Skip to main content

Google Cloud

OSO self-manages its core data infrastructure on Google Cloud rather than relying on managed equivalents. All of our GCP resources — the GKE cluster, its node pools, the Dagster GCS bucket, archive nodes, and more — are defined as code with Terraform and applied as OpenTofu (tofu), a drop-in, open-source fork of Terraform that reads and writes the same state format. Nothing is clicked together in the console.

Applying changes: GitOps, not a CI pipeline​

There is no CI job that runs tofu apply. Instead, changes are applied the same way Flux applies Kubernetes manifests: a controller running inside the cluster continuously reconciles infrastructure toward what's declared in Git.

tofu-controller runs in the tofu-system namespace on the production GKE cluster (installed via the tofu-controller Flux Kustomization, ops/k8s-operators/gke/tofu-controller). Each piece of infrastructure it manages is represented by a Terraform custom resource, whose spec.sourceRef points at a Flux GitRepository (or OCIRepository) and spec.path at a module under ops/tf-modules/. The controller watches that source, runs tofu plan/apply against the module in-cluster, and writes the result back to the CR's status — no local tofu apply, and no separate GCP service account key: on GKE the tofu-system/tofu-controller KSA authenticates directly via Workload Identity Federation and is granted GCP IAM roles as itself.

The GitOps wiring for the warehouse cluster's own infrastructure (its node pools and the Dagster bucket) lives in ops/core-infrastructure/: a common/ directory holding the Terraform CRs, and a gke/ overlay that re-points each CR's sourceRef at the cluster's own GitRepository. This is the GitOps replacement for what used to be a single, standalone tofu apply run by hand against the monolithic warehouse-cluster module.

Modules (ops/tf-modules/)​

Each module owns one narrow slice of infrastructure so it can be reconciled (and, if needed, destroyed) independently:

  • warehouse-cluster-core — the VPC, the GKE control plane, the warehouse-node service account, and a 0-node placeholder pool (GKE's cluster module requires at least one inline pool, so this one exists purely to satisfy that and runs no real workloads).
  • warehouse-node-pool — a single, fully-parameterized node pool. Every real pool (standard, spot, preemptible, Trino coordinator/worker, consumer-Trino coordinator/worker, MCS scheduler/worker, and the default pool) is one instance of this module, reconciled as its own Terraform CR.
  • warehouse-dagster-bucket — the GCS bucket Dagster uses for intermediate storage, plus IAM bindings for whichever principals need to read/write it.
  • warehouse-cluster — the original, monolithic module that used to provision the cluster and all of its node pools together. Deprecated in favor of the three modules above; kept functional only for consumers that haven't migrated yet.
  • warehouse — the data warehouse's own GCP resources: BigQuery, CloudSQL, and GCS.
  • archive-nodes — GCE VMs and disks for blockchain archive nodes.
  • vector-search — a Vertex AI Vector Search index and endpoint.
  • k3d-ci-node / k3d-ci-image-build — an ephemeral, Spot-VM-backed k3d cluster for CI, and the Packer pipeline that bakes its base image. See the k3d-ci-node README for how to exercise a module end-to-end against a local k3d cluster before it ever touches production.
  • gce-dev-vm / test-project — developer sandbox VMs and projects.

State​

Each module gets its own OpenTofu state, stored as a GCS object in the oso-private-tf-us bucket (one prefix per resource, e.g. oso/production/warehouse-nodepool-default). This is a deliberate move away from one shared state file per environment: a bug or a stuck lock in one resource's state can no longer block reconciliation of every other resource. The migration off the original monolithic state moved each resource's state entry with zero underlying resource churn (nothing in GKE or GCS was destroyed or recreated). It was a one-time job, and the tooling that drove it has since been removed. To bring an existing GCP resource under a new module's state today, declare it in an import {} block and let the CR's plan adopt it — see ops/clusters/warehouse-bootstrap/DR.md for the procedure.

Local testing​

You can exercise a module end-to-end with tofu-controller on a local k3d cluster instead of the production GKE cluster:

oso ops k3d-setup --with-tofu-controller

This is opt-in (a bare oso ops k3d-setup skips it) since it requires a GCP service account key Secret. See the k3d-ci-node README for the full walkthrough, including how to inspect the resulting Terraform CR and tear it down again.

What runs on top​

The infrastructure above is only the substrate. What actually runs on the GKE cluster — Trino, Nessie, Dagster, the backend API, the scheduler, and everything else in ops/k8s-apps/ — is deployed separately, as plain Kubernetes manifests reconciled by Flux rather than as Terraform. See the Ops Overview for how that half fits together, and the Dagster Playbook for day-to-day tasks like restating SQLMesh models and managing the Nessie consumer tag.