Ingesting data

Bring external data into your organization's private schema so you can query it alongside the public oso.* tables and model on top of it. OSO supports three ingestion mechanisms — a REST API pipeline, a CSV upload, and a live connector — all driven by MCP mutations from your agent. Whichever you use, the cadence is the same: create a config, request a run, then query the materialized table.

Watch: connect a source in the app

The fastest way to see the whole flow is in the OSO app — no code. This walkthrough connects a Google Drive spreadsheet as a source and queries it end to end:

• 0:00 — start a new integration and pick Google Drive.
• 0:15 — import the spreadsheet, selecting all tabs.
• 0:28 — preview the source data while it processes.
• 0:49 — confirm the data landed.
• 1:03 — add a SQL cell and query it.

The rest of this page covers the same mechanisms in more depth, including the MCP tools for driving them from an agent.

First: does this data already exist?

Ingesting is the last resort, not the first move. Before you pull anything in, check whether the data is already queryable — it's cheaper and stays fresh automatically.

• Public data. Entities, events, and metrics for open source projects already live in oso.*. Browse the core models reference before assuming you need to ingest.
• Your org's datasets. From an agent over MCP, run ListDatasets to see what's already in your namespace and ListTablesForDataset to list a dataset's tables. Something an earlier run ingested may already be there.
• The marketplace. MarketplaceDatasets shows datasets published by others that you can subscribe to. subscribeToDataset grants access in place — no copy, no new ingestion, nothing to keep in sync.

If none of those cover the need, pick a mechanism below.

Choosing a mechanism

Source	Mechanism	Dataset type
A REST API (endpoints returning JSON)	Data ingestion pipeline	DATA_INGESTION
A CSV file (URL, local path, or export)	Static model	STATIC_MODEL
A live system — Google Sheets, BigQuery, Postgres	Connector	DATA_CONNECTION

Rule of thumb: use a connector when the source is a live system you want to keep syncing, a REST pipeline when the source is an API, and a CSV static model for a one-off file or a snapshot you already have on disk.

For BigQuery and Google Drive, the connect step is a point-and-click flow in the OSO web app — see Connect BigQuery and Connect Google Drive, both with screenshots. The rest of this page covers the agent-driven path over MCP.

REST API → data ingestion

Each API endpoint becomes a table. Nested JSON is flattened into child tables automatically.

First create a DATA_INGESTION dataset (or reuse one with GetDatasetByName), then configure the ingestion with createDataIngestionConfig. The config names the base URL and one resource per endpoint:

createDataIngestionConfig(input: {
  rest: {
    datasetId: "<dataset_id>",
    factoryType: "REST",
    config: {
      client: { base_url: "https://api.llama.fi" },
      resources: [
        { name: "chains", endpoint: { config_type: "simple", path: "/v2/chains" } }
      ]
    }
  }
})

For authenticated APIs, set client.auth and pass tokens as a secret marker — {"$type": "secret", "value": "<raw_value>"} — so the key is stored separately from the config. Paginated APIs take a client.paginator; most simple public APIs return everything in one response and need none.

Then request a run:

createDataIngestionRunRequest(datasetId: "<dataset_id>")

The mutation returns a run group — one run per node it dispatched. Poll the group's runs with runs(where: {"runGroupId": {"eq": "<run_group_id>"}}) until every one reaches a terminal status (REST ingestions usually finish in 1–3 minutes), then list the resulting tables with ListTablesForDataset and verify with execute_sql.

A success: false reply is not an error: the dataset resolved but held nothing runnable, and message says why.

CSV → static model

Use this for a file you have as a URL or on disk. A static model holds one uploaded file and materializes it into a table.

Create the model, get a pre-signed URL, upload the file to it, then request a run:

createStaticModel({ orgId, datasetId: "<dataset_id>", name: "gitcoin_grants" })
  → static_model_id
createStaticModelUploadUrl(staticModelId: "<static_model_id>") → upload_url

The upload URL is a short-lived pre-signed slot — upload immediately with a plain HTTP PUT:

curl -X PUT \
  -H "Content-Type: text/csv" \
  --data-binary @gitcoin_grants.csv \
  "<upload_url>"

Then materialize:

createStaticModelRunRequest({
  datasetId: "<dataset_id>",
  selectedModels: ["<static_model_id>"]
})

selectedModels must be the static model UUID, not its name — the pipeline looks for the uploaded file at {dataset_id}/{static_model_id}, so passing the name gives a "No tables found" error. Poll the run, then query the table at {org}.{dataset}.{model} to confirm the row count matches your file.

Google Sheets → data connection

A connector keeps a live source in sync rather than snapshotting it. Create the connection, then trigger a sync:

createDataConnection(...)  → connection_id
syncDataConnection(connectionId: "<connection_id>")

syncDataConnection pulls the current contents of the connected sheet into your schema; re-running it refreshes the data. Because the Google account link and file picker are OAuth flows, the practical way to set a Google Sheets connection up is the web app — follow Connect Google Drive, which walks through authorizing the account and selecting sheets and tabs. The same connector model backs BigQuery and Postgres.

The common cadence

Every mechanism follows the same three beats, only the tool names differ:

Step	REST	CSV	Connector
Create	createDataIngestionConfig	createStaticModel + upload	createDataConnection
Run	createDataIngestionRunRequest	createStaticModelRunRequest	syncDataConnection
Materialize	poll GetRun, then execute_sql	poll the run, then execute_sql	query once the sync completes

You can also trigger these runs from the Data Catalog in the app — open the dataset and start a run there instead of the MCP run request.

Once the table is queryable in your org's namespace, treat it exactly like any public table: query it, model on top of it, or pull it into a notebook. See Connect over MCP to point your agent at the server so it can run these mutations for you.