Skip to main content

Technical Architecture

info

OSO's goal is to make it simple to contribute by providing an automatically deployed data pipeline so that the community can build this open data warehouse together. All of the code for this architecture is available to view/copy/redeploy from the OSO Monorepo.

Pipeline Overview​

OSO maintains an ETL data pipeline that is continuously deployed from our monorepo and regularly indexes all available event data about projects in the oss-directory.

There are two complementary ways data gets into OSO, and both land in the same data lakehouse:

  • Core pipeline: a fixed ETL pipeline that we maintain, indexing public data sources.
    • Extract: raw event data from a variety of public data sources (e.g., GitHub, blockchains, npm, Open Collective)
    • Transform: the raw data into impact metrics and impact vectors per project (e.g., # of active developers)
    • Load: the results into our data lakehouse
  • Self-serve platform: community members can connect their own data sources, author their own data models (in SQL or sandboxed Python), and upload datasets, all executed asynchronously and landed in the same lakehouse alongside the core pipeline's data.

The results are then served through our API, website, pyoso, and other data products.

Major Components​

The architecture has the following major components.

Data Orchestration​

Dagster orchestrates our core metrics pipeline, from data ingestion (e.g. via dlt connectors) to the sqlmesh transformations described below.

You can see our public Dagster dashboard at https://dagster.oso.xyz/.

Self-Serve Data Platform​

Alongside the core pipeline, OSO runs a scheduler: an asynchronous job system that powers community-driven, self-serve data. It's what executes work whenever someone connects a new data source, defines a custom data model, uploads a dataset, or runs an ad hoc SQL query through the API.

Data models can be written in SQL, or as sandboxed Python models (we call these "UDMs", or User-Defined Models). UDMs run in an isolated environment with no network access of their own — any query or outbound fetch a model makes is mediated and access-controlled, so community-contributed code can run safely alongside the rest of the platform. Both SQL and Python models land their output in the same Iceberg tables the core pipeline writes to.

Data Lakehouse​

Currently all data is stored in managed Iceberg tables.

We also make heavy use of public datasets from Google BigQuery. To see all BigQuery datasets that you can subscribe to, check out our Data Overview.

sqlmesh pipeline​

We use a sqlmesh pipeline to clean and normalize the data into a universal event table and metrics. You can read more about our event model here.

Trino clusters​

We maintain separate Trino clusters the operate over the Iceberg tables:

  • Production pipeline cluster - a read-write cluster to run the sqlmesh pipeline
  • Consumer query cluster - a read-only cluster to serve the API and pyoso

Backend API​

A GraphQL and REST API server fronts the data platform, backed by Postgres. It serves the OSO website, our MCP server, and other clients, and is also where self-serve platform requests (like creating a data model or a data connection) come in.

OSO Website​

The OSO website is served at https://www.oso.xyz. This website provides an easy to use public view into the data. We currently use Next.js hosted by Vercel, talking to the backend API described above.

Open Architecture for Open Source Data​

The architecture is designed to be fully open to maximize open source collaboration. With contributions and guidance from the community, we want OSO to evolve as we better understand what impact looks like in different domains.

All code is open source in our monorepo. All data, including every stage in our pipeline, is publicly available via pyoso. All data orchestration is visible in our public Dagster dashboard.

You can read more about our open philosophy on our blog.