Embedding Rocky

Rocky is an engine you drive from your own software. Every command emits machine-readable JSON. Four transports carry the same typed payloads, so the transport you pick changes how your code reaches the data, not the data itself.

  caller                transport                          engine
  ─────────────────────────────────────────────────────────────────
  any language ───► spawn `rocky <verb> --output json` ──┐
  Python code  ───► RockyClient (rocky-sdk) ─────────────┤
  LLM agent    ───► `rocky mcp` over stdio ──────────────┼──► rocky
  your service ───► HTTP GET/POST /api/v1 ───────────────┘    engine
                    (served by `rocky serve`)

This guide covers all four patterns, how to choose between them, and the operational details that matter when you build Rocky into a larger system.

Choose an integration pattern

The four patterns differ in how your code calls Rocky and in what process lifecycle they assume.

Pattern	How you call it	Best fit
Subprocess	Run rocky <verb> --output json and parse stdout	Shell scripts, CI steps, any language without a Rocky binding
Python SDK	RockyClient from the rocky-sdk package	Python applications, notebooks, and orchestrators (Dagster builds on this)
MCP	rocky mcp over stdio	AI agents that inspect, author, or verify models
Serve API	rocky serve, then HTTP GET/POST /api/v1/...	Long-lived services that need a persistent read surface and async job submission

Reach for the subprocess pattern first. It needs nothing but the binary on $PATH. Move to the SDK when you write Python and want typed results. Use MCP when the caller is an LLM agent. Stand up the serve API when you need a process that stays warm across many requests. Use it too when you want to submit long-running work as a job and poll for it.

Subprocess

Run a command with --output json and read the payload from stdout. The exit code tells you success or failure. The JSON carries the detail.

rocky plan --output json --filter client=acme > plan.json

Every command’s JSON shape is a typed contract, not an ad-hoc dump. The field reference lives in JSON output. The rules that keep those shapes stable are in the JSON contract. Parse the payload, branch on the exit code, and you have a working integration in any language.

Python SDK

The rocky-sdk package wraps the subprocess pattern in a typed client. RockyClient runs the binary for you and parses each payload into a Pydantic model. You get autocompletion and validation instead of raw dictionaries.

from rocky_sdk import RockyClient

client = RockyClient(config_path="rocky.toml")
plan = client.plan(filter="client=acme")
print(plan.plan_id)          # content-addressed; pass to client.apply() to execute
result = client.apply(plan.plan_id)

A plan is a reviewable record of what a run will do, keyed by a plan_id. You build it, inspect it, then apply it.

Start with the SDK introduction for setup and the client surface. The recipes cover common tasks. The Dagster integration is a thin adapter over this same client, so anything you express with RockyClient maps onto assets and checks.

MCP

rocky mcp runs a Model Context Protocol server over stdio. It hands an LLM agent Rocky’s read and authoring tools: compile, plan preview, lineage, schema inspection, and the write-path tools that draft and apply models. The agent then works from real engine data instead of guessing.

See authoring with MCP for the tool surface, and operating Rocky with agents for the workflow an agent should follow. Use MCP when the caller reasons about your project in natural language. For deterministic automation, prefer the subprocess or SDK patterns.

Serve API

rocky serve starts an HTTP server that holds a compiled graph in memory and answers requests under /api/v1. Use it for a long-lived service. It compiles once at startup, serves reads from the warm graph, and runs mutating work as a background job.

rocky --config rocky.toml serve --port 8080

The read routes return the same payloads as the matching CLI command, byte for byte. GET /api/v1/models/{name}/lineage returns exactly what rocky lineage <name> --output json prints. GET /api/v1/compile matches rocky compile --output json. GET /api/v1/products matches rocky product list --output json, and GET /api/v1/products/{name} matches rocky product status <name> --output json, and GET /api/v1/products/{name}/journal matches rocky product journal <name> --output json. GET /api/v1/review/queue matches rocky review --queue --output json, except the two fields that derive from the request instant (staleness_seconds and score), and GET /api/v1/review/{plan_id}/status matches rocky review <plan-id> --status --output json. GET /api/v1/review/{plan_id} matches rocky review <plan-id> --output json: the plan’s kind and its breaking-change findings against HEAD, with approved always false. This read route never approves and never writes the sign-off marker. To approve, use POST /api/v1/jobs/approve (below) or rocky review <plan-id> --approve. The governor’s three reads match too: GET /api/v1/brief matches rocky brief --output json except generated_at and since_timestamp, and never advances the digest cursor, so its since defaults to 7d rather than the CLI’s last; GET /api/v1/audit/scorecard matches rocky audit --scorecard --output json except window_start; GET /api/v1/custody/{subject} matches rocky audit --for <subject> --output json; GET /api/v1/audit matches rocky audit --output json, and with ?product=<name> matches rocky audit --product <name> --output json. GET /api/v1/policy matches rocky policy show --output json: the rules in file order with the fields that decide an outcome, and the freezes in force that the read could see. It carries a rule’s scope, effect and verify_after, not its conditions, which the engine parses and never evaluates. It reads and never writes, so it cannot freeze or thaw, and for the same reason it does not download the authoritative ledger on a remote [state] backend, where that download would replace the local file. There it reports freeze_sources.ledger as local_mirror, so a caller can tell a complete answer from a possibly stale one. The window, grouping and product are query parameters, and the OpenAPI document lists their accepted values. A caller on the HTTP API and a caller on the SDK see identical data.

Six routes have no CLI command to match: GET /api/v1/health, /project, /models, /models/{name}, /dag/layers and /dag/status. They still return typed payloads with published schemas; GET /api/v1/meta lists the estate capability for the five estate routes and project for the project route. /project is what the retired dashboard at / used to show: the config directory’s name, the pipelines and adapters, the compiled model count, the diagnostics counts and the newest run. Model detail caps the SQL text at 256 KiB. A cut is reported in the body through sql_truncated and sql_bytes, so a reader never mistakes a cut text for the whole. Each type-checked column carries its structured data_type, which keeps a struct field’s own nullability, and a data_type_display label such as DECIMAL(10,2) for showing to a person.

Two reads that compute, and what they cost

Most read routes project state the engine already has. Two of them do work per request, so they carry bounds the others do not.

GET /api/v1/review/{plan_id} compiles the project twice, once at the working tree and once at HEAD, to find what a plan would break. On a project of a thousand models each compile is about 140 milliseconds. One diff runs at a time; a second waits about two seconds and is then refused with 503 and a Retry-After. The base is always HEAD and is not a parameter, because a ref from a query string would become an argument to git.

GET /api/v1/models/{name}/rows runs the model’s compiled SELECT against the warehouse and returns at most limit rows, with classification-tagged columns masked. It is the same code rocky preview rows runs, so the two cannot answer differently.

curl -H "Authorization: Bearer $TOKEN" \
     -H "X-Rocky-Allow-Warehouse: true" \
     "http://127.0.0.1:8080/api/v1/models/customer_orders/rows?limit=50"

Four things bound it:

• Consent, per request. A remote adapter needs the X-Rocky-Allow-Warehouse: true header on every call. Without it the answer is 403 warehouse_gated, before anything compiles. A local DuckDB adapter needs none. The header is not a formality: a GET is issued by things that are not the user, such as a browser prefetch or a restored tab, and none of those can set a custom header.
• A row cap. limit is 1 to 500. Anything else is 400.
• A timeout, and one at a time. The call is given 30 seconds, the compile included; past that the answer is 504 sample_timeout, and the warehouse may still be running the query. A second concurrent sample is refused at once with 503 and a Retry-After, rather than queued behind a call that might take the full 30 seconds. A compile that outlives the deadline finishes on its own and keeps the sample lane until it does, so the next sample may be refused 503 until then.
• No caching. The response carries Cache-Control: no-store, because the body is warehouse rows.

Masking is applied inline, and a sample that cannot be masked is refused rather than served. Two limits are worth knowing. Rocky can express a mask on Databricks, Snowflake and DuckDB; on any other adapter a model with a masked column is refused with 422 masking_unsupported_by_adapter, every time. And the hash strategy is an unsalted SHA-256, so a column drawn from a small or guessable set, an email address or a phone number, is recoverable from its digest. Treat a hashed column as pseudonymous, not anonymous.

Ad-hoc SQL is not exposed over HTTP. The CLI’s --sql-file has no route: a compiled model’s SELECT is a different thing from arbitrary SQL, whatever the token.

Mutations are jobs you poll

A mutating route does not block. You submit the work, get an id back, and poll for the result:

  POST /api/v1/jobs/run ─────► 202 Accepted  { "job_id": ... }
       /api/v1/jobs/plan                │
       /api/v1/jobs/apply               │
       /api/v1/jobs/approve             │
        ┌───────────────────────────────┘
        ▼
  GET /api/v1/jobs/{id} ──► running ───► poll again
                        ├─► succeeded ─┐
                        ├─► failed ────┤
                        └─► cancelled ─┴─► body embeds the canonical
                                           RunOutput or PlanOutput

The polled result is the same payload the CLI would have produced.

POST /api/v1/jobs/approve takes {"plan_id": "<64 lowercase hex>"}. It runs rocky review <plan_id> --approve as a subprocess and takes the same single-mutation lock as run and apply. While one of them runs, it answers 409 mutation_in_progress. /jobs/apply and /jobs/approve answer 400 invalid_plan_id for any other plan_id. The approval records the approver source http_api and the server’s git identity. If git config user.email cannot be resolved, or it is literally unknown, the job fails with approver_identity_unresolved. Job subprocesses carry ROCKY_SESSION_SOURCE=http_api, so run records show the session source http_api. That names the HTTP API as the source. It does not prove a browser made the call.

A submitted job becomes running right away. The schema also declares a queued state, but this server never uses it, because submissions never sit in a queue. Poll until the state is succeeded, failed or cancelled rather than matching on the full set.

POST /api/v1/jobs/{id}/cancel stops a job this server process started. It takes no body. The first request sends the job’s process group SIGTERM and answers 202 {"job_id": ..., "signal": "terminate"}. A run of a replication pipeline handles it like Ctrl-C: it finishes the copies in flight, saves their state and exits. Other jobs stop at once. Once a SIGTERM was delivered, the next request sends SIGKILL and answers "signal": "kill". On Windows every request kills the job. The route answers 202 only after the signal reached the job. The job then ends cancelled, or succeeded if it still exits 0. It keeps the mutation lane until its subprocess has exited. A finished job, or one that exited before the signal reached it, answers 409 job_not_running. A job this process did not start, such as a scheduled run, answers 409 job_not_cancellable. An unknown id answers 404 job_not_found. The browser UI guide lists what a cancelled job leaves behind.

There is no timeout route today. A mutating job holds the single mutation lane until its subprocess exits. While that lane is held, the next run or apply submission returns 409, so a hung job blocks the next one until it is cancelled. On restart the server reconciles its durable job ledger: it marks any job the previous process left running or queued as failed, with the error interrupted by engine restart. A client that polls for a terminal state therefore always terminates — a state this version does not recognise is reported as failed when the record is read, so polling ends either way.

Every failure carries an error envelope

Every route except GET /api/v1/health returns a structured error body on failure, never an empty response. The body carries a stable code, a human message, and an optional remediation_hint. Switch on the code in your own code, and show the hint to an operator.

Router-level failures use the same envelope. An unknown path answers 404 with code: "route_not_found". A known path called with the wrong method answers 405 with code: "method_not_allowed". One case has no envelope: a request the HTTP stack rejects before it reaches the router, such as malformed HTTP framing or a connection dropped mid-body.

The full route reference, request and response schemas, and status codes are published as a generated OpenAPI 3.1 document: openapi.json. It is generated from the same typed schemas that back the CLI, so it cannot drift from what the server returns. Load it into any OpenAPI tool to explore the surface or generate a client.

The browser UI

rocky serve --ui serves a browser UI at /ui/, from files built into the release binaries. The page signs in through GET /login and a session cookie, and holds no token. Embedders keep using Authorization: Bearer <token>. The UI files themselves are public. The browser UI guide covers sign-in, operator mode and the host and origin checks.

For the command flags, see rocky serve. For where the server sits in the engine, see the architecture overview.

Pinning and upgrades

CI holds the shape of every /api/v1 payload stable. The codegen-drift check fails any change that alters a payload’s shape without regenerating the committed schemas, so a shape change cannot land silently. New fields arrive in a backward-compatible way: they are serde-defaulted and optional. Your parser must tolerate unknown fields. Adding a field, without removing or retyping any other, does not change the API version.

CI guarantees shape, not values. The drift check compares the structure of each payload. It does not compare the specific values inside a primitive field. A string field stays a string across a minor release, but the exact string it carries is best-effort. Pin against shape, and expect value semantics to change within a minor version. A frozen-value corpus that would trip on any value change is planned, and is not in force yet.

Read GET /api/v1/meta to identify the engine you are talking to at runtime. It reports the engine version, the state-schema version, and a hash of the full schema set. It also reports token_scope ("full", "read_only", or null when no token is configured). It also reports a per-request hash of the resolved config, and the routes and capabilities this build serves. Feature-detect against capabilities and routes rather than parsing the version string. Compare schemas_hash between deployments to see whether any payload shape moved.

Recommended practice:

• Pin the engine to a minor version and read /api/v1/meta on startup to confirm the deployed build matches what you tested against.
• Tolerate unknown fields in every parser.
• Watch schemas_hash across upgrades; a change there is your signal to re-review payloads.
• A backward-incompatible reshape (a removed field, a rename, a type change) would arrive as a new API version, not as a silent change to /api/v1.

Running the API as a sidecar

Run rocky serve next to your application as a single-tenant sidecar. It is not a shared multi-tenant server.

Binding. The server binds to 127.0.0.1:8080 by default, loopback only. It refuses to bind a non-loopback host such as 0.0.0.0 unless you also configure a bearer token. An unauthenticated API therefore cannot leak model SQL and run history onto the network by accident.

Authentication. Auth is one optional shared-secret bearer token, passed with --token or the ROCKY_SERVE_TOKEN environment variable. When you set a token, the check runs on every request whose path is not exempt. Two paths are exempt, whatever the method: /api/v1/health, and the webhook /api/v1/hooks/trigger/{pipeline} — bearer-exempt because it authenticates with its own HMAC over the raw body (and, on a loopback bind with --scheduler and no secret, accepts an unsigned POST). A CORS preflight is answered before the check, and the browser UI’s own files under /ui/ are public. When you set no token, a loopback server asks no request for one. It still refuses a request whose Host does not name it (421) or whose Origin is another site (403), so a web page cannot reach it by DNS rebinding; a client that calls http://127.0.0.1:<port> or http://localhost:<port> is unaffected. Behind a reverse proxy that sends its own name as Host, pass that name with --allowed-host. This is a single secret, not a user system.

Token scope. A token reaches every route by default. --token-scope read-only (or --read-only, or ROCKY_SERVE_TOKEN_SCOPE) narrows it to GET, HEAD, and OPTIONS; every other method gets 403 forbidden_read_only_token. Hand that token to anything you do not fully trust with a warehouse write — a browser UI above all, where a leaked token would otherwise reach POST /api/v1/jobs/run. The rule is the HTTP method, not a list of paths, so routes added in future are covered as well.

The scope restricts the token. It is not a whole-server perimeter, and one gap is worth knowing before a browser reaches this sidecar: the webhook route authenticates itself and ignores the token. Run --scheduler on loopback without ROCKY_WEBHOOK_SECRET and it accepts an unsigned POST, then queues work for the scheduler. Script running in the page can do that with no token at all. Set ROCKY_WEBHOOK_SECRET. Put the sidecar behind your own gateway if you need per-user identity, TLS, rate limiting, or a public perimeter. The server does not terminate TLS. The X-Rocky-Principal header is recorded for audit only, and never authorizes anything.

Job principals and policy gating. A job submitted over HTTP runs rocky <verb> as a subprocess. That subprocess inherits the sidecar’s environment, and takes nothing from the request. The X-Rocky-Principal header does not set the principal the policy plane evaluates.

So if agent-scoped [policy] rules gate run and apply in your setup, set ROCKY_PRINCIPAL=agent in the sidecar’s environment when you launch it. Without that variable, jobs run as the default human principal and the agent-scoped rules never fire, whatever the header says. Run one sidecar per principal class if you need both: one with ROCKY_PRINCIPAL=agent for agent traffic, and a separate one for human-driven automation.

Multi-tenancy. One rocky serve process serves one project configuration. For several tenants, run one sidecar per tenant with that tenant’s config. Do not multiplex tenants through one server. One process per tenant keeps each tenant’s state store, compiled graph, and credentials apart.

One mutation at a time. The server admits one mutating job (run, apply or approve) per project at a time. A second mutating submission during that job returns 409 and puts the in-flight job’s id in running_job_id. Your client can then poll that job instead of colliding with it. A plan job mutates nothing and is never blocked. Reads stay available throughout. A read that finds the state store held by another process, a rocky run in a terminal for example, returns a retryable 503 engine_busy rather than an error.

Reads queue, one at a time. Every state-backed read (/runs, /schedule, /audit, /brief, /products, /review/queue, and the rest) opens the state store for the request, and the store’s file lock allows one open at a time. The server queues its own reads on one permit, so two clients refreshing together wait for each other instead of racing for the lock. Routes served from the compile result (/models, /dag/layers, /dag) never take the queue. The queue sets the ceiling of one process.

A store a read cannot serve. A read takes no write transaction against a store that holds anything, and never creates a table in one. A store written by a Rocky older than schema v22 lacks tables this server reads. rocky serve migrates such a store at startup — its job sweep opens the store read-write before the router serves — so after an upgrade the first read already answers. What can still happen is a store that changes under a running server (a file copied or restored over it), or a startup sweep that could not open the store; then every state-backed read answers 409 state_needs_migration, naming the missing tables and the stamp on disk, until one read-write open migrates it: restart rocky serve, or run any read-write command such as rocky run, once. It is not the retryable 503: retrying never migrates a store. Tell an operator. A project that has never run is not this case: its store is created empty on first read and answers “nothing yet”.

What one process carries

Measured with scripts/serve-ceiling.py on 2026-09-05, against the playground project, on an Apple M3 Pro: a staircase of concurrent clients, each looping over the reads the browser UI makes, 20 seconds per step.

Concurrent clients	Requests per second	p99 latency
1	90	20 ms
2	95	45 ms
4	97	69 ms
8	96	140 ms
16	97	274 ms

Zero errors at every step, and the process’s memory did not move. Throughput saturates near 95 store-backed reads per second on that machine, about 10 ms per read; latency then grows with the queue, about 15 ms per waiting request. A viewer costs its refresh rate times the reads per refresh: at one read per second per viewer, that machine carries about 90 viewers, and 8 viewers refreshing at the same instant wait 140 ms. The number is machine-specific by construction. Reproduce it on yours:

rocky serve --port 8420 --token T &
python3 scripts/serve-ceiling.py --url http://127.0.0.1:8420 --token T --pid $!

The script names any route the server cannot answer, and refuses to report a ceiling its own client was the limit of.

Treat the sidecar as part of your deployment: one process per project, bound to loopback or sitting behind your gateway, with the bearer token provisioned as a secret. The OpenAPI document describes everything your application needs.