Architecture

Rocky’s engine is a single Cargo workspace. It holds 25 crates under engine/crates/, plus two binary crates. Each crate does one job.

This page is the map of those crates. For the step-by-step walkthrough of a run, read Execution Flow.

How data flows

A rocky run touches every layer. From config file to warehouse result:

rocky.toml
    │  parse config + env-var substitution
    ▼
Config struct
    │  call DiscoveryAdapter
    ▼
Source metadata  ◀── Fivetran REST API / DuckDB info_schema / manual config
    │  compile .sql + .toml models
    ▼
rocky-compiler  →  diagnostics (E001–E036 / W001–W031)
    │  produces typed ProjectIr on success
    ▼
ProjectIr  (one ModelIr per model, fully typed and validated)
    │  topological sort → execution layers
    ▼
Layer 0: [raw_a, raw_b]   ← run in parallel
Layer 1: [enriched]
Layer 2: [summary]
    │  per model: drift detect → skip gate → SQL gen → execute → checks
    ▼
WarehouseAdapter
    ├── rocky-databricks  →  Databricks SQL Statement API
    ├── rocky-snowflake   →  Snowflake REST API
    ├── rocky-bigquery    →  BigQuery REST API
    ├── rocky-trino       →  Trino /v1/statement
    └── rocky-duckdb      →  DuckDB in-process
    │  write results to warehouse; update state store
    ▼
State store (redb, embedded)
    │  watermarks, run progress, partitions, idempotency
    ▼
Hooks + JSON output on stdout

For a detailed walkthrough of each step, see Execution Flow.

Adapter model

Rocky has two kinds of adapter. An adapter is the plug between the engine and one outside system.

Source adapters handle discovery: finding which schemas and tables exist and are available for processing. They do NOT extract data. The data must already be in the warehouse, landed by an ingestion tool (Fivetran, Airbyte, etc.) or loaded manually.

rocky-fivetran: calls the Fivetran REST API to list connectors and their enabled tables in the destination
rocky-duckdb: queries information_schema to discover schemas and tables in a local DuckDB database
Manual source (built into rocky-core): reads schema/table definitions from rocky.toml

Warehouse adapters handle execution: running SQL, managing catalog lifecycle, and applying governance (tags, permissions, workspace isolation).

rocky-databricks: executes via the Databricks SQL Statement API and manages Unity Catalog, with adaptive concurrency (AIMD: additive increase, multiplicative decrease)
rocky-snowflake: executes via the Snowflake REST API with OAuth, key-pair JWT, and password auth
rocky-bigquery: executes via the BigQuery REST API with service account / ADC auth
rocky-trino: executes via Trino’s /v1/statement REST polling state machine with HTTP Basic / JWT bearer auth (Beta as of engine v1.28.0)
rocky-duckdb: local in-process execution for development, testing, and CI

rocky-ir defines the typed Intermediate Representation, or IR. The IR is the typed graph the compiler builds before it writes any SQL: ModelIr, MaterializationStrategy, and the source and target descriptors. It was split out of rocky-core in engine v1.30.0, so an adapter crate can depend on the IR without pulling in the SQL-generation code.

rocky-core sits between the adapters and the IR. It owns the warehouse-agnostic logic: DAG resolution, schema pattern parsing, SQL generation templates, checks, contracts, and state management. It depends on rocky-ir. It knows nothing about Databricks, Fivetran, or any other named system.

That split is why a new warehouse (Snowflake, Trino) or a new source (Airbyte) needs a new adapter crate. The core engine does not change.

Monorepo

Rocky is a monorepo with five subprojects:

rocky-data/
├── engine/                  # Rust CLI + engine (this section)
├── sdk/python/              # rocky-sdk — standalone typed Python client over the rocky CLI
├── integrations/dagster/    # dagster-rocky Python package (thin adapter over rocky-sdk)
├── editors/vscode/          # VS Code extension (LSP client)
├── examples/playground/     # POC catalog + benchmarks
└── docs/                    # Documentation site (Astro + Starlight)

The playground catalog currently ships 101 POCs across 8 categories. See the playground guide for the full list.

Crate overview

The 25 crates under engine/crates/, and the two binary crates that ship as executables:

engine/
├── crates/
│   ├── rocky-ir/            # Typed Intermediate Representation (ModelIr, MaterializationStrategy)
│   ├── rocky-core/          # Generic SQL transformation engine (depends on rocky-ir)
│   ├── rocky-core-compiletest/  # Compile-fail (trybuild) guards for rocky-core
│   ├── rocky-catalog-core/  # Catalog client abstraction (Iceberg REST, Unity Catalog, Polaris, Nessie)
│   ├── rocky-sql/           # SQL parsing + typed AST
│   ├── rocky-lang/          # Rocky DSL parser (.rocky files)
│   ├── rocky-compiler/      # Type checking + semantic analysis
│   ├── rocky-adapter-sdk/   # Adapter SDK + conformance tests
│   ├── rocky-databricks/    # Databricks warehouse adapter
│   ├── rocky-snowflake/     # Snowflake warehouse adapter
│   ├── rocky-bigquery/      # BigQuery warehouse adapter (Beta)
│   ├── rocky-trino/         # Trino warehouse adapter (Beta as of v1.28.0)
│   ├── rocky-fivetran/      # Fivetran source adapter
│   ├── rocky-airbyte/       # Airbyte source adapter
│   ├── rocky-iceberg/       # Iceberg + content-addressed Delta UniForm writer
│   ├── rocky-duckdb/        # DuckDB local execution adapter
│   ├── rocky-engine/        # Local execution engine (DuckDB-backed)
│   ├── rocky-server/        # HTTP API + LSP server
│   ├── rocky-cache/         # Three-tier caching
│   ├── rocky-ai/            # AI intent layer
│   ├── rocky-mcp/           # MCP server backing `rocky mcp`
│   ├── rocky-verify/        # Standalone offline verifier for rocky-manifest attestations
│   ├── rocky-observe/       # Observability
│   ├── rocky-wasm/          # WebAssembly exports for browser/edge
│   └── rocky-cli/           # CLI framework + Dagster Pipes
├── rocky/                   # Binary crate (the `rocky` CLI)
└── rocky-lsp/               # Binary crate (the `rocky lsp` language server)

rocky-ir

The typed Intermediate Representation. It carries data only: no traits, no logic.

ModelIr: the single in-memory plan representation. It replaced the legacy Plan enum and the ReplicationPlan / TransformationPlan structs, which were deleted in v1.30.0.
MaterializationStrategy: the strategy enum. Full list in the MaterializationStrategy section below.
Source and target descriptors, and partition windows.

rocky-core

The warehouse-agnostic transformation engine. It reads IR from rocky-ir and writes SQL for one dialect.

Key modules:

schema.rs: configurable schema pattern parsing (e.g., src__acme__us_west__shopify into structured components)
drift.rs: schema drift detection (compares column types between source and target)
sql_gen.rs: IR to dialect-specific SQL generation
state.rs: embedded state store backed by redb for watermarks and run history
state_sync.rs: remote state persistence: download/upload state from S3, Valkey, or tiered (Valkey + S3)
catalog.rs: catalog and schema lifecycle management (CREATE IF NOT EXISTS, tagging)
checks.rs: inline data quality checks (row counts, column matching, freshness, null rate, custom)
contracts.rs: data contracts (required columns, protected columns, allowed type changes)
unified_dag.rs / dag_executor.rs: unified DAG construction and layered execution across pipeline stages (topological sort itself lives in rocky-ir::dag)
models.rs: SQL model loading (sidecar .sql + .toml files)
source.rs: source adapter traits and manual source configuration
config.rs: TOML configuration parsing with environment variable substitution (${VAR} and ${VAR:-default})

rocky-sql

SQL parsing and validation, built on sqlparser-rs. This crate reads SQL. The per-warehouse SQL that Rocky writes comes from each adapter crate’s own SqlDialect implementation.

parser.rs: wraps sqlparser-rs with typed extensions for Rocky’s needs
dialect.rs: the one sqlparser-rs dialect Rocky parses every model with, whatever warehouse that model targets. It follows Databricks SQL syntax and accepts SELECT * EXCEPT (...).
validation.rs: SQL identifier validation using strict regex patterns. All identifiers must pass through this module before being interpolated into SQL. This prevents SQL injection by rejecting anything that doesn’t match ^[a-zA-Z0-9_]+$.

rocky-databricks

The Databricks warehouse adapter. Implements the warehouse traits defined in rocky-core.

connector.rs: SQL Statement Execution REST API client (POST /api/2.0/sql/statements, polling for results)
catalog.rs: Unity Catalog CRUD operations, tagging, and catalog isolation
permissions.rs: GRANT/REVOKE execution, SHOW GRANTS parsing
workspace.rs: workspace binding management for catalog isolation
auth.rs: authentication with auto-detection: tries PAT (DATABRICKS_TOKEN) first, falls back to OAuth M2M (DATABRICKS_CLIENT_ID + DATABRICKS_CLIENT_SECRET)
batch.rs: batched information_schema queries using UNION ALL (batches of 200)

rocky-fivetran

The Fivetran source adapter. Discovers what schemas and tables exist in the Fivetran destination. This is a metadata-only operation; the actual data is already in the warehouse, landed by Fivetran’s sync process.

client.rs: async REST client using reqwest with Basic Auth
connector.rs: connector discovery and filtering
schema.rs: schema configuration parsing (nested JSON structures from Fivetran’s API)
pagination.rs: cursor-based pagination for large result sets
sync.rs: sync detection via timestamp comparison (determines if new data is available)

rocky-cache

Three-tier caching system that reduces API calls and speeds up repeated operations.

memory.rs: in-process LRU cache with configurable TTL
valkey.rs: Valkey/Redis distributed cache with distributed locks
tiered.rs: fallback chain: memory -> Valkey -> API. A cache miss at one tier populates all tiers above it.

rocky-duckdb

DuckDB local execution adapter. Minimal implementation providing a local warehouse backend for development and testing without requiring a Databricks connection.

rocky-observe

Observability infrastructure.

metrics.rs: in-process metrics collection: counters (tables processed/failed, statements executed, retries, anomalies) and duration histograms (p50/p95/max for tables and queries). Thread-safe via atomics, serialized to JSON in run output.
tracing_setup.rs: structured JSON logging via the tracing crate
events.rs: event broadcasting over Valkey Pub/Sub for real-time monitoring
otel.rs: feature-gated OpenTelemetry exporter. When the engine is built with --features otel and OTEL_EXPORTER_OTLP_ENDPOINT is set, rocky run exports in-process metrics as OTLP via an OtelGuard RAII handle. Send metrics to any OTLP-compatible collector (Honeycomb, Datadog, Grafana Tempo, Prometheus via the OTel collector).

rocky-lang

The Rocky DSL parser, for .rocky files. It turns pipeline-oriented syntax into an AST, then lowers that AST to a SQL string. From there the model takes the same path as hand-written SQL.

token.rs: the token scanner, built on the logos crate
parser.rs: a recursive descent parser producing a typed AST
lower.rs: lower_to_sql(), which lowers the AST to a SQL string

rocky-compiler

Type checking and semantic analysis for Rocky models.

typecheck.rs: column-level type inference across the DAG
semantic.rs: tracks column lineage, dependencies, and contracts
diagnostic.rs: compiler errors and warnings with source locations and suggestions

rocky-adapter-sdk

Stable, versioned traits for building custom warehouse adapters:

WarehouseAdapter: execute SQL, describe tables, manage catalog objects
SqlDialect: format SQL for a specific warehouse
DiscoveryAdapter: discover connectors and tables
GovernanceAdapter: manage tags, grants, workspace bindings
Includes 26 conformance tests (18 always-run + 8 capability-gated)

rocky-snowflake

Snowflake warehouse adapter.

auth.rs: OAuth, password, RS256 key-pair JWT auth (auto-detection)
connector.rs: Snowflake REST API client
dialect.rs: Snowflake SQL dialect (dynamic tables, multi-statement transactions)

rocky-trino

Trino warehouse adapter (Beta as of engine v1.28.0).

connector.rs: async REST client driving Trino’s POST /v1/statement + nextUri polling state machine. Same-origin guard on every nextUri follow-up (same scheme + host + port as the configured coordinator).
dialect.rs: double-quoted identifiers, three-part <catalog>.<schema>.<table> references, DESCRIBE <table> column introspection, ANSI INSERT INTO / CREATE TABLE AS, TABLESAMPLE BERNOULLI.
auth.rs: HTTP Basic + JWT bearer with RedactedString-wrapped credentials.
adapter.rs: WarehouseAdapter impl: dialect, execute_statement, execute_query, describe_table.

A Docker conformance harness is gated behind the trino-conformance cargo feature: cargo test -p rocky-trino --features trino-conformance -- --ignored drives the adapter end-to-end against a real trinodb/trino coordinator. The default cargo test -p rocky-trino run stays credential- and network-free.

v0 limitations: no MERGE (Trino’s MERGE is connector-dependent), no OAuth/Kerberos, no governance, no checksum-bisection support.

rocky-iceberg

Apache Iceberg + Delta UniForm writer surface. Powers materialization = "content_addressed" (engine v1.30.0, Phases 1–5):

discover() reads the bootstrap Delta commit and surfaces schema + partition spec + rowTracking config.
write_batch() emits content-addressed Parquet files (blake3-hashed file names) and a Delta log commit referencing them.
sync_iceberg_metadata() keeps Iceberg-compatible readers in sync after each commit.
Partitioned + rowTracking + post-ALTER schema evolution all supported.

See Content-Addressed Materialization for the user-facing model.

rocky-engine

Local execution engine backed by DuckDB (via rocky-duckdb). Powers rocky test, branching, and CI-style local runs without a warehouse connection.

rocky-server

HTTP API and Language Server Protocol (LSP) server.

REST API (via axum): model metadata, lineage, DAG endpoints for rocky serve
LSP (via tower-lsp): diagnostics, hover, completion, go-to-definition, rename for rocky lsp

rocky-ai

AI intent layer using Claude for model generation, intent extraction, schema change sync, and test generation.

Implements the compile-verify loop (up to 3 retries on compilation failure)
Requires ANTHROPIC_API_KEY

rocky-cli

CLI framework built on clap.

commands/: 80+ command implementations organized by category
output.rs: typed JSON output structs (80+ exported schemas) with JsonSchema derivation for codegen
pipes.rs: Dagster Pipes protocol emitter (activates when DAGSTER_PIPES_CONTEXT is set)

rocky (binary)

The rocky binary crate. Contains only main.rs, which wires all the library crates together and dispatches CLI commands.

Intermediate Representation (IR)

Rocky compiles your config and SQL into one typed intermediate representation before it writes any executable SQL. Types, lineage, and validation all live on that representation, so every later stage reads structure rather than re-parsing text. The IR lives in the rocky-ir crate.

Two front ends feed it, and they converge before the IR:

  .rocky DSL ──► lower_to_sql() ──┐
  (rocky-lang/lower.rs)           │
                                  ├──► SQL string + sidecar TOML
  raw .sql + .toml sidecar ───────┘             │
                                                │ to_model_ir()
                                                │ (rocky-core/models.rs)
                                                ▼
                                            ModelIr
                                   the one typed intermediate
                                                │ sql_gen.rs
                                                ▼
                                          dialect SQL

There is no second intermediate form between them. A .rocky model and a raw-SQL model both arrive at the IR as a SQL string plus its sidecar config. The DSL is a front end onto SQL, never a replacement for it.

ModelIr

ModelIr is the single in-memory plan representation. As of engine v1.30.0 the legacy Plan enum is gone, along with the ReplicationPlan / TransformationPlan / From<&Plan> bridge layer.

ModelIr carries the source and target descriptors, the materialization strategy, and whatever dependencies, sources, contracts, and checks the model declares. Replication and transformation pipelines compile to the same ModelIr shape. Which descriptors are populated is what tells the two kinds apart.

MaterializationStrategy

Decides how a model’s rows land in the target table. One strategy per model.

FullRefresh: CREATE OR REPLACE TABLE ... AS SELECT .... Rebuilds the entire table on every run.
Incremental: INSERT INTO ... SELECT ... WHERE ts > watermark. Only processes new rows. The watermark is read from the embedded state store at SQL-generation time (not carried on the strategy itself, so recipe-hash inputs stay runtime-state-free).
Merge: MERGE INTO ... USING (...) ON key WHEN MATCHED THEN UPDATE WHEN NOT MATCHED THEN INSERT. Upserts based on a unique key.
View: CREATE OR REPLACE VIEW ... AS SELECT .... No physical storage; every read re-executes the SELECT. Supported on every warehouse.
MaterializedView: CREATE OR REPLACE MATERIALIZED VIEW ... AS SELECT .... Databricks-specific.
DynamicTable: CREATE OR REPLACE DYNAMIC TABLE ... TARGET_LAG = '...' AS SELECT .... Snowflake-specific.
TimeInterval: partition-keyed materialization. The model SQL uses @start_date and @end_date placeholders; the runtime substitutes per-partition timestamps. Supports --partition, --from/--to, --latest, --missing CLI flags.
Ephemeral: not materialized; inlined as a CTE in downstream consumers.
DeleteInsert: delete rows matching the partition key, then insert fresh data. Cheaper alternative to Merge when the partition key identifies the rows being rewritten.
Microbatch: alias for TimeInterval with hour-granularity defaults; dbt-compatible naming.
ContentAddressed: content-addressed Parquet + Delta log commit via the rocky-iceberg writer. Designed for cross-engine reads (DuckDB iceberg_scan, Trino, Spark). SQL generation does not run for this strategy; the runtime drives rocky-iceberg::uniform_writer directly. See Content-Addressed Materialization.

The full enum lives in rocky-ir::ir::MaterializationStrategy. SQL generation (where applicable) lives in rocky-core::sql_gen.

Adapter SDK

The rocky-adapter-sdk crate holds the stable, versioned traits: WarehouseAdapter, SqlDialect, DiscoveryAdapter, and GovernanceAdapter. Each adapter ships an AdapterManifest declaring which optional features it supports.

Write an adapter in Rust by implementing the traits directly. Write one in any other language through the process adapter protocol: JSON-RPC over stdio.

Scaffold a new adapter:

rocky init-adapter bigquery

Run conformance tests:

rocky test-adapter --adapter duckdb