Configuration

Rocky reads one rocky.toml file for everything. You name your adapters ([adapter.NAME]) and your pipelines ([pipeline.NAME]), so one file can hold several sources, several warehouses, and several pipelines side by side.

Config at a glance

A key that does not exist is usually an error, not a no-op

A typo (tooken for token, retires for retries) fails the config load in most blocks, and the message names the key. In the lenient blocks a clean rocky validate is not evidence that Rocky read your key. Check every key against this page.

Block	What happens to an unknown key
[pipeline.NAME.source], [pipeline.NAME.target], [pipeline.NAME.checks.quarantine]	Refused. The error names the key.
Directly under [pipeline.NAME]	Accepted and ignored, on every pipeline type. strategyy = "x" loads and rocky validate reports the config valid.
A check toggle such as [pipeline.NAME.checks.row_count], and an [[pipeline.NAME.checks.assertions]] entry	Ignored. severty = "warning" is dropped.
[hook.*]	An unknown event key is ignored silently.
[adapter.NAME.extra]	Every key passes through untouched.

One row per top-level block:

Block	What it does
[adapter.NAME]	One named connection to a warehouse or a source.
[pipeline.NAME]	One unit of work: source, target, checks, execution, schedule.
[state]	Where watermarks and run history live between runs.
[run]	Skip models that did not change. Off by default.
[reuse]	Reuse work for content-addressed models.
[resilience]	Retry a model that failed for a transient reason. On by default.
[retry]	One shared connector retry budget for the run.
[budget]	Limits on cost, time and bytes scanned per run.
[cost]	Warehouse prices for rocky optimize.
[cache]	The schema cache for typechecking.
[freshness]	Project-wide staleness budget.
[schedule]	Project defaults for [pipeline.NAME.schedule].
[contracts]	Strict handling of contract columns.
[imports.NAME]	Check models against another team’s published schema.
[lint], [portability]	Style rules and cross-warehouse SQL lint.
[mask], [classifications], [role.NAME]	Masking, classification tags and role inheritance.
[policy]	Who may change what, enforced at mutating steps.
[branch], [gc]	Branch promotion approval and rocky gc limits.
[selectors]	Saved --select expressions.
[ai], [fulfill]	Token cap for rocky ai*; agent driver for rocky fulfill (experimental).
[hook.*]	Shell commands and webhooks on pipeline events.
[schema_evolution]	Removed. A config that declares it fails to load. See below.

A minimal full config. It runs a DuckDB replication with no credentials:

[adapter.local]
type = "duckdb"
path = "warehouse.duckdb"

[pipeline.replication]
type = "replication"
strategy = "full_refresh"

[pipeline.replication.source]
adapter = "local"

[pipeline.replication.source.schema_pattern]
prefix = "raw__"
separator = "__"
components = ["source"]

[pipeline.replication.target]
adapter = "local"
catalog_template = "warehouse"
schema_template = "analytics"

[state]
backend = "local"

A config can declare more adapters and pipelines. A pipeline picks its adapters with adapter = "..." on source and target.

Config Inference and Defaults

Write less by leaning on the defaults. Rocky fills in most fields, so a small project needs only a handful of lines.

What	Default	When to omit
pipeline.type	"replication"	Always (unless using a different type)
Unnamed [adapter] with a type key	Auto-wraps as adapter.default. Then source.adapter and target.adapter can be left out.	Single-adapter projects
Pipeline adapter refs	"default"	When only one adapter is defined
[state]\nbackend = "local"	"local"	Local development (always the default)
auto_create_catalogs / auto_create_schemas	false	When you don’t need auto-creation
Model sidecar name	Filename stem	When file is fct_orders.toml and name is fct_orders
Model sidecar target.table	Same as name	When table name matches model name
models/_defaults.toml	N/A	Provides directory-level [target] defaults for catalog and schema

Directory-level defaults

Create a models/_defaults.toml so you do not repeat [target] in every model:

models/_defaults.toml

[target]
catalog = "analytics"
schema = "warehouse"

Each model inherits these values and overrides only what differs.

Environment Variables

Keep secrets out of the file. Write ${VAR_NAME} anywhere in the config and Rocky substitutes the environment variable’s value at parse time, before it reads the TOML.

[adapter.prod]
type = "databricks"
host = "${DATABRICKS_HOST}"
token = "${DATABRICKS_TOKEN}"

If a referenced variable is not set, Rocky returns a parse error listing the missing variable.

The same substitution runs over every TOML config Rocky loads, not just rocky.toml:

• Per-model sidecars (models/<name>.toml): useful for orchestrator-injected [target] overrides.
• models/_defaults.toml: directory-level defaults applied to every sibling sidecar.
• Inline ---toml frontmatter in .sql / .rocky files: only the frontmatter block is substituted. The SQL body below the closing --- is left untouched, so ${VAR} in SQL stays literal.

# models/customer_facts.toml — sidecar example
[target]
catalog = "${ROCKY_TARGET_CATALOG:-warehouse}"
schema  = "${ROCKY_TARGET_SCHEMA:-marts}"
table   = "${ROCKY_TABLE_OVERRIDE:-customer_facts}"

A worked example covering all three layers lives in examples/playground/pocs/00-foundations/07-config-layering/.

Where a resolved value is shown

Rocky prints the placeholder, not the value. Some fields print ${DATABRICKS_TOKEN}, never the token:

Surface	What prints ${NAME}
rocky serve responses	Every value of 8 bytes or more, anywhere in the response.
rocky policy show, text and --output json	Policy scope (models, tags, classifications, exclude_classifications, layer), verify_after and autonomy_budget.window.
rocky policy check	The model’s tags, classifications and layer.
rocky brief	A degraded rule’s budget window.
rocky compile	models_detail (strategy, freshness, tags, depends_on, name) and the diagnostics text.
rocky dag	Each model node’s strategy, freshness and partition_shape.
rocky list models, list adapters, list sources	A model’s depends_on; an adapter’s host; a source’s catalog, schema_prefix and components.
rocky catalog	An asset’s intent and column descriptions.
rocky validate	A pipeline’s strategy, catalog_template and schema_template, and the text of every message.
rocky doctor	Every check message, detail and suggestion.
rocky hooks list, rocky hooks test	A hook’s command, and the abort reason (which quotes a command or webhook URL).
rocky schedule status	timezone, and each pipeline’s cron, after and config error.
rocky audit	A decision’s model and reason, and a verification’s checks and reason.
The policy-decision ledger	A verify_after check name and its outcome text are stored as ${NAME}, so a later reader never needs the value.
rocky plan (replication) plan files	config_snapshot holds ${NAME}. A keyed digest of each config section is stored beside it, so rocky apply still refuses when an environment value changed. The key is .rocky/plan-digest.key; without it, apply refuses.
Config errors and model-file errors	Every value of 8 bytes or more that the error quotes, including a TOML parse error in a sidecar, _defaults.toml or frontmatter.
Adapter connection fields	The DuckDB path; the Databricks host, http_path and client_id; the Snowflake account, warehouse, username, private_key_path, role and database; the BigQuery project_id and location; the Fivetran destination_id. Rocky holds these with their placeholder, so a log line or error that quotes one (for example a DuckDB file that fails to open) prints ${NAME} when the value is 8 bytes or more (a shorter one, such as a location of US, prints as itself). Only the adapter connect path reads the value.

A value shorter than 8 bytes is not treated as a secret and prints as itself. The literal in ${VAR:-default} is already in the file, so it prints as itself too. A value inside a TOML basic string that the parser unescapes (for example \u0041 or \t) is matched in its unescaped form too.

Rocky matches the value, not the field. So a value of 8 bytes or more is rewritten wherever it appears in the outputs above, including inside a longer string: with BUCKET=analytics_prod, a hook command upload.sh s3://analytics_prod/out prints as upload.sh s3://${BUCKET}/out.

Some outputs still show a resolved value:

• Target coordinates, on purpose. A model’s or pipeline’s target catalog, schema and table print resolved wherever they appear as a target: rocky compile and rocky dag target, rocky list models target, rocky catalog fqn, rocky lineage target_schema, rocky ci-diff resolved_target and rocky history --audit target_catalog. They print exactly as rocky run prints them, because dagster-rocky matches the asset key it builds from rocky compile’s target against rocky run’s asset_key. A placeholder on one side and the value on the other would silently drop materializations. A target prints resolved even when the same value is also used as a secret elsewhere in the config.
• SQL that Rocky generates, in rocky plan, rocky run, rocky preview and their previews. It names the resolved catalog, schema and table.
• rocky run and rocky discover results, including each materialization’s asset_key. These carry the resolved target.
• The config_identity a governed plan stores in its plan file.
• rocky docs HTML.
• Log lines.

Do not put a secret in a field these outputs carry. In particular, never put a secret in a target catalog, schema or table name.

Default Values

Use ${VAR_NAME:-default} to provide a fallback when a variable is unset or empty:

backend = "${ROCKY_STATE_BACKEND:-local}"

---

[adapter.NAME]

Declare a connection once, then reference it by name from any number of pipelines. Each [adapter.NAME] block is one connection. You choose the name; type selects which adapter implementation handles it.

Field	Type	Required	Description
type	string	Yes	Adapter type. One of "databricks", "snowflake", "duckdb", "bigquery", "trino", "postgres", "redshift", "clickhouse", "sqlserver", "spark", "fivetran", "airbyte", "iceberg", "manual". An unrecognized value is a hard error.
kind	"data" | "discovery"	See description	The role of this block. "discovery" is required for the discovery-only types: fivetran, airbyte, iceberg and manual. Leave it out for databricks, snowflake, postgres, redshift, clickhouse, sqlserver and spark, which move data only. For duckdb and bigquery, which can do both, leaving it out registers both roles. Rocky does not check kind for trino.
retry	table	No	Retry policy (see [adapter.NAME.retry]).
extra	table	No	Escape hatch for adapter-specific keys Rocky’s typed config doesn’t model (see below).

The remaining fields depend on the adapter type.

Rocky validates the top-level adapter fields strictly. An unrecognized key — a typo like tooken — is a hard error, not a silent no-op. Put keys that a custom or process adapter needs but Rocky does not model in a nested [adapter.NAME.extra] table, which passes through untouched:

[adapter.my_warehouse]
type = "trino"

[adapter.my_warehouse.extra]
default_schema = "analytics"
x_custom_header = "service-account"

Per-adapter setup

The connection fields, authentication, and examples for each adapter type live on their own page:

• DuckDB — local in-process execution
• Databricks — SQL warehouse + Unity Catalog governance
• Snowflake — PAT, OAuth, key-pair, and password auth
• BigQuery — project/location plus environment-supplied credentials
• PostgreSQL — host, database and role, with libpq-style sslmode
• SQL Server — SQL Server; Azure SQL and Fabric Warehouse are Beta. SQL auth or Entra ID
• Redshift (Beta) — the PostgreSQL adapter’s fields plus dist/sort keys and late-binding views
• ClickHouse (Beta) — HTTP interface with user/password and TLS, plus table engine and sort keys
• Spark (Beta) — Spark Connect with an optional bearer token, on Delta Lake or Iceberg tables
• Fivetran — metadata-only source discovery

Three types have no page yet. They read these top-level fields:

type	Role	Fields
"trino"	Warehouse	host (coordinator URL, required). token for JWT bearer auth, else username + password for Basic auth. username also names the Trino user. database sets the default catalog. timeout_secs defaults to 300.
"airbyte"	Discovery	host (Airbyte API URL, required). token (optional bearer token).
"iceberg"	Discovery	host (REST catalog URL, required). token (optional bearer token).

type = "manual"

A discovery adapter that lists source schemas and tables inline in rocky.toml instead of discovering them from an API. Use it when the source has no discovery adapter of its own, such as a Databricks or Snowflake source with no Fivetran in front of it, for tests, and for small sources whose shape does not change.

It is discovery-only, so kind = "discovery" is required. List each schema as an [[adapter.NAME.schemas]] block:

Field	Type	Required	Description
schemas	array of tables	Yes	The source schemas this adapter returns. At least one.
schemas[].name	string	Yes	Schema name, matched against the pipeline’s schema_pattern.prefix like any discovered schema.
schemas[].tables	array of strings	Yes	Tables in the schema. At least one.

[adapter.local_discovery]
type = "manual"
kind = "discovery"

[[adapter.local_discovery.schemas]]
name = "raw__orders"
tables = ["orders", "order_items"]

[pipeline.poc.source.discovery]
adapter = "local_discovery"

The adapter discovers exactly what it lists, with no network call. It does not check that the tables exist. rocky plan plans every listed table, so a missing table fails at rocky run. rocky discover is different: when the pipeline sets source.catalog, it asks the source warehouse which tables exist and leaves out the missing ones, as it does for every discovery adapter.

A listed schema is used only when the pipeline’s schema_pattern parses it: the name must start with prefix (case-sensitive) and the rest must split into the declared components. Other listed schemas are skipped.

rocky validate reports each of these as a V057 error, and every command that loads the config (plan, run, discover) refuses it with the same message:

• a manual adapter with no schemas;
• a schema with no tables;
• a schema or table name that is not a plain identifier ([a-zA-Z0-9_]+);
• a schema listed twice, or a table listed twice in one schema (compared ignoring case);
• schemas on an adapter whose type is not manual.

When the schema_pattern of a pipeline that uses the adapter parses none of the listed schemas, rocky validate warns with V058: plan and run would find no table.

[adapter.NAME.retry]

Decide how hard this adapter tries again after a transient error: an HTTP 429 or 503, a rate limit, a timeout. Rocky backs off exponentially between attempts, with jitter by default.

Field	Type	Default	Description
max_retries	integer	3	Maximum retry attempts. Set to 0 to disable.
initial_backoff_ms	integer	1000	Initial backoff in milliseconds.
max_backoff_ms	integer	30000	Maximum backoff cap in milliseconds.
backoff_multiplier	float	2.0	Multiplier applied after each retry.
jitter	bool	true	Add random jitter to prevent thundering herd.
circuit_breaker_threshold	integer	5	Trip after this many consecutive failures. Set to 0 to disable.
circuit_breaker_recovery_timeout_secs	integer	null	When set, the breaker auto-recovers after this many seconds: it enters half-open, admits a single trial request, and either closes on success or re-opens on failure. When unset, a tripped breaker stays tripped for the rest of the run (manual-reset behaviour).
max_retries_per_run	integer	null	Per-adapter cross-statement retry budget for a single run. Use the top-level [retry] block when you want one shared budget across every adapter instead.

When the breaker trips, Rocky emits a circuit_breaker_tripped pipeline event; on auto-recovery it emits circuit_breaker_recovered. Hook subscribers can observe both without polling the adapter. See Hooks.

---

[pipeline.NAME]

Declare a unit of work: what it reads, where it writes, and how. Each [pipeline.NAME] block is one pipeline. You choose the name, and every command takes --pipeline NAME to select one when the config declares more than one.

Field	Type	Required	Default	Description
type	string	No	"replication"	Pipeline type. One of "replication", "transformation", "quality", "snapshot", "load". The remaining fields depend on the type; the fields below apply to "replication" (the default).
strategy	string	No	"incremental"	Replication strategy. The runner recognizes "incremental", "full_refresh", "merge", "view", "materialized_view" and "dynamic_table". "merge" needs merge_keys (or merge_keys_fallback); without them the config load fails. "dynamic_table" is recognized but not usable here: the pipeline block exposes no target_lag, so a run errors and asks you to declare it on a transformation model’s sidecar instead. Any other value is not an error. rocky validate warns (V035) and the run silently falls back to full_refresh.
timestamp_column	string	No	"_fivetran_synced"	Watermark column for incremental strategy.
metadata_columns	list	No	[]	Extra columns to add to copied data (see below).

[pipeline.bronze]
type = "replication"
strategy = "incremental"
timestamp_column = "_fivetran_synced"
metadata_columns = [
    { name = "_loaded_by", type = "STRING", value = "NULL" },
    { name = "_loaded_at", type = "TIMESTAMP", value = "CURRENT_TIMESTAMP()" },
]

Rocky inserts value into the copy statement as a SQL expression. Write "NULL" for a null and "CURRENT_TIMESTAMP()" for a computed value. Rocky checks the text when the config loads:

• A value must be one expression.
• Any function it calls must be on Rocky’s scalar-function allowlist. A function not on the list is refused, and the message names the list.
• A placeholder in a value must be quoted. value = "'{tenant}'" is a string literal and parses. value = "{tenant}" is not valid SQL, so the config is refused at load with a parse error.
• This applies to value only, because value is SQL. A placeholder in a name template stays unquoted: schema_template = "stage__{source}".

value = "CAST('x' AS VARCHAR)"       # passes
value = "CONCAT('{tenant}', '_', '{source}')"   # passes
value = "my_udf(1)"                  # refused: not on the allowlist

The check parses against the pipeline’s target adapter, not its source, since the target is the warehouse that receives the expression. DuckDB, Snowflake, BigQuery and Databricks each parse under their own dialect. Trino has none of its own in Rocky’s parser, so a Trino target parses under a generic dialect, which accepts more than those four share. A value that passes here can still be rejected by Trino itself. The allowlist applies either way.

[pipeline.NAME.source]

Point the pipeline at the system it reads from.

Field	Type	Required	Description
adapter	string	Yes	Name of the adapter that reads the source rows. Must match a [adapter.NAME] key, and that adapter must move data. A discovery-only adapter such as Fivetran goes in source.discovery instead.
catalog	string	No	Source catalog name (used by warehouse-resident sources like Databricks).

Fivetran lands its tables in your warehouse. The warehouse adapter reads the rows, and the Fivetran adapter (kind = "discovery") only lists what exists. Set it in source.discovery.

[pipeline.NAME.source.discovery]

Use a different adapter to list schemas and tables than the one that reads the rows. Reach for this when one system knows what exists (DuckDB, say) while the data itself lives elsewhere. Omit the block and Rocky discovers through the source adapter.

Field	Type	Required	Default	Description
adapter	string	Yes	—	Adapter name to use for discovery.
report_new_sources	bool	No	false	Diff the discovered source inventory against the prior persisted snapshot and report first-seen schemas in the discover output’s new_sources. The first discover of a pipeline records the baseline and reports nothing. Off by default — the diff and its state write only happen when opted in.
on_collision	"off" | "warn" | "error"	No	"off"	Cross-source collision detection. When the same external object (e.g. an ad account) is onboarded under two schemas, its data lands in two target tables and silently doubles any downstream UNION ALL. warn reports the pairs in the discover output’s collision_candidates and emits a source_collision_detected event; error additionally fails the discover so a colliding onboard can’t silently create a catalog. off (default) skips detection entirely. Only adapters that resolve external object ids (e.g. Fivetran) participate; others contribute nothing.

See Cross-source duplicate detection for the counterpart that runs during replication rather than at discovery.

[pipeline.NAME.source.schema_pattern]

Teach Rocky to read structure out of your source schema names. It can then route each source to the right target, and you can select sources with --filter. A schema like src__acme__us_west__shopify becomes a set of named components.

Field	Type	Required	Description
prefix	string	Yes	Prefix that identifies managed schemas (e.g., "src__").
separator	string	Yes	Separator between components (e.g., "__"). Letters, digits, _, . and - only.
components	list of strings	Yes	Ordered list of component names. A trailing "..." marks a component as multi-valued.

[pipeline.bronze.source.schema_pattern]
prefix = "src__"
separator = "__"
components = ["client", "regions...", "connector"]

Given src__acme__us_west__us_east__shopify, this pattern extracts:

Component	Value
client	"acme"
regions	["us_west", "us_east"]
connector	"shopify"

[pipeline.NAME.target]

Say where the pipeline writes. The two templates build the catalog and schema name per source, filling {component} placeholders from the components schema_pattern parsed above.

Field	Type	Required	Description
adapter	string	Yes	Name of the warehouse adapter. Must match a [adapter.NAME] key.
catalog_template	string	Yes	Template for the target catalog name. Uses {component} placeholders (or {component:SEP} to pin the join separator for variadic components).
schema_template	string	Yes	Template for the target schema name. Uses {component} placeholders (or {component:SEP} to pin the join separator for variadic components).

A template without a placeholder (warehouse) is static. The two blocks work as one path. schema_pattern takes a source schema apart, and the target templates put a destination name back together from the pieces:

  source schema                  parsed components        target name
  ─────────────                  ─────────────────        ───────────
                    schema_pattern                template
  src__acme__shopify ──────────► client  = acme  ─────────► warehouse
                       parse     source  = shopify  fill    stage__shopify
                                                    │
   prefix "src__"                                   └── catalog_template
   separator "__"                                       = "warehouse"
   components ["client","source"]                       schema_template
                                                        = "stage__{source}"

For per-tenant catalogs, put a placeholder in catalog_template too: catalog_template = "{tenant}_warehouse" with components = ["tenant", "regions...", "source"]. See Schema Patterns for the full pattern reference.

One name clash to know about. This schema_template routes replication targets from parsed source schemas. The config-group schema_template is a different feature: it fills its placeholders from a model’s [args] to route a fan-out of transformation models. See Config groups.

[pipeline.NAME.target.governance]

Have Rocky create and label the catalogs and schemas it writes to, instead of provisioning them by hand. The block covers four things: catalog and schema creation, tags, grants, and workspace isolation.

Support depends on the target adapter. The two auto_create_* flags work on every adapter that emits CREATE SCHEMA SQL.

Target	tags	grants / schema_grants	isolation
Databricks	Yes (Unity Catalog)	Yes	Yes
Snowflake	Yes	Yes	No
BigQuery	Schema and table labels only. A catalog (project) label logs a warning.	No: logs a warning. Use Google Cloud IAM.	No
Other adapters	No	No	No

Where a row says No, the call does nothing and the run continues. Catalog tags, grants and isolation run only with auto_create_catalogs = true. Schema tags and schema_grants run only with auto_create_schemas = true. Rocky adds grants and never revokes one. See Permissions.

Field	Type	Default	Description
auto_create_catalogs	bool	false	Create target catalogs if they do not exist.
auto_create_schemas	bool	false	Create target schemas if they do not exist. Honored on both replication and transformation pipeline targets.
tags	table	{}	Tags applied to managed catalogs, schemas, and tables.
grants	list	[]	Catalog-level grants. Each entry has principal (string) and permissions (list of strings).
schema_grants	list	[]	Schema-level grants. Same format as grants.
isolation	table		Workspace isolation settings (see below).

The Full Example shows a complete governance block.

[pipeline.NAME.target.governance.isolation]

Restrict which Databricks workspaces can reach the catalogs Rocky manages. Rocky binds each managed catalog to the workspaces you list, and can set the catalog’s isolation mode so no other workspace sees it.

Field	Type	Default	Description
enabled	bool	false	Set catalog isolation mode to ISOLATED.
workspace_ids	list of tables	[]	Workspace bindings; see below.

Each entry in workspace_ids is a table with two fields:

Field	Type	Default	Description
id	integer	required	Databricks workspace ID.
binding_type	string	"READ_WRITE"	Access level: "READ_WRITE" or "READ_ONLY".

binding_type maps to the Databricks API values BINDING_TYPE_READ_WRITE and BINDING_TYPE_READ_ONLY.

[pipeline.NAME.checks]

Assert that the data is right while the run is happening, not in a separate test step afterwards. Two surfaces share this section:

• Pipeline-level switches — row count, column match, freshness, null rate, custom SQL, anomaly detection.
• Model-level [[assertions]] blocks — not_null, unique, in_range, regex_match, and the rest.

See Data quality checks for what each one means.

Field	Type	Default	Description
enabled	bool	false	Master switch for the pipeline-level checks below.
fail_on_error	bool	true	When false, downgrades every error-severity assertion to a non-fatal result.
row_count	bool	false	Compare row counts between source and target.
column_match	bool	false	Verify source and target have the same column sets.
freshness	table		{ threshold_seconds = N }. Every table is checked against the single threshold_seconds. There is no per-schema override.
null_rate	table		{ columns = [...], threshold = 0.0–1.0, sample_percent = 10 }.
custom	list	[]	Custom SQL checks. Each entry has name, sql, and optional threshold.
anomaly_threshold_pct	float	50.0	Row count deviation percentage that triggers an anomaly. Set to 0 (or a negative value) to disable detection. Must be a finite number: nan and inf are refused when the config loads.
quarantine	table		{ enabled = true, mode = "split" | "tag" | "drop" }. Off unless enabled = true. See below.
assertions	list	[]	Repeated [[assertions]] blocks (DQX parity). See below.
cross_source_overlap	table		Flags the same business key appearing across sibling sources that feed one consolidation target. See Cross-source duplicate detection.

[[pipeline.NAME.checks.assertions]]

Declarative model-level assertions. Each block declares a type and type-specific parameters. All assertions share the same base fields:

Field	Type	Default	Description
table	string	(required)	Unqualified target table the assertion runs against. When assertions live in a model’s sidecar TOML the table is implied; in pipeline-level [[checks.assertions]] blocks (shown below) it must be set explicitly.
name	string		Optional identifier used as the result’s name; synthesized from {kind}:{column} when unset. Set it to disambiguate multiple assertions on the same table/kind/column. Five names are reserved for the engine’s own results and refused at config load: quarantine:compile, quarantine:execute, row_count, column_match and row_count_anomaly, plus freshness when the pipeline declares [checks.freshness]. The comparison maps every non-alphanumeric character to _ first, the way Dagster keys a check, so quarantine_compile and quarantine.execute are refused too. A [[checks.custom]] name is checked by the same rule. Beyond those reserved names, two check names that sanitize alike are refused too — a custom check, a null_rate column, or an assertion (named or unnamed) colliding with another. Config load catches every case it can see (any table-independent pair, or a pair sharing a table an assertion names explicitly); rocky run and rocky plan refuse the complete set again once discovery has resolved the real table set, catching a cross_source_overlap collision config load cannot see.
type	string	(required)	One of: not_null, unique, unique_expr, accepted_values, relationships, expression, row_count_range, in_range, regex_match, aggregate, composite, not_in_future, older_than_n_days.
column	string		Required for row-level column kinds (not_null, unique, accepted_values, relationships, in_range, regex_match, not_in_future, older_than_n_days).
severity	string	"error"	error fails the pipeline (subject to fail_on_error); warning reports but never fails.
filter	string		SQL boolean predicate that scopes the assertion to a subset of rows.

Type-specific parameters:

Type	Additional fields
accepted_values	values: [String]
relationships	to_table: String, to_column: String
expression	expression: String (SQL boolean predicate)
row_count_range	min: u64?, max: u64?
in_range	min: String?, max: String? (numeric literals)
regex_match	pattern: String (dialect-specific regex; no single quotes, backticks, or semicolons)
aggregate	op: sum|count|avg|min|max, cmp: lt|lte|gt|gte|eq|ne, value: String
composite	kind: "unique", columns: [String] (≥2)
unique_expr	key_expr: String (derived SQL key, e.g. md5(tenant || '-' || id); one expression over the row, allowlisted functions only, and no volatile function or COLLATE; see Per-assertion filter)
older_than_n_days	days: u32

[[pipeline.silver.checks.assertions]]
table = "orders"
type = "not_null"
column = "order_id"

[[pipeline.silver.checks.assertions]]
table = "orders"
type = "in_range"
column = "amount_cents"
min = "0"
filter = "region = 'US' AND status != 'cancelled'"

[[pipeline.silver.checks.assertions]]
table = "order_lines"
type = "composite"
kind = "unique"
columns = ["order_id", "line_item_id"]

Use unique_expr when the meaningful identity is a computed value rather than any stored column, for example a surrogate key built to be stable across a multi-tenant union, which neither unique (single column) nor composite (column tuple) can express.

[pipeline.NAME.checks.quarantine]

Keep the bad rows out of the clean table instead of only counting them. Rocky takes the rows that fail an error-severity row-level assertion and puts them in their own table, or marks them in place. Quarantine runs in quality pipelines only. Rocky accepts the block on other pipeline types and ignores it there, with no warning.

Field	Type	Default	Description
enabled	bool	false	Turn quarantine on. Without enabled = true, the block does nothing.
mode	"split" | "tag" | "drop"	"split"	What Rocky does with the failing rows. See the table below.
suffix_valid	string	"__valid"	Suffix for the table of passing rows. split and drop refuse a suffix that makes this name match the source table, so an empty suffix is refused. Names are compared exactly.
suffix_quarantine	string	"__quarantine"	Suffix for the table of failing rows. split refuses a suffix that makes this name match the source table or the valid table. drop writes no quarantine table, so it does not check this one. Names are compared exactly, so suffixes that differ only in case are not refused. On a warehouse that ignores case in a table name, such as DuckDB or Databricks, the two name one table, and the valid write replaces the quarantine rows there.

Mode	Behavior
split	Writes <table>__valid with the passing rows and <table>__quarantine with the failing rows. When the run completes, and the two suffixes name two different tables in your warehouse, each row lands in exactly one of them. Each failing row carries an _error_<name> column per assertion. The original <table> stays as it is. Point downstream models at <table>__valid. Refused on a dialect with no SELECT * EXCEPT form, such as Trino.
tag	Rewrites <table> in place and adds an _error_<name> column per assertion, set on the failing rows. Every row stays. This rewrites the source, so take care on a raw replication target.
drop	Writes only <table>__valid. Rocky discards the failing rows.

Only these row-level kinds are quarantined: not_null, accepted_values, expression, in_range, regex_match, not_in_future and older_than_n_days. All three modes accept all of them, and a clock function in an expression or filter: split evaluates each predicate once. A quarantine that is refused (quarantine:compile) or fails at the warehouse (quarantine:execute) fails the run, whatever fail_on_error says. Set-based, table-level and referential assertions (unique, unique_expr, composite, relationships, row_count_range, aggregate) run as ordinary checks whatever the mode.

Cross-source duplicate detection

The assertions above (especially unique / unique_expr / composite) also run on replication pipelines, not just transformation/quality ones, so a target table that’s silently doubled by the same source arriving twice is caught at load time, not three models downstream.

For the cross-table case (the same business key arriving through two sibling sources that later get UNION-ed into one consolidation target), use [pipeline.NAME.checks.cross_source_overlap]. A per-table unique check passes on each table individually; only an overlap check spanning the siblings sees the duplication.

Field	Type	Default	Description
keys	list of strings	[]	Business-key columns whose shared value across sibling tables signals a duplicate. Mutually exclusive with key_expr.
key_expr	string		Derived business-key expression (e.g. md5(a || '-' || b)) for sources without a single natural key. Mutually exclusive with keys. Same gate as unique_expr’s key_expr: one expression over the row, allowlisted functions only, no volatile function and no COLLATE.
severity	string	"error"	error fails the pipeline (subject to fail_on_error); warning reports but never fails.
max_overlap_rows	integer	0	Overlap-key count above which the check fails. 0 means any overlap fails.
sample	integer	20	Maximum overlapping keys attached to the result for triage.

Exactly one of keys or key_expr is required. Sibling tables whose key can’t be evaluated (missing column / keyless) are skipped with a logged reason rather than erroring.

This is the detective counterpart to discover-time on_collision (the preventive catch). See Data Quality Checks for the full semantics.

[pipeline.NAME.execution]

Tune how many tables this pipeline processes at once, and when it gives up. The defaults suit a remote warehouse; pin concurrency to a number when you need a predictable load.

Field	Type	Default	Description
concurrency	string | integer	"adaptive"	Max tables processed in parallel. "adaptive" (default) uses an AIMD throttle starting at 32 and adjusting to rate-limit signals — best for remote warehouses. A fixed integer (e.g. 8) pins concurrency; 1 is serial.
fail_fast	bool	false	Abort all remaining tables on first error.
error_rate_abort_pct	integer	50	Abort if error rate exceeds this percentage (0–100). Set to 0 to disable.
table_retries	integer	1	Times to retry failed tables after the initial parallel phase. Set to 0 to disable.

---

[[pipeline.NAME.sources]]

Declare the external tables a transformation pipeline reads, and how fresh each must be. A source is a table your models read but do not build, for example a landing table that Fivetran loads. rocky freshness checks each declared freshness block against the warehouse. Transformation pipelines only.

Field	Type	Default	Description
catalog	string	""	Catalog of the table. Leave it out for two-part names (DuckDB).
schema	string	(required)	Schema of the table.
table	string	(required)	Table name.
freshness.loaded_at_field	string	(required in freshness)	Column that holds each row’s load time. Use a DATE or TIMESTAMP column.
freshness.warn_after	string	(unset)	Age above which the source reports warn. Format: a positive integer and one unit, s, h or d ("3600s", "12h", "7d").
freshness.error_after	string	(unset)	Age above which the source reports error. Must not be shorter than warn_after.
freshness.filter	string	(unset)	SQL predicate that limits the rows Rocky takes the maximum over. Rocky adds it as WHERE (<filter>). A ; is refused.

Set at least one of warn_after and error_after.

[[pipeline.silver.sources]]
schema = "raw"
table  = "orders"

[pipeline.silver.sources.freshness]
loaded_at_field = "_loaded_at"
warn_after      = "12h"
error_after     = "24h"
filter          = "status <> 'test'"

rocky compile checks each block. It raises E050 for a block it cannot evaluate: no threshold, a duration that does not parse, error_after shorter than warn_after, a loaded_at_field that is not a plain column name, or a filter with a ;. It raises W050 when the compiler’s source schema shows the loaded_at_field as missing or not a date or time type. That schema comes from a seed or the schema cache and can be out of date, so this case only warns.

[pipeline.NAME.schedule]

Declare when this pipeline is due, so rocky tick can run it without an external orchestrator. Every field is optional. Omit the block and the pipeline has no standing demand: it runs only when you invoke it. Experimental while the reconciler soaks.

It works on replication, transformation, quality, and snapshot pipelines. A load pipeline cannot join the schedule graph yet, for two reasons. A load re-ingests every discovered file on each run rather than incrementally, so scheduling one would duplicate data. It also records no run the scheduler can observe. rocky validate rejects a scheduled load pipeline (V044) and an after that names one (V045).

Run one rocky tick timer per project. Schedule state is per-machine. A remote [state] backend never replicates the scheduler’s cursor and claim tables, and remote state has no cross-host locking. Two hosts ticking the same project therefore both fire the same occurrence.

Field	Type	Default	Description
cron	string	—	Standard 5-field cron expression (minute hour day-of-month month day-of-week). Due at each occurrence. First-ever evaluation anchors at “now” and waits for the next occurrence rather than firing immediately.
timezone	string	project [schedule].timezone, else "UTC"	IANA timezone name (e.g. "Europe/Lisbon") the cron occurrences are evaluated in. DST transitions are handled: a skipped wall-clock hour fires at the next valid instant once, a repeated hour fires on the first occurrence only.
after	array of strings	[]	Upstream pipeline names. Due once every listed pipeline has a successful run that completed after this pipeline’s own latest success started. A partial-success (exit 2) upstream run does not count. Cycles are a validation error.
freshness	bool	false	When true, the pipeline is due once its own run-staleness exceeds its freshness budget. That budget is the minimum of its member models’ max_lag_seconds, falling back to the project [freshness] expected_lag_seconds. This is run-staleness, not a warehouse data-freshness probe; the reconciler issues no queries. freshness = true with no resolvable budget is a validation error.
catchup	string	"latest"	Policy when more than one cron occurrence elapsed since the last fire. "latest" fires one demand at the most recent missed occurrence; "skip" advances the anchor and runs nothing. "all" is rejected at validation — Rocky runs are watermark-driven, not windowed, so replaying every missed occurrence is pure cost with no extra data.
retry	table	{ max = 0 }	In-tick retry on failure. retry.max is the number of additional immediate re-submissions after the first attempt fails. The reconciler never sleeps between attempts; minutes-scale spacing between ticks is the always-on cross-tick backoff, not this knob. Partial (exit 2) runs are never retried.
timeout_minutes	integer	0	Scheduler-level timeout for a launched run. 0 means none — the run’s own limits apply. On elapse the child is terminated gracefully, then forcibly.
enabled	bool	true	When false, demand is suppressed but the config is kept (it shows as disabled in the tick’s skipped[]).

A pipeline may combine sources — any one being due makes the pipeline due. A [schedule] block with none of cron/after/freshness set and enabled = true is an inert-config warning.

[pipeline.raw]
type = "replication"
[pipeline.raw.schedule]
cron = "0 3 * * *"
timezone = "Europe/Lisbon"

[pipeline.staging]
type = "transformation"
models = "models/**"
[pipeline.staging.schedule]
after = ["raw"]            # runs in the same tick, after raw, once raw has a newer success
freshness = true          # ...and also whenever staging's output goes stale
retry = { max = 2 }

Project-level defaults live in a top-level [schedule] block:

Field	Type	Default	Description
timezone	string	"UTC"	Default timezone for every pipeline’s cron.
poll_interval_seconds	integer	15	Cadence of rocky serve --scheduler. --poll-interval-seconds overrides it. The one-shot rocky tick does not read it.

---

[state]

Choose where Rocky keeps what it remembers between runs: watermarks, run history, plans, and checkpoint progress. Local is the default and needs no setup. Pick an object store or Valkey when runs happen on machines that do not survive — a CI runner, a Kubernetes pod.

Field	Type	Default	Description
backend	string	"local"	Storage backend: "local", "s3", "gcs", "valkey", or "tiered".
s3_bucket	string		S3 bucket name. Required when backend is "s3" or "tiered".
s3_prefix	string	"rocky/state/"	S3 key prefix for state files.
gcs_bucket	string		GCS bucket name. Required when backend is "gcs".
gcs_prefix	string	"rocky/state/"	GCS object prefix for state files.
valkey_url	string		Valkey/Redis connection URL. Required when backend is "valkey" or "tiered".
valkey_prefix	string	"rocky:state:"	Valkey key prefix for state entries.
transfer_timeout_seconds	int	300	Wall-clock budget for each transfer (upload or download). Retries share this budget rather than extending it; raise for large state or slow networks.
on_upload_failure	string	"skip"	Upload failure policy. "skip" warns and continues; recovery requires a surviving local ledger or successfully published recovery intent. "fail" propagates the error. Governed runs require durability.
namespacing	string	"none"	State-file namespacing policy. "none" (default) keeps one global state file — byte-identical to a project that omits this key. "pipeline" gives each pipeline its own state file (see State namespacing below).
concurrency_control	string	per backend	"cas" or "off". Unset means "cas" on s3, gcs, and tiered, and "off" on local and valkey. "cas" makes each state upload conditional on the remote object still carrying the generation this writer downloaded. A run that lost a cross-pod race then fails closed instead of erasing the winner. The ledger-seam commands (policy, gc, restore, apply) retry onto the winner instead. "off" uploads unconditionally (last writer wins). See Concurrent writers below.
on_schema_mismatch	string	"recreate"	What to do when the binary opens a state store written by a newer binary, which happens mid-way through a rolling upgrade. "recreate" logs one warning, starts from fresh local state, does one full-refresh run, and never writes the downgraded state back to the shared tier. "fail" aborts the open instead. Only the run path honours this; inspection and branch commands always hard-fail on a forward-incompatible store.
freeze_marker_writes	bool	false	Write durable freeze and unfreeze marker objects beside the remote state file (under <prefix>/freeze/ and <prefix>/unfreeze/) when rocky policy freeze / unfreeze run. Reading and enforcing markers is always on wherever a durable object tier exists. You can therefore upgrade a whole fleet to marker readers before any marker is written. Requires backend = "s3", "gcs", or "tiered"; setting it on "local" or "valkey" is a hard error rather than a silent no-op.

Set the fields each backend requires, as the table shows. "tiered" downloads from Valkey first (fast) and falls back to S3 (durable). It uploads to both.

[state]
backend = "tiered"
valkey_url = "${VALKEY_URL}"
s3_bucket = "${ROCKY_STATE_BUCKET}"

Concurrent writers

Stop a second run from erasing a first when both share one [state] prefix. Without protection, two concurrent runs both download the ledger, both change it, and the second upload wipes the first. The first run’s watermarks, run records, and policy rows are lost, and Rocky raises no error.

Compare-and-swap (CAS: a write that succeeds only if the object has not changed since you read it) closes that. It is the default on s3, gcs, and tiered, so an s3 config with only s3_bucket is protected with no extra setting. Each upload commits only if the remote object still carries the generation this run downloaded. A run that lost the race exits non-zero with a compare-and-swap conflict rather than overwriting the winner.

What it does not do is reconcile the two runs. You re-run the loser yourself. Rocky does not roll back the warehouse writes it already made, so a non-merge strategy can duplicate rows on that re-run. Prefer merge-style strategies for pipelines you expect to contend.

It needs a backend with a durable object tier: s3, gcs, or tiered. On local and valkey an unset mode is off, and an explicit "cas" downgrades to off with a warning, because neither offers a conditional write. cas also stops the mid-run periodic uploader on every backend. A crashed run then leaves the remote ledger at its last committed generation instead of a partial mid-run snapshot.

How Rocky picks the mode at startup

Some S3-compatible stores (older MinIO releases, some gateways) accept the conditional-write headers and ignore them. A CAS write there would silently act as an unconditional one. So every writer checks the store once per process before it relies on CAS:

concurrency_control unset ──► backend default
                               s3 / gcs / tiered → cas    local / valkey → off
            │
  cas on s3 / gcs / tiered ──► conditional-write probe
            ├─ supported     → cas
            ├─ unsupported   → unset: warn, use off     explicit "cas": error, writer stops
            └─ inconclusive  → cas  (never fall back to unconditional on a guess)

The probe writes a small object under <prefix>/cas-probe/ with a unique name. It checks that the store refuses a create over an existing object and a write with a stale generation, then deletes the object. It never touches the state object. It needs put and delete access under the state prefix; without delete access, one small probe object stays behind per process. rocky doctor runs the same probe and reports the result (see state_concurrency).

An inconclusive probe (a network error, a timeout, or no write access) keeps cas and logs a warning. That is not confirmed protection: on a store that ignores conditional headers, a cas write still overwrites. Fix the access problem, then run rocky doctor to confirm the probe passes.

Set concurrency_control = "off" to opt out. That is correct for a deployment with one writer per [state] prefix.

The cas-required marker

A writer on off uploads unconditionally, so one such writer could still overwrite every CAS writer. The marker stops that. The first CAS upload of a state object creates a small cas-required object beside it, for example <prefix>/v9/state.redb.cas-required. The write is create-once on its own key, like a freeze marker.

A writer whose mode resolves to off checks for the marker when it starts, before it does any work, and again before each upload. If the marker exists, the writer stops with an error that names the fix. Reads are not affected. The fix is to set concurrency_control = "cas", or to remove the explicit "off", on that writer. Delete the marker only on purpose, when every writer of that state is deliberately moving to "off". rocky doctor warns when the configured mode and the marker disagree.

The marker sits under the schema-version folder (v9/ above), and a schema bump does not move it. So the check also looks at each older version’s folder, down to the oldest one Rocky still carries state forward from. A marker left by the fleet before an upgrade still stops an off writer after it. To delete the marker on purpose, delete it in every version folder that has one.

Two limits are worth knowing:

• The check and the upload are separate steps. An off writer checks for the marker, then uploads. A first CAS upload that creates the marker between those two steps does not stop that one upload. After the marker exists, every later off upload is refused.
• A marker can exist on a store that ignores conditional writes. A CAS writer creates the marker even when its probe was inconclusive. If the store in fact ignores conditional headers, the cas writers are not protected from each other, yet the marker still blocks every off writer. Run rocky doctor until the probe passes, or set every writer to "off" and delete the marker on purpose.

Upgrading to the cas default

Engine 1.77.0 changed the default on s3, gcs, and tiered from off to cas. Older binaries do not know the marker exists and keep uploading unconditionally. So upgrade every writer that shares a [state] prefix together. Or keep one writer per prefix until the whole fleet runs 1.77.0 or later. To keep the old behaviour, set concurrency_control = "off" explicitly before you upgrade. Run rocky doctor on each writer afterwards to confirm the resolved mode.

What cas covers. Every write of the shared state object goes through it. The end-of-run upload fails closed when it loses a race. The ledger seams retry instead: rocky policy freeze / unfreeze, rocky gc, rocky restore (and a restore-shaped rocky apply <plan-id>), and the policy rows a run-shaped rocky apply writes when a [policy] rule sets verify_after. A seam that loses a race downloads the winner and replays its change on top. Where the change rests on a policy decision (a restore, or the rule decision a governed rocky apply records before it runs), the decision is re-made against the winner first. It gives up after three attempts with a conflict error and leaves the winner in place. It never falls back to an unconditional upload. A restore re-proves each artifact on every attempt. With a [policy] block it also re-checks freezes right before each object write, because a written object cannot be taken back. This applies equally to s3, gcs, and tiered.

The guarantee holds only between writers that all run with cas. Once the cas-required marker exists, a writer on off refuses to upload rather than overwrite. Before the first CAS upload, and for binaries older than 1.77.0, nothing stops it. Until every writer that shares a [state] prefix runs cas on 1.77.0 or later, keep one writer per [state] prefix.

On tiered, cas additionally makes the Valkey tier coherent with the durable object. The compare-and-swap runs against S3 first; only after it commits is the Valkey copy written, stored together with the generation it was committed at. A read may use the cached copy only after confirming that generation is still the durable object’s — otherwise it reads S3. So a Valkey write that fails, a process that dies between the two, or a cache entry left over from an earlier run can no longer shadow durable state. Cached copies are held under a separate key from the off path’s, so a fleet can move pods from off to cas one at a time.

With concurrency_control = "off" the tiered backend keeps its historical behaviour, including the stale-cache window: an off write carries no generation, so there is nothing for a read to validate a cached copy against.

Upload failures and recovery. Replication publishes recovery intent before copying on supported atomic adapters. A failed pre-copy upload with "fail" stops before INSERT. With "skip", it warns and continues. A fresh pod cannot recover unpublished intent after the local ledger disappears. See Interrupted replication.

A failed watermark flush keeps available progress and attempts its upload. Ordinary "skip" runs warn and retain their normal exit policy. Explicit "fail" and governed runs return nonzero after preserving available state.

[state.retry]

Decide how hard Rocky retries a state upload or download that failed for a transient reason: a network hiccup, a transient 5xx, an endpoint that hung past the per-request HTTP timeout. The block has the same shape as [adapter.NAME.retry], so both layers read the same way. Retries share the outer transfer_timeout_seconds budget rather than extending it, so the total wall-clock ceiling does not move.

Field	Type	Default	Description
max_retries	int	3	Maximum retry attempts per transfer. Set to 0 to disable retries.
initial_backoff_ms	int	1000	Initial backoff before the first retry.
max_backoff_ms	int	30000	Cap on exponential backoff growth.
backoff_multiplier	float	2.0	Multiplier applied between retries (e.g. 2.0 = doubling).
jitter	bool	true	Add random jitter to prevent concurrent runs from retrying in lockstep.
circuit_breaker_threshold	int	5	Trip the breaker after this many consecutive failures. 0 disables.
circuit_breaker_recovery_timeout_secs	int | null	null	Seconds in Open before a half-open trial is allowed. null = manual reset only.
max_retries_per_run	int | null	null	Cross-transfer retry budget for a single run. null = unbounded (per-transfer max_retries is the only cap).

Every transfer ends with a structured outcome field on its state.upload or state.download event: ok, absent, carried_forward, timeout, error_then_fresh, skipped_after_failure, transient_exhausted, circuit_open, or budget_exhausted. Build alerts on those values, not on the free-form log message.

[state.idempotency]

Tune how rocky plan --idempotency-key <KEY> deduplicates a retried run (the rocky run --idempotency-key alias reads the same block). Every field is optional and takes the default shown. The block does nothing on a run that passes no key, and Rocky rejects an unknown field rather than ignoring it.

Field	Type	Default	Description
retention_days	integer	30	Lifetime of a terminal idempotency stamp before garbage collection. GC runs during the state upload sweep; no separate cron.
dedup_on	string	"success"	Which terminal statuses count as “already processed”. "success" only stamps successful runs (failures stay claimable for retries); "any" stamps every terminal status.
in_flight_ttl_hours	integer	24	Hours after which an InFlight claim is treated as a crashed-pod corpse and adopted by a fresh caller. Informational on Valkey/tiered backends, which set the TTL server-side via SET NX EX.

Stamps live in the IDEMPOTENCY_KEYS redb table and replicate on tiered backends so sibling pods see the same entry. See rocky plan --idempotency-key for the three possible outcomes (fresh_run, skipped_idempotent, skipped_in_flight).

Bucket-native lifecycle for object-store backends

Rocky’s built-in sweep deletes idempotency stamps from state.redb after retention_days. On s3 / gcs / tiered backends, the sweep is correct but pays a per-key delete during state upload. For projects that emit thousands of stamps per day, configuring a bucket-native lifecycle rule is faster, cheaper, and keeps GC running even when no Rocky process is active.

Both rules below match the default state.s3_prefix / state.gcs_prefix of rocky/state/. Adjust the prefix if you’ve overridden it. The retention window should match [state.idempotency] retention_days, or be larger, so Rocky’s own sweep doesn’t try to delete an object the bucket has already removed.

S3, s3api put-bucket-lifecycle-configuration payload:

{
  "Rules": [
    {
      "ID": "rocky-idempotency-stamps-30d",
      "Status": "Enabled",
      "Filter": { "Prefix": "rocky/state/" },
      "Expiration": { "Days": 30 }
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration \
  --bucket "$ROCKY_STATE_BUCKET" \
  --lifecycle-configuration file://rocky-lifecycle.json

The same rule works for any object Rocky writes under rocky/state/, including state-store snapshots. If you want to retain snapshots longer than stamps, namespace them under separate prefixes via state.s3_prefix and configure two rules.

GCS, gcloud storage buckets update lifecycle JSON:

{
  "lifecycle": {
    "rule": [
      {
        "action": { "type": "Delete" },
        "condition": {
          "age": 30,
          "matchesPrefix": ["rocky/state/"]
        }
      }
    ]
  }
}

gcloud storage buckets update "gs://$ROCKY_STATE_BUCKET" \
  --lifecycle-file=rocky-lifecycle.json

Operational notes:

• Bucket lifecycle does not replace [state.idempotency] retention_days. The local redb mirror on each pod still has its own copy of the stamp; Rocky’s sweep is what evicts that. Bucket lifecycle handles the durable copy.
• In-flight claims (InFlight) are TTL-bounded by in_flight_ttl_hours, not by the lifecycle rule. Don’t set the lifecycle window shorter than in_flight_ttl_hours (default 24) or you risk reaping a live claim.
• Tiered backends already serve hits from Valkey first. A bucket lifecycle that’s slightly behind retention_days is harmless; Valkey’s own TTL evicts the hot copy long before the cold S3/GCS copy expires.

State namespacing

Let independent runs proceed at the same time instead of queueing behind one lock. redb permits one writer per state file. Run one rocky run process per pipeline or client against the single global file (<models>/.rocky-state.redb) and those runs serialize on one advisory lock. Namespacing gives each pipeline, or each client, its own state file with its own lock, its own redb handle, and its own remote object key.

Namespacing is opt-in and default-off.

Mode	Behavior
"none" (default)	One global <models>/.rocky-state.redb for the whole project.
"pipeline"	One state file per pipeline, under <models>/.rocky-state/<pipeline>.redb.

[state]
backend = "local"
namespacing = "pipeline"   # each pipeline gets its own state file

To fan out by client/tenant rather than by pipeline name, use the per-invocation --state-namespace <key> flag instead. Its precedence:

1. An explicit --state-path <path> is a hard override that disables namespacing for that invocation.
2. Otherwise --state-namespace <key> wins over the [state] namespacing config.
3. Neither set ⇒ the single global state file (default).

<key> must be a SQL identifier (^[a-zA-Z0-9_]+$): it becomes a path segment. Namespaced files start fresh; the legacy global file is never moved or auto-seeded, so carry watermarks forward manually if needed (copy the global file to <models>/.rocky-state/<key>.redb, or point --state-path at it for the first run).

---

[run]

Skip rebuilding a transformation model when neither its logic nor its upstream data appears to have moved since the last successful build. The gate is off unless you turn it on.

Best-effort, not a result-equivalence guarantee

Skipping is a best-effort optimization. It is not a promise that a fresh rebuild would produce identical bytes. It rests on two heuristics: a cosmetic-invariant IR hash for the logic, and MAX(ts) or rowcount movement for the upstream data. Every field defaults to the safe no-skip choice, and any missing, unreadable, or ambiguous input rebuilds. The feature stays off until you set skip_unchanged = true or pass --skip-unchanged.

Field	Type	Default	Description
skip_unchanged	bool	false	Master switch for the gate. false ⇒ every selected model always builds, exactly as before. The --skip-unchanged CLI flag turns the gate on for a single invocation regardless of this value; --force-rebuild overrides both.
skip_rowcount_fallback	bool	false	Allow a rowcount-only (COUNT(*)) data-stability signal when an upstream has no tracked timestamp column. Default off: without this, a model whose upstreams are not watermarkable is never skip-eligible. Rowcount equality is weaker than a watermark — it can miss a same-size in-place UPDATE (or a matched insert+delete) that mutates values without changing the row count.
lag_tolerance_seconds	integer	0	Treat an upstream MAX(ts) that moved by fewer than this many seconds as unchanged — the late-arriving-but-irrelevant micro-update analog of a freshness SLA threshold. Default 0: any movement at all forces a rebuild.
strict_scheduling	bool	false	Turn physical-read scheduling warnings into a run refusal. Rocky derives ordering edges from the physical schema.table names a model reads. It reports what it could not safely resolve. That covers contradicting bare-read pairs, models whose reference extraction failed, and colliding targets. It also covers a catalog.schema.table read of a model with no catalog in its [target], when Rocky cannot name that model’s catalog. And it covers a by-name edge that rocky run --dag dropped because an exact read contradicted it. false reports these as warnings and the run still exits 0, so a consumer that ignores them can read a stale target and see success. true refuses the run instead. It lives on [run] rather than [pipeline.NAME.execution] because rocky run --dag derives ordering across pipelines, where a per-pipeline switch would have no single answer.

What is and is not skip-eligible

Rocky skips a model only when both of these hold. Anything else rebuilds.

• (B) Eligible. The model uses a plain materialization strategy (not content_addressed or time_interval). Its [skip] eligible is not false, and Rocky can prove its SQL deterministic. It always rebuilds SQL containing CURRENT_TIMESTAMP / NOW(), RANDOM(), UUID(), CURRENT_USER, CURRENT_CATALOG, ANY_VALUE, ARRAY_AGG, COLLECT_LIST, COLLECT_SET, MODE, an unordered LIMIT / TOP / FETCH, or any function not on Rocky’s pure-function allowlist. full_refresh is eligible: a deterministic full refresh whose logic and inputs have not moved is safe to skip.
• (G) Upstreams provably unchanged. Rocky must be able to prove every upstream stable, and that needs the model’s lineage to be provably complete. Only a single plain SELECT over bare tables qualifies. A model that uses a CTE, a subquery in FROM, or an IN (SELECT …) / EXISTS / scalar sub-select is never skipped. Nor is one that uses a PIVOT / UNNEST / nested-join table factor, or a set operation (UNION / INTERSECT / EXCEPT). Rocky cannot prove it saw every upstream of those shapes.

--force-rebuild plus full_refresh always builds.

Per-model [skip] overrides

Override the automatic decision for one model with a [skip] block in its sidecar. It has two keys, eligible and deterministic, both unset by default. See [skip] in Model format for the fields.

---

[reuse]

Let a content-addressed model reuse work it already did, with a record an auditor can check. The two knobs are independent of each other. Both apply only to the content-addressed (S3/UniForm) write path. On DuckDB and plain warehouse targets they do nothing, and rocky run behaves exactly as if the block were absent.

Field	Type	Default	Description
enabled	bool	false	Byte-level point-to reuse. When true, a successful run records, per model, an input-match index entry and an offline-verifiable provenance record; on a later run, an eligible model whose recomputed input_hash hits the index for a prior strong run may point to that run’s already-written Parquet — a zero-copy commit that skips the SQL. false (the default) writes no input-match spine: no per-model hashing cost, no extra state write, no reuse decision.
column_level	bool	true	Column-level skip. An unpartitioned content-addressed model whose logic, environment, and every provably-consumed upstream column are unchanged since its last successful build is skipped — its SQL does not run and no new commit is written; the prior output stays authoritative. On by default. Independent of enabled.

Both decisions fail closed. Any unproven input forces a build: a non-deterministic model, a changed recipe or environment, a consumed-column set Rocky cannot enumerate, a missing or moved column hash. Doubt always resolves to running the SQL.

The kill switch for the default-on knob is column_level = false, which restores the always-build behavior. The per-invocation --no-reuse flag forces every model to build (the escape hatch parallel to --force-rebuild for --skip-unchanged). The provenance record this path writes is what an auditor reads in Verify a run.

---

[ai]

Cap what an AI generation can spend. The block applies to rocky ai, rocky ai-explain, rocky ai-sync, and rocky ai-test. Rocky rejects an unknown field here rather than ignoring it.

Field	Type	Default	Description
max_tokens	integer	4096	Per-request max_tokens sent to the Anthropic Messages API and the cumulative output-token budget enforced across the compile-verify retry loop. When the running total of output_tokens returned by the LLM across attempts exceeds this value, Rocky fail-stops with a TokenBudgetExceeded error instead of issuing another retry. Increase only when generations legitimately need more headroom (large model surfaces, verbose tests).

Every rocky ai* command reads this block. Rocky never reads the API key from rocky.toml. It comes from the ANTHROPIC_API_KEY environment variable instead, so the key never lands on disk in a project file.

---

[cache]

Stop paying a warehouse round-trip on every typecheck. Today the block holds one cache, the schema cache. It stores DESCRIBE TABLE results. rocky compile and rocky lsp then typecheck leaf models against real column types without asking the warehouse each time.

[cache.schemas]

Control how long a cached table shape stays trusted, and whether other machines see it.

Field	Type	Default	Description
enabled	bool	true	Enable schema cache reads + writes. Set to false for strict CI where every typecheck should resolve against the current warehouse.
ttl_seconds	integer	86400	TTL for cache entries in seconds (default 24h). Lower for high-DDL-churn teams.
replicate	bool	false	Replicate the schema cache via [state] sync. Default is off; a fresh clone should warm its cache from its own rocky apply, not inherit another machine’s stale types.
trusted_max_age_seconds	integer	unset	Cache entries younger than this are trusted as current. A reference to a column a trusted entry lacks is the E041 error. Older entries, and every entry when the key is unset, give the W041 warning.
strict_sources	bool	false	Trust every known source schema, including seeds and old cache entries. Every W041 becomes E041. rocky compile --strict-sources does the same for one invocation.

trusted_max_age_seconds and strict_sources control the missing-source-column check. See Missing columns in external sources.

A Valkey-backed runtime cache exists in the codebase but rocky.toml does not reach it yet. A future [cache.valkey] key is reserved for it.

---

[cost]

Tell rocky optimize what your warehouse costs, so its materialization recommendations reflect your prices rather than generic ones.

Field	Type	Default	Description
storage_cost_per_gb_month	float	0.023	Storage cost per GB-month.
compute_cost_per_dbu	float	0.40	Compute cost per DBU.
warehouse_size	string	"Medium"	Warehouse size for cost estimation (e.g., "Small", "Medium", "Large").
min_history_runs	integer	5	Runs Rocky needs in history before it makes a cost recommendation.

---

[budget]

Put a ceiling on what one run may cost, how long it may take, and how much it may scan. When a run crosses a limit, Rocky emits a budget_breach pipeline event and fires the HookEvent::BudgetBreach hook. Set on_breach = "error" and the run also exits non-zero. Rocky rejects an unknown field here rather than ignoring it.

Field	Type	Default	Description
max_usd	float		Maximum allowed run cost in USD. Cost is computed from per-materialization cost_usd values on RunOutput.cost_summary. None on runs where no adapter produced cost data (e.g. a BigQuery job with no bytes billed).
max_duration_ms	integer		Maximum allowed run wall time in milliseconds.
max_bytes_scanned	integer		Maximum allowed total bytes scanned across every materialization in the run. Useful for CI gates on scan volume even when the dollar cost stays inside max_usd (e.g. a BigQuery query that stops pruning partitions). Aggregated from the per-model bytes_scanned figures the adapter reports. Today that’s BigQuery’s totalBytesBilled; Databricks / Snowflake / DuckDB still inherit None and skip the dimension rather than treating “no data” as zero.
on_breach	string	"warn"	Either "warn" (fire the event, keep the run successful) or "error" (also fail the run).

The three limits are independent, and any one of them breaching trips the event. Rocky evaluates them once per run against the observed totals; per-model budgets are a follow-up. Subscribe to on_budget_breach under [hook.*] to route a breach into a notification system.

Each BudgetBreachOutput carries a limit_type tag: "max_usd", "max_duration_ms", or "max_bytes_scanned". A consumer can branch on the dimension that broke without matching strings in the human message.

---

[portability]

Find SQL that will not run on the warehouse you plan to move to, before you move. Set target_dialect and every rocky compile — and every in-editor check the LSP drives — runs the P001 portability lint against that target. The rocky compile --target-dialect <DIALECT> flag overrides this block. Rocky rejects an unknown field here rather than ignoring it.

Field	Type	Default	Description
target_dialect	string		"databricks", "snowflake", "bigquery", or "duckdb". When unset, no lint runs (the flag-only opt-in behavior).
allow	list of string	[]	Project-wide allow-list of construct labels (case-insensitive). Useful when a project standardizes on a non-portable extension like QUALIFY. Prefer per-model -- rocky-allow: … pragmas for targeted exemptions.

[portability]
target_dialect = "bigquery"
allow = ["QUALIFY"]

The effective target dialect is the --target-dialect flag if set, else target_dialect, else none (no lint).

See Linters for the full list of covered constructs and the per-model pragma syntax.

---

[lint]

Tune the style rules that rocky lint runs. Rocky rejects an unknown field here, and an unknown rule code is an error.

Field	Type	Default	Description
disable	list of string	[]	Rule codes that do not run, such as ["S003"].
severity	table	{}	Severity per rule code: "error", "warning" or "info". A rule keeps its default severity if it has no entry.

[lint]
disable = ["S004"]

[lint.severity]
S001 = "error"

rocky lint exits with code 1 when a finding has error severity. See rocky lint for the rules.

---

[retry]

Share one retry budget across every adapter in a run, instead of giving each its own. Rocky builds a single counter and passes it to all of them; once it runs out, no adapter retries again. This stops one failing endpoint from consuming retries the other adapters would have used. Omit the block and each [adapter.NAME.retry] keeps its own independent budget.

Field	Type	Default	Description
max_retries_per_run	integer		Total retries allowed across every adapter for this run. Omit the block to keep per-adapter budgets (each [adapter.NAME.retry] remains in isolation).

---

[contracts]

Project-level contract settings.

Key	Type	Default	Description
strict	bool	false	Refuse a contract column whose declared type Rocky cannot check. Every I003 note becomes the E059 error, which names the model and the column, says why the type is unknown, and says how to fix it. rocky compile --strict-contracts and rocky ci --strict-contracts do the same for one invocation. rocky run and rocky test read this key.

[mask]

Hide sensitive column values without editing a model. Tag a column with a [classification] in its sidecar, then map that tag to a strategy here. Every column carrying the tag gets the same treatment, across every model. See Governance for the narrative.

Strategy	Behavior
"hash"	SHA-256 hex digest of the column value. Deterministic, one-way.
"redact"	Replace the value with the literal string '***'.
"partial"	Keep the first and last two characters; replace the middle with ***. Values shorter than 5 chars are fully replaced with '***'.
"none"	Explicit identity; no masking applied. Useful as a per-env override to unmask a column that defaults to masked at the workspace level.

# models/customers.toml tags email + ssn with these classifications
[mask]
pii = "hash"
confidential = "redact"

An unknown strategy — a typo like "mask" — fails at config load. Rocky never accepts a spelling it cannot emit SQL for.

Adapter support

Masking works today against Databricks Unity Catalog, using column tags plus CREATE MASK / SET MASKING POLICY. Rocky emits one statement per column, because Unity Catalog rejects multi-column masking DDL. Other adapters do not support it yet. Rocky applies masks after a successful DAG run, best-effort: a failure logs a warning and does not abort the pipeline, the same as grants.

[mask.<env>]

Mask a column differently per environment. Rocky starts from the [mask] defaults and layers the block matching the active environment on top.

[mask]
pii = "hash"
confidential = "redact"

[mask.prod]
pii = "none"            # unmask pii in prod (e.g. service principal reads)
confidential = "partial"

[mask.staging]
pii = "partial"         # staging gets a softer mask than the dev default

Resolution precedence:

1. [mask.<env>] entry for the active env (when supplied to rocky plan --env <env>).
2. [mask] workspace default.
3. Unmatched tag: W004 warning unless the tag is listed in [classifications] allow_unmasked.

---

[classifications]

Silence the W004 warning for a classification tag you never intend to mask. This is a project-wide setting, distinct from the per-model [classification] sidecar block that applies the tags (see Model format).

Field	Type	Default	Description
allow_unmasked	list of strings	[]	Classification tags allowed to appear in a model sidecar without a matching [mask] strategy. Suppresses the W004 compiler warning.

Use it for tags that exist only for discovery or lineage tracking. rocky compliance still lists those columns and still suppresses their exceptions, so the report never pretends they are enforced.

---

[role.<name>]

Describe your access model as roles that inherit from each other, rather than repeating the same permission list everywhere. Each [role.<name>] block declares one role. Rocky flattens the inheritance graph before it reconciles against the warehouse’s own role or group system. It rejects a cycle or an unknown parent at config load.

Field	Type	Default	Description
inherits	list of strings	[]	Immediate parent role names. Rocky unions permissions transitively across every ancestor. Cycles and unknown parents are rejected at config-load time.
permissions	list of strings	[]	Permissions this role grants. Canonical uppercase spellings (e.g. "SELECT", "USE CATALOG", "USE SCHEMA", "MODIFY", "MANAGE"). Empty lists are legal; pure grouping roles exist only to aggregate children.

[role.reader]
permissions = ["SELECT", "USE CATALOG", "USE SCHEMA"]

[role.analyst]
inherits = ["reader"]
permissions = ["MODIFY"]

[role.admin]
inherits = ["analyst"]
permissions = ["MANAGE"]

Rocky flattens the graph into admin → {SELECT, USE CATALOG, USE SCHEMA, MODIFY, MANAGE} and forwards the resolved set to GovernanceAdapter::reconcile_role_graph after a successful DAG.

Databricks only, best-effort

On Databricks, Rocky creates a SCIM group named rocky_role_<name> for each role. It then grants each resolved permission on each managed catalog. A failed group create or GRANT logs a warning and the run continues. Without workspace auth, Rocky only checks the group names and logs the graph. Other adapters do nothing. The resolver catches cycles and unknown parents at config load on every adapter.

---

[selectors]

Saved selectors. Each entry maps a name to a --select expression. Use one with --select selector:<name> (or --exclude selector:<name>) on any command that takes --select. See Saved selectors.

[selectors]
nightly = "tag:nightly+ config.materialized:incremental"
finance = "path:marts/finance,tag:certified"

[freshness]

Set a project-wide staleness budget, so you do not repeat the same threshold in every model sidecar. Freshness is how far behind the newest row in a table is.

Field	Type	Default	Description
expected_lag_seconds	integer	(unset)	Maximum lag before a model counts as stale. This is the field that makes the block active: without it, Rocky treats the project as having no freshness default and inherits nothing.
time_column	string	(unset)	Timestamp column to measure lag from. Inherited by a model that declares no [freshness] block. rocky freshness reads MAX(time_column) from the model’s target table.
severity	string	(unset)	"error" or "warning". Inherited by a model that declares no [freshness] block. Under "error", a stale model makes rocky freshness exit 1.

A model that declares no [freshness] block of its own inherits this one, whole. Precedence, first match wins:

model sidecar [freshness]   ->  models/_defaults.toml [freshness]  ->  project [freshness]

Inheritance is whole-block, not field by field. A model that declares its own [freshness] keeps exactly what it wrote; it does not pick up time_column or severity from here for the fields it left out. A project block with no expected_lag_seconds supplies nothing at all, so time_column or severity on their own are still inert.

The compiler raises W005 on a model that has at least one temporal output column (DATE, TIMESTAMP, TIMESTAMP_NTZ) and no freshness declaration in scope. A model with no temporal column never raises it. Setting expected_lag_seconds here puts a declaration in scope for every model at once, which silences the warning project-wide; a per-model [freshness] block silences it for that model.

The scheduler uses it as the staleness budget for a pipeline that sets [pipeline.NAME.schedule] freshness = true. For a transformation pipeline, a member model’s own max_lag_seconds wins: the budget is the smallest one the members declare. When no member declares one, this project value applies. Every other pipeline type loads no member models, so this value is its only budget.

This block is separate from the [pipeline.NAME.checks] freshness check, which runs as part of the data-quality pipeline and has its own threshold_seconds.

---

[schema_evolution] (removed)

This block is gone. A config that still declares it fails to load, and the error says what to delete.

the `[schema_evolution]` section was removed because nothing ever read it: drift detection never reported a column that disappeared from the source, so Rocky never dropped one and `grace_period_days` never took effect. Delete the `[schema_evolution]` section from this config; removing it changes no behaviour. A source-side column removal alone never schedules a DROP COLUMN, and there is no opt-in for that; a full refresh or a drift-driven table recreation can still discard a target-only column. See https://github.com/rocky-data/rocky/issues/1616.

What to do: delete the section. Nothing about your pipeline changes. Rocky never dropped a column on the strength of that key, so there is no behaviour to replace.

A source-side column removal alone never schedules ALTER TABLE ... DROP COLUMN. The detector and generator for that exist in the engine. So does a state table for tracking a grace period. Nothing calls either function, and nothing writes a record to the table. There is no opt-in for that today. A full refresh or a drift-driven table recreation can still discard a target-only column. Both rebuild the table from only the source’s current columns.

See Schema drift for the changes Rocky does detect and act on.

---

[resilience]

Decide whether a model that fails for a transient reason gets another attempt. This is the run loop’s retry policy, and it re-runs a whole model. It is a different layer from [adapter.NAME.retry], which retries one statement inside a connector. It also differs from [retry], which caps connector retries across the run.

Unlike the skip and reuse gates, this block is on by default. What keeps that safe is the small budget: at most two retries per model, and at most eight across the whole run.

Rocky classifies each failure before deciding. Only a failure it can prove is transient is retried. Permanent and unknown failures are never retried, whatever these settings say.

Field	Type	Default	Description
enabled	bool	true	Master switch. false attempts every model exactly once, with no classification and no backoff.
transient_max_retries	integer	2	Re-runs allowed for a model that failed transiently, so three attempts in total. 0 disables retry but keeps the classifier running for observability.
initial_backoff_ms	integer	500	Wait before the first retry.
max_backoff_ms	integer	30000	Cap on exponential backoff growth.
backoff_multiplier	float	2.0	Multiplier applied to the backoff after each retry.
jitter	bool	true	Add ±25% jitter so concurrent runs do not retry in lockstep.
circuit_breaker_threshold	integer	3	Stop retrying for the rest of the run after this many consecutive transient model failures. Each model still gets its one attempt. 0 disables the breaker.
max_retries_per_run	integer | null	8	Ceiling on total retries across every model in one run. null removes the ceiling; 0 forbids all retries.
contain_failures	bool	false	false stops the run at a model that fails while it runs: no later layer starts, though models already running beside it finish. Every error-severity compile failure withholds its declared DAG descendants by default; unrelated branches can build. A project that does not compile at all builds nothing. See Failure containment. true also withholds the failed model and downstream closure after a runtime failure, then lets unrelated subtrees finish. It reports PartialFailure with the failure and its blast radius.
auto_apply_additive_drift	bool	false	false evolves the target for a new nullable upstream column with no policy gate, as Rocky has always done. true routes the mutation through the [policy] plane first. Rocky applies only a provably additive change with an allow verdict on the schema_change.additive capability, and leaves anything else for review. The verdict starts from a matching rule, or from default_agent_effect when no rule matches, so default_agent_effect = "allow" can grant it without one. A freeze, an exhausted autonomy budget, an unreadable or untrusted decision ledger, or a decision row that fails to persist still refuses. With no [policy] block, nothing is granted.

To restore single-attempt behaviour, for example in CI where a fast failure is what you want, set enabled = false.

---

[imports.<name>]

Check your models against another team’s published schema at compile time. A producer project publishes a snapshot of its compiled IR with rocky publish-ir. You vendor that file into your repository and point an [imports.<name>] block at it. Your rocky compile then fails when the producer drops a column you still read.

Field	Type	Required	Description
path	string	Yes	Directory holding the vendored snapshot files, relative to rocky.toml.
snapshot	string	Yes	Filename of the producer’s current snapshot, relative to path.
baseline	string	No	Filename of the reviewed “before” snapshot, relative to path. Rocky diffs baseline against snapshot to find columns the producer changed. Without it, the column-level diagnostics have nothing to compare.
pin	string	No	Recipe-hash pin. A concrete hash fails compilation (E033) when the vendored snapshot differs at all. "*", or leaving it unset, trusts whatever is vendored.

[imports.orders]
path     = "vendor/orders"
snapshot = "current.json"
baseline = "baseline.json"
pin      = "*"

pin and baseline answer different questions, so use both. pin is a whole-project tripwire that fires on any drift. baseline is the column-level “before” image that lets Rocky report exactly which of your reads broke.

Nothing advances the baseline on its own. Run rocky imports update once you have reviewed a producer change and decided to take it. Run rocky imports update --check in CI to fail when a baseline is behind.

See Cross-team contracts for the full diagnostic table (E030–E034, W030, W031).

---

[branch]

Require sign-off before a branch is promoted. Without this block the gate is off, and rocky branch promote runs unguarded.

[branch.approval]

Field	Type	Default	Description
required	bool	false	true makes rocky branch promote refuse unless enough valid approvals exist for the branch.
min_approvers	integer	1	Valid approvals needed when required = true.
allowed_signers	list of strings	[]	When non-empty, only approvals from these signer emails count. Empty accepts any signer.
max_age_seconds	integer	86400	Approvals older than this are rejected, even when they still match the branch state. Default is 24 hours.

An approval counts as valid only when all of these hold:

• Its signature verifies.
• Its recorded branch-state hash matches the branch as it stands now.
• It was signed inside max_age_seconds.
• Its signer is on the list, when allowed_signers is set.

---

[gc]

Control what rocky gc is allowed to delete. Eviction is recorded, not destructive: an approved rocky apply <gc-plan> writes a durable tombstone and retires the artifact’s ledger row, and the bytes stay where they are.

Field	Type	Default	Description
physical_delete	bool	false	Reserved for a future protocol-aware VACUUM. false (the only supported value today) tombstones and retires the ledger row only. true is a hard error at apply time today, so the flag fails loudly rather than deleting bytes or silently doing nothing.

rocky restore can rebuild an evicted artifact only when the recorded recipe is non-partitioned, content-addressed, and reads no recorded upstreams. Even then it can refuse — a missing tombstone, a re-derivation whose hash differs, or a lost race on ledger reinstatement all stop it. A recipe with any recorded upstream cannot be restored yet. What is durably retained is the eviction record and the provenance it points at, not a promise that the exact bytes come back. See the governance and reclamation commands.

---

[policy]

State who may change what, and let Rocky enforce it. Each rule maps a (principal, capability, scope) triple to one of three effects. Rocky evaluates the rules at the mutating seams — rocky apply, branch promote, the MCP write tools, and drift auto-apply in rocky run when auto_apply_additive_drift is on — and records every decision in the audit ledger.

An absent block does not gate agents

With no [policy] block, the gate returns NotConfigured and allows the action whoever the principal is. Two surfaces differ. rocky policy check predicts against the safe default posture: agents on mutating actions require review, and humans are never gated. With no block, its prediction is stricter than what enforcement actually does. Drift auto-apply with auto_apply_additive_drift = true also governs as require_review with no block, so it refuses every drift change.

Field	Type	Required	Description
version	integer	Yes	Schema version. Must be 1.
default_agent_effect	string	No	Effect for an agent on a mutating capability when no rule matches. Defaults to "require_review".
rules	list of tables	No	The rules themselves, written as [[policy.rules]].
tests	list of tables	No	Scenario assertions run by rocky policy test, written as [[policy.tests]]. Never read by enforcement.

Effects. "allow" permits the action. "require_review" permits it only after a human signs off. "deny" refuses it, and no allow overturns a deny.

Principals. "human" is a person. "agent" is a non-human caller. In this version you supply the principal explicitly, for example rocky policy check --principal agent.

Capabilities. read (always allowed), propose, apply, promote, backfill, gc, restore, retry, quarantine, plus three refinements of apply and promote: schema_change.additive, schema_change.breaking, and value_change. A rule naming apply or promote also matches those three refinements. Every other capability matches only itself, and a rule naming a refinement matches only that refinement.

[[policy.rules]]

Field	Type	Required	Description
principal	string	Yes	"human" or "agent".
capability	string	Yes	One of the capabilities above.
effect	string	Yes	"allow", "require_review", or "deny".
scope	table	No	Which models the rule covers. See below.
verify_after	list of strings	No	Named checks that must pass after a mutation this rule governed lands. A failing or absent check halts the apply and raises an alert. Empty means no post-apply gate.
autonomy_budget	table	No	{ failures = <n>, window = "<N>d" | "<N>h" }. When verify-after failures inside the rolling window reach failures, the rule degrades to require_review. It only ever tightens, never widens, and it recovers on its own as old failures age out.
conditions	table	No	Parsed and ignored. Reserved so a config written against a later version still loads.

[policy.rules.scope]

A model matches the scope only when it satisfies every key you set.

Field	Type	Description
any	bool	Match every model. Mutually exclusive with every other key.
models	list of strings	Glob patterns (*, ?) over the model name. Matches when the name matches at least one.
tags	table	Required key = value tags. Matches when the model carries all of them.
classifications	list of strings	Matches when the model has at least one column with any listed classification.
exclude_classifications	list of strings	Matches when the model has no column with any listed classification.
contracted	bool	Matches when the model’s contracted status equals this value.
layer	string	Matches when the model’s layer tag equals this value.
max_downstreams	integer	Blast-radius ceiling. Not a match predicate: the rule still matches, but an allow degrades to require_review when the transitive downstream count exceeds the ceiling or cannot be computed. deny and require_review rules are unaffected.

A scope with neither any = true nor any real predicate is rejected at config load, so an empty scope can never silently match everything.

Pipelines that run no models

A replication, snapshot, load or quality pipeline writes tables, not models. The apply gate names each table it writes, and models globs match that name. Use the bare table name, with no catalog or schema:

Pipeline	Name the gate sees
replication	Each target table (and <table><suffix> for a suffix shadow run).
snapshot	target.table, the history table.
load	target.table, and <table>__rocky_stg, the staging table a load with a contract writes first. With no table, Rocky loads one table per file, known only at run time, so the gate sees catalog.schema.
quality with [checks.quarantine] on	Per listed table, what the mode writes: split writes <table><suffix_valid> and <table><suffix_quarantine> (t__valid, t__quarantine by default) and a working table _quarantine_labels_<token> (the token is random per run, so the gate sees _quarantine_labels_; match it with _quarantine_labels_*), drop writes <table><suffix_valid>, tag rewrites <table>. A table entry with no table is seen as catalog.schema.
quality without quarantine	Nothing. It only reads, so no rule applies.

These names carry no tags, layer, classifications or contract, so a rule scoped by those keys does not match them. any = true does. One exception: a table named like a compiled model of the project takes that model’s attributes.

[policy]
version = 1
default_agent_effect = "require_review"

# Agents may add columns to staging models on their own.
[[policy.rules]]
principal = "agent"
capability = "schema_change.additive"
effect = "allow"
scope = { models = ["stg_*"], max_downstreams = 20 }

# Nobody drops a column from a model that carries PII without review.
[[policy.rules]]
principal = "agent"
capability = "schema_change.breaking"
effect = "deny"
scope = { classifications = ["pii"] }

Pin the behaviour you just wrote with [[policy.tests]] scenarios, and run rocky policy test in CI. A later policy edit then cannot open a hole unnoticed. See Testing policies for the scenario fields, and Operating Rocky with agents for where the gates sit.

---

[fulfill]

Configure the agent driver for the experimental rocky fulfill loop. The deterministic commands never read this block. Rocky refuses an unknown field here. See rocky fulfill configuration for the fields and the sandbox caveats.

---

Full Example

A complete Fivetran → Databricks pipeline with governance. The minimal config at the top of this page is the credential-free DuckDB version.

# ──────────────────────────────────────────────────
# Adapters: connections to source and warehouse
# ──────────────────────────────────────────────────
[adapter.fivetran]
type = "fivetran"
kind = "discovery"
destination_id = "${FIVETRAN_DESTINATION_ID}"
api_key = "${FIVETRAN_API_KEY}"
api_secret = "${FIVETRAN_API_SECRET}"

[adapter.prod]
type = "databricks"
host = "${DATABRICKS_HOST}"
http_path = "${DATABRICKS_HTTP_PATH}"
token = "${DATABRICKS_TOKEN}"

# ──────────────────────────────────────────────────
# Pipeline: bronze layer replication
# ──────────────────────────────────────────────────
[pipeline.bronze]
type = "replication"
strategy = "incremental"
timestamp_column = "_fivetran_synced"
metadata_columns = [
    { name = "_loaded_by", type = "STRING", value = "NULL" },
]

[pipeline.bronze.source]
adapter = "prod"

[pipeline.bronze.source.discovery]
adapter = "fivetran"

[pipeline.bronze.source.schema_pattern]
prefix = "src__"
separator = "__"
components = ["source"]

[pipeline.bronze.target]
adapter = "prod"
catalog_template = "warehouse"
schema_template = "stage__{source}"

[pipeline.bronze.target.governance]
auto_create_catalogs = true
auto_create_schemas = true

[pipeline.bronze.target.governance.tags]
managed_by = "rocky"
environment = "production"

[[pipeline.bronze.target.governance.grants]]
principal = "group:data_engineers"
permissions = ["USE CATALOG", "MANAGE"]

[[pipeline.bronze.target.governance.schema_grants]]
principal = "group:data_engineers"
permissions = ["USE SCHEMA", "SELECT", "MODIFY"]

[pipeline.bronze.target.governance.isolation]
enabled = true

[[pipeline.bronze.target.governance.isolation.workspace_ids]]
id = 123456789
binding_type = "READ_WRITE"

[pipeline.bronze.checks]
enabled = true
row_count = true
column_match = true
freshness = { threshold_seconds = 86400 }

[pipeline.bronze.execution]
concurrency = 8
fail_fast = false
table_retries = 1

# ──────────────────────────────────────────────────
# State: persistence backend for watermarks
# ──────────────────────────────────────────────────
[state]
backend = "${ROCKY_STATE_BACKEND:-local}"
# s3_bucket = "${ROCKY_STATE_BUCKET}"
# valkey_url = "${VALKEY_URL}"

---

Hooks

Run a shell command or post to an HTTP endpoint when something happens in a pipeline. A shell hook receives the event’s JSON context on stdin. A webhook posts a templated body to a URL.

# Shell hooks — run a command, pipe JSON context to stdin.
[[hook.on_pipeline_start]]
command = "bash scripts/notify.sh"
timeout_ms = 5000
on_failure = "warn"    # abort | warn | ignore

[[hook.on_materialize_error]]
command = "bash scripts/pagerduty.sh"
on_failure = "ignore"

# Webhooks — HTTP POST with template body
[hook.webhooks.on_pipeline_complete]
url = "https://hooks.slack.com/services/T.../B.../xxx"
preset = "slack"
secret = "${WEBHOOK_SECRET}"

[hook.webhooks.on_materialize_error]
url = "https://events.pagerduty.com/v2/enqueue"
preset = "pagerduty"

Hook Events

Name the table after the event’s config key, always in the on_<event> form. Rocky treats a key without the on_ prefix as unknown and ignores it silently, so check the spelling against this list.

Config key	Trigger
on_pipeline_start	Pipeline execution begins
on_discover_complete	Source discovery finishes
on_compile_complete	Compilation finishes
on_pipeline_complete	Pipeline execution succeeds
on_pipeline_error	Pipeline execution fails
on_before_materialize	Before a table is materialized
on_after_materialize	After a table is materialized
on_materialize_error	Table materialization fails
on_before_model_run	Before a compiled model runs
on_after_model_run	After a compiled model runs
on_model_error	Compiled model execution fails
on_before_checks	Before a table’s quality checks run
on_check_result	A quality check completes
on_after_checks	After a table’s quality checks run
on_drift_detected	Schema drift detected
on_anomaly_detected	Row count anomaly detected
on_state_synced	State store sync completes
on_budget_breach	A run-level budget limit is breached

Hook Config Fields

Field	Type	Default	Description
command	string	required	Shell command to execute
timeout_ms	number	30000	Max execution time in milliseconds
on_failure	string	"warn"	Behavior on failure: abort, warn, or ignore
env	object	{}	Extra environment variables

Webhook Config Fields

Field	Type	Default	Description
url	string	required	Webhook endpoint URL
preset	string	—	Built-in preset: slack, pagerduty, datadog, teams
method	string	"POST"	HTTP method
headers	object	{}	Additional HTTP headers
body_template	string	—	Mustache-style template ({{event}}, {{model}}, {{error}})
secret	string	—	HMAC-SHA256 signing key
timeout_ms	number	10000	Request timeout
async	boolean	false	Fire-and-forget (don’t wait for response)
on_failure	string	"warn"	Behavior on failure
retry_count	number	0	Number of retries
retry_delay_ms	number	1000	Delay between retries