Schema Patterns

Rocky uses a configurable schema pattern system to parse source schema names into structured components and resolve target catalog/schema names using templates.

Purpose

In multi-tenant data platforms, source schemas follow naming conventions that encode information: which tenant owns the data, which region it came from, which source system produced it. Rocky’s schema pattern system extracts this information and uses it to determine where data should land in the target warehouse.

Configuration

The schema pattern lives on the pipeline source; the templates live on the pipeline target. Both reference the same component names:

[pipeline.bronze.source.schema_pattern]
prefix = "src__"
separator = "__"
components = ["tenant", "regions...", "source"]

[pipeline.bronze.target]
adapter = "prod"
catalog_template = "{tenant}_warehouse"
schema_template = "staging__{regions}__{source}"

Fields

Field	Description
`prefix`	String prefix to strip before parsing. Schemas that don’t start with this prefix are skipped.
`separator`	Delimiter between components.
`components`	Ordered list of named components to extract from the schema name.

Component types

Each entry in the components list defines a named component. The suffix determines how it matches:

Variable (single segment)

A plain name like "tenant" matches exactly one segment.

tenant  →  matches one segment

Variable-length (multi-segment)

A name with ... suffix like "regions..." matches one or more segments. Only one variable-length component is allowed per pattern, and it must not be the last component.

regions...  →  matches 1..N segments

Terminal

The last component in the list always matches exactly one segment (the final segment of the schema name).

source  →  matches the last segment

Parsing examples

Given the pattern prefix = "src__", separator = "__", components = ["tenant", "regions...", "source"]:

Single region

src__acme__us_west__shopify
     │     │        │
     │     │        └── source = "shopify"
     │     └── regions = ["us_west"]
     └── tenant = "acme"

Multiple regions

src__acme__us_west__us_central__shopify
     │     │        │           │
     │     │        │           └── source = "shopify"
     │     └────────┘
     │     regions = ["us_west", "us_central"]
     └── tenant = "acme"

Deep region hierarchy

src__globex__emea__france__paris__zendesk
     │       │     │       │      │
     │       │     │       │      └── source = "zendesk"
     │       └─────┴───────┘
     │       regions = ["emea", "france", "paris"]
     └── tenant = "globex"

Template resolution

Templates use {component_name} placeholders that are replaced with parsed values:

[pipeline.bronze.target]
adapter = "prod"
catalog_template = "{tenant}_warehouse"
schema_template = "staging__{regions}__{source}"

Single-valued components

{tenant} is replaced with the parsed value directly:

{tenant}_warehouse  →  acme_warehouse

Multi-valued components

{regions} is replaced with all values joined by the separator:

staging__{regions}__{source}
→  staging__us_west__shopify              (single region)
→  staging__us_west__us_central__shopify  (multiple regions)

Full resolution example

Source: src__acme__us_west__shopify

Template	Result
`{tenant}_warehouse`	`acme_warehouse`
`staging__{regions}__{source}`	`staging__us_west__shopify`

Target table: acme_warehouse.staging__us_west__shopify.<table_name>

Pinning the join separator at the use site

By default, multi-valued components ({regions}) are joined with the caller-supplied separator. Different call sites supply different separators: target rendering uses target.separator while metadata_columns.value uses pattern.separator. The same placeholder can therefore resolve to different strings depending on which TOML field it appears in, a footgun for templates that hash or compare the rendered value (RLS keys, audit hashes).

Use {name:SEP} to pin the join separator at the use site:

[pipeline.bronze]
metadata_columns = [
    { name = "audit_key", type = "STRING",
      value = "md5('fivetran_{client}_{regions:_}_{source}')" }
    #                              ^^^ join `regions` with "_" regardless of caller default
]

Grammar:

Form	Behavior
`{name}`	Bare form — multi-valued components join with the caller-supplied default separator.
`{name:SEP}`	Explicit form — multi-valued components join with the literal string `SEP` (may be empty, single-, or multi-character). The closing `}` terminates `SEP`, so a literal `}` cannot appear inside it.

:SEP is silently ignored when name resolves to a single-valued component, so swapping a component from single to variadic does not require updating every template.

Error handling

Rocky produces clear errors for invalid schemas:

Condition	Error
Schema doesn’t start with prefix	Schema is skipped (not an error — it’s simply not a managed schema)
Not enough segments for all components	`"schema '<name>' has <actual> segments but pattern requires at least <minimum>"`
Missing required component	`"schema '<name>': no segments remaining for component '<component>'"`

Custom patterns

The schema pattern system is not limited to tenant/regions/source. You can define any components that match your naming convention:

[pipeline.bronze.source.schema_pattern]
prefix = "raw__"
separator = "__"
components = ["environment", "department", "system"]

This would parse raw__prod__finance__sap into:

environment = "prod"
department = "finance"
system = "sap"

And you could use templates like:

[pipeline.bronze.target]
adapter = "prod"
catalog_template = "{environment}_analytics"
schema_template = "{department}__{system}"

Config groups vs schema patterns

Schema patterns route at the pipeline level: they parse a source schema name into components, then fill a target catalog_template / schema_template from those parsed values. The values come from the schema name itself, and a ... component can be multi-valued.

Rocky has a separate model-level routing feature that shares the same template grammar but takes its values from a different place. A config group lives in models/groups/<name>.toml and defines a schema_template once. Each model opts in with group = "<name>" and fills the template’s placeholders from its own [args] block:

schema_template = "mart_{region}"

# models/fct_orders.toml
group = "daily_marts"

[args]
region = "emea"   # fills {region} -> schema "mart_emea"

A group’s schema_template uses the same {name} / {name:SEP} placeholder grammar as the target templates on this page and resolves through the same engine code. Use schema patterns when the routing information is encoded in source schema names; use config groups when a fan-out of models shares one routing and materialization that you set by hand.

See Config groups in the model format reference for the full [args] rules, precedence, enforced groups, and shared tags.

Filtering by parsed component

Once your sources are parsed into components, you can scope rocky plan and rocky compare (and the single-step rocky run sibling) to a subset via the --filter flag. The filter key is one of the component names you declared above (or the reserved id), and the value is matched against the parsed value, with containment semantics for multi-valued (...) components:

# Plan everything for tenant "acme" (then `rocky apply <plan-id>` to execute)
rocky plan --filter tenant=acme

# Compare every source that touches us-west (works because `regions...` is multi-valued)
rocky compare --filter regions=us_west

See the CLI Filters reference for the full syntax, grammar, and common patterns.