Skip to content
RockyRockyRocky

Quickstart

This page builds a Rocky pipeline that copies Fivetran-landed source tables into Databricks. It takes about five minutes.

There are two ways to reach the warehouse. Both start the same way.

  Shared setup
  ────────────
  rocky init        scaffold the project
  rocky validate    check config + adapter wiring, no network
  rocky discover    ask the source which tables exist
        │
        ├───────────────────────────┐
        │ PR-gated deploy           │ Local iteration
        ▼                           ▼
  rocky plan                    rocky run
        │ writes                    │ plans and applies
        ▼ .rocky/plans/<id>.json    │ in one step
  you review the SQL and drift      │
        │                           │
        ▼                           │
  rocky apply <id>                  │
        │                           │
        └────────────┬──────────────┘
                     ▼
                 warehouse

Steps 1 to 4 below are the shared setup. Steps 5 and 6 are the PR-gated path. The local path is one command, covered after the steps. Step 7 inspects what the run stored.

  1. Scaffold the project.

    Terminal window
    rocky init my-pipeline
    cd my-pipeline

    This writes a runnable DuckDB starter project. You get a rocky.toml, a models/ directory holding _defaults.toml and a sample stg_orders model, and a seeds/seed.sql. Step 2 replaces the generated rocky.toml with the Fivetran to Databricks config below.

  2. Configure a source, a target, and the pipeline that wires them together.

    [adapter.fivetran]
    type = "fivetran"
    kind = "discovery"
    destination_id = "${FIVETRAN_DESTINATION_ID}"
    api_key = "${FIVETRAN_API_KEY}"
    api_secret = "${FIVETRAN_API_SECRET}"
    

    [adapter.prod]
    type = "databricks"
    host = "${DATABRICKS_HOST}"
    http_path = "${DATABRICKS_HTTP_PATH}"
    token = "${DATABRICKS_TOKEN}"
    

    [pipeline.bronze]
    type = "replication"
    strategy = "incremental"
    timestamp_column = "_fivetran_synced"
    

    [pipeline.bronze.source]
    adapter = "prod"
    

    [pipeline.bronze.source.discovery]
    adapter = "fivetran"
    

    [pipeline.bronze.source.schema_pattern]
    prefix = "src__"
    separator = "__"
    components = ["source"]
    

    [pipeline.bronze.target]
    adapter = "prod"
    catalog_template = "warehouse"
    schema_template = "stage__{source}"
    

    [pipeline.bronze.target.governance]
    auto_create_catalogs = true
    auto_create_schemas = true
    

    [pipeline.bronze.checks]
    enabled = true
    row_count = true
    column_match = true
    freshness = { threshold_seconds = 86400 }
    

    [state]
    backend = "local"

    An [adapter.*] block defines one connection. A [pipeline.*] block ties adapters together into a unit of work. Select between pipelines with --pipeline NAME. Export the environment variables this config references (DATABRICKS_HOST, FIVETRAN_API_KEY, and the rest) before you run Rocky.

  3. Validate the config.

    Terminal window
    rocky validate

    Checks config syntax and adapter wiring. It does not call external APIs.

  4. Discover the sources.

    Terminal window
    rocky -o table discover

    Calls the Fivetran API and lists connectors matching the schema pattern.

  5. Build a plan.

    Terminal window
    plan_id=$(rocky plan --filter tenant=acme --output json | jq -r .plan_id)

    Rocky compiles the pipeline, detects drift, and records a plan keyed by plan_id. A plan is the exact work Rocky will do, written down before it does any of it. The same inputs always produce the same plan. Read the SQL, the drift actions, and the checks before you commit to a run. See the glossary.

  6. Apply the plan.

    Terminal window
    rocky apply "$plan_id"

    Rocky executes the plan in order: discover, create catalogs and schemas, apply drift, copy data, run checks. It prints a versioned JSON result holding the materializations, check results, drift actions, and permissions.

  7. Inspect the state.

    Terminal window
    rocky state

    Shows the stored watermark for every table. A watermark is the timestamp of the newest row Rocky has already loaded. See the glossary.

The one-command path

Section titled “The one-command path”

rocky run does the work of steps 5 and 6 in one command. Use it for local iteration and for automation.

Terminal window
rocky run --filter tenant=acme

Either path resumes from the last checkpoint after a failure. Add --resume-latest:

Terminal window
rocky run --filter tenant=acme --resume-latest

Next steps

Section titled “Next steps”