Skip to content
RockyRockyRocky

Quickstart

This page copies Fivetran-landed source tables into Databricks. It takes about five minutes.

  1. Scaffold the project.

    Terminal window
    rocky init my-pipeline
    cd my-pipeline

    This writes a DuckDB starter project. Step 2 replaces its rocky.toml.

  2. Configure a source, a target, and the pipeline that wires them together.

    [adapter.fivetran]
    type = "fivetran"
    kind = "discovery"
    destination_id = "${FIVETRAN_DESTINATION_ID}"
    api_key = "${FIVETRAN_API_KEY}"
    api_secret = "${FIVETRAN_API_SECRET}"
    

    [adapter.prod]
    type = "databricks"
    host = "${DATABRICKS_HOST}"
    http_path = "${DATABRICKS_HTTP_PATH}"
    token = "${DATABRICKS_TOKEN}"
    

    [pipeline.bronze]
    type = "replication"
    strategy = "incremental"
    timestamp_column = "_fivetran_synced"
    

    [pipeline.bronze.source]
    adapter = "prod"
    

    [pipeline.bronze.source.discovery]
    adapter = "fivetran"
    

    [pipeline.bronze.source.schema_pattern]
    prefix = "src__"
    separator = "__"
    components = ["source"]
    

    [pipeline.bronze.target]
    adapter = "prod"
    catalog_template = "warehouse"
    schema_template = "stage__{source}"
    

    [pipeline.bronze.target.governance]
    auto_create_catalogs = true
    auto_create_schemas = true
    

    [pipeline.bronze.checks]
    enabled = true
    row_count = true
    column_match = true
    freshness = { threshold_seconds = 86400 }
    

    [state]
    backend = "local"

    Export the environment variables this config references (DATABRICKS_HOST, FIVETRAN_API_KEY, and the rest) before you run Rocky. Other warehouses: Connect to a warehouse.

  3. Validate the config.

    Terminal window
    rocky validate

    Checks config syntax and adapter wiring. It does not call external APIs.

  4. Discover the sources.

    Terminal window
    rocky -o table discover

    Calls the Fivetran API and lists connectors matching the schema pattern.

  5. Build a plan.

    Terminal window
    plan_id=$(rocky plan --filter source=orders --output json | jq -r .plan_id)

    --filter source=orders selects the src__orders schema. Rocky compiles the pipeline, detects drift, and writes a plan to .rocky/plans/<plan_id>.json. A plan is the exact work Rocky will do, written down first. Read it before you apply it. See the glossary.

  6. Apply the plan.

    Terminal window
    rocky apply "$plan_id"

    Rocky runs the plan in order: create catalogs and schemas, apply drift, copy data, run checks. It prints a JSON result. Checks run after the copy, so a failed check cannot undo it.

The one-command path

Section titled “The one-command path”

rocky run --filter source=orders plans and applies in one step. Use it for local iteration and automation. Add --resume-latest to resume from the last checkpoint after a failure.

Next steps

Section titled “Next steps”