Observability

Rocky is instrumented from the inside. A run emits OpenTelemetry spans and metrics, writes structured logs to stderr, and keeps a local JSONL record of each log event with its span context on disk. You point the exporter at a collector, or you read the local files, and you can see exactly what a run did without adding a single line of code to your models.

This guide covers what Rocky emits, how to turn on OTLP export, and how to stand up a local Grafana stack that renders it.

What Rocky emits

Four signals come out of every run:

• Traces — OpenTelemetry spans for the phases of a run, exported over OTLP gRPC.
• Metrics — run counters and duration histograms, also exported over OTLP.
• Structured logs — log events on stderr, emitted as JSON (one object per line) when Rocky is producing JSON output, and in a compact human-readable form otherwise.
• Local trace files — each log event, with its active span chain, persisted as JSONL under .rocky/traces/.

The OTLP signals are opt-in (set one environment variable). The structured logs and local trace files are always on.

Spans

Spans are created around the phases of a run, so a trace reads as the shape of the run:

Span	Emitted by	What it covers
run	rocky run	the whole run, carrying the run_id — the root of a run’s trace unless an upstream context was supplied
discover_sources	source discovery	resolving configured sources before planning (an entered span — it parents the discovery work)
materialize.table	model execution	building one model; one span per model, carrying rocky.model.fqn and rocky.model.kind
statement.execute	warehouse adapter	one SQL statement sent to the warehouse
state.download / state.upload / state.probe	object-store state sync	reading and writing remote state, carrying the backend and bucket
scheduler.tick	resident scheduler	one reconciliation pass of rocky serve --scheduler, carrying rocky.scheduler.outcome and the pass’s due/executed/failed/skipped counts (experimental)

When Rocky drives a warehouse, the statement.execute spans nest under the materialize.table span for the model being built, so a single model’s SQL is grouped under it.

A few honest limits worth knowing:

• statement.execute spans come from the warehouse adapters. The Databricks, Snowflake, BigQuery, and Trino adapters emit them. The embedded DuckDB engine runs statements in-process and does not, so local DuckDB runs show model-level spans but no per-statement spans. The rocky.adapter.name, rocky.statement.kind, and rocky.warehouse.name attributes are carried by the Databricks, Snowflake, and BigQuery spans; Trino emits a simpler statement.execute span (and a separate statement.execute_arrow span on its Arrow path).
• scheduler.tick comes from the resident scheduler, not a run. It is emitted once per poll interval by rocky serve --scheduler (experimental), so it is its own trace root rather than a child of a run span — the runs it launches hang off it, not the other way around. Each pass records its outcome and counts even when nothing ran, so an idle loop and a wedged one stay distinguishable in the trace.
• rocky.query_duration_ms is populated by the Databricks adapter today. Other adapters record their statement spans without feeding that metric yet.
• governance_setup and batched_checks are marker spans. They are declared to delimit the governance and quality-check phases, but they do not currently parent the events emitted during those phases (those events stay at the root of the trace). Treat them as phase markers, not as subtrees.

Connected traces

Rocky speaks the W3C trace context convention in both directions, through the TRACEPARENT environment variable.

Inbound. When TRACEPARENT is set in a Rocky process’s environment, the root spans that process opens become children of the span it names, so the run appears inside the caller’s trace rather than starting a new one. That is what connects rocky serve --scheduler (experimental) to what it runs: each scheduler.tick describes itself, hands the value to the rocky run child it launches, and the child’s run span lands under that tick. One trace then answers both “what did this run do?” and “what decided to run it?”.

Outbound. The same variable is how anything else hands Rocky a trace to join. A CI job, a shell wrapper, or an orchestrator that exports TRACEPARENT before invoking Rocky gets its spans nested under its own — Rocky exports observability rather than owning it, so it joins whatever trace it is given.

Worth knowing before you turn it on:

• Any invocation is affected, not just scheduler children. If TRACEPARENT is exported in an environment, every Rocky command launched there roots its trace under it — run, compile, plan, and the rest. Unset it (or unset it for a specific invocation) when you want an independent trace. A command that keeps working after it starts (rocky run --watch) holds the parent it was launched with, so every one of its iterations lands in that same trace.
• In-process background work opens its own trace. Spans a command emits from a spawned task do not join: the resident scheduler’s own loop, the periodic state uploader, and the replication path’s per-table spans (which already root themselves today, independent of the run span). So a connected trace shows the run, not necessarily every span the process emits. Subprocesses are the opposite case — they inherit the environment, so they adopt the same context: a job submitted to rocky serve does its work in a child rocky run, which means every job a server runs joins the trace that server was launched with. Start a long-lived rocky serve with TRACEPARENT exported and every job it ever runs reports under that one launch-time span; unset the variable for the server process if you want each job to trace independently.
• An upstream sampling decision now applies to the spans that inherit it. Rocky samples with ParentBased(AlwaysOn), so a TRACEPARENT whose flags say “not sampled” (ending in -00) drops the run span and everything nested under it. The self-rooting spans in the bullet above are sampled on their own and still export, so a not-sampled upstream quietens a run’s trace rather than guaranteeing silence — on a replication run you will still see the per-table spans. The run itself is unaffected either way: it executes normally, only what gets exported changes.
• A malformed value is ignored. A truncated or corrupted TRACEPARENT never fails a command; the process simply roots its own trace, exactly as if the variable were unset.
• The Dagster integration does not set TRACEPARENT today. A Dagster-launched Rocky run joins Dagster’s trace only if something upstream of the process (the run launcher’s environment, for instance) exports it.

Inbound context needs OTLP export configured to be visible: without OTEL_EXPORTER_OTLP_ENDPOINT there is no exporter, so there is nothing to connect and nothing is emitted.

Metrics

Metrics are recorded in-process and exported when the run finishes (the recorded values are flushed as the run exits). Counters are exported as gauges (the last value over the run) and durations as histograms, so the backend computes its own percentiles.

Metric	Type	Meaning
rocky.tables_processed	gauge	tables materialized in the run
rocky.tables_failed	gauge	tables that failed
rocky.statements_executed	gauge	SQL statements executed
rocky.retries_attempted	gauge	retries attempted
rocky.retries_succeeded	gauge	retries that recovered
rocky.anomalies_detected	gauge	anomalies flagged by checks
rocky.error_rate_pct	gauge	failed tables as a percentage
rocky.table_duration_ms	histogram	per-table materialization time, in milliseconds
rocky.query_duration_ms	histogram	per-statement warehouse time, in milliseconds

Not every metric is populated on every run — instrumentation is adapter- and pipeline-specific today. rocky.statements_executed is emitted by the Databricks adapter; the retry counters by the Databricks and BigQuery adapters; and the table counters and durations (rocky.tables_processed, rocky.tables_failed, rocky.table_duration_ms) by replication runs. Other adapters and pipeline types still emit spans and logs, but do not feed these particular counters yet, so a metric you don’t see means that path isn’t instrumented, not that nothing happened.

Resident scheduler metrics

rocky serve --scheduler (experimental) is a long-lived process, so its reconciler exports these for the lifetime of the process rather than at the end of a run. Unlike the run metrics above, the counters are true monotonic counters — read them with rate(), not last-value. They appear only while the scheduler is running:

Metric	Type	Meaning
rocky.scheduler.ticks	counter	reconciliation passes the loop has run, split by an outcome label
rocky.scheduler.due	counter	demands that came due, summed across ticks
rocky.scheduler.executed	counter	demands that ran to a terminal outcome
rocky.scheduler.skipped	counter	demands suppressed, split by a reason label
rocky.scheduler.lag_seconds	histogram	seconds between a demand’s logical time and its execution
rocky.scheduler.consecutive_failures	gauge	current run of consecutive failed runs, per pipeline

The outcome label on rocky.scheduler.ticks is one of completed, config_error, permit_held, lock_skipped, or fault — summed across outcomes it is the loop’s liveness signal, and split by outcome it makes an execution-blocking pass visible: a scheduler that ticks every minute but fails to parse rocky.toml shows up as outcome=config_error rather than looking identical to a healthy idle loop. The reason label on rocky.scheduler.skipped is one of a small fixed set (not_due, disabled, in_flight, catchup_skipped, failure_backoff, partial_backoff, dedup, history_unavailable, state_busy), and the pipeline label is a configured pipeline name, so no label can grow unbounded. rocky.scheduler.consecutive_failures is kept in-process: it resets to zero when the scheduler restarts, and a pipeline removed from config leaves its last value behind as a stale series. Alert on it alongside a freshness check rather than on its own.

Structured logs

When Rocky is producing JSON output, it writes logs to stderr as JSON, one object per line (stdout stays reserved for the --output json result of discover, plan, and run); interactive runs use a compact human-readable form instead. The active run span carries a run_id field — the same id you see in rocky history and rocky trace — so events emitted within a run tie back to it.

{"timestamp":"2026-07-17T09:14:22.481Z","level":"INFO","fields":{"message":"idempotency key claimed — proceeding","backend":"local"},"span":{"run_id":"run-20260717-091422-481","name":"run"},"target":"rocky::run"}

Log verbosity is controlled by RUST_LOG. The default keeps Rocky’s own spans at info while quieting the query-cache trace:

RUST_LOG=info rocky run -c rocky.toml

Local trace files

Independently of any collector, Rocky writes one JSONL file per process under .rocky/traces/, named {timestamp}-{pid}.jsonl. Each line is a JSON log event with its timestamp, level, target, fields, and active span chain. This is on by default and needs no configuration.

• The last 20 files are kept; set ROCKY_TRACE_RETAIN_RUNS to change the count.
• Set ROCKY_TRACE_DISABLE=1 to turn the files off entirely (useful on a shared filesystem or under tight disk quota).

These files are structured event records with span context that you can inspect directly or ship to any log tool. They are separate from rocky trace and rocky replay, which render a run’s timeline from the persisted run record in Rocky’s state store rather than from these files.

Turning on OTLP export

Export activates when OTEL_EXPORTER_OTLP_ENDPOINT is set. That single variable switches on both traces and metrics; when it is unset, the exporter is never constructed and there is no runtime cost.

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_SERVICE_NAME=rocky   # optional; defaults to "rocky"
rocky run -c rocky.toml

• OTEL_EXPORTER_OTLP_ENDPOINT is the gRPC endpoint of your OTLP collector.
• OTEL_SERVICE_NAME sets the service.name resource attribute, which is how you filter Rocky’s data in the backend. It defaults to rocky.

If the endpoint is unreachable, Rocky logs a warning and keeps running; export is best-effort and never fails a run.

OTLP export requires a Rocky release built with the OpenTelemetry exporter. Earlier release binaries were compiled without it, so setting these variables had no effect on them — see the CHANGELOG for the release that adds it.

A local Grafana stack

The repository ships a single-node Grafana, Alloy, Tempo, Loki, and Prometheus stack under deploy/observability/ for watching runs locally. It is a development-grade quickstart, not a hardened production deployment.

From that directory:

# 1. Start the stack (Grafana, Alloy, Tempo, Loki, Prometheus).
docker compose up -d

# 2. Run a pipeline with export pointed at Alloy's OTLP receiver.
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_SERVICE_NAME=rocky \
  rocky run -c rocky.toml

# 3. Open Grafana at http://localhost:3000.

In Grafana, open the Tempo data source and search for traces where service.name is rocky. Pick a recent trace to see the run laid out span by span: the model-level materialize.table spans, and the statement.execute spans nested under them when the run drives a warehouse. The provisioned Rocky Runs dashboard renders the run metrics (tables processed and failed, error rate, and the duration histograms) from Prometheus. A second Rocky Scheduler dashboard renders the resident scheduler’s lag, due/executed/skipped-by-reason, and per-pipeline consecutive failures, and populates once a rocky serve --scheduler process is exporting.

Tear the stack down when you are done. The exact commands, ports, and dashboard details live in the deploy/observability/ README.

Collecting the structured logs

Traces and metrics reach the backend over OTLP. To get Rocky’s structured JSON logs into the same place, tail them with Alloy and lift the run’s run_id into a label so log lines line up with run records.

Capture Rocky’s stderr to a file, for example when cron or a systemd unit runs it:

rocky run -c rocky.toml 2>> /var/log/rocky/run.jsonl

Then have Alloy tail that file, parse each JSON line, and label it by run_id (which the run span carries in the span context):

local.file_match "rocky_logs" {
  path_targets = [{ __path__ = "/var/log/rocky/*.jsonl" }]
}

loki.source.file "rocky_logs" {
  targets    = local.file_match.rocky_logs.targets
  forward_to = [loki.process.rocky.receiver]
}

loki.process "rocky" {
  // Pull run_id out of the run span context on each log line.
  stage.json {
    expressions = { run_id = "span.run_id" }
  }
  stage.labels {
    values = { run_id = "" }
  }
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Because the run span carries the same run_id that rocky history and rocky trace report, events emitted during a run are labeled with it in Loki, and you can filter by a run_id to pull those lines and cross-reference them against the run’s record in Rocky’s own state. Events emitted outside the run span (early startup lines, for instance) won’t carry the label.