Fivetran state cache

Stop every rocky process from re-fetching the same Fivetran metadata. Give the adapter a persistent cache and concurrent processes against one Fivetran org share their discover fetches: the first process pays the API cost, and every process after it reads the same envelope from the cache until the window closes.

Why

Without a shared cache, every rocky process starts from nothing:

   process 1        process 2        process 3
       │                │                │
       ▼                ▼                ▼
   GET /destinations/{id}                       ─┐
   GET /groups/{id}/connectors  (paginated)      │  ~50+ calls
   GET /connectors/{id}/schemas × N connectors  ─┘  per process

                    Fivetran API  ──►  429 Too Many Requests

For a 50-connector tenant that is more than 50 calls per cold start. When several processes converge in one window — a sensor fan-out, say — they exhaust the org’s rate-limit budget and Fivetran answers with 429s.

The cache removes the repetition. A steady-state tenant pays one discover cycle per cache window instead of one per process:

   process 1 ──► cache MISS ──► Fivetran API ──► write envelope
   process 2 ──► cache HIT  ──► envelope
   process 3 ──► cache HIT  ──► envelope

Together with the rate-limit budget (see Cross-process coordination), this is the cold-start equivalent of an asset cache.

Configuration

Add an [adapter.<name>.cache] block under the Fivetran adapter you want to cache:

[adapter.fivetran_main]
type = "fivetran"
kind = "discovery"
api_key = "${FIVETRAN_API_KEY}"
api_secret = "${FIVETRAN_API_SECRET}"
destination_id = "popularity_cultivator"

[adapter.fivetran_main.cache]
backend = "tiered"
file_root = ".rocky/fivetran-state/"
object_store_url = "s3://my-bucket/rocky/fivetran-state/"
valkey_url = "rediss://valkey.internal:6379/"
valkey_ttl_seconds = 600

Omit the [adapter.<name>.cache] block and the backend defaults to "none": every fetch goes straight to the Fivetran API.

Backends

none (default)

No cache at all. Every fetch hits the Fivetran API. Fine for local development, where the API budget does not matter.

file

JSON files on the local filesystem under file_root. The cheapest backend: no external service, no credentials. Use it for a single-host deployment, or for a CI run that wants several Rocky invocations on one machine to share a single discover.

[adapter.fivetran_main.cache]
backend = "file"
file_root = ".rocky/fivetran-state/"

Files land at <account_hash>/<destination_id>.json under file_root. The account_hash is a short token derived from the Fivetran API key with SHA-256. Two orgs sharing one root therefore never collide on the same destination id.

object_store

S3, GCS, Azure, or a file:// path, through the object_store crate. It works across processes and machines, it is durable, and it needs no cache service of its own.

[adapter.fivetran_main.cache]
backend = "object_store"
object_store_url = "s3://my-bucket/rocky/fivetran-state/"

Supported URL schemes:

Scheme	Backend
s3://bucket/prefix/	AWS S3
gs://bucket/prefix/	Google Cloud Storage
az://container/prefix/	Azure Blob Storage
file:///absolute/path/	Local filesystem (mainly for testing)

Credentials come from each SDK’s own default provider chain: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, an IAM role, or ~/.aws/credentials for S3; GOOGLE_APPLICATION_CREDENTIALS for GCS; and so on. Rocky adds no credential surface of its own for cloud storage.

The backend writes single-part PUTs. It compares the response ETag, an MD5 for a single part, against the new bytes. No PUT crosses the wire when the envelope has not changed. It caps the serialized envelope at 5 MB to keep that comparison safe.

valkey

Redis or Valkey. Reads are fast, so this is the hot path for sensors and sync detection that need recent data. The released rocky binary includes it.

[adapter.fivetran_main.cache]
backend = "valkey"
valkey_url = "rediss://valkey.internal:6379/"
valkey_ttl_seconds = 600

Every key sits under the fivetran-state: namespace, so you can list them with KEYS fivetran-state:*. Every SET carries the configured TTL, so the layer never accumulates stale envelopes.

tiered

Valkey in front, an object store behind. A read tries Valkey first; on a miss it falls through to the object store and writes the result back into Valkey. A write goes to both.

[adapter.fivetran_main.cache]
backend = "tiered"
object_store_url = "s3://my-bucket/rocky/fivetran-state/"
valkey_url = "rediss://valkey.internal:6379/"
valkey_ttl_seconds = 600

Use this in production. Valkey serves the hot path fast, and the object store keeps the envelope through a Valkey outage or a pod restart.

--no-cache flag

rocky discover --no-cache skips the cache read for this one invocation and fetches from the API. Rocky still writes the fresh envelope back, so the next invocation sees the up-to-date data.

rocky discover --emit-fivetran-state-to /tmp/state.json --no-cache

Reach for it when you suspect the cache has gone stale, after rolling a Fivetran credential for instance. It gives you the next envelope straight from the wire.

Hash-dedupe

Every backend hashes the envelope before writing it, excluding fetched_at, and compares that digest against what is already cached. Matching hashes mean the write does nothing:

• FileCache — skips the rename entirely; on-disk mtime stays stable.
• ObjectStoreCache — HEAD the existing object; if its ETag matches MD5(new_bytes), skip the PUT.
• ValkeyCache — GET the prior value; if hashes match, skip the SET.
• TieredCache — both layers run their own dedupe.

This is the herd protection: however many processes converge on an identical envelope, the cache is written at most once per window.

Observability

Every cache decision emits a span event over OTLP, the wire format OpenTelemetry uses, and publishes the same event on the in-process pipeline event bus:

Event	Fields
fivetran.cache_hit	backend, key, age_seconds
fivetran.cache_miss	backend, key, reason ("no-entry" / "refresh-forced")
fivetran.cache_write	backend, key, bytes, outcome ("written")
fivetran.cache_write_skipped	backend, key, reason ("hash-match")
fivetran.cache_write_failed	backend, key, error

The cache fails open. When a backend is unreachable Rocky still fetches over HTTP, so a broken cache slows a run down but never stops it. Each failure emits a cache_write_failed event and a warning log line, so you can alert on a backend that fails consistently.

Cross-process coordination

Three more optional blocks on the same Fivetran adapter coordinate processes through Valkey. Each one fails open: when its backend is unreachable, Rocky calls the API directly.

Block	Default	What it does
[adapter.<name>.ratelimit]	backend = "file": a per-host budget under ${TMPDIR}/rocky-fivetran-ratelimit/	After one process gets a 429, the others on the same budget wait for the same wake_at. backend = "valkey" shares the budget across hosts. Keys: valkey_url, max_wake_seconds (default 600).
[adapter.<name>.stampede]	backend = "none": every process is the leader	On a cold cache, one leader calls the API and the others poll the cache. Keys: valkey_url, lock_ttl_seconds (default 60), poll_timeout_seconds (default 30).
[adapter.<name>.circuit_breaker]	backend = "none": no breaker	Stops API calls for the whole org after repeated failures, then probes again after a cooldown. Keys: valkey_url, failure_threshold (default 5), window_seconds (default 60), cooldown_seconds (default 300), cooldown_max_seconds (default 3600).

A valkey backend without valkey_url fails the config load. Rocky ignores these blocks on every adapter type except fivetran.

[adapter.fivetran_main.stampede]
backend = "valkey"
valkey_url = "rediss://valkey.internal:6379/"