Databricks Delta Change Data Feed

The databricks-import command captures existing rows and subsequent changes from Databricks Delta tables and writes Debezium-shaped events to Redis Streams for Redis Data Integration (RDI). It connects to an existing Databricks SQL warehouse over JDBC; it does not require a Spark cluster in the RIOT-X deployment.

Requirements

You need:

A running Databricks SQL warehouse and its JDBC URL and HTTP path.
A personal access token with permission to use the warehouse and read the source tables.
Delta Change Data Feed (CDF) enabled on every captured table.
Stable, unique, non-null key columns for every captured table.

Enable CDF before starting RIOT-X:

ALTER TABLE main.sales.orders
SET TBLPROPERTIES ('delta.enableChangeDataFeed' = true);

Alternatively, grant the collector permission to alter the table and pass --auto-enable-cdf. Without that option, RIOT-X fails with the exact ALTER TABLE statement required to enable CDF.

Basic usage

Store the PAT in an environment variable rather than placing it in shell history:

export DATABRICKS_TOKEN='<personal-access-token>'

riotx databricks-import main.sales.orders \
  --jdbc-url 'jdbc:databricks://dbc-example.cloud.databricks.com:443/default;transportMode=http;ssl=1' \
  --http-path '/sql/1.0/warehouses/0123456789abcdef' \
  --token-env DATABRICKS_TOKEN \
  --table-keys main.sales.orders:order_id

The default stream for this table is data:sales.orders.

RIOT-X reports rdi as the change-record source name by default. Use --source-name to match a different RDI source.server_name; the same name scopes the saved offsets.

--catalog and --schema provide defaults for shorter table names:

riotx databricks-import orders customers \
  --catalog main \
  --schema sales \
  --jdbc-url "$DATABRICKS_JDBC_URL" \
  --http-path "$DATABRICKS_HTTP_PATH" \
  --token-env DATABRICKS_TOKEN \
  --table-keys orders:order_id \
  --table-keys customers:customer_id

Table names may be catalog.schema.table, schema.table, or table when the missing catalog and schema are supplied as defaults.

Snapshots and offsets

The default --snapshot INITIAL lifecycle is:

INIT -> version-pinned SNAPSHOT -> CDF TAIL

RIOT-X captures the current Delta version before reading the first snapshot row, pages the snapshot with the configured key, and starts CDF at the following version. Snapshot cursors and tail offsets are stored in Redis after output chunks are written, so a crash can replay an uncheckpointed chunk but cannot skip it.

Use --snapshot NEVER to start from the latest Delta version without copying existing rows. Use --offset-clear to discard saved state and begin a new snapshot.

Every Databricks CDC table requires key columns in this order, including when using --snapshot NEVER:

--table-keys TABLE:COL1,COL2
Deprecated global --key-column
A primary key reported by Databricks JDBC metadata

If none is available, startup fails with instructions to provide --table-keys. Key columns must be stable, unique, and non-null for every source row. Updates that change any configured key column are rejected because v1 does not emit a delete for the old key.

Use --snapshot-chunk-size to control the maximum rows fetched per snapshot page; the default is 50000.

Column projection

Use --table-columns to limit the columns read and emitted for each table:

riotx databricks-import main.sales.orders \
  --jdbc-url "$DATABRICKS_JDBC_URL" \
  --http-path "$DATABRICKS_HTTP_PATH" \
  --token-env DATABRICKS_TOKEN \
  --table-keys main.sales.orders:order_id \
  --table-columns main.sales.orders:order_id,customer_id,status

Required key columns are added to the projection automatically. Unknown tables and columns are rejected before change processing begins.

Change-event semantics

Delta updates contain two physical CDF rows. RIOT-X filters the update_preimage row and emits the update_postimage as one logical update event:

Delta change type Emitted operation

Delta change type	Emitted operation
`insert`	`c` (create)
`delete`	`d` (delete)
`update_preimage`	Filtered
`update_postimage`	`u` (update)

insert

c (create)

delete

d (delete)

update_preimage

Filtered

update_postimage

u (update)

Updates contain after, deletes contain before, and update before values are not populated in v1.

RDI collector reference

The bundled rdi-riotx-collector chart contains the reference Databricks deployment shape. The Databricks PAT reuses the existing source database password secret slot: the pod reads the secret’s password key into RIOT_JDBC_PASS and invokes --token-env=RIOT_JDBC_PASS.

See rdi-riotx-collector/k8s/helm/values-databricks-example.yaml for a complete reference render. The pipeline and job examples are available under example$databricks-import-rdi/.

Native RDI validation and deployment require the follow-on RDI repository work described by tasks 004 and 005.

Limitations

Databricks v1 supports Redis Stream output for RDI only. Direct Redis commands, service-principal OAuth, multiple SQL warehouses in one job, Unity Catalog tag enrichment, and schema evolution during tailing are not supported.

Performance tuning

Use the common --batch, --stream-limit, --poll, and JDBC reader options to tune processing. Databricks CDC requires --threads=1 to preserve checkpoint ordering within each table. When multiple tables are configured, RIOT-X still processes them in parallel using one ordered flow per table. The JDBC pool must provide at least one connection per table plus two connections for polling and checkpoint work. Set --jdbc-pool to at least number of tables + 2; the default is 10. Benchmark recommendations will be added after representative SQL warehouse measurements are available.

For the complete option list, run:

riotx databricks-import --help