|
This version is still in development and is not considered stable yet. For the latest stable version, please use RIOT-X 2.0! |
Databricks Delta Change Data Feed
The databricks-import command captures existing rows and subsequent changes from Databricks Delta tables and writes Debezium-shaped events to Redis Streams for Redis Data Integration (RDI).
It connects to an existing Databricks SQL warehouse over JDBC; it does not require a Spark cluster in the RIOT-X deployment.
Requirements
You need:
-
A running Databricks SQL warehouse and its JDBC URL and HTTP path.
-
A personal access token with permission to use the warehouse and read the source tables.
-
Delta Change Data Feed (CDF) enabled on every captured table.
-
Stable, unique, non-null key columns for every captured table.
Enable CDF before starting RIOT-X:
ALTER TABLE main.sales.orders
SET TBLPROPERTIES ('delta.enableChangeDataFeed' = true);
Alternatively, grant the collector permission to alter the table and pass --auto-enable-cdf.
Without that option, RIOT-X fails with the exact ALTER TABLE statement required to enable CDF.
Basic usage
Store the PAT in an environment variable rather than placing it in shell history:
export DATABRICKS_TOKEN='<personal-access-token>'
riotx databricks-import main.sales.orders \
--jdbc-url 'jdbc:databricks://dbc-example.cloud.databricks.com:443/default;transportMode=http;ssl=1' \
--http-path '/sql/1.0/warehouses/0123456789abcdef' \
--token-env DATABRICKS_TOKEN \
--table-keys main.sales.orders:order_id
The default stream for this table is data:sales.orders.
RIOT-X reports rdi as the change-record source name by default. Use --source-name
to match a different RDI source.server_name; the same name scopes the saved offsets.
--catalog and --schema provide defaults for shorter table names:
riotx databricks-import orders customers \
--catalog main \
--schema sales \
--jdbc-url "$DATABRICKS_JDBC_URL" \
--http-path "$DATABRICKS_HTTP_PATH" \
--token-env DATABRICKS_TOKEN \
--table-keys orders:order_id \
--table-keys customers:customer_id
Table names may be catalog.schema.table, schema.table, or table when the missing catalog and schema are supplied as defaults.
Snapshots and offsets
The default --snapshot INITIAL lifecycle is:
INIT -> version-pinned SNAPSHOT -> CDF TAIL
RIOT-X captures the current Delta version before reading the first snapshot row, pages the snapshot with the configured key, and starts CDF at the following version. Snapshot cursors and tail offsets are stored in Redis after output chunks are written, so a crash can replay an uncheckpointed chunk but cannot skip it.
If reading a table, writing its changes to Redis, or saving its offset fails, databricks-import stops reading all tables and exits with a non-zero status.
With --termination-log, the error is also written to the termination message file (see Observability).
When restarted, e.g. by Kubernetes, each table resumes from its saved offset.
A check for new changes that fails, e.g. because of a dropped connection, is retried at the next poll; if the checks of a table keep failing for 5 minutes, databricks-import exits.
Use --snapshot NEVER to start from the latest Delta version without copying existing rows.
Use --offset-clear to discard saved state and begin a new snapshot.
Every Databricks CDC table requires key columns in this order, including when using --snapshot NEVER:
-
--table-keys TABLE:COL1,COL2 -
Deprecated global
--key-column -
A primary key reported by Databricks JDBC metadata
If none is available, startup fails with instructions to provide --table-keys.
Key columns must be stable, unique, and non-null for every source row.
Updates that change any configured key column are rejected because v1 does not emit a delete for the old key.
Use --snapshot-chunk-size to control the maximum rows fetched per snapshot page; the default is 50000.
Column projection
Use --table-columns to limit the columns read and emitted for each table:
riotx databricks-import main.sales.orders \
--jdbc-url "$DATABRICKS_JDBC_URL" \
--http-path "$DATABRICKS_HTTP_PATH" \
--token-env DATABRICKS_TOKEN \
--table-keys main.sales.orders:order_id \
--table-columns main.sales.orders:order_id,customer_id,status
Required key columns are added to the projection automatically. Unknown tables and columns are rejected before change processing begins.
Change-event semantics
Delta updates contain two physical CDF rows.
RIOT-X filters the update_preimage row and emits the update_postimage as one logical update event:
| Delta change type | Emitted operation |
|---|---|
|
|
|
|
|
Filtered |
|
|
Updates contain after, deletes contain before, and update before values are not populated in v1.
RDI collector reference
The bundled rdi-riotx-collector chart contains the reference Databricks deployment shape.
The Databricks PAT reuses the existing source database password secret slot: the pod reads the secret’s password key into RIOT_JDBC_PASS and invokes --token-env=RIOT_JDBC_PASS.
See rdi-riotx-collector/k8s/helm/values-databricks-example.yaml for a complete reference render.
The pipeline and job examples are available under example$databricks-import-rdi/.
Native RDI validation and deployment require the follow-on RDI repository work described by tasks 004 and 005.
Limitations
Databricks v1 supports Redis Stream output for RDI only. Direct Redis commands, service-principal OAuth, multiple SQL warehouses in one job, Unity Catalog tag enrichment, and schema evolution during tailing are not supported.
Performance tuning
Use the common --batch, --stream-limit, --poll, and JDBC reader options to tune processing.
Databricks CDC requires --threads=1 to preserve checkpoint ordering within each table.
When multiple tables are configured, RIOT-X still processes them in parallel using one ordered flow per table.
The JDBC pool defaults to 10 connections and must be at least number of tables + 2.
The default therefore supports up to eight tables; twelve tables require --jdbc-pool=14 or larger.
Smaller settings fail at startup with the required size and the configured value.
Exhausted tail cursors release their connections before cleanup and polling, so idle tables do not retain cursor connections.
Use --jdbc-pool or RIOT_JDBC_POOL to set the connection limit; RIOT-X does not increase it automatically with the table count.
The two extra connections are a conservative headroom policy, not a measured minimum for every workload.
Generator mode (--gen) does not connect to Databricks and skips the capacity check.
Large batches and slow downstream writes can still hold connections long enough for other tables to reach the JDBC connection timeout.
Benchmark recommendations will be added after representative SQL warehouse measurements are available.
For the complete option list, run:
riotx databricks-import --help