Redis integration guide for Ray Data¶
Redis must run separately with the Query Engine (Search). See installation and driver discovery for Python/native setup, supported wheel platforms, TLS and source builds. Tables must be created, ingested or adopted through the driver; arbitrary Redis keys do not automatically become SQL tables. Writes autocommit without rollback.
This guide describes current repository capabilities. A newly added helper or
extra may require a build containing its implementation. If a released wheel
lacks it, build the native driver from this checkout and install the relevant
extra from ./python; set REDIS_ADBC_LIBRARY to the built library. Do not infer
that an older published wheel contains every helper documented here.
Ray Data¶
Worker-side Arrow reader¶
Complete the shared example setup for the sample-data helper before running the first block. It creates and removes a unique input table.
The source-tree ray extra pins Ray 2.59.0. Install this build on the caller
and every worker (Python 3.12+), including its matching native library:
python -m pip install './python[ray]'
import os
import ray
from integration_guide_data import example_table
from adbc_driver_redis.ray import RedisDatasource
ray.init(num_cpus=2, include_dashboard=False)
try:
with example_table() as name, RedisDatasource(
os.environ["REDIS_URI"], f'SELECT id, label FROM "{name}" WHERE id >= ?',
parameters=(1,), partition_count=2, batch_rows=65536,
) as source:
ds = ray.data.read_datasource(source, concurrency=2)
rows = sorted(ds.take_all(), key=lambda row: row["id"])
assert rows == [{"id": 1, "label": "alpha"}, {"id": 2, "label": None}, {"id": 3, "label": "雪"}]
print(rows)
# Persist the tool's computed output, then read it through a fresh connection.
from integration_guide_data import write_result
import pyarrow as pa
write_result(pa.Table.from_pylist(rows))
finally:
ray.shutdown()
read_query and RedisDatasource eagerly plan a supported SELECT through ExecutePartitions and
closes the planning handles. Ray read tasks open independent connections and
consume ReadPartition Arrow batches incrementally. Neither Python rows nor the
complete result are materialized on the caller. Empty reads retain the planned
Dataset schema; Ray may omit zero-row blocks from to_arrow_refs().
The live partition SQL subset applies. Unsupported queries fail at planning; there is no automatic materializing fallback. The adapter explicitly selects live mode. Retained results, manual SQL splits, writes and Flight SQL are outside this adapter's scope. Existing Ray SQL read/write paths below are separate integrations.
partition_count requests 1..1024 splits; sparse or empty ranges can make them
uneven. concurrency caps simultaneous Ray read tasks independently of split
count. batch_rows controls native Arrow batch rows, not total pipeline memory;
Ray can combine or split blocks and buffer data in its object store. Output is
unordered. Reads and retries share the planned managed-table snapshot until its
partition group expires (five minutes by default) or is explicitly released.
Updates, TRUNCATE and DROP preserve that generation. Construct a new datasource
to observe later writes. Expired or released descriptors require a new plan;
there is no exactly-once processing guarantee.
read_query leaves the group available for lazy execution and retries until
expiry. For explicit early release, use the datasource as a context manager and
finish the complete computation inside it, as in the example above.
source.close() also releases the group and is idempotent. Already acquired
readers retain their own pins, while new task attempts using released descriptors
fail. Finish computations and desired retries before closing. Explicit release
avoids retaining completed plans and their old generations until expiry.
db_kwargs and conn_kwargs are passed to caller and worker connections.
For private TLS CAs, use a verified rediss:// URI and
db_kwargs={"adbc.redis.tls_ca_file": "/worker/local/ca.crt"}; the file must
exist at that path on every node. Use a worker-reachable endpoint, matching
native builds and a trusted Ray cluster: connection configuration can contain
credentials, and descriptors can contain SQL literals and bound values.
Named ACL users may need a positive, deployment-guaranteed
partition_scan_cap; the partition permission contract
also describes Search readiness and metadata permissions.
Reader, cursor and connection contexts close on completion, errors and cooperative cancellation/unwinding. Abrupt worker termination cannot guarantee Python cleanup; Redis cursor expiry remains the fallback. A failed attempt can have consumed batches before its failure; follow Ray's task retry behavior and avoid side effects in downstream processing that assumes exactly-once delivery.
See the worker validation record.
This is a repository adapter, not an upstream Ray integration or endorsement.
The ray extra and helper require a build containing this change; an older
published wheel will not supply them.
Existing SQL reader and writer¶
Install ray[data] alongside adbc-driver-redis[dbapi]. Ray's
read_sql
accepts a factory that opens a fresh DB-API connection for each task:
import os
from uuid import uuid4
from functools import partial
import ray
from adbc_driver_redis import dbapi
ray.init(num_cpus=2, include_dashboard=False)
connect = partial(dbapi.connect, os.environ["REDIS_URI"])
name = "guide_ray_sql_" + uuid4().hex
try:
ds = ray.data.read_sql("SELECT 1 AS one", connect, override_num_blocks=1)
with connect() as conn, conn.cursor() as cur:
try:
cur.execute(f'CREATE TABLE "{name}" (one bigint)')
ds.write_sql(f'INSERT INTO "{name}" (one) VALUES (?)', connect)
cur.execute(f'SELECT one FROM "{name}"')
assert cur.fetchall() == [(1,)]
print("REDIS_ADBC_READ_WRITE_OK")
finally:
cur.execute(f'DROP TABLE IF EXISTS "{name}"')
finally:
ray.shutdown()
Dataset.write_sql("INSERT INTO existing_table (id) VALUES (?)", connect)
also works through the driver's executemany. Create the destination table
first. Writes autocommit; failed or retried tasks can leave partial writes or
duplicates. This integration does not provide transactional or exactly-once writes.
To retain the driver's Arrow types and avoid Ray's SQL reader converting rows
through Python objects, fetch with cur.fetch_arrow_table() and pass the table
to ray.data.from_arrow(table). This materializes the query result on the
calling process before Ray distributes it; it is not a distributed database read.
On 2026-10-29, local Ray 2.59.0 with Python 3.12.14, PyArrow 26.0.0 and
ADBC driver manager 1.13.0 passed all three opt-in checks against Redis Software 8.6.2 with verified
active 4x QPF:
parameterized SQL reads with filters, NULLs and aggregates; a 300-row SQL write
round trip; and Arrow ingestion, fetch and Ray batch transformation. SQL reads
used one read task. Multiple input write blocks do not guarantee concurrent
write tasks because Ray may combine them. Multi-node execution, SQL read sharding
and performance were not validated. On a remote cluster, install the driver
and its native library on every worker and use a Redis endpoint reachable by
all workers; a client-local library path or localhost URI will not suffice.
Rerun this guide¶
From the repository root, with the guide's dependencies and native driver
installed, set REDIS_URI to the dedicated local Software fixture and run:
python python/scripts/check_integration_guide.py ray
This executes the guide's code blocks or its linked runnable example. For Superset it exercises the documented backend workflow in an isolated application; it does not automate the browser UI. See the per-guide results.
Write results back to Redis¶
The runnable example writes the tool's computed output through the shared
write_result Arrow sink and checks exact rows and types on a fresh connection.
This is custom application code and fully materializes the output; it does not
add a native connector, distributed sink, model/artifact store, or transactions.
The helper creates a unique output table and drops it after verification. In a
production workflow, choose an owned destination and explicit retry policy;
failed autocommit writes can partially apply and blind append retries duplicate rows.