Redis integration guide for Dask

Redis must run separately with the Query Engine (Search). See installation and driver discovery for Python/native setup, supported wheel platforms, TLS and source builds. Tables must be created, ingested or adopted through the driver; arbitrary Redis keys do not automatically become SQL tables. Writes autocommit without rollback.

This guide describes current repository capabilities. A newly added helper or extra may require a build containing its implementation. If a released wheel lacks it, build the native driver from this checkout and install the relevant extra from ./python; set REDIS_ADBC_LIBRARY to the built library. Do not infer that an older published wheel contains every helper documented here.

Dask

Install adbc-driver-redis[dask] in the client and every worker environment. This extra includes Dask DataFrame, Distributed, pandas and PyArrow. Platform wheels bundle the native driver; source installs require REDIS_ADBC_LIBRARY set to a local library path on every worker before starting it. Workers need network access to the Redis endpoint; localhost refers to each worker's machine. Keep package versions consistent across client, scheduler and workers. Use rediss:// and db_kwargs/conn_kwargs just as with dbapi.connect.

adbc_driver_redis.dask.read_query uses dd.from_map to run one bound SELECT for each supplied parameter tuple:

import pyarrow as pa
from adbc_driver_redis.dask import read_query

# Existing table, with non-NULL ids in [0, 3000).
frame = read_query(
    "redis://localhost:6379/0",
    "SELECT id, amount FROM orders WHERE id >= ? AND id < ?",
    [(0, 1000), (1000, 2000), (2000, 3000)],
    schema=pa.schema([("id", pa.int64()), ("amount", pa.float64())]),
    db_kwargs={"adbc.redis.query_workers": "1"},
)
result = frame.groupby("id").amount.sum().compute().reset_index()
print(result)
from integration_guide_data import write_result
write_result(pa.Table.from_pandas(result, preserve_index=False))

The driver supports these bound half-open range predicates. This helper does not rely on ADBC partition descriptors, SQLAlchemy, OFFSET pagination, automatic MIN/MAX probes, or fetching sample rows. The supplied Arrow schema describes the SELECT result exactly (column names, order, types and nullability; schema metadata is ignored). A mismatch fails the task, rather than coercing data. The result uses pandas Arrow dtypes, so nullable integers and empty partitions retain their types. Pass [()] for one unparameterized query. Parameters are bound through DB-API; identifiers belong in your SQL, not parameter values.

Choose disjoint predicates that cover your intended dataset. Numeric ranges exclude NULLs; add a separate NULL query partition when needed, as shown in the runnable example. Rows outside your bounds are excluded. Dask index divisions are unknown, indexes restart per partition, and row order is not guaranteed. There is no automatic inference or validation of partition coverage. Redis provides no shared snapshot across tasks: concurrent updates, key changes, and retried or repeated computations can yield different, missing, or duplicate rows. Read an immutable source or pause writers when exact coverage is required.

Each task opens and closes a connection and cursor on its worker, including on execution/fetch failures. Only query configuration, schema and parameters are sent to workers. Connection credentials are part of the task configuration; use a trusted Dask cluster and protect its scheduler and diagnostics.

A whole query partition is fetched into an Arrow table and converted to pandas in worker memory. Choose bounds using your data distribution, allow space for Arrow and pandas plus concurrent tasks, and split skewed ranges further. Streaming batch options do not cap this materialization. Limit active tasks and adbc.redis.query_workers to avoid multiplying driver concurrency across worker processes. This adapter does not push later Dask filters/projections into SQL; select only the required columns and predicates in the query. For large results, write partitions to Parquet or compute reductions instead of gathering the full DataFrame into the client.

Run the self-contained example against a disposable Redis with Search:

python -m pip install -e './python[dask]'
export REDIS_ADBC_LIBRARY="$PWD/go/build/libadbc_driver_redis.dylib" # .so on Linux
REDIS_URI=redis://localhost:6379/0 python python/examples/dask_read.py

The example creates a unique table, starts two Distributed worker processes, checks five partitions (including an empty range and a NULL partition), compares all rows and dtypes with the source, and drops its table. Its __main__ guard is required when spawning worker processes. To use an existing cluster, replace LocalCluster(...) with Client("tcp://scheduler:8786").

Dask's built-in read_sql_table/read_sql_query APIs require SQLAlchemy URIs, not direct ADBC connections. This package also provides a SQLAlchemy dialect (redis+adbc://); that is a separate route with Dask's SQLAlchemy query and partitioning requirements. The helper here directly uses ADBC with explicit partitions and schema.

Write computed results

The example uses the shared example setup sink to write a materialized computed pandas result through Arrow and verify it on a fresh connection. read_query remains a partitioned reader, without a distributed write API. Large outputs need a separately designed partitioned sink and retry policy. The linked runnable example also checks this output path. #273 tracks a packaged worker-local sink.

Rerun this guide

From the repository root, with the guide's dependencies and native driver installed, set REDIS_URI to the dedicated local Software fixture and run:

python python/scripts/check_integration_guide.py dask

This executes the guide's code blocks or its linked runnable example. For Superset it exercises the documented backend workflow in an isolated application; it does not automate the browser UI. See the per-guide results.

All integration guides