Redis integration guide for Dagster

Redis must run separately with the Query Engine (Search). See installation and driver discovery for Python/native setup, supported wheel platforms, TLS and source builds. Tables must be created, ingested or adopted through the driver; arbitrary Redis keys do not automatically become SQL tables. Writes autocommit without rollback.

This guide describes current repository capabilities. A newly added helper or extra may require a build containing its implementation. If a released wheel lacks it, build the native driver from this checkout and install the relevant extra from ./python; set REDIS_ADBC_LIBRARY to the built library. Do not infer that an older published wheel contains every helper documented here.

Dagster

The community dagster-adbc package provides ADBCResource for SQL and Arrow access inside assets. Install dagster-adbc alongside the Redis driver wheel. With the Redis driver manifest installed as described above:

from dagster import Definitions, asset
from dagster_adbc import ADBCResource

@asset
def order_count(redis: ADBCResource) -> int:
    with redis.get_connection() as conn, conn.cursor() as cur:
        cur.execute("SELECT count(*) FROM analytics.orders")
        return cur.fetchone()[0]

defs = Definitions(
    assets=[order_count],
    resources={"redis": ADBCResource(
        driver="redis",
        uri="redis://localhost:6379/0",
        autocommit=True,
    )},
)

Alternatively, set driver to the absolute native library path. Use autocommit=True: Redis does not implement transactions, and the community resource's default (False) produces a driver-manager warning. Native database options can be passed through db_kwargs, for example {"adbc.redis.default_schema": "analytics"}. The resource exposes a connection; assets explicitly execute queries or call cursor.adbc_ingest.

Install the wheel's dagster extra, plus pandas/polars when used. This repository ships RedisADBCIOManager; it is not part of the community dagster-adbc package.

import pyarrow as pa
from dagster import Definitions, asset
from adbc_driver_redis.dagster import RedisADBCIOManager

@asset(io_manager_key="redis", key_prefix=["analytics"])
def orders() -> pa.Table:
    return pa.table({"id": [1, 2], "amount": [10, 20]})

defs = Definitions(
    assets=[orders],
    resources={"redis": RedisADBCIOManager(uri="redis://localhost:6379/0")},
)

For a runnable smoke test from a source checkout, use a dedicated Redis database with Search and follow the development installation steps above. Run:

REDIS_URI=redis://localhost:6379/0 python python/examples/dagster_smoke.py
REDIS_URI=redis://localhost:6379/0 python -m pytest -q python/tests/test_dagster.py

The smoke example materializes orders, loads it from Redis into the dependent revenue asset, and checks the persisted result with SQL: three orders totaling 42. Each invocation uses a unique schema and drops it on exit. The integration suite also checks Arrow/pandas/polars round trips, column projection, partition retries, and failed-stream preservation; it creates and removes the fixed it_python_dagster schema, so run it serially.

Outputs may be pyarrow.Table, RecordBatchReader, pandas or polars DataFrames. Inputs default to Arrow tables; annotate them with pandas/polars DataFrame or RecordBatchReader to request that type. Input columns metadata selects columns. Reader outputs ingest in batches; reader inputs currently materialize the result before returning an in-memory reader.

Asset names preserve component boundaries and case. With no schema_name, the prefix path determines the schema (public without a prefix), and the final component determines the table. With schema_name, the entire asset path determines the table inside that schema (the schema setting is lowercased). A single lowercase ASCII identifier matching [a-z_][a-z0-9_]* stays readable, unless it begins with the reserved dagster__ marker. Other paths are encoded as dagster__ followed by each component's UTF-8 hex, separated by underscores. An explicit public prefix is also encoded to distinguish it from no prefix. For example, ["a_b", "c", "orders"] maps to dagster__615f62_63.orders, while ["a", "b_c", "orders"] maps to dagster__61_625f63.orders. Simple orders and sales.orders stay unchanged.

Migration: this replaces the earlier underscore join and lowercasing of asset keys. Multi-component prefixes, case-sensitive or encoded names, explicit public prefixes, and prefixed assets using schema_name may move to new targets. Inspect the manager's output table metadata, then deliberately migrate or rematerialize affected assets before downstream reads. The manager does not read, drop, or migrate ambiguous legacy tables automatically. Previously overwritten asset data must be recovered from its source or a backup.

Configure db_kwargs for native options and index_columns for ingestion indexing (* by default). Output metadata includes row count, table name and Arrow column schema.

Unpartitioned writes load a unique staging table before dropping the old table and renaming the stage. A failed ingest preserves the old version; a failure between DROP and RENAME can leave it absent. Readers can also observe this brief window. Static/dynamic string and daily/time-window partitions require asset metadata {"partition_expr": "column_name"}. The manager uses bound partition values or a half-open time range, deletes those rows, then appends. A retry replaces that partition. A failed append may leave it partially written; values returned by the asset must belong to the selected partition. Serialize writes to the same asset/partition. Multi-dimensional partitions are unsupported.

Rerun this guide

From the repository root, with the guide's dependencies and native driver installed, set REDIS_URI to the dedicated local Software fixture and run:

python python/scripts/check_integration_guide.py dagster

This executes the guide's code blocks or its linked runnable example. For Superset it exercises the documented backend workflow in an isolated application; it does not automate the browser UI. See the per-guide results.

All integration guides