Redis integration guide for Prefect

Redis must run separately with the Query Engine (Search). See installation and driver discovery for Python/native setup, supported wheel platforms, TLS and source builds. Tables must be created, ingested or adopted through the driver; arbitrary Redis keys do not automatically become SQL tables. Writes autocommit without rollback.

This guide describes current repository capabilities. A newly added helper or extra may require a build containing its implementation. If a released wheel lacks it, build the native driver from this checkout and install the relevant extra from ./python; set REDIS_ADBC_LIBRARY to the built library. Do not infer that an older published wheel contains every helper documented here.

Prefect custom Python tasks

Install pip install 'adbc-driver-redis[prefect]'. Create connections inside each task so concurrent tasks and retries each own their connection and cursor:

import os
from adbc_driver_redis import dbapi
from integration_guide_data import write_result
from prefect import flow, task
from prefect.cache_policies import NO_CACHE

@task(cache_policy=NO_CACHE, persist_result=False)
def query(uri):
    with dbapi.connect(uri) as conn, conn.cursor() as cur:
        cur.execute("SELECT 1 AS one")
        output = cur.fetch_arrow_table()
    return write_result(output)

@flow(persist_result=False)
def pipeline(uri):
    return query.submit(uri).result()

print(pipeline(os.environ["REDIS_URI"]).to_pylist())

The Prefect tests run real local flows with Arrow ingestion/fetch, task dependencies, concurrent queries, a retry after a committed write, and SQL error propagation. Writes autocommit even if a task fails. Make retried writes idempotent (the test replaces its output table); blindly retrying append/INSERT can duplicate rows. Caching and result persistence are disabled here so each invocation actually accesses Redis. Remote workers need the driver wheel and Redis connectivity installed separately.

Run against a disposable Redis with Search:

python -m pip install -e './python[prefect]' pytest
REDIS_URI=redis://localhost:6379/0 \
REDIS_ADBC_LIBRARY="$PWD/go/build/libadbc_driver_redis.dylib" \
PREFECT_SERVER_ANALYTICS_ENABLED=false \
python -m pytest -q python/tests/test_prefect.py

Build the library first as above; use .so on Linux. This verifies custom Python tasks, not a packaged Prefect ADBC connector, prefect-redis, Flight SQL, or a remote Prefect deployment.

Rerun this guide

From the repository root, with the guide's dependencies and native driver installed, set REDIS_URI to the dedicated local Software fixture and run:

python python/scripts/check_integration_guide.py prefect

This executes the guide's code blocks or its linked runnable example. For Superset it exercises the documented backend workflow in an isolated application; it does not automate the browser UI. See the per-guide results.

All integration guides