ADBC partitioned reads

This page describes the default partition_result_mode=live path. Opt-in retained results add bounded global SQL, additional source layouts and immutable replay through the same public APIs. The coverage comparison records remaining gaps and the source-consistency design boundaries.

ExecutePartitions / ReadPartition support deferred reads of a single permanent, driver-managed Redis table. See issue #163. This API supplies the split descriptions for an external scheduler such as Ray or Dask; it does not create workers or provide an installed Ray connector. The published 0.0.19 macOS ARM64 wheel passed native live/retained partition checks over verified TLS on Redis Software 8.6.2 with 4x QPF. See the dated validation record for exact scope. Other wheel/platform combinations require their own validation. Rebuild from source to use changes made after that release, including bounded cursor cleanup.

Current managed snapshot contract

Rebuild from PR #284 to use the current storage format. Managed live descriptors are version 3: the entire set retains the producer's committed generation. Workers see the original schema and values despite later DML, ALTER, TRUNCATE, DROP/recreation, compaction or producer close. Expired or explicitly released sets fail on new reads rather than silently acquiring the latest generation. Each active reader holds a separate child pin until completion/release.

The statement option adbc.redis.partition_result_ttl applies to live and retained production: default 300 seconds, range 1..86400. For live results it bounds when new workers may acquire the snapshot; an admitted reader retains its own lease. Get statement option adbc.redis.partition_snapshot_id after successful live production. DROP PARTITION SNAPSHOT '<id>' releases admission early, without invalidating active readers. Statement/connection close and freeing the C partition byte array do not revoke descriptor copies. Retained mode remains the option for independently stored durable replay. Legacy version-1 live descriptors are rejected by managed snapshot catalogs; regenerate them.

Private managed copies created by HASH adoption or foreign-index discovery also support live partitions. A regular worker connection reads the captured copy without enabling foreign-index discovery or importing unrelated sources. Ordinary SQL discovery and explicit refresh retain their existing options and read-only foreign-schema behavior. Old descriptors retain their original copied rows after refresh or retirement; application HASHes and index definitions remain external.

This contract supersedes the historical unpinned consistency/lifecycle details below, which describe earlier releases. Query-shape and permission restrictions remain unless explicitly described by the current snapshot contract.

Supported queries

Use column/star projections, arithmetic, casts, CASE, and these deterministic scalar functions: ABS, MOD (also %), LOWER, UPPER, LENGTH, COALESCE, NULLIF, GREATEST, LEAST. Predicates use the same row-local expression allowlist. For example:

SELECT id + ?, CASE WHEN label IS NULL THEN 'missing' ELSE UPPER(label) END AS label
FROM public.events WHERE id >= ?

One bound Arrow parameter row is supported, including BindStream. Empty or multiple parameter sets fail. The binding is consumed and released during planning; descriptors own copied typed values, not pointers into the binding. Types: NULL, bool, signed/unsigned integers fitting int64, floats, decimal (precision <=76), text, binary, date, time, timestamp (naive or UTC), and month/day/nanosecond interval, as mapped by the ordinary driver's Arrow type conversion. Dictionary values map to their underlying type. Arrays, records, and unsupported Arrow types fail explicitly. Maximum 1024 fields and 1024 input batches (including empty batches); maximum 256 KiB JSON-encoded parameter section and 1 MiB total descriptor. Bound timestamps with a non-UTC timezone name are normalized to UTC as in ordinary execution. NULL keeps its logical type; binary and string bytes use base64, including embedded NUL/invalid UTF-8. Numeric/temporal payloads use exact typed storage representations, never SQL interpolation. Parameter stream failures produce IO errors without including values.

Scripts, subqueries, views, CTEs, joins, aggregates, grouping, DISTINCT, windows, set operations, ORDER BY, LIMIT and OFFSET remain unsupported. Volatile, user-dependent and other session-dependent functions remain rejected. Captured time zone is the only supported session setting used by expressions. Temporary tables, live external HASH/index layouts and legacy tables without persistent managed row IDs are unsupported. Managed application/foreign copies use the native partition path described above. Flex remains unsupported. See the retained runtime contract for global SQL, additional layouts and stable replay in the opt-in mode.

Broad reads use Search cursors over disjoint numeric subranges. Each subrange spans at most the guaranteed aggregate cap's number of allocated IDs, so it cannot match more rows than that cap. Empty/sparse ranges and residual filters are handled without truncation, page replay, or whole-partition HASH scans. At most one cursor is open per reader. Ordinary ExecuteQuery and partition reads share the cursor/Arrow lifecycle and internal range source. Eligible ordinary reads use that source when a lowered cap requires it, while preserving ordinary ordering and applying LIMIT/OFFSET once across all ranges. Partition output remains unordered. Subranges are opened lazily and limited to 2^20 per descriptor; oversized sparse work is rejected explicitly.

adbc.redis.partition_scan_cap is an integer/string statement option in 0..2147483648 (default 0). On Search-backed reads, zero requires fresh discovery through CONFIG at read start and at each subrange, independently of the ordinary query capability cache. Hidden/denied/nonpositive settings fail explicitly. Named ACL users do not probe administrative commands; set a positive guaranteed lower bound when planning for such readers. This is a deployment contract, not a request to raise a server setting: administrators must ensure the effective cap on every serving node stays at least that bound during reads. With automatic discovery, keep the observed bound valid for each open cursor. Lowering it during an active cursor cannot be made atomic with Redis CONFIG; that operation is unsupported. Fresh reads/ranges discover subsequent changes; abandoned readers open no new ranges. Planning/releasing descriptors does not change server configuration or table data.

Planning and worker execution

The statement option adbc.redis.partition_count accepts an integer or string from 1 to 1024 (default 8). It limits the number of ranges, rather than choosing concurrency or guaranteeing equally sized partitions. Ranges divide allocated internal row IDs, not row counts; deletes and sequence gaps can leave empty or uneven partitions. The row sequence must be below 2^53, where Search numeric range boundaries are exact. A never-populated table returns one empty partition with its Arrow schema. A previously populated empty table may return more.

The string statement option adbc.redis.partition_strategy defaults to rowid (equal-width allocated-ID ranges). Opt-in balanced chooses live-row count quantiles within the inferred candidate envelope, without estimating residual selectivity. Managed planning counts visible versions under the producer's pin, using bounded per-segment masks to exclude replacements and tombstones. It performs at most 256 count queries across all segments, with a two-second internal deadline after readiness/cap discovery; exhaustion discards partial cuts and retains equal-width ranges. Sparse/empty tables with fewer live rows than requested partitions also retain the default splits. A candidate that would exceed the current cap's per-descriptor subrange budget falls back. Normal Redis metadata/transport timeouts still apply outside the probe deadline.

Counts estimate workload without estimating residual selectivity. Every captured allocated ID remains covered, including empty gaps. Managed workers retain the same committed generation through later writes and DDL; balancing preserves that membership. Balanced planning requires Search readiness and FT.SEARCH; failures remain explicit. It needs a discoverable cap or the configured positive contract to protect range width; if automatic cap information is unavailable it retains the default splits. If exact visibility masks exceed their bounds, planning also retains equal-width ranges. Independent workers need no new descriptor fields or scheduler. More planning work can lose on small/already-balanced queries; use the default unless measurements justify it.

The planner returns the Arrow schema, opaque byte descriptors and an unknown row count (-1). Send the descriptors to workers which open independently authenticated connections to the same logical database. The planning statement and connection can close before reads start. Different cluster seed/proxy addresses are allowed; the table's unique physical identity and schema are checked on the reading connection. Cloned databases with identical metadata are indistinguishable and must be routed by the caller.

Cluster API workers also need ASKING permission to follow temporary redirects during slot migration. An ACL that permits steady-state reads can still fail during resharding without it. The owned three-primary TLS check exercises ASK and MOVED using existing native connections and pre-planned descriptors. Search operations can report temporary topology availability errors or omit a migrated HASH while their local loads converge, even after slot ownership is final. Cached sortable IDs and document counts alone do not establish readiness. The fixture waits for complete HASH loads on every primary before its exact Search-result assertion; it does not add transparent query or stream retries. Committed retained results and direct-HASH partitions do not depend on this Search convergence.

Each worker reads Arrow batches using the reading database's adbc.redis.stream_batch_rows (default 65536). Partition reads always stream, even if ordinary query streaming is disabled. Client buffering is bounded by the Arrow batch setting plus Search cursor or HASH pipeline pages; it does not grow with the entire result. Output order is unspecified. Do not concatenate partitions and assume a global sort.

Standard incremental execution

adbc.statement.exec.incremental accepts canonical strings true/false, default false. It follows the canonical ADBC header: all deferred ranges are available together, so the first ExecutePartitions call returns them all. Subsequent calls return an empty partition set, the same schema and unknown row count (-1). Empty means terminal completion, never “temporarily unavailable.” An empty table still emits an empty-read descriptor first, then terminal completion. No worker/background task is started or polled; metadata/network planning and bound stream consumption still take their normal time. This option does not make arbitrary SQL evaluation asynchronous.

A discovery failure is terminal and repeated calls return that error until reset. SetSqlQuery, Bind/BindStream, setting the incremental option (including the same value), partition count/cap/strategy or ingest target resets discovery. Closing the statement clears its schema/error state; closing its connection makes further calls fail. ExecuteQuery/ExecuteSchema are unaffected. In-flight execution uses the existing context/C API cancellation path; once descriptors have been returned, there is no pending discovery work to cancel. Statement cancel then reports no active query unless it owns another streamed result. Reset before trying again after cancellation. Returned descriptors survive planner abandonment/close and are not revoked by resets or option changes. Readers belong to worker connections.

The standard double progress/max-progress options explicitly return NOT_IMPLEMENTED: metadata discovery has no useful intermediate execution progress. Consumer rows read are not query execution progress. No universal partition or incremental capability flag is invented in GetInfo.

Finite Arrow batch streaming fetches cursor pages as a reader advances. Partitioned reads distribute deferred read descriptions. Incremental execution controls repeated discovery of those descriptions. None of these is a continuous subscription, change feed, CDC stream or shared transactional snapshot.

Selective internal row IDs

Planning extracts exact integer __rowid equality, IN, BETWEEN and inequalities (in either operand order), including integer typed bindings and unary signs. It intersects bounds with the captured allocated-ID domain. AND can retain a safe conjunct; OR requires every branch to have a safe constraint. Float, decimal, text, NULL, casts, functions, NOT and mixed OR expressions retain the conservative range path. Business-column IDs never imply internal row IDs. Full residual predicates are rechecked, including SQL NULL semantics.

At most 1024 proven candidate IDs are enumerated. Their groups emit at most partition_count descriptors; an empty set emits one empty descriptor with its schema. Larger interval unions use a conservative envelope, including gaps, with the usual rowid/balanced splits. Extraction is bounded to 4096 nodes, 128 nesting levels and 1024 intervals; budget exhaustion widens candidates. The table's below-2^53 restriction and descriptor/discovery format still apply.

A worker whose descriptor proves at most 1024 candidate IDs reads authoritative HASHes in pipelines of at most 1000 keys. It shares Arrow batching, cancellation, connection ownership and exact decoding with the streamed reader; it checks physical identity before and after fetching. Missing/deleted keys produce no rows. Updates remain live; IDs above the captured bounds remain excluded.

These Search-free reads need connection/metadata and HASH read permissions, but do not issue CONFIG, FT.INFO, FT.SEARCH, FT.AGGREGATE or FT.CURSOR for their input. No scan-cap guarantee or index-readiness permission is needed on this path. Invalid descriptor cap values still fail validation. Other predicates retain all Search-backed cap/readiness checks below. Direct HASH lookup does not grant extra permissions or introduce a snapshot. Ordinary ExecuteQuery also retains normal numeric predicate evaluation for non-integer row-ID comparisons instead of rounding them into an equality key.

Search index readiness

Redis OSS 8.6.2/Search 8.6.0 emitted duplicate cursor rows while an index was still building. Bounded Search readers now check FT.INFO before and after opening each cursor and fetching each page. OSS Cluster readers check every primary; Software/proxy endpoints must expose the readiness fields through FT.INFO. A building/incomplete index fails with INVALID_STATE: wait for indexing to finish and retry the descriptor. Unsupported/missing readiness fields fail with NOT_IMPLEMENTED. Failed pages are discarded and their cursors are closed; the driver does not deduplicate or replay emitted pages.

Search-backed partition workers additionally need +ft.info and permission to inspect their index. A guaranteed positive scan cap does not bypass this check; CONFIG remains optional with that contract. A denied readiness check fails with UNAUTHORIZED. Eligible ordinary queries retain their complete HASH fallback when readiness or its permission cannot be established before selecting bounded scans. If readiness changes after a bounded read starts, that read fails explicitly.

A SQL COUNT alone does not establish Search readiness. Deployment owners can also verify indexing=0 and expected document counts before distributing work. The checks mitigate the observed building-index condition, not every possible backend defect or concurrent external index replacement. They do not create a snapshot or guarantee stable retries. See issue #175 and the original evidence.

Indexed ALTER operations wait for Search backfill before publishing their new metadata. Background dropped-field cleanup remains asynchronous. Search-backed partition reads reject a table with pending cleanup using INVALID_STATE; retry the descriptor after an authorized cleanup worker completes the work. Proven direct HASH reads do not depend on that index state and remain available. Balanced planning retains equal-width cuts during cleanup. Ordinary scans and aggregates use complete HASH inputs during that interval, with their normal global operators. No broader reader ACL or shared snapshot is introduced.

Consistency, lifecycle and permissions

Descriptors describe future reads, not retained results. There is no expiry timer, coordinator lease, temporary result table or server-side plan state to clean up. Releasing the C partitions structure releases its byte storage only; copies already sent to workers remain usable. Closing or reusing the planning statement does not revoke them. Table drop/recreation, truncate, rekey or physical schema changes invalidate descriptors; plan again after those changes.

Planning captures the allocated row-ID high watermark. Inserts allocated after that point are excluded. An insert already allocated but not finished may appear later. Updates and deletes are observed when each worker reads, and re-reading a descriptor may return different values. There is no shared transactional snapshot, stable retry result, or exactly-once guarantee. On an unchanging dataset, ranges cover the query's rows without overlaps or omissions.

Release/close each Arrow reader promptly: this deletes its Search cursor. Connection close, cancellation of the Go read context, and the C/Python connection cancellation API stop its readers. Python's cursor statement cancel does not own a partition read: use conn.adbc_cancel(). Redis also expires idle cursors. A failure can occur during batch iteration, so consume and check the reader before treating a partition as successful.

Descriptors contain SQL text/literals, typed bound values, schema and table names, time zone and physical identity hashes; treat them as potentially sensitive query data. They contain no URI, password or live pointer. They are validated and reparsed as untrusted input, are not signed, and are not authorization capabilities. Readers use their own Redis ACL permissions; changing descriptor bytes cannot grant permissions. Malformed/version/size/range errors fail with INVALID_ARGUMENT; stale/wrong-identity targets fail with INVALID_STATE. Missing objects and denied commands retain ordinary driver errors. Validation does not echo descriptor contents.

The planner captures the resolved schema and time zone, so workers do not depend on the planner's session or inherit its credentials. Other session dependent expressions are excluded by the query subset above.

Python example

Install adbc-driver-redis[dbapi] and PyArrow on each worker, and configure a supported Redis deployment with Search. With an existing public.events table:

from adbc_driver_redis import dbapi

uri = "redis://localhost:6379/0"  # worker-accessible endpoint
with dbapi.connect(uri) as conn:
    with conn.cursor() as cur:
        cur.adbc_statement.set_options(**{"adbc.redis.partition_count": 4})
        partitions, schema = cur.adbc_execute_partitions(
            "SELECT id, label FROM public.events WHERE id >= 100"
        )

# Run independently on each worker for its assigned descriptor.
def read_partition(descriptor):
    with dbapi.connect(uri) as conn:
        with conn.cursor() as cur:
            cur.adbc_read_partition(descriptor)
            with cur.fetch_record_batch() as batches:
                for batch in batches:
                    consume(batch)  # your processing code

For a runnable local process-pool example, see adbc_partitions.py. It counts rows incrementally and compares Arrow schemas on each worker; it creates no tables.

Follow-up scope

The capabilities beyond this initial implementation are tracked separately:

TLS and independent-worker validation

scripts/start_partition_tls.sh creates a disposable, TLS-only Redis 8.6.2 fixture with a fresh local CA and real aggregate cap 137. It accepts a new fixture directory, container name and localhost port. Set SSL_CERT_FILE to its ca.crt and REDIS_TLS_ADMIN_URI to the contents of admin.uri, then run scripts/check_partition_tls.py --output <fresh JSON>. Use REDIS_ADBC_LIBRARY for a built library, or install the native wheel and omit that override; --require-wheel verifies the installed-package path. The Python environment needs PyArrow, adbc-driver-manager and redis. The checker passes that CA using the database option adbc.redis.tls_ca_file, including to each fresh worker, rather than relying on inherited CA environment variables.

For application connections, set adbc.redis.tls_ca_file to a local PEM trust bundle when using a private CA. Nonempty replaces default system roots for new connections; empty uses system trust. It requires verified rediss:// and cannot be combined with skip_verify=true. Each worker supplies its own path; the file is not included in descriptors. Existing connections do not reload rotated files. Publicly trusted endpoints need no custom CA option.

The checker creates its own schema and distinct planner/read-only worker principals, closes the planner before starting fresh worker processes, and validates complete values and schemas. It checks untrusted certificate and hostname rejection, denied CONFIG/writes/HASH/readiness commands, cancellation, early release, live updates/deletes, later-insert exclusion and stale targets. Generated credentials, URI files and certificate keys must stay local; only the credential-free result JSON is a CI artifact. The checker removes its generated schema/users. Remove the fixture container and directory after use; it does not install a CA into the system trust store or change an existing Redis service.

PR tests require the TLS check on both Linux OSS-single architectures, reusing the built library. Release validation uses installed Linux and macOS wheels. The macOS job starts an owned native TLS-only Redis and checks default/minimum ADBC clients with 15 TLS checks, including native CA and hostname rejection. Local macOS packaged validation passed on 2026-10-21; hosted results are separate evidence. Cloud/proxy TLS and remaining compatibility gaps stay tracked in #170.

Redis Software proxy TLS uses scripts/software_partition_tls.py after the existing disposable Software setup. It verifies ownership of the Compose container and localhost ports, installs private API/proxy certificates, requires TLS/password authentication and verifies the real cap 137 and active primary count. Initial local management bootstrap uses the fixture's self-signed API; subsequent management and all native partition connections verify the private CA. Use --software-fixture DIRECTORY with the same checker to provision database-only roles/users through the management API. Software's unauthorized CONFIG GET returns an empty response; the checker verifies that policy and still requires CONFIG SET, writes, denied readiness and HASH reads to fail.

Both Linux PR architectures require Software single/three-shard checks after non-TLS suites; Linux release jobs also validate the installed wheel through a single-shard TLS proxy. Local macOS single/three-shard and Linux arm64 packaged runs passed all 16 Software checks. Each three-shard deployment uses one Software node, so this establishes proxy/shard routing coverage, not multi-node failover. See verification. Cloud TLS and remaining performance evidence are still open in #170.

Explicit layouts and byte lengths (#302/#304)

Non-dictionary adbc.redis.result_layouts are carried in revision 3 live partition descriptors. Revision 1 canonical descriptors remain readable. Schema compatibility is checked independently on every reader; conversion errors remain explicit. OCTET_LENGTH/BIT_LENGTH over VARCHAR/VARBINARY are admitted row-local functions. See Arrow result layouts and coverage.

Live revision-3 descriptors may carry a snapshot capability, supported result layouts, or both. Managed catalogs require a valid shared snapshot at read time; layout metadata alone never enables a live read of a managed catalog. Layout wrappers retain the underlying reader and its snapshot/transport ownership.