Arrow IPC and finite streaming

The driver serializes Arrow IPC inside the native driver, through standard ADBC statement options. No client-side serializer, new C entry point, Redis wire format, or custom Arrow C Stream layout is required. This implements the IPC portion of #78. Continuous subscriptions and CDC are tracked separately in #196.

Capability matrix

Capability Support Boundary and limits
Arrow C Stream query results Yes Standard ADBC ArrowArrayStream; an in-process interface, not an IPC byte format
Arrow C Stream ingestion Yes Standard ADBC BindStream with bulk-ingest options
Arrow IPC streaming format Yes Native local-file export/import through statement options; schema followed by sequential batches
Arrow IPC file format Yes Native local-file export/import; import requires a seekable regular file; footer metadata grows with batch count
Finite incremental query output Eligible indexed single-table SELECTs and UNION ALL trees Cursor pages fetched as batches are consumed; eligibility and cleanup apply
Incremental aggregate input Yes for eligible table scans Page-wise reduction from #72; output still waits for reduction and state grows with groups/distinct/ordered-set inputs
Continuous subscriptions / CDC Unsupported Separate scope in #196; no checkpoint, replay, or snapshot-to-live contract

Statement contract

String option Default Meaning
adbc.redis.ipc.format stream Exactly stream or file; selects encoding for input or output
adbc.redis.ipc.output Empty (disabled) Absolute local destination path that does not already exist
adbc.redis.ipc.input Empty (disabled) Absolute local regular-file source path for bulk ingestion

Set options using AdbcStatementSetOption. Execute using AdbcStatementExecuteQuery(statement, NULL, &rows, &error) in C, or execute_update() on a Python low-level ADBC statement. A non-null output stream or partition execution with IPC paths configured is rejected: use a separate statement for ordinary Arrow results. The paths and format persist until changed; set a path to the empty string to disable it.

Export requires exactly one SELECT, including SELECTs with blocking plans, and returns the fully written row count. Scripts, DML and ingest-target export are rejected before execution. Bound SELECT parameters work through the normal ADBC binding interface. Existing destinations, including symlinks, are never replaced. Output is created with owner-only permissions (subject to the process umask). A failed export removes its partial file. Consumers must wait for success before opening/publishing it: there is no atomic publication, fsync, or crash cleanup guarantee. Disk errors and late query errors fail the operation.

Import requires adbc.ingest.target_table and the normal ingest mode/schema options, with no independently bound data. Input/output cannot both be enabled. The native IPC reader feeds the existing ingestion path one batch at a time. An empty input preserves its schema and can create an empty table. SQL type support and schema validation remain those of ordinary Arrow ingestion. Invalid encoding/schema/batches fail normally. Ingestion autocommits: a late decode, Redis, or cancellation error can leave the table and earlier rows written.

Paths refer to the filesystem of the process loading the driver, with that process's permissions. Network URLs, pipe transport and automatic format detection are not provided. Treat IPC sources as trusted local files; there is no new decompression or allocation quota for imported Arrow data.

Executable example

Install the Python package (the driver performs serialization; this example does not require PyArrow). --output must name a new file. Optional --ingest-table creates a new table from that output through native ingestion:

python python/examples/arrow_ipc.py \
  --uri 'redis://localhost:6379/0' \
  --query 'SELECT id, name FROM example' \
  --output /absolute/new-result.arrow --format stream

The core calls are also usable from any ADBC C binding:

import adbc_driver_manager as adbc
from adbc_driver_redis import connect

with connect("redis://localhost:6379/0") as db, adbc.AdbcConnection(db) as conn:
    with adbc.AdbcStatement(conn) as stmt:
        stmt.set_sql_query("SELECT id, name FROM example")
        stmt.set_options(**{
            "adbc.redis.ipc.output": "/absolute/new-result.arrow",
            "adbc.redis.ipc.format": "stream",
            "adbc.redis.stream_batch_rows": "4096",
        })
        rows = stmt.execute_update()
    with adbc.AdbcStatement(conn) as stmt:
        stmt.set_options(**{
            "adbc.ingest.target_table": "example_copy",
            "adbc.ingest.mode": "adbc.ingest.mode.create",
            "adbc.redis.ipc.input": "/absolute/new-result.arrow",
            "adbc.redis.ipc.format": "stream",
        })
        ingested = stmt.execute_update()
        assert ingested == rows

Native IPC performance

The matched native-versus-PyArrow benchmark covers both IPC formats, export/import, narrow/wide scans, UNION ALL and blocking aggregation. At the tested local scale, native elapsed medians differed by -3.3% to +3.8%, with all trial ranges overlapping; no speed advantage was established. See the report for CPU, RSS, encoded size and scope limits.

Incremental execution and memory

IPC export consumes each Arrow batch before requesting the next. The serializer adds batch/dictionary buffers; it does not make a blocking SQL plan incremental. IPC file writers also keep footer metadata proportional to batch count. Import similarly holds the current decoded batch rather than collecting a complete Arrow table; one unusually large batch can still require substantial memory. Cancellation is checked between batches, and the underlying query/ingest checks continue to apply. A local read/write already in progress must return before it can observe cancellation. Readers, writers and files are released on success and error, including early query failure and late batch failure.

Query family Existing input/output behavior Further incremental output assessment
Indexed row-local filter/projection, eligible LIMIT/OFFSET and exact index order Incremental input and output Already streams; direct Arrow projections and byte budgets remain applicable
Aggregates/GROUP BY Eligible scans reduce input page by page; output after reduction General groups cannot be finalized before complete input; group/distinct state remains cardinality-dependent
Driver ORDER BY May require complete input; eligible top-K uses a bounded heap Top-K still needs all candidate input before output; bounds projected winners, not every upstream operator
DISTINCT and distinct set operations Retain deduplication/comparison state Emission-order and SQL duplicate/type rules need a separate executor design; no new path enabled here
Eligible UNION ALL trees without ORDER BY Incremental output over sequential managed indexed-table child scans Pairwise coercion and per-node LIMIT/OFFSET; only one active child; unsupported leaves/orderings retain materialization
Joins, windows, CTEs/derived tables Existing materialization paths Streaming requires operator-specific state and dependency analysis; no blanket eligibility expansion

Native IPC provides an incremental sink/source. #197 adds the eligible UNION ALL path described in streaming documentation, with fresh measurements. Neither feature adds spill or a query-wide memory cap. Existing output byte budgets apply only to eligible streamed batches. They do not bound process RSS, consumer-held buffers, server sort/group state, or all blocking operators.

Historical evidence, not measurements of this change:

  • Aggregate pages: at 100K rows, page-wise Q9 reduction measured 141.1 MiB peak RSS versus 348.2 MiB before; high-cardinality Q33 still used 622.7 MiB. This demonstrates why incremental input does not imply constant-memory aggregation.
  • Stream byte budgets records batch accounting, cursor-page overshoot, peak memory, cancellation and release coverage. These budgets do not cap blocking operators.
  • Incremental read comparison records independently validated complete IPC transfers with different row ceilings and explicit SQL ranges. It predates these native IPC options.

Other candidates are explicitly deferred in #197: general aggregates require complete input before finalization; DISTINCT set operations require cardinality-dependent deduplication state; joins/windows and CTE/derived-table dependency graphs need separate operator designs. Top-K can bound retained winners while still consuming all candidates. No new spill engine or limit for those operators is added. Further eligibility expansion requires its own correctness, lifecycle and memory evidence. CDC remains exclusively in #196.

Physical result layouts

The statement option adbc.redis.result_layouts applies to IPC output and schema discovery. Per-batch dictionaries require IPC stream format; file output rejects that layout. Other supported layouts work with either format. Result layouts do not change ingest normalization or the retained-partition IPC validation contract.