Retained partition results

Retained mode extends ExecutePartitions with global SQL and immutable Arrow replay. It runs a bounded coordinator query once, stores its completed final output, and returns version-2 canonical or version-4 explicit-layout descriptors over contiguous chunk ordinals. ExecuteQuery remains the ordinary SQL API. Live row-range partitions remain the default and let workers read the producer’s shared committed generation without storing a separate final result. This implementation addresses the runtime work in #166, #167 and #169; the deployment/performance matrix in #170 is tracked separately.

Configure and use

Set database option adbc.redis.partition_result_namespace to a tenant-owned name of 1–64 ASCII letters/digits/underscores/hyphens. Set statement option adbc.redis.partition_result_mode=retained, then call ExecutePartitions with one SELECT of at most 32 KiB. Existing typed single-row parameter bindings, adbc.redis.partition_count, adbc.redis.stream_batch_rows and the optional adbc.redis.stream_batch_bytes apply. The byte setting uses the ordinary stream's logical accounting and allows one oversized row to make progress; zero disables it. A planner statement limits output chunks; a worker database independently limits returned Arrow slices. A worker still holds one bounded IPC chunk, so the byte setting is not a resident memory ceiling. Very small planner batches can exhaust the configured chunk count. Partition count is an upper bound on contiguous output chunk groups, not source workers. The ordinary executor runs with pushdown disabled and one query worker on captured source rows; the output may contain fewer partitions than requested. An empty output still has one schema-bearing empty chunk/descriptor.

Independent consumers configure the same namespace and database option adbc.redis.partition_reader_only=true, authenticate independently, then call ReadPartition. This bootstrap performs no source catalog recovery, namespace initialization or SQL/ingestion. Set adbc.redis.cluster=true for a restricted Cluster API credential so setup need not probe INFO. TLS verification and adbc.redis.tls_ca_file apply to source, publisher and worker connections. RESP3 is required; RESP2 and protocol=2 are unsupported.

For ordered output, concatenate descriptors in the order returned, and batches within each descriptor in read order. Parallel arrival order is not global SQL order. Each descriptor is independently retryable; retries reproduce its stored values and order. Unspecified sort ties and aggregate input ordering are not a cross-execution ordering guarantee. Supply a total ORDER BY, including an aggregate's own ORDER BY where needed, for deterministic fresh executions.

SQL and source coverage

Operator Retained coordinator behavior Resource/lifecycle boundary
Projection, filters, casts, CASE, typed bindings Ordinary SQL evaluator and Arrow schema Bounded types, scalar size and expressions
ORDER BY, LIMIT/OFFSET One global sort and final slice Sort keys/indices admitted before allocation
DISTINCT / DISTINCT ON One global reduction Cumulative expression and intermediate credits
Aggregates, GROUP BY, HAVING Reduce once, including NULLs and duplicates Captured input and work/value credits
Windows and QUALIFY Evaluate once over global input Cache admission; conservative allowance for aggregate frames
UNION/ALL, INTERSECT, EXCEPT Ordinary set/multiset semantics Both input branches share job limits
Joins Shared captured dependencies; ordinary join semantics Candidate arrays/maps admitted before allocation; work cap
CTEs, permanent views, subqueries One coordinator session and dependency cache Definition checks; shared budgets and deadline

The initial stable-function allowlist includes ordinary aggregates, basic math, string case/length/trim/concatenation, conditional functions, session reads, positional string slicing (SUBSTR, LEFT, RIGHT), ranking/value windows, arrays and current time. Expanding generators, table functions, recursive CTEs, lateral items, volatile/side-effect functions, regex/JSON generators and catalog/OID aliases are rejected explicitly. Trim cutsets must be bounded literals; ROUND/TRUNC precision must be a literal in [-76,76]. An unsupported plan fails before returning descriptors.

Source layout Coverage and ownership
Permanent managed table Exact authoritative rows from one coordinator snapshot; native int64 IDs and values retain precision
Adopted HASHes and application indexes Private managed copy with original source-key provenance; explicit refresh recaptures application changes
Foreign HASH Search index Private managed copy of captured membership/schema; explicit refresh; ordinary foreign discovery options still apply
Producer-owned temporary tables/views Capture under the coordinator snapshot; readers replay committed output after producer closure
Legacy storage and unowned temporary relations Unsupported; no migration

Capture reads relevant source rows/columns before evaluating predicates. It does not yet push projections/predicates into capture. Duplicate projected rows remain duplicates. For non-UTF-8 application keys project CAST(__key AS BYTEA); invalid UTF-8 typed TEXT output is rejected. Native input preserves exact physical ID order; application keys do not imply SQL order.

The coordinator pins a committed catalog generation. Concurrent SQL DML, schema changes, view replacement, rekey and source replacement do not change its captured rows or definitions. An application-owned HASH/index is represented by its managed copy; external edits appear only after explicit refresh. Import and retained capture do not transfer ownership of application HASHes or index definitions. Publisher Open requires the current snapshot storage format, with no migration; reader-only retained replay bypasses ordinary catalog access.

The frozen object is the completed query output evaluated from that coordinator snapshot. Replay reads immutable Arrow chunks and never reruns SQL or consults source tables after publication. Cancellation before publication must expose no descriptors and release private upload/quota reservations. Stable replay does not give exactly-once effects in a downstream sink. Retained mode remains subject to its SQL/resource limits and control-persistence requirements below.

Current-protocol Cluster API, topology changes, failover and persistence recovery still require the deployment checks in the snapshot parity plan. Historical source-capture topology checks are not proof of that full matrix. See partition coverage for other partition limitations.

Limits and persistence

Limits are cumulative logical allocation/work credits, not process RSS or Redis allocator/replication/index-memory guarantees. Credits are not refunded within a query. Defaults can reject wide inputs well below the row limit. Resource exhaustion returns an error (SQLSTATE 53200), never truncated successful output. There is no disk spill. Concurrent coordinators each consume their execution allowance; output namespace quotas do not constitute a process-wide memory cap.

Statement adbc.redis.partition_execution_limits accepts a partial JSON object:

Field Default Hard maximum
transport_bytes 134217728 (128 MiB) 1 GiB
source_bytes 67108864 (64 MiB) 1 GiB
working_bytes 268435456 (256 MiB) 1 GiB
work 1000000 10000000
source_rows 100000 1048576
candidate_keys 200000 2000000

Dedicated source pools validate RESP headers before go-redis allocates decoded collections/bulk payloads: 1 MiB bulk, 4,096 collection entries, depth 32 and 4,096-byte headers. All connections/nodes share the transport credit. Known-field HASH and membership reads use bounded pipelines. The coordinator has a one-minute execution deadline; cancellation closes its source sockets. Scalar values are limited to 64 KiB, schema columns to 1,024, nested type depth to 32, expression nodes to 4,096, and decimal precision to 76. Arrow chunks use the restricted validated format in the storage contract.

Database adbc.redis.partition_result_limits accepts partial JSON fields job_bytes, total_bytes, chunk_bytes, chunks, jobs. Defaults are 64 MiB/job, 256 MiB/namespace, 1 MiB/chunk, 1,024 chunks and 1,024 jobs/tombstones. Hard maxima are 1 GiB/job, 8 GiB/namespace, 16 MiB/chunk and 1,024 chunks/jobs. All publishers in a namespace must agree on the persisted policy. Reserve the job output allowance before capture; return unused bytes on commit. All tenant keys share one Cluster slot, concentrating retained traffic on one primary.

Quota control keys must survive payload eviction. The default adbc.redis.partition_control_persistence=auto checks CONFIG GET maxmemory-policy on every primary, before upload and publication. It accepts noeviction and volatile eviction policies, rejects allkeys/unknown policies. Volatile payload eviction is allowed but fails affected replays. If the provider restricts CONFIG, an operator may set guaranteed only after independently verifying persistent control keys are protected. Preserve that condition for the namespace lifetime; this option is an operator assertion, not a driver check.

Lifetime, failures and authorization

Statement adbc.redis.partition_result_ttl is integer seconds, default 300, range 1–86400, measured from publication with no read renewal. Uncommitted uploads have a fixed ten-minute lease. A dead coordinator before publication leaves no readable manifest; its bounded reservation expires and is pruned on the next quota operation. Cancellation/error attempts nonce-authorized cleanup using a fresh deadline-aware pool. Transport-uncertain operations are not retried as a new SQL execution.

Closing producer handles preserves committed results. Early consumer release, worker cancellation/death and partial/full replay do not revoke shared output. Readers fetch one chunk at a time and check state/checksums/schema/counts before decoding. Already downloaded values can remain consumable until the next chunk; revocation is not a recall of local Arrow memory. Replay socket waits are bounded to at most 30 seconds. Missing/evicted/corrupt/expired/revoked output fails with an error and never silently falls back to source execution.

A namespace-authorized publisher can execute:

DROP PARTITION RESULT '0123456789abcdef0123456789abcdef';

Use the job ID from the descriptor. Revocation atomically deletes the manifest and complete bounded chunk key space and returns byte quota. A zero-byte tombstone counts against the job limit until its original expiry. Explicit revocation repairs missing/corrupt job metadata using the authoritative expiry ledger. It cannot repair destroyed quota ledgers; control corruption fails closed. Cleanup never deletes source data. Small namespace control keys persist until operator cleanup after all jobs expire.

Descriptors contain no credentials, endpoint, SQL or arbitrary Redis keys and are not bearer authorization. A minimal worker needs +ping +time +getrange and %R~adbc:{retained:TENANT}:*; Cluster additionally needs +cluster|slots +command +asking. Publishers need existing SQL/source privileges, source SCAN, known-field reads and index membership, topology checks when clustered, plus namespace EVAL and the scripts' declared String/ZSET operations. Publishers are trusted within their namespace. Give separate tenant credentials/namespace policies; key ACLs remain the authorization boundary.

Retained mode adds a complete coordinator read, serialization/upload, stored output until expiry and one download per worker/retry. It is useful for global SQL and stable retries, not an inherent speedup over ExecuteQuery/live scans. See implementation evidence and deployment validation for tested revisions, compatibility coverage and scoped measurements.

Explicit layouts (#302)

Revision 4 descriptors select non-dictionary Arrow result layouts; revision 2 canonical descriptors remain readable. The coordinator validates each converted batch before publishing. Stored chunks retain canonical restricted IPC and its schema digest; readers independently validate and apply the selected layouts. Closing the producer or truncating its source does not change committed replay. Dictionary layouts reject before publication. Existing expiry, revocation, namespace authentication and quotas apply to both descriptor revisions.