Bounded Substrait plan support

Date: 2026-11-04. Implementation baseline: f4d5b787e8c28c036baf49de9190c4e6f83ca2e8. Status: bounded runtime implementation for issue #213. Author: Codex, requested by the repository user. Maintainer acceptance is not recorded. Intent, plan, evidence.

SetSubstraitPlan accepts binary Substrait 0.108.0 plans in the subset below. InfoVendorSubstrait is true and min/max versions both report 0.108.0. This does not claim full Substrait coverage. See the runtime validation record. The earlier design records above remain historical evidence.

Feasibility decision

The supported grammar reads one base table, optionally filters it, and optionally projects fields and typed expressions. Validation establishes the relation shape, exact types and schema binding before execution.

Existing boundary Reuse and necessary restriction
statement.go: prepare, binding, execution and schema A separate plan state bypasses SQL parsing and revalidates table metadata.
sql.go: SelectStmt, ColumnRef, Literal, Binary Construct a validated AST with typed literals and canonical column names; keep positional Substrait fields in a separate ordered vector.
exec.go: resolveTable Reuse metadata and planning after enforcing base-table-only binding. Ordinary SQL planning also accepts views and functions, which this subset excludes.
query.go: planWhere, prepareScan, scan iterators Preserve exact HASH values and residual predicates. Passing a predicate to Search does not prove exact evaluation.
types.go, arrowconv.go Reuse physical Arrow mappings only for exact logical matches; prohibit implicit coercion and precision loss.
stream.go Reuse stream ownership/cancellation with a typed Substrait reader; both result modes share its admission limits.
pkg/driver.go: C entry point Validate size and pointer before the C-to-Go copy; a Go-only size check comes too late. This file is generated: update its generator/source workflow too.

The driver uses a strict wire validator, pinned generated protobuf messages, and a small driver-owned typed AST adapter. The existing scan iterators, expression evaluator and Arrow builders execute it. No SQL text is constructed from plan identifiers. Schema sources, hashes, licensing and regeneration are checked in. The upstream Go module release is tracked by substrait-protobuf#7.

Wire format and relation grammar

Protocol pin: Substrait 0.108.0, binary substrait.Plan only. The version range is deliberately exact. Require an explicit matching version and execution behavior VARIABLE_EVALUATION_MODE_PER_PLAN; reject other behavior, versions and nonempty git_hash. Producer text is diagnostic only. JSON, text protobuf and compressed payloads are outside the API. Pin the complete transitive descriptor set from v0.108.0, including plan.proto.

Exactly one root and this input-to-output grammar are candidates:

Read(named_table, base_schema) -> [Filter(condition)] -> [Project(expressions)] -> Root(names)

Each bracket means zero or one operator. No filter above a project, repeated projects, relation references, joins, aggregates, sorts, fetch/limit, writes, virtual tables, files, exchanges or extension relations. Reject all fields outside the allowlist, including unknown protobuf fields, advanced extensions, hints, expected type URLs, read masks and read filters/best-effort filters. Putting filtering in Filter alone avoids multiple initial semantics.

Read and Filter permit absent/direct RelCommon only. Project permits direct or emit: its ordered vector is input fields followed by newly computed fields; emit selects/reorders that vector and may repeat an index. Validate every index. Expressions reference the Project input only. Translate selected vector entries to explicit SELECT items; never implement Project as replacement by default. Require nonempty final output and exactly one Root name per output column; preserve names and duplicate names without treating them as SQL identifiers. These choices follow the pinned algebra definitions.

Table binding and schema discovery

Named-table components mean [table], [schema, table] or [redis, schema, table]. Components are literal identifiers, not dotted SQL, URIs, HASH prefixes, index names or Redis commands. Reject empty components, invalid UTF-8/NUL and other catalogs. Use resolveTable for current-schema and connection-owned pg_temp rules; never access another session's physical temp schema. Then require stored base-table metadata, including eligible adopted tables; reject views and inferred/unregistered HASH collections.

Require a flat base schema matching the table's full ordered public columns: names, logical types, precision/scale, units, timezone and nullability. No hidden row identifiers, missing columns, subsets or implicit widening. Resolve field references by zero-based ordinal against this checked vector, using canonical names only after validation. Allow root-relative direct struct-field references with no child; reject outer, expression-rooted and nested references.

GetTableSchema supplies discovery before plan generation. Prepare validates and binds metadata without scanning rows. ExecuteSchema returns the derived output schema without scanning. Recheck the base schema at each execution/schema request so altered or recreated tables cannot reuse stale ordinal bindings. This does not add snapshot isolation: existing autocommit concurrency limits still apply. Never retain a planned physical row prefix across executions.

Types, functions and parameters

All types require explicit required/nullable declarations and the default type variation. Reject user-defined types/variations even when declared but unused. The initial candidates map as follows; unsupported table types reject the whole plan even when projected away.

Substrait candidate Driver representation / restriction
bool, i16/i32/i64, fp32/fp64 Same-width ColType; i8 is excluded because there is no matching driver Kind. Floating values may be read/projected, but floating comparisons are excluded pending NaN/ordering proof.
string, binary Unbounded text/binary with exact bytes; reject fixed/length-constrained strings, aliases and nondefault collations.
decimal Decimal128, precision 1–38, scale 0–precision; validate the signed 16-byte little-endian literal and digit range without float conversion.
precision_timestamp / precision_timestamp_tz Precision 0/3/6/9 only, matching seconds/ms/us/ns; timezone is respectively naive/UTC. No session-local conversion, rounding or unit truncation.
typed NULL Retain the declared type through planning and Arrow output; do not let SQL's unknown-literal inference choose it.

Exclude date/time/interval, containers, UUID and deprecated timestamp encodings initially. See pinned type.proto and the literal definitions in algebra.proto. Decimal and timestamp eligibility requires the differential cases below; adding a mapping alone is insufficient.

Extension identifiers are offline lookup keys only; no plan-supplied URI or URN is fetched. Only extension:io.substrait:functions_comparison and extension:io.substrait:functions_boolean are accepted. Their exact definitions are pinned from Substrait v0.108.0 in the binding record. Resolve URN anchor -> function anchor -> full declared signature -> typed arguments/output. Anchor zero is valid. Duplicate/dangling anchors, unknown declarations (even unused), options and mismatched output types/nullability are rejected.

Accepted signatures are equal:any_any, not_equal:any_any, lt:any_any, lte:any_any, gt:any_any, gte:any_any, is_null:any, is_not_null:any, and:bool, or:bool, and not:bool. The variadic boolean signature includes its argument type once; this subset accepts exactly two arguments for and/or. Comparisons require identical supported operand types; boolean comparisons allow equality/inequality only. Floating comparisons reject; null tests and projection allow floats. No mixed-type coercion or arbitrary SQL function dispatch occurs. Boolean operations use three-valued logic; filters retain only TRUE. Null tests return required boolean; comparison nullability mirrors inputs, and boolean function output is nullable per its pinned overload.

Only identity casts with matching type attributes and nullability are candidates; require explicit throw-on-failure behavior. Reject unspecified/return-null cast behavior, narrowing, string/numeric conversion and timezone conversion. SQL supports more casts, but that is not proof of identical Substrait semantics.

Parameters are excluded in the first subset: reject every parameter expression. For an accepted plan GetParameterSchema returns an empty schema; Bind and BindStream return StatusNotImplemented without retaining new input. Preserve the caller's BindStream reference on refusal; the caller still releases it. Do not reinterpret bound batches as ingestion or silently repeat execution.

Resource, error and lifecycle contract

These are hard ceilings for Substrait execution. Check before allocating or expanding data, and at each traversal loop:

Budget Maximum
Serialized plan, checked at C boundary before copying and in native Go 1 MiB
Protobuf message nesting / semantic expression depth 64 / 32 (both apply)
Relations / expression nodes (whole plan) 3 / 4096
Input columns / output columns / project expressions 256 / 256 / 256
Extension URNs / declarations 2 / 64
One literal payload / identifier component 64 KiB / 1024 UTF-8 bytes
Total translated nodes, including emit duplication 8192

Use strict wire traversal plus the pinned protobuf runtime recursion limit and a bounded semantic validator; do not discard unknown fields. Inspect nested messages too. Reject duplicate singular/oneof wire fields rather than accepting last-value-wins ambiguities. Bounds must cover discarded/unreferenced fields and declarations. For the C wrapper reject null pointers with nonzero length and values above the budget before any narrowing to C.int or C.GoBytes. Never panic/poison the driver for an ordinary invalid plan.

Plan limits alone do not bound a table scan. The execution path uses shared accounting for both materialized and streamed paths: at most 1,000,000 decoded rows scanned (including filtered-out rows), 256 MiB cumulative decoded input/output payload, and a 30-second deadline capped by any earlier caller deadline. Check accounting before buffering rows; use overflow-safe counters. Exceeding a budget fails the query rather than truncating results. Streaming may already have delivered batches; surface a terminal reader error and clean up. Ordinary SQL keeps its existing limits. Accounting conservatively charges raw input payload and fixed per-value overhead before decoding, plus output and retained HASH-discovery keys. Redis reply pages and Arrow batches are bounded independently; the byte counter is not a process-RSS measurement. HASH discovery also has a 1,000,000-key-visit work limit, including duplicates.

Condition ADBC status
Closed statement / no executable query InvalidState
Malformed wire, missing required structure, bad ordinal/type/schema, duplicate anchor InvalidArgument
Version, relation, extension, cast or parameter outside the subset NotImplemented
Missing table/catalog NotFound
Size/depth/node/scan/byte budget exhausted InvalidArgument, with stable budget name and no plan payload
Caller cancellation / deadline (including internal deadline) Cancelled / Timeout

Validate into temporary owned state. A failed setter preserves the prior query, bindings and ingest mode; a successful setter atomically replaces them and releases prior parameters once. Copy/own caller bytes; mutation after return must have no effect. SetSqlQuery must clear plan state, and ingest selection must clear both query kinds. Check cancellation during decoding/translation and metadata lookup; no Redis row reads until the whole plan is validated. Release readers, cursors, worker tokens and any new budget state on failure, Cancel, Close and early reader release. Diagnostics identify a structural path/status; do not expose literals, raw plans, URIs with credentials or physical keys.

Capability reporting and execution boundaries

GetInfo reports true and min=max 0.108.0. Driver-manager tests cover both requested-code and all-code queries and completely drain metadata/results. A populated C function pointer alone is not evidence of execution support.

Prepare and ExecuteSchema revalidate metadata without scanning rows. A failed setter preserves prior SQL/plan/bindings/ingest selection. Successful query-kind changes clear bindings exactly once. Accepted plans have an empty parameter schema; Bind/BindStream reject with NotImplemented and preserve ownership. Partitions, CUDA, IPC and alternate result layouts are outside this subset and reject explicitly. Setting SQL or ingest mode clears the plan. Close releases active Substrait readers; caller-retained Arrow buffers remain valid.

The original acceptance matrix defines the broader design gates. The runtime record states the checks actually run and remaining gaps. Full Substrait coverage is outside this bounded implementation.