Live unmanaged HASH tables

Unmanaged tables expose application-owned HASHes through SQL without copying their data. Application changes, SQL writes and TTL expiry affect the same keys. The source Search index determines visibility during scans. Managed copies remain the default for adoption and foreign-index imports.

Adopt a HASH collection

Create an application-owned HASH Search index on exactly the collection prefix, without a FILTER. Then register it:

redis-adbc adopt -prefix app:user: -table users -index idx_users \
  -mode unmanaged -apply

Pass connection options using the CLI's normal connection flags. Without -apply, the command plans and reports blockers without registering anything. The Go API uses HashAdoptOptions{Mode: "unmanaged", ...}. Omitting Mode or selecting "managed" retains managed-copy behavior.

This mode requires an existing compatible source index. -own-index, -index-columns and -trust-strings are unsupported. The application owns the index's configuration, memory and lifecycle. Adoption considers HASH keys under the prefix with no further :; nested collections are excluded. Type, rename and text/native encoding overrides remain available. Schema inference examines the source; registration does not copy or change its HASHes.

Register foreign indexes

Set both database options before opening the connection:

db_kwargs = {
    "uri": "redis://localhost:6379/0",
    "adbc.redis.foreign_indexes": "true",
    "adbc.redis.foreign_indexes_mode": "unmanaged",
}

The mode option is also available on a connection; set it before enabling adbc.redis.foreign_indexes. Tables appear in redis_foreign under their exact index names. Quote punctuation, for example redis_foreign."idx:users". Multiple source prefixes and index FILTER membership are respected.

Foreign column inference samples up to 128 source HASHes and includes indexed attributes. Incompatible values outside the sample can fail a query. Explicit adbc.redis.foreign_indexes_refresh=true refreshes registered schema and source definitions; later source values do not require refresh. Repeating unmanaged adoption likewise refreshes the registration's schema. A changed source index definition causes queries to fail until explicitly refreshed or re-registered. Changing the registered source identity (its index name or primary prefix) requires dropping the old registration first.

Existing tables cannot silently switch modes. DROP the old table/registration first or choose another destination. Managed foreign imports remain read-only; unmanaged foreign registrations permit source DML.

Read and write the original keys

SELECT __key, name, age FROM users;
INSERT INTO users (__key, name, age) VALUES ('app:user:42', 'Ada', 37);
UPDATE users SET age = age + 1 WHERE __key = 'app:user:42';
DELETE FROM users WHERE __key = 'app:user:42';
DROP TABLE users;

__key identifies the original Redis key and is excluded from SELECT *. INSERT requires a new explicit key in the registered source prefixes; an existing key is refused. UPDATE preserves unrelated HASH fields and supported text encodings; it does not recreate an expired/deleted HASH. Setting a HASH's last field to NULL is refused because Redis would delete the row. DELETE removes the source HASH, including its unmodeled fields, and skips a key that has become a non-HASH value. UPDATE cannot rename __key.

For foreign indexes, INSERT is supported only without an index FILTER. UPDATE and DELETE use the source index's membership, including filters and multiple prefixes. Writes can change index membership according to the application index's normal behavior. Native Redis ACLs must permit the required source reads/writes and scripts; registration does not grant source-key access.

DROP removes only the table registration. Source HASHes, TTLs and Search indexes remain application-owned and survive managed sweep/compaction. Foreign indexes can be registered again by a later discovery/refresh.

Write Arrow data to source HASHes

Use ADBC ingestion in append or create_append mode against an existing unmanaged registration. Every row must include a non-NULL string __key under the registered source prefixes. Keys must be new: this is INSERT, not upsert. Logical column names, types and native/text encodings follow the registration, including renamed fields. A row must contain at least one non-NULL stored field.

import pyarrow as pa

# conn is an ADBC connection; public.users is already registered as unmanaged.
rows = pa.table({"__key": ["app:user:43"], "name": ["Grace"], "age": [38]})
with conn.cursor() as cursor:
    written = cursor.adbc_ingest("users", rows, mode="append", db_schema_name="public")

Arrow input is converted in bounded slices and writes source HASHes directly. There is no unmanaged publication step: each accepted row becomes visible as it is written. It does not create managed generations, acquire the managed Arrow append gate, or freeze application data. Existing HASHes, unrelated fields and TTLs are not overwritten. INSERT through filtered foreign indexes, shared read snapshots and retained/staged writes are unsupported. Schema-level refusals occur before consuming the stream; row values and keys are validated as each conversion slice is reached.

Once an unmanaged Arrow write passes target/schema preflight, a conversion, stream, key-conflict, cancellation or Redis error leaves earlier successful rows in place. The ADBC error message and UTF-8 error detail adbc.redis.ingest.acknowledged_rows report the number of acknowledged rows; Go ExecuteUpdate/ExecuteQuery also return that count with the error. This is a lower bound, since a lost reply can hide applied rows in a submitted pipeline. The count includes all acknowledged successes and need not describe a contiguous prefix of input. INSERT uses pipelines of at most 128 single-key scripts with a 1 MiB request target and a 64 MiB single-row ceiling. After a conflict, cancellation or lost reply, already-submitted rows may still apply; no further pipeline is submitted. The original ADBC status and SQLSTATE are retained. Failures before target resolution or unsupported-operation admission may have the usual unknown count instead of a progress detail; input has not been consumed. No input or uncertain mutation is replayed. Inspect source state before retrying; there is no rollback or whole-stream atomicity. This API does not require the managed snapshot NativeClient wrapper.

Create a new unmanaged collection

Provide the statement option adbc.redis.ingest.unmanaged_source to select unmanaged creation. Its JSON requires an explicit application prefix and index name; optional encodings maps public column names to native or text. Native storage is the default. Text encoding is supported for boolean, date and time columns. All input rows still require explicit string __key values.

import json

with conn.cursor() as cursor:
    cursor.adbc_statement.set_options(**{
        "adbc.redis.ingest.unmanaged_source": json.dumps({
            "prefix": "app:user:",
            "index": "app_users_membership",
            "encodings": {"enabled": "text"},
        })
    })
    rows = pa.table({"__key": ["app:user:1"], "name": ["Ada"], "enabled": [True]})
    cursor.adbc_ingest("users", rows, mode="create", db_schema_name="public")

create requires an unused table name, an empty prefix across all key types (including nested keys), and an unused index name. The Arrow schema defines the public fields and SQL types; __key is input-only. The driver creates a minimal application-owned Search index for row discovery: one TAG NOINDEX attribute, no hidden row-ID fields, and no field indexes for predicate pushdown. Creation uses SKIPINITIALSCAN and rechecks prefix emptiness after index creation, so a concurrent row that might have missed indexing causes refusal before registration or input consumption. SQL predicates continue to evaluate authoritative HASH values. The application can manage its Search indexes; the managed ingest.index_columns override is not supported for this creation path.

create_append creates when the registration is missing and otherwise appends using its existing schema, index and encodings. An existing unmanaged target accepts create_append without the source option. When the option is supplied, its prefix/index and any explicitly supplied encodings must match the existing registration. A managed target cannot silently switch ownership modes.

Creation coordinates registration metadata before consuming input. It commits the registration before writing rows, then releases creation admission. Data writes have no publication step. Failure can leave the new application index, registration and acknowledged partial rows; these resources are deliberately preserved, including after a lost creation reply or cancellation. Inspect them before retrying. To use an already existing source/index, explicitly register it with unmanaged adoption and then append. There is no automatic replay, rollback, index deletion or whole-call atomicity. Coordinate external writers during creation; the empty-prefix check does not freeze application activity.

The statement option persists until cleared. Set it to the empty string before reusing the statement to create a managed table. With no source option, new Arrow tables use the managed default. Temporary/virtual targets, retained preparation and staged writes are unsupported. Destructive replace requires the separate dedicated replacement contract; registration alone never authorizes replacement.

Replace a dedicated unmanaged collection

For a dedicated prefix/index with all application and SQL writers stopped, an explicit unmanaged_source.replace consent can replace both public schema and source rows. It deletes old HASHes (including unmodeled fields), discards their TTLs, resets only the named Search index to a membership index and streams new complete persistent HASHes with explicit __key values. Shared/nested scopes, foreign registrations, FILTERs and multi-prefix indexes are refused. The initial and final inventories each have a 100,000-key / 64 MiB bound. Input key tracking has the same bounds; exceeding a limit stops the call.

This is a direct destructive operation with partial visibility and failure, with no rollback or automatic replay. Managed replacement instead publishes schema and data atomically. Read the option example, phases, error details and recovery procedure before selecting unmanaged replacement.

Consistency and supported operations

An unmanaged statement pins catalog metadata, not source row values. Scans, joins and streamed results can observe intervening application changes or expiry. Source Search indexing determines when changes become visible to scans. For compatible native integer NUMERIC attributes, Search narrows candidate rows and SQL rechecks the complete predicate against current HASH values. Other attribute types and text encodings retain client evaluation. Source indexing lag can affect predicate candidate discovery as well as membership. Vanished/non-HASH keys are skipped when fetching live HASHes after discovery. Unfiltered full scans retain LOAD *; filtered unordered scans can fetch live HASHes after Search key discovery. Sorting and value aggregates do not trust mutable index values. A query mixing managed and unmanaged tables retains managed generations but does not acquire a common point-in-time view of all data.

Each source HASH mutation is individually atomic. INSERT and Arrow append use bounded pipelines; UPDATE and DELETE issue one mutation at a time. Multi-row INSERT, UPDATE, DELETE, MERGE and Arrow append can partially succeed. There is no rollback or atomic publication for them; application and SQL writers must coordinate themselves. The driver never automatically replays an uncertain source mutation. After an error, inspect source state before retrying a write.

Unmanaged tables reject ALTER, TRUNCATE (including truncate_adopted=true), managed index changes, replacement without dedicated-source consent, retained write preparation, shared read snapshots, native row-ID reads and live partition reads. Data-changing WITH statements cannot stage unmanaged writes inside managed publication. The always-on managed snapshot option applies to managed tables only.

Choose a managed copy when typed immutable generations, statement snapshots or atomic SQL/Arrow publication are needed. See the ownership comparison for tradeoffs.

Experimental Software Flex stopgap

Unmanaged registrations work with adbc.redis.flex=true on compatible Redis Software Flex deployments. Both entry paths are supported: unmanaged HASH adoption and foreign indexes with adbc.redis.foreign_indexes_mode=unmanaged. For example:

redis-adbc adopt -prefix app:user: -table users -index idx_users \
  -mode unmanaged -option adbc.redis.flex=true -apply

Use the normal URI/TLS connection flags. For foreign discovery, set the Flex, foreign-index and unmanaged-mode database options before Open, or set unmanaged mode on the connection before enabling foreign indexes. Managed-copy foreign imports remain unsupported on Flex; an enabled Flex connection cannot switch foreign discovery to managed mode.

The application must supply a ready, compatible HASH Search index. The driver never creates, repairs, reindexes or takes ownership of that index. Registration verifies singleton FT.SEARCH ... INKEYS support. Reads enumerate source HASHes using SCAN, check each candidate's index membership, and fetch authoritative HASH values. Even direct __key lookups check membership. This preserves index FILTERs, multiple/overlapping prefixes and adopted collections' nested-key exclusions; HASHes omitted by the application index remain invisible. Source indexing lag, application changes and expiry retain ordinary live semantics.

Enumeration requires SCAN permission; reads also require FT.INFO, FT.SEARCH and HASH read permissions. The stopgap's normal temporary capability-probe permissions also apply. Membership checks use pipelines of at most 128 singleton queries, with a 1 MiB request target and a 64 MiB single-key ceiling. Each query returns a count, so data enumeration does not depend on Search result caps, aggregate cursors, sortable fields or numeric driver row IDs. Deduplication retains one entry per observed key: memory grows with source key count. Sorting and SQL evaluation run in the client; no performance parity or constant-memory claim is made. Source ACLs, native encodings, TTLs, non-replayable writes and DROP/GC ownership rules above remain in force, as do the unmanaged operation exclusions.

See implementation and live validation.

Filtered reads can fetch only required HASH fields with native HMGET after Search discovers keys. A non-NULL selected value, including an empty string, proves the HASH existed at that atomic native read. If every selected field is NULL, a fresh HGETALL supplies the complete later live image or omits a now missing or non-HASH key. This does not mix fields from two observations or provide a statement snapshot; SQL still rechecks its existing residual predicate.

Full scans retain their established retrieval strategy. HMGET permission enables selected-field acceleration; an exact native command/key-permission refusal retains the existing HGETALL path. No TYPE/HSTRLEN or Lua permissions are needed, and large source values avoid an extra Lua copy. Requests retain bounded field and pipeline sizes. Unknown command/pipeline outcomes fail without replay; application HASHes/indexes are not modified, and permissions/values are not cached across statements. See the current concurrency work.