Server guide

Back to the README. This guide covers sessions, deployment, operations, architecture and development. Feature availability depends on the build profile.

Contents

Sessions

A session keeps a client's session state from one request to the next: SET parameters such as search_path and TimeZone, and temporary tables and views. The session has its own ADBC connection, opened by its first SQL request.

  • They are cookie sessions. A response to a request without a session cookie sets one, arrow_flight_session_id. A client that sends it back is in that session.
  • Which clients keep cookies:
  • The Arrow Flight SQL JDBC driver does by default.
  • The ADBC Flight SQL driver does with adbc.flight.sql.rpc.with_cookie_middleware=true. Each ADBC connection is then its own session.
  • arrow-go clients do with flight.NewCookieMiddleware().
  • What it costs. A freshly issued cookie reserves identity metadata for 30 seconds, with at most 10,000 issued/active entries combined. No dedicated ADBC connection is opened until session SQL or options need one. Clients that do not return cookies continue to use the pool.
  • When a session starts. It begins with the client's second request, the first one carrying the cookie. Make a cheap call first (for example GetSqlInfo) if the very first statement is a SET.
  • Session options (SetSessionOptions, GetSessionOptions):
Option Effect
catalog Must be redis
schema Sets the current schema (search_path); the schema must exist
timezone Sets the session time zone, as SET TIME ZONE does
  • Erasing an option goes back to its default.
  • An unknown name gets INVALID_NAME, and a bad value INVALID_VALUE.
  • SetSessionOptions starts a session if the request has none.
  • When a session ends. CloseSession ends it: its connection closes, and its temporary tables are dropped. So does --session-idle-timeout (10 minutes) without requests; a session made by SetSessionOptions whose cookie never came back closes after 30 seconds. A later request with the old cookie fails with ABORTED and receives a fresh cookie. Restore required options explicitly before retrying; the failed request was not executed.
  • Replica/restart behavior. Use session affinity so every cookie-bearing request reaches its owning process. An unknown cookie (wrong replica, restart, expiry or CloseSession) returns ABORTED before executing the request and issues a fresh cookie. Reconnect to the owner, or explicitly recreate settings and temporary objects before retrying. Shared bearer-token secrets allow authentication across replicas; they do not share sessions.
  • Ownership. A session belongs to the Redis user that made it. With --auth=redis-acl, another user's request with the same cookie gets a session of its own.
  • Publication conflicts. The native driver's definite catalog publication refusal (IO, SQLSTATE 40001, exact Redis catalog-conflict diagnostic) returns UNAVAILABLE while preserving the live session connection, SQL settings and temporary objects. Flight does not retry the request. A caller may submit a fresh safe statement after this refusal, but must not replay a script's committed prefix or reuse consumed Arrow input. Generic I/O, cancellation, timeouts and unknown publication outcomes do not receive this connection-preserving classification. Shared read-snapshot failures remain terminal.
  • Plain scans stream. A query that streams from Redis outside a session does in a session too, on a pooled connection, so the session's own stays free for its other statements (clients such as JDBC run some while they read a result):
  • The pooled connection gets the session's current schema (search_path) and time zone, as they are when the query starts (SET LOCAL included): a SET while the result is read doesn't change it. Of the SET parameters, only these two change what a scan returns. The connection's own are put back before the pool reuses it.
  • It sees the session's writes, since every statement commits as it runs.
  • The session's streams count against --max-streams and --max-table-streams like any; past them, results are read whole within --result-memory.
  • A session isn't closed as idle while one of its results streams.
  • Once the session has made a temporary table or view (or ingested a temporary table), its results are read whole on its own connection until the session ends: temporary objects hide other tables' names, and only that connection sees them.
  • Other queries (aggregates, joins, SHOW, …) run on the session's connection, built whole as outside a session.
  • Several servers. Sessions live in one server's memory. Route a client to the same server each time (sticky sessions on the load balancer).
  • Turning them off. --sessions=false does so: no cookies, and every request uses the pool.

Deployment and operations

Authentication and TLS

By default (--auth=none), every client runs with the credentials in --redis-uri. In that mode, keep the server on a trusted network.

Basic TLS/ACL authentication is available in the community build. Redis ACL mode requires both Flight TLS certificate/key files and a verified rediss:// backend; credential-bearing plaintext Redis URIs and skip_verify=true are rejected. For a private Redis CA, pass --adbc-option adbc.redis.tls_ca_file=/path/to/ca.pem (the same bundle is used by login, gateway Redis clients and native ADBC).

With --auth=redis-acl, clients log in with the Flight handshake and Basic credentials, which are a Redis ACL user's username and password. The server checks them against Redis and returns a bearer token, and each call then runs as that Redis user. Redis's ACLs decide what each client may read and change, for SQL and datasets alike.

# REDIS_URI must already name your verified rediss:// deployment.
./bin/redis-flight-server --auth redis-acl \
  --tls-cert server.crt --tls-key server.key
flight_sql.connect("grpc+tls://host:8815", db_kwargs={"username": "alice", "password": "…"}, autocommit=True)
client.authenticate_basic_token("alice", "…")  # pyarrow.flight: returns the header to send
  • Tokens are sealed with AES-256-GCM using a key derived from --token-secret. They hold the identity and the Redis password, encrypted, and expire after --token-ttl (12h). A configured secret must contain at least 32 bytes; use cryptographically random secret material through FLIGHT_TOKEN_SECRET. Without a configured secret, the server generates a fresh random key at startup.
  • Several servers that share the secret accept each other's tokens. A token secret and TLS are therefore what you need behind a load balancer. Without a secret, tokens last only until the server restarts.
  • ACL rules. A user needs the commands the Redis ADBC driver lists for SQL. For datasets, it also needs the key pattern ~arrowflight:* and these commands:
  • to write: +incr +hset +pexpire +persist +unlink +del +eval +evalsha (servers before the HASH layout also need +rpush)
  • to read: +hget +hgetall +hmget +scan +get, and +llen +lrange for generations stored by servers before the HASH layout

A user with exactly the driver's commands plus these ran examples/python/quickstart.py. A read-only user (the driver's read-only list plus the read commands above) can query and read datasets. Its writes fail with PermissionDenied. - Health checks need no login, so load balancers and probes can call them. They report whether the server serves and Redis answers its background PING; see Health checks.

Configuration

Every flag can also be set with the environment variable shown.

Flag Environment Default Meaning
-listen FLIGHT_LISTEN localhost:8815 Address to listen on
-redis-uri REDIS_URI redis://localhost:6379/0 redis://[user:pass@]host:port/db, or rediss:// for TLS to Redis
-auth FLIGHT_AUTH none none or redis-acl
-token-secret FLIGHT_TOKEN_SECRET random Seals bearer tokens; share it between replicas
-token-ttl FLIGHT_TOKEN_TTL 12h Bearer token lifetime
-tls-cert, -tls-key FLIGHT_TLS_CERT, FLIGHT_TLS_KEY PEM files; enable TLS
-key-prefix FLIGHT_KEY_PREFIX arrowflight: Prefix of the server's own keys
-dataset-ttl FLIGHT_DATASET_TTL 0 (keep) Expire uploaded datasets
-dataset-endpoint-batches FLIGHT_DATASET_ENDPOINT_BATCHES 0 (one endpoint) Give a dataset's FlightInfo one endpoint per this many batches, for parallel reads
-pool-max-idle FLIGHT_POOL_MAX_IDLE 8 Idle ADBC connections kept per Redis user
-pool-max-open FLIGHT_POOL_MAX_OPEN 32 ADBC connections in use at once per Redis user
-stream-overflow FLIGHT_STREAM_OVERFLOW buffer When an eligible scan has no stream capacity: buffer within the result budget, or reject before execution with RESOURCE_EXHAUSTED
-max-streams FLIGHT_MAX_STREAMS 8 Query results a Redis user can have streaming from Redis at once, at most -pool-max-open minus 1. Each keeps a connection; further queries are read whole first, within -result-memory
-max-table-streams FLIGHT_MAX_TABLE_STREAMS 32 Query results that can stream from one table at once, across all users. Each keeps a cursor on the table's index, which allows 128 across every client (on standalone Redis, streams also stop at three quarters of them open); further queries are read whole first, within -result-memory
-result-memory FLIGHT_RESULT_MEMORY 1 GiB Bytes of memory that query results read whole for want of a stream may take at once, across the server. A result that doesn't fit fails with RESOURCE_EXHAUSTED
-stream-stall-timeout FLIGHT_STREAM_STALL_TIMEOUT 2m End a streaming result whose client takes longer than this to read a batch
-sessions FLIGHT_SESSIONS true Sessions for clients that keep cookies
-session-idle-timeout FLIGHT_SESSION_IDLE_TIMEOUT 10m Close a session after this long without requests
-max-msg-size FLIGHT_MAX_MSG_SIZE 64 MiB Largest gRPC message accepted
-shutdown-timeout FLIGHT_SHUTDOWN_TIMEOUT 25s On SIGTERM or SIGINT, let running calls finish for this long, then cancel them
-adbc-option k=v (repeatable) FLIGHT_ADBC_OPTIONS (comma-separated) Redis ADBC driver options, such as adbc.redis.aggregate_pushdown=all, adbc.redis.default_schema=sales, adbc.redis.cluster=true
-metrics-listen FLIGHT_METRICS_LISTEN off Address to serve Prometheus metrics on, at /metrics (plain HTTP, no auth)
-log-level FLIGHT_LOG_LEVEL info debug logs every call but health checks
-healthcheck Check the server at -listen with the gRPC health service and exit: 0 if it serves, 1 if not. Trusts only -tls-cert when that is set
-version Print the version and exit

The server starts pinging Redis for health checks at startup. It opens ADBC connections when SQL requests arrive; opening one takes several round trips, so connections are pooled per Redis user. A user's connections are closed after 10 idle minutes.

Shutdown

On SIGTERM or SIGINT the server stops accepting calls and lets running ones (uploads, query streams) finish for up to --shutdown-timeout (25s). Then it cancels the rest, waits for them to release their connections and cursors, closes the sessions (dropping their temporary tables) and the Redis connections, and exits. A second signal exits at once.

Keep --shutdown-timeout under the time your platform waits before it kills the process: Kubernetes' terminationGracePeriodSeconds is 30s by default, and compose.yaml sets stop_grace_period: 30s (Docker's default is 10s).

The server pings a connection that has been idle for a minute, and closes it if the client doesn't answer within 20 seconds, so a client that has vanished doesn't hold a call, a connection or a cursor until TCP gives up.

Monitoring

Health checks

Backend health monitoring below requires enterprise. Community reports UNKNOWN, which is not readiness, and its image and Compose service have no backend probe.

The server has the standard gRPC health service (grpc.health.v1.Health), on the Flight port. Its calls need no authentication, even with --auth=redis-acl, and aren't logged.

  • SERVING while Redis answers a PING. The server pings every 5 seconds, on a connection of its own, so checks cost Redis nothing. With --auth=redis-acl, an authentication error counts as an answer (each client brings its own credentials). With --auth=none it doesn't, since every request uses the server's. On a cluster, the node in --redis-uri is pinged.
  • NOT_SERVING while Redis doesn't answer, and from the start of a shutdown.
  • The service names are "" (the server) and arrow.flight.protocol.FlightService.

redis-flight-server -healthcheck asks the server at -listen (an all-interfaces address is checked on the loopback address) and exits 0 if it serves, 1 if not. With -tls-cert it uses TLS and trusts only that certificate. The Docker image's HEALTHCHECK and the compose service run it; it reads the same environment variables as the server, not the server's flags. Any gRPC health client works too, such as grpc_health_probe:

grpc_health_probe -addr=localhost:8815

On Kubernetes, use a gRPC readiness probe, so a server whose Redis is down gets no traffic, and a TCP liveness probe, so it isn't restarted for that:

readinessProbe:
  grpc:
    port: 8815
livenessProbe:
  tcpSocket:
    port: 8815

The kubelet's gRPC probe doesn't do TLS. With TLS, use an exec probe that runs /redis-flight-server -healthcheck.

Metrics

The metric series below require enterprise. Community /metrics returns HTTP 204 with an empty body.

With --metrics-listen (FLIGHT_METRICS_LISTEN), the server serves Prometheus metrics at /metrics on that address. It is off by default. The endpoint is plain HTTP without authentication, so bind it to a private address; its labels never name a Redis user or a table. compose.yaml serves it on localhost:9815.

curl -s localhost:9815/metrics | grep '^redis_flight_'
Metric Type What it counts
redis_flight_calls_total{method, code} counter Calls finished, by method (DoGet, GetFlightInfo, …) and gRPC code. Health checks aren't counted
redis_flight_call_duration_seconds{method} histogram How long calls took. A DoGet or DoPut lasts until its stream ends, so it mostly measures the client
redis_flight_queries_streamed_total counter Query results streamed from Redis
redis_flight_queries_read_whole_total{reason} counter Query results read whole into memory first, by why (below)
redis_flight_query_streams_open gauge Results streaming now, each holding a pooled connection and a cursor
redis_flight_query_results_stalled_total counter Results ended by --stream-stall-timeout
redis_flight_result_memory_bytes gauge Memory held by results read whole for want of a stream, counted against --result-memory
redis_flight_result_memory_refusals_total counter Results refused with RESOURCE_EXHAUSTED for not fitting in --result-memory
redis_flight_index_cursor_limit_errors_total counter Calls that failed because an index had no cursor left (INDEX_CURSOR_LIMIT)
redis_flight_pool_connections{state} gauge Pooled ADBC connections, across Redis users: in_use (including ones being opened) and idle
redis_flight_pool_wait_seconds histogram How long requests that found --pool-max-open connections in use waited for one. Its _count is how many waited
redis_flight_session_connections gauge ADBC connections of sessions, outside the pool
redis_flight_sessions_open gauge Sessions the server keeps (with sessions on)
redis_flight_sessions_full_total counter Requests that used the pool because the server had 10,000 sessions
redis_flight_tenants_open gauge Redis users with an ADBC database, a connection pool and a Redis client open
redis_flight_dataset_rows_written_total, redis_flight_dataset_bytes_written_total counter Rows and Arrow IPC bytes of dataset uploads committed
redis_flight_dataset_rows_read_total, redis_flight_dataset_bytes_read_total counter Rows of datasets sent, and their bytes read from Redis (only the ticket's columns)
redis_flight_redis_up gauge 1 while Redis answers the health check's PING
redis_flight_build_info{version} gauge Always 1; the label is the server's version

The Prometheus client adds the Go runtime's (go_*: heap, goroutines, GC) and the process's (process_*: CPU, memory, open files).

reason is one of: - query: the driver doesn't stream the query (aggregates, joins, CTEs, …), or returned a small result whole - session: a session that has made temporary tables reads its results on its own connection - user_limit: the Redis user had --max-streams results streaming (with --auth=none, every client is one user) - table_limit: the table had --max-table-streams - index_cursors: the table's index had three quarters of its cursors open

The limits worth an alert:

rate(redis_flight_queries_read_whole_total{reason=~"user_limit|table_limit|index_cursors"}[5m]) > 0
histogram_quantile(0.99, rate(redis_flight_pool_wait_seconds_bucket[5m])) > 1
increase(redis_flight_index_cursor_limit_errors_total[5m]) > 0
increase(redis_flight_result_memory_refusals_total[5m]) > 0

An index out of cursors and a result over --result-memory are also logged at warn level.

Architecture

flowchart LR
    subgraph clients["Clients"]
        jdbc["JDBC / DBeaver<br/>Flight SQL JDBC driver"]
        adbc["ADBC Flight SQL driver<br/>Python, Go, R, …"]
        fl["pyarrow.flight, arrow-go,<br/>Arrow Java"]
    end

    subgraph server["redis-flight-server (gRPC)"]
        mw["Middleware<br/>log + metrics → recover → auth → sessions"]
        router{"router.go"}
        sql["Flight SQL handlers<br/>sqlserver.go, metadata.go"]
        ds["Dataset store<br/>datasets.go"]
        subgraph tenant["One tenant per Redis user (tenant.go)"]
            pool["Pooled ADBC connections<br/>pool.go"]
            sess["Session connections<br/>session.go"]
            gr["go-redis client"]
        end
        drv["Redis ADBC driver<br/>SQL → Redis commands"]
    end

    subgraph redis["Redis 8"]
        tables[("SQL tables<br/>a HASH per row,<br/>a RediSearch index per table")]
        dsk[("Datasets<br/>Arrow IPC in a HASH<br/>per generation")]
        res[("Large statement results<br/>kept 10 minutes")]
    end

    jdbc --> mw
    adbc --> mw
    fl --> mw
    mw --> router
    router -- "CMD descriptors,<br/>Flight SQL tickets" --> sql
    router -- "PATH descriptors, JSON tickets,<br/>drop_dataset" --> ds
    sql --> pool
    sql --> sess
    pool --> drv
    sess --> drv
    drv -- "FT.AGGREGATE, FT.SEARCH,<br/>HSET, …" --> tables
    sql --> gr
    ds --> gr
    gr -- "HSET, HMGET,<br/>EVAL" --> dsk
    gr -- "SET, GET" --> res
  • Middleware. auth is there only with --auth=redis-acl, and sessions unless --sessions=false. Health checks bypass logging, metrics, authentication and sessions; panic recovery still applies.
  • Tenants. Each Redis user gets its own ADBC database, connection pool and go-redis client, so every command runs with that user's ACLs.
  • The router tells the two paths apart by the descriptor type and the ticket's first byte (a protobuf Any, or {).
  • Code. Development lists what each file does.

How a request flows

A query (SELECT, SHOW) is planned when its FlightInfo is requested, and runs when its ticket is read:

sequenceDiagram
    participant C as Client
    participant S as Flight server
    participant D as Redis ADBC driver
    participant R as Redis
    C->>S: GetFlightInfo(CommandStatementQuery)
    S->>D: ExecuteSchema on a pooled connection
    D->>R: read the tables' metadata
    S-->>C: FlightInfo: the schema, a ticket holding the SQL
    C->>S: DoGet(ticket)
    S->>D: ExecuteQuery, on a connection it keeps until the end
    D->>R: FT.AGGREGATE … WITHCURSOR
    loop until the cursor ends
        D-->>S: an Arrow batch
        S-->>C: FlightData, about 2 MiB per message
        D->>R: FT.CURSOR READ, for the next page
    end

A plain scan like this streams. A query whose result must be built whole first (an aggregate, a join, a sort the driver does) reads every page before the first batch is sent, and gives its connection back at once.

A statement that changes something (INSERT, UPDATE, DDL, SET, …) runs when its FlightInfo is requested, so it runs exactly once. Its ticket carries the result:

sequenceDiagram
    participant C as Client
    participant S as Flight server
    participant D as Redis ADBC driver
    participant R as Redis
    C->>S: GetFlightInfo(CommandStatementQuery "INSERT … RETURNING id")
    S->>D: ExecuteQuery
    D->>R: HSET, …
    D-->>S: the result
    alt the result is up to 1 MiB
        S-->>C: FlightInfo, with the result in the ticket
    else larger
        S->>R: SET arrowflight:result:{id}, for 10 minutes
        S-->>C: FlightInfo, with the key in the ticket
    end
    C->>S: DoGet(ticket), any number of times
    S-->>C: the stored result, without running the statement again

CommandStatementUpdate (DoPut) runs the statement and returns its row count in the reply.

A dataset is written as a new generation, switched to atomically, and read back in pages:

sequenceDiagram
    participant C as Client
    participant S as Flight server
    participant R as Redis
    C->>S: DoPut(PATH ["features", "v1"]), the schema, then batches
    S->>R: INCR ds:{features/v1}:seq → generation N
    loop every 64 batches or 8 MiB
        S->>R: HSET ds:{features/v1}:gN, an IPC stream per column of each batch
    end
    S->>R: EVAL commit: point the HASH at N, unless a newer upload won
    S-->>C: PutResult {dataset, generation, rows, …}
    C->>S: GetFlightInfo(PATH ["features", "v1"])
    S->>R: HGETALL ds:{features/v1}
    S-->>C: FlightInfo: the schema, counts, one or more tickets
    C->>S: DoGet(ticket)
    S->>R: HMGET ds:{features/v1}:gN layout schema batches
    loop about 8 MiB of the ticket's columns at a time
        S->>R: HMGET ds:{features/v1}:gN, those columns of the next batches
    end
    S-->>C: FlightData: the ticket's columns and batches

Logging in and sessions. With --auth=redis-acl, the handshake checks the credentials against Redis. A client that keeps cookies gets a session, with its own connection:

sequenceDiagram
    participant C as Client
    participant S as Flight server
    participant R as Redis
    C->>S: Handshake, with Basic alice:…
    S->>R: PING, as alice
    S-->>C: a bearer token (sealed identity)
    C->>S: GetSqlInfo, with the token
    S-->>C: the result, and Set-Cookie: arrow_flight_session_id
    Note over S: no session state yet
    C->>S: SET TIME ZONE 'Asia/Tokyo', with the token and the cookie
    S->>S: start the session: an ADBC connection of its own, as alice
    S->>R: the statement, on the session's connection
    C->>S: SELECT …, with the same cookie
    S->>R: on the same connection, so the time zone still applies
    C->>S: CloseSession
    S->>S: close the connection, dropping its temporary tables

Development

Code layout:

Path What it is
cmd/redis-flight-server The command
server/router.go Sends PATH descriptors, JSON tickets and drop_dataset to the dataset store, and everything else to Flight SQL
server/sqlserver.go, server/metadata.go Flight SQL on the ADBC driver
server/datasets.go The Arrow IPC dataset store
server/tenant.go, server/pool.go Each Redis user's ADBC database, connection pool and go-redis client
server/session.go Sessions: the cookie middleware, each session's connection, and the session options
server/auth.go Redis ACL login and tokens
server/health.go, cmd/redis-flight-server/healthcheck.go The gRPC health service, and -healthcheck
server/metrics.go The Prometheus metrics

Tests:

make test
make test-integration
make test-cluster

make test runs unit tests without Redis. All integration testing requires a disposable Redis Enterprise cluster database with Search, verified TLS, and username/password authentication. Export its authenticated rediss:// URI as REDIS_URI, then run make test-integration through the qualified enterprise source composition. make test-cluster is an alias for the same suite. Neither target starts local Redis. Tests create and remove it_flight_* tables, keys and ACL users.

See testing and CI setup for credentials, CA trust, client checks and the private enterprise workflow recommendation. The community build supports the minimum TLS/ACL security baseline. Public CI exercises TLS and snapshot ownership with test doubles; it does not provide real Redis Enterprise or Airbyte integration qualification.

Releases: push a tag vX.Y.Z (or vX.Y.Z-rc1 for a prerelease). The release workflow (.github/workflows/release.yml) runs the tests on it, builds the binaries (make dist, which works locally too) and the image, runs each binary on its own platform, then pushes the image and attaches the binaries to the tag's GitHub release, making the release if there is none. Run it by hand (workflow_dispatch) to rebuild an existing tag. A pull request that changes it, the Dockerfile or the Makefile builds and checks everything without publishing.

The driver dependency is pinned in go.mod with a replace to the v0.0.24 release of github.com/redis/redis-adbc-driver/go (commit b052eedad9f2, checked 2026-11-11 and recorded as a Go pseudo-version). Always select the latest stable driver release when updating this dependency and verify its Go submodule revision upstream. Version 0.0.24 requires Go 1.27.3, uses SQL storage protocol 17 and rejects legacy SQL storage; it does not automatically migrate existing data. Validate existing-storage migration separately before deployment. See the update validation and the shared snapshot contract for fresh-storage requirements and pending transport qualification. To build against a local checkout instead:

go mod edit -replace github.com/adbc-drivers/redis/go=../redis-adbc-driver/go

The endpoint acceptance defaults define the 10,000-row local workload, reproducible measurements and proposed deployment objectives. See the document for measured results and remaining validation.

Both Flight build profiles use the full Redis ADBC driver. The paid-feature packaging rule applies to Flight-owned features only; see build profiles.