Testing and agent evaluation¶
Status: existing checks described from source; proposed agent evals are not installed. Evidence: Makefile, test workflow, and test files below.
Local verification¶
For a provisioned local TLS/ACL cluster and containerized enterprise runner, see the Redis Enterprise environment.
Use Go as pinned by go.mod. Integration tests create and delete tables, keys, and ACL users named it_flight_*; examples also mutate data. Use a disposable Redis Enterprise cluster database with Search, verified TLS, and a nonempty username and password. Core TLS/ACL authentication is available in community; paid-feature tests still need the qualified enterprise composition.
| Check | Command | Prerequisites and success signal |
|---|---|---|
| Unit tests | env -u REDIS_URI make test |
Go and dependencies available; exit 0. Redis-dependent tests intentionally skip |
| Formatting | gofmt -l . |
Empty output |
| Static analysis | go vet ./... |
Exit 0 |
| Static binary | make build |
Exit 0; output bin/redis-flight-server |
| Enterprise integration/race | make test-integration |
Qualified enterprise source composition; exported REDIS_URI=rediss://USERNAME:PASSWORD@HOST:PORT/0; exit 0 with backend tests executed |
| Compatibility alias | make test-cluster |
Same secured Enterprise endpoint and suite as make test-integration; no local Redis startup |
| Python client | make example |
Redis and server running; uv installs script dependencies; exit 0 |
| DBeaver manual checks | Setup, fixture and expected results | Running disposable endpoint, Generic driver and JDBC 19.0.0; record connection, navigator, columns, Data tab and queries separately |
| JDBC client | make jdbc-smoke |
Redis and server running; Java and network for initial driver download; exit 0 |
| Dataframe / DuckDB clients | make clients-smoke |
Disposable server and uv; pinned pandas, Polars, DuckDB and Flight dependencies; asserts results and reports pandas's ADBC params limitation |
| R Flight SQL client | make r-smoke FLIGHT_SQL_DRIVER=/path/to/library |
R, adbcdrivermanager, nanoarrow and an OS/architecture-matched Flight SQL shared library; exit 0 |
| Dataset benchmarks | go test ./server -run '^$' -bench Dataset -benchtime 10x |
Same enterprise composition and exported secured REDIS_URI; inspect reported timings/allocations, no pass/fail performance threshold exists |
make test always unsets the backend variables so it runs unit checks only.
make test-integration requires REDIS_URI and never starts Compose. Direct
go test skips backend tests when the URI is absent; set
REDIS_INTEGRATION_REQUIRED=1 in integration automation to make absence fatal.
The shared test helper rejects plaintext URIs, missing usernames/passwords,
and invalid TLS connection options without printing the URI. Provisioning must
establish that the endpoint belongs to Redis Enterprise: an Enterprise proxy
does not need to expose the OSS Redis Cluster API. A successful PING alone
cannot establish edition, shard count, or active 4x QPF qualification.
Export the URI through your secret manager or a hidden shell prompt, with URI
percent-encoding for special characters in credentials. Do not put credentials
in command arguments or checked-in files. The cluster's certificate must chain
to a trusted CA in the test host and server container; install a private CA in
those trust stores when needed. Do not disable certificate verification.
The test identity needs the commands exercised by the suite, including Search,
SQL data mutation and key cleanup. Enterprise ACL fixtures use the Software
management API with separate admin credentials; the full repaired suite and
original compatibility failures are recorded in the
local validation results.
Use a dedicated disposable database. Native managed tables share one catalog,
so unique table names alone do not isolate concurrent publications. The supported
make test-integration target serializes Go packages (-p 1) and runs through
scripts/run-integration.py, which locks a database scope for the command's
entire lifetime. make test-shared-snapshot uses the same runner.
Set the same nonsecret FLIGHT_INTEGRATION_SCOPE for every suite targeting one
database, especially when Redis and Flight use different hostnames/endpoints.
Without an explicit scope, the runner derives one from REDIS_URI's host/port/DB
and excludes credentials. Flight-only runs must supply an explicit scope.
Cooperating processes on one host must share FLIGHT_INTEGRATION_LOCK_DIR
(default: a directory under the host temporary directory). For separate containers,
wrap the docker run commands on the same host or mount the same lock directory;
separate hosts must use separate disposable databases. Direct go test/pytest
commands bypass this suite lock and must not publish concurrently into that DB.
No SQL write is automatically replayed by this isolation mechanism.
export FLIGHT_INTEGRATION_SCOPE=disposable-flight-db
make test-integration
make test-shared-snapshot
For a custom command, use python3 scripts/run-integration.py -- COMMAND ....
The runner requires POSIX file locking (Linux/macOS).
All Python, JDBC, R, BI-tool and benchmark validation must use a Flight server
built from the qualified enterprise composition and connected to this same
secured Redis Enterprise backend. Start that server using the composition's
build runner and exported REDIS_URI, then wait for a real SQL round trip with
scripts/wait-flight.py. Local community Compose is a development convenience;
it is not an accepted integration-test environment. make jdbc-smoke defaults
to JDBC driver 19.0.0 and adds memory options for the selected Java version;
enterprise CI should run it for Java 17, 21 and 25. No make lint target exists.
Recommended private CI setup¶
Run enterprise integration in a private workflow alongside the separately
supplied source composition. Check out the exact public commit under test,
qualify its full-module fingerprint, and execute make test-integration through
the composition runner. Ensure its Go overlay also reaches command subprocesses.
Then build the enterprise server from that same composition and run Python,
JDBC, and other client checks against it. Preserve read/write assertions and
record the public commit, composition revision and actual results.
Use a protected GitHub Environment named redis-enterprise-integration. Store
REDIS_URI as an environment secret containing the authenticated rediss://
endpoint. Grant only trusted workflow runs access to both the private composition
and this environment. Do not expose them to untrusted fork PRs or use
pull_request_target to execute untrusted changes with credentials. Use a runner
with network access to the cluster and the appropriate CA trust store.
Public CI runs formatting, vet, community unit tests and a static build. It no longer runs plaintext standalone/OSS Cluster or client integration tests. The public release workflow now requires the private qualification workflow and a matching receipt before building/publishing binaries or images. The private manifest binds the exact reviewed public commit before executing its source. The scoped dispatch credential is stored in GitHub Secrets, with organization approval pending. The private workflow and its qualification manifest live in the coordinated private change. Public unit CI and offline artifact checks alone are not enterprise integration qualification.
Behavior-to-test map¶
| Contract | Existing source evidence |
|---|---|
| Writes occur once when returned tickets are read repeatedly | TestIntegrationWritesRunOnce in integration_test.go |
| Prepared parameters/types and ingestion | TestIntegrationPrepared, TestIntegrationPreparedTypes, TestIntegrationIngest in integration_test.go |
| Temporary state does not leak through pooling | TestIntegrationTempObjectsDontLeak in integration_test.go |
| Atomic upload outcome, ordering, persistent generation sequence | TestIntegrationUploadCommitOutcome, TestIntegrationUploadOrder, TestIntegrationDatasetTTLKeepsSeq in integration_test.go |
| Projection, batch ranges, layouts, paging | TestIntegrationDatasetReads, TestIntegrationDatasetLayouts, TestIntegrationDatasetReadPages in integration_test.go |
| Redis ACL and changed credentials | TestIntegrationRedisACL, TestIntegrationStaleCredentials in integration_test.go |
| Sessions, ownership, settings | TestIntegrationSessions, TestIntegrationSessionOwner, TestIntegrationSessionSettings in session_integration_test.go |
| Definite publication refusals preserve SQL session state without replay | TestIntegrationSessionPublicationRefusal in publication_refusal_integration_test.go; source contract and before/after evidence |
| Streaming, fallback, cleanup | TestIntegrationStreamResults in stream_integration_test.go; budget/token/panic tests in unit_test.go |
| Health auth exception and draining | TestHealthCheckSkipsAuthAndSessions, TestHealthWatchDoesNotHoldShutdown in health_test.go |
| Shutdown and command behavior | TestShutdownDrainsThenCancels in unit_test.go; TestSIGTERM, TestIntegrationSIGTERM in main_test.go |
| Metrics | metrics_test.go |
This map identifies coverage to inspect; it does not assert all branches are covered or that tests passed in this session.
CI and completion evidence¶
Integration validation must cover both reads and writes unless the requester explicitly scopes an integration to reads. Tableau acceptance is read-only: validate discovery and application read queries; write support is not required. Record the actual read coverage and untested features. For integrations with write coverage, verify writes by reading back the resulting values, using the application's supported write path. Record unsupported write paths and application/environment blockers separately; driver-only success does not establish application compatibility. See the JDBC/Metabase read-write validation.
The public test workflow checks formatting/vet, community unit tests, and a statically linked binary. Enterprise integration and client evidence must come from the private qualification workflow described above. Workflow configuration is not historical evidence of green runs.
For a code change, run the relevant local checks and retain CI links/results in its PR. For a regression, reproduce the failure before fixing it where practical; do not weaken existing assertions to manufacture a pass. Explain intentional test contract changes. For documentation-only changes, check file links, source references, and whitespace; no Redis startup is necessary.
Proposed evaluation cases for agent instructions¶
These are candidate tasks for a future runner, not executable evals or measured scores. Run in isolated disposable checkouts with synthetic data and record the instruction/model version and source SHA.
| Task prompt | Expected evidence |
|---|---|
| Explain why reading a write ticket twice must not insert twice | Identifies execution timing and the write-once regression test |
| Plan a dataset storage change | Includes LIST/HASH compatibility, cluster hash tags, ordering, seq persistence, and mixed-version rollout |
Diagnose RESOURCE_EXHAUSTED |
Distinguishes pool/stream pressure, cursor limits, and fallback memory; does not claim the budget caps all memory |
| Review a session routing change | Checks owner identity, process-local state, setting restoration, temporary objects, and client cookies |
| Add a client compatibility statement | Requests actual client evidence; preserves untested BI-tool caveats |
| Propose a metrics or auth change | Preserves health's intentional exception, avoids credentials/high-cardinality labels, identifies public metrics exposure |
A future maintainer must choose scoring, budget, safe tool access, representative task fixtures, and gating thresholds. Do not enable paid model runs or assume a service integration from this document.