Performance Tuning

The replicate command reads from a source Redis database and write to a target Redis database.

To optimize throughput it is necessary to understand the two main possible scenarios:

Slow Producer: In this scenario the reader does not read from source as fast as the writer can write to the target. This means the writer is starved and we should look into ways to speed up the reader.
Slow Consumer: In this scenario the writer can not keep up with the reader and we should look into optimizing writes.

There are two ways to identify which scenario we fall into:

No-op writer: With the --dry-run option the replication process will use a no-op writer instead of a Redis writer. If throughput with dry-run is similar to throughput without then the writer is not the bottleneck. Follow steps below to improve reader throughput.

Reader

To improve reader performance tweak the options below until you reach optimal throughput.

Scan Performance Options

--scan-count: Number of keys each SCAN call should fetch (default: 100). Higher values reduce round-trips but use more memory. Recommended: 1000-5000 for large datasets.
--read-batch: Number of values each reader thread should read in a single pipelined call (default: 50). Controls Redis pipeline depth. Recommended: 100-500 depending on value sizes.
--read-threads: Number of threads reading key values concurrently (default: 1). More threads increase parallelism. Recommended: 4-16 for high-throughput scenarios.
--read-queue: Capacity of the async queue for key prefetching (default: batch * threads * 25). Larger queues improve throughput but use more memory. Recommended: 25000-50000 for high-throughput.
--source-pool: Number of Redis connections to the source database (default: 8). Should match or exceed read thread count for optimal performance.

High-Throughput Example

For maximum scan performance with large datasets:

riotx replicate redis://source:6379 redis://target:6379 \
  --scan-count 2000 \
  --read-batch 200 \
  --read-threads 8 \
  --read-queue 50000 \
  --source-pool 16 \
  --dry-run

The async queue architecture overlaps SCAN operations with value reading for significant performance improvements. Start with default values and increase gradually while monitoring memory usage.

Writer

To improve writer performance you can tweak the following options:

--batch: Number of items written in a single network round-trip to the Redis server (i.e. number of commands in the pipeline).
--threads: How many write operations can be performed concurrently (default: 1).
--target-pool: Number of Redis connections to the target database (default: 8). Keep in sync with the number of threads to have a dedicated connection per thread.