Replication Mode

Replication starts with identifying keys to be replicated from the source Redis database. The --mode option allows you to specify how RIOT-X identifies keys to be replicated:

  • iterate over keys with a key scan (--mode scan)

  • received by a keyspace notification subscriber (--mode liveonly)

  • or both (--mode live)

Scan

This key reader scans for keys using the Redis SCAN command:

SCAN cursor [MATCH pattern] [COUNT count] [TYPE type]
MATCH pattern

configured with the --key-pattern option

TYPE type

configured with the --key-type option

COUNT count

configured with the --scan-count option

INFO: In cluster mode keys are scanned in parallel across cluster nodes.

Async Queue Architecture

The scan reader uses an optimized async queue architecture for high-throughput scenarios:

  • Keys are scanned in a background thread and queued for processing

  • Multiple reader threads process keys in parallel and queue results

  • The main thread consumes processed values from the result queue

This architecture provides significant performance improvements by overlapping I/O operations.

The status bar shows progress with a percentage of keys that have been replicated. The total number of keys is estimated when the replication process starts and it can change by the time it is finished, for example if keys are deleted or added during replication.

Scan replication example
riotx replicate redis://source redis://target
High-throughput scan replication example
riotx replicate redis://source:6379 redis://target:6379 \
  --scan-count 2000 --read-batch 200 --read-threads 8 \
  --read-queue 50000 --source-pool 16

Live

The key notification reader listens for key changes using keyspace notifications.

Make sure the source database has keyspace notifications enabled using:

  • redis.conf: notify-keyspace-events = KEA

  • CONFIG SET notify-keyspace-events KEA

For more details see Redis Keyspace Notifications.

Live replication example
riotx replicate --mode live redis://source redis://target

The live replication mechanism does not guarantee data consistency. Redis sends keyspace notifications over pub/sub which does not provide guaranteed delivery. It is possible that RIOT-X can miss some notifications in case of network failures for example.

Also, depending on the type, size, and rate of change of data structures on the source it is possible that RIOT-X cannot keep up with the change stream. For example if a big set is repeatedly updated, RIOT-X will need to read the whole set on each update and transfer it over to the target database.

For those potentially problematic migrations it is recommend to perform some preliminary sizing using Redis statistics and bigkeys/memkeys in tandem with --mem-limit. If you need assistance please contact your Redis account team.