Sync2

The sync2 command synchronizes data from a source Redis database to a target Redis database. It does the same job as the replicate command but uses a new reactive engine and should be faster in most cases.

Usage

The basic usage is:

riotx sync2 [OPTIONS] SOURCE TARGET

where SOURCE and TARGET are Redis URIs:

riotx sync2 redis://source:6379 redis://target:6379

For the full list of options, run:

riotx sync2 --help

Replication Types

By default sync2 uses dump & restore, transferring each key’s serialized representation.

With --struct, keys are transferred using type-specific commands instead. Struct mode streams large keys in chunks (HSCAN, SSCAN, ZSCAN, LRANGE windows, GETRANGE windows, XRANGE pagination, TS.RANGE pagination), so memory usage is bounded regardless of key size. The --chunk option controls how many elements are read per chunk (e.g. 100, 10k). See Big Keys for details.

riotx sync2 redis://source:6379 redis://target:6379 --struct --chunk 10k

Struct-mode-only options:

  • --merge: merge collection data structures into existing target keys instead of overwriting them.

  • --no-stream-id: do not preserve stream message IDs (for CRDB targets).

  • --stream-prune: drop empty streams.

Replication Modes

The --mode option selects how source keys are identified:

  • SCAN (default): snapshot replication using SCAN.

  • LIVE: snapshot plus continuous replication using keyspace notifications.

  • LIVEONLY: keyspace notifications only, no initial scan.

In live modes, --idle-timeout stops replication after the given idle time.

Key Filtering

  • --key-pattern <glob>: pattern of keys to scan (default: *).

  • --key-type <type>: only sync keys of the given type (string, hash, list, set, zset, stream, ReJSON-RL, TSDB-TYPE).

  • --key-include <glob> / --key-exclude <glob>: include or exclude keys matching glob patterns (repeatable).

  • --key-slot <range>: restrict to a cluster slot range, e.g. 0-8000 (repeatable).

  • --mem-limit <size>: skip keys whose MEMORY USAGE exceeds this size (e.g. 10mb, 1gb). Optional in struct mode, which streams large keys in chunks.

  • --limit <n>: stop after syncing this many keys.

Key Transformation

The --key-proc option takes a JavaScript expression that transforms key names. Available variables: key (key name), type (Redis type), ttl (TTL in milliseconds, -1 if no expiry), and mem (memory usage in bytes from MEMORY USAGE, 0 if not fetched).

riotx sync2 redis://source:6379 redis://target:6379 --key-proc "'prefix:' + key"

Example expressions:

  • 'prefix:' + key: prepend a prefix

  • key.toUpperCase(): convert to uppercase

  • key.replace(/\d+/g, ''): regex replacement

  • type + ':' + key: prefix with the Redis type

  • ttl > 0 ? 'exp:' + key : key: conditional based on TTL

Verification

The --compare option verifies the sync after it completes:

  • NONE (default): no verification.

  • QUICK: compare key existence, type, and TTL.

  • FULL: compare values as well. In struct mode, full comparison uses the same chunked reads, so it also works on keys larger than the JVM heap.

Use --show-diffs to log key differences and --ttl-tolerance to adjust the allowed TTL difference (default: 100ms). The exit code is 2 when verification finds differences.

Performance Tuning

  • --readers <n> / --writers <n>: number of source and target connections operating concurrently.

  • --batch <n>: max number of keys in flight per connection.

  • --scan-count <n>: SCAN COUNT hint.

  • --read-queue <n>: max number of keys in the read queue.

Other Options

  • --dry-run: log keys that would be synced without writing them.

  • --no-replace / -n: do not overwrite existing keys on the target.

  • --remove-source-keys: delete keys from the source after successful replication (for migration).

  • --log-keys: log keys being read at INFO level.

  • --no-progress: disable the progress bar.