This version is still in development and is not considered stable yet. For the latest stable version, please use RIOT-X 1.15!

Sync

The sync command synchronizes data from a source Redis database to a target Redis database. It does the same job as the replicate command with a reactive engine (riot2-sync) built to replicate big keys: every key is copied in chunks with type-specific commands, so client memory stays bounded whatever the key size. Large keys are scheduled fairly against small ones, Redis Streams are replicated entry by entry, and every condition that prevents convergence is reported. The riot sync command of the riot distribution is the same command.

Usage

riotx sync [OPTIONS] SOURCE TARGET

where SOURCE and TARGET are Redis URIs:

riotx sync redis://source:6379 redis://target:6379

For the full list of options, run riotx sync --help.

What sync guarantees

Sync converges the target toward the current source state once source writes quiesce, provided the keyspace notifications and reads it needs are available. It is not a transactional snapshot, not an exactly-once change log, and it cannot recover data removed from the source before it was read.

The final summary says Sync completed WITH ERRORS and the exit code is 1 whenever the run cannot claim convergence:

  • any read, write or key error (for example a target-key collision or an unresolved transformed delete),

  • a notification for a new key rejected because --pending-keys was full,

  • a lost keyspace-notification connection (a gap: changes made meanwhile may be missing); on a cluster source this includes a primary that crashes during a live run, because notifications are subscribed on the primaries present at start and the promoted replica is not subscribed (a planned CLUSTER FAILOVER loses nothing),

  • a capability the source or target lacks (for example a module command),

  • a stream that could not be kept in sync (for example after three rebuilds in one run).

A write whose outcome is unknown (timeout, dropped connection) stops the run immediately with a nonzero exit code: RIOT never replays an ambiguous APPEND, RPUSH or source-key removal. sync disables Lettuce auto-reconnect on the target for this reason. With --retry, only operations that are safe to repeat are retried first (see Retries and failed keys).

Supported targets

Like replicate, sync refuses to write to a target that is not a supported Redis flavor (for example Valkey or ElastiCache) unless a valid unlock token is given with --unlock-token or the RIOT_UNLOCK_TOKEN environment variable. The check reads INFO on the target once, before anything is written, and stops the run with exit code 1 and a message naming the detected flavor. A token that is invalid, expired or lacks the riotx:replicate:unsupported-redis scope is refused the same way. --dry-run writes nothing and skips the check. Sources are not checked.

Replication modes

--mode selects the inputs:

SCAN (default)

One finite SCAN over every primary. Each stream is copied up to its last entry at the moment it is admitted; appends made later do not extend the copy.

LIVE

Keyspace notifications are subscribed first (on every primary of a cluster), then the SCAN runs while notifications are processed. After the scan, ordinary keys keep following notifications and streams keep tailing.

LIVEONLY

Notifications only: a key is copied when it changes. An existing key that never changes (including a stream nobody appends to) is never copied. A newly discovered stream is copied from its oldest retained entry.

Live modes need notify-keyspace-events to contain K and either A or all of g$lshzxet on the source. RIOT reads this setting and refuses to start a live run when it is insufficient; it never changes server configuration. Where the setting cannot be read (some managed services), the run proceeds with a warning.

--idle-timeout ends a live run after the given quiet period. It starts counting once the SCAN has finished (immediately in LIVEONLY). Pending, running or reconciliation work prevents it from firing; caught-up streams waiting for their next poll do not, and empty stream polls are not activity. Without --idle-timeout a live run continues until interrupted; an interrupted run does not print a successful summary.

Write modes

Mirror (default, OVERWRITE)

RIOT owns the target keys it writes: it may remove target-only members and entries, replace a key of another type, and rebuild a stream. Independently written targets are not supported. A stream rebuild destroys consumer groups that exist only on the target.

--merge

Upsert into existing target values and keep target-only hash fields, set and sorted-set members, time-series samples and stream entries; only a type mismatch replaces a key. Lists and strings have no merge and are replaced: appending a list onto its existing copy would duplicate it on every follow-up copy. Merge never deletes stream entries. For a target stream that already has history, RIOT verifies that every source entry up to the target’s last entry exists on the target with identical fields, then appends newer entries; a missing or different entry fails that stream, because XADD cannot insert below existing history. Deleting a whole source key still deletes the target key in live modes.

--no-replace / -n

Skip a key that already exists on the target when the key is admitted.

How keys are copied

Every key is copied in chunks with type-specific commands; there is no DUMP/RESTORE mode, so replication does not depend on the source and target sharing a serialization format.

  • Metadata (TYPE, PTTL and, with --mem-limit, MEMORY USAGE) is read in one read-only Lua script per key; sources that refuse scripts are read with individual commands instead.

  • Values are read and written in pages: GETRANGE windows of 256 KiB for strings (written with SET then APPEND), HSCAN/SSCAN/ZSCAN cursors (HSET/SADD/ZADD), LRANGE windows (RPUSH) and TS.RANGE windows (TS.ADD), with --chunk elements per page (default 1000). JSON documents are read and written whole.

  • Each page is read only after the previous one was acknowledged, and a large key yields its execution slot to other keys between pages.

  • Streams always use the dedicated entry path, whatever their size (see below).

  • Expirations are captured as a deadline when the key is first read and applied as a relative TTL on the target, so the source and target clocks need not agree. A key that expires during its copy is removed from a mirror target.

  • --no-ttl leaves expirations out: no PEXPIRE and no PERSIST (with --merge a target key keeps its own TTL), and stream polls ignore TTL changes. A key that expires during its copy is then kept on the target, persistent, with whatever was copied before the source key disappeared (possibly a partial value). In live modes the source key’s expiration still arrives as an expired notification, which deletes the target key like any other deletion.

  • --ttl-proc computes the target TTL with a JavaScript expression over key, type, ttl (the source TTL in milliseconds, -1 for none) and mem, for example "ttl > 0 ? ttl * 2 : -1". The result is a TTL in milliseconds relative to the moment the source TTL was read: negative means no expiration, 0 means already expired (the key is treated as expired during its copy). Streams re-evaluate it on every poll. An expression that fails or returns something other than a finite number is a key error. --ttl-proc cannot be combined with --no-ttl.

  • Changes to a key while it is being copied are folded into one follow-up copy that re-reads the key’s current state, so a delete followed by a recreate is never applied out of order.

Page sizes bound element counts, not bytes: a single huge element is still read in one piece.

Streams

Streams are replicated entry by entry with their original IDs, field order and repeated field names. An entry counts as copied only once XADD acknowledged it; an ID the target rejects as not greater than its top entry is read back and accepted only if its fields match exactly.

In live modes a caught-up stream is polled every --stream-poll-interval (default 1s) and immediately on an XADD notification. Each poll also detects deletion, type changes, TTL changes and a lower last-generated-id.

In mirror mode retention is reconciled:

  • entries trimmed from the head of the source (XTRIM, XADD MAXLEN/MINID) are removed from the target with exact XTRIM MINID (or bounded XDEL batches on servers without it);

  • arbitrary deletions (XDEL holes) are found by a reconciliation pass that walks source and target pages side by side, holding one page per side, on XTRIM/XDEL notifications and every --stream-reconcile-interval (default 5 minutes);

  • a mismatching or missing historical entry, a recreated stream, or a lower last-generated-id triggers a rebuild: the target stream is cleared and recopied, at most three times per stream per run;

  • an emptied source stream leaves an empty target stream (type preserved), created with the same last-generated-id.

Limitations:

  • Consumer groups, consumers and pending entries are not replicated.

  • Entries trimmed from the source before RIOT read them cannot be recovered.

  • A stream deleted and recreated while no notification is received is only detected by reconciliation.

  • --no-stream-id copies streams with target-generated IDs, in every mode. The stream cursor keeps the last source ID copied, so a live tail appends only newer source entries. Without matching IDs the target cannot be reconciled: trims and XDEL on the source are not propagated, and a mismatch is never detected. A mirror rebuild (recreated stream, lower last-generated-id, restart of RIOT) clears the target stream and copies it again; with --merge, which never clears, a restarted run or a retried transfer appends its entries again, so duplicates are possible.

Limits and bounds

--pending-keys <n> (default 1,000,000)

Admitted keys awaiting or undergoing transfer. SCAN slows down when it is reached; a notification for a new key is rejected (counted and logged at most once per minute), which makes the run unsuccessful. Notifications for keys already admitted are always folded in. --read-queue is a deprecated alias.

--in-flight <n> (default --readers × --batch)

Transfer steps executing at once, each holding at most one page.

--stream-watch-limit <n> (default 100,000)

Streams tailed at once in live modes. Exceeding it stops the run with a diagnostic instead of silently dropping a stream.

--limit <n>

Stop admitting after this many distinct keys (deletes do not count), finish the admitted transfers, then end. Keys skipped by filters or --no-replace count toward the limit and are reported separately. Stream tails stop at the limit.

--dry-run

Read metadata, apply filters and log what would be copied or deleted, without writing to the target.

Throttling and replica acknowledgment

--rate <n>

Starts at most n transfer steps per second, evenly spaced (no bursts; idle time is not saved up). A step reads and writes at most one page: one key for small keys, one page of --chunk elements (or 256 KiB of a string) for big ones, one stream poll or one reconciliation page. Stream polls therefore count against the rate. The wait is a timer: no thread sleeps, and other steps keep running up to --in-flight.

--wait-replicas <n>, --wait-timeout <duration> (default 1s)

After the writes of a step were acknowledged, and before the step completes, RIOT sends WAIT n timeout on the target connection those writes used (on a cluster, to the primary owning the key’s slot). WAIT runs once per step that wrote, and before a key counts as synced or deleted and before --remove-source-keys removes it from the source. Fewer acknowledging replicas than n is a write error for that key: the primary keeps what it applied, the run is unsuccessful, and the source key is not removed. --wait-timeout must be shorter than the stall timeout.

Retries and failed keys

--retry <n> (default 0, -1 for no limit) retries a failed operation, after a --backoff delay (EXPONENTIAL by default, starting at --backoff-delay 100ms, multiplied by --backoff-x 2 up to --backoff-max 30s; or FIXED, or NONE), only when repeating it leaves the same target state:

  • source reads (metadata, pages, stream ranges);

  • target writes SET of a string’s first page, HSET, SADD, ZADD, JSON.SET, TS.ADD … ON_DUPLICATE LAST, UNLINK, PEXPIRE (with the remaining TTL recomputed per attempt), PERSIST, XTRIM MINID, XDEL and WAIT.

A rejection, a timeout and a dropped connection are retried alike; a retry goes to the same connection, so a timed-out attempt is applied before it. APPEND, RPUSH, XADD (stream pages and empty-stream creation) and the removal of source keys are never retried: a rejection fails the key and an ambiguous outcome stops the run, as without --retry. Key errors (collisions, failed transformations) and missing capabilities are never retried. An operation that still fails after its retries fails as it would have without them.

--skip <n> (default -1, no limit) stops the run once more than n keys have failed; the failures counted are the read, write and key errors listed above. Every failed key makes the run unsuccessful whatever --skip says: skipping only decides whether the run continues.

--multi-exec is rejected: keys are copied in chunks, so a key larger than one page cannot be written in a single MULTI/EXEC, and making each key atomic would need temporary keys and swaps on the target.

The engine’s memory is roughly: registry entries (a few hundred bytes per pending key) + one page per executing step + one page per side during stream reconciliation + Lettuce client buffers. These defaults do not imply a particular heap size; lower --pending-keys, --in-flight and --chunk for small heaps.

Option reference

Every option maps to one engine setting.

Option Default Engine behavior

SOURCE, TARGET and --source-/--target-

Connections; the source database is taken from the URI (0 for clusters).

--mode

SCAN

SCAN, LIVE or LIVEONLY (see above).

--key-pattern

*

Key glob: SCAN MATCH for scanned keys, and a filter on notified keys in live modes.

--key-type

SCAN TYPE filter; in live modes notified keys of other types are skipped too.

--scan-count

1000

SCAN COUNT hint and SCAN request batch.

--key-include, --key-exclude

Globs with SCAN MATCH semantics, matched on raw key bytes before a key takes any state.

--key-slot

Cluster slot ranges, applied with the globs.

--mem-limit

Skip keys whose MEMORY USAGE is larger; a key whose usage cannot be read is an error, not a skip. Not needed for memory safety, since every key is copied in chunks.

--key-proc

JavaScript key transformation ('prefix:' + key; replicate’s SpEL templates are not supported); deletes resolve their target through a bounded map of earlier copies.

--ttl-proc

JavaScript target TTL in milliseconds, negative for none (see above); not with --no-ttl.

--no-ttl (RIOT_NO_TTL)

off

No PEXPIRE/PERSIST; keys expiring mid-copy are kept.

--unlock-token (RIOT_UNLOCK_TOKEN)

JWT allowing writes to unsupported target flavors.

--struct

Accepted for compatibility and ignored: every key is copied in chunks.

--chunk

1000

Elements per page (strings use 256 KiB windows).

--merge

off

MERGE write mode instead of mirror.

--no-replace, -n

off

Skip keys that exist on the target.

--no-stream-id

off

Target-generated stream IDs; live tails follow the last source ID (see Streams).

--stream-prune

off

Finite runs drop empty source streams instead of creating them; rejected with --merge; redundant in live mirror mode.

--remove-source-keys

off

Remove each source key after it was copied; --mode scan only, not with --key-proc.

--readers, --writers

8, 8

Source and target connections.

--batch

1000

With --readers, the default --in-flight.

--in-flight

readers × batch

Executing steps.

--pending-keys (--read-queue)

1,000,000

Admission bound, see above.

--stream-watch-limit

100,000

Tailed streams.

--stream-poll-interval

1s

Caught-up stream polls.

--stream-reconcile-interval

5m

Periodic stream reconciliation.

--idle-timeout

Live idle timeout.

--limit

Admission limit.

--dry-run

off

No target writes, no target check, no comparison.

--rate (RIOT_RATE)

Transfer steps started per second, e.g. 100 or 5k.

--wait-replicas, --wait-timeout

0, 1s

WAIT after each writing step; too few replicas is a write error.

--retry (RIOT_RETRY)

0

Retries of reads and idempotent writes; -1 for no limit.

--backoff, --backoff-delay, --backoff-max, --backoff-x

EXPONENTIAL, 100ms, 30s, 2

Delay between retries.

--skip (RIOT_SKIP)

-1

Failed keys tolerated before the run stops; -1 for no limit.

--multi-exec

Rejected (see above).

--log-keys

off

Log each key as it is read.

--diff (RIOT_DIFF), --show-diffs, --ttl-tolerance

QUICK

Post-sync verification of the keys the sync selected (pattern, type, globs and slots), each under the same name on the target. Skipped, with a warning, when the target differs from the source on purpose: --limit, --dry-run, --key-proc, --no-ttl, --ttl-proc, --mem-limit, --no-replace, and --merge with FULL.

--no-progress

off

Disable the progress bar.

The per-operation stall timeout is twice the longer of the source and target command timeouts (2 minutes with Lettuce defaults).

Exit codes

  • 0: the sync completed and verification (--diff, QUICK by default) matched.

  • 1: the sync failed or completed with errors, rejected notifications, notification gaps or failed streams (including too few replicas for --wait-replicas and runs stopped by --skip); also for an unsupported option combination and for an unsupported target without a valid unlock token.

  • 2: the sync succeeded but verification found differences. Since --diff defaults to QUICK, this can now happen without --diff on the command line; pass --diff NONE to skip verification.

Behavior changes from earlier sync2 versions

  • --remove-source-keys is rejected with --mode live and --mode liveonly, and with --key-proc, instead of deleting keys of an actively changing source or being silently ignored.

  • --no-stream-id works in every mode: live tails track the last source ID copied (it was rejected with --mode live and --mode liveonly in earlier versions of this engine).

  • --diff defaults to QUICK, as in replicate, so a run verifies keys and TTLs and can exit with code 2. It compares only the keys the sync selected, and it is skipped, with a warning, when options make the target differ on purpose (--limit, --dry-run, --mode LIVEONLY, --key-proc, --no-ttl, --ttl-proc, --mem-limit, --no-replace, --merge with FULL).

  • Unsupported target flavors (Valkey, ElastiCache, …​) are refused before any write unless --unlock-token/RIOT_UNLOCK_TOKEN holds a valid token, as in replicate.

  • New replicate options: --no-ttl, --ttl-proc (JavaScript, like --key-proc), --unlock-token, --wait-replicas/--wait-timeout, --rate, --retry/--backoff* (idempotent operations only, default no retries) and --skip (default no limit).

  • --multi-exec is rejected with a reason.

  • --key-proc and --ttl-proc are JavaScript expressions, not replicate’s SpEL templates; `--var, --date-format and --number-format do not exist.

  • Live runs require keyspace notifications to be enabled on the source when RIOT can read the setting.

  • --read-queue became --pending-keys with a default of 1,000,000; rejected notifications make the run unsuccessful.

  • --limit counts distinct keys admitted (including skipped ones), not successful writes.

  • An emptied stream stays an empty stream on the target instead of being deleted.

  • In mirror mode a target key of a different type is replaced instead of failing the write.

  • With --merge an existing target list is replaced instead of having the source list appended to it again (which grew it on every change in live modes).

  • --key-include/--key-exclude follow SCAN MATCH exactly; an unterminated [ no longer matches literally.

  • DUMP/RESTORE is gone: every key is copied in chunks with type-specific commands, and --struct is accepted but has no effect.

  • --merge, --no-stream-id and --stream-prune no longer require --struct.