Introduction

RIOT is a command-line utility to get data in and out of Redis. It supports any Redis-compatible database like Redis Cloud, Redis Community Edition, Redis Software.

RIOT includes the following features:

RIOT is supported by Redis, Inc. on a good faith effort basis. To report bugs, request features, or receive assistance, please file an issue or contact your Redis account team.

RIOT-X

RIOT-X is an extension to RIOT which provides the following additional features for Redis Cloud and Redis Software:

  • Observability

  • Memcached Replication

  • Stream Import/Export: Redis <→ Redis Stream

Full documentation for RIOT-X is available here: redis-field-engineering.github.io/riotx

riotx dashboard replication
riotx dashboard jvm

Install

RIOT can be installed on Linux, macOS, and Windows platforms and can be used as a standalone tool that connects remotely to a Redis database. It is not required to run locally on a Redis server.

Homebrew (macOS & Linux)

brew install redis/tap/riot

Scoop (Windows)

scoop bucket add redis https://github.com/redis/scoop.git
scoop install riot

Manual Installation (All Platforms)

Download the pre-compiled binary from RIOT Releases, uncompress and copy to the desired location.

riot-4.1.6.zip requires Java 11 or greater to be installed.

riot-standalone-4.1.6-*.zip includes its own Java runtime and does not require a Java installation.

Docker

You can run RIOT as a docker image:

docker run riotx/riot [OPTIONS] [COMMAND]

Concepts

RIOT is essentially an ETL tool where data is extracted from the source system, transformed (see Processing), and loaded into the target system.

architecture

Batching

Processing in RIOT is done in batches: a fixed number of records is read from the source, processed, and written to the target. The default batch size is 50, which means that an execution step reads 50 items at a time from the source, processes them, and finally writes then to the target. If the source/target is Redis, reading/writing of a batch is done in a single command pipeline to minimize the number of roundtrips to the server.

You can change the batch size (and hence pipeline size) using the --batch option. The optimal batch size in terms of throughput depends on many factors like record size and command types (see Redis Pipeline Tuning for details).

Multi-threading

By default processing happens in a single thread, but it is possible to parallelize processing by using multiple threads. In that configuration, each chunk of items is read, processed, and written in a separate thread of execution. This is different from partitioning where items would be read by multiple readers. Here, only one reader is being accessed from multiple threads.

To set the number of threads, use the --threads option.

Multi-threading example
riot db-import "SELECT * FROM orders" --jdbc-url "jdbc:postgresql://host:port/database" --jdbc-username appuser --jdbc-password passwd --threads 3 hset --keyspace order --key order_id

Processing

RIOT lets you transform incoming records using processors. These processors allow you to create/update/delete fields using the Spring Expression Language (SpEL). For example, import commands like file-import, db-import, and faker have a --proc option that allow for field-level processing:

  • field1='foo' → generate a field named field1 containing the string foo

  • temp=(temp-32)*5/9 → convert from Fahrenheit to Celsius

  • name=remove(first).concat(remove(last)) → concatenate first and last fields and delete them

  • field2=null → delete field2

Input fields are accessed by name (e.g. field3=field1+field2).

Processors have access to the following context variables and functions:

date

Date parsing and formatting object. Instance of Java SimpleDateFormat.

number

Number parsing and formatting object. Instance of Java DecimalFormat.

faker

Faker object.

redis

Redis commands object. Instance of Lettuce RedisCommands. The replicate command exposes 2 command objects named source and target.

geo

Convenience function that takes a longitude and a latitude to produce a RediSearch geo-location string in the form longitude,latitude (e.g. location=#geo(lon,lat))

Processor example
riot file-import --proc epoch="#date.parse(mydate).getTime()" location="#geo(lon,lat)" name="#redis.hget('person1','lastName')" ...
Faker processor example
riot file-import http://storage.googleapis.com/jrx/beers.csv --header --proc fakeid="#faker.numerify('########')" hset --keyspace beer --key fakeid

You can register your own variables using --var.

Custom variable example
riot file-import http://storage.googleapis.com/jrx/lacity.csv --var rnd="new java.util.Random()" --proc randomInt="#rnd.nextInt(100)" --header hset --keyspace event --key Id

Filtering

Filters allow you to exclude records that don’t match a SpEL boolean expression.

For example this filter will only keep records where the value field is a series of digits:

riot file-import --filter "value matches '\\d+'" ...

Redis URI

RIOT follows the Redis URI specification, which supports standalone, sentinel and cluster Redis deployments with plain, SSL, TLS and unix domain socket connections.

You can use the host:port short hand for redis://host:port.
You can provide the database, password and timeouts within the Redis URI.
Redis Standalone

redis :// [[username :] password@] host [:port][/database] [?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName] [&libraryName=libraryName] [&libraryVersion=libraryVersion] ]

Redis Standalone (SSL)

rediss :// [[username :] password@] host [: port][/database] [?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName] [&libraryName=libraryName] [&libraryVersion=libraryVersion] ]

Redis Standalone (Unix Domain Sockets)

redis-socket :// [[username :] password@]path [?[timeout=timeout[d|h|m|s|ms|us|ns]] [&database=database] [&clientName=clientName] [&libraryName=libraryName] [&libraryVersion=libraryVersion] ]

Redis Sentinel

redis-sentinel :// [[username :] password@] host1[:port1] [, host2[:port2]] [, hostN[:portN]] [/database] [?[timeout=timeout[d|h|m|s|ms|us|ns]] [&sentinelMasterId=sentinelMasterId] [&clientName=clientName] [&libraryName=libraryName] [&libraryVersion=libraryVersion] ]

Timeout Units
d

Days

h

Hours

m

Minutes

s

Seconds

ms

Milliseconds

us

Microseconds

ns

Nanoseconds

Usage

You can launch RIOT with the following command:

riot

This will show usage help, which you can also get by running:

riot --help

--help is available on any command:

riot COMMAND --help

Run the following command to give riot TAB completion in the current shell:

$ source <(riot generate-completion)

Data Generation

RIOT includes 2 commands for data generation:

Data Structure Generator

The generate command generates Redis data structures as well as JSON and Timeseries.

riot generate [OPTIONS]
Example
riot generate --type string hash json timeseries

Faker Generator

The faker command generates data using Datafaker.

riot faker [OPTIONS] EXPRESSION... [REDIS COMMAND...]

where EXPRESSION is a Faker expression field in the form field="expression".

To show the full usage, run:

riot faker --help

You must specify at least one Redis command as a target.

Redis connection options apply to the root command (riot) and not to subcommands.

In this example the Redis options will not be taken into account:

riot faker id="numerify '####'" hset -h myredis.com -p 6380
Keys

Keys are constructed from input records by concatenating the keyspace prefix and key fields.

mapping
Import into hashes
riot faker id="numerify '##########'" firstName="name.first_name" lastName="name.last_name" address="address.full_address" hset --keyspace person --key id
Import into sets
riot faker name="GameOfThrones.character" --count 1000 sadd --keyspace got:characters --member name
Data Providers

Faker offers many data providers. Most providers don’t take any arguments and can be called directly:

Simple Faker example
riot faker firstName="name.first_name"

Some providers take parameters:

Parameter Faker example
riot faker lease="number.digits '2'"

Here are a few sample Faker expressions:

  • regexify '(a|b){2,3}'

  • regexify '\\.\\*\\?\\+'

  • bothify '????','false'

  • name.first_name

  • name.last_name

  • number.number_between '1','10'

Refer to Datafaker Providers for a list of providers and their corresponding documentation.

Databases

RIOT includes two commands for interaction with relational databases:

  • db-import: Import database tables into Redis

  • db-export: Export Redis data structures to a database

Drivers

RIOT relies on JDBC to interact with databases. It includes JDBC drivers for the most common database systems:

  • Oracle

    jdbc:oracle:thin:@myhost:1521:orcl

  • MS SQL Server

    jdbc:sqlserver://[serverName[\instanceName][:portNumber]][;property=value[;property=value]]

  • MySQL

    jdbc:mysql://[host]:[port][/database][?properties]

  • PostgreSQL

    jdbc:postgresql://host:port/database

For non-included databases you must install the corresponding JDBC driver under the lib directory and modify the CLASSPATH:

  • *nix: bin/riotCLASSPATH=$APP_HOME/lib/myjdbc.jar:$APP_HOME/lib/…​

  • Windows: bin\riot.batset CLASSPATH=%APP_HOME%\lib\myjdbc.jar;%APP_HOME%\lib...

Database Import

The db-import command imports data from a relational database into Redis.

Ensure RIOT has the relevant JDBC driver for your database. See the Drivers section for more details.
riot db-import --jdbc-url <jdbc url> SQL [REDIS COMMAND...]

To show the full usage, run:

riot db-import --help

You must specify at least one Redis command as a target.

Redis connection options apply to the root command (riot) and not to subcommands.

In this example the Redis options will not be taken into account:

riot db-import "SELECT * FROM customers" hset -h myredis.com -p 6380

The keys that will be written are constructed from input records by concatenating the keyspace prefix and key fields.

mapping
PostgreSQL Import Example
riot db-import "SELECT * FROM orders" --jdbc-url "jdbc:postgresql://host:port/database" --jdbc-username appuser --jdbc-password passwd hset --keyspace order --key order_id
Import from PostgreSQL to JSON strings
riot db-import "SELECT * FROM orders" --jdbc-url "jdbc:postgresql://host:port/database" --jdbc-username appuser --jdbc-password passwd set --keyspace order --key order_id

This will produce Redis strings that look like this:

{
  "order_id": 10248,
  "customer_id": "VINET",
  "employee_id": 5,
  "order_date": "1996-07-04",
  "required_date": "1996-08-01",
  "shipped_date": "1996-07-16",
  "ship_via": 3,
  "freight": 32.38,
  "ship_name": "Vins et alcools Chevalier",
  "ship_address": "59 rue de l'Abbaye",
  "ship_city": "Reims",
  "ship_postal_code": "51100",
  "ship_country": "France"
}

Database Export

Use the db-export command to read from a Redis database and writes to a SQL database.

Ensure RIOT has the relevant JDBC driver for your database. See the Drivers section for more details.

The general usage is:

riot db-export --jdbc-url <jdbc url> SQL

To show the full usage, run:

riot db-export --help
Example: export to PostgreSQL
riot db-export "INSERT INTO mytable (id, field1, field2) VALUES (CAST(:id AS SMALLINT), :field1, :field2)" --jdbc-url "jdbc:postgresql://host:port/database" --jdbc-username appuser --jdbc-password passwd --key-pattern "gen:*" --key-regex "gen:(?<id>.*)"

Files

RIOT includes two commands to work with files in various formats:

File Import

The file-import command reads data from files and writes it to Redis.

The basic usage for file imports is:

riot file-import [OPTIONS] FILE... [REDIS COMMAND...]

To show the full usage, run:

riot file-import --help

File paths can include wildcard patterns.

RIOT will try to determine the file type from its extension (e.g. .csv or .json), but you can specify it with the --filetype option.

Gzipped files are supported and the extension before .gz is used (e.g. myfile.json.gzjson).

Examples
  • /path/file.csv

  • /path/file-*.csv

  • /path/file.json

  • http://data.com/file.csv

  • http://data.com/file.json.gz

Use - to read from standard input.

Amazon S3 and Google Cloud Storage buckets are supported.

Importing from Amazon S3
riot file-import s3://riot-bucket-jrx/beers.json --s3-region us-west-1 hset --keyspace beer --key id
Importing from Google Cloud Storage
riot file-import gs://riot-bucket-jrx/beers.json hset --keyspace beer --key id
Data Structures

If no REDIS COMMAND is specified, it is assumed that the input file(s) contain Redis data structures serialized as JSON or XML. See the File Export section to learn about the expected format and how to generate such files.

Example
riot file-import /tmp/redis.json
Redis Commands

When one or more `REDIS COMMAND`s are specified, these commands are called for each input record.

Redis client options apply to the root command (riot) and not to Redis commands.

In this example Redis client options will not be taken into account:

riot file-import my.json hset -h myredis.com -p 6380

Redis command keys are constructed from input records by concatenating keyspace prefix and key fields.

mapping
Import into hashes with keyspace blah:<id>
riot file-import my.json hset --keyspace blah --key id
Import into JSON
riot file-import http://storage.googleapis.com/jrx/es_test-index.json json.set --keyspace elastic --key _id
Import into hashes and set TTL on the key
riot file-import my.json hset --keyspace blah --key id expire --keyspace blah --key id
Import into hashes in keyspace blah:<id> and set TTL and add each id to a set named myset
riot file-import my.json hset --keyspace blah --key id expire --keyspace blah --key id sadd --keyspace myset --member id
Delimited (CSV)

The default delimiter character is comma (,). It can be changed with the --delimiter option.

If the file has a header, use the --header option to automatically extract field names. Otherwise specify the field names using the --fields option.

Let’s consider this CSV file:

Table 1. beers.csv
row abv ibu id name style brewery ounces

1

0.079

45

321

Fireside Chat (2010)

Winter Warmer

368

12.0

2

0.068

65

173

Back in Black

American Black Ale

368

12.0

3

0.083

35

11

Monk’s Blood

Belgian Dark Ale

368

12.0

The following command imports this CSV into Redis as hashes using beer as the key prefix and id as primary key.

riot file-import http://storage.googleapis.com/jrx/beers.csv --header hset --keyspace beer --key id

This creates hashes with keys beer:321, beer:173, …​

This command imports a CSV file into a geo set named airportgeo with airport IDs as members:

riot file-import http://storage.googleapis.com/jrx/airports.csv --header --skip-limit 3 geoadd --keyspace airportgeo --member AirportID --lon Longitude --lat Latitude
Fixed-Length (Fixed-Width)

Fixed-length files can be imported by specifying the width of each field using the --ranges option.

riot file-import http://storage.googleapis.com/jrx/accounts.fw --ranges 1 9 25 41 53 67 83 --header hset --keyspace account --key Account
JSON

The expected format for JSON files is:

[
  {
    "...": "..."
  },
  {
    "...": "..."
  }
]
JSON import example
riot file-import /tmp/redis.json

JSON records are trees with potentially nested values that need to be flattened when the target is a Redis hash for example.

To that end, RIOT uses a field naming convention to flatten JSON objects and arrays:

Table 2. Nested object

{ "field": { "sub": "value" } }

field.sub=value

Table 3. Array

{ "field": [1, 2, 3] }

field[0]=1 field[1]=2 field[2]=3

XML

Here is a sample XML file that can be imported by RIOT:

<?xml version="1.0" encoding="UTF-8"?>
<records>
    <trade>
        <isin>XYZ0001</isin>
        <quantity>5</quantity>
        <price>11.39</price>
        <customer>Customer1</customer>
    </trade>
    <trade>
        <isin>XYZ0002</isin>
        <quantity>2</quantity>
        <price>72.99</price>
        <customer>Customer2c</customer>
    </trade>
    <trade>
        <isin>XYZ0003</isin>
        <quantity>9</quantity>
        <price>99.99</price>
        <customer>Customer3</customer>
    </trade>
</records>
XML Import Example
riot file-import http://storage.googleapis.com/jrx/trades.xml hset --keyspace trade --key id

File Export

The file-export command reads data from a Redis database and writes it to a JSON or XML file, potentially gzip-compressed.

The general usage is:

riot file-export [OPTIONS] FILE

To show the full usage, run:

riot file-export --help
JSON
Export to JSON
riot file-export /tmp/redis.json
Sample JSON-export file
[
  {
    "key": "string:615",
    "ttl": -1,
    "value": "value:615",
    "type": "STRING"
  },
  {
    "key": "hash:511",
    "ttl": -1,
    "value": {
      "field1": "value511",
      "field2": "value511"
    },
    "type": "HASH"
  },
  {
    "key": "list:1",
    "ttl": -1,
    "value": [
      "member:991",
      "member:981"
    ],
    "type": "LIST"
  },
  {
    "key": "set:2",
    "ttl": -1,
    "value": [
      "member:2",
      "member:3"
    ],
    "type": "SET"
  },
  {
    "key": "zset:0",
    "ttl": -1,
    "value": [
      {
        "value": "member:1",
        "score": 1.0
      }
    ],
    "type": "ZSET"
  },
  {
    "key": "stream:0",
    "ttl": -1,
    "value": [
      {
        "stream": "stream:0",
        "id": "1602190921109-0",
        "body": {
          "field1": "value0",
          "field2": "value0"
        }
      }
    ],
    "type": "STREAM"
  }
]
Export to compressed JSON
riot file-export /tmp/beers.json.gz --key-pattern beer:*
XML
Export to XML
riot file-export /tmp/redis.xml

Replication

The replicate command reads data from a source Redis database and writes to a target Redis database.

replication architecture

The replication mechanism is as follows:

  1. Identify source keys to be replicated using scan and/or keyspace notifications depending on the replication mode.

  2. Read data associated with each key using dump or type-specific commands.

  3. Write each key to the target using restore or type-specific commands.

The basic usage is:

riot replicate [OPTIONS] SOURCE TARGET

where SOURCE and TARGET are Redis URIs.

For the full usage, run:

riot replicate --help
To replicate a Redis logical database other than the default (0), specify the database in the source Redis URI. For example riot replicate redis://source:6379/1 redis://target:6379 replicates database 1.

Replication Mode

Replication starts with identifying keys to be replicated from the source Redis database. The --mode option allows you to specify how RIOT identifies keys to be replicated:

  • iterate over keys with a key scan (--mode scan)

  • received by a keyspace notification subscriber (--mode liveonly)

  • or both (--mode live)

Scan

This key reader scans for keys using the Redis SCAN command:

SCAN cursor [MATCH pattern] [COUNT count] [TYPE type]
MATCH pattern

configured with the --key-pattern option

TYPE type

configured with the --key-type option

COUNT count

configured with the --scan-count option

INFO: In cluster mode keys are scanned in parallel across cluster nodes.

The status bar shows progress with a percentage of keys that have been replicated. The total number of keys is estimated when the replication process starts and it can change by the time it is finished, for example if keys are deleted or added during replication.

Scan replication example
riot replicate redis://source redis://target
Live

The key notification reader listens for key changes using keyspace notifications.

Make sure the source database has keyspace notifications enabled using:

  • redis.conf: notify-keyspace-events = KEA

  • CONFIG SET notify-keyspace-events KEA

For more details see Redis Keyspace Notifications.

Live replication example
riot replicate --mode live redis://source redis://target

The live replication mechanism does not guarantee data consistency. Redis sends keyspace notifications over pub/sub which does not provide guaranteed delivery. It is possible that RIOT can miss some notifications in case of network failures for example.

Also, depending on the type, size, and rate of change of data structures on the source it is possible that RIOT cannot keep up with the change stream. For example if a big set is repeatedly updated, RIOT will need to read the whole set on each update and transfer it over to the target database. With a big-enough set, RIOT could fall behind and the internal queue could fill up leading up to updates being dropped.

For those potentially problematic migrations it is recommend to perform some preliminary sizing using Redis statistics and bigkeys/memkeys in tandem with --mem-limit. If you need assistance please contact your Redis account team.

Replication Types

RIOT offers two different mechanisms for reading and writing keys:

  • Dump & restore (default)

  • Data structure replication (--struct)

Dump & Restore

The default replication mechanism is Dump & Restore:

  1. Scan for keys in the source Redis database. If live replication is enabled the reader also subscribes to keyspace notifications to generate a continuous stream of keys.

  2. Reader threads iterate over the keys to read corresponding values (DUMP) and TTLs.

  3. Reader threads enqueue key/value/TTL tuples into the reader queue, from which the writer dequeues key/value/TTL tuples and writes them to the target Redis database by calling RESTORE and EXPIRE.

Data Structure Replication

There are situations where Dump & Restore cannot be used, for example:

  • The target Redis database does not support the RESTORE command (Redis Enterprise CRDB)

  • Incompatible DUMP formats between source and target (Redis 7.0)

In those cases you can use another replication strategy that is data structure-specific: each key is introspected to determine its type and then use the corresponding read/write commands.

Type Read Write

Hash

HGETALL

HSET

JSON

JSON.GET

JSON.SET

List

LRANGE

RPUSH

Set

SMEMBERS

SADD

Sorted Set

ZRANGE

ZADD

Stream

XRANGE

XADD

String

GET

SET

TimeSeries

TS.RANGE

TS.ADD

This replication strategy is more intensive in terms of CPU, memory, and network for all the machines involved (source Redis, target Redis, and RIOT machines). Adjust number of threads, batch and queue sizes accordingly.
Type-based replication example
riot replicate --struct redis://source redis://target
Live type-based replication example
riot replicate --struct --mode live redis://source redis://target

Compare

Once replication is complete, RIOT performs a verification step by reading keys in the source database and comparing them against the target database.

The verification step happens automatically after the scan is complete (snapshot replication), or for live replication when keyspace notifications have become idle.

Verification can also be run on-demand using the compare command:

riot compare SOURCE TARGET [OPTIONS]

The output looks like this:

Verification failed (type: 225,062, missing: 485,450)
missing

Number of keys in source but not in target.

type

Number of keys with mismatched types (e.g. hash vs string).

value

Number of keys with mismatched values.

ttl

Number of keys with mismatched TTL i.e. difference is greater than tolerance (can be specified with --ttl-tolerance).

There are 2 comparison modes available through --compare (--quick for compare command):

  • Quick (default): compares key types and TTLs.

  • Full: compares key types, TTLs, and values.

To show which keys differ, use the --show-diffs option.

Performance

Performance tuning is an art but RIOT offers some options to identify potential bottlenecks. In addition to batch and threads options you have the --dry-run option which disables writing to the target Redis database so that you can tune the reader in isolation. Add that option to your existing replicate command-line to compare replication speeds with and without writing to the target Redis database.

Cookbook

Here are various recipes using RIOT.

Changelog

You can use RIOT to stream change data from a Redis database.

Streaming to stdout
riot file-export --mode live
{"key":"gen:1","type":"string","time":1718050552000,"ttl":-1,"memoryUsage":300003376}
{"key":"gen:3","type":"string","time":1718050552000,"ttl":-1,"memoryUsage":300003376}
{"key":"gen:6","type":"string","time":1718050552000,"ttl":-1,"memoryUsage":300003376}
...
Streaming to a file
riot file-export export.json --mode live

ElastiCache Migration

This recipe contains step-by-step instructions to migrate an ElastiCache (EC) database to Redis Cloud or Redis Software.

The following scenarios are covered:

  • One-time (snapshot) migration

  • Online (live) migration

It is recommended to read the Replication section to familiarize yourself with its usage and architecture.

Setup

Prerequisites

For this recipe you will require the following resources:

  • AWS ElastiCache: Primary Endpoint in case of Single Master and Configuration Endpoint in case of Clustered EC. Refer to this link to learn more

  • Redis Cloud or Redis Software

  • An Amazon EC2 instance to run RIOT

Keyspace Notifications

For a live migration you need to enable keyspace notifications on your ElastiCache instance (see AWS Knowledge Center).

Migration Host

To run the migration tool we will need an EC2 instance.

You can either create a new EC2 instance or leverage an existing one if available. In the example below we first create an instance on AWS Cloud Platform. The most common scenario is to access an ElastiCache cluster from an Amazon EC2 instance in the same Amazon Virtual Private Cloud (Amazon VPC). We have used Ubuntu 16.04 LTS for this setup but you can choose any Ubuntu or Debian distribution of your choice.

SSH to this EC2 instance from your laptop:

ssh -i “public key” <AWS EC2 Instance>

Install redis-cli on this new instance by running this command:

sudo apt update
sudo apt install -y redis-tools

Use redis-cli to check connectivity with the ElastiCache database:

redis-cli -h <ec primary endpoint> -p 6379

Ensure that the above command allows you to connect to the remote ElastiCache database successfully.

Installing RIOT

Let’s install RIOT on the EC2 instance we set up previously. For this we’ll follow the steps in Manual Installation.

Performing Migration

We are now all set to begin the migration process. The options you will use depend on your source and target databases, as well as the replication mode (snapshot or live).

ElastiCache Single Master → Redis
riot replicate source:port target:port
Live ElastiCache Single Master → Redis
riot replicate source:port target:port --mode live

In case ElastiCache is configured with AUTH TOKEN enabled, you need to pass --source-tls as well as --source-pass option:

riot replicate source:port target:port --source-tls --source-pass <password>
ElastiCache Cluster → Redis
riot replicate source:port target:port --source-cluster
--cluster is an important parameter used ONLY for ElastiCache whenever cluster-mode is enabled. Do note that the source database is specified first and the target database is specified after the replicate command and it is applicable for all the scenarios.
ElastiCache Single Master → Redis (with specific database index)
riot replicate redis://source:port/db target:port
ElastiCache Single Master → Redis with OSS Cluster
riot replicate source:port target:port --target-cluster
Live ElastiCache Cluster → Redis with OSS Cluster
riot replicate source:port target:port --source-cluster --target-cluster --mode live

Important Considerations

  • It is recommended to test migration in UAT before production use.

  • Once migration is completed, ensure that application traffic gets redirected to Redis endpoint successfully.

  • It is recommended to perform the migration process during low traffic hours so as to avoid chances of data loss.

Connectivity Test

The ping command can be used to test connectivity to a Redis database.

riot ping [OPTIONS]

To show the full usage, run:

riot ping --help

The command prints statistics like these:

riot ping -h localhost --unit microseconds
[min=491, max=14811, percentiles={99.9=14811, 90.0=1376, 95.0=2179, 99.0=14811, 50.0=741}]
[min=417, max=1286, percentiles={99.9=1286, 90.0=880, 95.0=1097, 99.0=1286, 50.0=606}]
[min=382, max=2244, percentiles={99.9=2244, 90.0=811, 95.0=1036, 99.0=2244, 50.0=518}]
...

FAQ

  1. Logs are cut off or missing

    This could be due to concurrency issues in the terminal when refreshing the progress bar and displaying logs. Try running with job option --progress log.

  2. Unknown options: '--keyspace', '--key'

    You must specify one or more Redis commands with import commands (file-import, faker, db-import).

  3. ERR DUMP payload version or checksum are wrong

    Redis 7 DUMP format is not backwards compatible with previous versions. To replicate between different Redis versions, use Type-Based Replication.

  4. Process gets stuck during replication and eventually times out

    This could be due to big keys clogging the replication pipes. In these cases it might be hard to catch the offending key(s). Try running the same command with --info and --progress log so that all errors are reported. Check the database with redis-cli Big keys and/or use reader options to filter these keys out.

  5. NOAUTH Authentication required

    This issue occurs when you fail to supply the --pass <password> parameter.

  6. ERR The ID argument cannot be a complete ID because xadd-id-uniqueness-mode is strict

    This usually happens in Active/Active (CRDB) setups where stream message IDs cannot be copied over to the target database. Use the --no-stream-id option to disable ID propagation.

  7. ERR Error running script…​ This Redis command is not allowed from scripts

    This can happen with Active/Active (CRDB) databases because the MEMORY USAGE command is not allowed to be run from a LUA script. Use the --mem-limit -1 option to disable memory usage.

  8. java.lang.OutOfMemoryError: Java heap space

    The RIOT JVM ran out of memory. Either increase max JVM heap size (export JAVA_OPTS="-Xmx8g") or reduce RIOT memory usage by lowering threads, batch, read-batch and read-queue.