Introduction

RIOT-X is a command-line utility to get data in and out of Redis. It supports Redis Cloud and Redis Software and includes the following features:

Files (CSV, JSON, XML, Parquet)
Databases
Data Generators
Replication

Redis → Redis

RIOT-X is supported by Redis, Inc. To report bugs, request features, or receive assistance, please file an issue or contact your Redis account team.

Install

RIOT-X can be installed on Linux, macOS, and Windows platforms and can be used as a standalone tool that connects remotely to a Redis database. It is not required to run locally on a Redis server.

Homebrew (macOS & Linux)

brew install redis/tap/riotx

Scoop (Windows)

scoop bucket add redis https://github.com/redis/scoop.git
scoop install riotx

Manual Installation (All Platforms)

Download the pre-compiled binary from RIOT-X Releases, uncompress and copy to the desired location.

riotx-1.4.1.zip requires Java 11 or greater to be installed.

riotx-standalone-1.4.1-*.zip includes its own Java runtime and does not require a Java installation.

Docker

You can run RIOT-X as a docker image:

docker run riotx/riotx [OPTIONS] [COMMAND]

Concepts

RIOT-X is essentially an ETL tool where data is extracted from the source system, transformed (see Processing), and loaded into the target system.

architecture

Redis URI

RIOT-X follows the Redis URI specification, which supports standalone, sentinel and cluster Redis deployments with plain, SSL, TLS and unix domain socket connections.

You can use the host:port short hand for redis://host:port.
Redis Standalone

redis :// [[username :] password@] host [:port][/database] [?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName] [&libraryName=libraryName] [&libraryVersion=libraryVersion] ]

Redis Standalone (SSL)

rediss :// [[username :] password@] host [: port][/database] [?[timeout=timeout[d|h|m|s|ms|us|ns]] [&clientName=clientName] [&libraryName=libraryName] [&libraryVersion=libraryVersion] ]

Redis Sentinel

redis-sentinel :// [[username :] password@] host1[:port1] [, host2[:port2]] [, hostN[:portN]] [/database] [?[timeout=timeout[d|h|m|s|ms|us|ns]] [&sentinelMasterId=sentinelMasterId] [&clientName=clientName] [&libraryName=libraryName] [&libraryVersion=libraryVersion] ]

You can provide the database, password and timeouts within the Redis URI. For example redis://localhost:6379/1 selects database 1.
Timeout Units
d

Days
h

Hours
m

Minutes
s

Seconds
ms

Milliseconds
us

Microseconds
ns

Nanoseconds

Batching

Processing in RIOT-X is done in batches: a fixed number of records is read from the source, processed, and written to the target. The default batch size is 50, which means that an execution step reads 50 items at a time from the source, processes them, and finally writes then to the target. If the source/target is Redis, reading/writing of a batch is done in a single command pipeline to minimize the number of roundtrips to the server.

You can change the batch size (and hence pipeline size) using the --batch option. The optimal batch size in terms of throughput depends on many factors like record size and command types (see Redis Pipeline Tuning for details).

Multi-threading

By default processing happens in a single thread, but it is possible to parallelize processing by using multiple threads. In that configuration, each chunk of items is read, processed, and written in a separate thread of execution. This is different from partitioning where items would be read by multiple readers. Here, only one reader is being accessed from multiple threads.

To set the number of threads, use the --threads option.

Multi-threading example
riotx db-import "SELECT * FROM orders" --jdbc-url "jdbc:postgresql://host:port/database" --jdbc-user appuser --jdbc-pass passwd --threads 3 hset order:#{order_id}

Importing

There are common features available when importing data into Redis (db-import, faker, file-import, redis-import, snowflake-import, stream-import).

Redis Key

RIOT-X offers two mechanisms to craft the Redis key:

Template Expression

This produces the key string using a SpEL expression.

Template expressions allow mixing literal text with one or more evaluation blocks. Each evaluation block is delimited with #{ } and within an evaluation block you can directly reference fields by their name, for example:

Template Expression Example
riotx file-import beers.json hset beer:#{id}

Here each key is constructed with the beer: prefix and suffixed with the value in the id field, producing keys beer:123 beer:494 etc.

Keyspace and Key Fields

This mechanism constructs keys using a keyspace string passed with --keyspace and key fields passed with --key, for example:

Keyspace/Key Fields Example
riotx file-import beers.json hset --keyspace beer --key id

Processing

Processors allow you to create/update/delete fields using the Spring Expression Language (SpEL). The SPEL processor system provides powerful data transformation capabilities with built-in type safety and extensive context support.

Field Processing Examples
--proc field1="'foo'"

Generate a field named field1 containing the string foo

--proc temp="(temp-32)*5/9"

Convert from Fahrenheit to Celsius

--proc name='remove("first").concat(remove("last"))'

Concatenate first and last fields and delete them

--proc field2=null

Delete field2

--proc index="T(java.lang.Integer).parseInt(id)"

Convert string field to integer

--proc fullName="firstName + ' ' + lastName"

Concatenate multiple fields

--proc upperName="name?.toUpperCase()"

Safe navigation with null checking

Input fields are accessed by name (e.g. field3=field1+field2).

Context Variables and Functions

Processors have access to the following context variables and functions:

date

Date parsing and formatting object. Instance of Java SimpleDateFormat.

number

Number parsing and formatting object. Instance of Java DecimalFormat.

faker

Faker object.

redis

Redis commands object. Instance of Lettuce RedisCommands. The replicate command exposes 2 command objects named source and target.

floatsToBytes

Convert an array of floats to a byte array so that the hash field can be indexed as vector (see Redis Vectors).

For example if the field vector is an array of floats: --proc embedding="#floatsToBytes(vector)"

Advanced Processing Patterns

Conditional Processing:

# Only process if field exists
--proc processedValue="value == null ? 0 : value * 2"

# Complex conditions
--proc category="price > 100 ? 'premium' : (price > 50 ? 'standard' : 'budget')"

Mathematical Operations:

# Statistical calculations
--proc average="(field1 + field2 + field3) / 3"
--proc percentage="(score / maxScore) * 100"
--proc rounded="T(java.lang.Math).round(value * 100.0) / 100.0"

String Processing:

# Text transformations
--proc slug="name.toLowerCase().replaceAll('[^a-z0-9]+', '-')"
--proc initials="firstName.substring(0,1) + lastName.substring(0,1)"
--proc masked="'***-**-' + ssn.substring(ssn.length()-4)"

Date and Time Processing:

# Date parsing and formatting
--proc timestamp="#date.parse(dateString, 'yyyy-MM-dd').getTime()"
--proc formatted="#date.format(new java.util.Date(epoch), 'MM/dd/yyyy')"
--proc age="T(java.time.Period).between(T(java.time.LocalDate).parse(birthDate), T(java.time.LocalDate).now()).getYears()"
geo

Convenience function for RediSearch geo-location strings in the form longitude,latitude.

For example with longitude and latitude fields lon and lat: --proc location="#geo(lon,lat)"

Comprehensive processor example
riotx file-import beers.json hset beer:#{id} \
  --proc epoch="#date.parse(mydate).getTime()" \
  --proc location="#geo(lon,lat)" \
  --proc fullName="#redis.hget('person1','lastName')" \
  --proc category="abv > 7.0 ? 'strong' : 'regular'" \
  --proc description="name + ' - ' + style + ' (' + abv + '% ABV)'"
Faker processor example
riotx file-import http://storage.googleapis.com/jrx/beers.csv --header --proc fakeid="#faker.numerify('###########')" hset beer:#{fakeid}

You can register your own variables using --var.

Custom variable example
riotx file-import http://storage.googleapis.com/jrx/lacity.csv --var rnd="new java.util.Random()" --proc randomInt="#rnd.nextInt(100)" --header hset event:#{Id}

Filtering

Filters allow you to exclude records that don’t match a SpEL boolean expression.

For example this filter will only keep records where the value field is a series of digits:

riotx file-import --filter "value matches '\\d+'" ...

Redis Search Operations

RIOT-X supports RediSearch commands for full-text search and aggregation operations as part of import workflows.

Search Command

Execute FT.SEARCH queries against RediSearch indexes during data import.

Syntax
riotx <cmd> ... search INDEX QUERY [OPTIONS...]

Parameters:

  • INDEX - Template expression for the search index name

  • QUERY - Template expression for the search query

  • OPTIONS - Optional search parameters (e.g., limit 0 10, withpayloads, sortby field)

Basic search example
riotx faker id="numerify('####')" search myindex "*"
Search with limit
riotx faker id="numerify('####')" search myindex "@title:redis" limit 0 10
Search with sorting and field selection
riotx faker id="numerify('####')" search products "@category:electronics" return 3 name price rating sortby price desc
Aggregate Command

Execute FT.AGGREGATE aggregations for complex data analysis and reporting during import operations.

Syntax
riotx <cmd> ... aggregate INDEX QUERY [OPTIONS...]

Parameters:

  • INDEX - Template expression for the search index name

  • QUERY - Template expression for the query to aggregate over

  • OPTIONS - Aggregation pipeline expressions (groupby, reduce, sortby, etc.)

Group by field and calculate average
riotx faker id="numerify('##########')" --count 1m --metrics-redis log --progress none aggregate beers "*" "groupby" "1" "@style_name" reduce avg 1 "@abv" as abv sortby 2 "@abv" desc
Both search and aggregate commands are available as Redis operations in any import command context. Template expressions support SpEL for dynamic field/query generation and integrate with RIOT-X’s batching and backpressure mechanisms. Results can be processed through standard RIOT-X processing pipelines.

Exporting

When exporting data from Redis the following options allow for filtering .

Key Filtering

Key filtering can be done through multiple options in RIOT-X:

--key-pattern

Glob-style pattern used for scan and keyspace notification registration.

--key-type

Type of keys to consider for scan and keyspace notification registration.

--key-include & --key-exclude

Glob-style pattern(s) to futher filter keys on the client (RIOT-X) side, i.e. after they are received through scan or keyspace notifications.

--mem-limit: Ignore keys whose memory usage exceeds the given limit. For example --mem-limit 10mb skips keys over 10 MB in size.

Usage

You can launch RIOT-X with the following command:

riotx

This will show usage help, which you can also get by running:

app --help

--help is available on any command:

riotx COMMAND --help

Run the following command to give riotx TAB completion in the current shell:

source <(riotx generate-completion)

Environment Variables

RIOT-X supports configuration via environment variables for some command-line options. This allows for convenient automation and containerized deployments without exposing sensitive information in command lines.

Job

Variable CLI Option Default Description

RIOT_THREADS

--threads

1

Number of concurrent threads to use for batch processing.

RIOT_BATCH

--batch

50

Number of items in each batch.

RIOT_RATE

--rate

none

Limit number of items written per second (e.g., 100 or 5k).

RIOT_DRY_RUN

--dry-run

false

Enable dummy writes.

RIOT_SKIP

--skip

none

Number of failed items before failing the job.

RIOT_RETRY

--retry

10

Number of times to retry failed items: 0 → never, -1 → always.

RIOT_FLUSH

--flush

50ms

Max duration between batch flushes.

RIOT_IDLE

--idle

none

Min duration to consider reader complete (e.g., 3s 5m).

Redis Connection

Variable CLI Option Default Description

RIOT_REDIS_URI

-u, --uri

none

Redis server URI.

RIOT_REDIS_HOST

-h, --host

localhost

Redis server hostname.

RIOT_REDIS_PORT

-p, --port

6379

Redis server port.

RIOT_REDIS_SOCKET

-s, --socket

none

Redis server socket (overrides hostname and port).

RIOT_REDIS_USER

--user

none

ACL style 'AUTH username pass'. Needs password.

RIOT_REDIS_PASS

-a, --pass

none

Password to use when connecting to the Redis server.

RIOT_REDIS_TIMEOUT

--timeout

60s

Redis command timeout (e.g., 30s or 5m).

RIOT_REDIS_DB

-n, --db

0

Redis database number.

RIOT_REDIS_TLS

--tls

false

Establish a secure TLS connection.

RIOT_REDIS_INSECURE

--insecure

false

Allow insecure TLS connection by skipping cert validation.

RIOT_REDIS_TLS_VERIFY

--verify

FULL

TLS verify mode: FULL, CA, NONE.

RIOT_REDIS_KEYSTORE

--keystore

none

Path to keystore.

RIOT_REDIS_KEYSTORE_PASS

--keystore-pass

none

Keystore password.

RIOT_REDIS_TRUST

--trust

none

Path to truststore.

RIOT_REDIS_TRUST_PASS

--trust-pass

none

Truststore password.

RIOT_REDIS_CERT

--cert

none

Client certificate to authenticate with (X.509 PEM).

RIOT_REDIS_KEY

--key

none

Private key file to authenticate with (PKCS#8 PEM).

RIOT_REDIS_KEY_PASS

--key-pass

none

Private key password.

RIOT_REDIS_CACERT

--cacert

none

CA Certificate file to verify with (X.509).

RIOT_REDIS_CLIENT

--client

none

Client name used to connect to Redis.

RIOT_REDIS_CLUSTER

-c, --cluster

false

Enable Redis cluster mode.

RIOT_REDIS_RESP

--resp

Newest supported

Redis protocol version: RESP2, RESP3.

RIOT_REDIS_POOL

--pool

8

Max number of Redis connections.

RIOT_REDIS_READ_FROM

--read-from

UPSTREAM

Which Redis cluster nodes to read from.

Redis Reader

Variable CLI Option Default Description

RIOT_KEY_PATTERN

--key-pattern

none

Pattern of keys to read (default: all keys).

RIOT_KEY_TYPE

--key-type

none

Type of keys to read (default: all types).

RIOT_SCAN_COUNT

--scan-count

100

How many keys to read at once on each scan iteration.

RIOT_READ_BATCH

--read-batch

50

Number of values each reader thread should read in a pipelined call.

RIOT_EVENT_QUEUE

--event-queue

10000

Capacity of the keyspace notification queue.

RIOT_MEM_LIMIT

--mem-limit

0

Max memory usage for a key to be read (e.g., 12KB, 5MB).

Database

Variable CLI Option Default Description

RIOT_TABLE

TABLE parameter

none

Fully qualified Snowflake Table or Materialized View (e.g., DB.SCHEMA.TABLE).

RIOT_MAX

--max

none

Max number of rows to import from database.

RIOT_FETCH

--fetch

none

Number of rows to return with each fetch from database.

RIOT_ROWS

--rows

none

Max number of rows the ResultSet can contain.

RIOT_QUERY_TIMEOUT

--query-timeout

none

Database query timeout duration.

RIOT_DATABASE

--database

none

Snowflake database for the specified table(s).

RIOT_SCHEMA

--schema

none

Snowflake schema for the specified table(s).

RIOT_ROLE

--role

none

Snowflake role to use.

RIOT_WAREHOUSE

--warehouse

none

Snowflake warehouse to use.

RIOT_CDC_DATABASE

--cdc-database

none

Snowflake CDC database to use for stream and temp table.

RIOT_CDC_SCHEMA

--cdc-schema

none

Snowflake CDC schema to use for stream and temp table.

RIOT_POLL

--poll

default

Snowflake stream polling interval.

RIOT_SNAPSHOT

--snapshot

INITIAL

Snapshot mode: INITIAL, NEVER, etc.

RIOT_KEY_COLUMN

--key-column

none

Table column name(s) to use as the key for the CDC event.

RIOT_GEN

--gen

none

Columns to simulate CDC activity for instead of connecting to database.

RIOT_STREAM_LIMIT

--stream-limit

10000

Max length of RDI stream. Use 0 for no limit.

RIOT_STREAM_PREFIX

--stream-prefix

data:

Key prefix for stream containing change events.

RIOT_OFFSET_PREFIX

--offset-prefix

riotx:offset:

Key prefix for offset stored in Redis.

RIOT_OFFSET_KEY

--offset-key

metadata:debezium:offsets

Key name for Debezium offset.

RIOT_JDBC_DRIVER

--jdbc-driver

none

Fully qualified name of the JDBC driver.

RIOT_JDBC_URL

--jdbc-url

none

JDBC URL to connect to the database.

RIOT_JDBC_USER

--jdbc-user

none

Login username of the database.

RIOT_JDBC_PASS

--jdbc-pass

none

Login password of the database.

RIOT_JDBC_POOL

--jdbc-pool

none

Maximum number of connections in the pool.

RIOT_JDBC_TIMEOUT

--jdbc-timeout

none

Maximum time to wait for a connection from the pool.

Replication

Variable CLI Option Default Description

RIOT_SYNC_MODE

--mode

SCAN

Replication mode: SCAN, LIVE, or LIVEONLY.

RIOT_SYNC_STRUCT

--struct

false

Enable data structure-specific replication.

RIOT_LOG_KEYS

--log-keys

false

Log keys being read and written.

RIOT_COMPARE

--compare

QUICK

Compare mode: QUICK, FULL, or NONE.

RIOT_REMOVE_SOURCE_KEYS

--remove-source-keys

false

Delete keys from source after successful replication.

RIOT_NO_TTL

--no-ttl

false

Do not propagate key expiration times.

RIOT_STREAM_ID

--stream-id

true

Propagate stream message IDs.

RIOT_STREAM_PRUNE

--stream-prune

false

Drop empty streams.

RIOT_MERGE

--merge

false

Merge collection data structures instead of overwriting them.
Source Redis
Variable CLI Option Default Description

RIOT_SOURCE_URI

Source URI parameter

none

Source server URI.

RIOT_SOURCE_USER

--source-user

none

Source ACL style 'AUTH username pass'. Needs password.

RIOT_SOURCE_PASS

--source-pass

none

Password to use when connecting to the source server.

RIOT_SOURCE_TIMEOUT

--source-timeout

60s

Source Redis command timeout (e.g., 30s or 5m).

RIOT_SOURCE_TLS

--source-tls

false

Establish a secure TLS connection to source.

RIOT_SOURCE_INSECURE

--source-insecure

false

Allow insecure TLS connection to source by skipping cert validation.

RIOT_SOURCE_TLS_VERIFY

--source-verify

FULL

Source TLS verify mode: FULL, CA, NONE.

RIOT_SOURCE_CLIENT

--source-client

none

Client name used to connect to source Redis.

RIOT_SOURCE_CLUSTER

--source-cluster

false

Enable source cluster mode.

RIOT_SOURCE_RESP

--source-resp

Newest supported

Redis protocol version used to connect to source.

RIOT_SOURCE_POOL

--source-pool

8

Max number of source Redis connections.

RIOT_SOURCE_READ_FROM

--source-read-from

UPSTREAM

Which source Redis cluster nodes to read from.
Target Redis
Variable CLI Option Default Description

RIOT_TARGET_URI

Target URI parameter

none

Target server URI.

RIOT_TARGET_USER

--target-user

none

Target ACL style 'AUTH username pass'. Needs password.

RIOT_TARGET_PASS

--target-pass

none

Password to use when connecting to the target server.

RIOT_TARGET_TIMEOUT

--target-timeout

60s

Target Redis command timeout (e.g., 30s or 5m).

RIOT_TARGET_TLS

--target-tls

false

Establish a secure TLS connection to target.

RIOT_TARGET_INSECURE

--target-insecure

false

Allow insecure TLS connection to target by skipping cert validation.

RIOT_TARGET_TLS_VERIFY

--target-verify

FULL

Target TLS verify mode: FULL, CA, NONE.

RIOT_TARGET_CLIENT

--target-client

none

Client name used to connect to target Redis.

RIOT_TARGET_CLUSTER

--target-cluster

false

Enable target cluster mode.

RIOT_TARGET_RESP

--target-resp

Newest supported

Redis protocol version used to connect to target.

RIOT_TARGET_POOL

--target-pool

8

Max number of target Redis connections.

RIOT_TARGET_READ_FROM

--target-read-from

UPSTREAM

Which target Redis cluster nodes to read from.
TLS Connection
export RIOT_SOURCE_TLS=true
export RIOT_SOURCE_INSECURE=true
export RIOT_TARGET_TLS=true
export RIOT_TARGET_INSECURE=true

riotx replicate redis://secure-source:6380 redis://secure-target:6380

Observability

Variable CLI Option Default Description

RIOT_LOG

--log-level

WARN

Set log level: ERROR, WARN, INFO, DEBUG, or TRACE.

RIOT_PROGRESS

--progress

ASCII

Progress style: ASCII, BAR, BLOCK, or NONE.

RIOT_NO_COLOR

none

false

Disable colored output in the console.

RIOT_METRICS

--metrics

false

Enable metrics.

RIOT_METRICS_JVM

--metrics-jvm

true

Enable/disable JVM metrics.

RIOT_METRICS_REDIS

--metrics-redis

false

Enable command latency metrics.

RIOT_METRICS_PORT

--metrics-port

8080

Port that Prometheus HTTP server should listen on.

Usage Examples

Basic Redis Connection
export RIOT_REDIS_HOST=my-redis-server
export RIOT_REDIS_PORT=6380
export RIOT_REDIS_USER=myuser
export RIOT_REDIS_PASS=mypassword
export RIOT_REDIS_DB=1
export RIOT_REDIS_TLS=true

riotx ping
Basic Replication
export RIOT_SOURCE_USER=myuser
export RIOT_SOURCE_PASS=mypassword
export RIOT_TARGET_USER=targetuser
export RIOT_TARGET_PASS=targetpassword
export RIOT_THREADS=4
export RIOT_BATCH=100

riotx replicate redis://source:6379 redis://target:6379
Import
export RIOT_THREADS=8
export RIOT_BATCH=500
export RIOT_FLUSH=10ms
export RIOT_RATE=10000
export RIOT_METRICS=true
export RIOT_PROGRESS=BAR

riotx file-import mydata.csv redis://localhost:6379
Snowflake Import
export RIOT_REDIS_HOST=redis
export RIOT_REDIS_PORT=12000
export RIOT_STREAM_LIMIT=100
export RIOT_ROLE=riotx_cdc
export RIOT_WAREHOUSE=compute_wh
export RIOT_CDC_SCHEMA=cdc_schema
export RIOT_JDBC_URL="jdbc:snowflake://company.snowflakecomputing.com?private_key_file=/path/to/key.p8"
export RIOT_JDBC_USER=username
export RIOT_JDBC_PASS=password
export RIOT_TABLE=db.schema.table

riotx snowflake-import
Docker
docker run --env-file riot.env riotx/riotx:latest replicate \
  redis://source:6379 redis://target:6379

Where riot.env contains:

RIOT_THREADS=4
RIOT_BATCH=100
RIOT_SOURCE_PASS=sourcepassword
RIOT_TARGET_PASS=targetpassword
RIOT_METRICS=true
RIOT_LOG=INFO

Notes

  • Environment variables take precedence over command-line defaults but are overridden by explicit command-line arguments.

  • Boolean values can be set to true or false.

  • Duration values support suffixes like s (seconds), m (minutes), h (hours).

  • Numeric values with suffixes like k (thousand) and m (million) are supported where applicable.

  • Sensitive values like passwords should be set via environment variables rather than command-line arguments for security.

Man Page

RIOT-X man page.

Data Generation

RIOT-X includes 2 commands for data generation:

generate

Generate Redis data structures

faker

Import data from Datafaker

Data Structure Generator

The gen command generates Redis data structures as well as JSON and Timeseries.

riotx gen [OPTIONS]
Example
riotx gen --type string hash json timeseries

Faker Generator

The faker command generates data using Datafaker.

riotx faker [OPTIONS] FIELD... [REDIS COMMAND...]

where FIELD is a Faker field in the form field="expression".

To show the full usage, run:

riotx faker --help

You must specify at least one Redis command as a target.

Redis connection options apply to the root command (riotx) and not to subcommands.

In this example the Redis options will not be taken into account:

riotx faker id="numerify('####')" hset -h myredis.com -p 6380
Use the index field to access the current record count. You can for example do this: `riotx faker text="lorem.words(2)" id="index" hset test:#{id}"

Keys

Keys are constructed from input records by concatenating the keyspace prefix and key fields.

mapping
Import into hashes
riotx faker id="numerify('##########')" firstName="name.firstName" lastName="name.lastName" address="address.fullAddress" hset person:#{id}
Import into sets
riotx faker name="gameOfThrones.character" --count 1000 sadd got:characters --member "#{name}"

Data Providers

Faker offers many data providers. Most providers don’t take any arguments and can be called directly:

Simple Faker example
riotx faker firstName="name.firstName"

Some providers take parameters:

Parameter Faker example
riotx faker lease="number.digits(2)"

Here are a few sample Faker expressions:

  • regexify('(a|b){2,3}')

  • regexify('\\.\\*\\?\\+')

  • bothify('????',false)

  • name.firstName

  • name.lastName

  • number.numbeBbetween(1,10)

Faker Providers and Expressions

Use these expressions in faker field definitions:

riotx faker firstName="name.firstName" age="number.numberBetween(18,99)"
ADDRESS

  • address.buildingNumber

  • address.city

  • address.cityName

  • address.cityPrefix

  • address.citySuffix

  • address.country

  • address.countryCode

  • address.countyByZipCode(string)

  • address.fullAddress

  • address.latLon

  • address.latLon(string)

  • address.latitude

  • address.lonLat

  • address.lonLat(string)

  • address.longitude

ANCIENT

  • ancient.god

  • ancient.hero

  • ancient.primordial

  • ancient.titan

ANIMAL

  • animal.genus

  • animal.name

  • animal.scientificName

  • animal.species

APP

  • app.author

  • app.name

  • app.version

APPLIANCE

  • appliance.brand

  • appliance.equipment

AQUATEENHUNGERFORCE

  • aquaTeenHungerForce.character

ARTIST

  • artist.name

AUSTRALIA

  • australia.animals

  • australia.locations

  • australia.states

AVATAR

  • avatar.image

AVIATION

  • aviation.METAR

  • aviation.aircraft

  • aviation.airline

  • aviation.airplane

  • aviation.airport

  • aviation.airportName

  • aviation.armyHelicopter

  • aviation.cargo

  • aviation.civilHelicopter

  • aviation.engineType

  • aviation.flight

  • aviation.flight(string)

  • aviation.flightStatus

  • aviation.gate

  • aviation.general

AWS

  • aws.accountId

  • aws.acmARN

  • aws.albARN

  • aws.albTargetGroupARN

  • aws.region

  • aws.route53ZoneId

  • aws.securityGroupId

  • aws.service

  • aws.subnetId

  • aws.vpcId

AZURE

  • azure.appServiceEnvironment

  • azure.appServicePlan

  • azure.applicationGateway

  • azure.bastionHost

  • azure.containerApps

  • azure.containerAppsEnvironment

  • azure.containerInstance

  • azure.containerRegistry

  • azure.cosmosDBDatabase

  • azure.firewall

  • azure.keyVault

  • azure.loadBalancer

  • azure.loadTesting

  • azure.logAnalytics

  • azure.managementGroup

BABYLON5

  • babylon5.character

  • babylon5.quote

BACKTOTHEFUTURE

  • backToTheFuture.character

  • backToTheFuture.date

  • backToTheFuture.quote

BARCODE

  • barcode.ean13

  • barcode.ean8

  • barcode.gtin12

  • barcode.gtin13

  • barcode.gtin14

  • barcode.gtin8

  • barcode.type

BASEBALL

  • baseball.coaches

  • baseball.players

  • baseball.positions

  • baseball.teams

BASKETBALL

  • basketball.coaches

  • basketball.players

  • basketball.positions

  • basketball.teams

BATTLEFIELD1

  • battlefield1.classes

  • battlefield1.faction

  • battlefield1.map

  • battlefield1.vehicle

  • battlefield1.weapon

BEER

  • beer.brand

  • beer.hop

  • beer.malt

  • beer.name

  • beer.style

  • beer.yeast

BIGBANGTHEORY

  • bigBangTheory.character

  • bigBangTheory.quote

BLOODTYPE

  • bloodtype.aboTypes

  • bloodtype.bloodGroup

  • bloodtype.pTypes

  • bloodtype.rhTypes

BOARDGAME

  • boardgame.artist

  • boardgame.category

  • boardgame.designer

  • boardgame.mechanic

  • boardgame.name

  • boardgame.publisher

  • boardgame.subdomain

BOJACKHORSEMAN

  • bojackHorseman.characters

  • bojackHorseman.quotes

  • bojackHorseman.tongueTwisters

BOOK

  • book.author

  • book.genre

  • book.publisher

  • book.title

BOOL

  • bool.bool

BOSSANOVA

  • bossaNova.artist

  • bossaNova.song

BRAND

  • brand.car

  • brand.sport

  • brand.watch

BREAKINGBAD

  • breakingBad.character

  • breakingBad.episode

BROOKLYNNINENINE

  • brooklynNineNine.characters

  • brooklynNineNine.quotes

BUFFY

  • buffy.bigBads

  • buffy.celebrities

  • buffy.characters

  • buffy.episodes

  • buffy.quotes

BUSINESS

  • business.creditCardExpiry

  • business.creditCardNumber

  • business.creditCardType

  • business.securityCode

CAMERA

  • camera.brand

  • camera.brandWithModel

  • camera.model

CANNABIS

  • cannabis.brands

  • cannabis.buzzwords

  • cannabis.cannabinoidAbbreviations

  • cannabis.cannabinoids

  • cannabis.categories

  • cannabis.healthBenefits

  • cannabis.medicalUses

  • cannabis.strains

  • cannabis.terpenes

  • cannabis.types

CAREPROVIDER

  • careProvider.hospitalName

  • careProvider.medicalProfession

CAT

  • cat.breed

  • cat.name

  • cat.registry

CHESS

  • chess.opening

  • chess.player

  • chess.title

  • chess.tournament

CHIQUITO

  • chiquito.expressions

  • chiquito.jokes

  • chiquito.sentences

  • chiquito.terms

CHUCKNORRIS

  • chuckNorris.fact

CLASHOFCLANS

  • clashOfClans.defensiveBuilding

  • clashOfClans.rank

  • clashOfClans.troop

CNPJ

  • cnpj.invalid

  • cnpj.invalid(boolean)

  • cnpj.invalid(boolean, boolean)

  • cnpj.valid

  • cnpj.valid(boolean)

  • cnpj.valid(boolean, boolean)

CODE

  • code.asin

  • code.ean13

  • code.ean8

  • code.gtin13

  • code.gtin8

  • code.imei

COFFEE

  • coffee.blendName

  • coffee.body

  • coffee.country

  • coffee.descriptor

  • coffee.intensifier

  • coffee.name1

  • coffee.name2

  • coffee.notes

  • coffee.region

  • coffee.region(country)

  • coffee.variety

COIN

  • coin.flip

COLLECTION

  • collection.build

  • collection.build

  • collection.faker(baseproviders)

  • collection.generate

  • collection.len(int)

  • collection.len(int, int)

  • collection.maxLen(int)

  • collection.minLen(int)

  • collection.nullRate(double)

  • collection.suppliers(Supplier[])

COLOR

  • color.hex

  • color.hex(boolean)

  • color.name

COMMERCE

  • commerce.brand

  • commerce.department

  • commerce.material

  • commerce.price

  • commerce.price(double, double)

  • commerce.productName

  • commerce.promotionCode

  • commerce.promotionCode(int)

  • commerce.vendor

COMMUNITY

  • community.character

  • community.quote

COMPANY

  • company.bs

  • company.buzzword

  • company.catchPhrase

  • company.industry

  • company.logo

  • company.name

  • company.profession

  • company.suffix

  • company.url

COMPASS

  • compass.abbreviation

  • compass.azimuth

  • compass.compassPoint(compasspoint)

  • compass.word

COMPUTER

  • computer.brand

  • computer.linux

  • computer.macos

  • computer.operatingSystem

  • computer.platform

  • computer.type

  • computer.windows

CONSTRUCTION

  • construction.heavyEquipment

  • construction.materials

  • construction.roles

  • construction.standardCostCodes

  • construction.subcontractCategories

  • construction.trades

CONTROL

  • control.alteredItem

  • control.alteredWorldEvent

  • control.character

  • control.hiss

  • control.location

  • control.objectOfPower

  • control.quote

  • control.theBoard

COSMERE

  • cosmere.allomancers

  • cosmere.aons

  • cosmere.feruchemists

  • cosmere.heralds

  • cosmere.knightsRadiant

  • cosmere.metals

  • cosmere.shardWorlds

  • cosmere.shards

  • cosmere.sprens

  • cosmere.surges

COUNTRY

  • country.capital

  • country.countryCode2

  • country.countryCode3

  • country.currency

  • country.currencyCode

  • country.flag

  • country.name

COWBOYBEBOP

  • cowboyBebop.character

  • cowboyBebop.episode

  • cowboyBebop.quote

  • cowboyBebop.song

CPF

  • cpf.invalid

  • cpf.invalid(boolean)

  • cpf.valid

  • cpf.valid(boolean)

CRICKET

  • cricket.formats

  • cricket.players

  • cricket.teams

  • cricket.tournaments

CRYPTOCOIN

  • cryptoCoin.coin

CULTURESERIES

  • cultureSeries.books

  • cultureSeries.civs

  • cultureSeries.cultureShipClassAbvs

  • cultureSeries.cultureShipClasses

  • cultureSeries.cultureShips

  • cultureSeries.planets

CURRENCY

  • currency.code

  • currency.name

DARKSOULS

  • darkSouls.classes

  • darkSouls.covenants

  • darkSouls.shield

  • darkSouls.stats

DATE

  • date.between(date, date)

  • date.between(date, date, string)

  • date.birthday

  • date.birthday(int, int)

  • date.birthday(int, int, string)

  • date.birthday(string)

  • date.birthdayLocalDate

  • date.birthdayLocalDate(int, int)

  • date.duration(long, chronounit)

  • date.duration(long, long, chronounit)

  • date.duration(long, long, string)

  • date.duration(long, string)

  • date.future(int, int, timeunit)

  • date.future(int, int, timeunit, string)

  • date.future(int, timeunit)

DCCOMICS

  • dcComics.hero

  • dcComics.heroine

  • dcComics.name

  • dcComics.title

  • dcComics.villain

DEMOGRAPHIC

  • demographic.demonym

  • demographic.educationalAttainment

  • demographic.maritalStatus

  • demographic.race

  • demographic.sex

DEPARTED

  • departed.actor

  • departed.character

  • departed.quote

DESSERT

  • dessert.flavor

  • dessert.topping

  • dessert.variety

DETECTIVECONAN

  • detectiveConan.characters

  • detectiveConan.gadgets

  • detectiveConan.vehicles

DEVICE

  • device.manufacturer

  • device.modelName

  • device.platform

  • device.serial

DISEASE

  • disease.anyDisease

  • disease.dermatology

  • disease.gynecologyAndObstetrics

  • disease.icd10

  • disease.internalDisease

  • disease.neurology

  • disease.ophthalmologyAndOtorhinolaryngology

  • disease.paediatrics

  • disease.surgery

DOCTORWHO

  • doctorWho.actor

  • doctorWho.catchPhrase

  • doctorWho.character

  • doctorWho.doctor

  • doctorWho.quote

  • doctorWho.species

  • doctorWho.villain

DOG

  • dog.age

  • dog.breed

  • dog.coatLength

  • dog.gender

  • dog.memePhrase

  • dog.name

  • dog.size

  • dog.sound

DOMAIN

  • domain.firstLevelDomain(string)

  • domain.fullDomain(string)

  • domain.secondLevelDomain(string)

  • domain.validDomain(string)

DORAEMON

  • doraemon.character

  • doraemon.gadget

  • doraemon.location

DOTA2

  • dota2.attribute

  • dota2.building

  • dota2.faction

  • dota2.hero

  • dota2.heroQuote(string)

  • dota2.item

  • dota2.neutralItem

  • dota2.player

  • dota2.rank

  • dota2.team

  • dota2.tier

DRAGONBALL

  • dragonBall.character

DRIVINGLICENSE

  • drivingLicense.drivingLicense(string)

DRONE

  • drone.batteryCapacity

  • drone.batteryType

  • drone.batteryVoltage

  • drone.batteryWeight

  • drone.chargingTemperature

  • drone.flightTime

  • drone.maxAltitude

  • drone.maxAngularVelocity

  • drone.maxAscentSpeed

  • drone.maxChargingPower

  • drone.maxDescentSpeed

  • drone.maxFlightDistance

  • drone.maxResolution

  • drone.maxShutterSpeed

  • drone.maxSpeed

DUMBANDDUMBER

  • dumbAndDumber.actor

  • dumbAndDumber.character

  • dumbAndDumber.quote

DUNE

  • dune.character

  • dune.planet

  • dune.quote

  • dune.quote(quote)

  • dune.saying

  • dune.saying(saying)

  • dune.title

DUNGEONSANDDRAGONS

  • dungeonsAndDragons.alignments

  • dungeonsAndDragons.backgrounds

  • dungeonsAndDragons.cities

  • dungeonsAndDragons.klasses

  • dungeonsAndDragons.languages

  • dungeonsAndDragons.meleeWeapons

  • dungeonsAndDragons.monsters

  • dungeonsAndDragons.races

  • dungeonsAndDragons.rangedWeapons

DURATION

  • duration.atMostDays(long)

  • duration.atMostHours(long)

  • duration.atMostMinutes(long)

  • duration.atMostSeconds(long)

EDUCATOR

  • educator.campus

  • educator.course

  • educator.secondarySchool

  • educator.subjectWithNumber

  • educator.university

ELDENRING

  • eldenRing.location

  • eldenRing.npc

  • eldenRing.skill

  • eldenRing.spell

  • eldenRing.weapon

ELDERSCROLLS

  • elderScrolls.city

  • elderScrolls.creature

  • elderScrolls.dragon

  • elderScrolls.firstName

  • elderScrolls.lastName

  • elderScrolls.quote

  • elderScrolls.race

  • elderScrolls.region

ELECTRICALCOMPONENTS

  • electricalComponents.active

  • electricalComponents.electromechanical

  • electricalComponents.passive

EMOJI

  • emoji.cat

  • emoji.smiley

  • emoji.vehicle

ENGLANDFOOTBALL

  • englandfootball.league

  • englandfootball.team

ESPORTS

  • esports.event

  • esports.game

  • esports.league

  • esports.player

  • esports.team

FAKEVALUESSERVICE

  • fakeValuesService.bothify(string, fakercontext)

  • fakeValuesService.bothify(string, fakercontext, boolean)

  • fakeValuesService.csv(int, String[])

  • fakeValuesService.csv(string, char, boolean, int, String[])

  • fakeValuesService.examplify(string, fakercontext)

  • fakeValuesService.expression(string, basefaker, fakercontext)

  • fakeValuesService.fetch(string, fakercontext)

  • fakeValuesService.fetchObject(string, fakercontext)

  • fakeValuesService.fetchString(string, fakercontext)

  • fakeValuesService.fileExpression(path, basefaker, fakercontext)

  • fakeValuesService.json(String[])

  • fakeValuesService.jsona(String[])

  • fakeValuesService.letterify(string, fakercontext)

  • fakeValuesService.letterify(string, fakercontext, boolean)

  • fakeValuesService.numerify(string, fakercontext)

FALLOUT

  • fallout.character

  • fallout.faction

  • fallout.location

  • fallout.quote

FAMILYGUY

  • familyGuy.character

  • familyGuy.location

  • familyGuy.quote

FAMOUSLASTWORDS

  • famousLastWords.lastWords

FILE

  • file.extension

  • file.fileName

  • file.fileName(string, string, string, string)

  • file.mimeType

FINALFANTASYXIV

  • finalFantasyXIV.character

  • finalFantasyXIV.dataCenter

  • finalFantasyXIV.job

  • finalFantasyXIV.race

  • finalFantasyXIV.zone

FINALSPACE

  • finalSpace.character

  • finalSpace.quote

  • finalSpace.vehicle

FINANCE

  • finance.bic

  • finance.creditCard

  • finance.creditCard(creditcardtype)

  • finance.iban

  • finance.iban(string)

  • finance.ibanSupportedCountries

  • finance.nasdaqTicker

  • finance.nyseTicker

  • finance.stockMarket

  • finance.usRoutingNumber

FINANCIALTERMS

  • financialTerms.adjective

  • financialTerms.adjective(category)

  • financialTerms.noun

  • financialTerms.noun(category)

  • financialTerms.verb

  • financialTerms.verb(category)

FOOD

  • food.allergen

  • food.dish

  • food.fruit

  • food.ingredient

  • food.measurement

  • food.spice

  • food.sushi

  • food.vegetable

FOOTBALL

  • football.coaches

  • football.competitions

  • football.players

  • football.positions

  • football.teams

FORMULA1

  • formula1.circuit

  • formula1.driver

  • formula1.grandPrix

  • formula1.team

FRESHPRINCEOFBELAIR

  • freshPrinceOfBelAir.celebrities

  • freshPrinceOfBelAir.characters

  • freshPrinceOfBelAir.quotes

FRIENDS

  • friends.character

  • friends.location

  • friends.quote

FULLMETALALCHEMIST

  • fullMetalAlchemist.character

  • fullMetalAlchemist.city

  • fullMetalAlchemist.country

FUNNYNAME

  • funnyName.name

FUTURAMA

  • futurama.character

  • futurama.hermesCatchPhrase

  • futurama.location

  • futurama.quote

GAMEOFTHRONES

  • gameOfThrones.character

  • gameOfThrones.city

  • gameOfThrones.dragon

  • gameOfThrones.house

  • gameOfThrones.quote

GARMENTSIZE

  • garmentSize.size

GENDER

  • gender.binaryTypes

  • gender.shortBinaryTypes

  • gender.types

GETFAKER

  • getFaker.address

  • getFaker.ancient

  • getFaker.animal

  • getFaker.app

  • getFaker.appliance

  • getFaker.aquaTeenHungerForce

  • getFaker.artist

  • getFaker.australia

  • getFaker.avatar

  • getFaker.aviation

  • getFaker.aws

  • getFaker.azure

  • getFaker.babylon5

  • getFaker.backToTheFuture

  • getFaker.barcode

GHOSTBUSTERS

  • ghostbusters.actor

  • ghostbusters.character

  • ghostbusters.quote

GRATEFULDEAD

  • gratefulDead.players

  • gratefulDead.songs

GREEKPHILOSOPHER

  • greekPhilosopher.name

  • greekPhilosopher.quote

HACKER

  • hacker.abbreviation

  • hacker.adjective

  • hacker.ingverb

  • hacker.noun

  • hacker.verb

HALFLIFE

  • halfLife.character

  • halfLife.enemy

  • halfLife.location

HARRYPOTTER

  • harryPotter.book

  • harryPotter.character

  • harryPotter.house

  • harryPotter.location

  • harryPotter.quote

  • harryPotter.spell

HASHING

  • hashing.md2

  • hashing.md5

  • hashing.sha1

  • hashing.sha256

  • hashing.sha384

  • hashing.sha512

HEARTHSTONE

  • hearthstone.battlegroundsScore

  • hearthstone.mainCharacter

  • hearthstone.mainPattern

  • hearthstone.mainProfession

  • hearthstone.standardRank

  • hearthstone.wildRank

HEROESOFTHESTORM

  • heroesOfTheStorm.battleground

  • heroesOfTheStorm.hero

  • heroesOfTheStorm.heroClass

  • heroesOfTheStorm.quote

HEYARNOLD

  • heyArnold.characters

  • heyArnold.locations

  • heyArnold.quotes

HIPSTER

  • hipster.word

HITCHHIKERSGUIDETOTHEGALAXY

  • hitchhikersGuideToTheGalaxy.character

  • hitchhikersGuideToTheGalaxy.location

  • hitchhikersGuideToTheGalaxy.marvinQuote

  • hitchhikersGuideToTheGalaxy.planet

  • hitchhikersGuideToTheGalaxy.quote

  • hitchhikersGuideToTheGalaxy.species

  • hitchhikersGuideToTheGalaxy.starship

HOBBIT

  • hobbit.character

  • hobbit.location

  • hobbit.quote

  • hobbit.thorinsCompany

HOBBY

  • hobby.activity

HOLOLIVE

  • hololive.talent

HORSE

  • horse.breed

  • horse.name

HOUSE

  • house.furniture

  • house.room

HOWIMETYOURMOTHER

  • howIMetYourMother.catchPhrase

  • howIMetYourMother.character

  • howIMetYourMother.highFive

  • howIMetYourMother.quote

HOWTOTRAINYOURDRAGON

  • howToTrainYourDragon.characters

  • howToTrainYourDragon.dragons

  • howToTrainYourDragon.locations

IDNUMBER

  • idNumber.inValidEnZaSsn

  • idNumber.invalid

  • idNumber.invalidEsMXSsn

  • idNumber.invalidPtNif

  • idNumber.invalidSvSeSsn

  • idNumber.peselNumber

  • idNumber.peselNumber(localdate, gender)

  • idNumber.singaporeanFin

  • idNumber.singaporeanFinBefore2000

  • idNumber.singaporeanUin

  • idNumber.singaporeanUinBefore2000

  • idNumber.ssnValid

  • idNumber.valid

  • idNumber.valid(idnumberrequest)

  • idNumber.validEnZaSsn

IMAGE

  • image.base64(base64imageruleconfig)

  • image.base64BMP

  • image.base64GIF

  • image.base64JPEG

  • image.base64JPG

  • image.base64PNG

  • image.base64SVG

  • image.base64TIFF

INDUSTRYSEGMENTS

  • industrySegments.industry

  • industrySegments.sector

  • industrySegments.subSector

  • industrySegments.superSector

INTERNET

  • internet.botUserAgent(botuseragent)

  • internet.botUserAgentAny

  • internet.domainName

  • internet.domainSuffix

  • internet.domainWord

  • internet.emailAddress

  • internet.emailAddress(string)

  • internet.emailSubject

  • internet.httpMethod

  • internet.image

  • internet.image(int, int)

  • internet.image(int, int, string)

  • internet.ipV4Address

  • internet.ipV4Cidr

  • internet.ipV6Address

JOB

  • job.field

  • job.keySkills

  • job.position

  • job.seniority

  • job.title

JOKE

  • joke.knockKnock

  • joke.pun

KAAMELOTT

  • kaamelott.character

  • kaamelott.quote

KPOP

  • kpop.boyBands

  • kpop.girlGroups

  • kpop.iGroups

  • kpop.iiGroups

  • kpop.iiiGroups

  • kpop.solo

LARGELANGUAGEMODEL

  • largeLanguageModel.embeddingModel

  • largeLanguageModel.textModel

  • largeLanguageModel.tokenizer

LEAGUEOFLEGENDS

  • leagueOfLegends.champion

  • leagueOfLegends.location

  • leagueOfLegends.masteries

  • leagueOfLegends.quote

  • leagueOfLegends.rank

  • leagueOfLegends.summonerSpell

LEBOWSKI

  • lebowski.actor

  • lebowski.character

  • lebowski.quote

LOCALITY

  • locality.allSupportedLocales

  • locality.allSupportedLocales(set)

  • locality.displayName

  • locality.localeString

  • locality.localeStringWithRandom(random)

  • locality.localeStringWithoutReplacement

  • locality.localeStringWithoutReplacement(random)

LOCATION

  • location.building

  • location.nature

  • location.otherworldly

  • location.privateSpace

  • location.publicSpace

  • location.work

LORDOFTHERINGS

  • lordOfTheRings.character

  • lordOfTheRings.location

LOREM

  • lorem.character

  • lorem.character(boolean)

  • lorem.characters

  • lorem.characters(boolean)

  • lorem.characters(int)

  • lorem.characters(int, boolean)

  • lorem.characters(int, boolean, boolean)

  • lorem.characters(int, boolean, boolean, boolean)

  • lorem.characters(int, int)

  • lorem.characters(int, int, boolean)

  • lorem.characters(int, int, boolean, boolean)

  • lorem.characters(int, int, boolean, boolean, boolean)

  • lorem.fixedString(int)

  • lorem.maxLengthSentence(int)

  • lorem.paragraph

MARKETING

  • marketing.buzzwords

MARVELSNAP

  • marvelSnap.character

  • marvelSnap.event

  • marvelSnap.rank

  • marvelSnap.zone

MASSEFFECT

  • massEffect.character

  • massEffect.cluster

  • massEffect.planet

  • massEffect.quote

  • massEffect.specie

MATZ

  • matz.quote

MBTI

  • mbti.characteristic

  • mbti.merit

  • mbti.name

  • mbti.personage

  • mbti.type

  • mbti.weakness

MEASUREMENT

  • measurement.height

  • measurement.length

  • measurement.metricHeight

  • measurement.metricLength

  • measurement.metricVolume

  • measurement.metricWeight

  • measurement.volume

  • measurement.weight

MEDICAL

  • medical.diagnosisCode

  • medical.diseaseName

  • medical.hospitalName

  • medical.medicalProfession

  • medical.medicineName

  • medical.procedureCode

  • medical.symptoms

MEDICALPROCEDURE

  • medicalProcedure.icd10

MEDICATION

  • medication.drugName

MILITARY

  • military.airForceRank

  • military.armyRank

  • military.dodPaygrade

  • military.marinesRank

  • military.navyRank

MINECRAFT

  • minecraft.animalName

  • minecraft.entityName

  • minecraft.itemName

  • minecraft.monsterName

  • minecraft.tileItemName

  • minecraft.tileName

MONEY

  • money.currency

  • money.currencyCode

  • money.currencyNumericCode

  • money.currencySymbol

MONEYHEIST

  • moneyHeist.character

  • moneyHeist.heist

  • moneyHeist.quote

MOOD

  • mood.emotion

  • mood.feeling

  • mood.tone

MOUNTAIN

  • mountain.name

  • mountain.range

MOUNTAINEERING

  • mountaineering.mountaineer

MOVIE

  • movie.name

  • movie.quote

MUSIC

  • music.chord

  • music.genre

  • music.instrument

  • music.key

MYST

  • myst.ages

  • myst.characters

  • myst.creatures

  • myst.games

  • myst.quotes

NAME

  • name.femaleFirstName

  • name.firstName

  • name.fullName

  • name.lastName

  • name.malefirstName

  • name.name

  • name.nameWithMiddle

  • name.prefix

  • name.suffix

  • name.title

  • name.username

NARUTO

  • naruto.character

  • naruto.demon

  • naruto.eye

  • naruto.village

NATION

  • nation.capitalCity

  • nation.flag

  • nation.language

  • nation.nationality

NATOPHONETICALPHABET

  • natoPhoneticAlphabet.codeWord

NEWGIRL

  • newGirl.characters

  • newGirl.quotes

NIGERIA

  • nigeria.celebrities

  • nigeria.food

  • nigeria.name

  • nigeria.places

  • nigeria.schools

NUMBER

  • number.digit

  • number.digits(int)

  • number.negative

  • number.numberBetween(double, double)

  • number.numberBetween(int, int)

  • number.numberBetween(long, long)

  • number.positive

  • number.randomDigit

  • number.randomDigitNotZero

  • number.randomDouble(int, int, int)

  • number.randomDouble(int, long, long)

  • number.randomNumber

  • number.randomNumber(int)

  • number.randomNumber(int, boolean)

OBSERVATION

  • observation.symptom

OLYMPICSPORT

  • olympicSport.ancientOlympics

  • olympicSport.summerOlympics

  • olympicSport.summerParalympics

  • olympicSport.unusual

  • olympicSport.winterOlympics

  • olympicSport.winterParalympics

ONEPIECE

  • onePiece.akumasNoMi

  • onePiece.character

  • onePiece.location

  • onePiece.quote

  • onePiece.sea

OPTIONS

  • options.nextElement(Object[])

  • options.nextElement(list)

  • options.option(Object[])

  • options.option(String[])

  • options.option(boolean[])

  • options.option(byte[])

  • options.option(char[])

  • options.option(class)

  • options.option(double[])

  • options.option(float[])

  • options.option(int[])

  • options.option(long[])

  • options.option(short[])

  • options.subset(int, Object[])

  • options.subset(int, String[])

OSCARMOVIE

  • oscarMovie.actor

  • oscarMovie.character

  • oscarMovie.movieName

  • oscarMovie.quote

  • oscarMovie.releaseDate

OVERWATCH

  • overwatch.hero

  • overwatch.location

  • overwatch.quote

PASSPORT

  • passport.valid

PHONENUMBER

  • phoneNumber.cellPhone

  • phoneNumber.cellPhoneInternational

  • phoneNumber.extension

  • phoneNumber.phoneNumber

  • phoneNumber.phoneNumberInternational

  • phoneNumber.phoneNumberNational

  • phoneNumber.subscriberNumber

  • phoneNumber.subscriberNumber(int)

PHOTOGRAPHY

  • photography.aperture

  • photography.brand

  • photography.camera

  • photography.genre

  • photography.imageTag

  • photography.lens

  • photography.shutter

  • photography.term

PLANET

  • planet.atmosphericComposition

  • planet.atmosphericComposition(planetname)

  • planet.axialTilt

  • planet.axialTilt(planetname)

  • planet.classification

  • planet.classification(planetname)

  • planet.density

  • planet.density(planetname)

  • planet.discoveryYear

  • planet.discoveryYear(planetname)

  • planet.eccentricity

  • planet.eccentricity(planetname)

  • planet.equatorialGravity

  • planet.equatorialGravity(planetname)

  • planet.equatorialRadius

POKEMON

  • pokemon.location

  • pokemon.move

  • pokemon.name

  • pokemon.type

PRINCESSBRIDE

  • princessBride.character

  • princessBride.quote

PROGRAMMINGLANGUAGE

  • programmingLanguage.creator

  • programmingLanguage.name

PRONOUNS

  • pronouns.objective

  • pronouns.possessive

  • pronouns.reflexive

  • pronouns.subjective

RANDOM

  • random.hex

  • random.hex(int)

  • random.hex(int, boolean)

  • random.nextBoolean

  • random.nextDouble

  • random.nextDouble(double, double)

  • random.nextFloat

  • random.nextInt

  • random.nextInt(int)

  • random.nextInt(int, int)

  • random.nextInt(range)

  • random.nextLong

  • random.nextLong(long)

  • random.nextLong(long, long)

  • random.nextLong(range)

REDDEADREDEMPTION2

  • redDeadRedemption2.animal

  • redDeadRedemption2.gangMember

  • redDeadRedemption2.majorCharacter

  • redDeadRedemption2.protagonist

  • redDeadRedemption2.quote

  • redDeadRedemption2.region

  • redDeadRedemption2.state

  • redDeadRedemption2.weapon

RELATIONSHIPS

  • relationships.any

  • relationships.direct

  • relationships.extended

  • relationships.inLaw

  • relationships.parent

  • relationships.sibling

  • relationships.spouse

RESIDENTEVIL

  • residentEvil.biologicalAgent

  • residentEvil.character

  • residentEvil.creature

  • residentEvil.equipment

  • residentEvil.location

RESTAURANT

  • restaurant.description

  • restaurant.name

  • restaurant.namePrefix

  • restaurant.nameSuffix

  • restaurant.review

  • restaurant.type

RICKANDMORTY

  • rickAndMorty.character

  • rickAndMorty.location

  • rickAndMorty.quote

ROBIN

  • robin.quote

ROCKBAND

  • rockBand.name

RUPAULDRAGRACE

  • ruPaulDragRace.queen

  • ruPaulDragRace.quote

SCIENCE

  • science.bosons

  • science.element

  • science.elementSymbol

  • science.leptons

  • science.quark

  • science.scientist

  • science.tool

  • science.unit

SEINFELD

  • seinfeld.business

  • seinfeld.character

  • seinfeld.quote

SHAKESPEARE

  • shakespeare.asYouLikeItQuote

  • shakespeare.hamletQuote

  • shakespeare.kingRichardIIIQuote

  • shakespeare.romeoAndJulietQuote

SHOW

  • show.adultMusical

  • show.kidsMusical

  • show.play

SILICONVALLEY

  • siliconValley.app

  • siliconValley.character

  • siliconValley.company

  • siliconValley.email

  • siliconValley.invention

  • siliconValley.motto

  • siliconValley.quote

  • siliconValley.url

SIMPSONS

  • simpsons.character

  • simpsons.location

  • simpsons.quote

SIP

  • sip.bodyBytes

  • sip.bodyString

  • sip.clientErrorResponseCode

  • sip.clientErrorResponsePhrase

  • sip.contentType

  • sip.globalErrorResponseCode

  • sip.globalErrorResponsePhrase

  • sip.messagingPort

  • sip.method

  • sip.nameAddress

  • sip.provisionalResponseCode

  • sip.provisionalResponsePhrase

  • sip.redirectResponseCode

  • sip.redirectResponsePhrase

  • sip.rtpPort

SIZE

  • size.adjective

SLACKEMOJI

  • slackEmoji.activity

  • slackEmoji.celebration

  • slackEmoji.custom

  • slackEmoji.emoji

  • slackEmoji.foodAndDrink

  • slackEmoji.nature

  • slackEmoji.objectsAndSymbols

  • slackEmoji.people

  • slackEmoji.travelAndPlaces

SONICTHEHEDGEHOG

  • sonicTheHedgehog.character

  • sonicTheHedgehog.game

  • sonicTheHedgehog.zone

SOULKNIGHT

  • soulKnight.bosses

  • soulKnight.buffs

  • soulKnight.characters

  • soulKnight.enemies

  • soulKnight.statues

  • soulKnight.weapons

SOUTHPARK

  • southPark.characters

  • southPark.quotes

SPACE

  • space.agency

  • space.agencyAbbreviation

  • space.company

  • space.constellation

  • space.distanceMeasurement

  • space.galaxy

  • space.meteorite

  • space.moon

  • space.nasaSpaceCraft

  • space.nebula

  • space.planet

  • space.star

  • space.starCluster

SPONGEBOB

  • spongebob.characters

  • spongebob.episodes

  • spongebob.quotes

STARCRAFT

  • starCraft.building

  • starCraft.character

  • starCraft.planet

  • starCraft.unit

STARTREK

  • starTrek.character

  • starTrek.klingon

  • starTrek.location

  • starTrek.species

  • starTrek.villain

STARWARS

  • starWars.alternateCharacterSpelling

  • starWars.callSign

  • starWars.character

  • starWars.droids

  • starWars.planets

  • starWars.quotes

  • starWars.species

  • starWars.vehicles

  • starWars.wookieWords

STARGATE

  • stargate.characters

  • stargate.planets

  • stargate.quotes

STOCK

  • stock.exchanges

  • stock.lseSymbol

  • stock.nsdqSymbol

  • stock.nseSymbol

  • stock.nyseSymbol

STRANGERTHINGS

  • strangerThings.character

  • strangerThings.quote

STREAM

  • stream.build

  • stream.build

  • stream.faker(baseproviders)

  • stream.generate

  • stream.len(int)

  • stream.len(int, int)

  • stream.maxLen(int)

  • stream.minLen(int)

  • stream.nullRate(double)

  • stream.suppliers(Supplier[])

STREETFIGHTER

  • streetFighter.characters

  • streetFighter.moves

  • streetFighter.quotes

  • streetFighter.stages

STUDIOGHIBLI

  • studioGhibli.character

  • studioGhibli.movie

  • studioGhibli.quote

SUBSCRIPTION

  • subscription.paymentMethods

  • subscription.paymentTerms

  • subscription.plans

  • subscription.statuses

  • subscription.subscriptionTerms

SUITS

  • suits.characters

  • suits.quotes

SUPERMARIO

  • superMario.characters

  • superMario.games

  • superMario.locations

SUPERSMASHBROS

  • superSmashBros.fighter

  • superSmashBros.stage

SUPERHERO

  • superhero.descriptor

  • superhero.name

  • superhero.power

  • superhero.prefix

  • superhero.suffix

SUPERNATURAL

  • supernatural.character

  • supernatural.creature

  • supernatural.weapon

SWORDARTONLINE

  • swordArtOnline.gameName

  • swordArtOnline.item

  • swordArtOnline.location

  • swordArtOnline.realName

TEA

  • tea.type

  • tea.variety

TEAM

  • team.creature

  • team.name

  • team.sport

  • team.state

TEXT

  • text.character

  • text.lowercaseCharacter

  • text.text

  • text.text(boolean)

  • text.text(int)

  • text.text(int, int)

  • text.text(int, int, boolean)

  • text.text(int, int, boolean, boolean)

  • text.text(int, int, boolean, boolean, boolean)

  • text.text(textruleconfig)

  • text.uppercaseCharacter

THEEXPANSE

  • theExpanse.characters

  • theExpanse.locations

  • theExpanse.quotes

  • theExpanse.ships

THEITCROWD

  • theItCrowd.actors

  • theItCrowd.characters

  • theItCrowd.emails

  • theItCrowd.quotes

THEKINGKILLERCHRONICLE

  • theKingkillerChronicle.book

  • theKingkillerChronicle.character

  • theKingkillerChronicle.creature

  • theKingkillerChronicle.location

THEROOM

  • theRoom.actors

  • theRoom.characters

  • theRoom.locations

  • theRoom.quotes

THETHICKOFIT

  • theThickOfIt.characters

  • theThickOfIt.departments

  • theThickOfIt.positions

THEVENTUREBROS

  • theVentureBros.character

  • theVentureBros.organization

  • theVentureBros.quote

  • theVentureBros.vehicle

TIME

  • time.between(localtime, localtime)

  • time.between(localtime, localtime, string)

  • time.future(int, chronounit)

  • time.future(int, chronounit, string)

  • time.future(int, int, chronounit)

  • time.future(int, int, chronounit, string)

  • time.past(int, chronounit)

  • time.past(int, chronounit, string)

  • time.past(int, int, chronounit)

  • time.past(int, int, chronounit, string)

TIMEANDDATE

  • timeAndDate.between(instant, instant)

  • timeAndDate.between(instant, instant, string)

  • timeAndDate.birthday

  • timeAndDate.birthday(int, int)

  • timeAndDate.birthday(int, int, string)

  • timeAndDate.birthday(string)

  • timeAndDate.duration(long, chronounit)

  • timeAndDate.duration(long, long, chronounit)

  • timeAndDate.future

  • timeAndDate.future(long, long, timeunit)

  • timeAndDate.future(long, long, timeunit, string)

  • timeAndDate.future(long, timeunit)

  • timeAndDate.future(long, timeunit, instant)

  • timeAndDate.future(long, timeunit, instant, string)

  • timeAndDate.future(long, timeunit, string)

TIRE

  • tire.aspectRatio

  • tire.code

  • tire.code(boolean)

  • tire.code(string)

  • tire.construction

  • tire.loadIndex

  • tire.rimSize

  • tire.speedrating

  • tire.vehicleType

  • tire.width

TOUHOU

  • touhou.characterFirstName

  • touhou.characterLastName

  • touhou.characterName

  • touhou.gameName

  • touhou.trackName

TRANSPORT

  • transport.type

TRON

  • tron.alternateCharacterSpelling

  • tron.alternateCharacterSpelling(alternatecharacterspelling)

  • tron.character

  • tron.character(character)

  • tron.game

  • tron.location

  • tron.quote

  • tron.quote(quote)

  • tron.tagline

  • tron.vehicle

TWINPEAKS

  • twinPeaks.character

  • twinPeaks.location

  • twinPeaks.quote

TWITTER

  • twitter.createdTime(boolean, date, date)

  • twitter.text(String[], int, int)

  • twitter.twitterId(int)

  • twitter.userId

  • twitter.userName

UNIQUE

  • unique.fetchFromYaml(string)

UNIVERSITY

  • university.degree

  • university.name

  • university.place

  • university.prefix

  • university.suffix

VFORVENDETTA

  • vForVendetta.characters

  • vForVendetta.quotes

  • vForVendetta.speeches

VEHICLE

  • vehicle.carOptions

  • vehicle.carOptions(int, int)

  • vehicle.carType

  • vehicle.color

  • vehicle.doors

  • vehicle.driveType

  • vehicle.engine

  • vehicle.fuelType

  • vehicle.licensePlate

  • vehicle.licensePlate(string)

  • vehicle.make

  • vehicle.makeAndModel

  • vehicle.manufacturer

  • vehicle.model

  • vehicle.model(string)

VERB

  • verb.base

  • verb.ingForm

  • verb.past

  • verb.pastParticiple

  • verb.simplePresent

VIDEOGAME

  • videoGame.genre

  • videoGame.platform

  • videoGame.title

VOLLEYBALL

  • volleyball.coach

  • volleyball.formation

  • volleyball.player

  • volleyball.position

  • volleyball.team

WARHAMMERFANTASY

  • warhammerFantasy.creatures

  • warhammerFantasy.factions

  • warhammerFantasy.heros

  • warhammerFantasy.locations

  • warhammerFantasy.quotes

WEATHER

  • weather.description

  • weather.temperatureCelsius

  • weather.temperatureCelsius(int, int)

  • weather.temperatureFahrenheit

  • weather.temperatureFahrenheit(int, int)

WITCHER

  • witcher.book

  • witcher.character

  • witcher.location

  • witcher.monster

  • witcher.potion

  • witcher.quote

  • witcher.school

  • witcher.sign

  • witcher.witcher

WORD

  • word.adjective

  • word.adverb

  • word.conjunction

  • word.interjection

  • word.noun

  • word.preposition

  • word.verb

WORLDOFWARCRAFT

  • worldOfWarcraft.hero

  • worldOfWarcraft.quotes

YODA

  • yoda.quote

ZELDA

  • zelda.character

  • zelda.game

ZODIAC

  • zodiac.sign

Special Functions

  • numerify('#') - Replace # with random digits

  • letterify('???') - Replace ? with random letters

  • bothify('##??') - Replace # with digits, ? with letters

  • regexify('[a-z]{4,8}') - Generate text matching regex pattern

Context Variables

  • index - Current item index (0-based)

  • thread.name - Current thread name

  • thread.id - Current thread ID

Examples
riotx faker id="numerify('##########')" firstName="name.firstName" \
            lastName="name.lastName" email="internet.emailAddress" \
            age="number.numberBetween(18,99)" \
            hset person:#{id}
riotx faker ip="number.digits(4)" lease="number.digits(2)" \
            time="number.digits(5)" \
            zadd leases:#{ip} --member "#{lease}" --score=time

For complete documentation, visit: https://www.datafaker.net/documentation/providers/

Databases

RIOT-X includes two commands for interaction with relational databases:

db-import

Import database tables into Redis

db-export

Export Redis data structures to a database

Drivers

RIOT-X relies on JDBC to interact with databases. It includes JDBC drivers for the most common database systems:

Db2

jdbc:db2://host:port/database

MySQL

jdbc:mysql://[host]:[port][/database][?properties]

Oracle

jdbc:oracle:thin:@myhost:1521:orcl

Postgres

jdbc:postgresql://host:port/database

Snowflake

jdbc:snowflake://<account_identifier>.snowflakecomputing.com/?<connection_params>

SQL Server

jdbc:sqlserver://[serverName[\instanceName][:portNumber]][;property=value[;property=value]]

For non-included databases, place the JDBC driver jar under the lib directory

Database Import

The db-import command imports data from a relational database into Redis.

Ensure RIOT-X has the relevant JDBC driver for your database. See the Drivers section for more details. 
riotx db-import --jdbc-url <jdbc url> -u <Redis URI> SQL [REDIS COMMAND...]

To show the full usage, run:

riotx db-import --help

You must specify at least one Redis command as a target.

Redis connection options apply to the root command (riotx) and not to subcommands.

The keys that will be written are constructed from input records by concatenating the keyspace prefix and key fields.

mapping
PostgreSQL Import Example
riotx db-import "SELECT * FROM orders" --jdbc-url "jdbc:postgresql://host:port/database" --jdbc-user appuser --jdbc-pass passwd hset order:#{order_id}
Import from PostgreSQL to JSON strings
riotx db-import "SELECT * FROM orders" --jdbc-url "jdbc:postgresql://host:port/database" --jdbc-user appuser --jdbc-pass passwd set order:#{order_id}

This will produce Redis strings that look like this:

{
  "order_id": 10248,
  "customer_id": "VINET",
  "employee_id": 5,
  "order_date": "1996-07-04",
  "required_date": "1996-08-01",
  "shipped_date": "1996-07-16",
  "ship_via": 3,
  "freight": 32.38,
  "ship_name": "Vins et alcools Chevalier",
  "ship_address": "59 rue de l'Abbaye",
  "ship_city": "Reims",
  "ship_postal_code": "51100",
  "ship_country": "France"
}

Snowflake Import

The snowflake-import command uses a Snowflake STREAM object to track changes (CDC) to a table and read them into a Redis data structure like hash or json. The Snowflake STREAM is created and managed by RIOTX. The user credentials you provide must have the ability to create a stream in the database and schema specified by the fully qualified object name.

RIOT-X optimizes stream polling by using Snowflake’s SYSTEM$STREAM_HAS_DATA function to check if the stream contains new data before creating expensive temporary tables. This reduces unnecessary processing and improves performance when no changes are available.

Side effects and limitations

  • SAMPLE_DATABASE.SAMPLE_SCHEMA.DATA_TABLE_changestream will be created or replaced. For security, this can be created in a different schema than the table you are importing from by specifying --cdc-schema.

  • riotx:offset:SAMPLE_DATABASE.SAMPLE_SCHEMA.DATA_TABLE_changestream - this key will be stored in the destination Redis database and is used to track the stream offset. If RIOT-X fails in the middle of copying data from the stream when restarted it will resume copying data from this offset. Removing this offset key from Redis will result in RIOT-X creating recreating the stream at time "NOW". With --snapshot INITIAL (default) the stream will include the initial table data plus changes going forward. If you do not want initial table data to be included specify --snapshot NEVER.

  • snowflake-import currently works on tables and materialized views

The basic usage is:

riotx snowflake-import [TABLE] [OPTIONS] [REDIS COMMAND...]

The recommended minimal necessary permissions for a snowflake role and user to run this command are:

CREATE OR REPLACE ROLE riotx_cdc
COMMENT = 'minimum cdc role for riotx';

-- replace compute_wh with the name of the warehouse you want to use
GRANT USAGE, OPERATE ON WAREHOUSE compute_wh TO ROLE riotx_cdc;

-- replace tb_101.raw_pos_cdc with the name of a database and schema for RIOT-X to create the stream in
CREATE OR REPLACE SCHEMA tb_101.raw_pos_cdc;
GRANT USAGE ON SCHEMA tb_101.raw_pos_cdc TO ROLE riotx_cdc;

-- replace tb_101 with the name of the database RIOT-X needs to read out of
GRANT USAGE ON DATABASE tb_101 TO ROLE riotx_cdc;

-- replace tb_101.raw_pos with the name of the schema RIOT-X needs to read out of
GRANT USAGE ON SCHEMA tb_101.raw_pos TO ROLE riotx_cdc;

-- replace with the name of the table(s) you want to read from
GRANT SELECT ON TABLE tb_101.raw_pos.incremental_order_header TO ROLE riotx_cdc;
GRANT REFERENCE_USAGE ON TABLE tb_101.raw_pos.incremental_order_header TO ROLE riotx_cdc;
ALTER TABLE tb_101.raw_pos.INCREMENTAL_ORDER_HEADER SET CHANGE_TRACKING = TRUE;

GRANT SELECT ON FUTURE TABLES IN SCHEMA tb_101.raw_pos_cdc TO ROLE riotx_cdc;
GRANT CREATE TABLE ON SCHEMA tb_101.raw_pos_cdc TO ROLE riotx_cdc;
GRANT CREATE STREAM ON SCHEMA tb_101.raw_pos_cdc TO ROLE riotx_cdc;
GRANT SELECT ON FUTURE STREAMS IN SCHEMA tb_101.raw_pos_cdc TO ROLE riotx_cdc;

CREATE OR REPLACE USER riotx_cdc
    DEFAULT_ROLE = 'riotx_cdc'
    DEFAULT_WAREHOUSE = 'compute_wh'
    PASSWORD = '{{PASSWORD}}';

GRANT ROLE riotx_cdc TO USER riotx_cdc;

For the full usage, run:

riotx snowflake-import --help
Example: CDC to Hashes

This command uses the example db, schema and table names from the minimal role setup above.

riotx snowflake-import \
      tb_101.raw_pos.incremental_order_header \
      --role riotx_cdc \
      --warehouse compute_wh \
      --cdc-schema raw_pos_cdc \
      --jdbc-url "jdbc:snowflake://abcdefg.abc12345.snowflakecomputing.com" \
      --jdbc-user databaseuser \
      --jdbc-pass databasepassword \
      --poll 10s \ (1)
      hset orderheader:#{order_id} (2)
1 Sleep 10 seconds after each CDC import
2 Column name to use as id

The command above imports CDC data from the Snowflake table tb_101.raw_pos.incremental_order_header into Redis hashes in the keyspace orderheader.

You can specify multiple tables to import. If you need to reference table information in the operation you can use the #table variable which exposes name, database, and schema fields.
Import Multiple Tables
riotx snowflake-import \
      db1.public.table1 db1.public.table2 \ (1)
      ...
      hset #{#table.name}:{id} (2)
1 Specify all tables to import
2 Use table name as key prefix

This command would generate keys table1:abc, table2:def, etc.

If you only need to do a one time import of data from Snowflake you can use the db-import command. This command will read all of the rows output from your SQL query and will write them to Redis. For more information see the db-import command.

Example: One-time Import
riotx db-import \
      "SELECT * FROM SAMPLE_DATABASE.SAMPLE_SCHEMA.DATA_TABLE" \
      --jdbc-url "jdbc:snowflake://abcdefg.abc12345.snowflakecomputing.com" \
      --jdbc-driver net.snowflake.client.jdbc.SnowflakeDriver \
      --jdbc-user databaseuser \
      --jdbc-pass databasepassword \
      hset datatable:#{data_id} (1)
1 Column name to use as id

This command performs a one-time import from Snowflake using the db-import command.

RDI Integration

This recipe contains step-by-step instructions for using RIOT-X as an external collector for Redis Data Integration (RDI).

Here is the end-to-end data flow:

Snowflake CDC Architecture
Snowflake

In your Snowflake UI (Snowsight) import this notebook: snowflake-cdc.ipynb. Run the first 2 steps in the notebook:

init

Set up roles, permissions and schema
populate

Create and populate table
Do not run the other steps yet. These are for later.
RDI

In Redis Insight use these pipeline config and job definitions:

Pipeline Config
sources:
  riotx:
    type: external
    # Redis Insight requires a connection element but it's not actually used by RDI
    connection: {}

targets:
  target:
    connection:
      type: redis
      host: # Target database hostname
      port: # Target database port
processors:
Job Config
source:
  table: incremental_order_header

output:
  - uses: redis.write
    with:
      connection: target
      data_type: hash
      key:
        expression: concat(['order:', ORDER_ID])
        language: jmespath
RIOT-X

Run this command to start RIOT-X:

riotx snowflake-import -h <target_host> -p <target_port> tb_101.raw_pos.incremental_order_header --role riotx_cdc --warehouse compute_wh --cdc-schema raw_pos_cdc --jdbc-url "jdbc:snowflake://<account>.snowflakecomputing.com" --jdbc-user $JDBC_USER --jdbc-pass $JDBC_PASS

RIOT-X will perform initial load of the incremental_order_header table which should translate to 1000 hashes in the target Redis database.

Snowflake Additional Data

Go back to your Snowflake UI and run the last step in the notebook. This will insert 100 rows in the incremental_order_header table which will be picked up by RIOT and written to the target database. You should now have 1,100 hashes in the target Redis database.

Service User

If your Snowflake account does not let you use JDBC password authentication you will have to use key-based authentication.

Key Pair

If you don’t already have a private/public key pair you need to generate one. Follow the steps at docs.snowflake.com/en/user-guide/key-pair-auth.

Use the following command to extract the actual public key for the last step: Assign the public key to a Snowflake user

grep -v "BEGIN\|END" rsa_key.pub | tr -d '\n'
JDBC URL

When using key authentication you need to modify the JDBC URL to:

jdbc:snowflake://<account>.snowflakecomputing.com?private_key_file=<path_to_key>/rsa_key.p8

Kubernetes Deployment

RIOT-X can be deployed on Kubernetes for production workloads using the provided deployment configuration.

Prerequisites

Before deploying, ensure you have:

  • A Kubernetes cluster with access to create deployments and secrets

  • Redis instance accessible from the cluster

  • Snowflake credentials and proper role permissions (see above)

  • Docker image riotx/riotx:latest available in your cluster

Configuration

The Kubernetes deployment configuration uses environment variables following the RIOT_REDIS_* and RIOT_* naming conventions.

Download the deployment configuration:

curl -O https://redis.github.io/riotx/snowflake-import-deployment.yaml

Or use the local file: snowflake-import-deployment.yaml

Secrets Setup

Before deploying the deployment, create the required secrets with your actual credentials:

# Create Redis credentials secret
kubectl create secret generic redis-credentials \
  --from-literal=username='redis_user' \
  --from-literal=password='redis_password'

# Create Snowflake credentials secret
kubectl create secret generic snowflake-credentials \
  --from-literal=jdbc-url='jdbc:snowflake://company.snowflakecomputing.com' \
  --from-literal=username='snowflake_user' \
  --from-literal=password='snowflake_password'

For key-based authentication, include the private key file path in the JDBC URL:

kubectl create secret generic snowflake-credentials \
  --from-literal=jdbc-url='jdbc:snowflake://company.snowflakecomputing.com?private_key_file=/path/to/key.p8' \
  --from-literal=username='snowflake_user' \
  --from-literal=password=''
Customization

Edit the deployment YAML to customize for your environment:

# Update Redis connection
- name: RIOT_REDIS_URI
  value: "rediss://your-redis-host:12000"  (1)

# Update Snowflake table and settings
- name: RIOT_TABLE
  value: "YOUR_DB.YOUR_SCHEMA.YOUR_TABLE"  (2)
- name: RIOT_ROLE
  value: "your_snowflake_role"  (3)
- name: RIOT_WAREHOUSE
  value: "your_warehouse"  (4)
- name: RIOT_CDC_SCHEMA
  value: "your_cdc_schema"  (5)
- name: RIOT_STREAM_LIMIT
  value: "20000"  (6)
1 Your Redis server URI (TLS enabled)
2 Fully qualified Snowflake table name
3 Snowflake role with CDC permissions
4 Snowflake warehouse to use
5 CDC schema for stream creation
6 Stream processing limit
Deployment

Deploy the application:

kubectl apply -f snowflake-import-deployment.yaml

Monitor the deployment:

# Check deployment status
kubectl get deployments -l app=riotx

# Check pod status
kubectl get pods -l app=riotx

# View logs
kubectl logs -f deployment/riotx-snowflake-import

# Check metrics (if enabled)
kubectl port-forward deployment/riotx-snowflake-import 8080:8080
curl http://localhost:8080/metrics
Production Considerations

For production deployments, consider:

  • Resource Limits: Adjust CPU and memory based on your workload

  • Persistent Storage: Mount volumes for key files if using key-based auth

  • Monitoring: Enable metrics and configure alerting

  • Security: Use proper RBAC and network policies

  • High Availability: The deployment ensures pods are restarted if they fail

  • Auto-scaling: Configure HPA based on metrics if needed

  • Replica Management: Keep replicas=1 for CDC consistency to avoid duplicate processing

Deployment Features

The snowflake-import-deployment.yaml includes:

  • Deployment: Ensures reliable pod management and restart policies

  • Service: Exposes metrics endpoint for monitoring

  • ServiceMonitor: Prometheus integration for metrics collection

  • Secrets: Secure credential management

  • ConfigMap: Non-sensitive configuration management

  • Health Checks: Liveness and readiness probes

  • Init Containers: Dependency checks before startup

Database Export

Use the db-export command to read from a Redis database and writes to a SQL database.

Ensure RIOT-X has the relevant JDBC driver for your database. See the Drivers section for more details.

The general usage is:

riotx db-export --jdbc-url <jdbc url> SQL

To show the full usage, run:

riotx db-export --help
Example: export to PostgreSQL
riotx db-export "INSERT INTO mytable (id, field1, field2) VALUES (CAST(:id AS SMALLINT), :field1, :field2)" --jdbc-url "jdbc:postgresql://host:port/database" --jdbc-user appuser --jdbc-pass passwd --key-pattern "gen:*" --key-regex "gen:(?<id>.*)"

Files

RIOT-X includes two commands to work with files in various formats:

file-import

Import data from files

file-export

Export Redis data structures to files

File Import

The file-import command reads data from files and writes it to Redis.

The basic usage for file imports is:

riotx file-import [OPTIONS] FILE... [REDIS COMMAND...]

To show the full usage, run:

riotx file-import --help

RIOT-X will try to determine the file type from its extension (e.g. .csv or .json), but you can specify it with the --type option.

Gzipped files are supported and the extension before .gz is used (e.g. myfile.json.gz → json).

Examples

  • /path/file.csv

  • /path/file-*.csv

  • /path/file.json

  • http://data.com/file.csv

  • http://data.com/file.json.gz

Use - to read from standard input.

Amazon S3 and Google Cloud Storage buckets are supported.

Importing from Amazon S3
riotx file-import s3://riotx/beers.json --s3-region us-west-1 hset beer:#{id}
Importing from Google Cloud Storage
riotx file-import gs://riotx/beers.json hset beer:#{id}

Data Structures

If no REDIS COMMAND is specified, it is assumed that the input file(s) contain Redis data structures serialized as JSON or XML. See the File Export section to learn about the expected format and how to generate such files.

Example
riotx file-import /tmp/redis.json

Redis Commands

When one or more `REDIS COMMAND`s are specified, these commands are called for each input record.

Redis client options apply to the root command (riotx) and not to Redis commands.

In this example Redis client options will not be taken into account:

riotx file-import my.json hset -h myredis.com -p 6380

Redis command keys are constructed from input records by concatenating keyspace prefix and key fields.

mapping
Import into hashes with key blah:<id>
riotx file-import my.json hset blah:#{id}
Import into JSON
riotx file-import http://storage.googleapis.com/jrx/es_test-index.json json.set elastic:#{_id}
Import into hashes and set TTL on the key
riotx file-import my.json hset blah:#{id} expire blah:#{id}
Import into hashes blah:<id> and set TTL and add each id to a set named myset
riotx file-import my.json hset blah:#{id} expire blah:#{id} sadd myset --member #{id}

Delimited (CSV)

The default delimiter character is comma (,). It can be changed with the --delimiter option.

If the file has a header, use the --header option to automatically extract field names. Otherwise specify the field names using the --fields option.

Let’s consider this CSV file:

Table 1. beers.csv
row abv ibu id name style brewery ounces

1

0.079

45

321

Fireside Chat (2010)

Winter Warmer

368

12.0

2

0.068

65

173

Back in Black

American Black Ale

368

12.0

3

0.083

35

11

Monk’s Blood

Belgian Dark Ale

368

12.0

The following command imports this CSV into Redis as hashes using beer as the key prefix and id as primary key.

riotx file-import http://storage.googleapis.com/jrx/beers.csv --header hset beer:#{id}

This creates hashes with keys beer:321, beer:173, …​

This command imports a CSV file into a geo set named airportgeo with airport IDs as members:

riotx file-import http://storage.googleapis.com/jrx/airports.csv --header --skip 3 geoadd airportgeo --member "#{AirportID}" --lon Longitude --lat Latitude

Fixed-Length (Fixed-Width)

Fixed-length files can be imported by specifying the width of each field using the --ranges option.

riotx file-import http://storage.googleapis.com/jrx/accounts.fw --type fw --ranges 1 9 25 41 53 67 83 --header hset account:#{Account}

JSON

The expected format for JSON files is:

[
  {
    "...": "..."
  },
  {
    "...": "..."
  }
]
JSON import example
riotx file-import /tmp/redis.json

JSON records are trees with potentially nested values that need to be flattened when the target is a Redis hash for example.

To that end, RIOT-X uses a field naming convention to flatten JSON objects and arrays:

Table 2. Nested object

{ "field": { "sub": "value" } }

→

field.sub=value
Table 3. Array

{ "field": [1, 2, 3] }

→

field[0]=1 field[1]=2 field[2]=3

XML

Here is a sample XML file that can be imported by RIOT-X:

<?xml version="1.0" encoding="UTF-8"?>
<records>
    <trade>
        <isin>XYZ0001</isin>
        <quantity>5</quantity>
        <price>11.39</price>
        <customer>Customer1</customer>
    </trade>
    <trade>
        <isin>XYZ0002</isin>
        <quantity>2</quantity>
        <price>72.99</price>
        <customer>Customer2c</customer>
    </trade>
    <trade>
        <isin>XYZ0003</isin>
        <quantity>9</quantity>
        <price>99.99</price>
        <customer>Customer3</customer>
    </trade>
</records>
XML Import Example
riotx file-import http://storage.googleapis.com/jrx/trades.xml hset trade:#{id}

Parquet

RIOT-X supports Parquet files.

Parquet file import example
riotx file-import s3://riotx/userdata1.parquet --s3-region us-west-1 hset user:#{id}

File Export

The file-export command reads data from a Redis database and writes it to a JSON or XML file, potentially gzip-compressed.

The general usage is:

riotx file-export [OPTIONS] FILE

To show the full usage, run:

riotx file-export --help

JSON

Export to JSON
riotx file-export /tmp/redis.json
Sample JSON-export file
[
  {
    "key": "string:615",
    "ttl": -1,
    "value": "value:615",
    "type": "STRING"
  },
  {
    "key": "hash:511",
    "ttl": -1,
    "value": {
      "field1": "value511",
      "field2": "value511"
    },
    "type": "HASH"
  },
  {
    "key": "list:1",
    "ttl": -1,
    "value": [
      "member:991",
      "member:981"
    ],
    "type": "LIST"
  },
  {
    "key": "set:2",
    "ttl": -1,
    "value": [
      "member:2",
      "member:3"
    ],
    "type": "SET"
  },
  {
    "key": "zset:0",
    "ttl": -1,
    "value": [
      {
        "value": "member:1",
        "score": 1.0
      }
    ],
    "type": "ZSET"
  },
  {
    "key": "stream:0",
    "ttl": -1,
    "value": [
      {
        "stream": "stream:0",
        "id": "1602190921109-0",
        "body": {
          "field1": "value0",
          "field2": "value0"
        }
      }
    ],
    "type": "STREAM"
  }
]
Export to compressed JSON
riotx file-export /tmp/beers.json.gz --key-pattern beer:*

XML

Export to XML
riotx file-export /tmp/redis.xml

Parquet

Parquet file export example
riotx file-export beers.parquet --parquet-field ounces=DOUBLE abv=DOUBLE id=INT32

Memcached Replication

The memcached-replicate command reads data from a source Memcached database and writes to a target Memcached database.

riotx memcached-replicate SOURCE TARGET [OPTIONS]

For the full usage, run:

riotx memcached-replicate --help
Example
riotx memcached-replicate mydb.cache.amazonaws.com:11211 mydb-12211.redis.com:12211 --source-tls

Redis Import

The redis-import command reads data from a Redis database and writes it to another Redis database.

The basic usage is:

riotx redis-import [OPTIONS] [REDIS COMMAND...]

For the full usage, run:

riotx redis-import --help
Example: migrate hashes to JSON
riotx redis-import --target-uri redis://localhost:6380 --key-pattern 'hash:*' --key-regex 'hash:(?<id>.+)' json.set doc:#{id}

Replication

The replicate command reads data from a source Redis database and writes to a target Redis database.

replication architecture

The replication mechanism is as follows:

  1. Identify source keys to be replicated using scan and/or keyspace notifications depending on the replication mode.

  2. Read data associated with each key using dump or type-specific commands.

  3. Write each key to the target using restore or type-specific commands.

The basic usage is:

riotx replicate [OPTIONS] SOURCE TARGET

where SOURCE and TARGET are Redis URIs.

For the full usage, run:

riotx replicate --help
To replicate a Redis logical database other than the default (0), specify the database in the source Redis URI. For example riotx replicate redis://source:6379/1 redis://target:6379 replicates database 1.

Redis Connections

RIOT-X supports multiple ways to connect to both source and target Redis databases, including various authentication methods and security configurations.

Connection Methods

Redis URI (Recommended)

The simplest way to specify Redis connections is using Redis URIs:

riotx replicate redis://source:6379 redis://target:6379

For secure connections:

riotx replicate rediss://source:6380 rediss://target:6380
Individual Connection Parameters

You can also specify connection details using individual options:

Source database options:

riotx replicate \
  --source-host source.example.com \
  --source-port 6379 \
  --source-user myuser \
  --source-pass mypassword \
  --target-uri rediss://target:6380

Target database options:

riotx replicate redis://source:6379 \
  --target-host target.example.com \
  --target-port 6380 \
  --target-tls \
  --target-user myuser \
  --target-pass mypassword

Authentication

RIOT-X supports multiple authentication methods for connecting to Redis databases.

Username/Password Authentication

For Redis ACL (Redis 6+) or basic AUTH:

Via URI:

riotx replicate redis://user:pass@source:6379 redis://user:pass@target:6379

Via options:

riotx replicate redis://source:6379 redis://target:6379 \
  --source-user myuser --source-pass mypass \
  --target-user myuser --target-pass mypass
Entra ID Authentication (Azure)

For Azure Cache for Redis Enterprise with Entra ID authentication:

Basic Entra ID (Managed Identity):

riotx replicate redis://source.redis.cache.windows.net:10000 \
                redis://target.redis.cache.windows.net:10000 \
  --source-entra \
  --target-entra

Entra ID with Client Credentials:

riotx replicate redis://source.redis.cache.windows.net:10000 \
                redis://target.redis.cache.windows.net:10000 \
  --source-entra \
  --source-entra-id "client-id" \
  --source-entra-secret "client-secret" \
  --target-entra \
  --target-entra-id "client-id" \
  --target-entra-secret "client-secret"

Advanced Entra ID Configuration:

riotx replicate redis://source.redis.cache.windows.net:10000 \
                redis://target.redis.cache.windows.net:10000 \
  --source-entra \
  --source-entra-auth "https://login.microsoftonline.com/tenant-id" \
  --source-entra-scope "https://redis.azure.com/.default" \
  --source-entra-refresh 0.75 \
  --target-entra

Environment Variables for Entra ID: You can also configure Entra ID authentication using environment variables:

export RIOT_SOURCE_ENTRA=true
export RIOT_SOURCE_ENTRA_ID=your-client-id
export RIOT_SOURCE_ENTRA_SECRET=your-client-secret
export RIOT_TARGET_ENTRA=true

{app} replicate redis://source.redis.cache.windows.net:10000 \
                redis://target.redis.cache.windows.net:10000
Certificate-Based Authentication

Client Certificate Authentication:

riotx replicate rediss://source:6380 rediss://target:6380 \
  --source-cert /path/to/client.crt \
  --source-key /path/to/client.key \
  --target-cert /path/to/client.crt \
  --target-key /path/to/client.key

Custom CA Certificate:

riotx replicate rediss://source:6380 rediss://target:6380 \
  --source-cacert /path/to/ca.crt \
  --target-cacert /path/to/ca.crt

TLS/SSL Configuration

Basic TLS

Enable TLS:

riotx replicate redis://source:6379 redis://target:6379 \
  --source-tls --target-tls

Or use TLS URIs:

riotx replicate rediss://source:6380 rediss://target:6380
TLS Verification Modes

Control how TLS certificates are verified:

riotx replicate rediss://source:6380 rediss://target:6380 \
  --source-verify FULL \
  --target-verify NONE

Available verification modes: * FULL - Verify hostname and certificate chain (default, most secure) * CA - Verify certificate chain only * NONE - No certificate verification (insecure)

Insecure TLS (Development Only)
Only use this in development environments. 
riotx replicate rediss://source:6380 rediss://target:6380 \
  --source-insecure --target-insecure

Cluster Configuration

For Redis Cluster deployments:

riotx replicate redis://source-node1:7000,source-node2:7000,source-node3:7000 \
                redis://target-node1:7000,target-node2:7000,target-node3:7000 \
  --source-cluster --target-cluster

Or using individual options:

riotx replicate redis://source:7000 redis://target:7000 \
  --source-cluster --target-cluster

Connection Tuning

Connection Pool Size

Configure the number of connections in the pool:

riotx replicate redis://source:6379 redis://target:6379 \
  --source-pool 16 --target-pool 16
Read Preference (Cluster Mode)

For source clusters, specify where to read from:

riotx replicate redis://source:7000 redis://target:6379 \
  --source-cluster \
  --source-read-from REPLICA_PREFERRED

Available read preferences: * UPSTREAM - Read from master nodes (default) * UPSTREAM_PREFERRED - Prefer master, fallback to replica * REPLICA - Read from replica nodes only * REPLICA_PREFERRED - Prefer replica, fallback to master * LOWEST_LATENCY - Read from node with lowest latency * ANY - Read from any available node

Command Timeout

Set timeout for Redis commands:

riotx replicate redis://source:6379 redis://target:6379 \
  --source-timeout 30s --target-timeout 30s

Connection Examples

Redis Cloud to Redis Cloud
riotx replicate \
  redis://default:source-password@source-endpoint.c1.cloud.redislabs.com:12000 \
  redis://default:target-password@target-endpoint.c1.cloud.redislabs.com:12001
AWS ElastiCache to Azure Cache for Redis
riotx replicate \
  rediss://source.abc123.cache.amazonaws.com:6380 \
  redis://target.redis.cache.windows.net:10000 \
  --source-pass elasticache-auth-token \
  --target-entra
Self-hosted Redis to Redis Enterprise Cloud
riotx replicate \
  redis://localhost:6379 \
  rediss://redis-endpoint.c1.cloud.redislabs.com:12000 \
  --source-user myuser --source-pass mypass \
  --target-user default --target-pass cloud-password

Replication Mode

Replication starts with identifying keys to be replicated from the source Redis database. The --mode option allows you to specify how RIOT-X identifies keys to be replicated:

  • iterate over keys with a key scan (--mode scan)

  • received by a keyspace notification subscriber (--mode liveonly)

  • or both (--mode live)

Scan

This key reader scans for keys using the Redis SCAN command:

SCAN cursor [MATCH pattern] [COUNT count] [TYPE type]
MATCH pattern

configured with the --key-pattern option

TYPE type

configured with the --key-type option

COUNT count

configured with the --scan-count option

INFO: In cluster mode keys are scanned in parallel across cluster nodes.

The status bar shows progress with a percentage of keys that have been replicated. The total number of keys is estimated when the replication process starts and it can change by the time it is finished, for example if keys are deleted or added during replication.

Scan replication example
riotx replicate redis://source redis://target

Live

The key notification reader listens for key changes using keyspace notifications.

Make sure the source database has keyspace notifications enabled using:

  • redis.conf: notify-keyspace-events = KEA

  • CONFIG SET notify-keyspace-events KEA

For more details see Redis Keyspace Notifications.

Live replication example
riotx replicate --mode live redis://source redis://target

The live replication mechanism does not guarantee data consistency. Redis sends keyspace notifications over pub/sub which does not provide guaranteed delivery. It is possible that RIOT-X can miss some notifications in case of network failures for example.

Also, depending on the type, size, and rate of change of data structures on the source it is possible that RIOT-X cannot keep up with the change stream. For example if a big set is repeatedly updated, RIOT-X will need to read the whole set on each update and transfer it over to the target database.

For those potentially problematic migrations it is recommend to perform some preliminary sizing using Redis statistics and bigkeys/memkeys in tandem with --mem-limit. If you need assistance please contact your Redis account team.

Replication Types

RIOT-X offers two different mechanisms for reading and writing keys:

  • Dump & restore (default)

  • Data structure replication (--struct)

Dump & Restore

The default replication mechanism is Dump & Restore:

  1. Scan for keys in the source Redis database. If live replication is enabled the reader also subscribes to keyspace notifications to generate a continuous stream of keys.

  2. Reader threads iterate over the keys to read corresponding values (DUMP) and TTLs.

  3. Reader threads enqueue key/value/TTL tuples into the reader queue, from which the writer dequeues key/value/TTL tuples and writes them to the target Redis database by calling RESTORE and EXPIRE.

Data Structure Replication

There are situations where Dump & Restore cannot be used, for example:

  • The target Redis database does not support the RESTORE command (Redis Enterprise CRDB)

  • Incompatible DUMP formats between source and target (Redis 7.0)

In those cases you can use another replication strategy that is data structure-specific: each key is introspected to determine its type and then use the corresponding read/write commands.

Type Read Write

Hash

HGETALL

HSET

JSON

JSON.GET

JSON.SET

List

LRANGE

RPUSH

Set

SMEMBERS

SADD

Sorted Set

ZRANGE

ZADD

Stream

XRANGE

XADD

String

GET

SET

TimeSeries

TS.RANGE

TS.ADD
This replication strategy is more intensive in terms of CPU, memory, and network for all the machines involved (source Redis, target Redis, and RIOT-X machines). Adjust number of threads, batch and queue sizes accordingly.
Type-based replication example
riotx replicate --struct redis://source redis://target
Live type-based replication example
riotx replicate --struct --mode live redis://source redis://target

Compare

Once replication is complete, RIOT-X performs a verification step by reading keys in the source database and comparing them against the target database.

The verification step happens automatically after the scan is complete (snapshot replication), or for live replication when keyspace notifications have become idle.

Verification can also be run on-demand using the compare command:

riotx compare SOURCE TARGET [OPTIONS]

The output looks like this:

Verification failed (type: 225,062, missing: 485,450)
missing

Number of keys in source but not in target.

type

Number of keys with mismatched types (e.g. hash vs string).

value

Number of keys with mismatched values.

ttl

Number of keys with mismatched TTL i.e. difference is greater than tolerance (can be specified with --ttl-tolerance).

There are 2 comparison modes available through --compare (--quick for compare command):

Quick (default)

Compare key types and TTLs.

Full

Compare key types, TTLs, and values.

To show which keys differ, use the --show-diffs option.

Performance

Performance tuning is an art but RIOT-X offers some options to identify potential bottlenecks. In addition to --batch and --threads options you have the --dry-run option which disables writing to the target Redis database so that you can tune the reader in isolation. Add that option to your existing replicate command-line to compare replication speeds with and without writing to the target Redis database.

Stats

The stats command analyzes the Redis database and displays keyspace statistics as well as keys that could be problematic during a live replication.

The basic usage is:

riotx stats [OPTIONS]

For the full usage, run:

riotx stats --help
--mem <size>

Memory usage threshold above which a key is considered big

--byterate <size>

Write bandwidth above which a key is considered problematic.

Example: Keys over 3MB in memory usage and 10MB in write rate
riotx stats --mem 3mb --byterate 10mb

Stream

Stream Import

The stream-import command reads data from a stream and writes it to Redis.

The basic usage is:

riotx stream-import STREAM...

For the full usage, run:

riotx stream-import --help
Example: Import stream into hashes
riotx stream-import stream:beers --idle-timeout 1s hset beer:#{id}

Stream Export

The stream-export command enables Redis CDC to a Redis stream.

riotx stream-export SOURCE TARGET [OPTIONS]

For the full usage, run:

riotx stream-export --help
Example: Export stream to another Redis instance
riotx stream-export redis://localhost:6379 redis://localhost:6380 --mode live

redis-cli -p 6380 xread COUNT 3 STREAMS stream:export 0-0
1) 1) "stream:export"
   2) 1) 1) "1718645537588-0"
         2)  1) "key"
             2) "order:4"
             3) "time"
             4) "1718645537000"
             5) "type"
             6) "hash"
             7) "ttl"
             8) "-1"
             9) "mem"
            10) "136"
            11) "value"
            12) "{\"order_date\":\"2024-06-13 22:19:35.143797\",\"order_id\":\"4\"}"

Cookbook

Here are various recipes using RIOT-X.

Observability

RIOT-X exposes several metrics over a Prometheus endpoint that can be useful for troubleshooting and performance tuning.

Getting Started

The riotx-dist repository includes a Docker compose configuration that set ups Prometheus and Grafana.

git clone https://github.com/redis/riotx-dist.git
cd riotx-dist
docker compose up

Prometheus is configured to scrape the host every second.

You can access the Grafana dashboard at localhost:3000.

Now start RIOT-X with the following command:

riotx replicate ... --metrics

This will enable the Prometheus metrics exporter endpoint and will populate the Grafana dashboard.

Configuration

Use the --metrics* options to enable and configure metrics:

--metrics

Enable metrics

--metrics-jvm

Enable JVM and system metrics

--metrics-redis

Enable command latency metrics. See https://github.com/redis/lettuce/wiki/Command-Latency-Metrics#micrometer

--metrics-name=<name>

Application name tag that will be applied to all metrics

--metrics-port=<int>

Port that Prometheus HTTP server should listen on (default: 8080)

--metrics-prop=<k=v>

Additional properties to pass to the Prometheus client. See https://prometheus.github.io/client_java/config/config/

Metrics

Below you can find a list of all metrics declared by RIOT-X.

riotx dashboard replication
Replication Metrics
Name Type Description

riotx_replication_bytes_total

Counter

Number of bytes replicated (needs memory usage with --mem-limit)

riotx_replication_lag_seconds

Summary

Replication end-to-end latency

riotx_replication_read_latency_seconds

Summary

Replication read latency

spring_batch_chunk_write_seconds

Timer

Batch writing duration

spring_batch_item_process_seconds

Timer

Item processing duration

spring_batch_item_read_seconds

Timer

Item reading duration

spring_batch_job_active_seconds

Timer

Active jobs

spring_batch_job_launch_count_total

Counter

Job launch count

spring_batch_redis_key_event_queue_capacity

Gauge

Gauge reflecting the remaining capacity of the queue

spring_batch_redis_key_event_queue_size

Gauge

Gauge reflecting the size (depth) of the queue

spring_batch_redis_key_scan_total

Counter

Number of keys scanned

spring_batch_redis_operation_seconds

Timer

Operation execution duration

spring_batch_redis_read_chunk

Gauge

Gauge reflecting the chunk size of the reader

spring_batch_redis_read_queue_capacity

Gauge

Gauge reflecting the remaining capacity of the queue

spring_batch_redis_read_queue_size

Gauge

Gauge reflecting the size (depth) of the queue
JVM Metrics

Use the --metrics-jvm option to enable the following additional metrics:

riotx dashboard jvm
Name Type Description

jvm_buffer_count_buffers

Gauge

An estimate of the number of buffers in the pool

jvm_buffer_memory_used_bytes

Gauge

An estimate of the memory that the Java virtual machine is using for this buffer pool

jvm_buffer_total_capacity_bytes

Gauge

An estimate of the total capacity of the buffers in this pool

jvm_gc_concurrent_phase_time_seconds

Timer

Time spent in concurrent phase

jvm_gc_live_data_size_bytes

Gauge

Size of long-lived heap memory pool after reclamation

jvm_gc_max_data_size_bytes

Gauge

Max size of long-lived heap memory pool

jvm_gc_memory_allocated_bytes_total

Gauge

Incremented for an increase in the size of the (young) heap memory pool after one GC to before the next

jvm_gc_memory_promoted_bytes_total

Counter

Count of positive increases in the size of the old generation memory pool before GC to after GC

jvm_gc_pause_seconds

Timer

Time spent in GC pause

jvm_memory_committed_bytes

Gauge

The amount of memory in bytes that is committed for the Java virtual machine to use

jvm_memory_max_bytes

Gauge

The maximum amount of memory in bytes that can be used for memory management

jvm_memory_used_bytes

Gauge

The amount of used memory

jvm_threads_daemon_threads

Gauge

The current number of live daemon threads

jvm_threads_live_threads

Gauge

The current number of live threads including both daemon and non-daemon threads

jvm_threads_peak_threads

Gauge

The peak live thread count since the Java virtual machine started or peak was reset

jvm_threads_started_threads_total

Counter

The total number of application threads started in the JVM

jvm_threads_states_threads

Gauge

The current number of threads

process_cpu_time_ns_total

Counter

The "cpu time" used by the Java Virtual Machine process

process_cpu_usage

Gauge

The "recent cpu usage" for the Java Virtual Machine process

process_start_time_seconds

Gauge

Start time of the process since unix epoch.

process_uptime_seconds

Gauge

The uptime of the Java virtual machine

system_cpu_count

Gauge

The number of processors available to the Java virtual machine

system_cpu_usage

Gauge

The "recent cpu usage" of the system the application is running in

system_load_average_1m

Gauge

The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time

Changelog

You can use RIOT-X to stream change data from a Redis database.

Streaming to stdout
riotx file-export --mode live
{"key":"gen:1","type":"string","time":1718050552000,"ttl":-1,"memoryUsage":300003376}
{"key":"gen:3","type":"string","time":1718050552000,"ttl":-1,"memoryUsage":300003376}
{"key":"gen:6","type":"string","time":1718050552000,"ttl":-1,"memoryUsage":300003376}
...
Streaming to a file
riotx file-export export.json --mode live

ElastiCache Migration

This recipe contains step-by-step instructions to migrate an ElastiCache (EC) database to Redis Cloud or Redis Software.

The following scenarios are covered:

  • One-time (snapshot) migration

  • Online (live) migration

It is recommended to read the Replication section to familiarize yourself with its usage and architecture.

Setup

Prerequisites

For this recipe you will require the following resources:

  • AWS ElastiCache: Primary Endpoint in case of Single Master and Configuration Endpoint in case of Clustered EC. Refer to this link to learn more

  • Redis Cloud or Redis Software

  • An Amazon EC2 instance to run RIOT-X (for manual CLI migration)

For automated migration using the CloudFormation template, you only need the ElastiCache cluster ID (the template handles all networking and infrastructure setup automatically).
Keyspace Notifications

For a live migration you need to enable keyspace notifications on your ElastiCache instance (see AWS Knowledge Center).
Migration Host

To run the migration tool we will need an EC2 instance.

You can either create a new EC2 instance or leverage an existing one if available. In the example below we first create an instance on AWS Cloud Platform. The most common scenario is to access an ElastiCache cluster from an Amazon EC2 instance in the same Amazon Virtual Private Cloud (Amazon VPC). We have used Ubuntu 16.04 LTS for this setup but you can choose any Ubuntu or Debian distribution of your choice.

SSH to this EC2 instance from your laptop:

ssh -i “public key” <AWS EC2 Instance>

Install redis-cli on this new instance by running this command:

sudo apt update
sudo apt install -y redis-tools

Use redis-cli to check connectivity with the ElastiCache database:

redis-cli -h <ec primary endpoint> -p 6379

Ensure that the above command allows you to connect to the remote ElastiCache database successfully.

Installing RIOT-X

Let’s install RIOT-X on the EC2 instance we set up previously. For this we’ll follow the steps in Manual Installation.

Performing Migration

We are now all set to begin the migration process. The options you will use depend on your source and target databases, as well as the replication mode (snapshot or live).

ElastiCache Single Master → Redis
riotx replicate source:port target:port
Live ElastiCache Single Master → Redis
riotx replicate source:port target:port --mode live

In case ElastiCache is configured with AUTH TOKEN enabled, you need to pass --source-tls as well as --source-pass option:

riotx replicate source:port target:port --source-tls --source-pass <password>
ElastiCache Cluster → Redis
riotx replicate source:port target:port --source-cluster
--cluster is an important parameter used ONLY for ElastiCache whenever cluster-mode is enabled. Do note that the source database is specified first and the target database is specified after the replicate command and it is applicable for all the scenarios.
ElastiCache Single Master → Redis (with specific database index)
riotx replicate redis://source:port/db target:port
ElastiCache Single Master → Redis with OSS Cluster
riotx replicate source:port target:port --target-cluster
Live ElastiCache Cluster → Redis with OSS Cluster
riotx replicate source:port target:port --source-cluster --target-cluster --mode live

Important Considerations

  • It is recommended to test migration in UAT before production use.

  • Once migration is completed, ensure that application traffic gets redirected to Redis endpoint successfully.

  • It is recommended to perform the migration process during low traffic hours so as to avoid chances of data loss.

CloudFormation

For the easiest migration experience, use the CloudFormation template that automates the entire process. This template automatically discovers your ElastiCache configuration and handles all the infrastructure setup.

Step-by-Step Migration Guide
Step 1: Gather Required Information

Before starting, you’ll need:

  1. ElastiCache Cluster ID:

    • Go to AWS Console → ElastiCache

    • Find your Redis cluster and copy the Name (not the endpoint)

    • Examples: my-redis-cluster, production-cache-001

  2. Target Redis URI:

    • Get this from your Redis Cloud or Redis Enterprise provider

    • Format: redis://username:password@host:port or rediss://username:password@host:port

    • Example: redis://default:mypassword@redis-12345.c1.us-west-2.ec2.redns.redis-cloud.com:12345

Step 2: Deploy the Migration Template

  1. Open the CloudFormation Console:

  2. Configure the Stack:

    • Stack name: Keep riotx-elasticache-sync or choose your own name

    • Click Next

  3. Fill Required Parameters:

    • Source ElastiCache Cluster ID: Enter your ElastiCache cluster name from Step 1

    • Target Redis URI: Enter your Redis Cloud/Enterprise URI from Step 1

    • Leave all other settings as defaults (they work for most cases)

    • Click Next

  4. Review and Deploy:

    • Scroll down and check ☑ I acknowledge that AWS CloudFormation might create IAM resources

    • Click Submit

Step 3: Monitor the Migration

The migration starts automatically when the CloudFormation stack is created.

  1. Watch Stack Progress:

    • Stay on the CloudFormation console

    • Watch the Events tab - you’ll see resources being created

    • The stack typically takes 5-10 minutes to complete

  2. Check Migration Status:

    • When the stack shows CREATE_COMPLETE, go to the Outputs tab

    • Look for MigrationExecutionMessage - it will show "migration completed successfully"

  3. View Migration Logs (optional):

    • In the Outputs tab, click the LogGroupUrl link

    • This takes you directly to CloudWatch logs showing migration details

    • Or copy the ViewLogsCommand and run it in AWS CLI

Step 4: Verify Migration Success

  1. Check Your Target Redis:

    • Connect to your Redis Cloud/Enterprise database

    • Verify that your data has been migrated

    • Check key counts: INFO keyspace

  2. Review Migration Details:

    • The CloudFormation Outputs tab shows discovered source configuration

    • Check SourceRedisURI, SourceClusterMode, SourceTransitEncryption

    • Verify these match your ElastiCache setup

Advanced Configuration (Optional)

If you need to customize the migration, expand the parameter sections when filling the template:

Migration Settings:

  • Sync Mode: Choose SCAN (one-time), LIVE (continuous), or LIVEONLY

  • Compare Mode: Choose QUICK for basic verification or FULL for complete comparison

  • Dry Run: Set to true to test without actually copying data

Performance Tuning:

  • Batch Size: Increase from 50 to 200+ for faster migration of small keys

  • Thread Count: Increase from 1 to 4+ for parallel processing

Authentication (if needed):

  • Source User/Password: Only if your ElastiCache has AUTH enabled

  • Target Cluster Mode: Set to true if your target Redis uses cluster mode

What the Template Does Automatically

  • ✅ Discovers your ElastiCache VPC, subnets, and security groups

  • ✅ Configures network access between ElastiCache and the migration container

  • ✅ Creates ECS infrastructure to run the migration

  • ✅ Runs the migration automatically during stack creation

  • ✅ Sets up CloudWatch logging for monitoring

  • ✅ Reuses existing infrastructure if available

Typical Workflow

For comprehensive testing and migration:

  1. Prerequisites for Live Migration:

  2. Migrate Data:

    • Deploy ec-sync.yaml to migrate from ElastiCache to target Redis

    • Choose migration mode: SCAN (snapshot), LIVE (scan + live changes), or LIVEONLY (live only)

    • Monitor progress via CloudWatch logs

  3. Verify Migration:

    • Check CloudFormation stack outputs for completion status

    • Review CloudWatch logs for any errors

    • Verify data in target Redis database

Monitoring and Logging

Both CloudFormation templates include enhanced logging capabilities that make it easy to monitor and troubleshoot RIOT-X container operations:

Enhanced Logging Features:

  • CloudWatch Log Group: Automatically created with 14-day retention

  • Direct Log Access URLs: CloudFormation outputs include direct links to view logs

  • AWS CLI Commands: Ready-to-use commands for viewing and tailing logs

  • Enhanced Log Configuration: Better timestamping and buffering

Accessing Logs:

After deploying a CloudFormation stack, you can access logs in several ways:

Via CloudFormation Outputs:

  1. Go to CloudFormation console → your stack → Outputs tab

  2. Look for these outputs:

    • LogGroupUrl: Direct link to CloudWatch logs in AWS Console

    • ViewLogsCommand: AWS CLI commands to view recent logs

    • LogGroup: Name of the CloudWatch log group

Via AWS Console:

  1. Go to CloudWatch in the AWS Console

  2. Navigate to "Log groups"

  3. Find the log group (default: /ecs/riot-migration or /ecs/riot-generation)

  4. Click on log streams to view container output

Via AWS CLI:

# List recent log streams
aws logs describe-log-streams --log-group-name "/ecs/riot-migration" --order-by LastEventTime --descending

# View latest logs (real-time tailing)
aws logs tail "/ecs/riot-migration" --follow

# Get specific log events
aws logs get-log-events --log-group-name "/ecs/riot-migration" --log-stream-name "ecs/container-name/task-id"

Log Stream Naming Convention:

ECS creates log streams with this format: ecs/[container-name]/[task-id]

For example: ecs/riot-migration/a1b2c3d4-e5f6-7890-abcd-ef1234567890

What You’ll See in the Logs:

  • Migration progress information ("Scanning", "Comparing")

  • Connection details (source ElastiCache, target Redis Cloud)

  • Job execution status and timing

  • Configuration parameters and validation

  • Error messages and troubleshooting information

Troubleshooting

Common Issues:

  • Stack Creation Failed: Check CloudWatch logs for detailed error messages

  • Permission Errors: Ensure your AWS user has CloudFormation and ECS permissions

  • Network Connectivity: Verify security groups allow Redis connectivity

  • ElastiCache Not Found: Double-check cluster ID and AWS region

  • Target Redis Unreachable: Verify Redis Cloud URI and credentials

Logging Troubleshooting:

If you don’t see logs:

  1. Check ECS Task Status: Ensure the task started successfully

  2. Verify IAM Permissions: The task execution role needs CloudWatch Logs permissions

  3. Check Network Connectivity: Tasks need internet access to pull container images

  4. Review Task Definition: Ensure the log configuration is properly set

Getting Help:

  • CloudWatch Logs: /ecs/riot-migration log group contains detailed execution logs

  • CloudFormation Events: Check stack events for resource creation details

  • Stack Outputs: Review outputs for auto-discovered configuration and status

Connectivity Test

The ping command can be used to test connectivity to a Redis database.

riotx ping [OPTIONS]

To show the full usage, run:

riotx ping --help

The command prints statistics like these:

riotx ping -h localhost --unit microseconds
[min=491, max=14811, percentiles={99.9=14811, 90.0=1376, 95.0=2179, 99.0=14811, 50.0=741}]
[min=417, max=1286, percentiles={99.9=1286, 90.0=880, 95.0=1097, 99.0=1286, 50.0=606}]
[min=382, max=2244, percentiles={99.9=2244, 90.0=811, 95.0=1036, 99.0=2244, 50.0=518}]
...

Best Practices

This section contains best practices and recipes for various RIOT-X use cases.

Replication Performance Tuning

replication architecture

The replicate command reads from a source Redis database and write to a target Redis database.

Replication Bottleneck

To optimize throughput it is necessary to understand the two main possible scenarios:

Slow Producer

In this scenario the reader does not read from source as fast as the writer can write to the target. This means the writer is starved and we should look into ways to speed up the reader.

Slow Consumer

In this scenario the writer can not keep up with the reader and we should look into optimizing writes.

There are two ways to identify which scenario we fall into:

No-op writer

With the --dry-run option the replication process will use a no-op writer instead of a Redis writer. If throughput with dry-run is similar to throughput without then the writer is not the bottleneck. Follow steps below to improve reader throughput.

Reader

To improve reader performance tweak the options below until you reach optimal throughput.

--scan-count

Number of keys each scan call should fetch (default: 100).

--read-batch

Number of values each reader thread should read in a single pipelined call (default: 50).

--source-pool

Number of Redis connections to the source database (default: 8). Keep in sync with the number of read threads to have a dedicated connection per thread.

Writer

To improve writer performance you can tweak the following options:

--batch

Number of items written in a single network round-trip to the Redis server (i.e. number of commands in the pipeline).

--threads

How many write operations can be performed concurrently (default: 1).

--target-pool

Number of Redis connections to the target database (default: 8). Keep in sync with the number of threads to have a dedicated connection per thread.

System Requirements

Operating System

RIOT-X works on all major operating systems but has been tested at scale on Linux X86 64-bit platforms.

CPU

CPU used by RIOT-X varies greatly dependending on specific replication settings and data structures at play. You can monitor CPU usage with the supplied Grafana dashboard (process_cpu_usage metric).

Disk

RIOT-X does not require any specific disk requirements since all state is kept in memory.

Memory

Memory requirements for RIOT-X itself are very light. Being JVM-based the default initial heap size is dependent on available system memory and on the operating system.

If you have very intensive replication requirements you will need to increase the JVM heap size. To estimate the worst case scenario for memory requirements you can use this formula: keySize * batchSize * threads where:

keySize

average key size as reported by the MEMORY USAGE command

batchSize

number of items to read in each batch

threads

Number of threads that process batches concurrently

Network

RIOT-X replication is essentially a network bridge between the source and target Redis databases so underlying network is crucial for the overall throughput and a 10 Gigabit network is the minimum recommended. Network latency will also have an impact on replication (and other RIOT-X uses) performance. Make sure the host running RIOT-X offers minimal latency to both the source and target databases. You can test the latency using the ping command].

CRDB

Active/active Redis databases (CRDB) need special considerations. If your target database is a CRDB deployment you will need to use the data-structure replication type (--struct) as the RESTORE command is not supported in CRDB.

FAQ

  1. Logs are cut off or missing

    This could be due to concurrency issues in the terminal when refreshing the progress bar and displaying logs. Try running with job option --progress log.

  2. Unknown options: '--keyspace', '--key'

    You must specify one or more Redis commands for *-import (file-import, faker, db-import).

  3. ERR DUMP payload version or checksum are wrong

    Redis 7 DUMP format is not backwards compatible with previous versions. To replicate between different Redis versions, use Type-Based Replication.

  4. ERR Unsupported Type 0

    The target database is most likely CRDB in which case you need to use type-based replication (--struct option).

  5. Process gets stuck during replication and eventually times out

    This could be due to big keys clogging the replication pipes. In these cases it might be hard to catch the offending key(s). Try running the same command with --info and --progress log so that all errors are reported. Check the database with redis-cli Big keys and/or use reader options to filter these keys out.

  6. NOAUTH Authentication required

    This issue occurs when you fail to supply the --pass <password> parameter.

  7. ERR The ID argument cannot be a complete ID because xadd-id-uniqueness-mode is strict

    This usually happens in Active/Active (CRDB) setups where stream message IDs cannot be copied over to the target database. Use the --no-stream-id option to disable ID propagation.

  8. ERR Error running script…​ This Redis command is not allowed from scripts

    This can happen with Active/Active (CRDB) databases because the MEMORY USAGE command is not allowed to be run from a LUA script. Use the --mem-limit -1 option to disable memory usage.

  9. java.lang.OutOfMemoryError: Java heap space

    The RIOT-X JVM ran out of memory. If you are running db-import this could be due to a large resultset being loaded upfront. Use the --fetch option to set a fixed fetch size (e.g. --fetch 1000). Otherwise increase max JVM heap size (export JAVA_OPTS="-Xmx8g") or reduce RIOT-X memory usage by lowering threads, batch, and read-batch values.