This version is still in development and is not considered stable yet. For the latest stable version, please use RIOT-X 2.0!

Importing

There are common features available when importing data into Redis (databricks-import, db-import, elastic-import, faker, file-import, redis-import, snowflake-import, stream-import).

Reader limit

Use --count (or RIOT_COUNT) to limit source items read before processing and filtering. The default is unlimited for imports and 1000 for faker and gen (generate). Values such as 10k and 5m are supported; 0 means unlimited for imports. faker and gen require a positive count and reject 0, including RIOT_COUNT=0. The count must be a nonnegative whole number no greater than 2147483646.

The limit applies separately to each reader: per table for snowflake-import and databricks-import, per file for file-import, and per index for elastic-import (including OpenSearch). For stream-import, it counts messages across the configured streams. For Snowflake and Databricks, the limit covers the snapshot and subsequent CDC batches, and also applies with --gen. Filtered source items still count toward the limit, so fewer items may be written to Redis.

--max has been removed. Use --count instead. For stream-import, --read-count controls the XREAD batch size and --read-block controls the XREAD blocking duration; neither is the overall reader limit.

For database readers, --max-rows sets JDBC Statement.setMaxRows() for each query result. It does not control fetch size or the overall reader count. Snowflake and Databricks reject positive --max-rows / RIOT_ROWS limits before opening connections, because truncating a CDC query result can skip changes. Use --count instead.

Redis Key

RIOT-X offers two mechanisms to craft the Redis key:

Template Expression

This produces the key string using a SpEL expression.

Template expressions allow mixing literal text with one or more evaluation blocks. Each evaluation block is delimited with #{ } and within an evaluation block you can directly reference fields by their name, for example:

Template Expression Example
riotx file-import beers.json hset beer:#{id}

Here each key is constructed with the beer: prefix and suffixed with the value in the id field, producing keys beer:123 beer:494 etc.

Keyspace and Key Fields

This mechanism constructs keys using a keyspace string passed with --keyspace and key fields passed with --key, for example:

Keyspace/Key Fields Example
riotx file-import beers.json hset --keyspace beer --key id

Processing

Processors allow you to create/update/delete fields using the Spring Expression Language (SpEL). The SPEL processor system provides powerful data transformation capabilities with built-in type safety and extensive context support.

Field Processing Examples

--proc field1="'foo'"

Generate a field named field1 containing the string foo

--proc temp="(temp-32)*5/9"

Convert from Fahrenheit to Celsius

--proc name='remove("first").concat(remove("last"))'

Concatenate first and last fields and delete them

--proc field2=null

Delete field2

--proc index="T(java.lang.Integer).parseInt(id)"

Convert string field to integer

--proc fullName="firstName + ' ' + lastName"

Concatenate multiple fields

--proc upperName="name?.toUpperCase()"

Safe navigation with null checking

Input fields are accessed by name (e.g. field3=field1+field2).

Context Variables and Functions

Processors have access to the following context variables and functions:

date

Date parsing and formatting object. Instance of Java SimpleDateFormat.

number

Number parsing and formatting object. Instance of Java DecimalFormat.

faker

Faker object.

redis

Redis commands object. Instance of Lettuce RedisCommands. The replicate command exposes 2 command objects named source and target.

floatsToBytes

Convert an array of floats to a byte array so that the hash field can be indexed as vector (see Redis Vectors).

For example if the field vector is an array of floats: --proc embedding="#floatsToBytes(vector)"

Advanced Processing Patterns

Conditional Processing:

# Only process if field exists
--proc processedValue="value == null ? 0 : value * 2"

# Complex conditions
--proc category="price > 100 ? 'premium' : (price > 50 ? 'standard' : 'budget')"

Mathematical Operations:

# Statistical calculations
--proc average="(field1 + field2 + field3) / 3"
--proc percentage="(score / maxScore) * 100"
--proc rounded="T(java.lang.Math).round(value * 100.0) / 100.0"

String Processing:

# Text transformations
--proc slug="name.toLowerCase().replaceAll('[^a-z0-9]+', '-')"
--proc initials="firstName.substring(0,1) + lastName.substring(0,1)"
--proc masked="'***-**-' + ssn.substring(ssn.length()-4)"

Date and Time Processing:

# Date parsing and formatting
--proc timestamp="#date.parse(dateString, 'yyyy-MM-dd').getTime()"
--proc formatted="#date.format(new java.util.Date(epoch), 'MM/dd/yyyy')"
--proc age="T(java.time.Period).between(T(java.time.LocalDate).parse(birthDate), T(java.time.LocalDate).now()).getYears()"
geo

Convenience function for RediSearch geo-location strings in the form longitude,latitude.

For example with longitude and latitude fields lon and lat: --proc location="#geo(lon,lat)"

Comprehensive processor example
riotx file-import beers.json hset beer:#{id} \
  --proc epoch="#date.parse(mydate).getTime()" \
  --proc location="#geo(lon,lat)" \
  --proc fullName="#redis.hget('person1','lastName')" \
  --proc category="abv > 7.0 ? 'strong' : 'regular'" \
  --proc description="name + ' - ' + style + ' (' + abv + '% ABV)'"
Faker processor example
riotx file-import http://storage.googleapis.com/jrx/beers.csv --header --proc fakeid="#faker.numerify('###########')" hset beer:#{fakeid}

You can register your own variables using --var.

Custom variable example
riotx file-import http://storage.googleapis.com/jrx/lacity.csv --var rnd="new java.util.Random()" --proc randomInt="#rnd.nextInt(100)" --header hset event:#{Id}

Filtering

Filters allow you to exclude records that don’t match a SpEL boolean expression.

For example this filter will only keep records where the value field is a series of digits:

riotx file-import --filter "value matches '\\d+'" ...

RIOT-X supports RediSearch commands for full-text search and aggregation operations as part of import workflows.

Search Command

Execute FT.SEARCH queries against RediSearch indexes during data import.

Syntax
riotx <cmd> ... search INDEX QUERY [OPTIONS...]

Parameters:

  • INDEX - Template expression for the search index name

  • QUERY - Template expression for the search query

  • OPTIONS - Optional search parameters (e.g., limit 0 10, withpayloads, sortby field)

Basic search example
riotx faker id="numerify('####')" search myindex "*"
Search with limit
riotx faker id="numerify('####')" search myindex "@title:redis" limit 0 10
Search with sorting and field selection
riotx faker id="numerify('####')" search products "@category:electronics" return 3 name price rating sortby price desc

Aggregate Command

Execute FT.AGGREGATE aggregations for complex data analysis and reporting during import operations.

Syntax
riotx <cmd> ... aggregate INDEX QUERY [OPTIONS...]

Parameters:

  • INDEX - Template expression for the search index name

  • QUERY - Template expression for the query to aggregate over

  • OPTIONS - Aggregation pipeline expressions (groupby, reduce, sortby, etc.)

Group by field and calculate average
riotx faker id="numerify('##########')" --count 1m --metrics-redis log --progress none aggregate beers "*" "groupby" "1" "@style_name" reduce avg 1 "@abv" as abv sortby 2 "@abv" desc
Both search and aggregate commands are available as Redis operations in any import command context. Template expressions support SpEL for dynamic field/query generation and integrate with RIOT-X’s batching and backpressure mechanisms. Results can be processed through standard RIOT-X processing pipelines.