Importing

There are common features available when importing data into Redis (db-import, faker, file-import, redis-import, snowflake-import, stream-import).

Redis Key

RIOT-X offers two mechanisms to craft the Redis key:

Template Expression

This produces the key string using a SpEL expression.

Template expressions allow mixing literal text with one or more evaluation blocks. Each evaluation block is delimited with #{ } and within an evaluation block you can directly reference fields by their name, for example:

Template Expression Example
riotx file-import beers.json hset beer:#{id}

Here each key is constructed with the beer: prefix and suffixed with the value in the id field, producing keys beer:123 beer:494 etc.

Keyspace and Key Fields

This mechanism constructs keys using a keyspace string passed with --keyspace and key fields passed with --key, for example:

Keyspace/Key Fields Example
riotx file-import beers.json hset --keyspace beer --key id

Processing

Processors allow you to create/update/delete fields using the Spring Expression Language (SpEL). The SPEL processor system provides powerful data transformation capabilities with built-in type safety and extensive context support.

Field Processing Examples

--proc field1="'foo'"

Generate a field named field1 containing the string foo

--proc temp="(temp-32)*5/9"

Convert from Fahrenheit to Celsius

--proc name='remove("first").concat(remove("last"))'

Concatenate first and last fields and delete them

--proc field2=null

Delete field2

--proc index="T(java.lang.Integer).parseInt(id)"

Convert string field to integer

--proc fullName="firstName + ' ' + lastName"

Concatenate multiple fields

--proc upperName="name?.toUpperCase()"

Safe navigation with null checking

Input fields are accessed by name (e.g. field3=field1+field2).

Context Variables and Functions

Processors have access to the following context variables and functions:

date

Date parsing and formatting object. Instance of Java SimpleDateFormat.

number

Number parsing and formatting object. Instance of Java DecimalFormat.

faker

Faker object.

redis

Redis commands object. Instance of Lettuce RedisCommands. The replicate command exposes 2 command objects named source and target.

floatsToBytes

Convert an array of floats to a byte array so that the hash field can be indexed as vector (see Redis Vectors).

For example if the field vector is an array of floats: --proc embedding="#floatsToBytes(vector)"

Advanced Processing Patterns

Conditional Processing:

# Only process if field exists
--proc processedValue="value == null ? 0 : value * 2"

# Complex conditions
--proc category="price > 100 ? 'premium' : (price > 50 ? 'standard' : 'budget')"

Mathematical Operations:

# Statistical calculations
--proc average="(field1 + field2 + field3) / 3"
--proc percentage="(score / maxScore) * 100"
--proc rounded="T(java.lang.Math).round(value * 100.0) / 100.0"

String Processing:

# Text transformations
--proc slug="name.toLowerCase().replaceAll('[^a-z0-9]+', '-')"
--proc initials="firstName.substring(0,1) + lastName.substring(0,1)"
--proc masked="'***-**-' + ssn.substring(ssn.length()-4)"

Date and Time Processing:

# Date parsing and formatting
--proc timestamp="#date.parse(dateString, 'yyyy-MM-dd').getTime()"
--proc formatted="#date.format(new java.util.Date(epoch), 'MM/dd/yyyy')"
--proc age="T(java.time.Period).between(T(java.time.LocalDate).parse(birthDate), T(java.time.LocalDate).now()).getYears()"
geo

Convenience function for RediSearch geo-location strings in the form longitude,latitude.

For example with longitude and latitude fields lon and lat: --proc location="#geo(lon,lat)"

Comprehensive processor example
riotx file-import beers.json hset beer:#{id} \
  --proc epoch="#date.parse(mydate).getTime()" \
  --proc location="#geo(lon,lat)" \
  --proc fullName="#redis.hget('person1','lastName')" \
  --proc category="abv > 7.0 ? 'strong' : 'regular'" \
  --proc description="name + ' - ' + style + ' (' + abv + '% ABV)'"
Faker processor example
riotx file-import http://storage.googleapis.com/jrx/beers.csv --header --proc fakeid="#faker.numerify('###########')" hset beer:#{fakeid}

You can register your own variables using --var.

Custom variable example
riotx file-import http://storage.googleapis.com/jrx/lacity.csv --var rnd="new java.util.Random()" --proc randomInt="#rnd.nextInt(100)" --header hset event:#{Id}

Filtering

Filters allow you to exclude records that don’t match a SpEL boolean expression.

For example this filter will only keep records where the value field is a series of digits:

riotx file-import --filter "value matches '\\d+'" ...

RIOT-X supports RediSearch commands for full-text search and aggregation operations as part of import workflows.

Search Command

Execute FT.SEARCH queries against RediSearch indexes during data import.

Syntax
riotx <cmd> ... search INDEX QUERY [OPTIONS...]

Parameters:

  • INDEX - Template expression for the search index name

  • QUERY - Template expression for the search query

  • OPTIONS - Optional search parameters (e.g., limit 0 10, withpayloads, sortby field)

Basic search example
riotx faker id="numerify('####')" search myindex "*"
Search with limit
riotx faker id="numerify('####')" search myindex "@title:redis" limit 0 10
Search with sorting and field selection
riotx faker id="numerify('####')" search products "@category:electronics" return 3 name price rating sortby price desc

Aggregate Command

Execute FT.AGGREGATE aggregations for complex data analysis and reporting during import operations.

Syntax
riotx <cmd> ... aggregate INDEX QUERY [OPTIONS...]

Parameters:

  • INDEX - Template expression for the search index name

  • QUERY - Template expression for the query to aggregate over

  • OPTIONS - Aggregation pipeline expressions (groupby, reduce, sortby, etc.)

Group by field and calculate average
riotx faker id="numerify('##########')" --count 1m --metrics-redis log --progress none aggregate beers "*" "groupby" "1" "@style_name" reduce avg 1 "@abv" as abv sortby 2 "@abv" desc
Both search and aggregate commands are available as Redis operations in any import command context. Template expressions support SpEL for dynamic field/query generation and integrate with RIOT-X’s batching and backpressure mechanisms. Results can be processed through standard RIOT-X processing pipelines.