This version is still in development and is not considered stable yet. For the latest stable version, please use RIOT-X 2.0!

Stream Import/Export

RIOT-X provides commands to import and export data from and to Redis streams.

Stream Import

The stream-import command reads data from a stream and writes it to Redis.

The basic usage is:

riotx stream-import STREAM...

For the full usage, run:

riotx stream-import --help
Example: Import stream into hashes
riotx stream-import stream:beers --idle-timeout 5s hset beer:#{id}

Each imported entry exposes the following metadata properties to SpEL expressions:

Field Description

__key

Source stream key

__id

Source stream entry ID

__timestamp

Stream entry ID timestamp component as a java.time.Instant

__sequence

Stream entry ID sequence component as a decimal string

These names are reserved. An input entry containing any of these fields is rejected to avoid overwriting its data. Metadata is available to expressions but is not included in destination payloads by default. Use --proc to copy a metadata property into a body field when it should be persisted, for example --proc 'sourceStream=__key'.

Example: Continuously materialize multiple streams into corresponding sets
riotx stream-import stream1 stream2 \
  sadd 'set:#{__key}' --member '#{value}'

The timestamp can be formatted or converted directly with SpEL. For example, {timestamp} produces an ISO-8601 timestamp. {timestamp.toEpochMilli()} produces epoch milliseconds as a signed Java long; it throws an overflow exception if the ID’s millisecond component exceeds 9223372036854775807 (Long.MAX_VALUE). To obtain the millisecond component as a decimal string over Redis’s full unsigned 64-bit range, use #{id.substring(0, id.indexOf('-'))} instead. For server-generated stream IDs, the timestamp represents the message creation time. Explicit stream IDs can use an arbitrary timestamp component.

Stream Export

The stream-export command enables Redis CDC to a Redis stream.

riotx stream-export SOURCE TARGET [OPTIONS]

For the full usage, run:

riotx stream-export --help
Example: Export stream to another Redis instance
riotx stream-export redis://localhost:6379 redis://localhost:6380 --mode live

redis-cli -p 6380 xread COUNT 3 STREAMS stream:export 0-0
1) 1) "stream:export"
   2) 1) 1) "1718645537588-0"
         2)  1) "key"
             2) "order:4"
             3) "time"
             4) "1718645537000"
             5) "type"
             6) "hash"
             7) "ttl"
             8) "-1"
             9) "mem"
            10) "136"
            11) "value"
            12) "{\"order_date\":\"2024-06-13 22:19:35.143797\",\"order_id\":\"4\"}"

Stream Export Field Descriptions

Each exported stream entry contains the following fields:

Field Description

key

The Redis key that was modified

time

Timestamp of the change event (milliseconds since epoch)

type

Redis data type (e.g., hash, string, set, zset, list, stream, json)

ttl

Time-to-live in seconds (-1 indicates no expiration)

mem

Memory usage of the key in bytes

value

JSON-serialized representation of the key’s value