This version is still in development and is not considered stable yet. For the latest stable version, please use RIOT-X 2.0!

Observability

RIOT-X exposes several metrics over a Prometheus endpoint that can be useful for troubleshooting and performance tuning.

Getting Started

The riotx-dist repository includes a Docker compose configuration that set ups Prometheus and Grafana.

git clone https://github.com/redis/riotx-dist.git
cd riotx-dist
docker compose up

Prometheus is configured to scrape the host every second.

You can access the Grafana dashboard at localhost:3000.

Now start RIOT-X with the following command:

riotx replicate ... --metrics

This will enable the Prometheus metrics exporter endpoint and will populate the Grafana dashboard.

Configuration

Use the --metrics* options to enable and configure metrics:

--metrics

Enable metrics

--metrics-jvm

Enable JVM and system metrics

--metrics-redis

Enable command latency metrics. See https://github.com/redis/lettuce/wiki/Command-Latency-Metrics#micrometer

--metrics-name=<name>

Application name tag that will be applied to all metrics

--metrics-port=<int>

Port that Prometheus HTTP server should listen on (default: 8080)

--metrics-prop=<k=v>

Additional properties to pass to the Prometheus client. See https://prometheus.github.io/client_java/config/config/

Metrics

Below you can find a list of all metrics declared by RIOT-X.

dashboard replication

Replication Metrics

Name Type Description

riot_replication_bytes_total

Counter

Number of bytes replicated (needs memory usage with --mem-limit)

riot_replication_lag_seconds

Summary

Replication end-to-end latency

riot_replication_read_latency_seconds

Summary

Replication read latency

spring_batch_chunk_write_seconds

Timer

Batch writing duration

spring_batch_item_process_seconds

Timer

Item processing duration

spring_batch_item_read_seconds

Timer

Item reading duration

spring_batch_job_active_seconds

Timer

Active jobs

spring_batch_job_launch_count_total

Counter

Job launch count

redis_batch_key_event_queue_capacity

Gauge

Remaining capacity of the keyspace notification queue

redis_batch_key_event_queue_size

Gauge

Number of keyspace notifications waiting in the queue

redis_batch_key_event_total

Counter

Keyspace notifications received, by event, type and status (FAILURE means the queue was full and the notification was dropped)

redis_batch_operation_seconds

Timer

Duration of each batch of Redis commands, by status. Commands that read from Redis, e.g. replicate, record reads from the source and writes to the target in the same series; imports into Redis, e.g. snowflake-import, record writes only

CDC Metrics

snowflake-import reports the state of each captured table when writing change records to Redis streams. All metrics carry the labels connector, source (the --source-name value) and table (the fully qualified table name). To get values for the whole source, aggregate over the tables in your query, e.g. max by (source) (riot_cdc_source_lag_seconds).

The metrics exist only while snowflake-import runs, and a failed table ends it. Each restart sets riot_cdc_source_lag_seconds back to 0 and riot_cdc_source_last_probe_age_seconds to the time since the restart, so the metrics do not show an import that keeps failing. See Snowflake Monitoring for how to detect it.

Name Type Description

riot_cdc_source_connected

Gauge

1 if the last round-trip to Snowflake succeeded, 0 otherwise. Setting up the Snowflake stream counts as a round-trip, so the gauge is 1 while the initial snapshot is copied

riot_cdc_source_pending

Gauge

1 while changes found by the last check in the Snowflake stream are not yet fetched, i.e. while their batch is prepared or after preparing it failed. The first batch after startup, including the initial snapshot, is fetched without a check, so the gauge stays 0 while it is prepared

riot_cdc_source_last_probe_age_seconds

Gauge

Seconds since the table was last read from Snowflake: a successful check for changes, a fetch of changes, or a row of the snapshot or of a batch of changes. Before the first read, it is the time since the import started. The table is checked every --poll interval, so a value well above it means the table is no longer being read. The table is not read while a write to Redis is in progress, so the value also grows while writes are slow or rejected because of --stream-limit

riot_cdc_source_lag_seconds

Gauge

Upper bound on how far Redis is behind the Snowflake table; 0 when caught up. After startup it stays 0 until the first batch is written and its offset saved, so it does not show the initial snapshot or changes made while RIOT-X was stopped

riot_cdc_events_total

Counter

Change records read from Snowflake, by operation (op label: create, update, delete). Snowflake streams report the net change of each row since the previous batch, so several updates of a row between two batches count as one update, a row inserted and then updated counts as one create, and a row inserted and then deleted is not counted. A record is counted when it is read, so records that later fail or are skipped stay counted. Rows of the initial snapshot are counted in riot_cdc_snapshot_rows_total instead. The series for an op value appears with its first record

riot_cdc_last_event_age_seconds

Gauge

Seconds since the last change record was read from Snowflake; NaN if none was read since the import started. Rows of the initial snapshot do not count

riot_cdc_events_erroneous_total

Counter

Change records skipped because they could not be processed or written (see --skip). They are also counted in riot_cdc_events_total or riot_cdc_snapshot_rows_total

riot_cdc_errors_total

Counter

Failed operations, by stage (stage label: fetch, read, probe, cleanup, write, checkpoint)

riot_cdc_backpressure_total

Counter

Write attempts rejected because the target stream is longer than --stream-limit. Every rejected attempt is counted, including repeated attempts for the same records

riot_cdc_fetch_seconds

Timer

Time to prepare a batch of changes in Snowflake and start reading it, including waiting for a connection from the pool, by status (success, failure). With --snapshot INITIAL and no saved offset, the first batch of a table is the initial snapshot, so it takes the longest. A check can report changes that turn out to be empty, e.g. a row inserted and then deleted, so a batch can be empty

riot_cdc_snapshot_running

Gauge

1 from the start of the initial snapshot, including creating the Snowflake stream and copying the rows, until all its rows are written to Redis and its offset is saved

riot_cdc_snapshot_completed

Gauge

1 once the initial snapshot has been written

riot_cdc_snapshot_skipped

Gauge

1 when the initial snapshot is disabled (--snapshot NEVER)

riot_cdc_snapshot_aborted

Gauge

1 when the initial snapshot stopped before it completed, e.g. because the import failed

riot_cdc_snapshot_rows_total

Counter

Rows of the initial snapshot read from Snowflake

riot_cdc_snapshot_rows_expected

Gauge

Rows in the initial snapshot; NaN if unknown, e.g. when the snapshot completed before the current run or counting its rows failed, so 0 always means an empty table. Compare with riot_cdc_snapshot_rows_total for progress

riot_cdc_snapshot_duration_seconds

Gauge

Duration of the running, completed or aborted initial snapshot, from its start, including creating the Snowflake stream, until its offset is saved; 0 if the snapshot completed before the current run or is disabled

snowflake-import also reports the health of its Snowflake connection pool with the standard HikariCP metrics, labeled pool="snowflake" by default. The most useful ones are listed below.

Name Type Description

hikaricp_connections_active

Gauge

Connections in use

hikaricp_connections_idle

Gauge

Open connections not in use

hikaricp_connections_pending

Gauge

Threads waiting for a connection

hikaricp_connections_max

Gauge

Maximum pool size (--jdbc-pool)

hikaricp_connections_timeout_total

Counter

Requests that timed out waiting for a connection

hikaricp_connections_acquire_seconds

Timer

Time to obtain a connection from the pool

JVM Metrics

Use the --metrics-jvm option to enable the following additional metrics:

dashboard jvm
Name Type Description

jvm_buffer_count_buffers

Gauge

An estimate of the number of buffers in the pool

jvm_buffer_memory_used_bytes

Gauge

An estimate of the memory that the Java virtual machine is using for this buffer pool

jvm_buffer_total_capacity_bytes

Gauge

An estimate of the total capacity of the buffers in this pool

jvm_gc_concurrent_phase_time_seconds

Timer

Time spent in concurrent phase

jvm_gc_live_data_size_bytes

Gauge

Size of long-lived heap memory pool after reclamation

jvm_gc_max_data_size_bytes

Gauge

Max size of long-lived heap memory pool

jvm_gc_memory_allocated_bytes_total

Gauge

Incremented for an increase in the size of the (young) heap memory pool after one GC to before the next

jvm_gc_memory_promoted_bytes_total

Counter

Count of positive increases in the size of the old generation memory pool before GC to after GC

jvm_gc_pause_seconds

Timer

Time spent in GC pause

jvm_memory_committed_bytes

Gauge

The amount of memory in bytes that is committed for the Java virtual machine to use

jvm_memory_max_bytes

Gauge

The maximum amount of memory in bytes that can be used for memory management

jvm_memory_used_bytes

Gauge

The amount of used memory

jvm_threads_daemon_threads

Gauge

The current number of live daemon threads

jvm_threads_live_threads

Gauge

The current number of live threads including both daemon and non-daemon threads

jvm_threads_peak_threads

Gauge

The peak live thread count since the Java virtual machine started or peak was reset

jvm_threads_started_threads_total

Counter

The total number of application threads started in the JVM

jvm_threads_states_threads

Gauge

The current number of threads

process_cpu_time_ns_total

Counter

The "cpu time" used by the Java Virtual Machine process

process_cpu_usage

Gauge

The "recent cpu usage" for the Java Virtual Machine process

process_start_time_seconds

Gauge

Start time of the process since unix epoch.

process_uptime_seconds

Gauge

The uptime of the Java virtual machine

system_cpu_count

Gauge

The number of processors available to the Java virtual machine

system_cpu_usage

Gauge

The "recent cpu usage" of the system the application is running in

system_load_average_1m

Gauge

The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time

Telegraf

RIOT-X logs can be collected and analyzed using Telegraf, an open-source server agent for collecting and sending metrics and logs.

Log Format

RIOT-X uses SLF4J Simple Logger with a configurable format.

Default format:

yyyy-MM-dd HH:mm:ss.SSS [LEVEL] logger.name - message

Example output:

2024-12-02 10:15:30.123 [INFO] com.redis.riot.FileImportCommand - Starting file import
2024-12-02 10:15:31.456 [WARN] com.redis.riot.ProgressStepExecutionListener - Slow processing detected
2024-12-02 10:15:32.789 [ERROR] com.redis.riot.ReplicateCommand - Connection failed

When --log-thread is enabled, the format includes thread information:

2024-12-02 10:15:30.123 [INFO] [main] com.redis.riot.FileImportCommand - Starting file import

Logging Options

Configure logging behavior with these options:

--log-file <file>

Write logs to a file

--termination-log <file>

Also write ERROR messages and fatal command exceptions to a termination message file. Disabled by default. Existing console and file logging is preserved.

--log-level <level>

Set log level: ERROR, WARN, INFO, DEBUG, or TRACE (default: WARN)

--log-time-fmt <format>

Date/time format (default: yyyy-MM-dd HH:mm:ss.SSS)

--no-log-time

Hide timestamp in log messages

--log-thread

Show thread name in log messages

--log-name

Show logger instance name

-d, --debug

Enable debug logging

-i, --info

Enable info logging

-q, --quiet

Show errors only

For Kubernetes collectors, use --termination-log=/dev/termination-log. The file is cleared before command execution and immediately updated with the latest error and its exception causes, without timestamps, stack traces, or ANSI colors. Messages are limited to 4096 UTF-8 bytes. File errors are reported to stderr without changing the command exit code.

Telegraf Setup

  1. Configure RIOT-X to write logs to a file:

    riotx --log-file /var/log/riotx/riotx.log file-import data.csv
  2. Download the complete Telegraf configuration: ../_attachments/telegraf-riotx.conf[telegraf-riotx.conf]

  3. Create a Telegraf configuration file (/etc/telegraf/telegraf.d/riotx.conf) with the following minimal setup:

    [[inputs.tail]]
      files = ["/var/log/riotx/*.log"]
      from_beginning = false
      data_format = "grok"
    
      grok_patterns = [
        '%{TIMESTAMP_ISO8601:timestamp} \[%{LOGLEVEL:level}\] \[%{DATA:thread}\] %{DATA:logger} - %{GREEDYDATA:message}',
        '%{TIMESTAMP_ISO8601:timestamp} \[%{LOGLEVEL:level}\] %{DATA:logger} - %{GREEDYDATA:message}',
      ]
    
      name_override = "riotx_logs"
      grok_timezone = "Local"
    
    [[processors.date]]
      field = "timestamp"
      field_key = "timestamp"
      date_format = ["2006-01-02 15:04:05.000"]
    
    [[processors.enum]]
      [[processors.enum.mapping]]
        field = "level"
        dest = "severity"
        [processors.enum.mapping.value_mappings]
          TRACE = 1
          DEBUG = 2
          INFO = 3
          WARN = 4
          ERROR = 5
    
    [[outputs.influxdb_v2]]
      urls = ["http://localhost:8086"]
      token = "$INFLUX_TOKEN"
      organization = "myorg"
      bucket = "riotx_logs"
  4. Start Telegraf:

    sudo systemctl restart telegraf

Parsed Fields

The Telegraf configuration extracts these fields:

Field Type Description

timestamp

timestamp

Log entry timestamp

level

string

Log level (TRACE, DEBUG, INFO, WARN, ERROR)

severity

integer

Numeric severity (1-5)

logger

string

Full logger name

thread

string

Thread name (if enabled)

message

string

Log message content

Docker Deployment

For containerized deployments, use the Docker log input:

[[inputs.docker_log]]
  endpoint = "unix:///var/run/docker.sock"
  container_name_include = ["riotx*"]
  data_format = "grok"
  grok_patterns = [
    '%{TIMESTAMP_ISO8601:timestamp} \[%{LOGLEVEL:level}\] %{DATA:logger} - %{GREEDYDATA:message}',
  ]
  name_override = "riotx_logs"

Querying Logs

Example InfluxDB Flux queries:

Get ERROR level logs:

from(bucket: "riotx_logs")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "riotx_logs")
  |> filter(fn: (r) => r.level == "ERROR")

Count logs by level:

from(bucket: "riotx_logs")
  |> range(start: -24h)
  |> filter(fn: (r) => r._measurement == "riotx_logs")
  |> group(columns: ["level"])
  |> count()

For a complete setup guide including Docker, Kubernetes, Elasticsearch, and Prometheus configurations, see ../_attachments/TELEGRAF_SETUP.adoc[Telegraf Setup Guide].