ElastiCache Migration

This recipe contains step-by-step instructions to migrate an ElastiCache (EC) database to Redis Cloud or Redis Software.

The following scenarios are covered:

  • One-time (snapshot) migration

  • Online (live) migration

It is recommended to read the Replication section to familiarize yourself with its usage and architecture.

Setup

Prerequisites

For this recipe you will require the following resources:

  • AWS ElastiCache: Primary Endpoint in case of Single Master and Configuration Endpoint in case of Clustered EC. Refer to this link to learn more

  • Redis Cloud or Redis Software

  • An Amazon EC2 instance to run RIOT-X (for manual CLI migration)

For automated migration using the CloudFormation template, you only need the ElastiCache cluster ID (the template handles all networking and infrastructure setup automatically).
Keyspace Notifications

For a live migration you need to enable keyspace notifications on your ElastiCache instance (see AWS Knowledge Center).

Migration Host

To run the migration tool we will need an EC2 instance.

You can either create a new EC2 instance or leverage an existing one if available. In the example below we first create an instance on AWS Cloud Platform. The most common scenario is to access an ElastiCache cluster from an Amazon EC2 instance in the same Amazon Virtual Private Cloud (Amazon VPC). We have used Ubuntu 16.04 LTS for this setup but you can choose any Ubuntu or Debian distribution of your choice.

SSH to this EC2 instance from your laptop:

ssh -i “public key” <AWS EC2 Instance>

Install redis-cli on this new instance by running this command:

sudo apt update
sudo apt install -y redis-tools

Use redis-cli to check connectivity with the ElastiCache database:

redis-cli -h <ec primary endpoint> -p 6379

Ensure that the above command allows you to connect to the remote ElastiCache database successfully.

Installing RIOT-X

Let’s install RIOT-X on the EC2 instance we set up previously. For this we’ll follow the steps in Manual Installation.

Performing Migration

We are now all set to begin the migration process. The options you will use depend on your source and target databases, as well as the replication mode (snapshot or live).

ElastiCache Single Master → Redis

riotx replicate source:port target:port

Live ElastiCache Single Master → Redis

riotx replicate source:port target:port --mode live

In case ElastiCache is configured with AUTH TOKEN enabled, you need to pass --source-tls as well as --source-pass option:

riotx replicate source:port target:port --source-tls --source-pass <password>

ElastiCache Cluster → Redis

riotx replicate source:port target:port --source-cluster
--cluster is an important parameter used ONLY for ElastiCache whenever cluster-mode is enabled. Do note that the source database is specified first and the target database is specified after the replicate command and it is applicable for all the scenarios.

ElastiCache Single Master → Redis (with specific database index)

riotx replicate redis://source:port/db target:port

ElastiCache Single Master → Redis with OSS Cluster

riotx replicate source:port target:port --target-cluster

Live ElastiCache Cluster → Redis with OSS Cluster

riotx replicate source:port target:port --source-cluster --target-cluster --mode live

Important Considerations

  • It is recommended to test migration in UAT before production use.

  • Once migration is completed, ensure that application traffic gets redirected to Redis endpoint successfully.

  • It is recommended to perform the migration process during low traffic hours so as to avoid chances of data loss.

CloudFormation

For the easiest migration experience, use the CloudFormation template that automates the entire process. This template automatically discovers your ElastiCache configuration and handles all the infrastructure setup.

Step-by-Step Migration Guide

Step 1: Gather Required Information

Before starting, you’ll need:

  1. ElastiCache Cluster ID:

    • Go to AWS Console → ElastiCache

    • Find your Redis cluster and copy the Name (not the endpoint)

    • Examples: my-redis-cluster, production-cache-001

  2. Target Redis URI:

    • Get this from your Redis Cloud or Redis Enterprise provider

    • Format: redis://username:password@host:port or rediss://username:password@host:port

    • Example: redis://default:mypassword@redis-12345.c1.us-west-2.ec2.redns.redis-cloud.com:12345

Step 2: Deploy the Migration Template

  1. Open the CloudFormation Console:

  2. Configure the Stack:

    • Stack name: Keep riotx-elasticache-sync or choose your own name

    • Click Next

  3. Fill Required Parameters:

    • Source ElastiCache Cluster ID: Enter your ElastiCache cluster name from Step 1

    • Target Redis URI: Enter your Redis Cloud/Enterprise URI from Step 1

    • Leave all other settings as defaults (they work for most cases)

    • Click Next

  4. Review and Deploy:

    • Scroll down and check ☑ I acknowledge that AWS CloudFormation might create IAM resources

    • Click Submit

Step 3: Monitor the Migration

The migration starts automatically when the CloudFormation stack is created.

  1. Watch Stack Progress:

    • Stay on the CloudFormation console

    • Watch the Events tab - you’ll see resources being created

    • The stack typically takes 5-10 minutes to complete

  2. Check Migration Status:

    • When the stack shows CREATE_COMPLETE, go to the Outputs tab

    • Look for MigrationExecutionMessage - it will show "migration completed successfully"

  3. View Migration Logs (optional):

    • In the Outputs tab, click the LogGroupUrl link

    • This takes you directly to CloudWatch logs showing migration details

    • Or copy the ViewLogsCommand and run it in AWS CLI

Step 4: Verify Migration Success

  1. Check Your Target Redis:

    • Connect to your Redis Cloud/Enterprise database

    • Verify that your data has been migrated

    • Check key counts: INFO keyspace

  2. Review Migration Details:

    • The CloudFormation Outputs tab shows discovered source configuration

    • Check SourceRedisURI, SourceClusterMode, SourceTransitEncryption

    • Verify these match your ElastiCache setup

Advanced Configuration (Optional)

If you need to customize the migration, expand the parameter sections when filling the template:

Migration Settings:

  • Sync Mode: Choose SCAN (one-time), LIVE (continuous), or LIVEONLY

  • Compare Mode: Choose QUICK for basic verification or FULL for complete comparison

  • Dry Run: Set to true to test without actually copying data

Performance Tuning:

  • Batch Size: Increase from 50 to 200+ for faster migration of small keys

  • Thread Count: Increase from 1 to 4+ for parallel processing

Authentication (if needed):

  • Source User/Password: Only if your ElastiCache has AUTH enabled

  • Target Cluster Mode: Set to true if your target Redis uses cluster mode

What the Template Does Automatically

  • ✅ Discovers your ElastiCache VPC, subnets, and security groups

  • ✅ Configures network access between ElastiCache and the migration container

  • ✅ Creates ECS infrastructure to run the migration

  • ✅ Runs the migration automatically during stack creation

  • ✅ Sets up CloudWatch logging for monitoring

  • ✅ Reuses existing infrastructure if available

Typical Workflow

For comprehensive testing and migration:

  1. Prerequisites for Live Migration:

  2. Migrate Data:

    • Deploy ec-sync.yaml to migrate from ElastiCache to target Redis

    • Choose migration mode: SCAN (snapshot), LIVE (scan + live changes), or LIVEONLY (live only)

    • Monitor progress via CloudWatch logs

  3. Verify Migration:

    • Check CloudFormation stack outputs for completion status

    • Review CloudWatch logs for any errors

    • Verify data in target Redis database

Monitoring and Logging

Both CloudFormation templates include enhanced logging capabilities that make it easy to monitor and troubleshoot RIOT-X container operations:

Enhanced Logging Features:

  • CloudWatch Log Group: Automatically created with 14-day retention

  • Direct Log Access URLs: CloudFormation outputs include direct links to view logs

  • AWS CLI Commands: Ready-to-use commands for viewing and tailing logs

  • Enhanced Log Configuration: Better timestamping and buffering

Accessing Logs:

After deploying a CloudFormation stack, you can access logs in several ways:

Via CloudFormation Outputs:

  1. Go to CloudFormation console → your stack → Outputs tab

  2. Look for these outputs:

    • LogGroupUrl: Direct link to CloudWatch logs in AWS Console

    • ViewLogsCommand: AWS CLI commands to view recent logs

    • LogGroup: Name of the CloudWatch log group

Via AWS Console:

  1. Go to CloudWatch in the AWS Console

  2. Navigate to "Log groups"

  3. Find the log group (default: /ecs/riot-migration or /ecs/riot-generation)

  4. Click on log streams to view container output

Via AWS CLI:

# List recent log streams
aws logs describe-log-streams --log-group-name "/ecs/riot-migration" --order-by LastEventTime --descending

# View latest logs (real-time tailing)
aws logs tail "/ecs/riot-migration" --follow

# Get specific log events
aws logs get-log-events --log-group-name "/ecs/riot-migration" --log-stream-name "ecs/container-name/task-id"

Log Stream Naming Convention:

ECS creates log streams with this format: ecs/[container-name]/[task-id]

For example: ecs/riot-migration/a1b2c3d4-e5f6-7890-abcd-ef1234567890

What You’ll See in the Logs:

  • Migration progress information ("Scanning", "Comparing")

  • Connection details (source ElastiCache, target Redis Cloud)

  • Job execution status and timing

  • Configuration parameters and validation

  • Error messages and troubleshooting information

Troubleshooting

Common Issues:

  • Stack Creation Failed: Check CloudWatch logs for detailed error messages

  • Permission Errors: Ensure your AWS user has CloudFormation and ECS permissions

  • Network Connectivity: Verify security groups allow Redis connectivity

  • ElastiCache Not Found: Double-check cluster ID and AWS region

  • Target Redis Unreachable: Verify Redis Cloud URI and credentials

Logging Troubleshooting:

If you don’t see logs:

  1. Check ECS Task Status: Ensure the task started successfully

  2. Verify IAM Permissions: The task execution role needs CloudWatch Logs permissions

  3. Check Network Connectivity: Tasks need internet access to pull container images

  4. Review Task Definition: Ensure the log configuration is properly set

Getting Help:

  • CloudWatch Logs: /ecs/riot-migration log group contains detailed execution logs

  • CloudFormation Events: Check stack events for resource creation details

  • Stack Outputs: Review outputs for auto-discovered configuration and status