How to Setup a Redis database on EC2

Introduction

Redis a fast, reliable in-memory datastore with a wide range of supported languages. Its speed and simplicity makes it a tool of choice for developers looking to add caching functionality, job queueing, durable key-value storage, and other uses. AWS makes it easy to run a Redis database - a couple of clicks through the Elasticache interface will deploy a new Redis cluster with daily backups.

You also have other options. Once the use case is established and your stack will depend on Redis going forward, you could choose to “self-host” (i.e. not managed) Redis on EC2.

After following this guide, you will have:

1. Redis v4.0.11 installed on an Ubuntu 16.04 instance
2. Automated backups to an S3 bucket of your choosing for recovery

But first, let’s discuss the decision between Redis on ElastiCache or EC2:

Reasons to run Redis on EC2

Cost Savings

ElastiCache charges double the price of the base instance for the convenience. Running an on-demand r4.large on EC2 and setting up Redis is 50% cheaper than using an cache.r4.large. Running Redis on EC2 can also be done on spot instances, enabling an additional saving of ~70% (assuming the use case permits using a spot instance). These savings could be up to thousands of dollars per month depending on the size of your Redis cluster.

More Flexibility

Configuring an instance for Redis allows an unmatched level of custom control. ElastiCache restricts the ability to install any version of Redis, easily set up backups on a custom schedule, and use any instance type available in EC2 (including some that are cheaper per GB of memory).

Reasons to stay with ElastiCache

Uncertain Needs

ElastiCache is a great choice if you’re not sure you’ll need Redis in the long-term. Experiment and prototype with a managed Redis to be sure it’s worth the investment to learn and host it yourself.

Maintenance burden

The cognitive load of maintaining infrastructure is meaningful. Learning how to use IAM roles, security groups, EC2, S3, etc can be daunting, and it could be worth the premium to offload the concern to a managed service.

Prerequisites

To complete this guide you’ll need:

• An AWS Account with a VPC. It should be created by default
• An SSH key you can use to log into EC2 instance

Create the infrastructure

This Redis setup will create

• an EC2 instance
• an S3 bucket for backups
• an IAM role and instance policy that will allow your instance to write to that bucket.
• A security group to restrict access to the EC2 instance

Via Terraform

One option is to use Terraform to automate the creation of these dependencies. Terraform is a tool that lets you describe and deploy infrastructure as code. We’ve written the following steps as a terraform script. Follow the instructions in the README and Terraform will deploy the necessary dependencies and you can skip ahead to configure the instance

Via AWS Console UI

First, go to the S3 section and create a new bucket. Make sure the bucket is private (that should be the default). You can optionally choose more custom options like deleting objects older than X days or storing all objects using less frequent access for cost saving.

Now go to IAM section, click Roles, then Create role. First, select EC2 for the service that will use this role. Then, under Attach permissions policies, select Create policy. Select the JSON tab to input the policy definition directly. Enter the following

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::name-of-your-bucket",
                "arn:aws:s3:::name-of-your-bucket/*"
            ]
        }
    ]
}

Make sure to replace the placeholder with the newly created bucket’s name. This statement allows any service with this role to view and modify content within the bucket. Proceed to the next step, give the policy a descriptive name, and then create it. Back in the Create Role interface, you should be able the refresh the dropdown by clicking the spinning arrows button and then find and select your policy by name. Then proceed to the next step, give your role a name, and then click Create. Now you have a role (and the UI automatically handles creating an instance policy for that role) that you can give your Redis instance.

Now, go to the security groups UI (EC2 -> Security Groups) and start to create a new group. ElastiCache instances by default are only accessible from resources within your VPC. To recreate that rule, we’ll need to add an inbound rule from your VPC’s private IPs. We’ll also need to allow an SSH connection to allow us to get onto the instance. Create your group with the settings below.

Now we’re ready to create the instance. Go again to the EC2 UI and select Launch Instance. From the Quick Start Menu, select the Ubuntu Server 16.04 LTS (HVM) option. Select your instance type and click Configure Instance Details. For IAM Role, select the newly created role from the dropdown and then select Next: Add Storage. Under Size (GiB) make sure to allocate enough disk space for the instance so that it can safely store the entire Redis database on disk and have space for logs, etc. Allowing 20GiB of disk space for an r4.large should be plenty. Proceed to the Tags step, add anything you’d like, then proceed to Configure Security Groups. Select Select an existing security group and choose the newly created one. Click Review and Launch, then Launch. Grab the public IP address from the EC2 instance in the UI.

Configure the instance

We’ve created a publicly available script that handles:

• Installing Redis v4.0.11
• Running Redis as a service with systemd
• Adjusting the default Redis settings
• Tweaking the system configuration to fix warnings when running Redis
• Creating a backup to S3 utility and configuring cron to run it daily

Feel free to tweak the script with settings based on your use case. You may choose to back up to S3 more frequently, change some redis settings, etc. We’ll walk through some important sections of the script that we’ve updated.

Redis configuration

40
41
42
43

################################## NETWORK #####################################
bind 0.0.0.0
protected-mode yes
port 6379

We enabled redis to serve traffic from anywhere as we’re going to let the instance’s security group do access control.

47
48
49

################################# GENERAL #####################################
daemonize no
supervised systemd

systemd will allow you to manage the redis server with commands like sudo service redis stop, sudo service redis start, etc.

55
56
57
58
59
60
61

################################ SNAPSHOTTING  ################################
save 450 1
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis

Redis will dump a backup to /var/lib/redis/dump.rdb every 450 seconds if 1 or more keys have changed.

70
71
72
73

################################### CLIENTS ####################################
maxclients 100000
############################## MEMORY MANAGEMENT ################################
maxmemory 4GB

System configuration

163
164
165
166
167
168
169

cat > /usr/local/bin/redis_backup << EOF
#!/usr/bin/env bash
bucket_name=FILL_ME_IN
dt=\$(date "+%Y%m%d-%H:%M")
aws s3 cp /var/lib/redis/dump.rdb s3://\${bucket_name}/redis-backups/\${dt}-dump.rdb
EOF
chmod +x /usr/local/bin/redis_backup

The S3 backup script will upload Redis’ dump.rdb file to S3 with the date in the name. Make sure to edit the bucket name

172
173
174
175
176
177

cat > cronconfig << EOF
PATH=/home/ubuntu/bin:/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
0 * * * * /usr/local/bin/redis_backup > /dev/null
EOF
crontab cronconfig
rm cronconfig

Cron will manage running the script. As is, it will run everyday at midnight UTC. Feel free to adjust it using cron syntax

Conclusion

Now you should have a functioning Redis server on EC2. To reach it from your other infrastructure, use the private ip of the instance like 172.31.xxx.xxx. Here are some additional ways to improve your setup

Use an infrastructure monitoring service

A service like Datadog or New Relic will monitor your instance health, report on key metrics specific to your Redis database, and alert you if something needs addressing. Datadog has a free tier of 5 instances.

Set up a Route53 internal hosted zone and create a DNS entry for your Redis instance

Rather than using the private IP address, consider setting up an internal hosted zone using Route53. You can then address your redis service with a more friendly name like redis.internal-domain.com in your application code. It’s also convenient if you want to migrate your redis to a different instance and not update your application’s source code.

Email alerts on backup failures

Cron can be configured to send emails when jobs have unexpected output to stderr. You’ll need to set up SMTP settings on your instance and modify your crontab slightly