Day 2

Focused Topics in Redis Day 1 - #4

Introduction to Redis Cluster

In this topic …

Redis Cluster provides one method of running a Redis installation where data is sharded across multiple Redis instances while providing a degree of availability for continuing operations in the event of partial node failure or communication failure.

This topics starts with an overview of a minimal Redis cluster with an demonstration of the composite hashing approach used by Redis

Partitioning

Composite Partitioning

Redis cluster uses a form of composite partitioning called consistent hashing that calculates what Redis instance any particular key is assigned called a hash slot.

The hash slot is the CRC16 hash algorithm applied to the key and then the computation of a modulo using 16384. The specific algorithm used by Redis cluster to calculate the hash slot for a key:

  1. The cyclic redundancy check (CRC) using a polynomial length of 17-bits or CRC16 is calculated on the Redis key
  2. The modulo 16384 of the value is then calculated to get the hash slot for the key.

Each master node is assigned a sub-range of hash slots and the key and value will reside on that Redis instance.

Cluster Topology & Redis Cluster Protocol

Mesh Network

Nodes in the Redis Cluster use the Redis cluster protocol to connect with every other node in the Redis cluster for a mesh network topology. Nodes communicate using a TCP gossip protocol and along with a configuration update mechanism in order to reduce the number of messages being exchanged between nodes.

When a Redis cluster is running, each node has two TCP sockets open:

Clients should never need to connect directly with the cluster bus port but with the lower, standard port.

Six-Node Redis Cluster

To illustrate all of the features of Redis Cluster, the minimum recommended Redis cluster setup is to have a 3 Master Redis nodes with each master node replicated with a single Redis slave instance node. Each Master Node hash slots are broken down as

Master One (M1) - allocated hash slots 0-5460
Slave One (S1) replicates Master One, is promoted if a quorum of nodes cannot reach Master One.
Master Two (M2) - allocated hash slots 5461-10922
Slave Two (S2) replicates Master Two, is promoted if a quorum of nodes cannot reach Master Two.
Master Three (M3) - allocated hash slots 10923-16383
Slave Three (S3) replicates Master Three, is promoted if a quorum of nodes cannot reach Master Three.

Manually Running a Redis Cluster

We'll start by manually create a minimal cluster with three master nodes and three slaves. 

$ mkdir cluster-test
$ cd cluster-test
$ mkdir 7000 7001 7002 7003 7004 7005
$ touch 7000/redis.conf
$ vi 7000/redis.conf
$ cp 7000/redis.conf 7001/.
$ vi 7001/redis.conf
...
$ cp ~/redis/src/redis-server .
$ cd 7000
$ ~/redis/src/redis-server redis.conf
# New terminal tab
$ cd ../7001
$ ~/redis/src/redis-server redis.conf
...
$ ~/redis/src/redis-server redis.conf

Install the Ruby Redis gem and run the redis-trib.rb

$ sudo gem install redis
$ cd ../../
$ .~/redis/src/redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 \
127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
Sample redis.conf

Here is the example redis.conf we are using for each of Redis cluster nodes

port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

We need to change the port number to the correct value for the node, i.e. for node 7002, we need specify the correct port number for that node's redis.conf configuration file.

Easier Redis Cluster with create-cluster script

Instead of manually setting up and running a Redis cluster, you can instead use the create-cluster located in the redis/utils/create-cluster.

Starting in the create-cluster directory, follow these steps to get a 6-node Redis Cluster with 3 masters and 3 slaves.

  1. Create a config.sh script: 
    #!/bin/bash
PORT=7000
TIMEOUT=2000
NODES=6
REPLICAS=1
  2. Start Redis Cluster: 
    ~/redis/utils/create-cluster$ ./create-cluster start
Starting 7001
Starting 7002
Starting 7003
Starting 7004
Starting 7005
Starting 7006
  3. Link and create the Redis Cluster
    4. >>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7001
127.0.0.1:7002
127.0.0.1:7003
Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
Adding replica 127.0.0.1:7006 to 127.0.0.1:7003
M: 1379949e7d8eaa27a0634285e521079eccc0cc1f 127.0.0.1:7001
   slots:0-5460 (5461 slots) master
M: 470bf4397e0002f211df09dadcd5ec12b458e9c3 127.0.0.1:7002
   slots:5461-10922 (5462 slots) master
M: 7e343391d165ccee34e0f1cf43590270130a9d5b 127.0.0.1:7003
   slots:10923-16383 (5461 slots) master
S: 3d8c532367f0846f292b538d09b7cafdc6b3c6b9 127.0.0.1:7004
   replicates 1379949e7d8eaa27a0634285e521079eccc0cc1f
S: 491abeb14973c0c9495f1b045b4e5d3f0729bcc8 127.0.0.1:7005
   replicates 470bf4397e0002f211df09dadcd5ec12b458e9c3
S: 933258e1d5ed8752c7e4ff7ce377dfd63543977f 127.0.0.1:7006
   replicates 7e343391d165ccee34e0f1cf43590270130a9d5b
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 127.0.0.1:7001)
M: 1379949e7d8eaa27a0634285e521079eccc0cc1f 127.0.0.1:7001
   slots:0-5460 (5461 slots) master
M: 470bf4397e0002f211df09dadcd5ec12b458e9c3 127.0.0.1:7002
   slots:5461-10922 (5462 slots) master
M: 7e343391d165ccee34e0f1cf43590270130a9d5b 127.0.0.1:7003
   slots:10923-16383 (5461 slots) master
M: 3d8c532367f0846f292b538d09b7cafdc6b3c6b9 127.0.0.1:7004
   slots: (0 slots) master
   replicates 1379949e7d8eaa27a0634285e521079eccc0cc1f
M: 491abeb14973c0c9495f1b045b4e5d3f0729bcc8 127.0.0.1:7005
   slots: (0 slots) master
   replicates 470bf4397e0002f211df09dadcd5ec12b458e9c3
M: 933258e1d5ed8752c7e4ff7ce377dfd63543977f 127.0.0.1:7006
   slots: (0 slots) master
   replicates 7e343391d165ccee34e0f1cf43590270130a9d5b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
  4. Stopping Redis Cluster 
    $ ./create-cluster stop
Stopping 7001
Stopping 7002
Stopping 7003
Stopping 7004
Stopping 7005
Stopping 7006

Using Redis Cluster with redis-cli

After successfully launching your Redis Cluster, an easy way to interact with it is to use the default Redis command-line client, redis-cli in cluster mode.