Introduction to Redis

Master-Slave Replication

Redis's Master-Slave replication starts with at least two Redis server instances; a master and one or more slave instances. All Redis server instances store a copy of data. The difference between the two is that the slaves are usually read-only while the master instance accepts reads and writes.

Exercise: Runtime Master and Slave instances

Open four command-line windows to run a Redis master, slave, and redis-cli instances

Redis Master

$ cd introduction-to-redis/replication-and-pub-sub/
$ ~/redis-3.0.2/src/redis-server --dbfilename master.rdb --port 6379      
       

Redis Slave

$ cd introduction-to-redis/replication-and-pub-sub/
$ ~/redis-3.0.2/src/redis-server --dbfilename slave.rdb --port 6380
       

Redis CLI

$ ~/redis-3.0.2/src/redis-cli -p 6379
127.0.0.1:6379> DBSIZE
(integer) 0
127.0.0.1:6379> exit
$
$ ~/redis-3.0.2/src/redis-cli -p 6380
127.0.0.1:6380> DBSIZE
(integer) 0
127.0.0.1:6380> SLAVEOF localhost 6379
OK
127.0.0.1:6380> exit
$
$ ~/redis-3.0.3/src/redis-cli -p 6379
127.0.0.1:6379> HMSET Book:1 name "Infinite Jest" author "David Foster Wallace"
OK
127.0.0.1:6379> exit
$
$ ~/redis-3.0.2/src/redis-cli -p 6380
127.0.0.1:6380> KEYS *
127.0.0.1:6380> KEYS *
1) "Book:1"
127.0.0.1:6380> HGETALL Book:1
1) "name"
2) "Infinite Jest"
3) "author"
4) "David Foster Wallace"
127.0.0.1:6380> HSET Book:1 copyrightYear 1996
(error) READONLY You can't write against a read only slave.
       

Exercise: Set-up of Master-Slave Replication with redis.config

Master-Slave replication can be set-up in the redis.conf file for each slave.

# Master-Slave replication. Use slaveof to make a Redis instance a copy of
# another Redis server.
slaveof <masterip> <masterport>
       
Facts about Master-Slave Replication
  • Master and Slaves use asynchronous replication
  • Masters can have multiple slaves
  • Slaves can accept connections from other slaves (i.e. a slave could have as it's master a Redis instance that is slave to a master instance)
  • Replication is non-blocking on both master and slave instances; a master can continue to respond to queries while slaves are synchronizing, a slave can be configured to respond to queries with old data while synchronizing with the master

Critical Commands for Master-Slave Replication

Caveats when using Replication

If persistence is turned off on the master A and it has two slaves B, C the following problem can occur:

  • We have a setup with node A acting as master, with persistence turned off, and nodes B and C replicating from node A.
  • A crashes, however it has some auto-restart system, that restarts the process. However since persistence is turned off, the node restarts with an empty data set.
  • Nodes B and C will replicate from A, which is empty, so they'll effectively destroy their copy of the data.

SYNC, Partial Resynchronization & Diskless Replication

Redis Replication

Steps:
  1. The master then starts background saving, and starts to buffer all new commands received that will modify the dataset.
  2. When the background saving is complete, the master transfers the database file to the slave, which saves it on disk, and then loads it into memory.
  3. The master will then send to the slave all buffered commands. This is done as a stream of commands and is structured in the Redis.

Partial Resynchronization

Replication is non-blocking on both master and slave instances; a master can continue to respond to queries while slaves are synchronizing, a slave can be configured to respond to queries with old data while synchronizing with the master

If the link between the master and slave goes down, the old behavior of Redis is to do a full synchronization with the slave when the connection is re-established. As of Redis 2.8 experimental support was added with PSYNC command that will only do a partial synchronization with MASTER based on if a time threshold has been exceeded or not. If time has been exceeded, a full SYNC is triggered between the master and slave.

Diskless replication

An experimental feature that sends the full RDB file to the slave without forking the master and using the disk as an intermediate storage.


Pub/Sub (Publication/Subscription)

Critical Commands for Pub/Sub

Exercise: Pub/Sub

In this exercise, we will use redis-cli as a publisher and use the a Python command line to run a consumer application using PUBLISH and SUBSCRIBE commands. The Python client redis-py has its own Publish/Subscribe class with its own method calls for responding to messages published to a channel.

Redis Python Subscription Client

>>> import redis
>>> local_redis = redis.StrictRedis()
>>> circ_consumer = local_redis.pubsub()
>>> circ_consumer.subscribe("checkouts", "checkins")
>>> circ_consumer.get_message()
       

Redis redis-cli Publisher Client

127.0.0.1:6379> publish checkouts Book:3
(integer) 1
127.0.0.1:6379> publish checkouts Book:1
(integer) 1
127.0.0.1:6379> publish checkins Book:3
(integer) 1
127.0.0.1:6379> publish checkins Book:3
(integer) 1
       

Pub/Sub Message Syntax

Redis Pub/Sub messages are defined as 3 or 4 part RESP (REdis Serialization Protocol) array. RESP is the communication protocol that Redis client communicate with Redis server and is used in other projects beside Redis. We see the raw RESP protocol if we use telnet and connect directly to Redis.

First, we will connect to a single Redis instance running at port 6379 with telnet.

$ telnet localhost 6379
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
        

Now, we will subscribe to a chat_room channel

and from a redis-cli session, we will post a message to the channel.
SUBSCRIBE chat_room
*3
$9
subscribe
$9
chat_room
:1
*3
$7
message
$9
chat_room
$18
Anybody out there?
        

Pattern Matching with PSUBSCRIBE & PUNSUBSCRIBE

To subscribe to more than one channel, you can use the PSUBSCRIBE that take a pattern to match to existing channels.

PSUBSCRIBE chat_*
*3
$10
psubscribe
$6
chat_*
:1
*4
$8
pmessage
$6
chat_*
$9
chat_room
$18
What is your name?
        

To unsubscribe to multiple channels at once, you can use the PUNSUBSCRIBE

PUNSUBSCRIBE chat*
*3
$12
punsubscribe
$5
chat*
:1
    

References and Resources

  1. From the redis.io website: topic on Replication
  2. Topic on *Pub/Sub