Add support for redis with read replication

[simoncoggins]

Currently we support a single Redis node but not a read-only replica setup. This has made it hard to test certain configurations, leading to bugs such as TL-36554 slipping through. Would be great to be able to easily test a production-like setup.

Sam has the configuration for Redis sentinel which we could use as a starting point.

[TotaraAdmin]

So prior to today's endeavour I had been running everything locally, not within docker. It worked for me but would be a hard thing to document with many steps, and it'd be a pain to maintain for most people unless they invested in actually understanding redis and sentinel.
I had luck getting a docker compose file up that spins up two redis instances one as master, one as a slave, and then 2 - 3 sentinel instances.
The compose file is:

version: '2'

networks:
    app-tier:
        driver: bridge

services:
  redis:
    image: 'bitnami/redis:latest'
    environment:
      - REDIS_REPLICATION_MODE=master
      - REDIS_PASSWORD=masterpassword123
    networks:
      - app-tier
    ports:
      - '6379:6379'
  redis-slave:
    image: 'bitnami/redis:latest'
    environment:
      - REDIS_REPLICATION_MODE=slave
      - REDIS_MASTER_HOST=redis
      - REDIS_MASTER_PASSWORD=masterpassword123
      - REDIS_PASSWORD=masterpassword123
    ports:
      - '6380:6379'
    depends_on:
      - redis
    networks:
      - app-tier
  redis-sentinel:
    image: 'bitnami/redis-sentinel:latest'
    links:
        - redis
        - redis-slave
    environment:
        - REDIS_SENTINEL_PASSWORD=masterpassword123
        - REDIS_MASTER_PASSWORD=masterpassword123
        - REDIS_SENTINEL_RESOLVE_HOSTNAMES=yes
        - REDIS_SENTINEL_ANNOUNCE_IP=127.0.0.1
    depends_on:
      - redis
      - redis-slave
    ports:
      - "26379-26381:26379"
    networks:
      - app-tier

This is then brought up by running:

docker-compose up --scale redis-sentinel=2 -d

There is a problem with it that someone better than I am at docker would still need to solve.
Totara requires that both sentinel and redis be available. The redis nodes get registered with sentinel and then Totara queries sentinel to get the redis services.
Presently, they are exposed via ports. However this isn't enough.
When you ask Sentinel to give you the hosts and ports for redis it returns an IP address associated with the container network, e.g. 172.18.0.3.
To work around this I hacked core code to map from internal IP's to exposed ports on the local machine. But a super hack.

diff --git a/server/lib/classes/redis/sentinel.php b/server/lib/classes/redis/sentinel.php
index f32bfbf71182..5691775842f8 100644
--- a/server/lib/classes/redis/sentinel.php
+++ b/server/lib/classes/redis/sentinel.php
@@ -192,6 +192,20 @@ final class sentinel {
                 }
                 $masterhost = $master[0];
                 $masterport = $master[1];
+
+                switch ($masterhost) {
+                    case '172.18.0.3':
+                        // Slave
+                        $masterhost = '127.0.0.1';
+                        $masterport = '6380';
+                        break;
+                    case '172.18.0.2':
+                        // Master
+                        $masterhost = '127.0.0.1';
+                        $masterport = '6379';
+                        break;
+                }
+
                 try {
                     $redis = new \Redis();
                     if (!$redis->connect($masterhost, $masterport, 3)) {

Then to configure Totara:

1. Add a new Redis cache store
2. Sentinel hosts: 127.0.0.1:26380
3. Sentinel master name: mymaster
4. Sentinel password: masterpassword123
5. Read replica server: 127.0.0.6379
6. Database number: 777
7. Key prefix: dev

Save that, and check on the next screen the cache store shows as ready.
Finally change you default application mapping to your new redis cache store.
Go!