Chapter 8. Redis high availability (HA) support for 3scale

High availability (HA) is provided for most components by the OpenShift Container Platform (OCP). For more information see OpenShift Container Platform 3.11 Chapter 30. High Availability.

Note

The information and procedures in this section is not officially supported by Red Hat. It is for reference only.

The database components for HA in Red Hat 3scale API Management include: * backend-redis: used for statistics storage and temporary job storage. * system-redis: provides temporary storage for background jobs for 3scale and is also used as a message bus for Ruby processes of system-app pods.

Both backend-redis and system-redis work with supported Redis high availability variants for Redis Sentinel and Redis Enterprise.

If the Redis pod comes to a stop, or if the OpenShift Container Platform stops it, a new pod is automatically created. Persistent storage will restore the data so the pod continues to work. In these scenarios, there will be a small amount of downtime while the new pod starts. This is due to a limitation in Redis that does not support a multi-master setup. You can reduce downtime by preinstalling the Redis images onto all nodes that have Redis deployed to them. This will speed up the pod restart time.

To set up Redis for zero downtime and configure back-end components for 3scale, perform the steps outlined in the following sections:

8.1. Setting up Redis for zero downtime

If zero downtime is required, you must configure Redis outside of OCP. There are several ways to set it up using the configuration options of 3scale pods:

Note

Red Hat does not provide support for the above mentioned services. The mention of any such services does not imply endorsement by Red Hat of the products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) any external content.

8.2. Configuring back-end components for 3scale

There are configuration settings in 3scale 2.7 to configure Redis HA (failover) for the back-end component. You can configure these settings as environment variables in the following deployment configurations: backend-cron, backend-listener, and backend-worker.

Note

If you want to use Redis with sentinels, you must create the system-redis secret with all fields in order to configure the Redis you want to point to before deploying 3scale. The fields are not provided as parameters in the back end as of 3scale.

8.2.1. Creating backend-redis and system-redis secrets

Follow these steps to create backend-redis and system-redis secrets accordingly:

8.2.2. Deploying a fresh installation of 3scale for HA

  1. Create the backend-redis and system-redis secrets with the fields below:

    backend-redis

    REDIS_QUEUES_SENTINEL_HOSTS
REDIS_QUEUES_SENTINEL_ROLE
REDIS_QUEUES_URL
REDIS_STORAGE_SENTINEL_HOSTS
REDIS_STORAGE_SENTINEL_ROLE
REDIS_STORAGE_URL

    system-redis

    MESSAGE_BUS_NAMESPACE
MESSAGE_BUS_SENTINEL_HOSTS
MESSAGE_BUS_SENTINEL_ROLE
MESSAGE_BUS_URL
NAMESPACE
SENTINEL_HOSTS
SENTINEL_ROLE
URL

    • When configuring for Redis with sentinels, the corresponding URL fields in backend-redis and system-redis refer to the Redis group in the format redis://[:redis-password@]redis-group[/db]`, where [x] denotes optional element x and redis-password, redis-group, and db are variables to be replaced accordingly:

      Example

      redis://:redispwd@mymaster/5

    • The SENTINEL_HOSTS fields are comma-separated lists of sentinel connection strings in the following format: 

      [redis://][:sentinel-password@]sentinel-hostname-or-ip[:port]

      • For each element of the list, [x] denotes optional element x and sentinel-password, sentinel-hostname-or-ip, and port are variables to be replaced accordingly:

        Example

        :sentinelpwd@123.45.67.009:2711,:sentinelpwd@other-sentinel:2722

    • The SENTINEL_ROLE fields are either master or slave.

  2. Deploy 3scale as indicated in Deploying 3scale on OpenShift using a template, using the latest version of the templates.

    • Ignore the errors due to backend-redis and system-redis already present.

8.2.3. Migrating a non-HA deployment of 3scale to HA

  1. Edit the backend-redis and system-redis secrets with all fields as shown in Deploying a fresh installation of 3scale for HA.

  2. Make sure the following backend-redis environment variables are defined for the back-end pods. 

    name: BACKEND_REDIS_SENTINEL_HOSTS
  valueFrom:
    secretKeyRef:
      key: REDIS_STORAGE_SENTINEL_HOSTS
      name: backend-redis
name: BACKEND_REDIS_SENTINEL_ROLE
  valueFrom:
    secretKeyRef:
      key: REDIS_STORAGE_SENTINEL_ROLE
      name: backend-redis

  3. Make sure the following system-redis environment variables are defined for the system-(app|sidekiq|sphinx) pods. 

    name: REDIS_SENTINEL_HOSTS
  valueFrom:
    secretKeyRef:
      key: SENTINEL_HOSTS
      name: system-redis
name: REDIS_SENTINEL_ROLE
  valueFrom:
    secretKeyRef:
      key: SENTINEL_ROLE
      name: system-redis
name: MESSAGE_BUS_REDIS_SENTINEL_HOSTS
  valueFrom:
    secretKeyRef:
      key: MESSAGE_BUS_SENTINEL_HOSTS
      name: system-redis
name: MESSAGE_BUS_REDIS_SENTINEL_ROLE
  valueFrom:
    secretKeyRef:
      key: MESSAGE_BUS_SENTINEL_ROLE
      name: system-redis

  4. Proceed with instructions to continue upgrading 3scale 2.6 to 2.7 using templates.

    1. Follow the steps in this order:

8.2.3.1. Using Redis Enterprise

  1. Use Redis Enterprise deployed in OpenShift, with three different redis-enterprise instances:

    1. Edit system-redis secret:

      1. Set distinct values to MESSAGE_BUS_NAMESPACE and NAMESPACE.
      2. Set URL and MESSAGE_BUS_URL to the same database.
    2. Set the back-end database in backend-redis to REDIS_QUEUES_URL.
    3. Set the third database to REDIS_STORAGE_URL for backend-redis.

8.2.3.2. Using Redis Sentinel

  1. Use Redis Sentinel, with three or four different Redis databases:

    1. Edit system-redis secret:

      1. Set distinct values to MESSAGE_BUS_NAMESPACE and NAMESPACE.
      2. Set URL and MESSAGE_BUS_URL to the proper Redis group, for example: redis://:redispwd@mymaster/5
      3. Set SENTINEL_HOSTS and MESSAGE_BUS_SENTINEL_HOSTS to a comma-separated list of sentinels hosts and ports, for example: :sentinelpwd@123.45.67.009:2711,:sentinelpwd@other-sentinel:2722
      4. Set SENTINEL_ROLE and MESSAGE_BUS_SENTINEL_ROLE to master

  2. Set the backend-redis secret for back-end with the values:

    • REDIS_QUEUES_URL
    • REDIS_QUEUES_SENTINEL_ROLE
    • REDIS_QUEUES_SENTINEL_HOSTS

  3. Set the variables in the third database as follows:

    • REDIS_STORAGE_URL
    • REDIS_STORAGE_SENTINEL_ROLE
    • REDIS_STORAGE_SENTINEL_HOSTS

Notes

  • The system-app and system-sidekiq components connect directly to back-end Redis for retrieving statistics.

    • As of 3scale 2.7, these system components can also connect to back-end Redis (storage) when using sentinels.

  • The system-app and system-sidekiq components uses only backend-redis storage, not backend-redis queues.

    • Changes made to the system components support backend-redis storage with sentinels.

8.3. Additional information