Menu Close
Settings Close

Language and Page Formatting Options

Server Installation and Configuration Guide

Red Hat Single Sign-On 7.6

For Use with Red Hat Single Sign-On 7.6

Red Hat Customer Content Services

Abstract

This guide consists of information to install and configure Red Hat Single Sign-On 7.6

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.

Chapter 1. Guide Overview

The purpose of this guide is to walk through the steps that need to be completed prior to booting up the Red Hat Single Sign-On server for the first time. If you just want to test drive Red Hat Single Sign-On, it pretty much runs out of the box with its own embedded and local-only database. For actual deployments that are going to be run in production you’ll need to decide how you want to manage server configuration at runtime (standalone or domain mode), configure a shared database for Red Hat Single Sign-On storage, set up encryption and HTTPS, and finally set up Red Hat Single Sign-On to run in a cluster. This guide walks through each and every aspect of any pre-boot decisions and setup you must do prior to deploying the server.

One thing to particularly note is that Red Hat Single Sign-On is derived from the JBoss EAP Application Server. Many aspects of configuring Red Hat Single Sign-On revolve around JBoss EAP configuration elements. Often this guide will direct you to documentation outside of the manual if you want to dive into more detail.

Chapter 2. Installing the software

You can install Red Hat Single Sign-On by downloading a ZIP file and unzipping it, or by using an RPM. This chapter reviews system requirements as well as the directory structure.

2.1. Installation prerequisites

These prerequisites exist for installing the Red Hat Single Sign-On server:

  • Can run on any operating system that runs Java
  • Java 8 JRE or Java 11 JRE
  • zip or gzip and tar
  • At least 512M of RAM
  • At least 1G of diskspace
  • A shared external database like PostgreSQL, MySQL, Oracle, etc. Red Hat Single Sign-On requires an external shared database if you want to run in a cluster. Please see the database configuration section of this guide for more information.
  • Network multicast support on your machine if you want to run in a cluster. Red Hat Single Sign-On can be clustered without multicast, but this requires a bunch of configuration changes. Please see the clustering section of this guide for more information.
  • On Linux, it is recommended to use /dev/urandom as a source of random data to prevent Red Hat Single Sign-On hanging due to lack of available entropy, unless /dev/random usage is mandated by your security policy. To achieve that on Oracle JDK 8 and OpenJDK 8, set the java.security.egd system property on startup to file:/dev/urandom.

2.2. Installing RH-SSO from a ZIP file

The Red Hat Single Sign-On server download ZIP file contains the scripts and binaries to run the Red Hat Single Sign-On server. You install the 7.6 server first, then the 7.6.1 server patch.

Procedure

  1. Go to the Red Hat customer portal.
  2. Download the Red Hat Single Sign-On 7.6 server.
  3. Unpack the ZIP file using the appropriate unzip utility, such as unzip, tar, or Expand-Archive.
  4. Return to the Red Hat customer portal.
  5. Click the Patches tab.
  6. Download the Red Hat Single Sign-On 7.6.1 server patch.
  7. Place the downloaded file in a directory you choose.
  8. Go to the bin directory of JBoss EAP.
  9. Start the JBoss EAP command line interface.

    Linux/Unix

    $ jboss-cli.sh

    Windows

    > jboss-cli.bat

  10. Apply the patch.

    $ patch apply <path-to-zip>/rh-sso-7.6.1-patch.zip

Additional resources

For more details on applying patches, see Patching a ZIP/Installer Installation.

2.3. Installing RH-SSO from an RPM

Note

With Red Hat Enterprise Linux 7 and 8, the term channel was replaced with the term repository. In these instructions only the term repository is used.

You must subscribe to both the JBoss EAP 7.4 and RH-SSO 7.6 repositories before you can install RH-SSO from an RPM.

Note

You cannot continue to receive upgrades to EAP RPMs but stop receiving updates for RH-SSO.

2.3.1. Subscribing to the JBoss EAP 7.4 repository

Prerequisites

  1. Ensure that your Red Hat Enterprise Linux system is registered to your account using Red Hat Subscription Manager. For more information see the Red Hat Subscription Management documentation.
  2. If you are already subscribed to another JBoss EAP repository, you must unsubscribe from that repository first.

For Red Hat Enterprise Linux 6, 7: Using Red Hat Subscription Manager, subscribe to the JBoss EAP 7.4 repository using the following command. Replace <RHEL_VERSION> with either 6 or 7 depending on your Red Hat Enterprise Linux version.

subscription-manager repos --enable=jb-eap-7.4-for-rhel-<RHEL_VERSION>-server-rpms --enable=rhel-<RHEL_VERSION>-server-rpms

For Red Hat Enterprise Linux 8: Using Red Hat Subscription Manager, subscribe to the JBoss EAP 7.4 repository using the following command:

subscription-manager repos --enable=jb-eap-7.4-for-rhel-8-x86_64-rpms --enable=rhel-8-for-x86_64-baseos-rpms --enable=rhel-8-for-x86_64-appstream-rpms

2.3.2. Subscribing to the RH-SSO 7.6 repository and installing RH-SSO 7.6

Prerequisites

  1. Ensure that your Red Hat Enterprise Linux system is registered to your account using Red Hat Subscription Manager. For more information see the Red Hat Subscription Management documentation.
  2. Ensure that you have already subscribed to the JBoss EAP 7.4 repository. For more information see Subscribing to the JBoss EAP 7.4 repository.

Procedure

  1. For Red Hat Enterprise Linux 6, 7: Using Red Hat Subscription Manager, subscribe to the RH-SSO 7.6 repository using the following command. Replace <RHEL_VERSION> with either 6 or 7 depending on your Red Hat Enterprise Linux version.

    subscription-manager repos --enable=rh-sso-7.6-for-rhel-<RHEL-VERSION>-server-rpms
  2. For Red Hat Enterprise Linux 8: Using Red Hat Subscription Manager, subscribe to the RH-SSO 7.6 repository using the following command:

    subscription-manager repos --enable=rh-sso-7.6-for-rhel-8-x86_64-rpms
  3. For Red Hat Enterprise Linux 6, 7: Install RH-SSO from your subscribed RH-SSO 7.6 repository using the following command:

    yum groupinstall rh-sso7
  4. For Red Hat Enterprise Linux 8: Install RH-SSO from your subscribed RH-SSO 7.6 repository using the following command:

    dnf groupinstall rh-sso7

Your installation is complete. The default RH-SSO_HOME path for the RPM installation is /opt/rh/rh-sso7/root/usr/share/keycloak.

Additional resources

For details on installing the 7.6.1 patch for Red Hat Single Sign-On, see RPM patching.

2.4. Important directories

The following are some important directories in the server distribution.

bin/
This contains various scripts to either boot the server or perform some other management action on the server.
domain/
This contains configuration files and working directory when running Red Hat Single Sign-On in domain mode.
modules/
These are all the Java libraries used by the server.
standalone/
This contains configuration files and working directory when running Red Hat Single Sign-On in standalone mode.
standalone/deployments/
If you are writing extensions to Red Hat Single Sign-On, you can put your extensions here. See the Server Developer Guide for more information on this.
themes/
This directory contains all the html, style sheets, JavaScript files, and images used to display any UI screen displayed by the server. Here you can modify an existing theme or create your own. See the Server Developer Guide for more information on this.

Chapter 3. Using operating modes

Before deploying Red Hat Single Sign-On in a production environment you need to decide which type of operating mode you are going to use.

  • Will you run Red Hat Single Sign-On within a cluster?
  • Do you want a centralized way to manage your server configurations?

Your choice of operating mode affects how you configure databases, configure caching and even how you boot the server.

Tip

The Red Hat Single Sign-On is built on top of the JBoss EAP Application Server. This guide will only go over the basics for deployment within a specific mode. If you want specific information on this, a better place to go would be the JBoss EAP Configuration Guide.

3.1. Using standalone mode

Standalone operating mode is only useful when you want to run one, and only one Red Hat Single Sign-On server instance. It is not usable for clustered deployments and all caches are non-distributed and local-only. It is not recommended that you use standalone mode in production as you will have a single point of failure. If your standalone mode server goes down, users will not be able to log in. This mode is really only useful to test drive and play with the features of Red Hat Single Sign-On

3.1.1. Booting in standalone mode

When running the server in standalone mode, there is a specific script you need to boot the server depending on your operating system. These scripts live in the bin/ directory of the server distribution.

Standalone Boot Scripts

standalone boot files

To boot the server:

Linux/Unix

$ .../bin/standalone.sh

Windows

> ...\bin\standalone.bat

3.1.2. Standalone configuration

The bulk of this guide walks you through how to configure infrastructure level aspects of Red Hat Single Sign-On. These aspects are configured in a configuration file that is specific to the application server that Red Hat Single Sign-On is a derivative of. In the standalone operation mode, this file lives in …​/standalone/configuration/standalone.xml. This file is also used to configure non-infrastructure level things that are specific to Red Hat Single Sign-On components.

Standalone Config File

standalone config file

Warning

Any changes you make to this file while the server is running will not take effect and may even be overwritten by the server. Instead use the command line scripting or the web console of JBoss EAP. See the JBoss EAP Configuration Guide for more information.

3.2. Using standalone clustered mode

Standalone clustered operation mode applies when you want to run Red Hat Single Sign-On within a cluster. This mode requires that you have a copy of the Red Hat Single Sign-On distribution on each machine where you want to run a server instance. This mode can be very easy to deploy initially, but can become quite cumbersome. To make a configuration change, you modify each distribution on each machine. For a large cluster, this mode can become time consuming and error prone.

3.2.1. Standalone clustered configuration

The distribution has a mostly pre-configured app server configuration file for running within a cluster. It has all the specific infrastructure settings for networking, databases, caches, and discovery. This file resides in …​/standalone/configuration/standalone-ha.xml. There’s a few things missing from this configuration. You can’t run Red Hat Single Sign-On in a cluster without configuring a shared database connection. You also need to deploy some type of load balancer in front of the cluster. The clustering and database sections of this guide walk you through these things.

Standalone HA Config

standalone ha config file

Warning

Any changes you make to this file while the server is running will not take effect and may even be overwritten by the server. Instead use the command line scripting or the web console of JBoss EAP. See the JBoss EAP Configuration Guide for more information.

3.2.2. Booting in standalone clustered mode

You use the same boot scripts to start Red Hat Single Sign-On as you do in standalone mode. The difference is that you pass in an additional flag to point to the HA config file.

Standalone Clustered Boot Scripts

standalone boot files

To boot the server:

Linux/Unix

$ .../bin/standalone.sh --server-config=standalone-ha.xml

Windows

> ...\bin\standalone.bat --server-config=standalone-ha.xml

3.3. Using domain clustered mode

Domain mode is a way to centrally manage and publish the configuration for your servers.

Running a cluster in standard mode can quickly become aggravating as the cluster grows in size. Every time you need to make a configuration change, you perform it on each node in the cluster. Domain mode solves this problem by providing a central place to store and publish configurations. It can be quite complex to set up, but it is worth it in the end. This capability is built into the JBoss EAP Application Server which Red Hat Single Sign-On derives from.

Note

The guide will go over the very basics of domain mode. Detailed steps on how to set up domain mode in a cluster should be obtained from the JBoss EAP Configuration Guide.

Here are some of the basic concepts of running in domain mode.

domain controller
The domain controller is a process that is responsible for storing, managing, and publishing the general configuration for each node in the cluster. This process is the central point from which nodes in a cluster obtain their configuration.
host controller
The host controller is responsible for managing server instances on a specific machine. You configure it to run one or more server instances. The domain controller can also interact with the host controllers on each machine to manage the cluster. To reduce the number of running process, a domain controller also acts as a host controller on the machine it runs on.
domain profile
A domain profile is a named set of configuration that can be used by a server to boot from. A domain controller can define multiple domain profiles that are consumed by different servers.
server group
A server group is a collection of servers. They are managed and configured as one. You can assign a domain profile to a server group and every service in that group will use that domain profile as their configuration.

In domain mode, a domain controller is started on a master node. The configuration for the cluster resides in the domain controller. Next a host controller is started on each machine in the cluster. Each host controller deployment configuration specifies how many Red Hat Single Sign-On server instances will be started on that machine. When the host controller boots up, it starts as many Red Hat Single Sign-On server instances as it was configured to do. These server instances pull their configuration from the domain controller.

Note

In some environments, such as Microsoft Azure, the domain mode is not applicable. Please consult the JBoss EAP documentation.

3.3.1. Domain configuration

Various other chapters in this guide walk you through configuring various aspects like databases, HTTP network connections, caches, and other infrastructure related things. While standalone mode uses the standalone.xml file to configure these things, domain mode uses the …​/domain/configuration/domain.xml configuration file. This is where the domain profile and server group for the Red Hat Single Sign-On server are defined.

domain.xml

domain file

Warning

Any changes you make to this file while the domain controller is running will not take effect and may even be overwritten by the server. Instead use the command line scripting or the web console of JBoss EAP. See the JBoss EAP Configuration Guide for more information.

Let’s look at some aspects of this domain.xml file. The auth-server-standalone and auth-server-clustered profile XML blocks are where you are going to make the bulk of your configuration decisions. You’ll be configuring things here like network connections, caches, and database connections.

auth-server profile

    <profiles>
        <profile name="auth-server-standalone">
            ...
        </profile>
        <profile name="auth-server-clustered">
            ...
        </profile>

The auth-server-standalone profile is a non-clustered setup. The auth-server-clustered profile is the clustered setup.

If you scroll down further, you’ll see various socket-binding-groups defined.

socket-binding-groups

    <socket-binding-groups>
        <socket-binding-group name="standard-sockets" default-interface="public">
           ...
        </socket-binding-group>
        <socket-binding-group name="ha-sockets" default-interface="public">
           ...
        </socket-binding-group>
        <!-- load-balancer-sockets should be removed in production systems and replaced with a better software or hardware based one -->
        <socket-binding-group name="load-balancer-sockets" default-interface="public">
           ...
        </socket-binding-group>
    </socket-binding-groups>

This configration defines the default port mappings for various connectors that are opened with each Red Hat Single Sign-On server instance. Any value that contains ${…​} is a value that can be overridden on the command line with the -D switch, i.e.

$ domain.sh -Djboss.http.port=80

The definition of the server group for Red Hat Single Sign-On resides in the server-groups XML block. It specifies the domain profile that is used (default) and also some default boot arguments for the Java VM when the host controller boots an instance. It also binds a socket-binding-group to the server group.

server group

    <server-groups>
        <!-- load-balancer-group should be removed in production systems and replaced with a better software or hardware based one -->
        <server-group name="load-balancer-group" profile="load-balancer">
            <jvm name="default">
                <heap size="64m" max-size="512m"/>
            </jvm>
            <socket-binding-group ref="load-balancer-sockets"/>
        </server-group>
        <server-group name="auth-server-group" profile="auth-server-clustered">
            <jvm name="default">
                <heap size="64m" max-size="512m"/>
            </jvm>
            <socket-binding-group ref="ha-sockets"/>
        </server-group>
    </server-groups>

3.3.2. Host controller configuration

Red Hat Single Sign-On comes with two host controller configuration files that reside in the …​/domain/configuration/ directory: host-master.xml and host-slave.xml. host-master.xml is configured to boot up a domain controller, a load balancer, and one Red Hat Single Sign-On server instance. host-slave.xml is configured to talk to the domain controller and boot up one Red Hat Single Sign-On server instance.

Note

The load balancer is not a required service. It exists so that you can easily test drive clustering on your development machine. While usable in production, you have the option of replacing it if you have a different hardware or software based load balancer you want to use.

Host Controller Config

host files

To disable the load balancer server instance, edit host-master.xml and comment out or remove the "load-balancer" entry.

    <servers>
        <!-- remove or comment out next line -->
        <server name="load-balancer" group="loadbalancer-group"/>
        ...
    </servers>

Another interesting thing to note about this file is the declaration of the authentication server instance. It has a port-offset setting. Any network port defined in the domain.xml socket-binding-group or the server group will have the value of port-offset added to it. For this sample domain setup, we do this so that ports opened by the load balancer server don’t conflict with the authentication server instance that is started.

    <servers>
        ...
        <server name="server-one" group="auth-server-group" auto-start="true">
             <socket-bindings port-offset="150"/>
        </server>
    </servers>

3.3.3. Server instance working directories

Each Red Hat Single Sign-On server instance defined in your host files creates a working directory under …​/domain/servers/{SERVER NAME}. Additional configuration can be put there, and any temporary, log, or data files the server instance needs or creates go there too. The structure of these per server directories ends up looking like any other JBoss EAP booted server.

Working Directories

domain server dir

3.3.4. Booting in domain clustered mode

When running the server in domain mode, there is a specific script you need to run to boot the server depending on your operating system. These scripts live in the bin/ directory of the server distribution.

Domain Boot Script

domain boot files

To boot the server:

Linux/Unix

$ .../bin/domain.sh --host-config=host-master.xml

Windows

> ...\bin\domain.bat --host-config=host-master.xml

When running the boot script you will need to pass in the host controlling configuration file you are going to use via the --host-config switch.

3.3.5. Testing with a sample clustered domain

You can test drive clustering using the sample domain.xml configuration. This sample domain is meant to run on one machine and boots up:

  • a domain controller
  • an HTTP load balancer
  • two Red Hat Single Sign-On server instances

Procedure

  1. Run the domain.sh script twice to start two separate host controllers.

    The first one is the master host controller that starts a domain controller, an HTTP load balancer, and one Red Hat Single Sign-On authentication server instance. The second one is a slave host controller that starts up only an authentication server instance.

  2. Configure the slave host controller so that it can talk securely to the domain controller. Perform these steps:

    If you omit these steps, the slave host cannot obtain the centralized configuration from the domain controller.

    1. Set up a secure connection by creating a server admin user and a secret that are shared between the master and the slave.

      Run the …​/bin/add-user.sh script.

    2. Select Management User when the script asks about the type of user to add.

      This choice generates a secret that you cut and paste into the …​/domain/configuration/host-slave.xml file.

      Add App Server Admin

      $ add-user.sh
       What type of user do you wish to add?
        a) Management User (mgmt-users.properties)
        b) Application User (application-users.properties)
       (a): a
       Enter the details of the new user to add.
       Using realm 'ManagementRealm' as discovered from the existing property files.
       Username : admin
       Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.
        - The password should not be one of the following restricted values {root, admin, administrator}
        - The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s)
        - The password should be different from the username
       Password :
       Re-enter Password :
       What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[ ]:
       About to add user 'admin' for realm 'ManagementRealm'
       Is this correct yes/no? yes
       Added user 'admin' to file '/.../standalone/configuration/mgmt-users.properties'
       Added user 'admin' to file '/.../domain/configuration/mgmt-users.properties'
       Added user 'admin' with groups to file '/.../standalone/configuration/mgmt-groups.properties'
       Added user 'admin' with groups to file '/.../domain/configuration/mgmt-groups.properties'
       Is this new user going to be used for one AS process to connect to another AS process?
       e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.
       yes/no? yes
       To represent the user add the following to the server-identities definition <secret value="bWdtdDEyMyE=" />

      Note

      The add-user.sh script does not add the user to the Red Hat Single Sign-On server but to the underlying JBoss Enterprise Application Platform. The credentials used and generated in this script are only for demonstration purposes. Please use the ones generated on your system.

  3. Cut and paste the secret value into the …​/domain/configuration/host-slave.xml file as follows:

         <management>
             <security-realms>
                 <security-realm name="ManagementRealm">
                     <server-identities>
                         <secret value="bWdtdDEyMyE="/>
                     </server-identities>
  4. Add the username of the created user in the …​/domain/configuration/host-slave.xml file:

         <remote security-realm="ManagementRealm" username="admin">
  5. Run the boot script twice to simulate a two node cluster on one development machine.

    Boot up master

    $ domain.sh --host-config=host-master.xml

    Boot up slave

    $ domain.sh --host-config=host-slave.xml

  6. Open your browser and go to http://localhost:8080/auth to try it out.

3.4. Using cross-site replication mode

Note

cross-site replication mode is Technology Preview and is not fully supported.

Use cross-site replication mode to run Red Hat Single Sign-On in a cluster across multiple data centers. Typically you use data center sites that are in different geographic regions. When using this mode, each data center will have its own cluster of Red Hat Single Sign-On servers.

This documentation will refer to the following example architecture diagram to illustrate and describe a simple cross-site replication use case.

Example Architecture Diagram

cross dc architecture

3.4.1. Prerequisites

As this is an advanced topic, we recommend you first read the following, which provide valuable background knowledge:

  • Clustering with Red Hat Single Sign-On When setting up for cross-site replication, you will use more independent Red Hat Single Sign-On clusters, so you must understand how a cluster works and the basic concepts and requirements such as load balancing, shared databases, and multicasting.
  • Red Hat Data Grid Cross-Site Replication Red Hat Single Sign-On uses Red Hat Data Grid (RHDG) for the replication of data between the data centers.

3.4.2. Technical details

This section provides an introduction to the concepts and details of how Red Hat Single Sign-On cross-site replication is accomplished.

Data

Red Hat Single Sign-On is stateful application. It uses the following as data sources:

  • A database is used to persist permanent data, such as user information.
  • An Infinispan cache is used to cache persistent data from the database and also to save some short-lived and frequently-changing metadata, such as for user sessions. Infinispan is usually much faster than a database, however the data saved using Infinispan are not permanent and is not expected to persist across cluster restarts.

In our example architecture, there are two data centers called site1 and site2. For cross-site replication, we must make sure that both sources of data work reliably and that Red Hat Single Sign-On servers from site1 are eventually able to read the data saved by Red Hat Single Sign-On servers on site2 .

Based on the environment, you have the option to decide if you prefer:

  • Reliability - which is typically used in Active/Active mode. Data written on site1 must be visible immediately on site2.
  • Performance - which is typically used in Active/Passive mode. Data written on site1 does not need to be visible immediately on site2. In some cases, the data may not be visible on site2 at all.

For more details, see Section 3.4.4, “Modes”.

3.4.3. Request processing

An end user’s browser sends an HTTP request to the front end load balancer. This load balancer is usually HTTPD or WildFly with mod_cluster, NGINX, HA Proxy, or perhaps some other kind of software or hardware load balancer.

The load balancer then forwards the HTTP requests it receives to the underlying Red Hat Single Sign-On instances, which can be spread among multiple data centers. Load balancers typically offer support for sticky sessions, which means that the load balancer is able to always forward all HTTP requests from the same user to the same Red Hat Single Sign-On instance in same data center.

HTTP requests that are sent from client applications to the load balancer are called backchannel requests. These are not seen by an end user’s browser and therefore can not be part of a sticky session between the user and the load balancer. For backchannel requests, the loadbalancer can forward the HTTP request to any Red Hat Single Sign-On instance in any data center. This is challenging as some OpenID Connect and some SAML flows require multiple HTTP requests from both the user and the application. Because we can not reliably depend on sticky sessions to force all the related requests to be sent to the same Red Hat Single Sign-On instance in the same data center, we must instead replicate some data across data centers, so the data are seen by subsequent HTTP requests during a particular flow.

3.4.4. Modes

According your requirements, there are two basic operating modes for cross-site replication:

  • Active/Passive - Here the users and client applications send the requests just to the Red Hat Single Sign-On nodes in just a single data center. The second data center is used just as a backup for saving the data. In case of the failure in the main data center, the data can be usually restored from the second data center.
  • Active/Active - Here the users and client applications send the requests to the Red Hat Single Sign-On nodes in both data centers. It means that data need to be visible immediately on both sites and available to be consumed immediately from Red Hat Single Sign-On servers on both sites. This is especially true if Red Hat Single Sign-On server writes some data on site1, and it is required that the data are available immediately for reading by Red Hat Single Sign-On servers on site2 immediately after the write on site1 is finished.

The active/passive mode is better for performance. For more information about how to configure caches for either mode, see: Section 3.4.14, “SYNC or ASYNC backups”.

3.4.5. Database

Red Hat Single Sign-On uses a relational database management system (RDBMS) to persist some metadata about realms, clients, users, and so on. See this chapter of the server installation guide for more details. In a cross-site replication setup, we assume that either both data centers talk to the same database or that every data center has its own database node and both database nodes are synchronously replicated across the data centers. In both cases, it is required that when a Red Hat Single Sign-On server on site1 persists some data and commits the transaction, those data are immediately visible by subsequent DB transactions on site2.

Details of DB setup are out-of-scope for Red Hat Single Sign-On, however many RDBMS vendors like MariaDB and Oracle offer replicated databases and synchronous replication. We test Red Hat Single Sign-On with these vendors:

  • Oracle Database 19c RAC
  • Galera 3.12 cluster for MariaDB server version 10.1.19-MariaDB

3.4.6. Infinispan caches

This section begins with a high level description of the Infinispan caches. More details of the cache setup follow.

Authentication sessions

In Red Hat Single Sign-On we have the concept of authentication sessions. There is a separate Infinispan cache called authenticationSessions used to save data during authentication of particular user. Requests from this cache usually involve only a browser and the Red Hat Single Sign-On server, not the application. Here we can rely on sticky sessions and the authenticationSessions cache content does not need to be replicated across data centers, even if you are in Active/Active mode.

Caching and invalidation of persistent data

Red Hat Single Sign-On uses Infinispan to cache persistent data to avoid many unnecessary requests to the database. Caching improves performance, however it adds an additional challenge. When some Red Hat Single Sign-On server updates any data, all other Red Hat Single Sign-On servers in all data centers need to be aware of it, so they invalidate particular data from their caches. Red Hat Single Sign-On uses local Infinispan caches called realms, users, and authorization to cache persistent data.

We use a separate cache, work, which is replicated across all data centers. The work cache itself does not cache any real data. It is used only for sending invalidation messages between cluster nodes and data centers. In other words, when data is updated, such as the user john, the Red Hat Single Sign-On node sends the invalidation message to all other cluster nodes in the same data center and also to all other data centers. After receiving the invalidation notice, every node then invalidates the appropriate data from their local cache.

User sessions

There are Infinispan caches called sessions, clientSessions, offlineSessions, and offlineClientSessions, all of which usually need to be replicated across data centers. These caches are used to save data about user sessions, which are valid for the length of a user’s browser session. The caches must handle the HTTP requests from the end user and from the application. As described above, sticky sessions can not be reliably used in this instance, but we still want to ensure that subsequent HTTP requests can see the latest data. For this reason, the data are usually replicated across data centers.

Brute force protection

Finally the loginFailures cache is used to track data about failed logins, such as how many times the user john entered a bad password. The details are described here. It is up to the admin whether this cache should be replicated across data centers. To have an accurate count of login failures, the replication is needed. On the other hand, not replicating this data can save some performance. So if performance is more important than accurate counts of login failures, the replication can be avoided.

For more detail about how caches can be configured see Section 3.4.13, “Tuning the RHDG cache configuration”.

3.4.7. Communication details

Red Hat Single Sign-On uses multiple, separate clusters of Infinispan caches. Every Red Hat Single Sign-On node is in the cluster with the other Red Hat Single Sign-On nodes in same data center, but not with the Red Hat Single Sign-On nodes in different data centers. A Red Hat Single Sign-On node does not communicate directly with the Red Hat Single Sign-On nodes from different data centers. Red Hat Single Sign-On nodes use external JDG (actually RHDG servers) for communication across data centers. This is done using the Infinispan HotRod protocol.

The Infinispan caches on the Red Hat Single Sign-On side must be configured with the remoteStore to ensure that data are saved to the remote cache. There is separate Infinispan cluster between JDG servers, so the data saved on JDG1 on site1 are replicated to JDG2 on site2 .

The receiving RHDG server notifies the Red Hat Single Sign-On servers in its cluster through Client Listeners, which are a feature of the Hot Rod protocol. Red Hat Single Sign-On nodes on site2 then update their Infinispan caches and the particular user session is also visible on Red Hat Single Sign-On nodes on site2.

See the Example Architecture Diagram for more details.

3.4.8. Setting up cross-site replication

Use the following procedures for RHDG 8.x to perform a basic setup of cross-site replication.

Note

The exact instructions for your version of RHDG may be slightly different from these instructions, which apply to RHDG 8.1. Check the RHDG documentation for any steps that do not work as described.

This example for RHDG 8.x involves two data centers, site1 and site2. Each data center consists of 1 RHDG server and 2 Red Hat Single Sign-On servers. We will end up with 2 RHDG servers and 4 Red Hat Single Sign-On servers in total.

  • Site1 consists of RHDG server, server1, and 2 Red Hat Single Sign-On servers, node11 and node12 .
  • Site2 consists of RHDG server, server2, and 2 Red Hat Single Sign-On servers, node21 and node22 .
  • RHDG servers server1 and server2 are connected to each other through the RELAY2 protocol and backup based RHDG caches in a similar way as described in the RHDG documentation.
  • Red Hat Single Sign-On servers node11 and node12 form a cluster with each other, but they do not communicate directly with any server in site2. They communicate with the Infinispan server server1 using the Hot Rod protocol (Remote cache). See Section 3.4.7, “Communication details” for more information.
  • The same details apply for node21 and node22. They cluster with each other and communicate only with server2 server using the Hot Rod protocol.

Our example setup assumes that the four Red Hat Single Sign-On servers talk to the same database. In production, we recommend that you use separate synchronously replicated databases across data centers as described in Section 3.4.5, “Database”.

3.4.8.1. Setting Up RHDG Servers

For cross-site replication, you start by creating remote RHDG clusters that can back up Red Hat Single Sign-On data.

Prerequisites

  • Download and install RHDG Server 8.x.
Note

RHDG Server 8.x requires Java 11.

Procedure

  1. Create a user to authenticate client connections from RHDG, for example:

    $ bin/cli.sh user create myuser -p "qwer1234!"
    Note

    You specify these credentials in the Hot Rod client configuration when you create remote caches on Red Hat Single Sign-On.

  2. Create an SSL keystore and truststore to secure connections between RHDG and Red Hat Single Sign-On, for example:

    1. Create a keystore to provide an SSL identity to your RHDG cluster

      keytool -genkey -alias server -keyalg RSA -keystore server.jks -keysize 2048
    2. Export an SSL certificate from the keystore.

      keytool -exportcert -keystore server.jks -alias server -file server.crt
    3. Import the SSL certificate into a truststore that Red Hat Single Sign-On can use to verify the SSL identity for RHDG.

      keytool -importcert -keystore truststore.jks -alias server -file server.crt
    4. Remove server.crt.

      rm server.crt

3.4.8.2. Configuring RHDG Clusters

Configure RHDG clusters to replicate Red Hat Single Sign-On data across data centers.

Prerequisites

  • Install and set up RHDG Server.

Procedure

  1. Open infinispan.xml for editing.

    By default, RHDG Server uses server/conf/infinispan.xml for static configuration such as cluster transport and security mechanisms.

  2. Create a stack that uses TCPPING as the cluster discovery protocol.

    <stack name="global-cluster" extends="tcp">
        <!-- Remove MPING protocol from the stack and add TCPPING -->
        <TCPPING initial_hosts="server1[7800],server2[7800]" 1
                 stack.combine="REPLACE" stack.position="MPING"/>
    </stack>
    1
    Lists the host names for server1 and server2.
  3. Configure the RHDG cluster transport to perform cross-site replication.

    1. Add the RELAY2 protocol to a JGroups stack.

      <jgroups>
         <stack name="xsite" extends="udp"> 1
            <relay.RELAY2 site="site1" 2
                          max_site_masters="1000"/> 3
            <remote-sites default-stack="global-cluster"> 4
               <remote-site name="site1"/>
               <remote-site name="site2"/>
            </remote-sites>
         </stack>
      </jgroups>
      1
      Creates a stack named xsite that extends the default UDP cluster transport.
      2
      Adds the RELAY2 protocol and names the cluster you are configuring as site1. The site name must be unique to each RHDG cluster.
      3
      Sets 1000 as the number of relay nodes for the cluster. You should set a value that is equal to or greater than the maximum number of nodes in your RHDG cluster.
      4
      Names all RHDG clusters that backup caches with RHDG data and uses the default TCP stack for inter-cluster transport.
    2. Configure the RHDG cluster transport to use the stack.

      <cache-container name="default" statistics="true">
            <transport cluster="${infinispan.cluster.name:cluster}"
                       stack="xsite"/> 1
      </cache-container>
      1
      Uses the xsite stack for the cluster.
  4. Configure the keystore as an SSL identity in the server security realm.

    <server-identities>
      <ssl>
        <keystore path="server.jks" 1
                  relative-to="infinispan.server.config.path"
                  keystore-password="password" 2
                  alias="server" /> 3
      </ssl>
    </server-identities>
    1
    Specifies the path of the keystore that contains the SSL identity.
    2
    Specifies the password to access the keystore.
    3
    Names the alias of the certificate in the keystore.
  5. Configure the authentication mechanism for the Hot Rod endpoint.

    <endpoints socket-binding="default">
       <hotrod-connector name="hotrod">
          <authentication>
             <sasl mechanisms="SCRAM-SHA-512" 1
                   server-name="infinispan" /> 2
          </authentication>
       </hotrod-connector>
       <rest-connector name="rest"/>
    </endpoints>
    1
    Configures the SASL authentication mechanism for the Hot Rod endpoint. SCRAM-SHA-512 is the default SASL mechanism for Hot Rod. However you can use whatever is appropriate for your environment, such as GSSAPI.
    2
    Defines the name that RHDG servers present to clients. You specify this name in the Hot Rod client configuration when you set up Red Hat Single Sign-On.
  6. Create a cache template.

    Note

    Add the cache template to infinispan.xml on each node in the RHDG cluster.

    <cache-container ... >
      <replicated-cache-configuration name="sessions-cfg" 1
                                      mode="SYNC"> 2
        <locking acquire-timeout="0" /> 3
        <backups>
          <backup site="site2" strategy="SYNC" /> 4
        </backups>
      </replicated-cache-configuration>
    </cache-container>
    1
    Creates a cache template named sessions-cfg.
    2
    Defines a cache that synchronously replicates data across the cluster.
    3
    Disables timeout for lock acquisition.
    4
    Names the backup site for the RHDG cluster you are configuring.
  7. Start RHDG server1.

    ./server.sh -c infinispan.xml -b PUBLIC_IP_ADDRESS -k PUBLIC_IP_ADDRESS -Djgroups.mcast_addr=228.6.7.10
  8. Start RHDG server2.

    ./server.sh -c infinispan.xml -b PUBLIC_IP_ADDRESS -k PUBLIC_IP_ADDRESS -Djgroups.mcast_addr=228.6.7.11
  9. Check RHDG server logs to verify the clusters form cross-site views.

    INFO  [org.infinispan.XSITE] (jgroups-5,${server.hostname}) ISPN000439: Received new x-site view: [site1]
    INFO  [org.infinispan.XSITE] (jgroups-7,${server.hostname}) ISPN000439: Received new x-site view: [site1, site2]

3.4.8.3. Creating Infinispan Caches

Create the Infinispan caches that Red Hat Single Sign-On requires.

We recommend that you create caches on RHDG clusters at runtime rather than adding caches to infinispan.xml. This strategy ensures that your caches are automatically synchronized across the cluster and permanently stored.

The following procedure uses the RHDG Command Line Interface (CLI) to create all the required caches in a single batch command.

Prerequisites

  • Configure your RHDG clusters.

Procedure

  1. Create a batch file that contains caches, for example:

    cat > /tmp/caches.batch<<EOF
    echo "creating caches..."
    create cache work --template=sessions-cfg
    create cache sessions --template=sessions-cfg
    create cache clientSessions --template=sessions-cfg
    create cache offlineSessions --template=sessions-cfg
    create cache offlineClientSessions --template=sessions-cfg
    create cache actionTokens --template=sessions-cfg
    create cache loginFailures --template=sessions-cfg
    echo "verifying caches"
    ls caches
    EOF
  2. Create the caches with the CLI.

    $ bin/cli.sh -c https://server1:11222 --trustall -f /tmp/caches.batch
    Note

    Instead of the --trustall argument you can specify the truststore with the -t argument and the truststore password with the -s argument.

  3. Create the caches on the other site.

3.4.8.4. Configuring Remote Cache Stores on Red Hat Single Sign-On

After you set up remote RHDG clusters, you configure the Infinispan subsystem on Red Hat Single Sign-On to externalize data to those clusters through remote stores.

Prerequisites

  • Set up remote RHDG clusters for cross-site configuration.
  • Create a truststore that contains the SSL certificate with the RHDG Server identity.

Procedure

  1. Add the truststore to the Red Hat Single Sign-On deployment.
  2. Create a socket binding that points to your RHDG cluster.

    <outbound-socket-binding name="remote-cache"> 1
      <remote-destination host="${remote.cache.host:server_hostname}" 2
                          port="${remote.cache.port:11222}"/> 3
    </outbound-socket-binding>
    1
    Names the socket binding as remote-cache.
    2
    Specifies one or more hostnames for the RHDG cluster.
    3
    Defines the port of 11222 where the Hot Rod endpoint listens.
  3. Add the org.keycloak.keycloak-model-infinispan module to the keycloak cache container in the Infinispan subsystem.

    <subsystem xmlns="urn:jboss:domain:infinispan:12.0">
        <cache-container name="keycloak"
                         modules="org.keycloak.keycloak-model-infinispan"/>
  4. Update the work cache in the Infinispan subsystem so it has the following configuration:

    <replicated-cache name="work"> 1
        <remote-store cache="work" 2
                      remote-servers="remote-cache" 3
                      passivation="false"
                      fetch-state="false"
                      purge="false"
                      preload="false"
                      shared="true">
            <property name="rawValues">true</property>
            <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
            <property name="infinispan.client.hotrod.auth_username">myuser</property>
            <property name="infinispan.client.hotrod.auth_password">qwer1234!</property>
            <property name="infinispan.client.hotrod.auth_realm">default</property>
            <property name="infinispan.client.hotrod.auth_server_name">infinispan</property>
            <property name="infinispan.client.hotrod.sasl_mechanism">SCRAM-SHA-512</property>
            <property name="infinispan.client.hotrod.trust_store_file_name">/path/to/truststore.jks</property>
            <property name="infinispan.client.hotrod.trust_store_type">JKS</property>
            <property name="infinispan.client.hotrod.trust_store_password">password</property>
        </remote-store>
    </replicated-cache>
    1
    Names the cache in the RHDG configuration.
    2
    Names the corresponding cache on the remote RHDG cluster.
    3
    Specifies the remote-cache socket binding.

    The preceding cache configuration includes recommended settings for RHDG caches. Hot Rod client configuration properties specify the RHDG user credentials and SSL keystore and truststore details.

    Refer to the RHDG documentation for descriptions of each property.

  5. Add distributed caches to the Infinispan subsystem for each of the following caches:

    • sessions
    • clientSessions
    • offlineSessions
    • offlineClientSessions
    • actionTokens
    • loginFailures

      For example, add a cache named sessions with the following configuration:

      <distributed-cache name="sessions" 1
                         owners="1"> 2
          <remote-store cache="sessions" 3
                        remote-servers="remote-cache" 4
                        passivation="false"
                        fetch-state="false"
                        purge="false"
                        preload="false"
                        shared="true">
              <property name="rawValues">true</property>
              <property name="marshaller">org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory</property>
              <property name="infinispan.client.hotrod.auth_username">myuser</property>
              <property name="infinispan.client.hotrod.auth_password">qwer1234!</property>
              <property name="infinispan.client.hotrod.auth_realm">default</property>
              <property name="infinispan.client.hotrod.auth_server_name">infinispan</property>
              <property name="infinispan.client.hotrod.sasl_mechanism">SCRAM-SHA-512</property>
              <property name="infinispan.client.hotrod.trust_store_file_name">/path/to/truststore.jks</property>
              <property name="infinispan.client.hotrod.trust_store_type">JKS</property>
              <property name="infinispan.client.hotrod.trust_store_password">password</property>
          </remote-store>
      </distributed-cache>
      1
      Names the cache in the RHDG configuration.
      2
      Configures one replica of each cache entry across the RHDG cluster.
      3
      Names the corresponding cache on the remote RHDG cluster.
      4
      Specifies the remote-cache socket binding.
  6. Copy the NODE11 to 3 other directories referred later as NODE12, NODE21 and NODE22.
  7. Start NODE11 :

    cd NODE11/bin
    ./standalone.sh -c standalone-ha.xml -Djboss.node.name=node11 -Djboss.site.name=site1 \
      -Djboss.default.multicast.address=234.56.78.1 -Dremote.cache.host=server1 \
      -Djava.net.preferIPv4Stack=true -b PUBLIC_IP_ADDRESS

    If you notice the following warning messages in logs, you can safely ignore them:

    WARN  [org.infinispan.CONFIG] (MSC service thread 1-5) ISPN000292: Unrecognized attribute 'infinispan.client.hotrod.auth_password'. Please check your configuration. Ignoring!
    WARN  [org.infinispan.CONFIG] (MSC service thread 1-5) ISPN000292: Unrecognized attribute 'infinispan.client.hotrod.auth_username'. Please check your configuration. Ignoring!
  8. Start NODE12 :

    cd NODE12/bin
    ./standalone.sh -c standalone-ha.xml -Djboss.node.name=node12 -Djboss.site.name=site1 \
      -Djboss.default.multicast.address=234.56.78.1 -Dremote.cache.host=server1 \
      -Djava.net.preferIPv4Stack=true -b PUBLIC_IP_ADDRESS

    The cluster nodes should be connected. Something like this should be in the log of both NODE11 and NODE12:

    Received new cluster view for channel keycloak: [node11|1] (2) [node11, node12]
    Note

    The channel name in the log might be different.

  9. Start NODE21 :

    cd NODE21/bin
    ./standalone.sh -c standalone-ha.xml -Djboss.node.name=node21 -Djboss.site.name=site2 \
      -Djboss.default.multicast.address=234.56.78.2 -Dremote.cache.host=server2 \
      -Djava.net.preferIPv4Stack=true -b PUBLIC_IP_ADDRESS

    It shouldn’t be connected to the cluster with NODE11 and NODE12, but to a separate one:

    Received new cluster view for channel keycloak: [node21|0] (1) [node21]
  10. Start NODE22 :

    cd NODE22/bin
    ./standalone.sh -c standalone-ha.xml -Djboss.node.name=node22 -Djboss.site.name=site2 \
      -Djboss.default.multicast.address=234.56.78.2 -Dremote.cache.host=server2 \
      -Djava.net.preferIPv4Stack=true -b PUBLIC_IP_ADDRESS

    It should be in cluster with NODE21 :

    Received new cluster view for channel keycloak: [node21|1] (2) [node21, node22]
    Note

    The channel name in the log might be different.

  11. Test:

    1. Go to http://node11:8080/auth/ and create the initial admin user.
    2. Go to http://node11:8080/auth/admin and login as admin to admin console.
    3. Open a second browser and go to any of nodes http://node12:8080/auth/admin or http://node21:8080/auth/admin or http://node22:8080/auth/admin. After login, you should be able to see the same sessions in tab Sessions of particular user, client or realm on all 4 servers.
    4. After making a change in the Red Hat Single Sign-On Admin Console, such as modifying a user or a realm, that change should be immediately visible on any of the four nodes. Caches should be properly invalidated everywhere.
    5. Check server.logs if needed. After login or logout, the message like this should be on all the nodes NODEXY/standalone/log/server.log :

      2017-08-25 17:35:17,737 DEBUG [org.keycloak.models.sessions.infinispan.remotestore.RemoteCacheSessionListener] (Client-Listener-sessions-30012a77422542f5) Received event from remote store.
      Event 'CLIENT_CACHE_ENTRY_REMOVED', key '193489e7-e2bc-4069-afe8-f1dfa73084ea', skip 'false'

3.4.9. Administration of cross-site deployment

This section contains some tips and options related to cross-site replication.

  • When you run the Red Hat Single Sign-On server inside a data center, it is required that the database referenced in KeycloakDS datasource is already running and available in that data center. It is also necessary that the RHDG server referenced by the outbound-socket-binding, which is referenced from the Infinispan cache remote-store element, is already running. Otherwise the Red Hat Single Sign-On server will fail to start.
  • Every data center can have more database nodes if you want to support database failover and better reliability. Refer to the documentation of your database and JDBC driver for the details how to set this up on the database side and how the KeycloakDS datasource on Keycloak side needs to be configured.
  • Every datacenter can have more RHDG servers running in the cluster. This is useful if you want some failover and better fault tolerance. The Hot Rod protocol used for communication between RHDG servers and Red Hat Single Sign-On servers has a feature that RHDG servers will automatically send new topology to the Red Hat Single Sign-On servers about the change in the RHDG cluster, so the remote store on Red Hat Single Sign-On side will know to which RHDG servers it can connect. Read the RHDG and WildFly documentation for more details.
  • It is highly recommended that a master RHDG server is running in every site before the Red Hat Single Sign-On servers in any site are started. As in our example, we started both server1 and server2 first, before all Red Hat Single Sign-On servers. If you still need to run the Red Hat Single Sign-On server and the backup site is offline, it is recommended to manually switch the backup site offline on the RHDG servers on your site, as described in Section 3.4.10, “Bringing sites offline and online”. If you do not manually switch the unavailable site offline, the first startup may fail or they may be some exceptions during startup until the backup site is taken offline automatically due the configured count of failed operations.

3.4.10. Bringing sites offline and online

For example, assume this scenario:

  1. Site site2 is entirely offline from the site1 perspective. This means that all RHDG servers on site2 are off or the network between site1 and site2 is broken.
  2. You run Red Hat Single Sign-On servers and RHDG server server1 in site site1
  3. Someone logs in on a Red Hat Single Sign-On server on site1.
  4. The Red Hat Single Sign-On server from site1 will try to write the session to the remote cache on server1 server, which is supposed to backup data to the server2 server in the site2. See Section 3.4.7, “Communication details” for more information.
  5. Server server2 is offline or unreachable from server1. So the backup from server1 to server2 will fail.
  6. The exception is thrown in server1 log and the failure will be propagated from server1 server to Red Hat Single Sign-On servers as well because the default FAIL backup failure policy is configured. See Backup failure policy for details around the backup policies.
  7. The error will happen on Red Hat Single Sign-On side too and user may not be able to finish his login.

According to your environment, it may be more or less probable that the network between sites is unavailable or temporarily broken (split-brain). In case this happens, it is good that RHDG servers on site1 are aware of the fact that RHDG servers on site2 are unavailable, so they will stop trying to reach the servers in the server2 site and the backup failures won’t happen. This is called Take site offline .

Take site offline

There are 2 ways to take the site offline.

Manually by admin - Admin can use the jconsole or other tool and run some JMX operations to manually take the particular site offline. This is useful especially if the outage is planned. With jconsole or CLI, you can connect to the server1 server and take the site2 offline. More details about this are available in the RHDG documentation.

Warning

These steps usually need to be done for all the Red Hat Single Sign-On caches mentioned in Section 3.4.14, “SYNC or ASYNC backups”.

Automatically - After some number of failed backups, the site2 will usually be taken offline automatically. This is done due the configuration of take-offline element inside the cache configuration.

<take-offline min-wait="60000" after-failures="3" />

This example shows that the site will be taken offline automatically for the particular single cache if there are at least 3 subsequent failed backups and there is no any successful backup within 60 seconds.

Automatically taking a site offline is useful especially if the broken network between sites is unplanned. The disadvantage is that there will be some failed backups until the network outage is detected, which could also mean failures on the application side. For example, there will be failed logins for some users or big login timeouts. Especially if failure-policy with value FAIL is used.

Warning

The tracking of whether a site is offline is tracked separately for every cache.

Take site online

Once your network is back and site1 and site2 can talk to each other, you may need to put the site online. This needs to be done manually through JMX or CLI in similar way as taking a site offline. Again, you may need to check all the caches and bring them online.

Once the sites are put online, it’s usually good to:

3.4.11. State transfer

State transfer is a required, manual step. RHDG server does not do this automatically, for example during split-brain, it is only the admin who may decide which site has preference and hence if state transfer needs to be done bidirectionally between both sites or just unidirectionally, as in only from site1 to site2, but not from site2 to site1.

A bidirectional state transfer will ensure that entities which were created after split-brain on site1 will be transferred to site2. This is not an issue as they do not yet exist on site2. Similarly, entities created after split-brain on site2 will be transferred to site1. Possibly problematic parts are those entities which exist before split-brain on both sites and which were updated during split-brain on both sites. When this happens, one of the sites will win and will overwrite the updates done during split-brain by the second site.

Unfortunately, there is no any universal solution to this. Split-brains and network outages are just state, which is usually impossible to be handled 100% correctly with 100% consistent data between sites. In the case of Red Hat Single Sign-On, it typically is not a critical issue. In the worst case, users will need to re-login again to their clients, or have the improper count of loginFailures tracked for brute force protection. See the RHDG/JGroups documentation for more tips how to deal with split-brain.

The state transfer can be also done on the RHDG server side through JMX. The operation name is pushState. There are few other operations to monitor status, cancel push state, and so on. More info about state transfer is available in the RHDG docs.

3.4.12. Clear caches

After split-brain it is safe to manually clear caches in the Red Hat Single Sign-On admin console. This is because there might be some data changed in the database on site1 and because of the event, that the cache should be invalidated wasn’t transferred during split-brain to site2. Hence Red Hat Single Sign-On nodes on site2 may still have some stale data in their caches.

To clear the caches, see Clearing Server Caches.

When the network is back, it is sufficient to clear the cache just on one Red Hat Single Sign-On node on any random site. The cache invalidation event will be sent to all the other Red Hat Single Sign-On nodes in all sites. However, it needs to be done for all the caches (realms, users, keys). See Clearing Server Caches for more information.

3.4.13. Tuning the RHDG cache configuration

This section contains tips and options for configuring your JDG cache.

Backup failure policy

By default, the configuration of backup failure-policy in the Infinispan cache configuration in the RHDG clustered.xml file is configured as FAIL. You may change it to WARN or IGNORE, as you prefer.

The difference between FAIL and WARN is that when FAIL is used and the RHDG server tries to back data up to the other site and the backup fails then the failure will be propagated back to the caller (the Red Hat Single Sign-On server). The backup might fail because the second site is temporarily unreachable or there is a concurrent transaction which is trying to update same entity. In this case, the Red Hat Single Sign-On server will then retry the operation a few times. However, if the retry fails, then the user might see the error after a longer timeout.

When using WARN, the failed backups are not propagated from the RHDG server to the Red Hat Single Sign-On server. The user won’t see the error and the failed backup will be just ignored. There will be a shorter timeout, typically 10 seconds as that’s the default timeout for backup. It can be changed by the attribute timeout of backup element. There won’t be retries. There will just be a WARNING message in the RHDG server log.

The potential issue is, that in some cases, there may be just some a short network outage between sites, where the retry (usage of the FAIL policy) may help, so with WARN (without retry), there will be some data inconsistencies across sites. This can also happen if there is an attempt to update the same entity concurrently on both sites.

How bad are these inconsistencies? Usually only means that a user will need to re-authenticate.

When using the WARN policy, it may happen that the single-use cache, which is provided by the actionTokens cache and which handles that particular key is really single use, but may "successfully" write the same key twice. But, for example, the OAuth2 specification mentions that code must be single-use. With the WARN policy, this may not be strictly guaranteed and the same code could be written twice if there is an attempt to write it concurrently in both sites.

If there is a longer network outage or split-brain, then with both FAIL and WARN, the other site will be taken offline after some time and failures as described in Section 3.4.10, “Bringing sites offline and online”. With the default 1 minute timeout, it is usually 1-3 minutes until all the involved caches are taken offline. After that, all the operations will work fine from an end user perspective. You only need to manually restore the site when it is back online as mentioned in Section 3.4.10, “Bringing sites offline and online”.

In summary, if you expect frequent, longer outages between sites and it is acceptable for you to have some data inconsistencies and a not 100% accurate single-use cache, but you never want end-users to see the errors and long timeouts, then switch to WARN.

The difference between WARN and IGNORE is, that with IGNORE warnings are not written in the RHDG log. See more details in the Infinispan documentation.

Lock acquisition timeout

The default configuration is using transaction in NON_DURABLE_XA mode with acquire timeout 0. This means that transaction will fail-fast if there is another transaction in progress for the same key.

The reason to switch this to 0 instead of default 10 seconds was to avoid possible deadlock issues. With Red Hat Single Sign-On, it can happen that the same entity (typically session entity or loginFailure) is updated concurrently from both sites. This can cause deadlock under some circumstances, which will cause the transaction to be blocked for 10 seconds. See this JIRA report for details.

With timeout 0, the transaction will immediately fail and then will be retried from Red Hat Single Sign-On if backup failure-policy with the value FAIL is configured. As long as the second concurrent transaction is finished, the retry will usually be successful and the entity will have applied updates from both concurrent transactions.

We see very good consistency and results for concurrent transaction with this configuration, and it is recommended to keep it.

The only (non-functional) problem is the exception in the RHDG server log, which happens every time when the lock is not immediately available.

3.4.14. SYNC or ASYNC backups

An important part of the backup element is the strategy attribute. You must decide whether it needs to be SYNC or ASYNC. We have 7 caches which might be cross-site replication aware, and these can be configured in 3 different modes regarding cross-site:

  1. SYNC backup
  2. ASYNC backup
  3. No backup at all

If the SYNC backup is used, then the backup is synchronous and operation is considered finished on the caller (Red Hat Single Sign-On server) side once the backup is processed on the second site. This has worse performance than ASYNC, but on the other hand, you are sure that subsequent reads of the particular entity, such as user session, on site2 will see the updates from site1. Also, it is needed if you want data consistency. As with ASYNC the caller is not notified at all if backup to the other site failed.

For some caches, it is even possible to not backup at all and completely skip writing data to the RHDG server. To set this up, do not use the remote-store element for the particular cache on the Red Hat Single Sign-On side (file KEYCLOAK_HOME/standalone/configuration/standalone-ha.xml) and then the particular replicated-cache element is also not needed on the RHDG server side.

By default, all 7 caches are configured with SYNC backup, which is the safest option. Here are a few things to consider:

  • If you are using active/passive mode (all Red Hat Single Sign-On servers are in single site site1 and the RHDG server in site2 is used purely as backup. See Section 3.4.4, “Modes” for more details), then it is usually fine to use ASYNC strategy for all the caches to save the performance.
  • The work cache is used mainly to send some messages, such as cache invalidation events, to the other site. It is also used to ensure that some special events, such as userStorage synchronizations, happen only on single site. It is recommended to keep this set to SYNC.
  • The actionTokens cache is used as single-use cache to track that some tokens/tickets were used just once. For example action tokens or OAuth2 codes. It is possible to set this to ASYNC to slightly improved performance, but then it is not guaranteed that particular ticket is really single-use. For example, if there is concurrent request for same ticket in both sites, then it is possible that both requests will be successful with the ASYNC strategy. So what you set here will depend on whether you prefer better security (SYNC strategy) or better performance (ASYNC strategy).
  • The loginFailures cache may be used in any of the 3 modes. If there is no backup at all, it means that count of login failures for a user will be counted separately for every site (See Section 3.4.6, “Infinispan caches” for details). This has some security implications, however it has some performance advantages. Also it mitigates the possible risk of denial of service (DoS) attacks. For example, if an attacker simulates 1000 concurrent requests using the username and password of the user on both sites, it will mean lots of messages being passed between the sites, which may result in network congestion. The ASYNC strategy might be even worse as the attacker requests won’t be blocked by waiting for the backup to the other site, resulting in potentially even more congested network traffic. The count of login failures also will not be accurate with the ASYNC strategy.

For the environments with slower network between data centers and probability of DoS, it is recommended to not backup the loginFailures cache at all.

  • It is recommended to keep the sessions and clientSessions caches in SYNC. Switching them to ASYNC is possible only if you are sure that user requests and backchannel requests (requests from client applications to Red Hat Single Sign-On as described in Section 3.4.3, “Request processing”) will be always processed on same site. This is true, for example, if:

    • You use active/passive mode as described Section 3.4.4, “Modes”.
    • All your client applications are using the Red Hat Single Sign-On JavaScript Adapter. The JavaScript adapter sends the backchannel requests within the browser and hence they participate on the browser sticky session and will end on same cluster node (hence on same site) as the other browser requests of this user.
    • Your load balancer is able to serve the requests based on client IP address (location) and the client applications are deployed on both sites.

      For example you have 2 sites LON and NYC. As long as your applications are deployed in both LON and NYC sites too, you can ensure that all the user requests from London users will be redirected to the applications in LON site and also to the Red Hat Single Sign-On servers in LON site. Backchannel requests from the LON site client deployments will end on Red Hat Single Sign-On servers in LON site too. On the other hand, for the American users, all the Red Hat Single Sign-On requests, application requests and backchannel requests will be processed on NYC site.

  • For offlineSessions and offlineClientSessions it is similar, with the difference that you even don’t need to backup them at all if you never plan to use offline tokens for any of your client applications.

Generally, if you are in doubt and performance is not a blocker for you, it’s safer to keep the caches in SYNC strategy.

Warning

Regarding the switch to SYNC/ASYNC backup, make sure that you edit the strategy attribute of the backup element. For example like this:

<backup site="site2" failure-policy="FAIL" strategy="ASYNC" enabled="true">

Note the mode attribute of cache-configuration element.

3.4.15. Troubleshooting

The following tips are intended to assist you should you need to troubleshoot:

  • We recommend that you go through the procedure for Setting up cross-site replication and have this one working first, so that you have some understanding of how things work. It is also wise to read this entire document to have some understanding of things.
  • For the Red Hat Single Sign-On servers, you should see a message such as this during the server startup:

    18:09:30,156 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (ServerService Thread Pool -- 54)
    Node name: node11, Site name: site1

    Check that the site name and the node name looks as expected during the startup of Red Hat Single Sign-On server.

  • Check that Red Hat Single Sign-On servers are in cluster as expected, including that only the Red Hat Single Sign-On servers from the same data center are in cluster with each other. This can be also checked in JConsole through the GMS view.
  • If there are exceptions during startup of Red Hat Single Sign-On server like this:

    17:33:58,605 ERROR [org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation] (ServerService Thread Pool -- 59) ISPN004007: Exception encountered. Retry 10 out of 10: org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport
    ...
    Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: 127.0.0.1:12232
    	at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:82)

    it usually means that Red Hat Single Sign-On server is not able to reach the RHDG server in his own datacenter. Make sure that firewall is set as expected and RHDG server is possible to connect.

  • If there are exceptions during startup of Red Hat Single Sign-On server like this:

    16:44:18,321 WARN  [org.infinispan.client.hotrod.impl.protocol.Codec21] (ServerService Thread Pool -- 57) ISPN004005: Error received from the server: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
     ...

    then check the log of corresponding RHDG server of your site and check if has failed to backup to the other site. If the backup site is unavailable, then it is recommended to switch it offline, so that RHDG server won’t try to backup to the offline site causing the operations to pass successfully on Red Hat Single Sign-On server side as well. See Section 3.4.9, “Administration of cross-site deployment” for more information.

  • Check the Infinispan statistics, which are available through JMX. For example, try to login and then see if the new session was successfully written to both RHDG servers and is available in the sessions cache there. This can be done indirectly by checking the count of elements in the sessions cache for the MBean jboss.datagrid-infinispan:type=Cache,name="sessions(repl_sync)",manager="clustered",component=Statistics and attribute numberOfEntries. After login, there should be one more entry for numberOfEntries on both RHDG servers on both sites.
  • If you updated the entity, such as user, on Red Hat Single Sign-On server on site1 and you do not see that entity updated on the Red Hat Single Sign-On server on site2, then the issue can be either in the replication of the synchronous database itself or that Red Hat Single Sign-On caches are not properly invalidated. You may try to temporarily disable the Red Hat Single Sign-On caches as described here to nail down if the issue is at the database replication level. Also it may help to manually connect to the database and check if data are updated as expected. This is specific to every database, so you will need to consult the documentation for your database.
  • Sometimes you may see the exceptions related to locks like this in RHDG server log:

    (HotRodServerHandler-6-35) ISPN000136: Error executing command ReplaceCommand,
    writing keys [[B0x033E243034396234..[39]]: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after
    0 milliseconds for key [B0x033E243034396234..[39] and requestor GlobalTx:server1:4353. Lock is held by GlobalTx:server1:4352

    Those exceptions are not necessarily an issue. They may happen anytime when a concurrent edit of the same entity is triggered on both DCs. This is common in a deployment. Usually the Red Hat Single Sign-On server is notified about the failed operation and will retry it, so from the user’s point of view, there is usually not any issue.

  • If you try to authenticate with Red Hat Single Sign-On to your application, but authentication fails with an infinite number of redirects in your browser and you see the errors like this in the Red Hat Single Sign-On server log:

    2017-11-27 14:50:31,587 WARN  [org.keycloak.events] (default task-17) type=LOGIN_ERROR, realmId=master, clientId=null, userId=null, ipAddress=aa.bb.cc.dd, error=expired_code, restart_after_timeout=true

    it probably means that your load balancer needs to be set to support sticky sessions. Make sure that the provided route name used during startup of Red Hat Single Sign-On server (Property jboss.node.name) contains the correct name used by the load balancer server to identify the current server.

  • If the RHDG work cache grows indefinitely, you may be experiencing this RHDG issue, which is caused by cache items not being properly expired. In that case, update the cache declaration with an empty <expiration /> tag like this:

        <replicated-cache name="work" configuration="sessions-cfg">
            <expiration />
        </replicated-cache>
  • If you see Warnings in the RHDG server log like:

    18:06:19,687 WARN  [org.infinispan.server.hotrod.Decoder2x] (HotRod-ServerWorker-7-12) ISPN006011: Operation 'PUT_IF_ABSENT' forced to
      return previous value should be used on transactional caches, otherwise data inconsistency issues could arise under failure situations
    18:06:19,700 WARN  [org.infinispan.server.hotrod.Decoder2x] (HotRod-ServerWorker-7-10) ISPN006010: Conditional operation 'REPLACE_IF_UNMODIFIED' should
      be used with transactional caches, otherwise data inconsistency issues could arise under failure situations

    you can just ignore them. To avoid the warning, the caches on RHDG server side could be changed to transactional caches, but this is not recommended as it can cause some other issues caused by the bug https://issues.redhat.com/browse/ISPN-9323. So for now, the warnings just need to be ignored.

  • If you see errors in the RHDG server log like:

    12:08:32,921 ERROR [org.infinispan.server.hotrod.CacheDecodeContext] (HotRod-ServerWorker-7-11) ISPN005003: Exception reported: org.infinispan.server.hotrod.InvalidMagicIdException: Error reading magic byte or message id: 7
    	at org.infinispan.server.hotrod.HotRodDecoder.readHeader(HotRodDecoder.java:184)
    	at org.infinispan.server.hotrod.HotRodDecoder.decodeHeader(HotRodDecoder.java:133)
    	at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.java:92)
    	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
    	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)

    and you see some similar errors in the Red Hat Single Sign-On log, it can indicate that there are incompatible versions of the Hot Rod protocol being used. This is likely happen when you try to use Red Hat Single Sign-On with an old version of the Infinispan server. It will help if you add the protocolVersion property as an additional property to the remote-store element in the Red Hat Single Sign-On configuration file. For example:

    <property name="protocolVersion">2.6</property>

Chapter 4. Managing the subsystem configuration

Low-level configuration of Red Hat Single Sign-On is done by editing the standalone.xml, standalone-ha.xml, or domain.xml file in your distribution. The location of this file depends on your operating mode.

While there are endless settings you can configure here, this section will focus on configuration of the keycloak-server subsystem. No matter which configuration file you are using, configuration of the keycloak-server subsystem is the same.

The keycloak-server subsystem is typically declared toward the end of the file like this:

<subsystem xmlns="urn:jboss:domain:keycloak-server:1.1">
   <web-context>auth</web-context>
   ...
</subsystem>

Note that anything changed in this subsystem will not take effect until the server is rebooted.

4.1. Configure SPI providers

The specifics of each configuration setting is discussed elsewhere in context with that setting. However, it is useful to understand the format used to declare settings on SPI providers.

Red Hat Single Sign-On is a highly modular system that allows great flexibility. There are more than 50 service provider interfaces (SPIs), and you are allowed to swap out implementations of each SPI. An implementation of an SPI is known as a provider.

All elements in an SPI declaration are optional, but a full SPI declaration looks like this:

<spi name="myspi">
    <default-provider>myprovider</default-provider>
    <provider name="myprovider" enabled="true">
        <properties>
            <property name="foo" value="bar"/>
        </properties>
    </provider>
    <provider name="mysecondprovider" enabled="true">
        <properties>
            <property name="foo" value="foo"/>
        </properties>
    </provider>
</spi>

Here we have two providers defined for the SPI myspi. The default-provider is listed as myprovider. However it is up to the SPI to decide how it will treat this setting. Some SPIs allow more than one provider and some do not. So default-provider can help the SPI to choose.

Also notice that each provider defines its own set of configuration properties. The fact that both providers above have a property called foo is just a coincidence.

The type of each property value is interpreted by the provider. However, there is one exception. Consider the jpa provider for the eventsStore SPI:

<spi name="eventsStore">
    <provider name="jpa" enabled="true">
        <properties>
            <property name="exclude-events" value="[&quot;EVENT1&quot;,
                                                    &quot;EVENT2&quot;]"/>
        </properties>
    </provider>
</spi>

We see that the value begins and ends with square brackets. That means that the value will be passed to the provider as a list. In this example, the system will pass the provider a list with two element values EVENT1 and EVENT2. To add more values to the list, just separate each list element with a comma. Unfortunately, you do need to escape the quotes surrounding each list element with &quot;.

Follow the steps in Server Developer Guide for more details on custom providers and the configuration of providers.

4.2. Starting the JBoss EAP CLI

Besides editing the configuration by hand, you also have the option of changing the configuration by issuing commands via the jboss-cli tool. CLI allows you to configure servers locally or remotely. And it is especially useful when combined with scripting.

To start the JBoss EAP CLI, you need to run jboss-cli.

Linux/Unix

$ .../bin/jboss-cli.sh

Windows

> ...\bin\jboss-cli.bat

This will bring you to a prompt like this:

Prompt

[disconnected /]

If you wish to execute commands on a running server, you will first execute the connect command.

connect

[disconnected /] connect
connect
[standalone@localhost:9990 /]

You may be thinking to yourself, "I didn’t enter in any username or password!". If you run jboss-cli on the same machine as your running standalone server or domain controller and your account has appropriate file permissions, you do not have to setup or enter in an admin username and password. See the JBoss EAP Configuration Guide for more details on how to make things more secure if you are uncomfortable with that setup.

4.3. CLI embedded mode

If you do happen to be on the same machine as your standalone server and you want to issue commands while the server is not active, you can embed the server into CLI and make changes in a special mode that disallows incoming requests. To do this, first execute the embed-server command with the config file you wish to change.

embed-server

[disconnected /] embed-server --server-config=standalone.xml
[standalone@embedded /]

4.4. Using CLI GUI mode

The CLI can also run in GUI mode. GUI mode launches a Swing application that allows you to graphically view and edit the entire management model of a running server. GUI mode is especially useful when you need help formatting your CLI commands and learning about the options available. The GUI can also retrieve server logs from a local or remote server.

Procedure

  1. Start the CLI in GUI mode

    $ .../bin/jboss-cli.sh --gui

    Note: to connect to a remote server, you pass the --connect option as well. Use the --help option for more details.

  2. Scroll down to find the node subsystem=keycloak-server.
  3. Right-click the node and select Explore subsystem=keycloak-server.

    A new tab displays only the keycloak-server subsystem.

    keycloak-server subsystem

    keycloak-server subsystem

4.5. CLI scripting

The CLI has extensive scripting capabilities. A script is just a text file with CLI commands in it. Consider a simple script that turns off theme and template caching.

turn-off-caching.cli

/subsystem=keycloak-server/theme=defaults/:write-attribute(name=cacheThemes,value=false)
/subsystem=keycloak-server/theme=defaults/:write-attribute(name=cacheTemplates,value=false)

To execute the script, you can follow the Scripts menu in CLI GUI, or execute the script from the command line as follows:

$ .../bin/jboss-cli.sh --file=turn-off-caching.cli

4.6. CLI recipes

Here are some configuration tasks and how to perform them with CLI commands. Note that in all but the first example, we use the wildcard path ** to mean you should substitute or the path to the keycloak-server subsystem.

For standalone, this just means:

** = /subsystem=keycloak-server

For domain mode, this would mean something like:

** = /profile=auth-server-clustered/subsystem=keycloak-server

4.6.1. Changing the web context of the server

/subsystem=keycloak-server/:write-attribute(name=web-context,value=myContext)

4.6.2. Setting the global default theme

**/theme=defaults/:write-attribute(name=default,value=myTheme)

4.6.3. Adding a new SPI and a provider

**/spi=mySPI/:add
**/spi=mySPI/provider=myProvider/:add(enabled=true)

4.6.4. Disabling a provider

**/spi=mySPI/provider=myProvider/:write-attribute(name=enabled,value=false)

4.6.5. Changing the default provider for an SPI

**/spi=mySPI/:write-attribute(name=default-provider,value=myProvider)

4.6.6. Configuring the dblock SPI

**/spi=dblock/:add(default-provider=jpa)
**/spi=dblock/provider=jpa/:add(properties={lockWaitTimeout => "900"},enabled=true)

4.6.7. Adding or changing a single property value for a provider

**/spi=dblock/provider=jpa/:map-put(name=properties,key=lockWaitTimeout,value=3)

4.6.8. Remove a single property from a provider

**/spi=dblock/provider=jpa/:map-remove(name=properties,key=lockRecheckTime)

4.6.9. Setting values on a provider property of type List

**/spi=eventsStore/provider=jpa/:map-put(name=properties,key=exclude-events,value=[EVENT1,EVENT2])

Chapter 5. Profiles

There are features in Red Hat Single Sign-On that are not enabled by default, these include features that are not fully supported. In addition there are some features that are enabled by default, but that can be disabled.

The features that can be enabled and disabled are:

NameDescriptionEnabled by defaultSupport level

account2

New Account Management Console

Yes

Supported

account_api

Account Management REST API

Yes

Supported

admin_fine_grained_authz

Fine-Grained Admin Permissions

No

Preview

ciba

OpenID Connect Client Initiated Backchannel Authentication (CIBA)

Yes

Supported

client_policies

Add client configuration policies

Yes

Supported

client_secret_rotation

Enables client secret rotation for confidential clients

Yes

Preview

par

OAuth 2.0 Pushed Authorization Requests (PAR)

Yes

Supported

declarative_user_profile

Configure user profiles using a declarative style

No

Preview

docker

Docker Registry protocol

No

Supported

impersonation

Ability for admins to impersonate users

Yes

Supported

openshift_integration

Extension to enable securing OpenShift

No

Preview

recovery_codes

Recovery codes for authentication

No

Preview

scripts

Write custom authenticators using JavaScript

No

Preview

step_up_authentication

Step-up authentication

Yes

Supported

token_exchange

Token Exchange Service

No

Preview

upload_scripts

Upload scripts

No

Deprecated

web_authn

W3C Web Authentication (WebAuthn)

Yes

Supported

update_email

Update Email Workflow

No

Preview

To enable all preview features start the server with:

bin/standalone.sh|bat -Dkeycloak.profile=preview

You can set this permanently by creating the file standalone/configuration/profile.properties (or domain/servers/server-one/configuration/profile.properties for server-one in domain mode). Add the following to the file:

profile=preview

To enable a specific feature start the server with:

bin/standalone.sh|bat -Dkeycloak.profile.feature.<feature name>=enabled

For example to enable Docker use -Dkeycloak.profile.feature.docker=enabled.

You can set this permanently in the profile.properties file by adding:

feature.docker=enabled

To disable a specific feature start the server with:

bin/standalone.sh|bat -Dkeycloak.profile.feature.<feature name>=disabled

For example to disable Impersonation use -Dkeycloak.profile.feature.impersonation=disabled.

You can set this permanently in the profile.properties file by adding:

feature.impersonation=disabled

Chapter 6. Setting up the relational database

Red Hat Single Sign-On comes with its own embedded Java-based relational database called H2. This is the default database that Red Hat Single Sign-On will use to persist data and really only exists so that you can run the authentication server out of the box. We highly recommend that you replace it with a more production ready external database. The H2 database is not very viable in high concurrency situations and should not be used in a cluster either. The purpose of this chapter is to show you how to connect Red Hat Single Sign-On to a more mature database.

Red Hat Single Sign-On uses two layered technologies to persist its relational data. The bottom layered technology is JDBC. JDBC is a Java API that is used to connect to a RDBMS. There are different JDBC drivers per database type that are provided by your database vendor. This chapter discusses how to configure Red Hat Single Sign-On to use one of these vendor-specific drivers.

The top layered technology for persistence is Hibernate JPA. This is an object to relational mapping API that maps Java Objects to relational data. Most deployments of Red Hat Single Sign-On will never have to touch the configuration aspects of Hibernate, but we will discuss how that is done if you run into that rare circumstance.

Note

Datasource configuration is covered much more thoroughly in the datasource configuration chapter in the JBoss EAP Configuration Guide.

6.1. Database setup checklist

Following are the steps you perform to get an RDBMS configured for Red Hat Single Sign-On.

  1. Locate and download a JDBC driver for your database
  2. Package the driver JAR into a module and install this module into the server
  3. Declare the JDBC driver in the configuration profile of the server
  4. Modify the datasource configuration to use your database’s JDBC driver
  5. Modify the datasource configuration to define the connection parameters to your database

This chapter will use PostgresSQL for all its examples. Other databases follow the same steps for installation.

6.2. Packaging the JDBC driver

Find and download the JDBC driver JAR for your RDBMS. Before you can use this driver, you must package it up into a module and install it into the server. Modules define JARs that are loaded into the Red Hat Single Sign-On classpath and the dependencies those JARs have on other modules.

Procedure

  1. Create a directory structure to hold your module definition within the …​/modules/ directory of your Red Hat Single Sign-On distribution.

    The convention is use the Java package name of the JDBC driver for the name of the directory structure. For PostgreSQL, create the directory org/postgresql/main.

  2. Copy your database driver JAR into this directory and create an empty module.xml file within it too.

    Module Directory

    Module Directory

  3. Open up the module.xml file and create the following XML:

    Module XML

    <?xml version="1.0" encoding="UTF-8"?>
    <module xmlns="urn:jboss:module:1.3" name="org.postgresql">
    
        <resources>
            <resource-root path="postgresql-VERSION.jar"/>
        </resources>
    
        <dependencies>
            <module name="javax.api"/>
            <module name="javax.transaction.api"/>
        </dependencies>
    </module>

    • The module name should match the directory structure of your module. So, org/postgresql maps to org.postgresql.
    • The resource-root path attribute should specify the JAR filename of the driver.
    • The rest are just the normal dependencies that any JDBC driver JAR would have.

6.3. Declaring and loading the JDBC driver

You declare your JDBC into your deployment profile so that it loads and becomes available when the server boots up.

Prerequisites

You have packaged the JDBC driver.

Procedure

  1. Declare your JDBC driver by editing one of these files based on your deployment mode:

    • For standalone mode, edit …​/standalone/configuration/standalone.xml.
    • For standalone clustering mode, edit …​/standalone/configuration/standalone-ha.xml.
    • For domain mode, edit …​/domain/configuration/domain.xml.

      In domain mode, make sure you edit the profile you are using: either auth-server-standalone or auth-server-clustered

  2. Within the profile, search for the drivers XML block within the datasources subsystem.

    You should see a pre-defined driver declared for the H2 JDBC driver. This is where you’ll declare the JDBC driver for your external database.

    JDBC Drivers

      <subsystem xmlns="urn:jboss:domain:datasources:6.0">
         <datasources>
           ...
           <drivers>
              <driver name="h2" module="com.h2database.h2">
                  <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
              </driver>
           </drivers>
         </datasources>
      </subsystem>

  3. Within the drivers XML block, declare an additional JDBC driver.

    • Assign any name to this driver.
    • Specify the module attribute which points to the module package that you created earlier for the driver JAR.
    • Specify the driver’s Java class.

      Here’s an example of installing a PostgreSQL driver that lives in the module example defined earlier in this chapter.

      Declare Your JDBC Drivers

        <subsystem xmlns="urn:jboss:domain:datasources:6.0">
           <datasources>
             ...
             <drivers>
                <driver name="postgresql" module="org.postgresql">
                    <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
                </driver>
                <driver name="h2" module="com.h2database.h2">
                    <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
                </driver>
             </drivers>
           </datasources>
        </subsystem>

6.4. Modifying the Red Hat Single Sign-On datasource

You modify the existing datasource configuration that Red Hat Single Sign-On uses to connect it to your new external database. You’ll do this within the same configuration file and XML block that you registered your JDBC driver in. Here’s an example that sets up the connection to your new database:

Declare Your JDBC Drivers

  <subsystem xmlns="urn:jboss:domain:datasources:6.0">
     <datasources>
       ...
       <datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true">
           <connection-url>jdbc:postgresql://localhost/keycloak</connection-url>
           <driver>postgresql</driver>
           <pool>
               <max-pool-size>20</max-pool-size>
           </pool>
           <security>
               <user-name>William</user-name>
               <password>password</password>
           </security>
       </datasource>
        ...
     </datasources>
  </subsystem>

Prerequisites

  • You have already declared your JDBC driver.

Procedure

  1. Search for the datasource definition for KeycloakDS.

    You’ll first need to modify the connection-url. The documentation for your vendor’s JDBC implementation should specify the format for this connection URL value.

  2. Define the driver you will use.

    This is the logical name of the JDBC driver you declared in the previous section of this chapter.

    It is expensive to open a new connection to a database every time you want to perform a transaction. To compensate, the datasource implementation maintains a pool of open connections. The max-pool-size specifies the maximum number of connections it will pool. You may want to change the value of this depending on the load of your system.

  3. Define the database username and password that is needed to connect to the database. This step is necessary for at least PostgreSQL. You may be concerned that these credentials are in clear text in the example. Methods exist to obfuscate these credentials, but these methods are beyond the scope of this guide.
Note

For more information about datasource features, see the datasource configuration chapter in the JBoss EAP Configuration Guide.

6.5. Database Configuration

The configuration for this component is found in the standalone.xml, standalone-ha.xml, or domain.xml file in your distribution. The location of this file depends on your operating mode.

Database Config

<subsystem xmlns="urn:jboss:domain:keycloak-server:1.1">
    ...
    <spi name="connectionsJpa">
     <provider name="default" enabled="true">
         <properties>
             <property name="dataSource" value="java:jboss/datasources/KeycloakDS"/>
             <property name="initializeEmpty" value="false"/>
             <property name="migrationStrategy" value="manual"/>
             <property name="migrationExport" value="${jboss.home.dir}/keycloak-database-update.sql"/>
         </properties>
     </provider>
    </spi>
    ...
</subsystem>

Possible configuration options are:

dataSource
JNDI name of the dataSource
jta
boolean property to specify if datasource is JTA capable
driverDialect
Value of database dialect. In most cases you don’t need to specify this property as dialect will be autodetected by Hibernate.
initializeEmpty
Initialize database if empty. If set to false the database has to be manually initialized. If you want to manually initialize the database set migrationStrategy to manual which will create a file with SQL commands to initialize the database. Defaults to true.
migrationStrategy
Strategy to use to migrate database. Valid values are update, manual and validate. Update will automatically migrate the database schema. Manual will export the required changes to a file with SQL commands that you can manually execute on the database. Validate will simply check if the database is up-to-date.
migrationExport
Path for where to write manual database initialization/migration file.
showSql
Specify whether Hibernate should show all SQL commands in the console (false by default). This is very verbose!
formatSql
Specify whether Hibernate should format SQL commands (true by default)
globalStatsInterval
Will log global statistics from Hibernate about executed DB queries and other things. Statistics are always reported to server log at specified interval (in seconds) and are cleared after each report.
schema
Specify the database schema to use
Note

These configuration switches and more are described in the JBoss EAP Development Guide.

6.6. Unicode considerations for databases

Database schema in Red Hat Single Sign-On only accounts for Unicode strings in the following special fields:

  • Realms: display name, HTML display name, localization texts (keys and values)
  • Federation Providers: display name
  • Users: username, given name, last name, attribute names and values
  • Groups: name, attribute names and values
  • Roles: name
  • Descriptions of objects

Otherwise, characters are limited to those contained in database encoding which is often 8-bit. However, for some database systems, it is possible to enable UTF-8 encoding of Unicode characters and use full Unicode character set in all text fields. Often, this is counterbalanced by shorter maximum length of the strings than in case of 8-bit encodings.

Some of the databases require special settings to database and/or JDBC driver to be able to handle Unicode characters. Please find the settings for your database below. Note that if a database is listed here, it can still work properly provided it handles UTF-8 encoding properly both on the level of database and JDBC driver.

Technically, the key criterion for Unicode support for all fields is whether the database allows setting of Unicode character set for VARCHAR and CHAR fields. If yes, there is a high chance that Unicode will be plausible, usually at the expense of field length. If it only supports Unicode in NVARCHAR and NCHAR fields, Unicode support for all text fields is unlikely as Keycloak schema uses VARCHAR and CHAR fields extensively.

6.6.1. Oracle database

Unicode characters are properly handled provided the database was created with Unicode support in VARCHAR and CHAR fields (e.g. by using AL32UTF8 character set as the database character set). No special settings is needed for JDBC driver.

If the database character set is not Unicode, then to use Unicode characters in the special fields, the JDBC driver needs to be configured with the connection property oracle.jdbc.defaultNChar set to true. It might be wise, though not strictly necessary, to also set the oracle.jdbc.convertNcharLiterals connection property to true. These properties can be set either as system properties or as connection properties. Please note that setting oracle.jdbc.defaultNChar may have negative impact on performance. For details, please refer to Oracle JDBC driver configuration documentation.

6.6.2. Microsoft SQL Server database

Unicode characters are properly handled only for the special fields. No special settings of JDBC driver or database is necessary.

6.6.3. MySQL database

Unicode characters are properly handled provided the database was created with Unicode support in VARCHAR and CHAR fields in the CREATE DATABASE command (e.g. by using utf8 character set as the default database character set in MySQL 5.5. Please note that utf8mb4 character set does not work due to different storage requirements to utf8 character set [1]). Note that in this case, length restriction to non-special fields does not apply because columns are created to accommodate given amount of characters, not bytes. If the database default character set does not allow storing Unicode, only the special fields allow storing Unicode values.

At the side of JDBC driver settings, it is necessary to add a connection property characterEncoding=UTF-8 to the JDBC connection settings.

6.6.4. PostgreSQL database

Unicode is supported when the database character set is UTF8. In that case, Unicode characters can be used in any field, there is no reduction of field length for non-special fields. No special settings of JDBC driver is necessary.

The character set of a PostgreSQL database is determined at the time it is created. You can determine the default character set for a PostgreSQL cluster with the SQL command

show server_encoding;

If the default character set is not UTF 8, then you can create the database with UTF8 as its character set like this:

create database keycloak with encoding 'UTF8';

Chapter 7. Use of the public hostname

Red Hat Single Sign-On uses the public hostname for a number of things. For example, in the token issuer fields and URLs sent in password reset emails.

The Hostname SPI provides a way to configure the hostname for a request. The default provider allows setting a fixed URL for frontend requests, while allowing backend requests to be based on the request URI. It is also possible to develop your own provider in the case the built-in provider does not provide the functionality needed.

7.1. Default provider

The default hostname provider uses the configured frontendUrl as the base URL for frontend requests (requests from user-agents) and uses the request URL as the basis for backend requests (direct requests from clients).

Frontend request do not have to have the same context-path as the Keycloak server. This means you can expose Keycloak on for example https://auth.example.org or https://example.org/keycloak while internally its URL could be https://10.0.0.10:8080/auth.

This makes it possible to have user-agents (browsers) send requests to Red Hat Single Sign-On through the public domain name, while internal clients can use an internal domain name or IP address.

This is reflected in the OpenID Connect Discovery endpoint for example where the authorization_endpoint uses the frontend URL, while token_endpoint uses the backend URL. As a note here a public client for instance would contact Keycloak through the public endpoint, which would result in the base of authorization_endpoint and token_endpoint being the same.

To set the frontendUrl for Keycloak you can either pass add -Dkeycloak.frontendUrl=https://auth.example.org to the startup or you can configure it in standalone.xml. See the example below:

<spi name="hostname">
    <default-provider>default</default-provider>
    <provider name="default" enabled="true">
        <properties>
            <property name="frontendUrl" value="https://auth.example.com"/>
            <property name="forceBackendUrlToFrontendUrl" value="false"/>
        </properties>
    </provider>
</spi>

To update the frontendUrl with jboss-cli use the following command:

/subsystem=keycloak-server/spi=hostname/provider=default:write-attribute(name=properties.frontendUrl,value="https://auth.example.com")

If you want all requests to go through the public domain name you can force backend requests to use the frontend URL as well by setting forceBackendUrlToFrontendUrl to true.

It is also possible to override the default frontend URL for individual realms. This can be done in the admin console.

If you do not want to expose the admin endpoints and console on the public domain use the property adminUrl to set a fixed URL for the admin console, which is different to the frontendUrl. It is also required to block access to /auth/admin externally, for details on how to do that refer to the Server Administration Guide.

7.2. Custom provider

To develop a custom hostname provider you need to implement org.keycloak.urls.HostnameProviderFactory and org.keycloak.urls.HostnameProvider.

Follow the instructions in the Service Provider Interfaces section in Server Developer Guide for more information on how to develop a custom provider.

Chapter 8. Setting up the network

The default installation of Red Hat Single Sign-On can run with some networking limitations. For one, all network endpoints bind to localhost so the auth server is really only usable on one local machine. For HTTP based connections, it does not use default ports like 80 and 443. HTTPS/SSL is not configured out of the box and without it, Red Hat Single Sign-On has many security vulnerabilities. Finally, Red Hat Single Sign-On may often need to make secure SSL and HTTPS connections to external servers and thus need a trust store set up so that endpoints can be validated correctly. This chapter discusses all of these things.

8.1. Bind addresses

By default Red Hat Single Sign-On binds to the localhost loopback address 127.0.0.1. That’s not a very useful default if you want the authentication server available on your network. Generally, what we recommend is that you deploy a reverse proxy or load balancer on a public network and route traffic to individual Red Hat Single Sign-On server instances on a private network. In either case though, you still need to set up your network interfaces to bind to something other than localhost.

Setting the bind address is quite easy and can be done on the command line with either the standalone.sh or domain.sh boot scripts discussed in the Choosing an Operating Mode chapter.

$ standalone.sh -b 192.168.0.5

The -b switch sets the IP bind address for any public interfaces.

Alternatively, if you don’t want to set the bind address at the command line, you can edit the profile configuration of your deployment. Open up the profile configuration file (standalone.xml or domain.xml depending on your operating mode) and look for the interfaces XML block.

    <interfaces>
        <interface name="management">
            <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
        </interface>
        <interface name="public">
            <inet-address value="${jboss.bind.address:127.0.0.1}"/>
        </interface>
    </interfaces>

The public interface corresponds to subsystems creating sockets that are available publicly. An example of one of these subsystems is the web layer which serves up the authentication endpoints of Red Hat Single Sign-On. The management interface corresponds to sockets opened up by the management layer of the JBoss EAP. Specifically the sockets which allow you to use the jboss-cli.sh command line interface and the JBoss EAP web console.

In looking at the public interface you see that it has a special string ${jboss.bind.address:127.0.0.1}. This string denotes a value 127.0.0.1 that can be overridden on the command line by setting a Java system property, i.e.:

$ domain.sh -Djboss.bind.address=192.168.0.5

The -b is just a shorthand notation for this command. So, you can either change the bind address value directly in the profile config, or change it on the command line when you boot up.

Note

There are many more options available when setting up interface definitions. For more information, see the network interface in the JBoss EAP Configuration Guide.

8.2. Socket port bindings

The ports opened for each socket have a pre-defined default that can be overridden at the command line or within configuration. To illustrate this configuration, let’s pretend you are running in standalone mode and open up the …​/standalone/configuration/standalone.xml. Search for socket-binding-group.

    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
        <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
        <socket-binding name="http" port="${jboss.http.port:8080}"/>
        <socket-binding name="https" port="${jboss.https.port:8443}"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="localhost" port="25"/>
        </outbound-socket-binding>
    </socket-binding-group>

socket-bindings define socket connections that will be opened by the server. These bindings specify the interface (bind address) they use as well as what port number they will open. The ones you will be most interested in are:

http
Defines the port used for Red Hat Single Sign-On HTTP connections
https
Defines the port used for Red Hat Single Sign-On HTTPS connections
ajp
This socket binding defines the port used for the AJP protocol. This protocol is used by Apache HTTPD server in conjunction mod-cluster when you are using Apache HTTPD as a load balancer.
management-http
Defines the HTTP connection used by JBoss EAP CLI and web console.

When running in domain mode setting the socket configurations is a bit trickier as the example domain.xml file has multiple socket-binding-groups defined. If you scroll down to the server-group definitions you can see what socket-binding-group is used for each server-group.

domain socket bindings

    <server-groups>
        <server-group name="load-balancer-group" profile="load-balancer">
            ...
            <socket-binding-group ref="load-balancer-sockets"/>
        </server-group>
        <server-group name="auth-server-group" profile="auth-server-clustered">
            ...
            <socket-binding-group ref="ha-sockets"/>
        </server-group>
    </server-groups>

Note

There are many more options available when setting up socket-binding-group definitions. For more information, see the socket binding group in the JBoss EAP Configuration Guide.

8.3. Setting up HTTPS/SSL

Warning

Red Hat Single Sign-On is not set up by default to handle SSL/HTTPS. It is highly recommended that you either enable SSL on the Red Hat Single Sign-On server itself or on a reverse proxy in front of the Red Hat Single Sign-On server.

This default behavior is defined by the SSL/HTTPS mode of each Red Hat Single Sign-On realm. This is discussed in more detail in the Server Administration Guide, but let’s give some context and a brief overview of these modes.

external requests
Red Hat Single Sign-On can run out of the box without SSL so long as you stick to private IP addresses like localhost, 127.0.0.1, 10.x.x.x, 192.168.x.x, and 172.16.x.x. If you don’t have SSL/HTTPS configured on the server or you try to access Red Hat Single Sign-On over HTTP from a non-private IP adress you will get an error.
none
Red Hat Single Sign-On does not require SSL. This should really only be used in development when you are playing around with things.
all requests
Red Hat Single Sign-On requires SSL for all IP addresses.

The SSL mode for each realm can be configured in the Red Hat Single Sign-On admin console.

8.3.1. Enabling SSL/HTTPS for the Red Hat Single Sign-On server

If you are not using a reverse proxy or load balancer to handle HTTPS traffic for you, you’ll need to enable HTTPS for the Red Hat Single Sign-On server. This involves

  1. Obtaining or generating a keystore that contains the private key and certificate for SSL/HTTP traffic
  2. Configuring the Red Hat Single Sign-On server to use this keypair and certificate.

8.3.1.1. Creating the Certificate and Java Keystore

In order to allow HTTPS connections, you need to obtain a self signed or third-party signed certificate and import it into a Java keystore before you can enable HTTPS in the web container where you are deploying the Red Hat Single Sign-On Server.

8.3.1.1.1. Self Signed Certificate

In development, you will probably not have a third party signed certificate available to test a Red Hat Single Sign-On deployment so you’ll need to generate a self-signed one using the keytool utility that comes with the Java JDK.

$ keytool -genkey -alias localhost -keyalg RSA -keystore keycloak.jks -validity 10950
    Enter keystore password: secret
    Re-enter new password: secret
    What is your first and last name?
    [Unknown]:  localhost
    What is the name of your organizational unit?
    [Unknown]:  Keycloak
    What is the name of your organization?
    [Unknown]:  Red Hat
    What is the name of your City or Locality?
    [Unknown]:  Westford
    What is the name of your State or Province?
    [Unknown]:  MA
    What is the two-letter country code for this unit?
    [Unknown]:  US
    Is CN=localhost, OU=Keycloak, O=Test, L=Westford, ST=MA, C=US correct?
    [no]:  yes

When you see the question What is your first and last name ?, supply the DNS name of the machine where you are installing the server. For testing purposes, localhost should be used. After executing this command, the keycloak.jks file will be generated in the same directory as you executed the keytool command in.

If you want a third-party signed certificate, but don’t have one, you can obtain one for free at cacert.org. However, you first need to use the following procedure.

Procedure

  1. Generate a Certificate Request:

    $ keytool -certreq -alias yourdomain -keystore keycloak.jks > keycloak.careq

    Where yourdomain is a DNS name for which this certificate is generated. Keytool generates the request:

    -----BEGIN NEW CERTIFICATE REQUEST-----
    MIIC2jCCAcICAQAwZTELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAk1BMREwDwYDVQQHEwhXZXN0Zm9y
    ZDEQMA4GA1UEChMHUmVkIEhhdDEQMA4GA1UECxMHUmVkIEhhdDESMBAGA1UEAxMJbG9jYWxob3N0
    MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAr7kck2TaavlEOGbcpi9c0rncY4HhdzmY
    Ax2nZfq1eZEaIPqI5aTxwQZzzLDK9qbeAd8Ji79HzSqnRDxNYaZu7mAYhFKHgixsolE3o5Yfzbw1
    29RvyeUVe+WZxv5oo9wolVVpdSINIMEL2LaFhtX/c1dqiqYVpfnvFshZQaIg2nL8juzZcBjj4as
    H98gIS7khql/dkZKsw9NLvyxgJvp7PaXurX29fNf3ihG+oFrL22oFyV54BWWxXCKU/GPn61EGZGw
    Ft2qSIGLdctpMD1aJR2bcnlhEjZKDksjQZoQ5YMXaAGkcYkG6QkgrocDE2YXDbi7GIdf9MegVJ35
    2DQMpwIDAQABoDAwLgYJKoZIhvcNAQkOMSEwHzAdBgNVHQ4EFgQUQwlZJBA+fjiDdiVzaO9vrE/i
    n2swDQYJKoZIhvcNAQELBQADggEBAC5FRvMkhal3q86tHPBYWBuTtmcSjs4qUm6V6f63frhveWHf
    PzRrI1xH272XUIeBk0gtzWo0nNZnf0mMCtUBbHhhDcG82xolikfqibZijoQZCiGiedVjHJFtniDQ
    9bMDUOXEMQ7gHZg5q6mJfNG9MbMpQaUVEEFvfGEQQxbiFK7hRWU8S23/d80e8nExgQxdJWJ6vd0X
    MzzFK6j4Dj55bJVuM7GFmfdNC52pNOD5vYe47Aqh8oajHX9XTycVtPXl45rrWAH33ftbrS8SrZ2S
    vqIFQeuLL3BaHwpl3t7j2lMWcK1p80laAxEASib/fAwrRHpLHBXRcq6uALUOZl4Alt8=
    -----END NEW CERTIFICATE REQUEST-----
  2. Send this CA request to your Certificate Authority (CA).

    The CA will issue you a signed certificate and send it to you.

  3. Obtain and import the root certificate of the CA.

    You can download the cert from CA (in other words: root.crt) and import as follows:

    $ keytool -import -keystore keycloak.jks -file root.crt -alias root
  4. Import your new CA generated certificate to your keystore:

    $ keytool -import -alias yourdomain -keystore keycloak.jks -file your-certificate.cer

8.3.1.2. Configure Red Hat Single Sign-On to Use the Keystore

Now that you have a Java keystore with the appropriate certificates, you need to configure your Red Hat Single Sign-On installation to use it.

Procedure

  1. Edit the standalone.xml, standalone-ha.xml, or host.xml file to use the keystore and enable HTTPS.
  2. Either move the keystore file to the configuration/ directory of your deployment or the file in a location you choose and provide an absolute path to it.

    If you are using absolute paths, remove the optional relative-to parameter from your configuration (See operating mode).

  3. Configure the keystore using the CLI:

    $ /subsystem=elytron/key-store=httpsKS:add(relative-to=jboss.server.config.dir,path=keycloak.jks,credential-reference={clear-text=secret},type=JKS)
    $ /subsystem=elytron/key-manager=httpsKM:add(key-store=httpsKS,credential-reference={clear-text=secret})
    $ /subsystem=elytron/server-ssl-context=httpsSSC:add(key-manager=httpsKM,protocols=[\"TLSv1.3\"])

    If using domain mode, the commands should be executed in every host using the /host=<host_name>/ prefix (in order to create the security-realm in all of them). Here is an example, which you would repeat for each host:

    $ /host=<host_name>/subsystem=elytron/key-store=httpsKS:add(relative-to=jboss.server.config.dir,path=keycloak.jks,credential-reference={clear-text=secret},type=JKS)
  4. Modify the https-listener to use the `server-ssl-context`previously created:

    $ /subsystem=undertow/server=default-server/https-listener=https:write-attribute(name=ssl-context, value=httpsSSC)

    If using domain mode, prefix the command with the profile that is being used with: /profile=<profile_name>/.

    The resulting element, server name="default-server", which is a child element of subsystem xmlns="urn:jboss:domain:undertow:12.0", should contain the following stanza:

    <subsystem xmlns="urn:jboss:domain:undertow:12.0">
       <buffer-cache name="default"/>
       <server name="default-server">
          <https-listener name="https" socket-binding="https" ssl-context="httpsSSC"/>
       ...
    </subsystem>

For more information on configuring TLS refer to the WildFly documentation.

8.4. Outgoing HTTP requests

The Red Hat Single Sign-On server often needs to make non-browser HTTP requests to the applications and services it secures. The auth server manages these outgoing connections by maintaining an HTTP client connection pool. There are some things you’ll need to configure in standalone.xml, standalone-ha.xml, or domain.xml. The location of this file depends on your operating mode.

HTTP client Config example

<spi name="connectionsHttpClient">
    <provider name="default" enabled="true">
        <properties>
            <property name="connection-pool-size" value="256"/>
        </properties>
    </provider>
</spi>

Possible configuration options are:

establish-connection-timeout-millis
Timeout for establishing a socket connection.
socket-timeout-millis
If an outgoing request does not receive data for this amount of time, timeout the connection.
connection-pool-size
How many connections can be in the pool (128 by default).
max-pooled-per-route
How many connections can be pooled per host (64 by default).
connection-ttl-millis
Maximum connection time to live in milliseconds. Not set by default.
max-connection-idle-time-millis
Maximum time the connection might stay idle in the connection pool (900 seconds by default). Will start background cleaner thread of Apache HTTP client. Set to -1 to disable this checking and the background thread.
disable-cookies
true by default. When set to true, this will disable any cookie caching.
client-keystore
This is the file path to a Java keystore file. This keystore contains client certificate for two-way SSL.
client-keystore-password
Password for the client keystore. This is REQUIRED if client-keystore is set.
client-key-password
Password for the client’s key. This is REQUIRED if client-keystore is set.
proxy-mappings
Denotes proxy configurations for outgoing HTTP requests. See the section on Proxy Mappings for Outgoing HTTP Requests for more details.
disable-trust-manager
If an outgoing request requires HTTPS and this config option is set to true you do not have to specify a truststore. This setting should only be used during development and never in production as it will disable verification of SSL certificates. This is OPTIONAL. The default value is false.

8.4.1. Proxy mappings for outgoing HTTP requests

Outgoing HTTP requests sent by Red Hat Single Sign-On can optionally use a proxy server based on a comma delimited list of proxy-mappings. A proxy-mapping denotes the combination of a regex based hostname pattern and a proxy-uri in the form of hostnamePattern;proxyUri, e.g.:

.*\.(google|googleapis)\.com;http://www-proxy.acme.com:8080

To determine the proxy for an outgoing HTTP request the target hostname is matched against the configured hostname patterns. The first matching pattern determines the proxy-uri to use. If none of the configured patterns match for the given hostname then no proxy is used.

If the proxy server requires authentication, include the proxy user’s credentials in this format username:password@. For example:

.*\.(google|googleapis)\.com;http://user01:pas2w0rd@www-proxy.acme.com:8080

The special value NO_PROXY for the proxy-uri can be used to indicate that no proxy should be used for hosts matching the associated hostname pattern. It is possible to specify a catch-all pattern at the end of the proxy-mappings to define a default proxy for all outgoing requests.

The following example demonstrates the proxy-mapping configuration.

# All requests to Google APIs should use http://www-proxy.acme.com:8080 as proxy
.*\.(google|googleapis)\.com;http://www-proxy.acme.com:8080

# All requests to internal systems should use no proxy
.*\.acme\.com;NO_PROXY

# All other requests should use http://fallback:8080 as proxy
.*;http://fallback:8080

This can be configured via the following jboss-cli command. Note that you need to properly escape the regex-pattern as shown below.

echo SETUP: Configure proxy routes for HttpClient SPI

# In case there is no connectionsHttpClient definition yet
/subsystem=keycloak-server/spi=connectionsHttpClient/provider=default:add(enabled=true)

# Configure the proxy-mappings
/subsystem=keycloak-server/spi=connectionsHttpClient/provider=default:write-attribute(name=properties.proxy-mappings,value=[".*\\.(google|googleapis)\\.com;http://www-proxy.acme.com:8080",".*\\.acme\\.com;NO_PROXY",".*;http://fallback:8080"])

The jboss-cli command results in the following subsystem configuration. Note that one needs to encode " characters with &quot;.

<spi name="connectionsHttpClient">
    <provider name="default" enabled="true">
        <properties>
            <property
            name="proxy-mappings"
            value="[&quot;.*\\.(google|googleapis)\\.com;http://www-proxy.acme.com:8080&quot;,&quot;.*\\.acme\\.com;NO_PROXY&quot;,&quot;.*;http://fallback:8080&quot;]"/>
        </properties>
    </provider>
</spi>

8.4.2. Using standard environment variables

Alternatively, it is possible to use standard environment variables to configure the proxy mappings, that is HTTP_PROXY, HTTPS_PROXY and NO_PROXY variables.

The HTTP_PROXY and HTTPS_PROXY variables represent the proxy server that should be used for all outgoing HTTP requests. Red Hat Single Sign-On does not differ between the two. If both are specified, HTTPS_PROXY takes the precedence regardless of the actual scheme the proxy server uses.

The NO_PROXY variable is used to define a comma separated list of hostnames that should not use the proxy. If a hostname is specified, all its prefixes (subdomains) are also excluded from using proxy.

Take the following example:

HTTPS_PROXY=https://www-proxy.acme.com:8080
NO_PROXY=google.com,login.facebook.com

In this example, all outgoing HTTP requests will use https://www-proxy.acme.com:8080 proxy server except for requests to for example login.google.com, google.com, auth.login.facebook.com. However, for example groups.facebook.com will be routed through the proxy.

Note

The environment variables can be lowercase or uppercase. Lowercase takes precedence. For example if both HTTP_PROXY and http_proxy are defined, http_proxy will be used.

If proxy mappings are defined using the subsystem configuration (as described above), the environment variables are not considered by Red Hat Single Sign-On. This scenario applies in case no proxy server should be used despite having for example HTTP_PROXY environment variable defined. To do so, you can specify a generic no proxy route as follows:

<spi name="connectionsHttpClient">
    <provider name="default" enabled="true">
        <properties>
            <property name="proxy-mappings" value=".*;NO_PROXY"/>
        </properties>
    </provider>
</spi>

8.4.3. Outgoing HTTPS request truststore

When Red Hat Single Sign-On invokes on remote HTTPS endpoints, it has to validate the remote server’s certificate in order to ensure it is connecting to a trusted server. This is necessary in order to prevent man-in-the-middle attacks. The certificates of these remote server’s or the CA that signed these certificates must be put in a truststore. This truststore is managed by the Red Hat Single Sign-On server.

The truststore is used when connecting securely to identity brokers, LDAP identity providers, when sending emails, and for backchannel communication with client applications.

Warning

By default, a truststore provider is not configured, and any https connections fall back to standard java truststore configuration as described in Java’s JSSE Reference Guide. If there is no trust established, then these outgoing HTTPS requests will fail.

You can use keytool to create a new truststore file or add trusted host certificates to an existing one:

$ keytool -import -alias HOSTDOMAIN -keystore truststore.jks -file host-certificate.cer

The truststore is configured within the standalone.xml, standalone-ha.xml, or domain.xml file in your distribution. The location of this file depends on your operating mode. You can add your truststore configuration by using the following template:

<spi name="truststore">
    <provider name="file" enabled="true">
        <properties>
            <property name="file" value="path to your .jks file containing public certificates"/>
            <property name="password" value="password"/>
            <property name="hostname-verification-policy" value="WILDCARD"/>
        </properties>
    </provider>
</spi>

Possible configuration options for this setting are:

file
The path to a Java keystore file. HTTPS requests need a way to verify the host of the server they are talking to. This is what the trustore does. The keystore contains one or more trusted host certificates or certificate authorities. This truststore file should only contain public certificates of your secured hosts. This is REQUIRED if any of these properties are defined.
password
Password of the keystore. This is REQUIRED if any of these properties are defined.
hostname-verification-policy
WILDCARD by default. For HTTPS requests, this verifies the hostname of the server’s certificate. ANY means that the hostname is not verified. WILDCARD Allows wildcards in subdomain names i.e. *.foo.com. STRICT CN must match hostname exactly.

Chapter 9. Configuring Red Hat Single Sign-On to run in a cluster

To configure Red Hat Single Sign-On to run in a cluster, you perform these actions:

Picking an operation mode and configuring a shared database have been discussed earlier in this guide. This chapter describes setting up a load balancer and supplying a private network as well as booting up a host in the cluster.

Note

It is possible to cluster Red Hat Single Sign-On without IP Multicast, but this topic is beyond the scope of this guide. For more information, see JGroups chapter of the JBoss EAP Configuration Guide.

9.2. Clustering example

Red Hat Single Sign-On does come with an out of the box clustering demo that leverages domain mode. Review the Clustered Domain Example chapter for more details.

9.3. Setting Up a load balancer or proxy

This section discusses a number of things you need to configure before you can put a reverse proxy or load balancer in front of your clustered Red Hat Single Sign-On deployment. It also covers configuring the built-in load balancer that was Clustered Domain Example.

The following diagram illustrates the use of a load balancer. In this example, the load balancer serves as a reverse proxy between three clients and a cluster of three Red Hat Single Sign-On servers.

Example Load Balancer Diagram

load balancer

9.3.1. Identifying client IP addresses

A few features in Red Hat Single Sign-On rely on the fact that the remote address of the HTTP client connecting to the authentication server is the real IP address of the client machine. Examples include:

  • Event logs - a failed login attempt would be logged with the wrong source IP address
  • SSL required - if the SSL required is set to external (the default) it should require SSL for all external requests
  • Authentication flows - a custom authentication flow that uses the IP address to for example show OTP only for external requests
  • Dynamic Client Registration

This can be problematic when you have a reverse proxy or loadbalancer in front of your Red Hat Single Sign-On authentication server. The usual setup is that you have a frontend proxy sitting on a public network that load balances and forwards requests to backend Red Hat Single Sign-On server instances located in a private network. There is some extra configuration you have to do in this scenario so that the actual client IP address is forwarded to and processed by the Red Hat Single Sign-On server instances. Specifically:

  • Configure your reverse proxy or loadbalancer to properly set X-Forwarded-For and X-Forwarded-Proto HTTP headers.
  • Configure your reverse proxy or loadbalancer to preserve the original 'Host' HTTP header.
  • Configure the authentication server to read the client’s IP address from X-Forwarded-For header.

Configuring your proxy to generate the X-Forwarded-For and X-Forwarded-Proto HTTP headers and preserving the original Host HTTP header is beyond the scope of this guide. Take extra precautions to ensure that the X-Forwarded-For header is set by your proxy. If your proxy isn’t configured correctly, then rogue clients can set this header themselves and trick Red Hat Single Sign-On into thinking the client is connecting from a different IP address than it actually is. This becomes really important if you are doing any black or white listing of IP addresses.

Beyond the proxy itself, there are a few things you need to configure on the Red Hat Single Sign-On side of things. If your proxy is forwarding requests via the HTTP protocol, then you need to configure Red Hat Single Sign-On to pull the client’s IP address from the X-Forwarded-For header rather than from the network packet. To do this, open up the profile configuration file (standalone.xml, standalone-ha.xml, or domain.xml depending on your operating mode) and look for the urn:jboss:domain:undertow:12.0 XML block.

X-Forwarded-For HTTP Config

<subsystem xmlns="urn:jboss:domain:undertow:12.0">
   <buffer-cache name="default"/>
   <server name="default-server">
      <ajp-listener name="ajp" socket-binding="ajp"/>
      <http-listener name="default" socket-binding="http" redirect-socket="https"
          proxy-address-forwarding="true"/>
      ...
   </server>
   ...
</subsystem>

Add the proxy-address-forwarding attribute to the http-listener element. Set the value to true.

If your proxy is using the AJP protocol instead of HTTP to forward requests (i.e. Apache HTTPD + mod-cluster), then you have to configure things a little differently. Instead of modifying the http-listener, you need to add a filter to pull this information from the AJP packets.

X-Forwarded-For AJP Config

<subsystem xmlns="urn:jboss:domain:undertow:12.0">
     <buffer-cache name="default"/>
     <server name="default-server">
         <ajp-listener name="ajp" socket-binding="ajp"/>
         <http-listener name="default" socket-binding="http" redirect-socket="https"/>
         <host name="default-host" alias="localhost">
             ...
             <filter-ref name="proxy-peer"/>
         </host>
     </server>
        ...
     <filters>
         ...
         <filter name="proxy-peer"
                 class-name="io.undertow.server.handlers.ProxyPeerAddressHandler"
                 module="io.undertow.core" />
     </filters>
 </subsystem>

9.3.2. Enabling HTTPS/SSL with a reverse proxy

Assuming that your reverse proxy doesn’t use port 8443 for SSL you also need to configure to what port the HTTPS traffic is redirected.

<subsystem xmlns="urn:jboss:domain:undertow:12.0">
    ...
    <http-listener name="default" socket-binding="http"
        proxy-address-forwarding="true" redirect-socket="proxy-https"/>
    ...
</subsystem>

Procedure

  1. Add the redirect-socket attribute to the http-listener element. The value should be proxy-https which points to a socket binding you also need to define.
  2. Add a new socket-binding element to the socket-binding-group element:

    <socket-binding-group name="standard-sockets" default-interface="public"
        port-offset="${jboss.socket.binding.port-offset:0}">
        ...
        <socket-binding name="proxy-https" port="443"/>
        ...
    </socket-binding-group>

9.3.3. Verifying the configuration

You can verify the reverse proxy or load balancer configuration

Procedure

  1. Open the path /auth/realms/master/.well-known/openid-configuration through the reverse proxy.

    For example if the reverse proxy address is https://acme.com/ then open the URL https://acme.com/auth/realms/master/.well-known/openid-configuration. This will show a JSON document listing a number of endpoints for Red Hat Single Sign-On.

  2. Make sure the endpoints starts with the address (scheme, domain and port) of your reverse proxy or load balancer. By doing this you make sure that Red Hat Single Sign-On is using the correct endpoint.
  3. Verify that Red Hat Single Sign-On sees the correct source IP address for requests.

    To check this, you can try to login to the Admin Console with an invalid username and/or password. This should show a warning in the server log something like this:

    08:14:21,287 WARN  XNIO-1 task-45 [org.keycloak.events] type=LOGIN_ERROR, realmId=master, clientId=security-admin-console, userId=8f20d7ba-4974-4811-a695-242c8fbd1bf8, ipAddress=X.X.X.X, error=invalid_user_credentials, auth_method=openid-connect, auth_type=code, redirect_uri=http://localhost:8080/auth/admin/master/console/?redirect_fragment=%2Frealms%2Fmaster%2Fevents-settings, code_id=a3d48b67-a439-4546-b992-e93311d6493e, username=admin
  4. Check that the value of ipAddress is the IP address of the machine you tried to login with and not the IP address of the reverse proxy or load balancer.

9.3.4. Using the built-in load balancer

This section covers configuring the built-in load balancer that is discussed in the Clustered Domain Example.

The Clustered Domain Example is only designed to run on one machine. To bring up a slave on another host, you’ll need to

  1. Edit the domain.xml file to point to your new host slave
  2. Copy the server distribution. You don’t need the domain.xml, host.xml, or host-master.xml files. Nor do you need the standalone/ directory.
  3. Edit the host-slave.xml file to change the bind addresses used or override them on the command line

Procedure

  1. Open domain.xml so you can registering the new host slave with the load balancer configuration.
  2. Go to the undertow configuration in the load-balancer profile. Add a new host definition called remote-host3 within the reverse-proxy XML block.

    domain.xml reverse-proxy config

    <subsystem xmlns="urn:jboss:domain:undertow:12.0">
      ...
      <handlers>
          <reverse-proxy name="lb-handler">
             <host name="host1" outbound-socket-binding="remote-host1" scheme="ajp" path="/" instance-id="myroute1"/>
             <host name="host2" outbound-socket-binding="remote-host2" scheme="ajp" path="/" instance-id="myroute2"/>
             <host name="remote-host3" outbound-socket-binding="remote-host3" scheme="ajp" path="/" instance-id="myroute3"/>
          </reverse-proxy>
      </handlers>
      ...
    </subsystem>

    The output-socket-binding is a logical name pointing to a socket-binding configured later in the domain.xml file. The instance-id attribute must also be unique to the new host as this value is used by a cookie to enable sticky sessions when load balancing.

  3. Go down to the load-balancer-sockets socket-binding-group and add the outbound-socket-binding for remote-host3.

    This new binding needs to point to the host and port of the new host.

    domain.xml outbound-socket-binding

    <socket-binding-group name="load-balancer-sockets" default-interface="public">
        ...
        <outbound-socket-binding name="remote-host1">
            <remote-destination host="localhost" port="8159"/>
        </outbound-socket-binding>
        <outbound-socket-binding name="remote-host2">
            <remote-destination host="localhost" port="8259"/>
        </outbound-socket-binding>
        <outbound-socket-binding name="remote-host3">
            <remote-destination host="192.168.0.5" port="8259"/>
        </outbound-socket-binding>
    </socket-binding-group>

9.3.4.1. Master bind addresses

Next thing you’ll have to do is to change the public and management bind addresses for the master host. Either edit the domain.xml file as discussed in the Bind Addresses chapter or specify these bind addresses on the command line as follows:

$ domain.sh --host-config=host-master.xml -Djboss.bind.address=192.168.0.2 -Djboss.bind.address.management=192.168.0.2

9.3.4.2. Host slave bind addresses

Next you’ll have to change the public, management, and domain controller bind addresses (jboss.domain.master-address). Either edit the host-slave.xml file or specify them on the command line as follows:

$ domain.sh --host-config=host-slave.xml
     -Djboss.bind.address=192.168.0.5
      -Djboss.bind.address.management=192.168.0.5
       -Djboss.domain.master.address=192.168.0.2

The values of jboss.bind.address and jboss.bind.address.management pertain to the host slave’s IP address. The value of jboss.domain.master.address needs to be the IP address of the domain controller, which is the management address of the master host.

Additional resources

  • See the load balancing section in the JBoss EAP Configuration Guide for information how to use other software-based load balancers.

9.4. Sticky sessions

Typical cluster deployment consists of the load balancer (reverse proxy) and 2 or more Red Hat Single Sign-On servers on private network. For performance purposes, it may be useful if load balancer forwards all requests related to particular browser session to the same Red Hat Single Sign-On backend node.

The reason is, that Red Hat Single Sign-On is using Infinispan distributed cache under the covers for save data related to current authentication session and user session. The Infinispan distributed caches are configured with one owner by default. That means that particular session is saved just on one cluster node and the other nodes need to lookup the session remotely if they want to access it.

For example if authentication session with ID 123 is saved in the Infinispan cache on node1, and then node2 needs to lookup this session, it needs to send the request to node1 over the network to return the particular session entity.

It is beneficial if particular session entity is always available locally, which can be done with the help of sticky sessions. The workflow in the cluster environment with the public frontend load balancer and two backend Red Hat Single Sign-On nodes can be like this:

  • User sends initial request to see the Red Hat Single Sign-On login screen
  • This request is served by the frontend load balancer, which forwards it to some random node (eg. node1). Strictly said, the node doesn’t need to be random, but can be chosen according to some other criterias (client IP address etc). It all depends on the implementation and configuration of underlying load balancer (reverse proxy).
  • Red Hat Single Sign-On creates authentication session with random ID (eg. 123) and saves it to the Infinispan cache.
  • Infinispan distributed cache assigns the primary owner of the session based on the hash of session ID. See Infinispan documentation for more details around this. Let’s assume that Infinispan assigned node2 to be the owner of this session.
  • Red Hat Single Sign-On creates the cookie AUTH_SESSION_ID with the format like <session-id>.<owner-node-id> . In our example case, it will be 123.node2 .
  • Response is returned to the user with the Red Hat Single Sign-On login screen and the AUTH_SESSION_ID cookie in the browser

From this point, it is beneficial if load balancer forwards all the next requests to the node2 as this is the node, who is owner of the authentication session with ID 123 and hence Infinispan can lookup this session locally. After authentication is finished, the authentication session is converted to user session, which will be also saved on node2 because it has same ID 123 .

The sticky session is not mandatory for the cluster setup, however it is good for performance for the reasons mentioned above. You need to configure your loadbalancer to sticky over the AUTH_SESSION_ID cookie. How exactly do this is dependent on your loadbalancer.

It is recommended on the Red Hat Single Sign-On side to use the system property jboss.node.name during startup, with the value corresponding to the name of your route. For example, -Djboss.node.name=node1 will use node1 to identify the route. This route will be used by Infinispan caches and will be attached to the AUTH_SESSION_ID cookie when the node is the owner of the particular key. Here is an example of the start up command using this system property:

cd $RHSSO_NODE1
./standalone.sh -c standalone-ha.xml -Djboss.socket.binding.port-offset=100 -Djboss.node.name=node1

Typically in production environment the route name should use the same name as your backend host, but it is not required. You can use a different route name. For example, if you want to hide the host name of your Red Hat Single Sign-On server inside your private network.

9.4.1. Disable adding the route

Some load balancers can be configured to add the route information by themselves instead of relying on the back end Red Hat Single Sign-On node. However, as described above, adding the route by the Red Hat Single Sign-On is recommended. This is because when done this way performance improves, since Red Hat Single Sign-On is aware of the entity that is the owner of particular session and can route to that node, which is not necessarily the local node.

You are permitted to disable adding route information to the AUTH_SESSION_ID cookie by Red Hat Single Sign-On, if you prefer, by adding the following into your RHSSO_HOME/standalone/configuration/standalone-ha.xml file in the Red Hat Single Sign-On subsystem configuration:

<subsystem xmlns="urn:jboss:domain:keycloak-server:1.1">
  ...
    <spi name="stickySessionEncoder">
        <provider name="infinispan" enabled="true">
            <properties>
                <property name="shouldAttachRoute" value="false"/>
            </properties>
        </provider>
    </spi>

</subsystem>

9.5. Setting up multicast networking

The default clustering support needs IP Multicast. Multicast is a network broadcast protocol. This protocol is used at boot time to discover and join the cluster. It is also used to broadcast messages for the replication and invalidation of distributed caches used by Red Hat Single Sign-On.

The clustering subsystem for Red Hat Single Sign-On runs on the JGroups stack. Out of the box, the bind addresses for clustering are bound to a private network interface with 127.0.0.1 as default IP address.

Procedure

  1. Edit your the standalone-ha.xml or domain.xml sections discussed in the Bind Address chapter.

    private network config

        <interfaces>
            ...
            <interface name="private">
                <inet-address value="${jboss.bind.address.private:127.0.0.1}"/>
            </interface>
        </interfaces>
        <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
            ...
            <socket-binding name="jgroups-mping" interface="private" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
            <socket-binding name="jgroups-tcp" interface="private" port="7600"/>
            <socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
            <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
            <socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
            <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
            ...
        </socket-binding-group>

  2. Configure the jboss.bind.address.private and jboss.default.multicast.address as well as the ports of the services on the clustering stack.

    Note

    It is possible to cluster Red Hat Single Sign-On without IP Multicast, but this topic is beyond the scope of this guide. For more information, see JGroups in the JBoss EAP Configuration Guide.

9.6. Secure cluster communication

When cluster nodes are isolated on a private network it requires access to the private network to be able to join a cluster or to view communication in the cluster. In addition you can also enable authentication and encryption for cluster communication. As long as your private network is secure it is not necessary to enable authentication and encryption. Red Hat Single Sign-On does not send very sensitive information on the cluster in either case.

If you want to enable authentication and encryption for clustering communication, see Securing a Cluster in the JBoss EAP Configuration Guide.

9.7. Serialized cluster startup

Red Hat Single Sign-On cluster nodes are allowed to boot concurrently. When Red Hat Single Sign-On server instance boots up it may do some database migration, importing, or first time initializations. A DB lock is used to prevent start actions from conflicting with one another when cluster nodes boot up concurrently.

By default, the maximum timeout for this lock is 900 seconds. If a node is waiting on this lock for more than the timeout it will fail to boot. Typically you won’t need to increase/decrease the default value, but just in case it’s possible to configure it in standalone.xml, standalone-ha.xml, or domain.xml file in your distribution. The location of this file depends on your operating mode.

<spi name="dblock">
    <provider name="jpa" enabled="true">
        <properties>
            <property name="lockWaitTimeout" value="900"/>
        </properties>
    </provider>
</spi>

9.8. Booting the cluster

Booting Red Hat Single Sign-On in a cluster depends on your operating mode

Standalone Mode

$ bin/standalone.sh --server-config=standalone-ha.xml

Domain Mode

$ bin/domain.sh --host-config=host-master.xml
$ bin/domain.sh --host-config=host-slave.xml

You may need to use additional parameters or system properties. For example, the parameter -b for the binding host or the system property jboss.node.name to specify the name of the route, as described in Sticky Sessions section.

9.9. Troubleshooting

  • Note that when you run a cluster, you should see message similar to this in the log of both cluster nodes:

    INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-10,shared=udp)
    ISPN000094: Received new cluster view: [node1/keycloak|1] (2) [node1/keycloak, node2/keycloak]

    If you see just one node mentioned, it’s possible that your cluster hosts are not joined together.

    Usually it’s best practice to have your cluster nodes on private network without firewall for communication among them. Firewall could be enabled just on public access point to your network instead. If for some reason you still need to have firewall enabled on cluster nodes, you will need to open some ports. Default values are UDP port 55200 and multicast port 45688 with multicast address 230.0.0.4. Note that you may need more ports opened if you want to enable additional features like diagnostics for your JGroups stack. Red Hat Single Sign-On delegates most of the clustering work to Infinispan/JGroups. For more information, see JGroups in the JBoss EAP Configuration Guide.

  • If you are interested in failover support (high availability), evictions, expiration and cache tuning, see Chapter 10, Server cache configuration.

Chapter 10. Server cache configuration

Red Hat Single Sign-On has two types of caches. One type of cache sits in front of the database to decrease load on the DB and to decrease overall response times by keeping data in memory. Realm, client, role, and user metadata is kept in this type of cache. This cache is a local cache. Local caches do not use replication even if you are in the cluster with more Red Hat Single Sign-On servers. Instead, they only keep copies locally and if the entry is updated an invalidation message is sent to the rest of the cluster and the entry is evicted. There is separate replicated cache work, which task is to send the invalidation messages to the whole cluster about what entries should be evicted from local caches. This greatly reduces network traffic, makes things efficient, and avoids transmitting sensitive metadata over the wire.

The second type of cache handles managing user sessions, offline tokens, and keeping track of login failures so that the server can detect password phishing and other attacks. The data held in these caches is temporary, in memory only, but is possibly replicated across the cluster.

This chapter discusses some configuration options for these caches for both clustered and non-clustered deployments.

Note

More advanced configuration of these caches can be found in the Infinispan section of the JBoss EAP Configuration Guide.

10.1. Eviction and expiration

There are multiple different caches configured for Red Hat Single Sign-On. There is a realm cache that holds information about secured applications, general security data, and configuration options. There is also a user cache that contains user metadata. Both caches default to a maximum of 10000 entries and use a least recently used eviction strategy. Each of them is also tied to an object revisions cache that controls eviction in a clustered setup. This cache is created implicitly and has twice the configured size. The same applies for the authorization cache, which holds the authorization data. The keys cache holds data about external keys and does not need to have dedicated revisions cache. Rather it has expiration explicitly declared on it, so the keys are periodically expired and forced to be periodically downloaded from external clients or identity providers.

The eviction policy and max entries for these caches can be configured in the standalone.xml, standalone-ha.xml, or domain.xml depending on your operating mode. In the configuration file, there is the part with infinispan subsystem, which looks similar to this:

<subsystem xmlns="urn:jboss:domain:infinispan:12.0">
    <cache-container name="keycloak">
        <local-cache name="realms">
            <object-memory size="10000"/>
        </local-cache>
        <local-cache name="users">
            <object-memory size="10000"/>
        </local-cache>
        ...
        <local-cache name="keys">
            <object-memory size="1000"/>
            <expiration max-idle="3600000"/>
        </local-cache>
        ...
    </cache-container>

To limit or expand the number of allowed entries simply add or edit the object element or the expiration element of particular cache configuration.

In addition, there are also separate caches sessions, clientSessions, offlineSessions, offlineClientSessions, loginFailures and actionTokens. These caches are distributed in cluster environment and they are unbounded in size by default. If they are bounded, it would then be possible that some sessions will be lost. Expired sessions are cleared internally by Red Hat Single Sign-On itself to avoid growing the size of these caches without limit. If you see memory issues due to a large number of sessions, you can try to:

  • Increase the size of cluster (more nodes in cluster means that sessions are spread more equally among nodes)
  • Increase the memory for Red Hat Single Sign-On server process
  • Decrease the number of owners to ensure that caches are saved in one single place. See Section 10.2, “Replication and failover” for more details
  • Disable l1-lifespan for distributed caches. See Infinispan documentation for more details
  • Decrease session timeouts, which could be done individually for each realm in Red Hat Single Sign-On admin console. But this could affect usability for end users. See Timeouts for more details.

There is an additional replicated cache, work, which is mostly used to send messages among cluster nodes; it is also unbounded by default. However, this cache should not cause any memory issues as entries in this cache are very short-lived.

10.2. Replication and failover

There are caches like sessions, authenticationSessions, offlineSessions, loginFailures and a few others (See Section 10.1, “Eviction and expiration” for more details), which are configured as distributed caches when using a clustered setup. Entries are not replicated to every single node, but instead one or more nodes is chosen as an owner of that data. If a node is not the owner of a specific cache entry it queries the cluster to obtain it. What this means for failover is that if all the nodes that own a piece of data go down, that data is lost forever. By default, Red Hat Single Sign-On only specifies one owner for data. So if that one node goes down that data is lost. This usually means that users will be logged out and will have to login again.

You can change the number of nodes that replicate a piece of data by change the owners attribute in the distributed-cache declaration.

owners

<subsystem xmlns="urn:jboss:domain:infinispan:12.0">
   <cache-container name="keycloak">
       <distributed-cache name="sessions" owners="2"/>
...

Here we’ve changed it so at least two nodes will replicate one specific user login session.

Tip

The number of owners recommended is really dependent on your deployment. If you do not care if users are logged out when a node goes down, then one owner is good enough and you will avoid replication.

Tip

It is generally wise to configure your environment to use loadbalancer with sticky sessions. It is beneficial for performance as Red Hat Single Sign-On server, where the particular request is served, will be usually the owner of the data from the distributed cache and will therefore be able to look up the data locally. See Section 9.4, “Sticky sessions” for more details.

10.3. Disabling caching

You can disable the realm or user cache.

Procedure

  1. Edit the standalone.xml, standalone-ha.xml, or domain.xml file in your distribution.

    The location of this file depends on your operating mode. Here is a sample config file.

        <spi name="userCache">
            <provider name="default" enabled="true"/>
        </spi>
    
        <spi name="realmCache">
            <provider name="default" enabled="true"/>
        </spi>
  2. Set the enabled attribute to false for the cache you want to disable.
  3. Reboot your server for this change to take effect.

10.4. Clearing cache at runtime

You can clear the realm cache, user cache, or the external public keys.

Procedure

  1. Log into the Admin Console.
  2. Click Realm Settings.
  3. Click the Cache tab.
  4. Clear the realm cache, the user cache or cache of external public keys.
Note

The cache will be cleared for all realms!

Chapter 11. Red Hat Single Sign-On Operator

The Red Hat Single Sign-On Operator automates Red Hat Single Sign-On administration in Openshift. You use this Operator to create custom resources (CRs), which automate administrative tasks. For example, instead of creating a client or a user in the Red Hat Single Sign-On admin console, you can create custom resources to perform those tasks. A custom resource is a YAML file that defines the parameters for the administrative task.

You can create custom resources to perform the following tasks:

Note

After you create custom resources for realms, clients, and users, you can manage them by using the Red Hat Single Sign-On admin console or as custom resources using the oc command. However, you cannot use both methods, because the Operator performs a one way sync for custom resources that you modify. For example, if you modify a realm custom resource, the changes show up in the admin console. However, if you modify the realm using the admin console, those changes have no effect on the custom resource.

Begin using the Operator by Installing the Red Hat Single Sign-On Operator on a cluster.

11.1. Installing the Red Hat Single Sign-On Operator on a cluster

To install the Red Hat Single Sign-On Operator, you can use:

11.1.1. Installing using the Operator Lifecycle Manager

Prerequisites

  • You have cluster-admin permission or an equivalent level of permissions granted by an administrator.

Procedure

Perform this procedure on an OpenShift cluster.

  1. Open the OpenShift Container Platform web console.
  2. In the left column, click Operators, OperatorHub.
  3. Search for Red Hat Single Sign-On Operator.

    OperatorHub tab in OpenShift

    operator openshift operatorhub

  4. Click the Red Hat Single Sign-On Operator icon.

    An Install page opens.

    Operator Install page on OpenShift

    operator olm installation

  5. Click Install.
  6. Select a namespace and click Subscribe.

    Namespace selection in OpenShift

    installed namespace

    The Operator starts installing.

Additional resources

11.1.2. Installing from the command line

You can install the Red Hat Single Sign-On Operator from the command line.

Prerequisites

  • You have cluster-admin permission or an equivalent level of permissions granted by an administrator.

Procedure

  1. Create a project.

    $ oc new-project <namespace>
  2. Create a file named rhsso-operator-olm.yaml to define a YAML group and a subscription operator.

    Update the targetNamespaces to the namespace for RH-SSO.

    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: rhsso-operator-group
    spec:
      targetNamespaces:
      -  <namespace> # change this to the namespace you will use for RH-SSO
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: rhsso-operator
    spec:
      channel: stable
      installPlanApproval: Manual
      name: rhsso-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
      # Here you can specify a specific Operator version, otherwise it will use the latest
      # startingCSV: rhsso-operator.7.6.0-opr-001
  3. Install the operator group and subscription.

    $ oc apply -f rhsso-operator-olm.yaml
  4. Approve the install plan and fill in the appropriate namespace.

    oc patch installplan $(oc get ip -n <namespace>  -o=jsonpath='{.items[?(@.spec.approved==false)].metadata.name}') -n <namespace> --type merge --patch '{"spec":{"approved":true}}'
  5. Verify the operator is running.

    $ oc get pods
    NAME                              READY   STATUS    RESTARTS   AGE
    rhsso-operator-558876f75c-m25mt   1/1     Running   0          28s

Additional resources

11.2. Using the Red Hat Single Sign-On Operator in production environment

  • The usage of embedded DB is not supported in a production environment.
  • Backup CRD is deprecated and not supported in a production environment.
  • The podDisruptionBudget field in the Keycloak CR is deprecated and will be ignored when the Operator is deployed on Kubernetes version 1.25 and higher.
  • We fully support using the rest of the CRDs in production, despite the v1alpha1 version. We do not plan to make any breaking changes in this CRDs version.

11.3. Installing Red Hat Single Sign-On using a custom resource

You can use the Operator to automate the installation of Red Hat Single Sign-On by creating a Keycloak custom resource. When you use a custom resource to install Red Hat Single Sign-On, you create the components and services that are described here and illustrated in the graphic that follows.

  • keycloak-db-secret - Stores properties such as the database username, password, and external address (if you connect to an external database)
  • credentials-<CR-Name> - Admin username and password to log into the Red Hat Single Sign-On admin console (the <CR-Name> is based on the Keycloak custom resource name)
  • keycloak - Keycloak deployment specification that is implemented as a StatefulSet with high availability support
  • keycloak-postgresql - Starts a PostgreSQL database installation
  • keycloak-discovery Service - Performs JDBC_PING discovery
  • keycloak Service - Connects to Red Hat Single Sign-On through HTTPS (HTTP is not supported)
  • keycloak-postgresql Service - Connects an internal and external, if used, database instance
  • keycloak Route - The URL for accessing the Red Hat Single Sign-On admin console from OpenShift

How Operator components and services interact

operator components

11.3.1. The Keycloak custom resource

The Keycloak custom resource is a YAML file that defines the parameters for installation. This file contains three properties.

  • instances - controls the number of instances running in high availability mode.
  • externalAccess - if the enabled is True, the Operator creates a route for OpenShift for the Red Hat Single Sign-On cluster. You can set host to override the automatically chosen host name for Route
  • externalDatabase - in order to connect to an externally hosted database. That topic is covered in the external database section of this guide. Setting it to false should be used only for testing purposes and will install an embedded PostgreSQL database. Be aware that externalDatabase:false is NOT supported in production environments.

Example YAML file for a Keycloak custom resource

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  name: example-sso
  labels:
    app: sso
spec:
  instances: 1
  externalAccess:
    enabled: True

Note

You can update the YAML file and the changes appear in the Red Hat Single Sign-On admin console, however changes to the admin console do not update the custom resource.

11.3.2. Creating a Keycloak custom resource on OpenShift

On OpenShift, you use the custom resource to create a route, which is the URL of the admin console, and find the secret, which holds the username and password for the admin console.

Prerequisites

  • You have a YAML file for this custom resource.
  • You have cluster-admin permission or an equivalent level of permissions granted by an administrator.

Procedure

  1. Create a route using your YAML file: oc create -f <filename>.yaml -n <namespace>. For example:

    $ oc create -f sso.yaml -n sso
    keycloak.keycloak.org/example-sso created

    A route is created in OpenShift.

  2. Log into the OpenShift web console.
  3. Select Networking, Routes and search for Keycloak.

    Routes screen in OpenShift web console

    route ocp

  4. On the screen with the Keycloak route, click the URL under Location.

    The Red Hat Single Sign-On admin console login screen appears.

    Admin console login screen

    login empty

  5. Locate the username and password for the admin console in the OpenShift web console; under Workloads, click Secrets and search for Keycloak.

    Secrets screen in OpenShift web console

    secrets ocp

  6. Enter the username and password into the admin console login screen.

    Admin console login screen

    login complete

    You are now logged into an instance of Red Hat Single Sign-On that was installed by a Keycloak custom resource. You are ready to create custom resources for realms, clients, and users.

    Red Hat Single Sign-On master realm

    new install cr

  7. Check the status of the custom resource:

    $ oc describe keycloak <CR-name>

Results

After the Operator processes the custom resource, view the status with this command:

$ oc describe keycloak <CR-name>

Keycloak custom resource Status

Name:         example-keycloak
Namespace:    keycloak
Labels:       app=sso
Annotations:  <none>
API Version:  keycloak.org/v1alpha1
Kind:         Keycloak
Spec:
  External Access:
    Enabled:  true
  Instances:  1
Status:
  Credential Secret:  credential-example-keycloak
  Internal URL:       https://<External URL to the deployed instance>
  Message:
  Phase:              reconciling
  Ready:              true
  Secondary Resources:
    Deployment:
      keycloak-postgresql
    Persistent Volume Claim:
      keycloak-postgresql-claim
    Prometheus Rule:
      keycloak
    Route:
      keycloak
    Secret:
      credential-example-keycloak
      keycloak-db-secret
    Service:
      keycloak-postgresql
      keycloak
      keycloak-discovery
    Service Monitor:
      keycloak
    Stateful Set:
      keycloak
  Version:
Events:

Additional resources

  • Once the installation of Red Hat Single Sign-On completes, you are ready to create a realm custom resource.
  • An external database is the supported option and needs to be enabled in the Keycloak custom resource. You can disable this option only for testing and enable it when you switch to a production environment. See Connecting to an external database.

11.4. Creating a realm custom resource

You can use the Operator to create realms in Red Hat Single Sign-On as defined by a custom resource. You define the properties of the realm custom resource in a YAML file.

Note

You can only create or delete realms by creating or deleting the YAML file, and changes appear in the Red Hat Single Sign-On admin console. However changes to the admin console are not reflected back and updates of the CR after the realm is created are not supported.

Example YAML file for a Realm custom resource

apiVersion: keycloak.org/v1alpha1
kind: KeycloakRealm
metadata:
  name: test
  labels:
    app: sso
spec:
  realm:
    id: "basic"
    realm: "basic"
    enabled: True
    displayName: "Basic Realm"
  instanceSelector:
    matchLabels:
      app: sso

Prerequisites

  • You have a YAML file for this custom resource.
  • In the YAML file, the app under instanceSelector matches the label of a Keycloak custom resource. Matching these values ensures that you create the realm in the right instance of Red Hat Single Sign-On.
  • You have cluster-admin permission or an equivalent level of permissions granted by an administrator.

Procedure

  1. Use this command on the YAML file that you created: oc create -f <realm-name>.yaml. For example:

    $ oc create -f initial_realm.yaml
    keycloak.keycloak.org/test created
  2. Log into the admin console for the related instance of Red Hat Single Sign-On.
  3. Click Select Realm and locate the realm that you created.

    The new realm opens.

    Admin console master realm

    test realm cr

Results

After the Operator processes the custom resource, view the status with this command:

$ oc describe keycloak <CR-name>

Realm custom resource status

Name:         example-keycloakrealm
Namespace:    keycloak
Labels:       app=sso
Annotations:  <none>
API Version:  keycloak.org/v1alpha1
Kind:         KeycloakRealm
Metadata:
  Creation Timestamp:  2019-12-03T09:46:02Z
  Finalizers:
    realm.cleanup
  Generation:        1
  Resource Version:  804596
  Self Link:         /apis/keycloak.org/v1alpha1/namespaces/keycloak/keycloakrealms/example-keycloakrealm
  UID:               b7b2f883-15b1-11ea-91e6-02cb885627a6
Spec:
  Instance Selector:
    Match Labels:
      App: sso
  Realm:
    Display Name:  Basic Realm
    Enabled:       true
    Id:            basic
    Realm:         basic
Status:
  Login URL:
  Message:
  Phase:      reconciling
  Ready:      true
Events:       <none>

Additional resources

11.5. Creating a client custom resource

You can use the Operator to create clients in Red Hat Single Sign-On as defined by a custom resource. You define the properties of the realm in a YAML file.

Note

You can update the YAML file and changes appear in the Red Hat Single Sign-On admin console, however changes to the admin console do not update the custom resource.

Example YAML file for a Client custom resource

apiVersion: keycloak.org/v1alpha1
kind: KeycloakClient
metadata:
  name: example-client
  labels:
    app: sso
spec:
  realmSelector:
     matchLabels:
      app: <matching labels for KeycloakRealm custom resource>
  client:
    # auto-generated if not supplied
    #id: 123
    clientId: client-secret
    secret: client-secret
    # ...
    # other properties of Keycloak Client

Prerequisites

  • You have a YAML file for this custom resource.
  • You have cluster-admin permission or an equivalent level of permissions granted by an administrator.

Procedure

  1. Use this command on the YAML file that you created: oc create -f <client-name>.yaml. For example:

    $ oc create -f initial_client.yaml
    keycloak.keycloak.org/example-client created
  2. Log into the Red Hat Single Sign-On admin console for the related instance of Red Hat Single Sign-On.
  3. Click Clients.

    The new client appears in the list of clients.

    clients

Results

After a client is created, the Operator creates a Secret containing the Client ID and the client’s secret using the following naming pattern: keycloak-client-secret-<custom resource name>. For example:

Client’s Secret

apiVersion: v1
data:
  CLIENT_ID: <base64 encoded Client ID>
  CLIENT_SECRET: <base64 encoded Client Secret>
kind: Secret

After the Operator processes the custom resource, view the status with this command:

$ oc describe keycloak <CR-name>

Client custom resource Status

Name:         client-secret
Namespace:    keycloak
Labels:       app=sso
API Version:  keycloak.org/v1alpha1
Kind:         KeycloakClient
Spec:
  Client:
    Client Authenticator Type:     client-secret
    Client Id:                     client-secret
    Id:                            keycloak-client-secret
  Realm Selector:
    Match Labels:
      App:  sso
Status:
  Message:
  Phase:    reconciling
  Ready:    true
  Secondary Resources:
    Secret:
      keycloak-client-secret-client-secret
Events:  <none>

Additional resources

11.6. Creating a user custom resource

You can use the Operator to create users in Red Hat Single Sign-On as defined by a custom resource. You define the properties of the user custom resource in a YAML file.

Note

You can update properties, except for the password, in the YAML file and changes appear in the Red Hat Single Sign-On admin console, however changes to the admin console do not update the custom resource.

Example YAML file for a user custom resource

apiVersion: keycloak.org/v1alpha1
kind: KeycloakUser
metadata:
  name: example-user
spec:
  user:
    username: "realm_user"
    firstName: "John"
    lastName: "Doe"
    email: "user@example.com"
    enabled: True
    emailVerified: False
    credentials:
      - type: "password"
        value: "12345"
    realmRoles:
      - "offline_access"
    clientRoles:
      account:
        - "manage-account"
      realm-management:
        - "manage-users"
  realmSelector:
    matchLabels:
      app: sso

Prerequisites

  • You have a YAML file for this custom resource.
  • The realmSelector matches the labels of an existing realm custom resource.
  • You have cluster-admin permission or an equivalent level of permissions granted by an administrator.

Procedure

  1. Use this command on the YAML file that you created: oc create -f <user_cr>.yaml. For example:

    $ oc create -f initial_user.yaml
    keycloak.keycloak.org/example-user created
  2. Log into the admin console for the related instance of Red Hat Single Sign-On.
  3. Click Users.
  4. Search for the user that you defined in the YAML file.

    You may need to switch to a different realm to find the user.

    realm user

Results

After a user is created, the Operator creates a Secret using the following naming pattern: credential-<realm name>-<username>-<namespace>, containing the username and, if it has been specified in the CR credentials attribute, the password.

Here’s an example:

KeycloakUser Secret

kind: Secret
apiVersion: v1
data:
  password: <base64 encoded password>
  username: <base64 encoded username>
type: Opaque

Once the Operator processes the custom resource, view the status with this command:

$ oc describe keycloak <CR-name>

User custom resource Status

Name:         example-realm-user
Namespace:    keycloak
Labels:       app=sso
API Version:  keycloak.org/v1alpha1
Kind:         KeycloakUser
Spec:
  Realm Selector:
    Match Labels:
      App: sso
  User:
    Email:           realm_user@redhat.com
    Credentials:
      Type:          password
      Value:         <user password>
    Email Verified:  false
    Enabled:         true
    First Name:      John
    Last Name:       Doe
    Username:        realm_user
Status:
  Message:
  Phase:    reconciled
Events:     <none>

Additional resources

11.7. Connecting to an external database

You can use the Operator to connect to an external PostgreSQL database by creating a keycloak-db-secret YAML file and setting Keycloak CR externalDatabase property to enabled. Note that values are Base64 encoded.

Example YAML file for keycloak-db-secret

apiVersion: v1
kind: Secret
metadata:
    name: keycloak-db-secret
    namespace: keycloak
stringData:
    POSTGRES_DATABASE: <Database Name>
    POSTGRES_EXTERNAL_ADDRESS: <External Database IP or URL (resolvable by K8s)>
    POSTGRES_EXTERNAL_PORT: <External Database Port>
    POSTGRES_PASSWORD: <Database Password>
    # Required for AWS Backup functionality
    POSTGRES_SUPERUSER: "true"
    POSTGRES_USERNAME: <Database Username>
    SSLMODE: <TLS configuration for the Database connection>
type: Opaque

The following properties set the hostname or IP address and port of the database.

  • POSTGRES_EXTERNAL_ADDRESS - an IP address or a hostname of the external database.
  • POSTGRES_EXTERNAL_PORT - (Optional) A database port.

The other properties work in the same way for a hosted or external database. Set them as follows:

  • POSTGRES_DATABASE - Database name to be used.
  • POSTGRES_USERNAME - Database username
  • POSTGRES_PASSWORD - Database password
  • POSTGRES_SUPERUSER - Indicates whether backups should run as super user. Typically true.
  • SSL_MODE - Indicates whether to use TLS on the connection to the external PostgreSQL database. Check the possible values

When SSL_MODE is enabled, the operator searches for a secret called keycloak-db-ssl-cert-secret containing the root.crt that has been used by the PostgreSQL database. Creating the secret is optional and the secret is used only when you want to verify the Database’s certificate (for example SSLMODE: verify-ca). Here is an example :

Example YAML file for TLS Secret to be used by the operator.

apiVersion: v1
kind: Secret
metadata:
  name: keycloak-db-ssl-cert-secret
  namespace: keycloak
type: Opaque
data:
  root.crt: {root.crt base64}

The Operator will create a Service named keycloak-postgresql. This Service is configured by the Operator to expose the external database based on the content of POSTGRES_EXTERNAL_ADDRESS. Red Hat Single Sign-On uses this Service to connect to the Database, which means it does not connect to the Database directly but rather through this Service.

The Keycloak custom resource requires updates to enable external database support.

Example YAML file for Keycloak custom resource that supports an external database

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  labels:
      app: sso
  name: example-keycloak
  namespace: keycloak
spec:
  externalDatabase:
    enabled: true
  instances: 1

Prerequisites

  • You have a YAML file for keycloak-db-secret.
  • You have modified the Keycloak custom resource to set externalDatabase to true.
  • You have cluster-admin permission or an equivalent level of permissions granted by an administrator.

Procedure

  1. Locate the secret for your PostgreSQL database: oc get secret <secret_for_db> -o yaml. For example:

    $ oc get secret keycloak-db-secret -o yaml
    apiVersion: v1
    data
      POSTGRES_DATABASE: cm9vdA==
      POSTGRES_EXTERNAL_ADDRESS: MTcyLjE3LjAuMw==
      POSTGRES_EXTERNAL_PORT: NTQzMg==

    The POSTGRES_EXTERNAL_ADDRESS is in Base64 format.

  2. Decode the value for the secret: echo "<encoded_secret>" | base64 -decode. For example:

    $ echo "MTcyLjE3LjAuMw==" | base64 -decode
    192.0.2.3
  3. Confirm that the decoded value matches the IP address for your database:

    $ oc get pods -o wide
    NAME                        READY  STATUS    RESTARTS   AGE   IP
    keycloak-0                  1/1    Running   0          13m   192.0.2.0
    keycloak-postgresql-c8vv27m 1/1    Running   0          24m   192.0.2.3
  4. Confirm that keycloak-postgresql appears in a list of running services:

    $ oc get svc
    NAME                 TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
    keycloak             ClusterIP  203.0.113.0    <none>       8443/TCP  27m
    keycloak-discovery   ClusterIP  None           <none>       8080/TCP  27m
    keycloak-postgresql  ClusterIP  203.0.113.1    <none>       5432/TCP  27m

    The keycloak-postgresql service sends requests to a set of IP addresses in the backend. These IP addresses are called endpoints.

  5. View the endpoints used by the keycloak-postgresql service to confirm that they use the IP addresses for your database:

    $ oc get endpoints keycloak-postgresql
    NAME                  ENDPOINTS         AGE
    keycloak-postgresql   192.0.2.3.5432    27m
  6. Confirm that Red Hat Single Sign-On is running with the external database. This example shows that everything is running:

    $ oc get pods
    NAME                        READY  STATUS    RESTARTS   AGE   IP
    keycloak-0                  1/1    Running   0          26m   192.0.2.0
    keycloak-postgresql-c8vv27m 1/1    Running   0          36m   192.0.2.3

11.8. Connecting to an external Red Hat Single Sign-On

This operator can also be used to partially manage an external Red Hat Single Sign-On instance. In it’s current state, it will only be able to create clients.

To do this, you’ll need to create unmanaged versions of the Keycloak and KeycloakRealm CRDs to use for targeting and configuration.

Example YAML file for external-keycloak

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  name: external-ref
  labels:
    app: external-sso
spec:
  unmanaged: true
  external:
    enabled: true
    url: https://some.external.url

In order to authenticate against this keycloak, the operator infers the secret name from the CRD by prefixing the CRD name with credential-.

Example YAML file for credential-external-ref

apiVersion: v1
kind: Secret
metadata:
  name: credential-external-ref
type: Opaque
data:
  ADMIN_USERNAME: YWRtaW4=
  ADMIN_PASSWORD: cGFzcw==

Example YAML file for external-realm

apiVersion: keycloak.org/v1alpha1
kind: KeycloakRealm
metadata:
  name: external-realm
  labels:
    app: external-sso
spec:
  unmanaged: true
  realm:
    id: "basic"
    realm: "basic"
  instanceSelector:
    matchLabels:
      app: external-sso

You can now use the realm reference in your client as usual, and it will create the client on the external Red Hat Single Sign-On instance.

11.9. Scheduling database backups

Warning

Backup CR is deprecated and could be removed in future releases.

You can use the Operator to schedule automatic backups of the database as defined by custom resources. The custom resource triggers a backup job and reports back its status.

You can use Operator to create a backup job that performs a one-time backup to a local Persistent Volume.

Example YAML file for a Backup custom resource

apiVersion: keycloak.org/v1alpha1
kind: KeycloakBackup
metadata:
  name: test-backup

Prerequisites

  • You have a YAML file for this custom resource.
  • You have a PersistentVolume with a claimRef to reserve it only for a PersistentVolumeClaim created by the Red Hat Single Sign-On Operator.

Procedure

  1. Create a backup job: oc create -f <backup_crname>. For example:

    $ oc create -f one-time-backup.yaml
    keycloak.keycloak.org/test-backup

    The Operator creates a PersistentVolumeClaim with the following naming scheme: Keycloak-backup-<CR-name>.

  2. View a list of volumes:

    $ oc get pvc
    NAME                          STATUS   VOLUME
    keycloak-backup-test-backup   Bound    pvc-e242-ew022d5-093q-3134n-41-adff
    keycloak-postresql-claim      Bound    pvc-e242-vs29202-9bcd7-093q-31-zadj
  3. View a list of backup jobs:

    $ oc get jobs
    NAME           COMPLETIONS     DURATION     AGE
    test-backup    0/1             6s           6s
  4. View the list of executed backup jobs:

    $ oc get pods
    NAME                               READY    STATUS       RESTARTS    AGE
    test-backup-5b4rf                  0/1      Completed    0           24s
    keycloak-0                         1/1      Running      0           52m
    keycloak-postgresql-c824c6-vv27m   1/1      Running      0           71m
  5. View the log of your completed backup job:

    $ oc logs test-backup-5b4rf
    ==> Component data dump completed
    .
    .
    .
    .

Additional resources

11.10. Installing extensions and themes

You can use the operator to install extensions and themes that you need for your company or organization. The extension or theme can be anything that Red Hat Single Sign-On can consume. For example, you can add a metrics extension. You add the extension or theme to the Keycloak custom resource.

Example YAML file for a Keycloak custom resource

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  name: example-keycloak
  labels:
   app: sso
spec:
  instances: 1
  extensions:
   - <url_for_extension_or_theme>
  externalAccess:
    enabled: True

You can package and deploy themes in the same way as any other extensions. See Deploying Themes manual entry for more information.

Prerequisites

  • You have a YAML file for the Keycloak custom resource.
  • You have cluster-admin permission or an equivalent level of permissions granted by an administrator.

Procedure

  1. Edit the YAML file for the Keycloak custom resource: oc edit <CR-name>
  2. Add a line called extensions: after the instances line.
  3. Add a URL to a JAR file for your custom extension or theme.
  4. Save the file.

The Operator downloads the extension or theme and installs it.

11.11. Command options for managing custom resources

After you create a custom request, you can edit it or delete using the oc command.

  • To edit a custom request, use this command: oc edit <CR-name>
  • To delete a custom request, use this command: oc delete <CR-name>

For example, to edit a realm custom request named test-realm, use this command:

$ oc edit test-realm

A window opens where you can make changes.

Note

You can update the Keycloak CR YAML file and changes will be applied to the deployment.

Updates to the other resources are limited:

Keycloak Realm CR only supports basic creation and deletion without sync options. Keycloak User and Client CRs support unidirectional updates (changes to the CR are reflected in Keycloak but changes done in Keycloak are not updated in the CR).

11.12. Upgrade strategy

You can configure how the operator performs Red Hat Single Sign-On upgrades. You can choose from the following upgrade strategies.

  • recreate: This is the default strategy. The operator removes all Red Hat Single Sign-On replicas, optionally creates a backup and then creates the replicas based on a newer Red Hat Single Sign-On image. This strategy is suitable for major upgrades as a single Red Hat Single Sign-On version is accessing the underlying database. The downside is Red Hat Single Sign-On needs to be shut down during the upgrade.
  • rolling: The operator removes one replica at a time and creates it again based on a newer Red Hat Single Sign-On image. This ensures a zero-downtime upgrade but is more suitable for minor version upgrades that do not require database migration since the database is accessed by multiple Red Hat Single Sign-On versions concurrently. Automatic backups are not supported with this strategy.

Example YAML file for a Keycloak custom resource

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  name: example-keycloak
  labels:
   app: sso
spec:
  instances: 2
  migration:
    strategy: recreate
    backups:
      enabled: True
  externalAccess:
    enabled: True

Note

Due to a bug introduced in a previous version of the Operator, Selector field on the Red Hat Single Sign-On StatefulSet might be misconfigured depending on your configuration. If such state is detected by the Operator and you are using the recreate strategy, it will delete and recreate the StatefulSet with the correct Selector field. This is required as the Selector field is immutable.

As a "delete" operation can have potentially dangerous side effects in very rare cases, for example when you have added custom functionality unknown to the Operator to the StatefulSet definition, you can instead delete the StatefulSet manually.

Legal Notice

Copyright © 2022 Red Hat, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.