Red Hat Single Sign-On is built on top of the JBoss EAP application server and its sub-projects like Infinispan (for caching) and Hibernate (for persistence). This guide only covers basics for infrastructure-level configuration. It is highly recommended that you peruse the documentation for JBoss EAP and its sub projects. Here is the link to the documentation:

These prerequisites exist for installing the Red Hat Single Sign-On server:

The Red Hat Single Sign-On server download ZIP file contains the scripts and binaries to run the Red Hat Single Sign-On server. You install the 7.6 server first, then the 7.6.7 server patch.

You must subscribe to both the JBoss EAP 7.4 and RH-SSO 7.6 repositories before you can install RH-SSO from an RPM.

subscription-manager repos --enable=jb-eap-7.4-for-rhel-<RHEL_VERSION>-server-rpms --enable=rhel-<RHEL_VERSION>-server-rpms

subscription-manager repos --enable=jb-eap-7.4-for-rhel-8-x86_64-rpms --enable=rhel-8-for-x86_64-baseos-rpms --enable=rhel-8-for-x86_64-appstream-rpms

The following are some important directories in the server distribution.

Standalone operating mode is only useful when you want to run one, and only one Red Hat Single Sign-On server instance. It is not usable for clustered deployments and all caches are non-distributed and local-only. It is not recommended that you use standalone mode in production as you will have a single point of failure. If your standalone mode server goes down, users will not be able to log in. This mode is really only useful to test drive and play with the features of Red Hat Single Sign-On

$ .../bin/standalone.sh

> ...\bin\standalone.bat

Standalone clustered operation mode applies when you want to run Red Hat Single Sign-On within a cluster. This mode requires that you have a copy of the Red Hat Single Sign-On distribution on each machine where you want to run a server instance. This mode can be very easy to deploy initially, but can become quite cumbersome. To make a configuration change, you modify each distribution on each machine. For a large cluster, this mode can become time consuming and error prone.

$ .../bin/standalone.sh --server-config=standalone-ha.xml

> ...\bin\standalone.bat --server-config=standalone-ha.xml

Domain mode is a way to centrally manage and publish the configuration for your servers.

Running a cluster in standard mode can quickly become aggravating as the cluster grows in size. Every time you need to make a configuration change, you perform it on each node in the cluster. Domain mode solves this problem by providing a central place to store and publish configurations. It can be quite complex to set up, but it is worth it in the end. This capability is built into the JBoss EAP Application Server which Red Hat Single Sign-On derives from.

Here are some of the basic concepts of running in domain mode.

In domain mode, a domain controller is started on a master node. The configuration for the cluster resides in the domain controller. Next a host controller is started on each machine in the cluster. Each host controller deployment configuration specifies how many Red Hat Single Sign-On server instances will be started on that machine. When the host controller boots up, it starts as many Red Hat Single Sign-On server instances as it was configured to do. These server instances pull their configuration from the domain controller.

    <profiles>
        <profile name="auth-server-standalone">
            ...
        </profile>
        <profile name="auth-server-clustered">
            ...
        </profile>

    <socket-binding-groups>
        <socket-binding-group name="standard-sockets" default-interface="public">
           ...
        </socket-binding-group>
        <socket-binding-group name="ha-sockets" default-interface="public">
           ...
        </socket-binding-group>
        <!-- load-balancer-sockets should be removed in production systems and replaced with a better software or hardware based one -->
        <socket-binding-group name="load-balancer-sockets" default-interface="public">
           ...
        </socket-binding-group>
    </socket-binding-groups>

$ domain.sh -Djboss.http.port=80

    <server-groups>
        <!-- load-balancer-group should be removed in production systems and replaced with a better software or hardware based one -->
        <server-group name="load-balancer-group" profile="load-balancer">
            <jvm name="default">
                <heap size="64m" max-size="512m"/>
            </jvm>
            <socket-binding-group ref="load-balancer-sockets"/>
        </server-group>
        <server-group name="auth-server-group" profile="auth-server-clustered">
            <jvm name="default">
                <heap size="64m" max-size="512m"/>
            </jvm>
            <socket-binding-group ref="ha-sockets"/>
        </server-group>
    </server-groups>

    <servers>
        <!-- remove or comment out next line -->
        <server name="load-balancer" group="loadbalancer-group"/>
        ...
    </servers>

    <servers>
        ...
        <server name="server-one" group="auth-server-group" auto-start="true">
             <socket-bindings port-offset="150"/>
        </server>
    </servers>

$ .../bin/domain.sh --host-config=host-master.xml

> ...\bin\domain.bat --host-config=host-master.xml

Cross-site replication, which was introduced as a Technology Preview feature in Red Hat Single Sign-On 7.2, is no longer available as a supported feature in any Red Hat SSO 7.x release including the latest RH-SSO 7.6 release. Red Hat does not recommend any customer implement or use this feature in their environment because it is not supported. Also, support exceptions for this feature are no longer considered or accepted.

A new solution for cross-site replication is being discussed and tentatively considered for a future release of Red Hat build of Keycloak (RHBK), which is the product that will be introduced instead of Red Hat SSO 8. More details will be available soon.

The specifics of each configuration setting is discussed elsewhere in context with that setting. However, it is useful to understand the format used to declare settings on SPI providers.

Red Hat Single Sign-On is a highly modular system that allows great flexibility. There are more than 50 service provider interfaces (SPIs), and you are allowed to swap out implementations of each SPI. An implementation of an SPI is known as a provider.

All elements in an SPI declaration are optional, but a full SPI declaration looks like this:

<spi name="myspi">
    <default-provider>myprovider</default-provider>
    <provider name="myprovider" enabled="true">
        <properties>
            <property name="foo" value="bar"/>
        </properties>
    </provider>
    <provider name="mysecondprovider" enabled="true">
        <properties>
            <property name="foo" value="foo"/>
        </properties>
    </provider>
</spi>

Here we have two providers defined for the SPI myspi. The default-provider is listed as myprovider. However it is up to the SPI to decide how it will treat this setting. Some SPIs allow more than one provider and some do not. So default-provider can help the SPI to choose.

Also notice that each provider defines its own set of configuration properties. The fact that both providers above have a property called foo is just a coincidence.

The type of each property value is interpreted by the provider. However, there is one exception. Consider the jpa provider for the eventsStore SPI:

<spi name="eventsStore">
    <provider name="jpa" enabled="true">
        <properties>
            <property name="exclude-events" value="[&quot;EVENT1&quot;,
                                                    &quot;EVENT2&quot;]"/>
        </properties>
    </provider>
</spi>

We see that the value begins and ends with square brackets. That means that the value will be passed to the provider as a list. In this example, the system will pass the provider a list with two element values EVENT1 and EVENT2. To add more values to the list, just separate each list element with a comma. Unfortunately, you do need to escape the quotes surrounding each list element with ".

Follow the steps in Server Developer Guide for more details on custom providers and the configuration of providers.

Besides editing the configuration by hand, you also have the option of changing the configuration by issuing commands via the jboss-cli tool. CLI allows you to configure servers locally or remotely. And it is especially useful when combined with scripting.

To start the JBoss EAP CLI, you need to run jboss-cli.

$ .../bin/jboss-cli.sh

> ...\bin\jboss-cli.bat

This will bring you to a prompt like this:

[disconnected /]

If you wish to execute commands on a running server, you will first execute the connect command.

[disconnected /] connect
connect
[standalone@localhost:9990 /]

You may be thinking to yourself, "I didn’t enter in any username or password!". If you run jboss-cli on the same machine as your running standalone server or domain controller and your account has appropriate file permissions, you do not have to setup or enter in an admin username and password. See the JBoss EAP Configuration Guide for more details on how to make things more secure if you are uncomfortable with that setup.

If you do happen to be on the same machine as your standalone server and you want to issue commands while the server is not active, you can embed the server into CLI and make changes in a special mode that disallows incoming requests. To do this, first execute the embed-server command with the config file you wish to change.

[disconnected /] embed-server --server-config=standalone.xml
[standalone@embedded /]

The CLI can also run in GUI mode. GUI mode launches a Swing application that allows you to graphically view and edit the entire management model of a running server. GUI mode is especially useful when you need help formatting your CLI commands and learning about the options available. The GUI can also retrieve server logs from a local or remote server.

The CLI has extensive scripting capabilities. A script is just a text file with CLI commands in it. Consider a simple script that turns off theme and template caching.

/subsystem=keycloak-server/theme=defaults/:write-attribute(name=cacheThemes,value=false)
/subsystem=keycloak-server/theme=defaults/:write-attribute(name=cacheTemplates,value=false)

To execute the script, you can follow the Scripts menu in CLI GUI, or execute the script from the command line as follows:

$ .../bin/jboss-cli.sh --file=turn-off-caching.cli

Here are some configuration tasks and how to perform them with CLI commands. Note that in all but the first example, we use the wildcard path ** to mean you should substitute or the path to the keycloak-server subsystem.

For standalone, this just means:

** = /subsystem=keycloak-server

For domain mode, this would mean something like:

** = /profile=auth-server-clustered/subsystem=keycloak-server

/subsystem=keycloak-server/:write-attribute(name=web-context,value=myContext)

**/theme=defaults/:write-attribute(name=default,value=myTheme)

**/spi=mySPI/:add
**/spi=mySPI/provider=myProvider/:add(enabled=true)

**/spi=mySPI/provider=myProvider/:write-attribute(name=enabled,value=false)

**/spi=mySPI/:write-attribute(name=default-provider,value=myProvider)

**/spi=dblock/:add(default-provider=jpa)
**/spi=dblock/provider=jpa/:add(properties={lockWaitTimeout => "900"},enabled=true)

**/spi=dblock/provider=jpa/:map-put(name=properties,key=lockWaitTimeout,value=3)

**/spi=dblock/provider=jpa/:map-remove(name=properties,key=lockRecheckTime)

**/spi=eventsStore/provider=jpa/:map-put(name=properties,key=exclude-events,value=[EVENT1,EVENT2])

Following are the steps you perform to get an RDBMS configured for Red Hat Single Sign-On.

This chapter will use PostgresSQL for all its examples. Other databases follow the same steps for installation.

Find and download the JDBC driver JAR for your RDBMS. Before you can use this driver, you must package it up into a module and install it into the server. Modules define JARs that are loaded into the Red Hat Single Sign-On classpath and the dependencies those JARs have on other modules.

You declare your JDBC into your deployment profile so that it loads and becomes available when the server boots up.

You modify the existing datasource configuration that Red Hat Single Sign-On uses to connect it to your new external database. You’ll do this within the same configuration file and XML block that you registered your JDBC driver in. Here’s an example that sets up the connection to your new database:

  <subsystem xmlns="urn:jboss:domain:datasources:6.0">
     <datasources>
       ...
       <datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true">
           <connection-url>jdbc:postgresql://localhost/keycloak</connection-url>
           <driver>postgresql</driver>
           <pool>
               <max-pool-size>20</max-pool-size>
           </pool>
           <security>
               <user-name>William</user-name>
               <password>password</password>
           </security>
       </datasource>
        ...
     </datasources>
  </subsystem>

The configuration for this component is found in the standalone.xml, standalone-ha.xml, or domain.xml file in your distribution. The location of this file depends on your operating mode.

<subsystem xmlns="urn:jboss:domain:keycloak-server:1.1">
    ...
    <spi name="connectionsJpa">
     <provider name="default" enabled="true">
         <properties>
             <property name="dataSource" value="java:jboss/datasources/KeycloakDS"/>
             <property name="initializeEmpty" value="false"/>
             <property name="migrationStrategy" value="manual"/>
             <property name="migrationExport" value="${jboss.home.dir}/keycloak-database-update.sql"/>
         </properties>
     </provider>
    </spi>
    ...
</subsystem>

Possible configuration options are:

Database schema in Red Hat Single Sign-On only accounts for Unicode strings in the following special fields:

Otherwise, characters are limited to those contained in database encoding which is often 8-bit. However, for some database systems, it is possible to enable UTF-8 encoding of Unicode characters and use full Unicode character set in all text fields. Often, this is counterbalanced by shorter maximum length of the strings than in case of 8-bit encodings.

Some of the databases require special settings to database and/or JDBC driver to be able to handle Unicode characters. Please find the settings for your database below. Note that if a database is listed here, it can still work properly provided it handles UTF-8 encoding properly both on the level of database and JDBC driver.

Technically, the key criterion for Unicode support for all fields is whether the database allows setting of Unicode character set for VARCHAR and CHAR fields. If yes, there is a high chance that Unicode will be plausible, usually at the expense of field length. If it only supports Unicode in NVARCHAR and NCHAR fields, Unicode support for all text fields is unlikely as Keycloak schema uses VARCHAR and CHAR fields extensively.

show server_encoding;

create database keycloak with encoding 'UTF8';

The default hostname provider uses the configured frontendUrl as the base URL for frontend requests (requests from user-agents) and uses the request URL as the basis for backend requests (direct requests from clients).

Frontend request do not have to have the same context-path as the Keycloak server. This means you can expose Keycloak on for example https://auth.example.org or https://example.org/keycloak while internally its URL could be https://10.0.0.10:8080/auth.

This makes it possible to have user-agents (browsers) send requests to Red Hat Single Sign-On through the public domain name, while internal clients can use an internal domain name or IP address.

This is reflected in the OpenID Connect Discovery endpoint for example where the authorization_endpoint uses the frontend URL, while token_endpoint uses the backend URL. As a note here a public client for instance would contact Keycloak through the public endpoint, which would result in the base of authorization_endpoint and token_endpoint being the same.

To set the frontendUrl for Keycloak you can either pass add -Dkeycloak.frontendUrl=https://auth.example.org to the startup or you can configure it in standalone.xml. See the example below:

<spi name="hostname">
    <default-provider>default</default-provider>
    <provider name="default" enabled="true">
        <properties>
            <property name="frontendUrl" value="https://auth.example.com"/>
            <property name="forceBackendUrlToFrontendUrl" value="false"/>
        </properties>
    </provider>
</spi>

To update the frontendUrl with jboss-cli use the following command:

/subsystem=keycloak-server/spi=hostname/provider=default:write-attribute(name=properties.frontendUrl,value="https://auth.example.com")

If you want all requests to go through the public domain name you can force backend requests to use the frontend URL as well by setting forceBackendUrlToFrontendUrl to true.

It is also possible to override the default frontend URL for individual realms. This can be done in the admin console.

If you do not want to expose the admin endpoints and console on the public domain use the property adminUrl to set a fixed URL for the admin console, which is different to the frontendUrl. It is also required to block access to /auth/admin externally, for details on how to do that refer to the Server Administration Guide.

To develop a custom hostname provider you need to implement org.keycloak.urls.HostnameProviderFactory and org.keycloak.urls.HostnameProvider.

Follow the instructions in the Service Provider Interfaces section in Server Developer Guide for more information on how to develop a custom provider.

By default Red Hat Single Sign-On binds to the localhost loopback address 127.0.0.1. That’s not a very useful default if you want the authentication server available on your network. Generally, what we recommend is that you deploy a reverse proxy or load balancer on a public network and route traffic to individual Red Hat Single Sign-On server instances on a private network. In either case though, you still need to set up your network interfaces to bind to something other than localhost.

Setting the bind address is quite easy and can be done on the command line with either the standalone.sh or domain.sh boot scripts discussed in the Choosing an Operating Mode chapter.

$ standalone.sh -b 192.168.0.5

The -b switch sets the IP bind address for any public interfaces.

Alternatively, if you don’t want to set the bind address at the command line, you can edit the profile configuration of your deployment. Open up the profile configuration file (standalone.xml or domain.xml depending on your operating mode) and look for the interfaces XML block.

    <interfaces>
        <interface name="management">
            <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
        </interface>
        <interface name="public">
            <inet-address value="${jboss.bind.address:127.0.0.1}"/>
        </interface>
    </interfaces>

The public interface corresponds to subsystems creating sockets that are available publicly. An example of one of these subsystems is the web layer which serves up the authentication endpoints of Red Hat Single Sign-On. The management interface corresponds to sockets opened up by the management layer of the JBoss EAP. Specifically the sockets which allow you to use the jboss-cli.sh command line interface and the JBoss EAP web console.

In looking at the public interface you see that it has a special string ${jboss.bind.address:127.0.0.1}. This string denotes a value 127.0.0.1 that can be overridden on the command line by setting a Java system property, i.e.:

$ domain.sh -Djboss.bind.address=192.168.0.5

The -b is just a shorthand notation for this command. So, you can either change the bind address value directly in the profile config, or change it on the command line when you boot up.

The ports opened for each socket have a pre-defined default that can be overridden at the command line or within configuration. To illustrate this configuration, let’s pretend you are running in standalone mode and open up the …​/standalone/configuration/standalone.xml. Search for socket-binding-group.

    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
        <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
        <socket-binding name="http" port="${jboss.http.port:8080}"/>
        <socket-binding name="https" port="${jboss.https.port:8443}"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="localhost" port="25"/>
        </outbound-socket-binding>
    </socket-binding-group>

socket-bindings define socket connections that will be opened by the server. These bindings specify the interface (bind address) they use as well as what port number they will open. The ones you will be most interested in are:

When running in domain mode setting the socket configurations is a bit trickier as the example domain.xml file has multiple socket-binding-groups defined. If you scroll down to the server-group definitions you can see what socket-binding-group is used for each server-group.

    <server-groups>
        <server-group name="load-balancer-group" profile="load-balancer">
            ...
            <socket-binding-group ref="load-balancer-sockets"/>
        </server-group>
        <server-group name="auth-server-group" profile="auth-server-clustered">
            ...
            <socket-binding-group ref="ha-sockets"/>
        </server-group>
    </server-groups>

This default behavior is defined by the SSL/HTTPS mode of each Red Hat Single Sign-On realm. This is discussed in more detail in the Server Administration Guide, but let’s give some context and a brief overview of these modes.

The SSL mode for each realm can be configured in the Red Hat Single Sign-On admin console.

If you are not using a reverse proxy or load balancer to handle HTTPS traffic for you, you’ll need to enable HTTPS for the Red Hat Single Sign-On server. This involves

$ keytool -genkey -alias localhost -keyalg RSA -keystore keycloak.jks -validity 10950
    Enter keystore password: secret
    Re-enter new password: secret
    What is your first and last name?
    [Unknown]:  localhost
    What is the name of your organizational unit?
    [Unknown]:  Keycloak
    What is the name of your organization?
    [Unknown]:  Red Hat
    What is the name of your City or Locality?
    [Unknown]:  Westford
    What is the name of your State or Province?
    [Unknown]:  MA
    What is the two-letter country code for this unit?
    [Unknown]:  US
    Is CN=localhost, OU=Keycloak, O=Test, L=Westford, ST=MA, C=US correct?
    [no]:  yes

The Red Hat Single Sign-On server often needs to make non-browser HTTP requests to the applications and services it secures. The auth server manages these outgoing connections by maintaining an HTTP client connection pool. There are some things you’ll need to configure in standalone.xml, standalone-ha.xml, or domain.xml. The location of this file depends on your operating mode.

<spi name="connectionsHttpClient">
    <provider name="default" enabled="true">
        <properties>
            <property name="connection-pool-size" value="256"/>
        </properties>
    </provider>
</spi>

Possible configuration options are:

.*\.(google|googleapis)\.com;http://www-proxy.acme.com:8080

.*\.(google|googleapis)\.com;http://user01:pas2w0rd@www-proxy.acme.com:8080

# All requests to Google APIs should use http://www-proxy.acme.com:8080 as proxy
.*\.(google|googleapis)\.com;http://www-proxy.acme.com:8080

# All requests to internal systems should use no proxy
.*\.acme\.com;NO_PROXY

# All other requests should use http://fallback:8080 as proxy
.*;http://fallback:8080

echo SETUP: Configure proxy routes for HttpClient SPI

# In case there is no connectionsHttpClient definition yet
/subsystem=keycloak-server/spi=connectionsHttpClient/provider=default:add(enabled=true)

# Configure the proxy-mappings
/subsystem=keycloak-server/spi=connectionsHttpClient/provider=default:write-attribute(name=properties.proxy-mappings,value=[".*\\.(google|googleapis)\\.com;http://www-proxy.acme.com:8080",".*\\.acme\\.com;NO_PROXY",".*;http://fallback:8080"])

<spi name="connectionsHttpClient">
    <provider name="default" enabled="true">
        <properties>
            <property
            name="proxy-mappings"
            value="[&quot;.*\\.(google|googleapis)\\.com;http://www-proxy.acme.com:8080&quot;,&quot;.*\\.acme\\.com;NO_PROXY&quot;,&quot;.*;http://fallback:8080&quot;]"/>
        </properties>
    </provider>
</spi>

HTTPS_PROXY=https://www-proxy.acme.com:8080
NO_PROXY=google.com,login.facebook.com

<spi name="connectionsHttpClient">
    <provider name="default" enabled="true">
        <properties>
            <property name="proxy-mappings" value=".*;NO_PROXY"/>
        </properties>
    </provider>
</spi>

$ keytool -import -alias HOSTDOMAIN -keystore truststore.jks -file host-certificate.cer

<spi name="truststore">
    <provider name="file" enabled="true">
        <properties>
            <property name="file" value="path to your .jks file containing public certificates"/>
            <property name="password" value="password"/>
            <property name="hostname-verification-policy" value="WILDCARD"/>
        </properties>
    </provider>
</spi>

The recommended network architecture for deploying Red Hat Single Sign-On is to set up an HTTP/HTTPS load balancer on a public IP address that routes requests to Red Hat Single Sign-On servers sitting on a private network. This isolates all clustering connections and provides a nice means of protecting the servers.

Red Hat Single Sign-On does come with an out of the box clustering demo that leverages domain mode. Review the Clustered Domain Example chapter for more details.

This section discusses a number of things you need to configure before you can put a reverse proxy or load balancer in front of your clustered Red Hat Single Sign-On deployment. It also covers configuring the built-in load balancer that was Clustered Domain Example.

The following diagram illustrates the use of a load balancer. In this example, the load balancer serves as a reverse proxy between three clients and a cluster of three Red Hat Single Sign-On servers.

<subsystem xmlns="urn:jboss:domain:undertow:12.0">
   <buffer-cache name="default"/>
   <server name="default-server">
      <ajp-listener name="ajp" socket-binding="ajp"/>
      <http-listener name="default" socket-binding="http" redirect-socket="https"
          proxy-address-forwarding="true"/>
      ...
   </server>
   ...
</subsystem>

<subsystem xmlns="urn:jboss:domain:undertow:12.0">
     <buffer-cache name="default"/>
     <server name="default-server">
         <ajp-listener name="ajp" socket-binding="ajp"/>
         <http-listener name="default" socket-binding="http" redirect-socket="https"/>
         <host name="default-host" alias="localhost">
             ...
             <filter-ref name="proxy-peer"/>
         </host>
     </server>
        ...
     <filters>
         ...
         <filter name="proxy-peer"
                 class-name="io.undertow.server.handlers.ProxyPeerAddressHandler"
                 module="io.undertow.core" />
     </filters>
 </subsystem>

<subsystem xmlns="urn:jboss:domain:undertow:12.0">
    ...
    <http-listener name="default" socket-binding="http"
        proxy-address-forwarding="true" redirect-socket="proxy-https"/>
    ...
</subsystem>

$ domain.sh --host-config=host-master.xml -Djboss.bind.address=192.168.0.2 -Djboss.bind.address.management=192.168.0.2

$ domain.sh --host-config=host-slave.xml
     -Djboss.bind.address=192.168.0.5
      -Djboss.bind.address.management=192.168.0.5
       -Djboss.domain.master.address=192.168.0.2

Typical cluster deployment consists of the load balancer (reverse proxy) and 2 or more Red Hat Single Sign-On servers on private network. For performance purposes, it may be useful if load balancer forwards all requests related to particular browser session to the same Red Hat Single Sign-On backend node.

The reason is, that Red Hat Single Sign-On is using Infinispan distributed cache under the covers for save data related to current authentication session and user session. The Infinispan distributed caches are configured with one owner by default. That means that particular session is saved just on one cluster node and the other nodes need to lookup the session remotely if they want to access it.

For example if authentication session with ID 123 is saved in the Infinispan cache on node1, and then node2 needs to lookup this session, it needs to send the request to node1 over the network to return the particular session entity.

It is beneficial if particular session entity is always available locally, which can be done with the help of sticky sessions. The workflow in the cluster environment with the public frontend load balancer and two backend Red Hat Single Sign-On nodes can be like this:

From this point, it is beneficial if load balancer forwards all the next requests to the node2 as this is the node, who is owner of the authentication session with ID 123 and hence Infinispan can lookup this session locally. After authentication is finished, the authentication session is converted to user session, which will be also saved on node2 because it has same ID 123 .

The sticky session is not mandatory for the cluster setup, however it is good for performance for the reasons mentioned above. You need to configure your loadbalancer to sticky over the AUTH_SESSION_ID cookie. How exactly do this is dependent on your loadbalancer.

It is recommended on the Red Hat Single Sign-On side to use the system property jboss.node.name during startup, with the value corresponding to the name of your route. For example, -Djboss.node.name=node1 will use node1 to identify the route. This route will be used by Infinispan caches and will be attached to the AUTH_SESSION_ID cookie when the node is the owner of the particular key. Here is an example of the start up command using this system property:

cd $RHSSO_NODE1
./standalone.sh -c standalone-ha.xml -Djboss.socket.binding.port-offset=100 -Djboss.node.name=node1

Typically in production environment the route name should use the same name as your backend host, but it is not required. You can use a different route name. For example, if you want to hide the host name of your Red Hat Single Sign-On server inside your private network.

<subsystem xmlns="urn:jboss:domain:keycloak-server:1.1">
  ...
    <spi name="stickySessionEncoder">
        <provider name="infinispan" enabled="true">
            <properties>
                <property name="shouldAttachRoute" value="false"/>
            </properties>
        </provider>
    </spi>

</subsystem>

The default clustering support needs IP Multicast. Multicast is a network broadcast protocol. This protocol is used at boot time to discover and join the cluster. It is also used to broadcast messages for the replication and invalidation of distributed caches used by Red Hat Single Sign-On.

The clustering subsystem for Red Hat Single Sign-On runs on the JGroups stack. Out of the box, the bind addresses for clustering are bound to a private network interface with 127.0.0.1 as default IP address.

When cluster nodes are isolated on a private network it requires access to the private network to be able to join a cluster or to view communication in the cluster. In addition you can also enable authentication and encryption for cluster communication. As long as your private network is secure it is not necessary to enable authentication and encryption. Red Hat Single Sign-On does not send very sensitive information on the cluster in either case.

If you want to enable authentication and encryption for clustering communication, see Securing a Cluster in the JBoss EAP Configuration Guide.

Red Hat Single Sign-On cluster nodes are allowed to boot concurrently. When Red Hat Single Sign-On server instance boots up it may do some database migration, importing, or first time initializations. A DB lock is used to prevent start actions from conflicting with one another when cluster nodes boot up concurrently.

By default, the maximum timeout for this lock is 900 seconds. If a node is waiting on this lock for more than the timeout it will fail to boot. Typically you won’t need to increase/decrease the default value, but just in case it’s possible to configure it in standalone.xml, standalone-ha.xml, or domain.xml file in your distribution. The location of this file depends on your operating mode.

<spi name="dblock">
    <provider name="jpa" enabled="true">
        <properties>
            <property name="lockWaitTimeout" value="900"/>
        </properties>
    </provider>
</spi>

Booting Red Hat Single Sign-On in a cluster depends on your operating mode

$ bin/standalone.sh --server-config=standalone-ha.xml

$ bin/domain.sh --host-config=host-master.xml
$ bin/domain.sh --host-config=host-slave.xml

You may need to use additional parameters or system properties. For example, the parameter -b for the binding host or the system property jboss.node.name to specify the name of the route, as described in Sticky Sessions section.

There are multiple different caches configured for Red Hat Single Sign-On. There is a realm cache that holds information about secured applications, general security data, and configuration options. There is also a user cache that contains user metadata. Both caches default to a maximum of 10000 entries and use a least recently used eviction strategy. Each of them is also tied to an object revisions cache that controls eviction in a clustered setup. This cache is created implicitly and has twice the configured size. The same applies for the authorization cache, which holds the authorization data. The keys cache holds data about external keys and does not need to have dedicated revisions cache. Rather it has expiration explicitly declared on it, so the keys are periodically expired and forced to be periodically downloaded from external clients or identity providers.

The eviction policy and max entries for these caches can be configured in the standalone.xml, standalone-ha.xml, or domain.xml depending on your operating mode. In the configuration file, there is the part with infinispan subsystem, which looks similar to this:

<subsystem xmlns="urn:jboss:domain:infinispan:12.0">
    <cache-container name="keycloak">
        <local-cache name="realms">
            <object-memory size="10000"/>
        </local-cache>
        <local-cache name="users">
            <object-memory size="10000"/>
        </local-cache>
        ...
        <local-cache name="keys">
            <object-memory size="1000"/>
            <expiration max-idle="3600000"/>
        </local-cache>
        ...
    </cache-container>

To limit or expand the number of allowed entries simply add or edit the object element or the expiration element of particular cache configuration.

In addition, there are also separate caches sessions, clientSessions, offlineSessions, offlineClientSessions, loginFailures and actionTokens. These caches are distributed in cluster environment and they are unbounded in size by default. If they are bounded, it would then be possible that some sessions will be lost. Expired sessions are cleared internally by Red Hat Single Sign-On itself to avoid growing the size of these caches without limit. If you see memory issues due to a large number of sessions, you can try to:

There is an additional replicated cache, work, which is mostly used to send messages among cluster nodes; it is also unbounded by default. However, this cache should not cause any memory issues as entries in this cache are very short-lived.

There are caches like sessions, authenticationSessions, offlineSessions, loginFailures and a few others (See Section 10.1, “Eviction and expiration” for more details), which are configured as distributed caches when using a clustered setup. Entries are not replicated to every single node, but instead one or more nodes is chosen as an owner of that data. If a node is not the owner of a specific cache entry it queries the cluster to obtain it. What this means for failover is that if all the nodes that own a piece of data go down, that data is lost forever. By default, Red Hat Single Sign-On only specifies one owner for data. So if that one node goes down that data is lost. This usually means that users will be logged out and will have to login again.

You can change the number of nodes that replicate a piece of data by change the owners attribute in the distributed-cache declaration.

<subsystem xmlns="urn:jboss:domain:infinispan:12.0">
   <cache-container name="keycloak">
       <distributed-cache name="sessions" owners="2"/>
...

Here we’ve changed it so at least two nodes will replicate one specific user login session.

You can disable the realm or user cache.

You can clear the realm cache, user cache, or the external public keys.

To install the Red Hat Single Sign-On Operator, you can use:

You can use the Operator to automate the installation of Red Hat Single Sign-On by creating a Keycloak custom resource. When you use a custom resource to install Red Hat Single Sign-On, you create the components and services that are described here and illustrated in the graphic that follows.

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  name: example-sso
  labels:
    app: sso
spec:
  instances: 1
  externalAccess:
    enabled: True

$ oc describe keycloak <CR-name>

Name:         example-keycloak
Namespace:    keycloak
Labels:       app=sso
Annotations:  <none>
API Version:  keycloak.org/v1alpha1
Kind:         Keycloak
Spec:
  External Access:
    Enabled:  true
  Instances:  1
Status:
  Credential Secret:  credential-example-keycloak
  Internal URL:       https://<External URL to the deployed instance>
  Message:
  Phase:              reconciling
  Ready:              true
  Secondary Resources:
    Deployment:
      keycloak-postgresql
    Persistent Volume Claim:
      keycloak-postgresql-claim
    Prometheus Rule:
      keycloak
    Route:
      keycloak
    Secret:
      credential-example-keycloak
      keycloak-db-secret
    Service:
      keycloak-postgresql
      keycloak
      keycloak-discovery
    Service Monitor:
      keycloak
    Stateful Set:
      keycloak
  Version:
Events:

You can use the Operator to create realms in Red Hat Single Sign-On as defined by a custom resource. You define the properties of the realm custom resource in a YAML file.

apiVersion: keycloak.org/v1alpha1
kind: KeycloakRealm
metadata:
  name: test
  labels:
    app: sso
spec:
  realm:
    id: "basic"
    realm: "basic"
    enabled: True
    displayName: "Basic Realm"
  instanceSelector:
    matchLabels:
      app: sso

$ oc describe keycloak <CR-name>

Name:         example-keycloakrealm
Namespace:    keycloak
Labels:       app=sso
Annotations:  <none>
API Version:  keycloak.org/v1alpha1
Kind:         KeycloakRealm
Metadata:
  Creation Timestamp:  2019-12-03T09:46:02Z
  Finalizers:
    realm.cleanup
  Generation:        1
  Resource Version:  804596
  Self Link:         /apis/keycloak.org/v1alpha1/namespaces/keycloak/keycloakrealms/example-keycloakrealm
  UID:               b7b2f883-15b1-11ea-91e6-02cb885627a6
Spec:
  Instance Selector:
    Match Labels:
      App: sso
  Realm:
    Display Name:  Basic Realm
    Enabled:       true
    Id:            basic
    Realm:         basic
Status:
  Login URL:
  Message:
  Phase:      reconciling
  Ready:      true
Events:       <none>

Additional resources

You can use the Operator to create clients in Red Hat Single Sign-On as defined by a custom resource. You define the properties of the realm in a YAML file.

apiVersion: keycloak.org/v1alpha1
kind: KeycloakClient
metadata:
  name: example-client
  labels:
    app: sso
spec:
  realmSelector:
     matchLabels:
      app: <matching labels for KeycloakRealm custom resource>
  client:
    # auto-generated if not supplied
    #id: 123
    clientId: client-secret
    secret: client-secret
    # ...
    # other properties of Keycloak Client

apiVersion: v1
data:
  CLIENT_ID: <base64 encoded Client ID>
  CLIENT_SECRET: <base64 encoded Client Secret>
kind: Secret

After the Operator processes the custom resource, view the status with this command:

$ oc describe keycloak <CR-name>

Name:         client-secret
Namespace:    keycloak
Labels:       app=sso
API Version:  keycloak.org/v1alpha1
Kind:         KeycloakClient
Spec:
  Client:
    Client Authenticator Type:     client-secret
    Client Id:                     client-secret
    Id:                            keycloak-client-secret
  Realm Selector:
    Match Labels:
      App:  sso
Status:
  Message:
  Phase:    reconciling
  Ready:    true
  Secondary Resources:
    Secret:
      keycloak-client-secret-client-secret
Events:  <none>

You can use the Operator to create users in Red Hat Single Sign-On as defined by a custom resource. You define the properties of the user custom resource in a YAML file.

apiVersion: keycloak.org/v1alpha1
kind: KeycloakUser
metadata:
  name: example-user
spec:
  user:
    username: "realm_user"
    firstName: "John"
    lastName: "Doe"
    email: "user@example.com"
    enabled: True
    emailVerified: False
    credentials:
      - type: "password"
        value: "12345"
    realmRoles:
      - "offline_access"
    clientRoles:
      account:
        - "manage-account"
      realm-management:
        - "manage-users"
  realmSelector:
    matchLabels:
      app: sso

Here’s an example:

kind: Secret
apiVersion: v1
data:
  password: <base64 encoded password>
  username: <base64 encoded username>
type: Opaque

Once the Operator processes the custom resource, view the status with this command:

$ oc describe keycloak <CR-name>

Name:         example-realm-user
Namespace:    keycloak
Labels:       app=sso
API Version:  keycloak.org/v1alpha1
Kind:         KeycloakUser
Spec:
  Realm Selector:
    Match Labels:
      App: sso
  User:
    Email:           realm_user@redhat.com
    Credentials:
      Type:          password
      Value:         <user password>
    Email Verified:  false
    Enabled:         true
    First Name:      John
    Last Name:       Doe
    Username:        realm_user
Status:
  Message:
  Phase:    reconciled
Events:     <none>

You can use the Operator to connect to an external PostgreSQL database by creating a keycloak-db-secret YAML file and setting Keycloak CR externalDatabase property to enabled. Note that values are Base64 encoded.

apiVersion: v1
kind: Secret
metadata:
    name: keycloak-db-secret
    namespace: keycloak
stringData:
    POSTGRES_DATABASE: <Database Name>
    POSTGRES_EXTERNAL_ADDRESS: <External Database URL (resolvable by K8s)>
    POSTGRES_EXTERNAL_PORT: <External Database Port>
    POSTGRES_PASSWORD: <Database Password>
    # Required for AWS Backup functionality
    POSTGRES_SUPERUSER: "true"
    POSTGRES_USERNAME: <Database Username>
    SSLMODE: <TLS configuration for the Database connection>
type: Opaque

The following properties set the hostname or IP address and port of the database.

The other properties work in the same way for a hosted or external database. Set them as follows:

When SSLMODE is enabled, the operator searches for a secret called keycloak-db-ssl-cert-secret containing the root.crt that has been used by the PostgreSQL database. Creating the secret is optional and the secret is used only when you want to verify the Database’s certificate (for example SSLMODE: verify-ca). Here is an example :

apiVersion: v1
kind: Secret
metadata:
  name: keycloak-db-ssl-cert-secret
  namespace: keycloak
type: Opaque
data:
  root.crt: {root.crt base64}

The Operator will create a Service named keycloak-postgresql. This Service is configured by the Operator to expose the external database based on the content of POSTGRES_EXTERNAL_ADDRESS. Red Hat Single Sign-On uses this Service to connect to the Database, which means it does not connect to the Database directly but rather through this Service.

The Keycloak custom resource requires updates to enable external database support.

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  labels:
      app: sso
  name: example-keycloak
  namespace: keycloak
spec:
  externalDatabase:
    enabled: true
  instances: 1

This operator can also be used to partially manage an external Red Hat Single Sign-On instance. In it’s current state, it will only be able to create clients.

To do this, you’ll need to create unmanaged versions of the Keycloak and KeycloakRealm CRDs to use for targeting and configuration.

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  name: external-ref
  labels:
    app: external-sso
spec:
  unmanaged: true
  external:
    enabled: true
    url: https://some.external.url

In order to authenticate against this keycloak, the operator infers the secret name from the CRD by prefixing the CRD name with credential-.

apiVersion: v1
kind: Secret
metadata:
  name: credential-external-ref
type: Opaque
data:
  ADMIN_USERNAME: YWRtaW4=
  ADMIN_PASSWORD: cGFzcw==

apiVersion: keycloak.org/v1alpha1
kind: KeycloakRealm
metadata:
  name: external-realm
  labels:
    app: external-sso
spec:
  unmanaged: true
  realm:
    id: "basic"
    realm: "basic"
  instanceSelector:
    matchLabels:
      app: external-sso

You can now use the realm reference in your client as usual, and it will create the client on the external Red Hat Single Sign-On instance.

You can use the Operator to schedule automatic backups of the database as defined by custom resources. The custom resource triggers a backup job and reports back its status.

You can use Operator to create a backup job that performs a one-time backup to a local Persistent Volume.

apiVersion: keycloak.org/v1alpha1
kind: KeycloakBackup
metadata:
  name: test-backup

You can use the operator to install extensions and themes that you need for your company or organization. The extension or theme can be anything that Red Hat Single Sign-On can consume. For example, you can add a metrics extension. You add the extension or theme to the Keycloak custom resource.

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  name: example-keycloak
  labels:
   app: sso
spec:
  instances: 1
  extensions:
   - <url_for_extension_or_theme>
  externalAccess:
    enabled: True

You can package and deploy themes in the same way as any other extensions. See Deploying Themes manual entry for more information.

The Operator downloads the extension or theme and installs it.

After you create a custom request, you can edit it or delete using the oc command.

For example, to edit a realm custom request named test-realm, use this command:

$ oc edit test-realm

A window opens where you can make changes.

You can configure how the operator performs Red Hat Single Sign-On upgrades. You can choose from the following upgrade strategies.

apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
  name: example-keycloak
  labels:
   app: sso
spec:
  instances: 2
  migration:
    strategy: recreate
    backups:
      enabled: True
  externalAccess:
    enabled: True