Chapter 17. Using a Vault to Obtain Secrets

Several fields in the administration support obtaining the value of a secret from an external vault.

To obtain a secret from a vault instead of entering it directly, enter the following specially crafted string into the appropriate field: ${vault.key} where you replace the key with the name of the secret as recognized by the vault.

In order to prevent secrets from leaking across realms, implementations may combine the realm name with the key obtained from the vault expression. This means that the key won’t directly map to an entry in the vault, but rather be used to create the final entry name according to the algorithm used to combine it with the realm name.

Currently, the secret can be obtained from the vault in the following fields:

SMTP password
In realm SMTP settings
LDAP bind credential
In LDAP settings of LDAP-based user federation.
OIDC identity provider secret
In Client Secret inside identity provider OpenID Connect Config

To use a vault, a vault provider must be registered within Red Hat Single Sign-On. It is possible to either use a built-in provider described below or implement your own provider. See the Server Developer Guide for more information.

Note

There is at most one vault provider active per Red Hat Single Sign-On instance at any given time, and the vault provider in each instance within the cluster has to be configured consistently.

17.1. Kubernetes / OpenShift Files Plaintext Vault Provider

Red Hat Single Sign-On supports vault implementation for Kubernetes secrets. These secrets can be mounted as data volumes, and they appear as a directory with a flat file structure, where each secret is represented by a file whose name is the secret name, and contents of that file is the secret value.

The files within this directory have to be named as secret name prefixed by realm name and an underscore. All underscores within the secret name or the realm name have to be doubled in the file name. For example, for a field within a realm called sso_realm, a reference to a secret with name secret-name would be written as ${vault.secret-name}, and the file name looked up would be sso__realm_secret-name (note the underscore doubled in realm name).

To use this type of secret store, you have to declare the files-plaintext vault provider in standalone.xml, and set its parameter for the directory that contains the mounted volume. The following example shows the files-plaintext provider with the directory where vault files are searched for set to standalone/configuration/vault relative to Red Hat Single Sign-On base directory:

<spi name="vault">
    <default-provider>files-plaintext</default-provider>
    <provider name="files-plaintext" enabled="true">
        <properties>
            <property name="dir" value="${jboss.home.dir}/standalone/configuration/vault/" />
        </properties>
    </provider>
</spi>

Here is the equivalent configuration using CLI commands:

/subsystem=keycloak-server/spi=vault/:add
/subsystem=keycloak-server/spi=vault/provider=files-plaintext/:add(enabled=true,properties={dir => "${jboss.home.dir}/standalone/configuration/vault"})

17.2. Elytron Credential Store Vault Provider

Red Hat Single Sign-On also provides support for reading secrets stored in an Elytron credential store. The elytron-cs-keystore vault provider is capable of retrieving secrets from the keystore-based implementation of the credential store, which is also the default implementation provided by Elytron.

This credential store is backed by a keystore (JCEKS is the default format, but it is possible to use other formats such as PKCS12) and users can create and manage the store contents using either the elytron subsystem in WildFly/JBoss EAP, or using the elytron-tool.sh script.

To use this provider, you have to declare the elytron-cs-keystore in the keycloak-server subsystem and set the location and master secret of the keystore that was created by Elytron. An example of the minimal configuration for the provider follows:

<spi name="vault">
    <default-provider>elytron-cs-keystore</default-provider>
    <provider name="elytron-cs-keystore" enabled="true">
        <properties>
            <property name="location" value="${jboss.home.dir}/standalone/configuration/vault/credential-store.jceks" />
            <property name="secret" value="secretpw1!"/>
        </properties>
    </provider>
</spi>

If the underlying keystore has a format other than JCEKS, this format has to be informed using the keyStoreType:

<spi name="vault">
    <default-provider>elytron-cs-keystore</default-provider>
    <provider name="elytron-cs-keystore" enabled="true">
        <properties>
            <property name="location" value="${jboss.home.dir}/standalone/configuration/vault/credential-store.p12" />
            <property name="secret" value="secretpw1!"/>
            <property name="keyStoreType" value="PKCS12"/>
        </properties>
    </provider>
</spi>

For the secret, the elytron-cs-keystore provider supports both clear-text values (as shown above) and also values that were masked using the elytron-tool.sh script:

<spi name="vault">
   ...
            <property name="secret" value="MASK-3u2HNQaMogJJ8VP7J6gRIl;12345678;321"/>
   ...
</spi>

For more detailed information on how to create/manage elytron credential stores, as well as how to mask keystore secrets, please refer to the Elytron documentation.

Note

The elytron-cs-keystore vault provider has been implemented as a WildFly extension and as such is only available if the Red Hat Single Sign-On server runs on WildFly/JBoss EAP.

17.3. Key Resolvers

All built-in providers support the configuration of one or more key resolvers. A key resolver essentially implements the algorithm or strategy for combining the realm name with the key (as obtained from the ${vault.key} expression} into the final entry name that will be used to retrieve the secret from the vault. The keyResolvers property is used to configure the resolvers that are to be used by the provider. The value is a comma-separated list of resolver names. An example of configuration for the files-plaintext provider follows:

<spi name="vault">
    <default-provider>files-plaintext</default-provider>
    <provider name="files-plaintext" enabled="true">
        <properties>
            <property name="dir" value="${jboss.home.dir}/standalone/configuration/vault/" />
            <property name="keyResolvers" value="REALM_UNDERSCORE_KEY, KEY_ONLY"/>
        </properties>
    </provider>
</spi>

The resolvers are executed in the same order that they are declared in the configuration. For each resolver, the final entry name produced by the resolver that combines the realm with the vault key is used to search for the secret in the vault. If a secret is found, it is immediately returned. If not, the next resolver is used and this continues until a non-empty secret is found or all resolvers have been tried, in which case an empty secret is returned. In the example above, first the REALM_UNDERSCORE_KEY resolver is used. If an entry is found in the vault with the name it produces, it is returned. If not, then the KEY_ONLY resolver is used. Again, if an entry is found in the vault with the name it produces, it is returned. If not, an empty secret is returned since there are no more resolvers to be used.

A list of the currently available resolvers follows:

  • KEY_ONLY: the realm name is ignored and the key from the vault expression is used as is.
  • REALM_UNDERSCORE_KEY: the realm and key are combined using an underscore _ character. Occurrences of underscore in either the realm or key are escaped by another underscore character. So if the realm is called master_realm and the key is smtp_key, the combined key will be master__realm_smtp__key.
  • REALM_FILESEPARATOR_KEY: the realm and key are combined using the platform file separator character. This is useful in situations where the keys are grouped by realm using a directory structure.

If no resolver is configured for the built-in providers, the REALM_UNDERSCORE_KEY is selected by default.

17.4. Sample Configuration

The following is an example of configuring a vault and credential store. The procedure involves two parts:

  • Creating the credential store and a vault, where the credential store and vault passwords are in plain text.
  • Updating the credential store and vault to have the password use a mask provided by elytron-tool.sh.

In this example, the test target used is an LDAP instance with BIND DN credential: secret12. The target is mapped using user federation in the realm ldaptest.

17.4.1. Configuring the credential store and vault without a mask

You create the credential store and a vault where the credential store and vault passwords are in plain text.

Prerequisites

  • A running LDAP instance has BIND DN credential: secret12.
  • The alias uses the format <realm-name>_< key-value> when using the default key resolver. In this case, the instance is running in the realm ldaptest and ldaptest_ldap_secret is the alias that corresponds to the value ldap_secret in that realm.
Note

The resolver replaces underscore characters with double underscore characters in the realm and key names. For example, for the key ldaptest_ldap_secret, the final key will be ldaptest_ldap__secret.

Procedure

  1. Create the Elytron credential store.

    [standalone@localhost:9990 /] /subsystem=elytron/credential-store=test-store:add(create=true, location=/home/test/test-store.p12, credential-reference={clear-text=testpwd1!},implementation-properties={keyStoreType=PKCS12})
  2. Add an alias to the credential store.

    /subsystem=elytron/credential-store=test-store:add-alias(alias=ldaptest_ldap__secret,secret-value=secret12)

    Notice how the resolver causes the key ldaptest_ldap__secret to use double underscores.

  3. List the aliases from the credential store to inspect the contents of the keystore that is produced by Elytron.

    keytool -list -keystore /home/test/test-store.p12 -storetype PKCS12 -storepass testpwd1!
    Keystore type: PKCS12
    Keystore provider: SUN
    
    Your keystore contains 1 entries
    
    ldaptest_ldap__secret/passwordcredential/clear/, Oct 12, 2020, SecretKeyEntry,
  4. Configure the vault SPI in Red Hat Single Sign-On.

    /subsystem=keycloak-server/spi=vault:add(default-provider=elytron-cs-keystore)
    
    /subsystem=keycloak-server/spi=vault/provider=elytron-cs-keystore:add(enabled=true, properties={location=>/home/test/test-store.p12, secret=>testpwd1!, keyStoreType=>PKCS12})

    At this point, the vault and credentials store passwords are not masked.

            <spi name="vault">
                    <default-provider>elytron-cs-keystore</default-provider>
                    <provider name="elytron-cs-keystore" enabled="true">
                        <properties>
                            <property name="location" value="/home/test/test-store.p12"/>
                            <property name="secret" value="testpwd1!"/>
                            <property name="keyStoreType" value="PKCS12"/>
                        </properties>
                    </provider>
                </spi>
    
             <credential-stores>
                    <credential-store name="test-store" location="/home/test/test-store.p12" create="true">
                        <implementation-properties>
                            <property name="keyStoreType" value="PKCS12"/>
                        </implementation-properties>
                        <credential-reference clear-text="testpwd1!"/>
                    </credential-store>
             </credential-stores>
  5. In the LDAP provider, replace binDN credential with ${vault.ldap_secret}.
  6. Test your LDAP connection.

    LDAP Vault

    LDAP Vault

17.4.2. Masking the password in the credential store and vault

You can now update the credential store and vault to have passwords that use a mask provided by elytron-tool.sh.

  1. Create a masked password using values for the salt and the iteration parameters:

    $ EAP_HOME/bin/elytron-tool.sh mask --salt SALT --iteration ITERATION_COUNT --secret PASSWORD

    For example:

    elytron-tool.sh mask --salt 12345678 --iteration 123 --secret testpwd1!
    MASK-3BUbFEyWu0lRAu8.fCqyUk;12345678;123
  2. Update the Elytron credential store configuration to use the masked password.

    /subsystem=elytron/credential-store=cs-store:write-attribute(name=credential-reference.clear-text,value="MASK-3BUbFEyWu0lRAu8.fCqyUk;12345678;123")
  3. Update the Red Hat Single Sign-On vault configuration to use the masked password.

    /subsystem=keycloak-server/spi=vault/provider=elytron-cs-keystore:remove()
    /subsystem=keycloak-server/spi=vault/provider=elytron-cs-keystore:add(enabled=true, properties={location=>/home/test/test-store.p12, secret=>”MASK-3BUbFEyWu0lRAu8.fCqyUk;12345678;123”, keyStoreType=>PKCS12})

    The vault and credential store are now masked:

            <spi name="vault">
                    <default-provider>elytron-cs-keystore</default-provider>
                    <provider name="elytron-cs-keystore" enabled="true">
                        <properties>
                            <property name="location" value="/home/test/test-store.p12"/>
                            <property name="secret" value="MASK-3BUbFEyWu0lRAu8.fCqyUk;12345678;123"/>
                            <property name="keyStoreType" value="PKCS12"/>
                        </properties>
                    </provider>
                </spi>
             ....
             .....
             <credential-stores>
                    <credential-store name="test-store" location="/home/test/test-store.p12" create="true">
                        <implementation-properties>
                            <property name="keyStoreType" value="PKCS12"/>
                        </implementation-properties>
                        <credential-reference clear-text="MASK-3BUbFEyWu0lRAu8.fCqyUk;12345678;123"/>
                    </credential-store>
             </credential-stores>
  4. You can now test the connection to the LDAP using ${vault.ldap_secret}.

Additional resources

For more information about the Elytron tool, see Using Credential Stores with Elytron Client.