Chapter 12. Customizing Data Grid for OpenShift
Use the Data Grid image with custom configuration either with the Source-to-Image (S2I) process or via the ConfigMap
API.
Red Hat encourages you to use the Data Grid for OpenShift image and datagrid-service
and cache-service
templates to create Data Grid clusters. While it is possible to create Data Grid pods with custom configuration, datagrid-service
and cache-service
are designed for high performance and are suitable for a wide range of use cases with little to no configuration required.
Go to Setting Up Data Grid for OpenShift Services and learn how to quickly create Data Grid clusters.
- Source-to-Image (S2I)
Use the S2I process and binary builds to create custom Data Grid images.
Add cache definitions and endpoint configurations to
openshift-clustered.xml
then use S2I capabilities to build a custom Data Grid image that uses that configuration file. You can then create Data Grid pods with the image as required.To modify the configuration you must build a new image. However, when you rebuild your custom image, the Data Grid pods automatically redeploy with the new configuration changes.
- ConfigMap API
Use the Red Hat OpenShift
ConfigMap
API to dynamically configure Data Grid pods.Define custom configuration in
standalone.xml
that maps to aConfigMap
object in the namespace as the Data Grid pod. You can modify the Data Grid configuration and then redeploy pods to load configuration changes. However, Data Grid pods do not automatically redeploy when you modifystandalone.xml
. You must manually redeploy pods to load the configuration changes.You should explicitly define all cache and endpoint configuration in
standalone.xml
before you create Data Grid pods.Environment variables for cache and endpoint configuration do not take effect unless the placeholder exists before deployment. For example, the following is a placeholder for the
JGROUPS_PING_PROTOCOL
:<!-- ##JGROUPS_PING_PROTOCOL## -->
See clustered-openshift.xml to review available placeholders.
-
To encrypt client to server traffic, you must configure the server identity in
standalone.xml
. -
The
DATAGRID_SPLIT
environment variable does not take effect with Data Grid for OpenShift pods that you customize via theConfigMap
API. These pods cannot use shared persistent volumes withDATAGRID_SPLIT
.
12.1. Cloning the Data Grid Examples
Clone the Data Grid for OpenShift image source repository.
$ git clone git@github.com:jboss-container-images/jboss-datagrid-7-openshift-image.git .
View the contents of the
docs/examples
directory, for example:$ cd jboss-datagrid-7-openshift-image/docs/examples/s2i/configuration $ tree . ├── s2i │ └── configuration │ └── clustered-openshift.xml └── user-configuration ├── standalone.xml └── user-config-template.yaml
- S2I
Injects
clustered-openshift.xml
to${JBOSS_HOME}/standalone/configuration/
inside the Data Grid for OpenShift image.TipAdd custom logging.properties and application-role.properties to the
configuration
directory to include them when you build a custom image.- ConfigMap
-
Maps
standalone.xml
to the/opt/datagrid/standalone/configuration/user
directory inside the Data Grid for OpenShift image.
12.2. Creating S2I Builds with Custom Configuration
12.2.1. Setting Up the Data Grid Image
You need the Data Grid for OpenShift image as source for your customization.
Confirm if the Data Grid for OpenShift image is available.
$ oc get images | grep jboss-datagrid73-openshift
If the image is not available, create an image stream and import it.
$ oc create -f https://raw.githubusercontent.com/jboss-container-images/jboss-datagrid-7-openshift-image/7.3-v1.9/templates/datagrid73-image-stream.json $ oc import-image jboss-datagrid73-openshift --from='registry.redhat.io/jboss-datagrid-7/datagrid73-openshift:1.9'
12.2.2. Creating Custom Data Grid Images
Create a new binary build using the Data Grid for OpenShift image.
$ oc new-build jboss-datagrid73-openshift:1.9 --binary=true --name=custom-datagrid
Navigate to the
s2i
directory so you can pass--from-dir="."
. You must include theconfiguration
directory so that the S2I process can detect your custom configuration.To use the example configuration, do the following:
$ cd jboss-datagrid-7-openshift-image/docs/examples/s2i/
Build the Data Grid for OpenShift image with your custom configuration.
$ oc start-build custom-datagrid --from-dir="." --follow Uploading directory "." as binary input for the build ... ... Push successful
Check that your image is available.
$ oc get images | grep custom-datagrid
12.2.3. Verifying Custom Data Grid Images
Confirm that the build pod is running.
$ oc get pods NAME READY STATUS RESTARTS AGE custom-datagrid-1-build 0/1 Completed 0 38s
Create a Data Grid application with your custom configuration.
$ oc new-app custom-datagrid \ -p APPLICATION_USER=${USERNAME} \ -p APPLICATION_PASSWORD=${PASSWORD}
Wait for your custom Data Grid application to start running.
$ oc get pods -w NAME READY STATUS RESTARTS AGE custom-datagrid-1-build 0/1 Completed 0 8m custom-datagrid-1-<id> 1/1 Running 0 11s
Remotely access the bash shell for your custom Data Grid pod.
$ oc exec -it ${pod-name} -- /bin/bash
View
clustered-openshift.xml
to verify the configuration, for example:$ cat /opt/datagrid/configuration/clustered-openshift.xml
If
clustered-openshift.xml
contains your custom configuration then the Data Grid pod is using it. You can optionally use the Data Grid command line interface to verify your configuration, for example:$ /opt/datagrid/bin/cli.sh -c
End the remote session after you verify your configuration.
$ exit
12.3. Creating Custom Data Grid for OpenShift Pods with the ConfigMap API
Create a custom template for your Data Grid for OpenShift pod.
- Expose the required ports and services in your template.
-
Add a
configMap
object to the custom template. -
Add a config volume for the container at
/opt/datagrid/standalone/configuration/user
. Import your custom template into OpenShift.
To use the example template, do the following:
$ cd jboss-datagrid-7-openshift-image/docs/examples/user-configuration/
$ oc create -f user-config-template.yaml
Create a ConfigMap in your OpenShift project, for example:
$ oc create configmap user-config --from-file="."
Create Data Grid pods with your custom configuration.
$ oc new-app user-config \ -p APPLICATION_NAME=${USERNAME} \ -e USER_CONFIG_MAP=true
Where:
-
APPLICATION_NAME
is a required parameter in the example template and defaults tocustom-datagrid
. USER_CONFIG_MAP=true
applies the ConfigMap to the Data Grid pod. This is set in the example template as follows:- env: - name: USER_CONFIG_MAP value: "true"
-
12.3.1. Verifying Custom Data Grid for OpenShift Pods with the ConfigMap API
Wait for your custom Data Grid application to start running.
$ oc get pods -w NAME READY STATUS RESTARTS AGE user-config-0 0/1 Running 7 17m
Check the container logs.
$ oc logs ${pod-name} | grep standalone.xml INFO Running jboss-datagrid-7/datagrid73-openshift image, version 1.9 with user standalone.xml
Try the Customizing Data Grid Service Deployments Quickstart Tutorial.
12.4. Configuring Persistent Datasources
Data Grid lets you persist data stored in the cache to a datasource. There are two types of datasources for Red Hat Data Grid for OpenShift:
Internal datasources that run on OpenShift. These datasources are available through the Red Hat Container Registry and do not require you to configure additional environment files.
NoteInternal datasources include PostgreSQL, MySQL, and MongoDB. However, Red Hat Data Grid for OpenShift currently supports PostgreSQL and MySQL only.
- External datasources that do not run on OpenShift. You must configure these external datasources with environment files that you add to OpenShift Secrets.
12.4.1. Configuring Internal Datasources
The DB_SERVICE_PREFIX_MAPPING
environment variable defines JNDI mappings for internal datasources.
You can define multiple JNDI mappings as comma-separated values for the DB_SERVICE_PREFIX_MAPPING
environment variable. When you run the Data Grid for OpenShift image, the launch script creates a separate datasource for each JNDI mapping. The Data Grid for OpenShift then automatically discovers each datasource.
To define a JNDI mapping, specify a value for the environment variable in the following format:
${POOL_NAME}-${DATABASE_TYPE}=${PREFIX}
-
${POOL_NAME}
is thepool-name
attribute for the datasource. Use any alphanumeric value that is meaningful and easy to identify. The value cannot contain special characters. Likewise, the value must contain lowercase characters only. ${DATABASE_TYPE}
specifies the database driver to use. The value must contain lowercase characters only.NoteOnly
mysql
andpostgresql
are supported values for{$DATABASE_TYPE}
.-
${PREFIX}
is used for the names of environment variables that configure the datasource.
12.4.1.1. Single Datasource Example
If you specify test-postgresql=TEST
as the value for the DB_SERVICE_PREFIX_MAPPING
environment variable, it creates a datasource with the following name:
java:jboss/datasources/test_postgresql
You must use the TEST
prefix when specifying other environment variables for the datasource. For example, to set the username and password, use TEST_USERNAME
and TEST_PASSWORD
as the environment variables.
12.4.1.2. Multiple Datasource Example
If you specify cloud-postgresql=CLOUD,test-mysql=TEST_MYSQL
as the value for the DB_SERVICE_PREFIX_MAPPING
environment variable, it creates two datasources with the following names:
-
java:jboss/datasources/test_mysql
-
java:jboss/datasources/cloud_postgresql
When specifying other environment variables for the datasources, you must use the TEST_MYSQL
prefix to configure the MySQL datasource. For example, use TEST_MYSQL_USERNAME
as the environment variable to specify the username.
Similarly, you must use the CLOUD
prefix to configure the PostgreSQL datasource. For example, use CLOUD_USERNAME
as the environment variable to specify the username.
12.4.2. Configuring External Datasources
To use an external datasource, you define a custom image template and then use the Source-to-Image (S2I) build tool to create an image. S2I is a framework that takes application source code as an input and produces a new image that runs the assembled application as output.
The following high-level steps provide an overview of the process:
Specify the
CUSTOM_INSTALL_DIRECTORIES
environment variable in the image template JSON. This variable defines the location where S2I artifacts reside, as in the following example:{ "name": "CUSTOM_INSTALL_DIRECTORIES", "value": "extensions/*" }
Create an
install.sh
script in that directory. This script installs the modules and drivers for the external datasource in the image.The following is an example
install.sh
script:#!/bin/bash # Import the common functions for # installing modules and configuring drivers source /usr/local/s2i/install-common.sh # Directory where this script is located injected_dir=$1 # Install the modules for the datasource install_modules ${injected_dir}/modules # Configure the drivers for the datasource configure_drivers ${injected_dir}/drivers.properties
Include a
modules
subdirectory that contains amodule.xml
file and the driver for the datasource. The resulting image uses the module to load classes and define dependencies.As an example, you plan to use Derby as an external datasource. You need to obtain a driver such as
derby-10.12.1.1.jar
and place it in the following directory:modules/org/apache/derby/main/
In the same directory, you also need to create a
module.xml
file that defines the driver as a resource and declares dependencies.The following is an example
module.xml
file:<?xml version="1.0" encoding="UTF-8"?> <module xmlns="urn:jboss:module:1.3" name="org.apache.derby"> <resources> <resource-root path="derby-10.12.1.1.jar"/> <resource-root path="derbyclient-10.12.1.1.jar"/> </resources> <dependencies> <module name="javax.api"/> <module name="javax.transaction.api"/> </dependencies> </module>
Define the driver configuration properties in a
drivers.property
environment variable file.The following is an example
drivers.property
file:#DRIVERS DRIVERS=DERBY DERBY_DRIVER_NAME=derby DERBY_DRIVER_MODULE=org.apache.derby DERBY_DRIVER_CLASS=org.apache.derby.jdbc.EmbeddedDriver DERBY_XA_DATASOURCE_CLASS=org.apache.derby.jdbc.EmbeddedXADataSource
After you build and deploy the image, specify environment variables for the datasource.
The following example shows a datasource definition with the
DATASOURCES
environment variable:# Set a unique prefix for the datasource DATASOURCES=ACCOUNTS_DERBY # Specify other environment variables using the prefix ACCOUNTS_DERBY_DATABASE=accounts ACCOUNTS_DERBY_JNDI=java:/accounts-ds ACCOUNTS_DERBY_DRIVER=derby ACCOUNTS_DERBY_JTA=true ACCOUNTS_DERBY_NONXA=false ACCOUNTS_DERBY_USERNAME=${USERNAME} ACCOUNTS_DERBY_PASSWORD=${PASSWORD} ACCOUNTS_DERBY_XA_CONNECTION_PROPERTY_DatabaseName=/opt/eap/standalone/data/databases/derby/accounts # _HOST and _PORT are required but not used ACCOUNTS_ORACLE_HOST=dummy ACCOUNTS_ORACLE_PORT=1527
12.4.3. Datasource Environment Variables
DB_SERVICE_PREFIX_MAPPING
Defines a comma-separated list of datasources to configure.
For example,
DB_SERVICE_PREFIX_MAPPING=test-mysql=TEST_MYSQL
. See Configuring Persistent Datasources for more information.${NAME}_${DATABASE_TYPE}_SERVICE_HOST
Defines the database server hostname or IP for the datasource
connection_url
property.For example,
EXAMPLE_MYSQL_SERVICE_HOST=192.0.2.0
${NAME}_${DATABASE_TYPE}_SERVICE_PORT
- Defines the database server port.
${PREFIX}_USERNAME
- Defines the user for the datasource.
${PREFIX}_PASSWORD
- Defines the password for the datasource.
${PREFIX}_DATABASE
Defines the database name for the datasource.
For example,
CLOUD_DATABASE=myDatabase
.${PREFIX}_DRIVER
Defines Java database driver for the datasource.
For example,
CLOUD_DRIVER=postgresql
${PREFIX}_BACKGROUND_VALIDATION
-
Specifies if a background thread validates database connections before they are used. The value is
true
orfalse
(default). By default, the<validate-on-match>
method is enabled. ${PREFIX}_BACKGROUND_VALIDATION_MILLIS
-
Specifies how often validation occurs, in milliseconds, if you set the
${PREFIX}_BACKGROUND_VALIDATION
environment variable totrue
. The default value is10000
. ${PREFIX}_CONNECTION_CHECKER
Specifies a connection checker class that validates connections to the database.
For example,
CLOUD_CONNECTION_CHECKER=org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLValidConnectionChecker
${PREFIX}_EXCEPTION_SORTER
Specifies the exception sorter class that detects and cleans up after fatal database connection exceptions.
For example,
CLOUD_EXCEPTION_SORTER=org.jboss.jca.adapters.jdbc.extensions.mysql.MySQLExceptionSorter
${PREFIX}_JNDI
Defines the JNDI name for the datasource.
Defaults to
java:jboss/datasources/<name>_<database_type>
. The launch script automatically generates the value from theDB_SERVICE_PREFIX_MAPPING
environment variable.For example,
CLOUD_JNDI=java:jboss/datasources/test-postgresql
${PREFIX}_JTA
-
Defines the Java Transaction API (JTA) option for non-XA datasources. The value is
true
(default) orfalse
. ${PREFIX}_MAX_POOL_SIZE
- Defines the maximum pool size for the datasource.
${PREFIX}_MIN_POOL_SIZE
- Defines the minimum pool size for the datasource.
${PREFIX}_NONXA
-
Defines the datasource as a non-XA datasource. The value is
true
orfalse
(default). ${PREFIX}_TX_ISOLATION
Defines the java.sql.Connection transaction isolation level for the database.
For example,
CLOUD_TX_ISOLATION=TRANSACTION_READ_UNCOMMITTED
${PREFIX}_URL
Defines the connection URL for a non-XA datasource.
If you do not specify a connection URL, the launch script automatically generates it from other environment variables as follows:
url="jdbc:${DRIVER}://${HOST}:${PORT}/${DATABASE}"
.However, the launch script constructs the correct connection URLs only for internal datasources such as PostgreSQL and MySQL. If you use any other non-XA datasource you must specify the connection URL.
For example,
CLOUD_URL=jdbc:postgresql://localhost:5432/postgresdb
${PREFIX}_XA_CONNECTION_PROPERTY_<PROPERTY_NAME>
Defines connection properties for an XA datasource.
Consult the appropriate driver documentation for your datasource to find which XA properties you can set on the connection.
For example,
CLOUD_XA_CONNECTION_PROPERTY_DatabaseName=/opt/eap/standalone/data/databases/db/accounts
This example adds the following to the configuration:
<xa-datasource-property name="DatabaseName">/opt/eap/standalone/data/databases/db/accounts</xa-datasource-property>