Chapter 15. Reference
15.1. Probes
Data Grid for OpenShift provides a liveness probe and a readiness probe to perform container health checks.
- Liveness probe
The liveness probe is located in the container at
/opt/datagrid/bin/livenessProbe.sh
.The liveness probe tests server status and restarts pods if the following events occur:
- Data Grid for OpenShift boots with errors.
- Custom deployment configurations do not successfully deploy.
- One or more caches fail to instantiate, which usually occurs if the cache configuration is not valid.
- Readiness probe
The readiness probe is located in the container at
/opt/datagrid/bin/readinessProbe.sh
.The readiness probe determines if the pod is ready to receive requests and checks Data Grid cache level
MBeans
to ensure the following:- All cache instances are initialized.
- All cache instances have joined the cluster, if using distributed cache mode.
- Initial state transfer is complete. If state transfer is in progress, the pod is not marked as ready.
- All cache instances in the Cache manager are running.
To configure custom deployments to use the liveness probe and readiness probe, run the following commands:
$ oc set probe dc/datagrid \ --readiness \ -- /bin/bash \ -c /opt/datagrid/bin/readinessProbe.sh $ oc set probe dc/datagrid \ --liveness \ -- /bin/bash \ -c /opt/datagrid/bin/livenessProbe.sh
15.2. Ports
Data Grid for OpenShift uses the following ports:
Port Number | Protocol | Use |
---|---|---|
8080 | TCP | HTTP Access |
8443 | TCP | HTTPS Access |
8888 | TCP | JGroups Ping |
11222 | TCP | Hot Rod Access |
Data Grid deployment configuration templates also use the following ports:
Port Number | Protocol | Use |
---|---|---|
11211 | TCP | Memcached Access |
11333 | TCP | External Hot Rod Access |
8778 | TCP | Remote JMX Access |
If you set the HOTROD_SERVICE_NAME environment variable with the deployment configuration templates, the Hot Rod external connector returns ${service_name}:11333
for the endpoint.
15.3. Administration Console
The Data Grid Administration Console is not supported on Red Hat OpenShift.
To monitor events and get statistics for Data Grid for OpenShift clusters, you should use Prometheus. See Setting Up Monitoring.
You can also use the Management CLI to troubleshoot Data Grid for OpenShift pods. See Launching the Command Line Interface.
15.4. Environment Variables
15.4.1. Monitoring and Logging
AB_PROMETHEUS_ENABLE
Allows you to collect JMX metrics to monitor and analyze Data Grid. The default value is
false
. Set the value totrue
to enable monitoring with the default Prometheus agent.The Prometheus Operator must be installed and running. You must also Set Up Monitoring.
AB_PROMETHEUS_JMX_EXPORTER_PORT
-
Defines the port on which Data Grid publishes JMX metrics. The default is
9779
. LOGGING_CATEGORIES
Adjusts the categories and levels for which Data Grid captures log messages, for example:
$ LOGGING_CATEGORIES=org.infinipan.core=WARN,org.infinispan.config=DEBUG
Log categories correspond to Java package names and use standard log levels:
TRACE
,DEBUG
,INFO
,WARN
,ERROR
, andFATAL
.ImportantIf you specify
LOGGING_CATEGORIES
, Data Grid does not set the following default loggers. Instead, Data Grid uses the default level ofINFO
for all packages that you do not explicitly specify withLOGGING_CATEGORIES
.Table 15.1. Default Loggers
Category Level com.arjuna
WARN
sun.rmi
WARN
org.jboss.as.config
DEBUG
15.4.2. Containers
USERNAME
Creates a user in the security realm who is authorized to access data.
NoteBy default, the Hot Rod endpoint uses the
ApplicationRealm
security realm and the REST endpoint uses thejdg-openshift
security realm.PASSWORD
- Specifies the password for the user.
DATAGRID_SPLIT
Determines if the data directory for each node should be split in a mesh. The value is
true
orfalse
(default).If you set the value to
true
, you must also configure a persistent volume mounted on/opt/datagrid/standalone/partitioned_data
.JAVA_OPTS_APPEND
Appends options to the
JAVA_OPTS
environment variable on startup.For example,
JAVA_OPTS_APPEND=-Dfoo=bar
OPENSHIFT_KUBE_PING_LABELS
Specifies the clustering labels selector.
For example,
OPENSHIFT_KUBE_PING_LABELS=application=eap-app
OPENSHIFT_KUBE_PING_NAMESPACE
- Specifies the clustering project namespace.
TRANSPORT_LOCK_TIMEOUT
Sets the time to wait to acquire a distributed lock. The default value is
240000
.Data Grid uses a distributed lock to maintain a coherent transaction log during state transfer or rehashing, which means that only one cache can perform state transfer or rehashing at a time. This constraint is in place because more than one cache could be involved in a transaction.
15.4.3. Caches
Creating and configuring caches with cache-service
and datagrid-service
Do not use environment variables to create and configure caches with cache-service
or datagrid-service
.
These environment variables are intended for use with the deployment configuration templates only and are deprecated.
You should remote create caches with cache-service
and datagrid-service
dynamically through the Hot Rod endpoint. For more information, see Remotely Creating Caches.
CACHE_NAMES
Defines cache instances in your configuration.
If you are using the Data Grid deployment configuration templates and you do not define any cache instances, the launch script adds a default distributed cache in
SYNC
mode.TipGive each cache instance in your configuration a unique name. Use underscore characters (_) and descriptive labels to help you distinguish between cache instances. This ensures that you do not have conflicts when applying cache-specific configuration.
For example,
CACHE_NAMES=addressbook,addressbook_indexed
CACHE_CONTAINER_START
Configures how the cache container starts. Specify one of the following:
-
LAZY
Starts the cache container when requested by a service or deployment. This is the default. -
EAGER
Starts the cache container when the server starts.
-
CACHE_CONTAINER_STATISTICS
-
Configures the cache container to collect statistics. The value is
true
(default) orfalse
. You can set the value tofalse
to improve performance. DEFAULT_CACHE
- Sets the default cache for the cache container.
15.4.3.1. Cache Container Security Configuration
CONTAINER_SECURITY_CUSTOM_ROLE_MAPPER_CLASS
Specifies the class of the custom principal to role mapper.
For example,
CONTAINER_SECURITY_CUSTOM_ROLE_MAPPER_CLASS=com.acme.CustomRoleMapper
CONTAINER_SECURITY_ROLE_MAPPER
Sets a role mapper for this cache container with the following values:
-
identity-role-mapper
Uses the Principal name as the role name. This is the default role mapper if you do not specify one and use theCONTAINER_SECURITY_ROLES
environment variable to define role names. -
common-name-role-mapper
Uses the Common Name (CN) as the role name if the Principal name is a Distinguished Name (DN). For example, the DNcn=managers,ou=people,dc=example,dc=com
is mapped to themanager
role name. -
cluster-role-mapper
Uses theClusterRegistry
to store Principal name to role mappings. -
custom-role-mapper
Takes the fully-qualified class name of an implementation of theorg.infinispan.security.impl.PrincipalRoleMapper
interface.
-
CONTAINER_SECURITY_ROLES
Defines role names and assigns permissions to them.
For example,
CONTAINER_SECURITY_ROLES=admin=ALL,reader=READ,writer=WRITE
15.4.3.2. Named Cache Configuration
You specify the cache name as a prefix for the environment variable in capital letters (all caps) otherwise the configuration does not take effect.
For example, you create two separate cache instances: MyCache
and MYCACHE
. You then set MyCache_CACHE_TYPE=replicated
to configure the MyCache
instance. This configuration does not take effect. However, if you set MYCACHE_CACHE_TYPE=replicated
the configuration takes effect for both the MyCache
and MYCACHE
instances.
${CACHE_NAME}_CACHE_TYPE
-
Determines whether this cache should be distributed or replicated. You can specify either
distributed
(default) orreplicated
. ${CACHE_NAME}_CACHE_START
Configures how the cache starts. Specify one of the following:
-
LAZY
Starts the cache when requested by a service or deployment. This is the default. -
EAGER
Starts the cache when the server starts.
-
${CACHE_NAME}_CACHE_BATCHING
-
Enables invocation batching for this cache. The value is
true
orfalse
(default). ${CACHE_NAME}_CACHE_STATISTICS
-
Configures the cache to collect statistics. The value is
true
(default) orfalse
. You can set the value tofalse
to improve performance. ${CACHE_NAME}_CACHE_MODE
Sets the clustered cache mode. Specify one of the following:
-
ASYNC
for asynchronous operations. -
SYNC
for synchronous operations.
-
${CACHE_NAME}_CACHE_QUEUE_SIZE
-
Sets the threshold at which the replication queue is flushed when the cache is in
ASYNC
mode. The default value is0
(flushing is disabled). ${CACHE_NAME}_CACHE_QUEUE_FLUSH_INTERVAL
-
Specifies the wakeup time, in milliseconds, for the thread that flushes the replication queue in
ASYNC
mode. The default value is10
. ${CACHE_NAME}_CACHE_REMOTE_TIMEOUT
-
Specifies the timeout, in milliseconds, to wait for acknowledgement when making remote calls in
SYNC
mode. If the timeout is reached, the remote call is aborted and an exception is thrown. The default value is17500
. ${CACHE_NAME}_CACHE_OWNERS
-
Specifies the number of cluster-wide replicas for each cache entry. The default value is
2
. ${CACHE_NAME}_CACHE_SEGMENTS
-
Specifies the number of hash space segments per cluster. The recommended value is
10 * cluster size
. The default value is80
. ${CACHE_NAME}_CACHE_L1_LIFESPAN
-
Specifies the maximum lifespan, in milliseconds, of an entry placed in the L1 cache. The default value is
0
(L1 is disabled). ${CACHE_NAME}_CACHE_MEMORY_EVICTION_TYPE
Defines the maximum limit for entries in the cache. You can set the following values:
-
COUNT
Measures the number of entries in the cache. When the count exceeds the maximum, Data Grid evicts unused entries. -
MEMORY
Measures the amount of memory that all entries in the cache take up. When the total amount of memory exceeds the maximum, Data Grid evicts unused entries.
-
${CACHE_NAME}_CACHE_MEMORY_STORAGE_TYPE
Defines how Data Grid stores entries in the cache. You can set the following values:
Storage Type Description Eviction Type Policy object
Stores entries as objects in the Java heap. This is the default storage type.
COUNT
TinyLFU
binary
Stores entries as
bytes[]
in the Java heap.COUNT
orMEMORY
TinyLFU
off-heap
Stores entries as
bytes[]
in native memory outside the Java.COUNT
orMEMORY
LRU
${CACHE_NAME}_CACHE_MEMORY_EVICTION_SIZE
Configures the size of the cache before eviction starts. Set the value to a number greater than zero.
-
For
COUNT
, the size is the maximum number of entries the cache can hold before eviction starts. For
MEMORY
, the size is the maximum number of bytes the cache can take from memory before eviction starts. For example, a value of10000000000
is 10 GB.Try different cache sizes to determine the optimal setting. A cache size that is too large can cause Data Grid to run out of memory. At the same time, a cache size that is too small wastes available memory.
NoteIf you configure a JDBC store, passivation is automatically enabled when you set the eviction size to a value that is greater than zero.
-
For
${CACHE_NAME}_CACHE_MEMORY_EVICTION_STRATEGY
Controls how Data Grid performs eviction. You can set the following values:
Strategy Description NONE
Data Grid does not evict entries. This is the default setting unless you configure eviction.
REMOVE
Data Grid removes entries from memory so that the cache does not exceed the configured size. This is the default setting when you configure eviction.
MANUAL
Data Grid does not perform eviction. Eviction takes place manually by invoking the
evict()
method from theCache
API.EXCEPTION
Data Grid does not write new entries to the cache if doing so would exceed the configured size. Instead of writing new entries to the cache, Data Grid throws a
ContainerFullException
.${CACHE_NAME}_CACHE_MEMORY_OFF_HEAP_ADDRESS_COUNT
Specifies the number of pointers that are available in the hash map to prevent collisions when using
OFFHEAP
storage. Preventing collisions in the hash map improves performance.Set the value to a number that is greater than the number of cache entries. By default
address-count
is 2^20, or 1048576. The parameter is always rounded up to a power of 2.
${CACHE_NAME}_CACHE_EXPIRATION_LIFESPAN
-
Specifies the maximum lifespan, in milliseconds, of a cache entry, after which the entry is expired cluster-wide. The default value is
-1
(entries never expire). ${CACHE_NAME}_CACHE_EXPIRATION_MAX_IDLE
-
Specifies the maximum idle time, in milliseconds, that cache entries are maintained in the cache. If the idle time is exceeded, then the entry is expired cluster-wide. The default value is
-1
(expiration is disabled). ${CACHE_NAME}_CACHE_EXPIRATION_INTERVAL
-
Specifies the interval, in milliseconds, between runs to purge expired entries from memory and any cache stores. The default value is
5000
. Set-1
to disable expiration. ${CACHE_NAME}_JDBC_STORE_TYPE
Sets the type of JDBC store to configure. You can set the following values:
-
string
-
binary
-
${CACHE_NAME}_JDBC_STORE_DATASOURCE
Defines the jndiname of the datasource.
For example,
MYCACHE_JDBC_STORE_DATASOURCE=java:jboss/datasources/ExampleDS
${CACHE_NAME}_KEYED_TABLE_PREFIX
-
Defines the prefix prepended to the cache name used when composing the name of the cache entry table. The defaule value is
ispn_entry
. ${CACHE_NAME}_CACHE_INDEX
Sets the indexing mode of the cache. You can set the following values:
-
NONE
This is the default. -
LOCAL
-
ALL
-
${CACHE_NAME}_INDEXING_PROPERTIES
Specifies a comma-separated list of properties to pass to the indexing system.
For example,
MYCACHE_INDEXING_PROPERTIES=default.directory_provider=ram
${CACHE_NAME}_CACHE_SECURITY_AUTHORIZATION_ENABLED
-
Enables authorization checks for this cache. The value is
true
orfalse
(default). ${CACHE_NAME}_CACHE_SECURITY_AUTHORIZATION_ROLES
Sets the roles required to access this cache.
For example,
MYCACHE_CACHE_SECURITY_AUTHORIZATION_ROLES=admin, reader, writer
${CACHE_NAME}_CACHE_PARTITION_HANDLING_WHEN_SPLIT
Configures the strategy for handling partitions between nodes in a cluster when network events isolate nodes from each other. Partitions function as independent clusters until Data Grid merges cache entries to re-form a single cluster. You can set the following values:
Partition Handling Strategy Description ALLOW_READ_WRITES
Nodes from any partition can read or write cache entries. This is the default value.
DENY_READ_WRITES
Nodes enter degraded mode if:
* One or more hash space segments in the partition have no owners. The
owners
are the number of cluster-wide replicas for cache entries.* The partition has less than half the nodes from the most recent stable cluster topology.
In degraded mode, only nodes in the same partition can read or write cache entries. All owners, or copies, for a cache entry must exist on the same partition, otherwise the read or write operation fails with an
AvailabilityException
.ALLOW_READS
Nodes enter degraded mode similarly to the
DENY_READ_WRITES
strategy. Nodes from any partition can read cache entries.In degraded mode, only nodes in the same partition can write cache entries. All owners, or copies, for a cache entry must exist on the same partition, otherwise the write operation fails with an
AvailabilityException
.${CACHE_NAME}_CACHE_PARTITION_MERGE_POLICY
Configures how Data Grid resolves conflicts between cache entries when merging partitions. You can set the following values:
Merge Policy Description NONE
Do not resolve conflicts when merging partitions. This is the default value.
PREFERRED_ALWAYS
Always use the
preferredEntry
. ThepreferredEntry
is the primary replica of a cache entry that resides in the partition that contains the most nodes. If the number of nodes is equal between partitions, thepreferredEntry
is the cache entry that resides in the partition with the highest topology ID, which means that topology is more recent.PREFERRED_NON_NULL
Use the
preferredEntry
if it has a value (non-null). If thepreferredEntry
does not have a value, use the first entry defined inotherEntries
.REMOVE_ALL
Remove entries (key and value) from the cache if conflicts exist.
${CACHE_NAME}_STATE_TRANSFER_TIMEOUT
Sets the amount of time, in milliseconds, to wait for other cache instances in the cluster to transfer state to the cache. If other cache instances do not transfer state before the timeout occurs, the application throws an exception and aborts startup. The default value is
240000
(4 minutes).You must use a custom template to set this environment variable. It does not take effect if you set the state transfer timeout in the default Data Grid for OpenShift templates.
15.4.4. Security Domain
SECDOMAIN_NAME
-
Defines additional security domains. For example:
SECDOMAIN_NAME=myDomain
SECDOMAIN_PASSWORD_STACKING
-
Enables the password staking module and sets the
useFirstPass
option. The value istrue
orfalse
(default). SECDOMAIN_LOGIN_MODULE
-
Specifies a login module to use. The default value is
UsersRoles
. SECDOMAIN_USERS_PROPERTIES
-
Specifies the properties file that contains user definitions. The default value is
users.properties
. SECDOMAIN_ROLES_PROPERTIES
-
Specifies the properties file that contains role definitions. The default value is
roles.properties
.
15.4.5. Endpoints
Clients can access Data Grid via REST, Hot Rod, and Memcached endpoints that you define in the cache configuration.
Clients that run in the same project as Data Grid for OpenShift can access the cache via Hot Rod and receive a full cluster view. These clients can also use consistent hashing capabilities.
However, when clients run in a different project to Data Grid for OpenShift, they need to access the Data Grid cluster using an OpenShift service that exposes the Hot Rod endpoint externally. Depending on your network configuration, clients might not have access to some pods and must use BASIC
client intelligence. In these cases, clients might require extra network hops to access data, which can increase network latency.
External access to clients running in OpenShift requires routes with passthrough encryption termination. Clients must also use BASIC
client intelligence and the fully qualified domain name as a TLS/SNI host name. Alternatively, you can expose the Data Grid cluster behind a Load Balancer service that is externally available.
Configure endpoints with the following environment variables:
INFINISPAN_CONNECTORS
-
Defines a comma-separated list of connectors to configure. Defaults to
hotrod
,memcached
,rest
. If authorization or authentication is enabled on the cache then you should removememcached
because this protocol is inherently insecure. MEMCACHED_CACHE
-
Sets the cache name for the Memcached connector. Defaults to
memcached
if you do not specify a cache name with theCACHE_NAMES
environment variable. HOTROD_SERVICE_NAME
Defines the name of the OpenShift service for the external Hot Rod connector.
The external Hot Rod connector is available with deployment configuration templates only if you define this environment variable.
cache-service
anddatagrid-service
do not use the external Hot Rod connector.For example, if you set
HOTROD_SERVICE_NAME=DATAGRID_APP_HOTROD
the Hot Rod external connector returnsDATAGRID_APP_HOTROD:11333
.REST_STORE_AS_STRING
Specifies if Data Grid saves entries as Java strings when written to the cache via the REST API. The value is
true
orfalse
(default).Set the value to
true
if you are upgrading the image from a previous version and plan to read persisted cache entries.NoteData Grid version 7.1 and earlier: When you write entries to the cache through the REST endpoint, Data Grid stores them as Java strings.
Data Grid version 7.2 and later: Data Grid stores cache entries as
bytes[]
to enable data interoperability between clients and protocols.If you upgrade Data Grid for OpenShift images to version 7.2 or later, Data Grid returns null values when you attempt to read cache entries that are persisted to a data store. To resolve the null values, set
REST_STORE_AS_STRING=true
.