Chapter 15. Reference

15.1. Probes

Data Grid for OpenShift provides a liveness probe and a readiness probe to perform container health checks.

Liveness probe

The liveness probe is located in the container at /opt/datagrid/bin/livenessProbe.sh.

The liveness probe tests server status and restarts pods if the following events occur:

  • Data Grid for OpenShift boots with errors.
  • Custom deployment configurations do not successfully deploy.
  • One or more caches fail to instantiate, which usually occurs if the cache configuration is not valid.
Readiness probe

The readiness probe is located in the container at /opt/datagrid/bin/readinessProbe.sh.

The readiness probe determines if the pod is ready to receive requests and checks Data Grid cache level MBeans to ensure the following:

  • All cache instances are initialized.
  • All cache instances have joined the cluster, if using distributed cache mode.
  • Initial state transfer is complete. If state transfer is in progress, the pod is not marked as ready.
  • All cache instances in the Cache manager are running.

To configure custom deployments to use the liveness probe and readiness probe, run the following commands:

$ oc set probe dc/datagrid \
  --readiness \
  -- /bin/bash \
  -c /opt/datagrid/bin/readinessProbe.sh

$ oc set probe dc/datagrid \
  --liveness \
  -- /bin/bash \
  -c /opt/datagrid/bin/livenessProbe.sh

15.2. Ports

Data Grid for OpenShift uses the following ports:

Port NumberProtocolUse

8080

TCP

HTTP Access

8443

TCP

HTTPS Access

8888

TCP

JGroups Ping

11222

TCP

Hot Rod Access

Data Grid deployment configuration templates also use the following ports:

Port NumberProtocolUse

11211

TCP

Memcached Access

11333

TCP

External Hot Rod Access

8778

TCP

Remote JMX Access

Note

If you set the HOTROD_SERVICE_NAME environment variable with the deployment configuration templates, the Hot Rod external connector returns ${service_name}:11333 for the endpoint.

15.3. Administration Console

The Data Grid Administration Console is not supported on Red Hat OpenShift.

To monitor events and get statistics for Data Grid for OpenShift clusters, you should use Prometheus. See Setting Up Monitoring.

You can also use the Management CLI to troubleshoot Data Grid for OpenShift pods. See Launching the Command Line Interface.

15.4. Environment Variables

15.4.1. Monitoring and Logging

AB_PROMETHEUS_ENABLE

Allows you to collect JMX metrics to monitor and analyze Data Grid. The default value is false. Set the value to true to enable monitoring with the default Prometheus agent.

The Prometheus Operator must be installed and running. You must also Set Up Monitoring.

AB_PROMETHEUS_JMX_EXPORTER_PORT
Defines the port on which Data Grid publishes JMX metrics. The default is 9779.
LOGGING_CATEGORIES

Adjusts the categories and levels for which Data Grid captures log messages, for example:

$ LOGGING_CATEGORIES=org.infinipan.core=WARN,org.infinispan.config=DEBUG

Log categories correspond to Java package names and use standard log levels: TRACE, DEBUG, INFO, WARN, ERROR, and FATAL.

Important

If you specify LOGGING_CATEGORIES, Data Grid does not set the following default loggers. Instead, Data Grid uses the default level of INFO for all packages that you do not explicitly specify with LOGGING_CATEGORIES.

Table 15.1. Default Loggers

CategoryLevel

com.arjuna

WARN

sun.rmi

WARN

org.jboss.as.config

DEBUG

15.4.2. Containers

USERNAME

Creates a user in the security realm who is authorized to access data.

Note

By default, the Hot Rod endpoint uses the ApplicationRealm security realm and the REST endpoint uses the jdg-openshift security realm.

PASSWORD
Specifies the password for the user.
DATAGRID_SPLIT

Determines if the data directory for each node should be split in a mesh. The value is true or false (default).

If you set the value to true, you must also configure a persistent volume mounted on /opt/datagrid/standalone/partitioned_data.

JAVA_OPTS_APPEND

Appends options to the JAVA_OPTS environment variable on startup.

For example, JAVA_OPTS_APPEND=-Dfoo=bar

OPENSHIFT_KUBE_PING_LABELS

Specifies the clustering labels selector.

For example, OPENSHIFT_KUBE_PING_LABELS=application=eap-app

OPENSHIFT_KUBE_PING_NAMESPACE
Specifies the clustering project namespace.
TRANSPORT_LOCK_TIMEOUT

Sets the time to wait to acquire a distributed lock. The default value is 240000.

Data Grid uses a distributed lock to maintain a coherent transaction log during state transfer or rehashing, which means that only one cache can perform state transfer or rehashing at a time. This constraint is in place because more than one cache could be involved in a transaction.

15.4.3. Caches

Creating and configuring caches with cache-service and datagrid-service

Do not use environment variables to create and configure caches with cache-service or datagrid-service.

These environment variables are intended for use with the deployment configuration templates only and are deprecated.

You should remote create caches with cache-service and datagrid-service dynamically through the Hot Rod endpoint. For more information, see Remotely Creating Caches.

CACHE_NAMES

Defines cache instances in your configuration.

If you are using the Data Grid deployment configuration templates and you do not define any cache instances, the launch script adds a default distributed cache in SYNC mode.

Tip

Give each cache instance in your configuration a unique name. Use underscore characters (_) and descriptive labels to help you distinguish between cache instances. This ensures that you do not have conflicts when applying cache-specific configuration.

For example, CACHE_NAMES=addressbook,addressbook_indexed

CACHE_CONTAINER_START

Configures how the cache container starts. Specify one of the following:

  • LAZY Starts the cache container when requested by a service or deployment. This is the default.
  • EAGER Starts the cache container when the server starts.
CACHE_CONTAINER_STATISTICS
Configures the cache container to collect statistics. The value is true (default) or false. You can set the value to false to improve performance.
DEFAULT_CACHE
Sets the default cache for the cache container.

15.4.3.1. Cache Container Security Configuration

CONTAINER_SECURITY_CUSTOM_ROLE_MAPPER_CLASS

Specifies the class of the custom principal to role mapper.

For example, CONTAINER_SECURITY_CUSTOM_ROLE_MAPPER_CLASS=com.acme.CustomRoleMapper

CONTAINER_SECURITY_ROLE_MAPPER

Sets a role mapper for this cache container with the following values:

  • identity-role-mapper Uses the Principal name as the role name. This is the default role mapper if you do not specify one and use the CONTAINER_SECURITY_ROLES environment variable to define role names.
  • common-name-role-mapper Uses the Common Name (CN) as the role name if the Principal name is a Distinguished Name (DN). For example, the DN cn=managers,ou=people,dc=example,dc=com is mapped to the manager role name.
  • cluster-role-mapper Uses the ClusterRegistry to store Principal name to role mappings.
  • custom-role-mapper Takes the fully-qualified class name of an implementation of the org.infinispan.security.impl.PrincipalRoleMapper interface.
CONTAINER_SECURITY_ROLES

Defines role names and assigns permissions to them.

For example, CONTAINER_SECURITY_ROLES=admin=ALL,reader=READ,writer=WRITE

15.4.3.2. Named Cache Configuration

You specify the cache name as a prefix for the environment variable in capital letters (all caps) otherwise the configuration does not take effect.

For example, you create two separate cache instances: MyCache and MYCACHE. You then set MyCache_CACHE_TYPE=replicated to configure the MyCache instance. This configuration does not take effect. However, if you set MYCACHE_CACHE_TYPE=replicated the configuration takes effect for both the MyCache and MYCACHE instances.

${CACHE_NAME}_CACHE_TYPE
Determines whether this cache should be distributed or replicated. You can specify either distributed (default) or replicated.
${CACHE_NAME}_CACHE_START

Configures how the cache starts. Specify one of the following:

  • LAZY Starts the cache when requested by a service or deployment. This is the default.
  • EAGER Starts the cache when the server starts.
${CACHE_NAME}_CACHE_BATCHING
Enables invocation batching for this cache. The value is true or false (default).
${CACHE_NAME}_CACHE_STATISTICS
Configures the cache to collect statistics. The value is true (default) or false. You can set the value to false to improve performance.
${CACHE_NAME}_CACHE_MODE

Sets the clustered cache mode. Specify one of the following:

  • ASYNC for asynchronous operations.
  • SYNC for synchronous operations.
${CACHE_NAME}_CACHE_QUEUE_SIZE
Sets the threshold at which the replication queue is flushed when the cache is in ASYNC mode. The default value is 0 (flushing is disabled).
${CACHE_NAME}_CACHE_QUEUE_FLUSH_INTERVAL
Specifies the wakeup time, in milliseconds, for the thread that flushes the replication queue in ASYNC mode. The default value is 10.
${CACHE_NAME}_CACHE_REMOTE_TIMEOUT
Specifies the timeout, in milliseconds, to wait for acknowledgement when making remote calls in SYNC mode. If the timeout is reached, the remote call is aborted and an exception is thrown. The default value is 17500.
${CACHE_NAME}_CACHE_OWNERS
Specifies the number of cluster-wide replicas for each cache entry. The default value is 2.
${CACHE_NAME}_CACHE_SEGMENTS
Specifies the number of hash space segments per cluster. The recommended value is 10 * cluster size. The default value is 80.
${CACHE_NAME}_CACHE_L1_LIFESPAN
Specifies the maximum lifespan, in milliseconds, of an entry placed in the L1 cache. The default value is 0 (L1 is disabled).
${CACHE_NAME}_CACHE_MEMORY_EVICTION_TYPE

Defines the maximum limit for entries in the cache. You can set the following values:

  • COUNT Measures the number of entries in the cache. When the count exceeds the maximum, Data Grid evicts unused entries.
  • MEMORY Measures the amount of memory that all entries in the cache take up. When the total amount of memory exceeds the maximum, Data Grid evicts unused entries.
${CACHE_NAME}_CACHE_MEMORY_STORAGE_TYPE

Defines how Data Grid stores entries in the cache. You can set the following values:

Storage TypeDescriptionEviction TypePolicy

object

Stores entries as objects in the Java heap. This is the default storage type.

COUNT

TinyLFU

binary

Stores entries as bytes[] in the Java heap.

COUNT or MEMORY

TinyLFU

off-heap

Stores entries as bytes[] in native memory outside the Java.

COUNT or MEMORY

LRU

${CACHE_NAME}_CACHE_MEMORY_EVICTION_SIZE

Configures the size of the cache before eviction starts. Set the value to a number greater than zero.

  • For COUNT, the size is the maximum number of entries the cache can hold before eviction starts.
  • For MEMORY, the size is the maximum number of bytes the cache can take from memory before eviction starts. For example, a value of 10000000000 is 10 GB.

    Try different cache sizes to determine the optimal setting. A cache size that is too large can cause Data Grid to run out of memory. At the same time, a cache size that is too small wastes available memory.

    Note

    If you configure a JDBC store, passivation is automatically enabled when you set the eviction size to a value that is greater than zero.

${CACHE_NAME}_CACHE_MEMORY_EVICTION_STRATEGY

Controls how Data Grid performs eviction. You can set the following values:

StrategyDescription

NONE

Data Grid does not evict entries. This is the default setting unless you configure eviction.

REMOVE

Data Grid removes entries from memory so that the cache does not exceed the configured size. This is the default setting when you configure eviction.

MANUAL

Data Grid does not perform eviction. Eviction takes place manually by invoking the evict() method from the Cache API.

EXCEPTION

Data Grid does not write new entries to the cache if doing so would exceed the configured size. Instead of writing new entries to the cache, Data Grid throws a ContainerFullException.

${CACHE_NAME}_CACHE_MEMORY_OFF_HEAP_ADDRESS_COUNT

Specifies the number of pointers that are available in the hash map to prevent collisions when using OFFHEAP storage. Preventing collisions in the hash map improves performance.

Set the value to a number that is greater than the number of cache entries. By default address-count is 2^20, or 1048576. The parameter is always rounded up to a power of 2.

${CACHE_NAME}_CACHE_EXPIRATION_LIFESPAN
Specifies the maximum lifespan, in milliseconds, of a cache entry, after which the entry is expired cluster-wide. The default value is -1 (entries never expire).
${CACHE_NAME}_CACHE_EXPIRATION_MAX_IDLE
Specifies the maximum idle time, in milliseconds, that cache entries are maintained in the cache. If the idle time is exceeded, then the entry is expired cluster-wide. The default value is -1 (expiration is disabled).
${CACHE_NAME}_CACHE_EXPIRATION_INTERVAL
Specifies the interval, in milliseconds, between runs to purge expired entries from memory and any cache stores. The default value is 5000. Set -1 to disable expiration.
${CACHE_NAME}_JDBC_STORE_TYPE

Sets the type of JDBC store to configure. You can set the following values:

  • string
  • binary
${CACHE_NAME}_JDBC_STORE_DATASOURCE

Defines the jndiname of the datasource.

For example, MYCACHE_JDBC_STORE_DATASOURCE=java:jboss/datasources/ExampleDS

${CACHE_NAME}_KEYED_TABLE_PREFIX
Defines the prefix prepended to the cache name used when composing the name of the cache entry table. The defaule value is ispn_entry.
${CACHE_NAME}_CACHE_INDEX

Sets the indexing mode of the cache. You can set the following values:

  • NONE This is the default.
  • LOCAL
  • ALL
${CACHE_NAME}_INDEXING_PROPERTIES

Specifies a comma-separated list of properties to pass to the indexing system.

For example, MYCACHE_INDEXING_PROPERTIES=default.directory_provider=ram

${CACHE_NAME}_CACHE_SECURITY_AUTHORIZATION_ENABLED
Enables authorization checks for this cache. The value is true or false (default).
${CACHE_NAME}_CACHE_SECURITY_AUTHORIZATION_ROLES

Sets the roles required to access this cache.

For example, MYCACHE_CACHE_SECURITY_AUTHORIZATION_ROLES=admin, reader, writer

${CACHE_NAME}_CACHE_PARTITION_HANDLING_WHEN_SPLIT

Configures the strategy for handling partitions between nodes in a cluster when network events isolate nodes from each other. Partitions function as independent clusters until Data Grid merges cache entries to re-form a single cluster. You can set the following values:

Partition Handling StrategyDescription

ALLOW_READ_WRITES

Nodes from any partition can read or write cache entries. This is the default value.

DENY_READ_WRITES

Nodes enter degraded mode if:

* One or more hash space segments in the partition have no owners. The owners are the number of cluster-wide replicas for cache entries.

* The partition has less than half the nodes from the most recent stable cluster topology.

In degraded mode, only nodes in the same partition can read or write cache entries. All owners, or copies, for a cache entry must exist on the same partition, otherwise the read or write operation fails with an AvailabilityException.

ALLOW_READS

Nodes enter degraded mode similarly to the DENY_READ_WRITES strategy. Nodes from any partition can read cache entries.

In degraded mode, only nodes in the same partition can write cache entries. All owners, or copies, for a cache entry must exist on the same partition, otherwise the write operation fails with an AvailabilityException.

${CACHE_NAME}_CACHE_PARTITION_MERGE_POLICY

Configures how Data Grid resolves conflicts between cache entries when merging partitions. You can set the following values:

Merge PolicyDescription

NONE

Do not resolve conflicts when merging partitions. This is the default value.

PREFERRED_ALWAYS

Always use the preferredEntry. The preferredEntry is the primary replica of a cache entry that resides in the partition that contains the most nodes. If the number of nodes is equal between partitions, the preferredEntry is the cache entry that resides in the partition with the highest topology ID, which means that topology is more recent.

PREFERRED_NON_NULL

Use the preferredEntry if it has a value (non-null). If the preferredEntry does not have a value, use the first entry defined in otherEntries.

REMOVE_ALL

Remove entries (key and value) from the cache if conflicts exist.

${CACHE_NAME}_STATE_TRANSFER_TIMEOUT

Sets the amount of time, in milliseconds, to wait for other cache instances in the cluster to transfer state to the cache. If other cache instances do not transfer state before the timeout occurs, the application throws an exception and aborts startup. The default value is 240000 (4 minutes).

You must use a custom template to set this environment variable. It does not take effect if you set the state transfer timeout in the default Data Grid for OpenShift templates.

15.4.4. Security Domain

SECDOMAIN_NAME
Defines additional security domains. For example: SECDOMAIN_NAME=myDomain
SECDOMAIN_PASSWORD_STACKING
Enables the password staking module and sets the useFirstPass option. The value is true or false (default).
SECDOMAIN_LOGIN_MODULE
Specifies a login module to use. The default value is UsersRoles.
SECDOMAIN_USERS_PROPERTIES
Specifies the properties file that contains user definitions. The default value is users.properties.
SECDOMAIN_ROLES_PROPERTIES
Specifies the properties file that contains role definitions. The default value is roles.properties.

15.4.5. Endpoints

Clients can access Data Grid via REST, Hot Rod, and Memcached endpoints that you define in the cache configuration.

Clients that run in the same project as Data Grid for OpenShift can access the cache via Hot Rod and receive a full cluster view. These clients can also use consistent hashing capabilities.

However, when clients run in a different project to Data Grid for OpenShift, they need to access the Data Grid cluster using an OpenShift service that exposes the Hot Rod endpoint externally. Depending on your network configuration, clients might not have access to some pods and must use BASIC client intelligence. In these cases, clients might require extra network hops to access data, which can increase network latency.

External access to clients running in OpenShift requires routes with passthrough encryption termination. Clients must also use BASIC client intelligence and the fully qualified domain name as a TLS/SNI host name. Alternatively, you can expose the Data Grid cluster behind a Load Balancer service that is externally available.

Configure endpoints with the following environment variables:

INFINISPAN_CONNECTORS
Defines a comma-separated list of connectors to configure. Defaults to hotrod, memcached, rest. If authorization or authentication is enabled on the cache then you should remove memcached because this protocol is inherently insecure.
MEMCACHED_CACHE
Sets the cache name for the Memcached connector. Defaults to memcached if you do not specify a cache name with the CACHE_NAMES environment variable.
HOTROD_SERVICE_NAME

Defines the name of the OpenShift service for the external Hot Rod connector.

The external Hot Rod connector is available with deployment configuration templates only if you define this environment variable. cache-service and datagrid-service do not use the external Hot Rod connector.

For example, if you set HOTROD_SERVICE_NAME=DATAGRID_APP_HOTROD the Hot Rod external connector returns DATAGRID_APP_HOTROD:11333.

REST_STORE_AS_STRING

Specifies if Data Grid saves entries as Java strings when written to the cache via the REST API. The value is true or false (default).

Set the value to true if you are upgrading the image from a previous version and plan to read persisted cache entries.

Note

Data Grid version 7.1 and earlier: When you write entries to the cache through the REST endpoint, Data Grid stores them as Java strings.

Data Grid version 7.2 and later: Data Grid stores cache entries as bytes[] to enable data interoperability between clients and protocols.

If you upgrade Data Grid for OpenShift images to version 7.2 or later, Data Grid returns null values when you attempt to read cache entries that are persisted to a data store. To resolve the null values, set REST_STORE_AS_STRING=true.