Chapter 2. Known and fixed issues

Learn about known issues in Data Grid and find out which issues are fixed.

2.1. Known Issues for Data Grid

For issues that affect Data Grid clusters that you manage with Data Grid Operator, you should refer to the Data Grid Operator 8.4 release notes.

JGroups address defaults to an external IP

Issue: JDG-6053

Description: In bare metal deployments, when JGroups bind to an external IP without authentication configured by default, the connection is not secure, posing a risk of unauthorized access or manipulation.

Workaround: Secure the connection in one of the following ways:

  • Configure JGroups security to control the network so only authorized nodes can join. For more information see, Encrypting cluster transport.
  • Use the -Djgroups.bind.address=<internal-network> parameter when starting Data Grid Server to set the JGroups address to secure internal network.

Inconsistent transactions when network partitions occur

Issue: JDG-3935

Description: In scenarios where a network partition occurs for a Data Grid cluster, transactions are rolled back after the partition is healed.

Workaround: There is no workaround for this issue.

Data Grid conflict resolution performance

Issue: JDG-3636

Description: In some test cases, Data Grid partition handling functionality took longer than expected to perform conflict resolution.

Workaround: There is no workaround for this issue.

2.2. Fixed in Data Grid 8.4.7

Data Grid 8.4.7 includes the following notable fixes:

  • JDG-6894 Response time between nodes is not correctly computed
  • JDG-6811 SSLSocket blocks on close()
  • JDG-6872 Avoid individual GET by Key in distributed query aggregation
  • JDG-6680 Attempt to iterate over cluster cache second time fails with segment already completed
  • JDG-6732 Cache startup failures on individual nodes can cause other caches to enter DEGRADED mode on restart
  • JDG-6652 Cache stats in REST should be clustered stats
  • JDG-6736 Check if MetricsRegistry is enabled before trying to register metrics
  • JDG-6591 Data Grid 8 automatic load cache from Oracle database limitation numeric values longer than 10
  • JDG-6824 Embedded JCache listener does not correctly return the previous value
  • JDG-6861 HR Rolling Upgrades fail with ___protobuf_metadata cache
  • JDG-6840 Hot Rod Client RemoteCache#computeAsync() can lose the result of remapping processing when executed concurrently against a new entry
  • JDG-6751 Keycloak integration broken with ACL call
  • JDG-6738 NullPointerException in case of failover brings to HR Client hanging
  • JDG-6678 Put operation fails on cache with remote-store configured with uri property
  • JDG-6847 SSLContext ignored by RemoteStore
  • JDG-6889 Shared Store Publisher should fetch data from remote nodes with SKIP_CACHE_LOAD
  • JDG-6793 NullPointerException in MBeans without getters

2.3. Fixed in Data Grid 8.4.6

Data Grid 8.4.6 includes the following notable fixes:

  • JDG-5435 Avoid executing hybrid queries if count targets aggregable fields
  • JDG-6585 IRAC state transfer might lead to data loss
  • JDG-6593 Soft Index File Store (SIFS) ExpirationCompaction on logFile might fail to update the index
  • JDG-6636 Index might receive NegativeArraySizeException
  • JDG-6586 PreferConsistencyStrategy throws UnsupportedOperationException with zero-capacity nodes
  • JDG-6540 Reduce OpenTelemetry log spam by logging the DEBUG message only once, when OpenTelemetry is not found on the client
  • JDG-6301 Deadlock occurs during expiration of entries with local cache and JDBC cache store
  • JDG-6489 Incorrect memory management when using Control Groups version 2
  • JDG-6644 SIFS index corruption might occur if a shutdown is initiated while compactor is processing a file
  • JDG-6682 ClassCastException occurs when executing PutAll operations during rebalancing
  • JDG-6681 JDBC PooledConnectionFactory retains stale connections after database restart
  • JDG-6633 Backup write operations should ignore previous value when replicating from the primary owner
  • JDG-6635 SIFS expiration compaction writes WARN message for concurrent write to logFile
  • JDG-6655 Remote cache store ignores defined configuration properties unless the blocking property is explicitly set
  • JDG-4534 Remote site goes offline due to an issue during the initial state transfer for cross-site replication
  • JDG-6581 Near Cache with Bloom filter enabled doesn’t work properly when configured with JBoss marshalling
  • JDG-6548 Exceptional HEAD requests return a non-zero Content-Length header
  • JDG-6458 Data Grid Console shows an error for non-admin users accessing a cache to which they have permission
  • JDG-6653 Hot Rod client repeatedly retries iteration operation against a failed server even after removal from topology
  • JDG-6634 SIFS removing a nonexistent key causes index to be incorrect

2.4. Fixed in Data Grid 8.4.4

Data Grid 8.4.4 includes the following notable fixes:

  • JDG-6447 Hot Rod query exception is missing stack trace to the caller
  • JDG-6446 Exclusion of 'byteman-bmunit' from JGroups dependencies
  • JDG-6443 Calling a Prometheus metrics endpoint collects both local and other node’s metrics
  • JDG-6428 Queued Hot Rod client operations should retry if the connection pool terminates
  • JDG-6416 InvalidatedNearRemoteCache can persist stale values in case of concurrent access
  • JDG-6391 RemoteStore leaks out netty event loop client threads
  • JDG-6389 An exception in Hibernate Search occurs when using search#hitCount() if the returned SearchResultTotal is not exact
  • JDG-6382 Startup failure occurs when dynamically created caches that use dynamic RBAC are enabled
  • JDG-6378 Query module deadlock caused by multiple threads waiting on the Class initialization monitor
  • JDG-6358 State transfer never completes
  • JDG-6278 Cache template parsing fails when the template is defined after the configuration element
  • JDG-6256 Circular actions during indexing might throw an exception
  • JDG-6161 Attributes with nested properties in the JGroups configuration are not applied

2.5. Fixed in Data Grid 8.4.3

Data Grid 8.4.3 includes the following notable fixes:

  • JDG-6197 REST API fails to load the data distribution for local secured caches in Data Grid Console
  • JDG-6194 The JSON configuration parser does not replace properties when retrieving values by name
  • JDG-6160 It is not possible to override the tracing.propagation_enabled property using the HotRod properties
  • JDG-6125 Empty authorization roles serialized as JSON cannot be parsed
  • JDG-6095 Remote getCacheNames does not exclude secured caches based on user roles when listing available caches
  • JDG-6251 The CacheLoaderInterceptor leaves orphaned pending operations when encountering an availability exception
  • JDG-6235 Incorrect index statistics are displayed after clearing or rebuilding index
  • JDG-6231 The Compactor.java class fails to properly close files, resulting in heap accumulation
  • JDG-6229 Missing CacheUpdateConfigurationAttributeTask in the ServerAdminOperationsHandler
  • JDG-6228 Protobuf metadata cache fails to provide validation error messages when the Protobuf schema is replaced
  • JDG-6227 Deadlock during expiration of entries with local cache mode
  • JDG-6226 ManagedConnectionFactory DataSource is initialized at component start time
  • JDG-6223 Incorrect detection of IPv6 multihomed addresses by the Data Grid Server
  • JDG-6222 Failed cache creation with backup and merge policy
  • JDG-6218 NoClassDefFoundError when using JDK 17 and JBossMarshalling
  • JDG-6217 Missing infinispan-hibernate-cache-v62 in the Data Grid BOM file

2.6. Fixed in Data Grid 8.4.2

Data Grid 8.4.2 includes the following notable fixes:

  • JDG-6057 RESP endpoint cannot parse request larger than packet size
  • JDG-6015 Not possible to update Helm Chart deploy.security.batch on upgrade
  • JDG-5997 Changing index schema incompatibly, for example, making a non-sortable field sortable, results in ISPN014827
  • JDG-5988 Helm Chart: DNS discovery fails when pods are not marked as ready
  • JDG-5972 REST API blocks key and entry retrieval, causing non-blocking threads to wait for stream elements
  • JDG-5943 Adding entry with putAll and replaceWithVersion causes a timeout
  • JDG-5927 SQL Cache store initiation fails on SQL type CHAR
  • JDG-5811 Data Grid Console: missing default value for indexing startup mode in the Cache creation wizard
  • JDG-5732 After the shutdown cluster command, caches remain accessible before all nodes rejoin the cluster
  • JDG-3967 Missing error message about incomplete restart of a cluster after the shutdown cluster command
  • JDG-5549 REST and CLI commands add schema definition for invalid files

2.7. Fixed in Data Grid 8.4.1

Data Grid 8.4.1 includes the following notable fixes:

  • JDG-5725 Creating Data Grid cluster backup over REST fails
  • JDG-5726 Retrieving Data Grid Server configuration with GET request over the browser fails
  • JDG-5724 Retrieving diagnostic reports for multiple Data Grid Server instances fails
  • JDG-5825 GET command blocks text/plain caches
  • JDG-5802 Data Grid Console: Queries return error when they run after the entries expire
  • JDG-5842 Data Grid Console: Metrics for misses and retrievals are updated only after the page is reloaded
  • JDG-5831 Data Grid Console: Cache creation wizard let users create indexed caches with invalid media type
  • JDG-5832 Data Grid Console: Data distribution chart resizing issue
  • JDG-5625 Updates to the Data Grid Console logo
  • JDG-5686 Thread Locking when doing Access Control List (ACL) permission check
  • JDG-5805 Getting statistics with AdvancedCache.getStats() fails
  • JDG-5801 The default value of datasource connection pool maxSize causes NullPointerException (NPE)
  • JDG-5799 JSON characters should be escaped with two equality checks
  • JDG-5800 Using Java 17 with Red Hat Single Sign-On configured on a Data Grid cluster throws an exception
  • JDG-5821 Incompatibility between Data Grid Server 8.3 and Hot Rod client 8.4
  • JDG-5790 CacheInputEntryStream escapes the value for every byte in the value

2.8. Fixed in Data Grid 8.4.0

Data Grid 8.4.0 includes the following notable fixes:

  • JDG-4863 Option to delete Protobuf schema from Data Grid CLI or the Console
  • JDG-5439 Topology update with multithread client is not secure
  • JDG-5320 Missing support for embedded query with Protobuf-encoded caches
  • JDG-5316 SoftIndexFileStore increases the size of the data folder
  • JDG-5318 SoftIndexFileStore loads incorrect size of the data that is stored
  • JDG-5221 ArrayIndexOutOfBoundsException when using indexed field in a remote query with startOffset=10 and maxResults=-1
  • JDG-5325 RELAY2 does not reconnect after a UnknownHostException
  • JDG-5219 Invalid Protobuf schema causes PUT operation to fail without specific error message
  • JDG-5253 Creating two caches that have name difference only in special characters and have statics enabled results in second cache being degraded
  • JDG-4766 Client should not close the server connection after timeout expires
  • JDG-5544 Potential deadlock in index writer threads under heavy load
  • JDG-5542 REST keys call fails if a cache has a value media-type that is not convertable to JSON
  • JDG-5553 Bulk operations do not work properly with embedded key SQL store
  • JDG-5641 Stores that don’t support expiration should not be configured to have expiration
  • JDG-5575 If a Protobuf entry contains number with fractional portion it fails to persist this value for MicrosoftSQL 2019 server or Postgresql 13
  • JDG-5531 YamlConfigurationReader throws NullPointerException for valid cache configuration
  • JDG-5622 NullPointerException when computing Netty global connections
  • JDG-5624 SoftIndexFileStore does not remove log files with expiration
  • JDG-5327 ConcurrentModificationException for a joining node with CrossSiteReplication
  • JDG-5232 Invalid YAML representation of a cache
  • JDG-5442 Configuring memory max-size as non-byte value causes failure on CacheManager restart
  • JDG-5411 If a security realm name contains special characters the HTTP library fails to parse it
  • JDG-3394 Hot Rod client throws an exception if a remote-task is executed in TaskExecutionMode.ALL_NODES
  • JDG-5629 A site sends empty batches with IRAC to the remote sites

2.9. Host system and dependency issues

In some cases Data Grid deployments can encounter errors that are caused by the host system or external dependency. This section provides details about any such known issues as well as troubleshooting and workaround procedures.

Nashorn JavaScript engine

If your Data Grid Server uses JavaScript to automate tasks, you must install the Nashorn JavaScript engine to ensure that these scripts can run on Data Grid 8.4. This is because OpenJDK 17 has removed support for the Nashorn JavaScript engine, its APIs, and the jjs tool.

For a bare-metal Data Grid Server, you can install Nashorn from the Maven central repository by issuing the following command in your Data Grid CLI: 

bin/cli.sh install org.openjdk.nashorn:nashorn-core:15.4 \
                                 org.ow2.asm:asm:7.3.1 \
                                 org.ow2.asm:asm-util:7.3.1

On OpenShift, you can create an Infinispan Custom Resource (CR) that sets the Data Grid Operator to install Nashorn for your Data Grid cluster. For example: 

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  name: infinispan
spec:
  replicas: 2
  dependencies:
    artifacts:
      - maven: org.openjdk.nashorn:nashorn-core:15.4
      - maven: org.ow2.asm:asm:7.3.1
      - maven: org.ow2.asm:asm-util:7.3.1
  service:
    type: DataGrid

TLS on Red Hat Enterprise Linux 7

RHEL 7 provides a version of the OpenSSL library that does not yet offer support for TLSv1.3. However Data Grid Server 8.4 enables TLSv1.3 and TLSv1.2 by default, which causes errors with client connections for encrypted Hot Rod and REST endpoints.

Data Grid Server also logs messages such as the following: 

WARN  [org.infinispan.HOTROD] ISPN004098: Closing connection due to transport error
org.infinispan.client.hotrod.exceptions.TransportException:: ISPN004077:
Closing channel due to error in unknown operation.

If you install Data Grid Server on RHEL 7 you should use the native Java SSL library by disabling OpenSSL with the following JVM option: 

-Dorg.infinispan.openssl=false

Additional resources