How to show the complete cache size for embedded caches in RHDG?

Solution Verified - Updated -

Environment

  • Red Hat Data Grid (RHDG, JDG)
    • 6
    • 7

Issue

  • If for an embedded replicated cache (library mode) the method .size() is called different nodes return different sizes, is that cache inconsistent? Where are the missing entries?
  • Is it possible to show the full size of a cache nevertheless whether eviction is enabled or not?
  • Is it possible to count the complete size of a cache if it is distributed?
  • Why is an invocation of cache.size() slower after migration from 6.x to 7.x

Resolution

Replicated caches

  • In this case each node of a cluster contain all entries, unless there is a ongoing replication for changes.
  • The size() operation will return the number of current entries in the cache, no matter whether they are evicted or not.

Note : This operation is very expensive as it has to access the store to count the evicted entries. It should be used for testing or monitor purpose only.

  • If the size should be only counted for entries which are current in memory the advanced API can be used with flags as below :
cache.getAdvancedCache().withFlags(Flag.SKIP_CACHE_LOAD).size()
  • This operation does not access the store, but notice in this case the number of entries can be different on every node as each node might evict the entries different.

Distributed caches

  • Entries are counted in the same manner as a replicated cache, but in this case the node does not have all entries because of the ConsistentHash distribution. The number of entries will be different on each node.

  • To count all entries of the cache the system property infinispan.accurate.bulk.ops can be set, in this case the size is counted for all entries in the clustered cache.

Warning : This operation is pretty expensive as it will count all entries for all available nodes which need to remotely invoke operations on all nodes and count the real number of entries. It is not recommended to be used in a production environment.

Root Cause

The 6.x versions are kind of incorrect/incomplete in counting cache entries, size() provides the size of the local, internal data container only. This does not take into account

  • in-fly transactions, entries stored in a cache store, or remote entries.
  • entries that have expired but haven't yet been removed from the internal container
  • as well as entries in the L1 cache if L1 is enabled along with distribution as a clustering mode.
  • persisted and/or passivated entries

This is fixed with RHDG 7 and the size will calculate all cache entries in a correct way.
But this will also lead to some performance drawbacks.

  • size() will be a global operation and need to check all remote instances
  • iteration is needed to check expiration to filter mortal entries
  • need to also run a query for persistent cache store

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.