Red Hat JBoss Data Grid is a distributed in-memory data grid, which provides the following capabilities:

The set of supported features, configurations, and integrations for Red Hat JBoss Data Grid (current and past versions) are available at the Supported Configurations page at https://access.redhat.com/articles/115883.

Red Hat JBoss Data Grid includes many components for Library and Remote Client-Server modes. A comprehensive (and up to date) list of components included in each of these usage modes and their versions is available in the Red Hat JBoss Data Grid Component Details page at https://access.redhat.com/articles/488833

The Red Hat JBoss Data Grid Performance Tuning Guide provides information about optimizing and configuring specific elements within the product in an attempt to improve the JBoss Data Grid implementation performance.

Due to each business case being different it is not possible to provide a "one size fits all" approach to tuning. Instead, this guide attempts to present various elements that have proven to be effective in increasing performance, along with potential values to begin testing for a user’s specific case. It is imperative that after each individual change performance be measured once again to isolate any improvements or negative effects; this approach allows a methodical approach to establishing a baseline of parameters.

When testing it is strongly recommended to use a workload that closely mirrors the expected production load. Using any other workload may result in performance differences between the testing and production environments.

Tuning a Java Virtual Machine (JVM) is a complex task because of the number of configuration options and changes with each new release.

The recommended approach for performance tuning Java Virtual Machines is to use as simple a configuration as possible and retain only the tuning that is beneficial, rather than all tweaks. A collection of tested configurations for various heap sizes are provided after the parameters are discussed.

In the majority of instances Xms and Xmx should be identical to prevent dynamic resizing of the heap, which will result in longer garbage collection periods.

The following sections, beginning in Garbage Collectors discuss common tunables for three different garbage collection policies.

For Library mode the server should have a minimum of 1 GB of RAM for a single JBoss Data Grid instance, and is configured using the JVM arguments on the command line.

For Client-Server mode the requirements are determined based on the configuration file in use:

It is strongly recommended to test each application, measuring throughput and collection times to determine if they are acceptable for the application in question.

JvmProcessMemory = JvmHeap + Metaspace + (ThreadStackSize * Number of Threads) + Jvm-native-c++-heap

Adjusting these values are discussed in Java Virtual Machine Settings.

The Jvm-native-c++-heap will vary based on the native threads and if any native libraries are used; however, for a default installation it is safe to assume this will use no more than 256 MB of memory.

The following configurations have been tested internally, and are provided as a baseline for customization. These configurations show various heap sizes, which allows users to find one appropriate for their environment to begin testing:

-server
-Xms8192m
-Xmx8192m
-XX:+UseLargePages
-XX:NewRatio=3
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:+DisableExplicitGC

-server
-Xmx32G
-Xms32G
-XX:+UseLargePages
-XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=70
-XX:MaxGCPauseMillis=3000
-XX:+DisableExplicitGC

-server
-Xmx64G
-Xms64G
-XX:+UseLargePages
-XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=70
-XX:MaxGCPauseMillis=3000
-XX:+DisableExplicitGC

A memory page is a fixed size, continuous block of memory and is used when transferring data from one storage medium to another, and to allocate memory. In some architectures, larger sized pages are available for improved memory allocation. These pages are known as large (or huge) pages.

The default memory page size in most operating systems is 4 kilobytes (kb). For a 32-bit operating system the maximum amount of memory is 4 GB, which equates to 1,048,576 memory pages. A 64-bit operating system can address 18 Exabytes of memory (in theory), resulting in a very large number of memory pages. The overhead of managing such a large number of memory pages is significant, regardless of the operating system.

Large memory pages are pages of memory which are significantly larger than 4 kb (usually 2 Mb). In some instances it is configurable from 2 MB to 2 GB, depending on the CPU architecture.

Large memory pages are locked in memory, and cannot be swapped to disk like normal memory pages. The advantage for this is if the heap is using large page memory it can not be paged or swapped to disk so it is always readily available. For Linux, the disadvantage is that applications must attach to it using the correct flag for the shmget() system call. Additionally, the proper security permissions are required for the memlock() system call. For any application that does not have the ability to use large page memory, the server behaves as if the large page memory does not exist, which can be a problem.

For additional information on page size refer to the Red Hat Enterprise Linux Configuring Hugetlb Huge Pages.

Page memory configuration to optimize Red Hat JBoss Data Grid’s performance must be implement at the operating system level and at the JVM level. The provided instructions are for the Red Hat Enterprise Linux operating system. Use both the operating system level and JVM level instructions for optimal performance.

As JBoss Data Grid is distributed data grid, adjusting the underlying network settings may improve performance. This chapter seeks to address the most common tunables used, and provide recommendations on their configuration.

In many environments packet loss may be caused by the buffer space not being large enough to receive all of the transmissions, resulting in packet loss and costly retransmissions. As with all tunables, it is important to test these settings with a workload that mirrors what is expected to determine an appropriate value for your environment.

The kernel buffers may be increased by following the below steps:

JGroups utilizes flow control for TCP connections to prevent fast senders from overflowing slower receivers. This process prevents packet loss by controlling the network transmissions, ensuring that the targets do not receive more information than they can handle.

Some network cards and switches also perform flow control automatically, resulting in a performance decrease due to duplicating flow control for TCP connections.

By default the maximum transmission unit (MTU) is 1500 bytes. Jumbo frames should be enabled when the MTU is larger than the default, or when smaller messages are aggregated to be larger than 1500 bytes. By enabling jumbo frames, more data is sent per ethernet frame. The MTU may be increased to a value up to 9000 bytes.

To enable jumbo frames add the following line to the configuration script of the network interface, such as /etc/sysconfig/network-scripts/ifcfg-eth0:

MTU=9000

The transmit queue determines how many frames are allowed to reside in the kernel transmission queue, with each device having its own queue. This value should be increased when a large number of writes will be expected over a short period of time, resulting in a potential overflow of the transmission queue.

To determine if overruns have occurred the following command may be executed against the device. If the value for overruns is greater than 0 then the transmission queue length should be increased:

ip -s link show $NIC

This value may be set per device by using the following command:

ip link set $NIC txqueuelen 5000

Multiple interfaces may be bound together to create a single, bonded, channel. Bonding interfaces in this manner allows two or more network interfaces to function as one, simultaneously increasing the bandwidth and providing redundancy in the event that one interface should fail. It is strongly recommended to bond network interfaces should more than one exist on a given node.

Full instructions on bonding are available in the Networking Guide, available in Red Hat Enterprise Linux’s Product Documentation.

Values returned by cache operations are referred to as return values. In Red Hat JBoss Data Grid, these return values remain reliable irrespective of which cache mode is employed and whether synchronous or asynchronous communication is used.

Marshalling is the process of converting Java objects into a format that is transferable over the wire. Unmarshalling is the reversal of this process where data read from a wire format is converted into Java objects.

Red Hat JBoss Data Grid uses marshalling and unmarshalling to:

Red Hat JBoss Data Grid uses the JBoss Marshalling Framework to marshall and unmarshall Java POJOs. Using the JBoss Marshalling Framework offers a significant performance benefit, and is therefore used instead of Java Serialization. Additionally, the JBoss Marshalling Framework can efficiently marshall Java POJOs, including Java classes.

The Java Marshalling Framework uses high performance java.io.ObjectOutput and java.io.ObjectInput implementations compared to the standard java.io.ObjectOutputStream and java.io.ObjectInputStream.

Instead of using the default Marshaller, which may be slow with payloads that are unnecessarily large, objects may implement java.io.Externalizable so that a custom method of marshalling/unmarshalling classes is performed. With this approach the target class may be created in a variety of ways (direct instantiation, factory methods, reflection, etc.) and the developer has complete control over using the provided stream.

import org.infinispan.marshall.AdvancedExternalizer;

public class Book {

   final String name;
   final String author;

   public Book(String name, String author) {
      this.name = name;
      this.author = author;
   }

   public static class BookExternalizer implements AdvancedExternalizer<Book> {
      @Override
      public void writeObject(ObjectOutput output, Book book)
            throws IOException {
         output.writeObject(book.name);
         output.writeObject(book.author);
      }

      @Override
      public Person readObject(ObjectInput input)
            throws IOException, ClassNotFoundException {
         return new Person((String) input.readObject(), (String) input.readObject());
      }

      @Override
      public Set<Class<? extends Book>> getTypeClasses() {
         return Util.<Class<? extends Book>>asSet(Book.class);
      }

      @Override
      public Integer getId() {
         return 2345;
      }
   }
}

Once the writeObject() and readObject() methods have been implemented the Externalizer may be linked up with the classes they externalize; this is accomplished with the getTypeClasses() method seen in the above example.

In addition, a positive identifier must be defined as seen in the getId() method above. This value is used to identify the Externalizer at runtime. A list of values used by JBoss Data Grid, which should be avoided in custom Externalizer implementations, may be found at JBoss Data Grid Externalizer IDs.

<cache-container>
  <serialization>
    <advanced-externalizer class="Book$BookExternalizer"/>
  </serialization>
</cache-container>

GlobalConfigurationBuilder builder = ...
builder.serialization()
   .addAdvancedExternalizer(new Book.BookExternalizer());

The following values are used as Externalizer IDs inside the Infinispan based modules or frameworks, and should be avoided while implementing custom marshallers.

Java Management Extension (JMX) is a Java based technology that provides tools to manage and monitor applications, devices, system objects, and service oriented networks. Each of these objects is managed, and monitored by MBeans.

JMX is the de facto standard for middleware management and administration. As a result, JMX is used in Red Hat JBoss Data Grid to expose management and statistical information.

Management in Red Hat JBoss Data Grid instances aims to expose as much relevant statistical information as possible. This information allows administrators to view the state of each instance. While a single installation can comprise of tens or hundreds of such instances, it is essential to expose and present the statistical information for each of them in a clear and concise manner.

In JBoss Data Grid, JMX is used in conjunction with JBoss Operations Network (JON) to expose this information and present it in an orderly and relevant manner to the administrator.

By default JMX is enabled locally on each JBoss Data Grid server, and no further configuration is necessary to connect via JConsole, VisualVM, or other JMX clients that are launched from the same system.

To enable remote connections it is necessary to define a port for the JMX remote agent to listen on. When using OpenJDK this behavior is defined with the com.sun.management.jmxremote.port parameter. In addition, it is recommended to secure the remote connection when this is used in a production environment.

## Default configuration, 1.3GB heap
JAVA_OPTS="-server -Xms1303m -Xmx1303m -XX:MetaspaceSize=96m -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true"

## Add the JMX configuration
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.port=3333"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.ssl=true"
JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keystore=keystore"
JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keystorePassword=password"

As JMX behavior is configured through JVM arguments, refer to the JDK vendor’s documentation for a full list of parameters and configuration examples.

Hot Rod is a binary TCP client-server protocol used in Red Hat JBoss Data Grid. It was created to overcome deficiencies in other client/server protocols, such as Memcached.

Hot Rod will failover on a server cluster that undergoes a topology change. Hot Rod achieves this by providing regular updates to clients about the cluster topology.

Hot Rod enables clients to do smart routing of requests in partitioned or distributed Red Hat JBoss Data Grid server clusters. To do this, Hot Rod allows clients to determine the partition that houses a key and then communicate directly with the server that has the key. This functionality relies on Hot Rod updating the cluster topology with clients, and that the clients use the same consistent hash algorithm as the servers.

Red Hat JBoss Data Grid contains a server module that implements the Hot Rod protocol. The Hot Rod protocol facilitates faster client and server interactions in comparison to other text-based protocols and allows clients to make decisions about load balancing, failover and data location operations.