Red Hat Training

A Red Hat training course is available for Red Hat JBoss Enterprise Application Platform

Chapter 5. Advanced Tutorials

5.1. Example Workflow: Automated Transaction Recovery Feature When Scaling Down a Cluster


This feature is provided as Technology Preview only. It is not supported for use in a production environment, and it might be subject to significant future changes. See Technology Preview Features Support Scope on the Red Hat Customer Portal for information about the support scope for Technology Preview features.

This tutorial demonstrates the automated transaction recovery feature of the JBoss EAP for OpenShift image when scaling down a cluster. The jta-crash-rec-eap7 quickstart example and the eap70-tx-recovery-s2i application template are used here to show how XA transactions issued on the OpenShift pod, when terminated within the cluster’s scale down, are recovered by the dedicated migration pod.


The jta-crash-rec-eap7 quickstart uses the H2 database that is included with JBoss EAP. It is a lightweight, relational example datasource that is used for examples only. It is not robust or scalable, is not supported, and should not be used in a production environment.

5.1.1. Prepare for Deployment

  1. Log in to your OpenShift instance using the oc login command.
  2. Create a new project.

    $ oc new-project eap-tx-demo
  3. Add the view role to the default service account, which will be used to run the underlying pods. This enables the service account to view all the resources in the eap-tx-demo namespace, which is necessary for managing the cluster.

    $ oc policy add-role-to-user view system:serviceaccount:$(oc project -q):default
  4. For automated transaction recovery to work, the JBoss EAP application must use a ReadWriteMany persistent volume.

    Provision the persistent volume expected by the eap70-tx-recovery-s2i application template to hold the data for the ${APPLICATION_NAME}-eap-claim persistent volume claim.

    This example uses a persistent volume object provisioned using the NFS method with the following definition:

    $ cat txpv.yaml
    apiVersion: v1
    kind: PersistentVolume
      name: txpv
        storage: 1Gi
        - ReadWriteMany
      persistentVolumeReclaimPolicy: Retain
        path: /mnt/mountpoint

    Update the path and server fields in the above definition for your environment, and provision the persistent volume with the following command:

    $ oc create -f txpv.yaml
    persistentvolume "txpv" created
    $ oc get pv
    txpv      1Gi        RWX           Retain         Available                                      26s

    When using the NFS method to provision persistent volume objects for the eap70-tx-recovery-s2i application template, ensure the mount point is exported with sufficient permissions. On the host from which the mount point is exported, perform the following:

    # chmod -R 777 /mnt/mountpoint
    # cat /etc/exports
    /mnt/mountpoint *(rw,sync,anonuid=185,anongid=185)
    # exportfs -va
    exporting *:/mnt/mountpoint
    # setsebool -P virt_use_nfs 1

    Replace /mnt/mountpoint path above as appropriate for your environment.

5.1.2. Deployment

  1. Deploy the jta-crash-rec-eap7 quickstart using the eap70-tx-recovery-s2i application template. Specify the following:

    $ oc new-app --template=eap70-tx-recovery-s2i \
    -p SOURCE_REPOSITORY_REF="master" \
    -p CONTEXT_DIR="jta-crash-rec-eap7" \
    -e CUSTOM_INSTALL_DIRECTORIES="extensions/*" \
    --> Deploying template "openshift/eap70-tx-recovery-s2i" to project eap-tx-demo
         JBoss EAP 7.0 (tx recovery)
         An example EAP 7 application. For more information about using this template, see
         A new EAP 7 based application has been created in your project.
         * With parameters:
            * Application Name=eap-app
            * Custom http Route Hostname=
            * Git Repository URL=
            * Git Reference=master
            * Context Directory=jta-crash-rec-eap7
            * Queues=
            * Topics=
            * A-MQ cluster password=nyneOXUm # generated
            * Github Webhook Secret=PUW8Tmov # generated
            * Generic Webhook Secret=o7uD7qrG # generated
            * ImageStream Namespace=openshift
            * JGroups Cluster Password=MoR1Jthf # generated
            * Deploy Exploded Archives=false
            * Maven mirror URL=
            * ARTIFACT_DIR=
            * MEMORY_LIMIT=1Gi
            * EAP Volume Size=1Gi
            * Split the data directory?=true
    --> Creating resources ...
        service "eap-app" created
        service "eap-app-ping" created
        route "eap-app" created
        imagestream "eap-app" created
        buildconfig "eap-app" created
        deploymentconfig "eap-app" created
        deploymentconfig "eap-app-migration" created
        persistentvolumeclaim "eap-app-eap-claim" created
    --> Success
        Build scheduled, use 'oc logs -f bc/eap-app' to track its progress.
        Run 'oc status' to view your app.
  2. Wait for the build to finish. You can see the status of the build using the oc logs -f bc/eap-app command.
  3. Modify the eap-app deployment configuration with the definition of JAVA_OPTS_APPEND and JBOSS_MODULES_SYSTEM_PKGS_APPEND environment variables.

    $ oc get dc
    eap-app             1          1         1         config,image(eap-app:latest)
    eap-app-migration   1          1         1         config,image(eap-app:latest)
    $ oc set env dc/eap-app \
    -e JBOSS_MODULES_SYSTEM_PKGS_APPEND="org.jboss.byteman" \
    -e JAVA_OPTS_APPEND="-javaagent:/tmp/src/extensions/byteman/byteman.jar=script:/tmp/src/src/main/scripts/xa.btm"
    deploymentconfig "eap-app" updated

    This setting will notify the Byteman tracing and monitoring tool to modify the XA transactions processing in the following way:

    • The first transaction is always allowed to succeed.
    • When an XA resource executes phase 2 of the second transaction, the JVM process of the particular pod is halted.

5.1.3. Using the JTA Crash Recovery Application

  1. List running pods in the current namespace:

    $ oc get pods | grep Running
    NAME                        READY     STATUS      RESTARTS   AGE
    eap-app-2-r00gm             1/1       Running     0          1m
    eap-app-migration-1-lvfdt   1/1       Running     0          2m
  2. Issue a new XA transaction.

    1. Launch the application by opening a browser and navigating to
    2. Enter Mercedes into the Key field, and Benz into the Value field. Click the Submit button.
    3. Wait for a moment, then click the Refresh Table link.
    4. Notice how the table row containing the Mercedes entry is updated with updated via JMS on eap-app-2-r00gm host. If it has not yet updated, click the Refresh Table link couple of times. Alternatively, you can inspect the log of the eap-app-2-r00gm pod to verify the transaction was handled properly:

      $ oc logs eap-app-2-r00gm | grep 'updated'
      INFO  [] (Thread-0 (ActiveMQ-client-global-threads-1566836606)) JTA Crash Record Quickstart: key value pair updated via JMS on eap-app-2-r00gm host.
  3. Issue a second XA transaction using your browser at

    1. Enter Land into the Key field, and Rover into the Value field. Click the Submit button.
    2. Wait for a moment, then click the Refresh Table link.
    3. Notice how the Land Rover entry was added without the updated via …​ suffix.
  4. Scale the cluster down.

    $ oc scale --replicas=0 dc/eap-app
    deploymentconfig "eap-app" scaled
    1. Notice how the eap-app-2-r00gm pod was scheduled for termination.

      $ oc get pods
      NAME                        READY     STATUS        RESTARTS   AGE
      eap-app-1-build             0/1       Completed     0          4m
      eap-app-2-r00gm             1/1       Terminating   0          2m
      eap-app-migration-1-lvfdt   1/1       Running       0          3m
  5. Watch the log of the migration pod and notice how transaction recovery is performed. Wait for the recovery to finish:

    $ oc logs -f eap-app-migration-1-lvfdt
    Finished Migration Check cycle, pausing for 30 seconds before resuming
    Finished, recovery terminated successfully
    Migration terminated with status 0 (T)
    Releasing lock: (/opt/eap/standalone/partitioned_data/split-1)
    Finished Migration Check cycle, pausing for 30 seconds before resuming
  6. Scale the cluster back up.

    $ oc scale --replicas=1 dc/eap-app
    deploymentconfig "eap-app" scaled
  7. Using the browser navigate back to
  8. Notice the table contains entries for both transactions. It looks similar to the following output:

    Table 5.1. Example: Database Table Contents

    Database Table Contents 




    Benz updated via JMS on eap-app-2-r00gm host.


    Rover updated via JMS on eap-app-migration-1-lvfdt host.

    The content in the above table indicates that, although the cluster was scaled down before the second XA transaction had chance to finish, the migration pod performed the transaction recovery and the transaction was successfully completed.