Getting 'No deployment content with hash 'xyz' is available in the deployment content repository' on JBoss EAP

Solution Verified - Updated -

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6
    • 7

Issue

  • Getting below error while undeploying the existing application and then deploying the new war on JBoss EAP 6:
JBAS014613: Operation ("add") failed - address: ([("deployment" => "example.war")]) - failure description: "JBAS018717: No deployment content with hash 'xyz' is available in the deployment content repository for deployment 'example.war'
  • Sometimes when restarting the domain controller of our EAP domain, the deployments cannot be found. Investigation reveals that the files are missing from the file system. What could be the cause of this issue?
  • I deployed an application using CLI command "deploy (module) --force". When the server group was started the slave node failed with error in EAP 7:
2019-05-17 19:12:18,363 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "example.war")]) - failure description: "WFLYSRV0137: No deployment content with hash 'xyz' is available in the deployment content repository for deployment 'example.war'. This is a fatal boot error. To correct the problem, either restart with the --admin-only switch set and use the CLI to install the missing content or remove it from the configuration, or remove the deployment from the xml configuration file and restart."
2019-05-17 19:12:18,367 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
  • The deployment content is getting deleted automatically after some time of deployment and throwing FATAL server boot error while restarting JBoss.

Resolution

  • The workaround is to delete the hash entries from domain.xml and then restart the server.

Example:

<deployments>
        <deployment name="example.war" runtime-name="example.war">
            <content sha1="5ba0640f6a32e8e60f847ede576f14bef6bbcb07"/>       << remove this
        </deployment>
</deployments>

For EAP 6.0 and 6.1 there is a Bug - 1021763 filed with this issue and it should be upgraded for the latest available version.

  • Do not run more than one JBoss standalone/domain with same jboss.server.base.dir or jboss.domain.base.dir system properties and different configuration files on the same host server.

Root Cause

  • The error usually occurs when standalone/data or domain/data directories are deleted and then an attempt to redeploy the same application war file. The JBoss EAP server is unable to delete entries defined in standalone.xml/domain.xml with the same hash for the deployed application.

  • When two or more JBoss standalone/domain runs with the same jboss.server.base.dir or jboss.domain.base.dir system properties and different configuration files, then the periodic scan from second JBoss which constructs a Map of files and references them against internal records to determine which files are part of currently deployed applications and which does not, deletes the deployment content of the first JBoss as that deployment entry is not present in second JBoss's record & vice-versa.

  • Any cron job or custom application present on the server which is reading/writing to $EAP_HOME/standalone/data directory.

Diagnostic Steps

  • Enable audit log on Linux to know which process/user is deleting the file/directory.
    e.g. on RHEL 7 , add the below lines in /etc/audit/rules.d/audit.rules file.
-a always,exit -F dir=</path/to/standalone/data/content> arch=b32 -S unlink -S unlinkat -S rename -S renameat  -S rmdir -k delete
-a always,exit -F dir=</path/to/standalone/data/content> arch=b64 -S unlink -S unlinkat -S rename -S renameat  -S rmdir -k delete

Rstart the service auditd and make it on in to retain across reboot.This will log the file deletion operations in the file /var/log/audit/audit.log.

service auditd restart
systemctl enable auditd

Sample log:

time->Mon Jan  6 19:49:16 2020
type=PROCTITLE msg=audit(1578320356.904:457545): proctitle=2F7530332F6A646B312E382E305F3232312F62696E2F6A617661002D445B5374616E64616C6F6E655D002D736572766572002D766572626F73653A6763002D586C6F6767633A2F7530332F6A
626F73732D6561702F4541502D372E322E302F7374616E64616C6F6E652F6C6F672F67632E6C6F67002D58583A2B5072696E7447\015
type=PATH msg=audit(1578320356.904:457545): item=3 name="/path/to/jboss/standalone/data/content/78/667dcfc7738c8db0e43c354530aca401ce386a/content" inode=918800 dev=fd:08 mode=0100640 ouid=1001 ogid=1001 rdev=00:00 obj=unconfined_u:object_r:default_t:s0 objtype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0\015
type=PATH msg=audit(1578320356.904:457545): item=2 name="/path/to/jboss/standalone/data/content/content321324978343012536.tmp" inode=918800 dev=fd:08 mode=0100640 ouid=1001 ogid=1001 rdev=00:00 obj=unconfined_u:object_r:default_t:s0 objtype=DELETE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0\015
type=PATH msg=audit(1578320356.904:457545): item=1 name="/path/to/jboss/standalone/data/content/78/667dcfc7738c8db0e43c354530aca401ce386a/" inode=918802 dev=fd:08 mode=040750 ouid=1001 ogid=1001 rdev=00:00 obj=unconfined_u:object_r:default_t:s0 objtype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0\015
type=PATH msg=audit(1578320356.904:457545): item=0 name="/path/to/jboss/standalone/data/content/" inode=918574 dev=fd:08 mode=040750 ouid=1001 ogid=1001 rdev=00:00 obj=unconfined_u:object_r:default_t:s0 objtype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0\015
type=CWD msg=audit(1578320356.904:457545):  cwd="/path/to/jboss/bin"\015
type=SYSCALL msg=audit(1578320356.904:457545): arch=c000003e syscall=82 success=yes exit=0 a0=3418410 a1=2740a60 a2=3418410 a3=7f0bd535677b items=4 ppid=5055 pid=5124 auid=1000 uid=1001 gid=1001 euid=1001 suid=1001 fsuid=1001 egid=1001 sgid=1001 fsgid=1001 tty=pts0 ses=5196 comm="java" exe="/u03/jdk1.8.0_221/bin/java" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="delete"\015
  • In the audit log, you can look for keyword delete or directory path configured in the audit.rules file and corresponding logline for analysis.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments