Manifest refresh on Satellite 6 failed with "Runtime Error java.lang.RuntimeException: java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively:-2 "

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6.

Issue

  • Manifest refresh on Satellite 6 failed with Runtime Error java.lang.RuntimeException: java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively:-2 . Below is the error message in tomcat logs:
INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [5,923] milliseconds.
Jan 31, 2017 9:29:15 AM org.apache.jasper.EmbeddedServletOptions <init>
SEVERE: The scratchDir you specified: /usr/share/tomcat/work/Catalina/localhost/candlepin is unusable.       <-----------------
Jan 31, 2017 9:29:15 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat/webapps/candlepin has finished in 23,706 ms

Resolution

  • Correct the group ownership for the incorrect directories:
# chgrp tomcat /usr/share/tomcat/
# chgrp tomcat /var/cache/tomcat/
# semanage fcontext -a -t tomcat_cache_t "/usr/share/tomcat(/.*)?"
# restorecon -R -v /usr/share/tomcat
  • Try to refresh the manifest on satellite webui.

Root Cause

  • There are some directories under tomcat home directory and cache directory which has incorrect group owner set
# namei -mol /var/cache/tomcat/work/Catalina/localhost/candlepin
f: /var/cache/tomcat/work/Catalina/localhost/candlepin
dr-xr-xr-x root   root   /
drwxr-xr-x root   root   var
drwxr-xr-x root   root   cache
drwxrwx--- root   38693  tomcat   <<<<<<<<<<<<<<<<<<<<<<<<
drwxrwx--- root   tomcat work
drwxr-xr-x tomcat tomcat Catalina
drwxr-xr-x tomcat tomcat localhost
drwxr-xr-x tomcat tomcat candlepin

# namei -mol /usr/share/tomcat/work/Catalina/localhost/candlepin
f: /usr/share/tomcat/work/Catalina/localhost/candlepin
dr-xr-xr-x root   root   /
drwxr-xr-x root   root   usr
drwxr-xr-x root   root   share
drwxrwxr-x root   38693  tomcat    <<<<<<<<<<<<<<<<<<<<<<<<
lrwxrwxrwx root   tomcat work -> /var/cache/tomcat/work
dr-xr-xr-x root   root     /
drwxr-xr-x root   root     var
drwxr-xr-x root   root     cache
drwxrwx--- root   38693    tomcat   <<<<<<<<<<<<<<<<<<<<<<<<
drwxrwx--- root   tomcat   work
drwxr-xr-x tomcat tomcat Catalina
  • The issue might be due to permission on /var/cache directory. If /var/cache has 750 permission, user apache will not able to access and write under directory /var/cache/pulp.
#chmod 755 /var/cache

Diagnostic Steps

  • production log captured when refreshing manifest:
2017-01-31 09:31:54 [katello/glue] [I] Task import manifest /tmp/0.7699204894583423.zip for owner: ACFC (3/5) > Katello::Provider.find(2).owner_import("/tmp/0.769920489458)
2017-01-31 09:31:54 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.7.36.235 at 2017-01-31 09:31:54 -0500
2017-01-31 09:31:54 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
2017-01-31 09:31:54 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"2026c460-bdc0-4758-91f8-20c12961c3fe", "search_id"=>"1"}], "task"=>{}}
2017-01-31 09:31:54 [app] [E] Error during manifest refresh: {"displayMessage"=>"Runtime Error java.lang.RuntimeException: java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively:-2", "requestUuid"=>"78b13ec2-eeda-4200-b8b5-ad3cc012f5e1"}
2017-01-31 09:31:54 [foreman-tasks/action] [E] Runtime Error java.lang.RuntimeException: java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively:-2 (Katello::Errors::CandlepinError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.90/app/lib/actions/middleware/propagate_candlepin_errors.rb:21:in `rescue in propagate_candlepin_errors'
  • candlepin log:
2017-01-31 09:31:54,440 [thread=http-bio-8443-exec-1] [req=78b13ec2-eeda-4200-b8b5-ad3cc012f5e1, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/ACFC/imports
2017-01-31 09:31:54,760 [thread=http-bio-8443-exec-1] [req=78b13ec2-eeda-4200-b8b5-ad3cc012f5e1, org=] ERROR org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error java.lang.RuntimeException: java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively:-2
org.jboss.resteasy.spi.ReaderException: java.lang.RuntimeException: java.io.IOException: Permission denied
    at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:201) ~[resteasy-jaxrs-2.3.10.Final.jar:na]
  • Check if tomcat user is external and has correct home directory:
# id tomcat
# grep tomcat /etc/passwd

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.