Manifest refresh on Red Hat Satellite 6 failed with "Runtime Error java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively:-2"

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6.

Issue

  • Manifest refresh on Satellite 6 failed with Runtime Error java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively:-2

  • Below is the error message in tomcat catalina logs:

    INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [5,923] milliseconds.
    Jan 31, 2017 9:29:15 AM org.apache.jasper.EmbeddedServletOptions <init>
    SEVERE: The scratchDir you specified: /usr/share/tomcat/work/Catalina/localhost/candlepin is unusable.       <-----------------
    Jan 31, 2017 9:29:15 AM org.apache.catalina.startup.HostConfig deployDirectory
    INFO: Deployment of web application directory /var/lib/tomcat/webapps/candlepin has finished in 23,706 ms
    

Resolution

  • Correct the group ownership for the incorrect directories:

    # chgrp tomcat /usr/share/tomcat/
    # chgrp tomcat /var/cache/tomcat/
    # chown tomcat.tomcat /var/cache/tomcat/temp/
    # chown -R tomcat:tomcat /var/cache/candlepin/
    # restorecon -R -v /var/cache/candlepin/
    # semanage fcontext -a -t tomcat_cache_t "/usr/share/tomcat(/.*)?"
    # restorecon -R -v /usr/share/tomcat
    
  • Check if Satellite is configured with AD server. If yes, check if tomcat group exists on AD server and note down the group ID of tomcat from AD server. Run below command on Satellite machine:

    # find / -group AD_GID -exec chgrp -h tomcat {} \;
    # find / -user AD_GID -exec chown -h tomcat {} \;
    

    Note: Replace AD_GID in above command with GID of tomcat group on AD server.

  • Try to refresh the manifest on satellite webUI.

For more KB articles/solutions related to Red Hat Satellite 6.x Manifest Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Manifest Issues

Root Cause

  • There are some directories under tomcat home directory and cache directory which has incorrect group owner set

    # namei -mol /var/cache/tomcat/work/Catalina/localhost/candlepin
    f: /var/cache/tomcat/work/Catalina/localhost/candlepin
    dr-xr-xr-x root   root   /
    drwxr-xr-x root   root   var
    drwxr-xr-x root   root   cache
    drwxrwx--- root   38693  tomcat   <<<<<<<<<<<<<<<<<<<<<<<<
    drwxrwx--- root   tomcat work
    drwxr-xr-x tomcat tomcat Catalina
    drwxr-xr-x tomcat tomcat localhost
    drwxr-xr-x tomcat tomcat candlepin
    
    # namei -mol /usr/share/tomcat/work/Catalina/localhost/candlepin
    f: /usr/share/tomcat/work/Catalina/localhost/candlepin
    dr-xr-xr-x root   root   /
    drwxr-xr-x root   root   usr
    drwxr-xr-x root   root   share
    drwxrwxr-x root   38693  tomcat    <<<<<<<<<<<<<<<<<<<<<<<<
    lrwxrwxrwx root   tomcat work -> /var/cache/tomcat/work
    dr-xr-xr-x root   root     /
    drwxr-xr-x root   root     var
    drwxr-xr-x root   root     cache
    drwxrwx--- root   38693    tomcat   <<<<<<<<<<<<<<<<<<<<<<<<
    drwxrwx--- root   tomcat   work
    drwxr-xr-x tomcat tomcat Catalina
    
  • The issue might be due to permission on /var/cache directory. If /var/cache has 750 permission, user apache will not able to access and write under directory /var/cache/pulp.

    #chmod 755 /var/cache
    

Diagnostic Steps

  • Below error is observed in /var/log/foreman/production log when refreshing manifest:

    2017-01-31 09:31:54 [katello/glue] [I] Task import manifest /tmp/0.7699204894583423.zip for owner: ACFC (3/5) > Katello::Provider.find(2).owner_import("/tmp/0.769920489458)
    2017-01-31 09:31:54 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.7.36.235 at 2017-01-31 09:31:54 -0500
    2017-01-31 09:31:54 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
    2017-01-31 09:31:54 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"2026c460-bdc0-4758-91f8-20c12961c3fe", "search_id"=>"1"}], "task"=>{}}
    2017-01-31 09:31:54 [app] [E] Error during manifest refresh: {"displayMessage"=>"Runtime Error java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively:-2", "requestUuid"=>"1234567-abcde-4567-b8b5-ad3cc012f5e1"}
    2017-01-31 09:31:54 [foreman-tasks/action] [E] Runtime Error java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively:-2 (Katello::Errors::CandlepinError)
    | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.90/app/lib/actions/middleware/propagate_candlepin_errors.rb:21:in `rescue in propagate_candlepin_errors'
    
  • Below error is observed in /var/log/candlepin/candlepin log:

    2017-01-31 09:31:54,440 [thread=http-bio-8443-exec-1] [req=78b13ec2-eeda-4200-b8b5-ad3cc012f5e1, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/owners/ACFC/imports
    2017-01-31 09:31:54,760 [thread=http-bio-8443-exec-1] [req=78b13ec2-eeda-4200-b8b5-ad3cc012f5e1, org=] ERROR  org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclusively:-2
    org.jboss.resteasy.spi.ReaderException: java.lang.RuntimeException: java.io.IOException: Permission denied
    at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:201) ~[resteasy-jaxrs-2.3.10.Final.jar:na]
    
  • Check if tomcat user is external and has correct home directory:

    # id tomcat
    # grep tomcat /etc/passwd
    
  • Run below command on Satellite to check the permission and ownership of mentioned directory structure in the command is correct.

    # namei -lom /var/cache/candlepin/
    # namei -lom /var/cache/candlepin
    # namei -lom /var/cache/tomcat/work/Catalina/localhost/candlepin
    # namei -lom /usr/share/tomcat/work/Catalina/localhost/candlepin
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments