katello-installer fails starting elasticsearch and tomcat6 if IPv6 is not configured properly

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 6
  • Red Hat Satellite 6.0
  • IPv6

Issue

katello-installer fails noting failure to start of elasticsearch or tomcat6 services. Here is an example output on screen

Installing             Debug: Executing '/sbin/chkconfig tomcat6'         [99%]
Could not start Service[tomcat6]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait tomcat6 start' returned 5: Starting tomcat6: [  OK  ]
Installing             Error: Could not start Service[tomcat6]: Execution [99%]

Following ERROR messages in /var/log/katello-installer.log

$ grep -e ^.ERR -e Errno katello-installer.log 
[ERROR 2014-07-11 14:23:13 main]  Could not start Service[elasticsearch]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait elasticsearch start' returned 5: Starting elasticsearch: [  OK  ]
[ERROR 2014-07-11 14:23:13 main]  /Stage[main]/Elasticsearch::Service/Service[elasticsearch]/ensure: change from stopped to running failed: Could not start Service[elasticsearch]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait elasticsearch start' returned 5: Starting elasticsearch: [  OK  ]
[ERROR 2014-07-11 14:23:13 main]  /Stage[main]/Elasticsearch::Service/Service[elasticsearch]: Failed to call refresh: Could not restart Service[elasticsearch]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait elasticsearch restart' returned 5: Stopping elasticsearch: [  OK  ]
[ERROR 2014-07-11 14:23:13 main]  /Stage[main]/Elasticsearch::Service/Service[elasticsearch]: Could not restart Service[elasticsearch]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait elasticsearch restart' returned 5: Stopping elasticsearch: [  OK  ]
[ERROR 2014-07-11 14:27:08 main]  Could not start Service[tomcat6]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait tomcat6 start' returned 5: Starting tomcat6: [  OK  ]
[ERROR 2014-07-11 14:27:08 main]  /Stage[main]/Candlepin::Service/Service[tomcat6]/ensure: change from stopped to running failed: Could not start Service[tomcat6]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait tomcat6 start' returned 5: Starting tomcat6: [  OK  ]
[ERROR 2014-07-11 14:27:23 main]  /Stage[main]/Candlepin::Service/Service[tomcat6]: Failed to call refresh: Could not restart Service[tomcat6]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait tomcat6 restart' returned 5: Stopping tomcat6: [  OK  ]
[ERROR 2014-07-11 14:27:23 main]  /Stage[main]/Candlepin::Service/Service[tomcat6]: Could not restart Service[tomcat6]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait tomcat6 restart' returned 5: Stopping tomcat6: [  OK  ]
[ WARN 2014-07-11 14:28:01 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: Task da6e728f-4ca6-46ea-b9b5-c0e224b37739: Errno::ECONNREFUSED: Connection refused - connect(2)
[ERROR 2014-07-11 14:28:01 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-07-11 14:28:01 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
  • katello-installer fails with error below:
katello-installer
 /Stage[main]/Candlepin::Database/Postgresql::Server::Db[candlepin]/Postgresql::Server::Database[candlepin]/Exec[/usr/bin/createdb
--port='5432' --owner='postgres' --template=template0  'candlepin']:
Failed to call refresh: /usr/bin/createdb --port='5432'--owner='postgres' --template=template0  'candlepin' returned 1 instead of one of [0]
 /Stage[main]/Candlepin::Database/Postgresql::Server::Db[candlepin]/Postgresql::Server::Database[candlepin]/Exec[/usr/bin/createdb
--port='5432' --owner='postgres' --template=template0  'candlepin']:
/usr/bin/createdb --port='5432' --owner='postgres' --template=template0  'candlepin' returned 1 instead of one of [0]
 /Stage[main]/Candlepin::Service/Service[tomcat6]: Failed to call refresh: Could not restart Service[tomcat6]: Execution of
'/usr/share/katello-installer/modules/service_wait/bin/service-wait 
tomcat6 restart' returned 5: Stopping tomcat6: [  OK  ]
 /Stage[main]/Candlepin::Service/Service[tomcat6]: Could not restart Service[tomcat6]: Execution of
'/usr/share/katello-installer/modules/service_wait/bin/service-wait
tomcat6 restart' returned 5: Stopping tomcat6: [  OK  ]
Installing             Done   [100%]

Resolution

  1. Make sure IPv6 is properly disabled (or enabled) as outlined in: How do I disable or enable the IPv6 protocol in Red Hat Enterprise Linux?

    • Pay particular attention to the modification of /etc/hosts file and the ::1 localhost IPv6 address.

  2. Validate the localhost address resolves to ::1 IPv6 address if and only if lo interface is using IPv6 as shown in the Diagnostic Steps below.

  3. Confirm whether hostname is configured in FQDN format on Red Hat Satellite server.

  4. Retry the satellite 6 install

        # katello-installer --reset

Root Cause

  • Despite having no IPv6 address assigned to lo interface, there is both an IPv4 & IPv6 entry for localhost in /etc/hosts. This can be validated either by using the getent command to query localhost address via standard resolver() routines.

  • Example showing that localhost is resolving to both ::1 IPv6 & 127.0.0.1 IPv4

        # getent ahosts localhost
        ::1             STREAM localhost
        ::1             DGRAM  
        ::1             RAW    
        127.0.0.1       STREAM 
        127.0.0.1       DGRAM  
        127.0.0.1       RAW
  • Example showing that localhost is resolving only to 127.0.0.1 IPv4 address
        # getent ahosts localhost
        127.0.0.1       STREAM localhost
        127.0.0.1       DGRAM  
        127.0.0.1       RAW
  • IPv4 and IPv6 were configured on system but the hostname was not configured in FQDN format, short name was set on the system.

Diagnostic Steps

  • Confirm whether IPv6 is supported in network or not. If supported then only configure IPv6 address and ping6 localhost command should work.

  • Check that localhost host entry is only resovling to ::1 IPv6 address if and only if the lo interface has both an IPv4 & IPv6 address assigned.

  • Here is the proper output for a system that has both 127.0.0.1 IPv4 & ::1 IPv6 address on lo interface.

            # ip addr show lo | grep -w -e inet -e inet6
                inet 127.0.0.1/8 scope host lo
                inet6 ::1/128 scope host 

            # getent ahosts localhost
            ::1             STREAM localhost
            ::1             DGRAM  
            ::1             RAW    
            127.0.0.1       STREAM 
            127.0.0.1       DGRAM  
            127.0.0.1       RAW
  • Here is the proper output for a system that has only 127.0.0.1 IPv4 address on lo interface.
            # ip addr show lo | grep -w -e inet -e inet6
                inet 127.0.0.1/8 scope host lo

            # getent ahosts localhost
            127.0.0.1       STREAM 
            127.0.0.1       DGRAM  
            127.0.0.1       RAW

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments