osa-dispatcher fails to start up on Red Hat Satellite-5 server.

Solution Verified - Updated -

Environment

  • Red Hat Satellite 5.x

Issue

  • osa-dispatcher service fails to start on Red Hat Satellite 5 server:
# rhn-satellite restart
[...]
Starting osa-dispatcher: RHN 591 2013/02/01 13:24:51 -07:00:
 ('Traceback (most recent call last):\n
  File "/usr/share/rhn/osad/jabber_lib.py", line 254, in setup_connection\n
    c = self._get_jabber_client(js)\n
  File "/usr/share/rhn/osad/jabber_lib.py", line 311, in _get_jabber_client\n
    c.connect()\n
  File "/usr/share/rhn/osad/jabber_lib.py", line 577, in connect\n
    jabber.Client.connect(self)\n
  File "/usr/lib/python2.4/site-packages/jabber/xmlstream.py", line 488, in connect\n
    raise socket.error("Unable to connect to the host and port specified")\n
error: Unable to connect to the host and port specified\n',)
  • Servers profile is not updated even after running rhn-profile-sync:
# rhn-satellite start
Starting spacewalk services...
Starting postgresql92-postgresql service:                  [  OK  ]
Initializing jabberd processes ...
Starting router:                                           [  OK  ]
Starting sm:                                               [  OK  ]
Starting c2s:                                              [  OK  ]
Starting s2s:                                              [  OK  ]
Starting tomcat6:                                          [  OK  ]
Waiting for tomcat to be ready ...
Starting httpd:                                            [  OK  ]
Starting osa-dispatcher: Spacewalk 23590 2016/01/21 07:47:17 +04:00: ('Error connecting to jabber server: Unable to connect to the host and port specified',)
                                                           [  OK  ]
Starting Monitoring ...
[ OK ]
Starting MonitoringScout ...
[ OK ]
Starting rhn-search...
Starting cobbler daemon:                                   [  OK  ]
Starting RHN Taskomatic...
Done.
  • After upgrading the Satellite server to 5.8, osa-dispatcher fails with the following errors.
2017/07/06 20:41:05 -00:00 4001 0.0.0.0: osad/jabber_lib.__init__
2017/07/06 20:41:05 -00:00 4001 0.0.0.0: osad/jabber_lib.connect('ERROR', 'Traceback caught:')
2017/07/06 20:41:05 -00:00 4001 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:')

# service osa-dispatcher start
Starting osa-dispatcher: Spacewalk 6371 2017/07/22 23:13:09 -04:00: ('Not able to reconnect - See https://access.redhat.com/solutions/45332 for possible solutions.\n',)
Spacewalk 6371 2017/07/22 23:13:09 -04:00: ('Error caught:',)

ERROR: unhandled exception occurred: (can't write str to text stream).                                                            [FAILED]

Resolution

  • For Red Hat Satellite v5.8:
    Monitoring is deprecated as of Satellite 5.8 and 6.x. Please see related Solution to remove the extraneous packages: "Red Hat Satellite 5.8 and extraneous Monitoring packages"

  • For Red Hat Satellite v5.7 and earlier:
    Verify that the Satellite's /etc/hosts file contains correct entries for the localhost and fully-qualified domain name (e.g., 'satellite.example.com'):

# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.10.10.1  satellite.example.com

# hostname
  • In /etc/sysconfig/network:
HOSTNAME=satellite.example.com
  • Then reconfigure jabberd and restart the Satellite services:
# spacewalk-setup-jabberd 
# rhn-satellite restart

Root Cause

  • The osa-dispatcher service was unable to establish a connection to the jabberd service, because jabberd either failed to listen on its configured IPv6 port, or osa-dispatcher could not resolve or connect to that port
  • The jabberd c2s process listens on port 5222 and osa-dispatcher connects to it. In the XMPP exchanges of XML between c2s and osa-dispatcher, osa-dispatcher expects to find a match with the Satellite's hostname.

Diagnostic Steps

  • Confirm whether the jabberd processes are up and listening:
# yum install nmap
# nmap -sT -p 5200-5400 localhost
[...]
PORT     STATE SERVICE
5222/tcp open  xmpp-client
5269/tcp open  xmpp-server
5347/tcp open  unknown
  • Verify whether the SSL server certificate used by jabberd has the correct Subject Common Name (CN), matching the Satellite's fully-qualified domain name, and verifies against the CA certificate:
# openssl verify -CAfile /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT /etc/pki/spacewalk/jabberd/server.pem
/etc/pki/spacewalk/jabberd/server.pem: OK

# openssl x509 -in /etc/pki/spacewalk/jabberd/server.pem -subject -noout | awk -F '/' '{print $6}'
CN=server.example.com

# hostname
server.example.com
  • Verify that the Satellite FQDN is being used in the jabberd configuration files. In /etc/jabberd/c2s.xml:
    <id require-starttls="false" pemfile="/etc/pki/spacewalk/jabberd/server.pem" realm="" register-enable="true">satellite.example.com</id>

In sm.xml:

  <id>satellite.example.com</id>
  • Verify the Satellite FQDN (Initial letter) is not in uppercase and should not varies in the various configuration files, It should be same across all the files present in the system :
# hostname 
satellite.example.com

# /etc/sysconfig/network
NETWORKINNG=yes
HOSTNAME=satellite.example.com

# hostname -f
satellite.example.com

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.