cdn-sync failing with syntax error on Red Hat Satellite 5.8

Solution Verified - Updated -

Environment

  • Red Hat Satellite 5.8

Issue

  • cdn-sync is failing with below error on satellite 5.8:
Exception: SYNC ERROR: attempting to display as much information as possible
 139910487316224:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: CERTIFICATE

Resolution

  • Check whether /var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf was modified.
  • Use below steps to fix the issue:

1) Delete the current entitlement certificate in rhnSatelliteCert table:

    # echo "delete from rhnSatelliteCert;" | spacewalk-sql -i

2) Uncomment # bytea_output = 'escape' line in /var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf

     bytea_output = 'escape'

3) Restart rh-postgresql95-postgresql service to apply the change:

     # service rh-postgresql95-postgresql restart

4) Activate Satellite again:

     # rhn-satellite-activate -vvv --manifest=/path/to/manifest.zip

5) Check that cdn-sync is working as well:

     # cdn-sync -l

Root Cause

  • Satellite entitlement certificate was stored in the db as hexadecimal value:
# echo "select * from rhnSatelliteCert;" | spacewalk-sql -i
53d226368616e6e656c2d66616d696c69657322207175616e746974793d22383032322220666c65783d2234303030222066616d696c793
d227268656c2d7365727665722d362d6f7074696f6e616c222f3e0a20203c72686e2d636572742d6669656c64206e616d653d226368616e6e656c2
d66616d696c69657322207175616e746974793d22383032322220666c65783d2234303030222066616d696c793d227268656c2d736572766572
2d362d6f7074696f6e616c2d62657461222f3e0a20203c72686e2d636572742d6669656c64206e616d653d226368616e6e656c2d66
616d696c69657322207175616e746974793d22383032322220666c65783d2234303030222066616d696c
793d227268656c2d7365727665722d362d6f7074696f6e616c2d626574612d6465627567222f3e0a20203c72686e2d63657274
2d6669656c64206e616d653d226368616e6e656c2d66616d696c69657322207175616e746974793d22383032322220666c65783d2234303030
222066616d696c793d227268656c2d7365727665722d362d6f7074662d72682d636f6d6d6f6e2d6465627567222f3e0a20203c
72686e2d636572742d6669656c64206e616d653d226368616e6e656c2d66616d696c69657322207175616e746974793d22383032
322220666c65783d2234303030222066616d696c793d227268656c2d7365727665722d362d726873636c69656e74222f3e0a202
03c72686e2d636572742d6669656c64206e616d653d226368616e6e656c2d66616d696c69657322207175616e746974793d2238.....
  • By default satellite store it as an escape value which is also expected by rhn-satellite-activate, its a hex value in this case and thus failing with the traceback we have seen. The reason for a hex value usage instead of escape is non-default config in /var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf, in last line there is:
# bytea_output = 'escape'

if its commented hex is used according to postresql docs.

  • However as per as a /usr/bin/spacewalk-setup-postgresql, which is executed during the upgrade, it should be set to escape:
bytea_output = 'escape'

# cat /usr/bin/spacewalk-setup-postgresql

139 ### spacewalk-setup-postgresql modified values
140 checkpoint_completion_target = 0.7
141 effective_cache_size = 1152MB
142 log_line_prefix = '%m '
143 maintenance_work_mem = 96MB
144 max_connections = 600
145 shared_buffers = 384MB
146 wal_buffers = 4MB
147 work_mem = 2560kB
148 bytea_output = 'escape'
149 EOF

Diagnostic Steps

  • Complete traceback:
# cdn-sync -l
Traceback (most recent call last):
  File "/usr/bin/cdn-sync", line 230, in 
    sys.exc_info()[2])
  File "/usr/bin/cdn-sync", line 156, in 
    cdnsync.print_channel_tree(repos=args.show_repos)
  File "/usr/lib/python2.6/site-packages/spacewalk/cdn_tools/cdnsync.py", line 731, in print_channel_tree
    channel_tree, not_available_channels = self._tree_available_channels()
  File "/usr/lib/python2.6/site-packages/spacewalk/cdn_tools/cdnsync.py", line 188, in _tree_available_channels
    self.cdn_repository_manager.check_channel_availability(x, self.no_kickstarts)]
  File "/usr/lib/python2.6/site-packages/spacewalk/cdn_tools/repository.py", line 163, in check_channel_availability
    if not self.check_repository_availability(source['relative_url'], channel_label=channel_label):
  File "/usr/lib/python2.6/site-packages/spacewalk/cdn_tools/repository.py", line 175, in check_repository_availability
    crypto_keys = self.get_repository_crypto_keys(relative_url)
  File "/usr/lib/python2.6/site-packages/spacewalk/cdn_tools/repository.py", line 241, in get_repository_crypto_keys
    keys = ssl_set.get_crypto_keys(check_dates=True)
  File "/usr/lib/python2.6/site-packages/spacewalk/cdn_tools/repository.py", line 496, in get_crypto_keys
    if not verify_certificate_dates(key[1]):
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/satCerts.py", line 89, in verify_certificate_dates
    _, _, not_before, not_after = get_certificate_info(cert_str)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/satCerts.py", line 79, in get_certificate_info
    cert = X509.load_cert_string(cert_str)
  File "/usr/lib64/python2.6/site-packages/M2Crypto/X509.py", line 655, in load_cert_string
    return load_cert_bio(bio, format)
  File "/usr/lib64/python2.6/site-packages/M2Crypto/X509.py", line 639, in load_cert_bio
    raise X509Error(Err.get_error())
Exception: SYNC ERROR: attempting to display as much information as possible
 140211184088832:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: CERTIFICATE

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.