satellite-sync fails with 'ORA-00604: error occurred at recursive SQL level 1' and 'ORA-12705: invalid or unknown NLS parameter value specified' [Satellite 4.2]

Solution Verified - Updated -

Environment

  • Red Hat Network Satellite 4.2

Issue

Unable to run satellite-sync from the server (error output below), and most attempts from clients to interact with the Satellite server result in an Internal Server Error message. Running rhn_check from a client produces a traceback:

# rhn_check
Traceback (most recent call last):
  File "/usr/sbin/rhn_check", line 298, in ?
    ACTION_VERSION, Status)
  File "/usr/lib64/python2.3/xmlrpclib.py", line 1029, in __call__
    return self.__send(self.__name, args)
  File "/usr/share/rhn/up2date_client/rpcServer.py", line 46, in _request1
    ret = self._request(methodname, params)
  File "/usr/lib/python2.3/site-packages/rhn/rpclib.py", line 330, in _request
    raise pe
xmlrpclib.ProtocolError: <ProtocolError for station.domain.example.com /XMLRPC: 500 Internal Server Error>

At the Satellite server interface, all client systems show as inactive, and when clicking on a host entry, no detail page is shown.

Similar errors regarding NLS consistently show up in the httpd error.log file.

Resolution

Ensure that the Oracle listener version matches the database version, and that the version is compatable with the Satellite Server release. For this scenario, the Oracle listener version 9.2 was used to match the database version.

Root Cause

Oracle listener versions are supposed to be backward-compatible with previous database versions (citation needed). The listener upgrade to 11g is expected to be inoccuous, but introduces the behavior above.

Diagnostic Steps

Complete error output from satellite-sync:

# /usr/bin/satellite-sync
RHN 409 2012/01/09 09:40:53 -04:00: ('Connection attempt failed', 604, 'ORA-00604: error occurred at recursive SQL level 1\nORA-12705: invalid or unknown NLS parameter value specified\n')
09:40:53 SQL ERROR during xml processing: (604, 'ORA-00604: error occurred at recursive SQL level 1\nORA-12705: invalid or unknown NLS parameter value specified\n', 'RHNP', 'Connection_Connect(): begin session')
09:40:53 TRACEBACK: Exception reported from satellite.example.com
Time: Mon Jan  9 09:40:53 2012
Exception type server.rhnSQL.sql_base.SQLConnectError

Exception Handler Information
Traceback (most recent call last):
  File "/usr/share/rhn/satellite_tools/satsync.py", line 2056, in processCommandline
    rhnSQL.initDB(CFG.DEFAULT_DB)
  File "/usr/share/rhn/server/rhnSQL/__init__.py", line 58, in initDB
    __init__DB(db)
  File "/usr/share/rhn/server/rhnSQL/__init__.py", line 36, in __init__DB
    __DB.connect()
  File "/usr/share/rhn/server/rhnSQL/int_oracle.py", line 282, in connect
    raise apply(sql_base.SQLConnectError, err_args)
SQLConnectError: (604, 'ORA-00604: error occurred at recursive SQL level 1\nORA-12705: invalid or unknown NLS parameter value specified\n', 'RHNP', 'Connection_Connect(): begin session')

Local variables by frame
Frame connect in /usr/share/rhn/server/rhnSQL/int_oracle.py at line 282
                           e = <type 'instance'> ORA-00604: error occurred at recursive SQL level 1
ORA-12705: invalid or unknown NLS parameter value specified

                       errno = <type 'int'> 604
                        self = <type 'instance'> <server.rhnSQL.driver_cx_Oracle.Database instance at 0xb7a717ec>
                         ret = <type 'tuple'> (604, 'ORA-00604: error occurred at recursive SQL level 1\nORA-12705: invalid or unknown NLS parameter value specified\n', 'Connection_Connect(): begin session')
                   reconnect = <type 'int'> 1
                    err_args = <type 'list'> ['RHNP', 604, 'ORA-00604: error occurred at recursive SQL level 1\nORA-12705: invalid or unknown NLS parameter value specified\n', 'Connection_Connect(): begin session']
                      errmsg = <type 'str'> ORA-00604: error occurred at recursive SQL level 1
ORA-12705: invalid or unknown NLS parameter value specified


Frame __init__DB in /usr/share/rhn/server/rhnSQL/__init__.py at line 36
                    db_class = <type 'classobj'> server.rhnSQL.driver_cx_Oracle.Database
                          db = <type 'str'> sserver/cycl0ps@RHNP

Frame initDB in /usr/share/rhn/server/rhnSQL/__init__.py at line 60
                          db = <type 'str'> sserver/cycl0ps@RHNP

Frame processCommandline in /usr/share/rhn/satellite_tools/satsync.py at line 2060
                optionsTable = <type 'list'> [<optparse.Option instance at 0xb7c5348c>, <optparse.Option instance at 0xb7c5336c>, <optparse.Option instance at 0xb7c5346c>, <optparse.Option instance at 0xb7c5330c>, <optparse.Option instance at 0xb7c61f6c>, <optparse.Option instance at 0xb7a6b08c>, <optparse.Option instance at 0xb7a6b0ec>, <optparse.Option instance at 0xb7a6b16c>, <optparse.Option instance at 0xb7a6b38c>, <optparse.Option instance at 0xb7a6b40c>, <optparse.Option instance at 0xb7a6b46c>, <optparse.Option instance at 0xb7a6b4cc>, <optparse.Option instance at 0xb7a6b52c>, <optparse.Option instance at 0xb7a6b58c>, <optparse.Option instance at 0xb7a6b5ec>, <optparse.Option instance at 0xb7a6b64c>, <optparse.Option instance at 0xb7a6b6ac>, <optparse.Option instance at 0xb7a6b72c>, <optparse.Option instance at 0xb7a6b78c>, <optparse.Option instance at 0xb7a6b80c>, <optparse.Option instance at 0xb7a6b86c>, <optparse.Option instance at 0xb7a6b8cc>, <optparse.Option ins
 tance at
                        args = <type 'list'> []
                           e = <type 'instance'> (604, 'ORA-00604: error occurred at recursive SQL level 1\nORA-12705: invalid or unknown NLS parameter value specified\n', 'RHNP', 'Connection_Connect(): begin session')
                optionParser = <type 'instance'> <optparse.OptionParser instance at 0xb7a6bacc>
                       tbOut = <type 'cStringIO.StringO'> <cStringIO.StringO object at 0xb7a6bfe0>

Frame main in /usr/share/rhn/satellite_tools/satsync.py at line 139
                   timeStart = <type 'float'> 1326120053.53
                        self = <type 'instance'> <satellite_tools.satsync.Runner instance at 0xb7c5310c>

Frame main in /usr/bin/satellite-sync at line 134

Frame ? in /usr/bin/satellite-sync at line 163
                  systemExit = <type 'function'> <function systemExit at 0xb7f14ae4>
                 rhnLockfile = <type 'module'> <module 'common.rhnLockfile' from '/usr/share/rhn/common/rhnLockfile.pyc'>
                      socket = <type 'module'> <module 'socket' from '/usr/lib/python2.3/socket.pyc'>
                     syncLib = <type 'module'> <module 'satellite_tools.syncLib' from '/usr/share/rhn/satellite_tools/syncLib.pyc'>
                __builtins__ = <type 'module'> <module '__builtin__' (built-in)>
                    _LIBPATH = <type 'str'> /usr/share/rhn
                    __file__ = <type 'str'> /usr/bin/satellite-sync
                         sys = <type 'module'> <module 'sys' (built-in)>
                    repr_str = <type 'function'> <function repr_str at 0xb7c5410c>
              fetchTraceback = <type 'function'> <function fetchTraceback at 0xb7c4dd84>
        systemExit_exception = <type 'function'> <function systemExit_exception at 0xb7c54144>
                         CFG = <type 'instance'> <RHNOptions instance at -1211849236: {'sync_cache_dir': '/var/cache/rhn/', 'http_proxy_password': None, 'http_proxy': 'proxy-server.example.com:8080', 'satellite': 1, 'cache_refresh_count_max': 5, 'max_user_len': 64, 'http_proxy_username': None, 'ca_chain': '/usr/share/rhn/RHNS-CA-CERT', 'freeloader_throttle_start': 9, 'disallow_user_creation': 1, 'min_user_len': 3, 'mount_point': '/var/satellite', 'throttle_freeloaders': 0, 'cache_refresh_level': 2, 'pam_auth_service': None, 'kickstart_mount_point': '/var/satellite', 'message_to_all': '/etc/rhn/message_to_all.txt', 'client_auth_timeout': 3600.0, 'throttle_grace_period': 168.0, 'pkg_list_dir': '/var/up2date/list', 'session_secret_4': '2827d848ca31df0c8fda746d2d988d2a', 'session_secret_2': 'a0dfaec322f09d53beff974895219f27', 'session_secret_3': 'e557bc1751715260789ed1be6dbfa7a8', 'session_lifetime': 1800, 'session_secret_1': 'c455b9dfa928f7eba15f76c00e8fe220', 'refer_to_www': 0, 'pr
 oxy_auth
                    __name__ = <type 'str'> __main__
                     satsync = <type 'module'> <module 'satellite_tools.satsync' from '/usr/share/rhn/satellite_tools/satsync.pyc'>
                        main = <type 'function'> <function main at 0xb7a6f374>
                          os = <type 'module'> <module 'os' from '/usr/lib/python2.3/os.pyc'>
                     __doc__ = <type 'NoneType'> None
                        LOCK = <type 'instance'> <common.rhnLockfile.Lockfile instance at 0xb7f165ec>
                 releaseLOCK = <type 'function'> <function releaseLOCK at 0xb7c5417c>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.