RHUA to CDS sync task fails with error "PulpException: Timeout occurred during sync; check the server log for more information"

Issue

• One of the CDS server is failing while synchronizing from RHUA server. We are getting the below error in the /var/log/pulp/client.log file of the CDS server.

2012-09-24 15:06:18,607 [ERROR][cds.example.com] run() @ consumer.py:72 - failed:
None
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/gofer/messaging/consumer.py", line 64, in run
    msg = self.__fetch(receiver)
  File "/usr/lib/python2.6/site-packages/gofer/messaging/consumer.py", line 100, in __fetch
    return receiver.fetch(timeout=self.WAIT)
  File "<string>", line 6, in fetch
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 998, in fetch
    self._ecwait(lambda: self.linked)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
    result = self._ewait(lambda: self.closed or predicate(), timeout)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 961, in _ewait
    result = self.session._ewait(lambda: self.error or predicate(), timeout)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 554, in _ewait
    result = self.connection._ewait(lambda: self.error or predicate(), timeout)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 197, in _ewait
    self.check_error()
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 190, in check_error
    raise self.error
InternalError: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 651, in write
    op.dispatch(self)
  File "/usr/lib/python2.6/site-packages/qpid/ops.py", line 84, in dispatch
    getattr(target, handler)(self, *args)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 864, in do_session_detached
    sst = self._sessions.pop(dtc.channel)
KeyError: 0

• While trying manual sync from the RHUA server to CDS, it fails with following error :

------ RHUI Manager -------
rhui (sync) => vc

Select a CDS: 
  1  - cds1.example.com
  2  - cds2.example.com
Enter value (1-2) or 'b' to abort: 2

  Start Time:     09-24-2012 09:37
  Finish Time:    09-24-2012 09:38
  Elapsed Time:   0:01:00
  Result:         Error
  Exception:      Timeout occurred during sync; check the server log for more information
  Traceback:
Traceback (most recent call last):

  File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 420, in run
    result = self.callable(*self.args, **self.kwargs)

  File "/usr/lib/python2.6/site-packages/pulp/server/api/cds.py", line 574, in cds_sync
    self.dispatcher.sync(cds, payload)

  File "/usr/lib/python2.6/site-packages/pulp/server/cds/dispatcher.py", line 137, in sync
    self._send(stub.sync, data)

  File "/usr/lib/python2.6/site-packages/pulp/server/cds/dispatcher.py", line 169, in _send
    result = func(*args)

  File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 72, in __call__
    return self.stub._send(request, opts)

  File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 133, in _send
    return self.__send(request, options)

  File "/usr/lib/python2.6/site-packages/gofer/rmi/stub.py", line 164, in __send
    any=opts.any)

  File "/usr/lib/python2.6/site-packages/gofer/rmi/policy.py", line 143, in send
    self.__getstarted(sn, reader)

  File "/usr/lib/python2.6/site-packages/gofer/rmi/policy.py", line 166, in __getstarted
    raise RequestTimeout(sn, 0)

PulpException: Timeout occurred during sync; check the server log for more information