Calamari upgrade from RHCS1.2.3 to RHCS1.3 fails with the message "LostRemote: Lost remote after 10s heartbeat".

Issue

• The Calamari node upgrade failed which was part of the upgrade process from RHCS1.2.3 to RHCS1.3.

• The upgrade process is detailed in RHCS1.3 Installation Guide

• The log file /var/log/calamari/calamari.log shows the following python trace-back.

2015-07-02 20:26:25,865 - ERROR - django.request Internal Server Error: /dashboard/
Traceback (most recent call last):
  File "/opt/calamari/venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/opt/calamari/venv/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 25, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/opt/calamari/venv/lib/python2.7/site-packages/calamari_web-0.1-py2.7.egg/calamari_web/views.py", line 38, in dashboard
    clusters = client.list_clusters()
  File "/opt/calamari/venv/lib/python2.7/site-packages/zerorpc/core.py", line 260, in <lambda>
    return lambda *args, **kargs: self(method, *args, **kargs)
  File "/opt/calamari/venv/lib/python2.7/site-packages/zerorpc/core.py", line 245, in __call__
    return self._process_response(request_event, bufchan, timeout)
  File "/opt/calamari/venv/lib/python2.7/site-packages/zerorpc/core.py", line 217, in _process_response
    reply_event = bufchan.recv(timeout)
  File "/opt/calamari/venv/lib/python2.7/site-packages/zerorpc/channel.py", line 267, in recv
    event = self._input_queue.get(timeout=timeout)
  File "/opt/calamari/venv/lib/python2.7/site-packages/gevent/queue.py", line 200, in get
    result = waiter.get()
  File "/opt/calamari/venv/lib/python2.7/site-packages/gevent/hub.py", line 568, in get
    return self.hub.switch()
  File "/opt/calamari/venv/lib/python2.7/site-packages/gevent/hub.py", line 331, in switch
    return greenlet.switch(self)
LostRemote: Lost remote after 10s heartbeat

• Cleaning up Clamari instances and initializing it again, didn't help.

# calamari-ctl clean

# calamari-ctl initialize