Issue

• THe metrics were running fine for a couple of days and today we saw multiple restarts for all metrics pods.
  I put rc of all to 0 and then tried to restart them one by one.
  I first started the cassandra pod and there i saw this message
  UnknownHostException for service 'hawkular-cassandra-nodes'

I have attached the logs to this ticket

I alaso found this bugzilla
https://bugzilla.redhat.com/show_bug.cgi?id=1307170

But it says upgrade to 3.2

How can we solve it in 3.1?

• #oc get events

oc get events

FIRSTSEEN   LASTSEEN   COUNT     NAME                         KIND      SUBOBJECT                               REASON    SOURCE                                       MESSAGE
4h          4m         55        hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Pulling   {kubelethost.example.com}   pulling image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
4h          4m         55        hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Pulled    {kubelethost.example.com}   Successfully pulled image "registry.access.redhat.com/openshift3/metrics-cassandra:latest"
4h          3s         1548      hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Backoff   {kubelethost.example.com}   Back-off restarting failed docker container
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Created   {kubelethost.example.com}   Created with docker id 0086b391997c
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Started   {kubelethost.example.com}   Started with docker id 0086b391997c
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Killing   {kubelethost.example.com}   Killing with docker id 0086b391997c
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Created   {kubelethost.example.com}   Created with docker id 22acd0882784
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Started   {kubelethost.example.com}   Started with docker id 22acd0882784
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Killing   {kubelethost.example.com}   Killing with docker id 22acd0882784
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Created   {kubelethost.example.com}   Created with docker id 0561b05466d2
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Started   {kubelethost.example.com}   Started with docker id 0561b05466d2
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Killing   {kubelethost.example.com}   Killing with docker id 0561b05466d2
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Created   {kubelethost.example.com}   Created with docker id 1d91373f29fc
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Started   {kubelethost.example.com}   Started with docker id 1d91373f29fc
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Killing   {kubelethost.example.com}   Killing with docker id 1d91373f29fc
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Created   {kubelethost.example.com}   Created with docker id 4300ab0c3109
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Started   {kubelethost.example.com}   Started with docker id 4300ab0c3109
1h          1h         1         hawkular-cassandra-1-2l6x3   Pod       spec.containers{hawkular-cassandra-1}   Killing   {kubelethost.example.com}   Killing with docker id 4300ab0c3109

• post-start hooks from cassandra and hawkular rcs removed to make the them stop restarting/crashing

Now cassandra and hawkular get into a running state.

Heapster gets into an an error state and keeps crashing

heapster-logs-----
oc logs heapster-po2e0 -n openshift-infra -p
exec: "./heapster-wrapper.sh": stat ./heapster-wrapper.sh: no such file or directory

Heapster image used in rc
image: registry.access.redhat.com/openshift3/metrics-heapster:latest