Hawkular Cassandra shows no metrics when cluster exceeds a certain size

Issue

• On large clusters, Heapster will collect metrics but won't send them to Hawkular Cassandra. The metrics data in the web console will show blank.
• This has been observed in a few different versions of OpenShift 3 when workloads are spread across more than 100 nodes, with thousands of pods and hundreds of namespaces.

• Logs show the following:

  W0627 13:23:27.254891       1 manager.go:119] Failed to push data to sink: Hawkular-Metrics Sink
  W0627 13:23:57.298169       1 manager.go:119] Failed to push data to sink: Hawkular-Metrics Sink
  W0627 13:24:27.370221       1 manager.go:119] Failed to push data to sink: Hawkular-Metrics Sink
  W0627 13:24:57.286841       1 manager.go:119] Failed to push data to sink: Hawkular-Metrics Sink
  W0627 13:25:27.324231       1 manager.go:119] Failed to push data to sink: Hawkular-Metrics Sink
  W0627 13:25:57.726255       1 manager.go:119] Failed to push data to sink: Hawkular-Metrics Sink
  I0627 13:26:20.557520       1 manager.go:116] Data push completed: Hawkular-Metrics Sink
  I0627 13:26:54.871042       1 manager.go:116] Data push completed: Hawkular-Metrics Sink
  W0627 13:27:27.352292       1 manager.go:119] Failed to push data to sink: Hawkular-Metrics Sink
  I0627 13:27:37.318129       1 manager.go:116] Data push completed: Hawkular-Metrics Sink
  I0627 13:28:10.550017       1 manager.go:116] Data push completed: Hawkular-Metrics Sink
  I0627 13:28:44.764977       1 manager.go:116] Data push completed: Hawkular-Metrics Sink
  I0627 13:29:18.995184       1 manager.go:116] Data push completed: Hawkular-Metrics Sink
  I0627 13:29:53.684293       1 manager.go:116] Data push completed: Hawkular-Metrics Sink
  W0627 13:30:27.293908       1 manager.go:119] Failed to push data to sink: Hawkular-Metrics Sink
  I0627 13:30:37.356895       1 manager.go:116] Data push completed: Hawkular-Metrics Sink
  

• The metrics-server autoscaler still works, which shows that metrics data is still being collected by Heapster.