Diagnostic Steps

The server.log file contained the following messages:

16:01:01,484 WARN  [org.rhq.server.metrics.StorageSession] (http-/10.33.22.44:7080-16) Encountered NoHostAvailableException due to following error(s): {myExample.server.com/10.33.22.44=null}
16:01:01,513 INFO  [org.rhq.enterprise.server.storage.StorageClusterMonitor] (http-/10.33.22.44:7080-16) Storage cluster is down

At the same time the error is logged in the rhq-storage.log file:

 INFO [main] 2015-02-09 15:56:55,309 StorageService.java (line 1518) Node myExample.server.com/10.33.22.44 state jump to normal
 INFO [main] 2015-02-09 15:56:55,328 StorageService.java (line 859) Startup completed! Now serving reads.
 INFO [main] 2015-02-09 15:56:55,829 Server.java (line 134) Starting listening for CQL clients on myExample.server.com/10.33.22.44:9142...
 INFO [main] 2015-02-09 15:56:55,899 CassandraDaemon.java (line 404) Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
 INFO [ScheduledTasks:1] 2015-02-09 16:00:16,924 GCInspector.java (line 119) GC for Copy: 242 ms for 1 collections, 76005184 used; max is 1060372480
 INFO [ScheduledTasks:1] 2015-02-09 16:00:31,242 GCInspector.java (line 119) GC for Copy: 614 ms for 1 collections, 69591568 used; max is 1060372480
 INFO [MemoryMeter:1] 2015-02-09 16:00:57,267 Memtable.java (line 514) CFS(Keyspace='rhq', ColumnFamily='metrics_idx') liveRatio is 35.7473544973545 (just-counted was 7.494708994708994).  calculation took 684ms for 4959 columns
 INFO [ScheduledTasks:1] 2015-02-09 16:01:01,273 GCInspector.java (line 119) GC for Copy: 707 ms for 1 collections, 80597672 used; max is 1060372480
 INFO [MemoryMeter:1] 2015-02-09 16:01:05,519 Memtable.java (line 514) CFS(Keyspace='rhq', ColumnFamily='raw_metrics') liveRatio is 21.183778079300467 (just-counted was 21.183778079300467).  calculation took 8178ms for 6509 columns
ERROR [Native-Transport-Requests:231] 2015-02-09 16:01:09,064 ErrorMessage.java (line 210) Unexpected exception during request
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:192)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
 INFO [CompactionExecutor:2] 2015-02-09 16:01:53,689 CompactionTask.java (line 105) Compacting [SSTableReader(path='../../../rhq-data/data/system/peers/system-peers-ic-5-Data.db'), SSTableReader(path='../../../rhq-data/data/system/peers/system-peers-ic-4-Data.db'), SSTableReader(path='../../../rhq-data/data/system/peers/system-peers-ic-3-Data.db'), SSTableReader(path='../../../rhq-data/data/system/peers/system-peers-ic-2-Data.db'), SSTableReader(path='../../../rhq-data/data/system/peers/system-peers-ic-1-Data.db')]
...

Check routing and DNS addresses for both JBoss ON Server and storage node:

• netstat -nr;
• netstat -id;
• use dig or nslookup to look up the DNS name (for instance - nslookup myExample.server.com);
• check /etc/hosts file;