JON fails to show metrics for monitored resources due storage node AuthenticationException

Solution Verified - Updated -

Issue

  • Stopped showing metrics for all A-MQ queues
  • Stopped receiving any alerts due to metrics not collected
  • Metrics are not being stored by the storage node
  • Storage client subsystem wasn't initialized
  • Server log contains the following warning:

    WARN  [org.rhq.enterprise.server.storage.StorageClientManagerBean] (pool-7-thread-1) Storage client subsystem wasn't initialized. The RHQ server will be set to MAINTENANCE mode. Please verify  that the storage cluster is operational.: java.lang.RuntimeException: Authentication error on host jboss-on-storage-01.example.com/192.168.5.50: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level ONE
        at org.rhq.enterprise.server.storage.StorageClientManagerBean.checkSchemaCompability(StorageClientManagerBean.java:257) [rhq-server.jar:4.9.0.JON320GA]
        at org.rhq.enterprise.server.storage.StorageClientManagerBean.createSession(StorageClientManagerBean.java:341) [rhq-server.jar:4.9.0.JON320GA]
        at org.rhq.enterprise.server.storage.StorageClientManagerBean.init(StorageClientManagerBean.java:154) [rhq-server.jar:4.9.0.JON320GA]
        ...
        at org.rhq.enterprise.server.storage.StorageClientManagerBean$$$view133.init(Unknown Source) [rhq-server.jar:4.9.0.JON320GA]
        at org.rhq.enterprise.server.core.StartupBean.initStorageClient(StartupBean.java:485) [rhq-server.jar:4.9.0.JON320GA]
        at org.rhq.enterprise.server.core.StartupBean.init(StartupBean.java:200) [rhq-server.jar:4.9.0.JON320GA]
        ...
    Caused by: com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host jboss-on-storage-01.example.com/192.168.5.50: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level ONE
        at com.datastax.driver.core.Connection.initializeTransport(Connection.java:168) [cassandra-driver-core-1.0.2.jar:]
        at com.datastax.driver.core.Connection.<init>(Connection.java:129) [cassandra-driver-core-1.0.2.jar:]
        at com.datastax.driver.core.Connection.<init>(Connection.java:56) [cassandra-driver-core-1.0.2.jar:]
        at com.datastax.driver.core.Connection$Factory.open(Connection.java:412) [cassandra-driver-core-1.0.2.jar:]
        at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:205) [cassandra-driver-core-1.0.2.jar:]
        ...
        at org.rhq.cassandra.util.ClusterBuilder.build(ClusterBuilder.java:130) [rhq-cassandra-util-4.9.0.JON320GA.jar:4.9.0.JON320GA]
        at org.rhq.cassandra.schema.SessionManager.initSession(SessionManager.java:33) [rhq-cassandra-schema-4.9.0.JON320GA.jar:4.9.0.JON320GA]
        at org.rhq.cassandra.schema.AbstractManager.initClusterSession(AbstractManager.java:100) [rhq-cassandra-schema-4.9.0.JON320GA.jar:4.9.0.JON320GA]
        at org.rhq.cassandra.schema.AbstractManager.initClusterSession(AbstractManager.java:90) [rhq-cassandra-schema-4.9.0.JON320GA.jar:4.9.0.JON320GA]
        at org.rhq.cassandra.schema.VersionManager.checkCompatibility(VersionManager.java:257) [rhq-cassandra-schema-4.9.0.JON320GA.jar:4.9.0.JON320GA]
        at org.rhq.cassandra.schema.SchemaManager.checkCompatibility(SchemaManager.java:112) [rhq-cassandra-schema-4.9.0.JON320GA.jar:4.9.0.JON320GA]
        at org.rhq.enterprise.server.storage.StorageClientManagerBean.checkSchemaCompability(StorageClientManagerBean.java:253) [rhq-server.jar:4.9.0.JON320GA]
        ... 149 more
    

Environment

  • Red Hat JBoss Operations Network (ON) 3.2
  • At least two JBoss ON storage nodes installed in a single storage cluster

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.