After upgrade from JBoss ON 3.2.3 to 3.3.0, some rhq column families are unavailable and compaction operations fail

Solution Verified - Updated -

Issue

  • After upgrade from JBoss ON 3.2.3 to JBoss ON 3.3.0, some column families are reported as unavailable: one_hour_metrics, six_hour_metrics, twenty_four_hour_metrics and metrics_index.
  • On disk the corresponding directories are not accessed any more and previously scheduled manual compaction task fails.
  • Manual compaction operation fails after JBoss ON upgrade from 3.2.3 to 3.3.0
  • Manual compaction operation fails with the exception:
org.mc4j.ems.connection.EmsInvocationException: Exception on invocation of [forceTableCompaction]javax.management.RuntimeMBeanException: java.lang.IllegalArgumentException: Unknown table/cf pair (rhq.metrics_index)
    at org.mc4j.ems.impl.jmx.connection.bean.operation.DOperation.invoke(DOperation.java:133)
    at org.rhq.plugins.cassandra.util.KeyspaceService.compact(KeyspaceService.java:101)
    at org.rhq.plugins.cassandra.KeyspaceComponent.compactKeyspace(KeyspaceComponent.java:168)
    at org.rhq.plugins.cassandra.ColumnFamilyComponent.invokeOperation(ColumnFamilyComponent.java:70)
...
Caused by: javax.management.RuntimeMBeanException: java.lang.IllegalArgumentException: Unknown table/cf pair (rhq.metrics_index)
...
Caused by: java.lang.IllegalArgumentException: Unknown table/cf pair (rhq.metrics_index)
    at org.apache.cassandra.db.Table.getColumnFamilyStore(Table.java:168)
    at org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2407)
    at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2246)
...

Environment

  • Red Hat JBoss Operations Network (JON) - upgrade done from 3.2.3 to 3.3.0 version;
  • In 3.2.3 version - scheduled Force Major Column Family Compaction operation for folders: metrics_index, one_hour_metrics, raw_metrics, schema_version, six_hour_metrics and twenty_four_hour_metrics.
  • Above operation is scheduled to be executed now and repeated indefinitely.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.