After upgrade from JBoss ON 3.2.3 to 3.3.0, some rhq column families are unavailable and compaction operations fail

Solution Verified - Updated -

Issue

  • After upgrade from JBoss ON 3.2.3 to JBoss ON 3.3.0, some column families are reported as unavailable: one_hour_metrics, six_hour_metrics, twenty_four_hour_metrics and metrics_index.
  • On disk the corresponding directories are not accessed any more and previously scheduled manual compaction task fails.
  • Manual compaction operation fails after JBoss ON upgrade from 3.2.3 to 3.3.0
  • Manual compaction operation fails with the exception:
org.mc4j.ems.connection.EmsInvocationException: Exception on invocation of [forceTableCompaction]javax.management.RuntimeMBeanException: java.lang.IllegalArgumentException: Unknown table/cf pair (rhq.metrics_index)
    at org.mc4j.ems.impl.jmx.connection.bean.operation.DOperation.invoke(DOperation.java:133)
    at org.rhq.plugins.cassandra.util.KeyspaceService.compact(KeyspaceService.java:101)
    at org.rhq.plugins.cassandra.KeyspaceComponent.compactKeyspace(KeyspaceComponent.java:168)
    at org.rhq.plugins.cassandra.ColumnFamilyComponent.invokeOperation(ColumnFamilyComponent.java:70)
...
Caused by: javax.management.RuntimeMBeanException: java.lang.IllegalArgumentException: Unknown table/cf pair (rhq.metrics_index)
...
Caused by: java.lang.IllegalArgumentException: Unknown table/cf pair (rhq.metrics_index)
    at org.apache.cassandra.db.Table.getColumnFamilyStore(Table.java:168)
    at org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2407)
    at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2246)
...

Environment

  • Red Hat JBoss Operations Network (JON) - upgrade done from 3.2.3 to 3.3.0 version;
  • In 3.2.3 version - scheduled Force Major Column Family Compaction operation for folders: metrics_index, one_hour_metrics, raw_metrics, schema_version, six_hour_metrics and twenty_four_hour_metrics.
  • Above operation is scheduled to be executed now and repeated indefinitely.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content