Galera will not start on one of the cluster nodes due to database corruption

Solution Verified - Updated -

Issue

  • Galera will not start on one of the cluster nodes and as a result other service which are dependent will not start either.

    # pcs status
    galera_start_0 on ctrl0 'unknown error' (1): call=222, status=complete, exit-reason='Slave instance did not start correctly in read-only mode,         Make sure local galera.cnf does not have wsrep_cluster_address set.', last-rc-change='Wed Sep  9 09:29:00 2015', queued=0ms, exec=2713m
    

    or

    Failed Actions:
    * galera_start_0 on overcloud-controller-0 'unknown error' (1): call=220, status=complete, exitreason='Unable to detect last known write sequence    number',
        last-rc-change='Fri Jan 22 09:26:03 2016', queued=0ms, exec=2533ms
    

From the mysql.log on the node which mysql will not start:

150909  9:29:02  InnoDB: Assertion failure in thread 139727328524352 in file fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
150909  9:29:02 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

Environment

  • Red Hat OpenStack

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content