Galera will not start on one of the cluster nodes due to database corruption
Issue
-
Galera will not start on one of the cluster nodes and as a result other service which are dependent will not start either.
# pcs status galera_start_0 on ctrl0 'unknown error' (1): call=222, status=complete, exit-reason='Slave instance did not start correctly in read-only mode, Make sure local galera.cnf does not have wsrep_cluster_address set.', last-rc-change='Wed Sep 9 09:29:00 2015', queued=0ms, exec=2713m
or
Failed Actions: * galera_start_0 on overcloud-controller-0 'unknown error' (1): call=220, status=complete, exitreason='Unable to detect last known write sequence number', last-rc-change='Fri Jan 22 09:26:03 2016', queued=0ms, exec=2533ms
From the mysql.log
on the node which mysql will not start:
150909 9:29:02 InnoDB: Assertion failure in thread 139727328524352 in file fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
150909 9:29:02 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
Environment
- Red Hat OpenStack
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.