Galera won't start after all cluster nodes went down and 'mysqld_safe --recover' and 'mysqld_safe --wsrep-recover --tc-heuristic-recover=ROLLBACK' get error.
Issue
- I am unable to get the galera service to start on two of the three control nodes.
pcs status
shows failed actions where galera cannot start because it is unable to detect the last known writ sequence number.
Failed Actions:
* galera_start_0 on overcloud-controller-2 'unknown error' (1): call=4169, status=complete, exitreason='Unable to detect last known write sequence number',
last-rc-change='Wed Feb 17 10:19:53 2016', queued=0ms, exec=2703ms
* galera_start_0 on overcloud-controller-0 'unknown error' (1): call=3943, status=complete, exitreason='Unable to detect last known write sequence number',
last-rc-change='Wed Feb 17 16:20:20 2016', queued=0ms, exec=2736ms
- attempting to recover the sequence number manually returns an error
# mysqld_safe --recover
160217 16:49:19 [ERROR] Found 1 prepared transactions! It means that mysqld was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start mysqld with --tc-heuristic-recover switch to commit or rollback pending transactions.
- attempting to use the
--tc-heuristic-recover
switch produces an error
[root@overcloud-controller-2 ~]# mysqld_safe --wsrep-recover --tc-heuristic-recover=ROLLBACK
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
160217 10:41:41 [ERROR] mysqld got signal 6 ;
Environment
- Red Hat OpenStack
- MySQL Database with Galera
- MariaDB Database with Galera
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.