Galera won't start after all cluster nodes went down and 'mysqld_safe --recover' and 'mysqld_safe --wsrep-recover --tc-heuristic-recover=ROLLBACK' get error.

Solution Verified - Updated -

Issue

  • I am unable to get the galera service to start on two of the three control nodes.
  • pcs status shows failed actions where galera cannot start because it is unable to detect the last known writ sequence number.
Failed Actions:
* galera_start_0 on overcloud-controller-2 'unknown error' (1): call=4169, status=complete, exitreason='Unable to detect last known write sequence number',
    last-rc-change='Wed Feb 17 10:19:53 2016', queued=0ms, exec=2703ms
* galera_start_0 on overcloud-controller-0 'unknown error' (1): call=3943, status=complete, exitreason='Unable to detect last known write sequence number',
    last-rc-change='Wed Feb 17 16:20:20 2016', queued=0ms, exec=2736ms
  • attempting to recover the sequence number manually returns an error
# mysqld_safe --recover
160217 16:49:19 [ERROR] Found 1 prepared transactions! It means that mysqld was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start mysqld with --tc-heuristic-recover switch to commit or rollback pending transactions.
  • attempting to use the --tc-heuristic-recover switch produces an error
[root@overcloud-controller-2 ~]#  mysqld_safe --wsrep-recover  --tc-heuristic-recover=ROLLBACK
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
160217 10:41:41 [ERROR] mysqld got signal 6 ;

Environment

  • Red Hat OpenStack
  • MySQL Database with Galera
  • MariaDB Database with Galera

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In