Galera won't start after all cluster nodes went down and 'mysqld_safe --recover' and 'mysqld_safe --wsrep-recover --tc-heuristic-recover=ROLLBACK' get error.

Solution Verified - Updated -

Issue

  • I am unable to get the galera service to start on two of the three control nodes.
  • pcs status shows failed actions where galera cannot start because it is unable to detect the last known writ sequence number.
Failed Actions:
* galera_start_0 on overcloud-controller-2 'unknown error' (1): call=4169, status=complete, exitreason='Unable to detect last known write sequence number',
    last-rc-change='Wed Feb 17 10:19:53 2016', queued=0ms, exec=2703ms
* galera_start_0 on overcloud-controller-0 'unknown error' (1): call=3943, status=complete, exitreason='Unable to detect last known write sequence number',
    last-rc-change='Wed Feb 17 16:20:20 2016', queued=0ms, exec=2736ms
  • attempting to recover the sequence number manually returns an error
# mysqld_safe --recover
160217 16:49:19 [ERROR] Found 1 prepared transactions! It means that mysqld was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start mysqld with --tc-heuristic-recover switch to commit or rollback pending transactions.
  • attempting to use the --tc-heuristic-recover switch produces an error
[root@overcloud-controller-2 ~]#  mysqld_safe --wsrep-recover  --tc-heuristic-recover=ROLLBACK
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
160217 10:41:41 [ERROR] mysqld got signal 6 ;

Environment

  • Red Hat OpenStack
  • MySQL Database with Galera
  • MariaDB Database with Galera

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content