Production Director experienced an FS error. DB corruption and last backup is > 7months

Solution In Progress - Updated -


  • The Production underclouds are VMs running on a dedicated OpenStack cloud.

  • The director seem to have experienced an ext4 error due to underlying storage on May 27th.

  • When the director came back up after fsck'ing the rootfs (June 1st(, mariadb went into a continuous crashloop showing a stack trace:

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
The manual page at contains
information that should help you find out what is causing the crash.
  • It appears that the latest backups for this Director are 8months+ (Nov 2022).

  • We have stopped mariadb for now and made a cold copy:

[root@director ~]# ls -ld /var/lib/mysql*
drwxr-xr-x. 16 mysql mysql 4096 Jun  2 12:03 /var/lib/mysql
drwxr-xr-x. 16 mysql mysql 4096 Jun  2 12:03 /var/lib/mysql.corrupted_20230602


  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content