Production Director experienced an FS error. DB corruption and last backup is > 7months

Issue

• The Production underclouds are VMs running on a dedicated OpenStack cloud.

• The director seem to have experienced an ext4 error due to underlying storage on May 27th.

• When the director came back up after fsck'ing the rootfs (June 1st(, mariadb went into a continuous crashloop showing a stack trace:

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x559b87b5229e]
/usr/libexec/mysqld(handle_fatal_signal+0x305)[0x559b87676925]
/lib64/libpthread.so.0(+0xf630)[0x7f096a348630]
/lib64/libc.so.6(gsignal+0x37)[0x7f0968a6f387]
/lib64/libc.so.6(abort+0x148)[0x7f0968a70a78]
/usr/libexec/mysqld(+0x847404)[0x559b87908404]
/usr/libexec/mysqld(+0x853817)[0x559b87914817]
/usr/libexec/mysqld(+0x85d02c)[0x559b8791e02c]
/usr/libexec/mysqld(+0x862b7e)[0x559b87923b7e]
/usr/libexec/mysqld(+0x7ea6b5)[0x559b878ab6b5]
/usr/libexec/mysqld(+0x7ec611)[0x559b878ad611]
/usr/libexec/mysqld(+0x7b1d4d)[0x559b87872d4d]
/usr/libexec/mysqld(+0x81a6ee)[0x559b878db6ee]
/usr/libexec/mysqld(+0x808486)[0x559b878c9486]
/lib64/libpthread.so.0(+0x7ea5)[0x7f096a340ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f0968b37b0d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

• It appears that the latest backups for this Director are 8months+ (Nov 2022).

• We have stopped mariadb for now and made a cold copy:

[root@director ~]# ls -ld /var/lib/mysql*
drwxr-xr-x. 16 mysql mysql 4096 Jun  2 12:03 /var/lib/mysql
drwxr-xr-x. 16 mysql mysql 4096 Jun  2 12:03 /var/lib/mysql.corrupted_20230602