The Production underclouds are VMs running on a dedicated OpenStack cloud.
The director seem to have experienced an ext4 error due to underlying storage on May 27th.
When the director came back up after fsck'ing the rootfs (June 1st(, mariadb went into a continuous crashloop showing a stack trace:
Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x40000 /usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x559b87b5229e] /usr/libexec/mysqld(handle_fatal_signal+0x305)[0x559b87676925] /lib64/libpthread.so.0(+0xf630)[0x7f096a348630] /lib64/libc.so.6(gsignal+0x37)[0x7f0968a6f387] /lib64/libc.so.6(abort+0x148)[0x7f0968a70a78] /usr/libexec/mysqld(+0x847404)[0x559b87908404] /usr/libexec/mysqld(+0x853817)[0x559b87914817] /usr/libexec/mysqld(+0x85d02c)[0x559b8791e02c] /usr/libexec/mysqld(+0x862b7e)[0x559b87923b7e] /usr/libexec/mysqld(+0x7ea6b5)[0x559b878ab6b5] /usr/libexec/mysqld(+0x7ec611)[0x559b878ad611] /usr/libexec/mysqld(+0x7b1d4d)[0x559b87872d4d] /usr/libexec/mysqld(+0x81a6ee)[0x559b878db6ee] /usr/libexec/mysqld(+0x808486)[0x559b878c9486] /lib64/libpthread.so.0(+0x7ea5)[0x7f096a340ea5] /lib64/libc.so.6(clone+0x6d)[0x7f0968b37b0d] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.
It appears that the latest backups for this Director are 8months+ (Nov 2022).
We have stopped mariadb for now and made a cold copy:
[root@director ~]# ls -ld /var/lib/mysql* drwxr-xr-x. 16 mysql mysql 4096 Jun 2 12:03 /var/lib/mysql drwxr-xr-x. 16 mysql mysql 4096 Jun 2 12:03 /var/lib/mysql.corrupted_20230602
- Red Hat OpenStack Platform 13.0 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.