Some instances fail to launch with the error "Failure prepping block device: RemoteError: Remote error: DBConnectionError (pymysql.err.OperationalError)"

Issue

• Customers are reporting the inability to spin up some instances with the following error:

Build of instance XXXXXXXXXXXXXX aborted: Failure prepping block device.

• cinder-manage db purge 30 fails to archive deleted volumes:

()[root@overcloud-controller-01 /]# cat /var/spool/cron/cinder 
# HEADER: This file was autogenerated at 2019-11-16 08:33:00 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: cinder-manage db purge
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 0 * * * cinder-manage db purge 30 >>/var/log/cinder/cinder-rowsflush.log 2>&1
()[root@overcloud-controller-01 /]# cinder-manage db purge 30
Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
Purge command failed, check cinder-manage logs for more details.
()[root@overcloud-controller-01 /]# cinder-manage logs errors
Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
DEPRECATED: The log commands are deprecated since Queens and are not maintained. They will be removed in an upcoming release.
/var/log/cinder/cinder-manage.log:-
Line 14 : 2019-11-28 14:19:10.181 1649 ERROR cinder.db.sqlalchemy.api [req-f9a046ec-1386-4485-948d-3318cc74982a - - - - -] DBError detected when purging from volumes: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_attachment`, CONSTRAINT `volume_attachment_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))') [SQL: u'DELETE FROM volumes WHERE volumes.deleted_at < %(deleted_at_1)s'] [parameters: {u'deleted_at_1': datetime.datetime(2019, 10, 29, 14, 19, 6, 707533)}] (Background on this error at: http://sqlalche.me/e/gkpj).: DBReferenceError: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_attachment`, CONSTRAINT `volume_attachment_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))') [SQL: u'DELETE FROM volumes WHERE volumes.deleted_at < %(deleted_at_1)s'] [parameters: {u'deleted_at_1': datetime.datetime(2019, 10, 29, 14, 19, 6, 707533)}] (Background on this error at: http://sqlalche.me/e/gkpj)

• Deleted volumes are no longer archived:

MariaDB [cinder]> select deleted, count(*) from volumes group by deleted;
+---------+----------+
| deleted | count(*) |
+---------+----------+
|       0 |     3553 |
|       1 |   168908 |
+---------+----------+
2 rows in set (0.24 sec)