Cinder backup is taking hours due to No dm was created

Solution Verified - Updated -

Environment

  • Red Hat OpenStack Platform 13.0.16

Issue

Why the cinder backup looks stuck, there no errors and only the warning No dm was created?

Resolution

Specify CinderBackupOptVolumes parameter in environment cinder-backup template.

  1. Add the following to templates.
parameter_defaults:
  CinderBackupOptVolumes:
    - /etc/multipath.conf:/etc/multipath.conf:ro
    - /etc/multipath/:/etc/multipath/:rw

  1. Run a overcloud deploy.

  2. Check if multipath was configured on cinder backup container.

[root@controller01 ~]# docker exec -it `docker ps|grep -i cinder-backup|awk '{print $1}'` cat /etc/multipath.conf
# For a list of configuration options with descriptions, see the multipath.conf man page
...
[root@controller01 ~]# docker exec -it `docker ps|grep -i cinder-backup|awk '{print $1}'` multipath -ll
<id> <dm> <storage>
...

Root Cause

The parameter to use multipath was not defined in templates, so the cinder backup container blacklisted all devices and affected the time to do a backup.

The parameter CinderBackupOptVolumes was included only in Openstack 13.0.16.

Diagnostic Steps

  1. Enable debug on Cinder.

  2. Check for the logs.

2023-10-16 15:10:00.832 43 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): multipath -a 360002ac0000000000000022c00023073 execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:372
2023-10-16 15:10:01.648 43 DEBUG oslo_concurrency.processutils [-] CMD "multipath -a 360002ac0000000000000022c00023073" returned: 0 in 0.815s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:409
2023-10-16 15:10:01.648 43 DEBUG oslo.privsep.daemon [-] privsep: reply[139830818971728]: (4, ("Oct 16 15:10:01 | /etc/multipath.conf does not exist, blacklisting all devices.\nOct 16 15:10:01 | A default multipath.conf file is located at\nOct 16 15:10:01 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf\nOct 16 15:10:01 | You can run /sbin/mpathconf --enable to create\nOct 16 15:10:01 | /etc/multipath.conf. See man mpathconf(8) for more details\nwwid '360002ac0000000000000022c00023073' added\n", '')) loop /usr/lib/python2.7/site-packages/oslo_privsep/daemon.py:456
2023-10-16 15:10:01.649 8 WARNING os_brick.initiator.connectors.iscsi [req-00a7a83e-b212-4361-ac5a-b74cf626563f 9c190750478c428d96c449d3252e57c7 d3a98bf45f7a4b85ad6f906c94a24140 - default default] No dm was created, connection to volume is probably bad and will perform poorly.
  1. Check if the multipath was configured in the container.
[root@controller01 ~]# docker exec -it `docker ps|grep -i cinder-backup|awk '{print $1}'` cat /etc/multipath.conf
[root@controller01 ~]# docker exec -it `docker ps|grep -i cinder-backup|awk '{print $1}'` multipath -ll

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments