Instances with volumes fail to start after a reboot of whole Red Hat OpenStack Platform environment

Issue

• After rebooting whole Red Hat OpenStack Platform environment, instances with volumes fail to start.

  $ openstack server show instance1 -c volumes_attached
  +------------------+--------------------------------------------------------------------------+
  | Field            | Value                                                                    |
  +------------------+--------------------------------------------------------------------------+
  | volumes_attached | delete_on_termination='False', id='cf28e3e0-2478-4d95-9b12-befc957e5dec' |
  +------------------+--------------------------------------------------------------------------+
    ==> Instance has attached volumes
  
  $ openstack server start instance1
  
    ...(wait for a while)...
  
  $ openstack server event list instance1 --long
  +------------------------------------------+--------------------------------------+---------------+----------------------------+---------+----------------------------------+----------------------------------+
  | Request ID                               | Server ID                            | Action        | Start Time                 | Message | Project ID                       | User ID                          |
  +------------------------------------------+--------------------------------------+---------------+----------------------------+---------+----------------------------------+----------------------------------+
  | req-aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa | aaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa | start         | 2023-03-13T05:38:45.000000 | Error   | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa |
    ==> Starting the instance failed with Error status
  

• The attached volumes shows error state

  $ openstack volume list
  +--------------------------------------+---------+--------+------+------------------------------------+
  | ID                                   | Name    | Status | Size | Attached to                        |              
  +--------------------------------------+---------+--------+------+------------------------------------+
  | cf28e3e0-2478-4d95-9b12-befc957e5dec | volume1 | error  |  100 | Attached to instance1 on /dev/vdb  |
  +--------------------------------------+---------+--------+------+------------------------------------+
  

• nova-compute.log shows error messages like the following example.

  ERROR os_brick.initiator.connectors.iscsi [- - - - -] Exception encountered during portal discovery: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  Command: iscsiadm -m discoverydb -o show -P 1
  Exit code: 21
  Stdout: 'SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nNo targets found.\nFIRMWARE:\nNo targets found.\n'
  Stderr: ''
  ERROR os_brick.initiator.connectors.iscsi Traceback (most recent call last):
     :
  ERROR os_brick.initiator.connectors.iscsi oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  ERROR os_brick.initiator.connectors.iscsi Command: iscsiadm -m discoverydb -o show -P 1
  ERROR os_brick.initiator.connectors.iscsi Exit code: 21
  ERROR os_brick.initiator.connectors.iscsi Stdout: 'SENDTARGETS:\nNo targets found.\niSNS:\nNo targets found.\nSTATIC:\nNo targets found.\nFIRMWARE:\nNo targets found.\n'
  ERROR os_brick.initiator.connectors.iscsi Stderr: ''
  ERROR os_brick.initiator.connectors.iscsi
  INFO nova.compute.manager [- - - - -] [instance: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa] Successfully reverted task state from powering-on on failure for instance.
  ERROR oslo_messaging.rpc.server [- - - - -] Exception during message handling: os_brick.exception.VolumeDeviceNotFound: Volume device not found at .