When redeploying getting the Failure prepping block device., Code: 500" error

Solution In Progress - Updated -

Issue

  • Overcloud deployment failed with this error:
ResourceInError: resources.Controller: Went to status ERROR due to "Message: Build of instance 31b53fad-c794-4d55-96cc-d64f18ff6652 aborted: Failure prepping block device., Code: 500"

  • An existing overcloud was previously deleted before another deployment attempt was done.

  • Overcloud nodes being deployed aren't obtaining an IP address but tcpdump seems to indicate the DHCP process replied:

 [root@undercloud ~]#  sudo tcpdump -i br-ctlplane | grep -i DHCP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-ctlplane, link-type EN10MB (Ethernet), capture size 262144 bytes
12:50:51.713028 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:41 (oui Unknown), length 347
12:50:51.714330 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:52.244759 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:79 (oui Unknown), length 347
12:50:52.245407 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:53.132358 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:31 (oui Unknown), length 347
12:50:53.132587 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:53.381357 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:01 (oui Unknown), length 347
12:50:53.381845 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:55.505469 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:41 (oui Unknown), length 347
12:50:55.506078 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:55.971510 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:79 (oui Unknown), length 347
12:50:55.971864 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:56.643771 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:31 (oui Unknown), length 347
12:50:56.645011 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:57.132227 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:01 (oui Unknown), length 347
12:50:57.132690 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:03.469801 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:41 (oui Unknown), length 347
12:51:03.470672 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:03.935859 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:79 (oui Unknown), length 347
12:51:03.936176 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:04.608063 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:31 (oui Unknown), length 347
12:51:04.608418 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:05.096579 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:01 (oui Unknown), length 347
12:51:05.097049 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:19.508559 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:41 (oui Unknown), length 347
12:51:19.510571 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:19.974505 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:79 (oui Unknown), length 347
12:51:19.974822 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:20.646680 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:31 (oui Unknown), length 347
12:51:20.647109 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:21.135255 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:01 (oui Unknown), length 347
12:51:21.135515 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:57.462730 IP6 0000::0000::0000::0000:cc01.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:52:02.110347 IP6 0000::0000::0000::0000:cc01.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:52:11.118295 IP6 0000::0000::0000::0000:cc01.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:52:28.090728 IP6 0000::0000::0000::0000:cc01.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:05.376769 IP6 0000::0000::0000::0000:c641.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:05.819622 IP6 0000::0000::0000::0000:d379.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:09.505584 IP6 0000::0000::0000::0000:c641.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:09.971604 IP6 0000::0000::0000::0000:d379.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:18.458615 IP6 0000::0000::0000::0000:c641.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:18.924605 IP6 0000::0000::0000::0000:d379.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
  • ironic-inspector dnsmasq process returns the following warnings:
 [root@undercloud ~]# tail -f /var/log/containers/ironic-inspector/dnsmasq.log
Feb 24 12:50:56 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:31 ignored
Feb 24 12:50:57 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:01 ignored
Feb 24 12:51:03 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:41 ignored
Feb 24 12:51:03 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:79 ignored
Feb 24 12:51:04 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:31 ignored
Feb 24 12:51:05 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:01 ignored
Feb 24 12:51:19 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:41 ignored
Feb 24 12:51:19 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:79 ignored
Feb 24 12:51:20 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:31 ignored
Feb 24 12:51:21 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:01 ignored

Environment

  • Red Hat OpenStack Platform 15.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content