When redeploying getting the Failure prepping block device., Code: 500" error

Solution In Progress - Updated -

Issue

  • Overcloud deployment failed with this error:
ResourceInError: resources.Controller: Went to status ERROR due to "Message: Build of instance 31b53fad-c794-4d55-96cc-d64f18ff6652 aborted: Failure prepping block device., Code: 500"
  • An existing overcloud was previously deleted before another deployment attempt was done.

  • Overcloud nodes being deployed aren't obtaining an IP address but tcpdump seems to indicate the DHCP process replied:

 [root@undercloud ~]#  sudo tcpdump -i br-ctlplane | grep -i DHCP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-ctlplane, link-type EN10MB (Ethernet), capture size 262144 bytes
12:50:51.713028 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:41 (oui Unknown), length 347
12:50:51.714330 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:52.244759 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:79 (oui Unknown), length 347
12:50:52.245407 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:53.132358 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:31 (oui Unknown), length 347
12:50:53.132587 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:53.381357 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:01 (oui Unknown), length 347
12:50:53.381845 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:55.505469 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:41 (oui Unknown), length 347
12:50:55.506078 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:55.971510 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:79 (oui Unknown), length 347
12:50:55.971864 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:56.643771 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:31 (oui Unknown), length 347
12:50:56.645011 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:50:57.132227 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:01 (oui Unknown), length 347
12:50:57.132690 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:03.469801 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:41 (oui Unknown), length 347
12:51:03.470672 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:03.935859 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:79 (oui Unknown), length 347
12:51:03.936176 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:04.608063 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:31 (oui Unknown), length 347
12:51:04.608418 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:05.096579 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:01 (oui Unknown), length 347
12:51:05.097049 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:19.508559 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:41 (oui Unknown), length 347
12:51:19.510571 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:19.974505 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:79 (oui Unknown), length 347
12:51:19.974822 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:20.646680 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:31 (oui Unknown), length 347
12:51:20.647109 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:21.135255 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:00:00:00:00:01 (oui Unknown), length 347
12:51:21.135515 IP 10.10.10.20.bootps > 255.255.255.255.bootpc: BOOTP/DHCP, Reply, length 359
12:51:57.462730 IP6 0000::0000::0000::0000:cc01.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:52:02.110347 IP6 0000::0000::0000::0000:cc01.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:52:11.118295 IP6 0000::0000::0000::0000:cc01.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:52:28.090728 IP6 0000::0000::0000::0000:cc01.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:05.376769 IP6 0000::0000::0000::0000:c641.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:05.819622 IP6 0000::0000::0000::0000:d379.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:09.505584 IP6 0000::0000::0000::0000:c641.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:09.971604 IP6 0000::0000::0000::0000:d379.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:18.458615 IP6 0000::0000::0000::0000:c641.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
12:54:18.924605 IP6 0000::0000::0000::0000:d379.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
  • ironic-inspector dnsmasq process returns the following warnings:
 [root@undercloud ~]# tail -f /var/log/containers/ironic-inspector/dnsmasq.log
Feb 24 12:50:56 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:31 ignored
Feb 24 12:50:57 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:01 ignored
Feb 24 12:51:03 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:41 ignored
Feb 24 12:51:03 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:79 ignored
Feb 24 12:51:04 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:31 ignored
Feb 24 12:51:05 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:01 ignored
Feb 24 12:51:19 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:41 ignored
Feb 24 12:51:19 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:79 ignored
Feb 24 12:51:20 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:31 ignored
Feb 24 12:51:21 dnsmasq-dhcp[7]: DHCPDISCOVER(br-ctlplane) 00:00:00:00:00:01 ignored

Environment

  • Red Hat OpenStack Platform 15.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In