Metadata url not accessible right after the VM boot

Issue

• It seems that more and more VMs are starting to have issues in our new rhosp deployments. We need cloud init to be able to fully setup the VMs, but the initial call to the metadata service is failing and cloud-init gives up on that datasource. As you can see below, the call to the metadata url is made about 20 seconds after the VM is spawned, but it fails. When we try to console the VM after, the metadata url works just fine and we can manually run cloud-init without any issues so it seems that there is a longer delay for that url to be accessible.

• Do you know of any reason that this would be the case or how to further troubleshoot it? We don't have a VM right now in this situation, but when it happens again i will try to take some sosreports from the compute and the ctrls and see with out dev teams if we can leave that specific VM there for a bit without re-deploying, but if there is anything specific that we can try, please let us know.

• cloud-init inside the VM:

2022-04-18 12:10:04,919 - util.py[DEBUG]: Resolving URL: http://169.254.169.254
took 0.016 seconds
2022-04-18 12:10:04,920 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.22
54/openstack' with {'url': 'http://169.254.169.254/openstack', 'headers': {'Userr
-Agent': 'Cloud-Init/19.4'}, 'allow_redirects': True, 'method': 'GET', 'timeout''
: 10.0} configuration
2022-04-18 12:10:07,957 - url_helper.py[DEBUG]: Calling 'http://169.254.169.254//
openstack' failed [3/-1s]: request error [('Connection aborted.', error(111, 'Coo
nnection refused'))]
2022-04-18 12:10:07,958 - DataSourceOpenStack.py[DEBUG]: Giving up on OpenStack
md from ['http://169.254.169.254/openstack'] after 3 seconds

• metadata:

2022-04-18 12:09:46.950 229473 INFO networking_ovn.agent.metadata.agent [-] Port 6842bbbc-6e6a-41bb-ab73-3866532f5ff2 in datapath ff200762-8a1d-4b5b-87bb-b945459875c9 bound to our chassis

• nova:

2022-04-18 12:09:48.840 7 INFO nova.virt.libvirt.driver [-] [instance: 0aabdc87-4de2-4ae5-bcd7-bea34a3217d7] Instance spawned successfully.

• This happens only for the first VM created in the provider network and if a 2nd VM is created, it will always work.

• If there's only a single VM prevent in the provider network and it is soft rebooted (ctrl-alt-del), it will always be able to reach the metadata service but if it's rebooted with "nova reboot", it will always fail to reach the metadata service.