Undercloud or overcloud keystone fails to start during FFU from 16.2 to 17.1

Issue

• We're performing major upgrade of OSP from 16.2 to 17.1. The upgrade is failing on:

2023-12-22 14:05:36.673470 | 14187734-7430-657b-ca63-00000000f3a0 |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | ctrl-01 | error={"changed": false, "msg": "Failed containers: keystone_db_sync, keystone_boo$
strap"}

• We can see the keystone container is excited:

[root@ctrl-01 tripleo-config]# podman ps -a | grep 'Exited (1'
fb8a314b7d94  registry.redhat.io/rhosp-rhel9/openstack-keystone:17.1                                  11 days ago  Exited (1) 10 days ago              container-puppet-keystone
ebc9173c702b  registry.redhat.io/rhosp-rhel9/openstack-keystone:17.1            /usr/bin/bootstra...  10 days ago  Exited (1) 10 days ago              keystone_db_sync
1b39f779dd48  registry.redhat.io/rhosp-rhel9/openstack-keystone:17.1            kolla_start           10 days ago  Exited (1) 10 days ago              keystone

• And we can see /var/lib/config-data/puppet-generated/keystone/etc/httpd/conf.modules.d is missing :

[root@ctrl-01 tripleo-config]# tail /var/log/containers/stdouts/keystone_db_sync.log.1
2023-12-22T19:34:13.766176573+05:30 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/substitute.load to /etc/httpd/conf.d/substitute.load
2023-12-22T19:34:13.766452471+05:30 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/suexec.load to /etc/httpd/conf.d/suexec.load
2023-12-22T19:34:13.766807321+05:30 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/systemd.load to /etc/httpd/conf.d/systemd.load
2023-12-22T19:34:13.767118039+05:30 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/unixd.load to /etc/httpd/conf.d/unixd.load
2023-12-22T19:34:13.767395120+05:30 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/usertrack.load to /etc/httpd/conf.d/usertrack.load
2023-12-22T19:34:13.767654350+05:30 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/version.load to /etc/httpd/conf.d/version.load
2023-12-22T19:34:13.767958303+05:30 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/vhost_alias.load to /etc/httpd/conf.d/vhost_alias.load
2023-12-22T19:34:13.768242488+05:30 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/wsgi.conf to /etc/httpd/conf.d/wsgi.conf
2023-12-22T19:34:13.768528440+05:30 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/httpd/conf.d/wsgi.load to /etc/httpd/conf.d/wsgi.load
2023-12-22T19:34:13.768817053+05:30 stderr F ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/src/etc/httpd/conf.modules.d file is not found