Stack Update Failing at Container Image Prep Image Not Found Error | Manifest unknown

Issue

Describe your problem. Include specific actions and error messages.
We were implementing the recommended changes from RedHat support team for another case id-04006481.
While running the stack update we encountered the issue and it failed at container image prep step stating Image Not Found . On re-run it failed again at the same step for other container image. Below are the logs. Also attaching the sosreport from undercloud/director VM.

Note: Initially, during the deployment of this OpenStack 17.1 cluster in 2024 the RedHat Satellite version was 6.14. which got recently upgraded to version 6.16 s 6.14 was coming to end of support.

Error captured in Ansible.log

#

(undercloud) [stack@dutbospdir02 ~]$ grep -i "Not found image" ansible.log
2025-03-06 15:18:22,888 p=1001444 u=stack n=ansible | 2025-03-06 15:18:22.886907 | 00505690-2217-17f0-c1e8-00000001c095 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: https://dutbsat02.tbiptvcloud.meydan.du/v2/dutbsat-osp17_containers-octavia-worker/manifests/17.1", "msg": "Error running container image prepare: Not found image: https://dutbsat02.tbiptvcloud.meydan.du/v2/dutbsat-osp17_containers-octavia-worker/manifests/17.1", "params": {}, "success": false}
2025-03-06 16:08:03,939 p=24414 u=stack n=ansible | 2025-03-06 16:08:03.937850 | 00505690-2217-b12f-592f-00000001bed5 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: https://dutbsat02.tbiptvcloud.meydan.du/v2/dutbsat-osp17_containers-mariadb/manifests/17.1", "msg": "Error running container image prepare: Not found image: https://dutbsat02.tbiptvcloud.meydan.du/v2/dutbsat-osp17_containers-mariadb/manifests/17.1", "params": {}, "success": false}
2025-03-06 17:20:10,030 p=117679 u=stack n=ansible | 2025-03-06 17:20:10.029198 | 00505690-2217-fdcd-f42f-00000001bef7 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: https://dutbsat02.tbiptvcloud.meydan.du/v2/dutbsat-osp17_containers-octavia-worker/manifests/17.1", "msg": "Error running container image prepare: Not found image: https://dutbsat02.tbiptvcloud.meydan.du/v2/dutbsat-osp17_containers-octavia-worker/manifests/17.1", "params": {}, "success": false}
(undercloud) [stack@dutbospdir02 ~]$

When manually curl we found this issue:

#

(undercloud) [stack@dutbospdir02 ~]$ curl -k -u director:s3cret.123! https://dutbsat02.tbiptvcloud.meydan.du/v2/dutbsat-osp17_containers-octavia-worker/manifests/17.1
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"Manifest not found.","detail":{"reference":"17.1"}}]}(undercloud) [stack@dutbospdir02 ~]$

[stack@dutbospdir02 ~]$ curl -k -u director:s3cret.123! https://dutbsat02.tbiptvcloud.meydan.du/v2/dutbsat-osp17_containers-rsyslog/tags/list
{"name":"dutbsat-osp17_containers-rsyslog","tags":["17.0","17.0.0","17.0.0-100","17.0.0-103","17.0.0-103.1667372749","17.0.0-107","17.0.1","17.0.1-11","17.0.1-13","17.0.1-15","17.0.1-17","17.0.1-18","17.0.1-5","17.0.1-7","17.0.1-7.1679399904","17.0.1-8","17.1","17.1.0","17.1.0-67","17.1.1","17.1.1-3","17.1.1-4","17.1.2","17.1.2-5","17.1.3","17.1.3-10","17.1.3-11","17.1.3-11.1724039701","17.1.3-11.1726787565","17.1.3-11.1729271197","17.1.3-8","17.1.3-8.1716909125","17.1.4","17.1.4-4","17.1.4-4.1733938165","17.1.4-4.1738707099","17.1.4-4.1740076576","sha256-02b29a588f268b267d85b87905f5952ed914f72d07056fc66fb81315430fc0bd.sig","sha256-03b54a2af1c15848f2c56f72247a7ebf608190b90a194398a23deb10d36cfbdd.sig","sha256-0725cfe491168d2ceca106ef1633522b0d105dc648dc7ed3e77c62577c4fd68f.sig","sha256-09cafe53828d5543d20a9d80531068c21d7814cd0d3388f2fa6f7268990856a5.sig","sha256-0ac2fb809feb1005040525b7e584bfb31487431b797db119ede8e8e6c6f5de1d.sig","sha256-0ebc77111edffe23c292247596e0d414c5230fe4e39c92e33ce7895895da106a.att","sha256-0ebc77111edffe23c292247596e0d414c5230fe4e39c92e33ce7895895da106a.sig","sha256-0fe4862f685311713e85472c36713434352b5fcdb3b3be566dbcc40db820b805.sig","sha256-15955997be84aa96c20b3e33d49eecb89ac4da577588309f0ce17ec500c85546.att","sha256-15955997be84aa96c20b3e33d49eecb89ac4da577588309f0ce17ec500c85546.sig","sha256-16922a184811a459ea2c30de57adcd1a2276f9a8067f750d91fcfba1cc8b4f97.att","sha256-16922a184811a459ea2c30de57adcd1a2276f9a8067f750d91fcfba1cc8b4f97.sig","sha256-16bf290e9028a359ca0ff431e1da4dbf541fd1a46d310d3066c8383d2f49388e.sig","sha256-17514251d87c7786fe298a67bc94eede395ebc37870d493b7c3bd2145d6fcd5d.att","sha256-17514251d87c7786fe298a67bc94eede395ebc37870d493b7c3bd2145d6fcd5d.sig","sha256-183f185eee17fe915800f845dcec921f285b601c7e3bed303b0e83657cc7ee81.sig","sha256-18e02f7fe0913c0dd66654e83023e8166b5936219190ee72f2e2cf30bc45e31b.sig","sha256-1b85612bcc9df79efcc41a811ff30af56e2a98ad4144fccc8340814603daeb2c.att","sha256-1b85612bcc9df79efcc41a811ff30af56e2a98ad4144fccc8340814603daeb2c.sig","sha256-1b909a30e9a9dec5f9ef4720332858a15c534e003511d7e384fd01a3491df728.sig","sha256-1e015406655845ca9599ab22e45f105a939deae7d34185b4947feb55c76991d3.sig","sha256-29c8d99cb007b819c6dfda848f05e163c512d240b7642bc86a4c334c70a1cef7.att","sha256-29c8d99cb007b819c6dfda848f05e163c512d240b7642bc86a4c334c70a1cef7.sig","sha256-2a97cf0008dca926fdcc5d224db882d43b6ccd0c05463a2a0d7d10ac682fbfd7.sig","sha256-2c2d29e0ca05f5340cb10d1a62568f14af465e6379b3a6ebc6ac40270658c702.att","sha256-2c2d29e0ca05f5340cb10d1a62568f14af465e6379b3a6ebc6ac40270658c702.sig","sha256-2d61ce6f17601ea90e377383d8b72551352231f15b793a719840150a88d6e002.sig","sha256-2d6b6b01c5539d367af6c84029bc9bfa875823c254d1d6a28e4f6dfade4b2908.sig","sha256-30b283603872bcdee9d916c5696a87d9929b59406c76e1e0666c32dbe9003914.sig","sha256-3303befde8bfc68a525484c89b07a9a864cd994b17807cdba3f70dbd3675d420.sig","sha256-344104ea8f221fb1db5d38c331ebe01729313d9e4580a157143b1fa4bfdd9147.sig","sha256-34ed003282de4dcd19b423a8789a7787d8ab9dd4557f67b89b78c41318d4f568.sig","sha256-37633b460e38795adc6f1186bad9d0fd12558257731fe215d1ebafa90e3a3134.sig","sha256-3b5fa9cf53c40a0c6f03b4d9506f3a2a313cd8ae1c83a285883eb57db4768c35.sig","sha256-3ee49bb5fd4840ec1291c6309a87e7b473e2ebdd53bdba07aedf92f411f5333c.sig","sha256-424d0d064013691563017ddfd87cfc33b333ffb6874632cc6cf6a8479c9ea720.sig","sha256-425e3191ad4aa40146ad98a4c3f41de0e0e7c7d66e76365c85387070171a5c88.sig","sha256-433407de731eb2f92b58aa11cd23ece5dc6955f853d7b568815070fde7c2632c.att","sha256-433407de731eb2f92b58aa11cd23ece5dc6955f853d7b568815070fde7c2632c.sig","sha256-43669c60c049f8b9e65019145aa547401e0293c735d5ec01c62b467cfdb6563e.sig","sha256-45c20c643b6eff7ff1b260c2e38090bd2cd81445c8ba453a4680a5d95f827b8f.sig","sha256-49fd892f1eea93e4ba6d0e6094b3496f777702507105d756e5a8504cd7159492.sig","sha256-4c47c0d16c02f9c1ddad876a88765a858d1be6a12c9cef8c69dc82af32658bf5.sig","sha256-4f6fea81b43d552180c21ed70b66952fb2318999b56fcd2faa50b2f670d35242.sig","sha256-53e180b46fb0b279eac67651b5ef38d1caf8a8684647a94376d7cbe5438455dc.sig","sha256-544c153992288666b8ac7535a450e7d6b4f1df08ee7a867406895e2105cbee14.sig","sha256-561ceeadec465cfd72743cb4eb52d646605daf328eb198bbbbd6210dcba8ec60.sig","sha256-56e1255a2a874816ddfc29bdcc078e363df1d0a49ad84da99ceeb3d950890128.sig","sha256-56e97dc7a846a0716682c79c5e090c807e2aac07b483cf104cead9093dced8d3.att","sha256-56e97dc7a846a0716682c79c5e090c807e2aac07b483cf104cead9093dced8d3.sig","sha256-5859059fc59fc0a8553904c127765b52f47d4b03adf3be693f1d97a3829cff6a.sig","sha256-585b0ea4f5796f2c398692b57a4975076217b0f8cb2855a2688c9d2fd51ce7d4.sig","sha256-59f79ef31a76e93dfc43ef717470722cec346e723c2002f0c5bb453d141be31e.sig","sha256-5b4080d590bf45cd242235ff1ef050ab0b319d1fc1c86b46cf5a3ad0713301f7.sig","sha256-5ba566c5fa005a47cea3c3417f3f041e4e100659b9efe72e3e5b84b424b501ab.sig","sha256-5c016f54be7bf3ca5d3c60144906205bce48e3be972d9500e12557f573ba9685.sig","sha256-5c8ad39c4dd91d21ca53043dbbdbd5f8ac25b977c1d3b8aa2afea56fc4e4ef55.sig","sha256-5efc8358890012d127d4db019e181e584076c7e4a6de15fd28d1a0b14f81e23c.sig","sha256-620865c672001979962d97c23f9ed6a0afbe2f4d5e550d6fa9d841b9fe65aa94.sig","sha256-6232eaa4a526e93649a5d2d5344f0995b3b035ae28ef020857c4c0da38d25931.sig","sha256-63a730881a96686b289fa0e6a703e525258cc28a4f35ed136af044723b933cd5.sig","sha256-6543b625d170687e016fbaf7255bbf9b600a8718aa0e2f41f05772d49de2a68d.sig","sha256-65dc3596cfca0b451918773472ee83751fd62aa972d5800aa1c0cb8ea9a26c95.sig","sha256-674bd434d3d31b096ecd1ef53da11bcef90f4e5a8f3261a102b66641af23c628.sig","sha256-6c30b4aacd66ffe0fda5ab15d4090735878407dec54eaff75d99383bf15bee29.sig","sha256-6d9e4c206d9dcc06d3f568d6ac7823bbad2beac130db5c98f9400ba28651a233.sig","sha256-6df9b07a79dcb8a80c65bf97e62b1f415cd60da885d3855ef9b5a04dc25f1ed1.sig","sha256-72642577efb5bc131c143aad1cdba264cfdf3ad3b4ea81342e2fef126b9fa2c3.att","sha256-72642577efb5bc131c143aad1cdba264cfdf3ad3b4ea81342e2fef126b9fa2c3.sig","sha256-7537cc9dd0fa5eb447fc49162e8afa36c7874aeb5d4a9b8a1387fb573ad5b732.att","sha256-7537cc9dd0fa5eb447fc49162e8afa36c7874aeb5d4a9b8a1387fb573ad5b732.sig","sha256-76b0038f65a22819146d6f865f00a97ba778eb67ee3f4bbd923e7aca0f1ecddf.sig","sha256-7e9b9453423c16eb300455ecb47a5abb48ad73f135dd6ca2dc10e78874228b49.sig","sha256-7ea5c5e7740eddea16a6111101a8bd2c9ea7e7153c341699464789e68299c572.sig","sha256-80b41f7b60b98fea1c719a9d7987ab6d734d3ef4ec2bd4e762f76185cd33983f.sig","sha256-812c9fda75134dc7d96e2822d95bb68f744825776b49056e9569137a52dd50b3.sig","sha256-82425b73d5900dbdd37581116b1aba6b8aab5c42e4b9dcd2d874b9ecf6eee957.sig","sha256-8361a46d198a09f84e691634c65ec933d2da1b11346f0c984e39af53b2d1bf08.sig","sha256-839a167667292616643ebf942dd8ca3eec15f2a080b0a4392fe1535f2a18dca5.sig","sha256-8971b841996841b78bb853335b6d57a82eee0edeb15dc00f21a18ed9e1051804.att","sha256-8971b841996841b78bb853335b6d57a82eee0edeb15dc00f21a18ed9e1051804.sig","sha256-8d0f21545ca2cab0ad1d58c70f58e6c2e20e3bae1ff27993a030e68a413f0cd2.sig","sha256-8d9cd76cd52c8f4ed771bf6dc098b1ea6596047561a9fcacf9629d127c91340b.sig","sha256-9374db9b85a2ebe717eefa7c72598d5b9c29e97b981ae33f86fc83fb2c348a77.att","sha256-9374db9b85a2ebe717eefa7c72598d5b9c29e97b981ae33f86fc83fb2c348a77.sig","sha256-9545200f3aa3f66c5f8db0705bbade5f3902e31010bc39d3736af99dee8d0e86.sig","sha256-9bd59d1116362102852236a973fd311aa288d4cd656843fbd376662745d970e1.sig","sha256-9d1f54f4212caf9f794be7da21de4a9e486fce7af7976c23d8708ca751ea47fc.sig","sha256-9d431ebc43ed0ec14e1bcc5c2780d052a115f1796e8a9d0ed9a47777c31501a6.sig","sha256-9d8cf1d600b837df8556c279f8e3caa4b274ecd439275ac28ade8b0adcae3132.sig","sha256-9f204a6a918619817ee920ba1615b41633c3b413c1e32b28e560278ad9486663.sig","sha256-a02afd15f79a7684000a870388ac6cdf9d7ba5680524073ac36ef711ae19bff4.att","sha256-a02afd15f79a7684000a870388ac6cdf9d7ba5680524073ac36ef711ae19bff4.sig","sha256-a2c79641036411313f062099ba701c33588964fff5251f699635ef71b26acf77.att","sha256-a2c79641036411313f062099ba701c33588964fff5251f699635ef71b26acf77.sig","sha256-a3c3a1cc38e3610f46ac6fd26e8b8164f2e0a8e303cfeb0568a9412e07949ce8.sig","sha256-a9121d470c957cbb373ca5172cec739311f8759552bd93e1f3ed6cd96c58c2b1.sig","sha256-ac96839e834e679499430de8c0fc13989d56e957fc09e604d9e63a1e84c3c79e.sig","sha256-b1248a898c7ded2a805452c2666b10ef4a8a9163c207e8ad8d4b51a729bb4024.sig","sha256-b2226400b25417e5443e109425d87cf6c5f95d7a7017e3a6da0134204fa0544e.sig","sha256-b2f0dac5d188cb3fcc278b827ed7b8ddef2b24eefc888d5b4b7fca1ee026b75c.sig","sha256-bb34090fa3ece0732358f8aad3b74c7c1c21d34ec5f669b85c0e239a2cd187df.att","sha256-bb34090fa3ece0732358f8aad3b74c7c1c21d34ec5f669b85c0e239a2cd187df.sig","sha256-bc672355b2f7120e4063e568bd2b02e252f2807a70a79a949c87b83200d8b499.sig","sha256-bd06e84eeba0513c58d84f70efffd024f4d33150836b22fd981f00d64e7fb1c4.sig","sha256-bd5a51d10e48c6130321c97deee0d534ecec5a39cba6a385cf394a5367838cc6.sig","sha256-c4e7238efbb6b4da94ea377d8c3ff341f281b08f197b44d87e688902bdbfbd1c.sig","sha256-c87e5c1dd5d7b0e11a595881312869a4fc1f84732ebadc47b2e97da0b5dbd3a5.sig","sha256-c8d38347cb7638199cbce6b49b55a5c99e504bcdfeef241ff554d541dd975154.att","sha256-c8d38347cb7638199cbce6b49b55a5c99e504bcdfeef241ff554d541dd975154.sig","sha256-cad8d1b5ec99cd5f62f745d20b9fdb5519380b2d3ea3aa9e837bd03cf7bc0f90.att","sha256-cad8d1b5ec99cd5f62f745d20b9fdb5519380b2d3ea3aa9e837bd03cf7bc0f90.sig","sha256-cafb36c71b5e4b07369640df89d8db0cd01a92a0ec368f201f68b2da97b122a4.sig","sha256-cc6174e15c03d6933462dc0375f547045d9b85efbb412d580e67e79c02b06483.sig","sha256-d13530b605bfbd1a43a4b37053b6849f25ae8e78ae4e58319de81fe839a8387b.sig","sha256-d22163b1fa5892bb413c4263b751f82c3de9bbd41613b2db51faaacb9aab08c4.sig","sha256-d3ddf9aecafba58c72305aeee58cf2e753973275435cc6c8f320f282e7698799.sig","sha256-d662a4787f214c95bfaf55b6022f0ac74595bcaeffe7c0b96641410187bbed2b.sig","sha256-d6c3707c448da8eaf88a11517945b2ff53b76fd52035b5ff3c1e33bb166f21a3.sig","sha256-daa797b80f4b250801e7ae425b55b53e7e7765fd8eaf6acd6466f1af43821164.att","sha256-daa797b80f4b250801e7ae425b55b53e7e7765fd8eaf6acd6466f1af43821164.sig","sha256-df5498e7d6b9a7bcfd05eaa96d14a7f2e36d520392a45378f1524afe1a314a29.sig","sha256-e1ad182b2e14777abf668cce1776c87a98b48c1e1e6e5c88e8d723607dc56a95.sig","sha256-e4da6e1288575c595cb6993069511b97e65d6f07be99307ba842ac58411dfd3c.att","sha256-e4da6e1288575c595cb6993069511b97e65d6f07be99307ba842ac58411dfd3c.sig","sha256-e660aaae550fec0650e8818b4f3bc4d3b9def8bc5e1b8d73ca298cb2288dfcbd.sig","sha256-ed4bfd26084c04fbbfa560f567a67478fcac87ba527102b53f48f5e585819572.sig","sha256-eea99dc08a1ca1bd86aae84a3e07e1dbe8da71971f60719b78b7a2862ba3d397.sig","sha256-eed5e3437a02359fd2389a49e1e7596fdade69925f8daea6645262a8e2eb5ab3.sig","sha256-f1e0545490810d105e804a6d5ab40f4b0182b96108f7e7986bacd79c320be0af.sig","sha256-f79ce0614d89e3b562d7c983e9b71a00756915d107538d7d00c1e89e21560902.sig","sha256-f9730e3060aa7181f80834bdd8407d66afdfe5345498d36d286ff9c3e8b25cef.sig","sha256-fb013d5ecf707e14e95a0833807c7ee56dd920c8d6d80a5b7511ca2166ea3574.sig","sha256-fbdab6c72827920249fc8ed6af93ef915ea429bcd70894c8a4336ff346d8756d.sig","sha256-fea4494467976ac9cb423239a83be889bccde9636f03921f4a7fda2cdcadbb21.sig"]}[stack@dutbospdir02 ~]$
[stack@dutbospdir02 ~]$ curl -k -u director:s3cret.123! https://dutbsat02.tbiptvcloud.meydan.du/v2/dutbsat-osp17_containers-rsyslog/manifests/17.1
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"Manifest not found.","detail":{"reference":"17.1"}}]}[stack@dutbospdir02 ~]$
[stack@dutbospdir02 ~]$
[stack@dutbospdir02 ~]$ curl -k -u director:s3cret.123! https://dutbsat02.tbiptvcloud.meydan.du/v2/dutbsat-osp17_containers-rsyslog/manifests/v17.1
{"errors":[{"code":"NAME_UNKNOWN","message":"v17.1 was not found!"}]}[stack@dutbospdir02 ~]$

RedHat Undercloud/Director version:

#

[root@dutbospdir02 ~]# cat /etc/*-release
NAME="Red Hat Enterprise Linux"
VERSION="9.2 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.2 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"
Red Hat Enterprise Linux release 9.2 (Plow)
Red Hat OpenStack Platform release 17.1.2 (Wallaby)
Red Hat Enterprise Linux release 9.2 (Plow)
[root@dutbospdir02 ~]#

RedHat Satellite version:

#

NAME="Red Hat Enterprise Linux"
VERSION="8.10 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.10 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.10
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.10"
Red Hat Enterprise Linux release 8.10 (Ootpa)
Red Hat Enterprise Linux release 8.10 (Ootpa)
[root@dutbsat02 ~]#

Describe the impact to you or the business
We are blocked as Stack Update is not working.

In what environment are you experiencing this behavior?
Pre-production