contrail_analytics_kafka was not updated/regenerated during stack71 upgrade from 13.5 to 13.6
Issue
-
After wrapping up our contrail 13.6 upgrade in stack71, we noted that one of the docker containers (contrail_analytics_kafka) was not running, and not upgraded.
-
Container which didn't upgrade (dead, old creation date, etc):
[heat-admin@overcloud-ca-1 ~]$ sudo docker ps -a | grep kafka
7bfe39ba3239 10.10.10.10:8787/contrail-external-kafka:1912.L3.77-rhel "/docker-entrypoin..." 5 months ago Dead contrail_analytics_kafka
- Example from other CA which is fine:
[heat-admin@overcloud-ca-2 ~]$ sudo docker ps -a | grep kafka
c1d0436e7b50 10.10.10.10:8787/contrail-external-kafka:1912.L4.97-rhel "/docker-entrypoin..." 47 hours ago Up 46 hours contrail_analytics_kafka
- Looks like it was shutdown during the upgrade:
[heat-admin@overcloud-ca-1 ~]$ sudo docker logs --tail 5 contrail_analytics_kafka
[2021-10-20 15:05:32,846] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-10-20 15:05:32,846] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-10-20 15:05:32,847] INFO [SocketServer brokerId=2] Shutting down socket server (kafka.network.SocketServer)
[2021-10-20 15:05:32,853] INFO [SocketServer brokerId=2] Shutdown completed (kafka.network.SocketServer)
[2021-10-20 15:05:32,859] INFO [KafkaServer id=2] shut down completed (kafka.server.KafkaServer)
"Deleting container (changed config_data): contrail_analytics_kafka",
"$ docker stop contrail_analytics_kafka",
"contrail_analytics_kafka",
"$ docker rm contrail_analytics_kafka",
"Error executing ['docker', 'rm', 'contrail_analytics_kafka']: returned 1",
"Error response from daemon: Driver overlay2 failed to remove root filesystem 7bfe39ba32396e762921539af8df2a0b09876424906cb45492bc608b4fbaae07: error while removing /var/lib/docker/overlay2/8e7579f668b1cf48526c81b1bdac7c3d5289cf87597419dd54516c84b83f259f: invalid argument",
"Error removing container: contrail_analytics_kafka",
- tripleo-heat-templates ignored this failure to stop the container
Environment
- Red Hat OpenStack Platform 13.0 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.