contrail_analytics_kafka was not updated/regenerated during stack71 upgrade from 13.5 to 13.6

Solution In Progress - Updated -

Issue

  • After wrapping up our contrail 13.6 upgrade in stack71, we noted that one of the docker containers (contrail_analytics_kafka) was not running, and not upgraded.

  • Container which didn't upgrade (dead, old creation date, etc):

[heat-admin@overcloud-ca-1 ~]$ sudo docker ps -a | grep kafka
7bfe39ba3239 10.10.10.10:8787/contrail-external-kafka:1912.L3.77-rhel "/docker-entrypoin..." 5 months ago Dead contrail_analytics_kafka
  • Example from other CA which is fine:
[heat-admin@overcloud-ca-2 ~]$ sudo docker ps -a | grep kafka
c1d0436e7b50 10.10.10.10:8787/contrail-external-kafka:1912.L4.97-rhel "/docker-entrypoin..." 47 hours ago Up 46 hours contrail_analytics_kafka
  • Looks like it was shutdown during the upgrade:
[heat-admin@overcloud-ca-1 ~]$ sudo docker logs --tail 5 contrail_analytics_kafka
[2021-10-20 15:05:32,846] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-10-20 15:05:32,846] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-10-20 15:05:32,847] INFO [SocketServer brokerId=2] Shutting down socket server (kafka.network.SocketServer)
[2021-10-20 15:05:32,853] INFO [SocketServer brokerId=2] Shutdown completed (kafka.network.SocketServer)
[2021-10-20 15:05:32,859] INFO [KafkaServer id=2] shut down completed (kafka.server.KafkaServer)

 "Deleting container (changed config_data): contrail_analytics_kafka",
        "$ docker stop contrail_analytics_kafka",
        "contrail_analytics_kafka",
        "$ docker rm contrail_analytics_kafka",
        "Error executing ['docker', 'rm', 'contrail_analytics_kafka']: returned 1",
        "Error response from daemon: Driver overlay2 failed to remove root filesystem 7bfe39ba32396e762921539af8df2a0b09876424906cb45492bc608b4fbaae07: error while removing /var/lib/docker/overlay2/8e7579f668b1cf48526c81b1bdac7c3d5289cf87597419dd54516c84b83f259f: invalid argument",
        "Error removing container: contrail_analytics_kafka",
  • tripleo-heat-templates ignored this failure to stop the container

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content