Red Hat Training
A Red Hat training course is available for Red Hat OpenStack Platform
Chapter 4. Working with containerized services
This chapter provides some examples of commands to manage containers and how to troubleshoot your OpenStack Platform containers
4.1. Managing containerized services
The overcloud runs most OpenStack Platform services in containers. In certain situations, you might need to control the individual services on a host. This section provides some common docker commands you can run on an overcloud node to manage containerized services. For more comprehensive information on using docker to manage containers, see Working with Docker formatted containers in the Getting Started with Containers guide.
Before running these commands, check that you are logged into an overcloud node and not running these commands on the undercloud.
Listing containers and images
To list running containers:
$ sudo docker ps
To also list stopped or failed containers, add the --all option:
$ sudo docker ps --all
To list container images:
$ sudo docker images
Inspecting container properties
To view the properties of a container or container images, use the docker inspect command. For example, to inspect the keystone container:
$ sudo docker inspect keystone
Managing basic container operations
To restart a containerized service, use the docker restart command. For example, to restart the keystone container:
$ sudo docker restart keystone
To stop a containerized service, use the docker stop command. For example, to stop the keystone container:
$ sudo docker stop keystone
To start a stopped containerized service, use the docker start command. For example, to start the keystone container:
$ sudo docker start keystone
Any changes to the service configuration files within the container revert after restarting the container. This is because the container regenerates the service configuration based upon files on the node’s local file system in /var/lib/config-data/puppet-generated/. For example, if you edit /etc/keystone/keystone.conf within the keystone container and restart the container, the container regenerates the configuration using /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf on the node’s local file system, which overwrites any the changes made within the container before the restart.
Monitoring containers
To check the logs for a containerized service, use the docker logs command. For example, to view the logs for the keystone container:
$ sudo docker logs keystone
Accessing containers
To enter the shell for a containerized service, use the docker exec command to launch /bin/bash. For example, to enter the shell for the keystone container:
$ sudo docker exec -it keystone /bin/bash
To enter the shell for the keystone container as the root user:
$ sudo docker exec --user 0 -it <NAME OR ID> /bin/bash
To exit from the container:
# exit
4.2. Troubleshooting containerized services
If a containerized service fails during or after overcloud deployment, use the following recommendations to determine the root cause for the failure:
Before running these commands, check that you are logged into an overcloud node and not running these commands on the undercloud.
Checking the container logs
Each container retains standard output from its main process. This output acts as a log to help determine what actually occurs during a container run. For example, to view the log for the keystone container, use the following command:
$ sudo docker logs keystone
In most cases, this log provides the cause of a container’s failure.
Inspecting the container
In some situations, you might need to verify information about a container. For example, use the following command to view keystone container data:
$ sudo docker inspect keystone
This provides a JSON object containing low-level configuration data. You can pipe the output to the jq command to parse specific data. For example, to view the container mounts for the keystone container, run the following command:
$ sudo docker inspect keystone | jq .[0].Mounts
You can also use the --format option to parse data to a single line, which is useful for running commands against sets of container data. For example, to recreate the options used to run the keystone container, use the following inspect command with the --format option:
$ sudo docker inspect --format='{{range .Config.Env}} -e "{{.}}" {{end}} {{range .Mounts}} -v {{.Source}}:{{.Destination}}{{if .Mode}}:{{.Mode}}{{end}}{{end}} -ti {{.Config.Image}}' keystone
The --format option uses Go syntax to create queries.
Use these options in conjunction with the docker run command to recreate the container for troubleshooting purposes:
$ OPTIONS=$( sudo docker inspect --format='{{range .Config.Env}} -e "{{.}}" {{end}} {{range .Mounts}} -v {{.Source}}:{{.Destination}}{{if .Mode}}:{{.Mode}}{{end}}{{end}} -ti {{.Config.Image}}' keystone )
$ sudo docker run --rm $OPTIONS /bin/bashRunning commands in the container
In some cases, you might need to obtain information from within a container through a specific Bash command. In this situation, use the following docker command to execute commands within a running container. For example, to run a command in the keystone container:
$ sudo docker exec -ti keystone <COMMAND>
The -ti options run the command through an interactive pseudoterminal.
Replace <COMMAND> with your desired command. For example, each container has a health check script to verify the service connection. You can run the health check script for keystone with the following command:
$ sudo docker exec -ti keystone /openstack/healthcheck
To access the container’s shell, run docker exec using /bin/bash as the command:
$ sudo docker exec -ti keystone /bin/bash
Exporting a container
When a container fails, you might need to investigate the full contents of the file. In this case, you can export the full file system of a container as a tar archive. For example, to export the keystone container’s file system, run the following command:
$ sudo docker export keystone -o keystone.tar
This command create the keystone.tar archive, which you can extract and explore.
