Red Hat Training
A Red Hat training course is available for Red Hat OpenStack Platform
Chapter 3. Troubleshooting
This chapter contains logging and support information to assist with troubleshooting your Red Hat OpenStack Platform deployment.
3.1. Support
If client commands fail or you run into other issues, contact Red Hat Technical Support with a description of what happened, the full console output, all log files referenced in the console output, and an sosreport
from the node that is (or might be) in trouble. For example, if you encounter a problem on the compute level, run sosreport
on the Nova node, or if it is a networking issue, run the utility on the Neutron node. For general deployment issues, it is best to run sosreport
on the cloud controller.
For information about the sosreport
command (sos
package), refer to What is a sosreport and how to create one in Red Hat Enterprise Linux 4.6 and later.
Check also the /var/log/messages
file for any hints.
3.2. Troubleshoot Identity Client (keystone) Connectivity Problems
When the Identity client (keystone
) is unable to contact the Identity service it returns an error:
Unable to communicate with identity service: [Errno 113] No route to host. (HTTP 400)
To debug the issue check for these common causes:
- Identity service is down
On the system hosting the Identity service check the service status:
# openstack-status | grep keystone openstack-keystone: active
If the service is not running then log in as the root user and start it.
# service openstack-keystone start
- Firewall is not configured properly
-
The firewall might not be configured to allow TCP traffic on ports
5000
and35357
. If so, see Configure the Firewall to Allow Identity Service Traffic in the Installation Reference for instructions on how to correct this. - Service Endpoints not defined correctly
On the system hosting the Identity service check that the endpoints are defined correctly.
Obtain the administration token:
# grep admin_token /etc/keystone/keystone.conf admin_token = 0292d404a88c4f269383ff28a3839ab4
Determine the correct administration endpoint for the Identity service:
http://IP:35357/VERSION
Replace IP with the IP address or host name of the system hosting the Identity service. Replace VERSION with the API version (
v2.0
, orv3
) that is in use.Unset any pre-defined Identity service related environment variables:
# unset OS_USERNAME OS_TENANT_NAME OS_PASSWORD OS_AUTH_URL
Use the administration token and endpoint to authenticate with the Identity service. Confirm that the Identity service endpoint is correct:
# keystone --os-token=TOKEN \ --os-endpoint=ENDPOINT \ endpoint-list
Verify that the listed
publicurl
,internalurl
, andadminurl
for the Identity service are correct. In particular ensure that the IP addresses and port numbers listed within each endpoint are correct and reachable over the network.If these values are incorrect then see Create the Identity Service Endpoint in the Installation Reference for information on adding the correct endpoint. Once the correct endpoints have been added, remove any incorrect endpoints using the
endpoint-delete
action of thekeystone
command:# keystone --os-token=TOKEN \ --os-endpoint=ENDPOINT \ endpoint-delete ID
Replace TOKEN and ENDPOINT with the values identified previously. Replace ID with the identity of the endpoint to remove as listed by the
endpoint-list
action.
3.3. Troubleshoot OpenStack Networking Issues
This section discusses the different commands you can use and procedures you can follow to troubleshoot the OpenStack Networking service issues.
- Debugging Networking Device
-
Use the
ip a
command to display all the physical and virtual devices. -
Use the
ovs-vsctl show
command to display the interfaces and bridges in a virtual switch. -
Use the
ovs-dpctl show
command to show datapaths on the switch.
-
Use the
- Tracking Networking Packets
Use the
tcpdump
command to see where packets are not getting through.# tcpdump -n -i INTERFACE -e -w FILENAME
Replace INTERFACE with the name of the network interface to see where the packets are not getting through. The interface name can be the name of the bridge or host Ethernet device.
The
-e
flag ensures that the link-level header is dumped (in which thevlan
tag will appear).The
-w
flag is optional. You can use it only if you want to write the output to a file. If not, the output is written to the standard output (stdout
).For more information about
tcpdump
, refer to its manual page by runningman tcpdump
.
- Debugging Network Namespaces
-
Use the
ip netns list
command to list all known network namespaces. Use the
ip netns exec
command to show routing tables inside specific namespaces.# ip netns exec NAMESPACE_ID bash # route -n
Start the
ip netns exec
command in a bash shell so that subsequent commands can be invoked without theip netns exec
command.
-
Use the
3.4. Troubleshoot Networks and Routes Tab Display Issues in the Dashboard
The Networks and Routers tabs only appear in the dashboard when the environment is configured to use OpenStack Networking. In particular note that by default the PackStack utility currently deploys Nova Networking and as such in environments deployed in this manner the tab will not be visible.
If OpenStack Networking is deployed in the environment but the tabs still do not appear ensure that the service endpoints are defined correctly in the Identity service, that the firewall is allowing access to the endpoints, and that the services are running.
3.5. Troubleshoot Instance Launching Errors in the Dashboard
When using the dashboard to launch instances if the operation fails, a generic ERROR
message is displayed. Determining the actual cause of the failure requires the use of the command line tools.
Use the nova list
command to locate the unique identifier of the instance. Then use this identifier as an argument to the nova show
command. One of the items returned will be the error condition. The most common value is NoValidHost
.
This error indicates that no valid host was found with enough available resources to host the instance. To work around this issue, consider choosing a smaller instance size or increasing the overcommit allowances for your environment.
To host a given instance, the compute node must have not only available CPU and RAM resources but also enough disk space for the ephemeral storage associated with the instance.
3.6. Troubleshoot Keystone v3 Dashboard Authentication
django_openstack_auth is a pluggable Django authentication back end, that works with Django’s contrib.auth framework, to authenticate a user against the OpenStack Identity service API. Django_openstack_auth uses the token object to encapsulate user and Keystone related information. The dashboard uses the token object to rebuild the Django user object.
The token object currently stores:
- Keystone token
- User information
- Scope
- Roles
- Service catalog
The dashboard uses Django’s sessions framework for handling user session data. The following is a list of numerous session back ends available, which are controlled through the SESSION_ENGINE setting in your local_settings.py file:
- Local Memory Cache
- Memcached
- Database
- Cached Database
- Cookies
In some cases, particularly when a signed cookie session back end is used and, when having many or all services enabled all at once, the size of cookies can reach its limit and the dashboard can fail to log in. One of the reasons for the growth of cookie size is the service catalog. As more services are registered, the bigger the size of the service catalog would be.
In such scenarios, to improve the session token management, include the following configuration settings for logging in to the dashboard, especially when using Keystone v3 authentication.
In /usr/share/openstack-dashboard/openstack_dashboard/settings.py add the following configuration:
DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'horizondb', 'USER': 'User Name', 'PASSWORD': 'Password', 'HOST': 'localhost', } }
In the same file, change SESSION_ENGINE to:
SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'
Connect to the database service using the mysql command, replacing USER with the user name by which to connect. The USER must be a root user (or at least as a user with the correct permission: create db).
# mysql -u USER -p
Create the Horizon database.
mysql > create database horizondb;
Exit the mysql client.
mysql > exit
Change to the openstack_dashboard directory and sync the database using:
# cd /usr/share/openstack-dashboard/openstack_dashboard $ ./manage.py syncdb
You do not need to create a superuser, so answer 'n' to the question.
Restart Apache http server. For Red Hat Enterprise Linux:
#service httpd restart