Server is not powering on and its power state is appearing as NOSTATE

Solution Verified - Updated -

Environment

Red Hat OpenStack Platform 13.0

Issue

  • All the servers on the compute node fails to power on.

  • Instances are stuck in powering-on task state and power_state NOSTATE after the host was rebooted.

    | OS-EXT-SRV-ATTR:instance_name       | instance-0000002c                                                    |
| OS-EXT-STS:power_state              | NOSTATE                                                              |
| OS-EXT-STS:task_state               | powering-on                                                          |

Resolution

  • First stop libvirtd service running on the compute node.

    # systemctl stop libvirtd

  • Then, restart the libvirtd & nova_compute containers on the compute node.

    # docker restart nova_libvirt
# docker restart nova_compute

  • After restarting both the services, containers enter a healthy state.

Root Cause

  • All the servers on the compute node failed to restart because libvirtd service was started with systemd.
  • Since this is RHOSP 13, services are supposed to be restarted, started, stopped using docker.

Diagnostic Steps

  • In nova-compute logs, below error is seen.

    ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
ERROR oslo_service.service HypervisorUnavailable: Connection to the hypervisor is broken on host: compute-0.redhat.local

  • Check the status of containerized services on the compute node. Initially, nova_libvirt and nova_compute services were found in restarting state.

    [root@compute-0 ~]# docker ps
CONTAINER ID        IMAGE                                                                                  COMMAND             CREATED             STATUS                          PORTS               NAMES
004ff0ec14d6        192.168.24.1:8787/rhosp13/openstack-neutron-openvswitch-agent:13.0-84   "kolla_start"       4 months ago        Up 19 hours (healthy)                               neutron_ovs_agent
525ad9c40059        192.168.24.1:8787/rhosp13/openstack-neutron-sriov-agent:13.0-83         "kolla_start"       4 months ago        Up 19 hours                                         neutron_sriov_agent
566cdc5ee52f        192.168.24.1:8787/rhosp13/openstack-nova-compute:13.0-92                "kolla_start"       4 months ago        Restarting (0) 21 minutes ago                       nova_compute   
015db85c1b3c        192.168.24.1:8787/rhosp13/openstack-nova-compute:13.0-92                "kolla_start"       4 months ago        Up 19 hours (healthy)                               nova_migration_target
d421d5b44d94        192.168.24.1:8787/rhosp13/openstack-cron:13.0-82                        "kolla_start"       4 months ago        Up 19 hours                                         logrotate_crond
24079716ca8a        192.168.24.1:8787/rhosp13/openstack-ceilometer-compute:13.0-75          "kolla_start"       4 months ago        Up 19 hours                                         ceilometer_agent_compute
76f62bb944f1        192.168.24.1:8787/rhosp13/openstack-iscsid:13.0-74                      "kolla_start"       4 months ago        Up 19 hours (healthy)                               iscsid         
71cd3efd79ec        192.168.24.1:8787/rhosp13/openstack-nova-libvirt:13.0-95                "kolla_start"       4 months ago        Restarting (1) 21 minutes ago                       nova_libvirt   
5b5b556a3508        192.168.24.1:8787/rhosp13/openstack-nova-libvirt:13.0-95                "kolla_start"       4 months ago        Up 19 hours                                         nova_virtlogd

  • libvirtd process file was found to have incorrect context

    [root@compute-0 ~]# ls -lZ /var/run/libvirt/libvirt-*
srwx------. root libvirt system_u:object_r:virt_var_run_t:s0 /var/run/libvirt/libvirt-admin-sock
srwxrwx---. root libvirt system_u:object_r:virt_var_run_t:s0 /var/run/libvirt/libvirt-sock
srwxrwxrwx. root libvirt system_u:object_r:virt_var_run_t:s0 /var/run/libvirt/libvirt-sock-ro

  • Correct context should look like this:

    [root@compute-0 ~]# ls -lZ /var/run/libvirt/libvirt-*
srwx------. root 42473 system_u:object_r:virt_var_run_t:s0 /var/run/libvirt/libvirt-admin-sock
srwxrwx---. root 42473 system_u:object_r:virt_var_run_t:s0 /var/run/libvirt/libvirt-sock
srwxrwxrwx. root 42473 system_u:object_r:virt_var_run_t:s0 /var/run/libvirt/libvirt-sock-ro

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments