new compute nodes do not show up in nova hypervisor-stats

Issue

• Added new computes but they do not show up in output of nova hypervisor-stats

(overcloud) [stack@dir ~]$ openstack hypervisor stats show
+----------------------+----------+
| Field                | Value    |
+----------------------+----------+
| count                | 48       |  < --- should be 50
| current_workload     | 0        |
| disk_available_least | 6464352  |
| free_disk_gb         | 25735788 |
| free_ram_mb          | 4628830  |
| local_gb             | 25779888 |
| local_gb_used        | 44100    |
| memory_mb            | 43952478 |
| memory_mb_used       | 39323648 |
| running_vms          | 2452     |
| vcpus                | 2200     |
| vcpus_used           | 10959    |
+----------------------+----------+

• nova-compute.log for the effected computes shows problems connecting to ceph

2021-07-12 19:19:15.697 7 DEBUG oslo_concurrency.processutils [req-8d6c30e5-f7c9-40bc-bd3f-28cbc5ecea36 - - - - -] 'ceph df --format=json --id openstack --conf /etc/ceph/ceph.conf' failed. Not Retrying. execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:457
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager [req-8d6c30e5-f7c9-40bc-bd3f-28cbc5ecea36 - - - - -] Error updating resources for node osrtpz41072.localdomain.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: ceph df --format=json --id openstack --conf /etc/ceph/ceph.conf
Exit code: 1
Stdout: ''
Stderr: "Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)\n"
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager Traceback (most recent call last):
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 8780, in _update_available_resource_for_node
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager     startup=startup)
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 911, in update_available_resource
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager     resources = self.driver.get_available_resource(nodename)
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8121, in get_available_resource
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager     disk_info_dict = self._get_local_gb_info()
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6575, in _get_local_gb_info
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager     info = rbd_utils.RBDDriver().get_pool_info()
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 382, in get_pool_info
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager     out, _ = processutils.execute(*args)
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager   File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager     cmd=sanitized_cmd)
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager Command: ceph df --format=json --id openstack --conf /etc/ceph/ceph.conf
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager Exit code: 1
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager Stdout: ''
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager Stderr: "Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)\n"
2021-07-12 19:19:15.697 7 ERROR nova.compute.manager
2021-07-12 19:19:15.699 7 DEBUG nova.service [req-8d6c30e5-f7c9-40bc-bd3f-28cbc5ecea36 - - - - -] Creating RPC server for service compute start /usr/lib/python3.6/site-packages/nova/service.py:197
2021-07-12 19:19:15.749 7 DEBUG nova.service [req-8d6c30e5-f7c9-40bc-bd3f-28cbc5ecea36 - - - - -] Join ServiceGroup membership for this service compute start /usr/lib/python3.6/site-packages/nova/service.py:215
2021-07-12 19:19:15.749 7 DEBUG nova.servicegroup.drivers.db [req-8d6c30e5-f7c9-40bc-bd3f-28cbc5ecea36 - - - - -] DB_Driver: join new ServiceGroup member osrtpz41072.localdomain to the compute group, service = <Service: host=xxx.localdomain, binary=nova-compute, manager_class_name=nova.compute.manager.ComputeManager> join /usr/lib/python3.6/site-packages/nova/servicegroup/drivers/db.py:47

• no keyrings or ceph.conf in /etc/ceph/

[root@dir heat-admin]# ls /etc/ceph/
[root@dir heat-admin]#