Unknown Hosts

I am working on deploying my 1st CEPH cluster in the lab and I pulled in 3 hosts (ceph1, ceph2, ceph3) all without issue. I was able to deploy a mon, mgr and rgw to each of these servers from my cephadm host.

The issue I am getting is that I am unable to create OSD's and more recently I tried to just drain the cluster of the ceph3 host.

All I see in my logs are "unknown hosts" for ceph3 as outlined in this dump below:

[ceph: root@cephadm /]# ceph orch ls --service_name mon --format yaml
service_type: mon
service_name: mon
placement:
hosts:
- cephadm
- ceph1
- ceph2
- ceph3
status:
created: '2023-03-29T22:14:41.173659Z'
last_refresh: '2023-04-04T13:19:21.158591Z'
running: 4
size: 3
events:
- '2023-04-04T13:08:29.624925Z service:mon [ERROR] "Failed to apply: Cannot place
on ceph3: Unknown hosts"'

The interesting thing is that all 4 hosts have the SAME hosts file:

[root@cephadm ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.254.7 cephadm cephadm.lab.local
192.168.254.8 ceph1 ceph1.lab.local
192.168.254.9 ceph2 ceph2.lab.local
192.168.254.10 ceph3 ceph3.lab.local

I am able to ping, resolve, ssh, and login to any of the hosts from each other, so I am unsure what's causing the "Unknown Hosts" to appear in my logs.

This is my last barrier to getting a good cluster running. A nudge in the right direction would be very helpful. Thanks in advance.