While performing introspection on the Ceph nodes the disk paths are reported incorrectly for one or two disks.

Issue

• We have the same set of hardware in multiple sites. This is a new site were we are doing a deploy of OSP-13 but during the introspection stage we noticed that for CEPH nodes we had few disks report different paths when compared to the other disks. We went ahead and added the path in the ceph.yaml file but the Openstack installed failed as expected as it couldn't recognize the disks.

• The same hardware in other sites have worked fine reporting the correct paths and installation have gone through fine without issues. We have contacted the hardware vendor and done a comparison of the hardware and everything is same and in line when we compare a working vs non-working node.

• In the below example, there is an additionnal disk/by-path/fc---lun-0 link:

P: /devices/pci0000:17/0000:17:00.0/0000:18:00.0/host17/port-17:0/expander-17:0/port-17:0:0/end_device-17:0:0/target17:0:0/17:0:0:0/block/sdb
N: sdb
S: disk/by-id/scsi-358ce38ee2048eb75
S: disk/by-id/wwn-0x58ce38ee2048eb75
S: disk/by-path/fc---lun-0
S: disk/by-path/pci-0000:18:00.0-sas-0x58ce38ee2048eb76-lun-0
S: disk/by-path/pci-0000:18:00.0-sas-exp0x500056b3e68a1dff-phy0-lun-0
E: DEVLINKS=/dev/disk/by-id/scsi-358ce38ee2048eb75 /dev/disk/by-id/wwn-0x58ce38ee2048eb75 /dev/disk/by-path/fc---lun-0 /dev/disk/by-path/pci-0000:18:00.0-sas-0x58ce38ee2048eb76-lun-0 /dev/disk/by-path/pci-0000:18:00.0-sas-exp0x500056b3e68a1dff-phy0-lun-0
E: DEVNAME=/dev/sdb
E: DEVPATH=/devices/pci0000:17/0000:17:00.0/0000:18:00.0/host17/port-17:0/expander-17:0/port-17:0:0/end_device-17:0:0/target17:0:0/17:0:0:0/block/sdb
E: DEVTYPE=disk
E: FC_TARGET_LUN=0
E: ID_BUS=scsi
E: ID_MODEL=PX05SVB096Y
E: ID_MODEL_ENC=PX05SVB096Y\x20\x20\x20\x20\x20
E: ID_PART_TABLE_TYPE=gpt
E: ID_PATH=pci-0000:18:00.0-sas-0x58ce38ee2048eb76-lun-0
E: ID_PATH_TAG=pci-0000_18_00_0-sas-0x58ce38ee2048eb76-lun-0
E: ID_REVISION=AS0E
E: ID_SAS_PATH=pci-0000:18:00.0-sas-exp0x500056b3e68a1dff-phy0-lun-0
E: ID_SCSI=1
E: ID_SCSI_SERIAL=78C0A0A7TEZE
E: ID_SERIAL=358ce38ee2048eb75
E: ID_SERIAL_SHORT=58ce38ee2048eb75
E: ID_TYPE=disk
E: ID_VENDOR=TOSHIBA
E: ID_VENDOR_ENC=TOSHIBA\x20
E: ID_WWN=0x58ce38ee2048eb75
E: ID_WWN_WITH_EXTENSION=0x58ce38ee2048eb75
E: MAJOR=8
E: MINOR=16
E: MPATH_SBIN_PATH=/sbin
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=40699

lspci

[ overcloud-cephstorage-0 ~ ]# lspci  | grep SATA
00:11.5 SATA controller [0106]: Intel Corporation C620 Series Chipset Family SSATA Controller [AHCI mode] [8086:a1d2] (rev 09) (prog-if 01 [AHCI 1.0])
    Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
00:17.0 SATA controller [0106]: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] [8086:a182] (rev 09) (prog-if 01 [AHCI 1.0])
    Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
d8:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller [1b4b:9230] (rev 11) (prog-if 01 [AHCI 1.0])
    Capabilities: [e0] SATA HBA v0.0 BAR4 Offset=00000004
[supportshell.prod.useraccess-us-west-2.redhat.com] [21:01:59+0

• From a node where this behavior isn't noticed, the path refered above is not present:

P: /devices/pci0000:17/0000:17:00.0/0000:18:00.0/host0/port-0:0/expander-0:0/port-0:0:0/end_device-0:0:0/target0:0:0/0:0:0:0/block/sda/sda1
N: sda1
S: disk/by-id/scsi-358ce38ee20574761-part1
S: disk/by-id/wwn-0x58ce38ee20574761-part1
S: disk/by-partlabel/ceph\x20journal
S: disk/by-partuuid/49b94d42-93f2-4e28-a495-5e48d17609db
S: disk/by-path/pci-0000:18:00.0-sas-0x58ce38ee20574762-lun-0-part1
S: disk/by-path/pci-0000:18:00.0-sas-exp0x500056b3db3ab2ff-phy0-lun-0-part1
E: DEVLINKS=/dev/disk/by-id/scsi-358ce38ee20574761-part1 /dev/disk/by-id/wwn-0x58ce38ee20574761-part1 /dev/disk/by-partlabel/ceph\x20journal /dev/disk/by-partuuid/49b94d42-93f2-4e28-a495-5e48d17609db /dev/disk/by-path/pci-0000:18:00.0-sas
-0x58ce38ee20574762-lun-0-part1 /dev/disk/by-path/pci-0000:18:00.0-sas-exp0x500056b3db3ab2ff-phy0-lun-0-part1
E: DEVNAME=/dev/sda1
E: DEVPATH=/devices/pci0000:17/0000:17:00.0/0000:18:00.0/host0/port-0:0/expander-0:0/port-0:0:0/end_device-0:0:0/target0:0:0/0:0:0:0/block/sda/sda1
E: DEVTYPE=partition
E: ID_BUS=scsi
E: ID_MODEL=PX05SVB096Y
E: ID_MODEL_ENC=PX05SVB096Y\x20\x20\x20\x20\x20
E: ID_PART_ENTRY_DISK=8:0
E: ID_PART_ENTRY_NAME=ceph\x20journal
E: ID_PART_ENTRY_NUMBER=1
E: ID_PART_ENTRY_OFFSET=2048
E: ID_PART_ENTRY_SCHEME=gpt
E: ID_PART_ENTRY_SIZE=10485760
E: ID_PART_ENTRY_TYPE=45b0969e-9b03-4f30-b4c6-b4b80ceff106
E: ID_PART_ENTRY_UUID=49b94d42-93f2-4e28-a495-5e48d17609db
E: ID_PART_TABLE_TYPE=gpt
E: ID_PATH=pci-0000:18:00.0-sas-0x58ce38ee20574762-lun-0
E: ID_PATH_TAG=pci-0000_18_00_0-sas-0x58ce38ee20574762-lun-0
E: ID_REVISION=AS0E
E: ID_SAS_PATH=pci-0000:18:00.0-sas-exp0x500056b3db3ab2ff-phy0-lun-0
E: ID_SCSI=1
E: ID_SCSI_SERIAL=9830A0BMTEZE
E: ID_SERIAL=358ce38ee20574761
E: ID_SERIAL_SHORT=58ce38ee20574761
E: ID_TYPE=disk
E: ID_VENDOR=TOSHIBA
E: ID_VENDOR_ENC=TOSHIBA\x20
E: ID_WWN=0x58ce38ee20574761
E: ID_WWN_WITH_EXTENSION=0x58ce38ee20574761
E: MAJOR=8
E: MINOR=1
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=285015933

• The storage controllers are the same on both nodes:

[root@overcloud-cephstorage-0 ~]#  lspci | grep SATA
00:11.5 SATA controller [0106]: Intel Corporation C620 Series Chipset Family SSATA Controller [AHCI mode] [8086:a1d2] (rev 09) (prog-if 01 [AHCI 1.0])
    Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
00:17.0 SATA controller [0106]: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] [8086:a182] (rev 09) (prog-if 01 [AHCI 1.0])
    Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
d8:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller [1b4b:9230] (rev 11) (prog-if 01 [AHCI 1.0])
    Capabilities: [e0] SATA HBA v0.0 BAR4 Offset=00000004