How can we attach more than 28 disks to a guest ?

Issue

• Recently we noticed that we can not add more then 28 disks(default: num of disks which can be attached to instance is unlimited) when num_pcie_ports is set to default 16 under libvirt configuration in nova.conf:

2023-01-13 16:48:56.051 7 ERROR nova.virt.block_device [instance: 07120906-e3ea-4100-9f96-7180e20a1c62] libvirt.libvirtError: internal error: No more available PCI slots
2023-01-13 16:48:56.051 7 ERROR nova.virt.block_device [instance: 07120906-e3ea-4100-9f96-7180e20a1c62]
2023-01-13 16:48:59.260 7 ERROR nova.compute.manager [req-432b58ba-edda-4bf3-906b-e6d11070a196 ae7c2e975f91433cb47a08651fd2a91f ab83426af6b04f818ddc0b2615e9afe5 - default default] [instance: 07120906-e3ea-4100-9f96-7180e20a1c62] Failed to attach 4c799126-aaa7-49e3-be8b-f4590241ffe9 at /dev/vdaa: libvirt.libvirtError: internal error: No more available PCI slots

[root@compute-2 ~]# cat /var/lib/config-data/puppetgenerated/nova_libvirt/etc/nova/nova.conf | grep -iB5 num_pcie_ports
# The number of PCIe ports an instance will get. For more information, refer to
# the documentation. (integer value)
# Minimum value: 0
# Maximum value: 28
#num_pcie_ports=0
num_pcie_ports=16 

• After changing the value to num_pcie_ports=28(max value), restarting the nova_libvirt and hard rebooting the instance and we were able to attach more then > 25 disks. ( we can attach i,e /dev/vd{a-z},/dev/vdaa and so on after this pcie port update.)

(overcloud) [stack@director ~]$ nova reboot --hard 50c69a2c-c28c-4f18-9ab4-8dad19d4ac93
Request to reboot server edebeaumont-server01 (50c69a2c-c28c-4f18-9ab4-8dad19d4ac93) has been accepted.

(overcloud) [stack@director ~]$ for i in testvol0{26..30}; do openstack server add volume 50c69a2c-c28c-4f18-9ab4-8dad19d4ac93  $i; done

(overcloud) [stack@director ~]$ nova volume-attachments 50c69a2c-c28c-4f18-9ab4-8dad19d4ac93  | grep -i "/dev/" | wc -l
30

• Here we would like to understand if there is any drawback of using max value(28) or if there is any other recommendation from RH for setting num_pcie_ports in nova.conf w.r.t to compute nodes.

• Kindly provide your feedback before we plan for permanent changes via "NovaLibvirtNumPciePorts" using stack update.