removal of SAS-2 controller drivers in RHEL 8
I've recently been made aware that RHEL 8 will remove certain devices that were previously supported by the mpt2sas driver, as listed here:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8-beta/html/8.0_beta_release_notes/removed_functionality
I can certainly understand the need to purge old drivers that are no longer relevant, especially those storage controllers from the SAS-1 or SATA-I/II era. However, among the removed drivers are the LSI SAS2008/2108/2116/etc. I point these out specifically for 2 reasons:
1) They are SAS-2/SATA-III storage controllers, which are still very much relevant if we're talking about storage servers that are still using traditional spinning HDDs. As much as SSDs and NVMe storage have grown in the storage space, I think HDDs still have very important roles to play (bulk storage, media storage, backup storage, etc.) Using faster SAS-3 controllers on HDDs would be a real waste. So, I don't fully understand the decision to remove these drivers?
2) LSI SAS2008/2108/2116 storage controllers are still widely deployed. They are also often re-used from the 2nd hand market for building storage servers for small/medium size businesses, start-up companies, as well as "home lab" hobbyists. They are also often re-branded by Dell, HP, IBM/Lenovo, etc., so are in very wide circulation. With this change, all these systems that still use these controllers would not have an easy path to upgrade to RHEL 8.
So, my questions are:
a) What was the reasoning for removing these specific drivers?
b) Will RedHat re-consider this decision, or is this etched in stone at this point?
c) If this decision cannot be modified, can RHEL provide a "legacy driver" kernel driver sub-package so that servers with these controllers can upgrade to RHEL8 in the future?
Thoughts?
Responses
Guru
35365 points
Hi ! :)
RHEL 8.0 seems to be based on fedora 28/29 ... I'm running fedora 29 on my main production machine and - although the system is installed on a SSD, the second built-in SATA disk is well supported. I'm running RHEL 8 Beta in a virtual machine where the system is installed on a virtual SATA disk and as I didn't face any problem, it must be supported there as well. Maybe we misunderstand the part about the deprecated drivers in the release notes, because you're right, a huge amount of systems are still installed on spinning disks.
Regards,
Christian
I can verify that the LSI 2008 drivers have been removed. The beta documentation (section 8.1) states that the MPTSAS drivers and Specifically the LSI 2008 adaptors have been removed. and there is a boot messages that says as much. The card still shows up but has no drivers. I also had the same issue with a Rocket Raid 2720 (which has given me issues before) although I have not researched if it has also been removed.
Newbie
15 points
I can confirm that sas2008 does not work in RHEL 8 beta, but it works in Fedora. So there is no technical reason for this strange decision.
Could we trouble you to please open a Severity 4 support case for this query?
It will be directed to our storage engineers who can answer your questions.
It's also important for us to capture customer demand for these removed drivers, and support cases linked to knowledgebase articles and to bugs are the way we gauge that demand.
Having recently purchased a secondhand server for a homelab, the 2108 card that came in there will prevent RHEL 8 from being my learning environment.
Poor choice for RH/RHEL. LSI sas2008/2308 were the de facto standards for SAS2 solutions for quite some time. Built like tanks and solid drivers for years now. I have LSI sas3 (3008) ctrls in IT mode that work great as well but a push to only using those for sas/sata capacity disks is dejecting when these sas2 LSI controllers are EVERYWHERE. mtp3sas drivers 'I believe' CAN support these sas2 gen LSI stg ctrls, seems in RHEL8 RH assembled the mpt3sas kernel module without the support of these older gen sas2 chips. Works in Fedora 29 but not RHEL8. Hoping mpt3sas driver can be recompiled to support the 2008/2308 and other deprecated chipsets for home labbers, production of course get the 'shiny new stuff'.
I also can confirm this as i'm running several Supermicro H8QG6-F systems and the latest release of Red Hat 8.0 will not recognize the LSI 2008 SAS2 on board
Guru
4337 points
I'm trying to find out exactly what device IDs have actually been removed from RHEL 8. Could someone who has the affected hardware post the IDs of your controller? "lspci -nn" should show them.
You will find a list of device IDs of the Removed adapters in RHEL 8.
For example, SAS2008, PCI ID 0x1000:0x0072 is one of them and its device ID pairing is [1000:0072]. What is curious is that this particular one (and some others) are in the mpt2sas driver in RHEL 7 but are now included in the mpt3sas driver in RHEL 8.
Guru
4337 points
Never mind. Turns out, those removed devices are listed as "xxx_pci_ids_removed" in the source code.
In Fedora 30...
Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
I am unable to install RH 8 as well, since I boot off of the LSI 9212 (2008), the drives don't show up.
Guru
4337 points
Right, [1000:0072] is one of the drivers listed in the "removed" section of the source code. Your solution is to use a DUD (driver update disk) during the installation. It is being developed. Will you be able to give it a try once it is made available for testing?
Undoubtedly. Where do I keep an eye out for that?
Thanks for the heads up!
Guru
4337 points
It will show up here soonish:
https://elrepo.org/linux/dud/el8/x86_64/
I will post a note as soon as it is uploaded.
Guru
4337 points
The iso images for mpt3sas and megaraid are available there now.
The installer is supposed to find the driver. If this does not happen, you need to append the inst.dd option to the boot command line. For details please see Performing an assisted driver update.
Thank you!
It worked. I placed in the resulting burned cd of the iso and it was automatically detected before the installer opened.
I assume it's normal that it takes a while to load?
Unfortunately after installing and running a dnf upgrade, the system no longer boots - goes into dracut... this happened twice in a row today (I just re-ran the installer after it happened the first time)... odd since I had RH8 running great booting off of the internal SATA II this morning... same drive, save exact procedures. Ahhh.. well... I'm still a bit of a RH noob, but I assume I need to custom build a kernel for the sas card...
However, thank you very much!
Guru
4337 points
It is good news that the DUD image worked.
Regarding the trouble after 'dnf update'... I assume the kernel was updated to 4.18.0-80.1.2.el8_0. Can you still boot the original kernel 4.18.0-80.el8? Can you check to see if the kmod-mpt3sas package is actually installed on the system? This package is supposed to survive kernel updates.
Yes, I am booted to the older kernel now... and kmod-mpt3sas is loaded...
mpt3sas 286720 3
And so far I dug these up...
/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas/mpt3sas.ko
/lib/modules/4.18.0-80.1.2.el8_0.x86_64/weak-updates/mpt3sas/mpt3sas.ko
And yes, the kernel updated and once I started going through journalctl @ dracut I see the issue was not finding the drives.
Guru
4337 points
Thanks for digging. Everything looks okay, except that, with the updated kernel, the driver is not found. Hopefully I can find the cause.
Thank you, Akemi, for your efforts! I am sure many will appreciate.
Shall I post in the elrepo mailing list of the experience, or should I wait until your able to figure out the issue?
Thank you and Best Regards, Eric
Guru
4337 points
Hi Eric,
Either the mailing list or our bug tracker will be good.
Active Contributor
173 points
Hi Eric,
I have opened the following bug on elrepo's bug tracker so we can track and hopefully solve the issue:
https://elrepo.org/bugs/view.php?id=919
I'm afraid you will have to register on the bug tracker to post, but that will allow you to follow the issue and receive email notifications of updates.
It would be really helpful if you could register and help us work through the issues to find a solution for you (and others) :-)
Clean install on my test bench Supermicro system with the latest updates and kernel update to kernel-4.18.0-80.4.2.el8_0.x86_64 system was unresponsive and i had to go back to the previous kernel-4.18.0-80.el8.x86_64 and it worked.
Will there be a permanent solution for this?
Guru
4337 points
Hi Raffaele,
Thanks for the testing. Yes, we have to investigate why the driver does not work in the updated kernel.
Guru
4337 points
Hi Raffaele,
I just wanted to make sure you have seen Phil Perry's comment above. We track the issue in ELRepo bug #919.
I have a device with
controller: 01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] [100:0070] (rev03)
Kernel: 4.18.0-425.3.1el8.x86_64
I think https://elrepo.org/linux/dud/el8/x86_64/dd-mptsas-3.04.20-8.el8_7.elrepo.iso may work for my device - but where do I put this ISO in my mirror repo? Under which path:
Index of /almalinux/8.7
[ICO] Name Last modified Size Description
[PARENTDIR] Parent Directory -
[DIR] AppStream/ 2022-11-10 06:45 -
[DIR] BaseOS/ 2022-11-10 06:24 -
[DIR] HighAvailability/ 2022-11-10 06:45 -
[DIR] NFV/ 2022-11-10 02:17 -
[DIR] PowerTools/ 2022-11-10 06:45 -
[DIR] RT/ 2022-11-10 02:17 -
[DIR] ResilientStorage/ 2022-11-10 06:45 -
[DIR] SAP/ 2022-11-10 06:45 -
[DIR] SAPHANA/ 2022-11-10 06:45 -
[DIR] cloud/ 2022-08-31 12:26 -
[DIR] devel/ 2022-11-10 08:40 -
[DIR] extras/ 2022-11-10 06:45 -
[DIR] isos/ 2022-11-10 06:45 -
[DIR] live/ 2022-02-16 16:42 -
[DIR] metadata/ 2022-11-10 06:45 -
[DIR] plus/ 2022-08-31 12:21 -
[DIR] raspberrypi/ 2021-11-10 21:04 -
can anyone here advice?
the iso won't load automatically. you need to load it with inst.dd when you boot the install media. so it's not important where you put the iso. you just need to make it accessible with inst.dd when you boot. ex: inst.dd=https://elrepo.org/linux/dud/el9/x86_64/dd-mpt3sas-39.100.00.00-1.el9_0.elrepo.iso
Active Contributor
173 points
In addition to the driver disks for mpt3sas, megaraid_sas and arcmsr that Akemi linked, elrepo will also shortly have available updated driver disk images for mptsas and mptspi drivers. All of these updated driver disks revert the changes RH has made in RHEL8 and restore all the missing PCI IDs for older unsupported devices. If there are any other drivers that people are aware of that have dropped support for older devices, please let us know at elrepo and we will endeavour to make these available to the community. If you are successfully using one of these updated driver disks, we would really appreciate feedback to confirm they are working as expected for you.
Great! Thank you. I will do an install later this morning and report my findings both here and elrepo.
Guru
4337 points
I failed to mention this, but this work by ELRepo is being done in collaboration with Red Hat developers. So, it is more than a workaround via 3rd party. :)
Regarding my case with Red Hat removing the drivers: Their response was that it was documented that they would be/have been deprecated.
SAS 1078 [1000:0060] SAS2004, PCI ID 0x1000:0x0070 SAS2008, PCI ID 0x1000:0x0072
One of my systems, SAS 1078 [1000:0060], had an onboard OS disk, so I was able to install the OS, but not see the raid controller drives. The DUD corrected that issue. Thank you for the support ELRepo/RH devs.
Guru
4337 points
Hi Eric, and all others affected by the current driver issue:
It is critically important that you perform the testing shown below and report the results. We do not have hardware to test.
The current status is that you were able to install RHEL 8.0 using the DUD image, but when the kernel was updated, the system no longer boots. You were able to boot the original GA kernel.
In the following example, GA kernel = 4.18.0-80.el8 and the updated kernel = 4.18.0-80.1.2.el8_0. If you have already done another update, your updated kernel may be 4.18.0-80.4.2.el8_0.
(1) While in the GA kernel (this is the only one that boots), run the dracut command for the updated kernel:
$ sudo dracut -f /boot/initramfs-4.18.0-80.1.2.el8_0.x86_64.img 4.18.0-80.1.2.el8_0.x86_64
(2) Check what is in the newly created initramfs file:
$ sudo lsinitrd -k 4.18.0-80.1.2.el8_0.x86_64 | grep mpt3sas
(3) Boot the updated kernel.
(4) If it fails to boot, then boot the GA kernel and remove the offending mpt3sas module, run depmod, and try rebuilding the initramfs again:
$ sudo mv /usr/lib/modules/4.18.0-80.1.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko.xz /usr/lib/modules/4.18.0-80.1.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas-orig.ko.xz
$ sudo depmod 4.18.0-80.1.2.el8_0.x86_64
$ sudo dracut -f /boot/initramfs-4.18.0-80.1.2.el8_0.x86_64.img 4.18.0-80.1.2.el8_0.x86_64
(5) Check what is in the newly created initramfs file:
$ sudo lsinitrd -k 4.18.0-80.1.2.el8_0.x86_64 | grep mpt3sas
(6) Does the updated kernel now boot?
[EDIT] I've added 'kernel version' to the depmod command in (4).
Sorry Akemi - haven't been around for a few days... working on it this morning and will report back!
Thanks.
Sorry to report, I was still unable to boot after both attempts...
[ TIME ] Timed out waiting for device dev-mapper-rhel\x2dswap.device.
[DEPEND] Dependency failed for Resume from hibernation using device /dev/mapper/rhel-swap.
...
[130.626999] dracut-initqueue[605]: Warning: dracut-initqueue timeout - starting timeout scripts.
Thanks the same!! Eric
Guru
4337 points
Hi Eric,
I am sorry, too. It would help if you post the output from (5).
Here you go, I copied it when I ran the commands...
[ericklinger@localhost ~]$ sudo lsinitrd -k 4.18.0-80.1.2.el8_0.x86_64 | grep mpt3sas
drwxr-xr-x 2 root root 0 Jan 15 17:49 usr/lib/modules/4.18.0-80.1.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas
-rw-r--r-- 1 root root 109980 Jan 15 17:49 usr/lib/modules/4.18.0-80.1.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas-orig.ko.xz
drwxr-xr-x 2 root root 0 Jan 15 17:49 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas
-rw-r--r-- 1 root root 597609 Jan 15 17:49 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas/mpt3sas.ko
Hi Eric,
Thanks for the output. We may come up with a temporary solution that hopefully works until RH fixes the dracut issue. Stay tuned.
Guru
4337 points
The error message you have posted looks similar to what appears in RHBZ #1641268. There is a patch to fix the issue. Fedora's dracut has been fixed but RHEL 8's dracut is not patched. We will look into the details.
Guru
4337 points
According to the RH devs, there is a problem in dracut. That may be the source of the current driver issue. Hopefully they can find a fix soon.
In the meantime, if you perform the testing in my previous post, that will serve as a workaround and let you run the updated kernel. So, please go ahead and give it a try.
Guru
35365 points
Hi Akemi,
I want to take the opportunity to thank you ... thank you so much for your ongoing effort in creating working solutions for
serious problems. This is very much appreciated and our community members can be glad to have you. I wish you a great
weekend. :)
Regards,
Christian
Guru
4337 points
Hi Eric, and anyone facing the same issue,
The current status is that you were able to install RHEL 8.0 using the DUD image for the mpt3sas driver, but were unable to update the kernel. Therefore, as a temporary solution, we offer the updated kernel that has the patched mpt3sas driver.
Go to http://elrepo.org/people/akemi/kernel/el8/. Download the following 3 packages to a directory:
kernel-4.18.0-80.1.2.el8_0.elrepo.bug919.x86_64.rpm
kernel-core-4.18.0-80.1.2.el8_0.elrepo.bug919.x86_64.rpm
kernel-modules-4.18.0-80.1.2.el8_0.elrepo.bug919.x86_64.rpm
Install them by running "yum localinstall *.rpm" and boot this updated kernel (and keep your fingers crossed).
Once we confirm this custom kernel works, we will provide the current kernel 4.18.0-80.4.2.el8_0 with the fix.