removal of SAS-2 controller drivers in RHEL 8

Latest response

I've recently been made aware that RHEL 8 will remove certain devices that were previously supported by the mpt2sas driver, as listed here:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8-beta/html/8.0_beta_release_notes/removed_functionality

I can certainly understand the need to purge old drivers that are no longer relevant, especially those storage controllers from the SAS-1 or SATA-I/II era. However, among the removed drivers are the LSI SAS2008/2108/2116/etc. I point these out specifically for 2 reasons:

1) They are SAS-2/SATA-III storage controllers, which are still very much relevant if we're talking about storage servers that are still using traditional spinning HDDs. As much as SSDs and NVMe storage have grown in the storage space, I think HDDs still have very important roles to play (bulk storage, media storage, backup storage, etc.) Using faster SAS-3 controllers on HDDs would be a real waste. So, I don't fully understand the decision to remove these drivers?

2) LSI SAS2008/2108/2116 storage controllers are still widely deployed. They are also often re-used from the 2nd hand market for building storage servers for small/medium size businesses, start-up companies, as well as "home lab" hobbyists. They are also often re-branded by Dell, HP, IBM/Lenovo, etc., so are in very wide circulation. With this change, all these systems that still use these controllers would not have an easy path to upgrade to RHEL 8.

So, my questions are:

a) What was the reasoning for removing these specific drivers?
b) Will RedHat re-consider this decision, or is this etched in stone at this point?
c) If this decision cannot be modified, can RHEL provide a "legacy driver" kernel driver sub-package so that servers with these controllers can upgrade to RHEL8 in the future?

Thoughts?

Responses

Great gesture Akemi ! So, thanks to Phil Perry and the whole ELRepo team too, well done guys ! :)

I saw pperry was working heavily on it as well at the bug tracker... you guys were all business there so I didn't want to muck it up, but I appreciate your (plural!) efforts!!

Hi Eric,

We now have the current version of our kernel (4.18.0-80.4.2.el8_0.elrepo.driver) that has patched drivers. In this version, in addition to mpt3sas, support for megaraid_sas ,mptsas, and mptspi have been added. Please download and install the following packages:

kernel-4.18.0-80.4.2.el8_0.elrepo.driver.x86_64.rpm
kernel-core-4.18.0-80.4.2.el8_0.elrepo.driver.x86_64.rpm
kernel-modules-4.18.0-80.4.2.el8_0.elrepo.driver.x86_64.rpm

from http://elrepo.org/people/akemi/kernel/el8/.

Hope updating to the latest kernel is a smooth ride!

I am using the dd-megaraid_sas-07.707.50.00-1.el8_0.elrepo.iso and it does now show the hard drives on the installation screen. However, during the install I get An unknown error has occurred. Parsing through some of the error message I see: "Some packages from local repository have incorrect checksum. pkg: kmod-megaraid_sas-07.707.50.00-1.el8_0.elrepo.x86_x64. There is more messages in the error log but I figured these might be the most important since they seem relevant. Any ideas on the issue? Thanks

Hi Joshua,

Thanks for the report. We will look into this.

After trying a different DVD drive I have it installed and working now. I guess something was wrong with that DVD drive despite it still passing and verifying the DVD from the installation. Sorry for the false report. Thanks for all your work.

Glad to hear it is working. We can stop scratching our heads.

Hi all

I have the similar problem with my SuperMicro X9DR3-F motherboard, it uses Intel C606, here is my SAS controller output, all drivers connected to the SAS controller won't be recognized on RHEL8, but RHEL7.6 works fine... Will there be a permanent solution?

Thanks Jas

03:00.0 Serial Attached SCSI controller [0107]: Intel Corporation C606 chipset Dual 4-Port SATA/SAS Storage Control Unit [8086:1d68] (rev 06)

Your device (Intel C600 SAS controller, IDs [8086:1d68]) uses the isci driver. It is listed under "Removed device drivers" in RHEL 8.

You may want to file a request to get a DUD image built for this driver at ELRepo's bug tracker.

Hi Jas,

I went ahead and built a driver disk image for the isci driver:

https://elrepo.org/linux/dud/el8/x86_64/dd-isci-1.2.0-1.el8.elrepo.iso

Please give it a try and let me know if it works.

Akemi, have any of y'all had any success with getting the fixes pushed into RHEL 8 dracut? Otherwise, do you know when you'll have kernel-4.18.0-80.7.2.el8_0 available?

I know that it's not easy rebuilding with each new release.

Thanks! - Alex

Hi Alex,

As I posted on Aug 24, there is a patched version of dracut:

http://elrepo.org/people/akemi/testing/el8/dracut/

Can you try installing it and then update the kernel and see if the problem is fixed?

I am on the last elrepo kernel published above on my elderly Dell megaraid, for which I am greatly appreciative.

Is there any hope of a general solution from elrepo for the SAS controllers here? Can dracut be coerced to provide these drivers permanently with the stock kernel?

I would also love to see the return of aacraid, as I have several older IBM x3650 servers that I would be able to use for development and testing.

Hi Charles,

We have a patched version of dracut that hopefully fixes the issue. We can make it available for testing. Are you willing to give it a try?

Ok! I am on this hardware (rebadged from Unisys):

dmesg | grep DMI:

[ 0.000000] DMI: Dell Inc. PowerEdge R710/00NH4P, BIOS 6.3.0 09/20/2012

lspci | grep RAID

03:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 04)

Can you check the output from:

lspci -nn

lspci -nn | grep -i raid 03:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] [1000:0079] (rev 04)

With that device IDs, I confirm that your hardware is one of the adapters whose support has been removed. I will upload the patched dracut so that you (or anyone affected) can run a test.

Here's the dracut package with the patch that may fix the current problem:

http://elrepo.org/people/akemi/testing/el8/dracut/

I only have squash, network, and config-rescue installed to start, I loaded yours with rpm -Uvh, and they replaced the base packages.

Now going back to find the command line invocation you sent previously...

I ran this, now I'm going to try to reboot...

dracut -f \ /boot/initramfs-4.18.0-80.7.2.el8_0.x86_64.img \ 4.18.0-80.7.2.el8_0.x86_64

...yes, the stock kernel now boots:

cat /proc/version | sed 's/[(].*//'

Linux version 4.18.0-80.7.2.el8_0.x86_64

lsmod | grep mega

megaraid_sas 155648 2

dmesg | grep -i mega | head -1

[ 1.801354] megaraid_sas: loading out-of-tree module taints kernel.

So far so good.

Just to double-check, can you show the dracut version that is now running?

These are the dracut versions that I pulled from you and installed:

# rpm -qa | grep ^dracut | sort
dracut-049-13.git20190614.p1.el8_0.elrepo.x86_64
dracut-config-rescue-049-13.git20190614.p1.el8_0.elrepo.x86_64
dracut-network-049-13.git20190614.p1.el8_0.elrepo.x86_64
dracut-squash-049-13.git20190614.p1.el8_0.elrepo.x86_64

@Jamie Bainbridge : considering the activity and discussion in this thread, perhaps Red Hat can consider re-enabling some of these drivers in a future update of RHEL 8.x? It seems pretty obvious from the discussion here that there is a demand for driver support for some of those SAS-2 controllers you guys disabled.

Love what the guys at elrepo are doing, but this seems like something that should be fixed with RHEL 8.x, not a sideloaded add-on.

If I can throw in my $0.02, I understand that this hardware is unsupported, should not be used in production, and elrepo storage drivers tainting the kernel should be instant permission to downgrade any support issue to minor.

However, I have been doing this for a few years, and I have acquired a decommissioned server stack that would be nice to draw upon for quick development or non-critical projects. A tainted kernel seems a small price to pay to bring those systems back to life.

I also actually have old PCI 3com Vortex cards that I use to help OS2200 developers, enabled with elrepo drivers. It's not production, but very important, it's all that fits, and I would replace the impacted servers before I'd think of opening a support ticket.

In any case, how do we get dracut fixed?

The patch applied to dracut came from:

https://github.com/dracutdevs/dracut/pull/614

We obtained this info from the Red Hat devs we are collaborating with. We can now report back to them with the current feedback. We hope this will help accelerate the process to get the patch into RHEL.

There is no official word but the current prospect is that the dracut patch will be in RHEL 8.1.

Latest 8.1 beta doesn't do any goods on that topic ;-(

Hi: it seems a sad situation for RHEL8 hardware support. although I can understand RedHat's choice. these old version of hardware are too stable to support, they won't come with money. I have used RHEL6.0/7.0 with shinning hardware and met tons of shinning bugs. RedHat is fixing bugs quickly, but hardware vendors are very slow. so I have learned to never use new version of hardware in production servers. although you will get support, but you don't want that kind of support at first place. we still buy new SAS2 servers for production usage today. we will wait new hardware to become old and stable. at that day we can use RHEL8.

Can verfify. We have many servers with LSI 2008 cards. Why, RH, why???

LSI 2108 here doesn't work. Either the vision of RH is to enforce new hardware, the old one, which is still working can be used for tests and builds, why should throw it. Red Hat, let back the hardware that we used with 7.x. It will save us tons of money.

2 Oct 2019, this is still broken with latest kernel. I have a Dell PERC 6/i RAID controller, uses the MegaRAID driver. Works fine on kernel 4.18.0-80.el8.x86_64, with the DUD sideloaded drivers provided in this thread on installation, but trying to update to 4.18.0-80.11.2.el8.x86_64 results in the system thinking there is no RAID volume.

We solved our problem by rolling our servers back to RHEL 7.7 until the MegaRAID driver is restored and dracut is fixed, though we will probably not try 8.x again for a long time. We need systems that are working now, and working reliably, nevermind running with proper support for Docker-ce in all things. I get why Podman is a thing, but it's too early to not just switch, but cut out all built in support for Docker.

Same issue here with Dell R410, CentOS 8.0.1905 doesn't see the drives during the install too. Back to Ubuntu 18.04... :(

Please Redhat for the sake of humanity bring back megaraid / LSI support. When you did remove that support from RHEL8 i am quite not sure you imagine how many thousands of DELL servers are running with those H700 RAID cards like the PowerEdge R700 / R710 / R720 used for big storage / NAS and so on. I guess i will stick with RHEL7 until EOL.

I would like to thank you all for trying to bring back driver for it. The DUD fails on the DELL H700 with latest kernel : 4.18.0-80.7 but fortunately upgrading with the dracut patch before upgrading the kernel solve the problem. Still no news on integration to RHEL 8.1 ?

Hello, I visited the URL listed earlier (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/considerations_in_adopting_rhel_8/hardware-enablement_considerations-in-adopting-rhel-8#removed-adapters_hardware-enablement) in the thread and have confirmed that we have numerous workstations and several servers that are using LSI MegaRAID cards which are not supported in RHEL-8. However I'm unable to find a listing of the actual products, either a list of unsupported or preferably an hardware compatibility list. For example when I shop to find replacement cards for the workstations, the information contained in the 'lspci -nn' is NOT available in product descriptions. So how do I know if a LSI MegaRAID 9280 (or 9265 or 9300 or 9400) will work, or if it's also on the no longer functioning list. Is there somewhere the lists all the products that the various chipsets are on? Or a list of all the products that are actually known to work on RHEL-8? Thanks.

I, too, have a few of the Dell Perc 6/i boxes. I have not attempted to update to RHEL8 on these, because of non-support for the RAID. I have installed RHEL8 on quite a few newer boxes, and am really happy with it. Is there any more word on these actually being supported again? Or, are we out of luck?

Good news!

The dracut package released today has the patch we've been waiting for. Its version is 049-10.git20190115.el8_0.1.

These drivers seem to work fine for me. https://www.broadcom.com/products/storage/raid-controllers/megaraid-sas-9361-8i

dracut-049-27.git20190906.el8 seems to have shown up in the last couple of days - it has a newer version number but doesn't contain the fix.

That is strange. The patch in question 0024.patch was added to dracut-049-10.git20190115.el8_0.1 and was carried over to dracut-049-27.git20190906.el8 dracut.spec.

I tried the newer dracut, but am not getting the desired result. Here is what I am seeing after running dracut:

lsinitrd -k 4.18.0-80.11.2.el8_0.x86_64 | grep mpt3
drwxr-xr-x   2 root     root            0 Oct 29 12:12 usr/lib/modules/4.18.0-80.11.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas
-rw-r--r--   1 root     root       109968 Sep 24 04:48 usr/lib/modules/4.18.0-80.11.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko.xz
drwxr-xr-x   2 root     root            0 Oct 29 12:12 usr/lib/modules/4.18.0-80.11.2.el8_0.x86_64/weak-updates/mpt3sas
lrwxrwxrwx   1 root     root           54 Oct 29 12:12 usr/lib/modules/4.18.0-80.11.2.el8_0.x86_64/weak-updates/mpt3sas/mpt3sas.ko -> ../../../4.18.0-80.el8.x86_64/extra/mpt3sas/mpt3sas.ko
drwxr-xr-x   2 root     root            0 Oct 29 12:12 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas
-rw-r--r--   1 root     root       597673 Jun 23 09:05 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas/mpt3sas.ko

So it seem to be loading the stock mpt3sas.ko.xz AND elrepo's mpt3sas.ko (kmod-mpt3sas-27.101.00.00-1.el8_0.elrepo.x86_64) So I then tried removing /usr/lib/modules/4.18.0-80.11.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas so that it only had elrepo's as an option and can confirm:

lsinitrd -k 4.18.0-80.11.2.el8_0.x86_64 | grep mpt
drwxr-xr-x   2 root     root            0 Oct 29 12:12 usr/lib/modules/4.18.0-80.11.2.el8_0.x86_64/weak-updates/mpt3sas
lrwxrwxrwx   1 root     root           54 Oct 29 12:12 usr/lib/modules/4.18.0-80.11.2.el8_0.x86_64/weak-updates/mpt3sas/mpt3sas.ko -> ../../../4.18.0-80.el8.x86_64/extra/mpt3sas/mpt3sas.ko
drwxr-xr-x   2 root     root            0 Oct 29 12:12 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas
-rw-r--r--   1 root     root       597673 Jun 23 09:05 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas/mpt3sas.ko

In either scenario I get this message in /var/log/messages

mpt3sas version 27.101.00.00 loaded Warning: mpt3sas 0000:61:00.0 [1000:0072] - Support for this device has been removed in this major release. Please check the removed functionality section of the release notes.

This seems to indicate that the elrepo mpt3sas 27.101.00.00 driver is still echoing that message? Sorry if I missed something obvious - just trying to get my card working (LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)).

I should note that on this machine... that I have no drives connected to the card since I was trying to test getting things working BEFORE doing this on a machine that uses drives connected to the card - therefore, its possible that it is working but just echos outs the support removed error message - I was assuming that since it is still showing that error message, that the card is not usable.

As a follow up to myself just to let others know - it works!

There is still an error message on boot letting you know it won't work, but it works just fine.

Glad to hear it worked. As to the error message, we'll see if that can be eliminated.

Add us to the list of customers that have servers with spinning HDDs and with LSI SAS2008 controllers.

This discussion is becoming too long to read through, so it is easy to miss important posts. Here, I've copied a note from Jamie Bainbridge (Red Hat), which I think is worth repeating.

[quote] Could we trouble you to please open a Severity 4 support case for this query?

It will be directed to our storage engineers who can answer your questions.

It's also important for us to capture customer demand for these removed drivers, and support cases linked to knowledgebase articles and to bugs are the way we gauge that demand. [/quote]

Using RHEL 8.1 and dd-mptsas-3.04.20-2.el8_1.elrepo.iso I get the same issue that Joshua Dantzler initially reported on 1 July 2019 2:24 PM. - hard drives are shown on the installation screen however during the installation we get a ""Some packages from local repository have incorrect checksum" error

Using RHEL 8.0 and dd-mptsas-3.04.20-1.el8_0.elrepo.iso the install is successful.

Has anybody successfully installed using RHEL 8.1 and dd-mptsas-3.04.20-2.el8_1.elrepo.iso?

Apparently the "markdown" formatting of these comments does not actually work. [I forgot to include an empty line before the list - would be nice to have a preview...]

I don't know why the installation using .el8_1 iso fails. This needs investigation.

I have tried multiple times but 8.1 combinations always fail (for me) and 8.0 combinations always work. My "method" is copying the dd-mptsas-xx.elrepo.iso to a usb stick and using the inst.dd=/dev/sda1 option.

Is there anything I can do to help you?

Pages