RHEL8 server bricked after STIG install

Latest response

I have run into this twice now.

I used the DISA STIG security profile to install RHEL 8.4.

During the installation I fully configured the connected NIC and told it to start automatically.

After the install finished, I rebooted and found that the network had not come up.

I tried to log into the console and the keyboard wouldn't work. It was a nice mechanical keyboard and I have seen issues at times with those, but mostly on KVMs. There was no KVM. I got a plain-jane Dell keyboard and attached it and got a message on the console that stated that the USB device was unauthorized.

Usbguard is supposed to start at boot and block unauthorized devices per this STIG finding:

https://www.stigviewer.com/stig/red_hat_enterprise_linux_8/2021-03-04/finding/V-230524

The key phrase there is unauthorized.

That would have been fine if the installer had run the usbguard configure routine, but it did not. Usbguard was running unconfigured and it was blocking everything, including the keyboard I had done the install with.

After installation of the usbguard package the installer should have run this command:

usbguard generate-policy > /etc/usbguard/rules.conf

That would have allowed the currently attached peripherals to work after a reboot.

The network failed to start because every NIC had ONBOOT=no set. Not sure if that is a STIG or a bug but it does seem to happen every time.

So, no ssh and no keyboard. I had to use a recovery CD to go in and fix the NIC and reboot to get SSH working. That allowed me to run the usbguard configuration and get the server fully operational.

So to my question: Is this the intended behavior of the installer, and if not, is it going to be fixed?

Responses

Hi Dave,

EDIT removed paragraph, it turns out usbguard is provided by Red Hat regardless of this link here seems to show the usbguard project is external to Red Hat. Below is a copy/paste of some info from there (please see the link in this paragraph, and the last new paragraph in this edited post).

I kindly/respectfully maintain that say that taking the entirety of a security profile is rather heavy handed because hardening a system often is done incrementally and taking the entirety of a profile can make it difficult to attempt to discover what among a number of things caused a loss of function. While it takes longer, it's best to incrementally add hardening features and evaluate as it goes. That being said, take a hard look at the /root/anaconda-ks.cfg file that is a result of your operating system load to see if it gives you any bread-crumbs to the hardening that was done.

EDIT: I looked in Red Hat's rpm search and see that usbguard is provided, but you will have to address the issue you have directly with them. We are volunteers here, and some Red-Hatters do look through the discussions, yet the best method Red Hat says to get something changed is through a ticket, and I'd recommend filing a report with bugzilla.redhat.com, because as you said in your next post following mine, they have the resources, and I'm just a volunteer and do not work for Red Hat.

Kind Regards,
RJ

Dave,

Also, consider Ansible with hardening possibly. Evaluate on a test system in your isolated test environment.

Kind Regards,
RJ

Thanks for the suggestions to manually STIG a Linux machine, but have you ever looked into how many thousands of individual changes that requires and then calculated the cost of having a senior admin/YAML developer sit around doing nothing but writing Ansible playbooks for at least 18 months? We have, about $900k. It took Redhat 18 months to get that policy written and they have, compared to us, practically unlimited resources to throw at it. It was a high priority for them because the use of that policy it is mandatory for their government customers - they weren't wasting any time doing it.

Funding an expensive development effort conducted to the exclusion of all other work while stalling multiple million dollar development projects just to replicate what we have already paid Redhat to do for us is a bit of a hard sell to management.

We are aware that you can download open-source STIG playbooks, but they have to be verified line by line to ensure no malware or vulnerabilities have been inserted. (Especially ones that OpenSCAP is unaware of)

And yes, we know that Redhat is not invulnerable to that sort of thing, but they are on the blame line if anything bad gets into ANY of their packages, not just this security policy.

We have very successfully and effectively used the STIG profile on earlier versions to get 99.99+% of the work done at install and then handled the remainder of the OpenSCAP scan findings with Ansible point solutions. That is a cost-effective use of admin time and development dollars. Other than the longstanding CIFS/FIPS interoperability issue, we haven't found anything that this policy breaks. (other than the usbguard thing, of course)

So, yeah, I am complaining about ONE annoying but relatively minor bug in a massive package containing thousands of individual operations. I'd have to say they did a damned good job of it. No way I'm going to be able to do that level of work with the resources I have on hand.

Sorry if this came off a little strong, but this is a bit of a hot-button topic for me.

If it costs exponentially more and takes exponentially longer to implement Linux than Windows, then no matter how obscenely bad Windows gets, we will be stuck with it. I don't want to be stuck with that bloated POS.

The CIO wants results last Tuesday. I'm not willing to tell her we won't be able to field the first server in this project until 2023Q3. I like having a paycheck.

Hi Dave Rutledge,

It's nice to know some more of your context that was not there at the initial post you provided. Had I known more of your context, I could have perhaps targeted my initial reply appropriate. Now that I do ...

Red Hat's security policy they provide is not necessarily what every entity wants to ingest at face value. Some systems require exceptions to specific findings. I know, because I've had to deal with this for quite some years. I had zero idea at what point you were in this process, and now, different from your first post, I have a better idea.

We have our systems at build time take in all the appropriate mitigations, at build.

So I wasn't initially clear you were just concerned of that one item, and I didn't know your context. I can look at that one later - I've been busy with production things last couple of days.

Yeah, I'm not thrilled with Windows, but here we are anyway. I believe everyone likes a paycheck too. In our environment, we do not use the profile that's available in Red Hat's ISO that you mentioned. We use our own developed methods, including one after-the-fact ansible means that resolves mitigations after build, per that one customer's request.

Now with the new context you provided, it's easier to see what you mean and to gauge a reply from an external view, I'll have a fresh look at what you mentioned.

Kind Regards,
RJ

Hello Dave Rutledge,

First of all, if I can help directly, I'd be honestly glad to do so where practically possible, given the priorities I already face. Just to let you know (I may have mentioned it), the Red Hat Discussion area (here) is not for production support. We're overwhelmingly volunteers, and the actual Red Hatters that come here in the discussion area do so not with production support in the discussion area. To get usbguard addressed properly, work directly with Red Hat.

I mentioned a link in my original reply, it does not seem usb-guard is provided by Red Hat I was incorrect, and edited my first post, please see the last paragraph in my first post.. While you can put in a ticket with Red Hat, you can take the resultant /root/anaconda.ks file and build a kickstart with the additional mitigations you discovered so that your system boots and you have the other mitigations you've discovered.

If this does not fit, I'd be happy to discuss, but from what I've experienced in over a decade of performing such mitigations, it is good to put in the bugzilla and ticket to Red Hat, because they must fix it. While you have a crushing deadline from your CIO, the choices to get this resolved are to address it with Red Hat since they made the rpm, and perform your coding in a kickstart for the things you've discovered. Getting the usbguard rpm fixed as you require is through Red Hat, they address this through tickets and in this case likely a bugzilla.

The usbguard link you mentioned is as you know not a CAT 1 (high) finding. You can put in a PO&AM for the medium finding and cite in there that you are process of addressing this with the vendor, we do this, An inspector will generally not hang you if you have a solid plan and demonstrate viable work and keep your security folks in the loop, I've done this for over a decade in my current area, and prior to that in a different scenario, and a documented rational plan for what you can do will generally work with an inspector.

I'm not a stranger to this process you describe - while our build makes the mitigations at build-time... -- I also contributed significantly in an ansible solution that is used after-build that is used somewhat widely, I can not say which project however here. It does work, but it too is heavy-handed in that there's no option to delineate the mitigations (I suggested it to the project lead). I say this to let you know, I recognize what you are going through and I bring up the nature of a security-profile or after-ansible solution versus a tight deadline.

I still have tickets in for CAT I items and I dropped features in my Red Hat satellites (some previous versions) because we did not want to incur the CAT I and we did not use foreman to the extent it had possible due to the insufferable unmitigated CAT I. So I understand what it is like to have that kind of issue. And I've only mentioned one example, I've put in numerous tickets to directly address these issues. I know from experience Red Hat dedicates efforts based on the need expressed in tickets and bugzillas, so I do recommend this.

Red Hat poured a lot of time into their build with the security profile you selected, but I can also tell you from our own experience, there are a number of things each location, site, entity may or may not be able to take due to operational reasons. So this is why I mentioned some of the items in a blanket security profile build may be excessive for your operational needs. Maybe, maybe not.

You will discover some STIG things saying things "unless required" or even "unless required and documented" which dismisses the issue in those very specific conditions. Additionally, there are some mitigations the zealous security profile will commit you to that you will find (besides your original post) may break something and you won't know what it is immediately, and for a legitimate operational need, you will find robust enough justification to not mitigate certain findings. This is the exception, not the rule. I say this because while Red Hat as you say has more resources than your current tight deadline versus manpower resources etc, you may discover you may be in a greater quandary due to having to unlock a specific set of things that the security profile committed you to. I have seen this multiple times before.

As my profile says, I deal with this on a day to day basis, and our builds, at build time our systems are over 94-97% compliant and functional, but it was the product of a lot of consistent work. Given your tight deadline, and lack of manpower, and the fact that Red Hat is the one to fix this through their typical channels of bugzillas and tickets, your best option in my opinion is to mitigate it in a kickstart you create after your initial load (as I mentioned in the first post, albeit briefly), start with the initial anaconda.ks.cfg file from your initial build, and use that as an initial basis. You could make a custom ISO file to build systems to include the kickstart you make, however, it would be easier (in my opinion) to make the kickstart and place it on your (hopefully you have a satellite server), including the mitigations you've briefly mentioned and build systems with the kickstart and point to that kickstart when you build, with the mitigations you've found.

If you take the scc report and look at the html version it spills out, and examine the items, you will discover much of it is even a false-positive, in RHEL 7, there's a tremendous amount of FIPS false positives, even where it says RHEL 7.9 is not a supported OS. They generally don't fix this until later and update their benchmark until later in the year, it is not prompt, and this false-positive occurs each time a new minor point release is made. There are other false positives as well.

You mentioned FIPS issues with RHEL 8? If this is an issue and you'd like , perhaps tell us more here. I made a script for RHEL 7 FIPS implementation that also includes RHEL 8 that boiled down a time-consuming yet viable Red Hat solution. The block a couple of paragraphs below is the method to implement FIPS on a RHEL 8 system, and my script at that discussion includes this.

At my discussion at that link, it cites this documentation here which says:

To switch the system to FIPS mode in RHEL 8:

# fips-mode-setup --enable
Setting system policy to FIPS
FIPS mode will be enabled.
Please reboot the system for the setting to take effect.
Restart your system to allow the kernel to switch to FIPS mode:

# reboot
Verification

After the restart, you can check the current state of FIPS mode:

# fips-mode-setup --check
FIPS mode is enabled.

If you need help with FIPS and RHEL 8, if you are having issues, I'd be happy to attempt to see what can be done. I recognize you are under a tight deadline, I do highly recommend you put in bugzillas, tickets for the things with usbguard outside your control (and mine) because Red Hat fixes their RPMS through tickets/bugzillas, not through the discussion forum, and we're mostly volunteers here. I recommend documenting the mitigations you made, the bugzilla and ticket you make if you can't fix this with a kickstart you create, or a script/ansible playbook you run after a build, then it could be a PO&AM especially since it is not a CAT I.

One other thing, a very well-thought out audit.rules took care of a severely large quantity of identified issues. If your scc scan with a current benchmark shows a good score, I still recommend considering the resultant /root/anaconda-ks.cfg file as an initial starting point. While you discovered issues in need of resolution, you could use that as an initial starting point. You already mentioned in your 2nd reply that you have some trust in Red Hat's security-profile versus the tight deadline/resources/manpower/etc, and you could perhaps add your mitigations and if needed, maybe make exceptions after the fact for the things you discover operationally you cannot incur with viable justification your organization can stand behind.

As I mentioned earlier, I'll be happy to attempt to assist, balancing other priorities, and recognizing this is a volunteer discussion area.

Kind Regards,
RJ

Dave Rutledge

Just to let you know, I tested the security profile "xccdf_org.ssgproject.content_profile_stig" (In the Anaconda installer, it appears as the bottom option DISA STIG) in a RHEL 8.4 build this morning. Important, the DISA STIG scc software shows the RHEL-provided security-profile only gets you to 60% compliance. (details below)

I added all the partitions the profile mandated, and grabbed the DISA SCC scan software and evaluated it.

[root@victim1] # wget https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/U_RHEL_8_V1R2_STIG_SCAP_1-2_Benchmark.zip
[root@victim1] # https://dl.dod.cyber.mil/wp-content/uploads/stigs/zip/scc-5.4.1_rhel8_oracle-linux8_x86_64_bundle.zip

I then unpacked these, installed the DISA provided rpm -ivh scc-5.4.1.rhel8.x86_64.rpm and installed the Benchmark. I ran the scan and You should know that security profile only corrects up to 60% compliance. I'll add more to this to bring it up to well over 90% compliance.**

mkdir /opt/scc/defaultnotused
find /opt/scc/Resources -type f -name "*Benchmark*xml" -exec mv {} /opt/scc/defaultnotused/ \;
# the next command should be on one line.  The "\" below means these two lines will be effectively be one after hitting "enter".
cp /var/tmp/sccunpack/U_RHEL_8_V1R2_STIG_SCAP_1-2_Benchmark.xml \ /opt/scc/Resources/Content/SCAP12_Content/

I then entered the scc scanner's configuration to ensure it uses the latest Benchmark I just placed in the proper location above:

/opt/scc/cscc --config

This enters a text-based menu, pick option one, follow the on-screen text menu, in this case, as I type this, option 1 reads: "1. Configure SCAP Content", then because I moved the erroneous Benchmarks out of the way above, pick the only benchmark remaining, in this case it is "RHEL_8_STIG" Version 001.002 2021-06-14, which as I type this is the most current available.

I added the directory below, else it may attempt to write to a home drive, and if it's an NFS provided drive, you'll get thwarted with no_root_squash, and you'll lose your result report. The "-u /opt/scc/Resources/Results`" directory I created with the mkdir command will put the results there, and not get lost.

I then ran the DISA scanner with:

[root@victim1] # /bin/mkdir /opt/scc/Resources/Results
[root@victim1] # /opt/scc/cscc -u /opt/scc/Resources/Results/
    <very long output, this takes a while, get a cup of coffee>

So after this finishes, it will give you output such as this:

15:47:05 - Adjusted Score - 60% [Red]
15:47:05 - Original Score - 60% [Red]

Scan session completed : 2021-09-04_154237
Analyzing Scan Session
Reading session data for 1 computers

Creating Summary Viewer
Summery Viewer created at /opt/scc/Resources/Sessions/2021-09_04_154237/SCC-Summary_Viewer_2021-09-04_154237.html

Total Errors: 0

Actions I took to build this, you mentioned using the Red Hat provided security profile, I mentioned this above.

  • I used the RHEL 8.4 DVD, selected a RHEL web server (minimal install removes some important rpms, I can remove httpd, but I disabled httpd after the build).
  • I picked the DISA STIG security profile
  • I added the mandated partitions in the web interface, using LVM.
  • I added a registration to an existing activation key I created in advance so the system would be registered at build time and patched at build. You can make an even more detailed activation key in Red Hat Satellite adding specific repositories and the organization name.
  • I added the intended system purpose during the anaconda build method.

I'll end up making a separate discussion with the efforts to use this method, improve the score and attempt to mitigate the usb guard - it would help greatly (so I can help you) if you provided the precise commands you used to mitigate the issues you provided, so I can include this. As much as I'm not a fan of using a security profile, and would rather add all mitigations my own way, I'll do it anyway. I'll probably do it my way too, and post later in the discussion forum.

Kind Regards,
RJ

Hello Dave Rutledge,

  • I spent a couple of days, and it may need polishing, however, I made a kickstart with the RHEL Security Policy "DISA STIG for Red Hat Enterprise Linux 8". You can see it at https://access.redhat.com/discussions/6308131
  • I built this initially from a standard RHEL 8.4 DVD selecting the Security Policy above.
  • Imagine my surprise when I discovered afterwards, it only has a score of 60% compliance when you check it with the publicly available DISA SCAP tool that I provide scant instructions at that discussion.
  • When you cat the kickstart, remove all blank lines, comments preceding with "#", it is currently 182 lines.
  • The "%post" section is where most of the mitigations I made that carries the compliance level up to 99.6% happens. Your actual mileage may vary after you redact specific mitigations specific to the operational needs you can justify for your own environment.
  • You may need to sort out where to place usbguard generate-policy > /etc/usbguard/rules.conf, but you can perhaps run it at the end of the kickstart, maybe.
  • In the kickstart I have "onboot=yes" which is our standard for some many years. I hope the resources/examples I gave at that separate discussion help.
  • I'll test the kickstart another time, it was a bit to carry this to this point.

Regards,
RJ

See previous post, really.

Bugzilla filed for usbguard RPM reported issue. This is up to Red Hat to fix, generally the customer with the issue either files a bug report or a ticket, however, I went ahead and did it as a volunteer effort for Red Hat's review.

Regards,
RJ

The usbguard issue was fixed upstream (I had submitted a bugzilla). The upstream fix is: https://github.com/ComplianceAsCode/content/pull/7105

Regards,
RJ