Red Hat Insights second malware scan not updating in dashboard

Latest response

Hello,

I have installed this week the malware collector for Red Hat Insights used for a Dev Red Hat linux licence running RHEL 9.1.

Having launched a first scan on 3 January, it worked smoothly and the results could be seen in the dashboard of the Insights console.

A second malware scan was run today with the CLI (actually twice), but this time, the results can't be seen in the Insight Malware dashboard.

I don't think it is because of a firewall setting since it worked right on the first scan but who knows. As the documentation suggested to do a weekly scan because of the load it generates - when scanning many workstations - I wonder if there is maybe a limited usage of scans, eg no more than once a week?

Many thanks,

Alex

Responses

Hi Alexandre,

You most probably are facing a bug in the Red Hat Insights SaaS solution.
Please open a support case with Red Hat or file a bug report on bugzilla. :)

Regards,
Christian

Hi Alexandre,

I just want to let you know that it must be an issue that seems to affect only random customers.
I ran a malware scan on several systems, and the results are reflected correctly on the console. :)

Regards,
Christian

Hi Alexandre,

I could reproduce the problem on the environment of another customer in our region.
Today I contacted the product manager of the Insights team and reported the issue. :)

Regards,
Christian

Many thanks Christian, much appreciated!

I actually saw your messages this morning, and tried afterwards to get more info or hopefully clear the issue by trying a simple move - remove insghts-client and install it again, sorry for not having tried it before :) - but two things happened :

1°) Before uninstalling the insights-client package, I ran 3 scans and the last one returned the following error message:

$ sudo insights-client --collector malware-detection Unable to fetch egg url https://cert-api.access.redhat.com/r/insights/platform/module-update-router/v1/channel?module=insights-core: 403: Forbidden. Defaulting to /release

The scan nevertheless proceeded but, as for the two previous ones today, the scan reports didn't appear in the dashboard, even after sometime and reloading the dashboard.

Since we last spoke here, as promised, I moved, for my daily RHEL config, from the STIG/FIPS one backto CIS workstation 2.

2°) After removing insights-client with dnf, I reinstalled it but this time, the "-- collector malware-detection" argument wasn't recognised in the CLI.

$ sudo insights-client --collector malware-detection usage: insights-client [-h] [--ansible-host ANSIBLE_HOST] [--checkin] [--compliance] [--conf CONF] [--disable-schedule] [--display-name DISPLAY_NAME] [--enable-schedule] [--group GROUP] [--keep-archive] [--list-specs] [--logging-file LOGGING_FILE] [--net-debug] [--no-upload] [--offline] [--output-dir OUTPUT_DIR] [--output-file OUTPUT_FILE] [--quiet] [--register] [--force-reregister] [--retry RETRIES] [--show-results] [--silent] [--status] [--support] [--test-connection] [--unregister] [--validate] [--verbose] [--version] [--payload PAYLOAD] [--content-type CONTENT_TYPE] [--diagnosis [DIAGNOSIS]] insights-client: error: unrecognized arguments: --collector malware-detection

Just removed/re-installed insights-client, and it looks the "--collector" is still not displayed at the moment.

Kind regards,

Alex

Hi Alexandre,

Please try if the following workaround (complete removal of insights-client and re-setup) resolves the problem ... :)

sudo insights-client --unregister

sudo dnf remove insight-client

sudo rm -r /etc/insights-client
sudo rm -r /etc/motd.d/insights-client
sudo rm -r /var/cache/insights
sudo rm -r /var/lib/insights
sudo rm -r /var/log/insights-client
sudo rm -r /var/tmp/insights-client

sudo dnf install insight-client

sudo reboot

sudo insights-client --register

sudo insights-client --collector malware-detection

sudo vi /etc/insights-client/malware-detection-config.yml

edit -> test_scan: false

sudo insights-client --collector malware-detection

Regards,
Christian

Hi again Christian,

Just ran the work around you shared and Inisights is back : a first scan was completed and I can see it in the dashboard - many thanks for having dealt with this so quickly !

For information on this workaround, the first and the last files could be removed, the other files in my computer were already removed so the command said the files were not there. I am going to do the full scan (not the initial one) later today and will share an update here - but again, many thanks !

You're welcome, Alexandre ! :)

Hi Christian,

I am back online : so...in fact, the workaround cleaned the situation and re-enabled the package to work properly - no more issue with the argument "--collector" - but it looks that the scan results don't get in the dashboard, except for the first one.

To recap, my initial installation was done following the guide Assessing and Reporting Malware Signatures on RHEL Systems with the Insights for RHEL Malware Service. I checked the command history and it looks it was all done, including the config file n /etc.

Here, I followed all the steps again, the issue with the argument is solved, the first initial scan appeared in the dashboard, that's ok. The config file has been changed so that test_scan is set to false, the scan is run and the CLI mentions that the results are being uploaded, but they don't appear in the dashboard.

Two things that I can think of that may have a link: - last week I installed Maldet to but it was quickly uninstalled it (removed all files) - since this reinstall, the registration for software subscription seems a bit unstable for some reason : I had to re-register the system once or twice, and even when it seems ok, notifications are being displayed from time to time a,d saying the registration is successful, while the system was already supposed to be registered.

Just in case it might bring some useful info.

Kind regards

Thanks for the information, Alexandre ! :)

Hi Alexandre,

Once I receive a response from the Insights team (in case they find the culprit), I'll share the results of the investigation with you. :)

Regards,
Christian

Many thanks Christian, again, very much appreciated !

As your are in touch with the Insights product team, may I just share one small comment / thought on the product?

When doing the initial set up of Insights, the CLI asks the user, at some point, to enter a PKCS#12 passphrase (not too sure btw when it is to be re-used). I don't know if that is because of a setting on my laptop, but the passphrase, when input, was displayed in clear text. Maybe other users would rather have it masked by dots or with no sign at all ? Personally, I would but again, sharing this just in case :)

Kind regards,

Alex

Hi Alexandre,

I have shared the link to this discussion with the team, so they may find your comments. :)
By the way, you can provide feedback on https://console.redhat.com/insights/dashboard.

Regards,
Christian

Hi Alexandre,

The product manager of the Insights team just came back to me - and here is what he told me ->

Quote :

Thank you for alerting me to that Portal discussion. It sounded familiar, so I went digging and found
this bug, which sounds incredibly similar - https://bugzilla.redhat.com/show_bug.cgi?id=2154098
I will follow-up with the team and see what further investigation has been done and get back to you when I know more.

You may want to add yourself to the "CC List" ... :)

Regards,
Christian

Think I found the bug. The scan result is only uploaded when test_scan is set to true. When you set it to false, to do an actual scan, 'last scan' is never updated in the console.redhat.com. (Well, for some reason we have one server that it still works on, but all others doesn't.)

Hi Siv Elisabeth,

That's an already known fact ... the Insights team is working on a solution as you can see in the
bugzilla. It might take some time to fix the problem, as it affects only random systems/users. :)

Regards,
Christian

Ah, my bad - in this thread I got the impression that it was only the first time it worked. But if I set it back to test_scan, it will change 'last scan'.

But I noticed now in the bugzilla, that it was mentioned there. I was looking most at the last post there - that said things was crashing, which I don't get the impression of on my servers. It looks like it successfully uploads the report.

The only difference between the (only) server that works, and all the others that doesn't - is that the one that works has a .'last_malware-detection_filesystem_scan' file that always gets updated with the time it scans. All other doesn't.

Hi Siv Elisabeth,

Thanks for the information ! You may want to add your finding about the missing
'last_malware-detection_filesystem_scan' file to the bug report - it might help. :)

Regards,
Christian

Hi Christian,

Many thanks for the update, for the above message too and for sharing the link to the bug filed on Red Hat bugzilla: I joined and also added myself to the cc list, thanks for the info :)

This looks indeed to be the same issue/bug so I'll stay posted.

Best regards,

Alex (nb: typo edited...)

Gladly, Alexandre ! :)

Hi again Christian,

Just back on this thread please, as I wonder if that issue with Insights, met on my computer and a few other ones (cf the bug in bugzilla) may not have a link with the status of registration / subscription management / system purpose etc.

Occasionally, I noticed that my system was getting notifications that my Red Hat workstation can't received updates, and then that the OS is registered again, without me doing anything about it in between.

I have tried to dig a little bit into that this morning, and noticed there were some mismatches between the status of my computer as to what's visible in CLI and what was visible in the subscription management console.

There was also a mismatch between the license usage (none visible while one was being used). That mismatch could be solved but still the status of "subscription management" and "system purpose" appear as unknown or not specified.

Then, even after updating the system status in CLI, it didn't appear as updated neither on the console nor when checking that with another CLI command.

For instance : in the subscription console, the system purpose is displayed as unknown, so I have updated this in CLI with

$ sudo subscription-manager syspurpose role --list
+-------------------------------------------+
               Available role
+-------------------------------------------+
 - Red Hat Enterprise Linux Workstation
 - Red Hat Enterprise Linux Server
 - Red Hat Enterprise Linux Compute Node

$ sudo subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Disabled
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

System Purpose Status: Disabled            

$ sudo subscription-manager syspurpose role --set="Red Hat Enterprise Linux Workstation"
role set to "Red Hat Enterprise Linux Workstation" 

However, after this command, the console still displays the system purpose as unknown (in the RH portal then systems > details), but also in CLI, as :

$ sudo subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Disabled
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

System Purpose Status: Disabled  

$sudo subscription-manager list --consumed
No consumed subscription pools were found.

$ sudo subscription-manager list --available --all
No available subscription pools to list

That looks a bit similar to the issue about system purpose mismatch reported on Red Hat 7 in 2020 here but using the CLI to set purpose doesn't seem to update this status.

Wondering please if there is a way to re-baseline the subscription management & system purpose settings post-installation ? I checked the RHEL 9 installation guide and the basic administration guide, but still, not too sure how see the right status for subscription status or the system purpose in the local web console.

I also wonder if the Simple Content Access should be enabled or disabled? At the moment, it is enabled (this morning, I disabled it, then re-enabled).

What I would like to do is to make sure that everything is clean on my system from a subscription / purpose perspective, and then see if there is a match between settings seen via CLI and those on the Red Hat subscription portal, if the un-registering/re-registering of subscription stops, and then see if there is a change with the display of malware scans in the Insights portal. Hoping not missing something too obvious there :-)

Kind regards,

Alex

Hi Alexandre,

What you describe is expected behavior, you will have to disable Simple Content Access in order to get what you
want. Un-register the system, disable Simple Content Access, and re-register it in the commonly known way ... :)

Regards,
Christian

Many thanks again Christian, disabling Simple Content Access allowed things to go much better ;)

So, everything went after that pretty well in CLI, I still ran though into that issue but after that, all subscription / system purpose topics are green and CLI commands return what expected.

I then re-registered with Insights-client, so everything was clean there as well. The malware scan was run twice, but the results only get displayed when test_scan set to true. Indeed...

You're welcome (again) , Alexandre ! :)

Hi Christian again,

I really appreciate how supportive you are - moreover seeing where you stand in terms of experience - but the fact that my RHEL install is not benefiting from the antimalware protection unlike others like you, is really unfortunate, as my system may have just been compromised by a (visible) malware.

Having downloaded a (well known) Linux open source security solution and verified it before preparing an install USB key for another laptop, the USB started to display multiple partitions that I didn't create, and which keep reappearing when I was closing them. This behavior repeated itself on my Fedora 36 usb key, which I plugged just in case to see if that was a mere hardware problem related to the first usb key. Except unplugging the network access, I couldn't do much more.

I guess that with a working Red Hat anti-malware, that kind of things could not happen, or would be much less likely, but I might have now to reinstall everything (knowing that malware know how to bypass such step...) and if on my system or subscription the anti-malware doesn't work; that issue occur again, so I am bit puzzled. :-/ Anyway, that happens.

Hi Alexandre,

When you create a bootable Linux USB medium with e.g. a tool like Fedora media writer, the tool
creates new partitions - so, nothing to worry about. RHEL is known to be one of the most secure
systems to use. Try to avoid third party software (if possible) - then you are on the safe side ... :)

Regards,
Christian

Hi Christian, thanks for your message and sorry, couldn't get back earlier on this topic.

In fact, the standard partitions on Fedora install USB wasn't the issue, maybe I wasn't clear :-) but in fact, this had nothing to do with the partitions you can see on such a installation medium with (Gnome) Disks, GParted or other. Right after I prepared a second install USB for that security project (which is a long-standing solution), these partitions were popping up like mushrooms after the rain, and trying to close them caused new ones to reappear.

On last Friday, I had the impression that my Red Hat OS was somehow "cleaner" but I noticed though that more than 400 lines of bash history were gone, 'ps' also stopped working and there were also other issues which didn't make me feel good.

As a home user, I don't have the possibility to have access to "security by layers" either, so the Insights Malware scan was really welcome. Anyway I'll keep an eye on Red Hat bugzilla.

But again, many thanks for your support on that issue!

Sure thing, Alexandre ... Always glad to assist and help. :)

Hi Alexandre,

I want to provide a short update on troubleshooting progress ... the Red Hat Insights engineers seem to have located the root
cause : It is a new timer that had been added to insights-client which prevents apps running too long. Unfortunately malware-
detection usually takes longer than two minutes to perform a scan. A patch to fix this behavior is in the works and will soon be released. Thanks a bunch to the Insights engineers, for their continuous efforts to solving the problem as quick as possible ! :)

Regards,
Christian

Hi Christian

Many thanks for your message and for the update, much appreciated. It's great to see Red Hatters doing a great job investigating this issue as, even, if it may have been concerning only a few users, it may be pointing at something else, so it's great to see the messages on Bugzilla.

I would have liked to help and try asap the potential solution or even provide the additional traces requested by support, but my RHEL 9.1 instance eventually broke so the exact environment to reproduce the issue is not available anymore on my end :-/ But I see that some other impacted users still can, and I am going to try to do something over the week-end and write updates here and in Bugzilla.

Many thanks again and best regards ,

Alexandre

Hi Alexandre,

Once you have a working environment, you can download the 'test file' insights.zip provided by Mark Huth.
Move the file to the /tmp directory and execute the commands below as the root user to check if it works. :)

export INSIGHTS_GPG=false
export BYPASS_GPG=false
export EGG=/tmp/insights.zip

insights-client --collector malware-detection

Regards,
Christian

Hi Christan,

Many thanks for you message and follow-up!

So, I have a brand new and clean install of Red Hat 9.1 (the previous system was installed as 9.0 and upgraded).

It runs at the moment as a VM in Gnome-Boxes, everything seems to work smoothly. The configurations are standard, I just applied Red Hat's security profile CIS workstation 2 during the installation.

All registration / subscription aspects worked fine from the install, including for Insights (in the previous system, there were some kind of instability with the subscription).

At present, the system seems fine, and I haven't installed any third-party software - not even the EPEL repo ;-).

Then, I went though RedHat Malware installation procedure, all steps went well and I didn't change the configuration file, except for the test_scan parameter.

The test scan was uploaded asap, the result of the full scan was that it was successfully completed and uploaded, but I can't see it in the Insights dashboard. I then applied the step outlined by Mark Huth.

Just had a quick look at the logs that I will upload in Bugzilla in a few minutes - I hope there no gross misconfiguration on my end :-/ but it seems there may be some network errors. Another thing I am wondering is could there be any link with glitches caused at firmware level ?

Anyway, I am just uploading the logs in a second.

Many thanks again and best regards,

Alexandre

Hi Alexandre,

Network errors ? You can check it by running sudo insights-client --test-connection ... :)

Regards,
Christian

Hi again Christian,

Your response speed is incredible :) Well, the connection test with the above command went fine, it is some mention of network problems I spotted in the log files. I have have just uploaded them in Bugzilla. Will write a quick update message and many thanks again for your support!

Best regards,

Alexandre

HaHaHa ... Seems your response speed is incredible too, Alexandre ! :D

Well, there is a (big) difference : you are providing support !)

Actually, I can't upload the log files in Buzilla - made a post about it. I don't know if I need extra permissions for that.

Hi Alexandre,

No, you don't need extra permissions. The other customer I assist was able to upload the logs.
By the way, I've left a suggestion for what you can try as a workaround on the bugzilla report. :)

Regards,
Christian