Red Hat Insights Blog
Hi again everyone, I'm Will Nix, Technical Evangelist for Red Hat Management & Automation and I'm headed into my 7th year here at Red Hat. I'm really excited for everyone to join us this year at Red Hat Summit 2018 in San Francisco's Moscone center.
For the past several years I've presented at Summit, and again this year I'll be presenting in several sessions, labs, and workshops. Check out a really brief description below and join me! Sign up for the events in your Red Hat Summit app, and don't forget to give us feedback on how the sessions went!
Check out a list of all sessions where we'll be spreading the proactive systems management goodness here: https://agenda.summit.redhat.com/SpeakerDetail.aspx?id=365781:
Session: Smarter infrastructure management with Red Hat Satellite & Red Hat Insights on Tuesday, May 8 at 10:30 AM - 11:15 AM Learn how to integrate Satellite and Insights systems managment with prescriptive analytics to prevent unplanned downtime and enable secure deployments in your clouds and datacenters. Session: Operations risk remediation in highly secure infrastructures on Tuesday, May 8 at 11:45 AM - 12:30 PM For customers who are operating in connected highly secure environments and want to take advantage of Red Hat's prescriptive analytics and systems management service, Red Hat Insights. Mini-session: Monitor and automate infrastructure risk remediation in 15 minutes or less on Tuesday, May 8 at 3:55 PM - 4:15 PM A quick 15 minute session on the fastest way to begin monitoring risk in your Red Hat infrastructure. Lab: Implementing proactive security and compliance automation on Wednesday, May 9 at 10:00 AM - 12:00 PM Replay: Implementing proactive security and compliance automation on Wednesday, May 9 at 1:00 PM - 3:00 PM Join us in this lab to see how to use various Red Hat management and platform technologies to automate security in your environments. Lab: Integrated management with Red Hat technologies on Thursday, May 10 at 10:45 AM - 12:45 PM Choose your own adventure with each of Red Hat's Management technologies in this integrated lab, highlighting Red Hat CloudForms, Red Hat Ansible Tower, Red Hat Insights, and Red Hat Satellite. BYOD Workshop: Red Hat Satellite and Red Hat Insights test drive on Thursday, May 10 at 1:45 PM - 3:45 PM Bring your own laptop and login to a virtual lab environment where you can test drive Red Hat's latest Satellite 6 to quickly learn how to do content management and prescriptive analytics.
One more thing! Drop by the Red Hat Management and Automation booth to check out demo's of Red Hat Ansible Automation technology, Satellite, CloudForms, and Insights. Let us know how we can help you better manage your mission critical environments.
Thanks, and see you there!
Before we begin...
Before we begin with how to configure Red Hat Insights to be tailored to your needs (in terms of controlling what is sent to Red Hat servers and how it is sent) let me please remind you of the very basics of Red Hat Insights…
Can I control what Red Hat Insights is doing behind the curtains?
Red Hat Insights collects metadata about the runtime configuration of a system. The data collected is 1% of what would be collected via sosreport during a support case. The data collected is a subset of an sosreport, so if a sosreport has been approved for usage, Insights data collection should also be acceptable.
The Red Hat Insights tool allows customers to review the data being collected by use of a
--no-uploadparameter. This runs the Insights client & collection, but does not transmit it to Red Hat for analysis. This collection is stored locally in a temporary directory where it can be inspected.
# ls -lh /var/tmp/TAFHhW/insights-amaya-insights2-20180129165816.tar.gz -rw-r--r--. 1 root root 138K Jan 29 16:58 /var/tmp/TAFHhW/insights-amaya-insights2-20180129165816.tar.gz # ls -lh /var/tmp/sosreport-amaya-insights2-20180129165924.tar.xz -rw-------. 1 root root 12M Jan 29 16:59 /var/tmp/sosreport-amaya-insights2-20180129165924.tar.xz
That data is sent to Red Hat’s servers over SSL and compared to our Support Knowledge Database, looking for matches, and results sent back to customer, in the form of actions, where they are displayed.
Insights on Red Hat Insights
Red Hat Insights requires Python-2.6.6-64 or later, being its main configuration file:
Red Hat Insights registration will auto-detect how the system is registered for software updates and can auto-configure the client based on that information. For auto-configuration, CERT is the default authmethod. Otherwise, authmethod can be set to BASIC, requiring a username and password for the target Insights server (Customer Portal or Satellite).
Red Hat Insights uses Satellite server as a Proxy to send diagnostic data to the Customer Portal so requires a connected environment.
The log file can be found at
/var/log/redhat-access-insights/redhat-access-insights.log. The logs rotate each time data is successfully collected, to
.log.1,.2,.n,so be aware that if an upload has occurred since the case was opened, relevant logs might now be in a different file.
The log file records the process of collecting data and uploading that data to the Insights server.
I still want to control more of it
Well, you can!
Insights can be configured by the customer to further restrict what's collected / sent, and optionally to obfuscate hostname and / or IP addresses from reports if desired. Customers can always look at the source code directly from the rpm - everything is made available for their perusal.
All data is trimmed down to the minimal necessary facts before being uploaded and encrypted both in transit and at rest. The customer may also choose to alter the name chosen to represent the system in the UI (eg,
apache01.prodinstead of a fully qualified domain name).
Customers can opt-out of sending any data they wish to the service via a configuration blacklist. The service will continue to function, and only health checks which depend on that specific piece of data will be impacted.
The Insights client will enable customers to ignore any specific file, keyword or pattern, making data redaction easy to use.
The data collected is sent over a secure TLS / https connection. It's encrypted at rest on Red Hat's systems using LUKS encryption, and is kept only until the next report is received, which by default is 24 hours. If another report doesn't arrive in the scan interval period, the data on file (encrypted) is kept for a maximum of 2 weeks and then deleted from our systems.
The Red Hat Insights client also provides an easy parameter to obfuscate hostname and IP information. The actual hostname and IP information is replaced with consistent obfuscated names sufficient for rule analysis.
How is data collected?
As new Insights rules are identified, there may be a need for additional metadata collection for analysis and detection. The list of System Information collected by Red Hat Insights is updated on an as-needed basis. The Red Hat Insights client, upon running, pings Red Hat to determine if any additional metadata is needed for rules which have been introduced since the last run. For example, if a new malware check is added, Insights may need to inspect new data sources to determine if a system is impacted.
This automatic check is enabled by default to ensure customers get all new rules and proactive alerts for their system. This ping to Red Hat can be disabled and manually updated via rpm version; however, this may cause customers to miss out on new health checks which depend on new information.
When Red Hat updates the collection rules, the rules are GPG signed by the redhat-tools GPG key. The Insights client will immediately abort if this signature cannot be verified. This file is also manually inspected carefully before each update is released.
These are some of the files Red Hat Insights collects and sends to be processed:
/etc/redhat-release /proc/meminfo /var/log/messages
Do not worry! we do not collect the entire messages file, but rather the lines that match a potential rule (i.e. page allocation failure).
Or some of the commands we run:
/bin/rpm -qa /bin/uname -a /usr/sbin/dmidecode
If you want to know the whole list of commands run and data collected you can take a look at this document.
As said, main configuration file is
/etc/redhat-access-insights/redhat-access-insights.conf, and it’s a very usual ini type of file with # delimited comments, let’s take a brief look at it:
[root@server ~]# cat /etc/redhat-access-insights/redhat-access-insights.conf # Example options in this file are the defaults # Change log level, valid options DEBUG, INFO, WARNING, ERROR, CRITICAL. Default DEBUG #loglevel=DEBUG # Log each line executed #trace=False # Attempt to auto configure with Satellite server #auto_config=True # Change authentication method, valid options BASIC, CERT. Default BASIC #authmethod=BASIC # username to use when authmethod is BASIC #username= # password to use when authmethod is BASIC #password= [...]
To obfuscate your IP addresses, simply add the line:
Or to obfuscate hostnames, simply add the line:
And addding items to the blacklist is as simple as using
We know you are concerned about the security of your data, yet there are times when it needs to be shared to provide the best capabilities for optimization and management. For this reason I wanted to let you know that the team here at Red Hat understands, and has worked hard to provide you with powerful tools that keep your data safe.
Wanna know more?? Find more info here.Posted: 2018-02-19T17:58:15+00:00
The new girl in the hood!
First of all, let me introduce myself, my name is Amaya Gil and I’m a new Insights SME (so you will see a lot of me from now on), working as a Technical Marketing Manager in the Management BU at Red Hat.
I’m a Computer Science Engineer from Madrid (Spain) and I’ve been a Redhatter since 2.010 evolving from different EMEA SA roles to the BU.
I also happen to speak four languages and since English is one of them… here you go!
What is Red Hat Insights?
Haven't heard of Red Hat Insights yet? No problem, this is Red Hat Insights first stop!!!
Red Hat Insights is a Software-as-a-Service (SaaS) offering that provides continuous, in-depth analysis of registered Red Hat-based systems to proactively identify threats to security, performance and stability across physical, virtual and cloud environments, and container deployments.
Red Hat Insights analyzes select files on a system, getting smarter with each additional piece of intelligence and data. Red Hat Insights can automatically discover relevant information, proactively recommend tailored next actions, and even automate tasks with Ansible Playbooks. Using Red Hat Insights, customers can benefit from the experience and technical knowledge of Red Hat Certified Engineers, making it easier to identify, prioritize, and resolve issues before business operations are affected.
As a SaaS offering, Red Hat Insights regularly updates and expands its knowledge base to reflect new IT challenges that can impact the stability of mission-critical systems.
We even got a video for you!
If I was still unable to make myself clear with what Red Hat Insights is, in this link I leave you a very quick video that would definitely clear your mind: Introduction to Red Hat Insights
Getting started with Red Hat Insights
To access Red Hat Insights, you just need a valid subscription to Red Hat and install the agent on each and every machine you want to monitor, it is as simple as running this command:
[root@server ~]# subscription-manager register --auto-attach [root@server ~]# yum install redhat-access-insights [root@server ~]# redhat-access-insights --register
But no worries! You can also evaluate Red Hat Insights before purchasing it,
to begin evaluating Red Hat Insights, visit Insights Evaluation to activate any available evaluations associated with your account. This step must be completed prior to any system registration with Insights (the steps above).
You must have RHEL 6 and later, or RHEL 7 and later, to use Insights (this process can be automated in different ways, further info can be found at Getting Started with Insights.
Finally after registration, the Insights client will upload initial system metadata to Red Hat Insights. You should be able to immediately see your system in the Insights user interface.
Insights from the UI
First stop is the Dashboard, which will give you a quick overview of your systems, latests news from Red Hat, actions summary and plan summary.
At a glance, Insights allows you to see your systems health, as well as some other intresting things like Latest news from Red Hat (under Latest box) or your Newest Systems (newly registered into Insights), Actions to be taken (in a summary form) or all of your Plans Summary, which gathers all the actions to be taken into groups of machines (previously defined by you).
In the left side of the screen you can see all the Insights features in the form of tabs.
Insights prioritizes security and displays graphical and easy to understand alerts, that gives you a clear idea of the impact a given risk has for your infrastructure.
- Hot items = Red Hat will occasionally update this to point you to new / interesting information or Vulnerabilities.
- High prio = Systems not checking in or critical suggestions from Red Hat about what to check out.
- Plan Summary = A glance into current Action Planner plans and their completion status.
- Actions Summary = numerical representation of topics and categories of problems in the infrastructure
It’s all very intuitive and easy to use, as you can see.
What can I monitor with Red Hat Insights?
Insights is delivered as Software-as-a-Service via Red Hat Customer Portal, and integrated with CloudForms (4.0+) and Satellite (5.7, 6.1+)
- Red Hat Enterprise Linux 6.4 and higher, RHEL 7 and higher
- Red Hat OpenStack 7 and higher
- Red Hat Virtualization 4 and higher
- Red Hat OpenShift Container Platform
- Red Hat Cloud Infrastructure 6 and higher, and Cloud Suite 6 and higher (included in RHCI/RHCS SKU’s)
One of the great features of Red Hat Insights is that there is no new infrastructure to manage, and also provides Ansible Tower integration which enables playbooks generated in Red Hat Insights to be automatically imported into Ansible Tower.
In the next blog post, I will give you some insights on insights, how data is collected, what data is collected, how you can control it... so stay tuned!
Wanna know more?? Find more info herePosted: 2018-02-08T15:10:40+00:00
Happy New Year! One way to get this new year started off right is to get started preventing some of the problems and downtime you may have experienced over the holidays. Using Insights can help future proof your infrastructure with integrated Ansible automation and a report on which systems you still need to patch for vulnerabilities like Meltdown and Spectre. Click here to see if you have systems that are missing the latest patches for these critical vulnerabilities.
We're pleased to announce the latest service release of Red Hat Insights.
Read below for more information or check out the new features and let us know your thoughts by using the "Provide Feedback" button from within the Insights UI in the top right.
For more information about the latest Insights release, refer to our Red Hat Insights Release Notes.
User Interface Layout
In an effort to ease workflow and visibility throughout the Insights application we are shifting to a full screen application mode. This both frees up valuable screen space to more efficiently bring you the latest information on critical threats to your infrastructure, and gives us some more room to continue to add features that you request. Remember, keep the feedback coming, there's a provide feedback button at the top of the user interface.
Webhooks for integration
Insights has now enabled our Webhook integration for custom alerts and notifications. Many of the Incident Management and response services that are used today allow for Webhook consumption which allows for Insights notifications in the tools and workflows you already use. For more information on getting Webhooks configured, please visit Understanding Red Hat Insights - Webhook Integration.
Our engineers, for internal demo purposes, integrated Insights Webhooks functionality with the Slack messaging service. This example allowed for our teams to be notified in chat immediately when new critical issues had been identified on our registered hosts with Insights. For those interested our example code for this integration is online at:
Please note, this is example code and is not officially supported by Red Hat.
CI/CD workflow examples
We previously added support for Jenkins and CI/CD workflow integration and now we have published some quick start example code. This code is available at https://github.com/RedHatInsights/insights-CI-examples/tree/master/examples/jenkins and we will have additional related materials coming soon right here on our blog.
Please note, this is example code and is not officially supported by Red Hat.
Newest rules widget
And to wrap up what's new in this service release, we've added a new widget to the UI with the latest rule additions to the Insights service. Since rules are constantly being added to Red Hat Insights on a regular basis you can now stay up to date with the latest additions on the Overview page. Additional filtering controls around when a rule was added have also been added to our rules page.
Thanks for your feedback and helping continue to improve the Insights service. Have a great new year!
It's that time of year again! Many of us in information technology don't have the luxury of letting our guard down during this crucial time for retail, finance, healthcare, and logistics. Being on call or glued to a monitor or app so that we can keep track of late breaking issues and quickly troubleshoot problems if the need arises is often a requirement in the infrastructure and operations world.
Downtime is unavoidable. It’s either planned or it's forced.
We can help you avoid unplanned downtime and save your organization time and effort with a quick introduction to Red Hat's proactive analytics service for security and infrastructure operations: Red Hat Insights.
When your systems are forced down or have known critical vulnerabilities how do you typically respond?
For those risk averse and security conscious, and to be honest -- who isn’t these days, being informed and having accurate intelligence to more quickly fix vulnerabilities and bugs so that we can safeguard valuable data and achieve greater infrastructure stability is only half the battle.
The other half is actually acting on that intelligence. Red Hat Insights can help you do both.
I'll also link you to the getting started guide so that you can begin using Insights right away with an evaluation, or you can contact your Red Hat account team for a larger scale proof of concept. It only takes a couple of minutes to get up and running on most deployments, and may take only a few seconds if you're already leveraging automation technologies like Ansible, or one of Red Hat's Management portfolio solutions like Ansible Tower, CloudForms, or Satellite.
What is Red Hat Insights?
Red Hat developed the Insights service for exactly the type of problems that keep us on edge during this time of year, to ease troubleshooting, find problems that crop up, and fix them faster!
Red Hat Insights takes advantage of both historical data and the latest knowledge from Red Hat and scales to your infrastructure. Insights is a constantly evolving and hyper-aware Software-as-a-Service (SaaS) platform that monitors and helps manage risk in both the traditional datacenter and hybrid cloud environments.
Insights is not an appliance and requires a very minimal setup with no added infrastructure cost. The goal with Insights has always been to help reduce complexity and risk in our customers’ infrastructures. We want to enable an easy to understand analysis of where problems exist and how to quickly resolve them with trusted, validated advice from Red Hat.
Get Started with Red Hat Management:
We do this in a few different ways. Insights is already integrated into each of the tools you may be using to manage your Red Hat Enterprise Linux infrastructure, and we continue to more tightly integrate with each management solution from Red Hat each release.
The latest releases of Red Hat Ansible Tower supports Insights host reporting directly from the inventory as well as project integration for Insights Ansible Playbook generation and automation jobs built with or imported from the Insights Planner.
Red Hat Satellite has had Insights support since Satellite 5.7 and Satellite 6.1, and continues to be a popular platform management solution for systems lifecycle provisioning and management, as well as risk reporting and remediation with Insights integration.
Red Hat CloudForms has had Insights reporting integration since CloudForms 4.0, and now with the latest 4.5 release has Ansible automation capabilities to further increase remediation efficiency across your managed infrastructures.
Red Hat's Customer Portal also makes it really easy to get started with Insights if you don't currently utilise any other software in the management portfolio.
Even if you're using the Ansible upstream unsupported offering you can take advantage of dynamically generated Ansible Playbooks from Insights to quickly remediate any problems that are identified. Insights can generate a playbook template that enables you to run immediately with ansible-playbook or AWX, or modify tasks therein for your own needs.
In addition, If you have a Technical Account Manager or Consulting engagement you can also engage those services to get more information about enabling Insights reporting for infrastructure health assessments.
How it works:
I'll drop a few screenshots here to give you an idea of how Red Hat Insights works and how easy it is to get started, but if you have a minute or two you should give it a try. Insights supports RHEL 6.4 and newer, so you will need a current RHEL subscription to take advantage of Insights.
You can also use Satellite 5.7 or newer for Insights reporting, but at the time of this post the newest Insights features are being developed for Red Hat Satellite 6 and newer, Red Hat CloudForms 4.5 and newer, and Ansible Tower 3.2 and newer.
We're always adding new Rules based off of the newest or hottest critical issues which are impacting our customers' infrastructures that trigger Actions.
A Rule is how we describe what kind of risks we're looking for from the analytics engine.
When a rule is generated for reporting an action is displayed.
Actions typically offer a description of the associated risk, tailored to the system with the problem, and how to resolve.
From here, you perform the remediation. If you’re using automation, it will call the Insights client to run again and verify if the problem has been resolved.
Using these techniques you can remediate vast amounts of systems with automation, or more surgically if you need to perform some targeted manual or custom automated remediations.
So if you’d like to get started this holiday season and take advantage of Insights proactive remediation capabilities click here to get started. We think Insights will help you avoid potential downtime and increase your infrastructure health score with a minimal time investment during this busy holiday season.
And finally, as with everything we do here at Red Hat - We truly value your feedback. There's a "Provide Feedback" button at the top of Red Hat Insights in the Customer portal, which you get access to as an Insights subscriber, so please give us your feedback about what's working great or what could help make Insights more valuable for you. Barring that, you may also email us at insights at redhat dot com as well.
Thanks for reading, and stay secure and informed,
The Red Hat Insights team is pleased to present our September 2017 service release.
Red Hat Insights is Software-as-a-Service (SaaS) that helps prioritize risk remediation and prevent unplanned downtime. Insights enables customers to proactively monitor and quickly remediate for infrastructure risks and critical security alerts detected in their environments. Using Insights requires no added infrastructure expansion, with no complex setup or configuration, and can be deployed across most infrastructures in a few seconds.
Insights offers automated remediation capabilities via Ansible Playbooks for both Ansible core and Red Hat Ansible Tower, as well as enhanced executive reporting features and overall infrastructure health scoring.
Our September 2017 release brings several new features to the Customer Portal Insights Web UI that are currently available for production environments, and next generation beta features that are offered for testing and feedback in our technology preview service called Insights Beta. Accompanying this release there is also a new Continuous Integration/Continuous Delivery (CI/CD) client for Insights integrations with containers and container images in technology preview. For more information about technology previews, see Technology Preview Features Support Scope.
Read below for more information or check out the new features and let us know your thoughts by using the "Provide Feedback" button from within the Insights UI.
For more information about the latest Insights release, refer to our Red Hat Insights Release Notes.
Expanded Red Hat Ansible Tower integration
Insights and Ansible Tower now offer improved reporting integration in the newly released Ansible Tower 3.2
With Ansible Tower 3.1, we built the first step of our integration with Red Hat Insights - allowing you to sync Insights remediation playbooks to Ansible Tower for use as needed. We’ve continued to enhance this integration in Ansible Tower 3.2. Now, we bring the ability to view Insights Actions directly in Ansible Tower. With this, you can more easily see your minor, major, and critical issues, and with just a few clicks, schedule remediation with Insights Plans.
Insights CI/CD Client available as Technology Preview
For those who would like to be on the cutting edge, a new Insights client that can be integrated directly into a container centric CI/CD pipeline is now available, enabling the identification of risk earlier in a development cycle within images or containers. The new client is designed for flexibility and allows for integration with the existing CI/CD tools such as Jenkins or GitLab CI.
If you'd like to run the new client in a non-production environment please make sure you update your available channels list for your product subscriptions and then follow "How do I get the new Insights client?".
For additional details on how to begin using Insights with CI/CD & examples please visit Red Hat Insights GitHub, RedHatInsights/insights-CI-examples: Sample project for Insights container build scanning demo
Risk of Change [Beta]
Some remediations are more complex than others and may require additional planning due to the impact they have on the environment it’s running on; such as those that require a kernel update and reboot vs a simple modification to a line in a configuration file. Insights now evaluates the “risk" of the change (Very low, Low, Moderate & High) as to the impact that change has on your environment and presents that to the user. By displaying this additional information we aim to provide you a quick idea of the effort required and to assist you in finding the easy fixes / low hanging fruit and remove that risk from your infrastructure. This is a filterable value while viewing lists of hits and is presented to users on rule & impacted system drill downs.
This information is also now presented when generating a playbook within Insights, where a user can select between a fix and a mitigation. By highlighting the risk of the change as well as if that selection requires a reboot, we hope to better inform the user on the impact this generated playbook will have on their environment by mitigating these detected issues within Insights.
Improved Executive Reporting functionality
Based on customer feedback, the Overall Score presentation has been redesigned as a gauge to provide a better understanding of your current score, the color-coded health of that score, and the overall range used. For more details on "Overall Score" please visit https://access.redhat.com/articles/3199992.
PDF Download now available in Next-gen Beta environment. Users now have the ability to export their account's executive report into PDF format allowing for the report to be easily shared.
Inventory view changes
The available inventory and criteria filters on each page of Insights have been redesigned to give you more control and less intrusive. The new page filters now come collapsed by default to give you as much screen real estate as possible. When expanding these filters you will now find additional options for filtering that have been added to each page. When selecting one or more filters, breadcrumbs are displayed to easily show you the active filters on your current view.
To improve the overall and experience for accounts with larger deployment we have removed the previously used "infinite scrolling" and have implemented pagination for these pages. This pagination will still allow for filtering while significantly reducing the amount of time needed for these pages to initially load and to be used.
Planner and Playbook builder changes
The addition of the Playbook summary provides users with clearer visibility of affected systems chosen, the resolution that has been selected, as well as a reboot summary to identify reboot requirements for that playbook, if applicable, to allow for adequate planning and personnel coordination for maintenance windows.
A big thank you from myself and the Insights team to all the customers and Red Hatters who provide feedback that helps us prioritize new features and quality of life value-added improvements. We hope you enjoy the new features and functionality. Please let us know how we can continue to improve the Insights service and what you think of the new enhancements!
Security-based Red Hat Insights rules attempt to analyze and detect issues that impact the security of your systems in different ways:
- Detect high profile, high priority, and 0-day vulnerabilities
- Detect misconfigurations of your software which may impact security
- Detect other issues that could have security implications, such as expired certificates
The Red Hat Product Security team works closely with the Red Hat Insights team to provide current, updated, and helpful content for these security rules. In this blog, we’ll focus on the first category of the rules, which are targeted at high profile vulnerabilities and the associated background work we do.
Customer Security Awareness
The Red Hat Product Security team continuously analyzes security vulnerabilities that affect Red Hat products. Some security flaws are recognized to be of especially great concern, or are expected to generate significant media attention. These issues might be branded (with a name, logo, website), are actively used in exploits “in the wild,” or are a severe problem in core packages or in the functionality of Red Hat products.
When such a high-priority vulnerability is presented, the Red Hat Product Security team starts a process known as Customer Security Awareness (CSAw). CSAw issues are frequently kept secret (embargoed) for a period of time so that proper fixes can be developed and prepared for release by those involved, before the vulnerability is publicly disclosed. During an embargo period, the Red Hat Product Security team and engineering team(s) work with upstream package maintainers, security researchers, and security teams of other Linux vendors with the goal of creating an optimal fix. The objective is to allow stakeholders to have simultaneous access to the fix(es) so that as many end users as possible have access to it before a potential attacker gets details on how the vulnerability can be exploited.
In addition to analyzing the vulnerability in-depth, making sure we have well-tested fixes available, creating an article to explain the issue, etc., the Red Hat Product Security and Red Hat Insights product teams start another process to make sure that Red Hat Insights brings a high level of value for security-conscious customers:
It is often a race against the clock to develop detections and remediations for a vulnerability before the exploit goes public. We treat the data and content around these issues with the highest priority, knowing that a significant number of customers depend on it. We must keep the information confidential, but we also have to cooperate with various internal parties and subject matter experts to create a response. To that end, we create a private repository, with access restricted to peers on a “need to know” basis. Team coordination is critical.
Here are three recent examples of CSAw issues:
- Samba - Loading shared modules from any path in the system leading to Remote Code Execution (CVE-2017-7494)
- sudo: Privilege escalation via improper get_process_ttyname() parsing (CVE-2017-1000367)
- Stack Guard Page Circumvention Affecting Multiple Packages (CVE-2017-1000364 & CVE-2017-1000366)
These issues utilized different engineering and testing groups. Each issue had unique technical nuances and potential impacts that had to be evaluated and corrected … all simultaneously. Often times we’ll have up to four developers collaborating on the final solution. With all of these moving parts, good lines of communication between collaborators are essential.
Identification of vulnerable packages
Once we have the shared, embargoed workspace setup, we collaborate on the vulnerability, with each engineer bringing their unique experience and expertise to bear. The first step is to identify which of our packages contains the vulnerable software and create a list of vulnerable packages that were released in various channels.
The breadth of the Red Hat Insights coverage goes beyond our initial analysis that kicked off the CSAw event. Red Hat Insights coverage includes outdated/out-of-support versions of the packages, and the solution(s) must support those older versions.
Like most major Linux distributions, Red Hat Enterprise Linux uses a process called backporting whereby security fixes are applied to existing stable versions instead of only using new, upstream packages that might introduce new features, changes, or unexpected behavior with the package. If you are not familiar with this concept, you can read Determining your risk, which discusses why commercial security scanners are often wrong when it comes to our products.
Because of our backporting policy, we do not rely on version comparison. We must be much more precise, so we create a tree of vulnerable package versions. If a system analyzed by Red Hat Insights uses a vulnerable package version, we flag it as such.
Close cooperation within the Red Hat Product Security team is our next step, especially between the engineers working on the Insights rule and a group of analysts who analyze the vulnerability. In the race against the clock to develop the Insights rule, we communicate with the technical analysts and package maintainers searching for updates on available information. The Insights rule development for CSAw is a parallel and iterative process.
The Red Hat Insights client can be configured to frequently gather the required data (such as installed packages, running services, or software configurations) so we can start analyzing the scope of impact. The Red Hat Product Security engineers who are working on the Insights rule develop several artifacts: rule server back-end logic, rule web UI front-end content, detection scripts, and Ansible Playbooks.
The goal is to provide the Insights rule components, scripts, and playbooks to the customers, as quickly as possible.
This is a complex task that can be interrupted many times and requires additional considerations and changes as new information emerges. To bring order and prevent missed steps, we utilize a set of extensive checklists that we update and improve based on our experience with them. These checklists assist us in delivering functionality that is up to our high quality standards, and ensures that important details are not overlooked.
We have one checklist for making sure that we follow a specific timeline so we can act quickly and correctly if the vulnerability goes public earlier than expected. We have another checklist with all tasks to make sure that any team member can see what is the state of work and cover for emergencies. And yet another checklist helps us communicate with analysts and make sure we are consistent when using their analyses.
One of the main questions we ask ourselves when developing a rule is, “Can we break out all of the affected systems into more categories?”. This is important because we want to build in flexibility and enable our customers to take a risk-based approach to remediation. Customers can quickly recognize systems they should act on first, if they have limited resources.
A good example of this categorization is a rule for another CSAw vulnerability, DROWN - Cross-protocol attack on TLS using SSLv2 (CVE-2016-0800), with eight different categorizations. This enables customers to easily evaluate the scale of impact on various systems – from systems that have the vulnerable package merely installed, to ones that are running software listening on externally accessible ports while using an old version of OpenSSL, making the system vulnerable to very effective exploitation.
This real-world advice helps our customers prioritize their efforts and initially focus on areas where they have the most exposure, allowing them to choose to defer remediation of lower risk systems until after the greatest risks are remediated.
Most vulnerabilities can be fixed by applying an update to affected packages. Ultimately, this is the best solution since it corrects the vulnerability. However, sometimes that is not desirable, or the fix might not be available immediately. Scheduling downtime for critical systems can sometimes be time-consuming, complex, or not feasible at that time.
If analysts are aware of effective mitigations of the issue – like using SELinux, changing software configurations or settings, running a short script, etc. – and the Red Hat Insights framework is able to detect those when they are used, the rule will also propose them. Having options for mitigations allows customers to better plan how they want to react without having to worry about exposure, and deploy the final fix at a time of their choosing.
In some cases, when we agree that the mitigation is as good as a fix, the Insights rule will disappear from the Red Hat Insights notifications list. If we deem it to be a good, but temporary solution, its severity is toned down.
Before a CSAw vulnerability becomes public, we review our rule against real production data. The rule is still embargoed and is accessible to as few people as necessary, even within Red Hat. Because of this, the rule back-end is uploaded to the production server out-of-band. It is moved from one private space to another as the access to the production server is also limited.
Everything is double-checked and tested as a whole. If testing goes as planned, the rule is ready when the CSAw vulnerability goes public. We share much of our work through our Red Hat Product Security Center articles and the Red Hat Insights service itself. Simultaneously, the vulnerability is announced on the mailing lists and to the public, a vulnerability article is published on the Customer Portal, fixes are made available in the repositories, and the rule status is changed to “active”.
And at that point, the long days of work by many tireless people become available to Red Hat Insights customers – so you can be aware of the things that matter the most at the moment.
For our final Ansible and Insights release blog, we will finish this three part series by showing you how to enable Tower to talk with the Insights API to enable jobs for site wide remediation. This builds on our previous blog post, Ansible and Insights Part 2 - Automating Ansible Core remediation, so if you have do not have the pre-requisites mentioned in Part 2, you should verify you have met those requirements and can build a Planner plan within Insights before trying to follow along.
Prerequisites for being able to utilise Ansible functionality with Insights are:
- Active RHEL subscription
- Active Insights evaluation or entitlement
- RHEL 7 or RHEL 6.4 and later
- Ansible Tower 3.1.2 for examples in this blog post
- Insights systems registered and reporting with an identifiable problem
- Ability to manage systems via Ansible with Insights system hostname or "display name" as the hostname in your ansible inventory in Ansible Tower.
- Ability to store credentials, projects, and create template's within Ansible Tower (administrator account is used in these examples).
- Your Red Hat Customer portal username/password (the same one you use to login to Red Hat Insights on the Customer Portal.
** Editor's Update Oct 12, 2017 **: For Tower 3.2 there are some new features and functionality available, as well as a few new steps to complete integration. Please see the latest Tower documentation for more information - These changes may make some steps in this instructional blog for Tower 3.1.2 obsolete, but will add improved functionality.
Creating a plan for your remediation in Red Hat Insights
Similar to building out a remediation plan for use with Ansible, you can create a plan for Ansible Tower with the same procedure. We outline this process in Ansible and Insights Part 2 - Automating Ansible Core remediation and the only minor difference is that once you save the plan it is able to be sync'd to your Tower. You do not have to download the playbook, although you can download and modify it for manual or process based (git, etc.) import into your Tower.
Since we have our example plans built from our previous blog, we need to now setup Tower to interface with the Red Hat Insights API. This only takes a few minor one time setup steps and a minimal amount of time.
Setting up Insights Credentials
- Login to your Tower and click the Settings icon to enter the Settings menu.
- Click Credentials to access the Credentials page.
- Click the add button located in the upper right corner of the Credentials screen.
- Enter the name of the credential to be used in the Name field. For example, "Red Hat Insights credentials"
- In the Type drop down menu list, select Source Control.
- Enter your Red Hat Customer Portal credentials associated with your Insights deployment in the Username and Password fields.
- Click Save when done.
Setting up Insights inventory
Insights will include a hosts: line that contains the hostname that Insights itself knows about which may be different than the hostname that Tower knows about. Therefore, make sure that those hostnames match up with what Tower has in its inventory by comparing the systems in the Red Hat Insights Portal to the systems listed in the Tower Inventory.
To create a new inventory for use with Insights:
- Click the Inventory main link to access the Inventories page.
- Click the Add button, which launches the New Inventory window.
Enter the name and organization to be used in their respective fields.
- Click Save to proceed to the Groups and Hosts Management screen.
In the Hosts (right side) of the Inventory display screen, click the Add Host button to open the Create Host dialog.
Enter the name in the Host Name field associated with the Insights host that will be used and click Save.
Now that we have our inventory to manage added or imported into Tower we should setup the Insights Planner sync project. Every time this project runs it will sync the available Planner plans between Insights and your Tower.
Setting up Insights project
In your Tower click the Projects main link to access the Projects page.
- Click the Add button, which launches the New Project window.
Enter the appropriate details into the required fields, at minimum.
SCM Type should be “Red Hat Insights”
Upon selecting the SCM type, the Source Details field expands. Enter the name of the credential you created in the previous step for use in this project in the text field provided, or click the search button to look up and select the name
- Click to select the update option for this project, and provide any additional values, if applicable. For information about each option, click the help button next to the options.
Creating an Insights job template
Now that we have our credentials, inventory, and projects added, we will create a job template to run an Insights playbook. You can do this for any large scale job you would want to enable across the enterprise or for groups within your enterprise.
- Click the Templates main link to access the Templates page.
- Click the Add button and select Job Template from the drop-down menu list, which launches the New Job Template window.
Enter the appropriate details into the required fields, at minimum. Note the following fields requiring specific Insights-related entries:
In the Inventory field, enter (or choose from lookup) the name of the inventory you created with the appropriate hostnames used by Insights.
In the Project field, enter (or choose from lookup) the name of the Insights project to be used with this job template.
In the Playbook drop-down menu list, choose the playbook to be launched with this job template from the available playbooks associated with the selected Insights project.
For additional information about each field, click its corresponding Help button or refer to Job Templates for details.
Complete the rest of the template with other attributes such as permissions, notifications, and surveys, as necessary. When done completing the job template, select Save.
- To launch the job template, click the Launch button (under Actions).
Once complete, the playbook's job results display in the Job Details page.
For the examples we have been following with this blog, you should use a similar Planner plan from our previous blog post, such as fixing a few systems in the Payload Injection Fix plan within Insights.
For that use case you would select the so-named "Payload Injection fix" from the Playbook dropdown and the remediation that we applied in the previous blog post can also be applied to machines from Tower.
You can then go back to the Insights Planner and see that the Payload Injection Fix Insights plan has remediated the selected systems via Tower.
We hope this helps you see how powerful the Insights and Tower integration is becoming, giving operations teams the ability to scale guided remediation out to the entire enterprise. Please let us know how we’re doing with Insights integrations and the service by emailing us at firstname.lastname@example.org or by using the Provide Feedback button at the top of every Insights Customer Portal page.
Stay tuned for more in depth and continued Red Hat Insights integrations into the Red Hat Management portfolio and other Red Hat software, and if you're interested in utilising these technologies in your own enterprise you can get started with an evaluation here.Posted: 2017-08-29T15:14:52+00:00
The Red Hat Insights team is pleased to highlight our first post-Summit 2017 service release for functionality and feature enhancement.
Red Hat Insights is a Software-as-a-Service (SaaS) that potentially prevents downtime by enabling customers to proactively monitor for infrastructure risks and critical security alerts detected in their environments, while requiring no added infrastructure. Insights offers automated remediation capabilities via Ansible Playbooks, as well as Executive Reporting features and Health Scoring, and recommends guidance on how to quickly and securely fix identified issues.
Our June 2017 release brings several new features to the Customer Portal Insights Web UI that are currently available for production environments, and beta features that are offered for testing and feedback in Insights Beta.
Read below for more information or go check them out and let us know your thoughts by using the "Provide Feedback" button.
For more information about the latest Insight release, refer to our Red Hat Insight Release Notes.
Incident Detection [Beta Release Pending]
Detecting "Incidents" within an infrastructure is a new concept added to Red Hat Insights. Previously, Insights would proactively detect issues you were at risk of encountering in the future and identify them early so they could be acted upon before they're encountered. This core functionality still exists; however, the Insights engine has been expanded to now detect critical issues we know are currently impacting your infrastructure at the time of analysis. By highlighting these incidents differently within the UI, we aim to direct immediate attention and prioritize these incidents to be addressed quickly, preventing further or impending disruption.
Insights Analysis of Openshift Infrastructure [Beta Release Pending]
Expands the capabilities of Insights to provide analysis of Openshift infrastructures (Master & Nodes).
Global Group Filtering [Beta & Stable]
Global group filters are now located throughout the UI, on almost all pages. These filters allow for modified views to only show the results within a selected group. The selected filter will remain with you as you navigate through Insights, until you reset or select another group.
Additional Page Filtering Capabilities [Beta & Stable]
Additional filtering capabilities have been added to Actions and Inventory views. Results can now be filtered by System Status (Checking-In or Stale), System Health (Affected or Healthy), and Incidents. Filtering is now designed to provide a consistent user experience no matter what page within Insights is being used.
Red Hat Insights Blog Subscription [Beta & Stable]
In an effort to keep users up to date with the latest news regarding Red Hat Insights, users are now automatically subscribed to the Red Hat Insights blog. New blog posts are submitted as new rules or features are added to Insights. Users can manage their subscriptions to this blog.
Red Hat’s Status Page Integration [Stable]
Integration with the Red Hat Status Page (status.redhat.com) has been completed and now provides up-to-date status of Red Hat Insights availability. The status page is used to communicate current outages, known availability issues or upcoming maintenance windows of Red Hat Insights stable, beta, and API.
Automatic Stale System Removal [Beta & Stable]
Automatic removal of stale systems helps users focus on the most up-to-date critical actions in their infrastructure, without the noise of older stale systems. A “stale” system is a system that is no longer checking-in with the Insights service daily, as expected. Once identified, the UI will highlight this system for action to be taken. After one month has passed with a stale status, the system will automatically be removed from Insights views.
Executive Reporting Enhancements [Beta]
Executive reporting was added in the April 2017 update of Red Hat Insights, providing users with views of historical trends and snapshots of infrastructure health. We have received multiple requests to enhance the reporting and have added the following features:
- Progress tracking and reporting on the number of issues resolved over the past 30 days.
- Appendix of all rule hits provides a quick report of issues identified by Insights within an account infrastructure, and the number of impacted systems.
- Overall Score improvements, on hover-over, provide additional details of what the score means and how it’s calculated. Additionally, the score color is modified based on the health of all systems.
- Export to PDF allows users to save and share their complete executive report. [Coming Soon]
Planner and Ansible Playbook Generation Improvements [Beta]
The Planner and playbook-builder UI has been improved to allow for more flexibility when adding to existing plans or creating new plans. Systems can now be added to previously specified groups, as individual systems, or all systems. Actions available to add are now displayed in intelligent views to allow for easier and quicker selection.
** The Insights team thanks all those who helped beta test. We're always hard at work adding new features and functionality. Let us know how we can continue to improve Insights.**Posted: 2017-06-07T17:04:51+00:00
- Progress tracking and reporting on the number of issues resolved over the past 30 days.
As we discussed in our previous blog post about enabling Ansible automation with Insights, we will look closer at taking findings from Insights and using the actionable intelligence provided to perform an automated remediation via Ansible playbook. Ansible Tower setup and remediation will be covered in an upcoming post.
Currently you can generate playbooks for Insights and Tower via Red Hat's customer portal. An upcoming release of Satellite 6 will further integrate Insights automated remediation into Satellite by allowing you to generate playbooks from the Satellite UI.
Prerequisites for being able to utilize Ansible functionality with Insights are:
- Active RHEL subscription
- Active Insights evaluation or entitlement
- RHEL 7 or RHEL 6.4 and later
- Ansible (or Ansible Tower) installed
- Insights systems registered and reporting with an identifiable problem
- Ability to manage systems via Ansible with Insights system hostname or "display name" as the hostname in your ansible inventory
Begin by logging into the Insights interface on the customer portal at https://access.redhat.com/insights
If you're already logged in, you'll be presented with the Insights Overview.
From the Overview you can see quickly if you have any systems that have automated remediation identified. In the lower right of the console under Planner you will see "# issues can be resolved automatically by Ansible" or something similar. You can use this to quickly see all items you can remediate with Ansible.
From here you have options. You can use Planner on the left nav menu to build a plan, you can click "Create a Plan/Playbook" from Overview, or you can use listed Actions (Actions -> Category) dropdowns for affected systems.
In this example we will navigate to Actions -> Security, and choose the "Kernel vulnerable to man-in-the-middle payload injection". We see that several systems are affected by this risk, and it has a medium likelihood, a critical impact, and a high overall total risk. This Action is also Ansible enabled.
Clicking into the Action itself gives us a description of the problem and a list of systems affected. From here we can create a playbook for the affected systems.
I'll choose the three affected systems and use the Actions dropdown dialogue to Create a New Plan/Playbook.
Give this plan a name (this is important; if you're using Tower integration this name is how we quickly identify the playbooks within Tower as well) and ensure the systems selected are correct. Click "Save" and the plan is created. From here you can delete or edit the plan to specify a maintenance window and duration, edit systems associated with this plan, or Generate Playbook and Export to CSV. We want to generate a playbook, so click that button.
If the playbook you're building has options (like this example) you will be presented with a dialogue to decide what tasks you want to include in your Ansible playbook. Currently you may need to goto "Playbook Summary", like the graphic above, to modify the playbook options. Since the selected machines are critical to my environment, and I can't afford to take downtime to fix them with a kernel update and reboot, I'll use the active mitigation and "Set sysctl ipv4 challenge ack limit". This will allow me to actively mitigate the system and make it non-vulnerable. A more permanent fix would be to update the kernel, but if I'm sure nothing is going to change my sysctl variable back (config management tools may reverse these changes if not also updated), then I would be safe with this active mitigation.
Click Save to confirm your selection and finalize playbook generation by Downloading Playbook.
You can then use this downloaded Ansible playbook YML file to remediate the systems with: $ ansible-playbook $downloaded_filename.yml
Filenames follow a scheme of plan_name-plan_number-unixtime.yml and contain information inside about which remediation systems and rule versions are being utilized.
After watching the playbook run, assuming there are no errors you need to further investigate, refreshing Planner shows us 3/3 systems have been remediated.
Upon refreshing the Planner interface we see that the remediations were performed successfully and these systems now have a check mark as their status.
That's how simple it is to start using Ansible playbooks to remediate systems reporting risks. Stay tuned for another upcoming blog post on how to scale this to your entire infrastructure with Ansible Tower.
Let us know your thoughts on the new features highlighted in our last post, in the comments on the blogs or with the Provide Feedback button inside of Insights!
Thanks from all of us here at the Insights engineering and product teams, and happy remediating. Stay tuned for part 3, where we will be using Ansible Tower and Insights for enterprise remediation.
-Will NixPosted: 2017-06-01T15:55:58+00:00