Chapter 7. Alerts

7.1. Assigning the Notifier Role

  1. Navigate to Configure → Configuration.
  2. Click the Settings accordion, and select the CloudForms Management Engine server.

  3. From the Server Control tab, select the Notifier role.

    6255
  4. Click Save.

7.2. Creating an Alert

In this section, the basics of creating an Alert are described. Detailed instructions for the specific types of Alerts are given in the sections following.

To send emails or SNMP traps from the CloudForms Management Engine server, you must have the Notifier server role enabled and have set up SMTP email or SNMP traps. For further information, see General Configuration.

  1. Navigate to Control → Explorer.

  2. Click the Alerts accordion, then click 1847 (Configuration), 1848 (Add a new Alert).

    • Type in a description for the alert.
    • Check Active when you feel that the alert is ready to be enabled.
    • From Based On, select the type of infrastructure item to base the alert on.
    • The options shown in What to Evaluate change based on what you selected in Based On.
    • In Notification Frequency, select how often you want to be notified if the event log threshold is reached.
  3. The parameters available are based on the What to Evaluate selection. See the following sections for additional details on each alert type.

  4. To send an email, check Send an E-mail. Parameters required for sending an email are displayed.

    1967
    • In From, type in the sending email.
    • Use Add a CloudForms Management Engine User to select a user. The CloudForms Management Engine user must have a valid email address entered under accounts.
    • Use Add (enter manually) to type in the address not registered to a CloudForms Management Engine user. Then, click 1848 (Add).

  5. If you check Send an SNMP Trap, type in the IP for the host to send the trap to, select the version of SNMP that you are using, and type in the Trap Object ID. Type in multiple hosts if you need the trap sent to multiple SNMP hosts.

    • If using SNMP V1, you will be prompted for a Trap Number. Type 1, 2, or 3, based on the appropriate suffix number from table below.
    • If using SNMP V2, you will be prompted for a Trap Object ID. Type info, warning, or critical, based on the table below.

    • Trap Object ID and suffix number

      Object IDSuffix Number Added to PENPEN with the Suffix Added

      info

      1

      1.3.6.1.4.1.33482.1

      warn, warning

      2

      1.3.6.1.4.1.33482.2

      crit, critical, error

      3

      1.3.6.1.4.1.33482.3

      1966
  6. To show the alert as an event on the CloudForms Management Engine timeline, check Show on Timeline. It shows as part of the Alarm/Status Change/Errors category.
  7. To invoke automation, check Send a Management Event. Type in the name of the event. This item exists in the Process/Event Class.
  8. Click Add.

7.3. Creating a Hardware Reconfigured Alert

Use a hardware reconfigure alert to detect changes to the amount of memory or the number of CPUs on a virtual machine.

  1. Navigate to Control → Explorer.
  2. Click the Alerts accordion, then click 1847 (Configuration), 1848 (Add a new Alert).

  3. In the Info area:

    • Type in a description for the alert.
    • From Based On, select VM and Instance.
    • From What to Evaluate, select Hardware Reconfigured.
    • In Notification Frequency, select how often you want to be notified if hardware reconfiguration is detected.

  4. From Hardware Attribute, select Number of CPUs. From the next dropdown, select Decreased.

    1973
  5. After setting the parameters, select what you want the alert to do. You can send an email, create an SNMP Trap, let the alert show on the timeline, or send a management event to start an automation process.
  6. Click Add.

7.4. Creating a Normal Operating Range Alert

Normal operating range alerts enables you to be notified when the normal operating range is exceeded, or falls below for a period of time from 1 minute to 2 hours. Capacity and utilization must be enabled for normal operating ranges to be calculated. See the General Configuration guide for more information.

  1. Navigate to Control → Explorer.
  2. Click the Alerts accordion, then click 1847 (Configuration), 1848 (Add a new Alert).

  3. In the Info area:

    • Type in a Description for the alert.
    • From Based On, select VM and Instance.
    • For What to Evaluate, select Normal Operating Range.
    • In Notification Frequency, select how often you want to be notified if the performance threshold is reached.

  4. Set the threshold in the Normal Operating Range Parameters area.

    1976
    • From Performance Field, select the field to check and whether you want to be notified if the field is exceeded or fell below.
    • In Field Meets Criteria for, select the amount of time that the threshold requires to be met to trigger the alert.
  5. After setting the parameters, you then select what you want the alert to do. You can send an email, create an SNMP Trap, let the alert show on the timeline, or send a management event to start an automation process. See Section 7.2, “Creating an Alert”.
  6. Click Add.

7.5. Creating a Real Time Performance Alert

Real Time Performance alerts enables you to be notified immediately when a performance threshold has been met for a virtual machine, host, or cluster. Capacity and Utilization must be enabled for performance thresholds to be detected. See General Configuration for more information.

  1. Navigate to Control → Explorer.
  2. Click the Alert accordion, then click 1847 (Configuration), 1848 (Add a new Alert).

  3. In the Info area:

    • Type in a Description for the alert.
    • From Based On, select VM and Instance.
    • For What to Evaluate, select Real Time Performance.
    • In Notification Frequency, select how often you want to be notified if the performance threshold is reached.

  4. Set the threshold in the Real Time Performance Parameters area.

    1978
    • From Performance Field, select the field to check and any other parameters required for that field.
    • In And is Trending, select Don’t Care if it does not matter how the performance metric is trending. Otherwise, choose from the possible trending options.
    • In Field Meets Criteria for, select the amount of time that the threshold requires to be met to trigger the alert.
    • Set Debug Tracing to true only when directed to do so by Red Hat Support. This provides an extremely detailed level of logging and can result in many more log lines being written.
  5. After setting the parameters, you then select what you want the alert to do. You can send an email, create an SNMP Trap, let the alert show on the timeline, or send a management event to start an automation process.
  6. Click Add.

7.6. Creating an Hourly Performance Alert

Hourly performance alerts enable you to be notified immediately when an hourly performance threshold has been met for a cluster. Capacity and Utilization must be enabled for performance thresholds to be detected. See General Configuration for instructions.

  1. Navigate to Control → Explorer.
  2. Click the Alerts accordion.
  3. Click 1847 (Configuration), 1848 (Add a new Alert).

  4. In the Info area:

    1979
    • Type in a Description for the alert.
    • From Based On, select Cluster.
    • For What to Evaluate, select Hourly Performance.
    • In Notification Frequency, select how often you want to be notified if threshold is met.
  5. In the Hourly Performance Parameters area select performance field and the criteria. You can also select options from the And is Trending dropdown box and whether the Debug Tracing is true or false.
  6. After setting the parameters, you then select what you want the alert to do. You can send an email, create an SNMP Trap, let the alert show on the timeline, or send a management event to start an automation process.
  7. Click Add.

7.7. Creating a hostd Log Threshold Alert

Use the hostd Log Threshold to send a notification when certain items are found in the event logs for a host. A default analysis profile with event log items is required for this feature. The following example shows steps to check the host’s log for a failure to validate a virtual machine’s IP address.

  1. Navigate to Control → Explorer.
  2. Click the Alert accordion.
  3. Click 1847 (Configuration), 1848 (Add a new Alert).

  4. In the Info area:

    1981
    • Type in a Description for the alert.
    • From Based On, select Host.
    • For What to Evaluate, select Hostd Log Threshold.
    • In Notification Frequency, select how often you want to be notified if the log item is detected.

  5. In the Hostd Log Threshold Parameters area, select the parameters for the event log message. You can set a threshold for a filter, level, or message source.

    1982
    • Use Message Filter to look for specific text in a message. Use Message Level to filter based on message level. CloudForms Management Engine reports on the specified level and above. Use Message Source to filter log messages based on its source.
    • Set How Far Back to Check in days you want to look for this message.
    • If you only want an alert triggered when the log message has occurred a certain number of times, type the number in Event Count Threshold.
  6. After setting the parameters, select what you want the alert to do. You can send an email, create an SNMP Trap, let the alert show on the timeline, or send a management event to start an automation process.
  7. Click Add.

7.8. Creating a VMware Alarm Alert

CloudForms Management Engine can use VMware alarms as a trigger for an alert. This type of alert can be created for a cluster, host, or virtual machine.

  1. Navigate to Control → Explorer.
  2. Click the Alerts accordion, then click 1847 (Configuration), 1848 (Add a new Alert).

  3. In the Info area:

    • Type in a description for the alert.
    • From Based On, select Cluster, Host, or VM.
    • For What to Evaluate, select VMware Alarm.
    • In Notification Frequency, select how often you want to be notified if the log item is detected.

  4. In the VMware Alarm Parameters area select the provider and alarm.

    1984
  5. After setting the parameters, you then select what you want the alert to do. You can send an email, create an SNMP Trap, let the alert show on the timeline, or send a management event to start an automation process.
  6. Click Add.

7.9. Creating an Expression Alert

Expression alerts enables you to create a notification based on any possible criteria for clusters, datastores, hosts, and virtual machines. The following procedure creates an alert for when a host’s datastore has less than 5% free space.

  1. Navigate to Control → Explorer.
  2. Click on the Alerts accordion, then click 1847 (Configuration), 1848 (Add a new Alert).

  3. In the Info area:

    1985
    • Type in a description for the alert.
    • From Based On, select Host.
    • For What to Evaluate, select Expression (Custom).
    • In Notification Frequency, select how often you want to be notified if the expression is evaluated to true.

  4. Use the expression editor to create your expression. This is the same expression editor used to create Conditions. For details on how to use the expression editor, see Defining Policies and Profiles.

    1986
  5. Click 1863 (Commit expression element changes) to accept the expression.
  6. After setting the parameters, you then select what you want the alert to do. You can send an email, create an SNMP Trap, let the alert show on the timeline, or send a management event to start an automation process.
  7. Click Add.

7.10. Create an Operational Alert

  1. Navigate to Control → Explorer.
  2. Click on the Alerts accordion, then click 1847 (Configuration), 1848 (Add a new Alert).

  3. In the Info area:

    • Type in a description for the alert.
    • Check Active when you feel that the alert is ready to be enabled.
    • From Based On, select Server.
    • Select the appropriate driving event.
    • In Notification Frequency, select how often you want to be notified if the event log threshold is reached.
  4. After setting the parameters, select what you want the alert to do. You can send an email, create an SNMP Trap, let the alert show on the timeline, or send a management event to start an automation process.
  5. Click Add.

7.11. Operational Alert Types

Table 7.1. Operational Alerts

Driving EventExplanation (Thresholds, Description)Proposed Action if Alert is Raised

EVM Server Start

Alert is raised when an EVM Server starts.

 

EVM Server Stop

Alert is raised when an EVM Server stops.

 

EVM Server Not Responding

Alert is raised when one EVM server detects that another EVM Server has not responded in (2 minutes).

This is a sign of a problem that should be investigated. Check logs.

EVM Server Exceeded Memory Limit

Alert is raised when an EVM server has exceeded its system memory limit and begins killing workers. Default is 80%.

Threshold configured in Advanced Settings.

server:

:worker_monitor:

:kill_algorithm:

:name: :used_swap_percent_gt_value

:value: 80

This may be caused by the following issues:

The server is running with too few resources.

The server is enabled with too many roles or number of workers.

The server picked up all the roles because another server has failed.

A runaway process has taken up most of the memory.

EVM Server is Master

When one EVM Server takes over as a master server.

Typically, this should only occur when first starting a set of servers, perhaps following expected outages. If a server picks up as master in other situations, the old master had an issue that needs to be researched (such as server not responding in time).

EVM Server High System Disk Usage

The EVM Servers system disk is 80% full. This check is run as part of a system schedule.

Threshold configured in Advanced Settings.

server:

events:

:disk_usage_gt_percent: 80

Something is filling the disk such as temp files used by the operating system such as, yum updates and normal /tmp files, or EVM temp files in /var/lib/data/miqtemp/.

EVM Server High App Disk Usage

The EVM Servers app disk is 80% full. This check is run as part of a system schedule.

Threshold configured in Advanced Settings.

server:

events:

:disk_usage_gt_percent: 80

Possibly EVM temp files are being left around.

EVM Server High Log Disk Usage

The EVM Servers log disk is 80% full. This check is run as part of a system schedule.

Threshold configured in Advanced Settings.

server:

events:

:disk_usage_gt_percent: 80

Logs are getting too big or are not being log rotated properly every day. Check most recent logs.

EVM Server High DB Disk Usage

The EVM Servers db disk is 80% full. This check is run as part of a system schedule. Applies if using PostgreSQL as the VDMB.

Threshold configured in Advanced Settings.

server:

events:

:disk_usage_gt_percent: 80

Database or database logging is getting too large. May need FULL vacuuming of PostgreSQL database.

EVM Worker Started

Alert is raised when a worker is about to start.

 

EVM Worker Stopped

Alert is raised when a worker is requested to stop.

 

EVM Worker Killed

Alert is raised when a non- responsive worker does not restart on its own and is killed.

 

EVM Worker Not Responding

Alert is raised when a worker has not responded for 2 minutes (:heartbeat_timeout) or has not started within 10 minutes (:starting_timeout).

 

EVM Worker Exceeded Memory Limit

Alert is raised when a worker exceeds the memory threshold. The default is 150 MB, but some workers have their own value in the :memory_threshold section for that specific worker.

 

EVM Worker Exceeded Uptime Limit

Alert is raised when a worker has been running longer than the :restart_interval. (Most workers are set to never restart using the 0.hours setting.) The EMS Refresh SmartProxy workers are set to restart every 2 hours.

 

EVM Worker Exit File

Alert is raised when the scheduler worker exits due to a pending large ntp time change.

 

7.12. Editing an Alert

After creating an alert, you can edit the threshold, expression, or the notification type.

  1. Navigate to Control → Explorer
  2. Click on the Alerts accordion, then click on the alert that you need to edit.
  3. Click 1847 (Configuration), 1851 (Edit this Alert).
  4. Make the required changes.
  5. Click Save.

7.13. Copying an Alert

You can copy an existing alert to create a new alert that is similar to the existing one, then change the values associated with it.

  1. Navigate to Control → Explorer.
  2. Click on the Alert accordion, then click on the alert that you want to copy.
  3. Click 1847 (Configuration), 1859 (Copy this Alert). Click OK to confirm.
  4. Make the required changes.
  5. Click Add.

7.14. Deleting an Alert

When an alert is no longer needed, you can remove it from your VMDB.

  1. Navigate to Control → Explorer.
  2. Click on the Alerts accordion, then click on the alert that you want to delete.
  3. Click 1847 (Configuration), 1861 (Delete this Alert).
  4. Click OK to confirm.

7.15. Evaluating an Alert

  1. Navigate to Control → Explorer.
  2. Click the Actions accordion, then click 1847 (Configuration), 1848 (Add a new Action).

  3. Type in a Description for the action.

    1911
  4. Select Evaluate Alerts from Action Type.

  5. Select the alerts to be evaluated and click 1876 (Move selected Alerts into this Action). Use the Ctrl key to select multiple alerts.

    1912
  6. Click Add.