Many provider components are named according to a name-spacing schema that follows the style of:

ManageIQ::Providers::<ProviderName>::<ProviderCategory>

Some examples of this are as follows:

The Automation Engine role enables a CFME appliance to process queued automation tasks.^[2]. There should be at least one CFME appliance with this role set in each zone. The role does not have a dedicated worker, automate tasks are processed by either a MiqGenericWorker or a MiqPriorityWorker, depending on message priority.

Capacity and utilization (C&U) metrics processing is a relatively resource-intensive operation, and there are three roles associated with its operation.

The Database Operations role enables a CFME appliance to run certain database maintenance tasks such as purging old metrics. This role does not have a dedicated worker, the database operations tasks are processed by a MiqGenericWorker.

The Embedded Ansible role enables the use of the built-in Ansible automation manager, which allows Ansible playbooks to be run from service catalogs, or from control actions and alerts. If more than one CFME appliance in a region has this role enabled, only one will be active at a time. This role has a dedicated worker called the EmbeddedAnsibleWorker, but enabling the role also starts the following event catcher and refresh workers:

The Event Monitor role is responsible for detecting and processing provider events such as a VM starting or stopping, a cloud instance being created, or a hypervisor rebooting. Enabling the role starts at least 2 workers; one or more provider-specific, and one common event handler.

The provider-specific event catcher maintains a connection to a provider’s event source (such as the Google Cloud Pub/Sub API for Google Compute Engine) and detects or 'catches' events and passes them to the common event handler. An event catcher worker is started for each provider in the appliance’s zone; a zone containing a VMware provider would contain a ManageIQ::Providers::Vmware::InfraManager::EventCatcher worker, for example.

Some cloud providers automatically add several types of manager, and these might each have an event catcher worker. To illustrate this, enabling the event monitor role on an appliance in an OpenStack Cloud provider zone would start the following event catcher workers:

The event handler worker, called MiqEventHandler, is responsible for feeding the events from all event catchers in the zone into the automation engine’s event switchboard for processing.

There should be at least one CFME appliance with the event monitor role set in any zone containing a provider, however if more than one CFME appliance in a zone has this role, only one will be active at a time.

A CFME appliance with the Git Repositories Owner role enabled is responsible for synchronising git repository data from a git source such as Github or Gitlab, and making it available to other appliances in the region that have the automation engine role set. The git repository data is copied to /var/www/miq/vmdb/data/git_repos/<git_profile_name>/<git_repo_name> on the CFME appliance. This role does not have a dedicated worker.

The Notifier role should be enabled if CloudForms is required to forward SNMP traps to a monitoring system, or to send e-mails. These might be initiated by an automate method or from a control policy, for example.

If more than one CFME appliance in a region has this role enabled, only one will be active at a time. This role does not have a dedicated worker, notifications are processed by either a MiqGenericWorker or a MiqPriorityWorker, depending on message priority.

The Provider Inventory role is responsible for refreshing provider inventory data for all provider objects such as virtual machines, hosts, clusters, tenants, or orchestration templates. It is also responsible for capturing datastore file lists. If more than one CFME appliance in a zone has this role enabled, only one will be active at a time.

Setting this role starts the provider-specific refresh workers for any providers in the appliance’s zone; a zone containing a RHV provider would contain a ManageIQ::Providers::Redhat::InfraManager::RefreshWorker worker, for example.

VMware providers add an additional MiqEmsRefreshCoreWorker, while cloud providers that use several types of manager add a worker per manager. For example enabling the Provider Inventory role on an appliance in an Azure provider zone would start the following Refresh workers:

A CFME appliance with the Provider Operations role performs certain managed object operations such as stop, start, suspend, shutdown guest, clone, reconfigure, etc., to provider objects such as VMs. These operations might be initiated from the WebUI, from Automate, or from a REST call. It also handles some storage-specific operations such as creating cloud volume snapshots. The role does not have a dedicated worker, provider operations tasks are processed by either a MiqGenericWorker or a MiqPriorityWorker, depending on message priority. There is no limit to the number of concurrent workers handling this role in a zone.

A CFME appliance with the RHN Mirror role acts as a repository server for the latest CloudForms Management Engine RPM packages. It also configures other CFME appliances within the same region to point to itself for updates. This provides a low bandwidth method to update environments with multiple appliances. The role does not have a dedicated worker.

The Reporting role allows a CFME appliance to generate reports. There should be at least one CFME appliance with this role in any zone in which reports are automatically scheduled or manually requested/queued.^[3] (such as from a WebUI zone).

Enabling this server role starts one or more MiqReportingWorker workers.

The Scheduler sends messages to start all scheduled activities such as report generation, database backups, or to retire VMs or services. One server in each region must be assigned this role or scheduled CloudForms events will not occur. Enabling this server role starts the MiqScheduleWorker worker.

Enabling the SmartProxy role turns on the embedded SmartProxy on the CFME appliance. The embedded SmartProxy can analyse virtual machines that are registered to a host and templates that are associated with a provider. Enabling this role starts three MiqSmartProxyWorker workers.

The SmartState Analysis role controls which CFME appliances can control SmartState Analyses and process the data from the analysis. There should be at least one of these in each zone that contains a provider. This role does not have a dedicated worker, SmartState tasks are processed by either a MiqGenericWorker or a MiqPriorityWorker, depending on message priority.

This role enables access to a CFME appliance using the Red Hat CloudForms Operations WebUI console. More than one CFME appliance can have this role in a zone (the default behaviour is to have this role enabled on all appliances). Enabling this server role starts one or more MiqUiWorker workers.

This role enables the RESTful Web service API on a CFME appliance. More than one CFME appliance can have this role in a zone. Enabling this server role starts one or more MiqWebServiceWorker workers.

This role enables a CFME appliance to be used as a websocket proxy for the VNC and SPICE HTML5 remote access consoles. It is also used by the WebUI notification service. Enabling this server role starts one or more MiqWebsocketWorker workers.

Many server roles - or more accurately their worker processes - have an affinity to the zone with which the hosting CFME appliance is associated. For example messages intended for zone "A" will generally not be processed by worker processes in zone "B".

The following server roles have zone affinity:

Monitoring the health status of workers becomes important as a CloudForms installation is scaled. A server thread called validate_worker checks that workers are alive (they have recently issued a 'heartbeat' ping.^[4]), and are within their time limits and memory thresholds. Some workers such as Refresh and SmartProxy workers have a maximum lifetime of 2 hours to restrict their resource consumption.^[5]. If this time limit is exceeded, the validate_worker thread will instruct the worker to exit at the end of its current message processing, and spawn a new replacement.

The following evm.log line shows an example of the normal timeout processing for a RefreshWorker:

INFO -- : MIQ(MiqServer#validate_worker) Worker ⏎
[ManageIQ::Providers::Vmware::InfraManager::RefreshWorker] ⏎
with ID: [1000000258651], PID: [17949], ⏎
GUID: [77362eba-c179-11e6-aaa4-00505695be62] uptime has reached ⏎
the interval of 7200 seconds, requesting worker to exit

The following log line shows an example of an abnormal exit request for a MiqEmsMetricsProcessorWorker that has exceeded its memory threshold (see Section 2.6.2.1, “Worker Memory Thresholds”:

WARN -- : MIQ(MiqServer#validate_worker) Worker [MiqEmsMetricsProcessorWorker] ⏎
with ID: [1000000259290], PID: [15553], ⏎
GUID: [40698326-c18a-11e6-aaa4-00505695be62] process memory usage [598032000] ⏎
exceeded limit [419430400], requesting worker to exit

grep 'MiqServer#validate_worker' evm.log

It is often a requirement to tune the number of per-appliance workers and their memory thresholds when CloudForms is deployed to manage larger clouds or virtual infrastructures.

2.6.2.2. Adjusting Worker Settings

The count and maximum memory thresholds for most worker types can be tuned from the CloudForms WebUI, in the Workers tab of the Configuration → Settings page for each appliance (see Figure 2.3, “Worker Settings”).

Figure 2.3. Worker Settings

​

For other workers not listed in this page, the memory threshold settings can be tuned (with caution) in the Configuration → Advanced settings by directly editing the YAML, for example:

:workers:
  :worker_base:
  ...
    :ui_worker:
      :connection_pool_size: 8
      :memory_threshold: 1.gigabytes
      :nice_delta: 1
      :count: 1

Tasks are dispatched to the various workers in one of three ways:

Many of the queued messages are created by workers dispatching work to other workers. For example, the Schedule worker will queue a message for the SmartProxy workers to initiate a SmartState Analysis. An Event Catcher worker will queue a message for an Event Handler worker to process the event. This will in turn queue a message for a Priority worker to process the event through the automate event switchboard.

To improve the performance of the messaging system, each CFME appliance prefetches a batch of messages into its local memcache. When a worker looks for work by searching for a "ready" state message, it calls an MiqQueue method get_message_via_drb that transparently searches the prefetched message copies in the memcache. If a suitable message is found, the message’s state in the VMDB miq_queue table is changed to "dequeue", and the message is processed by the worker.

A message contains a number of fields. The useful ones to be aware of for troubleshooting purposes are described below.

Message processing is so critical to the overall performance of a CloudForms installation, that understanding how to follow messages in evm.log is an important skill to master when scaling CloudForms. There are generally four stages of message processing that can be followed in the log file. For this example a message will be traced that instructs the Automation Engine (role "automate" in queue "generic") to run the method AutomationTask.execute on automation task ID 7829.

... INFO -- : Q-task_id([automation_request_6298]) MIQ(MiqQueue.put) ⏎
Message id: [32425368], ⏎
id: [], ⏎
Zone: [RHV], ⏎
Role: [automate], ⏎
Server: [], ⏎
Ident: [generic], ⏎
Target id: [], ⏎
Instance id: [7829], ⏎
Task id: [automation_task_7829], ⏎
Command: [AutomationTask.execute], ⏎
Timeout: [600], ⏎
Priority: [100], ⏎
State: [ready], ⏎
Deliver On: [], ⏎
Data: [], ⏎
Args: []

... INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) ⏎
Message id: [32425368], ⏎
MiqWorker id: [260305], ⏎
Zone: [RHV], ⏎
Role: [automate], ⏎
Server: [], ⏎
Ident: [generic], ⏎
Target id: [], ⏎
Instance id: [7829], ⏎
Task id: [automation_task_7829], ⏎
Command: [AutomationTask.execute], ⏎
Timeout: [600], ⏎
Priority: [100], ⏎
State: [dequeue], ⏎
Deliver On: [], ⏎
Data: [], ⏎
Args: [], ⏎
Dequeued in: [6.698342458] seconds

... State: [dequeue], Deliver On: [2017-04-27 09:00:00 UTC], ⏎
Data: [], Args: ["2017-04-27T08:00:00Z", "hourly"], ⏎
Dequeued in: [2430.509191336] seconds

... INFO -- : Q-task_id([automation_task_7829]) ⏎
MIQ(MiqQueue#deliver) Message id: [32425368], Delivering...

... INFO -- : Q-task_id([automation_task_7829]) ⏎
MIQ(MiqQueue#delivered) ⏎
Message id: [32425368], ⏎
State: [ok], ⏎
Delivered in [23.469068759] seconds

The overall performance of any multi-appliance CloudForms installation is largely dependant on the timely processing of messages. Fortunately the internal log_system_status method writes the queue states to evm.log every 5 minutes, and this information can be used to assess message throughput.

To find the numbers of messages currently being processed (in state "dequeue") in each zone, use the following bash command:

grep 'count for state=\["dequeue"\]' evm.log

... Q-task_id([log_status]) MIQ(MiqServer.log_system_status) ⏎
[EVM Server (2768)] MiqQueue count for state=["dequeue"] ⏎
by zone and role: {"RHV"=>{nil=>1, "automate"=>1, ⏎
"ems_metrics_coordinator"=>1, "ems_metrics_collector"=>2, ⏎
"ems_metrics_processor"=>2, "smartproxy"=>1, "smartstate"=>2}, ⏎
nil=>{"database_owner"=>1}}

To find the numbers of messages in state "error" in each zone, use the following bash command:

grep 'count for state=\["error"\]' evm.log

... Q-task_id([log_status]) MIQ(MiqServer.log_system_status) ⏎
[EVM Server (2768)] MiqQueue count for state=["error"] ⏎
by zone and role: {"RHV"=>{nil=>36}, "default"=>{nil=>16}, ⏎
"UI Zone"=>{nil=>35}}

To find the numbers of messages in state "ready" that are waiting to be dequeued in each zone, use the following bash command:

grep 'count for state=\["ready"\]' evm.log

... Q-task_id([log_status]) MIQ(MiqServer.log_system_status) ⏎
[EVM Server (2768)] \ MiqQueue count for state=["ready"] ⏎
by zone and role: {"UI Zone"=>{"smartstate"=>15, "smartproxy"=>2, ⏎
nil=>4}, "default"=>{"automate"=>2, nil=>21, "smartstate"=>1, ⏎
"smartproxy"=>1}, "RHV"=>{"automate"=>6, "ems_inventory"=>1, ⏎
nil=>19, "smartstate"=>2, "ems_metrics_processor"=>1259, ⏎
"ems_metrics_collector"=>641}}

All CFME appliances in a region access the same PostgreSQL database, and so the I/O and CPU performance of the database server is a significant factor in determining the maximum size to which a region can grow (in terms of numbers of managed objects) whilst maintaining acceptable performance.

When sizing a region, some thought needs to be given to the number of CloudForms worker processes that are likely to be needed to handle the expected workload, and hence the number of CFME appliances. The workload will depend on the capabilities of the providers that will be configured, and the CloudForms features that are likely to be used.

Two of the most resource-intensive tasks are those performed by the C&U Data collector and Data Processor workers, particularly where there is a limited time window for the collection of realtime data as there is with VMware or OpenStack providers (see Chapter 6, Capacity & Utilization). It has been established through testing that one C&U Data Collector worker can retrieve and store the metrics from approximately 150 VMware VMs or OpenStack instances in the rolling 60 minute time window that realtime metrics are retained for. As an out-of-the-box CFME appliance is configured with 2 C&U Data Collector workers, it should be able to handle the collection of realtime metrics for 300 VMs. If the number of workers is increased to 4, the appliance could handle the collection of realtime metrics for 600 VMs, although the increased CPU and memory load may adversely affect other processing taking place on the appliance.

Using the 1:300 ratio of CFME appliances to VMs is a convenient starting point for scaling the number of CFME appliances required for a region containing VMware, RHV or OpenStack providers. For other provider types this ratio is often increased to 1:400.

Table 3.2, “Objects per CFME Appliance Guidelines” provides suggested guideline ratios for each of the provider types. It should be noted that these numbers are approximate and are only suitable for planning and design purposes. The final numbers of CFME appliances required for a region or zone can only be determined from analysis of the specific region workload, and the performance of existing CFME appliances.

There are a number of considerations for region design and layout, but the most important are the anticipated number of managed objects (discussed above), and the location of the infrastructure components being managed, or the public cloud endpoints.

vmdb
tools/db_ping.rb
0.358361 ms
1.058845 ms
0.996966 ms
1.029908 ms
1.048192 ms

Average: 0.898454 ms

bin/rails runner tools/db_ping.rb

tools/db_ping_remote.rb 10.3.0.22 5432 root vmdb_production
Enter the password for database user root on host 10.3.0.22
Password:
10.874407 ms
10.984994 ms
11.040376 ms
11.119602 ms
11.031609 ms

Average: 11.010198 ms

As illustrated in Figure 3.1, “Regions and Zones” regions can be linked in such a way that several subordinate regions replicate their object data to a single global region. The global region has no providers of its own, and is typically used for enterprise-wide reporting as it has visibility of all objects. A new feature introduced with CloudForms 4.2 allows some management operations to be performed directly from the global region, utilising a RESTful API connection to the correct child region to perform the action. These operations include the following:

Regions have associated with them a region number that is allocated when the VMDB appliance is first initialised. When several regions are linked in a global/subregion hierarchy, all of the region numbers must be unique. Region numbers can be up to three digits long, and the region number is encoded into the leading digits of every object ID in the region. For example the following 3 message IDs are from different regions:

Global regions are often allocated a higher region number (99 is frequently used) to distinguish them from subordinate regions whose numbers often start with 0 and increase as regions are added. There is no technical restriction on region number allocation in a connected multi-region CloudForms deployment, other than uniqueness.

The following guidelines can be used when designing a region topology:

The following sections describe some of the advantages of implementing zones within a CloudForms region.

The following guidelines can be used when designing a zone topology:

The PostgreSQL configuration file on a CFME appliance is /var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf. Some of the values in this file have been defined based on the assumption of a small CloudForms installation. For larger installations - particularly when using a dedicated database instance - these values should be changed.

shared_buffers = 128MB       # MIQ Value SHARED CONFIGURATION
#shared_buffers = 1GB        # MIQ Value DEDICATED CONFIGURATION

max_connections = 1000                  # MIQ Value;
#max_connections = 100

SELECT datname,application_name,client_addr FROM pg_stat_activity;

netstat -tp | grep postgres

#log_directory = 'pg_log'      # directory where log files are written,
                               # can be absolute or relative to PGDATA

mkdir -p /var/log/pg_log
chown postgres:postgres /var/log/pg_log

log_directory = '/var/log/pg_log'

systemctl restart rh-postgresql95-postgresql.service

/var/www/miq/vmdb/log/*.log /var/www/miq/vmdb/log/apache/*.log ⏎
        /var/log/pg_log/*.log /var/log/tower/*.log {
  daily
  dateext
  missingok
  rotate 14
  notifempty
  compress
  copytruncate
  prerotate
    source /etc/default/evm; /bin/sh ⏎
       ${APPLIANCE_SOURCE_DIRECTORY}/logrotate_free_space_check.sh $1
  endscript
  lastaction
    /sbin/service httpd reload > /dev/null 2>&1 || true
  endscript
}

sysctl -w vm.nr_hugepages=600
echo "vm.nr_hugepages=600" >> /etc/sysctl.d/rh-postgresql95.conf

The :ems_refresh section of the Configuration → Advanced settings is listed as follows:

:ems_refresh:
  :capture_vm_created_on_date: false
  :ec2:
    :get_private_images: true
    :get_shared_images: true
    :get_public_images: false
    :public_images_filters:
    - :name: image-type
      :values:
      - machine
    :ignore_terminated_instances: true
  :ansible_tower_configuration:
    :refresh_interval: 15.minutes
  :foreman_configuration:
    :refresh_interval: 15.minutes
  :foreman_provisioning:
    :refresh_interval: 1.hour
  :full_refresh_threshold: 100
  :hawkular:
    :refresh_interval: 15.minutes
  :kubernetes:
    :refresh_interval: 15.minutes
  :openshift:
    :refresh_interval: 15.minutes
  :openshift_enterprise:
    :refresh_interval: 15.minutes
  :raise_vm_snapshot_complete_if_created_within: 15.minutes
  :refresh_interval: 24.hours
  :scvmm:
    :refresh_interval: 15.minutes
  :vmware_cloud:
    :get_public_images: false

... INFO -- : MIQ(ManageIQ::Providers::Redhat::InfraManager:: ⏎
RefreshWorker::Runner#do_before_work_loop) EMS [rhvm] as [admin] ⏎
Queueing initial refresh for EMS

  :rhevm:
    :full_refresh_threshold: 200

... MIQ(ManageIQ::Providers::Vmware::InfraManager::Refresher# ⏎
preprocess_targets) Escalating to full refresh for EMS: [vCenter6], ⏎
id: [1000000000002].

(225 - 6) / 3 = 73

Every 3 minutes a message is queued for the C&U Coordinator to begin processing.^[15]. The Coordinator schedules the data collections for all of the managed objects in the VMDB, and queues messages for the C&U Data Collector to retrieve metrics for any objects for which a collection is due. The default time interval between collections (the capture threshold) is 10 minutes, but for VMs, Clusters and Hosts this is expanded to 50 minutes, and for storage, 60 minutes.

The capture thresholds are defined in the :performance section of the Configuration → Advanced settings, as follows:

:performance:
  :capture_threshold:
    :default: 10.minutes
    :ems_cluster: 50.minutes
    :host: 50.minutes
    :storage: 60.minutes
    :vm: 50.minutes
  :capture_threshold_with_alerts:
    :default: 1.minutes
    :host: 20.minutes
    :vm: 20.minutes

If a control alert is defined for an object type, the shorter capture thresholds defined under :capture_threshold_with_alerts are used to ensure a faster response.

The message created by the C&U Coordinator specifies a counter type to retrieve ("realtime" or "hourly"), and an optional time range to collect data for.

The data collection phase of C&U processing is split into two parts: capture, and initial processing and storage, both performed by the C&U Data Collector.

... MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#perf_capture) ⏎
[realtime] Capture for ⏎
ManageIQ::Providers::Vmware::InfraManager::Vm name: [VP23911], ⏎
id: [1000000000789]...Complete - Timings: ⏎
{:capture_state=>0.08141517639160156, ⏎
:vim_connect=>0.06982016563415527, ⏎
:capture_intervals=>0.014894962310791016, ⏎
:capture_counters=>0.20465683937072754, ⏎
:build_query_params=>0.0009250640869140625, ⏎
:num_vim_queries=>1, ⏎
:vim_execute_time=>0.2935605049133301, ⏎
:perf_processing=>0.10299563407897949, ⏎
:num_vim_trips=>1, ⏎
:total_time=>0.7732744216918945}

... MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm#perf_process) ⏎
[realtime] Processing for ⏎
ManageIQ::Providers::Vmware::InfraManager::Vm name: [VR11357], ⏎
id: [1000000000043], ⏎
for range [2017-01-25T05:59:00Z - 2017-01-25T06:50:20Z]... ⏎
Complete - Timings:  ⏎
{:process_counter_values=>0.019364118576049805, ⏎
:db_find_prev_perfs=>0.015059232711791992, ⏎
:process_perfs=>0.2053236961364746, ⏎
:process_perfs_db=>1.8572983741760254, ⏎
:total_time=>2.1722793579101562}

The C&U Data Processors periodically perform the task of 'rolling up' the realtime data. Rollups are performed hourly and daily, and counters for more granular objects such as virtual machines are aggregated into the counters for their parent objects. For example for a virtual infrastructure such as VMware or Red Hat Virtualization, the parent rollup process would include the following objects:

VM {hourly,daily} → Host {realtime,hourly,daily} → Cluster {hourly,daily} → Provider {hourly,daily} → Region {hourly,daily} → Enterprise

Rollup data is stored in the metrics_rollups table and in one of 12 sub-tables called metric_rollups_01 to metric_rollups_12 (each table corresponds to a month).

Additional analysis is performed on the hourly rollup data to identify bottlenecks, calculate chargeback metrics, and determine normal operating range and right-size recommendations. The completion of a successful rollup is written to evm.log, as follows:

... INFO -- : MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm# ⏎
perf_rollup) [hourly] Rollup for ManageIQ::Providers::Vmware:: ⏎
InfraManager::Vm name: [ranj001], id: [1000000000752] for time: ⏎
[2016-12-13T02:00:00Z]...Complete - Timings: ⏎
{:server_dequeue=>0.0035326480865478516, ⏎
:db_find_prev_perf=>3.514737129211426, ⏎
:rollup_perfs=>27.559985399246216, ⏎
:db_update_perf=>7.901974678039551, ⏎
:process_perfs_tag=>1.1872785091400146, ⏎
:process_bottleneck=>2.1828694343566895, ⏎
:total_time=>54.16198229789734}

With a very large number of managed objects the C&U Coordinator becomes unable to create and queue all of the required perf_capture_realtime messages within its own message timeout period of 600 seconds. An indeterminate number of managed objects will have no collections scheduled for that time interval. An extraction of lines from evm.log that illustrates the problem is as follows:

... INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) ⏎
Message id: [10000221979280], MiqWorker id: [10000001075231], ⏎
Zone: [OCP], Role: [ems_metrics_coordinator], Server: [], ⏎
Ident: [generic], Target id: [], Instance id: [], Task id: [], ⏎
Command: [Metric::Capture.perf_capture_timer], Timeout: [600], ⏎
Priority: [20], State: [dequeue], Deliver On: [], Data: [], ⏎
Args: [], Dequeued in: [2.425676767] seconds

... INFO -- : MIQ(Metric::Capture.perf_capture_timer) Queueing ⏎
performance capture...

... INFO -- : MIQ(MiqQueue.put) Message id: [10000221979391],  ⏎
id: [], Zone: [OCP], Role: [ems_metrics_collector], Server: [], ⏎
Ident: [openshift_enterprise], Target id: [], ⏎
Instance id: [10000000000113], Task id: [], ⏎
Command: [ManageIQ::Providers::Kubernetes::ContainerManager:: ⏎
ContainerNode.perf_capture_realtime], Timeout: [600], ⏎
Priority: [100], State: [ready], Deliver On: [], Data: [], ⏎
Args: [2017-03-23 20:59:00 UTC, 2017-03-24 18:33:23 UTC]

...

... INFO -- : MIQ(MiqQueue.put) Message id: [10000221990773],  ⏎
id: [], Zone: [OCP], Role: [ems_metrics_collector], Server: [], ⏎
Ident: [openshift_enterprise], Target id: [], ⏎
Instance id: [10000000032703], Task id: [], ⏎
Command: [ManageIQ::Providers::Kubernetes::ContainerManager:: ⏎
ContainerGroup.perf_capture_realtime], Timeout: [600], ⏎
Priority: [100], State: [ready], Deliver On: [], Data: [], ⏎
Args: [2017-03-24 18:10:20 UTC, 2017-03-24 18:43:15 UTC]

... ERROR -- : MIQ(MiqQueue#deliver) Message id: [10000221979280], ⏎
timed out after 600.002976954 seconds.  Timeout threshold [600]

Such problems can be detected by looking for message timeouts in the log using a command such as the following:

egrep "Message id: \[\d+\], timed out after" evm.log

Any lines matched by this search can be traced back using the PID field in the log line to determine the operation that was in process when the message timeout occurred.

Some providers keep realtime performance data for a limited time period, and if not retrieved in that time period, it is lost. For example VMware ESXi servers sample performance counter instances for themselves and the virtual machines running on them every 20 seconds, and maintain 180 realtime instance data points for a rolling 60 minute period. Similarly the OpenStack Gnocchi 'low' and 'high' archive policies on OSP 10+ only retain the finest granularity collection points for one hour (although this is configurable). There is therefore a 60 minute window during which performance information for each VMware or OpenStack element must be collected. If the performance data samples are not collected before that rolling 60 minutes is up, the data is lost.

The C&U Coordinator schedules a new VM, host or cluster realtime performance collection 50 minutes after the last data sample was collected for that object. This allows up to 10 minutes for the message to be dequeued and processed, before the realtime metrics are captured. In a large VMware or OpenStack environment the messages for the C&U Data Collectors can take longer than 10 minutes to be dequeued, meaning that some realtime data samples are lost. As the environment grows (more VMs) the problem slowly becomes worse. 

There are several types of log line written to evm.log that can indicate C&U data collection problems.

...  INFO -- : MIQ(Metric::Capture.perf_capture_health_check) ⏎
520 "realtime" captures on the queue for zone [VMware Zone] - ⏎
oldest: [2016-12-13T07:14:44Z], recent: [2016-12-13T08:02:32Z]
... INFO -- : MIQ(Metric::Capture.perf_capture_health_check) ⏎
77 "hourly" captures on the queue for zone [VMware Zone] - ⏎
oldest: [2016-12-13T08:02:15Z], recent: [2016-12-13T08:02:17Z]
... INFO -- : MIQ(Metric::Capture.perf_capture_health_check) ⏎
0 "historical" captures on the queue for zone [VMware Zone]

... INFO -- : MIQ(ManageIQ::Providers::Vmware::InfraManager:: ⏎
MetricsCollectorWorker::Runner#get_message_via_drb) ⏎
Message id: [1000032258093], MiqWorker id: [1000000120960], ⏎
Zone: [VMware], Role: [ems_metrics_collector], Server: [], ⏎
Ident: [vmware], Target id: [], Instance id: [1000000000060], ⏎
Task id: [], Command: [ManageIQ::Providers::Vmware::InfraManager:: ⏎
Vm.perf_capture_realtime], Timeout: [600], Priority: [100], ⏎
State: [dequeue], Deliver On: [], Data: [], Args: [], ⏎
Dequeued in: [789.95923544] seconds

... WARN -- : MIQ(ManageIQ::Providers::Vmware::InfraManager::HostEsx ⏎
#perf_capture) [realtime] For ManageIQ::Providers::Vmware:: ⏎
InfraManager::HostEsx name: [esx04], id: [1000000000023], ⏎
expected to get data as of [2016-12-13T01:20:00Z], ⏎
but got data as of [2016-12-13T02:00:20Z].

...  INFO -- : MIQ(ManageIQ::Providers::Vmware::InfraManager::Vm# ⏎
perf_process) [realtime] Processing 138 performance rows...Complete ⏎
- Added 138 / Updated 0

6.5.2.5. Unresponsive Provider

In some cases the CloudForms processes are working as expected, but the provider EMS is overloaded and not responding to API requests. To determine the relative EMS connection and query times for a VMware provider, the ':vim_connect' and ':vim_execute_time' timing counters from evm.log can be plotted. For this example the perf_process_timings.rb script can be used, as follows:

ruby ~/git/cfme-log-parsing/perf_process_timings.rb ⏎
-i evm.log -o perf_process_timings.out

egrep -A 22 "Worker PID:\s+10563" perf_process_timings.out | ⏎
grep vim_connect | awk '{print $2}' > vim_connect_times.txt

egrep -A 22 "Worker PID:\s+10563" perf_process_timings.out | ⏎
grep vim_execute_time | awk '{print $2}' > vim_execute_times.txt

The contents of the two text files can then be plotted, as shown in Figure 6.1, “VMware Provider C&U Connect and Execute Timings, Single Worker, 24 Hour Period”.

Figure 6.1. VMware Provider C&U Connect and Execute Timings, Single Worker, 24 Hour Period

​

In this example the stacked lines show a consistent connect time, and an execute time that is slightly fluctuating but still within acceptable bounds for reliable data collection.

The rollup and associated bottleneck and performance processing of the C&U data is less time sensitive, although must still be completed in the 4 hour realtime performance data retention period.

With a very large number of managed objects and insufficient worker processes, the time taken to process the realtime data can exceed the 4 hour period, meaning that that data is lost. The time taken to process the hourly rollups can exceed an hour, and the rollup process never keeps up with the rate of messages.

The count of messages queued for processing by the Data Processor can be extracted from evm.log, as follows:

grep 'count for state=\["ready"\]' evm.log | ⏎
egrep -o "\"ems_metrics_processor\"=>[[:digit:]]+"

"ems_metrics_processor"=>16612
"ems_metrics_processor"=>16494
"ems_metrics_processor"=>12073
"ems_metrics_processor"=>12448
"ems_metrics_processor"=>13015
...

The "Dequeued in" and "Delivered in" times for messages processed by the MiqEmsMetricsProcessorWorkers can be used as guidelines for overall throughput, for example:

... INFO -- : MIQ(MiqEmsMetricsProcessorWorker::Runner# ⏎
get_message_via_drb) Message id: [1000032171247], MiqWorker id: ⏎
[1000000253077], Zone: [VMware], Role: [ems_metrics_processor], ⏎
Server: [], Ident: [ems_metrics_processor], Target id: [], ⏎
Instance id: [1000000001228], Task id: [], ⏎
Command: [ManageIQ::Providers::Vmware::InfraManager::Vm.perf_rollup], ⏎
Timeout: [1800], Priority: [100], State: [dequeue], ⏎
Deliver On: [2016-12-13 03:00:00 UTC], Data: [], ⏎
Args: ["2016-12-13T02:00:00Z", "hourly"], ⏎
Dequeued in: [243.967960013] seconds

... INFO -- : MIQ(MiqQueue#delivered) Message id: [1000032171247], ⏎
State: [ok], ⏎
Delivered in [0.202901147] seconds

When C&U is operating correctly, for each time-profile instance there should be one daily record and at least 24 hourly records for each powered-on VM. There should also be at most 5 of the metrics_## tables that contain more than zero records. 

The following SQL query can be used to confirm that the records are being processed correctly:

select resource_id, date_trunc('day',timestamp) as collect_date, ⏎
resource_type, capture_interval_name, count(*)
from metric_rollups
where resource_type like '%Vm%'
group by resource_id, collect_date, resource_type, capture_interval_name
order by resource_id, collect_date, resource_type, capture_interval_name, count
;
 ..._id | collect_date        | resource_type | capture_int... | count
--------+---------------------+---------------+----------------+-------
...
      4 | 2017-03-17 00:00:00 | VmOrTemplate  | daily          |     1
      4 | 2017-03-17 00:00:00 | VmOrTemplate  | hourly         |    24
      4 | 2017-03-18 00:00:00 | VmOrTemplate  | daily          |     1
      4 | 2017-03-18 00:00:00 | VmOrTemplate  | hourly         |    24
      4 | 2017-03-19 00:00:00 | VmOrTemplate  | daily          |     1
      4 | 2017-03-19 00:00:00 | VmOrTemplate  | hourly         |    24
      4 | 2017-03-20 00:00:00 | VmOrTemplate  | daily          |     1
      4 | 2017-03-20 00:00:00 | VmOrTemplate  | hourly         |    24
...

Messages for the ems_metrics_coordinator (C&U coordinator) server role are processed by a Generic or Priority worker. These workers also process automation messages, which are often long-running. For larger CloudForms installations it can be beneficial to separate the C&U Coordinator and Automation Engine server roles onto different CFME appliances.

The metrics_00 to metrics_23 VMDB tables have a high rate of insertions and deletions, and benefit from regular reindexing. The database maintenance scripts that can be installed from appliance_console run a /usr/bin/hourly_reindex_metrics_tables script that reindexes one of the tables every hour.

If realtime data samples are regularly being lost, there are two remedial measures that can be taken.

:performance:
  :capture_threshold:
    :ems_cluster: 50.minutes
    :host: 50.minutes
    :storage: 60.minutes
    :vm: 50.minutes

If C&U data processing is taking too long to process the rollups for all objects, the number of C&U Data Processor workers can be increased from the default of 2 up to a maximum of 4 per appliance. As before, consideration should be given to the additional CPU and memory requirements that an increased number of workers will place on an appliance. Adding further CFME appliances to the zone may be more appropriate.

For larger CloudForms installations it can be beneficial to separate the C&U Data Processor and Automation Engine server roles onto different CFME appliances, as both are resource intensive. CloudForms installations managing several thousand objects may benefit from dedicated CFME appliances in the provider zones exclusively running the C&U Data Processor role.

There are some zone-related implications that should be considered when running automation tasks in a large multi-zone CloudForms deployment.

By default each CFME appliance has 2 Priority and 2 Generic workers. If both of the Generic workers are busy processing long-running automate tasks or high priority messages (such as from an event storm), no further priority 100 automate messages will be dequeued and processed until either of the workers completes their current task. This is often observed in larger deployments when service or automation requests appear to remain in a "Pending" state for a long time.

The default message timeout for automate messages is 600 seconds, which means that the combined execution times of all automate methods that share a common $evm.root object must be less than 10 minutes. If this time limit is exceeded the automate method will be deemed "non responsive" and terminated, and the Generic worker running the automation will exit and be re-spawned. This timer is only reset if a state machine method exits with $evm.root['ae_result'] = 'retry'.

The timeout mechanism can be observed in evm.log, as follows:

... ERROR -- : <AutomationEngine> Terminating non responsive method ⏎
with pid 29188

... ERROR -- : <AutomationEngine> <AEMethod test> The following error ⏎
occurred during method evaluation:

... ERROR -- : <AutomationEngine> <AEMethod test> SignalException: ⏎
SIGTERM

... ERROR -- : MIQ(MiqQueue#deliver) Message id: [1054092], timed ⏎
out after 600.03190583 seconds.  Timeout threshold [600]

... INFO -- : MIQ(MiqQueue#delivered) Message id: [1054092], ⏎
State: [timeout], Delivered in [600.047235602] seconds

... ERROR -- : MIQ(MiqGenericWorker::Runner) ID [3758] PID [3149] ⏎
GUID [d8bbe584-0e0f-11e7-a1a8-001a4aa0151a] ⏎
Exiting worker due to timeout error Worker exiting.

If the number of retries attempted by a state machine state reaches the limit defined in the class schema, an error will be logged to evm.log, as follows:

... ERROR -- : Q-task_id([automation_task_13921]) State=<pre4> running  ⏎
raised exception: <number of retries <6> exceeded maximum of <5>>

The number of Priority workers per CFME appliance can be increased up to a maximum of 4, and Generic workers up to a maximum of 9. This will increase the concurrency at which automate messages can be processed, however worker count should only be increased after consideration of the additional CPU and memory requirements that an increased number of workers will place on an appliance.

For larger CloudForms installations it can be beneficial to separate any of the Capacity and Utilization, and the Automation Engine server roles onto different CFME appliances, as both are resource intensive. In very large CloudForms installations it can be beneficial to have dedicated appliances per zone with the Automation Engine role enabled, each with the maximum numbers of Generic and Priority workers.

There are two useful techniques that can be used to help keep the overall execution time of custom Ruby-based automation workflows within the 10 minute timeout period. The first is to use state machines as much as possible to model workflows, and to include CheckCompleted states after any asynchronous and potentially long-running operation. The CheckCompleted state methods check for completion of the prior state, and issue an ae_result="retry" if the operation is incomplete.

The second is to use $evm.execute('create_automation_request',…​) rather than $evm.instantiate to execute long-running instances. Using $evm.instantiate to start another instance from a currently running method will execute the called instance synchronously. The calling method will wait until the instantiated instance completes before continuing. If the instantiated method integrates with an external system for example, this delay might be significant, and contributes towards the total message processing time.

The use of these two techniques can be illustrated with the following example. In this case a call is made using $evm.instantiate to run an instance update_cmdb that updates the IP address for a virtual machine in an external CMDB, but the external API call to the CMDB sometimes takes several minutes to complete. The existing in-line call is as follows:

$evm.instantiate("/Integration/Methods/update_cmdb?name=dbsrv01& ⏎
  ip=10.1.2.3")

To run the update_cmdb instance asynchronously, the call can be rewritten to run as a new automation request, for example:

options = {}
options[:namespace]     = 'Integration'
options[:class_name]    = 'Methods'
options[:instance_name] = 'update_cmdb'
options[:user_id]       = $evm.root['user'].id
options[:attrs]         = {
                          'name' => 'dbsrv01',
                          'ip'   => '10.1.2.3'
                          }
auto_approve            = true

update_cmdb_request = $evm.execute('create_automation_request', ⏎
  options, 'admin', auto_approve)

If the calling method does not need to wait for the completion of update_cmdb then processing can continue, and minimal delay has been incurred. If update_cmdb should complete before the main processing can continue, the request ID can be saved, and a 'CheckCompleted' state added to the state machine, as follows:

update_cmdb_request = $evm.execute('create_automation_request', ⏎
  options, 'admin', auto_approve)
$evm.set_state_var(:request_id, update_cmdb_request.id)
$evm.root['ae_result'] = 'ok'
exit MIQ_OK

The following state in the state machine would be check_cmdb_request, containing code similar to the following:

update_cmdb_request =
  $evm.vmdb(:miq_request, $evm.get_state_var(:request_id))
case update_cmdb_request.state
when "pending", "active"
  $evm.log(:info, "Request still active, waiting for 30 seconds...")
  $evm.root['ae_retry_interval'] = '30.seconds'
  $evm.root['ae_result']         = 'retry'
when "finished"
  $evm.log(:info, "Request complete!")
  $evm.root['ae_result'] = 'ok'
else
  $evm.log(:warn, "Unexpected request status")
  $evm.root['ae_result'] = 'error'
end
exit MIQ_OK

Sometimes the called method needs to pass data back to the caller, and this can be returned via the request object’s options hash. The called method update_cmdb can retrieve its own request object and use the set_option method to encode a key/value pair (where the value is a JSON-encoded hash) as follows:

request = $evm.root['automation_task'].automation_request
request.set_option(:return, JSON.generate({:status => 'success',
                   :cmdb_return => 'update successful'}))

The options hash can be read from the request object by the caller using the get_option method, as follows:

update_cmdb_request =
  $evm.vmdb(:miq_request, $evm.get_state_var(:request_id))
returned_data = update_cmdb_request.get_option(:return)

Executing long-running tasks asynchronously in this way using a state machine retry loop to check for completion, is an efficient way of reducing overall processing time, and increasing concurrency and throughput of automate operations.

The default behaviour of services and API requests with regard to zones may not necessarily be suitable for all cases.

attrs = {}
attrs['dialog_stack_name'] = $evm.root['dialog_stack_name']
attrs['dialog_password']   = $evm.root['dialog_password']
options = {}
options[:namespace]     = 'Service/Provisioning/StateMachines'
options[:class_name]    = 'ServiceProvision_Template'
options[:instance_name] = 'create_stack'
options[:user_id]       = $evm.vmdb(:user).find_by_userid('admin').id
options[:miq_zone]      = 'Generic'
options[:attrs]         = attrs
auto_approve            = true
$evm.execute('create_automation_request', options, 'admin', ⏎
  auto_approve)

  :requester => {
    :auto_approve => true
  },
  :parameters => {
     :miq_zone => 'Zone Name'
  }

The default automate datastore state machine has the fields shown in Figure 8.1, “VM Provision State Machine”:

Figure 8.1. VM Provision State Machine

​

As can be seen, many of the fields are empty "placeholder" states such as AcquireIPAddress that can be used to extend the functionality of the state machine and integrate the workflow with the wider enterprise.

The internal state machine is a nested state machine that is launched asynchronously at the Provision state of the VM provision state machine. The subsequent CheckProvisioned state of the VM provision state machine performs a check-and-retry loop until the internal state machine completes.

Internal provision state machines are provider-specific, and are not exposed to automate (they are not designed to be user-customizable). They perform the granular steps of creating the virtual machine; communicating with the EMS using its native API, and customizing the VM using the parameters defined in the provisioning options hash. A typical set of internal state machine steps to provision a VMware virtual machine are as follows:

The final state of the internal state machine marks the provision task object as having a state of provisioned. The outer VM provision state machine CheckProvisioned state polls for this status, and continues to its own PostProvision state when detected.

As mentioned in Chapter 7, Automate, the message to initiate a VM provisioning workflow has a timeout value of 600 seconds. The VM provision state machine therefore has a maximum time of 10 minutes to execute down to the first retry stage, which is CheckProvisioned.

8.2.1.1. External Integration

In larger CloudForms deployments it is common to add enterprise integration to the VM provisioning workflow. Custom instances are often added to the placeholder fields such as AcquireIPAddress to retrieve an IP address from a corporate IP Address Management (IPAM) solution, for example. If the methods run by these stages take minutes to run under high load, the state machine may timeout before the CheckProvisioned state is reached.

To reduce this possibility the VM provision state machine can be expanded to include check-and-retry states after the custom methods, such as the CheckIPAddressAcquired state in Figure 8.2, “Modified VM Provision State Machine”.

Figure 8.2. Modified VM Provision State Machine

​

$evm.root['ae_retry_interval'] = '1.minute'

It can sometimes be beneficial in large virtual environments to create a separate provisioning or incubation CloudForms region that manages a small sub-set of the overall infrastructure. This can be used to provision new virtual machines, which can then be migrated to the production data centers or clusters once they are patched and ready for use.

The polling frequency of each of the provider-specific event catchers is defined in the :event_catcher section of the Configuration→Advanced settings. The default settings for CloudForms Management Engine 5.8 are as follows:

    :event_catcher:
        :poll: 1.seconds
      :event_catcher_ansible_tower:
        :poll: 20.seconds
      :event_catcher_embedded_ansible:
        :poll: 20.seconds
      :event_catcher_redhat:
        :poll: 15.seconds
      :event_catcher_openstack:
        :poll: 15.seconds
      :event_catcher_openstack_infra:
        :poll: 15.seconds
      :event_catcher_openstack_network:
        :poll: 15.seconds
      :event_catcher_hawkular:
        :poll: 10.seconds
      :event_catcher_hawkular_datawarehouse:
        :poll: 1.minute
      :event_catcher_google:
        :poll: 15.seconds
      :event_catcher_kubernetes:
        :poll: 1.seconds
      :event_catcher_lenovo:
        :poll: 4.minutes
      :event_catcher_openshift:
        :poll: 1.seconds
      :event_catcher_cinder:
        :poll: 10.seconds
      :event_catcher_swift:
        :poll: 10.seconds
      :event_catcher_amazon:
        :poll: 15.seconds
      :event_catcher_azure:
        :poll: 15.seconds
      :event_catcher_vmware:
        :poll: 1.seconds
      :event_catcher_vmware_cloud:
        :poll: 15.seconds

VMware vCenter management systems use an event type called EventEx as a catch-all event. Several VMware components issue EventEx events with a subtype to record state changes, problems, and recovery from problems. They appear as [EventEx]-[subtype], for example: 

Until the cause of the event storm is identified and corrected, the quickest way to restore any operation for the CloudForms environment is to to prevent the continued growth of the miq_queue table. The simplest techniques are to blacklist the event(s) causing the storm (see Section 9.4.1, “Blacklisting Events”), or to disable the event monitor role on all CFME appliance in the provider’s zone. 

In critical situations with many hundreds of thousands to millions of queued messages, it may be necessary to selectively delete message instances from the miq_queue table. Since the overwhelming number of messages expected to be in this table will be of type 'event', the following SQL statement can be used to remove all such instances from the miq_queue table:

delete from miq_queue where role = 'event' and class_name = 'EmsEvent';

Before running this query the following points should be noted:

Some provider events occur relatively frequently, but are either uninteresting to CloudForms, or processing them would consume excessive resources (such as those typically associated with event storms). Events such as these can be skipped or blacklisted. The event catchers write a list of blacklisted events to evm.log when they start, for example:

... MIQ(ManageIQ::Providers::Redhat::InfraManager::EventCatcher:: ⏎
Runner#after_initialize) EMS [rhevm.bit63.net] as [cfme@internal] ⏎
Event Catcher skipping the following events:
... INFO -- :   - UNASSIGNED
... INFO -- :   - USER_REMOVE_VG
... INFO -- :   - USER_REMOVE_VG_FAILED
... INFO -- :   - USER_VDC_LOGIN
... INFO -- :   - USER_VDC_LOGIN_FAILED
... INFO -- :   - USER_VDC_LOGOUT

These events are defined in the blacklisted_events table in the VMDB. The default rows in the table are as follows:

vmdb_production=# select event_name,provider_model ⏎
from blacklisted_events;
               event_name               |    provider_model
----------------------------------------+------------------------------
 storageAccounts_listKeys_BeginRequest  | ...Azure::CloudManager
 storageAccounts_listKeys_EndRequest    | ...Azure::CloudManager
 identity.authenticate                  | ...Openstack::CloudManager
 scheduler.run_instance.start           | ...Openstack::CloudManager
 scheduler.run_instance.scheduled       | ...Openstack::CloudManager
 scheduler.run_instance.end             | ...Openstack::CloudManager
 ConfigurationSnapshotDeliveryCompleted | ...Amazon::CloudManager
 ConfigurationSnapshotDeliveryStarted   | ...Amazon::CloudManager
 ConfigurationSnapshotDeliveryFailed    | ...Amazon::CloudManager
 UNASSIGNED                             | ...Redhat::InfraManager
 USER_REMOVE_VG                         | ...Redhat::InfraManager
 USER_REMOVE_VG_FAILED                  | ...Redhat::InfraManager
 USER_VDC_LOGIN                         | ...Redhat::InfraManager
 USER_VDC_LOGOUT                        | ...Redhat::InfraManager
 USER_VDC_LOGIN_FAILED                  | ...Redhat::InfraManager
 AlarmActionTriggeredEvent              | ...Vmware::InfraManager
 AlarmCreatedEvent                      | ...Vmware::InfraManager
 AlarmEmailCompletedEvent               | ...Vmware::InfraManager
 AlarmEmailFailedEvent                  | ...Vmware::InfraManager
 AlarmReconfiguredEvent                 | ...Vmware::InfraManager
 AlarmRemovedEvent                      | ...Vmware::InfraManager
 AlarmScriptCompleteEvent               | ...Vmware::InfraManager
 AlarmScriptFailedEvent                 | ...Vmware::InfraManager
 AlarmSnmpCompletedEvent                | ...Vmware::InfraManager
 AlarmSnmpFailedEvent                   | ...Vmware::InfraManager
 AlarmStatusChangedEvent                | ...Vmware::InfraManager
 AlreadyAuthenticatedSessionEvent       | ...Vmware::InfraManager
 EventEx                                | ...Vmware::InfraManager
 UserLoginSessionEvent                  | ...Vmware::InfraManager
 UserLogoutSessionEvent                 | ...Vmware::InfraManager
 identity.authenticate                  | ...Openstack::InfraManager
 scheduler.run_instance.start           | ...Openstack::NetworkManager
 scheduler.run_instance.scheduled       | ...Openstack::NetworkManager
 scheduler.run_instance.end             | ...Openstack::NetworkManager
 ConfigurationSnapshotDeliveryCompleted | ...Amazon::NetworkManager
 ConfigurationSnapshotDeliveryStarted   | ...Amazon::NetworkManager
 ConfigurationSnapshotDeliveryFailed    | ...Amazon::NetworkManager
(37 rows)

If processing of any of the events in the blacklisted_events table is required, the enabled field can be set to false and the provider-specific event catcher restarted.

An EMS can also report some minor object property changes as events, even though these are not modelled in the CloudForms VMDB. For VMware providers such event types can be added to the "Vim Broker Exclude List" so that they can be discarded without processing. The exclude list is found under :broker_notify_properties in the Configuration → Advanced settings, as follows:

:broker_notify_properties:
  :exclude:
    :HostSystem:
    - config.consoleReservation
    - config.dateTimeInfo
    - config.network
    - config.service
    - summary
    - summary.overallStatus
    - summary.runtime.bootTime
    - summary.runtime.healthSystemRuntime.systemHealthInfo. ⏎
         numericSensorInfo
    :VirtualMachine:
    - config.locationId
    - config.memoryAllocation.overheadLimit
    - config.npivWorldWideNameType
    - guest.disk
    - guest.guestFamily
    - guest.guestFullName
    - guest.guestId
    - guest.ipStack
    - guest.net
    - guest.screen
    - guest.screen.height
    - guest.screen.width
    - guest.toolsRunningStatus
    - guest.toolsStatus
    - resourceConfig
    - summary
    - summary.guest.guestFullName
    - summary.guest.guestId
    - summary.guest.toolsRunningStatus
    - summary.overallStatus
    - summary.runtime.bootTime
    - summary.runtime.memoryOverhead
    - summary.runtime.numMksConnections
    - summary.storage
    - summary.storage.committed
    - summary.storage.unshared

CloudForms recently introduced the concept of flood monitoring for the provider-specific event catchers. This stops provider events from being queued when too many duplicates are received in a short time. By default an event is considered as flooding if it is received 30 times in one minute.

Flood monitoring is a generic concept for event processing, but requires the appropriate supporting methods to be added to each provider. As of CloudForms Management Engine 5.8 only the VMware provider supports this functionality.

The :event_catcher section is one of the largest of the Configuration → Advanced settings, and it defines the configuration of each type of event catcher. For example the following extract shows the settings for the ManageIQ::Providers::Openstack::InfraManager::EventCatcher worker:

    :event_catcher:
...
      :event_catcher_openstack:
        :poll: 15.seconds
        :topics:
          :nova: notifications.*
          :cinder: notifications.*
          :glance: notifications.*
          :heat: notifications.*
        :duration: 10.seconds
        :capacity: 50
        :amqp_port: 5672
        :amqp_heartbeat: 30
        :amqp_recovery_attempts: 4
        :ceilometer:
          :event_types_regex: "\\A(?!firewall|floatingip|gateway| ⏎
          net|port|router|subnet|security_group|vpn)"
...

The configuration settings rarely need to be changed from their defaults.

The MiqSmartProxyWorker processes scan VMware virtual machines using the VixDiskLib API functionality provided by the VMware Virtual Disk Development Kit (VDDK). Any CFME appliance in the provider’s zone that is running the SmartProxy role must therefore have the VDDK installed.^[19].

:coresident_miqproxy:
...
  :scan_via_host: false

For SmartState Analysis of Red Hat Virtualization (RHV) virtual machines to complete successfully, the CFME appliances running the SmartProxy server roles must be in the same RHV datacenter as the VM being scanned. The storage domains much also be accessible to the SmartProxy appliances. Fibre channel or iSCSI storage domains should be presented to each SmartProxy appliance as shareable direct LUNs. NFS datastores must be mountable by each SmartProxy appliance, which may mean adding secondary network interfaces to the CFME appliances, connected to the storage network.

The management engine relationship must also be set for each CFME appliance. This enables the VM SmartState Analysis job to determine the datacenter where the CFME appliance is running and thus to identify which storage it has access to.

CloudForms is capable of performing a SmartState Analysis of both Overcloud images, and Undercloud Nova compute nodes.

 10) Extend Temporary Storage

SmartState scanning of OpenShift containers is performed by an image_inspector pod that is pulled from the Red Hat registry as required. The image inspector dynamically downloads and uses the latest OpenScap definition/rules file from Red Hat before scanning.^[20]

With CloudForms 4.5 the registry and repository are configurable in Configuration → Advanced settings, as follows:

:ems_kubernetes:
...
  :image_inspector_registry: registry.access.redhat.com
  :image_inspector_repository: openshift3/image-inspector

A virtual machine SmartState Analysis is always performed on a temporary snapshot of the VM. The snaphot is taken using the native means exposed by the EMS, however most snapshotting technology does not take into account the requirements of any application running in the virtual machine. Taking a virtual machine snapshot can have unintended and unexpected consequences for some applications that maintain state data such as Microsoft Exchange Server.^[21].

Virtual machines running such applications must not be snapshotted, and should therefore be excluded from SmartState Analysis.

A control policy can be created to prevent SmartState Analysis from running on any VM tagged with "exclusions/do_not_analyze", as shown in Figure 10.1, “Control Policy to Block SmartState Analysis”.

Figure 10.1. Control Policy to Block SmartState Analysis

​

Virtual machines running stateful workloads can be tagged accordingly to prevent the snapshot from being taken.

Problems with SmartState Analysis are logged to evm.log, and can be identified using the following bash command:

grep 'VmScan#process_abort' evm.log

Many of the most common errors are caused as a result of scaling parts of the infrastructure - hosts or CFME appliances - and forgetting to update the provider-specific considerations for SmartState Analysis.

... MIQ(VmScan#process_abort) job aborting, No eligible proxies for VM ⏎
:[[NFS_PROD] odrsrv001/odrsrv001.vmx] - [No active SmartProxies found ⏎
to analyze this VM], aborting job [8064001a-e2ea-11e6-9140-005056b19b0f].

... MIQ(VmScan#process_abort) job aborting, No eligible proxies for VM ⏎
:[[FCP_MID] osdweb01/osdweb01.vmx] - [Provide credentials for this VM's ⏎
Host to perform SmartState Analysis], aborting job ⏎
[d2e08e70-c26b-11e6-aaa4-00505695be62].

... MIQ(VmScan#process_abort) job aborting, Unable to mount filesystem. ⏎
Reason:[mount.nfs: Connection timed out

The default number of "VM Analysis Collector" (MiqSmartProxyWorker) workers per appliance is 3. This can be increased to a maximum of 5, although consideration should be given to the additional CPU and memory requirements that an increased number of workers will place on an appliance. It may be more appropriate to add further appliances and scale horizontally.

CloudForms installations managing several thousand objects may benefit from dedicated CFME appliances in the provider zones exclusively running the SmartState Analysis and SmartProxy roles.

Hosts and datastores can be can be 'pinned' to specific embedded SmartProxy servers using the SmartProxy Affinity setting in the Configuration → Settings → Zones area of the WebUI, as shown in Figure 10.2, “SmartProxy Affinity”:

Figure 10.2. SmartProxy Affinity

​

This can help ensure that only the most optimally placed or suitably configured CFME appliances are used for SmartState Analysis scans.

Multiple CFME appliances in a WebUI zone are often placed behind a load balancer. The load balancer should be configured with sticky sessions enabled, which will force it to send requests to the same UI worker during a session.

The load balancer should also be configured to test for connectivity using the CloudForms ping response page at https://cfme_appliance/ping. The expected reply from the appliance is the text string “pong”. Using this URL is preferable to the standard login URL as it does not establish a connection to the database.

By default the CloudForms UI workers store session data in the local appliance’s memcache. When operating behind a load balancer the UI workers should be configured to store session data in the database. This prevents a user from having to re-login if the load balancer redirects them to an alternative server if their original UI worker is unresponsive.

The location of the session store is defined in the Configuration → Advanced settings. The default value for session_store is as follows:

:server:
...
  :session_store: cache

This should be changed to:

  :session_store: sql

Red Hat Virtualization 4.0 is installed as the Engineering/R&D virtual infrastructure. It currently comprises 2 clusters, 20 hosts, 10 storage domains and approximately 500 virtual machines. The number of VMs is not expected to grow significantly over the next two years.

A Red Hat OpenStack Platform 10 private IaaS cloud is installed. This contains approximately 900 images and instances spread between 50 tenant/projects, and also hosts the OpenShift PaaS. An OpenStack Director manages the Undercloud, comprising 42 Nova compute nodes.

The number of Overcloud instances is forecast to grow by approximately 400 per year over the next two years, giving a projected total of around 1900 managed objects.

A recently acquired subsidiary uses Amazon EC2 for cloud workloads. There are approximately 250 EC2 instances used by two accounts (separate access key IDs), but this number is expected to gradually reduce over the next two years as work is migrated to the OpenStack IaaS. Ansible playbooks are frequently used to configure Amazon EC2 cloud components such as Elastic Load Balancers.

A Red Hat OpenShift Container Platform 3.4 PaaS is installed, hosted in OpenStack, and currently comprising approximately 100 nodes, 750 pods and 1000 containers. This number is expected to rise to 300 nodes, 2000 pods and 3500 containers over the next two years.

All in-house networking components are split between two campus datacenters. There is LAN-speed (<1MSec) latency between all points on this network. For security isolation the Engineering/R&D RHV, OpenStack and OpenShift environments are on separate vLANs, with only limited connectivity to the 'Enterprise' network.

Additional firewall routes into and out of the Enterprise network are possible, but require security change approval.

User workstations are connected to a 'Desktops' network, which has very limited access to servers in the Enterprise or Engineering/R&D networks. User who wish to access the CloudForms environment must connect to WebUI servers accessible from this Desktops network.

The Enterprise network hosts common components such as a Configuration Management Database (CMDB) and an IPAM solution, however strict security policies are in place that restrict access to these components from non-Enterprise networks.

Virtual machines provisioned into the Engineering/R&D RHV and OpenStack networks may require registration with one or more of these enterprise tools.

The following capabilities of CloudForms are required:

The rightsizing calculation process uses metrics gathered by C&U, so this must also be enabled for cloud providers.

Latency from worker appliance to VMDB should be LAN speed, around 1ms or less. This will dictate where the VMDB appliance should be situated, and also the optimum location of the worker CFME appliances. For this design network latency is good, so the VMDB server should be placed in the most centrally accessible location.

The optimum VMDB server for this design will be a CFME appliance configured as a standalone PostgreSQL server. Although database high availability (HA) has not been specified as an initial requirement, installing a standalone database appliance allows for HA to be configured in future if required.

The database server will be installed in the Enterprise network, hosted by the VMware 6.0 virtual infrastructure. The estimated size of the database after two years, based on the formula presented in Chapter 4, Database Sizing and Optimization is approximately 468 GBytes. To allow for unexpected growth and a margin of uncertainty, a 750 GByte disk will be presented from a datastore backed by fast FC SAN storage, and used as the database volume.

The database server will have 8 GBytes memory, and a PostgreSQL shared_buffers region of 2 GBytes. A 2 GByte hugepage region will be created for PostgreSQL to use.

The planned zone design contains 13 CFME appliances. Referring to the table in Appendix A, Database Appliance CPU Count shows that the database server will need 6 vCPUs to maintain an idle CPU load under 20%.

The database maintenance scripts will be enabled on the VMDB server.

A zone should be created per provider (unless the EMS only manages a small number of systems; around 100 VMs or so). There should be a minimum of 2 CFME appliances per zone for resilience, and zones should not span networks.

For this design scenario the following zones are proposed.

A WebUI zone will be created that contains 2 CFME appliances, each running the following server roles:

The CFME appliances in this zone will be hosted by the enterprise VMware 6.0 environment, in a vLAN accessible from user workstations. User access to them will be via a hardware load-balancer and common Fully-Qualified Domain Name.

A Management zone will be created that contains 2 CFME appliances, each running the following server roles:

The CFME appliances in this zone will be hosted by the enterprise VMware 6.0 environment. The zone will not contain any providers, but automate workflows that interact with the CMDB and IPAM solutions will run in this zone.

The RHV zone will contain approximately 600 managed objects. The table Table 3.2, “Objects per CFME Appliance Guidelines” suggests that 2 appliances should be sufficient, each running the following server roles:

The CFME appliances in this zone will be hosted by the RHV environment, and so firewall ports must be opened to allow these appliances to connect to the VMDB server in the Enterprise network. The RHV provider will be in this zone.

The OpenStack zone will initially contain approximately 900 instances (for example instances, images, tenants,or networks), increasing to 1700 in two years time. The table Table 3.2, “Objects per CFME Appliance Guidelines” suggests that 3 appliances should be sufficient initially, each running the following server roles:

The CFME appliances in this zone will be hosted by the OpenStack environment, and so firewall ports must be opened and routes created to allow these appliances to connect to the VMDB server in the Enterprise network, and to the OpenStack Director. Both OpenStack Cloud and Infrastructure Manager (Undercloud) providers will be in this zone.

Further appliances will need to be added to this zone as the number of managed objects increases.

The OpenShift zone will contain approximately 800 managed objects. The table Table 3.2, “Objects per CFME Appliance Guidelines” suggests that 2 appliances should be sufficient initially, each running the following server roles:

The CFME appliances in this zone will also be hosted by the OpenStack environment, and so firewall ports must be opened and routes created to allow these appliances to connect to the VMDB server in the Enterprise network, and to the OpenShift master. The OpenShift provider will be in this zone.

Further appliances will need to be added to this zone as the number of managed objects increases.

The Amazon zone will contain approximately 250 managed objects. The table Table 3.2, “Objects per CFME Appliance Guidelines” suggests that 1 appliance should be sufficient, however for resilience and load balancing 2 will be installed, each running the following server roles:

The CFME appliances in this zone will be hosted on a separate vLAN in the RHV environment, and so firewall ports must be opened to allow these appliances to connect to the VMDB server in the Enterprise network, and to the Amazon EC2 network. The Embedded Ansible role will be enabled on these CFME appliances so that Ansible playbooks can be run from service catalogs. The Amazon EC2 providers for both accounts will be in this zone.

The proposed zone design is shown in Figure 13.1, “Networks and Zones”.

Figure 13.1. Networks and Zones

​