How to open a proactive case for a planned activity on Red Hat OpenStack Platform ?

Solution Verified - Updated -

Environment

  • Red Hat OpenStack Platform (RHOSP)

Issue

  • Patching production OpenStack servers with OS patches and OpenStack patches
  • Minor/Major Upgrade.
  • Scale In/Out Activity.
  • How should I open a case to notify Red Hat of patches or upgrades taking place so appropriate resources are ready?
  • How do I open a proactive case for system upgrades?

Resolution

Introduction - what is a Proactive Case?

Proactive cases are a way for Premium customers to provide an advance notice to Red Hat Support of planned upgrades, migrations, patching, or any other maintenance activity. Typically, these events happen outside regular business hours.
Having prior knowledge of the planned activity provides Red Hat with an opportunity to ensure the right information is made available to better prepare technical support should you encounter an issue.

Note: New installations only entitle a Proactive Case for Severity 3/4 as per our Severity Level Definitions.

Note: If there are any issues faced during activity we request customers to always open a separate case which will be treated as child case whereas Proactive case will be used to track the activity. This helps to keep the track of issues isolated and also maintains health of the case. All child cases should have parent case.

Prerequisites

  • The account must have an active Premium OpenStack subscription.
  • It is recommended to submit Proactive Cases several days before the activity is scheduled to begin (preferably 7-14 days in advance), in order to allocate appropriate resources for the upcoming event.
    • If the activity requires weekend coverage, the Proactive Case must be submitted at least 4 business days before the upcoming activity (by close of the business day, for example on the Tuesday prior to the upcoming activity).
    • Note: Weekend Proactive Cases submitted less than 4 business days before the activity will be considered "Late Entry", and treated through normal Follow the Sun workflow. For such, Red Hat will not be able to guarantee that the appropriate resources would be staffed for the weekend.
  • In order to allocate the most appropriate engineer from different teams, no matter where they may be worldwide, Proactive Support Cases are conducted in the English language only.
  • It is recommended to familiarize yourself with the Reference Guide for Engaging with Red Hat Support article, which provides best practices for engaging Red Hat Support.
    • In addition, there is a Blog Post mentioning Best Practices when engaging with Red Hat.
  • If you have a Technical Account Manager (TAM), a Solution Architect (SA), or a Customer Success Manager (CSM) - please engage them when planning a Proactive Case.
  • Please ensure that proper maintenance of your OpenStack environment is performed on a regular basis before any upgrade (for example: token, cinder, nova and heat database cleanup jobs).

Opening a Proactive Support Case

Proactive means knowing about the bullet points in advance about any scheduled outage to react quickly in case assistance is needed. When opening a proactive Support Case , please, provide the following:

  • Case Title - should be prefixed with "[Proactive]" and should summarize the change.
  • Severity - open the case on a Severity level of 3.
  • Case Description - describe as many details as possible, for example: Environment and any third-party software that was integrated, applications/services that may be affected, other components being
    upgraded, patches being applied, etc.
  • Date / Time of Event - state the planned start and end date/time of the maintenance window duration with the timezone (preferably referencing the offset from UTC).
  • Contact Information - provide the primary and secondary contact name, email addresses and phone number with country and region code. This is especially important if the person who opened the case is
    not the person doing the maintenance activity.
  • Data Attachments - upload an SOS Report of the relevant system/s before the activity is performed (preferably as soon as you open the case).

    • If there are only a few systems affected by the activity, please attach their respective SOS Reports to the case. Otherwise, please provide one or two as an example and keep the rest locally.
    • Also, we recommend you review the Article specific to OpenStack Logs requirement.

    Pre and Post Checks to be shared while opening a Proactive Case

  • Collect the ouptut from any of the Controller nodes (though it's collected by sosreport but good to have directly on comment itself)

    pcs status 
    pcs property 
    
  • From Director

    source stackrc
    openstack stack list
    openstack overcloud status --stack $(openstack stack list -c "Stack Name" -f value)
    openstack stack resource list $(openstack stack list -c "Stack Name" -f value) |grep -iv complete
    
    source overcloudrc  [or anyother rc as per stack Name]
    openstack compute service list
    openstack volume service list
    openstack network agent list
    
  • Additionally it's highly recommended to take backups before any activity is performed on the OpenStack Setup. ReaR recommendations for Undercloud are provided RHOSP 16.1, RHOSP 16.2. Broader set choices is available for RHOSP 17.1

  • For Overcloud node you can run the below command to take the DB backup (run as root user on any one controller)

    GALERACONTAINER=$(podman ps -f name=galera --format={{.Names}})
    
    for i in $(podman exec  $GALERACONTAINER mysql -s -N -e "select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';");do podman exec $GALERACONTAINER /usr/bin/mysqldump -uroot --single-transaction --databases $i >> $(hostname)_openstack_db-$(date +"%Y-%m-%d").sql; done
    
    for i in $(podman exec $GALERACONTAINER mysql -s -N -e "SELECT CONCAT('''',user,'''@''',host,'''') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')"); do podman exec $GALERACONTAINER mysql -s -N -e "SHOW GRANTS FOR $i" >> $(hostname)_openstack_db_grants-$(date +"%Y-%m-%d").sql; done
    

    NOTE: Replace podman with docker for RHOSP13 in above command

  • Collect the file created by running script pre_checks.sh attached to the KCS.

Attachments

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments