How to open a proactive case for a planned activity on Red Hat OpenStack Platform ?

Resolution

Introduction - what is a Proactive Case?

Proactive cases are a way for Premium customers to provide an advance notice to Red Hat Support of planned upgrades, migrations, patching, or any other maintenance activity. Typically, these events happen outside regular business hours.
Having prior knowledge of the planned activity provides Red Hat with an opportunity to ensure the right information is made available to better prepare technical support should you encounter an issue, to proactively book developer's resources and ensure that enough engineers are online during weekend activities.

Note: New installations only entitle a Proactive Case for Severity 3/4 as per our Severity Level Definitions.

Note: If there are any issues faced during planned activity, Red Hat will request to open a separate case per issue. A separate case will be treated as a child case for original proactive case, while a proactive case will be used to track the activity. This helps to keep the track of issues isolated and also maintains the health of the case. All child cases should have a parent case.

Prerequisites

• The account must have an active Premium OpenStack subscription.
• Proactive Cases should be reported 10 working days before the activity is scheduled to begin, in order to allocate appropriate resources for the upcoming event. If reported later, then Red Hat will not be able to book resources properly, but Red Hat technical support will still complete some checks and do its best to properly support upcoming activity.
  • If the activity requires weekend coverage, the Proactive Case must be submitted at least 10 business days before the upcoming activity (by close of the business day, for example on the Tuesday prior to the upcoming activity).
  • Note: Weekend Proactive Cases submitted less than 10 business days before the activity will be considered "Late Entry", and treated through normal Follow the Sun workflow. For such, Red Hat will not be able to guarantee that the appropriate resources would be staffed for the weekend.
• In order to allocate the most appropriate engineer from different teams, no matter where they may be worldwide, Proactive Support Cases are conducted in the English language only.
• It is recommended to familiarize yourself with the Reference Guide for Engaging with Red Hat Support article, which provides best practices for engaging Red Hat Support.
  • In addition, there is a Blog Post mentioning Best Practices when engaging with Red Hat.
• If you have a Technical Account Manager (TAM), a Solution Architect (SA), or a Customer Success Manager (CSM) - please engage them when planning a Proactive Case.
• Please ensure that proper maintenance of your OpenStack environment is performed on a regular basis before any upgrade (for example: token, cinder, nova and heat database cleanup jobs).

Opening a Proactive Support Case

Proactive means knowing about the bullet points in advance about any scheduled outage to react quickly in case assistance is needed. When opening a proactive Support Case , please, provide the following:

• Case Title - should be prefixed with "[Proactive]" and should summarize the change.
• Severity - open the case on a Severity level of 3.
• Case Description - describe as many details as possible, for example: Environment and any third-party software that was integrated, applications/services that may be affected, other components being upgraded, patches being applied, etc.
• Date / Time of Event - state the planned start and end date/time of the maintenance window duration with the timezone (preferably referencing the offset from UTC).
• Contact Information - provide the primary and secondary contact name, email addresses and phone number with country and region code. This is especially important if the person who opened the case is
  not the person doing the maintenance activity.

• Data Attachments:

  • Documentation for upcoming activities (if exists): action plan, HLD/LLD documents, step by step procedures and so on.
  • Documentation explaining how other Red Hat products rely on RHOSP infrastructure or other way around (if exists). If not available, then a brief overview of the RHOSP environment, products used together with RHOSP, or products relying on RHOSP should be provided.
  • Environment history (which RHOSP version was deployed originally, which upgrade steps were taken afterwards).
  • [RHOSP 17.1 and prior] Deployment command and deployment templates.
  • [RHOSP 18+] Must gather for RHOSO with common openstack ... outputs for a cloud and optional DB dump.
  • Upload an SOS Report from all system/s affected by upcoming activity:
    • [RHOSP 17.1 and prior] Sosreport from director node
    • Sosreports from relevant overcloud/EDPM nodes in case of upgrade, scale of deployment activities. Ideally, but not a must is one sosreport per role/node type; sosreport from at least one controller for RHOSP 17.1 and prior is a must.
  • [RHOSP 17.1 and prior] Undercloud DB dump.
  • Also, we recommend you review the Article specific to OpenStack Logs requirement.

  Pre and Post Checks to be shared while opening a Proactive Case

• Collect the output from any of the Controller nodes (though it's collected by sosreport but good to have directly on comment itself)

  pcs status 
  pcs property 
  

• [RHOSP 17.1 and prior] The following outputs from Director (ephemeral heat process should be started for RHOSP 17.1)

  source stackrc
  openstack stack list
  openstack overcloud status --stack $(openstack stack list -c "Stack Name" -f value)
  openstack stack resource list $(openstack stack list -c "Stack Name" -f value) |grep -iv complete
  
  source overcloudrc  [or anyother rc as per stack Name]
  openstack compute service list
  openstack volume service list
  openstack network agent list
  

• Additionally it's highly recommended to take backups before any activity is performed on the OpenStack Setup. ReaR recommendations for Undercloud are provided RHOSP 16.1, RHOSP 16.2. Broader set choices is available for RHOSP 17.1

• For Overcloud node you can run the below command to take the DB backup (run as root user on any one controller)

  GALERACONTAINER=$(podman ps -f name=galera --format={{.Names}})
  
  for i in $(podman exec  $GALERACONTAINER mysql -s -N -e "select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';");do podman exec $GALERACONTAINER /usr/bin/mysqldump -uroot --single-transaction --databases $i >> $(hostname)_openstack_db-$(date +"%Y-%m-%d").sql; done
  
  for i in $(podman exec $GALERACONTAINER mysql -s -N -e "SELECT CONCAT('''',user,'''@''',host,'''') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')"); do podman exec $GALERACONTAINER mysql -s -N -e "SHOW GRANTS FOR $i" >> $(hostname)_openstack_db_grants-$(date +"%Y-%m-%d").sql; done
  

  NOTE: Replace podman with docker for RHOSP13 in above command

• Collect the file created by running script pre_checks.sh attached to the KCS.