How to open a PROACTIVE case for patching or upgrading Red Hat OpenShift Container Platform

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)

Issue

  • Upgrading production OpenShift Container Platform clusters.
  • How to open a case to notify Red Hat of upgrades taking place so appropriate resources are available?
  • What information is useful for Red Hat support when opening a proactive ticket for weekend or after hours work?

Note :

  • For new installations it is not necessary to create a proactive case. For those situations, simply raise a regular support ticket if any problem arises, please also note that new installations can't be considered severity 1 (urgent) accordingly with our Red Hat Support Severity Level Definitions.
  • One cluster Upgrade per Support Case as per our Guidelines for a Successful Support Engagement. If Red Hat OpenShift Data Foundation (ODF) and/or Red Hat OpenShift Virtualization are installed in the cluster, different support cases for those components are required.
  • Pre-Upgrade or Post-Upgrade issue shall require a separate case if there are any potential or existing issues seen.

Resolution

Introduction - What is a Proactive Case?

Proactive cases are a way for Premium customers to provide an advance notice to Red Hat Support of planned upgrades, migrations, patching, or any other maintenance activity. Typically, these events happen outside regular business hours.

Having prior knowledge of the planned activity provides Red Hat with an opportunity to review the status of the cluster before the upgrade, and to ensure the right information is made available to better prepare technical support should an issue is faced.

Note: New installations do not entitle a Proactive Case, as they do not fit with the Severity Level Definitions for Severity 1. For issues encountered during new installations, please open a regular Support Case.

Consideration for opening a proactive ticket

During the timeframe or window of maintenance Red Hat will not remotely access systems in scenarios where there is no software defect. Such scenarios include, but are not limited to:

Performing system administration functions
Standing by during migrations or upgrades
Debugging non-Red Hat software
Monitoring your environment
Walking through documentation and upgrade procedure/workflow.
Configurations or advisement of pre-upgrade configurations/issues.


NOTE


Red Hat support will not join any maintenance windows calls/bridges/channels, but will actively monitor the proactive case for updates after the maintenance window has been opened.

Prerequisites

  • The account must have an active Premium OpenShift subscription.

    • If the account has an active Standard: Red Hat Cloud Suite, Standard (2 Sockets, 32 Cores) subscription, or the cluster is ARO (Azure Red Hat OpenShift) as a Partner Product (for which the Microsoft provides the subscription and not Red Hat) then the account is entitled to submit and engage in a Proactive Case
  • It is needed to submit Proactive Cases several days before the activity is scheduled to begin (preferably 7-14 days in advance), in order to check if the cluster is able to be upgraded, and to allocate appropriate resources for the upcoming event.

    • Review if a cluster is in a good shape for an upgrade takes time, and in some cases, actions could be required to be performed before the upgrade, which can lead in delays for the upgrade, or issues if not performed.
    • If the activity requires weekend coverage, the Proactive Case must be submitted at least 4 business days before the upcoming activity (by close of the business day, for example on the Tuesday prior to the upcoming activity).
      Note: weekend Proactive Cases submitted less than 4 business days before the activity will be considered "Late Entry", and treated through normal Follow the Sun workflow. For such, Red Hat will not be able to guarantee that the appropriate resources would be staffed for the weekend.
    • In order to allocate the most appropriate engineer from different teams, no matter where they may be worldwide, Proactive Support Cases are conducted in the English language only.
  • It is recommended to familiarize with the Reference Guide for Engaging with Red Hat Support article, which provides best practices for engaging Red Hat Support.

  • Prepare all relevant details for the upcoming event, that will help communicate what changes and activities are taking place.

  • If the account has a Technical Account Manager (TAM), a Solution Architect (SA), or a Customer Success Manager (CSM) - please engage them when planning a Proactive Case.

  • Check the overall state of the cluster following OpenShift 4 cluster upgrade pre-checks requirements.

  • Review Navigating Kubernetes API deprecations and removals for getting information about the Kubernetes deprecated APIs used in the cluster.

  • If Red Hat OpenShift Data Foundation (RHODF) is installed on the cluster, refer also to implications to consider when upgrading OpenShift Data Foundation, and open a separate support case for the ODF upgrade.

  • If Red Hat OpenShift Virtualization or Migration Toolkit for Virtualization (MTV) are installed on the cluster, refer also to how to open a Proactive case for OpenShift Virtualization/MTV and open a separate support case for them.

Opening a Proactive Support Case

Open a Support Case and provide the required information in the following manner:

  • Case Title - should be prefixed with "[Proactive]" and should summarize the version upgrade. Example:

    [PROACTIVE] Production/Integration/Development OpenShift cluster upgrade from 4.16 to 4.17
    
  • Severity - open the case on a Severity level of 3. On the day of activity or when the activity begins, Red Hat recommend the customers to reset the Severity to 1 so that the proper resources would be allocated to assist in case any issue arises.

  • Case Description - describe as many details as possible, for example:
    • Specific activity occurring during the event.
    • Environment.
    • Infrastructure.
    • System(s) involved.
    • Other Components being upgraded.
  • Date / Time of Event - state the planned start and end date/time of the maintenance window duration including time zone.
  • Contact Information - provide the primary and secondary contact name, email addresses and phone number with country and region code. This is especially important if the person who opened the case is not the person doing the maintenance activity.
  • Data Attachments - Collect a fresh must-gather and, if needed, sosreports of all the systems involved before any upgrade work is performed (refer to the Diagnostic Steps section below).
    • When collecting sosreports if there are only a few systems, please attach them to the case. Otherwise, please keep them locally and put a note in the case description confirming that sosreports have been collected for all of them. This will help to prevent possible issues that were already there and identify new issues after the upgrade.
    • Collect the information about the Kubernetes deprecated APIs used in the cluster following Navigating Kubernetes API deprecations and removals.

NOTE: Unless exceptionally recommended by a TAM or CSM (for accounts with any of them), it is not supported to update more than one minor version at a time, if in spite of this it is still needed to upgrade more than one version in the same maintenance window, please keep in mind that for workload reasons Red Hat ca not guarantee a dedicated resource in place.

Example of a Proactive Case:

  • Case Title:

    [PROACTIVE] Production OpenShift cluster upgrade from 4.16.23 to 4.18.12
    
  • Case Description:

    In two weeks, on Saturday July 22nd,  we are planning to upgrade our production cluster from OCP X.Y.Z to OCP X2.Y2.Z2.
    These guest systems are installed on Red Hat Virtualization 4.2. All systems have 8 vCPU's and 32GB of RAM. No memory or CPU over commitment is being used.
    
    Cluster details:
      3 MASTERS.
      3 ETCD NODES.
      3 INFRA NODES.
      20 APP NODES.
      50 PROJECTS.
      200 SERVICES.
      400 PODS.
    
    Maintenance Window Information:
       - Date - Sunday, July 22nd
       - Time frame - 6 hours
       - Timezone - CEST
       - Start Time - 08:00 AM UTC+2
       - End Time - 14:00 PM UTC+2
    
    Primary Contact Information:
       - Name: Jane Doe
       - Email: jane.doe@example.com
       - Phone: +1-123-456-7890
    
    Secondary Contact Information:
      - Name: John Doe
      - Email: john.doe@example.com
      - Phone: +1-123-456-7891
    
  • Case Attachments:

    must-gather.tar.gz
    sos-report.tar.gz
    

Diagnostic Steps

OpenShift 4

Collect a must-gather log bundle containing a full picture of the cluster general status and configurations. You can refer to the instructions below or to the gathering data about your cluster documentation.

  • Navigate to the directory where you want to store the must-gather data.
  • Run the oc adm must-gather command:

    $ oc adm must-gather
    
  • Create a compressed file from the must-gather directory that was just created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

    $ tar cvaf must-gather.tar.gz must-gather.local.[string]/ 
    

    Make sure to replace must-gather-local.[string]/ with the actual directory name & attach the compressed file to the support case on the Red Hat Customer Portal.

  • Only if requested, sosreports can also be collected from the RHCOS underlying nodes within OCP4. If needed, refer to how to provide a sos report from a RHEL CoreOS OpenShift 4 node.

OpenShift 3

To properly collect a sosreports on OCP 3, follow the instructions below or refer to the link:

$ sosreport -e docker -k docker.all=on -k docker.logs=on -k origin.diag=off

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments