Chapter 4. Creating the Test Plan
4.1. Test Plan Overview
This chapter describes the process followed by the Red Hat Hardware Certification team to create a test plan for a system or hardware component. Section Section 4.7, “Hardware Class Requirements” provides a useful reference that defines the testing required for each hardware class item.
A hardware certification engineer creates a test plan by following these steps:
- Define the model by its specification (Section 4.2, “Models”)
- Determine the options (Section 4.3, “Options”)
- Remove unsupported operating system features and unintentional hardware (Section 4.4, “Non-OS Features and Unintentional Features”)
- Apply the minimum test set criteria (Section 4.5, “Minimum Test Set”)
- Add the install, boot, and kdump requirements (Section 4.6, “Installation, Boot, and Kdump Requirements”)
- Add additional policy requirements (Chapter 3, Hardware Certification Policies)
After performing the steps above, the items remaining determine the test plan for your hardware. The Hardware Catalog records the test plan under the Test Plan Progress.
Figure 4.1. Test Plan Progress
Red Hat Hardware Certification Test Plans are not meant to substitute for proper and complete internal quality assurance testing, criteria, and processes. Each vendor is responsible for their own internal shipment criteria and is encouraged to do testing in excess of the required certification test plan items.
4.2. Models
The Red Hat Hardware Certification program certifies models, not specific configurations of models. Red Hat defines a model as inclusive of all Integrated Hardware and all Optional Hardware described by the Hardware Partner on the hardware specification. Integrated Hardware is hardware required to be present in all configurations of a model. Optional Hardware is hardware which is present in some configurations of a model. Additional Hardware may also appear on the model specification. Additional Hardware is hardware that can be purchased in addition to but is not included as part of any configuration of the model. Additional Hardware is not required to be tested but must be clearly identifiable as Additional Hardware and not confused with Integrated Hardware or Optional Hardware. A Red Hat Knowledge Base Article may be associated with the certification listing for clarity of Additional Hardware.
Optional Hardware was previously referred to as Config-To-Order (CTO). Additional Hardware was previously referred to as After-Market (AM).
Model names are required to be unique and have a particular hardware specification.
Tiered model naming schemes are allowed and supported by the Red Hat Hardware Certification program. A tiered naming scheme is any naming scheme which includes a hierarchical collection of models and submodels. When employing tiered naming schemes for the purposes of certification the specification is considered to include all submodels which would reasonably be represented by the name provided in the certification request. For example; three model names, 3000, 3000a, and 3000s. If 3000 reflects the collection which includes the 3000a and 3000s models and 3000 is submitted, the specification would include the content of the 3000a and 3000s models. If, however, 3000s was submitted the specification would be limited to only the hardware listed in the 3000s specification. If 3000 is instead a model separate from 3000a and 3000s this would not be a tiered scheme but similar model naming and only the hardware listed in the 3000 specification would be considered.
Red Hat may alter the listed model name for clarity; for example in NUMA and cluster situations when a quantity of systems/nodes alters the specification and a Red Hat Knowledge Base entry is not considered sufficient to avoid customer confusion; e.g. the addition of "(up to 2 nodes)" after a model name.
For simplicity, a leverage pool certification model name may utilize the component vendor’s model information in the make and model fields. The model name must be unique within the system vendor’s pool and will remain unpublished.
4.3. Options
4.3.1. Integrated Hardware
All integrated hardware, CPU options, memory options, integrated graphic controllers, integrated displays, and other non field removable hardware of a model must be tested. This includes features integrated into System-on-Chip (SoC), System-in-Package (SiP), and other full or partial integrated systems solution designs.
Specific portions of integrated hardware may be excluded from the certification when they provide features which qualify for exclusion based on the policies outlined in Section 4.4, “Non-OS Features and Unintentional Features” or system processor leveraging.
4.3.2. Optional Hardware
All Optional Hardware must be tested except when the Optional Hardware is field removable, does not provide a unique function within the model [1] , and is clearly noted for use with another operating system [2] or marked to disclose any Service Level impacts as appropriate on at least one of the model specification or the model support URL and on all materials using the Red Hat Hardware Certification mark(s) in association with the model.
As an example, if a system is available with and without a SAS controller, and is also available with three functionally identical optional network cards, one of which is identified for use with Fedora only, the required testing would include the SAS controller and the two remaining optional network cards.
4.3.3. Special Cases
The Hardware Changes policy (see Section 3.4.4, “Changes”) may be utilized when Optional Hardware or a (series of) CPU(s) causes a higher than desired minor release to be required during an original certification. This may allow the model to be tested and posted with the desired release with an associated Red Hat Knowledge Base Article to reflect the higher release required by the Optional Hardware or CPU(s).
4.4. Non-OS Features and Unintentional Features
Hardware feature classes not offered by the operating system are not required to be tested if the remaining hardware continues to be fully functional. For example, IEEE 1394, a type of storage not currently supported in Red Hat Enterprise Linux 6, would not require testing in a Red Hat Enterprise Linux 6.x certification. If, however, the system only had IEEE 1394 storage, it would not qualify for a Red Hat Enterprise Linux 6 certification. A Red Hat Knowledge Base Article may be added to the certification listing for clarity.
An Unintentional Feature is defined as any feature offered on integrated or optional hardware that is not intentionally included by the hardware partner. This feature must not be mentioned in the hardware specification unless it is called out as not supported. Unintentional features can not be supported by the hardware partner on any OS. Unintentional features are not required to be tested if the remaining hardware continues to be fully functional, even if the provided feature is unique. We recommend that unintentional features are masked from end users where possible, i.e. by disabling or removing features from the BIOS, not providing power, not including connectors, headers, etc. to minimize confusion. A Red Hat Knowledge Base Article may be added to add clarity. Changes to unintentional features are considered to be hardware changes and subject to the hardware changes policies and requirements.
Unintentional features can also cover items that are not available on all architectures. For example, if an Infiniband storage controller were supported by a system vendor on the Intel 64 and AMD64 architecture only, the controller could be considered an unintentional feature for the system’s i386 certification. The feature must not be supported on any i386 architecture operating system for the unintentional feature status to be granted.
4.5. Minimum Test Set
The Red Hat Hardware Certification program encourages testing with all configurations including the maximum and minimum supported configuration of your hardware. It is also recognized that resourcing these configuration can be difficult due to availability, cost, timing, and other constraints.
For these reasons we have defined a minimum requirements policy by hardware class in the Table 4.1, “Hardware Requirements by Class”. This column in combination with Section 3.1.11, “Component Leveraging” and Section 3.1.14, “Component Pass-Through Certifications” are designed to ensure an appropriate and reasonable testing effort to achieve certification.
The minimum testing requirements are not intended as product release criteria and it is expected that internal Red Hat Enterprise Linux and other Red Hat product interoperability, and qualification testing is conducted in addition to and prior to certification testing.
All hardware used during testing is required to be part of the model specification. Similar hardware that might otherwise qualify as part of the minimum test set if it were part of the model is not accepted. For example, only those CPUs which appear in the model specification may be used. Results from other members of the same CPU product family are not accepted.
The maximum supported limits for Red Hat Enterprise Linux are defined at https://access.redhat.com/articles/rhel-limits.
4.6. Installation, Boot, and Kdump Requirements
The installation of Red Hat Enterprise Linux may require testing via a number of mediums (Optical Media and Network for example). Additionally, all boot devices must be tested to ensure a successful boot of Red Hat Enterprise Linux. The Table 4.1, “Hardware Requirements by Class” table shows the hardware that requires installation and boot testing. A complete installation is not required to fulfill the boot testing requirement.
For increased testing efficiency, we recommend combining boot and install testing where possible. For example, booting from the Red Hat Enterprise Linux installation media on a CD and performing a full installation fulfills the CD boot and installation testing requirement.
Kdump is a common feature of both Red Hat Enterprise Linux 6 and 7. Kdump utilizes the Linux kernel kexec feature to boot a kernel without a hardware reset in the event of a crash and capture the state of the previous kernel. This feature is enabled by default and must be tested to ensure this critical debug information can be captured properly. The kdump test will automatically be planned when the kdump service is enabled.
Kdump testing is required on an integrated storage controller and an integrated network adapter when these items are available in the model. These requirements apply to all Red Hat Enterprise Linux 6 and 7 certifications on the 32-bit and 64-bit Intel and AMD systems , and 64-bit IBM PowerPC architectures. Additionally, Red Hat Enterprise Linux allows testing of Kdump on IBM System z architectures.
Red Hat Enterprise Linux 6.0 and 6.1 certifications may optionally use Red Hat Enterprise Linux 6.2 kdump results to satisfy the kdump testing criteria as the kdump requirement policy was established subsequent to these releases. A Red Hat Knowledge Base Article will be added to these certs noting that the minimum version for kdump functionality is Red Hat Enterprise Linux 6.2.
4.7. Hardware Class Requirements
Table 4.1. Hardware Requirements by Class
| Device Class | Required Tests | Minimum Requirements | Install, Boot, Kdump[a][b] | General Notes | Leverage Notes[c] |
|---|---|---|---|---|---|
| All | Info | Info is required for all test runs. | Test runs without Info are invalid and will be automatically discarded by the Hardware Catalog. | ||
| Battery | Battery and Suspend | Required for all models capable of running from battery power. | |||
| Bluetooth | Bluetooth | A minimum of two machines with Bluetooth capability in proximity to each other. | Identical integrated chipsets and removable adapters. | ||
| CD-ROM drive, DVD drive, or Blu-ray | CDROM drive, DVD drive, or BLURAY | The highest media type in order of BD-RW (highest), BD-R, DVD-RW[d], DVD-R, CD-RW, CD-R, BD, DVD, CD (lowest) on each storage controller, based on the collective media support of all drives available on that storage controller. | Install, Boot | The hardware partner is required to support all drives that are part of the model regardless of the specific drive or number of drives used during testing. Equivalent production cycle drive changes are required to be tested internally by the hardware partner. Production cycle drive change test results are not required to be submitted to Red Hat. | Drives with identical or lesser media support on the storage controller following the storage controller leveraging policies. |
| Display Adapters | Video | The lower of VRAM/VBIOS limits, panel capabilities, or 1024x768 at 24 or 32 BPP. | Install,Boot; 3D is not currently tested. GPGPUs are not currently tested. | Identical removable cards or integrated chips without shared memory,processor-integrated. Decreases in video memory. | |
| Express Card Sockets | Express Card | All accessible sockets | Devices and inaccessible ports require additional class testing as applicable. Both USB and PCI-E portions are required. | ||
| Fibre Channel | Network or Storage | Each interface at maximum connection speed. | Install ,Boot ,Kdump | Nominal connection speed is considered a feature. Remote attached storage devices may require additional testing. (see Section 4.8.1, “External Storage and Multipath HBAs”) | Identical integrated chipsets, removable adapters, drives and arrays. |
| IDE/SCSI/SATA/SAS Drives, Arrays, and HBAs | Storage | Maximum storage capacity of local attach arrays if greater than OS limit. | Install,Boot,Kdump | Drive capacity are not tracked in context of a system. SATA Controllers require testing in a SATA mode. SAS Controllers require testing with SAS drives. | Identical integrated chipsets, removable adapters, drives and arrays. |
| SATA[SSD with M2, U2], SAS[SSD], NVMe[PCIE, M2, U2] | Storage | Maximum storage capacity of local attach arrays if greater than OS limit. | Install,Boot,Kdump | Drive capacity are not tracked in context of a system. SATA Controllers require testing in a SATA mode. SAS Controllers require testing with SAS drives. | Identical integrated chipsets, removable adapters, drives and arrays. |
| NVDIMM | Storage | Maximum storage capacity of local attach arrays if greater than OS limit.[e] | Install,Boot,Kdump | Drive capacity are not tracked in context of a system. SATA Controllers require testing in a SATA mode. SAS Controllers require testing with SAS drives. | Identical integrated chipsets, removable adapters, drives and arrays. |
| I/O Chassis, Port Expanders | Applicable testing for integrated additional CTO hardware. | Install,Boot | Identical integrated chipsets, removable adapters, drives and arrays. | ||
| Integrated Graphics Display | Video [LID] | Native resolution[f] [g] at adaptive or native color depths with available display + graphics controller combinations[h][i] | Install [j] | Backlight must respond to lid switch if present | Identical display + graphics controller + BIOS |
| iSCSI Adapters | Network and Storage | Each interface at maximum connection speed. | Install,Boot,Kdump | Nominal network connection speed is considered a feature. | Identical integrated chipsets and removable adapters. |
| (e)MMC, SD, CompactFlash Memory Cards/Chips/Readers[k], USB Keys/Thumb drives | Storage | Each interface. Maximum storage capacity and format feature set. | Install,Boot | Multi-Readers follow the Multi-Port Adapter criteria. | Identical integrated chipsets, removable adapters. Identical, smaller capacity or feature cards and sticks. |
| Motherboard/Mainboard | System Processor+Memory requirements+all integrated feature classes. | Follows the combined class leverage requirements. | |||
| Multi-Function Adapters | Applicable test for each function. | Follows the combined functional leverage requirements. | |||
| Multi-Port Adapters | Functional test(s) on all ports -or- single port where identical chips are replicated for each port -or- maximum number of ports per controller where heads are capped to create a port reduced variation. | Follows functional leverage requirements. | |||
| Network Interface Adapters | Network | Each interface at maximum connection speed[l] | Install,Boot,Kdump | Nominal network connection speed is considered a feature. | Identical integrated chipsets and removable adapters. |
| PC Card Sockets | PC Card | All accessible sockets | Devices and inaccessible ports require additional class testing as applicable | ||
| PCIE, SAS, and SATA SSD Storage Cards and Drives | Storage | Maximum storage capacity and feature set. Capacity is not tracked for SATA and SAS attached SSDs in context of a system certification and should follow the standard SAS and SATA policies instead. | Install,Boot,Kdump | PCIE SSD Storage testing will be required on option cards starting May 1st 2015[m]. | Identical integrated chipsets, removable adapters and drives. Models with the same host interface, hardware and firmware designs including reduced features and capacity. |
| RAID Controllers | Storage | Each OS code path (e.g. where multiple drivers are used) for each interface. Maximum storage capacity of arrays if greater than OS limit. | Install,Boot,Kdump | Host RAID/Driver RAID/Fake RAID/etc. use the applicable regular controller policy. Remote attached storage devices may require additional testing. (see Section 4.8.1, “External Storage and Multipath HBAs”) | Identical integrated chipsets, removable adapters, drives and arrays following type criteria. Reduced RAID levels, changes in memory amounts or battery presence. |
| Sound Cards | Audio | Stereo record and playback as applicable. | Identical integrated chipsets+codec and removable adapters. | ||
| System Buses | Each bus by proxy testing[n]. | Install,Boot | Buses without possible active devices do not require testing. | Identical integrated chipsets, removable adapters | |
| System Processors, System-on-Chip(SoC), System-in-Package(SiP) | CORE[o] and CPUSCALING and PROFILER [FV_CORE and FV_MEMORY] [REALTIME OR RTEVAL and HWLATDETECT] | Maximum number of processors, cores[p], and feature set[q][r]; Multiple test runs with different processors on different kernels [s]may be required. | Core clock speed, FSB speed, cache size, cache depth and manufacturing size are not considered for feature set review.[t] A fully virtualized guest environment using KVM. PROFILER test runs may utilize timer mode. [u] REALTIME, RTEVAL and HWLATDETECT require the realtime kernel. | Equal or lesser feature set within a model. Processor/core count downward on scaling designs. Feature set and core count upgrades[v] to existing certifications. | |
| System Memory | Memory | Lower of 1GB Per Logical CPU, system limit, or OS limit[w]. | The vendor must support the maximum amount of RAM listed in the specification irrespective of the test results submitted. Memory clock speed is not considered for feature review. | Equal or lesser quantities where RAM type and memory controller match. | |
| System Virtualization | INFO and CORE and MEMORY on the guest | A fully virtualized guest environment (Xen on Red Hat Enterprise Linux 5, KVM on Red Hat Enterprise Linux 6) | 32-bit and 64-bit Intel and AMD systems with pae and vmx on Red Hat Enterprise Linux 5; 64-bit Intel and AMD systems only on Red Hat Enterprise Linux 6. | None[x] | |
| Tape Drives and Changers | TAPE | Each drive | Changers require manual testing with test description and results report. | Identical drives and changers. Internal and external versions of the same drives. Models with the same host interface, hardware and firmware designs including reduced features, capacity, media size and/or total slots and drive count in changers/libraries. | |
| USB ports | USB, USB 2, USB 3[y] | All pluggable ports at the stated connection version[z]. | Inaccessible ports may substitute attached device class testing. | ||
| USB devices | Each function class at the stated connection version. | Class testing as applicable | Identical devices. | ||
| Virtual Hardware | Each virtualized hardware class. | Class testing as applicable | |||
| Wireless Network Interface Adapters | WirelessG WirelessN WirelessAC[aa] | Each interface at maximum connection in N(highest), G, B, A(lowest) order. | Install,Boot | Nominal network connection speed is considered a feature. | Identical integrated chipsets and removable adapters. |
| Ethernet Adapters | Ethernet or the aligning test from one of the following: 1GbE (Gigabit Ethernet), 2GbE, 5GbE, 10GbE, 20GbE, 25GbE, 40GbE, 50GbE or 100GbE. | Each interface with the corresponding test for the maximum claimed connection speed. | Install ,Boot ,Kdump | Maximum network connection speed is considered a feature. | Identical integrated chipsets and removable adapters |
| Infiniband | Infiniband_QDR, Infiniband_FDR, or Infiniband_EDR | Each interface with the corresponding test for the maximum claimed connection speed. | Install,Boot,Kdump | Maximum network connection speed is considered a feature. | Identical integrated chipsets, removable adapters, drives and arrays. |
| iWARP Adapters | 10GbE (Gigabit Ethernet), 20GbE, 25GbE, 40GbE, 50GbE or 100GbE. | Each interface with the corresponding test for the maximum claimed connection speed. | Maximum network connection speed is considered a feature. | Identical integrated chipsets and removable adapters. | |
| OmniPath | OmniPath | Each interface with the corresponding test for the maximum claimed connection speed. | Maximum network connection speed is considered a feature. | Identical integrated chipsets,processor-integrated,and removable adapters. | |
| RoCE | 10GbE (Gigabit Ethernet), 20GbE, 25GbE, 40GbE, 50GbE or 100GbE. | Each interface with the corresponding test for the maximum claimed connection speed | Maximum network connection speed is considered a feature. | Identical integrated chipsets and removable adapters. | |
[a]
Red Hat Enterprise Linux 6 only.
[b]
Not required on s390x.
[c]
Refer to section, Component Leveraging for more information on Leveraging
[d]
"+" and "-" are considered equal for feature review.
[e]
It is available for RHEL versions 7.3 and above
[f]
Compensation/Stretching does not qualify as native testing.
[g]
A horizontal resolution of 1360 may be used on 1366 native panels.
[h]
Optional graphics controllers excluded by other policies are not required to be tested. At least one display + controller combination is required for each display.
[i]
Display and graphics controller combinations may be clarified in a Red Hat Knowledge Base Article entry to avoid confusion.
[j]
Native resolutions not required during install
[k]
Including variants for each (eg. mini, micro, etc.).
[l]
Network devices that support NIC partitioning are required to demonstrate both the complete bandwidth and a single partition in one or more test runs.
[m]
A knowledge base entry may be added to the certification to clarify when passing results are not provided before the requirement date.
[n]
For example, testing a PCI NIC with the NETWORK test tests the PCI bus.
[o]
The minimum installed memory is required to match the System Memory testing requirement.
[p]
System or OS maximum, whichever is lower.
[q]
Integrated non-usable GPUs are ignored during feature set comparison.
[r]
Features includes any integrated classes and are be subject to additional requirements of that class.
[s]
A knowledge base entry may be added to the certification to clarify combinations of CPUs and kernels for customers.
[t]
Additional testing may be required where maximum system core count is greater than the currently listed OS Certified maximum.
[v]
Processor upgrades are defined as field installable physical packages and may require field installable BIOS/firmware upgrades (Section 3.3.3, “Settings” apply).
[w]
Additional testing may be required where maximum system memory is greater than the currently listed OS Certified maximum.
[x]
System Virtualization is leverageable in CPU upgrade leveraging where the existing certification already includes System Virtualization testing.
[y]
While running the USB 3.x tests, a dropdown list gives an option to select between gen1 and gen2.
[z]
USB 3.1 gen2 ports require testing with RHEL 7.3 or higher in all RHEL 7.x certification; all USB 3.x ports will be tested as USB 3.0 in RHEL 6.x. USB 3.1 gen2 ports that are tested only with the gen1 devices can be certified.
[aa]
Red Hat Enterprise Linux 7 only supports 802.11ac devices at 802.11n speeds. Results will be accepted from the WirelessN test on 802.11ac devices until an erratum that provides full 802.11ac connection speeds to Red Hat Enterprise Linux 7 is available.
| |||||
4.8. Additional Manual Testing
4.8.1. External Storage and Multipath HBAs
In addition to the base requirements for storage controllers/devices; vendors must verify that their internal quality assurance processes have tested full functionality with Red Hat Enterprise Linux under the following scenarios as appropriate:
- multi-controllers/single host
- multi-host/single controller
- multi-controller/multi-host
- with/without multi-path
- with/without LUN masking (i.e., dedicating LUNs to specific hosts)
- a short cable pull (remove cable and restore prior to failure detection)
- any special features listed as supported on Red Hat Enterprise Linux
Testing result packages are not required to be submitted to Red Hat for the above testing.
Revised on 2018-11-01 11:44:39 UTC
Where did the comment section go?
Red Hat's documentation publication system recently went through an upgrade to enable speedier, more mobile-friendly content. We decided to re-evaluate our commenting platform to ensure that it meets your expectations and serves as an optimal feedback mechanism. During this redesign, we invite your input on providing feedback on Red Hat documentation via the discussion platform.