Red Hat Knowledgebase
Welcome to Red Hat’s Knowledgebase information center. Find resources for resolving problems and troubleshooting. Log in to see all our solutions and articles. Some are restricted to verified users.
If you don’t have a Red Hat account yet, register and open one! For details about these accounts, see Developer Subscription Information or Customer Account Information.
Find what you need
Latest resources
Browse the latest published and updated knowledgebaseHosted Control Plane (HCP) dashboards or metrics are not visible on ACM Hub cluster after importing an external MCE cluster
- After discovering Hosted Control Planes (HCP) managed by an external MultiCluster Engine (MCE) operator into the Red Hat Advanced Cluster Management (RHACM) Hub cluster by following the Red Hat [documentation](https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.15/html/multicluster_engine_operator_with_red_hat_advanced_cluster_management/hosted-acm#discover-hosted-acm), the metrics and monitoring dashboards for the Hub's own locally managed hosted clusters are not visible or fail to display after enabling HCP observability by following the procedure described in the [documentation](https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html-single/hosted_control_planes/index#observability-for-hosted-control-planes).
VM fails to attach to Logical Switch Port (LSP) after migration
- Loss of VM network connectivity after an OpenShift upgrade- Logs in the VM showed that the Logical Switch Port (LSP) was not connected. ~~~- virt-handler-xxxxx logs {"component":"virt-handler","kind":";","level":"info","msg":"The target node received the running migrated domain","name":";testvm01","namespace":"test-vm","pos":"vm.go:816"...}... stderr F E0508... error: failed to add logical port of Pod stg-vm/virt-launcher-testvm01-xxxxx for NAD stg-vm/nad-virt: could not find OVN pod annotation in map[descheduler.alpha.kubernetes.io/request-evict-only: k8s.v1.cni.cncf.io/networks:[{";name";:"nad-virt","namespace":";test-vm",";mac":"0a:0b:0c:0d:0e:0f","interface":"pod12341234abc"}]...] ~~~
ROSA GovCloud and FIPS-validated Cryptography
This article describes the relationship between underlying OpenShift version and the status of FIPS-validation in ROSA GovCloud clusters.
Kernel panic after "Poison overwritten" in kmalloc-64 allocated in request_key_auth_new() and freed in keyctl_instantiate_key_common() on RHEL 8.10.z kernel 4.18.0-553.166.1.el8_10 and later
### Crash pattern without slub_debug + **Without slub_debug enabled, the bug can show as kernel crashes in different places, depending on what has reused the freed memory.** + The crash can be in the keys code itself, for example in free_request_key_auth() or search_process_keyrings() called from the request-key helper such as nfsidmap, or in an unrelated code path. One example is a fault in kfree() called from free_request_key_auth() and key_revoke() while the request-key helper (here nfsidmap) instantiates a key: ~~~ stack segment: 0000 [#1] SMP PTI CPU: 2 PID: 163271 Comm: nfsidmap Kdump: loaded Not tainted 4.18.0-553.166.1.el8_10.x86_64 #1 RIP: 0010:kfree+0x66/0x250 Call Trace:?__die_body+0x1a/0x60? die+0x2a/0x50? do_trap+0xe7/0x110? do_stack_segment+0x21/0x30? stack_segment+0x1e/0x30? free_request_key_auth.part.4+0x32/0x50? kfree+0x66/0x250? generic_file_buffered_read+0x84e/0xbb0 free_request_key_auth.part.4+0x32/0x50 key_revoke+0x3a/0x80__key_instantiate_and_link+0xb5/0x140 key_instantiate_and_link+0x16a/0x180 ~~~ + The bug can also cause a hang instead of a panic. In one such case, one CPU reports Soft lockups or Hard lockups spinning on a socket's sk_lock.slock, and the holder is the other CPU, stuck with interrupts off in tcp_v4_rcv()-> sock_def_readable()->__wake_up_common_lock() on the same socket's sk-> sk_wq-> wait.lock, a kmalloc-64 object. ``` [4215.465256] watchdog: BUG: soft lockup- CPU#0 stuck for 22s! [swapper/0:0] [4220.598257] Kernel panic- not syncing: NMI: Not continuing PID: 0 TASK: ffff8e3780c74000 CPU: 1 COMMAND: ";swapper/1"; [exception RIP: native_queued_spin_lock_slowpath+0x24] RIP: ffffffffbc15ee64 RSP: ffffb6f90001cae8 RFLAGS: 00000086 RAX: 00000000c0000000 RBX: 0000000000000246 RCX: 0000000000000002 RDX: 0000000000000001 RSI: 00000000c0000000 RDI: ffff8e36c56c5ec0 < <----- #5 [ffffb6f90001cae8] native_queued_spin_lock_slowpath at ffffffffbc15ee64 #6 [ffffb6f90001cae8]_raw_spin_lock_irqsave at ffffffffbca28764 #7 [ffffb6f90001caf8]__wake_up_common_lock at ffffffffbc14def6 #8 [ffffb6f90001cb68] sock_def_readable at ffffffffbc82b2e7 #9 [ffffb6f90001cb78] tcp_rcv_established at ffffffffbc90e990 #10 [ffffb6f90001cbb8] tcp_v4_do_rcv at ffffffffbc91b4b7 #11 [ffffb6f90001cbd8] tcp_v4_rcv at ffffffffbc91dab6 crash> px &((struct sock *)0xffff8e37a1810000)->; sk_wq-> wait-> lock $5 = (spinlock_t *) 0xffff8e36c56c5ec0 crash> kmem 0xffff8e36c56c5ec0 CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME ffff8e37800028c0 64 120483 121728 1902 4k kmalloc-64 SLAB MEMORY NODE TOTAL ALLOCATED FREE ffffeb6c4115b140 ffff8e36c56c5000 0 64 59 5 FREE / [ALLOCATED] [ffff8e36c56c5ec0] ``` + The characteristic is the spinlock word value 0xc0000000 (INT_MIN / 2) written by the x86 refcount exception handler ex_handler_refcount() when the second put of the double put in request_key_auth_destroy() decrements the unlocked spinlock of a socket_wq that reused the freed request_key_auth address, from 0 to-1. ``` crash> px ((struct sock *)0xffff8e37a1810000)->; sk_wq-> wait-> lock-> rlock-> raw_lock $9 = {{val = {counter = 0xc0000000 < <-----}, {locked = 0x0, pending = 0x0}, {locked_pending = 0x0, tail = 0xc000}}} ``` ### Crash pattern with slub_debug * The kernel panics or reports memory corruption in the kmalloc-64 slab cache. * With slub_debug enabled, a "Poison overwritten" report with the allocation point request_key_auth_new() and the free point keyctl_instantiate_key_common() is printed, followed by a list_del corruption panic. ~~~ BUG kmalloc-64 (Not tainted): Poison overwritten 0x00000000be29508a-0x00000000be29508a @offset=7232. First byte 0x69 instead of 0x6b Allocated in request_key_auth_new+0x5d/0x1f0 age=39 cpu=1 pid=29322 request_key_auth_new+0x5d/0x1f0 request_key_and_link+0x2dc/0x6f0 request_key+0x3c/0x80 nfs_idmap_get_key+0x12f/0x1e0 [nfsv4] nfs_idmap_lookup_id+0x30/0x80 [nfsv4] nfs_map_group_to_gid+0x11e/0x140 [nfsv4] Freed in keyctl_instantiate_key_common+0x140/0x1a0 age=36 cpu=0 pid=29338 keyctl_instantiate_key_common+0x140/0x1a0 keyctl_instantiate_key+0x4d/0x80 do_syscall_64+0x5b/0x1d0 entry_SYSCALL_64_after_hwframe+0x66/0xcb FIX kmalloc-64: Restoring 0x00000000be29508a-0x00000000be29508a=0x6b FIX kmalloc-64: Marking all objects used list_del corruption, fffff924c6452a08->; next is LIST_POISON1 (dead000000000100) kernel BUG at lib/list_debug.c:47! RIP: 0010:__list_del_entry_valid.cold.1+0x12/0x48 Call Trace: free_debug_processing+0x390/0x4c0 kfree+0x22e/0x250 Kernel panic- not syncing: Fatal exception in interrupt ~~~
Why using MTV to migrate VMs from VMware to OpenShift Virtualization is causing the disk sizes to use fully allocated size with NetApp Storage?
- The disk was thin provisioned in Vmware vSphere. However, after importing into OpenShift Virtualization using Migration Toolkit for Virtualization (MTV), the disk in the imported VM is using fully allocated size (virtual size in vSphere) when using NetApp storage devices.- No same issue was not encountered with other storage devices.
Ansible beyondtrust.secrets_safe lookup fails with 401 after BeyondTrust Password Safe upgrade to 26.2
* After upgrading BeyondTrust Password Safe from 24.2 to 26.2, Ansible playbooks fail to retrieve secrets using the `beyondtrust.secrets_safe` collection. * The lookup plugin fails with the following error: ~~~ The lookup plugin ';beyondtrust.secrets_safe.secrets_safe_lookup'; failed: Error getting secret by path, message: "Failed to authenticate due to one or more authentication rules.", statuscode: 401 ~~~ * On the BeyondTrust Password Safe appliance logs, requests to `/BeyondTrust/api/public/v3/secrets-safe/secrets` return HTTP 401 Unauthorized: ~~~ GET /BeyondTrust/api/public/v3/secrets-safe/secrets title=svc_unixad&;path=...- 401 0 0 616 393 4 ~~~
Can I download a VM golden image from the OpenShift dashboard (web UI)?
- Is there a feature in the OpenShift dashboard that allows users to download a DV as a VM image in their desired format?
MetricsService deployment fails with "must be no more than 63 characters" after upgrading to Ansible Automation Platform 2.7
- After upgrading Red Hat Ansible Automation Platform from version 2.6 to 2.7 on OpenShift, the MetricsService instance fails to deploy.- The parent AnsibleAutomationPlatform CR status or the Metricsservice CR status displays a Failure state, and inspecting the namespace events or the operator logs reveals the following validation error: ~~~ Failed to create object: b9;{"kind":";Status","apiVersion";:"v1";,"metadata":{},"status":";Failure",";message":";Deployment.apps \"ansible-automation-platform-automationmetricsservice-web\" is invalid: [spec.template.spec.volumes[5].name: Invalid value: \"ansible-automation-platform-automationmetricsservice-bundle-cacert\": must be no more than 63 characters, spec.template.spec.containers[0].volumeMounts[5].name: Not found: \"ansible-automation-platform-automationmetricsservice-bundle-cacert\", spec.template.spec.initContainers[0].volumeMounts[1].name: Not found: \"ansible-automation-platform-automationmetricsservice-bundle-cacert\"]","reason";:"Invalid";,... ";code";:422}9; ~~~
Trending
Solutions
Solve specific problems that has been raised with Red HatMachine Using KPatch-Patch Still Showing As Vulnerable In Red Hat Insights
+ A server registered to Red Hat Insights is shown to be vulnerable to a Common Vulnerabilities and Exposures (CVE), despite having and using a kpatch-patch to address the vulnerability. + Example: A stock RHEL 7.9 server registered to Red Hat Insights is shown to be vulnerable to [CVE-2022-32250](https://access.redhat.com/security/cve/CVE-2022-32250). That CVE is addressed by either a [kernel update](https://access.redhat.com/errata/RHSA-2022:5232) or a [kpatch-patch](https://access.redhat.com/errata/RHSA-2022:5216). The day after applying the kpatch-patch, Red Hat Insights Inventory still shows the machine (under its Vulnerability tab) vulnerable to that CVE, despite also showing the kpatch-patch RPM installed (under its General Information tab's Installed Packages list).
External ODF upgrades unexpectedly despite Manual upgrade strategy
- External ODF bypasses `Manual` upgrade strategy and updates automatically - External ODF ignores `Manual` upgrade policy and triggers upgrade - External ODF upgrades automatically regardless of `Manual` strategy setting
"goa-daemon" spams journal with "secret_password_lookup_sync() returned NULL"
- After every 5 seconds `goa-daemon` spams the journal with below messages. ~~~ goa-daemon[****]: secret_password_lookup_sync() returned NULL ~~~ - The `/var/log/messages` contains huge amount of below logs. ~~~ journal[****]: secret_password_lookup_sync() returned NULL ~~~ - How to disable `goa-deamon` on the system?
Articles
Technical briefs, reference architectures, and early stages of documentationRed Hat Case Management - Attachment Service API Guide
This document is the API usage guide for the Red Hat Case Management Attachment Service. It covers authentication, upload (single part and multipart up to 5 TB), download, deletion, status tracking, and metadata listing for file attachments on Red Hat support cases. The intended audience is developers and integration engineers building clients that interact with the attachment APIs at `https://api.access.redhat.com/support`.
Uninstalling an OpenShift cluster
Should you need to uninstall an openshift cluster, here are the general instructions for uninstalling a cluster.
Red Hat OpenShift Data Foundation Disaster Recovery Offerings
This is one place to know about the Red Hat OpenShift Data Foundation Disaster Recovery Offerings
Latest discussions in the Red Hat Community
RHCOS ISO for aarch64 architecture
Generated Agentic AI
OpenShift 4.22 UPI bootstrap stuck: bootstrap MCS 22623 works but Kubernetes API 644...
Get support
Support cases
Get answers quickly by opening a support case with us.
Live chat
Directly access our support engineers during weekday business hours.
Call or email
Speak directly with a Red Hat Support expert by phone or through email.
