Red Hat Knowledgebase
Welcome to Red Hat’s Knowledgebase information center. Find resources for resolving problems and troubleshooting. Log in to see all our solutions and articles. Some are restricted to verified users.
If you don’t have a Red Hat account yet, register and open one! For details about these accounts, see Developer Subscription Information or Customer Account Information.
Find what you need
Latest resources
Browse the latest published and updated knowledgebaseSystem hangs with many tasks blocked on cgroup_mutex while kswapd loops in do_shrink_slab() on RHEL8
System hangs with many tasks blocked on cgroup_mutex while kswapd loops in do_shrink_slab() on RHEL8 Red Hat Enterprise Linux 8 Memory cgroups The system hangs and many tasks stay in D state until the system is restarted. Messages like the following are logged for many tasks: [917190.287287] INFO: task systemd:1 blocked for more than 120 seconds. [917190.287294] Tainted: G OE X -------- - - 4.18.0-553.170.1.el8_10.x86_64 #1 [917190.287296] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [917190.287297] task:systemd state:D stack:0 pid:1 ppid:0 flags:0x90000000 [917190.287302] Call Trace: [917190.287342] __schedule+0x2d1/0x870 [917190.287386] schedule+0x55/0xf0 [917190.287389] schedule_preempt_disabled+0xa/0x10 [917190.287392] rwsem_down_write_slowpath+0x370/0x570 [917190.287412] down_write+0x29/0x50 [917190.287416] alloc_shrinker_info+0x1f/0x100 [917190.287422] mem_cgroup_css_online+0xf/0xc0 [917190.287428] online_css+0x1c/0x60 [917190.287433] cgroup_apply_control_enable+0x20d/0x310 [917190.287439] cgroup_mkdir+0x1c2/0x490 [917190.287441] kernfs_iop_mkdir+0x5a/0xa0 [917190.287447] vfs_mkdir+0x11b/0x1e0 [917190.287452] do_mkdirat+0xec/0x120 [917190.287456] do_syscall_64+0x5b/0x1d0 [917190.287462] entry_SYSCALL_64_after_hwframe+0x66/0xcb -- [917190.287511] INFO: task systemd-journal:635 blocked for more than 120 seconds. [917190.287513] Tainted: G OE X -------- - - 4.18.0-553.170.1.el8_10.x86_64 #1 [917190.287514] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [917190.287515] task:systemd-journal state:D stack:0 pid:635 ppid:1 flags:0x90000100 [917190.287518] Call Trace: [917190.287520] __schedule+0x2d1/0x870 [917190.287523] ? fsnotify+0x1/0x3d0 [917190.287528] ? proc_cgroup_show+0x30/0x370 [917190.287531] schedule+0x55/0xf0 [917190.287534] schedule_preempt_disabled+0xa/0x10 [917190.287536] __mutex_lock.isra.11+0x349/0x420 [917190.287539] ? kmem_cache_alloc_trace+0x142/0x280 [917190.287543] proc_cgroup_show+0x4a/0x370 [917190.287545] proc_single_show+0x4e/0x90 [917190.287550] seq_read+0x163/0x420 [917190.287555] vfs_read+0x91/0x150 [917190.287560] ksys_read+0x4f/0xb0 [917190.287562] do_syscall_64+0x5b/0x1d0 [917190.287565] entry_SYSCALL_64_after_hwframe+0x66/0xcb [917190.287567] RIP: 0033:0x7fbaccdfb424 -- [917190.288485] INFO: task app-A:5160 blocked for more than 120 seconds. [917190.288488] Tainted: G OE X -------- - - 4.18.0-553.170.1.el8_10.x86_64 #1 [917190.288489] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [917190.288490] task:app-A state:D stack:0 pid:5160 ppid:5175 flags:0x10000000 [917190.288493] Call Trace: [917190.288496] __schedule+0x2d1/0x870 [917190.288502] schedule+0x55/0xf0 [917190.288505] schedule_preempt_disabled+0xa/0x10 [917190.288506] __mutex_lock.isra.11+0x349/0x420 [917190.288509] task_cgroup_path+0x3a/0x170 [917190.288517] endr_proc_utils_get_extended_info+0x179/0xa20 [module-A] [917190.288529] ? security_file_alloc+0x1/0x90 [917190.288534] ? security_file_alloc+0x1/0x90 [917190.288537] _endr_get_klids+0xd1/0xe30 [module-A] [917190.288546] _endr_get_klids+0x54e/0xe30 [module-A] [917190.288552] ? udp_lib_get_port+0x4c0/0x4c0 [917190.288568] _endr_get_klids+0x664/0xe30 [module-A] [917190.288574] inet_autobind+0x2c/0x70 [917190.288577] inet_dgram_connect+0x6e/0xb0 [917190.288579] __sys_connect+0x9e/0xd0 Red Hat is aware of this issue, and a fix for Red Hat Enterprise Linux 8.10.z is in progress. The fix is not included in 8.10.z kernels up to the latest version. The issue is fixed upstream by 41d17431df4a ("mm: list_lru: fix the return value of list_lru_count_one()"). Red Hat Enterprise Linux 9.1 and later already include the fix. We opened a JIRA, RHEL-300411, for RHEL8.10.z backport of the same patch. The RHEL8.10.z backport of the patch is in progress in that JIRA. list_lru_count_one() returns the nr_items counter of a list_lru as unsigned long, without checking for a negative value. nr_items can be negative for a short time while a memory cgroup is being removed and its list_lru items are moved to the parent. A shrinker that reads such a count takes a small negative value as a huge number of freeable objects, and do_shrink_slab() gets a scan target that it practically never finishes. do_shrink_slab() holds shrinker_rwsem for read during the scan, so a memory cgroup creation, which takes shrinker_rwsem for write while holding cgroup_mutex, waits forever, and every task that needs cgroup_mutex hangs behind it. VMCORE ANALYSIS Tasks in D state crash> sys ... LOAD AVERAGE: 181.79, 181.07, 178.72 TASKS: 1119 ... RELEASE: 4.18.0-553.170.1.el8_10.x86_64 PANIC: "Kernel panic - not syncing: NMI: Not continuing" crash> ps -S RU: 11 UN: 181 IN: 759 ID: 142 PA: 8 ZO: 18 crash> foreach UN bt | awk '/#[2-4] /{print $3,$5}' | paste - - - | sort | uniq -c | sort -nr 167 schedule_preempt_disabled ffffffff90c2277a __mutex_lock ffffffff90c24409 task_cgroup_path ffffffff903c276a 7 schedule_preempt_disabled ffffffff90c2277a __mutex_lock ffffffff90c24409 proc_cgroup_show ffffffff903c74fa 4 schedule_preempt_disabled ffffffff90c2277a __mutex_lock ffffffff90c24409 cgroup_path_ns ffffffff903c26e3 2 schedule_preempt_disabled ffffffff90c2277a __mutex_lock ffffffff90c24409 cgroup_kn_lock_live ffffffff903c58b0 1 schedule_preempt_disabled ffffffff90c2277a rwsem_down_write_slowpath ffffffff9035e1d0 down_write ffffffff90c24ca9 crash> foreach UN ps -ms | sort | tail [ 0 08:32:41.948] [UN] PID: 5249 TASK: ffff968125c34000 CPU/NUMA: 6/0 COMMAND: "app-A" [ 0 08:32:52.337] [UN] PID: 5247 TASK: ffff968119fe4000 CPU/NUMA: 1/0 COMMAND: "app-A" [ 0 08:33:05.533] [UN] PID: 5231 TASK: ffff968d24610000 CPU/NUMA: 5/0 COMMAND: "app-A" [ 0 08:33:05.539] [UN] PID: 5227 TASK: ffff9681127f0000 CPU/NUMA: 7/0 COMMAND: "app-A" [ 0 08:33:05.641] [UN] PID: 5226 TASK: ffff9683650bc000 CPU/NUMA: 7/0 COMMAND: "app-A" [ 0 08:33:06.016] [UN] PID: 635 TASK: ffff968122444000 CPU/NUMA: 7/0 COMMAND: "systemd-journal" [ 0 08:33:06.125] [UN] PID: 5160 TASK: ffff96812500c000 CPU/NUMA: 7/0 COMMAND: "app-A" [ 0 08:33:06.137] [UN] PID: 5174 TASK: ffff96811227c000 CPU/NUMA: 5/0 COMMAND: "app-A" [ 0 08:33:06.143] [UN] PID: 129036 TASK: ffff96811b2dc000 CPU/NUMA: 1/0 COMMAND: "app-B" [ 0 08:33:06.323] [UN] PID: 1 TASK: ffff9681067dc000 CPU/NUMA: 7/0 COMMAND: "systemd" The system had been hung for 8 hours 33 minutes when the vmcore was taken by NMI. 180 tasks wait for a mutex in cgroup functions, and 1 task waits for an rwsem for write. cgroup_mutex is held by systemd, which waits for shrinker_rwsem crash> p &cgroup_mutex.wait_list $8 = (struct list_head *) 0xffffffff9203a850 crash> list -o mutex_waiter.list -s mutex_waiter.task -H 0xffffffff9203a850 | paste - - | wc -l 180 crash> p cgroup_mutex.owner -x $7 = { counter = 0xffff9681067dc001 } PID: 1 TASK: ffff9681067dc000 CPU/NUMA: 7/0 COMMAND: "systemd" #0 [ffffba71c624fc70] __schedule at ffffffff90c21e31 #1 [ffffba71c624fcc8] schedule at ffffffff90c22425 #2 [ffffba71c624fce0] schedule_preempt_disabled at ffffffff90c2277a #3 [ffffba71c624fce8] rwsem_down_write_slowpath at ffffffff9035e1d0 #4 [ffffba71c624fd80] down_write at ffffffff90c24ca9 #5 [ffffba71c624fd90] alloc_shrinker_info at ffffffff904ba4ff #6 [ffffba71c624fdb8] mem_cgroup_css_online at ffffffff9054dc0f #7 [ffffba71c624fdd0] online_css at ffffffff903be32c #8 [ffffba71c624fde8] cgroup_apply_control_enable at ffffffff903c468d #9 [ffffba71c624fe48] cgroup_mkdir at ffffffff903c6d02 #10 [ffffba71c624fe98] kernfs_iop_mkdir at ffffffff9060a29a #11 [ffffba71c624feb0] vfs_mkdir at ffffffff9057ea8b #12 [ffffba71c624fee0] do_mkdirat at ffffffff9058208c #13 [ffffba71c624ff38] do_syscall_64 at ffffffff90203d1b #14 [ffffba71c624ff50] entry_SYSCALL_64_after_hwframe at ffffffff90e0012e RIP: 00007f0638225dcb RSP: 00007ffcd7618a48 RFLAGS: 00000206 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0638225dcb RDX: 00007ffcd7618910 RSI: 00000000000001ed RDI: 0000558dc680be90 RBP: 00007ffcd7618a90 R8: 0000000000000022 R9: 002f70756f726763 R10: 0000000000000100 R11: 0000000000000206 R12: 0000558dc6718bd0 R13: 00000000000000f3 R14: 00007f0639ab2efe R15: 0000000000000020 ORIG_RAX: 0000000000000053 CS: 0033 SS: 002b crash> gdb f 8 #8 0xffffffff904ba4ff in alloc_shrinker_info (memcg=memcg@entry=0xffff968005289000) at mm/vmscan.c:286 286 down_write(&shrinker_rwsem); crash> list -H 0xffffffff920579c8 -o rwsem_waiter.list -s rwsem_waiter.task,type | paste - - - ffffba71c624fd10 task = 0xffff9681067dc000, type = RWSEM_WAITING_FOR_WRITE, The 180 tasks are all on the wait list of cgroup_mutex. cgroup_mutex is held by systemd, which is creating a memory cgroup in cgroup_mkdir() and
Mitigation for CVE-2026-99288 in OpenShift Container Platform
Mitigation for CVE-2026-99288 in OpenShift Container Platform Red Hat OpenShift Container Platform (RHOCP) 4 OpenShift Container Platform is confirmed as affected by CVE-2026-99288 How to mitigate this vulnerability in OpenShift until the fix is implemented? Important: This procedure implements a workaround of disabling SYN cookies and should be reverted once a fixed kernel is installed. Doing so removes the node's protection against SYN floods; if oc debug node/ -q -- chroot /host nstat -az | grep SyncookiesSent returns a non-zero, growing value, that node is actively relying on SYN cookies and will drop connections instead once they are disabled. The second mitigation "reducing the interface to a single RX queue and disabling RPS/RFS." Is not covered here as serializing receive processing to one queue could bottleneck OVN-Kubernetes traffic and can destabilize the cluster. net.ipv4.tcp_syncookies is scoped per network namespace. Apply part 1 for the host network namespace, and part 2 for workloads exposed through NodePort or LoadBalancer services, net.* sysctls cannot be set on pods using hostNetwork: true. Part 1 — Host network namespace; (Node Tuning Operator) Covers the Ingress Controller (If host networked) when it runs on the host network, and the Kubernetes API server endpoint on the control plane nodes. apiVersion: tuned.openshift.io/v1 kind: Tuned metadata: name: cve-2026-99288-mitigation namespace: openshift-cluster-node-tuning-operator spec: profile: - name: cve-2026-99288-control-plane data: | [main] summary=Mitigation for CVE-2026-99288 (disable SYN cookies) - control plane include=openshift-control-plane [sysctl] net.ipv4.tcp_syncookies=0 - name: cve-2026-99288-worker data: | [main] summary=Mitigation for CVE-2026-99288 (disable SYN cookies) - worker include=openshift-node [sysctl] net.ipv4.tcp_syncookies=0 recommend: - match: - label: node-role.kubernetes.io/control-plane - label: node-role.kubernetes.io/infra priority: 10 profile: cve-2026-99288-control-plane - priority: 20 profile: cve-2026-99288-worker Apply the profile. A node restart is no required for the settings to take affect. $ oc apply -f cve-2026-99288-tuned.yaml Verify the profile applied. $ oc get profile -n openshift-cluster-node-tuning-operator NAME TUNED APPLIED DEGRADED MESSAGE master-0 cve-2026-99288-control-plane True False TuneD profile applied. worker-0 cve-2026-99288-worker True False TuneD profile applied. $ oc debug node/ -q -- chroot /host /bin/bash -c 'cat /proc/sys/net/ipv4/tcp_syncookie' 0 To revert: $ oc delete tuned cve-2026-99288-mitigation -n openshift-cluster-node-tuning-operator Nodes return to their previous profile without a reboot. Part 2 — Pod network namespaces Pods reached through NodePort or LoadBalancer services terminate external TCP connections in their own network namespace and are not covered by part 1. net.ipv4.tcp_syncookies is classified by Kubernetes as a safe sysctl, Add the following to the pod template of the affected workload: spec: securityContext: sysctls: - name: net.ipv4.tcp_syncookies value: "0" Verify: $ oc exec -n -- cat /proc/sys/net/ipv4/tcp_syncookies 0 This must be applied to each affected workload and will not apply to workloads created later. Reverting after a fixed kernel is released Delete the Tuned resource created in part 1. Remove the securityContext.sysctls stanza from any workload modified in part 2. Confirm the value has returned to 1 in both the host and pod network namespaces: $ oc debug node/ -q -- chroot /host cat /proc/sys/net/ipv4/tcp_syncookies $ oc exec -n -- cat /proc/sys/net/ipv4/tcp_syncookies An unauthenticated remote attacker can trigger an immediate operating system crash across the network without requiring privileges. In standard Red Hat Enterprise Linux deployments, TCP networking is universally enabled, making any system with active network listeners susceptible to high-volume socket recycling traffic. While this flaw causes a kernel panic that disrupts host availability, it does not allow data alteration or arbitrary code execution. 1. Confirm SYN cookies are enabled on the node at the host level $ oc debug node/ -q -- chroot /host cat /proc/sys/net/ipv4/tcp_syncookies 1 2. Confirm the setting is not inherited by other network namespaces First identify a pod to read from, and the node it is running on: $ oc get pods -n -o wide NAME READY STATUS RESTARTS AGE IP NODE ... my-app-7d9f8b6c4-xk2pq 1/1 Running 0 6d 10.131.0.42 worker-0 ... Set the value to 0 in the host network namespace of that node: $ oc debug node/worker-0 -q -- chroot /host sysctl -w net.ipv4.tcp_syncookies=0 net.ipv4.tcp_syncookies = 0 While the node is set to 0, read the value from the pod on that same node: $ oc exec -n my-app-7d9f8b6c4-xk2pq -- cat /proc/sys/net/ipv4/tcp_syncookies 1 The pod still reports 1. The kernel initializes net.ipv4.tcp_syncookies to 1 in every newly created network namespace, and the value is never inherited from the host. Restore the node to its default and confirm: $ oc debug node/worker-0 -q -- chroot /host sysctl -w net.ipv4.tcp_syncookies=1 net.ipv4.tcp_syncookies = 1 $ oc debug node/worker-0 -q -- chroot /host cat /proc/sys/net/ipv4/tcp_syncookies 1 Setting the sysctl on the node covers onlythe host network namespace — which includes the Ingress Controller when ituses host networking, and the Kubernetes API server endpoint. Every pod on the cluster SDN keeps net.ipv4.tcp_syncookies=1 regardless. Pods reached from outside the cluster through a NodePort or LoadBalancer service terminate external TCP connections in their own network namespace and therefore require the separate, pod-level change described in part 2 of the Resolution. Mitigation for CVE-2026-99288 in OpenShift Container Platform
Verify and Remediate OCPBUGS-62985 in ROSA HCP
Verify and Remediate OCPBUGS-62985 in ROSA HCP OpenShift Hosted Control Plane (HCP) 4.17+ Red Hat OpenShift Service on AWS (ROSA) In ROSA HCP clusters, a race condition in the Cluster API (CAPI) controller can lead to creation and running of excessive Amazon EC2 instances than required by the worker nodes. This script provides the instructions to detect such leaked EC2 instances until the fix for the identified bug (OCPBUGS-62985) is available in the cluster through a cluster upgrade. Perform the steps outlined in this article to identify if there are any orphaned EC2 instance. If your audit identifies any EC2 instance, reach out to Red Hat support to verify that there are no associated active nodes for those EC2 instances. Upon confirmation from Red Hat, you can remove the EC2 instances. Upgrade your cluster to the OpenShift version(s) to fix the OCPBUGS-62985: 4.22.12, 4.21.39, 4.20.46, 4.19.55, 4.18.60 Issue reported in https://redhat.atlassian.net/browse/OCPBUGS-62985 can lead to orphaned EC2 instances without an associated worker node in the ROSA HCP cluster. 1. Gather the cluster and AWS values Use the ROSA cluster name, cluster ID, worker region, and a read-only AWS profile for the account that contains the worker instances. export CLUSTER_ID={your_cluster_id} export CLUSTER_NAME={your_cluster_name} export AWS_REGION={cluster_region} Optionally, if you use different AWS profiles, configure them: export ROSA_PROFILE={your_rosa_profile} export AUDIT_AWS_PROFILE={optional_audit_aws_profile} 2. Log into your cluster Cluster-admin access is not required. The OpenShift identity needs permission to list Nodes and Certificate Signing Requests (CSRs). It also needs permission to list NodeClaims if Red Hat build of Karpenter is enabled on 4.22+ clusters. 3. Verify the cluster context and AWS account The AWS identity needs ec2:DescribeInstances; the script uses AWS STS to display the account ID. rosa describe cluster --cluster="$CLUSTER_NAME" –profile=”$ROSA_PROFILE" -o json oc config current-context oc whoami --show-server aws sts get-caller-identity --profile "$AUDIT_AWS_PROFILE" 4. Audit the environment and cluster: Download the attached Shell script "rosa-node-ec2-audit.sh". #!/usr/bin/env bash # Read-only diagnostic for ROSA worker nodes and their EC2 instances. # Run with --help to see access requirements and options. set -euo pipefail usage() { cat <<'EOF' Usage: bash rosa-node-ec2-audit.sh --cluster-id --region [--startup-grace-seconds ] Optional: Set AWS_PROFILE to select the AWS CLI profile. Without it, the AWS CLI default credential chain is used. Unmatched EC2 instances younger than the startup grace (default 300 seconds) are listed in a WARNING because they may still be bootstrapping or registering. Instances at or beyond the grace, and those with unknown launch age, are listed separately. Set it to 0 for a raw diff. Before running, log in to the ROSA cluster you want to check and set KUBECONFIG to that cluster's kubeconfig. Install the OpenShift CLI (oc) if needed. The ROSA CLI can also download it with: rosa download oc Get the cluster ID and region from: rosa describe cluster --cluster= Required tools: bash, oc connected to the target cluster, AWS CLI v2, and standard Unix utilities. Cluster access is read-only; cluster-admin is not required. The logged-in user needs cluster-scoped list permission for: core/nodes certificates.k8s.io/certificatesigningrequests karpenter.sh/nodeclaims (only if that API is installed) AWS access is required for the EC2-to-Node diff. The identity needs: ec2:DescribeInstances on Resource "*" The AWS account ID is displayed using AWS STS; this call requires no additional IAM permission. The AWS query is limited to the supplied region and EC2 instances with both red-hat-managed=true and api.openshift.com/id=. EOF } cluster_id="" region="" startup_grace_seconds=300 while [[ "$#" -gt 0 ]]; do case "$1" in --cluster-id) [[ "$#" -ge 2 ]] || { usage >&2; exit 2; } cluster_id="$2" shift 2 ;; --region) [[ "$#" -ge 2 ]] || { usage >&2; exit 2; } region="$2" shift 2 ;; --startup-grace-seconds) [[ "$#" -ge 2 ]] || { usage >&2; exit 2; } startup_grace_seconds="$2" shift 2 ;; -h|--help) usage exit 0 ;; *) printf 'Unknown argument: %s\n\n' "$1" >&2 usage >&2 exit 2 ;; esac done if [[ -z "$cluster_id" || -z "$region" ]]; then echo 'Both --cluster-id and --region are required.' >&2 usage >&2 exit 2 fi if [[ ! "$startup_grace_seconds" =~ ^[0-9]+$ ]]; then echo 'Startup grace must be a non-negative integer number of seconds.' >&2 exit 2 fi for command in oc aws sort comm grep mktemp cat date sed; do if ! command -v "$command" >/dev/null 2>&1; then printf 'Required command not found: %s\n' "$command" >&2 exit 1 fi done timestamp_to_epoch() { local timestamp="$1" local normalized normalized=$(printf '%s\n' "$timestamp" | sed -E 's/([+-][0-9]{2}):([0-9]{2})$/\1\2/') date -j -f '%Y-%m-%dT%H:%M:%S%z' "$normalized" +%s 2>/dev/null || date -d "$timestamp" +%s 2>/dev/null } if ! oc whoami >/dev/null 2>&1; then echo 'No working oc login found. Log in to the target cluster and retry.' >&2 exit 1 fi tmp_dir=$(mktemp -d) trap 'rm -rf "$tmp_dir"' EXIT context=$(oc config current-context 2>/dev/null || true) server=$(oc whoami --show-server 2>/dev/null || true) if [[ -z "$context" ]]; then context=unknown; fi if [[ -z "$server" ]]; then server=unknown; fi printf 'Context: %s\nAPI server: %s\nROSA cluster ID: %s\nAWS region: %s\n' \ "$context" "$server" "$cluster_id" "$region" echo 'Read-only collection; no cluster or AWS resources will be changed.' if ! aws_account=$(aws sts get-caller-identity --query Account --output text 2>"$tmp_dir/aws-identity.err"); then cat "$tmp_dir/aws-identity.err" >&2 echo 'Could not identify the AWS caller. Check AWS_PROFILE or the AWS credential chain.' >&2 exit 1 fi if [[ -z "$aws_account" || "$aws_account" == None ]]; then aws_account=unknown; fi printf 'AWS account: %s\n' "$aws_account" if ! nodes=$(oc get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"|"}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"|"}{.spec.providerID}{"|"}{.metadata.creationTimestamp}{"|"}{.metadata.labels.hypershift\.openshift\.io/nodePool}{"|"}{.metadata.labels.karpenter\.sh/nodepool}{"\n"}{end}' 2>"$tmp_dir/nodes.err"); then cat "$tmp_dir/nodes.err" >&2 echo 'Unable to list cluster nodes. Check that your account can list core/nodes.' >&2 exit 1 fi echo echo 'Nodes' printf 'NAME\tREADY\tPROVIDER_ID\tCREATED\n' : > "$tmp_dir/node-ids" : > "$tmp_dir/nodes-without-pool-labels" while IFS='|' read -r name ready provider_id created hypershift_pool karpenter_pool; do [[ -n "$name" ]] || continue [[ -n "$ready" ]] || ready=Unknown [[ -n "$provider_id" ]] || provider_id=- printf '%s\t%s\t%s\t%s\n' "$name" "$ready" "$provider_id" "$created" if [[ -z "$hypershift_pool" && -z "$karpenter_pool" ]]; then printf '%s\t%s\t%s\n' "$name" "$provider_id" "$created" >> "$tmp_dir/nodes-without-pool-labels" fi if [[ "$provider_id" == aws:///* ]]; then instance_id=${provider_id##*/} [[ "$instance_id" == i-* ]] && printf '%s\n' "$instance_id" >> "$tmp_dir/node-ids" fi done <<< "$nodes" sort -u -o "$tmp_dir/node-ids" "$tmp_dir/node-ids" if ! csrs=$(oc get csr -o=jsonpath='{range .items[*]}{.metadata.name}{"|"}{.metadata.creationTimestamp}{"|"}{.spec.signerName}{"|"}{.spec.username}{"|"}{range .status.conditions[*]}{.type}{","}{end}{"\n"}{end}' 2>"$tmp_dir/csrs.err"); then cat "$tmp_dir/csrs.err" >&2 echo 'Unable to list certificate requests. Check that your account can list certificatesigningrequests.certificates.k8s.io.' >&2 exit 1 fi echo echo 'Kubelet certificate requests not approved (includes denied requests)' printf 'NAME\tCREATED\tSIGNER\tREQUESTOR\tCONDITIONS\n' unapproved_csrs=0 while IFS='|' read -r name created signer requestor conditions; do [[ -n "$name" ]] || continue case "$signer" in kubernetes.io/kube-apiserver-client-kubelet|kubernetes.io/kubelet-serving) ;; *) continue ;; esac conditions=${conditions%,} case ",$conditions," in *,Approved,*) continue ;; esac [[ -n "$conditions" ]] || conditions=Pending [[ -n "$requestor" ]] || requestor=- printf '%s\t%s\t%s\t%s\t%s\n' "$name" "$created" "$signer" "$requestor" "$conditions" unapproved_csrs=$((unapproved_csrs + 1)) done <<< "$csrs" if (( unapproved_csrs == 0 )); then echo 'No unapproved kubelet CSRs found.' fi : > "$tmp_dir/nodeclaim-ids" if claims=$(oc get nodeclaims.karpenter.sh -o=jsonpath='{range .items[*]}{.metadata.name}{"|"}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"|"}{.status.providerID}{"|"}{.metadata.creationTimestamp}{"\n"}{end}' 2>"$tmp_dir/nodeclaims.err"); then echo echo 'Karpenter NodeClaims' printf 'NAME\tREADY\tPROVIDER_ID\tCREATED\n' while IFS='|' read
Token-based authentication for fence_vmware_rest
Token-based authentication for fence_vmware_rest Red Hat Enterprise Linux Server (RHEL) with the High Availability or Resilient Storage Add-Ons pacemaker cluster fence-agents-vmware-rest Is there token-based authentication for VMware fence devices? Do I need to use a password on the fence_vmware_rest device? Can we have a passwordless fence_vmware_rest device? Red Hat Enterprise Linux 10 This RFE (Feature Enhancement) was tracked with the Jira RHEL-191488, and released with RHBA-2026:97258: fence-agents-vmware-rest-4.16.0-34.el10 Red Hat Enterprise Linux 9 This RFE (Feature Enhancement) was tracked with the Jira RHEL-193739, and released with RHBA-2026:97569: fence-agents-vmware-rest-4.10.0-122.el9 Red Hat Enterprise Linux 8 It won't be backported to RHEL 8, as this major release is not in its Full Support Phase since June 1st, 2024. Usage steps: The token_script option is now available for usage with fence_vmware_rest: # fence_vmware_rest --help Usage: fence_vmware_rest [options] Options: -a, --ip=[ip] IP address or hostname of fencing device -l, --username=[name] Login name ... ... --token-script=[script] Script to retrieve a token Steps to implement this option: Create a token fetcher script and any supplemental files for fetching token. Below is an example script, but your own custom script can be used: $ vim /usr/local/bin/fetch_token.sh #!/bin/bash set -e VCENTER="vcenter.example.com" CREDS_FILE="/etc/pacemaker/vcenter_creds.txt" API_URL="https://${VCENTER}/rest/com/vmware/cis/session" if [ ! -f "$CREDS_FILE" ]; then echo "Error: Credentials file $CREDS_FILE not found." >&2 exit 1 fi CREDS=$(cat "$CREDS_FILE") RESPONSE=$(curl -s -k -X POST -u "$CREDS" "$API_URL") TOKEN=$(echo "$RESPONSE" | jq -r '.value') if [ -z "$TOKEN" ] || [ "$TOKEN" == "null" ]; then echo "Error: Failed to fetch token. vCenter responded with: $RESPONSE" >&2 exit 1 fi # Output token: echo "$TOKEN" # Script ends Here# # Credentials file for POST API call to vcenetr. $ vim /etc/pacemaker/vcenter_creds.txt @vcenter.example.com: Secure permissions as needed: Same as with password_script option, secure permissions can be used: $ chmod 700 /usr/local/bin/fetch_token.sh $ chmod 600 /etc/pacemaker/vcenter_creds.txt Test manually as needed: # Manual run of script returns a token: $ /usr/local/bin/fetch_token.sh 752bda7217a51118a10cfdac686fff00 # Power off and confirm status test: $ fence_vmware_rest -a vcenter.example.com --ssl-insecure -o status -n rhel9-vm1 --token-script="/usr/local/bin/fetch_token.sh" Status: ON $ fence_vmware_rest -a vcenter.example.com --ssl-insecure -o off -n rhel9-vm1 --token-script="/usr/local/bin/fetch_token.sh" Success: Powered OFF $ fence_vmware_rest -a vcenter.example.com --ssl-insecure -o status -n rhel9-vm1 --token-script="/usr/local/bin/fetch_token.sh" Status: OFF Update the stonith config to include the new option or create stonith with option: If token_script is used, then normal credentials ( login and passwd ) should not be used. $ pcs stonith update vmfence token_script="/usr/local/bin/fetch_token.sh" login= passwd= Accordingly with Vmware documentation for “Authentication with vSphere APIs”, Vmware Rest API can use Token-based authentication for login. Token-based authentication for fence_vmware_rest
The replaying of a journal for a gfs2 filesystem was delayed after a cluster node was successfully fenced.
The replaying of a journal for a gfs2 filesystem was delayed after a cluster node was successfully fenced. Red Hat Enterprise Linux Server 6, 7 (with the High Availability Add On and Resilient Storage Add Ons) A Global Filesystem 2(gfs2) The replaying of a journal for a gfs2 filesystem was delayed after a cluster node was successfully fenced. The delay in starting the journal replay is not an indication of a problem with the gfs2 filesystem or its data. After a cluster node is successfully fenced, the dlm must finish recovering the lock space before the gfs2 journal locks can be acquired. Once they can be, each of the fenced node's gfs2 journals is replayed by exactly one of the surviving cluster nodes: the surviving nodes race to acquire the journal lock, and the nodes that lose the race report that the journal is "Busy" and leave the recovery to the winning node. The length of the delay before the replay starts depends on the number of dlm locks and the amount of cluster activity, so it can be several minutes on busy filesystems. Once the replay has started it completes correctly: no corruption or data loss occurs and no action is required. If a gfs2 filesystem was withdrawn (rather than the cluster node being fenced), or if corruption is found on the filesystem after the recovery, that is a different issue: A gfs2 filesystem was corrupted after a withdraw of the filesystem. Review the /var/log/messages file for a membership change that leads to a fencing event. The gfs2 filesystem will not start the replaying of a gfs2 filesystem's journal until the cluster node rejoins the cluster or the cluster node is successfully fenced off from the cluster. The log snippets below are from a RHEL 6 cluster using the rgmanager stack; the gfs2 journal replay messages are identical on clusters using pacemaker (RHEL 6 with pacemaker or RHEL 7 and later). In the snippet below we can see that a cluster node was successfully fenced (so replaying of the gfs2 journals should begin after that) and that the surviving cluster nodes are trying to acquire the journal locks in order to start replaying journals for the gfs2 filesystem. The replaying of the journal will not start until one of the cluster nodes is able to get a journal lock on the journal used by the evicted cluster node. The journal lock cannot be obtained until after the cluster node rejoins the cluster (after a restart) or fencing is successful. Jan 23 15:30:27 node317 corosync[4253]: [TOTEM ] A processor failed, forming new configuration. Jan 23 15:30:39 node317 kernel: dlm: closing connection to node 1 Jan 23 15:30:39 node317 rgmanager[19922]: State change: node316 DOWN Jan 23 15:30:39 node317 fenced[4420]: fencing node node316 Jan 23 15:30:39 node317 kernel: GFS2: fsid=mycluster:gfs2one.0: jid=2: Trying to acquire journal lock... Jan 23 15:30:39 node318 kernel: GFS2: fsid=mycluster:gfs2one.1: jid=2: Trying to acquire journal lock... Jan 23 15:30:53 node317 fenced[4420]: fence node316 success After the cluster node has been successfully fenced, the replaying of the gfs2 filesystem's journals can begin. One of the filesystems successfully replays its journal soon after the fencing is successful. Please note that only 1 cluster node will replay the journal of a gfs2 filesystem and the other cluster node will report that it is "busy". Jan 23 15:30:54 node317 kernel: GFS2: fsid=mycluster:gfs2one.0: jid=2: Looking at journal... Jan 23 15:30:54 node317 kernel: GFS2: fsid=mycluster:gfs2one.0: jid=2: Done Jan 23 15:30:54 node318 kernel: GFS2: fsid=mycluster:gfs2one.1: jid=2: Busy, retrying... Jan 23 15:30:55 node318 kernel: GFS2: fsid=mycluster:gfs2one.1: jid=2: Looking at journal... Jan 23 15:30:55 node318 kernel: GFS2: fsid=mycluster:gfs2one.1: jid=2: Done Then there is a gap of time between the first gfs2 filesystem starting and finishing the replaying of the journal of a gfs2 filesystem and the next gfs2 filesystem. Jan 23 15:30:39 node317 kernel: GFS2: fsid=mycluster:gfs2two.0: jid=2: Trying to acquire journal lock... Jan 23 15:35:34 node317 kernel: GFS2: fsid=mycluster:gfs2two.0: jid=2: Looking at journal... Jan 23 15:35:35 node317 kernel: GFS2: fsid=mycluster:gfs2two.0: jid=2: Acquiring the transaction lock... Jan 23 15:35:35 node317 kernel: GFS2: fsid=mycluster:gfs2two.0: jid=2: Replaying journal... Jan 23 15:35:37 node317 kernel: GFS2: fsid=mycluster:gfs2two.0: jid=2: Replayed 93 of 6031 blocks Jan 23 15:35:37 node317 kernel: GFS2: fsid=mycluster:gfs2two.0: jid=2: Found 1441 revoke tags Jan 23 15:35:37 node317 kernel: GFS2: fsid=mycluster:gfs2two.0: jid=2: Journal replayed in 3s Jan 23 15:35:37 node317 kernel: GFS2: fsid=mycluster:gfs2two.0: jid=2: Done Jan 23 15:30:39 node318 kernel: GFS2: fsid=mycluster:gfs2two.1: jid=2: Trying to acquire journal lock... Jan 23 15:35:34 node318 kernel: GFS2: fsid=mycluster:gfs2two.1: jid=2: Busy, retrying... Jan 23 15:35:35 node318 kernel: GFS2: fsid=mycluster:gfs2two.1: jid=2: Busy, retrying... Jan 23 15:35:36 node318 kernel: GFS2: fsid=mycluster:gfs2two.1: jid=2: Busy, retrying... Jan 23 15:35:37 node318 kernel: GFS2: fsid=mycluster:gfs2two.1: jid=2: Busy, gave up. The third gfs2 filesystem was delayed as well: Jan 23 15:30:54 node317 kernel: GFS2: fsid=mycluster:gfs2three.0: jid=2: Trying to acquire journal lock... Jan 23 15:35:34 node317 kernel: GFS2: fsid=mycluster:gfs2three.0: jid=2: Looking at journal... Jan 23 15:35:34 node317 kernel: GFS2: fsid=mycluster:gfs2three.0: jid=2: Acquiring the transaction lock... Jan 23 15:35:34 node317 kernel: GFS2: fsid=mycluster:gfs2three.0: jid=2: Replaying journal... Jan 23 15:35:34 node317 kernel: GFS2: fsid=mycluster:gfs2three.0: jid=2: Replayed 0 of 0 blocks Jan 23 15:35:34 node317 kernel: GFS2: fsid=mycluster:gfs2three.0: jid=2: Found 0 revoke tags Jan 23 15:35:34 node317 kernel: GFS2: fsid=mycluster:gfs2three.0: jid=2: Journal replayed in 0s Jan 23 15:35:34 node317 kernel: GFS2: fsid=mycluster:gfs2three.0: jid=2: Done Jan 23 15:30:55 node318 kernel: GFS2: fsid=mycluster:gfs2three.1: jid=2: Trying to acquire journal lock... Jan 23 15:35:34 node318 kernel: GFS2: fsid=mycluster:gfs2three.1: jid=2: Busy, retrying... Jan 23 15:35:35 node318 kernel: GFS2: fsid=mycluster:gfs2three.1: jid=2: Looking at journal... Jan 23 15:35:35 node318 kernel: GFS2: fsid=mycluster:gfs2three.1: jid=2: Done There is nothing found in the logs that explains why there was a delay in the second gfs2 filesystem from starting its replay after the cluster node was successfully fenced: the dlm lock space recovery that must complete before the journal lock can be acquired is not logged by default. If fencing did not complete successfully then you are not hitting this issue. The replaying of a journal for a gfs2 filesystem was delayed after a cluster node was successfully fenced.
A kernel panic occurred after a gfs2 filesystem had a withdraw on RHEL 7
A kernel panic occurred after a gfs2 filesystem had a withdraw on RHEL 7 Red Hat Enterprise Linux Server 6, 7 (with the High Availability Add On and Resilient Storage Add Ons) A Global Filesystem 2(gfs2) A kernel panic occurred after a gfs2 filesystem had a withdraw on RHEL 7. [30165.991660] sd 7:0:3:3: Parameters changed [30166.021772] sd 9:0:0:6: Parameters changed [30166.022293] sd 7:0:3:6: Parameters changed [30166.028221] sd 7:0:3:6: reservation conflict [30166.029949] scsi_io_completion: 64 callbacks suppressed [30166.031156] sd 9:0:0:6: reservation conflict [30166.031161] blk_update_request: critical nexus error, dev sdbh, sector 94346824 [30166.031181] blk_update_request: critical nexus error, dev dm-62, sector 94346824 [30166.031626] sd 9:0:3:6: Parameters changed [30166.037216] sd 9:0:3:6: reservation conflict [30166.037221] blk_update_request: critical nexus error, dev sdej, sector 73900672 [30166.037235] blk_update_request: critical nexus error, dev dm-62, sector 73900672 [30166.037242] GFS2: fsid=mycluster:myfs.1: Error -52 writing to journal, jid=1 [30166.037250] GFS2: fsid=mycluster:myfs.1: error -52: withdrawing the file system to prevent further damage. [30166.037251] GFS2: fsid=mycluster:myfs.1: about to withdraw this file system [30166.037676] sd 7:0:2:6: Parameters changed [30166.044159] sd 7:0:2:6: reservation conflict [30166.044165] blk_update_request: critical nexus error, dev sdar, sector 73900696 [30166.044182] blk_update_request: critical nexus error, dev dm-62, sector 73900696 [30166.044188] GFS2: fsid=mycluster:myfs.1: Error -52 writing to journal, jid=1 [30166.044505] sd 9:0:0:6: reservation conflict [.....] [30166.351079] sd 7:0:2:6: [sdar] tag#2 CDB: Write(10) 2a 00 00 a5 ea 00 00 04 00 00 [30166.351084] GFS2: fsid=mycluster:myfs_data.1: Error -52 writing to journal, jid=1 [30166.351092] GFS2: fsid=mycluster:myfs_data.1: error -52: withdrawing the file system to prevent further damage. [30166.351094] GFS2: fsid=mycluster:myfs_data.1: about to withdraw this file system [30166.351142] sd 7:0:2:6: reservation conflict [.....] [30166.582493] sd 9:0:0:6: reservation conflict [30166.582499] sd 9:0:3:6: reservation conflict [30166.582789] sd 7:0:2:6: reservation conflict [30166.695838] GFS2: fsid=mycluster:myfs_data.1: gfs2_evict_inode: -5 [30166.882047] GFS2: fsid=mycluster:myfs_data.1: gfs2_evict_inode: -5 [30166.948853] GFS2: fsid=mycluster:myfs_data.1: gfs2_evict_inode: -5 [30166.949259] G: s:EX n:2/335518 f:yIqb t:EX d:EX/0 a:1 v:0 r:-128 m:200 [30166.949616] ------------[ cut here ]------------ [30166.949917] kernel BUG at fs/gfs2/glock.c:250! [30166.950250] invalid opcode: 0000 [#1] SMP [30166.950591] Modules linked in: gfs2 dlm secvm2(POE) secfs2(POE) mptctl mptbase rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache team_mode_activebackup team dm_round_robin sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel dm_multipath lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr mgag200 i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm i2c_i801 hpilo lpc_ich drm_panel_orientation_quirks hpwdt ioatdma dca sg wmi ipmi_si ipmi_devintf ipmi_msghandler [30166.954775] pcc_cpufreq acpi_power_meter binfmt_misc auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic qla2xxx bnx2x ahci libahci libata crct10dif_pclmul crct10dif_common crc32c_intel tg3 serio_raw nvme_fc hpsa nvme_fabrics nvme_core mdio libcrc32c scsi_transport_fc ptp scsi_transport_sas pps_core scsi_tgt dm_mirror dm_region_hash dm_log dm_mod [30166.957168] CPU: 23 PID: 7081 Comm: transform Kdump: loaded Tainted: P OE ------------ 3.10.0-957.21.3.el7.x86_64 #1 [30166.958317] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 10/17/2018 [30166.958929] task: ffff9d977a2330c0 ti: ffff9d97a1728000 task.ti: ffff9d97a1728000 [30166.959566] RIP: 0010:[] [] __gfs2_glock_put+0xdb/0xf0 [gfs2] [30166.960239] RSP: 0018:ffff9d97a172bcc8 EFLAGS: 00010296 [30166.960917] RAX: 0000000000000000 RBX: ffff9db21ef497a8 RCX: 0000000000000006 [30166.961626] RDX: ffffffffc104a500 RSI: ffff9db21ef495f0 RDI: 0000000000000000 [30166.962562] RBP: ffff9d97a172bce8 R08: 0000000000000000 R09: ffff9db274e26f00 [30166.963282] R10: 000000000000292d R11: 0000000000000001 R12: ffff9db21ef495f0 [30166.964006] R13: ffff9dd1e2340000 R14: ffff9db21ef49620 R15: ffff9dd03ef45160 [30166.965019] FS: 00007f7902d70740(0000) GS:ffff9db27fdc0000(0000) knlGS:0000000000000000 [30166.965909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30166.966680] CR2: 00007f96d55c61d4 CR3: 0000002d1e37a000 CR4: 00000000001607e0 [30166.967481] Call Trace: [30166.968292] [] gfs2_glock_put+0x22/0x30 [gfs2] [30166.969121] [] gfs2_holder_uninit+0x1f/0x40 [gfs2] [30166.969947] [] gfs2_glock_dq_uninit+0x1a/0x20 [gfs2] [30166.970797] [] gfs2_create_inode+0x24e/0xe50 [gfs2] [30166.971643] [] ? gfs2_create_inode+0x87c/0xe50 [gfs2] [30166.972502] [] gfs2_mknod+0x35/0x40 [gfs2] [30166.973372] [] vfs_mknod+0xe7/0x160 [30166.974253] [] SyS_mknodat+0x184/0x230 [30166.975130] [] ? SyS_mknodat+0x230/0x230 [30166.976013] [] SyS_mknod+0x1d/0x20 [30166.976943] [] m_sys_mknod+0x41/0x50 [secfs2] [30166.977852] [] system_call_fastpath+0x22/0x27 [30166.978774] Code: 48 8b 7b 08 48 83 c3 10 4c 89 e6 e8 f0 e7 33 ed 48 8b 03 48 85 c0 75 e8 eb b0 48 83 7b 60 00 74 a4 4c 89 e6 31 ff e8 65 f6 ff ff <0f> 0b 0f 1f 00 4c 89 e6 31 ff e8 56 f6 ff ff 0f 0b 0f 1f 40 00 [30166.980814] RIP [] __gfs2_glock_put+0xdb/0xf0 [gfs2] [30166.981812] RSP Red Hat Enterprise Linux 6 The issue 1618511 - A gfs2 withdrawal occurred: function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1432 was closed as WONT_FIX as the fix would require too many patches to resolve the issue in the current life cycle of RHEL 6 and a workaround exists shown below. Red Hat Enterprise Linux 7 The issue is being tracked with bugzilla 1364234: Bug 1364234 - A gfs2 withdrawal occurs function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 109 (RHEL 7). An errata was released to resolve this issue: RHSA-2022:0063 - Security Advisory. In addition to the above, this error is commonly seen with storage issues, such as the reservation conflict errors observed above. It may be worth checking for any storage side issues or fencing events with fence_scsi or fence_mpath, and preventing these kinds of errors may additionally help to avoid issues like this: 'blk_update_request: critical nexus error dev dm-X, sector xyz' messages logged reservation conflicts in a RHEL 7 High Availability cluster using fence_scsi after adding a new storage device to the cluster Recovery After a gfs2 withdrawal has occurred it is recommended that the filesystem is checked which will fix any corruption on the filesystem that it finds. For more information on this procedure then review the following: How do I recover from a gfs2 withdrawal and fix filesystem corruption on a Red Hat Enterprise Linux High Availability cluster? Workaround The corruption that occurs in this issue occurs after the gfs2 filesystem withdraws. This corruption can be avoided by allowing the gfs2 filesystem to panic instead of causing a withdraw. To cause a gfs2 filesystem to panic instead of withdraw then add the following to the mount options for the gfs2 filesystem. This should be done after a fsck.gfs2 mention above is done to fix the filesystem: errors="panic" For more information then see the following articles: RHEL 6: Global File System 2 - 4.14. The GFS2 Withdraw Function RHEL 7: Global File System 2 - 3.12. The GFS2 Withdraw Function Related Issues These issues are related to the gfs2 withdrawal and journal recovery problems described in this article, but they manifest with different signatures or symptoms: A gfs2 filesystem was corrupted after a withdraw of the filesystem A RHEL 6, 7 gfs2 filesystem had a withdraw: function = gfs2\_setbit, file = fs/gfs2/rgrp.c, line = 109 A gfs2 withdrawal occurred on RHEL 6, 7: fatal: filesystem consistency error function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1414 A kernel panic
A gfs2 withdrawal occurred on RHEL 6, 7: fatal: filesystem consistency error function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1414
A gfs2 withdrawal occurred on RHEL 6, 7: fatal: filesystem consistency error function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1414 Red Hat Enterprise Linux Server 6, 7 (with the High Availability Add On and Resilient Storage Add Ons) A Global Filesystem 2(gfs2) A gfs2 filesystem withdrawal occurred after an IO error on RHEL 7: Jul 23 01:06:11 node42 kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK Jul 23 01:06:11 node42 kernel: sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 00 08 1d 68 00 00 08 00 Jul 23 01:06:11 node42 kernel: end_request: I/O error, dev sdd, sector 531816 Jul 23 01:06:11 node42 kernel: GFS2: fsid=cluster2-gfs2:clslv.1: gfs2_quotad: statfs error -5 [....] Jul 23 10:30:39 node42 kernel: GFS2: fsid=cluster2-gfs2:clslv.1: fatal: filesystem consistency error Jul 23 10:30:39 node42 kernel: GFS2: fsid=cluster2-gfs2:clslv.1: inode = 721 145887 Jul 23 10:30:39 node42 kernel: GFS2: fsid=cluster2-gfs2:clslv.1: function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1432 Jul 23 10:30:39 node42 kernel: GFS2: fsid=cluster2-gfs2:clslv.1: about to withdraw this file system A gfs2 filesystem withdrawal occurred on RHEL 6: 2018-07-07T02:51:28.527586+02:00 node42 kernel - - - GFS2: fsid=wave_clust_04:db.0: fatal: filesystem consistency error 2018-07-07T02:51:28.527586+02:00 node42 kernel - - - GFS2: fsid=wave_clust_04:db.0: inode = 50371 12652009 2018-07-07T02:51:28.527586+02:00 node42 kernel - - - GFS2: fsid=wave_clust_04:db.0: function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1414 2018-07-07T02:51:28.527644+02:00 node42 kernel - - - GFS2: fsid=wave_clust_04:db.0: about to withdraw this file system 2018-07-07T02:51:28.530550+02:00 node42 kernel - - - GFS2: fsid=wave_clust_04:db.0: telling LM to unmount 2018-07-07T02:51:28.534719+02:00 node42 kernel - - - GFS2: fsid=wave_clust_04:db.0: withdrawn Red Hat Enterprise Linux 6 The issue 1618511 - A gfs2 withdrawal occurred: function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1432 was closed as WONT_FIX as the fix would require too many patches to resolve the issue in the current life cycle of RHEL 6 and a workaround exists shown below. Red Hat Enterprise Linux 7 Update to kernel-3.10.0-1160.53.1.el7 or later. The issue is being tracked with bugzilla 1618513: Bug 1618513 - A gfs2 withdrawal occurred: function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1432 (RHEL 7). As of Thu, August 16 2018, the status of 1618513 is CLOSED as duplicate of bz1364234. The issue is being tracked with bugzilla 1364234: Bug 1364234 - A gfs2 withdrawal occurs function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 109 (RHEL 7). As of Wed, January 12 2022, the status of 1364234 is CLOSED, and RHSA-2022:0063 has been released. Recovery After a gfs2 withdrawal has occurred it is recommended that the filesystem is checked which will fix any corruption on the filesystem that it finds. For more information on this procedure then review the following: How do I recover from a gfs2 withdrawal and fix filesystem corruption on a Red Hat Enterprise Linux High Availability cluster? Workaround The corruption that occurs in this issue occurs after the gfs2 filesystem withdraws. This corruption can be avoided by allowing the gfs2 filesystem to panic instead of causing a withdraw. To cause a gfs2 filesystem to panic instead of withdraw then add the following to the mount options for the gfs2 filesystem. This should be done after a fsck.gfs2 mention above is done to fix the filesystem: errors="panic" For more information then see the following articles: RHEL 6: Global File System 2 - 4.14. The GFS2 Withdraw Function RHEL 7: Global File System 2 - 3.12. The GFS2 Withdraw Function Related Issues These issues are related to the gfs2 withdrawal and journal recovery problems described in this article, but they manifest with different signatures or symptoms: A gfs2 filesystem was corrupted after a withdraw of the filesystem A RHEL 6, 7 gfs2 filesystem had a withdraw: function = gfs2\_setbit, file = fs/gfs2/rgrp.c, line = 109 A gfs2 withdrawal occurred on RHEL 6, 7: fatal: filesystem consistency error function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1414 A kernel panic occurred after a gfs2 filesystem had a withdraw on RHEL 7 The withdraw signature in logs is a valid gfs2 withdraw (gfs2 did detect some type of corruption or something incorrect) that occurred. When the withdraw occurs in some situations (and timing) more corruption can occur that was unintended. This is the reason that is you continue to hit this issue that a panic should be triggered instead of withdrawing. Run the following as root on the node where the file system became unavailable. A withdrawal is logged only on the node that detected the problem, so check the logs of every node that had the file system mounted. Check whether a file system is currently withdrawn on the node (run on each node). The file contains 1 if that file system has withdrawn and 0 if it has not. This only shows withdrawals that occurred since the last reboot: the withdrawn file system stays mounted, but unusable, until the node is rebooted, and after a reboot the entry either no longer exists (file system not mounted) or shows 0 again (file system freshly mounted) — a withdrawal from before the reboot is no longer visible. In that case, find the withdrawal in the logs instead (steps 2 and 3): # cat /sys/fs/gfs2/:/withdraw To check every gfs2 file system on the node at once: # for locktable in $(ls /sys/fs/gfs2/); do printf '%s: %s\n' "$locktable" "$(cat "/sys/fs/gfs2/$locktable/withdraw")"; done Search the kernel messages for the withdrawal report. The current boot, via journald: # journalctl -k | grep -iE 'gfs2.*(fatal|withdraw|assertion)' Via rsyslog: # grep -iE 'gfs2.*(fatal|withdraw|assertion)' /var/log/messages To narrow the output to a time window: # journalctl -k --since "YYYY-MM-DD HH:MM:SS" --until "YYYY-MM-DD HH:MM:SS" If the fsid is known, show every kernel message for that file system: # journalctl -k | grep -i 'fsid=:' # grep -i 'fsid=:' /var/log/messages If the node was rebooted or fenced after the withdrawal, dmesg no longer contains the report. Use the previous boot of the persistent journal (persistent by default on RHEL 7 and later; if /var/log/journal is missing, use /var/log/messages instead): # journalctl -k -b -1 | grep -iE 'gfs2.*(fatal|withdraw|assertion)' A gfs2 withdrawal occurred on RHEL 6, 7: fatal: filesystem consistency error function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1414
A RHEL 6, 7 gfs2 filesystem had a withdraw: function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 109
A RHEL 6, 7 gfs2 filesystem had a withdraw: function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 109 Red Hat Enterprise Linux Server 6, 7 (with the High Availability Add On and Resilient Storage Add Ons) A Global Filesystem 2(gfs2) A withdrawal occurred on a RHEL 7 gfs2 filesystem shortly after it was mounted at boot time: [ 1163.633594] GFS2 installed [ 1163.634124] GFS2: fsid=rh7_cluster:mydata: Trying to join cluster "lock_dlm", "rh7_cluster:mydata" [ 1164.138968] GFS2: fsid=rh7_cluster:mydata: first mounter control generation 0 [ 1164.138970] GFS2: fsid=rh7_cluster:mydata: Joined cluster. Now mounting FS... [ 1164.168104] GFS2: fsid=rh7_cluster:mydata.0: jid=0, already locked for use [ 1164.168106] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Looking at journal... [ 1164.201735] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Acquiring the transaction lock... [ 1164.201768] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Replaying journal... [ 1164.230452] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Replayed 39 of 47 blocks [ 1164.230454] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Found 5 revoke tags [ 1164.230860] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Journal replayed in 1s [ 1164.230875] GFS2: fsid=rh7_cluster:mydata.0: jid=0: Done [ 1164.230912] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Trying to acquire journal lock... [ 1164.231893] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Looking at journal... [ 1164.287488] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Acquiring the transaction lock... [ 1164.287523] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Replaying journal... [ 1164.290579] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Replayed 39 of 47 blocks [ 1164.290581] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Found 5 revoke tags [ 1164.291151] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Journal replayed in 1s [ 1164.291168] GFS2: fsid=rh7_cluster:mydata.0: jid=1: Done [ 1164.291203] GFS2: fsid=rh7_cluster:mydata.0: first mount done, others may mount [.....] [ 1187.852687] GFS2: buf_blk = 0x417 old_state=0, new_state=2 [ 1187.852690] GFS2: rgrp=0x1f0005 bi_start=0x0 [ 1187.852691] GFS2: bi_offset=0x80 bi_len=0xf80 [ 1187.852694] CPU: 0 PID: 12055 Comm: db2start Not tainted 3.10.0-327.13.1.el7.x86_64 #1 [ 1187.852695] Hardware name: Cisco Systems Inc UCSC-C240-M4S/UCSC-C240-M4S, BIOS C240M4.2.0.4a.0.042220151400 04/22/2015 [ 1187.852697] 00000000001f0422 00000000756d825d ffff881010a0fc80 ffffffff816356f4 [ 1187.852702] ffff881010a0fce0 ffffffffa07705e8 00000002577d53a5 ffff881ff76ab110 [ 1187.852705] ffff88201f6af200 0000000000000417 00000000756d825d ffff881005260dc0 [ 1187.852709] Call Trace: [ 1187.852718] [] dump_stack+0x19/0x1b [ 1187.852731] [] rgblk_free+0x1f8/0x230 [gfs2] [ 1187.852736] [] gfs2_unlink_di+0x37/0x100 [gfs2] [ 1187.852742] [] gfs2_unlink_inode+0x6e/0x76 [gfs2] [ 1187.852747] [] gfs2_unlink+0x219/0x220 [gfs2] [ 1187.852752] [] ? gfs2_unlink+0x90/0x220 [gfs2] [ 1187.852756] [] ? gfs2_unlink+0xb8/0x220 [gfs2] [ 1187.852759] [] ? gfs2_unlink+0xec/0x220 [gfs2] [ 1187.852764] [] vfs_unlink+0xe1/0x150 [ 1187.852767] [] do_unlinkat+0x26e/0x2b0 [ 1187.852771] [] ? ____fput+0xe/0x10 [ 1187.852776] [] ? task_work_run+0xac/0xe0 [ 1187.852780] [] SyS_unlink+0x16/0x20 [ 1187.852784] [] system_call_fastpath+0x16/0x1b [ 1187.852799] GFS2: fsid=rh7_cluster:mydata.0: fatal: filesystem consistency error GFS2: fsid=rh7_cluster:mydata.0: RG = 2031621 GFS2: fsid=rh7_cluster:mydata.0: function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 109 [ 1187.852818] GFS2: fsid=rh7_cluster:mydata.0: about to withdraw this file system [ 1187.864039] GFS2: fsid=rh7_cluster:mydata.0: telling LM to unmount [ 1187.864507] GFS2: fsid=rh7_cluster:mydata.0: withdrawn A withdrawal occurred on a RHEL 7 gfs2 filesystem: Jul 31 11:41:06 node42 kern.warning kernel:GFS2: fsid=mycluster:share.0: fatal: filesystem consistency error Jul 31 11:41:06 node42 GFS2: fsid=mycluster:share.0: RG = 202579796 Jul 31 11:41:06 node42 GFS2: fsid=mycluster:share.0: function = gfs2_setbit, file = fs/gfs2/rgrp.c, line = 109 Jul 31 11:41:06 node42 kern.err kernel:GFS2: fsid=mycluster:share.0: about to withdraw this file system Jul 31 11:41:06 node42 kern.err kernel:GFS2: fsid=mycluster:share.0: telling LM to unmount Jul 31 11:41:06 node42 kern.err kernel:GFS2: fsid=mycluster:share.0: withdrawn Jul 31 11:41:06 node42 kern.warning kernel:CPU: 3 PID: 11249 Comm: java Not tainted 3.10.0-514.2.2.el7.x86_64 #1 Jul 31 11:41:06 node42 kern.warning kernel:Hardware name: Dell Inc. PowerEdge M620/0VHRN7, BIOS 1.7.6 05/23/2013 Jul 31 11:41:06 node42 kern.warning kernel: ffff881fb8176000 0000000078f8a0ae ffff8818d9fff800 ffffffff816860cc Jul 31 11:41:06 node42 kern.warning kernel: ffff8818d9fff878 ffffffffa0ab5c56 ffff881800000030 ffff8818d9fff8a0 Jul 31 11:41:06 node42 kern.warning kernel: ffff8818d9fff830 0000000078f8a0ae ffff881fbee43fc0 ffff8818d9fff8e0 Jul 31 11:41:06 node42 kern.warning kernel:Call Trace: Jul 31 11:41:06 node42 kern.warning kernel: [] dump_stack+0x19/0x1b Jul 31 11:41:06 node42 kern.warning kernel: [] gfs2_lm_withdraw+0x146/0x180 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] gfs2_consist_rgrpd_i+0x47/0x50 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] rgblk_free+0x217/0x230 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] __gfs2_free_blocks+0x34/0x120 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] recursive_scan+0x6c7/0x6f0 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] ? __find_get_page+0x1e/0xa0 Jul 31 11:41:06 node42 kern.warning kernel: [] trunc_dealloc+0x11a/0x140 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] ? __fput+0x80/0x260 Jul 31 11:41:06 node42 kern.warning kernel: [] do_shrink+0x1e3/0x550 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] gfs2_setattr_size+0x173/0x270 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] ? gfs2_glock_nq+0x260/0x450 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] gfs2_setattr+0x11f/0x2b0 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] ? gfs2_glock_add_to_lru+0x68/0x80 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] ? gfs2_setattr+0x78/0x2b0 [gfs2] Jul 31 11:41:06 node42 kern.warning kernel: [] notify_change+0x279/0x3d0 Jul 31 11:41:06 node42 kern.warning kernel: [] ? process_measurement+0x8e/0x250 Jul 31 11:41:06 node42 kern.warning kernel: [] do_truncate+0x75/0xc0 Jul 31 11:41:06 node42 kern.warning kernel: [] do_last+0x5f2/0x12a0 Jul 31 11:41:06 node42 kern.warning kernel: [] path_openat+0xc2/0x490 Jul 31 11:41:06 node42 kern.warning kernel: [] ? unlock_page+0x2b/0x30 Red Hat Enterprise Linux 6 The issue 1618511 - A gfs2 withdrawal occurred: function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1432 was closed as WONT_FIX as the fix would require too many patches to resolve the issue in the current life cycle of RHEL 6 and a workaround exists shown below. Red Hat Enterprise Linux 7 The issue (bugzilla bug: 1364234) has been resolved with the errata RHSA-2022:0063 with the following package(s): kernel-3.10.0-1160.53.1.el7 or later for RHEL 7.9.z. Red Hat Enterprise Linux 8 The issue (bz1724769) has been resolved with errata RHSA-2020:4431 with the following package(s): kernel-4.18.0-240.el8 or later. It is recommended that a filesystem check be performed on the gfs2 filesystem with the latest version of fsck.gfs2 from the package gfs2-utils. Update gfs2-utils before running the filesystem check and make sure that the gfs2 filesystem is not mounted on any clustered node when the filesystem check is ran. Once a patched kernel is installed and running on all cluster nodes then the gfs2 filesystems no longer need to be mounted with errors=panic. Workaround The corruption that occurs in this issue occurs after the gfs2 filesystem withdraws. This corruption can be avoided by allowing the gfs2 filesystem to panic instead of causing a withdraw. To cause a gfs2 filesystem to panic instead of withdraw then add the following to the mount options for the gfs2 filesystem. This should be done after a fsck.gfs2 mention above is done to fix the filesystem: errors="panic" For more information then see the following articles: RHEL 6: Global File System 2 - 4.14. The GFS2 Withdraw Function RHEL
Trending
Solutions
Solve specific problems that has been raised with Red HatThere are no trending items at this time.
Articles
Technical briefs, reference architectures, and early stages of documentationAs a customer how does Red Hat support me when I use third party components?
Policy and support definitions for how Red Hat manages non-Red Hat components/software when customers have issues.
Red Hat Online Learning (ROLE) Technical Support Information
Updating Global Contacts link
Latest discussions in the Red Hat Community
Rootless Podman Guided Exercise: Clarification Needed Regarding Image Builds for Roo...
Ansible Automation Portal Appliance has no network connection
Get support
Support cases
Get answers quickly by opening a support case with us.
Live chat
Directly access our support engineers during weekday business hours.
Call or email
Speak directly with a Red Hat Support expert by phone or through email.