Kernel panic at do_munmap+0x1dc

Diagnostic Steps

Pre-requisites

1. Deploy kdump in Order to Collect a vmcore:

   • Vmcore analyis is required to determine if you are being impacted by this issue. This first requires that a vmcore is dumped successfully.
   • If the kexec-tools package is absent or the kdump service is inactive, please reference the following article to install, enable, start, and configure kdump:
     How to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux

2. Prepare crash Environment for vmcore Analysis

   • Ensure that you have the crash package installed, and if necessary install the package:

     # yum install crash
     

   • Ensure the necessary debuginfo package is installed. See the following article for more information:
     How can I download or install debuginfo packages for RHEL systems?

Vmcore Analysis

1. The backtrace of the panicking task shows the function where the panic occurred as indicated by the RIP. Here the panic occurs in do_munmap:

         crash> bt
         PID: 12853  TASK: ffff88132bf73f40  CPU: 4   COMMAND: "java"
          #0 [ffff88122ca97b68] machine_kexec at ffffffff8105c4cb
          #1 [ffff88122ca97bc8] __crash_kexec at ffffffff81104a32
          #2 [ffff88122ca97c98] crash_kexec at ffffffff81104b20
          #3 [ffff88122ca97cb0] oops_end at ffffffff816ad278
          #4 [ffff88122ca97cd8] no_context at ffffffff8169d29a
          #5 [ffff88122ca97d28] __bad_area_nosemaphore at ffffffff8169d330
          #6 [ffff88122ca97d70] bad_area_nosemaphore at ffffffff8169d49a
          #7 [ffff88122ca97d80] __do_page_fault at ffffffff816b013e
          #8 [ffff88122ca97de0] do_page_fault at ffffffff816b02e5
          #9 [ffff88122ca97e10] page_fault at ffffffff816ac508
             [exception RIP: do_munmap+0x1dc]
             RIP: ffffffff811b794c  RSP: ffff88122ca97ec8  RFLAGS: 00010246
             RAX: ffff881397b3e000  RBX: ffff887301b1abc0  RCX: ffff883327a5a020
             RDX: ffffffff811b4ff0  RSI: ffff881397b3e000  RDI: ffff881302ad2a40
             RBP: ffff88122ca97f08   R8: ffff887264e91028   R9: 0000000000000000
             R10: 0000000000000000  R11: 0000000000000206  R12: ffff881397b3e000
             R13: 00007f0894000000  R14: 00007f0898000000  R15: 0000000000000000
             ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
         #10 [ffff88122ca97f10] vm_munmap at ffffffff811b7c45
         #11 [ffff88122ca97f60] sys_munmap at ffffffff811b8d52
         #12 [ffff88122ca97f80] system_call_fastpath at ffffffff816b4fc9
             RIP: 00007f08d5a36a27  RSP: 00007f08b9f0afb0  RFLAGS: 00000202
             RAX: 000000000000000b  RBX: ffffffff816b4fc9  RCX: ffffffffffffffff
             RDX: 0000000000000000  RSI: 0000000004000000  RDI: 00007f0894000000
             RBP: 0000000000021000   R8: ffffffffffffffff   R9: 0000000000000000
             R10: 0000000000004022  R11: 0000000000000206  R12: 0000000000000000
             R13: 00007f0890000000  R14: ffffffff811b8d52  R15: ffff88122ca97f78
             ORIG_RAX: 000000000000000b  CS: 0033  SS: 002b
   

2. The disassembly of the panicking function shows the instructions leading up to the panic and where we are in the source code:

         crash> dis -rl do_munmap+0x1dc | tail
         /usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/mm/internal.h: 173
         0xffffffff811b7931 <do_munmap+0x1c1>: mov    0x8(%r12),%rdx
         0xffffffff811b7936 <do_munmap+0x1c6>: mov    (%r12),%rsi
         0xffffffff811b793a <do_munmap+0x1ca>: callq  0xffffffff811b44e0 <munlock_vma_pages_range>
         0xffffffff811b793f <do_munmap+0x1cf>: jmp    0xffffffff811b7900 <do_munmap+0x190>
         0xffffffff811b7941 <do_munmap+0x1d1>: nopl   0x0(%rax)
         /usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/mm/mmap.c: 2624
         0xffffffff811b7948 <do_munmap+0x1d8>: mov    -0x40(%rbp),%rsi
         /usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/mm/mmap.c: 2625
         0xffffffff811b794c <do_munmap+0x1dc>: movq   $0x0,0x18(%r15)   
   
         2612 /*
         2613  * Create a list of vma's touched by the unmap, removing them from the mm's
         2614  * vma list as we go..
         2615  */
         2616 static void
         2617 detach_vmas_to_be_unmapped(struct mm_struct *mm, struct vm_area_struct *vma,
         2618         struct vm_area_struct *prev, unsigned long end)
         2619 {
         2620         struct vm_area_struct **insertion_point;
         2621         struct vm_area_struct *tail_vma = NULL;
         2622         unsigned long addr;
         2623 
         2624         insertion_point = (prev ? &prev->vm_next : &mm->mmap);
         2625         vma->vm_prev = NULL;           <<<---- Panicked here
   

3. Get the value of the vma, held in %r15 from %rax at offset 0x16f. Two instructions later we jump to the area of code where we panicked:

          /usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/mm/mmap.c: 2798
         0xffffffff811b78d0 <do_munmap+0x160>:   cmpq   $0x0,-0x40(%rbp)
         0xffffffff811b78d5 <do_munmap+0x165>:   je     0xffffffff811b7b90 <do_munmap+0x420>
         0xffffffff811b78db <do_munmap+0x16b>:   mov    -0x40(%rbp),%rax
         0xffffffff811b78df <do_munmap+0x16f>:   mov    0x10(%rax),%r15    <----
         /usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/mm/mmap.c: 2803
         0xffffffff811b78e3 <do_munmap+0x173>:   cmpq   $0x0,0xc0(%rbx)
         0xffffffff811b78eb <do_munmap+0x17b>:   je     0xffffffff811b7948 <do_munmap+0x1d8>   
   
         2736 int do_munmap(struct mm_struct *mm, unsigned long start, size_t len,
         2737               struct list_head *uf)
         -----------------<snip>-------------------------
         2798         vma = prev? prev->vm_next: mm->mmap;       <--- here we set the vma
         2799 
         2800         /*
         2801          * unlock any mlock()ed ranges before detaching vmas
         2802          */
         2803         if (mm->locked_vm) {
         2804                 struct vm_area_struct *tmp = vma;
         2805                 while (tmp && tmp->vm_start < end) {
         2806                         if (tmp->vm_flags & VM_LOCKED) {
         2807                                 mm->locked_vm -= vma_pages(tmp);
         2808                                 munlock_vma_pages_all(tmp);
         2809                         }
         2810                         tmp = tmp->vm_next;
         2811                 }
         2812         }
         2813 
         2814         /*
         2815          * Remove the vma's, and unmap the actual pages
         2816          */
         2817         detach_vmas_to_be_unmapped(mm, vma, prev, end);
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                             Call to detach_vmas_to_be_unmapped:
   
                    vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv  
               2616 static void
               2617 detach_vmas_to_be_unmapped(struct mm_struct *mm, struct vm_area_struct *vma,
               2618         struct vm_area_struct *prev, unsigned long end)
               2619 {
               2620         struct vm_area_struct **insertion_point;
               2621         struct vm_area_struct *tail_vma = NULL;
               2622         unsigned long addr;
               2623 
               2624         insertion_point = (prev ? &prev->vm_next : &mm->mmap);
               2625         vma->vm_prev = NULL;         <--- here is where we panic
   

4. So %rax should be our vma, however this vma is already freed. Notice that kmem is reporting that this is a duplicate freelist object:

         crash> kmem ffff881397b3e000
         CACHE             OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE  NAME
         ffff88017fc05900      216     137156    162800   4400     8k  vm_area_struct
           SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
           ffffea004e5ecf80  ffff881397b3e000     0     37      65509  -65472
           FREE / [ALLOCATED]
            ffff881397b3e000  (cpu 3 cache)
         kmem: vm_area_struct: slab: ffffea004e5ecf80 duplicate freelist object: ffff881397b3e000
   
               PAGE         PHYSICAL      MAPPING       INDEX CNT FLAGS
         ffffea004e5ecf80 1397b3e000                0 ffff881397b3e000  1 2fffff00004080 slab,head
   

5. Look at the vm_area_struct and fill it, vm_prev is already set to null:

         struct vm_area_struct {
           vm_start = 0xffff881397b3e510, 
           vm_end = 0x7f0894000000, 
           vm_next = 0x0, 
           vm_prev = 0x0, 
           vm_rb = {
             __rb_parent_color = 0xffff887264e91028, 
             rb_right = 0x0, 
             rb_left = 0x0
           }, 
          --------<snip>---------------------
   

6. Get the vma value from calling find_vma() earlier, and the corresponding source code:

         0xffffffff811b781b <do_munmap+0xab>:    callq  0xffffffff811b5120 <find_vma>
         /usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/mm/mmap.c: 2750
         0xffffffff811b7820 <do_munmap+0xb0>:    test   %rax,%rax
         /usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/mm/mmap.c: 2749
         0xffffffff811b7823 <do_munmap+0xb3>:    mov    %rax,-0x40(%rbp)     <-- here is where we save the value at -0x40(rbp)
   
         2736 int do_munmap(struct mm_struct *mm, unsigned long start, size_t len,
         2737               struct list_head *uf)
         ------------------<snip>-----------------
         2748         /* Find the first overlapping VMA */
         2749         vma = find_vma(mm, start);
         2750         if (!vma)
         2751                 return 0;
         2752         prev = vma->vm_prev;
   

7. Within find_vma() we walk the mm_rb tree within the mm_struct to get the next vma for this process, however this vma is not in the list:

         crash> tree -t rbtree -r mm_struct.mm_rb 0xffff8851c77d21d0 -o 0x38 | grep ffff881397b3e000 | wc -l
         0
   
         2215 /* Look up the first VMA which satisfies  addr < vm_end,  NULL if none. */
         2216 struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr)
         2217 {
         2218         struct vm_area_struct *vma = NULL;
         2219 
         2220         /* Check the cache first. */
         2221         /* (Cache hit rate is typically around 35%.) */
         2222         vma = ACCESS_ONCE(mm->mmap_cache);
         2223         if (!(vma && vma->vm_end > addr && vma->vm_start <= addr)) {
         2224                 struct rb_node *rb_node;
         2225 
         2226                 rb_node = mm->mm_rb.rb_node;
         2227                 vma = NULL;
         2228 
         2229                 while (rb_node) {
         2230                         struct vm_area_struct *vma_tmp;
         2231 
         2232                         vma_tmp = rb_entry(rb_node,
         2233                                            struct vm_area_struct, vm_rb);
         2234 
         2235                         if (vma_tmp->vm_end > addr) {
         2236                                 vma = vma_tmp;
         2237                                 if (vma_tmp->vm_start <= addr)
         2238                                         break;
         2239                                 rb_node = rb_node->rb_left;
         2240                         } else
         2241                                 rb_node = rb_node->rb_right;
         2242                 }
         2243                 if (vma)
         2244                         mm->mmap_cache = vma;
         2245         }
         2246         return vma;
         2247 }
   

8. This task is part of a thread group and is not the thread group leader. The significance here is that they share the same mm_struct and corresponding vmas:

         crash> set
             PID: 12853
         COMMAND: "java"
            TASK: ffff88132bf73f40  [THREAD_INFO: ffff88122ca94000]
             CPU: 4
           STATE: TASK_RUNNING (PANIC)
   
         crash> ps -g 12853
         PID: 12846  TASK: ffff8872f5ea6eb0  CPU: 99  COMMAND: "java"
           PID: 12847  TASK: ffff8872f5ea0fd0  CPU: 57  COMMAND: "java"
           PID: 12848  TASK: ffff88132bf75ee0  CPU: 71  COMMAND: "java"
           PID: 12849  TASK: ffff88132bf72f70  CPU: 75  COMMAND: "java"
           PID: 12850  TASK: ffff88132bf74f10  CPU: 16  COMMAND: "java"
           PID: 12851  TASK: ffff88132bf76eb0  CPU: 2   COMMAND: "java"
           PID: 12852  TASK: ffff88132bf71fa0  CPU: 3   COMMAND: "java"
           PID: 12853  TASK: ffff88132bf73f40  CPU: 4   COMMAND: "java"
           PID: 12854  TASK: ffff88132bf70000  CPU: 5   COMMAND: "java"
           PID: 12855  TASK: ffff88132bf70fd0  CPU: 6   COMMAND: "java"
           PID: 12856  TASK: ffff8806513a8000  CPU: 7   COMMAND: "java"
           PID: 12857  TASK: ffff8806513a9fa0  CPU: 64  COMMAND: "java"
           PID: 12858  TASK: ffff8806513aeeb0  CPU: 65  COMMAND: "java"
           PID: 12859  TASK: ffff8806513a8fd0  CPU: 66  COMMAND: "java"
           PID: 12860  TASK: ffff8806513abf40  CPU: 11  COMMAND: "java"
           PID: 12861  TASK: ffff8806513acf10  CPU: 12  COMMAND: "java"
           PID: 12862  TASK: ffff8806513adee0  CPU: 13  COMMAND: "java"
           PID: 12863  TASK: ffff8833176c1fa0  CPU: 56  COMMAND: "java"
           PID: 12864  TASK: ffff8833176c6eb0  CPU: 1   COMMAND: "java"
           PID: 12865  TASK: ffff8833176c5ee0  CPU: 10  COMMAND: "java"
           PID: 12866  TASK: ffff8820d328cf10  CPU: 17  COMMAND: "java"
           PID: 12867  TASK: ffff8820d328af70  CPU: 18  COMMAND: "java"
           PID: 12868  TASK: ffff88324eebeeb0  CPU: 72  COMMAND: "java"
           PID: 12869  TASK: ffff88324eeb8000  CPU: 76  COMMAND: "java"
           PID: 12870  TASK: ffff88324eebaf70  CPU: 77  COMMAND: "java"
           PID: 12871  TASK: ffff88330b7bbf40  CPU: 78  COMMAND: "java"
           PID: 12872  TASK: ffff88330b7baf70  CPU: 0   COMMAND: "java"
           PID: 12873  TASK: ffff88330b7bdee0  CPU: 79  COMMAND: "java"
           PID: 12874  TASK: ffff88330b7b8000  CPU: 16  COMMAND: "java"
           PID: 12875  TASK: ffff88330b7beeb0  CPU: 81  COMMAND: "java"
           PID: 12876  TASK: ffff8803b17a0000  CPU: 8   COMMAND: "java"
           PID: 12877  TASK: ffff8803b17a0fd0  CPU: 20  COMMAND: "java"
           PID: 12878  TASK: ffff8803b17a1fa0  CPU: 26  COMMAND: "java"
           PID: 12879  TASK: ffff8803b17a2f70  CPU: 27  COMMAND: "java"
           PID: 12880  TASK: ffff8803b17a3f40  CPU: 14  COMMAND: "java"
           PID: 12881  TASK: ffff8803b17a4f10  CPU: 15  COMMAND: "java"
           PID: 12882  TASK: ffff8803b17a5ee0  CPU: 19  COMMAND: "java"
           PID: 12883  TASK: ffff8803b17a6eb0  CPU: 19  COMMAND: "java"
           PID: 12884  TASK: ffff88334c7fbf40  CPU: 19  COMMAND: "java"
           PID: 12885  TASK: ffff88334c7f8fd0  CPU: 22  COMMAND: "java"
           PID: 12886  TASK: ffff88334c7feeb0  CPU: 22  COMMAND: "java"
           PID: 12887  TASK: ffff88334c7fcf10  CPU: 14  COMMAND: "java"
           PID: 12888  TASK: ffff88334c7faf70  CPU: 24  COMMAND: "java"
           PID: 12889  TASK: ffff88334c7f8000  CPU: 22  COMMAND: "java"
           PID: 12890  TASK: ffff88334c7fdee0  CPU: 25  COMMAND: "java"
           PID: 12891  TASK: ffff883328a32f70  CPU: 21  COMMAND: "java"
           PID: 12892  TASK: ffff88335d3f0000  CPU: 18  COMMAND: "java"
           PID: 12893  TASK: ffff88335d3f4f10  CPU: 23  COMMAND: "java"
           PID: 12894  TASK: ffff88335d3f1fa0  CPU: 70  COMMAND: "java"
           PID: 12895  TASK: ffff88335d3f0fd0  CPU: 20  COMMAND: "java"
           PID: 12896  TASK: ffff88335d3f5ee0  CPU: 20  COMMAND: "java"
           PID: 12897  TASK: ffff88335d3f2f70  CPU: 24  COMMAND: "java"
           PID: 12898  TASK: ffff883ffc06eeb0  CPU: 24  COMMAND: "java"
           PID: 12899  TASK: ffff883ffc068000  CPU: 25  COMMAND: "java"
           PID: 12900  TASK: ffff88287b776eb0  CPU: 55  COMMAND: "java"
           PID: 12901  TASK: ffff88287b775ee0  CPU: 23  COMMAND: "java"
           PID: 12902  TASK: ffff88287b770000  CPU: 103  COMMAND: "java"
           PID: 12903  TASK: ffff88287b774f10  CPU: 103  COMMAND: "java"
           PID: 12904  TASK: ffff88287b773f40  CPU: 103  COMMAND: "java"
           PID: 12905  TASK: ffff88287b770fd0  CPU: 103  COMMAND: "java"
           PID: 12906  TASK: ffff88287b772f70  CPU: 50  COMMAND: "java"
           PID: 12907  TASK: ffff88335ea83f40  CPU: 50  COMMAND: "java"
           PID: 12908  TASK: ffff88335ea80fd0  CPU: 85  COMMAND: "java"
           PID: 12909  TASK: ffff88335ea86eb0  CPU: 30  COMMAND: "java"
           PID: 12910  TASK: ffff88335ea81fa0  CPU: 54  COMMAND: "java"
           PID: 12911  TASK: ffff88335ea84f10  CPU: 55  COMMAND: "java"
           PID: 12912  TASK: ffff88335ea80000  CPU: 55  COMMAND: "java"
           PID: 12913  TASK: ffff88334435dee0  CPU: 55  COMMAND: "java"
           PID: 12914  TASK: ffff88334435cf10  CPU: 55  COMMAND: "java"
           PID: 12915  TASK: ffff88334435eeb0  CPU: 55  COMMAND: "java"
           PID: 12916  TASK: ffff8832332acf10  CPU: 27  COMMAND: "java"
           PID: 12917  TASK: ffff8832332adee0  CPU: 27  COMMAND: "java"
           PID: 12918  TASK: ffff8832332abf40  CPU: 26  COMMAND: "java"
           PID: 12919  TASK: ffff8832332aaf70  CPU: 26  COMMAND: "java"
           PID: 12920  TASK: ffff8832332a8fd0  CPU: 14  COMMAND: "java"
   

9. These tasks have a relatively small number of vmas. Also note that this is the only running task at the time of the crash:

         crash> vm
         PID: 12853  TASK: ffff88132bf73f40  CPU: 4   COMMAND: "java"
                MM               PGD          RSS    TOTAL_VM
         ffff887301b1abc0  ffff88736b212000  13912k  32710532k
               VMA              START        END     FLAGS FILE
         ffff8866d52ce1b0       400000       401000 8000875 /oracle/app/19.3.0/grid/jdk/bin/java
         ffff8866d52ce5e8       600000       601000 8100871 /oracle/app/19.3.0/grid/jdk/bin/java
         ffff8873387c6e58       601000       602000 8100873 /oracle/app/19.3.0/grid/jdk/bin/java
         ffff8873387c71b8      1704000      1725000 8100073 
         ffff88386f645290     80200000     d5780000 8100073 
         ffff883327a5b290     d5780000    580100000 8200070 
         ffff883327a5b950    580100000    5aab80000 8100073 
         ffff883327a5a000    5aab80000    800000000 8200070 
         ffff887264e91008 7f0880000000 7f088c000000 8200070 
         ffff881302ad2a20 7f0894000000 7f0898000000 8200070 
         ffff881302ad21b0 7f0898000000 7f08b0000000 8200070 
         ffff887264e90510 7f08ac000000 7f08b0000000 8200070 
   
         crash> bt -a | grep "exception RIP:" | sort | uniq -c | sort
             111     [exception RIP: intel_idle+0xd5]
               1     [exception RIP: do_munmap+0x1dc]
   

10. To modify the vmas mapped to a mm_struct, you need to first take the mmap_sem lock. Looking at the lock, it looks valid with the exception of an incorrect owner:

          crash> struct mm_struct.mmap_sem 0xffff887301b1abc0 
            mmap_sem = {
              {
                count = {
                  counter = 0xfffffffe00000001
                }, 
                __UNIQUE_ID_rh_kabi_hide0 = {
                  count = 0xfffffffe00000001
                }, 
                {<No data fields>}
              }, 
              wait_lock = {
                raw_lock = {
                  val = {
                    counter = 0x0
                  }
                }
              }, 
              osq = {
                tail = {
                  counter = 0x0
                }
              }, 
              wait_list = {
                next = 0xffff8813943dbe50
              }, 
              owner = 0x1
            }
    

11. Note there are 73 tasks currently blocked, all of these blocked tasks are waiting on a rw_semaphore, and notice that the wait_list from the mmap_sem has 73 waiters as well:

          crash> ps -S
            RU: 113
            IN: 3295
            UN: 73
    
          crash> foreach UN bt | awk '/#3 /{print $3,$5,$6}' | sort | uniq -c | sort -nr
               71 call_rwsem_down_write_failed ffffffff81331b07 
                2 call_rwsem_down_read_failed ffffffff81331ad8 
    
          crash> list 0xffff8813943dbe50 | wc -l
          73