Issue

Experiencing hung task kernel panics in a cloudera environment.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-253:21  D 0000000000000002     0  2258      2 0x00000000
 ffff881024fdb9a0 0000000000000046 ffff881024fdb968 ffff881024fdb964
 0000000001000000 ffff88103fe70c00 ffff880061936840 0000000000000002
 ffff881024c3b058 ffff881024fdbfd8 000000000000fbc8 ffff881024c3b058
Call Trace:
 [<ffffffffa006e08a>] start_this_handle+0x25a/0x480 [jbd2]
 [<ffffffff8116ec9b>] ? cache_alloc_refill+0x15b/0x240
 [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa006e495>] jbd2_journal_start+0xb5/0x100 [jbd2]
 [<ffffffffa00adeb6>] ext4_journal_start_sb+0x56/0xe0 [ext4]
 [<ffffffffa009ce1f>] ext4_da_writepages+0x26f/0x620 [ext4]
 [<ffffffff812825b9>] ? cpumask_next_and+0x29/0x50
 [<ffffffff8105afee>] ? find_busiest_group+0x90e/0x9f0
 [<ffffffff81134c91>] do_writepages+0x21/0x40
 [<ffffffff811b4ffd>] writeback_single_inode+0xdd/0x290
 [<ffffffff811b53fd>] writeback_sb_inodes+0xbd/0x170
 [<ffffffff811b555b>] writeback_inodes_wb+0xab/0x1b0

crash> ps | grep ">"
>     0      0   2  ffff882029378040  RU   0.0       0      0  [swapper]
>     0      0   3  ffff881029b21500  RU   0.0       0      0  [swapper]
>     0      0   4  ffff882029378aa0  RU   0.0       0      0  [swapper]
>     0      0   5  ffff881029b67540  RU   0.0       0      0  [swapper]
>     0      0   6  ffff882029379500  RU   0.0       0      0  [swapper]
>     0      0   7  ffff881029bb1500  RU   0.0       0      0  [swapper]
>     0      0   8  ffff8820293c6080  RU   0.0       0      0  [swapper]
>     0      0   9  ffff881029bd1540  RU   0.0       0      0  [swapper]
>     0      0  10  ffff8820293c6ae0  RU   0.0       0      0  [swapper]
>     0      0  11  ffff88102900f500  RU   0.0       0      0  [swapper]
>     0      0  12  ffff8820293c7540  RU   0.0       0      0  [swapper]
>     0      0  13  ffff88102902f540  RU   0.0       0      0  [swapper]
>     0      0  15  ffff881029051500  RU   0.0       0      0  [swapper]
>     0      0  16  ffff882028ce0aa0  RU   0.0       0      0  [swapper]
>     0      0  17  ffff881029099540  RU   0.0       0      0  [swapper]
>     0      0  18  ffff882028ce1500  RU   0.0       0      0  [swapper]
>     0      0  19  ffff8810290df500  RU   0.0       0      0  [swapper]
>     0      0  20  ffff882028d10080  RU   0.0       0      0  [swapper]
>     0      0  21  ffff881029129540  RU   0.0       0      0  [swapper]
>     0      0  22  ffff882028d10ae0  RU   0.0       0      0  [swapper]
>     0      0  23  ffff88102916f500  RU   0.0       0      0  [swapper]
>     0      0  24  ffff882028d11540  RU   0.0       0      0  [swapper]
>     0      0  25  ffff881029191540  RU   0.0       0      0  [swapper]
>     0      0  26  ffff882028d62040  RU   0.0       0      0  [swapper]
>     0      0  27  ffff8810291b1500  RU   0.0       0      0  [swapper]
>     0      0  28  ffff882028d62aa0  RU   0.0       0      0  [swapper]
>     0      0  29  ffff8810291f3540  RU   0.0       0      0  [swapper]
>     0      0  30  ffff882028d63500  RU   0.0       0      0  [swapper]
>     0      0  31  ffff881029213500  RU   0.0       0      0  [swapper]
>   338      2   1  ffff88102645d500  RU   0.0       0      0  [khungtaskd]
> 39242  39122   0  ffff880fb6fc1540  RU   0.5 35145172 651596  java
> 50863  40006  14  ffff880ea7240aa0  RU   0.3 1675556 416064  java

No active swapping.

crash> kmem -i
              PAGES        TOTAL      PERCENTAGE
 TOTAL MEM  33029772       126 GB         ----
      FREE  18961137      72.3 GB   57% of TOTAL MEM
      USED  14068635      53.7 GB   42% of TOTAL MEM
    SHARED   475659       1.8 GB    1% of TOTAL MEM
   BUFFERS   283126       1.1 GB    0% of TOTAL MEM
    CACHED  3188956      12.2 GB    9% of TOTAL MEM
      SLAB   586613       2.2 GB    1% of TOTAL MEM

TOTAL SWAP  33554430       128 GB         ----
 SWAP USED        0            0    0% of TOTAL SWAP
 SWAP FREE  33554430       128 GB  100% of TOTAL SWAP

Call Trace:
 [<ffffffffa006e08a>] start_this_handle+0x25a/0x480 [jbd2]
 [<ffffffff8116ec9b>] ? cache_alloc_refill+0x15b/0x240
 [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa006e495>] jbd2_journal_start+0xb5/0x100 [jbd2]
 [<ffffffffa00adeb6>] ext4_journal_start_sb+0x56/0xe0 [ext4]
 [<ffffffffa009ce1f>] ext4_da_writepages+0x26f/0x620 [ext4]
 [<ffffffff812825b9>] ? cpumask_next_and+0x29/0x50
 [<ffffffff8105afee>] ? find_busiest_group+0x90e/0x9f0