System uptime not keeping pace with other CPUs time events on VMware VMs due to host overload

Solution In Progress - Updated -

Issue

Time stamps of event in the /var/log/messages appear to be "in the future" relative to the overall system uptime. Tasks are hitting soft lockup and NMI watchdog reports tasks stuck for a long time

The system uptime is reported as

crash> sys
...
UPTIME: 08:25:14

which would correspond to

crash> pd 8*3600+25*60+14
$2 = 30314   seconds

Yet the latest timestamps reported in the message buffer are over 34112 seconds, nearly 4000 seconds MORE than the system uptime

crash> bt 3149
PID: 3149   TASK: ffff9056799bd140  CPU: 4   COMMAND: "xfsaild/dm-1"
    [exception RIP: mpt_put_msg_frame+0x5e]
     RIP: ffffffffc03473be  RSP: ffff9056766dbb28  RFLAGS: 00000246
     RAX: ffffb63ac35c0000  RBX: ffff90567ac42000  RCX: 000000000000000c
     RDX: ffff905679914400  RSI: ffff90567ac42000  RDI: 000000000000000e
     RBP: ffff9056766dbb18   R8: 0000000000000003   R9: 00000006398d93ff
     R10: ffff90567963f0d8  R11: ffff90562837f880  R12: 0000000039b02c83
     R13: ffff905679b02c80  R14: ffff90567ac42000  R15: ffff90562837f880
 #0 [ffff9056766dbb30] mptscsih_qcmd at ffffffffc03bd71e [mptscsih]
 #1 [ffff9056766dbbd8] mptspi_qcmd at ffffffffc03622e0 [mptspi]
 #2 [ffff9056766dbc00] scsi_dispatch_cmd at ffffffff9a8d1540
 #3 [ffff9056766dbc28] scsi_request_fn at ffffffff9a8daadc
 #4 [ffff9056766dbc90] __blk_run_queue at ffffffff9a7436b9
 #5 [ffff9056766dbca8] queue_unplugged at ffffffff9a74377a
 #6 [ffff9056766dbce0] blk_flush_plug_list at ffffffff9a746fb8
 #7 [ffff9056766dbd40] blk_finish_plug at ffffffff9a747424
 #8 [ffff9056766dbd58] xfs_buf_delwri_submit_buffers at ffffffffc06190f9 [xfs]
 #9 [ffff9056766dbdf8] xfs_buf_delwri_submit_nowait at ffffffffc0619e50 [xfs]
#10 [ffff9056766dbe08] xfsaild at ffffffffc0648d3f [xfs]
#11 [ffff9056766dbec8] kthread at ffffffff9a4c1c71

Environment

  • Red Hat Enterprise Linux 5, 6, 7
  • Virtual guests running on a VMware host

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content