System uptime not keeping pace with other CPUs time events on VMware VMs due to host overload
Issue
Time stamps of event in the /var/log/messages appear to be "in the future" relative to the overall system uptime. Tasks are hitting soft lockup and NMI watchdog reports tasks stuck for a long time
The system uptime is reported as
crash> sys
...
UPTIME: 08:25:14
which would correspond to
crash> pd 8*3600+25*60+14
$2 = 30314 seconds
Yet the latest timestamps reported in the message buffer are over 34112 seconds, nearly 4000 seconds MORE than the system uptime
crash> bt 3149
PID: 3149 TASK: ffff9056799bd140 CPU: 4 COMMAND: "xfsaild/dm-1"
[exception RIP: mpt_put_msg_frame+0x5e]
RIP: ffffffffc03473be RSP: ffff9056766dbb28 RFLAGS: 00000246
RAX: ffffb63ac35c0000 RBX: ffff90567ac42000 RCX: 000000000000000c
RDX: ffff905679914400 RSI: ffff90567ac42000 RDI: 000000000000000e
RBP: ffff9056766dbb18 R8: 0000000000000003 R9: 00000006398d93ff
R10: ffff90567963f0d8 R11: ffff90562837f880 R12: 0000000039b02c83
R13: ffff905679b02c80 R14: ffff90567ac42000 R15: ffff90562837f880
#0 [ffff9056766dbb30] mptscsih_qcmd at ffffffffc03bd71e [mptscsih]
#1 [ffff9056766dbbd8] mptspi_qcmd at ffffffffc03622e0 [mptspi]
#2 [ffff9056766dbc00] scsi_dispatch_cmd at ffffffff9a8d1540
#3 [ffff9056766dbc28] scsi_request_fn at ffffffff9a8daadc
#4 [ffff9056766dbc90] __blk_run_queue at ffffffff9a7436b9
#5 [ffff9056766dbca8] queue_unplugged at ffffffff9a74377a
#6 [ffff9056766dbce0] blk_flush_plug_list at ffffffff9a746fb8
#7 [ffff9056766dbd40] blk_finish_plug at ffffffff9a747424
#8 [ffff9056766dbd58] xfs_buf_delwri_submit_buffers at ffffffffc06190f9 [xfs]
#9 [ffff9056766dbdf8] xfs_buf_delwri_submit_nowait at ffffffffc0619e50 [xfs]
#10 [ffff9056766dbe08] xfsaild at ffffffffc0648d3f [xfs]
#11 [ffff9056766dbec8] kthread at ffffffff9a4c1c71
Environment
- Red Hat Enterprise Linux 5, 6, 7
- Virtual guests running on a VMware host
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.