soft lockups in opal_get_rtc_time()/opal_set_rtc_time() on ppc64le under RHEL7

Solution In Progress - Updated -

Issue

Kernel experiences numerous softlockup events that look like:

[8800618.529482] NMI watchdog: BUG: soft lockup - CPU#106 stuck for 22s! [systemd-timedat:178328]
[8800618.529594] Modules linked in: unix_diag 8021q garp mrp ip6t_rpfilter bonding ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6
 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip
6_tables iptable_filter i2c_dev ext4 mbcache jbd2 ses enclosure scsi_transport_sas sg ofpart powernv_flash ibmpowernv shpchp ipmi_powernv mtd ipmi_msghandler at24 powernv_rng opal_prd i2c_opal binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect
[8800618.530433]  sysimgblt fb_sys_fops ttm drm ahci libahci i40e libata ixgbe megaraid_sas i2c_core mdio ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_devintf]
[8800618.530692] CPU: 106 PID: 178328 Comm: systemd-timedat Not tainted 3.10.0-693.37.4.el7.ppc64le #1
[8800618.530762] task: c00000394969ab80 ti: c000003338770000 task.ti: c000003338770000
[8800618.530822] NIP: c00000000077e338 LR: c000000000083898 CTR: 00000000300378e8
[8800618.530882] REGS: c000003338753970 TRAP: 0901   Not tainted  (3.10.0-693.37.4.el7.ppc64le)
[8800618.530952] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 44022882  XER: 20000000
[8800618.531103] CFAR: c0000000000838d0 SOFTE: 1 
                 GPR00: 0000000030005128 c000003338753bf0 c00000000121f200 0000000000000000 
                 GPR04: 0000000044022882 c00000000077e338 9000000000009033 0000000000000422 
                 GPR08: 0000000000000000 00000000300b3218 c0000000000838b4 9000000000001003 
                 GPR12: c00000000008388c c000000007b9ba00 
[8800618.531478] NIP [c00000000077e338] opal_get_rtc_time+0x98/0xb0
[8800618.531530] LR [c000000000083898] opal_return+0xc/0x48
[8800618.531570] Call Trace:
[8800618.531592] [c000003338753bf0] [c00000000077e338] opal_get_rtc_time+0x98/0xb0 (unreliable)
[8800618.531673] [c000003338753c30] [c0000000007769fc] rtc_read_time+0xac/0xf0
[8800618.531733] [c000003338753c70] [c000000000778b68] rtc_dev_ioctl+0x478/0x610
[8800618.531793] [c000003338753d20] [c00000000034c748] do_vfs_ioctl+0x428/0x890
[8800618.531887] [c000003338753dd0] [c00000000034cc74] SyS_ioctl+0xc4/0xe0
[8800618.531949] [c000003338753e30] [c00000000000a284] system_call+0x38/0xfc
[8800618.532008] Instruction dump: ...

Eventually, kernel panics.

Environment

  • RHEL7
  • Power machine with OPAL (OpenPower Abstraction Layer)/OpenPOWER firmware

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content