Unknown program exception: 0028 in cpu_relax() on IBM system Z (s390x)

Issue

Linux z/VM guest on IBM Z Series crashes with "Unknown program exception: 0028 [#1] SMP" in random places.

In this example the crash happened when executing a no operation assembly instruction.

The kernel panic stack trace looked like:

[2113471.872311] Unknown program exception: 0028 [#1] SMP 
[2113471.872322] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver appldata_net_sum appldata_os appldata_mem nfsv3 nfs_acl nfs lockd grace fscache qeth_l3 vmur auth_rpcgss sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 dasd_fba_mod qeth ccwgroup qdio dasd_diag_mod dasd_eckd_mod dasd_mod dm_mirror dm_region_hash dm_log dm_mod sha512_s390 ghash_s390 des_s390 des_generic aes_s390 prng
[2113471.872373] CPU: 1 PID: 3323 Comm: Thread-44 Not tainted 3.10.0-1062.4.1.el7.s390x #1
[2113471.872377] task: 000000059aa32ec0 ti: 000000036d9f4000 task.ti: 000000036d9f4000
[2113471.872381] Krnl PSW : 0704f00180000000 00000000001067ae (cpu_relax+0x1e/0x20)
[2113471.872393]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 EA:3
                 Krnl GPRS: 0000000000531230 0000000000000000 000000007f971040 0000000000c00000
[2113471.872400]            000003ff8030d952 0000000000000010 0000000000000000 07000005aec2f2b0
[2113471.872404]            000000010c9882f8 0000000000ac9300 00000000016f8000 0000000663772960
[2113471.872471]            000000036d9f7d50 00000006637729a0 000003ff8030d9e4 000000036d9f7ab0
[2113471.872482] Krnl Code: 00000000001067a0: a7840004      brc 8,1067a8
                       00000000001067a4: 83000044       diag    %r0,%r0,68
                      #00000000001067a8: c0f40032e094   brcl    15,7628d0
                      >00000000001067ae: 0707       bcr 0,%r7
                       00000000001067b0: c0000000000c   larl    %r0,1067c8
                       00000000001067b6: c0f400000009   brcl    15,1067c8
                       00000000001067bc: c60000000005   exrl    %r0,1067c6
                       00000000001067c2: a7f40000       brc 15,1067c2
[2113471.872497] Call Trace:
[2113471.872499] ([<000003ff8030d952>] qeth_send_control_data+0x212/0x6f8 [qeth])
[2113471.872515]  [<000003ff80313812>] qeth_send_ipa_cmd+0xaa/0x140 [qeth]
[2113471.872520]  [<000003ff8031424a>] qeth_core_ethtool_get_link_ksettings+0xe2/0x328 [qeth]
[2113471.872524]  [<00000000005fe932>] __ethtool_get_link_ksettings+0x7a/0x118
[2113471.872530]  [<000000000061bf68>] speed_show+0x88/0xc0
[2113471.872533]  [<000000000052c09c>] dev_attr_show+0x54/0x98
[2113471.872537]  [<00000000003ac568>] sysfs_kf_seq_show+0x108/0x1a0
[2113471.872543]  [<000000000033515a>] seq_read+0xea/0x498
[2113471.872546]  [<0000000000302b18>] vfs_read+0xa0/0x180
[2113471.872549]  [<0000000000303bc4>] SyS_read+0x84/0x128
[2113471.872551]  [<000000000075abc4>] sysc_tracego+0x18/0x1e
[2113471.872555]  [<000002003bafc974>] 0x2003bafc974
[2113471.872558] Last Breaking-Event-Address:
[2113471.872559]  [<000003ff80306c98>] 0x3ff80306c98

The symptoms could vary widely, the z/VM guest could crash almost anytime. The key is the unexpected exception 0028. Please see the root cause for further details.