blkio controller: use after free crash

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 7.0

Issue

  • RHEL7 kernel crash in _raw_spin_lock_irq()
  • Strange corruption in my Docker images.

Resolution

Root Cause

__blkg_release_rcu() may be invoked after the associated request_queue
is released with a RCU grace period inbetween. As such, the function
and callbacks invoked from it must not dereference the associated
request_queue.

Unfortunately, a previous patch added [un]locking of @blkg->q->queue_lock
to __blkg_release_rcu(). This can cause oops as the
request_queue may be long gone by the time this code gets executed.

Diagnostic Steps

Steps to reproduce the issue.

  • Boot kernel with slub debugging enabled
    slub_debug=FZPU
  • Install docker package from RHEL7 Extras repo
    yum install docker
  • Start docker service
    systemctl start docker
  • Run a command in container.
    docker run rhel7 /bin/echo "Hello World"
  • Watch it crash:
[3628168.100141] BUG: unable to handle kernel NULL pointer dereference at           (null)
[3628168.100272] IP: [<ffffffff815e9865>] _raw_spin_lock_irq+0x15/0x60
[3628168.100370] PGD 0 
[3628168.100407] Oops: 0002 [#1] SMP 
[3628168.100464] Modules linked in: fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat binfmt_misc veth xt_addrtype xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop iptable_filter ip_tables nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache sg ext4 mbcache jbd2 iTCO_wdt iTCO_vendor_support bnx2x ipmi_si ipmi_msghandler i7core_edac mdio hpilo coretemp pcspkr acpi_power_meter edac_core hpwdt kvm crc32c_intel serio_raw lpc_ich mfd_core shpchp pcc_cpufreq mperf nfsd auth_rpcgss nfs_acl lockd openafs(POF) sunrpc xfs libcrc32c sd_mod crc_t10dif crct10dif_common radeon i2c_algo_bit drm_kms_helper ttm drm i2c_core hpsa dm_mirror dm_region_hash dm_log dm_mod
[3628168.101867] CPU: 7 PID: 47 Comm: rcuos/4 Tainted: PF          O--------------   3.10.0-123.6.3.el7.x86_64 #1
[3628168.102008] Hardware name: HP ProLiant BL460c G6, BIOS I24 12/01/2010
[3628168.102101] task: ffff880bfe1996c0 ti: ffff880bfe196000 task.ti: ffff880bfe196000
[3628168.102208] RIP: 0010:[<ffffffff815e9865>]  [<ffffffff815e9865>] _raw_spin_lock_irq+0x15/0x60
[3628168.102334] RSP: 0018:ffff880bfe197de0  EFLAGS: 00010082
[3628168.102410] RAX: 0000000000020000 RBX: 0000000000000010 RCX: 0000000000000000
[3628168.102513] RDX: 000000008000000b RSI: 0000000000000286 RDI: 0000000000000000
[3628168.102615] RBP: ffff880bfe197de0 R08: 0000000000000286 R09: 00007e0100007be0
[3628168.102740] R10: 00007e0100007be0 R11: 00007ba100007b60 R12: ffff880baa01e0b0
[3628168.102867] R13: ffff880baa01e000 R14: 000000000000001d R15: ffff8800da9865d0
[3628168.102970] FS:  0000000000000000(0000) GS:ffff880607c60000(0000) knlGS:0000000000000000
[3628168.103085] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[3628168.103169] CR2: 0000000000000000 CR3: 00000000018ce000 CR4: 00000000000007e0
[3628168.103272] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3628168.103375] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[3628168.103478] Stack:
[3628168.103509]  ffff880bfe197e08 ffffffff812a90b2 ffff880baa01e0b0 0000000000000001
[3628168.103634]  ffff880be847bf50 ffff880bfe197ec0 ffffffff810fef69 ffff880bfe1996c0
[3628168.103757]  ffff880bfe1996c0 ffff880c1fa4eef0 ffff880bfe197e80 ffff880bfe1996c0
[3628168.103943] Call Trace:
[3628168.103986]  [<ffffffff812a90b2>] __blkg_release_rcu+0x72/0x130
[3628168.104074]  [<ffffffff810fef69>] rcu_nocb_kthread+0x229/0x370
[3628168.104160]  [<ffffffff81086ac0>] ? wake_up_bit+0x30/0x30
[3628168.104238]  [<ffffffff810fed40>] ? rcu_start_gp+0x40/0x40
[3628168.104317]  [<ffffffff81085aff>] kthread+0xcf/0xe0
[3628168.104387]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[3628168.104482]  [<ffffffff815f26ec>] ret_from_fork+0x7c/0xb0
[3628168.104559]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[3628168.104652] Code: 75 f1 66 66 66 90 66 66 90 eb e3 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 fa 66 66 90 66 66 90 b8 00 00 02 00 <f0> 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 
[3628168.109670] RIP  [<ffffffff815e9865>] _raw_spin_lock_irq+0x15/0x60
[3628168.114319]  RSP <ffff880bfe197de0>
[3628168.118960] CR2: 0000000000000000

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments