Ceph volumes hang or stall when iptables/nftables is running

Solution In Progress - Updated -

Issue

  • Whenever we start virtual machines backed by a Ceph volume, disk I/O on these VMs will stall soon after the initial boot. dmesg of affected machines contains the following messages:
[  615.074762]       Not tainted 4.18.0-147.el8.x86_64 #1
[  615.075418] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  615.076434] xfsaild/vda1    D    0   416      2 0x80000000
[  615.077173] Call Trace:
[  615.077512]  ? __schedule+0x253/0x830
[  615.077997]  schedule+0x28/0x70
[  615.078563]  xfs_log_force+0x160/0x2e0 [xfs]
[  615.079155]  ? wake_up_q+0x70/0x70
[  615.079628]  xfsaild+0x17b/0x7f0 [xfs]
[  615.080161]  ? xfs_trans_ail_cursor_first+0x80/0x80 [xfs]
[  615.080864]  kthread+0x112/0x130
[  615.081286]  ? kthread_flush_work_fn+0x10/0x10
[  615.081880]  ret_from_fork+0x35/0x40
  • Another footprint is that Ceph Volume creation can hang or take a long time.

Environment

  • Red hat Enterprise Linux 8.1
    • kernel-4.18.0-147.5.1.el8
  • OSP16
  • iptables using nftables backend

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content