How do I recover or expand docker storage when I hit an XFS ENOSPC error?

Issue

• I was trying to use docker to do kernel builds of source rpms and the "docker build" command hung at the yum-builddep step:

Step 5 : RUN yum-builddep -y --enablerepo=\* ~/rpmbuild/SPECS/kernel.spec && yum clean all
 ---> Running in 46211acdaf6d
Loaded plugins: product-id

• In the messages file, I see the container start up, then soon after I see various errors

Jul  4 14:24:43 foo systemd: Starting docker container 46211acdaf6d25e4cd1d8310037355a0525d7a9dbbac882e6bde6b26ffeea3fe.
Jul  4 14:24:43 foo systemd: Started docker container 46211acdaf6d25e4cd1d8310037355a0525d7a9dbbac882e6bde6b26ffeea3fe.
...
Jul  4 14:24:46 foo lvm[826]: Thin vg_foo-docker--pool is now 89% full.
Jul  4 14:24:46 foo lvm[826]: Insufficient free space: 23 extents needed, but only 10 available
Jul  4 14:24:46 foo lvm[826]: Failed to extend thin vg_foo-docker--pool.
...
Jul  4 14:26:58 foo kernel: device-mapper: thin: 253:7: reached low water mark for data device: sending event.
Jul  4 14:26:58 foo kernel: device-mapper: thin: 253:7: switching pool to out-of-data-space mode
Jul  4 14:26:58 foo lvm[826]: Thin vg_foo-docker--pool is now 100% full.
Jul  4 14:26:58 foo lvm[826]: Insufficient free space: 23 extents needed, but only 10 available
...
Jul  4 14:27:58 foo kernel: device-mapper: thin: 253:7: switching pool to read-only mode
Jul  4 14:27:58 foo kernel: device-mapper: thin: 253:7: metadata operation 'dm_pool_commit_metadata' failed: error = -1
Jul  4 14:27:58 foo kernel: device-mapper: thin: 253:7: aborting current metadata transaction
Jul  4 14:27:58 foo lvm[826]: Thin vg_foo-docker--pool is now 100% full.
Jul  4 14:27:58 foo kernel: device-mapper: thin: 253:7: switching pool to read-only mode
Jul  4 14:27:58 foo lvm[826]: Insufficient free space: 23 extents needed, but only 10 available
Jul  4 14:27:58 foo lvm[826]: Failed to extend thin vg_foo-docker--pool.
...
Jul  4 14:27:58 foo kernel: XFS (dm-9): metadata I/O error: block 0xc7fc00 ("xfs_buf_iodone_callbacks") error 5 numblks 16
Jul  4 14:27:58 foo kernel: Buffer I/O error on device dm-9, logical block 1037664
Jul  4 14:27:58 foo kernel: lost page write due to I/O error on dm-9
...
Jul  4 14:27:58 foo kernel: XFS (dm-9): Detected failing async write on buffer block 0x12c6c10. Retrying async write.
Jul  4 14:27:58 foo kernel: XFS (dm-9): Detected failing async write on buffer block 0x12c6c00. Retrying async write.

• The above sequence of lvm and XFS errors repeates ad infinitum.

Jul  4 14:28:03 foo kernel: XFS (dm-9): metadata I/O error: block 0x12c6c00 ("xfs_buf_iodone_callbacks") error 5 numblks 16
Jul  4 14:28:06 foo lvm[826]: Thin vg_foo-docker--pool is now 100% full.
Jul  4 14:28:06 foo lvm[826]: Insufficient free space: 23 extents needed, but only 10 available
Jul  4 14:28:06 foo lvm[826]: Failed to extend thin vg_foo-docker--pool.
Jul  4 14:28:08 foo kernel: XFS (dm-9): metadata I/O error: block 0x12c6c00 ("xfs_buf_iodone_callbacks") error 5 numblks 16
Jul  4 14:28:14 foo kernel: XFS (dm-9): metadata I/O error: block 0x12c6c00 ("xfs_buf_iodone_callbacks") error 5 numblks 16

Jul  4 14:28:59 foo kernel: XFS (dm-9): Detected failing async write on buffer block 0x12c6c10. Retrying async write.

Jul  4 14:28:59 foo kernel: XFS (dm-9): Detected failing async write on buffer block 0x12c6c00. Retrying async write.

Jul  4 14:28:59 foo kernel: XFS (dm-9): metadata I/O error: block 0x12c6c00 ("xfs_buf_iodone_callbacks") error 5 numblks 16
Jul  4 14:29:04 foo kernel: XFS (dm-9): metadata I/O error: block 0x12c6c00 ("xfs_buf_iodone_callbacks") error 5 numblks 16
...

• Eventually we see a kworker task and yum-builddep task both blocked in xfs_log_reserve -> xfs_log_grant_head_check -> xfs_log_grant_head_wait indefinitely.

• It looks like docker is out of space, but all my filesystems have free space, so why is docker out of space?

[root@foo kerneltest]# docker info
Containers: 4
Images: 19
Storage Driver: devicemapper
 Pool Name: vg_foo-docker--pool
 Pool Blocksize: 524.3 kB
 Backing Filesystem: extfs
 Data file: 
 Metadata file: 
 Data Space Used: 3.792 GB
 Data Space Total: 3.792 GB
 Data Space Available: 0 B
 Metadata Space Used: 618.5 kB
 Metadata Space Total: 268.4 MB
 Metadata Space Available: 267.8 MB
 Udev Sync Supported: true
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Kernel Version: 3.10.0-229.7.2.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.1 (Maipo)
CPUs: 8
Total Memory: 33.16 GiB
Name: foo..bar.com
ID: BMBZ:4QU7:ZCJT:R5XO:U76M:2FN7:TTCP:RVWL:RMXM:XAC2:IPZD:XKKS
[root@foo kerneltest]# df -h
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/vg_foo-lv_root        50G   37G   10G  79% /
devtmpfs                             17G     0   17G   0% /dev
tmpfs                                17G     0   17G   0% /dev/shm
tmpfs                                17G  114M   17G   1% /run
tmpfs                                17G     0   17G   0% /sys/fs/cgroup
/dev/sda2                           202G  145G   48G  76% /mnt/sda2
/dev/sda1                            74G   61G  9.0G  88% /mnt/sda1
/dev/sdb1                           477M  141M  307M  32% /boot
/dev/mapper/vg_foo-lv_home       9.8G  204M  9.1G   3% /home
/dev/mapper/vg_foo-lv_data_xfs    10G  4.7G  5.3G  48% /mnt/fs_xfs
/dev/mapper/vg_foo-lv_data_ext4  121G   84G   31G  74% /mnt/fs_ext4