Error starting POD because of docker storage "error"

Solution In Progress - Updated -

Issue

  • When trying to start a POD, sometimes the POD is simply "hanging", because the image can't be pulled from the docker registry. We are trying to narrow the error, without success.

In the Event log we see:

Failed to pull image "172.30.x.x:5000/proj/example-mq@sha256:5ce3e037b11373286fa63078a15525372de5630872ecc8e43eecf165148c7671": Error activating devmapper device for '48665f80f9f5a1ea16c8ad5435039418d3fbf6d107e967b225c21929caee8da0': Error running DeviceCreate (ActivateDevice) dm_task_run failed

On the OpenShift node:

# docker info
Containers: 123
Images: 1258
Storage Driver: devicemapper
 Pool Name: docker--vg-docker--pool
 Pool Blocksize: 524.3 kB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 80.28 GB
 Data Space Total: 159.4 GB
 Data Space Available: 79.14 GB
 Metadata Space Used: 40.19 MB
 Metadata Space Total: 83.89 MB
 Metadata Space Available: 43.7 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Library Version: 1.02.107-RHEL7 (2015-12-01)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: OpenShift Enterprise
CPUs: 4
Total Memory: 7.64 GiB
Name: cvm01722.bmwgroup.net
ID: 27WA:2F7C:WFQJ:LRSO:UXOM:7NEJ:Q7OR:DWDL:VQ4P:W2P3:TPPE:26TY
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

The pod then is hanging endless.

When I scale the POD up to '2', the new POD is created successfully, so we have one "blue" and one yellow POD (s. screenshot). So the main configuration seems to work, as for the most time.

The only hint I have is on the node in 'dmesg':

[1719455.183114] device-mapper: thin: 252:2: growing the data device from 253384 to 304064 blocks
[1719482.213470] device-mapper: thin: 252:2: growing the metadata device from 16384 to 20480 blocks

This may correlate with the error.
Is there any docker storage configuration recomandation except the official documentation?

Environment

  • Openshift Enterprise 3.1.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.