Error starting POD because of docker storage "error"
Issue
- When trying to start a POD, sometimes the POD is simply "hanging", because the image can't be pulled from the docker registry. We are trying to narrow the error, without success.
In the Event log we see:
Failed to pull image "172.30.x.x:5000/proj/example-mq@sha256:5ce3e037b11373286fa63078a15525372de5630872ecc8e43eecf165148c7671": Error activating devmapper device for '48665f80f9f5a1ea16c8ad5435039418d3fbf6d107e967b225c21929caee8da0': Error running DeviceCreate (ActivateDevice) dm_task_run failed
On the OpenShift node:
# docker info
Containers: 123
Images: 1258
Storage Driver: devicemapper
Pool Name: docker--vg-docker--pool
Pool Blocksize: 524.3 kB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 80.28 GB
Data Space Total: 159.4 GB
Data Space Available: 79.14 GB
Metadata Space Used: 40.19 MB
Metadata Space Total: 83.89 MB
Metadata Space Available: 43.7 MB
Udev Sync Supported: true
Deferred Removal Enabled: true
Library Version: 1.02.107-RHEL7 (2015-12-01)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: OpenShift Enterprise
CPUs: 4
Total Memory: 7.64 GiB
Name: cvm01722.bmwgroup.net
ID: 27WA:2F7C:WFQJ:LRSO:UXOM:7NEJ:Q7OR:DWDL:VQ4P:W2P3:TPPE:26TY
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
The pod then is hanging endless.
When I scale the POD up to '2', the new POD is created successfully, so we have one "blue" and one yellow POD (s. screenshot). So the main configuration seems to work, as for the most time.
The only hint I have is on the node in 'dmesg':
[1719455.183114] device-mapper: thin: 252:2: growing the data device from 253384 to 304064 blocks
[1719482.213470] device-mapper: thin: 252:2: growing the metadata device from 16384 to 20480 blocks
This may correlate with the error.
Is there any docker storage configuration recomandation except the official documentation?
Environment
- Openshift Enterprise 3.1.1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
