Ceph: Unable to bring OSDs in when using IPv6. Error: "unable to find any IPv4 address in networks '192.x.y.0/24,192.x.y.0/24'" or "unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64'"

Solution Verified - Updated -

Issue

Cephadm unable to bring OSDs in when using IPv6.
OSD log: "unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64' interfaces ''".
OSD log: "unable to find any IPv4 address in networks '192.x.y.0/24,192.x.y.0/24' interfaces ''".
OSD log: "unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64'"

Wrong public IP example:

[root@edon-4 ~]# journalctl -u ceph-8d2323ac-0820-49ad-b917-059ba00794f2@osd.36.service -xe --no-pager  --since "10 min ago"
Aug 21 17:06:35 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  0 set uid:gid to 167:167 (ceph:ceph)
Aug 21 17:06:35 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  0 ceph version 16.2.10-187.el8cp (5d6355e2bccd18b5c6457a34cb666d773f21823d) pacific (stable), process ceph-osd, pid 7
Aug 21 17:06:35 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  0 pidfile_write: ignore empty --pid-file
Aug 21 17:06:35 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  1 bdev(0x561864503800 /var/lib/ceph/osd/ceph-36/block) open path /var/lib/ceph/osd/ceph-36/block
Aug 21 17:06:35 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  1 bdev(0x561864503800 /var/lib/ceph/osd/ceph-36/block) open size 3200606273536 (0x2e933000000, 2.9 TiB) block_size 4096 (4 KiB) non-rotational discard supported
Aug 21 17:06:35 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  1 bluestore(/var/lib/ceph/osd/ceph-36) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06
Aug 21 17:06:35 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  1 bdev(0x561864503400 /var/lib/ceph/osd/ceph-36/block) open path /var/lib/ceph/osd/ceph-36/block
Aug 21 17:06:35 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  1 bdev(0x561864503400 /var/lib/ceph/osd/ceph-36/block) open size 3200606273536 (0x2e933000000, 2.9 TiB) block_size 4096 (4 KiB) non-rotational discard supported
Aug 21 17:06:35 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-36/block size 2.9 TiB
Aug 21 17:06:35 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  1 bdev(0x561864503400 /var/lib/ceph/osd/ceph-36/block) close
Aug 21 17:06:36 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  1 bdev(0x561864503800 /var/lib/ceph/osd/ceph-36/block) close
Aug 21 17:06:36 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  1  objectstore numa_node 0
Aug 21 17:06:36 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200  0 starting osd.36 osd_data /var/lib/ceph/osd/ceph-36 /var/lib/ceph/osd/ceph-36/journal
Aug 21 17:06:36 edon-4 ceph-redacted-osd-36[3812225]: 7f108c3a6200 -1 unable to find any IPv4 address in networks '192.78.x.0/24,192.78.y.0/24' interfaces ''  <-- Here
Aug 21 17:07:04 edon-4 ceph-redacted-osd-36[3814241]: 7ffa12290200  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-36/block size 2.9 TiB
Aug 21 17:07:04 edon-4 ceph-redacted-osd-36[3814241]: 7ffa12290200  1 bdev(0x562a404f5400 /var/lib/ceph/osd/ceph-36/block) close
Aug 21 17:07:04 edon-4 ceph-redacted-osd-36[3814241]: 7ffa12290200  1 bdev(0x562a404f5800 /var/lib/ceph/osd/ceph-36/block) close
Aug 21 17:07:04 edon-4 ceph-redacted-osd-36[3814241]: 7ffa12290200  1  objectstore numa_node 0
Aug 21 17:07:04 edon-4 ceph-redacted-osd-36[3814241]: 7ffa12290200  0 starting osd.36 osd_data /var/lib/ceph/osd/ceph-36 /var/lib/ceph/osd/ceph-36/journal
Aug 21 17:07:04 edon-4 ceph-redacted-osd-36[3814241]: 7ffa12290200 -1 unable to find any IPv4 address in networks '192.78.x.0/24,192.78.y.0/24' interfaces ''  <-- Here
Aug 21 17:07:04 edon-4 ceph-redacted-osd-36[3814241]: 7ffa12290200 -1 Failed to pick public address.
Aug 21 17:07:04 edon-4 systemd[1]: ceph-redacted@osd.36.service: Main process exited, code=exited, status=1/FAILURE
Aug 21 17:07:05 edon-4 systemd[1]: ceph-redacted@osd.36.service: Failed with result 'exit-code'.

IPV4/IPv6 confusion example (code issue in RHCS 5.0.0):

$ journalctl u ceph-313c101e-9947-43ee-b60a-348a3aa357fd@osd.8 -xe --no-pager --since today
...
Oct 06 14:51:37 ceph-0 conmon39128: 7fd3bb3f6080 1 bdev(0x5650f3c20c00 /var/lib/ceph/osd/ceph-8/block) close
Oct 06 14:51:38 ceph-0 conmon39128: 7fd3bb3f6080 0 starting osd.8 osd_data /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
Oct 06 14:51:38 ceph-0 conmon39128: 7fd3bb3f6080 -1 unable to find any IPv4 address in networks 'fd00:fd00:fd00:3000::/64' interfaces ''  <-- Here
Oct 06 14:51:38 ceph-0 conmon39128: 7fd3bb3f6080 -1 Failed to pick public address.
...

Environment

Red Hat Ceph Storage 4.x
Red Hat Ceph Storage 5.x
Red Hat Ceph Storage 6.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content