How to segregate RBD images within isolated namespaces in ceph?

Solution Verified - Updated -

Environment

  • Red Hat Ceph Storage 4.x
  • rbd

Issue

  • How to segregate RBD images within isolated namespaces in ceph?
  • While trying to map the rbd image it fails with:
[12854.233079] Key type dns_resolver registered
[12854.253573] Key type ceph registered
[12854.253701] libceph: loaded (mon/osd proto 15/24)
[12854.259652] rbd: loaded (major 251)
[12854.268854] libceph: bad option at '_pool_ns=<namespace>'

Resolution

1. Create one pool for rbd:

# ceph osd pool create rbd 100 100
pool 'rbd' created
# ceph osd pool application enable rbd rbd
enabled application 'rbd' on pool 'rbd'
# rbd pool init -p rbd

2. Create namespaces:

# rbd namespace ls
# rbd namespace create --namespace testnamespace
# rbd namespace create --namespace othernamespace
# rbd namespace ls --format=json
[{"name":"othernamespace"},{"name":"testnamespace"}]

3. Give the access to to the namespace for two users:

# ceph auth get-or-create client.rbduser mon 'profile rbd' osd 'profile rbd pool=rbd namespace=testnamespace' -o /etc/ceph/client.rbduser.keyring
# ceph auth get-or-create client.otheruser mon 'profile rbd' osd 'profile rbd pool=rbd namespace=othernamespace' -o /etc/ceph/client.otheruser.keyring
# ceph auth get client.rbduser
exported keyring for client.rbduser
[client.rbduser]
        key = AQCMqlxe19tAARAAeu5btSGePf9QbHOJTZZ5hg==
        caps mon = "profile rbd"
        caps osd = "profile rbd pool=rbd namespace=testnamespace"
# ceph auth get client.otheruser
exported keyring for client.otheruser
[client.otheruser]
        key = AQBix1xeJvx2JxAAgz8UZwEX/Z1mmdVf+n5L6A==
        caps mon = "profile rbd"
        caps osd = "profile rbd pool=rbd namespace=othernamespace"

4. Create images and use the pre-defined namespace within a pool:

# rbd create --namespace testnamespace rbdimg1 --size 1G
# rbd create --namespace othernamespace rbdimg2 --size 1G
# rbd ls --long
# rbd --namespace testnamespace ls --long
NAME    SIZE  PARENT FMT PROT LOCK
rbdimg1 1 GiB          2
# rbd --namespace othernamespace ls --long
NAME    SIZE  PARENT FMT PROT LOCK
rbdimg2 1 GiB          2
# rbd feature disable rbd/testnamespace/rbdimg1 object-map fast-diff deep-flatten
# rbd feature disable rbd/othernamespace/rbdimg2 object-map fast-diff deep-flatten

5. Copy key and ceph.conf files to the RBD client.

6. Map the RBD image:

# rbd map --namespace testnamespace rbdimg1  -n  client.rbduser --keyring=/etc/ceph/client.rbduser.keyring
/dev/rbd0
# rbd map --namespace othernamespace rbdimg2 -n client.otheruser --keyring=/etc/ceph/client.otheruser.keyring
/dev/rbd1
  • But it won't permit to access in the other namespaces in the pool:
# rbd map --namespace othernamespace rbdimg2  -n  client.rbduser --keyring=/etc/ceph/client.rbduser.keyring
rbd: sysfs write failed
rbd: error asserting namespace: (1) Operation not permitted
In some cases useful info is found in syslog - try "dmesg | tail".
2020-03-04 12:12:29.427 7f6b90778040 -1 librbd::api::Namespace: exists: error asserting namespace: (1) Operation not permitted
rbd: map failed: (1) Operation not permitted
# rbd map --namespace testnamespace rbdimg1 -n client.otheruser --keyring=/etc/ceph/client.otheruser.keyring
rbd: warning: image already mapped as /dev/rbd0
rbd: sysfs write failed
rbd: error asserting namespace: (1) Operation not permitted
In some cases useful info is found in syslog - try "dmesg | tail".
2020-03-04 12:16:24.011 7fcad776a040 -1 librbd::api::Namespace: exists: error asserting namespace: (1) Operation not permitted
rbd: map failed: (1) Operation not permitted

7. Verify the device:

# rbd showmapped
id pool namespace      image   snap device
0  rbd  testnamespace  rbdimg1 -    /dev/rbd0
1  rbd  othernamespace rbdimg2 -    /dev/rbd1

8. Create the filesystem on the devices and use it.

Root Cause

  • RBD images can now be segregated within isolated namespaces within the same pool.
  • When using Ceph Block Devices directly without a higher-level system, such as OpenStack or OpenShift Container Storage, it was not possible to restrict user access to specific RBD images.
  • When combined with CephX capabilities, users can be restricted to specific pool namespaces to restrict access to RBD images.

Diagnostic Steps

  • To map a rbd (+ namespace) it needs a kernel >= 4.x. Otherwise it may throw the below error:
# uname -r
3.10.0-1062.el7.x86_64
# rbd map --namespace testnamespace rbd/rbdimg1 -n  client.rbduser --keyring=/etc/ceph/client.rbduser.keyring
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (22) Invalid argument
# dmesg | tail
[    9.485487] Bridge firewalling registered
[    9.813644] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[   10.682032] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   10.841660] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   10.955307] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[12854.233079] Key type dns_resolver registered
[12854.253573] Key type ceph registered
[12854.253701] libceph: loaded (mon/osd proto 15/24)
[12854.259652] rbd: loaded (major 251)
[12854.268854] libceph: bad option at '_pool_ns=testnamespace'

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments