Ceph Monitor has hardcoded IPs in Nova database
Environment
- Red Hat OpenStack Platform 8 (Liberty)
Issue
- Ceph monitor IPs are hardcode in the nova.block_device_mappings database.
- If Ceph monitors are replaced, it could be a real problem because there is not a tool to update all fields in the Nova database.
Resolution
- The resolution is in progress in the BZ 1414124.
Diagnostic Steps
- Create a new instance:
# nova boot --flavor A2 --block-device source=image,id=35a49dbb-c975-46e1-b5d7-e77d704b39e3,dest=volume,size=60,shutdown=remove,bootindex=0 --availability-zone AZ1-2 test-consulting-ceph-monitor-before
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | AZ1-2 |
| OS-EXT-SRV-ATTR:host | node1.tam.local |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node1.tam.local |
| OS-EXT-SRV-ATTR:instance_name | inst-00000300 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2017-02-16T14:27:52.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2017-02-16T14:27:39Z |
| flavor | A2 (1002) |
| hostId | c01ca1f1e941c704d84106fe0a67708f1897bd4d3e0788a773993d2a |
| id | d61d23a4-b085-4f12-b3c3-eaacaa7ac33a |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| metadata | {} |
| name | test-consulting-ceph-monitor-before |
| os-extended-volumes:volumes_attached | [{"id": "a5a69591-5aab-4aa8-86bf-f638a424990b"}] |
| progress | 0 |
| status | ACTIVE |
| tenant_id | f7697e2b770540558695168119dfbf08 |
| updated | 2017-02-16T14:27:52Z |
| user_id | admin |
+--------------------------------------+----------------------------------------------------------+
- Check Ceph monitor IPs in nova.block_device_mapping:
MariaDB [nova]> select * from nova.block_device_mapping where instance_uuid="d61d23a4-b085-4f12-b3c3-eaacaa7ac33a"\G
*************************** 1. row ***************************
created_at: 2017-02-16 14:27:40
updated_at: 2017-02-16 14:27:52
deleted_at: NULL
id: 756
device_name: /dev/vda
delete_on_termination: 1
snapshot_id: NULL
volume_id: a5a69591-5aab-4aa8-86bf-f638a424990b
volume_size: 60
no_device: 0
connection_info: {"driver_volume_type": "rbd", "connector": {"initiator": "iqn.1994-05.com.redhat:a114ee6e254", "ip": "10.229.156.21", "platform": "x86_64", "host": "node1.tam.local", "os_type": "linux2", "multipath": false}, "serial": "a5a69591-5aab-4aa8-86bf-f638a424990b", "data": {"secret_type": "ceph", "name": "volumes/volume-a5a69591-5aab-4aa8-86bf-f638a424990b", "encrypted": false, "secret_uuid": "d9bdd614-d20d-4a1f-8446-65a30b1bb009", "qos_specs": {"read_bytes_sec": "52428800", "write_iops_sec": "100", "write_bytes_sec": "52428800", "read_iops_sec": "100"}, "hosts": ["10.232.0.73", "10.232.28.21", "10.232.28.22", "10.232.28.23", "10.232.28.24"], "volume_id": "a5a69591-5aab-4aa8-86bf-f638a424990b", "auth_enabled": true, "access_mode": "rw", "auth_username": "volumes", "ports": ["6789", "6789", "6789", "6789", "6789"]}}
instance_uuid: d61d23a4-b085-4f12-b3c3-eaacaa7ac33a
deleted: 0
source_type: image
destination_type: volume
guest_format: NULL
device_type: disk
disk_bus: virtio
boot_index: 0
image_id: 35a49dbb-c975-46e1-b5d7-e77d704b39e3
1 row in set (0.00 sec)
Ceph monitor IPs: "hosts": ["10.232.0.73", "10.232.28.21", "10.232.28.22", "10.232.28.23", "10.232.28.24"]
- Add a new monitor: We have followed procedure attached replace_monitors.pdf
# ceph -s
cluster 129430ee-827a-41f7-8385-d40422742400
health HEALTH_OK
monmap e2: 6 mons at {srvmes102cnodep01=10.232.28.21:6789/0,srvmes102cnodep02=10.232.28.22:6789/0,srvmes102cnodep03=10.232.28.23:6789/0,srvmes102cnodep04=10.232.28.24:6789/0,vmmes100cephmp01=10.232.0.45:6789/0,vmmes100cmonp02=10.232.0.73:6789/0}
election epoch 10, quorum 0,1,2,3,4,5 vmmes100cephmp01,vmmes100cmonp02,srvmes102cnodep01,srvmes102cnodep02,srvmes102cnodep03,srvmes102cnodep04
osdmap e19728: 38 osds: 38 up, 38 in
pgmap v902293: 1792 pgs, 5 pools, 363 GB data, 82273 objects
1121 GB used, 104 TB / 105 TB avail
1792 active+clean
- Boot a new instance:
# nova boot --flavor A2 --block-device source=image,id=35a49dbb-c975-46e1-b5d7-e77d704b39e3,dest=volume,size=60,shutdown=remove,bootindex=0 --availability-zone AZ1-2 test-consulting-ceph-monitor-after
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | AZ1-2 |
| OS-EXT-SRV-ATTR:host | node2.tam.local |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node2.tam.local |
| OS-EXT-SRV-ATTR:instance_name | inst-00000333 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2017-02-16T15:50:26.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2017-02-16T15:50:14Z |
| flavor | A2 (1002) |
| hostId | 42352d6d514527f13061236f3a182461ab4c7058f651b25fd92fd0a5 |
| id | 5b29b652-7d48-4a2a-b58e-fc795a9a6e25 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| metadata | {} |
| name | test-consulting-ceph-monitor-after |
| os-extended-volumes:volumes_attached | [{"id": "9b8008de-e704-432c-b837-53030e030456"}] |
| progress | 0 |
| status | ACTIVE |
| tenant_id | f7697e2b770540558695168119dfbf08 |
| updated | 2017-02-16T15:50:26Z |
| user_id | admin |
+--------------------------------------+----------------------------------------------------------+
- Check Ceph monitor IPs in nova.block_device_mapping (new instance):
MariaDB [nova]> select * from nova.block_device_mapping where instance_uuid="5b29b652-7d48-4a2a-b58e-fc795a9a6e25"\G
*************************** 1. row ***************************
created_at: 2017-02-16 15:50:14
updated_at: 2017-02-16 15:50:26
deleted_at: NULL
id: 807
device_name: /dev/vda
delete_on_termination: 1
snapshot_id: NULL
volume_id: 9b8008de-e704-432c-b837-53030e030456
volume_size: 60
no_device: 0
connection_info: {"driver_volume_type": "rbd", "connector": {"initiator": "iqn.1994-05.com.redhat:4f18d539d1", "ip": "10.229.156.23", "platform": "x86_64", "host": "node2.tam.local", "os_type": "linux2", "multipath": false}, "serial": "9b8008de-e704-432c-b837-53030e030456", "data": {"secret_type": "ceph", "name": "volumes/volume-9b8008de-e704-432c-b837-53030e030456", "encrypted": false, "secret_uuid": "d9bdd614-d20d-4a1f-8446-65a30b1bb009", "qos_specs": {"read_bytes_sec": "52428800", "write_iops_sec": "100", "write_bytes_sec": "52428800", "read_iops_sec": "100"}, "hosts": ["10.232.0.45", "10.232.0.73", "10.232.28.21", "10.232.28.22", "10.232.28.23", "10.232.28.24"], "volume_id": "9b8008de-e704-432c-b837-53030e030456", "auth_enabled": true, "access_mode": "rw", "auth_username": "volumes", "ports": ["6789", "6789", "6789", "6789", "6789", "6789"]}}
instance_uuid: 5b29b652-7d48-4a2a-b58e-fc795a9a6e25
deleted: 0
source_type: image
destination_type: volume
guest_format: NULL
device_type: disk
disk_bus: virtio
boot_index: 0
image_id: 35a49dbb-c975-46e1-b5d7-e77d704b39e3
1 row in set (0.00 sec)
6-Check Ceph monitor IPs in nova.block_device_mapping (old instance):
MariaDB [nova]> select * from nova.block_device_mapping where instance_uuid="d61d23a4-b085-4f12-b3c3-eaacaa7ac33a"\G
*************************** 1. row ***************************
created_at: 2017-02-16 14:27:40
updated_at: 2017-02-16 14:27:52
deleted_at: NULL
id: 756
device_name: /dev/vda
delete_on_termination: 1
snapshot_id: NULL
volume_id: a5a69591-5aab-4aa8-86bf-f638a424990b
volume_size: 60
no_device: 0
connection_info: {"driver_volume_type": "rbd", "connector": {"initiator": "iqn.1994-05.com.redhat:a114ee6e254", "ip": "10.229.156.21", "platform": "x86_64", "host": "node1.tam.local", "os_type": "linux2", "multipath": false}, "serial": "a5a69591-5aab-4aa8-86bf-f638a424990b", "data": {"secret_type": "ceph", "name": "volumes/volume-a5a69591-5aab-4aa8-86bf-f638a424990b", "encrypted": false, "secret_uuid": "d9bdd614-d20d-4a1f-8446-65a30b1bb009", "qos_specs": {"read_bytes_sec": "52428800", "write_iops_sec": "100", "write_bytes_sec": "52428800", "read_iops_sec": "100"}, "hosts": ["10.232.0.73", "10.232.28.21", "10.232.28.22", "10.232.28.23", "10.232.28.24"], "volume_id": "a5a69591-5aab-4aa8-86bf-f638a424990b", "auth_enabled": true, "access_mode": "rw", "auth_username": "volumes", "ports": ["6789", "6789", "6789", "6789", "6789"]}}
instance_uuid: d61d23a4-b085-4f12-b3c3-eaacaa7ac33a
deleted: 0
source_type: image
destination_type: volume
guest_format: NULL
device_type: disk
disk_bus: virtio
boot_index: 0
image_id: 35a49dbb-c975-46e1-b5d7-e77d704b39e3
1 row in set (0.00 sec)
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments