OSD pods in CrashLoopBackOff after ODF upgrade logging : VG name on command line not found in list of VGs
Issue
- After an upgrade from ODF 4.9 to ODF 4.10 rook is unable to successfully start the OSD pods and they remain in CrashLoopBackOff state.
Example of the errors from the osd pod log:
2023-02-05T21:07:25.313080407Z 2023-02-05 21:07:25.312697 I | cephosd: shutdown signal received, exiting...
2023-02-05T21:07:25.318999465Z 2023-02-05 21:07:25.317954 I | cephosd: process ID for ceph-osd: 111
2023-02-05T21:07:26.512949696Z 2023-02-05 21:07:26.512905 E | cephosd: failed to start osd or shutting down. signal: killed
2023-02-05T21:07:26.710758351Z 2023-02-05 21:07:26.706305 E | cephosd: failed to release device from lvm. failed to deactivate LVM . output: Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will manage logical volume symlinks in device directory.
2023-02-05T21:07:26.710758351Z Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will obtain device list by scanning device directory.
2023-02-05T21:07:26.710758351Z devices/hints not found in config: defaulting to "all"
2023-02-05T21:07:26.710758351Z metadata/record_lvs_history not found in config: defaulting to 0
2023-02-05T21:07:26.710758351Z devices/search_for_devnames not found in config: defaulting to "auto"
2023-02-05T21:07:26.710758351Z global/locking_type not found in config: defaulting to 1
2023-02-05T21:07:26.710758351Z devices/md_component_checks not found in config: defaulting to "auto"
2023-02-05T21:07:26.710758351Z report/output_format not found in config: defaulting to "basic"
2023-02-05T21:07:26.710758351Z log/report_command_log not found in config: defaulting to 0
2023-02-05T21:07:26.710758351Z devices/use_devicesfile not found in config: defaulting to 0
2023-02-05T21:07:26.710758351Z /dev/initctl: stat failed: No such file or directory
2023-02-05T21:07:26.710758351Z /dev/log: stat failed: No such file or directory
2023-02-05T21:07:26.710758351Z /dev/sdb: size is 7500201984 sectors
2023-02-05T21:07:26.710758351Z devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
2023-02-05T21:07:26.710758351Z Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
2023-02-05T21:07:26.710758351Z devices/devicesfile not found in config: defaulting to "system.devices"
2023-02-05T21:07:26.710758351Z /dev/sdb: using cached size 7500201984 sectors
2023-02-05T21:07:26.710758351Z /dev/sdb: using cached size 7500201984 sectors
2023-02-05T21:07:26.710758351Z /dev/sdb: using cached size 7500201984 sectors
2023-02-05T21:07:26.710758351Z /dev/sdb: lvm2 label detected at sector 1
2023-02-05T21:07:26.710758351Z Obtaining the complete list of VGs before processing their LVs
2023-02-05T21:07:26.710758351Z VG name on command line not found in list of VGs:
2023-02-05T21:07:26.710758351Z Processing VG
2023-02-05T21:07:26.710758351Z Volume group name "" has invalid characters.
2023-02-05T21:07:26.710758351Z Cannot process volume group: exit status 5
Environment
- OpenShift Data Foundation 4.8
- OpenShift Data Foundation 4.9
- OpenShift Data Foundation 4.10
- Upgrade
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.