OSD pods in CrashLoopBackOff after ODF upgrade logging : VG name on command line not found in list of VGs

Solution Verified - Updated -

Issue

  • After an upgrade from ODF 4.9 to ODF 4.10 rook is unable to successfully start the OSD pods and they remain in CrashLoopBackOff state.

Example of the errors from the osd pod log:

2023-02-05T21:07:25.313080407Z 2023-02-05 21:07:25.312697 I | cephosd: shutdown signal received, exiting...
2023-02-05T21:07:25.318999465Z 2023-02-05 21:07:25.317954 I | cephosd: process ID for ceph-osd: 111
2023-02-05T21:07:26.512949696Z 2023-02-05 21:07:26.512905 E | cephosd: failed to start osd or shutting down. signal: killed
2023-02-05T21:07:26.710758351Z 2023-02-05 21:07:26.706305 E | cephosd: failed to release device from lvm. failed to deactivate LVM . output: Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will manage logical volume symlinks in device directory.
2023-02-05T21:07:26.710758351Z   Udev is running and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will obtain device list by scanning device directory.
2023-02-05T21:07:26.710758351Z   devices/hints not found in config: defaulting to "all"
2023-02-05T21:07:26.710758351Z   metadata/record_lvs_history not found in config: defaulting to 0
2023-02-05T21:07:26.710758351Z   devices/search_for_devnames not found in config: defaulting to "auto"
2023-02-05T21:07:26.710758351Z   global/locking_type not found in config: defaulting to 1
2023-02-05T21:07:26.710758351Z   devices/md_component_checks not found in config: defaulting to "auto"
2023-02-05T21:07:26.710758351Z   report/output_format not found in config: defaulting to "basic"
2023-02-05T21:07:26.710758351Z   log/report_command_log not found in config: defaulting to 0
2023-02-05T21:07:26.710758351Z   devices/use_devicesfile not found in config: defaulting to 0
2023-02-05T21:07:26.710758351Z   /dev/initctl: stat failed: No such file or directory
2023-02-05T21:07:26.710758351Z   /dev/log: stat failed: No such file or directory
2023-02-05T21:07:26.710758351Z   /dev/sdb: size is 7500201984 sectors
2023-02-05T21:07:26.710758351Z   devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
2023-02-05T21:07:26.710758351Z   Setting devices/filter to filter = [ "a|^/mnt/.*|", "r|.*|" ]
2023-02-05T21:07:26.710758351Z   devices/devicesfile not found in config: defaulting to "system.devices"
2023-02-05T21:07:26.710758351Z   /dev/sdb: using cached size 7500201984 sectors
2023-02-05T21:07:26.710758351Z   /dev/sdb: using cached size 7500201984 sectors
2023-02-05T21:07:26.710758351Z   /dev/sdb: using cached size 7500201984 sectors
2023-02-05T21:07:26.710758351Z   /dev/sdb: lvm2 label detected at sector 1
2023-02-05T21:07:26.710758351Z   Obtaining the complete list of VGs before processing their LVs
2023-02-05T21:07:26.710758351Z   VG name on command line not found in list of VGs: 
2023-02-05T21:07:26.710758351Z   Processing VG  
2023-02-05T21:07:26.710758351Z   Volume group name "" has invalid characters.
2023-02-05T21:07:26.710758351Z   Cannot process volume group: exit status 5

Environment

  • OpenShift Data Foundation 4.8
  • OpenShift Data Foundation 4.9
  • OpenShift Data Foundation 4.10
  • Upgrade

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content