Ghosts of Block-devs Past?

Recently, a fairly junior technician was added to our staff. As part of the tech's onboarding, they were given the task of reviewing current standard operating procedures. This was done both to ensure that the SoPs were clear enough for someone very junior to use and to familiarize the tech with day-to-day tasksings.

This week, while working through SoPs related to SAN-based storage for physical Linux systems, the tech either ran into or created a really odd configuration error. Specifically, when attempting to run through the procecures to take some CLARiiON based LUN and get the pulled under control of dm-multipath and LVM, two mpathX devices (sort of) ended up in dm-multipath's database and both seemed to refer to the same LUN.

When the tech called me over to help - was getting "filesystem mounted?" errors when trying to run `pvcreate` the first bit of odditiy I noticed was that there was an "mpath1" device in multipath -ll's output but no "mpath0" device. The fdisk utility's output was showing, in addition to the /dev/sdX devices for the CLARiiON, a dm-7p1 device and a dm-8 device.

In tryint to work through the "filesystem mounted?" errors, I'd had the tech clear and re-do the partition tables (fdisk had shown layout errors on the /dev/sdX devices and was trying to render them consistent/good). However, after resetting the partitions, we noticed that the mis-partitioning came back and that the pooched partitioning matched what was on the dm-7p1 fdisk entries rather than the dm-8 entries.

Ultimately, ended up having the tech do a `multipath -f <DEVIDSTRING>` to flush out dm-multipath's entries. This got rid of the ghost and allowed the LUN to be correctly partitioned and got rid of the "filesystem mounted?" errors that `pvcreate` was spitting out. I just can't figure out how the system got into such an odd state in the first place. Anyone ever encounter something like this?