Raw Disk Mapping.

Latest response

We did P2V all redhat 5.5 physical server to virtual.
Here is the situation we ran into.After we P2V’d the servers, and moved all of their data to virtual drives, when I went to remove the RDMs from one server as a test, it actually rearranged all of the drives within the operating system after a reboot and oracle services won't start. We were able recover from it but I would like some help in understanding what caused the issue and how we can move forward with the remaining servers while avoiding the same thing. Customer using Oracle ASM.

Thanks

Responses

You need to create mappings in UDEV to create the ASM devices based on the UUID of the disk (or another unique identifier), this ensures that the disks falling out of order will not cause issues in ASM. The /dev/sd* devices shouldn't be used directly in ASM. The reason it likely occurred is that other block devices were mapped through to the VM or the existing disks were setup in a different order in the VM (order of devices isn't guaranteed)

Oracle recommends using UUID of the disk to identify it and generate the rule in the udev configuration. If you are using VMWare you need to ensure you have the configuration option 'disk.EnableUUID' set to true to ensure the UUID is being passed through. Without this option in VMWare, the SCSI command to retrieve the UUID will fail.

The commend in the Oracle documentation is as follows:

scsi_id --page=0x83 --whitelisted --device=/dev/sdb

This will return a unique device string off the disk that looks similar to an md5 hash.

Using this hash you can create udev rules to create the ASM device mappings (which are just symlinks). The udev configuration line looks like this:

KERNEL=="sd*", PROGRAM=="scsi_id --page=0x83 --whitelisted --device=/dev/%k", RESULT=="3600605b00162841221e285f92326df21", SYMLINK+="oracleasm/disks/DSK1", OWNER="oracle", GROUP="dba", MODE="0660"

This will map all the ASM disks into /dev/oracleasm/disks/XXXX, the /dev/oracleasm/disks path should then be used as the base path in the ASM configuration to search for block devices.

ASM is very temperamental when disks are out of order (which is painful), but if you need to recover from this there are commands you can pass to the raw disk devices to pull their ASM labels off them. From there you can assemble the device mappings in order and get ASM to mount cleanly

I maybe a day later and a dollar short on this one. However I have to ask, as you mention there are ways to pass the raw disk devices to pull their ASM labels off of them. as I'm dealing with a situation that I detailed in another post here, however I'm looking for more info in order to troubleshoot to see if I can save this system.

I didn't have the RHEL OS setup with either UUID or udev and we had an issue with our SAN and now we can't get map back to those disks on the SAN that hold the Oracleasm info.

Here is the status for Oracleasm on that RHEL server:

[root@server init.d]# ./oracleasm status

Checking if ASM is loaded: yes

Checking if /dev/oracleasm is mounted: yes



[root@server init.d]# ./oracleasm listdisks

DATA0

DATA1

DATA2

DATA3

DATA4

DATA5

FRA1

FRA2



[root@server init.d]# ./oracleasm scandisks

Scanning the system for Oracle ASMLib disks:               [  OK  ]


[root@server init.d]#


[root@server init.d]# oracleasm querydisk -d DATA0

Disk "DATA0" is a valid ASM disk on device /dev/sdd1[8,49]



[root@server init.d]# oracleasm querydisk -d DATA1

Disk "DATA1" is a valid ASM disk on device /dev/sde1[8,65]



[root@server init.d]# oracleasm querydisk -d DATA2

Disk "DATA2" is a valid ASM disk on device /dev/sdf1[8,81]



[root@server init.d]# oracleasm querydisk -d DATA3

Disk "DATA3" is a valid ASM disk on device /dev/sdg1[8,97]



[root@server init.d]# oracleasm querydisk -d DATA4

Disk "DATA4" is a valid ASM disk on device /dev/sdh1[8,113]



[root@server init.d]# oracleasm querydisk -d DATA5

Disk "DATA5" is a valid ASM disk on device /dev/sdi1[8,129]



[root@server init.d]# oracleasm querydisk -d FRA1

Disk "FRA1" is a valid ASM disk on device /dev/sdj1[8,145]



[root@server init.d]# oracleasm querydisk -d FRA2

Disk "FRA2" is a valid ASM disk on device /dev/sdk1[8,161]

Oracle is having the issue and can't see some of the disks that are allocated to the Oracleasm.

So we stuck. We're not sure if we want to just add another disks to the OS and then allocate it to Oracleasm. However we want to do it right and would have to set it up under UUID and udev so if the SAN fails again, we don't have to go thru this again.

We have a Netapp SAN and I'm looking for commands that I can use to see if I can get mapping info so we could try to re-map from Oracleasm back to the SAN.

Do you know what the name of the missing disk should be? Your DATA disks are numbered starting from 0, but the FRA disks starting from 1. If your naming is consistent, that would mean that a disk called FRA0 would be missing. And the listing also shows the visible ASM disks as /dev/sd[d-k]. I assume that /dev/sda is probably your system disk... but do /dev/sdb and /dev/sdc contain filesystems or should one of them be used by ASM?

The presence of the "oracleasm querydisk" command indicates you have ASMLib, and the command tells you the standard device name of each ASM disk device it sees. So, you now know that DATA0 is stored on /dev/sdd1, the first partition of /dev/sdd. Then, you can find more information on that device from the corresponding sysfs directory: /sys/block/sdd.

For example, /sys/block/sdd/device/wwid should contain the WWID of the LUN, which should be useful information for the NetApp SAN administrator. By working together in this way with the SAN admin, you should be able to map the visible devices all the way back to the SAN storage.

Alternatively, the scsi_id command can be used to query the WWID of any visible LUN using its /dev/sd* whole-disk device.

To identify an ASM disk without any Oracle-specific tools, you can do this:

# dd if=/dev/sdX1 bs=128 count=1 | od -t x1z 
0000000 00 00 00 00 00 00 00 00 00 00 00 00 4e 45 57 32  >............NEW2<
0000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
0000040 4f 52 43 4c 44 49 53 4b 4e 45 57 32 00 00 00 00  >ORCLDISKNEW2....<
0000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
1+0 records in
1+0 records out

If the third line of the hex dump begins with the characters "ORCLDISK", then the disk has been at least prepared for use for ASM. The ASM name of the disk is also visible: in this case, this disk would be named NEW2 in ASM.

Yes, I know the name of the missing FRA disks. But to complicate matters more, we aren't using Raw Disks from the various SANs that we have. We are using VMFS on those LUNs.

So I'm not how much this will impact this.

Hi
Thanks for response.

Ok I did enabled disk.EnabledUUID part in VMware. And this are my vmdks that are getting changed after reboot.

lrwxrwxrwx 1 root root 10 Dec 13 14:36 0acb4b96-c2c3-4a62-8510-ee96a0c1ff07 -> ../../sda3
lrwxrwxrwx 1 root root 10 Dec 13 14:36 1e57c4a8-fa01-49be-adea-9383a958ff17 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Dec 13 14:36 7c589c31-976e-44d2-b0fa-63d691c6f9f9 -> ../../sda2
lrwxrwxrwx 1 root root 10 Dec 13 14:36 91a38cf5-b49c-417d-a3be-3f37bcbfd977 -> ../../sdd1
lrwxrwxrwx 1 root root 16 Dec 13 14:38 94077cfd-c567-4c42-9379-dd055595d709 -> ../../emcpowerw1
lrwxrwxrwx 1 root root 10 Dec 13 14:36 a9ce81c4-ed77-4d31-b65e-cee7ed433a40 -> ../../sdc1
lrwxrwxrwx 1 root root 16 Dec 13 14:38 b395e8e0-85c7-4e14-bd12-f54454012a34 -> ../../emcpoweru1
lrwxrwxrwx 1 root root 10 Dec 13 14:36 dd02ce0c-f1e3-4a75-9a5f-84e479ef6d52 -> ../../sda1
lrwxrwxrwx 1 root root 16 Dec 13 14:38 f7d62fb5-fb70-432a-b3cc-f601dc98565c -> ../../emcpowerv1

so how should my rules look like in udev? As listed below? but every rule is going to be with new UUID number.

KERNEL=="sd*", PROGRAM=="scsi_id --page=0x83 --whitelisted --device=/dev/%k", RESULT=="3600605b00162841221e285f92326df21", SYMLINK+="oracleasm/disks/DSK1", OWNER="oracle", GROUP="dba", MODE="0660"

Thanks again.

Bit confused about what you're getting here. Do you know what partition types are on those disks?

As for the rules, if you were using raw disks, yes, new UUID each link and then a different corresponding disk name (eg. DSK2, DSK3) in the 'SYMLINK' section. In your case you will also need to change the "--device" option as well to look at your emcpower* devices if they also contain ASM data.

I usually generate the rules and place them in a file eg.

/etc/udev/rules.d/99-asm.rules

You can then trigger a udev reload without rebooting by using:

 /sbin/udevadm trigger

Then check to see the symlinks are created in /dev/oracleasm/disks (or whichever directory you decide to use).

These will automatically be re-mapped on reboot once the rules are in rules.d.

Yeap, I'm confused too =) and sorry for confusing you too.

My disks list on the previous comment was wrong ones. But they have something in common they are partitioned too. These are the disks list. this is actually output of the command "blkid"
In this case how I should process with my rules? Should I use scsi_id command to get their UUID numbers and create rules from those disks? Or what is the best practice?

/dev/sde1: LABEL="DATA1" TYPE="oracleasm"
/dev/sdf1: LABEL="DATA2" TYPE="oracleasm"
/dev/sdg1: LABEL="DATA3" TYPE="oracleasm"
/dev/sdh1: LABEL="DATA4" TYPE="oracleasm"
/dev/sdi1: LABEL="DATA5" TYPE="oracleasm"
/dev/sdj1: LABEL="RECOV1" TYPE="oracleasm"
/dev/sdk1: LABEL="RECOV2" TYPE="oracleasm"
/dev/sdl1: LABEL="RECOV3" TYPE="oracleasm"
/dev/sdm1: LABEL="RECOV4" TYPE="oracleasm"

Appreciate your help.

Thanks

The fact that the system can identify that these are 'oracleasm' partitions to me suggests that ASMLib is configured on this host. Is it Oracle Unbreakable Linux (has ASMLib rolled into their standard kernel) or has someone added the module for your running kernel to the host?

I would suggest looking up the Red Hat documentation for ASM Lib, I will try and find a link and post it here.

The doco here may help if you confirm you are using ASMLib:
https://aacess.redhat.com/site/sites/default/files/attachments/oracle_on_rhel_with_or_without_asm_asmlib_0.pdf

I suspect that your ASM configuration (in Oracle) may be configured to use block devices (the /dev/sd* disk names) when it could/should be configured to use the ASM devices.

The UUID should still work if you use the disk device and ignore that the kernel can see the oracle ASM partition.

ie. add rules for sde -> sdm disk devices to create the /dev/oracleasm/disks symlinks and then configure ASM to point at these block devices instead of /dev/sd*. ASM in Oracle will then be able to identify the disk contents. The benefit of having ASMLib is that you can scan the disk headers and then map them through in the correct order.

I'd map them (by retrieving their UUID and generating UDEV rules) using something like the following (ie. use their disk label as their mapped device name):

/dev/oracleasm/disks/DATA1 -> /dev/sde
/dev/oracleasm/disks/DATA2 -> /dev/sdf
/dev/oracleasm/disks/DATA3 -> /dev/sdg
...

Which would be rules like (obviously UUID's will differ):

KERNEL=="sd*", PROGRAM=="scsi_id --page=0x83 --whitelisted --device=/dev/%k", RESULT=="3600605b00162841221e285f92326d1423", SYMLINK+="oracleasm/disks/DATA1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd*", PROGRAM=="scsi_id --page=0x83 --whitelisted --device=/dev/%k", RESULT=="3600605b00162841221e285f9232600114", SYMLINK+="oracleasm/disks/DATA2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sd*", PROGRAM=="scsi_id --page=0x83 --whitelisted --device=/dev/%k", RESULT=="3600605b00162841221e285f92326e0123", SYMLINK+="oracleasm/disks/DATA3", OWNER="oracle", GROUP="dba", MODE="0660"
...

The benefit of continuing down the path of UUID mapping is the host will no longer have a dependence on ASMLib configured in the kernel.

-edit-

This document from Red Hat may assist with mapping with partitions:
https://access.redhat.com/site/sites/default/files/attachments/oracle_11gr2_on_rhel6_0.pdf

At 5.1.2 they create a blank partition on each disk before adding to ASM.

Page 42 includes the UDEV rules which I have also included in this thread.

I believe there are 2 "camps" regarding best-practices. UDEV and ASMlib - using ASMlib, most everything discussed in this thread is primarily dynamic once it is setup correctly - which I believe would handle any renumbering of devices and their subsequent order. I thought it simply scanned the first portion of the drive to determine what the ASM setup is for the device.

I believe a full partition is a best-practice for ASMlib (as opposed to a full-disk option).

A few things to note
/etc/sysconfig/oracleasm -> should be a symlink to another file in the same directory (I can't recall the exact name it points to). Also - in that file you define the User/Group that the ASM devices should belong to, and the SCANORDER.
Lastly - and this has caught me a few times... you may have to run kpartx to add the dm-XX links.

If you run

/etc/init.d/oracleasm scandisks
/etc/init.d/oracleasm listdisks

Do you get an expected result?

I'll try to remember to provide links and more examples tomorrow.

My understanding was that ASMLib did this too (read the partition header and provided mapping for ASM), but haven't used this method.

Looking at a dev box with ASM I note that even without ASMLib it identifies an 'oracleasm' disk so this isn't the indicator of ASMLib being present as I originally expected.

The full disk partition for ASM is an Oracle recommendation which is all correct in this config.

If you do determine that you have ASMLib installed, James will be able to provide more specific advice for that. In saying that, I believe creating these UUID rules can resolve your immediate problem.

-edit- cleaned up post to stay on topic

Well,I checked the box and asmlib wasn't installed in this case so I'm assuming that from Pixel... explanation that system can identify asm disk type without the asmlib tool. So I will try to come up with action and task plan and will share with you guys to make sure that tasks were assigned right.

Thanks

I have quick question, Is these solutions are the same for single path and multipath scenarios? My new environment we are not using the multipathing.

Thanks

These rules will differ slightly if you are using multipathing. There is a Red Hat doc (linked below) that includes UDEV rules for dev mapper devices which would be a better fit if using multipath.

As it stands, the above rules are under the assumption that your multipathing is occurring at the hypervisor layer (eg. VMWare) and the guest is essentially seeing single pathed scsi disks.

I want to note that there are several issues/problems regarding UDEV configuration for ASM in the above Red Hat document.

There is also a more general Red Hat document in regards to configuring Oracle (and ASM) which I believe is close but still has minor issues with ASM UDEV configuration here:
https://access.redhat.com/site/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.0_2.pdf

When configuring ASM using UDEV without ASMLib I believe there are 3 basic things you want to achieve.

  1. Device naming persistence - The hardware device names remain the same even if hardware changes occur (eg. disk order)
  2. Device separation - The block device names are segregated into a different file structure (eg. /dev/oracleasm/disks) so this path can be used as the base path in ASM and incorrect devices cannot be accidentally included by the Oracle administrator
  3. Device permissions - Ensure that the devices that Oracle/ASM needs to access have the correct permissions to do so

It should also be noted that it appears the Oracle best practice has been revised over time (can find specific Oracle documents if required) so that all disks should have a single full disk partition on them before adding to ASM. In Oracle documentation (1528148.1) this partition is of type Linux and is created using MBR but can just as easily be created with GPT too. Unfortunately this Oracle documentation fails at both device naming persistence and device separation (which the Red Hat documentation is better at).

I think the best rules I have found in documentation so far (don't believe they were from Red Hat documentation) are as follows.

KERNEL=="sd?1", PROGRAM=="scsi_id -g -u -d /dev/%P", RESULT=="47000c2115a47e1deeb6a40867fdefb9", SYMLINK+="oracleasm/disks/disk1", OWNER="oracle", GROUP="dba", MODE="0660"

This UDEV rule accounts for the best practice of creating full disk partitions on the disks (the rules map partitions through), which I don't believe the original UDEV rules I have posted above from the Red Hat documentation account for (and this configuration matches your situation).

The rule reads as follows:

Find all first partitions on disks (sd?1), if the parent device ie. raw disk (%P) has the scsi_id of "47000c2115a47e1deeb6a40867fdefb9" create a symlink in /dev/oracleasm/disks/ to the partition and set permissions to oracle:dba mode 0660

Note: The rule will need adjusting when you go further than sdz.

Worket DC:
With the above rules you will still need to get the scsi_id of the disk using "scsi_id -g -u -d /dev/sda" for example, but the udev rules will map to the partition /dev/sda1 (what you want with your configuration).

So I made it kind of documentation type if this steps are true I will post it so others can benefit.

RDM remove process. RHEL 5.5 (Single Path)
Stage-1 VMware Level.
1. Start the vSphere Client, and log in to a vCenter Server.
2. Select Virtual Machines and Templates and click the Virtual Machines tab.
3. Right-click the Linux virtual machine for which you are enabling the disk UUID attribute, and select Power > Power Off.
The virtual machine powers off.
4. Right-click the virtual machine, and click Edit Settings.
5. Click the Options tab, and select the General entry in the settings column.
6. Click Configuration Parameters.
The Configuration Paramters window appears.
7. Click Add Row.
8. In the Name column, enter disk.EnableUUID.
9. In the Value column, enter TRUE.
10. Click OK and click Save.
11. Power on the virtual machine.

Stage-2

At this point we should able to grab the UUID number of ASM disks.

Run this command to get the UUID number of the disk.

# scsi_id -g -u -d /dev/sde or sde1
# 36000c29127c3ae0670242b058e863393 output should be something like this.

Next, we need to make udev rule file.

cd /etc/udev/rules.d/

touch 99-oracle-asmdevices.rules

vi 99-oracle-asmdevices.rules

KERNEL=="sde1", PROGRAM=="scsi_id -g -u -d /dev/%P", RESULT=="47000c2115a47e1deeb6a40867fdefb9", SYMLINK+="oracleasm/disks/disk1", OWNER="oracle", GROUP="oinstall", MODE="0660"

should this part be /dev/%k" instead of /dev/%P"???

And in my case OWNER=oracle GROUP=oinstall
This is according to this output.

brw-rw---- 1 oracle oinstall 8, 65 May 13 14:02 DATA1
brw-rw---- 1 oracle oinstall 8, 81 May 13 14:02 DATA2
brw-rw---- 1 oracle oinstall 8, 97 May 13 14:02 DATA3
brw-rw---- 1 oracle oinstall 8, 113 May 13 14:02 DATA4
brw-rw---- 1 oracle oinstall 8, 129 May 13 14:02 DATA5
brw-rw---- 1 oracle oinstall 8, 145 May 13 14:02 RECOV1
brw-rw---- 1 oracle oinstall 8, 161 May 13 14:02 RECOV2
brw-rw---- 1 oracle oinstall 8, 177 May 13 14:02 RECOV3
brw-rw---- 1 oracle oinstall 8, 193 May 13 14:02 RECOV4

After you have created rules to these disks.
Start the udev.

start_udev

Should I add this to rc.local file so after each reboot it will start the udev???
And is there any other configuration need to be setted up?

Thanks PixelDrift.NET in advance.

Stage 1, Step 3 says Windows 2008... hope you weren't talking about Windows all along!

The UDEV rule you have is close, but you need to leave the 'KERNEL==' parameter as '"sd?1" otherwise it will only check the sde1 device for the UUID which won't work if the devices move around / change order (eg. if someone adds a device later on).

So to clarify, the rule you posted should look exactly as follows:

KERNEL=="sd?1", PROGRAM=="scsi_id -g -u -d /dev/%P", RESULT=="47000c2115a47e1deeb6a40867fdefb9", SYMLINK+="oracleasm/disks/disk1", OWNER="oracle", GROUP="oinstall", MODE="0660"

%P is correct because the kernel filter "sd?1" is looking at partitions, and the scsi_id works on the raw disk device, so %P references the parent device. In your case, /dev/sde1 will match the filter, /dev/sde is the parent device we are running the scsi_id command on to return the UUID to match the 'RESULT'. I also notice in your example you still have 'disk1' but in your case you likely want this to be DATA1, DATA2 etc. (but that comes down to whatever you want to call the devices).

There is no need to add the rules to rc.local, they should be run automatically on startup by udev (confirm the permissions of the rule file matches other rules in the directory).

To confirm the rules are configured correctly I run

/sbin/udevadm trigger

Then

ls -l /dev/oracleasm/disks

You should then see all the symlinks pointing to your disk partitions.

Sorry I have edited the post, its linux boxes.

So Stage 2 needs to look like this.

Stage-2

At this point we should able to grab the UUID number of ASM disks.

Run this command to get the UUID number of the disk.

scsi_id -g -u -d /dev/sde or sde1

36000c29127c3ae0670242b058e863393 output should be something like this.

Next, we need to make udev rule file.

cd /etc/udev/rules.d/
touch 99-oracle-asmdevices.rules
vi 99-oracle-asmdevices.rules
KERNEL=="sd?1", PROGRAM=="scsi_id -g -u -d /dev/%P", RESULT=="47000c2115a47e1deeb6a40867fdefb9", SYMLINK+="oracleasm/disks/DATA1", OWNER="oracle", GROUP="oinstall", MODE="0660"

should this part be /dev/%k" instead of /dev/%P"???

And in my case OWNER=oracle GROUP=oinstall
This is according to this output.

brw-rw---- 1 oracle oinstall 8, 65 May 13 14:02 DATA1
brw-rw---- 1 oracle oinstall 8, 81 May 13 14:02 DATA2
brw-rw---- 1 oracle oinstall 8, 97 May 13 14:02 DATA3
brw-rw---- 1 oracle oinstall 8, 113 May 13 14:02 DATA4
brw-rw---- 1 oracle oinstall 8, 129 May 13 14:02 DATA5
brw-rw---- 1 oracle oinstall 8, 145 May 13 14:02 RECOV1
brw-rw---- 1 oracle oinstall 8, 161 May 13 14:02 RECOV2
brw-rw---- 1 oracle oinstall 8, 177 May 13 14:02 RECOV3
brw-rw---- 1 oracle oinstall 8, 193 May 13 14:02 RECOV4

After you have created rules to these disks.
Start the udev.

start_udev

To confirm the rules are configured correctly I run

/sbin/udevadm trigger

Then

ls -l /dev/oracleasm/disks
You should then see all the symlinks pointing to your disk partitions.

Thanks PixelDrift.NET in advance.

In previous responses you have posted a bit different rules like this:

KERNEL=="sd*", PROGRAM=="scsi_id --page=0x83 --whitelisted --device=/dev/%k", RESULT=="3600605b00162841221e285f92326d1423", SYMLINK+="oracleasm/disks/DATA1", OWNER="oracle", GROUP="dba", MODE="0660"

Do I have to do whitelisting of any divices? or that is just for whole disk not partitioned ones?

Thanks

This UDEV rule is from the original Red Hat document I posted but from my testing will only map the disk device, not the partition, these rules aren't required at all if you are using the newer rules I posted.

The filter in this case "sd*" will return for eg. sde and sde1 (disk and the partition) but the scsi_id command will only return a matching UUID on the 'sde' device from my experience (ie. the scsi_id command listed doesn't return a UUID if pointed to the partition) so the disk device not the partition will be linked to.

The revised rule I have posted (which you have in your documentation) specifically filters for single partition disks and maps to the partition by validating parent disk device UUID which I believe is far more reliable (and creates the link to the partition in the same way the Oracle document I have referenced does).

I am interested to know if anyone has a configuration that returns a UUID from the partition (ie. of the disk it is on) if the scsi_id command is run against it, I believe this returned nothing on RHEL 6 when I tested (in a VMWare guest).

Hi
when I'm running scsi_id -g -u -s /dev/sdb command output is null, nothing in return but when i'm running it scsi_id -g -u -s /block/sdb option its returning UUID number. Is dev and block options are the same or something is wrong? I have enabled the VMware UUID option.

Thanks

The command should be

scsi_id -g -u -d /dev/sdb

As in above posts. You have -s instead of -d switch.

Can you confirm that works?

Nope it did not work with -d option.

scsi_id -g -u -d /dev/sdb

So out of interest does the following return a UUID for you?:

scsi_id --page=0x83 --whitelisted --device=/dev/sdb

The 'scsi_id -g -u -d' command worked in a RHEL 6 VMWare environment I tested in.

This does not return anything too, should we change rule from dev to block? What do you think. This box is RHEL 5.

Interesting reading about the inconsistencies in scsi_id. It goes a long to explaining why there are so many different sets of rules out there in different vendor documentation.

KERNEL=="sd?1", PROGRAM=="scsi_id -g -u -s /block/%P", RESULT=="47000c2115a47e1deeb6a40867fdefb9", SYMLINK+="oracleasm/disks/disk1", OWNER="oracle", GROUP="oinstall", MODE="0660"

This rule (with block instead of dev) should work in your situation. I will test it again in my RHEL 6 environment when I get a chance as it would be nice to have a rule the worked in all situations / configurations.

I found this documentation in Redhat and this shows to get UUID from scsi_id -g -u -d /block/sdb. Is this valid configuration.

Single path configuration
This procedure implements LUN device persistence using udev. Only use this procedure for hosts which are not using multipath.
Edit the /etc/scsi_id.config file.
Ensure the options=-b is line commented out.

options=-b

Add the following line:
options=-g
This option configures udev to assume all attached SCSI devices return a UUID.
To display the UUID for a given device run the scsi_id -g -s /block/sd* command. For example:

scsi_id -g -s /block/sd*

3600a0b800013275100000015427b625e
The output may vary from the example above. The output displays the UUID of the device /dev/sdc.
Verify the UUID output by the scsi_id -g -s /block/sd* command is identical from computer which accesses the device.
Create a rule to name the device. Create a file named 20-names.rules in the /etc/udev/rules.d directory. Add new rules to this file. All rules are added to the same file using the same format. Rules follow this format:
KERNEL=="sd[a-z]", BUS=="scsi", PROGRAM="/sbin/scsi_id -g -s /block/%k", RESULT="UUID", NAME="devicename"
Replace UUID and devicename with the UUID retrieved above, and a name for the device. This is a rule for the example above:
KERNEL="sd", BUS="scsi", PROGRAM="/sbin/scsi_id -g -s", RESULT="3600a0b800013275100000015427b625e", NAME="rack4row16"
The udev daemon now searches all devices named /dev/sd
for the UUID in the rule. Once a matching device is connected to the system the device is assigned the name from the rule. In the a device with a UUID of 3600a0b800013275100000015427b625e would appear as /dev/rack4row16.
Append this line to /etc/rc.local:
/sbin/start_udev
Copy the changes in the /etc/scsi_id.config, /etc/udev/rules.d/20-names.rules, and /etc/rc.local files to all relevant hosts.
/sbin/start_udev
Networked storage devices with configured rules now have persistent names on all hosts where the files were updated This means you can migrate guests between hosts using the shared storage and the guests can access the storage devices in their configuration files.

See my above post. The rule I posted (using block) as you have confirmed working for you should work in your environment.

Can you also provide the version number of your scsi_id command? for future reference.

scsi_id -g -u -d /block/sdb this command doesn't return anything.

scsi_id -g -u -s /block/sdb this returns UUID.

scsi_id -g -u -s /block/sdb
36000c2959beef52d1df6d6b7672b5e27

So this UDEV rule (change the symlink value) should work for you and map to /dev/sdb1:

KERNEL=="sd?1", PROGRAM=="scsi_id -g -u -s /block/%P", RESULT=="36000c2959beef52d1df6d6b7672b5e27", SYMLINK+="oracleasm/disks/disk1", OWNER="oracle", GROUP="oinstall", MODE="0660"

Just wanted to share the final Doc of Removing RDMs process steps. Please let me know if I have missed anything?

Thanks

Stage1
So I made it kind of documentation type if this steps are true I will post it so others can benefit.
RDM remove process. RHEL 5.5 (Single Path)
Stage-1 VMware Level.
1. Start the vSphere Client, and log in to a vCenter Server.
2. Select Virtual Machines and Templates and click the Virtual Machines tab.
3. Right-click the Linux virtual machine for which you are enabling the disk UUID attribute, and select Power > Power Off.
The virtual machine powers off.
4. Right-click the virtual machine, and click Edit Settings.
5. Click the Options tab, and select the General entry in the settings column.
6. Click Configuration Parameters.
The Configuration Paramters window appears.
7. Click Add Row.
8. In the Name column, enter disk.EnableUUID.
9. In the Value column, enter TRUE.
10. Click OK and click Save.
11. Power on the virtual machine.

Stage-2
At this point we should able to grab the UUID number of ASM disks.
Run this command to get the UUID number of the disk.

scsi_id -g -u -d /block/sde

36000c29127c3ae0670242b058e863393 output should be something like this.

Next, we need to make udev rule file.

cd /etc/udev/rules.d/

touch 99-oracle-asmdevices.rules

vi 99-oracle-asmdevices.rules

KERNEL=="sd?1", PROGRAM=="scsi_id -g -u -s /dev/%P", RESULT=="47000c2115a47e1deeb6a40867fdefb9", SYMLINK+="oracleasm/disks/DATA1", OWNER="oracle", GROUP="oinstall", MODE="0660"

And in my case OWNER=oracle GROUP=oinstall

This is according to this output.
brw-rw---- 1 oracle oinstall 8, 65 May 13 14:02 DATA1
brw-rw---- 1 oracle oinstall 8, 81 May 13 14:02 DATA2
brw-rw---- 1 oracle oinstall 8, 97 May 13 14:02 DATA3
brw-rw---- 1 oracle oinstall 8, 113 May 13 14:02 DATA4
brw-rw---- 1 oracle oinstall 8, 129 May 13 14:02 DATA5
brw-rw---- 1 oracle oinstall 8, 145 May 13 14:02 RECOV1
brw-rw---- 1 oracle oinstall 8, 161 May 13 14:02 RECOV2
brw-rw---- 1 oracle oinstall 8, 177 May 13 14:02 RECOV3
brw-rw---- 1 oracle oinstall 8, 193 May 13 14:02 RECOV4

After you have created rules to these disks.
Start the udev.

start_udev

To confirm the rules run this command

/sbin/udevadm trigger

Then

ls -l /dev/oracleasm/disks

You should then see all the symlinks pointing to your disk partitions.

Hi again,

What would be the back up plan in case if the udev rules do not work and the devise mapping will just messed up?

Appritiate for any comments.

Thanks

If the UDEV rules don't work, the disks won't be visible to ASM, so the disk group won't mount correctly. Similarly, if the disks are out of order ASM will fail to mount the ASM disk group.

Hello everyone,

I have implemented the rules we have covered here, and want to say big thanks to PixelDrift.Net Support and other members. I was able to make asm disks persistent after removing the RDM disk from VMware.

Thanks

Great news, glad it all went smoothly!

Thanks for the feedback.

Hi, Just to add, reading the RedHat KB https://access.redhat.com/solutions/76673 is worth as well.

Thanks.

Hi, I was using the following UDEV rule on VMware RHEL6 server for ASM disks with a single partition.

ACTION=="add|change", KERNEL=="sd[!0-9]", ENV{ID_SERIAL}=="", IMPORT{program}="/lib/udev/scsi_id --whitelisted --device=/dev/%k" ACTION=="add|change", KERNEL=="sd[0-9]", ENV{ID_SERIAL}=="", IMPORT{parent}=="ID_" ACTION=="add|change", KERNEL=="sd1", ENV{ID_SERIAL}=="xxxxxxxxxxxxxx", SYMLINK+="asmdisk01", GROUP="asmadmin", OWNER="grid", MODE="0660"

Replace xxxxxx with the UUID of the parent disk, name, group and owner accordingly to suit your environment.

Thanks

@Nujoom Ereelot:

Since you did not use a code block for your rules, the forum's Markdown parser deleted some essential asterisks from your rules and displays them all on one line. With the missing asterisks fixed, your udev rule actually looks like a combination of three rules:

ACTION=="add|change", KERNEL=="sd*[!0-9]", ENV{ID_SERIAL}=="", IMPORT{program}="/lib/udev/scsi_id --whitelisted --device=/dev/%k" 
ACTION=="add|change", KERNEL=="sd*[0-9]", ENV{ID_SERIAL}=="", IMPORT{parent}=="ID_*" 
ACTION=="add|change", KERNEL=="sd*1", ENV{ID_SERIAL}=="xxxxxxxxxxxxxx", SYMLINK+="asmdisk01", GROUP="asmadmin", OWNER="grid", MODE="0660"

The first rule says "when adding or changing devices, for all devices matching sd*[!0-9], run the scsi_id command and capture its output as ENV{something} keys for subsequent rules". This rule has a problem: the IMPORT{program} rule needs the program to output its results in an environment key format, i.e. like this:

ID_SERIAL=xxxxxxxxxxxxxx

But without the "--export" option, the scsi_id command will output just the "xxxxxxxxxxxxxx" part, so the information won't be available for the subsequent rules. So the first rule fails to do anything useful as-is. Fortunately, in RHEL 6.x, /lib/udev/rules.d/60-persistent-storage.rules already contains a functional version of this rule, so the first line could be removed as long as your custom rules are located after the standard 60-persistent-storage.rules file in the udev rule execution order.

The second rule says: "on adding or changing devices, for all devices whose default name matches sd*[0-9] and don't have ENV{ID_SERIAL}, import all ID_* environment parameters from the parent device." This looks correct, and allows partition devices to "inherit" the UUID/serial number of their parent whole-disk device. That's a good idea: it allows you to write rules for any number of ASM partition devices in a very simple way.

The third rule will match only if the device's default name is equal to sd*1 and the device has a specific serial number. So it will match the first partition on the disk with that UUID/serial number. The rule will assign the right owner, group and protection mode for ASM use, and add a symlink /dev/asmdisk01 for it. The symlink name should then match the ASM_DISKSTRING parameter set within the ASM configuration.

Reading and re-reading this thread, trying to soak in all of the information here.

It seems that setting these udev rules applies to RHEL6. I haven't had much chance to work with udev under RHEL5 and this is where we have our Oracle 11g installed on. Can we apply these udev rules for OracleASM on RHEL5 with no issues?

The udev rules in RHEL 5.x /etc/udev/rules.d/50-udev.rules seem to already be doing the work of Nujoom Ereelot's first two rules. So, as long as your custom rules file is named so that it sorts after 50-udev.rules, then you'll only need the third rule:

ACTION=="add|change", KERNEL=="sd*1", ENV{ID_SERIAL}=="xxxxxxxxxxxxxx", SYMLINK+="asmdisk01", GROUP="asmadmin", OWNER="grid", MODE="0660"

If you run a heavily customized udev ruleset or otherwise need to specify all three rules explicitly instead of relying on the rules in the default ruleset, Nujoom's first rule would need to be adapted for RHEL 5.x, as its version of scsi_id does not yet have long-form options. So the two first rules would be:

KERNEL=="sd*[!0-9]", ENV{ID_SERIAL}=="", IMPORT{program}="/lib/udev/scsi_id -g -x -s %p -d $tempnode"
KERNEL=="sd*[0-9]", ENV{ID_SERIAL}=="", IMPORT{parent}=="ID_*"

Note: on RHEL 5.x, this works on non-multipathed disks. But if dm-multipath is involved, then RHEL 5.x throws a curveball by using the keyword OPTIONS="last_rule" in the default /etc/udev/rules.d/40-multipath.rules, so for multipath devices, the rule processing ends at 40-multipath.rules. This makes it inconvenient to append your own custom rules for multipathed devices.

I ended up solving this by copying the standard 40-multipath.rules into 39-multipath-asm.rules and then adding the necessary rules for adding the ASM symlinks and device permissions to that copy. Because all multipath devices will encounter the OPTIONS="last_rule" line in my customized 39-multipath-asm.rules file, the standard 40-multipath.rules will not get executed at all for multipath devices. Removing the standard file would have been futile, as the next dm-multipath update would have restored it, and in RHEL 5.x, udev did not yet have the "/etc/udev/rules.d/ files override any standard rule files with the same name in /lib/udev/rules.d/" semantics.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.