udev and multipath setup for Oracle ASM 11.2.0.3 on RHEL 6.x

Recently I was tasked to install Oracle RAC 11.2.0.3 on RHEL 6 (kernel 2.6.32-xx) using HP P4000 as our iSCSI shared storage solution. One would assume that Oracle ASMLib would be available to configure Oracle ASM devices but to my surprise realized that Oracle has now decided to yank support for it in RHEL 6.x releases, BUT offer ASMLib for OEL!

Therein begun my journey into unix system admin udev and multipath utilities to manual configure ASM devices. It took a lot of digging to figure it out and I thought that documenting the steps might make it easier for somebody out there facing the same issue.

HP P4000 has a unique solution that allows us to create volumes across multiple storage servers using Network RAID. I have yet to test out how well it performs in our case since we are sharing the same network switch to interconnect 2 storage servers and the connection to 2 database servers. More on that later…

We created 2 volumes:

1. CRS (OCR and Voting Disk)

2. DATA (datafiles, redologfiles, etc).

If using FRA, then I presume you would want to create a 3rd volume, but that was not needed in my case. Since each DB Server serves as ISCSI “initiator” that needs to connect to shared storage volume, defined as “target”, we found the ISCSI Initiatorname as follows:

# more /etc/iscsi/initiatorname.iscsi
 InitiatorName=iqn.1994-05.com.redhat:xxxxxxxxxxxx

and input the above when adding Servers in HP P4000 CMC Software.

ISCSI Discovery / Setup

 From each DB Server, we need to “discover” the “targets”. Targets are the 2 volumes created on the shared storage device HP P4000 using Network RAID 10.

  1. Login as root 
  2. Define two paths to reach the target (for redundancy).

# iscsiadm -m iface -I oraeth0 –op=new

New interface oraeth0 added

 

# iscsiadm -m iface -I oraeth2 –op=new

New interface oraeth2 added

 

# iscsiadm -m iface -I oraeth0 –op=update -n iface.net_ifacename -v eth0

oraeth0 updated.

 

# iscsiadm -m iface -I oraeth2 –op=update -n iface.net_ifacename -v eth2

oraeth2 updated.

 

3. Discover the targets using both interfaces defined above on HP P4000 VIP (10.116.49.88):

 

# iscsiadm -m discovery -t st -p 10.116.49.88 -I oraeth0 -I oraeth2

10.116.49.88:3260,1 iqn.2003-10.com.lefthandnetworks:rac-mg:285:data

10.116.49.88:3260,1 iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs

10.116.49.88:3260,1 iqn.2003-10.com.lefthandnetworks:rac-mg:285:data

10.116.49.88:3260,1 iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs

 

4. Enable automatic login to targets on reboot (copy iqn…. name from above):

 

# iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:rac-mg:285:data -p 10.116.49.88 –op update -n node.startup -v automatic

# iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs -p 10.116.49.88 –op update -n node.startup -v automatic

 

5. Manual Login to targets

#iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs -p 10.116.49.88 -l

#iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:rac-mg:285:data -p 10.116.49.88 -l

Multipath

 

 Set Alias and multipath policy in /etc/multipath.conf file

 

  1. Determine the /dev/sd* device name that is mapped to target.

# ls -l /dev/disk/by-path | grep ip

total 0

lrwxrwxrwx 1 root root  9 Jul 19 20:43 ip-10.116.49.88:3260-iscsi-iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs-lun-0 -> ../../sde

lrwxrwxrwx 1 root root  9 Jul 19 20:43 ip-10.116.49.88:3260-iscsi-iqn.2003-10.com.lefthandnetworks:rac-mg:285:data-lun-0 -> ../../sdh

  1. The above displays sdh and sde as devices that are mapped to CRS and DATA volumes exposed as LUN 0.
  1. Create partition on sdh device

# fdisk /dev/sdh

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel with disk identifier 0x76993a12.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won’t be recoverable.

 

Warning: invalid flag 0×0000 of partition table 4 will be corrected by w(rite)

 

WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to

         switch off the mode (command ‘c’) and change display units to

         sectors (command ‘u’).

 

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-52216, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-52216, default 52216):

Using default value 52216

 

Command (m for help): w

The partition table has been altered!

 

Calling ioctl() to re-read partition table.

Syncing disks.

  1. Create partition on sde device

fdisk /dev/sde

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel with disk identifier 0xdf7be671.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won’t be recoverable.

 

Warning: invalid flag 0×0000 of partition table 4 will be corrected by w(rite)

 

WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to

         switch off the mode (command ‘c’) and change display units to

         sectors (command ‘u’).

 

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

1

Invalid partition number for type `1′

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-1014, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-1014, default 1014):

Using default value 1014

 

Command (m for help): w

The partition table has been altered!

 

Calling ioctl() to re-read partition table.

Syncing disks.

  1. Determine the target and uuid of sdh device:

# ls -l /dev/disk/by-path |grep sdh

lrwxrwxrwx 1 root root  9 Jul 19 20:43 ip-10.116.49.88:3260-iscsi-iqn.2003-10.com.lefthandnetworks:rac-mg:285:data-lun-0 -> ../../sdh

 

scsi_id –whitelisted /dev/sdh

36000eb3262f406b2000000000000011d

 

Note target data uuid is 36000eb3262f406b2000000000000011d

  1. Determine the target and uuid of sde device:

# ls -l /dev/disk/by-path |grep sde

lrwxrwxrwx 1 root root  9 Jul 19 20:43 ip-10.116.49.88:3260-iscsi-iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs-lun-0 -> ../../sde

 

scsi_id –whitelisted /dev/sde

36000eb3262f406b2000000000000011b

 

 

Note target crs uuid is 36000eb3262f406b2000000000000011b

  1. Enter the above values into /etc/multipath and give friendly aliases to uuid.

vi /etc/multipathd.conf

 

Uncomment and modify as follows:

 

blacklist {

        devnode “^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*”

        devnode “^hd[a-z]“

}

multipaths {

        multipath {

                wwid                    36000eb3262f406b2000000000000011b

                alias                   crs

                path_grouping_policy    multibus

                path_selector           “round-robin 0″

                failback                immediate

                rr_weight               priorities

                no_path_retry           5

        }

        multipath {

                wwid                    36000eb3262f406b2000000000000011d

                alias                   data

                path_grouping_policy    multibus

                path_selector           “round-robin 0″

                failback                immediate

                rr_weight               priorities

                no_path_retry           5

        }

}

  1. Restart multipathd service.

service multipathd restart

  1. Check if multipath is setup properly:

# multipath -ll

 

crs (36000eb3262f406b2000000000000011b) dm-2 LEFTHAND,iSCSIDisk

size=3.0G features=’1 queue_if_no_path’ hwhandler=’0′ wp=rw

`-+- policy=’round-robin 0′ prio=1 status=active

  |- 66:0:0:0 sdc 8:32  active ready running

  |- 65:0:0:0 sdb 8:16  active ready running

  |- 67:0:0:0 sdd 8:48  active ready running

  `- 68:0:0:0 sde 8:64  active ready running

data (36000eb3262f406b2000000000000011d) dm-3 LEFTHAND,iSCSIDisk

size=400G features=’1 queue_if_no_path’ hwhandler=’0′ wp=rw

`-+- policy=’round-robin 0′ prio=1 status=active

  |- 70:0:0:0 sdf 8:80  active ready running

  |- 71:0:0:0 sdg 8:96  active ready running

  |- 72:0:0:0 sdh 8:112 active ready running

  `- 69:0:0:0 sdi 8:128 active ready running

  1. Determine partitioned alias for target device

# dmsetup ls |grep p1

datap1  (253, 5)

crsp1   (253, 4)

 

  1. Edit /etc/udev/rules.d/12-dm-permissions.rules file to set the above device ownership as oracle:dba

# Set permissions for first two partitions created on a multipath device (and detected by kpartx)

ENV{DM_NAME}==”crsp1″, OWNER:=”oracle”, GROUP:=”dba”, MODE:=”660″, SYMLINK+=”iscsi/oraasm-$env{DM_NAME}”

ENV{DM_NAME}==”datap1″, SYMLINK+=”iscsi/oraasm-$env{DM_NAME}”, OWNER:=”oracle”, GROUP:=”dba”, MODE:=”660″

 

  1. Restart multipathd service.

service multipathd restart

 

  1. Check if permissions are correct

# ls -l /dev/dm*

brw-rw—- 1 root   disk 253, 0 Jul 16 19:40 /dev/dm-0

brw-rw—- 1 root   disk 253, 1 Jul 16 19:40 /dev/dm-1

brw-rw—- 1 root   disk 253, 2 Jul 19 21:47 /dev/dm-2

brw-rw—- 1 root   disk 253, 3 Jul 19 21:47 /dev/dm-3

brw-rw—- 1 oracle dba  253, 4 Jul 19 21:47 /dev/dm-4

brw-rw—- 1 oracle dba  253, 5 Jul 19 21:47 /dev/dm-5

 

Note: Multipath /dev/dm-4 and /dev/dm-5 devices are owned by oracle:dba, which point to CRS and DATA partitioned volumes.

  1. Check if oracle ASM symbolic links exists for multipath devices.

# ls -l /dev/iscsi/*

lrwxrwxrwx 1 root root 7 Jul 19 21:47 /dev/iscsi/oraasm-crsp1 -> ../dm-4

lrwxrwxrwx 1 root root 7 Jul 19 21:47 /dev/iscsi/oraasm-datap1 -> ../dm-5

About these ads
This entry was posted in RAC, udev and tagged , , , , , , , , , . Bookmark the permalink.

2 Responses to udev and multipath setup for Oracle ASM 11.2.0.3 on RHEL 6.x

  1. jcnars says:

    You are the best !
    I have been going crazy crazy crazy.

    I tried every udev rule … all of them used scsi_id…which is basically useless if you want to present the partition to the ASM. Because scsi_id returns null for partitions and only returns values for indiviudal disks.

    Thanks a million

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s