This isn't very well documented anywhere so here is the rest of the story when it comes to sharing raw disk between two or more virtual machines that will be in a cluster. The cluster you choose doesn't really matter if it is Microsoft Cluster (no thanks) or Symantec (Veritas) Cluster for Linux systems. The focus here is on the correct mapping of Raw Disk in VMware.
One of the better documents (though incomplete) can be found here. Direct from the source. Here is the idea visually represented. This is well documented on many sites up to this point - creating the Raw Disk Mapping in VMware.
Here are the steps required to create Raw Disk Mapping from the beginning.
1. In vSphere Client click on the VM which will be the active or Node A in the cluster.
2. Select Edit Settings (right click on the VM or Edit Settings from the Getting Started or Summary tab).
3. Select Hard Disk then Next to continue in the Add Hardware Wizard.
4. Click the Raw Device Mappings radio button.
5. Next you will need to coordinate with your SAN team to ensure raw disk shows in the size you requested. If you don't seen any raw disk here then you may need to rescan your storage configuration at the ESX/ESXi host. For ESX 4.x you may need to rescan twice to discover the raw disk. If you still don't see the disk and you have also refreshed the storage configuration at the host then you may need to check with the SAN team to ensure the correct WWPNs from the host are logged in (FLOGI).
6. It is recommended to store raw disk (and all virtual disk assigned) with the virtual machine. Why? To avoid conflicts and loss of availability. VMware has configurations (HA, DRS, EVC) that make storage and virtual machines more fluid among other hosts and resources. Take advantage of those features to maintain high availability for your clustered machines.
7. When sharing disk between clustered virtual machines ensure physical compatibility mode is selected. This is the recommended (and required) mode from VMware according to their documentation.
8. This next part is very important. All shared raw disk must be on a different SCSI controller than the boot disk. Node B (see below) must also share the exact same SCSI ID to share the virtual disk properly.
9. The final step is the summary screen. Click Finish to complete.
When you click complete a new SCSI controller will be presented. You MUST select that device (also in bold) and ensure Physical compatibility modes is checked in the SCSI Bus Sharing configuration.
Security Fun Fact. Avoid using a whole disk as shared disk. Sharing an entire disk can expose the data on the disk to other hosts that may also share the disk. This is a side effect of virtualization. Even though vendors have taken steps to ensure resource sharing is unique to each instance for memory and CPUs disk can be exposed to data loss when presented as a whole disk and then shared to multiple virtual machines.
Another Security Fun Fact. You must power off a VM before you can edit or vMotion a clustered VM to another host/datastore. This affects availability - unless you first fail over cluster services from the active node to the passive node and promote the passive node to active. It is possible to avoid any downtime when vMotioning a clustered VM.
Now the rest of the story...
What's not very well documented anywhere is the mapping part from the point of view of Server 2 or Node B in a two node cluster.
There are a few steps to mapping the raw disk to Node B.
1. First verify the "Mapped Raw LUN" from Node A. You'll need it to when mapping Node B. It looks something like this. You'll need to power OFF the VMs to perform these steps. Node A or the active node in a cluster is where you will Map the Raw LUN. Node B is where you will add the shared Raw Disk as a connection to an existing disk (not as another Raw Disk Mapping). There are three choices you get when adding a new virtual Hard Drive, create new, choose from existing and raw disk. Node B will always use the "choose from existing" virtual disk.
It's important to note the full path of the [datastore] vm-node-a/vm-node-a-disk_2.vmdk which I have blackened out. Write that path down. Also note that Raw Disk always get their very own SCSI Controller - in this case SCSI 1:0. Compatibility Mode is actually very important. We really have to have this mode set to Physical to take advantage of the sharing capabilities and it's a requirement for the cluster also.
Part of the process which you will need to know is the SCSI Controller must also be in Physical SCSI Bus Sharing mode to match the disk. It's a requirement for this RDM configuration to work properly with the clustering software you will configure at the OS.
2. Set up Node B - or Server 2 by adding a Mapped Raw LUN which will point directly to the full path of Node A's Raw Disk. It is absolutely critical that your full path to the shared disk matches exactly what Node A shows. You must also ensure your "Virtual Device Node" is set to the same SCSI controller and ID (SCSI (1:0) as shown in the example below).
Remember to set SCSI Bus Sharing to Physical too.
Save the settings and power on the VMs.
Why create a file system on a Raw Disk? Answer: Because that's the only way to share the disk between two or more virtual machines in VMware. Every server vendor has this capability to share the same disk across multiple servers. Solaris does it differently than IBM. VMware does it their own way too.
In a Linux Cluster you may experience SCSI Reservation Conflicts when sharing a Raw Disk between two or more nodes in a cluster. In VMware you can sometimes clear those reservations on the host as described in this KB article. One work around may be to ensure all nodes in the cluster reside on the same ESX host. This may clear the disk errors but it creates a single point of failure. See my article on setting up Affinity Rules with DRS (You Can't Triple Stamp A Double Stamp!) to ensure nodes remain static on separate ESX hosts. Make sure your ESX/ESXi hosts are fully patched, firmware is up to date for your HBAs and your clustering software is also fully patched to avoid SCSI Reservation Conflicts.
The point of this posting is to complete the process of how raw disk are mapped between two or more virtual machines. I didn't find the complete story when researching this back in 2012. So I decided to document it. This concludes Mapped Raw LUNs in VMware. At the storage array the disk must be marked as raw disk. At the operating system (like Oracle Enterprise Linux) the raw disk can remain raw and ownership give to say Oracle for ASM. Or you can create a file system with the raw disk (I know that sounds like it completely defeats the purpose of raw disk in the first place but it works for clustering software like SteelEye LifeKeeper Linux Clusters).
No comments:
Post a Comment