Tag Archive for NMP

Perform command line configuration of multipathing options

January 18, 2013 Objective 1 Storage, Objective 6 Advanced Troubleshooting No comments

signpost

Multipathing Considerations

Specific considerations apply when you manage storage multipathing plug-ins and claim rules. The following considerations help you with multipathing

  • If no SATP is assigned to the device by the claim rules, the default SATP for iSCSI or FC devices is VMW_SATP_DEFAULT_AA. The default PSP is VMW_PSP_FIXED.
  • When the system searches the SATP rules to locate a SATP for a given device, it searches the driver rules first. If there is no match, the vendor/model rules are searched, and finally the transport rules are searched. If no match occurs, NMP selects a default SATP for the device.
  • If VMW_SATP_ALUA is assigned to a specific storage device, but the device is not ALUA-aware, no claim rule match occurs for this device. The device is claimed by the default SATP based on the device’s transport type.
  • The default PSP for all devices claimed by VMW_SATP_ALUA is VMW_PSP_MRU. The VMW_PSP_MRU selects an active/optimized path as reported by the VMW_SATP_ALUA, or an active/unoptimized path if there is no active/optimized path. This path is used until a better path is available (MRU). For example, if the VMW_PSP_MRU is currently using an active/unoptimized path and an active/optimized path becomes available, the VMW_PSP_MRU will switch the current path to the active/optimized one.
  • If you enable VMW_PSP_FIXED with VMW_SATP_ALUA, the host initially makes an arbitrary selection of the preferred path, regardless of whether the ALUA state is reported as optimized or unoptimized. As a result, VMware does not recommend to enable VMW_PSP_FIXED when VMW_SATP_ALUA is used for an ALUA-compliant storage array. The exception is when you assign the preferred path to be to one of the redundant storage processor (SP) nodes within an active-active storage array. The ALUA state is irrelevant.
  • By default, the PSA claim rule 101 masks Dell array pseudo devices. Do not delete this rule, unless you want to unmask these devices.

What can we use to configure Multipath Options

  • vCLI
  • vMA
  • Putty into DCUI console

What we can view and adjust

  • You can display all multipathing plugins available on your host
  • You can list any 3rd Party MPPs as well as your hosts PSP and SATPs and review the paths they claim
  • You can also define new paths and specify which multipathing plugin should claim the path

The ESXCLI Commands

Click the link to take you to the vSphere 5 Documentation Center for each command

These are the 2 commands you need to use to perform configuration of multipathing

nmp

nmp2

esxcli storage nmp psp Namespaces

generic1

Display NMP PSPs

  • esxcli storage nmp psp list

This command list all the PSPs controlled by the VMware NMP

psplist

More complicated commands with esxcli storage nmp psp namespace

  • esxcli storage nmp psp fixed deviceconfig set - -device naa.xxx –path vmhba3:C0:T5:L3

The command sets the preferred path to vmhba3:C0:T5:L3. Run the command with – -default to clear the preferred path selection

esxcli storage nmp satp Namespaces

generic2

Display SATPs for the Host

  • esxcli storage nmp satp list

For each SATP, the output displays information that shows the type of storage array or system this SATP supports and the default PSP for any LUNs using this SATP. Placeholder (plugin not loaded) in the Description column indicates that the SATP is not loaded.

satplist

More complicated commands with esxcli storage nmp satp namespaces

  • esxcli storage nmp satp rule add -V NewVend -M NewMod -s VMW_SATP_INV

The command assigns the VMW_SATP_INV plug-in to manage storage arrays with vendor string NewVend and model string NewMod.

esxcli storage nmp device NameSpaces

generic3

Display NMP Storage Devices

  • esxcli storage nmp device list

This command list all storage devices controlled by the VMware NMP and displays SATP and PSP information associated with each device

devicelist

More complicated commands with esxcli storage nmp device namespaces

  • esxcli storage nmp device set - -device naa.xxx - -psp VMW_PSP_FIXED

This command sets the path policy for the specified device to  VMW_PSP_FIXED

esxcli storage nmp path Namespaces

generic4

Display NMP Paths

  • esxcli storage nmp path list

This command list all the paths controlled by the VMware NMP and displays SATP and PSP information associated with each device

pathlist

More complicated commands with esxcli storage nmp path namespaces

There is only really the list command associated with this command

esxcli storage core Command Namespaces

storagecore

esxcli storage core adapter Command Namespaces

storagecore2

esxcli storage core device Command Namespaces

core3

esxcli storage core path Command Namespaces

core4

esxcli storage core plugin Command Namespaces

core5

esxcli storage core claiming Command Namespaces

core6

The esxcli storage core claiming namespace includes a number of troubleshooting commands. These  commands are not persistent and are useful only to developers who are writing PSA plugins or troubleshooting a system. If I/O is active on the path, unclaim  and reclaim actions fail

The help for esxcli storage core claiming includes the autoclaim command. Do not use this command unless instructed to do so by VMware support staff

esxcli storage core claimrule Command Namespaces

core7

The PSA uses claim rules to determine which multipathing module should claim the paths to a particular device and to manage the device. esxcli storage core claimrule manages claim rules.

Claim rule modification commands do not operate on the VMkernel directly. Instead they operate on the configuration file by adding and removing rules

To change the current claim rules in the VMkernel
1
Run one or more of the esxcli storage core claimrule modification commands (add, remove, or move).
2
Run esxcli storage core claimrule load to replace the current rules in the VMkernel with the modified rules from the configuration file.

Claim rules are numbered as follows.

  • Rules 0–100 are reserved for internal use by VMware.
  • Rules 101–65435 are available for general use. Any third party multipathing plugins installed on your system use claim rules in this range. By default, the PSA claim rule 101 masks Dell array pseudo devices. Do not remove this rule, unless you want to unmask these devices.
  • Rules 65436–65535 are reserved for internal use by VMware.

When claiming a path, the PSA runs through the rules starting from the lowest number and determines is a path matches the claim rule specification. If the PSA finds a match, it gives the path to the corresponding plugin. This is worth noticing because a given path might match several claim rules.

The following examples illustrate adding claim rules.  

  • Add rule 321, which claims the path on adapter vmhba0, channel 0, target 0, LUN 0 for the NMP plugin.
  • esxcli storage core claimrule add -r 321 -t location -A vmhba0 -C 0 -T 0 -L 0 -P NMP
  • Add rule 429, which claims all paths provided by an adapter with the mptscsi driver for the MASK_PATH plugin.
  • esxcli storage core claimrule add -r 429 -t driver -D mptscsi -P MASK_PATH
  • Add rule 914, which claims all paths with vendor string VMWARE and model string Virtual for the NMP plugin.
  • esxcli storage core claimrule add -r 914 -t vendor -V VMWARE -M Virtual -P NMP
  • Add rule 1015, which claims all paths provided by FC adapters for the NMP plugin.
  • esxcli storage core claimrule add -r 1015 -t transport -R fc -P NMP

Example: Masking a LUN

In this example, you mask the LUN 20 on targets T1 and T2 accessed through storage adapters vmhba2 and vmhba3.

  • esxcli storage core claimrule list
  • esxcli  storage core claimrule add -P MASK_PATH -r 109 -t location -A
    vmhba2 -C 0 -T 1 -L 20
  • esxcli storage core claimrule add -P MASK_PATH -r 110 -t location -A
    vmhba3 -C 0 -T 1 -L 20
  • esxcli  storage core claimrule add -P MASK_PATH -r 111 -t location -A
    vmhba2 -C 0 -T 2 -L 20
  • esxcli storage core claimrule add -P MASK_PATH -r 112 -t location -A
    vmhba3 -C 0 -T 2 -L 20
  • esxcli storage core claimrule load
  • esxcli storage core claimrule list
  • esxcli storage core claiming unclaim -t location -A vmhba2
  • esxcli storage core claiming unclaim -t location -A vmhba3
  • esxcli storage core claimrule run

What is Pluggable Storage Architecture (PSA) and Native Multipathing (NMP)?

April 5, 2012 Objective 1 Storage, PSA and NMP, VMware No comments

Pluggable Storage Architecture

To manage storage multipathing, ESX/ESXi uses a special VMkernel layer which sits in the SCSI middle layer of the VMKernel I/O Stack, Pluggable Storage Architecture (PSA). The PSA is an open modular framework that coordinates the simultaneous operation of multiple multipathing plugins (MPPs). PSA is a collection of VMkernel APIs that allow third party hardware vendors to insert code directly into the ESX storage I/O path. This allows 3rd party software developers to design their own load balancing techniques and failover mechanisms for particular storage array. The PSA coordinates the operation of the NMP and any additional 3rd party MPP

What does PSA do?

  • Load and unload multipathing plug-ins
  • Uses predefined claim rules to assign each device to an MPP (One claim rule per device)
  • Handle physical path discovery and removal (through scanning)
  • Route I/O requests for a specific logical device to an appropriate multipathing plug-in
  • Handle I/O queuing to the physical storage HBAs and to the logical devices
  • Implement logical device bandwidth sharing between virtual machines
  • Provide logical device and physical path I/O statistics

Native Multipathing Plugin

The VMkernel multipathing plugin that ESX/ESXi provides, by default, is the VMware Native Multipathing Plugin (NMP). The NMP is an extensible module that manages subplugins. There are two types of NMP subplugins: Storage Array Type Plugins (SATPs), and Path Selection Plugins (PSPs). SATPs and PSPs can be built-in and provided by VMware, or can be provided by a third party.If more multipathing functionality is required, a third party can also provide an MPP to run in addition to, or as a replacement for, the default NMP.

VMware provides a generic Multipathing Plugin (MPP) called Native Multipathing Plugin (NMP) which supports all storage arrays on the Compatibility list

A single MPP can support multiple SATPs and PSPs. If a storage vendor has not supplied an MPP, SATP or PSP, VMware will use its own assigned by default

PSA

What does NMP do?

  1. Manages physical path claiming and unclaiming.
  2. Registers and de-registers logical devices
  3. Associates physical paths with logical devices.
  4. Processes I/O requests to logical devices
  5. Selects an optimal physical path for the request (load balance)
  6. Performs actions necessary to handle failures and request retries.
  7. Supports management tasks such as abort or reset of logical devices.

How it works

The ESX kernel (VMkernel) goes down through three layers when communicating with storage:

  1. In the top layer, VMware native NMP or third-party MPP software decides which SATP to use, or whether to use the native interface.
  2. The SATP layer includes native generic path selection (active/active, active/passive), standard ALUA, as well as allowing third-party plugins (SATP) to override its behavior. The SATP monitors these paths, reports changes, and initiates fail-over on the array as needed.
  3. At the PSP layer, software decides which physical channel to use for I/O requests.

In more detail

  • NMP assigns a SATP to every physical path to the logical device (datastore)
  • NMP associates paths to logical devices
  • NMP decides which PSP to use with the logical device.
  • The VM tells NMP an I/O is ready to send.
  • I/O is issued.
  • PSP is selected. Load-balances if applicable.
  • I/O is sent to  device.
  • Success:Device driver (Storage array) indicates I/O is complete. Failure: NMP calls appropriate SATP.
  • Success: NMP tells PSP I/O is complete. Failure: SATP interprets error codes and fails over to inactive paths.
  • Failure: PSP is called again to select which path to use for I/O excluding the failed path.
  • PSP checks every 300 seconds if the path is active again. SATP is responsible for doing the failover.

PSA Plugins

There are three types of PSA plugins for vSphere 4:

  1. Storage Array Type Plug-In (SATP)
  2. Path Selection Plug-in (PSP)
  3. A complete third-party multipathing software stack (MPP)

As is the case with VAAI, VMware includes a number of third-party plug-ins in the ESXi install. Users can simply activate many of these according to their needs, though some require additional fees and licensing.

SATP Plugins

SATPs allow load balancing across multiple paths, intelligent path selection, and over troubled conditions such as “chatter”, when passed rapidly fail back and forth between controllers.

The SATP has critical tasks to perform in the PSA stack:

  1. Decide which method of communication to use with the storage (PSA or native)
  2. Monitor the health of the physical I/O channels or paths
  3. Report any changes in the state of the paths up the stack
  4. Perform actions required to fail over storage between controllers on the array

VMware vSphere includes a variety of generic SATP plugins for storage arrays.

  • VMW_SATP_LOCAL – Local SATP for direct-attached devices
  • VMW_SATP_DEFAULT_AA – Generic for active/active arrays
  • VMW_SATP_DEFAULT_AP – Generic for active/passive arrays
  • VMW_SATP_ALUA – Asymmetric Logical Unit Access-compliant arrays
  • VMW_SATP_LSI – LSI/NetApp arrays from Dell, HDS, IBM, Oracle, SGI
  • VMW_SATP_SVC – IBM SVC-based systems (SVC, V7000, Actifio)
  • VVMW_SATP_SYMM – EMC Symmetrix DMX-3/DMX-4/VMAX, Invista
  • MW_SATP_CX – EMC/Dell CLARiiON  and Celerra (also VMW_SATP_ALUA_CX)
  • VMW_SATP_INV – EMC Invista and VPLEX
  • VMW_SATP_EQL – Dell EqualLogic systems

You can see which SATP plug-ins are available using the following esxcli command:

esxcli storage nmp satp list

PSP Plugins

In contrast to the diversity of VAAI and SATP plug-ins, the universe of path selection plug-ins is fairly small. Most storage arrays are supported with either Most Recently Used (MRU) or Fixed path selection approaches. Many also support Round Robin (RR) path selection. The only vendor with a specific PSP that is not also part of a full MPP (like EMC PowerPath or HDS HDLM) is Dell, which offers a special routed path selection plug-in for the EqualLogic iSCSI arrays.

  • VMW_PSP_MRU – Most-Recently Used (MRU) – Supports hundreds of storage arrays
  • VMW_PSP_FIXED – Fixed – Supports hundreds of storage arrays
  • VMW_PSP_RR – Round-Robin – Supports dozens of storage arrays
  • DELL_PSP_EQL_ROUTED – Dell EqualLogic iSCSI arrays

You can view the Path Polices in vCenter

  • Click on the Host
  • Click Configuration
  • Click Storage
  • Click on a Datastore and click Properties
  • Click Manage paths and you should see the below

paths

Array Types

  • Active /Active arrays use Fixed PSP Plugins
  • Active/Passive arrays use Most Recently Used PSP Plugins

ESXCLI Commands

  • esxcli storage nmp psp list

Capture1

  • esxcli nmp satp list

Capture2

  • esxcli storage core claimrule list

Capture3

  • esxcli storage nmp device list

Capture4