Pluggable Storage Architecture
To manage storage multipathing, ESX/ESXi uses a special VMkernel layer which sits in the SCSI middle layer of the VMKernel I/O Stack, Pluggable Storage Architecture (PSA). The PSA is an open modular framework that coordinates the simultaneous operation of multiple multipathing plugins (MPPs). PSA is a collection of VMkernel APIs that allow third party hardware vendors to insert code directly into the ESX storage I/O path. This allows 3rd party software developers to design their own load balancing techniques and failover mechanisms for particular storage array. The PSA coordinates the operation of the NMP and any additional 3rd party MPP
What does PSA do?
- Load and unload multipathing plug-ins
- Uses predefined claim rules to assign each device to an MPP (One claim rule per device)
- Handle physical path discovery and removal (through scanning)
- Route I/O requests for a specific logical device to an appropriate multipathing plug-in
- Handle I/O queuing to the physical storage HBAs and to the logical devices
- Implement logical device bandwidth sharing between virtual machines
- Provide logical device and physical path I/O statistics
Native Multipathing Plugin
The VMkernel multipathing plugin that ESX/ESXi provides, by default, is the VMware Native Multipathing Plugin (NMP). The NMP is an extensible module that manages subplugins. There are two types of NMP subplugins: Storage Array Type Plugins (SATPs), and Path Selection Plugins (PSPs). SATPs and PSPs can be built-in and provided by VMware, or can be provided by a third party.If more multipathing functionality is required, a third party can also provide an MPP to run in addition to, or as a replacement for, the default NMP.
VMware provides a generic Multipathing Plugin (MPP) called Native Multipathing Plugin (NMP) which supports all storage arrays on the Compatibility list
A single MPP can support multiple SATPs and PSPs. If a storage vendor has not supplied an MPP, SATP or PSP, VMware will use its own assigned by default
What does NMP do?
- Manages physical path claiming and unclaiming.
- Registers and de-registers logical devices
- Associates physical paths with logical devices.
- Processes I/O requests to logical devices
- Selects an optimal physical path for the request (load balance)
- Performs actions necessary to handle failures and request retries.
- Supports management tasks such as abort or reset of logical devices.
How it works
The ESX kernel (VMkernel) goes down through three layers when communicating with storage:
- In the top layer, VMware native NMP or third-party MPP software decides which SATP to use, or whether to use the native interface.
- The SATP layer includes native generic path selection (active/active, active/passive), standard ALUA, as well as allowing third-party plugins (SATP) to override its behavior. The SATP monitors these paths, reports changes, and initiates fail-over on the array as needed.
- At the PSP layer, software decides which physical channel to use for I/O requests.
In more detail
- NMP assigns a SATP to every physical path to the logical device (datastore)
- NMP associates paths to logical devices
- NMP decides which PSP to use with the logical device.
- The VM tells NMP an I/O is ready to send.
- I/O is issued.
- PSP is selected. Load-balances if applicable.
- I/O is sent to device.
- Success:Device driver (Storage array) indicates I/O is complete. Failure: NMP calls appropriate SATP.
- Success: NMP tells PSP I/O is complete. Failure: SATP interprets error codes and fails over to inactive paths.
- Failure: PSP is called again to select which path to use for I/O excluding the failed path.
- PSP checks every 300 seconds if the path is active again. SATP is responsible for doing the failover.
PSA Plugins
There are three types of PSA plugins for vSphere 4:
- Storage Array Type Plug-In (SATP)
- Path Selection Plug-in (PSP)
- A complete third-party multipathing software stack (MPP)
As is the case with VAAI, VMware includes a number of third-party plug-ins in the ESXi install. Users can simply activate many of these according to their needs, though some require additional fees and licensing.
SATP Plugins
SATPs allow load balancing across multiple paths, intelligent path selection, and over troubled conditions such as “chatter”, when passed rapidly fail back and forth between controllers.
The SATP has critical tasks to perform in the PSA stack:
- Decide which method of communication to use with the storage (PSA or native)
- Monitor the health of the physical I/O channels or paths
- Report any changes in the state of the paths up the stack
- Perform actions required to fail over storage between controllers on the array
VMware vSphere includes a variety of generic SATP plugins for storage arrays.
- VMW_SATP_LOCAL – Local SATP for direct-attached devices
- VMW_SATP_DEFAULT_AA – Generic for active/active arrays
- VMW_SATP_DEFAULT_AP – Generic for active/passive arrays
- VMW_SATP_ALUA – Asymmetric Logical Unit Access-compliant arrays
- VMW_SATP_LSI – LSI/NetApp arrays from Dell, HDS, IBM, Oracle, SGI
- VMW_SATP_SVC – IBM SVC-based systems (SVC, V7000, Actifio)
- VVMW_SATP_SYMM – EMC Symmetrix DMX-3/DMX-4/VMAX, Invista
- MW_SATP_CX – EMC/Dell CLARiiON and Celerra (also VMW_SATP_ALUA_CX)
- VMW_SATP_INV – EMC Invista and VPLEX
- VMW_SATP_EQL – Dell EqualLogic systems
You can see which SATP plug-ins are available using the following esxcli command:
esxcli storage nmp satp list
PSP Plugins
In contrast to the diversity of VAAI and SATP plug-ins, the universe of path selection plug-ins is fairly small. Most storage arrays are supported with either Most Recently Used (MRU) or Fixed path selection approaches. Many also support Round Robin (RR) path selection. The only vendor with a specific PSP that is not also part of a full MPP (like EMC PowerPath or HDS HDLM) is Dell, which offers a special routed path selection plug-in for the EqualLogic iSCSI arrays.
- VMW_PSP_MRU – Most-Recently Used (MRU) – Supports hundreds of storage arrays
- VMW_PSP_FIXED – Fixed – Supports hundreds of storage arrays
- VMW_PSP_RR – Round-Robin – Supports dozens of storage arrays
- DELL_PSP_EQL_ROUTED – Dell EqualLogic iSCSI arrays
You can view the Path Polices in vCenter
- Click on the Host
- Click Configuration
- Click Storage
- Click on a Datastore and click Properties
- Click Manage paths and you should see the below
Array Types
- Active /Active arrays use Fixed PSP Plugins
- Active/Passive arrays use Most Recently Used PSP Plugins
ESXCLI Commands
- esxcli storage nmp psp list
- esxcli nmp satp list
- esxcli storage core claimrule list
- esxcli storage nmp device list