Tuning Configurations

• Deploy single-threaded applications on uniprocessor virtual machines, instead of on SMP virtual machines, for the best performance and resource use.
• VMware advise against using CPU Affinity as it generally constrains the scheduler BD can cause an improperly balanced load
• Use DRS where you can as this will balance the load for you
• Don’t configure your VMs for more vCPUs then their workloads require. Configuring a VM with more vCPUs then it needs will cause additional, unnecessary CPU utilization due to the increased overhead relating to multiple vCPUs
• Enable Hyperthreading in the BIOS. Check this in the BIOS and in vCenter, click on the host, select Configuration, select Properties and check that Hyperthreading is enabled
• When dealing with NUMA systems, ensure that node interleaving is disabled in the BIOS. If node interleaving is set to enabled it essentially disables NUMA capability on that host
• When possible configure the number of vCPUs to equal or less than the number of physical cores on a single NUMA node. When you configure equal or less vCPUs:physical cores the VM will get all its memory from that single NUMA node, resulting in lower memory access and latency times
• Sometimes for certain machines it may be beneficial to schedule all of the vCPUs on the same sockets in under committed systems which gives that VM full access to a Shared Last Level cache rather than spread across multiple processors. Adjust sched.cpu.vsmpConsolidate=”true” in the VMX Configuration file
• Pay attention to the Manufacturers recommendations for resources, especially application multithreading support
• Use processors which support Hardware-Assisted CPU Virtualization (Intel VT-x and AMD AMD-V)
• Use processors which support Hardware-Assisted MMU Virtualization (Intel EPT and AMD RVI)
• When configuring virtual machines, the total CPU resources needed by the virtual machines running on the system should not exceed the CPU capacity of the host. If the host CPU capacity is overloaded, the performance of individual virtual machines may degrade

Tuning Configurations

• Use Network I/O Control to utilise Limits, Shares and Qos priority tags for traffic
• Team NICs across PCI cards and switches for complete redundancy
• Using vDS switches gives you more features than the Standard Switch and minimises configuration time
• Utilise NIC Teaming where possible to provide failover and extra bandwidth
• Use Jumbo Frames where you can – MTU 9000 rather than 1500. Must be set the same end to end
• Keep physical NIC firmware updated
• Use VMXNET-3 Virtual Network Adapters where possible. Must be supported by the O/S you are running. Shares a ring buffer between the VM and VMKernel and uses zero copy which saves CPU cycles. Takes advantage of transmission packet coalescing to reduce address space switching
• DirectPath I/O may provide you a bump in network performance, but you really need to look at the use case. You can lose a lot of core functionality when using this feature, such as vMotion and FT (some special exceptions when running on UCS for vMotion) so you really need to look at the cost:benefit ratio and determine if it’s worth the tradeoffs
• Enable Discovery Protocols CDP and LLDP for extra information on your networks
• Make sure your NIC teaming policies on your Virtual switches match the correct policies on the physical switches
• Make sure your physical switches support cross stack etherchannel if you are planning on using this in a fully redundant networking solution
• Use static or ephemeral port bindings due to the deprecation of Dynamic Binding
• Choose 10GB ethernet over 1GB. This gives you Netqueue, a feature which use multiple transmit and receive queues to allow I/O processing across multiple CPUs
• Choose physical NICs with TCP Checksum Offload which reduces the load on the physical CPU by allowing the NIC to perform checksum operations on network packets
• Choose physical adapters with TCP Segmentation offload as this can reduce the CPU Overhead involved with sending large amounts of TCP traffic.
• To speed up packet handling, network adapters can be configured for direct memory access to high memory/ This bypasses the CPU and allows the NIC direct access to memory
• You can use DirectPath which allows a VM to directly access the physical NIC instead of using an emulated or paravirtual device however it is not compatible with certain features such as vMotion, Hot Add/Hot Remove, HA, DRS and Snapshots
• Use Split RX Mode on VMXNet-3 adapters is an ESXi feature that uses multiple physical CPUs to process network packets received in a single work queue. It is individually configured on each NIC. Good for Stock Exchanges and Multimedia companies
• Use VMCI if you have 2 VMs on the same host which require a high-speed communication channel which bypasses the guest or VMKernel networking stack
• In a native environment, CPU utilization plays a significant role in network throughput. To process higher levels of throughput, more CPU resources are needed. The effect of CPU resource availability on the network throughput of virtualized applications is even more significant. Because insufficient CPU resources will limit maximum throughput, it is important to monitor the CPU utilization of high-throughput workloads.

Appropriate BIOS and firmware settings

• Make sure you have the most up to date firmware for your Servers including all 3rd party cards
• Enable Hyperthreading. Note, you cannot enable hyperthreading on a system with great then 32 physical cores because of the logical limit of 64 CPUs
• Make sure the BIOS is set to enable all populated processor sockets and to enable all cores in each socket.
• Enable “Turbo Boost” in the BIOS if your processors support it
• Some NUMA-capable systems provide an option in the BIOS to disable NUMA by enabling node interleaving. In most cases you will get the best performance by disabling node interleaving (in other words, leaving NUMA enabled) These technologies automatically trap sensitive calls, eliminating the overhead required to
  do so in software. This allows the use of a hardware virtualization (HV) virtual machine monitor (VMM) as opposed to a binary translation (BT) VMM.
• Hardware-Assisted CPU Virtualization (Intel VT-x and AMD AMD-V) The first generation of hardware virtualization assistance, VT-x from Intel and AMD-V from AMD, became available in 2006. These technologies automatically trap sensitive calls, eliminating the overhead required to do so in software. This allows the use of a hardware virtualization (HV) virtual machine monitor (VMM) as opposed to a binary translation (BT) VMM.
• Hardware-Assisted MMU Virtualization (Intel EPT and AMD RVI) Some recent processors also include a new feature that addresses the overheads due to memory management unit (MMU) virtualization by providing hardware support to virtualize the MMU. ESX 4.0 supports this feature in both AMD processors, where it is called rapid virtualization indexing (RVI) or nested page tables (NPT), and in Intel processors, where it is called extended page tables (EPT).
• Cache prefetching mechanisms (sometimes called DPL Prefetch, Hardware Prefetcher, L2 Streaming Prefetch, or Adjacent Cache Line Prefetch) usually help performance, especially when memory access patterns are regular. When running applications that access memory randomly, however, disabling these mechanisms might result in improved performance.
• ESX 4.0 supports Enhanced Intel SpeedStep® and Enhanced AMD PowerNow!™ CPU power management technologies that can save power when a host is not fully utilized. However because these and other power-saving technologies can reduce performance in some situations, you should consider disabling them when performance considerations outweigh power considerations.
• Disable C1E halt state in the BIOS.
• Disable any other power-saving mode in the BIOS.
• Disable any unneeded devices from the BIOS, such as serial and USB ports.

Identify appropriate driver revisions required for optimal ESXi host performance

Check out the VMware HCL and (or) VMware KB 2030818 for recommended drivers and firmware for different vSphere versions.