Tag Archive for HA

Installing vCenter HA – 6.5U2c

October 12, 2018 VCSA 6.5 No comments

 

 

 

 

 

Installing vCenter HA

The vCenter High Availability architecture uses a three-node cluster to provide availability against multiple types of hardware and software failures. A vCenter HA cluster consists of one Active node that serves client requests, one Passive node to take the role of Active node in the event of failure, and one quorum node called the Witness node. Any Active and Passive node-based architecture that supports automatic failover relies on a quorum or a tie-breaking entity to solve the classic split-brain problem, which refers to data/availability
inconsistencies due to network failures within distributed systems maintaining replicated data. Traditional architectures use some form of shared storage to solve the split-brain problem. However, in order to support a vCenter HA cluster spanning multiple datacenters, our design does not assume a shared storage–based deployment. As a result, one node in the vCenter HA cluster is permanently designated as a quorum node, or a Witness node. The other two nodes in the cluster dynamically assume the roles of Active and Passive nodes.
vCenter Server availability is assured as long as there are two nodes running inside a cluster. However, a cluster is considered to be running in a degraded state if there are only two nodes in it. A subsequent failure in a degraded cluster means vCenter services are no longer available.

A vCenter Server appliance is stateful and requires a strong, consistent state for it to work correctly. The appliance state (configuration state or runtime state) is mainly composed of:

• Database data (stored in the embedded PostgreSQL database)
• Flat files (for example, configuration files).

The appliance state must be backed up in order for VCHA failover to work properly. For the state to be stored inside the PostgreSQL database, we use the PostgreSQL native replication mechanism to keep the database data of the primary and secondary in sync. For flat files, a Linux native solution, rsync, is used for replication.
Because the vCenter Server appliance requires strong consistency, it is a strong requirement to utilize a synchronous form of replication to replicate the appliance state from the Active node to the Passive node

 

 

 

 

 

 

 

 

 

 

 

Installing vCenter HA

  • Download the relevant vCenter HA iso from the VMware download page
  • Mount the iso from a workstation or server

 

 

 

 

 

 

 

 

  • We’ll now go through the process of installing the first vCenter Server. I have mounted the iso on my Windows 10 machine
  • Go to vcsa-ui-installer > win32 > installer.exe

 

 

 

 

 

 

 

 

  • Click Install

 

 

 

 

 

 

 

 

 

 

 

 

  • Click Next

 

 

 

 

 

 

 

 

 

 

 

 

  • Click Accept License Agreement

 

 

 

 

 

 

 

 

 

 

 

 

  • Select Embedded Platform Services Controller. Note you can deploy an external PSC. I am doing the install this way as I want to test the embedded linked mode functionality now available in 6.5U2+ between embedded platform services controllers (This will require the build of another vCenter HA with an embedded PSC which I’ll try and cover in an another blog)

 

 

 

 

 

 

 

 

 

 

 

 

  • Next put in the details for a vCenter or host as the deployment target

 

 

 

 

 

 

 

 

 

 

 

 

  • Select the Certificate

 

 

 

 

 

 

 

 

 

 

 

 

  • Put in an appliance, username and password for the new vCenter appliance

 

 

 

 

 

 

 

 

 

 

  • Choose the deployment size and the Storage Size. Click Next

 

 

 

 

 

 

 

 

 

 

  • Choose the datastore to locate the vCenter on. Note: I am running vSAN.

 

 

 

 

 

 

 

 

 

 

  • Configure network settings. Note: As I chose a host to deploy to, it does not give me any existing vDS port groups. I have chosen to deploy to a host rather than an existing vCenter as I am testing this for a Greenfield build at work which does not have any existing vCenters etc to start with, just hosts.
  • Note: It would be useful at the point to make sure you have entered the new vCenter name and IP address into DNS.

 

 

 

 

 

 

 

 

 

 

  • Check all the details are correct

 

 

 

 

 

 

 

 

 

 

  • Click Finish. It should now say Initializing and start deploying

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  • You should see the appliance is being deployed.

 

  • When the deployment has finished, you should see this screen.

  • You can carry on with Step 2 at this point but I closed the wizard at this point and I’m now going to log in to my vCenter and configure the appliance settings on https://techlabvca002.techlab.local:5480
  • Click Set up vCenter Server Appliance

 

 

 

 

 

 

 

 

 

  •  Log in to the vCenter

 

 

 

 

 

 

 

 

 

  • The below screen will pop up. Click Next

 

 

 

 

 

 

 

 

 

  • Check all details
  • Put in time servers. I’m connected to the internet through my environment so I use some generic time servers
  • Enable SSH if you need to – can be turned off again after configuration for security.

 

 

 

 

 

 

 

 

 

  • Put in your own SSO configuration
  • Click Next

 

 

 

 

 

 

 

 

 

  • Select or unselect the CEIP

 

 

 

 

 

 

 

 

 

  • Check all the details and click Finish

 

 

 

 

 

 

 

 

 

  • A message will pop up

 

 

 

 

 

 

 

 

 

  • The vCenter Appliance will begin the final installation

 

 

 

 

 

 

 

 

  • When complete, you should see the following screen

 

 

 

 

 

 

 

 

 

  • You can now connect to the vCenter Appliance on the 5480 port and the Web Client

 

 

 

 

 

 

  • Note: at this point I actually changed to enable VCHA on my normal first built vCenter called techlabvca001 as I should have added my second vCenter into the same SSO domain as techlabvca001 but I actually set it up as a completely different vCenter so it wouldn’t let me enable VCHA in the way I set it up. Log into the vSphere Web Client for techlabvca001
  • Highlight vCenter
  • Click the Configure tab
  • Choose Basic

  • Put in the Active vCenters HA address and subnet mask
  • Choose a port group

 

 

 

 

 

 

 

 

 

  • Click Next
  • Select Advanced and change the IP settings to what you want

 

 

 

 

 

 

 

 

 

  • Passive Node

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  • And the Witness Node

 

  • Click Next and you will be on the next screen which allows you to specify what location and datastores you can use to place the nodes

 

 

 

 

 

 

 

 

 

  • Click Edit on the Passive Node

 

 

 

 

 

 

 

 

 

  • Select the Compute Resource

 

 

 

 

 

 

 

 

 

  • Choose a datastore – In my case this will be my vSAN

 

 

 

 

 

 

 

 

 

  • Check the Compatibilty checks – In my case it is just notifying me about snapshots being lost when this created.

  • Next adjust the Witness settings – I am not going to go through them all again as they will be the same as the Passive node we just did.

 

 

 

 

 

 

 

 

 

  • Check the Management network and vCenter HA networks

 

 

 

 

 

 

 

 

 

  • Next and check the final details and click Finish

 

 

 

 

 

 

 

 

 

  • It will now say vCenter HA being deployed in the vSphere Web client

 

 

 

 

 

 

 

 

 

  • You should see a Peer machine and a Witness machine being deployed

 

 

 

 

 

 

 

 

  • Once complete you will see VCHA is enabled and you should see your Active vCenter, Passive vCenter and Witness

 

 

 

 

 

 

  • Click the Test Failover to check everything is working as expected

 

 

 

 

 

 

 

  • You can also place the HA Cluster in several modes

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

HA in VMware vSphere 5.x – What actually happens?

July 8, 2014 HA One comment

HEARTBEAT

The HA Question?

We were asked what actually happens to the hosts and VMs in vSphere 5.5 if an isolation event was triggered and we completely lost our host Management Network. (Which I have seen happen in the past!) I have written several blog posts about HA in the HA Category so I am not going to go back over these. I am just going to focus on this question with our settings which are set as below.

It is important to note that the restarting by VMware HA of virtual machines on other hosts in the cluster in the event of a host isolation or host failure is dependent on the “host monitoring” setting. If host monitoring is disabled, the restart of virtual machines on other hosts following a host failure or isolation is also disabled

On our Non Production Cluster and our Production Cluster we have HA enabled and Enable Host Monitoring turned on with Leave Powered On as our default

HA1

HA2

HA3

The vSphere architecture comprises of Master and Slave HA agents. Except during network partitions there is one master in the cluster. A master agent is responsible for monitoring the health of virtual machines and restarting any that fail. The Slaves are responsible for sending information to the master and restarting virtual machines as instructed by the master.

HA4

HA5

When a HA cluster is created it will begin by electing a master which will try and gain ownership of all the datastores it can directly access or by proxying requests to one of the slaves using the management network. It does this by locking a file called protectedlist that is stored on the datstores in an existing cluster. The master will also try and take ownership of any datastores that it discovers on the way and will periodically try any datatstores it could not access previously.

The master uses the protectlist file to store the inventory and keeps track of the virtual machines protected by HA. It then distributes the inventory across all the datastores

HA6

There is also a file called poweron located on a shared datastore which contains a list of powered on virtual machines. This file is used by slaves to inform the master that they are isolated by the top line of the file containing a 0 or 1 with 1 meaning isolated

HA7

Datastore Heartbeating

In vSphere versions prior to 5.x, machine restarts were always attempted, even if it was only the Management network which went down and the rest of the VM networks were running fine. This was not a desirable situation. VMware have introduced the concept of Datastore heartbeating which adds much more resiliency and false positives which resulted in VMs restarting unnecessarily.

Datastore Heartbeating is used when a master has lost network connectivity with a slave. The Datastore Heartbeating mechanism is then used to validate if a host has failed or is isolated/network partitioned which is validated through the poweron file as mentioned previously. By default HA picks 2 heartbeat datastores. To see which datastores, click on the vCenter name and select Cluster Status

HA3

Isolation and Network Partitioning

A host is considered to be either isolated or network partitioned when it loses network access to a master but has not completely failed.

Isolation

  • A host is not receiving any heartbeats from the master
  • A host is not receiving any election traffic
  • A host cannot ping the isolation address
  • Virtual machines may be restarted depending on the isolation response
  • A VM will only be shut down or powered off when the isolated host knows there is a master out there that has taken ownership for the VM or when the isolated host loses access to the home datastore of the VM

Network Partitioning

  • A host is not receiving any heartbeats from the master
  • A host is receiving election traffic
  • An election process will take place and the state reported to vCenter and virtual machines may be restarted depending on the isolation response

What happens if? 

  • The Master fails

If the slaves have not received any network heartbeats from the master, then the slaves will try and elect a new master. The new master will gather the required information and restart the VMs. The Datastore lock will expire and a newly elected master will relock the file if it has access to the Datastore

  • A Slave fails

The master along with monitoring the slave hosts also receives heartbeats from the slaves every second. If a slave fails or become isolated, the master will check for connectivity for 15 seconds then it will see if the host is still heartbeating to the datastore. Next it will try and ping the management gateway. If the datastore and management gateway prove negative then the host will be declared failed and determine which VMs need to be restarted and will try and distribute them fairly across the remaining hosts

  • Power Outage

If there is a Power Outage and all hosts power down suddenly then as soon as the power for the hosts returned, an election process will be kicked off and a master will be elected. The Master reads protected list which contains all VMs which are protected by HA and then the Master initiates restarts for those VMs which are listed as protected but not running

  • Complete Management Network failure

First of all it’s a very rare scenario where the Management Network becomes unavailable at the same time from all the running Host’s in the Cluster. VMware recommend to have redundant vmnics configured for the Host and each vmkernel management vmnic going into a different management switch for full redundancy. See pic below.

vmkernelredundant

If all the ESXi Hosts lose the Management Network then the Master and the Slaves will remain at the same state as there will be no election happening because the FDM agents communicate through the Management Network. Because the VMs will be accessible on the Datastores which the master knows by reading the protectedlist file and the poweron file on the Datastores, it will know if there is a complete failure of the Management network or a failure of itself or a slave or an isolation/network partition event. Each host will ping the isolation address and declare itself isolated. It will then trigger the isolation response which is to leave VMs powered on

A host remains isolated until it observes HA network traffic, like for instance election messages or it starts getting a response from an isolation address. Meaning that as long as the host is in an “isolated state” it will continue to validate its isolation by pinging the isolation address. As soon as the isolation address responds it will initiate an election process or join an existing election process and the cluster will return to a normal state.

Useful Link

Thanks to Iwan Rahabok 🙂

http://virtual-red-dot.blogspot.co.uk/2012/02/vsphere-ha-isolation-partition-and.html

 

 

Analyze HA cluster capacity to determine optimum cluster size

February 4, 2013 Objective 4 Business Continuity No comments

Capture

Cluster Capacity Recommendations

  • How many hosts do you have
  • How many VMs do you have
  • Do you have reservations on your VMs
  • Remember the limit of 32 hosts per cluster
  • Do you have Oracle clustering considerations (CPU Licensing)?
  • Do you have to have separate clusters because of large VMs?
  • By pooling together hosts into larger clusters, DRS is far more efficient at VM placement and providing resource management. It also allows for more efficient HA policy management since the absorption of spare capacity needed for infrequent host failures is now spread out over a larger set of hosts
  • Check all hosts can see shared storage
  • Check your networking. Inconsistency in the network configuration may result in virtual machines losing network connectivity after redistribution, in the event they are moved to a physical host lacking the required VLAN.
  • It’s critical to ensure there is complete network redundancy in all paths between hosts in the cluster. The greatest risk with VMware HA implementations is a false isolation event where an ESX host is incorrectly identified as being offline, triggering an isolation response.
  • DRS load balancing is not instantaneous, so balancing virtual machines with rapidly oscillating load with more consistent VMware HA and DRS Capacity Planning workload characteristics will likely require a larger cluster to ensure a more rapid redistribution response.
  • There are also performance considerations as to the number of hosts that should concurrently access a single LUN, as well as storage IO considerations. Too many hosts with concurrent access to a single LUN can inflict a performance penalty due to LUN-level SCSI reservations associated with virtual machine file operations and LUN metadata updates.
  • Although an HA cluster can contain up to sixteen hosts, recommended cluster sizes generally fall somewhere between eight and twelve, based on the individual organization’s needs.
  • VMware ESX host hardware in the cluster should be as homogeneous as possible, specifically in terms of memory capacity, CPU clock speed, and core count. Because DRS relies on VMotion to migrate running workloads, all hosts must be VMotion compatible
  • It is best to have hosts of the same spec so you don’t create a skewed HA Failover situation.

Analyse performance metrics to calculate host failure requirements

February 4, 2013 Objective 4 Business Continuity No comments

images

Performance Metrics to use

  • Use the inbuilt vCenter Performance charts

The vCenter Performance charts have 3 advanced settings which allow you to see Effective CPU Resources, Effective Memory Resources and Current Failover level

ClusterPerf

ClusterPerf2

ClusterPerf3

  • Use ESXTOP/RESXTOP

ESXTOP and RESXTOP can be used in batch mode to monitor hosts over a day maximum. You can monitor all statistics or select the counters you want to monitor. The resulting csv file can then be imported into EXCEL, ESXPLOT or Perfmon on a Windows server to analyse the results. From these, you can probably see if there are any resources that may restrict the choice of cluster setting you apply. If you are low on resources then selecting a Specify Failover Host setting is probably not the best idea as wasting a host as a hot standby when you are already contrained for resources will take even more resource away

Analyze vSphere environment to determine appropriate HA admission control policy

February 4, 2013 Objective 4 Business Continuity No comments

index

Choosing an Admission Control Policy

You should choose a vSphere HA admission control policy based on your availability needs and the characteristics of your cluster. When choosing an admission control policy, you should consider a number of factors.

  • Avoiding Resource Fragmentation

Resource fragmentation occurs when there are enough resources in aggregate for a virtual machine to be failed over. However, those resources are located on multiple hosts and are unusable because a virtual machine can run on one ESXi host at a time. The Host Failures Cluster Tolerates policy avoids resource fragmentation by defining a slot as the maximum virtual machine reservation. The Percentage of Cluster Resources policy does not address the problem of resource fragmentation. With the Specify Failover Hosts policy, resources are not fragmented because hosts are reserved for failover.

  • Flexibility of Failover Resource Reservation

Admission control policies differ in the granularity of control they give you when reserving cluster resources for failover protection. The Host Failures Cluster Tolerates policy allows you to set the failover level as a number of hosts. The Percentage of Cluster Resources policy allows you to designate up to 100% of cluster CPU or memory resources for failover. The Specify Failover Hosts policy allows you to specify a set of failover hosts.

  • Heterogeneity of Cluster

Clusters can be heterogeneous in terms of virtual machine resource reservations and host total resource capacities. In a heterogeneous cluster, the Host Failures Cluster Tolerates policy can be too conservative because it only considers the largest virtual machine reservations when defining slot size and assumes the largest hosts fail when computing the Current Failover Capacity. The other two admission control policies are not affected by cluster heterogeneity.

Understand interactions between DRS and HA

February 4, 2013 Objective 4 Business Continuity No comments

index

Using vSphere HA and DRS Together

Using vSphere HA with Distributed Resource Scheduler (DRS) combines automatic failover with load balancing. This combination can result in a more balanced cluster after vSphere HA has moved virtual machines to different hosts.
When vSphere HA performs failover and restarts virtual machines on different hosts, its first priority is the immediate availability of all virtual machines. After the virtual machines have been restarted, those hosts on which they were powered on might be heavily loaded, while other hosts are comparatively lightly loaded.
vSphere HA uses the virtual machine’s CPU and memory reservation to determine if a host has enough spare capacity to accommodate the virtual machine.

In a cluster using DRS and vSphere HA with admission control turned on, virtual machines might not be evacuated from hosts entering maintenance mode. This behavior occurs because of the resources reserved for restarting virtual machines in the event of a failure. You must manually migrate the virtual machines off of the hosts using vMotion.
In some scenarios, vSphere HA might not be able to fail over virtual machines because of resource constraints. This can occur for several reasons.

  • HA admission control is disabled and Distributed Power Management (DPM) is enabled. This can result in DPM consolidating virtual machines onto fewer hosts and placing the empty hosts in standby mode leaving insufficient powered-on capacity to perform a failover.
  • VM-Host affinity (required) rules might limit the hosts on which certain virtual machines can be placed.
  • There might be sufficient aggregate resources but these can be fragmented across multiple hosts so that they can not be used by virtual machines for failover.

In such cases, vSphere HA can use DRS to try to adjust the cluster (for example, by bringing hosts out of standby mode or migrating virtual machines to defragment the cluster resources) so that HA can perform the failovers.
If DPM is in manual mode, you might need to confirm host power-on recommendations. Similarly, if DRS is in manual mode, you might need to confirm migration recommendations. If you are using VM-Host affinity rules that are required, be aware that these rules cannot be violated. vSphere HA does not perform a failover if doing so would violate such a rule.

Configure HA Related alarms and monitor a HA Cluster

February 4, 2013 Objective 4 Business Continuity No comments

images

Monitoring a cluster

  • Highlight the cluster
  • Select Summary
  • You will see the following page

ha3

  • Click Advanced Runtime. This tells you the current state of your cluster including the slot information and host information

ha4

  • Click on Cluster Operational Status to see informational and warning messages associated with your cluster

ha5

  • Click Cluster Status and tab through the below options

ha1

ha2

ha3

Configuring Alarms

You may setup custom alerts of your own to monitor the health of your HA cluster. However, VMware provides a number of default alerts for you described below. These alarms may be updated to take custom actions. To access these settings, connect to the vCenter server using the vSphere client:

  1. Enter the Host and Clusters view (Ctrl + Shift + H)
  2. Highlight the vCenter Server
  3. Click on the Alarms tab
  4. Select the Definitions view
  5. Select one of the below described HA default alarms
  6. Right click the alarm and select Edit Settings
  7. Click the Actions tab

ha6

Default System Actions

  1. Send a Notification email
  2. Send a Notification trap
  3. Run a command
  4. Power on VM
  5. Power off VM
  6. Suspend VM
  7. Reset VM
  8. Migrate VM
  9. Reboot Guest VM
  10. Shutdown Guest VM

To Configure a Sender Email Account for the vCenter Server

  1. Enter the Server Settings dialog (Ctrl + Shift + I)
  2. Click on Mail
  3. Enter an SMTP Server
  4. Enter a Sender Account
  5. Note. Additional settings will be required to allow vCenter to use a pre-defined account to send emails. Check with your third party email server documentation when setting up this account.

To Configure SNMP

  1. Enter the Server Settings dialog (Ctrl + Shift + I)
  2. Click on SNMP
  3. Enter up to four different Receiver URLs and Community Strings for your SNMP environment

Configure customised isolation response settings

February 4, 2013 HA, Objective 4 Business Continuity No comments

HA Isolation Responses

As seen in the below diagram, when HA detects a failure on one of the hosts, a response is triggered to deal with the Virtual Machines on that host

ha

Host Isolation Responses

First of all we need to look at the Host Monitoring which is a selectable box within the HA Settings

Host Monitoring

The restarting by VMware HA of virtual machines on other hosts in the cluster in the event of a host isolation or host failure is dependent on the “host monitoring” setting. If host monitoring is disabled, the restart of virtual machines on other hosts following a host failure or isolation is also disabled. Disabling host monitoring also impacts VMware Fault Tolerance because it controls whether HA will restart a Fault Tolerance (FT) secondary virtual machine after an event. Essentially a host will always perform the programmed host isolation response when it determines it is isolated. The host monitoring setting determines if virtual machines will be restarted elsewhere following this event.

iso2

Isolation  Responses

When an isolation response is triggered, the isolated host must determine whether it must take any action based upon the configuration settings for the isolation response for each virtual machine that is powered on. The isolation response setting provides a means to dictate the action desired for the powered-on virtual machines maintained by a host when that host is declared isolated. There are three possible isolation response values that can be configured and applied to a cluster or individually to a specific virtual machine.These are

  • Leave Powered On
  • Power Off
  • Shut Down

isolation

Leave Powered On

With this option, virtual machines hosted on an isolated host are left powered on. In situations where a host loses all management network access, it might still have the ability to access the storage subsystem and the virtual machine network. Selecting this option enables the virtual machine to continue to function if this were to occur. This is now the default isolation response setting in vSphere High Availability 5.0.

Power Off
When this isolation response option is used, the virtual machines on the isolated host are immediately stopped. This is similar to removing the power from a physical host. This can induce inconsistency with the file system of the OS used in the virtual machine. The advantage of this action is that VMware HA will attempt to restart the virtual machine more quickly than when using the third option.

Shut Down

Through the use of the VM Tools package installed within the guest operating system of a virtual machine, this option attempts to gracefully shut down the operating system with the virtual machine before powering off the virtual machine. This is more desirable than using the Power Off option because it provides the OS with time to commit any outstanding I/O activity to disk. HA will wait for a default of 300 seconds (5 minutes) for this graceful shutdown to occur. If the OS is not gracefully shut down by this time, it will initiate a power-off of the virtual machine. Changing the das.isolationshutdowntimeout attribute will modify this timeout if it is determined that more time is required to gracefully shut down an OS. The Shut Down option requires that the VM Tools package be installed in the guest OS. Otherwise, it is equivalent to the Power Off setting.

Best Practices

From a best practices perspective, Leave Powered On is the recommended isolation response setting for the majority of environments. Isolated hosts are a rare event in a properly architected environment, given the redundancy built in. In environments that use network-based storage protocols, such as iSCSI and NFS, the recommended isolation response is Power Off. With these environments, it is highly likely that a network outage that causes a host to become isolated will also affect the host’s ability to communicate to the datastores.
An isolated host will initiate the configured isolation response for a running virtual machine if either of the following is true

  • The host lost access to the datastore containing the configuration (.vmx) file for the virtual machine
  • The host still has access to the datastore and it determined that a master is responsible for the virtual machine.

To determine this, the isolated host checks for the accessibility of the “home datastore” for each virtual machine and whether the virtual machines on that datastore are “owned” by a master, which is indicated by a master’s having exclusively locked a key file that HA maintains on the datastore. After declaring itself as being isolated, the isolated host releases any locks it might have held on any datastores. It then checks periodically to see whether a master has obtained a lock on the datastore. After a lock is observed on the datastore by the isolated host, the HA agent on the isolated host applies the configured isolation response. Ensuring that a virtual machine is under continuous protection by a master provides an additional layer of protection. Because only one master can lock a datastore at a given time, this significantly reduces chances of “split-brain” scenarios. This also protects against situations where a complete loss of the management networks without a complete loss of access to storage would make all the hosts in a cluster determine they were isolated.
In certain environments, it is possible for a loss of the management network to also affect access to the heartbeat datastores. This is the case when the heartbeat datastores are hosted via NFS that is tied to the management network in some manner. In the event of a complete loss of connectivity to the management network and the heartbeat datastores, the isolation response activity resembles that observed in vSphere 4.x.
In this configuration, the isolation response should be set to Power Off so another virtual machine with access to the network can attempt to power on the virtual machine.
There is a situation where the isolation response will likely take an extended period of time to transpire. This occurs when all paths to storage are disconnected, referred to as an all-paths-down (APD) state, and the APD condition does not impact all of the datastores mounted on the host. This is due to the fact that there might be outstanding write requests to the storage subsystem that must time out. Establishing redundant paths to the storage subsystem will help prevent an APD situation and this issue.

Configure HA Redundancy

February 4, 2013 Objective 4 Business Continuity No comments

HA in vSphere 5

The way HA works in vSphere 5 is quite different to the way it worked in vSphere 4. vSphere HA now uses a new tool called FDM (Fault Domain Manager) which has been developed to replace AAM (Automated Availability Manager) AAM had limitations in reliance on name resolution and scalability limits. Improvements such as

  • FDM uses a Master/Slave architecture which does not rely on Primary/Secondary host designations
  • FDM uses both the management network and storage devices for communications
  • FDM introduces support for IPv6
  • FDM addresses the issues of network partition and network isolation

How it works

  • When vSphere HA is enabled, the vSphere HA agents enter an election to pick a vSphere HA Master.
  • The vSphere HA master monitors slave hosts and will restart VMs in the event of a failover
  • The vSphere HA master monitors the power state of all protected machines and if the VM fails, it will be restarted
  • The vSphere HA master manages the list of hosts that are members of the cluster and manages the adding/removing of hosts into a cluster
  • The vSphere HA master manages the list of protected VMs
  • The vSphere HA master caches the cluster configuration. The master notifies and informs slave hosts of changes in the cluster
  • The vSphere HA master sends heartbeat messages to the slave hosts so the slaves know the master is still alive
  • The vSphere HA master reports state information to vCenter (Only the master does this)

HA  Process

  • The hosts within an HA cluster constantly heartbeat with the host designated as the master over the management network. The first step in determining whether a host is isolated is detecting a lack of these heartbeats.
  • After a host stops communicating with the master, the master attempts to determine the cause of the issue.
  • Using heartbeat datastores, the master can distinguish whether the host is still alive by determining if the affected host is maintaining heartbeats to the heartbeat datastores. This enables the master to differentiate between a management network failure, a dead host and a partitioned/isolated situation.
  • The time elapsed before the host declares itself isolated varies depending on the role of the host (master or slave) at the time of the loss of heartbeats.
  • If the host was a master, it will declare itself isolated within 5 seconds.
  • If the host was a slave, it will declare itself isolated in 30 seconds.
  • The difference in time is due to the fact that if the host was a slave, it then must go through an election process to identify whether any other hosts exist or if the master host simply died. This election process starts for the slave at 10 seconds after the loss of heartbeats is detected.
  • If the host sees no response from another host during the election for 15 seconds, the HA agent on a host then elects itself as a master, checks whether it is isolated and, if so, drops into a startup state.
  • In short, a host will begin to check to see whether it is isolated whenever it is a master in a cluster with more than one other host and has no slaves. It will continue to do so until it becomes a master with a slave or connects to a master as a slave.
  • At this point, the host will attempt to ping its configured isolation addresses to determine the viability of the network. The default isolation address is the gateway specified for the management network.
  • Advanced settings can be used to modify the isolation addresses used for your particular environment. The option das. isolationaddress[X] (where X is 1–10) is used to configure multiple isolation addresses.
  • Additionally das. usedefaultisolationaddress is used to indicate whether the default isolation address (the default gateway) should be used to determine if the host is network isolated. If the default gateway is not able to receive ICMP ping packets, you must set this option to “false.”
  • It is recommended to set one isolation address for each management network used, keeping in mind that the management network links should be redundant, as previously mentioned.
  • The isolation address used should always be reachable by the host under normal situations, because after 5 seconds have elapsed with no response from the isolation addresses, the host then declares itself isolated.
  • After this occurs, it will attempt to inform the master of its isolated state by use of the heartbeat datastores.

What does HA use?

  • Management Network
  • Datastore Heartbeats

Management Network

Ideally the Management network should be set up as a fully redundant network team at the adapter level or at the Management network level. It can either be setup as Etherchannel with Route based on IP Hash or in an Active/Standby configuration allowing for failover should one network card fail

In the event that the vSphere HA Master cannot communicate with a slave through the management network isolation address, it can then check its Heartbeat Datastores to see if the host is still up and running. This helps vSphere HA deal with Network Partitioning and Network Isolation

Switch Setup

Requirements:

  • Two physical network adaptors
  • VLAN trunking
  • Two physical switches

The vSwitch should be configured as follows:

  • Load balancing = route based on the originating virtual port ID (default)
  • Failback = no
  • vSwitch0: Two physical network adaptors (for example: vmnic0 and vmnic2)
  • Two port groups (for example, vMotion and management)

In this example, the management network runs on vSwitch0 as active on vmnic0 and as standby on vmnic2. The vMotion network runs on vSwitch0 as active on vmnic2 and as standby on vmnic0
Each port group has a VLAN ID assigned and runs dedicated on its own physical network adaptor. Only in the case of a failure is it switched over to the standby network adaptor. Failback is set to “no” because in the case of physical switch failure and restart, ESXi might falsely recognize that the switch is back online when its ports first come online. In reality, the switch might not be forwarding on any packets until it is fully online. However, when failback is set to “no” and an issue arises, both your management network and vMotion network will be running on the same network adaptor and will continue running until you manually intervene.

Network Partitioning

This is what happens when one or more of the slaves cannot communicate with the vSphere HA Master even though they still have network connectivity. Checking the Datastore heartbeats is then used to see whether the slave hosts are alive

When a management network failure occurs for a vSphere HA cluster, a subset of the cluster’s hosts might be unable to communicate over the management network with the other hosts. Multiple partitions can occur in a cluster.
A partitioned cluster leads to degraded virtual machine protection and cluster management functionality. Correct the partitioned cluster as soon as possible.

Virtual machine protection. vCenter Server allows a virtual machine to be powered on, but it is protected only if it is running in the same partition as the master host that is responsible for it. The master host must be communicating with vCenter Server. A master host is responsible for a virtual machine if it has exclusively locked a system-defined file on the datastore that contains the virtual machine’s configuration file.

Cluster management. vCenter Server can communicate with only some of the hosts in the cluster, and it can connect to only one master host. As a result, changes in configuration that affect vSphere HA might not take effect until after the partition is resolved. This failure could result in one of the partitions operating under the old configuration.

Advanced Cluster Settings which can be used

  • das.isolationaddressx Used to configure multiple isolation addresses.
  • das.usedefaultisolationaddress Set to true/false and used in the cse where a default gateway is not pingable, in which case this set to false in conjunction with configuring another address for das.isolationaddress
  • das.failuredetectiontime. Increase to 30 seconds (30000) to decrease the likelyhood of a false positive

NOTE: If you change the value of any of the following advanced attributes, you must disable and then re-enable vSphere HA before your changes take effect.

Network Isolation

This is where one or more slave hosts have lost all management network connectivity. Isolated hosts can’t communicate with the master or other slaves. The slave host uses the heartbeat datastore to notify the master that it is isolated via special binary file called host-X-poweron file.

Datastore Heartbeating

By default, vCenter will automatically select two datastores to use for storage heartbeats. An algorithm designed to maximize availability and redundancy of the storage heartbeats selects these datastores. This algorithm attempts to select datastores that are connected to the highest number of hosts. It also attempts to select datastores that are hosted on different storage arrays/NFS servers. A preference is given to VMware vSphere VMFS–formatted datastores, although NFS-hosted datastores can also be used.

  • Highlight your cluster
  • Click Edit Settings
  • Select Datastore Heartbeating

Datastore Heartbeat

  • Select only from my preferred datastores restricts HA to using only those selected from the list
  • Select any of the cluster datastores disables the selection of datastores from the list. Any cluster datastore can be used by HA for heartbeating
  • Select any of the cluster datastores taking into account my preferences is a mix of the previous 2 options. The Admin selects the preferred datastores that HA should use. vSphere selects which ones to use from these. If any become unavailable, HA will choose another from the list.
  • The vSphere HA Cluster Status box will show you which datastores are being used

Datastore Heartbeat2

VMware Availability Guide

http://pubs.vmware.com

vSphere High Availability Deployment Best Practices

http://www.vmware.com/files/pdf/techpaper/vmw-vsphere-high-availability.pdf

 

Can you have VMware hosts with different amounts of RAM?

October 2, 2012 HA, VMware No comments

Yes you can have hosts with different amounts of RAM in a cluster.

HA will work ok as long as one host doesn’t have less RAM than an actual VM needs as a reservation

For example: If you have 3 hosts (2 with 64 GB RAM) and one with 32 GB RAM. If you then have a large VM with a 36 GB reservation then each host needs to be able to power on the largest VM in the event of a failover and in this case the host with 32 GB ram would not be able to power on the VM.

« Older Entries