This is a short post because I came across a really useful link to a PDF document showing how vCenter and ESXi are connected. This shows what ports and what direction the ports travel which is really useful to understand the internal and external communication with explanations for each port.
Link
Downloadable PDF within the KB
https://kb.vmware.com/s/article/2131180
The Problem
A host’s VDS Status says Out of Sync in the Networking View
If network connectivity is interrupted between the vCenter Server and one or more hosts, a synchronization interval may be missed resulting in this alert being displayed. This type of interruption can occur during vCenter Service restarts, vCenter Server reboots as well as ESX/ESXi host reboots or network maintenance.
The Solution
To manually synchronize the host vDS information from the vSphere web client (vSphere 5.5):
The Task
Change an E1000 NIC to a VMXNET3 NIC
Instructions
What happens if you encounter ghost adapters?
Sometimes you will remove an adapter but the Windows servers will still think it is there especially if you try and rename it and it says the adapter exists but you can’t see it.
If you want to see this ghost adapter, try the following
VLANs provide for logical groupings of stations or switch ports, allowing communications as if all stations or ports were on the same physical LAN segment. Confining broadcast traffic to a subset of the switch ports or end users saves significant amounts of network bandwidth and processor time.
In order to support VLANs for VMware Infrastructure users, one of the elements on the virtual or physical network has to tag the Ethernet frames with 802.1Q tag as per below
The most common tagging is 802.1Q, which is an IEEE standard that nearly all switches support. The tag is there to identify which VLAN the layer 2 frame belongs to. vSphere can both understand these tags (receive them) as well as add them to outbound traffic (send them)
There are three different configuration modes to tag (and untag) the packets for virtual machine frames
1. VST (Virtual Switch Tagging)
This is the most common configuration. In this mode, you provision one port group on a virtual switch for each VLAN, then attach the virtual machine’s virtual adapter to the port group instead of the virtual switch directly.
The virtual switch port group tags all outbound frames and removes tags for all inbound frames. It also ensures that frames on one VLAN do not leak into a different VLAN.
Use of this mode requires that the physical switch provide a trunk. E.g The ESX host network adapters must be connected to trunk ports on the physical switch.
The port groups connected to the virtual switch must have an appropriate VLAN ID specified
switchport trunk encapsulation dot1q
switchport mode trunk
switchport trunk allowed vlan x,y,z
spanning-tree portfast trunk
Note: The Native VLAN is not tagged and thus requires no VLAN ID to be set on the ESX/ESXi portgroup.
2. VGT (Virtual Guest Tagging)
You may install an 802.1Q VLAN trunking driver inside the virtual machine, and tags will be preserved between the virtual machine networking stack and external switch when frames are passed from or to virtual switches. Use of this mode requires that the physical switch provide a trunk
3. EST (External Switch Tagging)
You may use external switches for VLAN tagging. This is similar to a physical network, and VLAN configuration is normally transparent to each individual physical server.
There is no need to provide a trunk in these environments.
All VLAN tagging of packets is performed on the physical switch.
ESX host network adapters are connected to access ports on the physical switch.
The portgroups connected to the virtual switch must have their VLAN ID set to 0.
See this example snippet of a code from a Cisco switch port configuration:
switchport mode access
switchport access vlan x
Virtual Distributed Switches
In vSphere, there’s a new networking feature which can be configured on the distributed virtual switch (or DVS). In VI3 it is only possible to add one VLAN to a specific port group in the vSwitch. in the DVS, you can add a range of VLANs to a single port group. The feature is called VLAN trunking and it can be configured when you add a new port group. There you have the option to define a VLAN type, which can be one of the following:
The VLAN policy allows virtual networks to join physical VLANs.
What is a VLAN Trunk?
A VLAN trunk is a port on a physical switch that has the ability to listen and pass traffic for multiple VLANs. Trunks are used primarily to pass traffic between multiple switches.
In Cisco networks, trunking is a special function that can be assigned to a port, making that port capable of carrying traffic for any or all of the VLANs accessible by a particular switch. Such a port is called a trunk port, in contrast to an access port, which carries traffic only to and from the specific VLAN assigned to it. A trunk port marks frames with special identifying tags (either ISL tags or 802.1Q tags) as they pass between switches, so each frame can be routed to its intended VLAN. An access port does not provide such tags, because the VLAN for it is pre-assigned, and identifying markers are therefore unnecessary.
A quick note on the relationship between VLANs and vSwitch port groups. A VLAN can contain multiple port groups, but a port group can only be associated with one VLAN at any given time. A prerequisite for VLAN functionality on a vSwitch (vSS or vDS) is that the vSwitch uplinks must be connected to a trunk port on the physical switch. This trunk port will also need to include the associated VLAN ID range, enabling the physical switch to pass VLAN tags to the ESXi host. So why is any of this important? A trunk port can store and distribute multiple VLAN tags, enabling multiple traffic types to flow independently (at least logically), across the same uplink or group of uplinks in the case of teamed NICs
Use case for using VLAN trunking would be if you have multiple VLANs in place for logical separation or to isolate your VM traffic but you have a limited amount of physical uplink ports dedicated for your ESXi hosts
Networking Policies
Policies set at the standard switch or distributed port group level apply to all of the port groups on the standard switch or to ports in the distributed port group. The exceptions are the configuration options that are overridden at the standard port group or distributed port level.
Useful Post (Thanks to Mohammed Raffic)
Benefits of NIC teaming include load balancing and failover: However, those policies will affect outbound traffic only. In order to control inbound traffic, you have to get the physical switches involved.
NIC Teaming Policies
|
Network Teaming Setting |
Description |
| Route based on the originating virtual port | Choose an uplink based on the virtual port where the traffic entered the virtual switch. |
| Route based on IP hash | Choose an uplink based on a hash of the source and destination IP addresses of each packet. For non-IP packets, whatever is at those offsets is used to compute the hash.Used for Etherchannel when set on the switch |
| Route based on source MAC hash | Choose an uplink based on a hash of the source Ethernet. |
| Route based on physical NIC load | Choose an uplink based on the current loads of physical NICs. |
| Use explicit failover order | Always use the highest order uplink from the list of Active adapters which passes failover detection criteria |
There are two ways of handling NIC teaming in VMware ESX:
There is a corresponding vSwitch configuration that matches each of these types of NIC teaming:
Considerations for NIC teaming without physical switch configuration
Something to be aware of when setting up NIC Teaming without physical switch configuration is that you don’t get true load balancing as you do with Etherchannel. The following applies to the NIC Teaming Settings
Route based on the originating virtual switch port ID
Choose an uplink based on the virtual port where the traffic entered the virtual switch. This is the default configuration and the one most commonly deployed.
When you use this setting, traffic from a given virtual Ethernet adapter is consistently sent to the same physical adapter unless there is a failover to another adapter in the NIC team.
Replies are received on the same physical adapter as the physical switch learns the port association.
* This setting provides an even distribution of traffic if the number of virtual Ethernet adapters is greater than the number of physical adapters.
Route based on source MAC hash
Choose an uplink based on a hash of the source Ethernet MAC address.
When you use this setting, traffic from a given virtual Ethernet adapter is consistently sent to the same physical adapter unless there is a failover to another adapter in the NIC team.
Replies are received on the same physical adapter as the physical switch learns the port association.
* This setting provides an even distribution of traffic if the number of virtual Ethernet adapters is greater than the number of physical adapters.
When creating a Virtual machine, VMware will normally offer you several choices of network adaptor depending on what O/S you select.
Network Adaptor Types
New Features
You can enable Jumbo frames on a vSphere Distributed Switch or Standard switch by changing the maximum MTU. TSO (TCP Segmentation Offload is enabled on the VMKernel interface by default but must be enabled at the VM level. Just change the nic to VMXNet 3 to take advantage of this feature
A Message Signaled Interrupt is a write from the device to a special address which causes an interrupt to be received by the CPU. The MSI capability was first specified in PCI 2.2 and was later enhanced in PCI 3.0 to allow each interrupt to be masked individually. The MSI-X capability was also introduced with PCI 3.0. It supports more interrupts – 26 per device than MSI and allows interrupts to be independently configured.
MSI, Message Signaled Interrupts, uses in-band pci memory space message to raise interrupt, instead of conventional out-band pci INTx pin. MSI-X is an extension to MSI, for supporting more vectors. MSI can support at most 32 vectors while MSI-X can support up to 2048. Using msi can lower interrupt latency, by giving every kind of interrupt its own vector/handler. When kernel see the message, it will directly vector to the interrupt service routine associated with the address/data. The address/data (vector) were allocated by system, while driver needs to register handler with the vector.
When Receive Side Scaling (RSS) is enabled, all of the receive data processing for a particular TCP connection is shared across multiple processors or processor cores. Without RSS all of the processing is performed by a single processor, resulting in inefficient system cache utilization
RSS is enabled on the Advanced tab of the adapter property sheet. If your adapter does not support RSS, or if your operating system does not support it, the RSS setting will not be displayed.
IPv6 TCP Segmentation Offloading significantly helps to reduce transmit processing performed by the vCPUs and improves both transmit efficiency and throughput. If the uplink NIC supports TSO6, the segmentation work will be offloaded to the network hardware; otherwise, software segmentation will be conducted inside the VMkernel before passing packets to the uplink. Therefore, TSO6 can be enabled for VMXNET3 whether or not the hardware NIC supports it
The VMXNET3 driver is NAPI‐compliant on Linux guests. NAPI is an interrupt mitigation mechanism that improves high‐speed networking performance on Linux by switching back and forth between interrupt mode and polling mode during packet receive. It is a proven technique to improve CPU efficiency and allows the guest to process higher packet loads
New API (also referred to as NAPI) is an interface to use interrupt mitigation techniques for networking devices in the Linux kernel. Such an approach is intended to reduce the overhead of packet receiving. The idea is to defer incoming message handling until there is a sufficient amount of them so that it is worth handling them all at once.
A straightforward method of implementing a network driver is to interrupt the kernel by issuing an interrupt request (IRQ) for each and every incoming packet. However, servicing IRQs is costly in terms of processor resources and time. Therefore the straightforward implementation can be very inefficient in high-speed networks, constantly interrupting the kernel with the thousands of packets per second. Overall performance of the system as well as network throughput can suffer as a result.
Polling is an alternative to interrupt-based processing. The kernel can periodically check for the arrival of incoming network packets without being interrupted, which eliminates the overhead of interrupt processing. Establishing an optimal polling frequency is important, however. Too frequent polling wastes CPU resources by repeatedly checking for incoming packets that have not yet arrived. On the other hand, polling too infrequently introduces latency by reducing system reactivity to incoming packets, and it may result in the loss of packets if the incoming packet buffer fills up before being processed.
As a compromise, the Linux kernel uses the interrupt-driven mode by default and only switches to polling mode when the flow of incoming packets exceeds a certain threshold, known as the “weight” of the network interface
VMXNET3 also supports Large Receive Offload (LRO) on Linux guests. However, in ESX 4.0 the VMkernel backend supports large receive packets only if the packets originate from another virtual machine running on the same host.
Security Options
Promiscuous Mode
Promiscuous mode eliminates any reception filtering that the virtual network adapter would perform so that the guest operating system receives all traffic observed on the wire. By default, the virtual network adapter cannot operate in promiscuous mode.
Although promiscuous mode can be useful for tracking network activity, it is an insecure mode of operation, because any adapter in promiscuous mode has access to the packets regardless of whether some of the packets are received only by a particular network adapter. This means that an administrator or root user within a virtual machine can potentially view traffic destined for other guest or host operating system
In some situations, you might have a legitimate reason to configure a standard switch to operate in promiscuous mode (for example, if you are running network intrusion detection software or a packet sniffer
MAC Address Changes
The setting for the MAC Address Changes option affects traffic that a virtual machine receives.
When the option is set to Accept, ESXi accepts requests to change the effective MAC address to other than the initial MAC address.
When the option is set to Reject, ESXi does not honor requests to change the effective MAC address to anything other than the initial MAC address, which protects the host against MAC impersonation. The port that the virtual adapter used to send the request is disabled and the virtual adapter does not receive any more frames until it changes the effective MAC address to match the initial MAC address. The guest operating system does not detect that the MAC address change was not honored.
The iSCSI initiator relies on being able to get MAC address changes from certain types of storage. If you are using ESXi iSCSI and have iSCSI storage, set the MAC Address Changes option to Accept.
In some situations, you might have a legitimate need for more than one adapter to have the same MAC address on a network—for example, if you are using Microsoft Network Load Balancing in unicast mode. When Microsoft Network Load Balancing is used in the standard multicast mode, adapters do not share MAC addresses.
MAC address changes settings affect traffic leaving a virtual machine. MAC address changes will occur if the sender is permitted to make them, even if standard switches or a receiving virtual machine does not permit MAC address chan
Forged Transmits
The setting for the Forged Transmits option affects traffic that is transmitted from a virtual machine.
When the option is set to Accept, ESXi does not compare source and effective MAC addresses.
To protect against MAC impersonation, you can set this option to Reject. If you do, the host compares the source MAC address being transmitted by the operating system with the effective MAC address for its adapter to see if they match. If the addresses do not match, ESXi drops the packet.
The guest operating system does not detect that its virtual network adapter cannot send packets by using the impersonated MAC address. The ESXi host intercepts any packets with impersonated addresses before they are delivered, and the guest operating system might assume that the packets are dropped
Note
This option is enabled by default, because it is occasionally needed to avoid software licensing problems. For example, if software on a physical machine is licensed to a specific MAC address, it will not work in a virtual machine because the VM’s MAC address is different. In this case, allowing forged transmits enables you to use the software by forging the VM’s MAC address.
However, allowing forged transmits poses a security risk.If an administrator has only authorized specific MAC addresses to enter the network, an intruder may be able to change his unauthorized MAC address to an authorized one
There are 3 types of Port Binding
Static Binding
When you connect a virtual machine to a port group configured with static binding, a port is immediately assigned and reserved for it, guaranteeing connectivity at all times. The port is disconnected only when the virtual machine is removed from the port group. You can connect a virtual machine to a static-binding port group only through vCenter Server.
Dynamic Binding
In a port group configured with dynamic binding, a port is assigned to a virtual machine only when the virtual machine is powered on and its NIC is in a connected state. The port is disconnected when the virtual machine is powered off or the virtual machine’s NIC is disconnected. Virtual machines connected to a port group configured with dynamic binding must be powered on and off through vCenter.
Dynamic binding can be used in environments where you have more virtual machines than available ports, but do not plan to have a greater number of virtual machines active than you have available ports. For example, if you have 300 virtual machines and 100 ports, but never have more than 90 virtual machines active at one time, dynamic binding would be appropriate for your port group.
Note: Dynamic binding is deprecated in ESXi 5.0.
Ephemeral Binding
In a port group configured with ephemeral binding, a port is created and assigned to a virtual machine by the host when the virtual machine is powered on and its NIC is in a connected state. The port is deleted when the virtual machine is powered off or the virtual machine’s NIC is disconnected.
You can assign a virtual machine to a distributed port group with ephemeral port binding on ESX/ESXi and vCenter, giving you the flexibility to manage virtual machine connections through the host when vCenter is down. Although only ephemeral binding allows you to modify virtual machine network connections when vCenter is down, network traffic is unaffected by vCenter failure regardless of port binding type.
Note: Ephemeral port groups should be used only for recovery purposes when you want to provision ports directly on host bypassing vCenter Server, not for any other case. This is true for several reasons:
The disadvantage is that if you configure ephemeral port binding your network will be less secure. Anybody who will gain host access can create rogue virtual machine and place it on the network or to move VMs between networks. The security hardening guide even recommends to lower the number of ports for each distributed portgroup so there are none unused.
AutoExpand (New Feature)
Note: vSphere 5.0 has introduced a new advanced option for static port binding called Auto Expand. This port group property allows a port group to expand automatically by a small predefined margin whenever the port group is about to run out of ports. In vSphere 5.1, the Auto Expand feature is enabled by default.
In vSphere 5.0 Auto Expand is disabled by default. To enable it, use the vSphere 5.0 SDK via the managed object browser (MOB):
<spec>
<configVersion>3</configVersion>
<autoExpand>true</autoExpand>
</spec>
Useful VMware Article
Useful Blog on why to use Static Port Binding on vDS Switches
http://blogs.vmware.com/vsphere/2012/05/why-use-static-port-binding-on-vds-.html
Private VLANs are used to solve VLAN ID limitations and waste of IP addresses for certain network setups.
PVLANs segregate VLANs even further than normal, they are basically VLANs inside of VLANs. The ports share a subnet, but can be prevented from communicating. They use different port types:
Promiscuous ports – These will be the “open ports” of the PVLANs, they can communicate with all other ports.
Community ports – These ports can communicate with other community ports and promiscuous ports.
Isolated ports – These can ONLY communicate with promiscuous ports.
There are different uses for PVLANs. They are used by service providers to allow customer security while sharing a single subnet. Another use could be for DMZ hosts in an enterprise environment. If one host is compromised its ability to inflict damage to the other hosts will be severely limited.
How vSphere implements private VLANs
Configuring and Assigning a Primary VLAN and Secondary VLAN
Diagram of Configuration courtesy of VMware
After the primary and secondary private VLANs are associated for the VDS, use the association to configure the VLAN policy for the distributed port group
Useful KB Article
Private VLAN (PVLAN) on vNetwork Distributed Switch – Concept Overview KB
Troubleshooting PVLANs
The terms Ingress source and Egress source are with respect to the VDS.
For example:
Ingress – When you want to monitor the traffic that is going out of a virtual machine towards the VDS, it is called Ingress Source traffic. The traffic seeks ingress to the VDS and hence the source is called Ingress.
Egress – When you want to monitor the traffic that is going out of the VDS towards the VM, it is called egress
Traffic Shaping concepts:
Average Bandwidth: Kbits/sec
Target traffic rate cap that the switch tries to enforce. Every time a client uses less than the defined Average Bandwidth, credit builds up.
Peak Bandwidth: Kbits/sec
Extra bandwidth available, above the Average Bandwidth, for a short burst. The availability of the burst depends on credit accumulated so far.
Burst Size: Kbytes
Amount of traffic that can be transmitted or received at Peak speed (Combining Peak Bandwidth and Burst Size you can calculate the maximum allowed time for the burst
Traffic Shaping on VSS and VDS
VSS
Traffic Shaping can be applied to a vNetwork Standard Switch port group or the entire vSwitch for outbound traffic only
VDS
Traffic Shaping can be applied to a vNetwork Distributed Switch dvPort or the entire dvPort Group for inbound and outbound traffic
