Archive for February 2013

Use esxcli to troubleshoot VMkernel storage module configurations

February 12, 2013 Objective 6 Advanced Troubleshooting No comments

Lightbulb

Storage Modules

The VMkernel is a high-performance operating system that runs directly on the ESXi host. The VMkernel manages most of the physical resources on the hardware, including memory, physical processors, storage, and networking controllers

To manage storage, VMkernel has a storage subsystem that supports several Host Bus Adapters (HBAs) including parallel SCSI, SAS, Fibre Channel, FCoE, and iSCSI. These HBAs connect a wide variety of active-active, active-passive, and ALUA storage arrays that are certified for use with the VMkernel.

The primary file system that the VMkernel uses is the VMware Virtual Machine File System (VMFS). VMFS is a cluster file system designed and optimized to support large files such as virtual disks and swap files. The VMkernel also supports the storage of virtual disks on NFS file systems.

The storage I/O path provides virtual machines with access to storage devices through device emulation. This device emulation allows a virtual machine to access files on a VMFS or NFS file system as if they were SCSI devices. The VMkernel provides storage virtualization functions such as the scheduling of I/O requests from multiple virtual machines and multipathing.

In addition, VMkernel offers several Storage APIs that enable storage partners to integrate and optimize their products for vSphere.

The following graphic illustrates the basics of the VMkernel core, with special attention to the storage stack. Storage‐related modules reside between the logical device I/O scheduler and the adapter I/O scheduler layerThe esxcli system module namespace allows you to view load and enable VMKernel modules.

Capture

To get an overview use this command:

  • esxcli system module

Module

  • esxcli system module list

module2

  •  esxcli system module parameters list –module ModuleName

Capture

 

Identify Logs used to troubleshoot storage issues

February 12, 2013 Objective 6 Advanced Troubleshooting No comments

images

Logs

Located in var/log

Log

The logs you will want to look at for storage issues are likely to be

  • /var/log/vmkeventd.log

VMkernel deamon related log

  • /var/log/vmkernel.log

Generic NMP messages, iSCSI and fibre channel messages, driver, device discovery, storage and networking devices

  • /var/log/vpxa.log

vCenter Server vpxa agent logs, including communication with vCenter Server and the Host Management hostd agent

  • /var/log/hostd.log

Host management service logs, including virtual  machine and host Task and Events, communication with the vSphere Client  and vCenter Server vpxa agent, and SDK connections

  • /var/log/vmkwarning.log

Generic storage messages, like disconnects. A summary of Warning and Alert log messages excerpted from the VMkernel logs.

  • /var/log/storagerm

If SIOC is enabled then all the logs regarding that will be here

  • vCenter logs

Analyse troubleshooting data to see if the problem lies in the Virtual or the Physical layer

February 12, 2013 Objective 6 Advanced Troubleshooting No comments

images

Troubleshooting

Troubleshooting can often be frustrating and challenging, and knowing where to look and what to do is the key to quickly finding and resolving problems. You shouldn’t just look through log files when you are experiencing known problems, however. Often, many problems might not be that obvious, and the log files are a good place to look for signs of them happening. You should keep a list of all the log files handy so that you can quickly access them if needed and so not have to waste time when a problem is happening trying to remember their path and filenames. You might not know how to resolve or troubleshoot every problem you encounter, so be sure to rely on the resources available to you, including documentation, support forums, knowledge base, and VMware’s technical support. Being properly prepared to handle problems when they occur is one of the best troubleshooting skills that you can have.

What you can do

  • Check Monitoring Systems if you have them. SCOM, Nagios etc. Some companies have real-time screens showing monitoring
  • Check with your Network Team as they will more than likely be alerted to physical problems faster than you
  • Can you isolate the problem to a VM, Host, Switch or router or is the issue affecting the whole network
  • Ensure that the Port Group name(s) associated with the virtual machine’s network adapter(s) exists in your vSwitch or Virtual Distributed Switch and is/are spelt correctly.
  • Check any warning Triangles or exclamation marks on the standard or distributed switches
  • Verify the virtual network adapter is present and connected for all VMkernel ports
  • Verify that the networking within the virtual machine’s guest operating system is correct
  • Verify that the vSwitch has enough ports for the virtual machine
  • Ensure the physical switch ports are configured as port-channel
  • Shut down all but one of the physical ports the NICs are connected to, and toggle this between all the ports by keeping only one port connected at a time. Take note of the port/NIC combination where the virtual machines lose network connectivity.
  • Check Logs

Configure and Administer Port Mirroring

February 12, 2013 Objective 6 Advanced Troubleshooting No comments

images

What is Port Mirroring?

Port mirroring is technology that duplicates network packets of a switch port to another port where it is monitored at the destination port. Most switch vendors implement Port Mirroring in their switches. Supported on vDS’s only and overcomes the issue of enabling Promiscuous Mode on a port where this port then sees all the traffic going through it

What is it used for?

  • Troubleshooting
  • Input for network analysis
  • Intrusion Detection systems

Instructions for configuring Port Mirroring

Note: Both source and destination must be on the same ESXi Host

  • Log into vCenter
  • Go to Networking
  • Right click your vDS and select Edit Settings
  • Click the Port Mirroring tab

Mirror1

  • Click Add

Mirror2

  • Put in a name
  • Put in a Description
  • If you do not select Allow normal I/O on destination ports then mirrored traffic is allowed out on destination ports but no traffic is allowed in
  • If you select Encapsulate VLAN then this VLAN ID encapsulates all frames at the destination port. If packets already have a VLAN then the VLAN is replaced with VLAN ID specified here
  • If you select Preserve Original VLAN then the original VLAN is kept and a packet is added with another VLAN tag specified
  • If you select Mirrored Packet Length then this puts a limit on the size of the mirrored frames. Increasing this length increases the time taken to process packets. Used for capturing protocols of a certain length
  • Click Next

Mirror3

  • Traffic direction can be Ingress/Egress or Both

Traffic Direction can be thought of in terms of the vDS. Ingress is traffic from the VM to the vDS and Egress is traffic from the vDS to the VM

  • As an example I chose Port 5 and Ingress/Egress

Mirror4

  • On the destination page you can choose Port or Uplink for a destination and choose more than one of either

There are Caveats

  1. In a session, a port cannot be both a Source and a Destination
  2. A port cannot be a destination for more than one session
  3. A promiscuous port cannot be an Egress source destination
  4. An egress source cannot be a destination of any session to avoid cycles of mirrored paths
  • As an example I have chosen dvUplink 1

Mirroring5

  • Click Enable this Port Mirroring Session. By default it is disabled

Mirroring6

  • Click Finish and check the overview

Mirroring7

Troubleshoot DNS and routing related issues

February 12, 2013 Objective 6 Advanced Troubleshooting No comments

dns

Using vMA to troubleshoot DNS

dns8

Using Command Line Tools

  • ping
  • netstat
  • ipconfig
  • nslookup
  • ipconfig /flushdns

Using ESXCLI to troubleshoot

  • SSH into the host and run the following commands

DNS11

  • Example below

DNS5

Using the DCUI to check DNS information

  • SSH into a a host
  • Select Configure Management Network
  • Select DNS information to see what the details are

DNS2

  • Press Enter to adjust the config

DNS3

  • You can also check the DNS Suffixes

DNS4

Troubleshoot vmkernel related network configuration issues

February 12, 2013 Objective 6 Advanced Troubleshooting No comments

LogIcon

Troubleshooting

The following are basic Service Console TCP/IP configuration requirements to check first

  • The ESX host has working physical network adapters connecting to physical network switches appliances
  • Proper/functional Ethernet Cable
  • Gateway appliance that can be either a router or switch appliance is working
  • Establishing method of VLAN Tagging configuration (VST,EST, or VGT)
  • Proper IP address, network sub mask, and gateway configuration
  • Successful pinging of all relevant network addresses associated with the VMKernel
  • You can only have 1 Management Gateway

Checking the Logs

The VMkernel logs can be found in the locations below

vmkernel

Restarting the Management Network

To restart the management network on ESXi:

  1. Connect to the console of your ESXi host.
  2. Press F2 to customize the system.
  3. Login as root
  4. Use the Up/Down arrows to navigate to Restart Management Network

network

  • Click Enter to restart

Using vCLI to troubleshoot the VMkernel

Note: With the release of 5.0, the majority of the legacy esxcfg-*/vicfg-* commands have been migrated over to esxcli. At some point, hopefully not in the distant future, esxcli will be parity complete and the esxcfg-*/vicfg-* commands will be completely deprecated and removed including the esxupdate/vihostupdate utilities.

  • esxcfg-nics
  • vicfg-nics
  • esxcfg-route
  • vicfg-route
  • esxcfg-vmknic
  • vicfg-vmknic
  • esxcfg-vswitch
  • vicfg-vswitch
  • esxcli network nic
  • esxcli network interface
  • esxcli network vswitch
  • esxcli network ip

Using vkernel Commands

Retrieving Network Port Information

vmkernel1

Managing a VMKernel Port

vmkernel2