Troubleshooting iSCSI
ESXi systems include iSCSI technology to access remote storage using an IP network. You can use the vSphere Client, commands in the esxcli iscsi namespace, or the vicfg-iscsi command to configure both hardware and software iSCSI storage for your ESXi system.
Command Chart
VMware Link (Pages 53 onwards)
Storage Modules
The VMkernel is a high-performance operating system that runs directly on the ESXi host. The VMkernel manages most of the physical resources on the hardware, including memory, physical processors, storage, and networking controllers
To manage storage, VMkernel has a storage subsystem that supports several Host Bus Adapters (HBAs) including parallel SCSI, SAS, Fibre Channel, FCoE, and iSCSI. These HBAs connect a wide variety of active-active, active-passive, and ALUA storage arrays that are certified for use with the VMkernel.
The primary file system that the VMkernel uses is the VMware Virtual Machine File System (VMFS). VMFS is a cluster file system designed and optimized to support large files such as virtual disks and swap files. The VMkernel also supports the storage of virtual disks on NFS file systems.
The storage I/O path provides virtual machines with access to storage devices through device emulation. This device emulation allows a virtual machine to access files on a VMFS or NFS file system as if they were SCSI devices. The VMkernel provides storage virtualization functions such as the scheduling of I/O requests from multiple virtual machines and multipathing.
In addition, VMkernel offers several Storage APIs that enable storage partners to integrate and optimize their products for vSphere.
The following graphic illustrates the basics of the VMkernel core, with special attention to the storage stack. Storage‐related modules reside between the logical device I/O scheduler and the adapter I/O scheduler layerThe esxcli system module namespace allows you to view load and enable VMKernel modules.
To get an overview use this command:
Logs
Located in var/log
The logs you will want to look at for storage issues are likely to be
VMkernel deamon related log
Generic NMP messages, iSCSI and fibre channel messages, driver, device discovery, storage and networking devices
vCenter Server vpxa agent logs, including communication with vCenter Server and the Host Management hostd agent
Host management service logs, including virtual machine and host Task and Events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections
Generic storage messages, like disconnects. A summary of Warning and Alert log messages excerpted from the VMkernel logs.
If SIOC is enabled then all the logs regarding that will be here
Troubleshooting
Troubleshooting can often be frustrating and challenging, and knowing where to look and what to do is the key to quickly finding and resolving problems. You shouldn’t just look through log files when you are experiencing known problems, however. Often, many problems might not be that obvious, and the log files are a good place to look for signs of them happening. You should keep a list of all the log files handy so that you can quickly access them if needed and so not have to waste time when a problem is happening trying to remember their path and filenames. You might not know how to resolve or troubleshoot every problem you encounter, so be sure to rely on the resources available to you, including documentation, support forums, knowledge base, and VMware’s technical support. Being properly prepared to handle problems when they occur is one of the best troubleshooting skills that you can have.
What you can do
What is Port Mirroring?
Port mirroring is technology that duplicates network packets of a switch port to another port where it is monitored at the destination port. Most switch vendors implement Port Mirroring in their switches. Supported on vDS’s only and overcomes the issue of enabling Promiscuous Mode on a port where this port then sees all the traffic going through it
What is it used for?
Instructions for configuring Port Mirroring
Note: Both source and destination must be on the same ESXi Host
Traffic Direction can be thought of in terms of the vDS. Ingress is traffic from the VM to the vDS and Egress is traffic from the vDS to the VM
There are Caveats
Using vMA to troubleshoot DNS
Using Command Line Tools
Using ESXCLI to troubleshoot
Using the DCUI to check DNS information
Troubleshooting
The following are basic Service Console TCP/IP configuration requirements to check first
Checking the Logs
The VMkernel logs can be found in the locations below
Restarting the Management Network
To restart the management network on ESXi:
Using vCLI to troubleshoot the VMkernel
Note: With the release of 5.0, the majority of the legacy esxcfg-*/vicfg-* commands have been migrated over to esxcli. At some point, hopefully not in the distant future, esxcli will be parity complete and the esxcfg-*/vicfg-* commands will be completely deprecated and removed including the esxupdate/vihostupdate utilities.
Using vkernel Commands
Retrieving Network Port Information
Managing a VMKernel Port
Logs
Methods of accessing logs you need
vSphere Logs
The main logs you will need to look at are
Use the vSphere vCenter Performance Charts to monitor Memory usage and CPU usage of clusters, hosts, virtual machines, and vApps. Really useful statistics in blue
Host Memory
VM Memory
Host CPU
VM CPU
When problems occur in your virtual environment, you need to know where to look for clues to the cause and what to do to resolve them. Often, just trying to figure out the exact cause is the most difficult part, because virtual servers are more complicated than physical servers and there are more potential causes of problems. When you know where to look to find the cause of a problem, the process becomes a lot easier.
Note: By default, VMware ESXi logs do not persist upon a reboot. If a VMware ESXi host experiences an abrupt reboot due to reasons other than a VMkernel error, the logs do not persist and you do not have access to the logs prior to the reboot to determine the cause
Note: Many logfiles are time stamped using UTC – if you’re host isn’t configured to use UTC this may make correlating events and logs difficult
Types of Searches
The following types of server log entries are generated:
How to search through logs
grep is a unix command that allows you to search for a pattern in a list of files
Many times you need to view a constantly updating file. This is a common case with logs. People usually think that tail command is only used to view the last parts of a file, but it even provides you with the ability to view growing/changing files.
Please note that, by growing I mean files to whom data is being appended constantly. Using the -f option, tail lets us view the data that is being added to the file in real time.
Useful Links
http://www.vmworld.com/docs/DOC-3765
http://media.techtarget.com/searchServerVirtualization/downloads/0137008589_CH10.pdf
Scrolling through logs using DCUI
http://blogs.vmware.com/vsphere/2012/06/viewing-esxi-logs-from-the-dcui.html
