Archive for February 2012

Dilbert

RESXTOP and ESXTOP

ESXTOP and RESXTOP

Are used to analyze real-time performance data from an individual ESX or ESXi server.

The fundamental difference between resxtop and esxtop is that you can use resxtop remotely, whereas you can start extop only through the ESXi Shell of a local ESXi host.

You can start either utility in one of three modes:

  • Interactive (default)
  • Batch
  • Replay

Running ESXTOP/RESXTOP

Type esxtop/resxtop into one of the following consoles

  • Putty
  • vMA (vSphere Management Assistant) virtual appliance.
  • vCLI
  • Power CLI

esxtop59

When running RESXTOP you will have to specify the ESX or ESXi server hostname, username, and password, as you see below

What you will see first

  • Global Statistics

  • Up time

The elapsed time since the server has been powered on.

  • Number of worlds

The total number of worlds on ESX(i) Server (Like Processes)

  • CPU load average

The arithmetic mean of CPU loads in 1 minute, 5 minutes, and 15 minutes, based on 6-second samples. CPU load accounts the run time and ready time for all the groups on the host.

A load average of 0.50 means that the physical CPUs on the ESXi system
are half utilized.

A load average of 1.00 means that the physical CPUs on the ESXi system
are fully utilized.

A load average of 2.00 means that means that the physical CPUs on the ESXi system
are doubly utilized and the ESXi system might need twice as many physical CPUs as are currently available.

Accessing the 8 different displays

You’ll find that ESXTOP/RESXTOP has 8 different “displays” that show CPU, interrupt, memory, network, disk adapter, disk interface, disk VM, and power management. These are accessed by typing the letters below

Commands by letter

esxtop

Running esxtop in Batch Mode

  • Log into the host using whichever console you feel comfortable with. E.g. Putty
  • Type esxtop
  • Type V (Capital V) to just show the VMs

esxtop1

  • By default you are on the CPU Screen. If you then type f (lower case) you can toggle between what CPU fields to view. Type the letter to activate the relevant field

esxtop2

  • Press any key to return to the main screen and now press m (lower case) for Memory and then press f to see the fields. Type the letter to activate the relevant field

esxtop3

  • Press any key to return to the main screen then type n (lower case) for Network and type f to see the fields. Type the letter to activate the relevant field

esxtop4

  • Press any key to return to the main screen and now press v (lower case) for VM Disk and then press f to see the fields. Type the letter to activate the relevant field.

esxtop5

  • Now you have selected all your fields, you need to press W (Capital W) to save your settings then press Enter

esxtop6

  • You should see the following screen flash up quickly

esxtop7

  • Type q to quit and go back to your normal command line

esxtop8

  • You now need to run it in batch mode and save the results to a .csv file:
  • Type esxtop -b -a -d 2 -n 1800 > /tmp/esxtopcapture.csv

Where “-b” stands for batch mode, “-d 2″ is a delay of 2 seconds and “-n 1800″ are 3600 iterations. In this specific case esxtop will log all metrics for 1 Hour. If you want to record all metrics make sure to add “-a” to your string.

esxtopbatch

Analysing Data

You can use multiple tools to analyze the captured data. Underlined are links to the software

  1. VisualEsxtop
  2. perfmon
  3. excel
  4. esxplot

VisualEsxtop

VisualEsxtop is an enhanced version of resxtop and esxtop. VisualEsxtop can connect to VMware vCenter Server or ESX hosts, and display ESX server stats with a better user interface and more advanced features.

Features

  1. Live connection to ESX host or vCenter Server
  2. Flexible way of batch output
  3. Load batch output and replay them
  4. Multiple windows to display different data at the same time
  5. Line chart for selected performance counters
  6. Flexible counter selection and filtering
  7. Embedded tooltip for counter description
  8. Color coding for important counters

Instructions

  • Once it is download you must make sure that Java is installed or VisualEsxtop will not run. We have JRE 6 Update 29 installed. You can check this by running cmd.exe and typing java

java

  • If you don’t have Java installed correctly then you will get the following message

esxtop60

  • For Windows, navigate to your VisualEsxtop folder and run the VisualEsxtop.bat file

esxtop56

  • It should open the below application
  • Click File > Load Batch Output and open your CSV output file from running ESXTOP in Batch Mode

esxtop57

  • You can then filter as well

esxtop58

https://labs.vmware.com/flings/visualesxtop

http://blogs.vmware.com/kb/2013/09/using-visualesxtop-to-troubleshoot-performance-issues-in-vsphere-2.html

Perfmon

  • On your Windows Server, click Start > Run > Type perfmon
  • Right click on the graph and select “Properties”.

esxtop50

  • Select the “Source” tab.
  • Select the “Log files:” radio button from the “Data source” section.
  • Click the “Add” button.

esxtop51

  • Select the CSV file created by esxtop and click “OK”.

esxtop52

  • Click the “Apply” button.
  • Optionally: reduce the range of time over which the data will be displayed by using the sliders under the “Time Range” button.
  • Select the “Data” tab.
  • Remove all Counters.

esxtop53

  • Click “Add” and select appropriate counters. When you click on some of the counters, you can select the instance or VM/Machine you want to monitor directly
  • Click Add

esxtop54

  • Click “OK”
  • Click “OK”
  • You should now see the graph of values

esxtop55

Using ESXPLOT

Please see the below link for instructions

  1. Run: esxplot
  2. Click File -> Import -> Dataset
  3. Select file and click “Open”
  4. Double click host name and click on metric

http://www.electricmonk.org.uk/2012/09/05/esxplot/

Using MS Excel

Within Excel it is also possible to import the data as a CSV. You need to be careful of the size of the file though as the amount of captured data is sometimes quite large so you might want to limit it by first importing it into perfmon and then select the correct timeframe and counters and export this to a CSV. You can import the CSV as per below instructions

  1. Run: Excel
  2. Click on “Data”
  3. Click “Import External Data” and click “Import Data”
  4. Select “Text files” as “Files of Type”
  5. Select file and click “Open”
  6. Make sure “Delimited” is selected and click “Next”
  7. Deselect “Tab” and select “Comma”
  8. Click “Next” and “Finish

Looking at esxtop values and results (Realtime)

General CPU Statistics

First visible CPU Statistics

CPUesxtop

Optional Fields for CPU Performance Monitoring

General Memory Statistics

 First Visible Memory Statistics

esxtop5

Optional Fields for Memory Performance Monitoring

esxtopmem5

General Disk Statistics

General Network Statistics

esxtopnetwork

Running ESXTOP in Replay Mode

In replay mode, esxtop replays resource utilization statistics collected using vm-support.

After you prepare for replay mode, you can use esxtop in this mode.

In replay mode, esxtop accepts the same set of interactive commands as in interactive mode and runs until no more snapshots are collected by vm-support to be read or until the requested number of iterations are completed.

To run in replay mode, you must prepare for replay mode.

  • Run vm-support in snapshot mode on the ESX service console
  • Type vm-support -S -d duration -I interval
  • -S = Snapshot mode, prompts for the delay between updates, in seconds
  • -R = Path to the vm-support collected snapshot’s directory
  • Unzip and untar the resulting tar file so that esxtop can use it in replay mode.
  • tar -xf /root/esx*.tgz
  • Now run the following
  • esxtop -R root/vm-support*

http://www.vmwarearena.com/2012/08/esxtop-replay-mode.html

5 of the best posts for analysing results and statistics

http://www.yellow-bricks.com/esxtop/

http://communities.vmware.com/docs/DOC-9279

http://www.vmware.com/pdf/esx2_using_esxtop.pdf

http://simongreaves.co.uk/blog/esxtop-guide

http://communities.vmware.com/docs/DOC-5240

Analysing CPU/RAM/Network/Performance

http://communities.vmware.com/docs/DOC-3930

VMFS Block Sizes

VMFS-3

VMFS-3 Block Sizes are assigned to Datastores when created and determine the maximum file size that can be stored.

Here are the VMFS-3 Block sizes and corresponding file sizes

1MB = 256GB

2MB = 512GB

4MB = 1TB

8MB = 2TB

Once set this cannot be changed.

Sub-block allocation size

64KB

VMFS-5

Only hosts running ESXi 5 or later support VMFS-5. Hosts running ESXi 4.x will not be able to see or access VMFS-5 Datastores

VMFS-5 Datastores can be up to 64TB on a single extent

Datastores built on extents are still limited to 64TB as well.

Here is the VMFS-5 Block sizes and corresponding file size

1MB = 2TB

Sub-block allocation size

8KB

Adding Extents to Disks

vSphere 4 uses VMFS version 3 (VMFS-3) and vSphere 5 continues to provide support for VMFS-3.

VMFS-3 supports datastores with a maximum size of 2TB -512Bytes) This size stems from the partition management features which use MBR (Master Boot Record) instead of GPT (GUID Partition Tables)

So what happens if you need a larger datastore on a VMFS-3 datastore?

Fortunately VMFS-3 has the ability to reside on one or more partitions which are also known as extents.

VMFS-3 supports up to 32 extents in a single VMFS-3 Datastore with a maximum size of 64TB

For a VMFS volume, there is a rule of one VMFS per LUN. SCSI-2 Reservation locking in ESX locks the entire LUN and not a specific partition. Therefore, the best practice is to have one LUN per VMFS only (exception being local storage). When using extents, gather multiple LUNs under one logical VMFS and not multiple partitions per LUN.

Note: Ensure a complete rescan is done on all ESX hosts after adding an extent to a VMFS volume on one ESX host. Otherwise, the same extent might be inadvertently added by another node in the cluster, which could potentially cause loss of data. Best practice is to add extents to an existing VMFS volume from a single node and then rescan the storage resources from all ESX hosts capable of accessing that shared storage resource.

How to Extend a VMFS Datastore
How to create a single VMFS datastore and increase the size by adding an extent.

First of all make sure you have created a new LUN on your storage or SAN and rescanned in VMware so all hosts can see the new storage

1. Select your datastore in the Datastore inventory.
2. Click Configurations tab.
3. In Datastore Details pane, click Properties link.
4. Click Increase on the Properties dialog box.
5. Do the following when prompted by the Increase Datastore Capacity wizard (field):
a. Select your LUN ID. (Extent Device)
b. Review current disk layout. (Current Disk Layout)
c. Leave the Maximum capacity check box selected. (Capacity)
d. Click Finish. (Ready to Complete)

Errors

When going through the Add Extent wizard on a datastore residing on a LUN that has already been extended at the hardware/array level you may receive a warning message indicating that there may be data loss.

The warning message is as follows:

Warning: The current disk layout will be destroyed.  All file systems and data will be lost permanently

This error message is in relation to the area of the disk that you are extending to, not the area of the disk you are extending from.

If you look in Target Identifier, you see the vmhba:W:X:Y:Z where the extent is to be placed. If your original disk was vmhba1:0:0:1 , that extent is created on vmhba:1:0:0:2