Archive for February 2012

ESX, ESXi and VM Log Locations

February 7, 2012 Logs, VMware No comments
Logs can help you find out what happened if commands do not have the desired results. On ESXi 5.0 systems, find all logs in the /var/log directory. Some of the items in that directory are symbolic links from the /var/run/log directory.

On ESXi 4.1 systems, you can find the following logs.

Location of Log Files for VMware Products

http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup=true&languageId=&externalID=1021806

Host and VM Log Locations

Location

Component

 
/var/log/vmware/vmware-serverd.log ESX Server 2.x service log
var/log/vmware/hostd.log Host management service logs, including virtual machine and host Tasks and Events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections.
/var/log/vmware/vpx/vpxa.log vSphere client agent log
After you reboot your machine, files /root/vmkernel-log.<date> and /root/vmkernel-core.<date> are present. Virtual machine kernel core file
/var/log/messages Messages from the Service Console Linux kernel, including service startup and shutdown.
/var/log/vmksummary Summary of ESX host startup and shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service console resource consumption
/var/log/vmkernel VMkernel messages, alerts, and availability report, Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup.
/var/log/vmkwarning Summary of Warning and Alert log messages excerpted from the VMkernel logs.
vmware.logIn the same directory as the VMX file for the virtual machine Virtual machine log file
/.vmx Located on a datastore associated with the managed host. Use the virtual machine summary page in the vSphere Client to determine the datastore on which this file is located. Virtual machine configuration file
var/log/boot-logs/sysboot.log Early VMkernel startup, module loading, and host initialization.
/var/log/vmkiscsid.log Software iSCSI Client logs.

vCenter Log Locations

/var/log/vmware/vpx/vpxa.log vCenter Server vpxa agent logs, including communication with vCenter Server and the Host Management hostd agent.
/var/log/vmware/fdm/* VMware High Availability Logs for vCenter Server 5
/var/log/vmware/aam/* VMware High Availability Logs for vCenter Server 4

VMware vCLI for vSphere 5

February 7, 2012 Command Line, Objective 1 Storage, vCLI, vCLI, VMware No comments

VMware vCLI Instructions

The vSphere Command-Line Interface (vSphere CLI) command set allows you to run common system administration commands against ESX/ESXi systems from any machine with network access to those systems. You can also run most vSphere CLI commands against a vCenter Server system and target any ESX/ESXi system that vCenter Server system manages. vSphere CLI includes the ESXCLI command set, vicfg- commands, and some other commands.

  • Download and Install vCLI
  • http://www.vmware.com/support/developer/vcli/
  • Right click on the vCLI icon and select Run as Administrator
  • Navigate to c:\Program Files (x86)\VMware\VMware vSphere CLI\bin
  • You will see the below vCLI commands (Note the .pl extension on the end)

vcli2

  • An example of running a command would be as per below with vifs.pl
  • Type vifs.pl –help to see the associated switches for this command

vifs

  • Try typing vifs.pl –server esxihostserver –listdc

vifs3

  • Another example of this command as per below screenprints shows how you can create a folder on a Datstore
  • vifs.pl –server esxiserver –mkdir “[Datastore] test”

vifs4

Documentation

vSphere Command-Line Interface Documentation

Getting Started with vSphere Command-Line Interfaces

vSphere Command‐Line Interface Concepts and Examples

vSphere Command‐Line Interface Reference

YouTube Video

You Tube Video

Running Commands on Windows.

In order to stop having to put in credentials everytime you run a command you can can the following

save_session.pl –server esxiserver01 –username usera –password passswordxyz –savesessionfile c:\temp\vclisessionfile

The next time you run a command you can type the following

esxcli –server MyESXiHost –sessionfile c:\temp\vclisessionfile storage core filesystem list

vCLI Poster

http://blogs.vmware.com/tp/files/vmware-management-with-vcli-5.0.pdf

VMware Visio Action Pack

February 7, 2012 IT, VMware No comments

Overview

http://xtravirt.com/visio-action-pack-re-released-free-member-download

This excellent Visio icon pack for VMware & virtualization has been re-released as a free member download. It contains over 70 unique icons together with a user guide and sample diagram templates.

It is designed for Windows Operating Systems and has been designed to run on Microsoft Visio 2003 and Microsoft Visio 2007, although also runs on Mac Omnigraffle. They are not compatible with Microsoft Visio 2000 or 2002

vSphere 4 Client RDP Plug-in

February 7, 2012 VMware No comments

There is a really useful and informative website called http://xtravirt.com which provides lots of free tools for virtualisation admins, one of these being the RDP Plugin which allows you to right click on any VM and Remote Desktop to this machine from the vClient

The Xtravirt vSphere RDP Plug-in provides integration of the Windows Remote Desktop tool with the VMware vSphere Client.

Utilising Remote Desktop to connect virtual machines provides a better user experience compared to the built-in VMware console as well as performing better across WAN connection

System Requirements

1. VMware vCenter Server version 4

2. Version 4.0 of the VMware vSphere Client

3. Microsoft .NET Framework 3.5

4. Version 7.0 or greater of the Microsoft Remote Desktop Connection client

Download File Contents

1. vSphere RDP Plug-in installation software

2. Install and User Guide

Screenprints

 

Port Group Security

February 7, 2012 Networking, Objective 2 Networking, Objective 7 Secure a vSphere environment, VMware No comments

Security Options

portsecurity

Promiscuous Mode

Promiscuous mode eliminates any reception filtering that the virtual network adapter would perform so that the guest operating system receives all traffic observed on the wire. By default, the virtual network adapter cannot operate in promiscuous mode.

Although promiscuous mode can be useful for tracking network activity, it is an insecure mode of operation, because any adapter in promiscuous mode has access to the packets regardless of whether some of the packets are received only by a particular network adapter. This means that an administrator or root user within a virtual machine can potentially view traffic destined for other guest or host operating system

Note

In some situations, you might have a legitimate reason to configure a standard switch to operate in promiscuous mode (for example, if you are running network intrusion detection software or a packet sniffer

MAC Address Changes

The setting for the MAC Address Changes option affects traffic that a virtual machine receives.

When the option is set to Accept, ESXi accepts requests to change the effective MAC address to other than the initial MAC address.

When the option is set to Reject, ESXi does not honor requests to change the effective MAC address to anything other than the initial MAC address, which protects the host against MAC impersonation. The port that the virtual adapter used to send the request is disabled and the virtual adapter does not receive any more frames until it changes the effective MAC address to match the initial MAC address. The guest operating system does not detect that the MAC address change was not honored.

Note

The iSCSI initiator relies on being able to get MAC address changes from certain types of storage. If you are using ESXi iSCSI and have iSCSI storage, set the MAC Address Changes option to Accept.

In some situations, you might have a legitimate need for more than one adapter to have the same MAC address on a network—for example, if you are using Microsoft Network Load Balancing in unicast mode. When Microsoft Network Load Balancing is used in the standard multicast mode, adapters do not share MAC addresses.

MAC address changes settings affect traffic leaving a virtual machine. MAC address changes will occur if the sender is permitted to make them, even if standard switches or a receiving virtual machine does not permit MAC address chan

Forged Transmits

The setting for the Forged Transmits option affects traffic that is transmitted from a virtual machine.

When the option is set to Accept, ESXi does not compare source and effective MAC addresses.

To protect against MAC impersonation, you can set this option to Reject. If you do, the host compares the source MAC address being transmitted by the operating system with the effective MAC address for its adapter to see if they match. If the addresses do not match, ESXi drops the packet.

The guest operating system does not detect that its virtual network adapter cannot send packets by using the impersonated MAC address. The ESXi host intercepts any packets with impersonated addresses before they are delivered, and the guest operating system might assume that the packets are dropped

Note

This option is enabled by default, because it is occasionally needed to avoid software licensing problems. For example, if software on a physical machine is licensed to a specific MAC address, it will not work in a virtual machine because the VM’s MAC address is different. In this case, allowing forged transmits enables you to use the software by forging the VM’s MAC address.

However, allowing forged transmits poses a security risk.If an administrator has only authorized specific MAC addresses to enter the network, an intruder may be able to change his unauthorized MAC address to an authorized one

Dilbert

February 6, 2012 IT No comments

RESXTOP and ESXTOP

February 6, 2012 ESXTOP + RESXTOP, Objective 3 Tuning and Optimisation, VMware 2 comments

ESXTOP and RESXTOP

Are used to analyze real-time performance data from an individual ESX or ESXi server.

The fundamental difference between resxtop and esxtop is that you can use resxtop remotely, whereas you can start extop only through the ESXi Shell of a local ESXi host.

You can start either utility in one of three modes:

  • Interactive (default)
  • Batch
  • Replay

Running ESXTOP/RESXTOP

Type esxtop/resxtop into one of the following consoles

  • Putty
  • vMA (vSphere Management Assistant) virtual appliance.
  • vCLI
  • Power CLI

esxtop59

When running RESXTOP you will have to specify the ESX or ESXi server hostname, username, and password, as you see below

What you will see first

  • Global Statistics

  • Up time

The elapsed time since the server has been powered on.

  • Number of worlds

The total number of worlds on ESX(i) Server (Like Processes)

  • CPU load average

The arithmetic mean of CPU loads in 1 minute, 5 minutes, and 15 minutes, based on 6-second samples. CPU load accounts the run time and ready time for all the groups on the host.

A load average of 0.50 means that the physical CPUs on the ESXi system
are half utilized.

A load average of 1.00 means that the physical CPUs on the ESXi system
are fully utilized.

A load average of 2.00 means that means that the physical CPUs on the ESXi system
are doubly utilized and the ESXi system might need twice as many physical CPUs as are currently available.

Accessing the 8 different displays

You’ll find that ESXTOP/RESXTOP has 8 different “displays” that show CPU, interrupt, memory, network, disk adapter, disk interface, disk VM, and power management. These are accessed by typing the letters below

Commands by letter

esxtop

Running esxtop in Batch Mode

  • Log into the host using whichever console you feel comfortable with. E.g. Putty
  • Type esxtop
  • Type V (Capital V) to just show the VMs

esxtop1

  • By default you are on the CPU Screen. If you then type f (lower case) you can toggle between what CPU fields to view. Type the letter to activate the relevant field

esxtop2

  • Press any key to return to the main screen and now press m (lower case) for Memory and then press f to see the fields. Type the letter to activate the relevant field

esxtop3

  • Press any key to return to the main screen then type n (lower case) for Network and type f to see the fields. Type the letter to activate the relevant field

esxtop4

  • Press any key to return to the main screen and now press v (lower case) for VM Disk and then press f to see the fields. Type the letter to activate the relevant field.

esxtop5

  • Now you have selected all your fields, you need to press W (Capital W) to save your settings then press Enter

esxtop6

  • You should see the following screen flash up quickly

esxtop7

  • Type q to quit and go back to your normal command line

esxtop8

  • You now need to run it in batch mode and save the results to a .csv file:
  • Type esxtop -b -a -d 2 -n 1800 > /tmp/esxtopcapture.csv

Where “-b” stands for batch mode, “-d 2″ is a delay of 2 seconds and “-n 1800″ are 3600 iterations. In this specific case esxtop will log all metrics for 1 Hour. If you want to record all metrics make sure to add “-a” to your string.

esxtopbatch

Analysing Data

You can use multiple tools to analyze the captured data. Underlined are links to the software

  1. VisualEsxtop
  2. perfmon
  3. excel
  4. esxplot

VisualEsxtop

VisualEsxtop is an enhanced version of resxtop and esxtop. VisualEsxtop can connect to VMware vCenter Server or ESX hosts, and display ESX server stats with a better user interface and more advanced features.

Features

  1. Live connection to ESX host or vCenter Server
  2. Flexible way of batch output
  3. Load batch output and replay them
  4. Multiple windows to display different data at the same time
  5. Line chart for selected performance counters
  6. Flexible counter selection and filtering
  7. Embedded tooltip for counter description
  8. Color coding for important counters

Instructions

  • Once it is download you must make sure that Java is installed or VisualEsxtop will not run. We have JRE 6 Update 29 installed. You can check this by running cmd.exe and typing java

java

  • If you don’t have Java installed correctly then you will get the following message

esxtop60

  • For Windows, navigate to your VisualEsxtop folder and run the VisualEsxtop.bat file

esxtop56

  • It should open the below application
  • Click File > Load Batch Output and open your CSV output file from running ESXTOP in Batch Mode

esxtop57

  • You can then filter as well

esxtop58

https://labs.vmware.com/flings/visualesxtop

http://blogs.vmware.com/kb/2013/09/using-visualesxtop-to-troubleshoot-performance-issues-in-vsphere-2.html

Perfmon

  • On your Windows Server, click Start > Run > Type perfmon
  • Right click on the graph and select “Properties”.

esxtop50

  • Select the “Source” tab.
  • Select the “Log files:” radio button from the “Data source” section.
  • Click the “Add” button.

esxtop51

  • Select the CSV file created by esxtop and click “OK”.

esxtop52

  • Click the “Apply” button.
  • Optionally: reduce the range of time over which the data will be displayed by using the sliders under the “Time Range” button.
  • Select the “Data” tab.
  • Remove all Counters.

esxtop53

  • Click “Add” and select appropriate counters. When you click on some of the counters, you can select the instance or VM/Machine you want to monitor directly
  • Click Add

esxtop54

  • Click “OK”
  • Click “OK”
  • You should now see the graph of values

esxtop55

Using ESXPLOT

Please see the below link for instructions

  1. Run: esxplot
  2. Click File -> Import -> Dataset
  3. Select file and click “Open”
  4. Double click host name and click on metric

http://www.electricmonk.org.uk/2012/09/05/esxplot/

Using MS Excel

Within Excel it is also possible to import the data as a CSV. You need to be careful of the size of the file though as the amount of captured data is sometimes quite large so you might want to limit it by first importing it into perfmon and then select the correct timeframe and counters and export this to a CSV. You can import the CSV as per below instructions

  1. Run: Excel
  2. Click on “Data”
  3. Click “Import External Data” and click “Import Data”
  4. Select “Text files” as “Files of Type”
  5. Select file and click “Open”
  6. Make sure “Delimited” is selected and click “Next”
  7. Deselect “Tab” and select “Comma”
  8. Click “Next” and “Finish

Looking at esxtop values and results (Realtime)

General CPU Statistics

First visible CPU Statistics

CPUesxtop

Optional Fields for CPU Performance Monitoring

General Memory Statistics

 First Visible Memory Statistics

esxtop5

Optional Fields for Memory Performance Monitoring

esxtopmem5

General Disk Statistics

General Network Statistics

esxtopnetwork

Running ESXTOP in Replay Mode

In replay mode, esxtop replays resource utilization statistics collected using vm-support.

After you prepare for replay mode, you can use esxtop in this mode.

In replay mode, esxtop accepts the same set of interactive commands as in interactive mode and runs until no more snapshots are collected by vm-support to be read or until the requested number of iterations are completed.

To run in replay mode, you must prepare for replay mode.

  • Run vm-support in snapshot mode on the ESX service console
  • Type vm-support -S -d duration -I interval
  • -S = Snapshot mode, prompts for the delay between updates, in seconds
  • -R = Path to the vm-support collected snapshot’s directory
  • Unzip and untar the resulting tar file so that esxtop can use it in replay mode.
  • tar -xf /root/esx*.tgz
  • Now run the following
  • esxtop -R root/vm-support*

http://www.vmwarearena.com/2012/08/esxtop-replay-mode.html

5 of the best posts for analysing results and statistics

http://www.yellow-bricks.com/esxtop/

http://communities.vmware.com/docs/DOC-9279

http://www.vmware.com/pdf/esx2_using_esxtop.pdf

http://simongreaves.co.uk/blog/esxtop-guide

http://communities.vmware.com/docs/DOC-5240

Analysing CPU/RAM/Network/Performance

http://communities.vmware.com/docs/DOC-3930

VMFS Block Sizes

February 6, 2012 Storage, VMware No comments

VMFS-3

VMFS-3 Block Sizes are assigned to Datastores when created and determine the maximum file size that can be stored.

Here are the VMFS-3 Block sizes and corresponding file sizes

1MB = 256GB

2MB = 512GB

4MB = 1TB

8MB = 2TB

Once set this cannot be changed.

Sub-block allocation size

64KB

VMFS-5

Only hosts running ESXi 5 or later support VMFS-5. Hosts running ESXi 4.x will not be able to see or access VMFS-5 Datastores

VMFS-5 Datastores can be up to 64TB on a single extent

Datastores built on extents are still limited to 64TB as well.

Here is the VMFS-5 Block sizes and corresponding file size

1MB = 2TB

Sub-block allocation size

8KB

Adding Extents to Disks

February 6, 2012 Storage, VMware No comments

vSphere 4 uses VMFS version 3 (VMFS-3) and vSphere 5 continues to provide support for VMFS-3.

VMFS-3 supports datastores with a maximum size of 2TB -512Bytes) This size stems from the partition management features which use MBR (Master Boot Record) instead of GPT (GUID Partition Tables)

So what happens if you need a larger datastore on a VMFS-3 datastore?

Fortunately VMFS-3 has the ability to reside on one or more partitions which are also known as extents.

VMFS-3 supports up to 32 extents in a single VMFS-3 Datastore with a maximum size of 64TB

For a VMFS volume, there is a rule of one VMFS per LUN. SCSI-2 Reservation locking in ESX locks the entire LUN and not a specific partition. Therefore, the best practice is to have one LUN per VMFS only (exception being local storage). When using extents, gather multiple LUNs under one logical VMFS and not multiple partitions per LUN.

Note: Ensure a complete rescan is done on all ESX hosts after adding an extent to a VMFS volume on one ESX host. Otherwise, the same extent might be inadvertently added by another node in the cluster, which could potentially cause loss of data. Best practice is to add extents to an existing VMFS volume from a single node and then rescan the storage resources from all ESX hosts capable of accessing that shared storage resource.

How to Extend a VMFS Datastore
How to create a single VMFS datastore and increase the size by adding an extent.

First of all make sure you have created a new LUN on your storage or SAN and rescanned in VMware so all hosts can see the new storage

1. Select your datastore in the Datastore inventory.
2. Click Configurations tab.
3. In Datastore Details pane, click Properties link.
4. Click Increase on the Properties dialog box.
5. Do the following when prompted by the Increase Datastore Capacity wizard (field):
a. Select your LUN ID. (Extent Device)
b. Review current disk layout. (Current Disk Layout)
c. Leave the Maximum capacity check box selected. (Capacity)
d. Click Finish. (Ready to Complete)

Errors

When going through the Add Extent wizard on a datastore residing on a LUN that has already been extended at the hardware/array level you may receive a warning message indicating that there may be data loss.

The warning message is as follows:

Warning: The current disk layout will be destroyed.  All file systems and data will be lost permanently

This error message is in relation to the area of the disk that you are extending to, not the area of the disk you are extending from.

If you look in Target Identifier, you see the vmhba:W:X:Y:Z where the extent is to be placed. If your original disk was vmhba1:0:0:1 , that extent is created on vmhba:1:0:0:2

N_Port ID Virtualization Explained

February 5, 2012 Storage, VMware No comments

NPIV was initially developed by Emulex, IBM, and McDATA (now Brocade) to provide more scalable access to Fibre Channel storage from mainframe Linux Virtual Machine instances by allowing the assignment of a virtual WWN to each Linux OS partition.

N_Port ID Virtualization or NPIV is a Fibre Channel facility allowing multiple N_Port IDs to share a single physical N_Port. This allows multiple Fibre Channel initiators to occupy a single physical port, easing hardware requirements in Storage Area Network design, especially where virtual SANs are called for. NPIV is defined by the Technical Committee T11 in the Fibre Channel – Link Services (FC-LS) specification

With NPIV in place you can create a zone in a SAN that only one virtual machine can access, thus restoring that security between the applications even if they are both running on the same virtual machine

NPIV really pays off in a virtual environment because the virtual WWN follows the virtual machine. This means if you migrate the virtual machine from one host to another, there are no special requirements to make sure the target host has the correct access to the LUN. The virtual machine has that access and as a result the host inherits the ability to access it.

This greatly simplifies storage provisioning and zoning in a virtual environment by allowing the storage admin to interact with the lowest level of granularity in storage access. Once in place the storage admin can monitor SAN utilization statistics to track how each virtual machine is using SAN resources. With this level of detail the SAN administrator is better able to balance utilization.

What is required for NPIV?

To enable NPIV in the environment requires several components, the first of which is a NPIV aware fabric. The switches in the SAN must all support NPIV and again using Brocade as an example, all Brocade FC switches running Fabric OS (FOS) 5.1.0 or later support NPIV.

In addition the HBA’s must support NPIV as well and they need to expose an API for the VM monitor to create and manage the virtual fabric ports; it is relatively common for HBA’s to support this today.

Finally the virtualization software itself must support NPIV and be able to manage the relationship between the virtual NPIV ports and the virtual machines. Most virtualization software also requires the use of a specific type of disk mapping; VMware calls this Raw Disk Mapping (RDM).

In the VMware case, by default when a virtual machine is created it is mapped to a virtual disk in a Virtual Machine File System (VMFS). When the operating system inside the virtual machine issues disk access commands to the virtual disk, the virtualization hypervisor translates this to a VMFS file operation. RDMs are an alternative to VMFS. They are special files within a VMFS volume that act as a proxy for a raw device.

RDM gives some of the advantages of a virtual disk in the VMFS file system while keeping some advantages of direct access to physical devices. In addition to being used in a virtual environment with NPIV, RDM might be required if you use server clustering, or for better SAN snapshot control or some other layered application in the virtual machine. RDMs better enable systems to use the hardware features inherent to SAN arrays and the SAN fabric, NPIV being an example.

NPIV is completely transparent to disk arrays, so the storage systems themselves require no special support.

« Older Entries   Recent Entries »