Archive for vCenter

Testing port connectivity from vCenters and Hosts

Using Curl to test port conectivity from vCenter

Curl is available in the VMware vCenter Server Appliance command line interface. This small blog provides a simple example of using Curl to simulate a telnet connection to test port connectivity

To test port connectivity in VMware vCenter Server Appliance:

  1. Log in as root user through the VMware vCenter Server Appliance console.
  2. Run this command on the vCenter Server Appliance:

curl -v telnet://target ip address:port number

Example of testing port connectivity

All vCenter servers must have access to the UMDS server on port 80 (http)

The below screen-print shows a working curl test from a vCenter to a Windows UMDS Server on IP address 10.124.74.65 over port 80.

Using Netcat to test port connectivity from hosts

The telnet command is not available in any versions of ESXi and, therefore, you must use netcat (nc) to confirm connectivity to a TCP port on a remote host. The syntax of the nc command is:

nc -z <destination-ip> <destination-port>

vCenter 6.5U2c VCHA Error

 

 

 

 

Random problem when setting up vCenter HA

So this is an interesting one because I don’t have a solution but it is now working so I can only explain what happened. The 3 blades hosting the vCenter were HPE Proliant BL460c Gen10 servers. Once I reached the end of configuring a 6.5U2c vCenter for vCenter HA, I received the following error message.

So after going back and double checking typos and distributed switch and port group settings, everything looked fine but this error as you can see specifically mentioned a host vmh01. So i decided to run the VCHA wizard again which produced the same error but listed the second host


As I had 3 hosts in the cluster, I decided to run the wizard a third time which errored on the third host but running it a fourth time meant the VCHA setup ran perfectly and finished without any problems. There was no problem with the vDS or port groups or general networking.

The great thing about VCHA is that in this instance, it rolls everything back so you can simply start again. You might ask why I havent taken a snapshot – well it doesn’t allow you to do this! The rollback works very well, in fact 3 times in this scenario. Obviously not so good if you have hundreds of hosts 😀 A very strange problem where the NICs seemed to need a push before deciding to work however it did work in the end.

 

VMware vCenter 5 Attributes and Tags

Capture

What are Tags?

Tags allow you to attach metadata to objects in the vSphere inventory to make these objects more sortable and searchable. A tag is a label that you can apply to objects in the vSphere inventory. When you create a tag, you assign that tag to a category.

What are Categories?

Categories allow you to group related tags together. When you define a category, you can also specify which object types its tags can be applied to and whether more than one tag in the category can be applied to an object. For example, if you wanted to tag your virtual machines by guest operating system type, you could create a category called ‘operating system’, and specify that it applies to virtual machines only and that only a single tag can be applied to a virtual machine at any time. The tags in this category could be “Windows”, “Linux”, and “Mac OS”.

Tagging replaces the custom attributes functionality found in previous versions of vCenter Server. If you have existing custom attributes, you can convert them into tags.

Migrate Custom Attributes to Tags

Tags replace the custom attributes functionality found in previous versions of vSphere. If you have existing custom attributes, you can migrate them to tags.

During the migration

  • The custom attribute names are converted to categories.
  • Custom attribute values are converted to tag names.

Procedure

  • In the vSphere Web Client object navigator, browse to Home
  • Click Tags
  • Click Convert Custom Attributes

Attributes1

  • Select the Custom Attributes you want to migrate

Attributes2

  • Create Tag Categories
  • You can change the Category Name
  • Choose the Cardinality. For example, if you wanted to tag your virtual machines by Function, you could create a category called ‘Function’, and specify that it applies to virtual machines only and that only a single or multiple tag can be applied to a virtual machine at any time. The tags in this category could be “Application”, “Database”, “Active Directory”, “DNS” and “Web”.
  • In my case, I may want to assign 2 Functions to one VM so I will choose Many Tags per object. E.g Active Directory and DNS Server
  • Choose the Associable Object Types which could be anything from Datacenter, Cluster, Datastore or All Objects. In this case I would choose VM which you can see if you scroll down on this option

Attributes4

  • Next Create Tags. Check you are happy with the Tag name

Attributes5

  • You are now ready to complete. Click Finish

Attributes6

  • Now if you click on Tags > Items, you should see the Migrated Custom Attribute

Attributes7

Optimising SQL Server for VMware vCenter

images

SQL Modifications

I am using Microsoft SQL Server 2008 R2 running on Microsoft Windows Server 2008 R2. It is always worth having some knowledge about your Database software whether it be Oracle, SQL or DB2 etc and worth knowing how to optimise this software to work correctly for VMware vCenter whilst maintaining backups and maintenance plans for further minimization of issues and/or performance problems

Memory

  • Right-click the topmost SQL Server object, usually named with the machine name or local.
  • Choose Properties.
  • Choose the Memory page.
  • Set “Maximum Server Memory (in MB)” to something useful for the server. Probably something like 25%-50% of the RAM on the host.
  • The more memory you can give it the better, as the database will cache data in RAM, but you also want to leave room in RAM for the OS (2 GB) and some file cache.

sql1a

Recovery Model

  • Right-click the relevant Database in SQl Management Studio
  • Click Properties
  • Select Options
  • Set the Recovery Model to “Simple.” Click OK.

sql2

Configure Microsoft SQL Server TCP/IP for JDBC

If the Microsoft SQL Server database has TCP/IP disabled and the dynamic ports are not set, the JDBC connection remains closed. The closed connection causes the vCenter Server statistics to malfunction. You can configure the server TCP/IP for JDBC.

This task applies to remote Microsoft SQL Server database servers. You can skip this task if your database is local.

  • Select Start > All Programs > Microsoft SQL Server > Configuration Tool > SQL Server Configuration Manager
  • Select SQL Server Network Configuration
  • Protocols for Instance name
  • Enable TCP/IP
  • Open TCP/IP Properties and set the entries as per the below screen print
  • Click on the IP Addresses tab

sql3

  • Restart the SQL Server service from SQL Server Configuration Manager > SQL Server Services.
  • Start the SQL Server Browser service from SQL Server Configuration Manager > SQL Server Services.

Maintenance of your SQL Server Databases

  • Start the Microsoft SQL Server Management Studio again and log in as the sa user. Open the Management folder.

sql4

  • Right-click Maintenance Plans. Select Maintenance Plan Wizard.

sql5

  • Click Next
  • On the Select Plan Properties page give it the name WeeklyMaintenancePlan. Select Single schedule for the entire plan or no schedule

sql6

  • Click the Change button to pick when you want it to run.

sql7

  •  Schedule the job to occur when there is little occurring on the system. E.g No backups or antivirus scanning
  • Click Next and choose your Maintenance Tasks

sql8

  • Select the order for the Maintenance Tasks to run in

sql9

  • For Define Database Integrity Check Select All databases, including indexes.
  • You have the choices below

sql11

  • Click OK and it will bring you back to the Define Database Integrity Check

sql10

  •  For Define Reorganize Index select All databases, compact large objects.

sql12

  • For Define Rebuild Index select All Databases, reorganize pages with the default amount of free space. Also check Keep index online while reindexing. Note: The Keep index online option appears to be an Enterprise version feature, and you may see failures with it enabled on other SQL Server versions.

sql13

  • For Define Update Statistics select All Databases, all existing statistics, full scan

sql14

  •  Next on the Define Backup Database (Full) Task, enter the following

sql15

  • Backup Type = Full
  • Databases = All Databases
  • Backup Set will expire after = 14 Days
  • Backup to Disk = Selected
  • Create a backup file for every Database = Selected
  • Choose a folder according to where you want to back up
  • Backup File Extension = bak
  • Set backup compression = Use the default server settings. The Compress Backup option seems like a good one but it isn’t supported on 64-bit SQL Server. It’ll let you set it, then fail on execution
  • Next Define Maintenance Cleanup Task

sql16

  •  Delete files of the following type = Backup Files
  • Search Folder and delete files based on an extension = Choose your backup folder
  • File extension = bak
  • File age = 4 weeks or your choice
  • Next you are on to the Report Options Page

sql17

  •  Check the Summaries and Click Finish

sql18

  • Go into the Maintenance Plans folder now, right click on this job, and choose Execute to see if it runs. Check the logs if it doesn’t.
  • Your location may be different but as a rough guide, the log location is c:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\Log

Defragmenting VirtualCenter performance data indexes on a Microsoft SQL database

For troubleshooting or maintenance purposes it may be necessary to defragment the indexes on your Microsoft SQL database server.
Fragmentation of indexes occurs when the logical order of pages is different from the physical order on the disk. In VirtualCenter fragmentation occurs most noticeably due to the statistics collection and consolidation.

When the indexes are excessively fragmented, performance of queries to the VirtualCenter database is slow.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003990

Warning: If you do not have experienced DB administrators, shutdown the VMware VirtualCenter Server service and do a backup prior to performing any kind of database maintenance. If you have experienced DB administrators you can do the tasks online

Regular Reorganize Database Task

One of the performance suggestions buried in the VMware KB is to regularly reorganize the indexes, since the historical statistics tables get unwieldy. You can do this manually or schedule a job to do it by running the Maintenance Plan Wizard. Choose only Reorganize Indexes and set the schedule to recur every six hours, every day (or however often you want.This keeps the logical fragmentation of the indices down.

Click through the pages of the wizard until you get to “Define Reorganize Index Task.” Have it only reindex VCDB, choose “Tables and views” in the Object selection, and check “Compact large objects.” Click through until you’re done.

Installing VMware vCenter Server 5.1 using the Simple Install method

Architectural Changes

The vCenter Server 5.1 release includes significant architectural changes. You must understand these changes before attempting to freshly install or upgrade to vCenter Server 5.1 from older versions of the product. There are four separate services that constitute the vCenter Server 5.1 platform. These are below and must be installed in order

  • vCenter Single Sign On (SSO)
  • vCenter Inventory Service
  • vCenter Server
  • vSphere Web Client

Pre Requisites

Before installing vCenter Server 5.1, vSphere 5.1 requires you to install vCenter Single Sign On and install the Inventory Service. You can install vCenter Single Sign On, Inventory Service, and vCenter Server all on a single host machine using the vCenter Server Simple Install option. This option is appropriate for small deployments.

Alternatively, you can install vCenter Single Sign On, vCenter Inventory Service, and vCenter Server separately to customize the location and configuration of the components. (I found this to be the best way)

You also need Adobe Flash installed for the vSphere Web Client

If you are running vCenter as a Virtual Machine in Workstation then you will need at least 2GB RAM or more!

SSO-MultiSItesso-arch1

This blog will focus on installing vCenter Single Sign On, Inventory Service, and vCenter Server all on a single host machine running Windows Server 2008 R2 Enterprise

Instructions

Note: It may be best to install each component separately. I encountered a few errors when I went through the Simple Install Method. See screenprint below

Error

  • Download the ISO or installer from the VMware website
  • Make sure you have the .NET Framework installed
  • Attach the ISO or run the installer on your designated vCenter Server

vCenterInstaller

  • Select VMware vCenter Server Simple Install and Click Install

vcenter

  • Click Next

vCenter0

  • Click Next

vCenter1

  • Select I accept the terms in the license agreement and click Next

vCenter2

  • Put in a Strong Password and click Next. This is a local account not tied to AD or the Windows host. After SSO is installed, you can configure it for one or more LDAP/AD server and other identity sources.

Capture

  • For this demo, we will just be using the Express Instance

vCenter4

  • Put in the passwords for the RSA_USER and RSA_DBA accounts

vCenter5

  • The FQDN should be in here automatically. If you get an error saying nslookup cannot perform a lookup against this address then check your DNS server

vCenter6

  • Use Network Service Account or put in a Username and Password
  • Click Next

Capture1

  • Choose the location to save into and click Next

Capture3

  • Check HTTP Port

Capture4

  •  Click Install and allow SQL DB to be installed and SSO
  • When this has finished you will get the screen below
  • Put in a license key or just click Next if you are using it in Evaluation Mode

Capture

  • Click Next

Capture1

  • Click Next

Capture2

  • Click Next

Capture4

  • Click Next

Capture3

  • Click Next

Capture5

  • Click Install and then Finish
  • Install the vSphere 5 Web Client (Just follow the prompts)

Capture

  • Next check all the vCenter and Webservices and SSO are running

service

  • Install the vSphere 5 client

client5

  • Make sure you have downloaded and installed Adobe Flash
  • Just a quick point, make sure your vCenter Server has +2GB RAM or things just don’t work very well especially if you are running SSO, Inventory and vCenter on the same box as a test
  • You also may need to adjust your firewall for port 9443
  • If you are running 5.1 rather than 5.0, it is best to log into the vSphere Web Client first before on https://localhost:9443/vsphere-client/ using your SSO Login admin@System-Domain + inital setup password before logging into the vSphere Web Admin Assistant on https://localhost:9443/admin-app or you will get an error such as the below
The vSphere Web Client Administration Tool only supports registration of vCenter
 Server version 5.0.  For newer versions, the vCenter Server system must be regi
stered with the Lookup Service to allow the vSphere Web Client to discover the s
ystem.
  • Log into vSphere web client as admin (admin@System-Domain, this is the default user added during install of vcenter)
  • Go to Administration -> SSO Users and Groups
  • Go to Groups tab and click on __Administrators__
  • Click on the little man icon to Add Principals

Capture

  • Select the local vCenter server as the identity source and search for a local user. See Screenprint) you can add your Admins Group or any other group
  • Then add that user and click OK.
  • Log in as the local user.
  • You should see the vCenter listed after you log in, if not, you may need to reboot.

perms

Useful VMware KB for troubleshooting known certificate error

http://blogs.vmware.com/kb/2012/10/implementing-ca-signed-ssl-certificates-with-vsphere-5-1.html

SSO Issues (Cannot log in using Domain account to vClient etc)

http://blogs.vmware.com/kb/2012/10/vsphere-sso-resources.html

http://longwhiteclouds.com/2012/09/26/vsphere-5-1-gotcha-with-single-sign-on-sso/

  • Go to Administration
  • Single Sign On and Discovery > Configuration
  • Click the + sign to add a new identity source. E.g Active Directory Server.
  • Fill in as per your domain. Note my lab domain is dacmt.local

identity

  • Click Test Connection
  • Wait for it to say Connection successful

connection good

  • Change the order of the domains so AD is at the top

Registering vCenter Server 5 (Not 5.1)

  • Log into the vSphere Web Admin Assistant on https://localhost:9443/admin-app

web

  • Click Register vCenter
  • Enter the vCenter FQDN
  • Enter the Username and Password
  • Enter the vCenter hostname

vcenter

  • Click Register
  • Accept the certificate

cert

Can you run vCenter 5 on Windows Servers 2012?

vCenter isn’t officially supported on Server 2012

What you will find is that the installation fails just before it tries to install the vSphere Profile-Driven Storage Service.

The reason why

To install that service, the vCenter service needs to be running. However, the vCenter service does not start properly in Windows Server 2012. This is due to a missing dependency. In particular, the VirtualCenter Server service relies on the ProtectedStorage service which was removed from Win8/Server 2012. The work around is the following open regedit and go to \System\CurrentControlSet\Services\vpxd and then open the DependOnService key and remove ProtectedStorage from the list. Reboot the machine and the vCenter service should come alive (might take a while). Then restart only the vCenter installation again once everything has come up (you need to wait for vCenter service to come alive which can take a few minutes). The install will continue from where it kicked off and finish

So the short version is, when the vCenter install fails, go to registry and remove ProtectedStorage dependency from the vpxd service, reboot and it should work. Restart the vcenter install and it will finish as per normal.

Changing vCenters IP Address

The Challenge

Currently at my work, our network team have decided they want to create a new VMware Management VLAN (Headache Time) They want us to move vCenter on to this new VLAN and assign a new…

  1. IP Address
  2. Subnet Mask
  3. Gateway
  4. VMware Port Group VLAN ID

So what can possibly go wrong?…. Apparently quite a lot

Once the networking is changed on your vCenter, the ESX(i) hosts disconnect because they store the IP address of the vCenter Server in configuration files on each of the individual servers. This incorrect address continues to be used for heartbeat packets to vCenter Server.

You may also experience connectivity issues with vSphere Update Manager, Autodeploy, Syslog and Dump Collector.

Things to remember

  1. Ensure you have a vCenter database backup.
  2. Once the vCenter IP address has changed all that should be necessary is to reconnect the hosts back into vCenter.
  3. Please ensure that the vCenter DNS entry gets updated with the correct IP address. In addition ensure you have intervlan routing configured correctly.
  4. In the worst case scenario and you have to recreate the vCenter database then all you will lose is historic performance data and resource pools.
  5. You will need to change the Port Group VLAN
  6. Creating a second nic on the vCenter and assigning it the IP address of the new VLAN won’t be of assistance as you will need to select a managed vCenter IP address if you do this

How to resolve this

There are two methods to get the ESX hosts connected again. Try each one in order

Method 1
  1. Log in as root to the ESX host with an SSH client.
  2. Using a text editor, edit the /etc/opt/vmware/vpxa/vpxa.cfg file and change the parameter to the new IP of the vCenter Server.
  3. Or for ESXi 4 and 5, navigate to the folder /etc/vmware/vpxa and with vi open the file: vpxa.cfg. Search for the line that starts with: and then change this parameter to the new IP address of the vCenter Server.
  4. Save your changes and exit.
  5. Restart the management agents on the ESX.
  6. Restart the VirtualCenter Server service with this command: # services.sh
  7. Return to the vCenter Server and restart the “VMware VirtualCenter Server” Service.

Note: This procedure can be performed on an ESXi host through Tech Support mode with the help of a VMware Technical Support Engineer.

Method 2
  1. From vSphere Client, right-click the ESX host and click Disconnect.
  2. From vSphere Client, right-click the ESX host and click Reconnect. If the IP is still not correct, go to step 3.
  3. From vSphere Client, right-click the ESX host and click Remove.
  4. Caution: After removing the host from vCenter Server, all the performance data for the virtual machines and the performance data for the host will be lost
  5. Reinstall the VMware vCenter Server agent.
  6. Select New > Add Host.
  7. Enter the information used for connecting to the host

Firewall/Router Passthrough

If the IP traffic between the vCenter Server and ESX host is passing through a NAT device like a firewall or router and the vCenter Server’s IP is translated to an external or WAN IP, update the Managed IP address:
  1. From vSphere Client connected to the vCenter Server top menu, click Administration and choose VirtualCenter Management Server Configuration.
  2. Click Runtime Settings from the left panel.
  3. Change the vCenter Server Managed IP address.
  4. If the DNS name of the vCenter Server has changed, update the vCenter Server Name field with the new DNS name

How we changed IP Address step by step on vSphere 4.1

  • First of all Remote Desktop into your vCenter Server and change the IP Address, Subnet Mask and Gateway.
  • Make sure inter vlan routing is configured between your new subnet and the subnet your DNS servers are on if this is the case
  • Go to your DNS Server and delete the entry for your current vcenter server
  • Add the new A Record for your vCenter Server
  • You may need to run an ipconfig /flushdns on the systems you are working on.
  • Try reconnecting via Remote Desktop to your vCenter Server to establish connectivity
  • Click Home and go to vCenter Server Settings and adjust vCenter’s IP address
  • At this point, all your hosts will have disconnected? (Don’t panic!)
  • At this point we logged into the host which runs vCenter using the vClient and changed the VLAN on the port group vCenter was on.
  • Go back to your logon into vCenter
  • Right click on the first disconnected hosts and click Connect
  • The below error message will appear

  • Click Close and then an Add host box will appear as per below screenprint

  • The host should now connect back in and adjust for HA

If you get any error messages afterwards then the IP Addres will need to be updated in a couple of other places. See link below

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1014213

 

Virtual vCenter – Pros and Cons

Over the years there have been some controversy over this topic. Should vCenter Server be a physical or a virtual machine?

The most important aspect is that both solutions are supported by VMware.

http://www.vmware.com/pdf/vi3_vc_in_vm.pdf

Physical Solution Pro’s

  • More scalable
  • Hardware upgrades can be carried out
  • It is not susceptible to a potential VI outage

Physical Solution Cons

  • A dedicated physical server is required
  • Extra Power usage
  • Extra cooling considerations
  • UPS considerations
  • Backup must be done using tradition tools
  • DR may be more difficult

Virtual Solution Pro’s

  • You do not need a dedicated physical server (a way to reach a greater consolidation)
  • Server Consolidation: instead of dedicating an entire physical server to VirtualCenter, you can run it in a virtual machine along with others on the same ESX Server host.
  • Mobility: by encapsulating the VirtualCenter server in a virtual machine, you can transfer it from one host to another, enabling maintenance and other activities.
  • Each backup solution that works for a VM work also in this case
  • Snapshots: A snapshot of the VirtualCenter virtual machine can be used for backup,
    archiving, and other similar purposes.
  • Availability: using VMware HA, you can provide high availability for the VirtualCenter server
  • You can via DRS rules place the vCenter on certain hosts so you know where it is.

Virtual Solution Con’s

  • It is susceptible to a potential VI outage
  • No cold migration
  • No cloning
  • It must contend for resources along with other VMs
  • If you wish to modify the hardware properties for the VirtualCenter virtual machine, you will need to schedule downtime for VirtualCenter. Then, you will need to connect to the ESX Server host directly with the VI Client, shut down the VirtualCenter virtual machine, and make the modifications.
  • Careful consideration and design thinking needs to built into a vSphere environment where a vDS will be used – See below

Virtual vCenter and vDS

VMware specifically state about running vCenter within a distributed switch and they said point blank, “it is not supported”. They said “Because vCenter governs the distributed switch environment, you can’t have vCenter within the distributed switch.”

If you lose your Virtual Center you will have no way in moving virtual machines between different port groups on the vNetwork Distributed Switch. In addition, you will not be able to get a virtual machine from the traditional virtual switch to a port group on the vNetwork Distributed Switch. Extra to that, you can’t move a VM to another VMware vNetwork Distributed Switch. So that means if you are using VMware vSphere vNetwork Distributed Switches & you lose virtual center you are almost disabled on the networking part. If you lose connectivity on the classic virtual switch & your adapter on the distributed switch are OK you still can’t move your virtual machines to that distributed switch till Virtual Center is back.”

Does this mean a virtual infrastructure design should keep a vSS around? I would say “yes!”. Perhaps it’s now more important to dedicate 2 of the ESX host’s pNICs for the ESX Service Console / ESXi Management VMKernel isolated as a vSS. The 2 pNICs are not only for redundancy anymore, but also to support one or more standby VM portgroups in case they’re needed as a recovery network for VMs normally using the vDS. Of course, that means creating the appropriate trunking, and VLANs ahead of time. Have everything ready for a quick and easy change of critical VMs when needed.

Therefore, a hybrid design using both a vSS and a vDS is a smart “safety net” to have. Especially when an admin has to point the vSphere client directly at an ESX/ESXi host. The “safety net” vSS portgroups will be available from each host and the VMs can be easily switched via the vSphere Client GUI.

See this useful article by Duncan Epping

http://www.yellow-bricks.com/2012/02/08/distributed-vswitches-and-vcenter-outage-whats-the-deal/

In the event that the worst happens and you lose connectivity

VMware has provided a KB Article 1010555 which will allow an admin to create a vSS and move the vCenter VM on to this switch

KB1010555