vRealize Automation large scale deployment Part 2 IaaS Server Install

In a distributed installation, the system administrator can deploy multiple instances of the appliances and install IaaS components over multiple machines in the deployment environment.

This install will include the following

• 2 x Windows 2012 R2 Server running IaaS
• 2 x Windows 2012 R2 Servers running SQL 2012 in a SQL failover cluster

IP Addresses

IaaS Service Account

Step 1 – Check Pre-requisites

Make sure the server is fully patched and snapshotted prior to installation to allow easy rollback in the event of any issues

There is a great PowerShell script which will install the pre-requisites for you but it is always worth checking all the steps I’ve listed following this for your own sanity. Reboot after running the script

https://github.com/vtagion/Scripts

SQL

• TCP/IP protocol enabled for SQL Server

• Microsoft Distributed Transaction Coordinator Service (MS DTC) enabled on all SQL nodes and IaaS nodes in the system. MS DTC is required to support database transactions and actions such as workflow creation. Start > Run > dcomcnfg > Computer > My Computer > Distributed Transaction Coordinator > Local DTC > Properties
• Note there may be a clustered DTC, in which case modify this as well

• No firewalls between Database Server and the Web server or IaaS Server, or ports opened as described in Port Requirement
• If using SQL Server Express, the SQL Server Browser service must be running
• For 6.0.x installations, the database name cannot contain a space. For 6.1 and later installations, the use of spaces in names is supported
• Log into SQL Management Studio and add Domain Admins to Logins

IaaS Pre-requisites

• Configuration of Active Directory Domain Service Accounts for Local Administrators Group

• Configuration of Windows Server 2012 R2 Firewall

The firewall can either be turned off or there are certain rules which need enabling as per below if it is turned on

• Installation of Microsoft .NET 4.5.2 Framework
• Installation of Java Runtime 64-bit Environment (jre-7u67-windows-x64.exe; required to install the database)
• Note I had to use the below version. 1.8 did not work and you can use the latest 1.7 version which is jre-7u79 currently I think

• Click New

• Type the following path to the Java installation directory

• Installation and configuration of IIS Server

You can run these commands in PowerShell

• Add-WindowsFeature -Name Web-Webserver,Web-Http-Redirect,Web-Asp-Net,Web-Windows-Auth,Web-Mgmt-Console,Web-Mgmt-Compat, web-metabase

• Add-WindowsFeature -Name Was, Was-config-apis, was-Net-Environment,NET-Non-HTTP-Activ

• Add-WindowsFeature -Name Web-Webserver,Web-Http-Redirect,Web-Asp-Net,Web-Windows-Auth,Web-Mgmt-Console,Web-Mgmt-Compat, web-metabase

• Add-WindowsFeature -Name Was, Was-config-apis, was-Net-Environment,NET-Non-HTTP-Activ

• Add-WindowsFeature -Name NET-WCF-HTTP-Activation45

• Enabling the Secondary Login Service. You can just start this for the installation process then it can be stopped afterwards

• Configuration of the batch login access and service login
• Open Local Security Policy
• Modify the Log on as a batch job and Log on as a service with the account you are going to install IaaS on

• Next open IIS Manager and navigate to the default website

• Click on Authentication

• Next click on Providers and remove NTLM and Negotiate then add Negotiate back in followed by NTLM

• Next click on Advanced Settings
• Change it from Off to Accept. Click on OK then change it back to Off

• Do an iisreset

• Next we need to register asp.net
• Go to c:\Windows\Microsoft.Net\Framework64\v4.0.30319
• Type aspnet_regiis -i

• Do another iisreset
• The following registry modification is required for the IaaS web server to include Local Security Authority host names that can be referenced in in the NTLM authentication requests for CNAME and load balancer FQDN addresses.
• Open the Windows registry and browse HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0.
• Right-click MSV1_0, point to New, and click Multi-String Value.
• In the Name column, type BackConnectionHostNames, and press Enter.
• In the Value text box, type the CNAME or DNS alias that is used for the local shares on the computer, and click OK.
• Example for IaaS Web Servers: f5.ias.techlab.local

• Before the installation of the IaaS components, verify system cryptography
• Go to the Local Group Policy Editor, expand Computer Configuration, expand Windows Settings, expand Security Settings, expand Local Policies, expand Security Options and use FIPS-compliant algorithms for encryption and hashing.  Verify that signing is set to Disabled.

• Next I also like to add my IaaS service account to the Local Admins group on the server or if it is the Domain Admins group then add this for lab purposes

• Add REG_DWORD key DisableLoopbackCheck 1
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\
• Add REG_DWORD key DisableStrictNameChecking 1 HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\LanmanServer\Parameters
• Next I like to shutdown the server and take a snapshot at this point
• Do exactly the same procedure on the second IaaS server

Note: Once DTC was enabled on both the IaaS and the remote SQL server, the installation still failed. After some searching, I found that since the IaaS server and SQL server VMs were provisioned using the same Virtual Machine template in vSphere, DTC had to be uninstalled and re-installed on one of the servers, either the IaaS server or the SQL server. To perform this task, execute the following commands from an elevated command prompt (run cmd.exe as an Administrator):

• msdtc -uninstall
• msdtc -install
• Reconfigure settings
• Reboot

Step 2 – Install certificates

You will need to refer to my other blog about creating and installing vRA IaaS certificates here if you haven’t created them already.

http://www.electricmonk.org.uk/2015/12/03/installing-vra-6-x-certificates/

Import the certificate into IIS

Step 3 – Install IaaS Website and Model Manager Data

• Go to https://yourvRAserver.FQDN:5480/installer
• Download the IaaS installer

• Launch the installer from where you saved it and Run as Administrator

• Click Next

• Accept the License agreement

• Put in root and your password

• Choose Custom Install
• Select IaaS Server

• Select the Database checkbox
• I have a Windows Server 2012 / SQL2012 cluster called SQLCLUSTER which was picked up when I put in my SQL server name and clicked Scan
• I then unticked Use existing empty database and called it vcac

• Fix any warnings which appear in the Verify Pre-requisites box

• Click Check again

• Click Next and click Install

• Hopefully you should now see the below screen

• Untick the box which says Guide me through the initial system configuration and click Finish

Installing the Primary IaaS Web and Model Manager Data Server

• If you haven’t already, import the certificate you previously created. This is the PFX cert
• Double click on the certificate
• Choose Local Machine

• Check the path to your cert file is correct
• Click Next

• Enter the password if you created one
• Select Mark this key as exportable
• Click Next

• Accept the default store

• Check the final box and click Finish

• Add certificate into the IIS Console under Server Certificates. It may already be there. Check 443 bindings are linked to your certificate
• Just double check in Local Security that System Cryptography: Use FIPS compliant algorithms is disabled

• Launch the IaaS installer as Administrator again
• Click Next, accept the license agreement put in the root username and password
• Select Custom Install and IaaS server

• Select Website and ModelManagerData checkboxes
• On the Administration and Model Manager Website tab select the certificate that you previously imported
• Select the Suppress certificate mismatch box

• You should get a message back when you click Test Binding

• Click on the Model Manager Data tab
• Enter the FQDN of the vRA appliance load balanced address. In my case f5.vra.techlab.local
• On SSO Default Tenant, click Load
• Under certificate click Download (This is the certificate which should be pre-created from my other blog and imported into IIS
• Click View Certificate and check it
• Add in all the rest of the details

• On the Verify pre-requisites screen, make sure everything is ticked green and fix any issues

• Under Server and Account Settings put
• Passwords
• Passphrase
• SQL Servername and Database name

• You may get a message coming up about the user account needed adding to the Local Security Policy if you hadn’t added it there already
• Click Install

• It should start installing

• And hopefully say Completed

Useful Troubleshooting info

http://www.virtualvcp.com/vmware-vrealize-automation-vcac/208-vrealize-automation-6-2-installation-and-configuration-gotchas

Installing IaaS server on the second Iaas Server

This procedure is exactly the same except as the above process. We just install the website component on the second server

• Don’t forget all the pre-requisites
• Don’t forget to import your certificate
• Start the installer
• Enter your root and password for the vRA appliance screen
• Enter your details below choosing just the website component

• Enter all the relevant details again

• Follow the next prompts to install and finish

 

Configuring the vRA Appliances

VMware vRealize Automation Center documentation recommended the utilization of an external instance of VMware vFabric Postgres when setting up a high availability (HA) environment. However, since the release of VMware vRealize Automation standalone, VMware vFabric Postgres is End Of Availability and no longer available as a standalone product. To address customers needs, VMware developed a way to utilize the database instance located in the VMware vRealize Automation appliance in a high availability (HA) mode, without having to incur additional licensing.

Useful Links

http://pubs.vmware.com/vra-62/index.jsp#com.vmware.vra.install.doc/GUID-8E631C5E-97D7-4D2B-945A-33B5DDBA452F.html

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2108923

Instructions Part 1

Follow the below instructions for both appliances until you get to Part 2

• Shutdown both vRA appliances and snapshot in vCenter
• Download the 2108923_dbCluster.zip file from the VMware Knowledge Base.
• Add a 20GB disk to the primary vRA appliance and secondary appliances
• Power on the primary and secondary vRA appliances
• Log into both vRA_Appliance:5480 in a web browser
• Log into both vRA appliances in Putty and WinSCP
• Extract the tar file from the 2108923_dbCluster.zip file attached to this article to both the appliances (I created a /tmp/prostgres folder)
• Using winscp copy the 2108923_dbcluster.tar file to a tmp folder on both appliances
• In Putty (See screen below) extract the .tar file on both appliances
• tar xvf 2108923_dbCluster.tar

• type parted -l on both appliances
• You should see Error: /dev/sdd: unrecognized disk label. See the bottom of the screen

• Run ./configureDisk.sh /dev/sdd

• At this point it is normally a good idea to snapshot both appliances as they seem to be sensitive to the password you use especially the special characters. Do not use = anywhere in the password
• Run the pgClusterSetup.sh script to prepare the appliance databases for clustering
• Note: In our case the db_fqdn was the Load balanced DB FQDN for the Postgres database

./pgClusterSetup.sh [-d] db_fqdn [-w] db_pass [-r] replication_password [-p]postgres_password

[-d] Database load balancer fully qualified domain name
[-w] Database password (will set password to this value)
[-r] Replication password (Optional: will use Database password if not set)
[-p] Postgres password (Optional: will use Database password if not set

• cd /tmp/postgres
• ./pgClusterSetup.sh -d f5.db.techlab.local -w password -r password -p password

• This is the end of configuration on both appliances

Instructions Part 2

Configuring the database replication on appliance B

• Type su – postgres
• Type cd /opt/vmware/vpostgres/current/share/
• Type ./run_as_replica -h vRA_FQDN -b -W -U replicate (Note don’t copy and paste as needed typing in manually)

./run_as_replica –h Primary Appliance -b -W -U replicate
[-U] The user who will perform replication. For the purpose of this KB this user is replicate
[-W] Prompt for the password of the user performing replication
[-b] Take a base backup from the master. This option destroys the current contents of the data directory
[-h] Hostname of the master database server. Port 5432 is assumed

• Enter the same password which was created previously
• It should now look like the below
• Type yes

• Type yes

• Type the password

• Type yes to enable WAL archiving on the primary

• It will now say shutting down and ignore the error message

• Type yes to the base backup message
• Note to myself really, I had an issue where I needed to run a command as root on the second vRA appliance to stop the vpostgres service (service vpostgres stop) to get the installer to finish!

• Next test replication
• cd /opt/vmware/vpostgres/current/share/
• Type ./show_replication_status

Validate replication

• Connect to the appliance with the primary (master) database using SSH.
• Validate if the WAL process is running. You should see the WAL process by running this command:
• ps -ef | grep wal

Validate if the master is ready for read-write connections by running these commands:

• su – postgres
• cd /opt/vmware/vpostgres/current/bin
• ./psql vcac
• SELECT pg_is_in_recovery();

• You see output similar to the above
• Quit psql by running \q
• Connect to the appliance with the replica database using SSH.
• Validate if the replica is read only using these commands
• su – postgres
• cd /opt/vmware/vpostgres/current/bin
• ./psql vcac
• SELECT pg_is_in_recovery();

• Quit psql by running \q

Instructions Step 3

Testing Failover between the Postgres Databases. Performing a test failover (appliance A to appliance B)

• Validate if the WAL process is running. You should see the WAL process by running this command:
• Type ps -ef | grep wal

• Connect to appliance A using SSH as root
• Stop the vpostgres service by running service vpostgres stop

• Connect to appliance B using SSH as root.
• Promote the replica database to master as the postgres user by running these commands
• su – postgres
• cd /opt/vmware/vpostgres/current/share
• ./promote_replica_to_primary

• SSH into appliance A as root.
• Configure database replication as user postgres by running these commands
• su – postgres
• cd /opt/vmware/vpostgres/current/share/
• ./run_as_replica -h FQDNofServer -b -W -U replicate
• Note the FQDN of the server was the second node which was been promoted to primary

1. Enter the replicate users password when prompted.
2. Type yes after verifying the thumbprint of the primary machine when prompted.
3. Enter the postgres users password when prompted.
4. Type yes when prompted with Warning: the base backup operation will replace the current contents of the data directory. Please confirm by typing yes
5. Do a quick check to test which machine is the primary and which is the secondary

Instructions Step 4

Perform a test failback (appliance B to appliance A)

• Connect to appliance B using SSH as root.
• Stop the vpostgres service by running this command:
• service vpostgres stop

•  Connect to appliance A using SSH as root.
• Promote the replicate database to master as user postgres by running these commands
• su – postgres
• cd /opt/vmware/vpostgres/current/share/
• ./promote_replica_to_primary

• Connect to appliance B using SSH as root.
• Configure database replication as user postgres by running these commands:
• su – postgres
• cd /opt/vmware/vpostgres/current/share
• ./run_as_replica -h FQDNofServer -b -W -U replicate
• Enter the replicate users password when prompted
• Type yes when prompted with:WARNING: the base backup operation will replace the current contents of the data

Validate replication

• Connect to the appliance with the primary (master) database using SSH.
• Validate if the WAL process is running. You should see the WAL process by running this command:
• ps -ef | grep wal
• Validate if the master is ready for read-write connections by running the commands below
• It should say f indicating it is the master

• You see output similar to the above
• Quit psql by running \q
• Connect to the appliance with the replica database using SSH.
• Validate if the replica is read only using these commands:

• Quit psql by running \q
• If you now log into the VAMI page of the vRA appliances and check the database and cluster page you should see the following

Configuring monitoring of the VMware vRealize Automation appliance databases

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2127052