www.electricmonk.org.uk

vSphere 6 Platform Services Controller HA Setups – High Availability with an F5 Load Balancer

vSphere 6 Platform Services Controller HA Setups – High Availability

Useful Links

Useful VMware Feature Walkthrough Link here

VMware vCenter Server 6.0 Deployment Guide here Includes F5 setup steps in the Appendix.

vCenter Single Sign-On and Platform Services Controller High Availability Compatibility Matrix here

Configuring Windows PSC 6.0 High Availability for vSphere 6.0 (2113085) here

Information

When configuring PSC High Availability, the load balanced pair are required to be the same type; it is not supported to mix Appliance-Base and Windows-Based PSCs in the same load balanced pair.

New to vSphere 6.0, both the Appliance-based PSC and Windows-based PSC can be deployed in both multi-site or high availability configurations. Additionally, if you need multi-site in conjunction with high availability, you can now setup your vSphere environment to have multi-sites and then configure each site with secondary PSCs. A load balancer is still required per site to provide high-availability. Only local load balancers (often times referred to as LTM, or Local Traffic Manager) are supported for PSC HA

PSC Config (Lab Setup)

1 x Windows 2012 Server – techlabpsc001 (192.168.2.152/24)

1 x Windows 2012 Server – techlabpsc002 (192.168.2.153/24)

Load Balanced Name – psclb.techlab.local

Load Balanced IP – 192.168.2.155

F5 Load Balancer is Version 12

Steps to enable High Availability on 2 Platform Service Controllers

Install Windows 2012 on a new server
Attach the vCenter 6 ISO to the server
In the software directory, double click the autorun installer

Accept the License Agreement
Choose External Deployment > Platform Services Controller

Put in a FQDN System Network Name for the Platform Services Controller

Ignore the warning below but do make sure you have added a DNS entry for the PSC into your DNS server and that it is joined to the domain

As this is the first PSC, you will need to select Create a new vCenter Single Sign-On domain.
Enter an SSO password
Put in the Site Name. In this case I am just using the name England-Site

Check the ports which need to be available

Select the destination directory

Choose whether to join the VMware Customer experience program

Double check the details you have entered

Once installed you should see the below screen

NEXT Install the second Platform Services Controller

In the software directory, double click the autorun installer

Accept the License Agreement
Choose External Deployment > Platform Services Controller

Put in a FQDN System Network Name for the Platform Services Controller

Select Join a vCenter Single Sign-On domain
Put in the first PSC FQDN
Put in the SSO password

Accept the Certificate

Select to join an existing site which is my England site

Check the Configure Ports screen

Choose your Destination Directory

Choose whether to join the Customer Experience Program
Check the final details and Install

NEXT: Now we need to download the scripts used to setup a cluster of PSC nodes into a highly available configuration from here

Download and unzip the scripts into a folder called c:\sso-ha

You should see the scripts here

Screen Shot 2016-07-06 at 15.43.35

Go to the first Platform Services Controller
Open a Command Prompt and add Python to the path
Type path=%PATH%;%VMWARE_PYTHON_HOME%

Type cd c:\sso-ha
Type python gen-lb-cert.py –primary-node –lb-fqdn=loadbalancerFQDN Where loadbalancerFQDN is the FQDN of the load balancer’s virtual IP used for load balancing the Platform Service Controllers
In my case I typed python gen-lb-cert.py –primary-node –lb-fqdn=psclb.techlab.local

This also generates a ha folder on the C Drive
Next Setup your load balancer to balance between the two Platform Service Controllers on ports 443, 2012, 2014, 2020, 389, and 636. See the vCenter Server 6.0 Deployment Guide – Page 88 for specific instructions on configuring the load balancer or read my notes below

My F5 v12 Load Balancer Notes below

Download the lb.p12 file from the c:\ha folder from the first Platform Services Controller.

Log in to the F5 BIG-IP configuration Web page.
Click System.
Open File Management, SSL Certificate List.

Click Import.
For Import Type, select PKCS

Provide a descriptive Certificate Name. Browse for the Certificate downloaded earlier. Click Import.

You should now see the certificate as per below screenprint (psclb)

Click Local Traffic.
Open Profiles, SSL, Client.

Click Create.
Provide a descriptive Name. In my case psclb
Click Custom under Configuration
Click Add under Certificate Key Chain

Choose the Certificate and Key installed earlier.
Enter the Passphrase for the certificate. In this case it was changeme
Click Add.

Scroll to the bottom and click Finished. You will be taken back to the screen below

Open Profiles, SSL, Server.

Click Create.
Provide a descriptive Name.
Click Custom.

Choose the Certificate and Key installed earlier.

Scroll to the bottom and click Finished

Open Nodes, Node List.
Click Create.

Add all Platform Services Controllers as a node. (I added my 2 PSC Nodes techlabpsc002 and techlabpsc003)
Use Repeat to speed up the process.

Open Pools, Pool List.
Click Create.

Create six pools, one each for port 443, 2012, 2014, 2020, 389, and 636.
All pools have the same Configuration, tcp for monitoring, and Round Robin for Load Balancing Method.
Add both psc servers to he New Members box
Use Repeat to save time: Remove the existing members from the list.

Open Virtual Servers, Virtual Server List.

Click Create.
All virtual servers—except the one for port 443—have the same configuration.
Provide a descriptive Name.
Enter the Destination Address. (The Load Balanced address)
For Service Port, enter 443 and HTTPS
For SSL Profile (Client), select the client profile created earlier.
For SSL Profile (Server), select the client profile created earlier.
For Source Address Translation, select Auto Map.
For the Default Pool, select the pool created for port 443.
For the Default Persistence Profile, select source_addr.
Click Finished
Repeat the steps above from Click Create to create virtual servers for all other ports: 2012, 2014, 2020, 389, and 636. All settings are the same as port 443, except there is no SSL Profile (Client) or SSL Profile (Server) and the Service Port and Default Pool should match. For example, if the Service Port is 2012, the Default Pool should be the pool set up for port 2012.

Open Profiles, Persistence.
Click source_addr.

Check Match Across Services and click Update

After both Platform Services Controller nodes have been installed and configured, click Network Map and verify that all services are up (green).

Next log into the second PSC
Copy the sso-ha and ha folder from the first Platform Services Controller into the c: drive.
Copy C:\ProgramData\VMware\vCenterServer\cfg\sso\keys from the first Platform Services Controller to c:\ha\keys.
Open a command prompt.
Add Python to your path by typing: path=%PATH%;%VMWARE_PYTHON_HOME%

Change directories to c:\sso-ha.
Run: python gen-lb-cert.py –secondary-node –lb-fqdn=loadbalancerFQDN –lb-certfolder=C:\ha –sso-serversign-folder=c:\ha\keys\ where loadbalancerFQDN is the FQDN of the load balancer’s VIP used for load-balancing the Platform Services Controllers

Repeat this step on any additional PSCs
On one Platform Services Controller, update the endpoint URL by running: python lstoolHA.py –hostname=FQDNofLocalMachine –lb-fqdn=loadbalancerFQDN –lb-cert-folder=C:\ha –user=Administrator@SSODomain –password=”password” where FQDNofLocalMachine is the FQDN of the machine where the script is being run, loadbalancerFQDN is the FQDN of the load balancer’s VIP used for load balancing the Platform Services Controllers, SSODomain is the vCenter Single Sign-On domain (by default vsphere.local), and password is the password for the vCenter Single Sign-On administrator. The password parameter is optional; if not specified, you will be prompted for it.

C:\sso-ha> python lstoolHA.py –hostname=techlabpsc002.techlab.local –lb-fqdn=psclb.techlab.local –lb-cert-folder=C:\ha –user=Administrator@vsphere.local

To verify the endpoints have been updated correctly run these commands using the First PSC Node FQDN entry:
Obtain the Site ID by running the following

“C:\Program Files\VMware\vCenter Server\python\python.exe” “C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\lstool.py” get-site-id –url https://psc_node_1_fqdn/lookupservice/sdk

Using the output sitename from the previous step, run these commands to verify the endpoints have been updated with the Load Balanced FQDN:

“C:\Program Files\VMware\vCenter Server\python\python.exe” “C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\lstool.py” list –url https://psc_node_1_fqdn/lookupservice/sdk –site My_Site_ID –type cs.license | findstr “URL:”

“C:\Program Files\VMware\vCenter Server\python\python.exe” “C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\lstool.py” list –url https://psc_node_2_fqdn/lookupservice/sdk –site My_Site_ID –type cs.identity | findstr “URL:”

Should bring back the same information as the above screenprint
Follow the steps to install a new external vCenter Server. When asked for the Platform Services Controller, enter the FQDN of the load balancer’s VIP.

vSphere 6 Platform Services Controller HA Setups – Enhanced Linked Mode

vSphere 6 Platform Services Controller HA Setups – Enhanced Linked Mode

To install vCenter Server with 2 or more external Platform Services Controllers, first install a Platform Services Controller for Windows followed by a second Platform Services Controller joined to the same domain The Platform Services Controller contains the common services, such as vCenter Single Sign-On and the License service, which can be shared across several vCenter Server instances.

You can install many Platform Services Controllers and join them to the same vCenter Single Sign-On domain. Concurrent installations of Platform Services Controllers are not supported. You must install the Platform Services Controllers in a sequence.

1. Enhanced Linked Mode

When you select to join an existing vCenter Single Sign-On domain, you enable the Enhanced Linked Mode feature. Your Platform Services Controller will replicate infrastructure data with the joined vCenter Single Sign-On server.

Note: You can use the appliance or a Windows Server. In the steps below, I have 2 Windows servers I am using as an example

Steps to enable Enhanced Linked Mode on 2 Platform Service Controllers

Install Windows 2012 on a new server
Attach the vCenter 6 ISO to the server
In the software directory, double click the autorun installer

Accept the License Agreement
Choose External Deployment > Platform Services Controller

Put in a FQDN System Network Name for the Platform Services Controller

Ignore the warning below but do make sure you have added a DNS entry for the PSC into your DNS server and that it is joined to the domain

As this is the first PSC, you will need to select Create a new vCenter Single Sign-On domain.
Enter an SSO password

Check the ports which need to be available

Select the destination directory

Choose whether to join the VMware Customer experience program

Double check the details you have entered

Once installed you should see the below screen

Now we need to move on to the second PSC and install this in Enhanced Linked Mode

Install Windows 2012 on a new server
Attach the vCenter 6 ISO to the server
In the software directory, double click the autorun installer

Accept the License Agreement
Choose External Deployment > Platform Services Controller

Put in a name for your second PSC Controller

Ignore the warning below but do make sure you have added a DNS entry for the PSC into your DNS server and that it is joined to the domain

Screen Shot 2016-07-06 at 10.48.47

As this is the second PSC, you will need to Join an existing vCenter Single Sign-On domain and put in the FQDN of the first PSC created earlier
Enter the Single Sign-On password

Accept the certificate

Select to join an existing site which in this case is the Default-First-Site

Check the Ports screen

Choose the Destination Directory

Select whether to join the Customer Experience Program

Check the final details

Finish.
The 2 PSCs are now set up in Enhanced Linked Mode

Determining replication agreements and status with the Platform Services Controller using vdcrepadmin

Useful VMware KB Link here

Use these parameters using the vdcrepadmin CLI:

showservers – Displays all of the PSCs in a vSphere domain.
showpartners – Displays the current partnerships from a single PSC within a vSphere domain.
showpartnerstatus – Displays the current replication status of a PSC and any of the replication partners of the PSC.
createagreement and removeagreement – Allows for creation and removal of additional replication agreements between PSCs within a vSphere domain.

Steps for vdcrepadmin showservers

This steps below provide information on using the vdcrepadmin command-line interface (CLI) for reviewing the existing vSphere domain, Platform Services Controllers (PSC) that make up your vSphere domain as well as checking the replication agreements configured and replication status within your environment. Although the utility can be used for other operations, at this time, only what is documented must be executed by technical support staff and customers.

Open a Command Prompt as Administrator
Navigate to cd c:\Program Files\VMware\vCenter Server\vmdird
Type the below command to show all the PSC Controllers in the vSphere domain

vdcrepadmin -f showservers -h PSC_FQDN -u administrator -w Administrator_Password

Example

vdcrepadmin -f showservers -h techlabpsc002.techlab.local -u administrator -w Password123!

You should now see the below showing you your 2 PSCs

Steps for vdcrepadmin showpartners

Next type the following command to show the psc partners

vdcrepadmin -f showpartners -h psc1.vmware.local -u administrator -w VMw@re123

Example

vdcrepadmin -f showpartners -h techlabpsc002.techlab.local -u administrator -w Password123!

You could run this showpartners command against all PSCs to map out the topology of the current vSphere domain by re-running this command against each of the PSCs in order to determine all of the partnerships.
You can see that some environments will be installed in an in-line fashion, with each PSC installed against the previous PSC, rather than a hub-and-spoke fashion where all of the PSCs would terminate to a central PSC

Steps for vdcrepadmin showpartnerstatus

Next type the following command to show the partner replication status.
This CLI is limited to execution only against the local PSC. Using the command to query the replication status from one PSC to a different PSC is not yet supported.

vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w Administrator_password

Example

vdcrepadmin -f showpartnerstatus -h techlabpsc002.techlab.local -u administrator -w Password123!

If you have problems with replication failing, review the /var/log/vmware/vmdird/vmdird-syslog.log or %VMWARE_LOG_DIR%\vmdird\vmdird-syslog.log file for details. This provides all information related to replication status and the objects that are replicated

Steps for vdcrepadmin createagreement – Example only with 4 PSCs as I only have 2 PSCs

Note: This cannot be used to create replication agreements between disparate (separate) vSphere domains
Map out the topology of the current vSphere domain by re-running the showpartners command against each of the PSCs in order to determine all of the partnerships

For example you have 4 PSCs

psc1
psc2
psc3
psc4

You can use the showservers parameter to get a list of all of the PSCs in the domain.

vdcrepadmin -f showpartners -h psc1.vmware.local -u administrator -w VMw@re123
ldap://psc2. vmware.local

vdcrepadmin -f showpartners -h psc2.vmware.local -u administrator -w VMw@re123
ldap://psc1. vmware.local
ldaps://psc3. vmware.local

vdcrepadmin -f showpartners -h psc3.vmware.local -u administrator -w VMw@re123
ldap://psc4. vmware.local
ldaps://psc2. vmware.local

vdcrepadmin -f showpartners -h psc4.vmware.local -u administrator -w VMw@re123
ldap://psc3. vmware.local

With the topology defined, we can now generate new replication agreements. Using the PSCs 1-4 in this section as a model, we need to generate additional replication agreements between:
PSC1.* and PSC3.*
PSC1.* and PSC4.*
PSC2.* and PSC4.*
Use the following command to create a new replication agreement between PSCs to generate a mesh topology:

vdcrepadmin -f createagreement -2 -h Source_PSC_FQDN -H New_PSC_FQDN_to_Replicate -u administrator -w Administrator_Password

For example:

vdcrepadmin -f createagreement -2 -h psc1.vmware.local -H psc3.vmware.local -u Administrator -w VMw@re123

vdcrepadmin -f createagreement -2 -h psc1.vmware.local -H psc4.vmware.local -u Administrator -w VMw@re123

vdcrepadmin -f createagreement -2 -h psc2.vmware.local -H psc4.vmware.local -u Administrator -w VMw@re123

Repeat this operation for additional PSCs until you have created an entire mesh topology.
After completion, repeat Step 5 to confirm that you have generated a mesh topology.
Note: Due to replication time, it may take a few seconds to minutes for a complete mesh topology to be configured.

Steps for vdcrepadmin removeagreement – Example only with 4 PSCs as I only have 2 PSCs

Map out the topology of the current vSphere domain by re-running the showpartners command against each of the PSCs in order to determine all of the partnerships

For example you have 4 PSCs

psc1
psc2
psc3
psc4

You can use the showservers parameter to get a list of all of the PSCs in the domain.

vdcrepadmin -f showpartners -h psc1.vmware.local -u administrator -w VMw@re123
ldap://psc2. vmware.local
ldap://psc3. vmware.local
ldap://psc4. vmware.local

vdcrepadmin -f showpartners -h psc2.vmware.local -u administrator -w VMw@re123
ldap://psc1. vmware.local
ldap://psc3. vmware.local
ldap://psc4. vmware.local

ldap://psc4. vmware.local

vdcrepadmin -f showpartners -h psc3.vmware.local -u administrator -w VMw@re123
ldap://psc4. vmware.local
ldap://psc2. vmware.local
ldap://psc1. vmware.local

vdcrepadmin -f showpartners -h psc4.vmware.local -u administrator -w VMw@re123
ldap://psc3. vmware.local
ldap://psc1. vmware.local
ldap://psc2. vmware.local

Use the following command to remove a replication agreement

vdcrepadmin -f removeagreement -2 -h Source_PSC_FQDN -h PSC_FQDN_to_Remove_from_Replication -u administrator -w Administrator_Password

For example:

vdcrepadmin -f removeagreement -2 -h psc1.vmware.local -h psc3.vmware.local -u administrator -w Administrator_Password

vRealize Log Insight 3.3 and vRealize Operations Manager Integration

May 12, 2016 vRealize Log Insight No comments

vRealize Log Insight and Operations Manager Integration

VMware vRealize Log Insight delivers heterogeneous and highly scalable log management with intuitive, actionable dashboards, sophisticated analytics and broad third party extensibility, providing deep operational visibility and faster troubleshooting.

Sophisticated and scalable log analytics and log management organizes chaotic log data and gives you meaningful, actionable insights across multiple tiers of a hybrid cloud environments

Useful link

http://www.vmware.com/uk/products/vrealize-log-insight

Sizing

Steps

Download the Log Insight appliance from here
Import the OVF into vCenter
Power on the Log Insight Appliance
Connect to the IP address you set as your Log Insight Appliance Address – https://<Log Insight FQDN>
Click Next

Click Start New Deployment

Put in Admin Credentials

Put in a License key

Put in an email and check whether you want to join the customer experience program

Set the Time Configuration and test it. You can choose your own NTP server or sync with your ESXi hosts

Set your NTP Configuration

Finish the Configuration

Click Configure vSphere Integration
Put in your vCenter Server and username and password and test connection

It will then configure your hosts

A quick look through the Admin Pages

System Monitor

Cluster

Access Control

Hosts

Agents

Event Forwarding

License

vRealize Operations Integration

When you enable launch in context you will then get another menu option on an object in vROps as seen below

General

Time

Authentication

SMTP

Archiving

Next The Default Dashboards Screen

Dashboards are a collection of different charts or queries.

The screen is divided into four parts parts:

The menubar, all the way to the top
The dashboard selection. It’s the left part of the screen
The widget/chart area, which is the bottom part of the screen on the right
The filtering area, which is the top part of the screen on the right

in the top right hand corner, you can click on the drop down by Admin to change your password and e-mail address or if you want to change settings or add management packs to Log Insight (the three bars)

What can you do with dashboards?

You can create your own dashboards with useful metrics that you want to monitor closely.
Any query can be turned into a dashboard widget and visualized for any range in time.
You can check the performance of your system for the last hour, day, or week.
You can view a break down of errors by hour and observe the trends in log events.

You can filter by hostname

You can open the Interactive Analytics by clicking on the Search icon highlighted in yellow below

Within the Interactive Analytics page we can click on the highlighted icon Area to choose a type of chart to display

We can start typing a keyword into the box which will bring up other keywords you could use as well

Clicking on the gear icon to the left on an error message will bring up even more options allowing you to filter further and colourise events and errors

You can set the time interval you want to look at

There are 4 icons next to the time interval

You can add a current query you have built to your Favourites
You can add the current query to a dashboard
You can create or manage alerts
You can export or share a current query

There are another 4 tabs above the events where you can also see different information

Events

This lists all the events seen under the current query or default view

Field Table

A Field Table that contains events where each field represents a column. A dashboard field table widget contains the latest events for the given query in a table format where each field represents a column.

You can use a field table widget for the following reasons.

■	To see the latest events for the given query. This can be useful for change management or for security reasons.
■	To see only the fields you care about for a given query. This can be useful to limit event output

Event Types

The event Types tab is located on the Interactive Analytics page, under the search bar. When you click the event Types tab you see a list of similar events that are grouped together.

Machine learning analyzes events and discovers the types of fields that similar log messages contain. For example, the types may be timestamp, string, int, hex and others. The discovered types appear as hyperlinks within the event Types list.

Each type that machine learning discovers represents a new type of field called smart field. The default name of a smart field follows the format smart field – type number [event_type]. You can change the default name of a smart field. After you name a smart field, it appears under the Fields section just like other fields. You can rename or delete a smart field but you cannot modify its definition.

Machine learning introduces a new static field called event_type. You can use the event_type as a filter to include or exclude certain event types from queries

Event Trends

You can analyze log events for trends and anomalies.

Procedure

1	Navigate to the Interactive Analytics tab.
2	Construct and run your query by using the search text box and applying filters.
3	In the Set Time Range From Event dialog box, use the drop-down menus to select the period and direction of the time range.
4	Click the Event Trends tab. Realize Log Insight compares your query to the same time period immediately before and displays the result

Fields

You can create your own custom fields to search from by doing the following

Look at Events and the keywords you may want to reuse in future searches
Highlight the word and select Extract to field

Name the field

This can then be reused

vRealize Log Insight Management Pack Configuration – vRealize Operations Management Pack

May 12, 2016 Management Packs, vRealize Log Insight No comments

vRealize Log Insight Management Pack Configuration – vRealize Operations Management Pack

VMware vRealize Operations Manager content pack is provided to present log data in a more meaningful way and to analyze all the logs redirected from a vRealize Operations Manager instance(s). The content pack contains various dashboards, queries and alerts to provide better diagnostics and troubleshooting capabilities to the vRealize Operations Manager administrator

Description

The content pack for vRealize Configurations Manager can be used to aggregate and analyze the logs from multiple vRealize Operations Manager instances. Operators can then select the particular vRealize Operations cluster or node for further analysis of the current state of the environment.

Highlights

Proactive monitoring and alert notifications of the vRealize Operations clusters – Specific alerts focused on important events that indicate problems can be enabled to get the alerts in vR Ops as well as for sending emails to the administrator(s).
Cluster-role specific breakdown of vRealize Operations events – The dashboards are grouped based on the cluster role of the vR Ops nodes/slices like Master, Data, Replica and Remote Collector to provide better manageability.
Cluster-role specific breakdown of vRealize Operations events – The dashboards are also grouped based on the cluster role of the vR Ops nodes/slices like Master, Data, Replica and Remote Collector to provide better manageability.

What’s New in v 1.6

Added vRealize operations Telemetry and vRealize operations cassandra Components in the content pack
Added new dashboard & widgets relevant to 6.1+, with backwards compatibility to 6.0.x
New Dashboards, alerts and queries

Components

The vRealize Operation Manager content pack comprises of the following components:

12 Dashboard Groups
81 Dashboard Widgets
Queries
Alerts
Extracted Fields

Download Link

The Management Pack can be downloaded here from http://solutionexchange.com

Instructions

Once you have downloaded the Management Pack and saved it you will need to look at the documentation here
What we need to do next is modify a file called liagent.ini which is located in /var/lib/loginsight-agent on the vROps appliance

The vRealize Log Insight agent enables the integration and manages communication between vRealize Operations Manager and vRealize Log Insight. The liagent.ini file contains configuration properties that control how the vRealize Log Insight agent sends events to vRealize Log Insight servers, sets the communication protocol and port, and configures flat file log collection.
To identify the source and cluster role, tags need to be updated in the
liagent.ini configuration file. As administrator, configure the following tags for each node role and on each node in the cluster. The applicable values for Cluster roles are the following.

Master
Replica
Data
RemoteCollector

Within the file below I have highlighted in blue everything which needs adjusting according to the instructions below

vmw_vr_ops_appname: do not update this tag
vmw_vr_ops_logtype: do not update this tag
vmw_vr_ops_clustername: this tag can be updated
vmw_vr_ops_clusterrole: change the tag to either the Master, Replica, Data, or Remote Collector
vmw_vr_ops_nodename: this tag can be updated as per below can be picked up from Administration > Cluster Management in the vROps console

vmw_vr_ops_hostname: The IP or FQDN of the vRealize Operations Manager node as per below can be picked up from Administration > Cluster Management in the vROps console

The liagent.ini file

The information below is what is contained in the liagent.ini file

Note you will need to update the [Sever] section only once with the LogInsight Server name

; Client-side configuration of VMware Log Insight Agent
; See liagent-effective.ini for the actual configuration used by VMware Log Insight Agent

[server]
; Log Insight server hostname or ip address
; If omitted the default value is LOGINSIGHT
hostname=techlabvrl001.techlab.local

; Set protocol to use:
; cfapi – Log Insight REST API
; syslog – Syslog protocol
; If omitted the default value is cfapi
;
;proto=cfapi

; Log Insight server port to connect to. If omitted the default value is:
; for syslog: 512
; for cfapi without ssl: 9000
; for cfapi with ssl: 9543
;port=9000

;ssl – enable/disable SSL. Applies to cfapi protocol only.
; Possible values are yes or no. If omitted the default value is no.
;ssl=no

; Time in minutes to force reconnection to the server
; If omitted the default value is 30
;reconnect=30

[storage]
;max_disk_buffer – max disk usage limit (data + logs) in MB:
; 100 – 2000 MB, default 200
;max_disk_buffer=200

[logging]
;debug_level – the level of debug messages to enable:
;   0 – no debug messages
;   1 – trace essential debug messages
;   2 – verbose debug messages (will have negative impact on performace)
;debug_level=0

[filelog|messages]
directory=/var/log
include=messages;messages.?

[filelog|syslog]
directory=/var/log
include=syslog;syslog.?

[filelog|ANALYTICS-analytics]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”ANALYTICS”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log
include = analytics*.log*
exclude_fields=hostname

[filelog|COLLECTOR-collector]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”COLLECTOR”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log
include = collector.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|COLLECTOR-collector_wrapper]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”COLLECTOR”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log
include = collector-wrapper.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\.\d{3}

[filelog|COLLECTOR-collector_gc]
directory = /data/vcops/log
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”COLLECTOR”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
include = collector-gc*.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\w]\d{2}:\d{2}:\d{2}\.\d{3}

[filelog|WEB-web]
directory = /data/vcops/log
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”WEB”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
include = web*.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|GEMFIRE-gemfire]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”GEMFIRE”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log
include = gemfire*.log*
exclude_fields=hostname

[filelog|VIEW_BRIDGE-view_bridge]
tags = {“vmw_vr_ops_appname”:”vROps”,”vmw_vr_ops_logtype”:”VIEW_BRIDGE”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log
include = view-bridge*.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|VCOPS_BRIDGE-vcops_bridge]
tags = {“vmw_vr_ops_appname”:”vROps”,”vmw_vr_ops_logtype”:”VCOPS_BRIDGE”,”vmw_vr_ops_clustername”:”vropscluster” vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log
include = vcops-bridge*.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|SUITEAPI-api]
directory = /data/vcops/log
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”SUITEAPI”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
include = api.log*;http_api.log*;profiling_api.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|SUITEAPI-suite_api]
directory = /data/vcops/log/suite-api
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”SUITEAPI”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
include = *.log*
exclude_fields=hostname
event_marker=^\d{2}-\w{3}-\d{4}[\s]\d{2}:\d{2}:\d{2}\.\d{3}

[filelog|ADMIN_UI-admin_ui]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”ADMIN_UI”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log/casa
include = *.log*;*_log*
exclude_fields=hostname

[filelog|CALL_STACK-call_stack]
tags = {“vmw_vr_ops_appname”:”vROps”,”vmw_vr_ops_logtype”:”CALL_STACK”, “vmw_vr_ops_clustername”:”vropscluster“,”vmw_vr_ops_clusterrole”:”Master“, “vmw_vr_ops_nodename”:”vropscluster“,”vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log/callstack
include = analytics*.txt;collector*.txt
exclude_fields=hostname

[filelog|TOMCAT_WEBAPP-tomcat_webapp]
tags = {“vmw_vr_ops_appname”:”vROps”,”vmw_vr_ops_logtype”:”TOMCAT_WEBAPP”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log/product-ui
include = *.log*;*_log*
exclude_fields=hostname

[filelog|OTHER-other1]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”OTHER”,”vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“,”vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log
include = aim*.log*;calltracer*.log*;casa.audit*.log*;distributed*.log*;hafailover*.log;his*.log*;installer*.log*;locktrace*.log*;opsapi*.log*;query-service-timer*.log*;queryprofile*.log*;vcopsConfigureRoles*.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|OTHER-other2]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”OTHER”, “vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“, “vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log
include = env-checker.log*
exclude_fields=hostname
event_marker=^\d{2}\D{1}\d{2}\D{1}\d{4}\s\d{2}:\d{2}:\d{2}

[filelog|OTHER-other3]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”OTHER”, “vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“, “vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log
include = gfsh*.log*;HTTPPostAdapter*.log*;meta-gemfire*.log*;migration*.log*
exclude_fields=hostname

[filelog|OTHER-watchdog]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”OTHER”, “vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master”, “vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log/vcops-watchdog
include = vcops-watchdog.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|ADAPTER-vmwareadapter]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”ADAPTER”, “vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“, “vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log/adapters/VMwareAdapter
include = *.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|ADAPTER-vcopsadapter]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”ADAPTER”, “vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“, “vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log/adapters/VCOpsAdapter
include = *.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

[filelog|ADAPTER-openapiadapter]
tags = {“vmw_vr_ops_appname”:”vROps”, “vmw_vr_ops_logtype”:”ADAPTER”, “vmw_vr_ops_clustername”:”vropscluster“, “vmw_vr_ops_clusterrole”:”Master“, “vmw_vr_ops_nodename”:”vropscluster“, “vmw_vr_ops_hostname”:”techlabvro001.techlab.local“}
directory = /data/vcops/log/adapters/OpenAPIAdapter
include = *.log*
exclude_fields=hostname
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}

Next we need to copy this file into the vROps appliance via WinScp into the /var/lib/loginsight-agent folder. Note: Take a backup of the original liagent.ini file first
Next restart the liagentd service in Putty by typing /etc/init.d/liagentd restart
Following this we can go to our LogInsight server and check whether we have data coming in
Go to Dashboards and click on the dropdown on the left hand side

You should now see data starting to come in

Note: If you had previously configured vRealize Operations 6.0.x to send its logs to Log Insight directly by editing the logger configuration, you should now undo this configuration. Leaving it in place will result in some logs being sent to Log Insight twice, and may even confuse the content pack

vRA 7 Part 1 Minimal Installation of vRA7

February 18, 2016 vRA7, vRA7 Minimal Deployment No comments

What is vRA7?

VMware vRealize Automation 7 sets a new standard in cloud automation by radically changing how fast and easy it is to automate the delivery of IT services and thereby accelerating your time to value. This major update has a simplified architecture and includes an installation wizard, the unified blueprint model, and enhanced NSX support.

IT organizations can use VMware vRealize™ Automation to deliver services to their lines of business.

vRealize Automation provides a secure portal where authorized administrators, developers or business users can request new IT services and manage specific cloud and IT resources, while ensuring compliance with business policies. Requests for IT service, including infrastructure, applications, desktops, and many others, are processed through a common service catalog to provide a consistent user experience.

You can improve cost control by using vRealize Automation to monitor resource and capacity usage. For further cost control management, you can integrate vRealize Business Advanced or Enterprise Edition with your vRealize Automation instance to expose the cost of cloud and virtual machine resources, and help you better manage capacity, cost, and efficiency

Support Documentation

https://www.vmware.com/support/pubs/vrealize-automation-pubs.html

New Features

http://pubs.vmware.com/New Features

Support Matrix

https://www.vmware.com/pdf/vrealize-automation-70-support-matrix.pdf

Reference Architecture

http://pubs.vmware.com/vra-70/topic/com.vmware.ICbase/PDF/vrealize-automation-70-reference-architecture.pdf

Installing vRealize Automation (Minimal Install in lab)

Depending on your deployment requirements, you can install and configure vRealize Automation components by using the Installation Wizard, or manually, through the management console. With either method, you can choose to create a minimal installation, or distribute components over separate servers in a custom distributed installation, with or without load balancers.

Choose a minimal installation to deploy a proof of concept (PoC) or development environment with a basic topology. Choose an enterprise installation to deploy a production environment with the topology best suited to your organizational needs

To complete a minimal deployment, a system administrator installs the vRealize Automation appliance and Infrastructure as a Service (IaaS) components.

■	vRealize Automation appliance includes the Web console interface and support for single sign-on capabilities. It is installed as a virtual appliance.
■	Infrastructure as a Service (IaaS) is installed on a Windows Server machine.
■	The IaaS uses an SQL database that can be installed on the same machine as IaaS or on its own server.

The following figure shows the relationship and purpose of components of a minimal installation.

Step 1 DNS

vRealize Automation requires the system administrator to identify all hosts by using a fully qualified domain name (FQDN).
In a distributed deployment, all vRealize Automation components must be able to resolve each other by using a FQDN.
The Model Manager Web service, Manager Service, and Microsoft SQL Server database must also be able to resolve each other by their Windows Internet Name Service (WINS) name. You must configure the Domain Name System (DNS) to resolve these host names in your environment.
So I created an A record in DNS for my vRA7 appliance and an A record in DNS for my IaaS Server

Step 2 Check minimum hardware requirements

Your deployment must meet minimum system resources to install virtual appliances and minimum hardware requirements to install IaaS components on the Windows Server.
For operating system and high-level environment requirements, including information about supported browsers and operating systems, see the vRealize Automation Support Matrix.
The Hardware Requirements table shows the minimum configuration requirements for deployment of virtual appliances and installation of IaaS components. Appliances are preconfigured virtual machines that you add to your vCenter Server or ESXi inventory. IaaS components are installed on physical or virtual Windows 2008 R2 SP1, or Windows 2012 R2 servers. An Active Directory is considered small when there are up to 25,000 users in the OU to be synced in the ID Store configuration. An Active Directory is considered large when there are more than 25,000 users in the O

Step 3 Browser Considerations

Some restrictions exist for browser use with vRealize Automation.

■	vRealize Automation does not support Compatibility View mode for Internet Explorer 10 on Windows 7 platforms. If you are unable to log in to appliance management consoles or you receive an error on the SSO tab when using Internet Explorer 10, use the Developer Tools to set the browser mode to Internet Explorer 7.
■	Multiple browser windows and tabs are not supported. vRealize Automation supports one session per user.
■	VMware Remote Consoles provisioned on support a subset of vRealize Automation-supported browsers.

For operating system and high-level environment requirements, including information about supported browsers and operating systems, see the vRealize Automation Support Matrix

Step 4 Password requirements

The vRealize Automation administrator password cannot contain a trailing “=” character.
Verify that the adminstrator password you assign during installation does not end with an “=” character. Such passwords are accepted when you assign them, but result in errors when you perform operations such as saving endpoints

Step 5 Database requirements

The vRealize Automation administrator password cannot contain a trailing “=” character.Verify that the adminstrator password you assign during installation does not end with an “=” character. Such passwords are accepted when you assign them, but result in errors when you perform operations such as saving endpoints
If you clone an IaaS node, install MS DTC on each node after it has been cloned. When you clone a node that has MS DTC installed, its unique identifier is copied to each clone, which causes communication to fail. See Error in Manager Service Communication for further information.
The database can reside on the IaaS (Windows) server host or on a remote host.
Java-related requirements apply for databases on the IaaS (Windows) server host. They do not apply for external databases.

Step 6 IaaS Server requirements

You can use the following script to install all pre-requisites on your IaaS server but do a double check of everything first

https://github.com/vtagion/Scripts/blob/master/vRA%206.2%20PreReq%20Automation%20Script.ps1

Step 7 Port requirements

vRealize Automation uses designated ports for communication and data access.

Although vRealize Automation uses only port 443 for communication, there might be other ports open on the system.
Because open, unsecure ports can be sources of security vulnerabilities, review all open ports on your system and ensure that only the ports that are required by your business applications are open

Step 8 Certificates

vRealize Automation uses SSL certificates for secure communication among IaaS components and instances of the vRealize Automation appliance. The appliances and the Windows installation machines exchange these certificates to establish a trusted connection. You can obtain certificates from an internal or external certificate authority, or generate self-signed certificates during the deployment process for each component.

For important information about troubleshooting, supportability, and trust requirements for certificates, see the VMware knowledge base article at http://kb.vmware.com/kb/2106583.

You can update or replace certificates after deployment. For example, a certificate may expire or you may choose to use self-signed certificates during your initial deployment, but then obtain certificates from a trusted authority before going live with your vRealize Automation implementation

Step 9 Time Synchronization

A system administrator must set up accurate timekeeping as part of the vRealize Automation installation.

Installation fails if time synchronization is set up incorrectly.

Timekeeping must be consistent and synchronized across the vRealize Automation appliance and Windows servers. By using the same timekeeping method for each component, you can ensure this consistency.

For virtual machines, you can use the following methods:

■	Configuration by using Network Time Protocol (directly)
■	Configuration by using Network Time Protocol through ESXi with VMware Tools. You must have NTP set up on the ESXi.

Step 10 Deploy the vRealize Automation appliance

Note: If you have to cancel out of the wizard and when you log back in to the appliance, the wizard doesn’t automatically come up then you can do the following

ssh into the appliance and run vcac-vami installation-wizard activate
Put /#wizard.wizard at the end of the vRA portal address

Follow the instructions below

Download the vRealize Automation appliance from the VMware Web site. Click here

Optionally on the same page you can download the VMware vRealize Orchestrator appliance

Procedure

Select File > Deploy OVF Template from the vSphere client.

Browse to the vRealize Automation appliance file you downloaded and click Open.

Click Next.

Click Next on the OVF Template Details page.

Accept the license agreement and click Next.

Type a unique virtual appliance name according to the IT naming convention of your organization in the Name text box, select the datacenter and location to which you want to deploy the virtual appliance, and click Next.

Follow the prompts until the Disk Format page appears.

Verify on the Disk Format page that enough space exists to deploy the virtual appliance and click Next.

Follow the prompts to the Properties page.

The options that appear depend on your vSphere configuration.

Configure the values on the Properties page.

a	Type the root password to use when you log in to the virtual appliance console in the Enter password and Confirm password text boxes.
b	Select or uncheck the SSH service checkbox to choose whether SSH service is enabled for the appliance. This value is used to set the initial status of the SSH service in the appliance. If you are installing with the Installation Wizard, enable this before you begin the wizard. You can change this setting from the appliance management console after installation.
c	Type the fully qualified domain name of the virtual machine in the Hostname text box, even if you are using DHCP.
d	Configure the networking properties.

Click Next.

Start the host machine.

■

If Power on after deployment is available on the Ready to Complete page.

a	Select Power on after deployment and click Finish.
b	Click Close after the file finishes deploying into vCenter.
c	Wait for the machine to start. This could take up to five minutes.

■

If Power on after deployment is not available on the Ready to Complete page.

a	Click Close.
b	Power on the machine. This could take up to five minutes. Check the Remote console window

After a few moments, a success message appears.

Open a command prompt and ping the FQDN to verify that the fully qualified domain name can be resolved against the IP address of vRealize Automation appliance.

Step 11 Run the Installation Wizard for a Minimal Deployment

Open a Web browser.

Navigate to the vRealize Automation appliance management console by using its fully qualified domain name, https://vra-va-hostname.domain.name:5480/.

When the Installation Wizard appears, click Next.

Accept the End User License Agreement and click Next.

Select Minimal Deployment and Install Infrastructure as a Service on the Deployment Type screen and click Next.

Check that the prerequisites listed on the Installation Prerequisites page have been met and that the Windows servers on which you installed a Management Agent are listed.

Click the link and obtain the Management Agent software and install this agent on your IaaS server

The Mangement Agent executes work items which are issued by the VAMI. the context under whom the management agent is running executes the installer. Certificate changes can now be performed from the VAMI for infrastructure machines as well and this is handled by the management agent

The Management agent requires a direct connection to 5480 on all virtual appliances. It becomes aware of all the appliances in the system after the initial connection is established to the first VA. It is also used for log collection and telemetry etc.

The next screen will ask you for account information that has administrative rights on your IaaS Server. This account will be used to install services and additional pre-requisite software

Once the installer finishes, go back to your wizard. Notice that at the bottom of the screen you were on, there is now an IaaS Server listed. Set your NTP settings (THIS IS VERY IMPORTANT !) and click next

If needed, you can change the timekeeping method for your vRealize Automation appliance. Click Change Time Settings, if you make changes.

Click Next.

Click Run on the Run the Prerequisite Checker screen to verify that the Windows servers in your deployment are correctly configured for vRealize Automation use.

Because this step runs remotely, it can take several minutes for the step to run.

a	If a failed status is returned for a machine, click Fix to start automatic corrections or click Show Details and follow the instructions. Automatic corrections also restart
b	Click Run to rerun the checker.
c	When all statuses show success, click Next.

Proceed through the next screens, supplying the requested information to configure your deployment components, including the Web server, Manager Service, Distributed Execution Manager, vSphere proxy agent, and certificate information.

Additional information is available from the Help buttons.

DNS of the vRA appliance

SSO Password

IaaS server details

Database Information

DEM Information

Agents Information

vRealize Appliance Certificate

Web Certificate

Manager Service Certificate

Validate: Click Validate – Can take between 10 minutes and half an hour

Hopefully you should then see

A reminder to take snapshots

Read the message and click Install

The installation can take between 30 minutes and one hour

And hopefully should say completed

Update the license key

Choose Telemetry settings

Initial Content creation

Optionally, you can start an initial content workflow for a vSphere endpoint.
The process uses a local user called configurationadmin that is granted administrator rights.

A configuration admin user is created and a configuration catalog item is created in the default tenant. The

configuration admin is granted the following rights:

Approval Administrator
Catalog Administrator
IaaS Administrator
Infrastructure Architect
Tenant Administrator
XaaS Architect

What to do next

After you finish the wizard, log in to the default tenant as the configurationadmin user or as administrator.
Go to the service catalog, request the Initial Content catalog item
Complete the request form for the Initial Content workflow

Step 12 – Login using the configurationadmin account or administrator

Note you don’t have to put administrator@vsphere.local in, just administrator and your SSO password

Type https://vra-appliance-fqdn/vcac

vSphere 6 Platform Services Controller

February 7, 2016 vSphere 6 No comments

What is the Platform Services Controller?

Starting with vSphere 6.0, all prerequisite services for running vCenter Server and the vCenter Server components are bundled in the VMware Platform Services Controller. You can deploy vCenter Server with an embedded or external Platform Services Controller, but you must always install or deploy the

Platform Services Controller before installing or deploying vCenter Server

Installation Scenarios (Embedded or External)

When you install vCenter Server with an embedded Platform Services Controller, or deploy the vCenter Server Appliance with an embedded Platform Services Controller, vCenter Server, the vCenter Server components, and the services included in the Platform Services Controller are deployed on the same system.
When you install vCenter Server with an external Platform Services Controller, or deploy the vCenter Server Appliance with an external Platform Services Controller, vCenter Server and the vCenter Server components are deployed on one system, and the services included in the Platform Services Controller are deployed on another system.

Components included in the vCenter Server and vCenter Server Appliance installations

The VMware Platform Services Controller group of infrastructure services contains:

vCenter Single Sign-On
License service
Lookup Service
VMware Certificate Authority.

The vCenter Server group of services contains:

vCenter Server
vSphere Web Client
Inventory Service
vSphere Auto Deploy
vSphere ESXi Dump Collector
VMware vSphere Syslog Collector on Windows
VMware Sphere Syslog Service for the vCenter Server Appliance

Scenario 1: vCenter with an embedded PSC

Advantages of vCenter with an embedded PSC

The connection between vCenter Server and the Platform Services Controller is not over the network and vCenter Server is not prone to outages because of connectivity and name resolution issues between vCenter Server and the Platform Services Controller.
You will need fewer Windows licenses.
You will have to manage fewer virtual machines or physical servers.
You do not need a load balancer to distribute the load across Platform Services Controller.

Disadvantages of vCenter with an embedded PSC

There is a Platform Services Controller for each product which might be more than required. This consumes more resources.
The model is suitable for small-scale environments

Scenario 2: vCenter Server with an External Platform Services Controller

vCenter Server and the Platform Services Controller are deployed on separate virtual machine or physical server. The Platform Services Controller can be shared across several vCenter Server instances. You can install a Platform Services Controller and then install several vCenter Server instances and register them with the Platform Services Controller. You can then install another Platform Services Controller, configure it to replicate data with the first Platform Services Controller, and then install vCenter Server instances and register them with the second Platform Services Controller.

Advantages of vCenter Server with an External Platform Services Controller

Less resources consumed by the combined services in the Platform Services Controllers enables a reduced footprint and reduced maintenance
Your environment can consist of more vCenter Server instances

Disadvantages of vCenter Server with an External Platform Services Controller

The connection between vCenter Server and Platform Services Controller is over the network and is prone to connectivity and name resolution issues.
If you install vCenter Server on Windows virtual machines or physical servers, you need more Microsoft Windows licenses.
You must manage more virtual machines or physical servers

Scenario 3: Mixed Operating Systems

A vCenter Server instance installed on Windows can be registered with either a Platform Services Controller installed on Windows or a Platform Services Controller appliance.

Example of a Mixed Operating Systems Environment with an External Platform Services Controller on Windows

Example of a Mixed Operating Systems Environment with an External Platform Services Controller Appliance

Both vCenter Server and the vCenter Server Appliance can be registered with the same Platform Services Controller within a domain
Having many Platform Services Controllers that replicate their infrastructure data, allows you to ensure high availability of your system.
If an external Platform Services Controller with which your vCenter Server instance or vCenter Server Appliance was initially registered, stops responding, you can repoint your vCenter Server or vCenter Server Appliance to another external Platform Services Controller in the domain

Enhanced Linked Mode Overview (http://kb.vmware.com/kb/210854)

Enhanced Linked Mode connects multiple vCenter Server systems together by using one or more Platform Services Controllers.
Enhanced Linked Mode lets you view and search across all linked vCenter Server systems and replicate roles, permissions, licenses, policies, and tags.
When you install vCenter Server or deploy the vCenter Server Appliance with an external Platform Services Controller, you must first install the Platform Services Controller.
With Enhanced Linked Mode, you can connect not only vCenter Server systems running on Windows but also many vCenter Server Appliances. You can also have an environment where multiple vCenter Server systems and vCenter Server Appliances are linked together.

During installation of the Platform Services Controller, you can select whether to create a new vCenter Single Sign-On domain or join an existing domain. You can select to join an existing vCenter Single Sign-On domain if you have already installed or deployed a Platform Services Controller, and have created a vCenter Single Sign-On domain. When you join an existing vCenter Single Sign-On domain, the data between the existing Platform Services Controller and the new Platform Services Controller is replicated, and the infrastructure data is replicated between the two Platform Services Controllers

If you install vCenter Server with an external Platform Services Controller, you first must deploy the Platform Services Controller on one virtual machines or physical server and then deploy vCenter Server on another virtual machines or physical server. While installing vCenter Server, you must select the external Platform Services Controller. Make sure that the Platform Services Controller you select is an external standalone Platform Services Controller. Selecting an existing Platform Services Controller that is a part of an embedded installation is not supported and cannot be reconfigured after the deployment.

Repoint the Connections Between vCenter Server and Platform Services Controller

Joining external Platform Services Controller instances in the same vCenter Single Sign-On domain, ensures high availability of your system.

If your environment contains external Platform Services Controller instances within a site that replicate the infrastructure data within a single domain, you can redirect the vCenter Server instances to another Platform Services Controller. If an external Platform Services Controller stops responding, you can repoint the vCenter Server instances to another Platform Services Controller within the same domain.

If you want to distribute the load of an external Platform Services Controller, you can repoint some of the vCenter Server instances to other Platform Services Controller instances in the same domain.
You can repoint the connections between a vCenter Server instance and the external Platform Services Controller instances in different vCenter Single Sign-On sites if the Platform Services Controller instances replicate the infrastructure data within a single domain. A site in the VMware Directory Service is a logical container in which you can group Platform Services Controller instances within a domain. You can name the sites in an intuitive way for easier implementation. Currently, the use of sites is for configuring Platform Services Controller High Availability groups behind a load balancer. vCenter Single Sign-On sites can be, for example, external Platform Services Controller instances that are deployed in multiple physical locations.

For more information, see the VMware knowledge base article at
http://kb.vmware.com/kb/2131191

Prerequisites

Verify that the external Platform Services Controller instances are within a single site and replicate the infrastructure data within a single domain.

Procedure

Log in to the vCenter Server instance.
For vCenter Server Appliance, log in to the vCenter Server Appliance shell as root
For a vCenter Server instance installed on Windows, log in as an administrator to the virtual machine or physical server that you installed vCenter Server on.
Run the cmsso-util script.
cmsso-util repoint –repoint-psc psc_fqdn_or_static_ip [–dc-port port_number]
where the square brackets [ ] enclose the command options.
Here, psc_fqdn_or_static_ip is the system name used to identify the Platform Services Controller. This system name must be an FQDN or a static IP address.
Use the –dc-port port_number option if the Platform Services Controller runs on a custom HTTPS port. The default value of the HTTPS port is 443.
Log in to the vCenter Server instance by using the vSphere Web Client to verify that the vCenter Serveris running and can be managed.

The vCenter Server instance is registered with the new Platform Services Controller

vRealize Automation large scale deployment Part 3 IaaS Server Install

February 1, 2016 Part 3, vRA Distributed Deployment v6.2.3 No comments

vRealize Automation large scale deployment Part 2 IaaS Server Install

In a distributed installation, the system administrator can deploy multiple instances of the appliances and install IaaS components over multiple machines in the deployment environment.

This install will include the following

2 x Windows 2012 R2 Server running IaaS
2 x Windows 2012 R2 Servers running SQL 2012 in a SQL failover cluster

IP Addresses

IaaS Service Account

Step 1 – Check Pre-requisites

Make sure the server is fully patched and snapshotted prior to installation to allow easy rollback in the event of any issues

There is a great PowerShell script which will install the pre-requisites for you but it is always worth checking all the steps I’ve listed following this for your own sanity. Reboot after running the script

https://github.com/vtagion/Scripts

SQL

TCP/IP protocol enabled for SQL Server

Microsoft Distributed Transaction Coordinator Service (MS DTC) enabled on all SQL nodes and IaaS nodes in the system. MS DTC is required to support database transactions and actions such as workflow creation. Start > Run > dcomcnfg > Computer > My Computer > Distributed Transaction Coordinator > Local DTC > Properties
Note there may be a clustered DTC, in which case modify this as well

No firewalls between Database Server and the Web server or IaaS Server, or ports opened as described in Port Requirement
If using SQL Server Express, the SQL Server Browser service must be running
For 6.0.x installations, the database name cannot contain a space. For 6.1 and later installations, the use of spaces in names is supported
Log into SQL Management Studio and add Domain Admins to Logins

IaaS Pre-requisites

Configuration of Active Directory Domain Service Accounts for Local Administrators Group

Configuration of Windows Server 2012 R2 Firewall

The firewall can either be turned off or there are certain rules which need enabling as per below if it is turned on

Installation of Microsoft .NET 4.5.2 Framework
Installation of Java Runtime 64-bit Environment (jre-7u67-windows-x64.exe; required to install the database)
Note I had to use the below version. 1.8 did not work and you can use the latest 1.7 version which is jre-7u79 currently I think

Click New

Type the following path to the Java installation directory

Installation and configuration of IIS Server

You can run these commands in PowerShell

Add-WindowsFeature -Name Web-Webserver,Web-Http-Redirect,Web-Asp-Net,Web-Windows-Auth,Web-Mgmt-Console,Web-Mgmt-Compat, web-metabase

Add-WindowsFeature -Name Was, Was-config-apis, was-Net-Environment,NET-Non-HTTP-Activ

Add-WindowsFeature -Name Web-Webserver,Web-Http-Redirect,Web-Asp-Net,Web-Windows-Auth,Web-Mgmt-Console,Web-Mgmt-Compat, web-metabase

Add-WindowsFeature -Name Was, Was-config-apis, was-Net-Environment,NET-Non-HTTP-Activ

Add-WindowsFeature -Name NET-WCF-HTTP-Activation45

Enabling the Secondary Login Service. You can just start this for the installation process then it can be stopped afterwards

Configuration of the batch login access and service login
Open Local Security Policy
Modify the Log on as a batch job and Log on as a service with the account you are going to install IaaS on

Next open IIS Manager and navigate to the default website

Click on Authentication

Next click on Providers and remove NTLM and Negotiate then add Negotiate back in followed by NTLM

Next click on Advanced Settings
Change it from Off to Accept. Click on OK then change it back to Off

Do an iisreset

Next we need to register asp.net
Go to c:\Windows\Microsoft.Net\Framework64\v4.0.30319
Type aspnet_regiis -i

Do another iisreset
The following registry modification is required for the IaaS web server to include Local Security Authority host names that can be referenced in in the NTLM authentication requests for CNAME and load balancer FQDN addresses.
Open the Windows registry and browse HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0.
Right-click MSV1_0, point to New, and click Multi-String Value.
In the Name column, type BackConnectionHostNames, and press Enter.
In the Value text box, type the CNAME or DNS alias that is used for the local shares on the computer, and click OK.
Example for IaaS Web Servers: f5.ias.techlab.local

Before the installation of the IaaS components, verify system cryptography
Go to the Local Group Policy Editor, expand Computer Configuration, expand Windows Settings, expand Security Settings, expand Local Policies, expand Security Options and use FIPS-compliant algorithms for encryption and hashing. Verify that signing is set to Disabled.

Next I also like to add my IaaS service account to the Local Admins group on the server or if it is the Domain Admins group then add this for lab purposes

Add REG_DWORD key DisableLoopbackCheck 1
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\
Add REG_DWORD key DisableStrictNameChecking 1 HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\LanmanServer\Parameters
Next I like to shutdown the server and take a snapshot at this point
Do exactly the same procedure on the second IaaS server

Note: Once DTC was enabled on both the IaaS and the remote SQL server, the installation still failed. After some searching, I found that since the IaaS server and SQL server VMs were provisioned using the same Virtual Machine template in vSphere, DTC had to be uninstalled and re-installed on one of the servers, either the IaaS server or the SQL server. To perform this task, execute the following commands from an elevated command prompt (run cmd.exe as an Administrator):

msdtc -uninstall
msdtc -install
Reconfigure settings
Reboot

Step 2 – Install certificates

You will need to refer to my other blog about creating and installing vRA IaaS certificates here if you haven’t created them already.

http://www.electricmonk.org.uk/2015/12/03/installing-vra-6-x-certificates/

Import the certificate into IIS

Step 3 – Install IaaS Website and Model Manager Data

Go to https://yourvRAserver.FQDN:5480/installer
Download the IaaS installer

Launch the installer from where you saved it and Run as Administrator

Click Next

Accept the License agreement

Put in root and your password

Choose Custom Install
Select IaaS Server

Select the Database checkbox
I have a Windows Server 2012 / SQL2012 cluster called SQLCLUSTER which was picked up when I put in my SQL server name and clicked Scan
I then unticked Use existing empty database and called it vcac

Fix any warnings which appear in the Verify Pre-requisites box

Click Check again

Click Next and click Install

Hopefully you should now see the below screen

Untick the box which says Guide me through the initial system configuration and click Finish

Installing the Primary IaaS Web and Model Manager Data Server

If you haven’t already, import the certificate you previously created. This is the PFX cert
Double click on the certificate
Choose Local Machine

Check the path to your cert file is correct
Click Next

Enter the password if you created one
Select Mark this key as exportable
Click Next

Accept the default store

Check the final box and click Finish

Add certificate into the IIS Console under Server Certificates. It may already be there. Check 443 bindings are linked to your certificate
Just double check in Local Security that System Cryptography: Use FIPS compliant algorithms is disabled

Launch the IaaS installer as Administrator again
Click Next, accept the license agreement put in the root username and password
Select Custom Install and IaaS server

Select Website and ModelManagerData checkboxes
On the Administration and Model Manager Website tab select the certificate that you previously imported
Select the Suppress certificate mismatch box

You should get a message back when you click Test Binding

Click on the Model Manager Data tab
Enter the FQDN of the vRA appliance load balanced address. In my case f5.vra.techlab.local
On SSO Default Tenant, click Load
Under certificate click Download (This is the certificate which should be pre-created from my other blog and imported into IIS
Click View Certificate and check it
Add in all the rest of the details

On the Verify pre-requisites screen, make sure everything is ticked green and fix any issues

Under Server and Account Settings put
Passwords
Passphrase
SQL Servername and Database name

You may get a message coming up about the user account needed adding to the Local Security Policy if you hadn’t added it there already
Click Install

It should start installing

And hopefully say Completed

Useful Troubleshooting info

http://www.virtualvcp.com/vmware-vrealize-automation-vcac/208-vrealize-automation-6-2-installation-and-configuration-gotchas

Installing IaaS server on the second Iaas Server

This procedure is exactly the same except as the above process. We just install the website component on the second server

Don’t forget all the pre-requisites
Don’t forget to import your certificate
Start the installer
Enter your root and password for the vRA appliance screen
Enter your details below choosing just the website component

Enter all the relevant details again

Follow the next prompts to install and finish

vRealize Automation large scale deployment Part 1 Identity and vRA appliance install

January 7, 2016 Part 1, vRA Distributed Deployment v6.2.3 No comments

vRA Distributed deployment.

This series will cover a larger distributed deployment of vRealize Automation 6.2.3

Software required

Components

Only the Identity and vRA appliances are covered in this blog. The rest will be covered in the series to follow.

1 x Identity appliance
2 x vRA appliances (Postgres Database only)
2 x IaaS servers
2 x Manager Servers
1 x Orchestrator appliance
1 x F5 Load Balancer

Important

DNS must be configured for all servers/appliances you use and test it
Whatever you use for time sync must be identical for all servers/appliances you use

F5 Load Balancer setup and information

http://kaloferov.com/blog/configuring-vrealize-automation-load-balancing-using-f5-big-ip/

Certificates

Please follow one of my other blogs for generating and importing certificates into vRA appliances and servers

http://www.electricmonk.org.uk/2015/12/03/installing-vra-6-x-certificates/

Step 1 – Deploying the Identity Appliance

In the vSphere client or web client select File > Deploy OVF Template

Check the details

Accept the license agreeement
Put in a name for the vRA Identity appliance

Choose your storage

Leave the defaults for storage

Check the details and click Finish

Note: The identity appliance cannot be clustered but can be put on a vSphere HA cluster to provide redundancy in the event of hardware failure but not in the event of the Identity appliance having an issue.
You may need to go into the vCenter console for the machine and set a root password
You will then see this screen where you can see the web browser link to log into the Identity appliance

Log into the web link

Set the time zone

Set the SSO password

It should then look like the below screenprint

Click on host settings and put in the name of the identity appliance
Make sure there is a DNS entry for the identity appliance

Click on Network then the Address tab and put in the relevant details

You will then need to reboot and relogin
Next click on SSO > SSL
Generate a certificate for now. Example below

Click on Active Directory and put in your details

It will then look like the below

Go to the Admin tab and click Admin
Tick SSH service enabled and Administrator SSH login enabled

Click on Time settings and adjust if you have a time server. I left mine on Use host time

This should now be complete.
Note: You may want to adjust the CPU and RAM depending on customer requirements
Note. It might be wise at this point to shutdown the appliance and take a snapshot

Step 2 Deploy 2 vRealize Automation Appliances

Note: Follow the below steps for each appliance

In the vSphere client or vSphere Web Client click File > Deploy OVF template

Check the details

Accept the license agreement
Put in a name

Choose your storage

Choose your storage options

Next you will need to type in the hostname, ssh enabled, IP address, subnet mask, gateway and DNS servers

Click Next and check all your details

Once this is deployed, make sure you have a DNS entry added
Log into the appliance
Change the time settings first

Click on the Network tab and select Host Settings.
Fill in your details

Reboot the appliance

Click on the vRA Host Settings
Add in your host settings – this should be your load balanced name
Import your certificate in which should have been pre-created from the instructions in my previous vRA certificate blog

Click on SSO
Put in the SSO details (The identity appliance details)
If everything is ok then you will see a certificate message

Click Save Settings and note the SSO seems to take a long time

You should see the following

You should slowly see the services begin to come up
Note: To monitor service startup run the following command:
tail -f /var/log/vcac/catalina.out

Do exactly the same process on the second appliance
Add your license in – Go to vRA Settings > Licensing

Next please go to Part 2 for the Postgres clustering of the vRA appliances

http://www.electricmonk.org.uk/2016/01/07/vrealize-automation-large-scale-deployment-part-2-clustering-the-postgres-databases-on-the-vra-appliances-v6-0-2/

Licensing Problems

I had an issue where my license suddenly became invalid which was a little bizarre as it is test non expiring one.

However I followed the steps in the below article on both appliances and it came back fine

Thanks @ vmguru 🙂

https://www.vmguru.com/2015/09/downgrade-the-vrealize-automation-license/

vRealize Automation large scale deployment Part 2 Clustering the Postgres Databases on the vRA Appliances v6.2.3

January 7, 2016 Part 2, vRA Distributed Deployment v6.2.3 One comment

Configuring the vRA Appliances

VMware vRealize Automation Center documentation recommended the utilization of an external instance of VMware vFabric Postgres when setting up a high availability (HA) environment. However, since the release of VMware vRealize Automation standalone, VMware vFabric Postgres is End Of Availability and no longer available as a standalone product. To address customers needs, VMware developed a way to utilize the database instance located in the VMware vRealize Automation appliance in a high availability (HA) mode, without having to incur additional licensing.

Useful Links

http://pubs.vmware.com/vra-62/index.jsp#com.vmware.vra.install.doc/GUID-8E631C5E-97D7-4D2B-945A-33B5DDBA452F.html

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2108923

Instructions Part 1

Follow the below instructions for both appliances until you get to Part 2

Shutdown both vRA appliances and snapshot in vCenter
Download the 2108923_dbCluster.zip file from the VMware Knowledge Base.
Add a 20GB disk to the primary vRA appliance and secondary appliances
Power on the primary and secondary vRA appliances
Log into both vRA_Appliance:5480 in a web browser
Log into both vRA appliances in Putty and WinSCP
Extract the tar file from the 2108923_dbCluster.zip file attached to this article to both the appliances (I created a /tmp/prostgres folder)
Using winscp copy the 2108923_dbcluster.tar file to a tmp folder on both appliances
In Putty (See screen below) extract the .tar file on both appliances
tar xvf 2108923_dbCluster.tar

type parted -l on both appliances
You should see Error: /dev/sdd: unrecognized disk label. See the bottom of the screen

Run ./configureDisk.sh /dev/sdd

At this point it is normally a good idea to snapshot both appliances as they seem to be sensitive to the password you use especially the special characters. Do not use = anywhere in the password
Run the pgClusterSetup.sh script to prepare the appliance databases for clustering
Note: In our case the db_fqdn was the Load balanced DB FQDN for the Postgres database

./pgClusterSetup.sh [-d] db_fqdn [-w] db_pass [-r] replication_password [-p]postgres_password

[-d] Database load balancer fully qualified domain name
[-w] Database password (will set password to this value)
[-r] Replication password (Optional: will use Database password if not set)
[-p] Postgres password (Optional: will use Database password if not set

cd /tmp/postgres
./pgClusterSetup.sh -d f5.db.techlab.local -w password -r password -p password

This is the end of configuration on both appliances

Instructions Part 2

Configuring the database replication on appliance B

Type su – postgres
Type cd /opt/vmware/vpostgres/current/share/
Type ./run_as_replica -h vRA_FQDN -b -W -U replicate (Note don’t copy and paste as needed typing in manually)

./run_as_replica –h Primary Appliance -b -W -U replicate
[-U] The user who will perform replication. For the purpose of this KB this user is replicate
[-W] Prompt for the password of the user performing replication
[-b] Take a base backup from the master. This option destroys the current contents of the data directory
[-h] Hostname of the master database server. Port 5432 is assumed

Enter the same password which was created previously
It should now look like the below
Type yes

Type yes

Type the password

Type yes to enable WAL archiving on the primary

It will now say shutting down and ignore the error message

Type yes to the base backup message
Note to myself really, I had an issue where I needed to run a command as root on the second vRA appliance to stop the vpostgres service (service vpostgres stop) to get the installer to finish!

Next test replication
cd /opt/vmware/vpostgres/current/share/
Type ./show_replication_status

Validate replication

Connect to the appliance with the primary (master) database using SSH.
Validate if the WAL process is running. You should see the WAL process by running this command:
ps -ef | grep wal

Validate if the master is ready for read-write connections by running these commands:

su – postgres
cd /opt/vmware/vpostgres/current/bin
./psql vcac
SELECT pg_is_in_recovery();

You see output similar to the above
Quit psql by running \q
Connect to the appliance with the replica database using SSH.
Validate if the replica is read only using these commands
su – postgres
cd /opt/vmware/vpostgres/current/bin
./psql vcac
SELECT pg_is_in_recovery();

Quit psql by running \q

Instructions Step 3

Testing Failover between the Postgres Databases. Performing a test failover (appliance A to appliance B)

Validate if the WAL process is running. You should see the WAL process by running this command:
Type ps -ef | grep wal

Connect to appliance A using SSH as root
Stop the vpostgres service by running service vpostgres stop

Connect to appliance B using SSH as root.
Promote the replica database to master as the postgres user by running these commands
su – postgres
cd /opt/vmware/vpostgres/current/share
./promote_replica_to_primary

SSH into appliance A as root.
Configure database replication as user postgres by running these commands
su – postgres
cd /opt/vmware/vpostgres/current/share/
./run_as_replica -h FQDNofServer -b -W -U replicate
Note the FQDN of the server was the second node which was been promoted to primary

Enter the replicate users password when prompted.
Type yes after verifying the thumbprint of the primary machine when prompted.
Enter the postgres users password when prompted.
Type yes when prompted with Warning: the base backup operation will replace the current contents of the data directory. Please confirm by typing yes
Do a quick check to test which machine is the primary and which is the secondary

Instructions Step 4

Perform a test failback (appliance B to appliance A)

Connect to appliance B using SSH as root.
Stop the vpostgres service by running this command:
service vpostgres stop

Connect to appliance A using SSH as root.
Promote the replicate database to master as user postgres by running these commands
su – postgres
cd /opt/vmware/vpostgres/current/share/
./promote_replica_to_primary

Connect to appliance B using SSH as root.
Configure database replication as user postgres by running these commands:
su – postgres
cd /opt/vmware/vpostgres/current/share
./run_as_replica -h FQDNofServer -b -W -U replicate
Enter the replicate users password when prompted
Type yes when prompted with:WARNING: the base backup operation will replace the current contents of the data

Validate replication

Connect to the appliance with the primary (master) database using SSH.
Validate if the WAL process is running. You should see the WAL process by running this command:
ps -ef | grep wal
Validate if the master is ready for read-write connections by running the commands below
It should say f indicating it is the master

You see output similar to the above
Quit psql by running \q
Connect to the appliance with the replica database using SSH.
Validate if the replica is read only using these commands:

Quit psql by running \q
If you now log into the VAMI page of the vRA appliances and check the database and cluster page you should see the following

Configuring monitoring of the VMware vRealize Automation appliance databases

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2127052

Installing Oracle Linux 7 and Oracle Grid Infrastructure and Database 12c RAC cluster on vSphere 6 (Lab)

January 3, 2016 Oracle RAC 15 comments

What is Oracle RAC?

The Oracle Database with the Oracle Real Application Clusters (RAC) option allows the running of multiple database instances on different servers in the cluster against a shared set of data files, also known as the database. The database spans multiple hardware systems and yet appears as a single unified database to the application.

This enables the utilization of commodity hardware to reduce total cost of ownership and to provide a scalable computing environment that supports various application workloads. Oracle RAC is Oracle’s premier shared disk database clustering technology

Pre-Requisites

vSphere 5 or 6 or VirtualBox also works very well
2 x Oracle Linux 7 VMs (2 vCPUs and 6GB RAM)
Oracle Grid Infrastructure 12c Release 1 (12.1.0.2)
Oracle Database Enterprise Edition 12.1.0.2.0 for Linux x86-64
1 x 50GB O/S disk on each node
3 x 20GB CSR disks (Multi-writer enabled on Disk 2/3/4) on node 1 only and added to node 2 as existing disks
1 x 25GB DB Disk (Multi-writer enabled on Disk 5) on node 1 only and added to node 2 as an existing disk
1 x 25GB Redo Disk (Multi-writer enabled on Disk 6) on node 1 only and added to node 2 as an existing disk

IP Addresses

2 x LAN IP addresses
2 x Private IP addresses
2 x VIP IP addresses
3 x SCAN IP addresses

Note: I had to add 3 A records in DNS pointing to 192.168.1.132, 192.168.1.133 and 192.168.1.133 for the scan IPs
See this link for more info on DNS and SCAN names http://www.oracle.com/technetwork/products/clustering/overview/scan-129069.pdf

Software

Download the following software.

Oracle Linux; Once in the Oracle Cloud service, select “Oracle Linux” and “x86 64-bit”.
VirtualBox (Use if you don’t have a vSphere 5/6 installation)
Oracle Database 12c Release 1 (12.1.0.1.0) (use Linux x86-64, then “See All” link).
Oracle Database 12c Release 1 Grid Infrastructure (12.1.0.1.0) for Linux x86-64 (use Linux x86-64, then “See All” link).

Instructions for the first VM

Note: There may be points where it is good to snapshot your VMs if you need a point to go back to if anything goes wrong. It took me a while to work all this out and snapshots were very useful 🙂

Download Oracle Linux from http://edelivery.oracle.com/linux

Create 2 VMs
Add 6 disks to the first VM and just the O/S disk on the second VM
1 x O/S disk on each node
3 x CSR disk (Multi-writer enabled on Disk 2/3/4) on node 1 only and added to node 2 as an existing disks. (Must be independent persistent)
1 x DB Disk (Multi-writer enabled on Disk 5) on node 1 only and added to node 2 as an existing disk (Must be independent persistent)
1 x Redo Disk (Multi-writer enabled on Disk 6) on node 1 only and added to node 2 as an existing disk (Must be independent persistent)

Log into the web client and go to the second VM
You will need to add all the extra disks as existing disks to the second VM
Next you need to set the multi-writer flag on both VMs – The multi-writer has to be set on the non O/S disks in the vSphere Web Client not the c# client in vSphere 6 or the settings do not stay and you will find you cannot power on the second VM. In vSphere 5 you can still use the c# client to set the multi-writer flag in the Configuration parameters of the VM as seen below.
Link for Information here

VMFS is a clustered file system that disables (by default) multiple virtual machines from opening and writing to the same virtual disk (.vmdk file). This prevents more than one virtual machine from inadvertently accessing the same .vmdk file. The multi-writer option allows VMFS-backed disks to be shared by multiple virtual machines. This option is used to support VMware fault tolerance along with Oracle RAC, which allows a primary virtual machine and a standby virtual machine to simultaneously access a .vmdk file.
You will also need to add the configuration parameter disk.enableUUID and set the value to true. This step is necessary so that the VMDK always presents a consistent UUID to the VM, thus allowing the disk to be mounted properly. For each of the virtual machine nodes (VMs) that will be participating in the cluster, follow the steps below from the vSphere client. VMware by default does not pass through the device UUID’s for the disks so the Oracle documentation (the udev rules) will fail to work because they depend on the UUID’s to create the ASM symlinks in DEV
Attach the Oracle Linux iso to the first VM
This link is useful – https://oracle-base.com/articles/linux/oracle-linux-7-installation

Let it run through bootup then select your language

You will now see the below screen

Click on System and check the hard drive layout
I left it on automatic partitioning

Click on network and put in host and network name settings
Click Configure to put in IPv4 IP Address, Netmask and Gateway settings

Click Save then move the network connection slider to On

Click Done
Click on Software Selection > Select Server with GUI
You may want to choose to install a GUI desktop. This is never recommended for a production environment. For ease of maintenance, you can click on the SOFTWARE SELECTION button and choose to install a “Server with GUI” option and choose the KDE desktop.
I also selected Java Platform / Performance Tools and Development Tools

Click on the Begin Installation button after you select all the options that you want installed

You will now see the screen below

Set a root password

Next enter a username who will be the administrator

It will now install

Once installation is complete, reboot your machine
You should now see the below screen

Accept the license agreement

Check the network configuration and click Finish Configuration

You will now see the login page
Log in using your admin account which you setup

You will now see the below screen

On the typing screen select your language

Skip the email account setup

You should now see the screen below

Install Oracle on the second node

Next update VMware Tools on both nodes – Attach the iso to the VM

Log in to Putty and type the below commands
ls media to check nothing is previously mounted to /media
mount /dev/cdrom /media/
cd /media
ls
cp VMwareTools9.10.5.tar.gz /tmp/
cd /tmp
ls
tar xvzf VMwareTools9.10.5.tar.gz
cd vmware-tools-distrib
./vmware-tools.pl -default
rm -rf vmware-tools-distrib
rm -rf VMwareTools9.10.5.tar.gz
If there is an issue with VMware Tools saying VMware Tools not installed on the system then type
rm -f /etc/vmware-tools/locations
Next Open Putty and run yum update -y on both nodes

Next type yum install ntp -y on both nodes

Next Oracle have provided a pre-requisite checker
Type yum install oracle-rdbms-server-12cR1-preinstall -y on both nodes

Note: If you get an message saying No package oracle-rdms-server-12cR1-preinstall available then try typing
cd /etc/yum.repos.d/
wget http://public-yum.oracle.com/public-yum-ol7.repo
yum repolist
yum install oracle-rdbms-server-12cR1-preinstall -y

WINSCP into the VM and modify the host file to reflect both nodenames and IP addresses on both nodes
The SCAN address is commented out of the hosts file because it must be resolved using a DNS, so it can round-robin between 3 addresses on the same subnet as the public IPs. The DNS can be configured on the host machine using BIND or Dnsmasq, which is much simpler. If you are using Dnsmasq, put the RAC-specific entries in the hosts machines “/etc/host” file, with the SCAN entries uncommented, and restart Dnsmasq.

Set the password for the Oracle user on both nodes
Type passwd oracle and change password

Make sure the “/etc/resolv.conf” file on both nodes includes a nameserver entry that points to the correct nameserver. Also, if the “domain” and “search” entries are both present, comment out one of them. For this installation my “/etc/resolv.conf” looked like this.

On both nodes amend the “/etc/security/limits.d/20-nproc.conf” file from 4096 to 16384

Change the setting of SELinux on both nodes to disabled by editing the “/etc/selinux/config” file, making sure the SELINUX flag is set as follows.

Next we are going to disable the Linux firewall on both nodes, which will otherwise interfere with the RAC install. First let’s make sure the firewall service is stopped
Type systemctl stop firewalld
Type systemctl disable firewalld

Either configure NTP, or make sure it is not configured so the Oracle Cluster Time Synchronization Service (ctssd) can synchronize the times of the RAC nodes. If you want to deconfigure NTP do the following, which is what I did for this installation.

Add the following lines to the “/etc/security/limits.conf” file.

oracle   soft   nofile    1024
oracle   hard   nofile    65536
oracle   soft   nproc    2047
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768
grid     soft   nofile   1024
grid     hard   nofile   65536
grid     soft   nproc    2047
grid     hard   nproc    16384
grid     soft   stack    10240
grid     hard   stack    32768

Add or amend the following lines to the “/etc/sysctl.conf” file on both nodes. Note this file is probably OK… I didn’t need to adjust it.

fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500

Run /sbin/sysctl -p
Create the directories in which the Oracle software will be installed on both nodes
mkdir -p /u01/app/12.1.0.2/grid
mkdir -p /u01/app/oracle/product/12.1.0.2/db_1
chown -R oracle:oinstall /u01
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/

Add in groups on both nodes
Note you will see from the screenprint below that the pre-requisites installer from Oracle creates 2 groups but not the rest so we still need to create them
groupadd -g 54321 oinstall
groupadd -g 54322 dba
groupadd -g 54323 oper
groupadd -g 54324 backupdba
groupadd -g 54325 dgdba
groupadd -g 54326 kmdba
groupadd -g 54327 asmdba
groupadd -g 54328 asmoper
groupadd -g 54329 asmadmin

The user oracle was already created by the rdbms preinstall package with oinstall and dba groups. But it is not member of all newly created groups. Also, it has no rights for ASM, and is not able to see ASM disks to create a database. For that, the oracle user needs the asmdba group. You can drop and recreate the user, or just alter its parameters with this command:
/usr/sbin/usermod -g oinstall -G dba,backupdba,dgdba,kmdba,oper,asmdba,asmadmin oracle
Grid Infrastructure needs a software owner, member of oinstall, which is not created by the rdbms preinstall package. It is advised to create a dedicated user:
/usr/sbin/useradd -u 54322 -g oinstall -G asmadmin,asmdba,asmoper grid
Type passwd grid and change the password
These are the groups which are set up

You now need to edit the following lines at the end of the “/home/oracle/.bash_profile” file
To do this I opened WinSCP and navigated to /home/oracle
Click on Hidden at the bottom right hand corner
You should now see the hidden .bash_profile file

Copy the file to your desktop and you now need to add the following lines to the end of the file

# Oracle Settings
export TMP=/tmp
export TMPDIR=$TMP

export ORACLE_HOSTNAME=techlabora001.techlab.local
export ORACLE_UNQNAME=cdbrac
export ORACLE_BASE=/u01/app/oracle
export GRID_HOME=/u01/app/12.1.0.2/grid
export DB_HOME=$ORACLE_BASE/product/12.1.0.2/db_1
export ORACLE_HOME=$DB_HOME
export ORACLE_SID=cdbrac1
export ORACLE_TERM=xterm
export BASE_PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$BASE_PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -p 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

alias grid_env='. /home/oracle/grid_env'
alias db_env='. /home/oracle/db_env'
umask 022

Next Create a file called “/home/oracle/grid_env” with the following contents.

export ORACLE_SID=+ASM1
export ORACLE_HOME=$GRID_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

Next Create a file called “/home/oracle/db_env” with the following contents.

export ORACLE_SID=cdbrac1
export ORACLE_HOME=$DB_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

We now need to edit the “/home/oracle/.bash_profile” file on the “techlabora002” node to correct the ORACLE_SID and ORACLE_HOSTNAME values.
Note: At the bottom of the file it says unmask and meant to say umask

Create and edit the /home/oracle/grid_env file and edit as below

Create and edit the /home/oracle/db_env file and edit as below

Once the “/home/oracle/.bash_profile” has been run, you will be able to switch between environments as follows.
I logged in as oracle to both nodes and ran the following

Follow this link below or my instructions underneath to set a swap file space on both nodes.

http://it-toolkit.blogspot.co.uk/2015/07/increase-swapfile-space-on-oracle.html

Putty into each node and run the following
dd if=/dev/zero of=/home/swapfile bs=6048 count=1048576
mkswap /home/swapfile
swapon /home/swapfile
swapon -a
swapon -s

WinSCP into each node and update /etc/fstab (Important)

Next do the following
Open /etc/fstab/ and add the highlighted line to the bottom
tmpfs /dev/shm /tmpfs rw,exec 0 0

Then run in Putty the following command
mount -o remount,size=4g /dev/shm

The reason we do this is during the installer pre-requisite checker, it will sometimes error falsely. This error of /dev/shm is because of a bug in cluvfy which comes with 12.1.0.2 Grid Infrastructure. I don’t get any errors after doing the above
Next Type chkconfig avahi-daemon off on both nodes
On each node add the following to /etc/pam.d/login file on both nodes
session required /lib64/security/pam_limits.so session required pam_limits.so
session required pam_limits.so
Next on both nodes unzip the grid files
cd /tmp/grid
unzip linuxamd64_12102_grid_1of2.zip -d /tmp/grid
unzip linuxamd64_12102_grid_2of2.zip -d /tmp/grid
Restart both nodes

When both nodes have started, check they can both ping all the public and private IP addresses

At this point I shut both servers down and took a snapshot
Power both nodes back on again
Next we need to format the disks originally added to Node 1 only on node 1
Format the 5 disks which will be sdb, sdc, sdd, sde, and sdf
As an example for the first disk, type fdisk /dev/sdb
Type n for new
Type p for primary
Type 1 for partitionPress enter
Press enter
Press enter
Press w

SSH

The OUI (Oracle Universal Installer) is run on one node, and uses SSH and scp commands to run command and copy files on remote nodes during the install. SSH must be configured on all members of the cluster to avoid each command to prompt for a password. This configuration can be done during install process, with the OUI, or with a script provided by Oracle and available in /software/grid/sshsetup.

You need to run the sshUserSetup.sh script for both oracle and grid accounts:

Log into Node 1 using Putty and navigate to the grid folder you placed in the /tmp folder followed by navigating to the sshsetup folder
Note I checked this in WinSCP before logging in to Putty. You can see the sshUserSetup.sh script below

Within Putty, you need to enable execute permissons (chmod +x sshUserSetup.sh)
Then type ./sshUserSetup.sh -user grid -hosts “techlabora001 techlabora002” -advanced -exverify –noPromptPassphrase

You will be prompted to enter some inputs
Do you want to continue and let the script make the changes Say Yes
Say No to creating a passphrase
Put in the grid password several times
You should now see the below

Next you need to run the same command but change the user to oracle
Type ./sshUserSetup.sh -user oracle -hosts “techlabora001 techlabora002” -advanced -exverify –noPromptPassphrase
Putty into the second node
Within Putty, you need to enable execute permissons (chmod +x sshUserSetup.sh)
Then type ./sshUserSetup.sh -user grid -hosts “techlabora001 techlabora002” -advanced -exverify –noPromptPassphrase
Then type ./sshUserSetup.sh -user oracle -hosts “techlabora001 techlabora002” -advanced -exverify –noPromptPassphrase
Once you have set up the ssh, test it in Putty by typing ssh techlabora001 or techlabora002 from one node to another to test connectivity

ASM

The installation and configuration procedures in this section should be performed on both of the Oracle RAC nodes in the cluster. Creating the Automatic Storage Management (ASM) disks, however, will only need to be performed on a single node within the cluster (Node1).

ASM will be used as the shared file system and volume manager for Oracle Clusterware files (OCR and voting disk), Oracle Database files (data, online redo logs, control files, archived redo logs), and the Fast Recovery Area.

ASM distributes the I/O load across all available resources to optimize performance, while removing the need for manual I/O tuning. ASM helps DBAs manage a dynamic database environment by allowing them to grow the database size without any database downtime for storage allocation modification. Oracle ASM is easier to manage than conventional file systems, and is designed to provide optimal performance for Oracle database deployments.

ASM provides redundancy by mirroring against disk failures. ASM stripes data evenly across a number of disks in the disk group, and maintains a uniform distribution of extents as the storage configuration changes

With the introduction of Oracle ASM, administrators can expand storage capacity without having to take the database offline; additional ASM disks can be added to an existing ASM disk group for increased storage capacity. It is a best practice that all ASM disks belonging to the same disk group are similarly sized. Therefore, when adding new ASM disks to an existing disk group, it is recommended to configure the new disks with similar sizes as the existing ASM disks of the same disk group.

Adding or dropping ASM disks will automatically trigger ASM to rebalance the I/O that spreads data across all available disks in the disk group for optimal performance and utilization; this rebalancing involves physical movement of file extents. The impact to the online database activity is generally low, because the rebalance is done at one allocation unit at a time. However, Oracle recommends that you schedule the rebalance operation during off‐peak hours.

To use Oracle ASM, two or three packages are required, depending on the situation. Here, we consider using ASMLib instead of Raw devices. So three packages will be required:

oracleasm-support
oracleasmlib
The driver oracle asm (For this package, it depends of your kernel. If your server is configured with the default Unbreakable Enterprise Kernel (UEK), the driver is already compiled in the kernel. If you are using the Red Hat compatible kernel, you must install the following package (for Oracle Linux 6)

Instructions

Log into Putty
Install ASM Support
Type yum install oracleasm-support

Next we need ASMLIB
ASMLib is a support library for the Automatic Storage Management feature of the Oracle Database. Oracle provides a Linux specific implementation of this library
Type wget http://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.12-1.el7.x86_64.rpm
Type rpm -iv oracleasmlib-2.0.12-1.el7.x86_64.rpm

Type yum install kmod-oracleasm (May not need this. Oracle Linux 7 has the unbreakable kernel)

Next do the same on the second node
Next we need to configure and load the asm kernel module on both nodes
Type /usr/sbin/oracleasm configure -i
grid
asmadmin
y

The script completes the following tasks:

Creates the /etc/sysconfig/oracleasm configuration file
Creates the /dev/oracleasm mount point
Mounts the ASMLib driver file system
Next, initialize ASM module on both nodes:
Type /usr/sbin/oracleasm init

Next add the disks to the first node only

Next go to the second node and you can type
/usr/sbin/oracleasm scandisks
You can also type /usr/sbin/oracleasm status

If you want to check the disks type
/usr/sbin/oracleasm listdisks

Now we can start to install our Grid Infrastructure
Previously we downloaded and zipped the relevant grid files into the /tmp/grid/grid folder
Install cvuqdisk. The file you are looking for to start with is cvuqdisk in the /tmp/grid/grid/rpm folder

Type rpm -Uvh cvuqdisk*

Next there is a tool in the Grid software called runcluvfy. This is a tool provided by Oracle. It checks all software and hardware pre requisites on each node to make sure the install can be done. This tool is called before installing grid infrastructure, before adding a node, or before adding a database.

Run the following as the Oracle user
Type cd /tmp/grid/grid
Type ./runcluvfy.sh stage -pre crsinst -osdba asmdba -orainv oinstall -n techlabora001,techlabora002 -r 12.1 -fixup -verbose
Check the output for any tests which didn’t pass. I initially had an issue where it wouldn’t start at all in the below screenprint

Fixed by adding the hostname in /etc/hostname. for some reason it has unknown:00:50:56 etc
So a correct view is below

Next log into your Node 1 as the grid user
Open a terminal session
Navigate to /tmp/grid/
Type ./runInstaller

The Oracle Installer will now run
Select the “Install and Configure Oracle Grid Infrastructure for a Cluster” option, then click the “Next” button.

Choose Configure a Standard cluster

Choose Typical

Put in the correct SCAN name and public and virtual addresses

Click Add and put in your second node details

You should now see both your nodes

Click SSH Connectivity and test it. It should come back with the below screen

Check the public and private networks are specified correctly. If the NAT interface is displayed, remember to mark it as “Do Not Use”. Click the “Next” buttonCheck the network interfaces

Enter /u01/app/oracle as the Oracle Base
Enter “/u01/app/12.1.0.2/grid” as the software location and “Automatic Storage Manager” as the cluster registry storage type. Enter the ASM password, select “dba” or “asmadmin” as the group and click the “Next” button.

I left my disk group name as CRS
Set the redundancy to “External” and select your disks

NORMAL REDUNDANCY – Two-way mirroring, requiring two failure groups.
HIGH REDUNDANCY – Three-way mirroring, requiring three failure groups.
EXTERNAL REDUNDANCY – No mirroring for disks that are already protected using hardware RAID or mirroring

If you don’t see your disks straight away then click the “Change Discovery Path” button and set the path to “/dev/oracleasm/disks/*” OR ORCL:*. (See a few screenprints down after the troubleshooting)
I selected just my CRS disks as I will create another disk group later

Also I found in one installation I tried the disks just wouldn’t appear because I had run through the installer once and it had failed but used the disks, I needed to delete and recreate my disks as per below which was a pain but at least easy to rectify

Back to the ASM Disk Group wizard
Accept the default inventory

If you want the root scripts to run automatically, enter the relevant credentials
Or run them manually
run the /u01/app/oraInventory/orainstRoot.sh
and /u01/app/11.2.0/grid/root.sh scripts

You will now see the pre-requisites being checked

You will see I have some errors
Some are fixable, some will just need to be investigated and then you can run a check again (Note: After taking this screenprint I have added all the steps to stop these messages in the steps previously!)

Once the pre-requisites are complete, click Next and check the summary

It should then run through the install

It will stop and ask you about running scripts – Say yes

You can click Details on the installer screen to see what is happening
Highlighted in yellow is the install log which you can use in the event of any issues

Once it has completed, you should see the below screen

Next we can run some commands to check the cluster

CRSCTL

CRSCTL is an interface between you and Oracle Clusterware, parsing and calling Oracle Clusterware APIs for Oracle Clusterware objects.

Oracle Clusterware 11g release 2 (11.2) introduced cluster-aware commands with which you can perform check, start, and stop operations on the cluster. You can run these commands from any node in the cluster on another node in the cluster, or on all nodes in the cluster, depending on the operation.

You can use CRSCTL commands to perform several operations on Oracle Clusterware, such as:

Starting and stopping Oracle Clusterware resources
Enabling and disabling Oracle Clusterware daemons
Checking the health of the cluster
Managing resources that represent third-party applications
Integrating Intelligent Platform Management Interface (IPMI) with Oracle Clusterware to provide failure isolation support and to ensure cluster integrity
Debugging Oracle Clusterware components

Testing the cluster

type /u01/app/12.1.0.2/grid/bin/crsctl status resource -t

Check for cluster nodes
Type /u01/app/12.1.0.2/grid/bin/olsnodes -n

Check the CRS status on the first node as the grid user
Type /u01/app/12.1.0.2/grid/bin/crsctl check crs

Check for cluster registry (OCR)
Type /u01/app/12.1.0.2/grid/bin/ocrcheck

Check voting disk
Type /u01/app/12.1.0.2/grid/bin/crsctl query css votedisk

The number of voting disks in ASM is derived by the redundancy of the diskgroup.
1 Vote disk with external redundancy, 3 with normal and 5 with high redundancy.

Usefu Cluster Troubleshooting Link

https://www.toadworld.com/platforms/oracle/b/weblog/archive/2014/01/28/troubleshooting-oracle-clusterware-common-startup-failures

Next Oracle Database 12c software installation

Useful video link

http://www.oracle.com/webfolder/technetwork/tutorials/demos/db/11g/r2/grid_rac/05_asm_asmca/asm_asmca_viewlet_swf.html

Copy the database files into the Oracle Linux /tmp folder and unzip them

cd /tmp/database
unzip V46095-01_1of2.zip -d /tmp/database
unzip V46095-01_2of2.zip -d /tmp/database

In this section, we will create two additional ASM disk groups using the ASM Configuration Assistant ( asmca). These new ASM disk groups will be used later in this guide when creating the clustered database
Log into Node 1 as grid
Type cd /u01/app/12.1.0.2/grid/bin/asmca and you should now get the below box

From the disk group tab click Create
Put in a name for your Disk Group
Select disk/disks

You should see the confirmation message

Next create the FRA disk group
From the disk group tab click Create
Put in a name for your Disk Group – FRA
Select disk/disks

Make sure you get a confirmation message

You should now see all your disks

Check the cluster services are up before starting the installer
Type /u01/app/12.1.0.2/grid/bin/crsctl status resource -t

Log into the first RAC node as the oracle user and start a terminal session
Type cd /tmp/database/database
./runInstaller

The installer should start

Uncheck the security updates checkbox and click the “Next” button and “Yes” on the subsequent warning dialog.

Select Install Database software only

Accept the “Oracle Real Application Clusters database installation” option by clicking the “Next” button.

Make sure both nodes are selected

Choose your language

Choose your database edition

Enter “/u01/app/oracle” as the Oracle base and “/u01/app/oracle/product/12.1.0.2/db_1” as the software location, then click the “Next” button

It will then run through pre-requisite checking

I got one error to fix

I clicked fix and check again and got the below screen

So I ran the below on both nodes

Then clicked OK on the message and it worked fine and took me to the next screen below

It will then run through the installer
You can click on details to see where it up to

During the install the following message will pop up

I putty’d into each node as root and ran /u01/app/oracle/product/12.1.0.2/db_1/root.sh

Go back to the installer and click on OK
Hopefully now your installation should be successful

Creating a new database

Oracle Link to follow and for explanations and descriptions

http://docs.oracle.com/cd/B16276_01/doc/server.102/b14196/install003.htm

Log into Putty as oracle
You can type dbca or type the path to the runInstaller (instructions paths are below but I used dbca)

If you get the below message you will need to modify the permissions on the /u01/app/oracle/cfgtoollogs/dbca folder

At this point I have found you also need to change the permissions on the /u01/app/oracle/admin folder on both nodes or you will get to the end of the wizard and it will fail straightaway with permissions

You will now see the below screen
Choose Advanced Mode then we can see all the options you get

You then see the below screen

Administrator-managed: Database administrators define on which servers a database resource should run, and place resources manually as needed. This is the management strategy used in previous releases.
Policy managed: Database administrators specify in which server pool (excluding generic or free) the database resource will run. Oracle Clusterware is responsible for placing the database resource on a server.
Put in cdbrac for the name

Good Link for understanding Server Pools

https://martincarstenbach.wordpress.com/2013/06/17/an-introduction-to-policy-managed-databases-in-11-2-rac/

You can run the following command to check what server pools there are
/u01/app/oracle/product/12.1.0.2/db_1/bin/srvctl config srvpool

You will now be at the below screen

Put in a Global Database Name – In my case cdbrac.techlab.local
The SID prefix is cdbrac
Select Create as Container Database
Select Create a Container database with one or more PDBs and name it pdb1
On the Database placement, select Nodes

Next choose your management options

Next set your database credentials
I chose to use the same for all accounts

Click Next
The next screen sets Storage Locations

If you click on File Location variables you will see the below screen

On the Database Schemas page, just leave this as it is

In the Initialisation Parameters window for memory, select Automatic memory management

In the Sizing tab, keep the default

For character sets, just keep the defaults although read the below
Oracle recommends using Unicode for all new system deployments. Migrating legacy systems to Unicode is also recommended. Deploying your systems today in Unicode offers many advantages in usability, compatibility, and extensibility. Oracle Database enables you to deploy high-performing systems faster and more easily while utilizing the advantages of Unicode. Even if you do not need to support multilingual data today, nor have any requirement for Unicode, it is still likely to be the best choice for a new system in the long run and will ultimately save you time and money as well as give you competitive advantages in the long term