What is Transparent Page Sharing?
When multiple virtual machines are running, some of them may have identical sets of memory content. This presents opportunities for sharing memory across virtual machines (as well as sharing within a single virtual machine). For example, several virtual machines may be running the same guest operating system, have the same applications, or contain the same user data. With page sharing, the hypervisor can reclaim the redundant copies and keep only one copy, which is shared by multiple virtual machines in the host physical memory. As a result, the total virtual machine host memory consumption is reduced and a higher level of memory overcommitment is possible.
What is the security problem related to Transparent Page Sharing currently?
There has been recent academic research that leverages Transparent Page Sharing (TPS) to gain unauthorized access to data under certain highly controlled conditions and documents VMware’s precautionary measure of restricting TPS to individual virtual machines by default in upcoming ESXi releases. At this time, VMware believes that the published information disclosure due to TPS between virtual machines is impractical in a real world deployment.
Published academic papers have demonstrated that by forcing a flush and reload of cache memory, it is possible to measure memory timings to try and determine an AES encryption key in use on another virtual machine running on the same physical processor of the host server if Transparent Page Sharing is enabled between the two virtual machines. This technique works only in a highly controlled system configured in a non-standard way that VMware believes would not be recreated in a production environment. .
Even though VMware believes information being disclosed in real world conditions is unrealistic, out of an abundance of caution upcoming ESXi Update releases will no longer enable TPS between Virtual Machines by default (Inter-VM TPS). TPS will still be utilized within individual VMs. (Intra-VM TPS)
What is meant by Intra-VM and Inter-VM in the context of Transparent Page Sharing?
- Intra-VM means that TPS will de-duplicate identical pages of memory within a virtual machine, but will not share the pages with any other virtual machines.
- Inter-VM mean that TPS will de-duplicate identical pages of memory within a virtual machine and will also share the duplicates with one of more other virtual machines with the same content.
VMware will disable the ability to share memory pages “between” virtual machines (Inter-VM Transparent Page Sharing) by default (in ESXi 5.0/5.1 and 5.5) in coming updates and the next major ESXi release and inter-Virtual Machine TPS is not enabled by default as of ESXi 6.0. Administrators may revert to the previous behavior if they so wish.
What could potentially be the effect?
Disabling inter-Virtual Machine TPS may impact performance in environments that rely heavily on memory over-commitment although we still have memory resource management techniques such as
- Ballooning – Reclaims memory by artificially increasing the memory pressure inside the guest
- Hypervisor/Host swapping – Reclaims memory by having ESX directly swap out the virtual machine’s memory
- Memory Compression – Reclaims memory by compressing the pages that need to be swapped out
Please keep reading KB52337 for further information
So what options do we have?
The concept of salting has been introduced to help address concerns system administrators may have over the security implications of TPS. Salting is used to allow more granular management of the virtual machines participating in TPS than was previously possible. As per the original TPS implementation, multiple virtual machines could share pages when the contents of the pages were same. With the new salting settings, the virtual machines can share pages only if the salt value and contents of the pages are identical. A new host config option Mem.ShareForceSalting is introduced to enable or disable salting.
By default, salting is enabled after the ESXi update releases mentioned above are deployed, (Mem.ShareForceSalting=2) and each virtual machine has a different salt. This means page sharing does not occur across the virtual machines (inter-VM TPS) and only happens inside a virtual machine (intra VM).
When salting is enabled (Mem.ShareForceSalting=1 or 2) in order to share a page between two virtual machines both salt and the content of the page must be same. A salt value is a configurable vmx option for each virtual machine. You can manually specify the salt values in the virtual machine’s vmx file with the new vmx option sched.mem.pshare.salt. If this option is not present in the virtual machine’s vmx file, then the value of vc.uuid vmx option is taken as the default value. Since the vc.uuid is unique to each virtual machine, by default TPS happens only among the pages belonging to a particular virtual machine (Intra-VM).
If a group of virtual machines are considered trustworthy, it is possible to share pages among them by setting a common salt value for all those virtual machines (inter-VM).
The following table shows how different settings for TPS are used together to affect how TPS operates for individual virtual machines:
What is the default behavior of Transparent Page Sharing in above mentioned Update releases?
By default, the setting is (Mem.ShareForceSalting=2) and each virtual machine has a different salt (that is sched.mem.pshare.salt is not present) which means that only Intra-VM page sharing is enabled. This behavior is new as per these ESXi update releases and page sharing will not happen across the virtual machines (inter-VM TPS) by default.
How can I enable or disable salting?
- Log in to ESX (i)/vCenter with the VI-Client.
- Select ESX (i) relevant host.
- In the Configuration tab, click Advanced Settings (link) under the software section.
- In the Advanced Settings window, click Mem.
- Search for Mem.ShareForceSalting and set the value to 1 or 2 (enable salting), 0 (disable salting).
- Click OK.
- For the changes to take effect do either of the two:
- Migrate all the virtual machines to another host in cluster and then back to original host. Or
- Shutdown and power-on the virtual machines.
How can I allow inter-VM TPS between two or more virtual machines?
Inter-VM TPS is enabled for two or more virtual machines by enabling salting and by giving them the same salt value.
How can I specify salt value of a virtual machine?
- Power off the virtual machine on which you want to set salt value.
- Right click on virtual machine, click on Edit settings.
- Select options menu, click on General under Advanced section.
- Click on Configuration Parameters
- Click on Add Row, new row will be added.
- On LHS add text sched.mem.pshare.salt and on RHS specify the unique string.
- Power on the virtual machine to take effect of salting.
- Repeat steps 1 to 7 to set the salt value for individuals virtual machine.
What is the difference in behavior of page sharing when MEM_SHARE_FORCE_SALTING value is set to 1 and 2?
MEM_SHARE_FORCE_SALTING 1: By default salt value is taken from sched.mem.pshare.salt. If not specified, falls back to old TPS (inter-VM) behavior by considering salt values for the virtual machine as 0.
MEM_SHARE_FORCE_SALTING 2: By default salt value is taken from vc.uuidz. If it does not exist, then the page sharing algorithm generates random and unique value for salting per virtual machine, which is not configurable by users.
How can I prepare for the ESXi Update releases that no longer allow inter-VM TPS by default?
VMware recommends you to monitor free memory available on the host along with the total ballooned and total swapped memory before deploying the ESXi update releases listed above that disallow inter-VM TPS. Once inter-VM TPS is disallowed, available free memory might drop which further can lead to increased ballooning and swapping. If increased ballooning and swapping activity is observed along with noticeable performance issues, more physical memory can be added on the host or the memory load on the host can be reduced.
To monitor the stats – Run esxtop command:
- Run esxtop on host, click m to switch to memory mode.
- free from PMEM /MB row displays the free memory available on the host.
- curr from MEMCTL/MB row displays the total ballooned memory.
- curr from SWAP/MB row displays the total swapped memory.
How can I enable or disable salting for multiple ESXi hosts?
To enable or disable salting for multiple ESXi hosts. Refer to the attached powercli script in KB2097593
. This script allows toggling pshare salting for update releases.
Usage
.\pshare-salting.ps1 <vcenter IP/hostname> -s -> Enables pshare salting.
.\pshare-salting.ps1 <vcenter IP/hostname> -o -> Turn offs pshare salting and falls back to default TPS behaviour
Links
Are there any tools we are able to use to compare TPS savings before and after disabling Inter-VM transparent page sharing?
There is a PowerShell script (VMware recommended) called the “Host Memory Assessment Tool” to look at shared memory per host, and report it in a tabular form, so you can easily review the current shared memory savings, and the worst case impact in contrast with the free memory on the host. The script uses plink.exe to remotely SSH into each ESXi host and record memory counters using vsish. There is very low risk and impact to the ESXi hosts as it is a read only process
https://www.brianjgraf.com/2015/04/03/assess-impact-tps-vsphere-6/
What the script does:
- Connects to vCenter and enumerates all ESXi hosts
- Allows you to enable SSH on selected hosts
- Generates an assessment report
- Allows you to export the assessment report to .csv
- Allows you to easily turn off SSH again if necessary
This tool would need to be run on an normal existing system with workloads with TPS on and Off to see the different outputs.
https://www.brianjgraf.com/2015/04/03/assess-impact-tps-vsphere-6/
VMware Update (20/03/2018)
The last update and updates going forward on performance impact associated with applying the security patches are now found in https://kb.vmware.com/s/article/52337
Virtualization Layer Mitigations: The latest ESXi patches** and the relevant Intel CPU microcode but without Guest Operating System mitigation patches. These mitigations have a minimal performance impact (< 2%) for most workloads on a representative range of recent Intel Xeon server processors.
Full Stack Mitigations: All levels of mitigation. This includes all virtualization layer mitigations above with the addition of Guest Operating System mitigation patches. As reported in the press, the impact of these mitigations will vary depending on your application. Applications with very heavy system call usage, including those with very high IO rates, will show a more significant impact than their counterparts with lower system call usage. For information regarding the performance impact of Operating System Mitigations on your application, please consult with your Operating system and/or Application vendor. Consistent with our findings above, the virtualization layer mitigations that are part of these full stack mitigations have minimal influence to the overall impact. As a general best practice, we recommend you test the appropriate patches with your applications prior to deploying in production environments.
