Chapter 1: About This Manual
Using AKM to encrypt vSphere VMs
VMware vSphere supports KMIP integration for key management of encrypted vSphere virtual machines (VMs) and virtual disks. Alliance Key Manager supports the KMIP 1.1 operations required by vSphere and vSAN to accomplish key management functions. Customers using vSphere can automatically create a new AES key on AKM using the KMIP interface. vSphere can then use this encryption key to encrypt virtual machines and virtual disk images.
For more information see Chapters 6 and 7 of the vSphere Security guide at:
Who is this for?
This guide is designed to help VMware administrators and project managers use an encryption key stored on Alliance Key Manager as the vSphere data encryption key. An existing encryption key on AKM may be retrieved, or a new encryption key can be created from vSphere and stored on AKM for later retrieval and use.
The following documents provide additional information on the installation and use of Alliance Key Manager:
This product and documentation is covered by U.S. and International copyright law. This product may incorporate software licensed under one or more open source license agreements. Government users please note that this product is provided under restricted government use license controls. Please refer to the AKM End User License Agreement for more information.
The following table provides information on the changes to this documentation:
|4.6.1.001||5/9/2019||Updated required items checklist.|
|4.6.1.002||5/20/2019||Updated pre-requisites. Added instructions for Refreshing KMS cert.|
|4.6.1.003||7/25/2019||Added vSAN Configuration & Updated Security Guide|
|4.6.1.004||7/1/2020||Added note to ‘Failover” section regarding bidirectional mirroring.|
Chapter 2: Preparation
You will need to complete the following steps before continuing:
Install and set up the primary AKM server
Download Administrative certificates from the AKM server
Know the unique hostname assigned to the AKM server during initialization (this will be the prefix of the admin1.zip located in home/admin/downloads following initialization), the IP address of the AKM server, and KMIP port number (the default is 5696)
See below for more information.
IMPORTANT: You should thoroughly review the vSphere Security guide before proceeding. Operations, limitations and best practices are discussed in this guide: https://docs.vmware.com/en/VMware-vSphere/6.7/vsphere-esxi-vcenter-server-67-security-guide.pdf
VMware vSphere (Enterprise Plus or Platinum edition of version 6.5 and higher) for encryption of virtual machine images
VMware vSAN (version 6.6 or higher) for encryption of virtual disk
vTPM is available starting with vCenter Server 6.7
Alliance Key Manager (version 4.6.0 or higher)
VM encryption in vSphere can be configured from the vCenter management application. A vCenter administrator or user must have the following privileges to configure integration with an external Key Management System (KMS):
Host.Local operations.Manage user groups
To enable encryption on a vSAN cluster you will need the following priviliges:
Host. Inventory. Modify Cluster
Cryptographic Operations. Manage KMS
Cryptographic Operations. Manage Encryption Polices
Cryptographic Operations. Manage Keys
The vCenter administrator account has these privileges, but it is a common practice to assign different privileges to specific user roles. Note that
Cryptographic Operations is a set of privileges, which will be individually identified when relevant for specific KMS integration steps.
An AKM server administrator will also need access to AKM certificates, which will be imported into vSphere. The AKM certificates are created during AKM initialization and can be downloaded from the
After starting Alliance Key Manager start an SSH session to the key manager on the default SSH port. Accept the initial SSH security questions regarding the trust for the connection, and access the initial AKM menu. After changing the password you can configure a primary AKM server. Answer the prompts to establish the PKI certificates and keys required by AKM.
After creating the PKI infrastructure you are asked if you want to create an initial set of unique encryption keys. This is optional as vSphere will create the data encryption keys as needed using the KMIP interface. At any point in the future you can use SSH to access the akm-menu to create encryption keys, or you can use the AKM Administrative Console to create keys. The administrative certificates are created and stored in the
/home/admin/downloads directory. You will need these certificates and private keys when setting up vSphere encryption. For more detailed instructions please refer to the AKM for VMware Quick Start Guide. Note that you can initialize the AKM server in the cloud (Azure, AWS) and on the AKM hardware security module in the same way.
vSphere and the AKM server use certificates and private keys to establish a secure TLS connection and perform authentication. You will need to install the following certificates and private keys on the vSphere server in order to authenticate with the AKM server:
The primary AKM’s certificate authority (CA) certificate in
Administrative (Crypto Officer) certificate (
AKMAdminCertificate.pem) and private key (
These certificates and private keys are generated on the initialization of AKM and stored on the AKM server. Administrative certificates (used to send key creation requests and key retrieval requests from vSphere to AKM) are located in a file with the name format
SECURITY ALERT: Private key files must be protected during creation, distribution, and storage to prevent loss. The loss of these files will compromise the security of the AKM server. Depending on the file format, the private key files may be bundled with a certificate or they may be separate files. Transfer the private key files by sharing them over a secure network, placing them in a password-protected zip file, sending them using sFTP, or another secure method. Use the same level of care you would employ to protect encryption keys, including encryption. In the event the private keys are compromised or lost, you should immediately replace the certificate authority on the AKM server and all client certificates in that chain of trust. See the AKM User Guide for more information.
You will need the following server information before continuing:
The hostname and IP address of the AKM server
The port number that AKM has been configured to use for KMIP (the default is 5696)
Before continuing, you will need the following items:
An administrative (Crypto Officer) client certificate and private key files in
.pemformat and password for the private key. The default files are
The IP address or DNS name of the AKM server and the port number it will use for KMIP
Chapter 3: Configuring vSphere to Connect to AKM
VM encryption can be configured using vSphere API’s or CLI tools, or using the vCenter web client, which is covered in this document. There are often multiple ways to access many functions in the web client; not all user interface options are described here.
Adding a KMS cluster
The first step is to add a
KMS Cluster to vSphere, which requires the
Cryptographic operations.Manage key servers privilege. A KMS Cluster contains information about one or more key servers. Since Alliance Key Manager implements real-time key mirroring between the primary key server and one or more secondary, failover key servers, you will define the primary key server in the KMS Cluster and one or more secondary key servers following the primary key server in the same KMS Cluster. There can be more than one cluster, but this is not required. If you define multiple Key Clusters the first Key Cluster is the default. The default cluster designation can be changed at a later time, if more KMS Clusters are added.
KMS Clusters are managed at the vCenter Server level. With a vCenter Server instance selected, the
Configure menu exposes a
Key Management Servers option:
Add KMS option brings up a dialog to configure the KMS:
The cluster name and server alias are arbitrary, but may be referred to later. The network address and port for the external KMS (AKM) are entered, either in dotted IP notation or a domain qualified hostname. The AKM server should be network accessible from the vSphere Data Center.
The proxy values would only be needed if the network configuration requires, which can be determined by a network admin.
Some KMS products require the username and password values. AKM uses certificates to verify trust, entering a username and password at this stage will cause the process to fail.
When adding the first KMS cluster, a confirmation dialog indicates that it will be the default:
The next step is to set up the certificates between vSphere and AKM.
When adding a KMS, if vSphere can begin a TLS connection, the external server’s certificate may be shown in a pop-up dialog, with the option to select
Trust here configures vSphere to allow connections from that external KMS server. If the certificate does not appear, or this step is skipped, the external KMS certificate can be explicitly trusted later with the
refresh kms certificate action with the KMS instance selected.
KMS Cert Refresh procedure:
- Log in to the vSphere Web Client, and select a vCenter Server system.
Key Management Servers.
- Select the KMS instance with which you want to establish a trusted connection.
- To establish the trust relationship, Click
All Actions, and select
Refresh KMS certificate. In the dialog box that appears, click
AKM uses mutual client-server authentication, thus trust is not yet established after accepting AKM’s certificate into vSphere:
It will be necessary to configure certificates for AKM to trust vSphere as a client by importing an AKM client certificate and private key. The vCenter Server
Configure screen shows existing KMS Clusters. A KMS server instance should be selected, rather than a cluster. With a KMS instance selected, choose
Establish KMS Trust From the All Actions’ menu.
Establish Trust with KMS option will present a dialog offering several trust methods:
For AKM, the method to choose is
Upload certificate and private key to upload an AKM administrative certificate and corresponding private key. The certificate to upload should be an AKM administrator role certificate, with an Organizational Unit (OU) value of “akm_admin”. Note that OU corresponds to the group name in AKM. This certificate and key can be downloaded from AKM after initialization. The default files are
The dialog allows to either paste the
.pem file’s text into each respective box, or to run an open file dialog by choosing the
Upload file option.
If the earlier step had been completed for vSphere to trust the AKM server certificate, then once the AKM admin certificate and corresponding key are uploaded, vSphere will attempt an initial secure connection to AKM. If successful, the connection status for the KMS (AKM) server instance is shown in green.
At this point an initial KMIP request can be seen in the AKM server logs located on AKM at /var/log/townsend/akmaudit.log. For errors please check /var/log/townsend/akmerror.log.
Note that If the earlier step had not been completed to configure vSphere to trust the AKM server certificate, it must be explicitly completed before vSphere can complete the connection. See the “Complete the Trust Setup” in section 7 of the vSphere Security guide.
Chapter 4: VM/vSan Encryption and vTPM
Enabling encryption on ESXi hosts and VMs
After a KMS cluster is configured for the vCenter server to connect with Alliance Key Manager, individual ESXi hosts must be enabled for encryption before VM instances can be encrypted on that host.
Different permissions are required for the host and VM steps; it is possible that different users will perform these functions. The
Cryptographic operations.Register host privilege is required to enable or disable host encryption mode.
An ESXi host’s encryption mode is managed through Security settings. With the host’s Configure tab selected, the
System category expands to show
Security Profile. After selecting Security Profile, scroll down the details display pane to the bottom to show the
Host Encryption Mode. The
Edit button allows to enable encryption mode, if not already enabled.
Edit button here enables a simple dialog to set the encryption mode to
Enabled. Once Host Encryption Mode is enabled, status is updated in the vCenter Web Client:
Encrypting a VM
After a KMS cluster is configured for the vCenter server instance and Host Encryption Mode is enabled on an ESXi host, virtual machines can be encrypted on that ESXi host. Encrypting a VM requires the
Cryptographic operations.Encrypt new privilege.
After enabling host mode encryption, vCenter will ask the KMS to create a new one on the KMS, and then retrieves the key. Only the key id is persistently stored on the vSphere host. Virtual machines are encrypted using a dynamically generated data encryption key which is then protected by the key retrieved from the KMS. When encrypting and decrypting VM contents, the actual ESXi key is used transiently and in memory only.
VM encryption is controlled via Storage Policy in the VM configuration. The default storage policy encrypts both the VM files and virtual disk files. Details about creating custom storage policies are described in the vSphere Security guide.
With an encryption enabled ESXi host selected, select a VM to encrypt (the VM must be powered off). From the right-click menu, choose
VM Policies, and then
Edit VM Storage Policies:
The next dialog allows changing the storage policy:
After applying the VM Encryption Policy, the virtual disk and certain other VM files are encrypted; this can take a few minutes or longer, depending on the size of the virtual disk. This operation is reported in the
Recent Tasks status area of the Web Client:
At this point the VM is encrypted and can be powered on. vCenter and ESXi will cooperate in retrieving the encryption key from AKM and decrypting the VM at runtime. When powered off, the VM contents are
encrypted at rest.
The VM’s encryption state can also be seen in the VM’s
Decrypting a VM
To decrypt an individual VM requires the
Cryptographic operations.Decrypt privilege, and uses the same Edit Storage Policies dialog to restore the
Datastore Default policy. As with encryption, the decryption operation can take some minutes and the result shown in the
Recent Tasks status area.
NOTE: If the host had previously been set for encrypted mode, an earlier key id may be still stored. The key id stored in the host may have the original KMS cluster name appended. Using a different KMS cluster or key may involve rebooting the host. See the section “Disable Host Encryption Mode” in the vSphere Security guide
Enabling encryption on vSAN Cluster
After a KMS cluster is configured for the vCenter server to connect with Alliance Key Manager, individual vSAN clusters must have encryption enabled.
First, make sure that you have the correct permissions to enable encryption on a vSAN cluster.
- Navigate to the vSAN host cluster.
- Click the Configure tab.
- Under vSAN, select Services.
- Click the Encryption Edit button.
- On the vSAN Services dialog, enable Encryption, and select a KMS cluster.
- (Optional) If the storage devices in your cluster contain sensitive data, select Erase Disks Before Use. This setting directs vSAN to wipe existing data from the storage devices as they are encrypted. This option can increase the time to process each disk, so do not choose it unless you have unwanted data on the disks.
Finally Click Apply.
vTPMs perform cryptographic coprocessor capabilities in software. When added to a virtual machine, a vTPM enables the guest operating system to create and store keys that are private. These keys are not exposed to the guest operating system itself. Therefore, the virtual machine attack surface is reduced. Usually, compromising the guest operating system compromises its secrets, but enabling a vTPM greatly reduces this risk. These keys can be used only by the guest operating system for encryption or signing. With an attached vTPM, a third party can remotely attest to (validate) the identity of the firmware and the guest operating system.
You can add a vTPM to either a new virtual machine or an existing virtual machine. A vTPM depends on virtual machine encryption to secure vital TPM data. When you configure a vTPM, VM encryption automatically encrypts the virtual machine files but not the disks. You can choose to add encryption explicitly for the virtual machine and its disks.
You can also back up a virtual machine enabled with a vTPM. The backup must include all virtual machine data, including the *.nvram file. If your backup does not include the *.nvram file, you cannot restore a virtual machine with a vTPM. Also, because the VM home files of a vTPM-enabled virtual machine are encrypted, ensure that the encryption keys are available at the time of a restore.
To use a vTPM your VM must meet the following requirements:
- EFI Firmware
- Hardware version 14
- Be running a supported OS
- Windows Server 2016 (64 bit)
- Windows 10 (64 bit)
NOTE: A vTPM does not require a physical Trusted Platform Module (TPM) 2.0 chip to be present on the ESXi host. However, if you want to perform host attestation, an external entity, such as a TPM 2.0 physical chip, is required. For more details, see the vSphere Security documentation.
Key server failover
In the event a key server becomes unavailable for any reason, vSphere will attempt to connect to a secondary, failover key server if one is configured in the KMS Cluster definition. vSphere always uses the first key server definition in the KMS Cluster as the default. If the default key server cannot be contacted, vSphere will attempt to connect to the second key server definition in the KMS Cluster. There can be a 60 second delay between the failed attempt to contact the primary key server, and the attempt to connect to the secondary key server. This delay is not configurable. While most Alliance Key Manager customers define only a primary and one secondary key servers, you can define multiple secondary key servers if needed. vSphere uses the secondary, failover key servers in the order they are defined in the KMS Cluster.
The best way to configure redundant keyservers in your KMS Cluster is to take advantage of vCenter’s VM cloning ability and using AKM Admin Console to configure mirroring between the new key server and any active keyservers in your KMS Cluster. It’s important to note that this is different from the steps outlined in our mirroring guide.
- Do not
- Initialize a secondary AKM
- Configure mirroring through the AKM Shell Menu
To add an additional keyserver you must:
- Use vCenter to create a clone of your primary keyserver
- Configure a new IP address for the clone. Follow the instructions in the Server Management Guide
- At this point you’ll need to contact us for a license.
- Add the keyserver to your KMS Cluster. Like we did in Chapter 3
- Using the AKM Admin Console you will Set Mirror Address. We recommend that the mirroring be set up bi-directionally to allow keys to sync both ways.
- Use the List Mirror Names command on each keyserver to verify that the other servers in the cluster are listed as mirrors.
- Finally, connect to the primary keyserver with the Admin Console and Force Key Sync specifying the name of the newly configured mirror.
- NOTE: Force sync is only required when adding a new AKM to the KMS cluster. Keys created after mirroring has been configured will automatically be available on each keyserver.
Chapter 5: Key Rollover (Key Rotation)
** Key rotation for encrypted VMs is not done with the AKM key rollover feature; instead vSphere provides specific procedures that must be used. **
Key rollover in vSphere
There are two levels of key rollover: * Shallow Re-Key: A new KEK(Key Encryption Key) is supplied from the KMS and each DEK(Data Encrytpion Key) is re-encrypted using the new KEK. * Deep Re-Key: A new DEK is generated and all data is re-encrypted.
You can perform vSphere key rollover of the data encryption key for a VM or for virtual disks using the vSphere command line interface. See this VMware blog for more information. See especially the sections on “Set-VMEncryptionKey” and “Set-VMDiskEncryptionKey”.
vSAN key rollovers can be managed through the vSphere Client.
Chapter 6: Support
Technical support for customers with valid software licenses is available via the website at http://townsendsecurity.com/support/. All others should contact your sales representative.
Please see the Townsend Security Maintenance Policy you received with your purchase for information on fees, online technical support, documentation, license transferability and upgrades, software maintenance, hardware maintenance, customer responsibilities, limitations, disclaimers, lapsed maintenance, enhanced maintenance services, and other topics.