Chapter 1: About This Manual

Using AKM to encrypt vSphere VMs

VMware vSphere supports KMIP integration for key management of encrypted vSphere virtual machines (VMs) and virtual disks. Alliance Key Manager supports the KMIP 1.1 operations required by vSphere and vSAN to accomplish key management functions. Customers using vSphere can automatically create a new AES key on AKM using the KMIP interface. vSphere can then use this encryption key to encrypt virtual machines and virtual disk images.

For more information see Chapters 6 and 7 of the vSphere Security guide at:

https://docs.vmware.com/en/VMware-vSphere/6.5/vsphere-esxi-vcenter-server-65-security-guide.pdf

IMPORTANT: The vSphere web console supports two interfaces - one based on Adobe Flash, and another based on HTML5. At the time this manual was written the functions to define vSphere encryption are only available in the Flash version of the web console. You can select the flash version when you open a vSphere web console session:

Who is this for?

This guide is designed to help VMware administrators and project managers use an encryption key stored on Alliance Key Manager as the vSphere data encryption key. An existing encryption key on AKM may be retrieved, or a new encryption key can be created from vSphere and stored on AKM for later retrieval and use.

Other resources

The following documents provide additional information on the installation and use of Alliance Key Manager:

Notices

This product and documentation is covered by U.S. and International copyright law. This product may incorporate software licensed under one or more open source license agreements. Government users please note that this product is provided under restricted government use license controls. Please refer to the AKM End User License Agreement for more information.

Change log

The following table provides information on the changes to this documentation:

Version Date Description
4.6.0.001 7/14/2018 Initial release.
4.6.1.001 5/9/2019 Updated required items checklist.
4.6.1.002 5/20/2019 Updated pre-requisites. Added instructions for Refreshing KMS cert.

 

Chapter 2: Preparation

You will need to complete the following steps before continuing:

  • Review pre-requisites

  • Install and set up the primary AKM server

  • Download Administrative certificates from the AKM server

  • Know the unique hostname assigned to the AKM server during initialization (this will be the prefix of the admin1.zip located in home/admin/downloads following initialization), the IP address of the AKM server, and KMIP port number (the default is 5696)

See below for more information.

IMPORTANT: You should thoroughly review the vSphere Security guide before proceeding. Operations, limitations and best practices are discussed in this guide: https://docs.vmware.com/en/VMware-vSphere/6.5/vsphere-esxi-vcenter-server-65-security-guide.pdf

Pre-requisites

IMPORTANT: vSphere supports a web browser interface with two options: flash or HTML5. At the time this manual was written only the flash interface supports the KMS Cluster definitions and you cannot use the HTML5 interface.

  • VMware vSphere (Enterprise Plus or Platinum edition of version 6.5 and higher) for encryption of virtual machine images

  • VMware vSAN (version 6.6 or higher) for encryption of virtual disk

  • Alliance Key Manager (version 4.6.0 or higher)

VM encryption in vSphere can be configured from the vCenter management application. A vCenter administrator or user must have the following privileges to configure integration with an external Key Management System (KMS):

  • Cryptographic Operations

  • Global.Diagnostics

  • Host.Local operations.Manage user groups

The root vCenter administrator has these privileges, but it is a common practice to assign different privileges to specific user roles. Note that Cryptographic Operations is a set of privileges, which will be individually identified when relevant for specific KMS integration steps.

An AKM server administrator will also need access to AKM certificates, which will be imported into vSphere. The AKM certificates are created during AKM initialization and can be downloaded from the /home/admin/downloads directory.

AKM initialization

After starting Alliance Key Manager start an SSH session to the key manager on the default SSH port. Accept the initial SSH security questions regarding the trust for the connection, and access the initial AKM menu. After changing the password you can configure a primary AKM server. Answer the prompts to establish the PKI certificates and keys required by AKM.

After creating the PKI infrastructure you are asked if you want to create an initial set of unique encryption keys. This is optional as vSphere will create the data encryption keys as needed using the KMIP interface. At any point in the future you can use SSH to access the akm-menu to create encryption keys, or you can use the AKM Administrative Console to create keys. The administrative certificates are created and stored in the /home/admin/downloads directory. You will need these certificates and private keys when setting up vSphere encryption. For more detailed instructions please refer to the AKM for VMware Quick Start Guide. Note that you can initialize the AKM server in the cloud (Azure, AWS) and on the AKM hardware security module in the same way.

Certificates

vSphere and the AKM server use certificates and private keys to establish a secure TLS connection and perform authentication. You will need to install the following certificates and private keys on the vSphere server in order to authenticate with the AKM server:

  • The primary AKM’s certificate authority (CA) certificate in .pem format (AKMRootCACertificate.pem)

  • Administrative (Crypto Officer) certificate (AKMAdminCertificate.pem) and private key (AKMAdminPrivateKey.pem) in .pem format

These certificates and private keys are generated on the initialization of AKM and stored on the AKM server. Administrative certificates (used to send key creation requests and key retrieval requests from vSphere to AKM) are located in a file with the name format . For example, if the server name is `akmmongodb` the name of the zip file would be `akmmongodb_admin1_20180719` (your server name and date may be different). See your platform specific AKM deployment guide for instructions on downloading these certificates.

SECURITY ALERT: Private key files must be protected during creation, distribution, and storage to prevent loss. The loss of these files will compromise the security of the AKM server. Depending on the file format, the private key files may be bundled with a certificate or they may be separate files. Transfer the private key files by sharing them over a secure network, placing them in a password-protected zip file, sending them using sFTP, or another secure method. Use the same level of care you would employ to protect encryption keys, including encryption. In the event the private keys are compromised or lost, you should immediately replace the certificate authority on the AKM server and all client certificates in that chain of trust. See the AKM User Guide for more information.

Server information

You will need the following server information before continuing:

  • The hostname and IP address of the AKM server

  • The port number that AKM has been configured to use for KMIP (the default is 5696)

Checklist

Before continuing, you will need the following items:

  • An administrative (Crypto Officer) client certificate and private key files in .pem format and password for the private key. The default files are AKMAdminCertificate.pem and AKMAdminPrivateKey.pem

  • The IP address or DNS name of the AKM server and the port number it will use for KMIP

 

Chapter 3: Configuring vSphere to Connect to AKM

VM encryption can be configured using vSphere API’s or CLI tools, or using the vCenter web client, which is covered in this document. There are often multiple ways to access many functions in the web client; not all user interface options are described here.

Adding a KMS cluster

The first step is to add a KMS Cluster to vSphere, which requires the Cryptographic operations.Manage key servers privilege. A KMS Cluster contains information about one or more key servers. Since Alliance Key Manager implements real-time key mirroring between the primary key server and one or more secondary, failover key servers, you will define the primary key server in the KMS Cluster and one or more secondary key servers following the primary key server in the same KMS Cluster. There can be more than one cluster, but this is not required. If you define multiple Key Clusters the first Key Cluster is the default. The default cluster designation can be changed at a later time, if more KMS Clusters are added.

KMS Clusters are managed at the vCenter Server level. With a vCenter Server instance selected, the Configure menu exposes a Key Management Servers option:

image alt text

The Add KMS option brings up a dialog to configure the KMS:

image alt text

The cluster name and server alias are arbitrary, but may be referred to later. The network address and port for the external KMS (AKM) are entered, either in dotted IP notation or a domain qualified hostname. The AKM server should be network accessible from the vSphere Data Center.

The proxy values would only be needed if the network configuration requires, which can be determined by a network admin.

Some KMS products require the username and password values, but AKM does not. AKM manages user access to keys via the Common Name (CN) field on the client certificate which is referred to on the AKM server as the User.

When adding the first KMS cluster, a confirmation dialog indicates that it will be the default:

image alt text

Configuring certificates

The next step is to set up the certificates between vSphere and AKM.

When adding a KMS, if vSphere can begin a TLS connection, the external server’s certificate may be shown in a pop-up dialog, with the option to select Trust:

image alt text

Choosing Trust here configures vSphere to allow connections from that external KMS server. If the certificate does not appear, or this step is skipped, the external KMS certificate can be explicitly trusted later with the refresh kms certificate action with the KMS instance selected.

KMS Cert Refresh procedure:

  1. Log in to the vSphere Web Client, and select a vCenter Server system.
  2. Click Configure and select Key Management Servers.
  3. Select the KMS instance with which you want to establish a trusted connection.
  4. To establish the trust relationship, Click All Actions, and select Refresh KMS certificate. In the dialog box that appears, click Trust.

AKM uses mutual client-server authentication, thus trust is not yet established after accepting AKM’s certificate into vSphere:

image alt text

It will be necessary to configure certificates for AKM to trust vSphere as a client by importing an AKM client certificate and private key. The vCenter Server Configure screen shows existing KMS Clusters. A KMS server instance should be selected, rather than a cluster. With a KMS instance selected, choose Establish KMS Trust From the All Actions’ menu.

image alt text

The Establish Trust with KMS option will present a dialog offering several trust methods:

image alt text

For AKM, the method to choose is Upload certificate and private key to upload an AKM administrative certificate and corresponding private key. The certificate to upload should be an AKM administrator role certificate, with an Organizational Unit (OU) value of “akm_admin”. Note that OU corresponds to the group name in AKM. This certificate and key can be downloaded from AKM after initialization. The default files are AKMAdminCertificate.pem and AKMAdminPrivateKey.pem

image alt text

The dialog allows to either paste the .pem file’s text into each respective box, or to run an open file dialog by choosing the Upload file option.

If the earlier step had been completed for vSphere to trust the AKM server certificate, then once the AKM admin certificate and corresponding key are uploaded, vSphere will attempt an initial secure connection to AKM. If successful, the connection status for the KMS (AKM) server instance is shown in green.

image alt text

At this point an initial KMIP request can be seen in the AKM server logs located on AKM at /var/log/townsend/akmaudit.log. For errors please check /var/log/townsend/akmerror.log.

Note that If the earlier step had not been completed to configure vSphere to trust the AKM server certificate, it must be explicitly completed before vSphere can complete the connection. See the “Complete the Trust Setup” in section 7 of the vSphere Security guide.

Enabling encryption on ESXi hosts and VMs

After a KMS cluster is configured for the vCenter server to connect with Alliance Key Manager, individual ESXi hosts must be enabled for encryption before VM instances can be encrypted on that host.

Different permissions are required for the host and VM steps; it is possible that different users will perform these functions. The Cryptographic operations.Register host privilege is required to enable or disable host encryption mode.

An ESXi host’s encryption mode is managed through Security settings. With the host’s Configure tab selected, the System category expands to show Security Profile. After selecting Security Profile, scroll down the details display pane to the bottom to show the Host Encryption Mode. The Edit button allows to enable encryption mode, if not already enabled.

image alt text

The Edit button here enables a simple dialog to set the encryption mode to Enabled. Once Host Encryption Mode is enabled, status is updated in the vCenter Web Client:

image alt text

Encrypting a VM

After a KMS cluster is configured for the vCenter server instance and Host Encryption Mode is enabled on an ESXi host, virtual machines can be encrypted on that ESXi host. Encrypting a VM requires the Cryptographic operations.Encrypt new privilege.

After enabling host mode encryption, vCenter will ask the KMS to create a new one on the KMS, and then retrieves the key. Only the key id is persistently stored on the vSphere host. Virtual machines are encrypted using a dynamically generated data encryption key which is then protected by the key retrieved from the KMS. When encrypting and decrypting VM contents, the actual ESXi key is used transiently and in memory only.

VM encryption is controlled via Storage Policy in the VM configuration. The default storage policy encrypts both the VM files and virtual disk files. Details about creating custom storage policies are described in the vSphere Security guide.

With an encryption enabled ESXi host selected, select a VM to encrypt (the VM must be powered off). From the right-click menu, choose VM Policies, and then Edit VM Storage Policies:

image alt text

The next dialog allows changing the storage policy:

image alt text

After applying the VM Encryption Policy, the virtual disk and certain other VM files are encrypted; this can take a few minutes or longer, depending on the size of the virtual disk. This operation is reported in the Recent Tasks status area of the Web Client:

image alt text

At this point the VM is encrypted and can be powered on. vCenter and ESXi will cooperate in retrieving the encryption key from AKM and decrypting the VM at runtime. When powered off, the VM contents are encrypted at rest.

The VM’s encryption state can also be seen in the VM’s Configure tab:

image alt text

Decrypting a VM

To decrypt an individual VM requires the Cryptographic operations.Decrypt privilege, and uses the same Edit Storage Policies dialog to restore the Datastore Default policy. As with encryption, the decryption operation can take some minutes and the result shown in the Recent Tasks status area.

NOTE: If the host had previously been set for encrypted mode, an earlier key id may be still stored. The key id stored in the host may have the original KMS cluster name appended. Using a different KMS cluster or key may involve rebooting the host. See the section “Disable Host Encryption Mode” in the vSphere Security guide

Key server failover

In the event a key server becomes unavailable for any reason, vSphere will attempt to connect to a secondary, failover key server if one is configured in the KMS Cluster definition. vSphere always uses the first key server definition in the KMS Cluster as the default. If the default key server cannot be contacted, vSphere will attempt to connect to the second key server definition in the KMS Cluster. There can be a 60 second delay between the failed attempt to contact the primary key server, and the attempt to connect to the secondary key server. This delay is not configurable. While most Alliance Key Manager customers define only a primary and one secondary key servers, you can define multiple secondary key servers if needed. vSphere uses the secondary, failover key servers in the order they are defined in the KMS Cluster.

For more information please refer to the vSphere Security guide. You may also find this blog by Mike Foley helpful.

 

Chapter 4: Key Rollover (Key Rotation)

Key rollover in vSphere

Individual ESXi hosts generate their own symmetric data encryption keys (DEK) that are encrypted at rest by the AES Key Encryption Key (KEK) from Alliance Key Manager. See Chapter 6 of the vSphere Security document for more information.

You can perform vSphere key rollover of the data encryption encryption key for a VM or for virtual disks using the vSphere command line interface. See this VMware blog for more information. See especially the sections on “Set-VMEncryptionKey” and “Set-VMDiskEncryptionKey”.

Key rotation for encrypted VMs is not done with the AKM key rollover feature; instead vSphere provides specific procedures that must be used.

 

Chapter 5: Support

Technical support for customers with valid software licenses is available via the website at http://townsendsecurity.com/support/. All others should contact your sales representative.

Please see the Townsend Security Maintenance Policy you received with your purchase for information on fees, online technical support, documentation, license transferability and upgrades, software maintenance, hardware maintenance, customer responsibilities, limitations, disclaimers, lapsed maintenance, enhanced maintenance services, and other topics.