Thursday, May 4, 2017

Virtual Machine Encryption

A new security feature introduced in vSphere 6.5 is virtual machine encryption. The encryption is VM agnostic as it takes place at hypervisor level before the I/O is stored to disk. It uses vSphere APIs for I/O filtering framework that allows interception of VM I/Os in the virtual SCSI emulation (vSCSI) layer. It encrypts virtual machine files (nvram, vswp), virtual disk files and core dump files. However it does not support log files, VM configuration files or virtual disk descriptor files since these are considered to contain non-sensitive data.

How it works

There are several components necessary to implement VM encryption. The process uses two different sets of keys - key encryption keys (KEKs) and data encryption keys (DEKs). The components are:
  • external key management server (KMS) - generates and stores key encryption keys (KEKs) 
  • vCenter Server - requests KEKs from KMS and distributes them to ESXi hosts; Key Management Interoperability Protocol (KMIP) v1.1 is supported
  • ESXi hosts - generates data encryption keys (DEKs) and encrypts them with KEK; encrypted DEKs are stored in configuration files. DEKs are used to encrypt/decrypt virtual machine files. KEKs need to be in ESXi memory for a VM to be powered on.

Since KEKs are only stored in KMS and they are used to encrypt/decrypt DEKs, KMS should be made highly available. Loosing KMS generated keys means DEKs cannot be decrypted and access to VM data is gone.

Another important aspect is VM encryption uses the data block's address to protect against snooping by generating different encrypted data for identical data blocks. However it does not provide protection against data corruption.

How to configure

First we need to configure KMS solution. For demo purpose I've used the docker container created by William Lam. Please note that the keys for this KMS are held in memory and they will be lost on restart. To configure it, logon to your docker host and start the docker image with KMS by running the following commands:
docker pull lamw/vmwkmip
docker run -d -p 5696:5696 law/lamw/vmwkmip

Check the container is running by executing the following command on the docker host: docker ps. 

Next configure vCenter Server. Login to web client, select the vCenter Server in Hosts view, go to configure tab, Key Management Servers and press Add KMS. In the window add the KMS cluster name, server alias, server address, TCP port and optionally proxy details:

Press Yes to set the KMS as your default KMS cluster:

Trust the certificate presented by KMS:

vCenter Server is now configured to use KMS and the details are displayed in the web client:

To encrypt a VM becomes a matter of applying the correct storage policy to the VM. Before applying the encryption policy make sure the VM is powered off, otherwise you will get the following error:

To change the storage policy, in web client right click the VM you want to encrypt, go to VM Policies -> Edit VM Storage Policies. Change the default policy with VM Encrpyption Policy (which is the default encryption policy) and press OK button:

The encryption process will take some time. You can monitor it in events log:

Once the process is finished you can power on the VM. On the summary tab of the VM you can also check that the VM is encrypted:

Access control
Since cryptography is used when one needs to restrict access to certain data, the question is do we need all admins to have access to cryptographic functions in vCenter Server. To restrict access, a new role has been introduced - No cryptography administrator. It does not have the following privileges: 

  • Cryptographic Operations
  • Global.Diagnostics
  • Host.Inventory.Add host to cluster
  • Host.Inventory.Add standalone host
  • Host.Local operations.Manage user groups

To further restrict the access, the role can be cloned and modified accordingly.

The are restrictions and limitations when VM encryption is being used. One of the most important is that backup solutions using VMware vSphere Storage API - Data Protection are restricted to hot add backup and NBD-SSL network transport mode. SAN backup is not supported.

VMs with existing snapshots cannot by encrypted. All snapshots must be first consolidated. Guest memory cannot be saved during a snapshot of encrypted VMs.

If you are looking for details on performance impact of VM encryption, there is a performance study from VMware. Dedicated encryption hardware is not necessary, but using a processor that support AES-NI instruction set will speed up encryption and decryption.

Tuesday, April 18, 2017

Updating vSphere 6.5 with vSphere Update Manager

Update Manager has been and is the tool for upgrading and patching ESXi hosts, virtual appliances and VMs. There was only one caveat: it needed a Windows server to run on. It needed it, because starting with version 6.5, vSphere Update Manager is no longer dependent on Microsoft Windows. Update Manager 6.5 is embedded in vCenter Server Appliance 6.5 and uses the internal vPostgres database. The Windows offering still exists, but now there is a choice between going all Windows or all Linux.

From the point of view of functionality, one of the features Update Manger offers is orchestrated ESXi host updates. Since I got a few questions related to host updates, I will describe in the post how to do the update. In my case I updated from vSphere 6.5 to 6.5a. Not a big step, but enough to prove a point.

I've started by downloading the latest ESXi ISO from VMware site, then logged in vCenter Web Client and uploade the image in Update Manager. In Web Client go to Home - Update Manager - Manage tab - ESXi Images and select Import ESXi Image. Browse to the location of the ISO image and upload it.

After the upload is successful, the image will be displayed in the list imported images. Select it and you can see the software packages that are included in the image:

Now the desired image is in the repository. To use the image, we need to attach it to a baseline. Then the baseline will be attached to the hosts or clusters that we want to update. Right click on the image and select create baseline, type the name of the baseline, optionally add a description and press OK. 

The baseline is displayed in Hosts Baseline tab:

Attach the new baseline to cluster of hosts. In Web Client go to Home - Hosts and Cluster, right click cluster, in the action list go to Update Manager and select Attach Baseline. Select the appropriate baseline an click OK. Next, go to Update Manager tab and press Scan for Updates to check the compliance status of the hosts in the cluster.

After the scan finishes, the Non-Compliant message is displayed and we can start the remediation process. Pressing Remediate starts the wizard. First, select the baseline to apply (there might be other baselines attached to the hosts). 

Then select the targets, in my case I've selected all the hosts in the cluster:

Accept the EULA, go to Adanced options and schedule the remediation to take place during a specific maintenance window or whether to ignore warnings that may appear during update. In this case we'll run the remediation immediately:

On Host remediation options select what to do with the VMs when hosts are put into maintenance mode: leave them on, suspend or power off. Removable media mounted to VMs can be automatically disabled. In case entering maintenance mode fails, we can specify how many times to retry and how long to wait between retries. The settings can be saved as default host remediation options for future upgrades.

Finally, the Cluster remediation options enables changes at cluster level - disable DPM, disable Fault Tolerance, disable HA admission control. Powered off and suspended VMs can be migrated to other hosts in the cluster while a host enters maintenance mode. Finally you may select to enable parallel cluster remediation and to specify how many hosts to process in parallel or let Update Manager decide the number based on the cluster settings. Parallel remediation is applied only on hosts where VMs are powered off or suspended. Also, VSAN cluster allows only for one host to be in maintenance mode at a time. Hosts in VSAN cluster will be updated sequentially.

If remediation is sequential and one of the host fails to enter maintenance mode, Update Manager will report an error and stop the remediation process. When parallel processing is selected, if Update Manager encounters an error, it ignores the host and it continues with the next host in the cluster. 

On the summary page you can run a a pre-check remediation report which will provide information about issues with the environment and will suggest what changes to apply:

Press OK and the remediation process will start. At the end of the process, the status of overall compliance in Update Manger tab will display Compliant. 

Wednesday, March 29, 2017

Veeam Backup and Replication: Offsite Backup Repository

One of the common scenarios for backup infrastructures is to send the local backups to a secondary site. In a Veeam Backup and Replication (VBR) environment, this can be easily done by deploying a backup repository in the secondary location and then configuring backup copy jobs for the backups that you want to be sent offsite. VBR  v9.5 supports 4 types of backup repositories:

  • Windows server with local or direct attached storage - local disk, USB drive, iSCSI/FC LUN
  • Linux server with local, direct attached storage or mounted NFS - local disk, USB drive, iSCSI/FC LUN, NFS mounts
  • CIFS (SMB) share
  • deduplicating storage appliance - only the following three are supported EMC Data Domain, ExaGrid and HPE StoreOnce and there is also Enterprise license required.
For the current implementation, the chosen solution is implemented in a VMware environment across two vCenter Servers. VBR Server and main repository are located in the primary site. In the secondary site a backup repository has been installed on top of a Windows VM. Data mover service is installed in both sites. 

Having a data mover service in secondary site also enables backups directly to secondary site. Now, let's see how to configure the offsite repository.

First we need to deploy the Windows VM. The process is "standard" procedure: reserve an IP address, deploy from template, select VM name, compute resource and storage, customize the guest OS (including joining to AD).

After the VM has been deployed, configure the VM hardware if necessary: repository space depending on the size of the backups and RAM (4GB for OS and up to 4GB for each concurrent backup job).

Once the Windows VM is up, go to VBR management console, backup infrastructure tab and start repository configuration wizard by right clicking on Backup Repository -> Add backup repository.
Add a name and a description for the new repository:

Select the type of repository (Windows, Linux, CIFS, Appliance):

On the repository server list page press "Add new"

This will open a new wizard that configures a new windows server repository. Add DNS name or IP address of the repository server (optionally a description):

Add the credentials to use for connecting to the VM. If you've saved them in the credential manager select them from the drop down list, otherwise click Add button and enter the username and password.

Review the components to be installed and press Apply. The wizard will install VBR components on the repository server and it displays the progress in the window:

A summary page is displayed with info about the target server:

Once server is configured the  new backup repository is displayed in the list of servers. Press Populate button to retrieve all the available storage locations. Select the appropriate storage, press Next.

Configure the repository parameters: backup folder path, maximum number of concurrent tasks, read and write data rates (if necessary). 

Advanced configuration features are realated mostly to storage appliances. Pressing Advanced button allows to select the following:
  • Align backup file data blocks - useful for better deduplication ratios on storage appliances that use constant block size deduplication
  • decompress backup data before storing - achieves better deduplication ratio on most storage appliances at the cost of performance
  • this repository is backed by rotated hard drives - if hard drives are rotated and removed from the server
  • user per-VM backup files - multiple I/O streams per VM will improve performance with storage appliances 
Next, select the mount server and whether to enable vPower NFS or not. Default TCP ports for mount server and vPower NFS could be changed if necessary (press Ports button).

Review the configuration page of the server and press Apply.

The summary page will display the tasks, the progress and their status - creating repository folder, installing components (mount server, vPower NFS), configuring components. 

The following services are installed on the repository server:
  • Veeam Data Mover Service - sends and receives data 
  • Veeam Mount Server - mounts backups and replicas for file-level access
  • Veeam Installer Service - installs, updates and configures VBR components
  • Veeam vPower NFS Service - enables running VMs directly from backup files by "publishing" VM vdmk's from backup files to vPower NFS datastore. The datastore is then mounted on the ESXi host.
Once the process finishes, the new repository appears in the repository list of the VBR console and it is ready to use. Use it as a destination for a copy job or as an offsite backup job.

Wednesday, January 11, 2017

Veeam VMware vSphere Web Client Plug-in

Somehow I keep on getting back to restarting this activity, which is very rewarding, but also time consuming. Since I've accepted a new position with another company, I am reviving the blog. Again. I'm starting to see a pattern here. 

First, let me start with a disclaimer: the posts and opinions herein are my own. Not representing the company I work for, or the vendor, or anything official.

First topic on the list is Veeam vSphere Web Client Plugin. I've chosen this because it integrates Veeam Backup & Replication console (VBR from now on) with vSphere Web Client allowing the virtualization admin to have an understanding of what is going on with "those backups".

The plugin provides the following widgets:
- VMs Overview - details about protected VMs, restore points, source VM size, backup size
- Processed VMs - backed up and replicated
- Repositories - capacity, bakup size, free space
- Jobs statistics - running jobs,  successful jobs, errors

In addition to all the information displayed, it also enables the virtualization admin to run VeeamZIP (full backup file that acts as an independent restore point) and QuickBackup (on-demand incremental backup - needs the existence of a full backup) tasks.

Now let's see how we get the plugin installed and configured.

First, we need to have Veeam Enterprise Manager (EM) installed in the infrastructure. It is not usually installed since EM is utilized to manage multiple VBR installations from a single console. Having only one VBR instance doesn't make a use case for EM, unless you are using encryption in your environment. In this case, EM is used in the encryption/decryption process. There are a few other considerations for Enterprise Manager, one them being deploying vSphere Web Client Plugin. The installation of EM is not covered in this article. It needs a Windows server and SQL database. In our demo lab, EM was installed on the same server with Veeam ONE using the already deployed SQL Express.

After installation, open EM console and login. Go to Configuration and add VBR servers that you will manage from EM:

Next, go to vCenter Server tab and you will see the list of Veeam managed servers populated. Looking at the plugin status you will see "Unknown" meaning no check has been done for that particular vCenter Server:

In the above image, there are 2 servers that have already been configured with the plugin. For those it shows the status "Installed" and the plugin version. In order to install the plugin on a new vCenter Server, select the server and then press Check version. Fill in username and password for connecting to vCenter Server and press ok. The status will change from "Pending" to "Not installed".

You will notice the "Install" button is activated. Press "Install" and wait for the status to change to "Installed":

Next, login to vSphere Web Client and on you should see the plugin on the home page of vCenter Server client:

The plugin displays two tabs - Summary and Settings. The Summary tab contains four widgets:
- VMs Overview

- Jobs statistics
- Processed VMs

- Repositories 

Settings tab allows configuration of EM server, the user account for web plugin to connect to EM and Veeam ONE (if it exists in the infrastructure). 
When Veeam ONE is configured, predefined reports can be accessed directly from the widgets. For example the Repositories widget provides a link to Capacity Planning for Backup Repositories report:

Finally, the web client plugin implements VeeamZIP and Quick Backup functionality in the VM context menu, empowering the virtualization administrator to create full and incremental on demand backups. 

Before being able to run the backup actions, we need to configure the plugin by selecting VeeamZIP to.... Next you configure the destination backup server, repository, retention policy and compression level 

After selecting the desired values, press VeeamZIP button to save the configuration. From now on, you can run backups directly from vSphere Web Client. 

Wednesday, October 22, 2014

Distributed virtual switch IP hash mismatch warning

While migrating a standard switch to a distributed switch I've enabled health check for Teaming and failover. In the migration wizard, there are no errors displayed on impact analysis screen, so I just pressed GO! The host is migrated and I get the following warning: IP Hash mismatch
At this point I get a bit confused - where is that coming from? There is no ether channel configured on any of the nics (at least that's how the physical switches should've been configured). The failover policy is Load Based Teaming. And I start looking at each distributed portgroup - and I find that one of the portgroups is configured with Route based on IP Hash. Found it! I've changed the policy, refreshed the view and the warning disappeared.

Monday, July 1, 2013

TCP ports "greater than 65535"

Let me tell this nice story. And yes, there are no TCP ports above 65535, but earlier today after receiving the credentials, I connected without a second thought to A.B.C.D:73382. I was even happy it worked, because the first time I tried "by mistake" port 63382 and I had problems with the login. And this time it worked. But there is a problem - the port number should have been smaller than 65535. Still, take a look at the following picture and notice the strange TCP/IP connection:

It is TCP port 73382. And the connection works (more or less optimal, but it works). But what actually happens is the interesting part. The connection in the image is a port translation from 73382 to 3389. This means there is a device doing the translation - in our case a linux box. And there is a program listening on port 73382. Well, not exactly...
TCP header uses 16 bits for representing the source port and 16 bits for representing the destination port. 

The 16 bits limit the maximum number of ports to 65536 (2^16) having the numbers between 0 and 65535. Number 73382 needs actually 17 bits for its binary form - 10001111010100110. The only way to fit 17 bits into 16 bits is to truncate the 17 bits. The only question is what part to truncate - left bit or the right bit. To find out the answer I used good old wireshark:

The communication takes place on port 7846 - which on 16 bits looks like this 0001111010100110. So, it truncates the left most bit.What happens is that a small piece of code (perl or python) will successfully  accept to open a socket using numbers bigger 65535 - only it  fits the number on 16 bits. Remember, on a linux system "everything is a file; if it is not file, it is a process".
There are client programs that using a simple input check do not allow using strange numbers, however RDC is not one of them. And after all, it was actually a nice debug.