Tuesday, April 18, 2017

Updating vSphere 6.5 with vSphere Update Manager

Update Manager has been and is the tool for upgrading and patching ESXi hosts, virtual appliances and VMs. There was only one caveat: it needed a Windows server to run on. It needed it, because starting with version 6.5, vSphere Update Manager is no longer dependent on Microsoft Windows. Update Manager 6.5 is embedded in vCenter Server Appliance 6.5 and uses the internal vPostgres database. The Windows offering still exists, but now there is a choice between going all Windows or all Linux.

From the point of view of functionality, one of the features Update Manger offers is orchestrated ESXi host updates. Since I got a few questions related to host updates, I will describe in the post how to do the update. In my case I updated from vSphere 6.5 to 6.5a. Not a big step, but enough to prove a point.

I've started by downloading the latest ESXi ISO from VMware site, then logged in vCenter Web Client and uploade the image in Update Manager. In Web Client go to Home - Update Manager - Manage tab - ESXi Images and select Import ESXi Image. Browse to the location of the ISO image and upload it.

After the upload is successful, the image will be displayed in the list imported images. Select it and you can see the software packages that are included in the image:

Now the desired image is in the repository. To use the image, we need to attach it to a baseline. Then the baseline will be attached to the hosts or clusters that we want to update. Right click on the image and select create baseline, type the name of the baseline, optionally add a description and press OK. 

The baseline is displayed in Hosts Baseline tab:

Attach the new baseline to cluster of hosts. In Web Client go to Home - Hosts and Cluster, right click cluster, in the action list go to Update Manager and select Attach Baseline. Select the appropriate baseline an click OK. Next, go to Update Manager tab and press Scan for Updates to check the compliance status of the hosts in the cluster.

After the scan finishes, the Non-Compliant message is displayed and we can start the remediation process. Pressing Remediate starts the wizard. First, select the baseline to apply (there might be other baselines attached to the hosts). 

Then select the targets, in my case I've selected all the hosts in the cluster:

Accept the EULA, go to Adanced options and schedule the remediation to take place during a specific maintenance window or whether to ignore warnings that may appear during update. In this case we'll run the remediation immediately:

On Host remediation options select what to do with the VMs when hosts are put into maintenance mode: leave them on, suspend or power off. Removable media mounted to VMs can be automatically disabled. In case entering maintenance mode fails, we can specify how many times to retry and how long to wait between retries. The settings can be saved as default host remediation options for future upgrades.

Finally, the Cluster remediation options enables changes at cluster level - disable DPM, disable Fault Tolerance, disable HA admission control. Powered off and suspended VMs can be migrated to other hosts in the cluster while a host enters maintenance mode. Finally you may select to enable parallel cluster remediation and to specify how many hosts to process in parallel or let Update Manager decide the number based on the cluster settings. Parallel remediation is applied only on hosts where VMs are powered off or suspended. Also, VSAN cluster allows only for one host to be in maintenance mode at a time. Hosts in VSAN cluster will be updated sequentially.

If remediation is sequential and one of the host fails to enter maintenance mode, Update Manager will report an error and stop the remediation process. When parallel processing is selected, if Update Manager encounters an error, it ignores the host and it continues with the next host in the cluster. 

On the summary page you can run a a pre-check remediation report which will provide information about issues with the environment and will suggest what changes to apply:

Press OK and the remediation process will start. At the end of the process, the status of overall compliance in Update Manger tab will display Compliant. 

Wednesday, March 29, 2017

Veeam Backup and Replication: Offsite Backup Repository

One of the common scenarios for backup infrastructures is to send the local backups to a secondary site. In a Veeam Backup and Replication (VBR) environment, this can be easily done by deploying a backup repository in the secondary location and then configuring backup copy jobs for the backups that you want to be sent offsite. VBR  v9.5 supports 4 types of backup repositories:

  • Windows server with local or direct attached storage - local disk, USB drive, iSCSI/FC LUN
  • Linux server with local, direct attached storage or mounted NFS - local disk, USB drive, iSCSI/FC LUN, NFS mounts
  • CIFS (SMB) share
  • deduplicating storage appliance - only the following three are supported EMC Data Domain, ExaGrid and HPE StoreOnce and there is also Enterprise license required.
For the current implementation, the chosen solution is implemented in a VMware environment across two vCenter Servers. VBR Server and main repository are located in the primary site. In the secondary site a backup repository has been installed on top of a Windows VM. Data mover service is installed in both sites. 

Having a data mover service in secondary site also enables backups directly to secondary site. Now, let's see how to configure the offsite repository.

First we need to deploy the Windows VM. The process is "standard" procedure: reserve an IP address, deploy from template, select VM name, compute resource and storage, customize the guest OS (including joining to AD).

After the VM has been deployed, configure the VM hardware if necessary: repository space depending on the size of the backups and RAM (4GB for OS and up to 4GB for each concurrent backup job).

Once the Windows VM is up, go to VBR management console, backup infrastructure tab and start repository configuration wizard by right clicking on Backup Repository -> Add backup repository.
Add a name and a description for the new repository:

Select the type of repository (Windows, Linux, CIFS, Appliance):

On the repository server list page press "Add new"

This will open a new wizard that configures a new windows server repository. Add DNS name or IP address of the repository server (optionally a description):

Add the credentials to use for connecting to the VM. If you've saved them in the credential manager select them from the drop down list, otherwise click Add button and enter the username and password.

Review the components to be installed and press Apply. The wizard will install VBR components on the repository server and it displays the progress in the window:

A summary page is displayed with info about the target server:

Once server is configured the  new backup repository is displayed in the list of servers. Press Populate button to retrieve all the available storage locations. Select the appropriate storage, press Next.

Configure the repository parameters: backup folder path, maximum number of concurrent tasks, read and write data rates (if necessary). 

Advanced configuration features are realated mostly to storage appliances. Pressing Advanced button allows to select the following:
  • Align backup file data blocks - useful for better deduplication ratios on storage appliances that use constant block size deduplication
  • decompress backup data before storing - achieves better deduplication ratio on most storage appliances at the cost of performance
  • this repository is backed by rotated hard drives - if hard drives are rotated and removed from the server
  • user per-VM backup files - multiple I/O streams per VM will improve performance with storage appliances 
Next, select the mount server and whether to enable vPower NFS or not. Default TCP ports for mount server and vPower NFS could be changed if necessary (press Ports button).

Review the configuration page of the server and press Apply.

The summary page will display the tasks, the progress and their status - creating repository folder, installing components (mount server, vPower NFS), configuring components. 

The following services are installed on the repository server:
  • Veeam Data Mover Service - sends and receives data 
  • Veeam Mount Server - mounts backups and replicas for file-level access
  • Veeam Installer Service - installs, updates and configures VBR components
  • Veeam vPower NFS Service - enables running VMs directly from backup files by "publishing" VM vdmk's from backup files to vPower NFS datastore. The datastore is then mounted on the ESXi host.
Once the process finishes, the new repository appears in the repository list of the VBR console and it is ready to use. Use it as a destination for a copy job or as an offsite backup job.

Wednesday, January 11, 2017

Veeam VMware vSphere Web Client Plug-in

Somehow I keep on getting back to restarting this activity, which is very rewarding, but also time consuming. Since I've accepted a new position with another company, I am reviving the blog. Again. I'm starting to see a pattern here. 

First, let me start with a disclaimer: the posts and opinions herein are my own. Not representing the company I work for, or the vendor, or anything official.

First topic on the list is Veeam vSphere Web Client Plugin. I've chosen this because it integrates Veeam Backup & Replication console (VBR from now on) with vSphere Web Client allowing the virtualization admin to have an understanding of what is going on with "those backups".

The plugin provides the following widgets:
- VMs Overview - details about protected VMs, restore points, source VM size, backup size
- Processed VMs - backed up and replicated
- Repositories - capacity, bakup size, free space
- Jobs statistics - running jobs,  successful jobs, errors

In addition to all the information displayed, it also enables the virtualization admin to run VeeamZIP (full backup file that acts as an independent restore point) and QuickBackup (on-demand incremental backup - needs the existence of a full backup) tasks.

Now let's see how we get the plugin installed and configured.

First, we need to have Veeam Enterprise Manager (EM) installed in the infrastructure. It is not usually installed since EM is utilized to manage multiple VBR installations from a single console. Having only one VBR instance doesn't make a use case for EM, unless you are using encryption in your environment. In this case, EM is used in the encryption/decryption process. There are a few other considerations for Enterprise Manager, one them being deploying vSphere Web Client Plugin. The installation of EM is not covered in this article. It needs a Windows server and SQL database. In our demo lab, EM was installed on the same server with Veeam ONE using the already deployed SQL Express.

After installation, open EM console and login. Go to Configuration and add VBR servers that you will manage from EM:

Next, go to vCenter Server tab and you will see the list of Veeam managed servers populated. Looking at the plugin status you will see "Unknown" meaning no check has been done for that particular vCenter Server:

In the above image, there are 2 servers that have already been configured with the plugin. For those it shows the status "Installed" and the plugin version. In order to install the plugin on a new vCenter Server, select the server and then press Check version. Fill in username and password for connecting to vCenter Server and press ok. The status will change from "Pending" to "Not installed".

You will notice the "Install" button is activated. Press "Install" and wait for the status to change to "Installed":

Next, login to vSphere Web Client and on you should see the plugin on the home page of vCenter Server client:

The plugin displays two tabs - Summary and Settings. The Summary tab contains four widgets:
- VMs Overview

- Jobs statistics
- Processed VMs

- Repositories 

Settings tab allows configuration of EM server, the user account for web plugin to connect to EM and Veeam ONE (if it exists in the infrastructure). 
When Veeam ONE is configured, predefined reports can be accessed directly from the widgets. For example the Repositories widget provides a link to Capacity Planning for Backup Repositories report:

Finally, the web client plugin implements VeeamZIP and Quick Backup functionality in the VM context menu, empowering the virtualization administrator to create full and incremental on demand backups. 

Before being able to run the backup actions, we need to configure the plugin by selecting VeeamZIP to.... Next you configure the destination backup server, repository, retention policy and compression level 

After selecting the desired values, press VeeamZIP button to save the configuration. From now on, you can run backups directly from vSphere Web Client. 

Wednesday, October 22, 2014

Distributed virtual switch IP hash mismatch warning

While migrating a standard switch to a distributed switch I've enabled health check for Teaming and failover. In the migration wizard, there are no errors displayed on impact analysis screen, so I just pressed GO! The host is migrated and I get the following warning: IP Hash mismatch
At this point I get a bit confused - where is that coming from? There is no ether channel configured on any of the nics (at least that's how the physical switches should've been configured). The failover policy is Load Based Teaming. And I start looking at each distributed portgroup - and I find that one of the portgroups is configured with Route based on IP Hash. Found it! I've changed the policy, refreshed the view and the warning disappeared.

Monday, July 1, 2013

TCP ports "greater than 65535"

Let me tell this nice story. And yes, there are no TCP ports above 65535, but earlier today after receiving the credentials, I connected without a second thought to A.B.C.D:73382. I was even happy it worked, because the first time I tried "by mistake" port 63382 and I had problems with the login. And this time it worked. But there is a problem - the port number should have been smaller than 65535. Still, take a look at the following picture and notice the strange TCP/IP connection:

It is TCP port 73382. And the connection works (more or less optimal, but it works). But what actually happens is the interesting part. The connection in the image is a port translation from 73382 to 3389. This means there is a device doing the translation - in our case a linux box. And there is a program listening on port 73382. Well, not exactly...
TCP header uses 16 bits for representing the source port and 16 bits for representing the destination port. 

The 16 bits limit the maximum number of ports to 65536 (2^16) having the numbers between 0 and 65535. Number 73382 needs actually 17 bits for its binary form - 10001111010100110. The only way to fit 17 bits into 16 bits is to truncate the 17 bits. The only question is what part to truncate - left bit or the right bit. To find out the answer I used good old wireshark:

The communication takes place on port 7846 - which on 16 bits looks like this 0001111010100110. So, it truncates the left most bit.What happens is that a small piece of code (perl or python) will successfully  accept to open a socket using numbers bigger 65535 - only it  fits the number on 16 bits. Remember, on a linux system "everything is a file; if it is not file, it is a process".
There are client programs that using a simple input check do not allow using strange numbers, however RDC is not one of them. And after all, it was actually a nice debug.

Thursday, June 20, 2013

Command line upgrade of ESXi 5.1 to 5.1 U1

The reason I`ve chosen this method is not having VUM installed and also wanting to remember a bit of esxcli. The process is pretty simple:
  • download the zip bundle (update-from-esxi5.1-5.1_update01.zip) and put it on a datastore accessible by all hosts (I am using a NFS share)
  • check to see if the host has the datastore mounted, if not - add the datastore
~ # esxcli storage nfs list

~ # esxcli storage nfs add -H -s /mnt/vol1-nfs -v shared1-nfs-sata
  • check the host version
~ # esxcli system version get
   Product: VMware ESXi
   Version: 5.1.0
   Build: Releasebuild-799733
   Update: 0
  • check the update file and see if it requires the host in maintenance mode (in this case, it does not - however since the update will request a system reboot, I think it is better to place the host in maintenance mode and have it cleaned up of VMs)
~ # esxcli software sources vib get --depot=/vmfs/volumes/shared1-nfs-sata/update-from-esxi5.1-5.1_update01.zip
   Maintenance Mode Required: False
  • install the update
~ # esxcli software vib update --depot=/vmfs/volumes/shared1-nfs-sata/update-from-esxi5.1-5.1_update01.zip

Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
  • reboot the system (after all VMs have been migrated to other hosts) and check the new host version
~ # reboot
~ # esxcli system version get
   Product: VMware ESXi
   Version: 5.1.0
   Build: Releasebuild-1065491
   Update: 1

Saturday, June 15, 2013

vCloud Director 5.1 upgrade and missing sysadmin permissions

I`ve recently managed to crash my test lab. One of the rebuild tasks was to reinstall vCloud Director - the hard way: dropping the DB and recreating it. Next I was running the vCloud Director installation script (5.1.1), recreating the DB schema and so on. After a week I`ve decided to upgrade the vCD installation from 5.1.1 to 5.1.2.
After downloading the bin file (vmware-vcloud-director-5.1.2-1068441.bin) from VMware site and uploading it on my vCD cell , I`ve executed the bin and a long list of "Yes" ends up with an error:

Unable to update database statistics. Database user has insufficient privileges to update database statistics. To complete this step manually, run 'EXEC sp_updatestats' as the DBO or a member of the sysadmin fixed server role.

Next, I`ve been looking in the DB server (MSSQL) - logging in with vCD DB admin user (vcddbadmin) and checking if the rights are ok. And they were, since the user was dbowner. I`ve ran 'EXEC sp_updatestats' in query editor and ended up with another error:

Msg 15247, Level 16, State 1, Procedure sp_updatestats, Line 15
User does not have permission to perform this action.

It was stored procedure time: vcddb01 -> Programmability ->Stored Procedures -> System Store Procedures and after a few pages of scrolling down: sys.sp_updatestats. I`ve opened the stored procedure (Context menu - Modify) and on line 15 there was a check to see if the current user was sysadmin. Which, obviously it was not.

Below, a small SQL script that will display if the current user is sysadmin or not:
if is_srvrolemember('sysadmin') = 1
print  ' It is sys admin'
else if is_srvrolemember('sysadmin') = 0
print  ' It is NOT sys admin'

To solve the problem, I`ve added sysadmin role to the vCloud Director DB admin user (vcddbadmin) and ran again the stored procedure:

This time it worked ok and updated database statistics. I`ve checked back the installation procedure for vCloud DB and I`ve taken a look at KB 2034540 to see if I`veforgotten anything while creating the user for vCD DB. But it seems I did not. The only reference is to db_owner role. Since I am not a DB guy, I`ll just leave it as is, happy to have solved my problem.