To stretch or not to stretch?

In recent projects the use of stretched cluster solutions has been a recurring topic. But why is it a recurring topic, and what are the benefits and drawbacks?

The primary benefit of a stretched cluster solution is to enable active-active and workload balanced data centers. The solution has the ability to migrate virtual machines between sites, which enables cross-site mobility of workloads. To be more specific, stretched cluster solutions offer the following benefits:

  • Workload mobility
  • Cross-site automated load balancing
  • Enhanced downtime avoidance
  • Disaster avoidance

While the first two benefits are almost self-explanatory, I want to dive a little deeper into the disaster avoidance aspects. While most customers see a stretched cluster solution as a disaster recovery solution, this may not be entirely true.

Disaster Avoidance vs Disaster Recovery

While generally explained as the same, disaster avoidance and disaster recovery are two different things.

Disaster avoidance focuses on keeping the service available while avoiding an impending disaster. This can be done through the use of availability features in the application or infrastructure, such as vSphere vMotion.

Disaster recovery focuses on getting the service available after a disaster has occurred in a timely and orderly fashion. This can be done through the use of  availability features in the application or infrastructure, such as vSphere HA, or SRM.

But how do these reflect on a failure/disaster?

Host Level

  • Disaster avoidance = vMotion to avoid disaster and outage (non-disruptive)
  • Disaster recovery = vSphere HA restarts virtual machines (disruptive)

Site Level

  • Disaster avoidance = vMotion over distance to avoid disaster and outage (non-disruptive)
  • Disaster recovery = VMware SRM or scripted register/power-on of VMs at recovery site (disruptive)

Service Level Agreement

While talking with customers about disaster avoidance/recovery the first question I always ask is: “Do you have an SLA with the business?”. Sadly, the answer is most of the time “uhh…”.

Without an SLA, the business expectations of the services delivered can be very different from what can be delivered. For example, the business wants an application to be available all the time (100% availability),  but the infrastructure and application only provide an availability of 95%.

Why this brief outline about SLA’s? The SLA is an important piece of information to help decide if a stretched cluster solution is something that helps to meet the agreed upon service level objectives.

Application or Infrastructure?

Let’s say you have an SLA or at least some RPO and RTO values to which you need to design your environment. The next question is “Where in the environment do you want to ensure the application meets the availability requirements?”

On other words, who is responsible for the availability of the application? Is it the application itself, or does the infrastructure provide the availability?

With the rise of cloud infrastructures (AWS, Azure), and cloud native apps, there is a shift in who is responsible for the availability. Applications need to be able to run anywhere with no dependencies on the underlying infrastructure. For example, does it need to survive a data center failure? The application needs to make sure that data is synchronized to another location. In this scenario, the application itself is responsible for the availability.

Availability features in the more traditional applications are for example; Exchange Database Availability Groups (DAG), and SQL Always On Availability Groups (AAG). These features do not rely on the infrastructure to provide availability for the application.

It may not be possible or even desirable to provide availability in the application due to constraints such as network latency/bandwidth or even organizational constraints such as budget or security policies.


There are different options to provide availability through the infrastructure. In my opinion, all the different options can be categorized in a couple of solutions. Each solution has its own pro’s and con’s, and should be matched against the availability requirements of the business.

To stretch or not to stretch?

While a stretched cluster is a solution that fits you organization, it may not be for any other. Each organization has its own requirements and they may or may not match with a stretched cluster solution. The decision to use a stretched cluster solution should not only be based on the current application landscape and requirements but also on the future ones. What is the vision for the application landscape? Are you going to use public cloud infrastructures? Does your organization develop cloud native apps? Where do you want to responsibility for the availability of the application? All these questions and more should help you in choosing the correct solution. And maybe one type of solution is not enough, and you will need to combine different solutions. It’s is all possible but keep in mind you are doing it to meet the requirements of the business.

VCSA Migration – Syslog and Dump Collector

In preparation for an upcoming project it was time to try out the vCenter Server Migration Tool for 6.0 (vCenter Server 6 U2M). I started with reading the vSphere Migration Guide and Release Notes to find out what the prerequisites are and if any known issues are reported.

vSphere Migration Guide

Release Notes

In the release notes I found the following known issue.

Release Notes Known Issue - Syslog and Dump Collector

In the vCenter Server Windows 5.5 the Syslog and Dump Collector are extra components you could deploy but in the vCenter Server Appliance 6.x the Syslog and Dump Collector are integrated in the appliance.

The pictures below show the difference in the vSphere Web Client. The left picture shows the vCenter Server Windows 5.5 with Syslog and Dump Collector and the right picture shows the vCenter Server Appliance 6.0 with the integrated Syslog and Dump Collector.

As you can see there are no more items for the Syslog and Dump Collector in the vSphere Web Client.

When you migrate from the vCenter Server 5.5 Windows to vCenter Server Appliance 6.0 and you have the Syslog and/or Dump Collector installed and configured as integrated with vCenter Server these items will still exist in the Web Client and will show an error even after configuring the services.


According to the release notes there is no workaround. The only way the avoid this issue is to uninstall the Syslog and/or Dump Collector before migrating to the vCenter Server Appliance.

As mentioned earlier in this post, this only happens when the Syslog and Dump Collector are configured as integrated with vCenter Server. If you have deployed them as standalone instances this issue does not occur.


VCSA Upgrade and VM Monitoring

With the release of vCenter Server 6.0 Update 2a and ESXi 6.0 Patch 4 I decided it was time the update the lab environment. Both releases contain a lot of fixes, some specifically for VSAN.

Release notes

vCenter Server 6.0 Update 2a

ESXi 6.0 Patch 4

The update went perfect on the Platform Service Controllers but then disaster struck. During the update of the vCenter Server the VM got a reset and the VM would not boot anymore with the error message:

“Error 15: Could not find file”

When using the Embedded Host Client to check the status of the VM I found out that the VM has had a reset started by the vpxuser. At first I suspected somebody else had giving the VM a restart but this was not the case.

Because I did not make a VM snapshot before the update (it’s a lab environment so he..) I could not recover the VM to a point before the update.

Luckily we had vSphere Replication configured for the vCenter Server to a different lab environment with PIT’s so I could recover the VM to an earlier state. After recovering the vCenter Server and logging on to the Web Client the cause of the reset was made clear.

Virtual Machine Monitoring was enabled for this cluster. Apparently no VMware Tools heartbeats have been received for 120 seconds (low sensivity) and no storage or network traffic was happening for a period of 120 seconds (default). This triggered a reset of the VM and therefore breaking it.

This vCenter Server has been upgraded several times and I never had any issue with VM Monitoring . I have no idea why this happened this time but to be sure I recommend disabling VM Monitoring for the vCenter Server during an update. And of course always have a backup of the vCenter Server in case of a failure.



vSphere 6.5 Storage – What’s new

This VMworld EU 2016 VMware announced the long awaited vSphere 6.5. This blogpost focuses on the new and enhanced storage features in vSphere 6.5.


A new version of the VMFS file system is introduced providing an all-round performance improvement including faster file creation, device discovery and device rescanning. Maybe the biggest change is that VMFS-6 is 4K aligned which will allow the support of 4K drives when they become supported.

There will be no upgrade path to VMFS-6 because of the amount of on-disk changes. Moving to VMFS-6 should be considered a migration using Storage vMotion.

Limits Increase

There are two major limit increases in vSphere 6.5. First off, ESXi hosts running version 6.5 can now support up to 2,000 paths in total. Second, ESXi hosts running version 6.5 can now support up to 512 devices. This is a 2x increase from previous versions of ESXi where the number of devices supported was limited to 256.

NFS 4.1 Improvements

The major improvement for NFS 4.1 is the support for hardware acceleration. This allows certain operations to be offloaded to the array. Other improvements are:

  • NFS 4.1 will now be fully supported with IPv6
  • NFS 4.1 Kerberos AES encryption support (AES256-CTS-HMAC-SHA1-96 and AES128-CTS-HMAC-SHA1-96)
  • NFS 4.1 Kerberos integrity checking support (SEC-KRB5I)

iSCSI Improvements

iSCSI routed conections

The first improvement is that iSCSI routed connections are now supported. Another improvement is that it is now possible to use different gateway settings per VMKernel interface. This implies that port binding can be used to reacht targets in different subnets.

UEFI boot

It is also now possible to use UEFI iSCSI boot. With this you can boot an ESXi host from a iSCSI LUN using UEFI settings in the host BIOS.


Storage I/O Control will be policy driven via I/O Filters. This allows you to expand Storage Policies with SIOC settings such as Limits, Reservations and Shares. By default there will be 3 configuration options for these settings called Low, Normal and High. It is possible to customize the options to your likings.

SIOC Storage policy integration
SIOC Storage policy integration


In the initial release of SIOC v2 there will be no support for VSAN or VVOLs. SIOC v2 is only supported with virtual machines that run on VMFS or NFS backed datastores.

VSAN 6.5

VSAN 6.5 is included in vSphere 6.5 and adds a few new features and a different licensing setup.

iSCSI service

The VSAN iSCSI service allows you to create iSCSI targets and LUNs on top of the VSAN datastore. These LUNs are VSAN objects and have a Storage Policy assigned to them. This feature is targeted for physical workloads such as Microsoft Clustering with shared disks.  It is not intended for connecting to other vSphere Clusters. It is possible to create a maximum of 1024 LUNs and 128 iSCSI targets per cluster. The LUN capacity limit is 62TB.

2-Node Direct Connect

VSAN 2-Node Direct Connect allows you the create a VSAN ROBO configuration without a switch by simple connection the 2 hosts together with cross connect cables. This can make a huge difference in total cost of ownership because it is no longer required to purchase 10 Gbit switches to connect the hosts together.

VSAN 2-Node Direct Connect
VSAN 2-Node Direct Connect


Furthermore, for this type of configuration it is possible to tag a VMkernel interface for the witness traffic to separate this type of traffic.


The different VSAN licenses have been changed and an All-Flash configuration is now possible with the VSAN standard license. This means all VSAN licenses now support an All-Flash configuration. If you want to use data services like deduplication, compression or erasure coding you still have to buy the VSAN Advanced license. For a quick overview of the different licensing options visit the VMware website at

Hardware support

VSAN 6.5 also introduces support for 512e drives, which will enable larger capacities.

VVOLs 2.0

Array-based Replication

VVOLs 2.0 adds the support for array-based replication. Unlike traditional array-based replication like NetApp MetroCluster which replicates the entire datastore, VVol replication allows you to use fine grained control for virtual machine replication. This means you have the flexibility to replicate not all virtual machines but a group of or individual virtual machine(s).

VVol Array-based Replication
VVol Array-based Replication


vSphere 6.5 also offers public APIs for triggering DR operations as well PowerCLI cmdlets for administrator-level orchestration and automation.

  • Replication Discovery – VVol disaster recovery discovers the current replication relationships between two fault domains.
  • Sync Replication Group – Synchronizes the data between source and replica.
  • Test Failover – To ensure that the recovered workloads will be functional after a failover, administrators periodically run the test-failover workflow. After a test, administrators can optionally move the devices from test to production when ready for a real failover.
  • Disaster Recovery and Planned Migration – For planned migration, on-demand sync can be initiated at the recovery site.
  • Setting up protection after a DR event – After the recovery on a peer site, administrators can set protection in the reverse direction.

Oracle RAC

VVols is also now validated to support Oracle RAC workloads.

VASA 3.0

VASA 3.0 introduces a new concept called ‘line of service’. A line of service is a group of related capabilities with a specific purpose, such as inspection, compression, encryption, replication, caching, or persistence. Now in addition to configuring replication at the individual Storage Policy, it is possible to create a line of service for replication and assign it to multiple Storage Polices.

As an example, imagine you have 3 Storage Policies: Gold, Silver and Bronze. While these three categories have very different storage capabilities assigned, it is possible to manage the replication once with a line of service replication instead of configuring replication on each individual Storage Policy.

Cross vCenter vMotion – Cannot connect to host

21 November – VMware engineering has given a fix for this issue. Results are posted at the end of this post.

17 October – Currently this issue is under investigation by VMware and the SR has been referred to VMware engineering. The workaround for this issue at this time is to not use the provision VMkernel interface and TCP/IP stack.

Part of a solution for a customer was the ability to perform migrations of VM’s between vCenter Servers. Furthermore, company policy dictates the management network is used only for management purposes.

By default, data for VM cold migration, cloning, and snapshots is transferred through the management network. This traffic is called provisioning traffic. On a host, you can dedicate a separate VMkernel interface to the provisioning traffic, for example, to isolate this traffic on another VLAN.

To comply with the policy the design used the provisioning VMkernel interface and TCP/IP stack to isolate the provisioning traffic to another VLAN.

When performing the validation of the design we ran into an issue with the Cross vCenter vMotion possibilities. If the VM was powered on the x-vCenter vMotion would perform successful but when the VM was powered off the x-vCenter vMotion would fail with the error ‘Cannot connect to host’.


Because the failure happened when the VM was powered off we immediately suspected the provisioning vmkernel interface and TCP/IP stack. We double checked the configuration and checked if the vmkernel interfaces could reach each other.

esxa01_vmkernel_config esxb01_vmkernel_config

esxa01_firewall esxa01_firewall_out

esxa01_ping_esxb01 esxb01_ping_esxa01

After this we examined the vpxa log on the source host and found connection errors between the provisioning vmkernels on the source and destination hosts.

esxa01_vpxa_log_error esxa01_vpxa_log_error_2

Because pinging between the vmkernel interfaces worked, we wanted to verify if the provisioning network packets reached the vmkernel interface. To verify this we used the pktcap-uw tool which is included by default in ESXi 5.5 and later versions. The pktcap-uw tool is an enhanced packet capture and analysis tool that can be used in place of the legacy tcpdump-uw tool.

With the pktcap-uw tool we generated receive and transmit traffic captures on the provisioning vmkernel interfaces on both the source and destination hosts. The capture files were analyzed with wireshark to verify if the provisioning network packets are exchanged between the hosts.


The picture above is taken from the receive packet capture on the destination host. As you can see packets are received from the IP address  on port 902. This is the IP address of the provisioning vmkernel interface on the source host and port 902 is used for the provisioning traffic (NFC).

Because traffic was flowing between the vmkernel interfaces we checked if the NFC service is listening to accept connections on the provisioning vmkernel interfaces.

esxa01_ssh_esxb01_prov esxb01_ssh_esxa01_prov

The NFC service was not listening on the provisioning vmkernel interfaces on both hosts. To verify if the NFC service was listening on the host we performed the same test on the management vmkernel interfaces.

esxa01_ssh_esxb01_mgmt esxb01_ssh_esxa01_mgmt

This time the test was successful. It looks like the NFC service only responds to incoming connections on the management vmkernel interfaces. To further investigate this issue we opened a SR at VMware.

The SR was transfered to VMware engineering and we had to wait a very long time.

The fix VMware engineering provided us was to increase the maximum memory that could be used by the NFC process on the vSphere hosts. These commands where:

“grpID=$(vsish -e set /sched/groupPathNameToID host vim vmvisor nfcd | cut -d’ ‘ -f 1)”
“vsish -e set /sched/groups/$grpID/memAllocationInMB max=16”

And to check the result:

“vsish -e get /sched/groups/$grpID/memAllocationInMB”


After configuring the hosts with this, I performed a new x-vCenter vMotion action and this time at was successful. We decided to do a few more between different hosts and all of these were successful.


Many thanks to the VMware support representative for keeping the SR open and giving us updates on the progress!

VMCA subordinate CA caveats

vSphere 6 comes with the VMware Certificate Authority service on the VCSA. It’s possible to configure this as a subordinate CA in the existing CA infrastructure.

We tested this configuration in our lab environment and ran into some issues with this configuration. As I don’t have any screenshots of the issues, I will try and explain them shortly.

*Updated with VSAN Health Check error*


Java: Certificate does not comply to algorithm constraints

First off, when we configured the VMCA as a subordinate CA and replaced all certificates of the PSC and vCenter it wasn’t any longer possible to connect vRealize Operations or Trend Micro Deep Security to the vCenter Server. Both products would give you an Java error complaining that the certificate did not comply to the algorithm constraints. After a lot of debugging we found the problem. Our Windows CA infrastructure uses the SHA256 algorithm with the option AlternateSignatureAlogritm=1. This option sets the algorithm to RSASSA-PSS.

vROPS and Deep Security do not work with this algorithm and will trow you an error. We have setup a second CA infrastructure but this time with the option AlternateSignatureAlogritm=0. The algorithm that is used now is SHA256. After re-configuring the VMCA as subordinate CA and replacing the PSC and vCenter certificates it was possible again to connect vROPS and Deep Security to the vCenter Server.


NSX Manager SSO Lookup Service: Failed to Register

After replacing the certificates we were not able to configure the NSX Manager to use the SSO Lookup service.

This happens because after replacing the certificates the corresponding service registration does not get updated in the lookup service. VMware does have a KB article explaining this and provides a solution in these articles if you use an External or Embedded PSC.

vCenter Server with External PSC

vCenter Server with Embedded PSC


NSX Manager Host preparation: Not Ready

After replacing the certificates and updating the lookup services with the instructions from the KB article above the host preparation tab would show the status “Not Ready” for the cluster.

This happens because the EAM service on the VCSA doesn’t recognize the new certificate. VMware does also have a KB article for this.


VSAN Health Check: Unexpected status code:400 

After replacing the certificates the VSAN Health Check Plugin wouldn’t run anymore with the error “Unexpected status code: 400”. I have had already encountered this once before so I knew how to fix this.

Not very surprising but VMware have a KB article for this.

Log Browser and ESXi RAM disk full

With the vSphere Web Client it is possible to view, search and export vCenter Server and ESXi log files using the Log Browser.


When you use the log browser to retrieve the log files of an ESXi host, the log bundle is stored on the ESXi host in /tmp/scratch/downloads. The location /tmp is a symbolic link to the RAM disk created by ESXi to store temporary files.

You can view the log bundle and the size of it using the ‘ls ‘ command when logged in to the ESXi shell.


Because the RAM disk is not very big it might be possible that the Log Browser cannot retrieve the log files of the ESXi host. If the RAM disk is full you will receive an error that no support bundle could be generated.


This error doesn’t show you why the log bundle couldn’t be created but the events tab of the ESXi host does.


You can view the RAM disk usage using the ‘vdf’ command when logged in to the ESXi shell.


It is possible to remove previously created log bundles using the log browser  but the failed log bundle will still be on the RAM disk. You can remove this log bundle using the ‘rm’ command when logged in to the ESXi shell.