Another CBT bug found in VMware ESXi 6.0

VMware just published KB 2136854 regarding a new bug found in ESXi 6.0 that causes virtual machine backups, which use Changed Block Tracking (CBT), to be inconsistent. VMware says the cause of the issue is this:

This issue occurs due to an issue with CBT in the disklib area, this causes the change tracking information of I/Os that occur during snapshot consolidation to be lost. The main backup payload data is never lost and it is always written to the backend device. However, the corresponding change tracking information entries which occur during the consolidation task are missed. Subsequent QueryDiskChangedAreas() calls do not include these missed blocks, hence a backup based on this CBT data is inconsistent.

Unfortunately at the moment VMware do not have a fix for the issue and the “workarounds” really are not all the much better either:

• Downgrade your hosts to ESXi 5.5 as well as downgrade the virtual machine hardware version from 11 to 10.
• Power down the virtual machine before doing an incremental backup.
• Do a full backup on the virtual machine rather than an incremental backup.

Hopefully VMware is able to knock out a fix for this bug sooner rather than later. Once a fix has been released I’ll be sure to update this post as well.

Updated 11/29: VMware has released a fix for this bug now, VMware ESXi 6.0, Patch Release ESXi600-201511001 (2137545)