VMWare ESX 6 Change Block Tracking (CBT) corruption

Solution In Progress - Updated -

Environment

  • VMWare ESX 6
    • Virtual machine backup
    • Virtual machine Power on
    • Expanding the size of the virtual disk
    • Taking virtual machine quiesced snapshots
  • Red Hat Enterprise Linux 5, 6, 7
  • Backup software using Change Block Tracking (CBT)
  • Oracle DB 11.X, 12.X
  • Any file system

Issue

  • DB files getting corrupted frequently
  • File system corruptions

Resolution

Root Cause

Known issues

Diagnostic Steps

  • Oracle error messages like:
ORA-00742: Log read detects lost write in thread 1 sequence 92 block 30215
  • Per VMWare KB note
You can observe error log similar to the following in the /var/log/vmkernel.log file:
<YYYY-MM-DD>T<TIME>.nnnn cpu5:nnnnnn)WARNING: CBT: 191: No memory available! Called from 0xnnnnnnnnnnnn

The vSphere Client might display an error message similar to the following when you attempt quiesced snapshot:
An error occurred while taking a snapshot: msg.snapshot.error-QUIESCINGERROR

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments