What to be aware when you move from ESX to ESXi.
With more customers moving their infrastructure from ESX to ESXi (including version vSphere 5) I would like to cover one topic related to this upgrade.
I am not going to talk about a performance drop while moving from Service Console based (LAN) backup mode to the corresponding vStorage API NBD. This is all covered in Deployment guide along with some suggestions how to improve situation. We play by VMware rules: Service Console has been locked, end of story. We miss this part same way you do.
Here is another serious topic – Restore. Take following scenario. You had few ESX hosts and performed LAN-based backup. Some of your VMs had a snapshot or two. They were successfully backed up. With LAN-based backup we had an opportunity to save each vmdk file no matter if it was flat.vmdk or delta.vmdk (snapshot). During restore to the ESX host, all information is recovered including snapshots. Basically, after restore you still see a snapshot and have an option to revert or commit changes.
Now fast forward to ESXi. The only way to work with virtual disks is by using VMware vStorage API. You probably noticed already that if you backup VM with a snapshot residing on ESXi, you don’t see any delta files in savepoint directory. vStorage API consolidates all snapshots on the fly. What it means is that on restore you have a VM without any snapshots, with all changes, previously stored in snapshot, perfectly merged into the base flat vmdk.
If you try to configure a restore job from the savepoint with snapshots (delta files), job will fail:
[2012-04-04 15:31:38.383]: Restore task failed: Restoring savepoints with snapshots using VDDK is not supported.
[2012-04-04 15:31:38.415]: An internal error occurred during execution, please contact Quest support if the error persists. Error Message: Restoring savepoints with snapshots using VDDK is not supported.
VDDK is just another term for vStorage API. Obviously, since it does not save snapshots during backup, why should it work with them on restore?!
What can we (Support) do to address this issue? Well, we have some options:
1. if you still have any ESX host you are saved. Just point a restore job to this host and run the job. VM will be restored with all snapshots.
2. we still have a manual restore procedure. Igor’s var-extractor restores vmdk on NTFS partition. It is a three-step procedure:
2.1 create VM with defined parameters (number of disks, size, number of snapshots);
2.2 extract vmdk from each var file locally, on Windows side;
2.3 upload all extracted vmdk to the datastore overwriting existing files.
It is a long and cumbersome procedure, but it does the trick.
There are two other possible solutions but both involve metadata editing. I won’t go into details, just give a brief description.
All references to the snapshot/delta should be removed from the Manifest and VmConfig files. Then ‘Restore from Manifest’ should be performed. Manifest file is pretty easy. The hard part is to properly edit VmConfig.metadata. Information about snapshot is stored in two major spots: at the very beginning (RootSnapshot, CurrentSnapshot/SnapshotLayouts) and in each disk configuration tag.
Here are two remaining solutions:
3. If you can leave snapshot behind (say, you do not care about changes accumulated in the snapshot), you can restore VM as if it never had a snapshot.
3.1 modify metadata files;
3.2 create ‘Restore from Manifest’ job.
4. If you still want to restore VM with all snapshots
4.1 modify metadata files;
4.2 create ‘Restore from Manifest’ job and run it;
4.3 take one or more snapshots of restored VM (see in metadata files how many snapshots existed);
4.4. extract delta files on Windows end using var-extractor. Make sure names match those created by VC/ESX in VM directory;
4.5 upload delta files to the datastore.
Every method, except for the first one, looks pretty complicated. And it is. Main goal of this post is to make customers aware of possible issues during restore.
Keeping VMs clean of any snapshots helps avoiding problems in the future!