Virtualization is still a big issue among companies regarding their IT infrastructure. As the technology matures, the benefits remain the same:  lower hardware costs, lower energy consumption and less administrative work hours for maintaining the guest operating systems and the applications that run on them. Market analysts from Gartner estimate that companies already reach an internal server virtualization rate of over 75%. Today every main computer technology can be virtualized including: compute, storage, operating systems, network or applications.

Even though virtualization hypervisors have become better and better, there is still a good chance of having a data loss event. Since 2008 when Ontrack first introduced data recovery for the two most widely used virtualized platforms - VMware and Microsoft Hyper-V - their data recovery specialists have seen a variety of data losses.

From the beginning Ontrack experts involved in data recovery projects with virtualization gathered information and data of the reasons that led to data loss from a VMware or Hyper-V based system. What comes as surprise, is that after almost a decade of computer virtualization in companies, is that users make almost the same mistakes today in both VMware and Microsoft Hyper-V based virtual platforms as they did when it was first introduced.

Common reasons for data losses in VMware and Hyper-V

Ontrack used analysis and insights from his internal database of data recoveries from all of their worldwide data recovery labs to publish statistics on the most common problems that lead to data loss in virtualized systems.

According to these figures, the five most common reasons for data loss in virtualized environments in 2016 are:

• Deleted VMs (Virtual Machines) (40%)
• Hardware failures (30 %)
• Migration Failures (10%)
• Snapshots gone wrong (10%)
• Other (10%)

When you compare these figures with the ones from 2010 they differ only slightly: In 2010, hardware/RAID failures accounted for 40% of all data losses in virtual systems, while deleted virtual disks and/or snapshots totaled 36%. These figures clearly show that the most common reasons remained almost the same. What has become less a factor over the years is formatting and reinstallation problems that lead to data loss. While this accounted for 10% of all cases in 2010, the figure has dropped and is now just one of the many reasons combined in the 10% of “other reasons”.

What is somewhat a good sign is that data loss due to the used virtualization platform itself became less a factor over the years. As an example: “VMFS Metadata Corruption” accounted for 13% and “virtual disk corruption” for 1% of all “virtual” data losses in 2010. In 2016, these reasons for data loss have fallen dramatically and   are now part of the 10% “other reasons” section. This shows clearly that the virtualization platform software products have become better and more reliable other the years.

While virtual platforms became better – many users didn’t!

What has remained the same is that many failures and data losses are due to human error.

• Deleted Virtual Machines,
• Migration Failures,
• Snapshots gone wrong

are all examples of human error. In most of these cases the IT administrator simply made poor decisions. With the better user interfaces and administration panels, deleting or migrating hundreds or even thousands of virtual machines or virtual disks for example can nowadays be achieved by just a simple click of a button. The main risk here is that sometimes the user is either too fast or he doesn’t really know what he is doing. The administrator then tries, after he notices his mistake, to recover the missing data by using internal or 3^rd party recovery tools.

How can data recovery experts help in virtualization related data losses?

For almost a decade now Ontrack engineers have developed highly specialized tools to recover data from both VMware virtual platforms, like VMware ESX, VMware Virtual SAN (VSAN), as well as Microsoft virtual platforms including Hyper-V, Virtual Server and Virtual PC. For Hyper-V alone Ontrack researchers have developed a unique Hyper-V tool suite for their data recovery specialists which can help in almost all common data loss scenarios listed above.

Even though Ontrack has the all the necessary tools to recover data from virtual platforms and systems like VMware ESX or Microsoft Hyper-V every case is different. So what comes really into play is the needed deep knowledge of the involved data recovery specialists.

How do the Ontrack experts recover the missing VMs when they were deleted?

First the specialists try to find the original pointers to the missing virtual disk. If found they virtually rebuild the pointers to the missing virtual disk and extract the data.  If there is corruption or blocks are missing, Ontrack engineers can extract the data inside the virtual disk.  If the pointers are missing, engineers will scan the entire data store for the blocks that make up the missing virtual disk.  They will then they virtually put all the identified blocks of data together in the right order and extract the data.

How do they recover lost virtual data due to hardware failure?

First all individual drives that belong to the system are imaged in the clean room. If some drives are ruined, the experts then replace them and use special tools to virtually rebuild the RAID or storage system. Once the storage has been virtually rebuilt, the engineers will attempt to recover the missing data.

How is virtual data recovered when a migration goes wrong?

Typically when a migration goes wrong, the experts examine the internal structure of the virtualized source storage first and then rebuild the original virtual machine.

How is virtual data recovered when snapshots have gone awry?

If they snapshots are missing or deleted than the specialists try to find and virtually rebuild the original pointers to address the original files.  If they cannot recover the pointers, then they will recover the raw blocks that make up the snapshot and virtually rebuild them. When a snapshot is corrupted then they use the Ontrack’s special tools to merge the flat file with the corrupted snapshot. Afterwards they virtually repair any file system damage at the guest level and then extract the data inside of the virtual disk and/or the snapshots to get to the desired original files in both cases.

Data recovery from virtual platforms and systems is too difficult and dangerous for non-professionals

As one can see data recovery from virtual platforms and systems is definitely not an easy task. It involves a lot of knowledge, specialized tools as well as time. In many cases several technological challenges are combined in one single data recovery project due to the different data structure layers that have to be virtually “recreated” to finally regain access to the original files. Therefore it is wise for the user in case of a data loss to stay calm and not to try any do-it-yourself data recovery attempts. In many of these attempts, the missing data is still available on the hard disks but can be deleted for good when trying DYI. Additionally it is wise to power down the affected hardware, so no applications can run any scripts on the disk and overwrite the “missing” data. In any case it is far better to contact a professional data recovery service provider like Ontrack immediately after an incident occurred. The experts can give sound advice and a professional estimation of what the needed steps for a successful data recovery from a virtual platform or system have to be.

Picture copyright: Windorias  / pixelio.de