The BBC reports on the unusual incident, announced by Google in an online statement. The affected disks were part of Google Computer Engine (GCE), a utility that lets people run virtual computers in the cloud on Google’s servers.
Google says that there were “four successive lightning strikes” on a data centre in Belgium on Thursday August 13. Most of the storage systems were connected to battery backups and survived the power outage that followed the lightning strikes. Google quickly restored power to all of its systems but apparently it wasn’t quickly enough.
The company writes that “some recently written data” was stored on systems more likely to fail after experiencing extended power loss. Of these, most disks were quickly mirrored and committed to stable storage but a small minority ended up with “unrecoverable” recent writes.
Google continues to say this has led to “permanent data loss” on some drives holding the most recently changed files. It is unknown which clients have been affected by the outage and have now got to deal with the data loss.
After the power was restored, issues remained for four days. Some customers found that the GCE platform was inaccessible up until Sunday evening as storage systems were gradually checked and brought back online.
Google says that service was almost totally restored by Monday August 17 and that “less than 0.000001 percent” of disks in the europe-west1-b server region remain affected. It is these drives that have encountered permanent data loss.
The company has said it will upgrade hardware to improve resilience to power losses in case of a similar outage in the future. It writes that it has an “ongoing program” of storage upgrades to new systems that are less likely to be damaged when struck by extended power outages.
The incident did not affect Google’s consumer features and services like Search, Maps, YouTube, Drive or Gmail. Google Compute Engine is aimed at businesses looking to host websites and data in the cloud, taking advantage of Google’s powerful servers rather than having to build their own.
