Web hosts like 123-reg provide servers that people can upload their websites to so they are accessible online. Companies use web hosts to get their websites visible on the Internet without having to invest in their own costly hardware.
123-reg is one of the largest web hosts in the UK with 800,000 customers and a further 900,000 international users. Since Saturday morning, many of those sites — described by the company as a “small proportion” — have been inaccessible and remain offline more than two days later. Sixty-seven servers out of a total of 115,000 in Europe were affected.
After initially describing the incident as a connectivity problem, 123-reg has since admitted it inadvertently deleted customer websites during a routine maintenance operation. It confirmed to the BBC that it had “effectively deleted” sites due to a bug in a clean-up script running on its VPS systems, virtual private servers that customers have access to like a real machine.
The company said it does not have backups of its customers’ data, instead relying on them to keep their own copies. It is working with data recovery experts to try to restore what it can. Engineering teams have been devoted to getting systems back online since the script failed on Saturday night.
It remains unclear what 123-reg is doing as part of its “recovery process” or how it intends to compensate customers who have just lost their online presence. It apologized to all its users and promised to “leave no stone unturned” without explaining what its next steps will be. The company said to Ars Technica that it has brought in data recovery firm Kroll but admitted it is “unable to estimate” when the process will be complete.
In an email to customers yesterday, 123-reg said it expected to begin restoring service overnight with more sites coming back online during the week. Some sites are unlikely to ever be restored though, leaving customers without a website to trade from and potentially ending their business.
Customers have reacted angrily to the news, the sixth in the past 12 months where VPS connections at 123-reg have been compromised. “It is clear that 123-reg does not employ sufficient expertise or maintain sufficient hardware capacity to ensure that potentially catastrophic hardware failures do not occur,” one customer told V3. “It is also clear that they do not have processes in place which allow them to learn from previous mistakes.”
123-reg is continuing its efforts to get customer sites back online, posting regular updates on Twitter and its status page. It has contacted every affected user to explain what went wrong and how it is trying to put it right. It warned customers that response times on communication channels may be higher than usual, asking for patience while it deals with the aftermath of running the buggy script.
