CBL and Dell Come to the Rescue, Bringing Digital Journal Back Online

Advertising

Welcome back DJers, it’s been far too long. In case you missed the earlier announcement, DigitalJournal.com did not implode from heated debate on daily news. We unfortunately experienced catastrophic hardware failure in the core of our server architecture and it took down the whole system, with multiple hard drives failing at once. It's very rare that so many hard drives fail at the same moment, but that's what happened in our case. We have spent the last four days running around like chickens with their heads cut off, frantically working to bring life back to the fallen news hub.

In today’s online era, we’re at the mercy of hardware that keeps things operational; nobody is immune from glitches or hardware failure. The technical problems we experienced were incredibly rare, but we’re happy to say we’re back and stronger than ever.

Vote it up! Alex Chumak working in DigitalJournal.com's server farm. And no, chairs are not in our budget.

First off, we’d like to thank all of our incredible users for their unprecedented support during our down-time, and for being patient while we brought things back online. We received countless emails from dedicated readers who offered well wishes, we received hilarious e-cards that offered comedic relief (courtesy of Picasso), and even a few offers to bring coffee to our offices to keep the DJ staff of worker monkeys fuelled for the long haul. We cannot express our gratitude for this support — it means more than any of you will ever know.

So what happened and how did we get back? Well, thanks to the efforts of DJ staff, CBL Data Recovery Technologies Inc. and Dell, DigitalJournal.com has recovered. The crash gave me 15 heart attacks, it made Alex revert to only speaking in Russian (he kept demanding pivo and a hard drive defibrillator) and resulted in literally four sleepless days for most DJ staffers. But we’re back, and more tired than ever!

Because our crash involved multiple drives in a complex RAID structure, a lot of our data was damaged. We run backups regularly, but the crash corrupted a lot of that information. Without the help of CBL Data Recovery, we would not have been able to recover to this point.

Vote it up! Alex Chumak (right) works on DigitalJournal.com database as Editor Chris Hogg sits doing nothing (he's actually playing solitaire)

“When CBL received the call from Digital Journal at 2 a.m. on Friday, it was clearly evident from the initial assessment of need, that the personnel at Digital Journal were facing a critical data loss,” said Bill Margeson, President, CEO and Co-founder CBL Data Recovery Technologies Inc. “The business continuity of Digital Journal was at the forefront of everyone's mind.”

CBL is no stranger to Digital Journal: In 2004, we profiled Margeson and his company. They are widely regarded as one of the best data recovery companies in the world, and they have extensive experience working on enterprise-level recovery with severely damaged drives. CBL has worked extensively with government, corporations, the public health sector, and they are widely recommended by manufacturers.

When we realized the magnitude of our crash and how much data was corrupted in the fall, we knew CBL would be the only company that could save us.

Vote it up! Bill Margeson, president and CEO of CBL Data Recovery Technologies Inc., inspecting hard drives in the company's lab.

“Through 72 hours of round-the-clock work in our data recovery laboratory, CBL kept personnel abreast of developments along this digital recovery journey, while remaining focused on the goal of successfully recovering the data and getting Digital Journal back online,” says Margeson.

CBL deployed a team of forensic data recovery specialists who evaluated our hard drives and deployed proprietary techniques and processes to address both logical and hardware specific-problems to retrieve the data.

“CBL Data Recovery Technologies is pleased to have come to the rescue of both the data and the Digital Journal team, as well as play a role to help facilitate the return of Digital Journal for its readers' enjoyment,” Margeson said.

CBL successfully recovered 100 per cent of our lost data. We are greatly appreciative of all of the efforts of the entire team at CBL, and we would like to express our most sincere gratitude for all the work they did. Their team pulled nothing short of a miracle.

We would also like to say a BIG thank you to Dell who has spent a great deal of time with us to resolve this issue.

Vote it up! 1 vote DigitalJournal.com Programmer-in-Chief, Alex Chumak.

When our website went down, we were humbled to see how quickly dozens of Dell employees stepped up to the plate to find a solution to bring DJ back online.

We spent quite a bit of time diagnosing problems, figuring out solutions and working together with Dell to recover the fallen site. Dell also dispatched a support team to go through all of the server parts to make sure nothing was defective.

Because the problem of multiple hard drives failing at once is so rare (unheard of in any of my experiences), we decided to play it safe and replace virtually every part inside our main server. It was probably overkill, but we are grateful that Dell went above and beyond what was expected of them to make sure Digital Journal's community could be back up and running again. They kept us informed the entire time, with customer service support that rivalled most of my experiences with any other company in the tech world. When things go down in a crisis (such as what DJ experienced), you truly see what a support company is capable of. Dell has impressed and surpassed our expectations all along the way.

“We encourage everyone at Dell to take personal ownership of the customer experience and to be the reason our customers remain loyal," said Debora Jensen vice president of the Advanced Solutions Group at Dell Canada. "We understand the critical role Digital Journal plays in providing digital content. I am tremendously proud of how quickly our team responded in addressing and resolving their business challenges.”

We also think it's important to note that we will be continuing to work with Dell even now that we're back up and running. We will be discussing more hardware support and more levels of redundancy. While this crash was unpredictable and very unlikely to happen again, we believe it's important to plan for even the worst case scenario, no matter how fluky it is. Dell will be working with us to provide the hardware to make sure your favourite news hub doesn't suffer this pain again. Thanks again to the incredible team at Dell who have shown true professionalism and business excellence. We look forward to working more with them.

There was an incredibly large team that devoted four days to bringing DigitalJournal.com back online successfully. We recovered all of our data, our hardware has been fixed and upgraded and we will be adding more Dell hardware to prevent anything like this from happening again.

And thank you once again DJ users. We are forever humbled by your support.


Cheers. It’s time to get some sleep.

— Christopher Hogg, Editor-in-Chief, DigitalJournal.com