Update: DJ is back online!
If you have spent the last little while scratching your head, wondering where your favourite news hub went, you aren't the only one. It's been a crazy few days and DJ experienced some very damaging hardware failure. This is what happened...
First off, we'd like to apologize for the serious delay in getting things back up and running again: It's been a very hectic few days and we've dealt with some very major technical complications. We've always believed that an open discourse with our user base is important, so we'd like to offer our apologies along with a synopsis of what is currently happening:
On early Friday morning, our IT department was sent scrambling after we received multiple error messages being sent by our server. According to early reports, all we knew was a hard drive was not responding. The problem, in fact, was a whole lot worse.
After spending some time to determine the exact problem, we realized we had experienced catastrophic hard drive failure on our servers. In the world of tech, it's common for a hard drive to fail here and there. In fact, DJ anticipated this and designed our system to be able to run even if a hard drive drops offline. But in our case (and probably one of the biggest flukes you will ever read about) we had all of our hard disks fail virtually at once and a domino effect brought our system to a grinding halt. With mass failure of hard drives, the "brain" of our system has been paralyzed and is not functioning until we can replace parts and rebuild data that was lost.
Bill Margeson, president and CEO of CBL Data Recovery Technologies Inc., inspecting hard drives in the company's lab.
image:27796:0::0
|
With no vital signs, we took a look into what we could recover from the drives. But hurdle No. 2 was not an easy one, as we learned the drives were not spinning properly and sectors had become so corrupt that only a data forensic expert would be able to recompile the information we lost.
On Friday morning, we took our damaged server hardware to one of the best data recovery companies in the world - CBL Data Recovery Technologies Inc. They specialize in recovering data from severely damaged drives, and the company has worked extensively with government, corporations, the public health sector, and they are widely recommended by manufacturers. We knew if they couldn't recover the data nobody could. CBL jumped right on the case, beginning the laborious process of trying to rebuild our very complex RAID structure and recompile hundreds of gigabytes of data that had become corrupted.
At this point, we know that our hard drives all suffered electronic failure as well as countless bad sectors that have made the UNRAID process incredibly difficult. CBL continues to work on trying to recover the data, and we hope to have it from them in the very near future. They have their entire Toronto-based team of forensic data recovery specialists working on this case, and have showed a remarkable amount of passion and dedication to Digital Journal. We offer our most sincere thanks to their entire team for putting so much dedication into recovering data, and we'll keep you updated as more information becomes available.
We are now focusing on doing two things. We are working to diagnose what exactly caused this malfunction, waiting for new hardware to be shipped to us to replace the pieces that are no longer functioning, and we are attempting to recompile backup data that was also partially corrupted in this process. We have spent three sleepless days rebuilding DJ, pulling whatever data we could from other corrupted drives and recompiling into the interface you are familiar with. With millions of comments, articles, images, scoring, voting history and countless database tables that keep DJ up and running, rebuilding has not been easy and the results are less than favourable, as we are still down. We are dealing with delivery hurdles, as new parts take time to be processed through shipping channels, and we must wait for CBL to finish recovering the data.
So what happens now?
We will keep you updated as we learn more information, but as of right now it all comes down to hardware arriving, and recompiling hundreds of gigs of data. Unfortunately we cannot say exactly when this will happen at this point, but we hope it won't be too much longer. We will keep you updated as we learn more information.
We would like to thank everyone for their ongoing support, as we have received many messages of encouragement and well wishes. We really appreciate your patience as we work to resolve this issue and we offer our most sincere apologies for the inconvenience.
Update: DJ is back online!
