Connect with us

Hi, what are you looking for?

Tech & Science

Simple router IP address change took Microsoft offline

Incredibly, at Microsoft, a configuration change caused a widespread outage.

Microsoft is reported to be in talks to invest $10 billion in OpenAI to challenge Google's world dominating search engine. — © AFP
Microsoft is reported to be in talks to invest $10 billion in OpenAI to challenge Google's world dominating search engine. — © AFP

According to Microsoft, a change made to the Microsoft Wide Area Network (WAN) made Microsoft services inaccessible to users around the globe. The networking outage took down the cloud platform Azure together with business services including Teams and Outlook.

This recent Microsoft outage caused significant concerns among business users. The issue presents ongoing issues for network operations teams to solve in order to prevent future network outages and to be prepared to minimise future downtime.

To understand the significance, Digital Journal caught up with Josh Stephens, who has been involved with network engineering for over 30 years — first as an engineer with the U.S. Air Force, and more recently as CTO for BackBox.

Stephens begins by expressing incredulity over the incident: “It’s incredible that even the simplest configuration change or even a typo can sometimes cause a ripple effect and bring down a network and/or disrupt a supposedly fault-tolerant business service. Even tech giants like Microsoft aren’t immune.”

Stephens  continues: “In many cases, the outage may not occur immediately after the configuration change was made and so it can be difficult to correlate the change during root cause analysis.”

Looking deeper into the central issue, Stephens  notes: “While many news reports have keyed upon the fact that a configuration change caused such a widespread outage, the real headline is that it took them four hours to restore service.”

Stephens  adds: “While this sounds exorbitant, without more technical details about the cause of the outage and, more specifically, the extenuating circumstances that extended the time it took to restore service, rather than pass judgment I will just honestly say, I’ve been there.”

In terms of lessons to be learned, Stephens has thoughts on how other network teams at organizations can be proactive now to avoid a similar disaster.

His first recommendation is: “Accelerate the speed of solving difficult technical problems to ensure there is solid documentation and up-to-date network maps.”

Secondly, Stephens advises: “Continuous, automated configuration auditing and remediation to ensure that all network devices are up-to-date and compliant with operational policies and industry standards.”

Stephens also adds, as his third consideration, that automated network configuration backups “allow you to instantly restore backups and have automated weekly or frequent OS updates and patches”.

Stephens’ fourth and final recommendation is: “At a minimum, your automation platform should create backups daily, before and after changes, and store a long history of backups within an autoscaling, fault-tolerant data store. Furthermore, it should be able to reliably conduct upgrades at scale and while employing at least mildly complex workflows.” While these approaches are useful, Stephens warns: “No single tool or approach can guarantee business continuity, but there are ways to be better prepared if the worst does happen.”

Avatar photo
Written By

Dr. Tim Sandle is Digital Journal's Editor-at-Large for science news. Tim specializes in science, technology, environmental, business, and health journalism. He is additionally a practising microbiologist; and an author. He is also interested in history, politics and current affairs.

You may also like:

Social Media

Wanna buy some ignorance? You’re in luck.

Tech & Science

Under new legislation that passed the House of Representatives last week, TikTok could be banned in the United States.

Life

Platforms like Instagram and Pinterest often suggest travel destinations based on your likes and viewing habits.

Social Media

From vampires and wendigos to killer asteroids, TikTok users are pumping out outlandish end-of-the-world conspiracy theories.