This type of contingency planning came to the fore recently with the outage that locked out thousands of Salesforce Marketing Cloud and Pardot users. This was reportedly caused by a faulty Pardot database script, which allowed users to see and edit all of their company’s data, regardless of the permission settings.
In response, Salesforce quickly cut off access for current and past Pardot customers while working to resolve the faulty script. But, without access to data, analytics, contacts and more, many teams struggled to operate. In light of this, Todd Matters of RackWare raises why it is important that sellers and marketers develop contingency plans for handling worst-case scenarios and outages outside of their control and how enterprises can identify risks and outline an emergency plan for disaster recovery.
Digital Journal: What was the reason for the Salesforce incident?
Todd Matters: Salesforce officially announced that the incident was caused by a faulty database deployment script. This created downstream permissions issues that took multiple days to resolve. The issue was not cloud specific, but rather more of an IT malfunction in general.
DJ: What were the consequences of the outage?
Matters: While the outage prohibited some users from syncing their data (or doing so with degraded performance), the manual intervention required by users to resolve their issues was a huge burden. An outage is bad enough, but requiring users to fiddle with redefining permissions and reconfiguring the data connector adds insult to injury.
DJ: How common are these types of incidents?
Matters: These types of issues occur fairly frequently; however, they are typically resolved more quickly and without major manual intervention from the users.
DJ: What types of contingency plans should sellers and marketers be developing?
Matters:Companies need a plan that is documented, vetted, and tested. This third component is extremely important. Testing must be done on a regular basis with the gaps and audit findings clearly documented. This is the only way to test the strength of a contingency plan.
The cloud plays a major part in a well-developed contingency plan, as well. This should include considerations for disparate geographies and cross-cloud providers. Implementing a hybrid cloud strategy ensures enterprises don’t put all their eggs in one basket.
A cloud strategy allows businesses to fallback to a working recovery point, which should be a main goal of every sellers’ and marketers’ contingency plan. If Salesforce had been able to rewind to a version of software prior to the bad deployment script, it would have significantly smoothed out the outage.
DJ: What should companies be doing in terms of risk identification?
Matters:There are three key things companies need to be doing to identify risks.
First, a triage of applications, and the components necessary for running those apps, need to be well defined. Next, it is extremely important to have a well-documented plan around handling failures. Lastly, while recovery points should be part of any plan, ensuring a solid recovery point prior to the roll out is particularly important when implementing new systems or upgrades.
DJ: Do organizations need new technology to address such risks?
Matters:Yes, organizations need new technology to handle these risks. With today’s demands for availability and uptime, just having a backup of data isn’t enough. Keeping functional recovery points that can be retrieved on a moments’ notice is imperative. Many traditional solutions don’t have this capability, making a more modern recovery product warranted.
