I follow cloud vendors’ outages. Broadly, I believe that cloud vendors can deliver higher availability than most SMBs can do themselves. Enterprises are a different discussion.
Recently Salesforce had a DNS outage. Like other vendors, e.g. Microsoft, the Salesforce outage even took down their status page!
And look at the spin they tried to put on it.
“We’re not blaming one employee,” said Chief Availability Officer Darryn Dieken.
And then they threw him under the bus.
“For whatever reason that we don’t understand, the employee decided to do a global deployment,” Dieken went on.
But wait, there’s more…
“In this case,” he went on, “we found a circular dependency where the tool that we use to get into production had a dependency on the DNS servers being active.”
If you’re going to run a cloud service, you’ve GOT to design to avoid these kinds of problems.
Originally published at https://blog.benmoore.info.
