CrowdStrike and CDK: Your Vendors Are Now Single Points of Failure

Two Outages, One Lesson

This summer delivered two very different failures with the same moral.

On July 19, a faulty content update to CrowdStrike's Falcon security software caused Windows computers around the world to crash. Microsoft estimated that about 8.5 million Windows devices were affected. Airlines, hospitals, banks and broadcasters went down. Delta Air Lines, which took longer than most to recover, said in an August filing that it had cancelled about 7,000 flights over five days. There was no attack. A trusted vendor shipped a bad update, and it went everywhere at once.

A month earlier, a cyberattack on CDK Global, which says it serves about 15,000 dealer locations in North America, forced it to shut down its systems.

A month earlier, a cyberattack on CDK Global, which says it serves about 15,000 dealer locations in North America, forced it to shut down its systems. For about two weeks, many dealerships went back to paper, handwriting sales contracts and tracking repairs manually, until CDK said in early July that most were back on its system.

Concentration Is the New Risk

For decades, the main risk to a manufacturer's systems was local: a server failure, a fire, a bad patch applied by the company's own IT staff. Those risks have not gone away, but they have been joined by a different kind. Many companies now depend on a small number of software providers, often delivered as a service, for functions they cannot operate without.

When one of those providers fails, every customer fails at the same time, and none of them can do anything about it except wait. The efficiency of a shared service becomes a shared vulnerability.

For a manufacturer, the obvious concentration is the ERP, especially when it is delivered as software as a service. But it extends further: the identity provider that every login depends on, the security agent installed on every computer, the electronic data interchange network that connects you to your largest customers, the shipping platform that prints every label.

Map Your Critical Vendors

The first step is to know which vendors you cannot operate without. For each critical business process, receiving, production, shipping, invoicing, payroll, ask what systems it depends on, and which vendors supply those systems. Include the infrastructure that sits underneath everything, such as identity, security software and network services.

Many companies find that the list is longer than expected, and that some of the vendors on it were bought by a single department without much review.

Ask the Hard Questions

For each critical vendor, ask how it deploys changes. CrowdStrike's update reached millions of computers almost simultaneously. Vendors that roll out changes gradually, test them on a small group first, and let customers delay non-urgent updates reduce the chance that a single mistake affects everyone at once.

Ask what happens to your data and your operations if the vendor is unavailable for a week. Can you get a copy of your data? In what format? Can you operate in a degraded mode? What does the contract say about service levels and remedies, and do those remedies compensate for anything close to the actual cost of an outage? Most do not.

Build a Plan B for the Most Important Processes

You cannot eliminate dependence on vendors, but you can decide in advance how the plant will keep running while one is down. For a manufacturer, that usually means a few concrete measures: regular exports of open orders, inventory and work in progress so the plant can operate from a recent snapshot; the ability to print shipping documents and labels outside the main system; a documented manual process for receiving and shipping; and a clear method for entering the backlog once the system returns.

A business that can fall back on paper quickly keeps shipping while it waits. A recovery procedure that has been rehearsed runs far faster than one being written during the outage.

The Bottom Line

The CrowdStrike failure and the CDK cyberattack showed that a single vendor's problem can stop thousands of businesses at once, through no fault of their own. Manufacturers should map the vendors they cannot operate without, question how those vendors deploy changes and handle outages, keep their data accessible, and prepare a practiced fallback for the processes that matter most.

At Cherry Street, this is the work we do. We help manufacturers identify their critical system dependencies, evaluate the resilience of their ERP and supporting vendors, and build the fallback procedures that keep products shipping when a provider goes dark.