Recalls are skyrocketing in 2022, with more than 1 billion units in the first seven months – on track to break records. Automotive recall units for this past quarter were 9.2 million. Medical device recalls increased by 34%, a 2-year high.

70-80% of defects are avoidable if fixed during the requirements stage, and 40-50% of the time a software specialist spends is on avoidable rework rather than value-added work.

We are in an era of:

• increased automation
• complexity in development
• shorter time-to-market
• reduced business operation models
• increased cost pressures
• globalization
• evolving regulations

We are seeing clients and consumers becoming more concerned about safety and health, as well as the impact of ESG (environmental, social, and governance) issues. Social media and the 24/7 news cycle means that product defects and recalls become public knowledge, immediately and globally.

All this means that the enterprise now operates with much tighter margins for success and profit. There is less time for development, in every phase, and outsourcing is encouraged in the name of increased competitive advantage. Organizations are looking for ways to increase their productivity; when —not if! — done incorrectly, the consequences are often dire.

The performance standard is zero defects (relative to requirements), not “that’s close enough”.

• Philip B Crosby

General Motors (GM) Ignition Switch Recall

GM’s ignition switch recall provides insight into the disastrous and painful results that can come from errors in your design phase cascading into production.

GM’s engineers violated two requirements. Specifically, requirements in respect to the torque (or rotational force) needed to turn the ignition switch and the vibrational environment for the ignition switch. The torque, or rotational force that prevents the ignition switch from changing

modes needed to be between 10 N·cm and 20 N·cm (Newton centimeters). GM’s was less than 10 N·cm.

The ignition switch installed into GM’s vehicles required much less effort than a user expected to switch the vehicle into a different mode. A bumpy road, a keychain with decorations, or even a nudge from a knee could turn off the engine. If hit with the right amount of force the ignition switch would cause the car to change into Accessory mode, removing power from the vehicle’s airbags and causing difficulty for the driver to steer and brake.

GM knew of the defect in 2001, while specific models were in production. They issued a Technical Service Bulletin in 2005 and again in 2006. These instructed dealers to notify owners to remove accessories on their keychains. Unfortunately, millions of vehicles were already on the road with the defect. Rather than issue an immediate recall, GM instead started production on a new ignition switch that required greater torque, thus avoiding the problems of the defective switch.

GM eventually issues a recall, but the damage was done. GM compensated claimants for 124 deaths. The toll was almost certainly higher because 90% of claims were rejected. This number also does not include claims that are part of the Multidistrict Litigation: a larger lawsuit filed by several individuals who have all suffered similar injuries by the same defendant.

The recall cost GM more than $3 billion in shareholders’ value over four weeks. The organization forfeited $900 million to the US, paid $600 million in compensation to surviving victims of accidents caused by the faulty switch, and paid over $35 million in fines for delaying the recall of the defective cars. In the end, they recalled 30 million vehicles worldwide.

Prior to the ignition switch crisis, GM engineers worked in silos where they had a mindset of, “I’m responsible for this part, someone else will handle another part, and it’s not my responsibility to connect the dots,” said Combs. “Now, part of our process is to make sure we are looking at things from a system perspective.”

Deepwater Horizon Oil Spill

Eleven workers dead, 17 injured. According to the United States, Environment Protection Agency (EPA) the Deepwater Horizon oil spill was the largest marine oil spill in history. At its peak, 60,000 barrels of oil every day leaked into the Gulf of Mexico – nearly 134 million gallons. Before sealing the leak, petroleum covered an area of over 57,500 square miles — an area approximately equal to the country of Greece. An estimated 1,100 miles of polluted shoreline devastated the ecosystem. As many as 105,400 sea birds and nearly 170,000 sea turtles died, and up to a 51% decrease in dolphins in Louisiana and Barataria Bay.

The panel investigators could not fault the explosion on any one decision of BP or its contractors. However, they did establish that the organization focused on the pace of completion rather than safety. The well was behind schedule and costing BP $1.5 million per day – a significant factor in the accident. As Donald Winter told The New York Times, “A large number of decisions were made that were highly questionable and potentially contributed to the blowout of the Macondo well… Virtually all were made in favor of approaches which were shorter in time lower in cost. That gives us concern that there was not proper consideration of the trade-offs between cost and schedule and risk and safety.

It was a case of exchanging accusations, and no one accepted liability for the catastrophic event.

Harry Theirens, BP’s Vice President for drilling and completions, told the hearing that he found the blowout preventer connected to a test pipe rather than the pipe transporting oil to the surface.

The blowout preventer provided by Transocean to BP did not properly function with  the structure on the ocean floor. The equipment intended to be a failsafe, halting the flow of highly pressurized gas and oil rising through pipes from beneath the ocean floor. The shear rams, components inside the blowout preventer, were not designed to function on the joints where the drill pipes were screwed together or on tools passed through the blowout prevent during well construction.

BP also maintains that Transocean did not correctly manage the emergency after the fatal failure.

BP accepted partial responsibility for the disaster but pointed the finger at what it claimed were significant failures by Transocean Ltd, the operator of the ill-fated Deepwater Horizon oil rig, and oil services company Halliburton who cemented the deep-sea well.

Transocean, for their part, wasn’t willing to take the blame either.

“The well barriers–the cementing and the casing — were responsible for controlling any pressure from the reservoir,” Newman was set to say — according to his prepared testimony.

Halliburton insists they completed all the work concerning the Macondo well following BP’s specifications and that BP was to blame for poor well-design specifications.

In the end, 11 decisions may have increased the risk on the rig, and ultimately BP was decided to be majority responsible. What followed was also the largest settlement in US history – $20.8 billion. On top of that, BP paid a $4.5 billion criminal penalty, $15 billion in cleanup costs, and $20 billion in economic damages to companies and individuals harmed by the spill – in total, $65 billion for the Deepwater compensation process. Halliburton paid $1.1 billion for their involvement, Transocean paid $1.4 billion for their role in the oil spill, and Moex Offshore, a minority investor in the Macondo well, agreed to pay $90 million.

Better management of decision-making processes within BP and other companies, better communication within and between BP and its contractors, and effective training of key engineering and rig personnel would have prevented the Macondo incident.

What If…

These are worst-case scenarios: loss of life, billions in compensation, company value, and brand destruction, but these were both preventable situations. Errors slipped through the initial design phases, compounding into the disastrous situations these organization paid millions to resolve. Investing in the beginning stages of a project may feel unproductive, but both BP and GM’s examples show the true burden of paying for a mistake in the end stage of a project. ROI increases as you invest in error management earlier in the design phase, specifically the requirements phase.

If only 34 highest-risk requirements, requirements that pose the most significant threat to the organization, make it through the product development lifecycle they can have a heavy burden on the overall project.

Controlled figures say that should 34 high-risk requirements get to the Release phase, the number of hours to fix the mistakes is 180, or 22.5 working days. On the other hand, if the errors only make it to your project’s next step, the Design phase – you’re 1.5 working days behind schedule.

“One day’s delay is another day’s lack of progress.” Stuart Bowen.

Using the same breakdown, what sort of subsequent cost does your organization incur? For example, in the Test phase, a conservative cost estimate for 34 high-risk requirements is $100,000. In the Release phase, the cost increases to $250,000 to address the errors. Although this cost may seem small compared to the examples detailed earlier, eliminating cost overruns and time spent fixing errors frees up resources for continued innovation.

Your development projects may have thousands of requirements, and only 34 high-risk requirements is a very modest number. This scenario also does not include requirements that are midway, they aren’t wrong, but they also open the door to misinterpretation, ambiguity, and confusion on intent.

These analyses include one person’s time and salary; substantially more compounding costs are to be considered. The costs associated with go-to-market delays, the loss of trust from investors who deem your organization too risky, your company’s value depreciation (what that means for future acquisitions or mergers), and your discredited reputation from a consumer, contractor, or client perspective. And what if your product goes to market? As we have discovered in the incidents above, disastrous repercussions and devastating impacts.

Final Thoughts

As an organization, being strategic versus reactive is the ideal method – risk management is favored over crisis management.

There is a philosophy exhibited in ISO 9001 that correcting a problem is less effective than identifying the root cause and then updating the process to ensure the problem doesn’t recur – corrective action. That same philosophy states that ensuring the problem doesn’t reoccur isn’t as helpful as preventing it from happening – preventive action. Bizmanualz summarizes that corrective action is based on a nonconformance event that has occurred in the past. Preventive action is based on preventing a nonconformance event in the future.

If we know that the root causes of over 50% of defects identified in projects are introduced in the requirements phase and now see the breakdown of the cost and time overruns with just 34 high-risk requirements… logic would say, why not prevent those in the first place.

The level of a project’s complexity determines what level of project rigor is needed to complete the project effectively. As products advance in intricacy, the required precision at the requirements stage must increase. If not, cost and schedule delays will only grow in severity and number. Like other stages of the product development process, tools are needed in this early stage to offset the cost of this rigor without sacrificing quality. Tools like QVscribe can help augment employees to achieve the required complexity while keeping velocity and quality consistent.

“Zero defects through prevention – “Zero defects” doesn’t mean mistakes never happen, rather that there is no allowable number of errors built into a product or process and that you get it right the first time.”

              -Philip B Crosby