Testing contingency plans... too expensive!?

[doc_strange]

"If the risk or cost of testing failover is too high, the risk of actual failure is too high.”

That has become a catchphrase of mine. It made me wonder:

"If the risk or cost of testing a contingency plan is too high, the risk presented by actual disaster is too high.”

These may not be equivalent in value or accuracy. Discuss?

Comments

[ivy]

I think the first is better, because "failover" is a much more clear term than "contingency plan". Contingency plans could involve things that you just aren't willing to do in non-crisis situations (nuke Russia, or something), and making that simulated rather than real may undermine the value of the test. If you don't know what happens if you really nuke Russia, and you don't want to do it, making up a mythical "so now we have nuked Russia and X happens" may be contrived and inaccurate. Some things just can't be tested well.

[docstrange.livejournal.com]

I think the first means an intention to have operations continue relatively unimpeded, to the extent that the cost of the plan/fault-tolerance per period-of-time doesn't exceed (some percentage of) the likely loss over period-of-time without the plan/fault-tolerance. In effect, to prevent harm to the extent reasonable.

The second means an intention to reduce harm, but assumes prevention of harm is unreasonable from a cost perspective. It may be for events forseeable but so unlikely that the cost of testing is not reasonable. It also may therefore be the mobilization of very expensive resources to deal with incredibly expensive, but very rare, failure.

In that light, say, the New Orleans flood was a failure of the first type, followed by a failure of the second type.

[marsgov.livejournal.com]

I've had a busy morning; I meant to comment earlier.

A "drill" wherein Chicago is evacuated (dirty bomb? metorite strike?) would cost millions of dollars, result in the wholesale destruction of empty neighborhoods by fire, and kill a few dozen people. Regardless of potential threat, a drill evacuation won't happen. I'll go out on a limb and say that I'd resist to the best of my ability an order for a "test" evacuation of my home, much less my neighborhood.