Testing contingency plans... too expensive!?

[doc_strange]

"If the risk or cost of testing failover is too high, the risk of actual failure is too high.”

That has become a catchphrase of mine. It made me wonder:

"If the risk or cost of testing a contingency plan is too high, the risk presented by actual disaster is too high.”

These may not be equivalent in value or accuracy. Discuss?

Comments

[ivy]

I think the first is better, because "failover" is a much more clear term than "contingency plan". Contingency plans could involve things that you just aren't willing to do in non-crisis situations (nuke Russia, or something), and making that simulated rather than real may undermine the value of the test. If you don't know what happens if you really nuke Russia, and you don't want to do it, making up a mythical "so now we have nuked Russia and X happens" may be contrived and inaccurate. Some things just can't be tested well.

[docstrange.livejournal.com]

I tend to agree. My take on it in the following comment.

[docstrange.livejournal.com]

I think the first means an intention to have operations continue relatively unimpeded, to the extent that the cost of the plan/fault-tolerance per period-of-time doesn't exceed (some percentage of) the likely loss over period-of-time without the plan/fault-tolerance. In effect, to prevent harm to the extent reasonable.

The second means an intention to reduce harm, but assumes prevention of harm is unreasonable from a cost perspective. It may be for events forseeable but so unlikely that the cost of testing is not reasonable. It also may therefore be the mobilization of very expensive resources to deal with incredibly expensive, but very rare, failure.

In that light, say, the New Orleans flood was a failure of the first type, followed by a failure of the second type.

[marsgov.livejournal.com]

I've had a busy morning; I meant to comment earlier.

A "drill" wherein Chicago is evacuated (dirty bomb? metorite strike?) would cost millions of dollars, result in the wholesale destruction of empty neighborhoods by fire, and kill a few dozen people. Regardless of potential threat, a drill evacuation won't happen. I'll go out on a limb and say that I'd resist to the best of my ability an order for a "test" evacuation of my home, much less my neighborhood.

[docstrange.livejournal.com]

Hmm. And yet they used to do air raid drills.

Should one just assume any evacuation will fail, or maybe that they should just wing it according to plan and hope for the best (even if in practice the plan is a flop)?

Fault tolerance planning is clearly not contingency planning - but if you can't test a contingency plan because it's too expensive to test, maybe that's strong evidence your situation is inherently too risky?

[marsgov.livejournal.com]

I can't really chat pre-Shabbat, but my philosophy is a bit different: Is a plan really needed to evacuate Chicgao? Is someone going to tell me I have to drive instead of bicycling (my current plan, BTW)?

I suspect that central planing may create more disaster than an unplanned evactuation.

[cruiser.livejournal.com]

For the most part, the cost of air raid drills was only time, something which most people waste a lot of anyway (like me, reading LJ, for example).
The decision to do a Chicago evacuation drill or not has a number of factors, not the least of which would be how many people would just ignore the drill (look at the number of people who ignore *real* mandatory evacuations because of hurricanes). The most important factor, though, is how many people would die and how much would a drill evacuation cost vs. how many additional people would be saved multiplied by the chance of an evacuation being necessary. Having participated in lots of drills in the military, it often takes a lot of practice to get something right - which means we're not talking just one drill to have a positive effect in evacuating Chicago, but several.

[docstrange.livejournal.com]

So I say again:

"but if you can't test a contingency plan because it's too expensive to test, maybe that's strong evidence your situation is inherently too risky?"

[cruiser.livejournal.com]

It also could be evidence that the contingency that the contingency plan is designed to mitigate is so bad that any plan, even a bad one, is better than no plan at all. It also probably means that the situation is so unlikely to happen that the cost of testing the plan is greater than the cost of the problems created by the situation. multiplied by the probablility of it happening.

[docstrange.livejournal.com]

I agree. It could be evidence that the harm to which the risk points is hard to mitigate. One should then check the likelihood of the risk materializing over /n/ timeframe. Your approach seems akin to the Learned Hand rule in essence.

But if the cost testing of the contingency plan is high, and the risk is also likely within a given timeframe, then one should look to remediating/reducing not only the effect of the harm (as you say, the plan being better then none at all) but the risk of occurrence. That's where I come to the point that the "too expensive to test" contingency plan can be evidence of too high a risk: risk that perhaps could have been reduced in depth or likelihood. Such a plan can also be, I agree, for unlikely-but-high-cost risks, a cost that may be as much as it's worth given the low likelihood.

While the military makes plans for all kinds of extremely unlikely scenarios, I don't think business tends to - and for straight-up economic reasons; ergo if there is a plan it would be for a not-entirely-unlikely risk. That's why the comparison of my first quote (very business oriented) with the second (much broader) is interesting to me.

Good comments - thanks!