10 June 2012

Probability Management for Project Estimates

Calculating with averages, point estimates and expected values runs afoul of the Flaw of Averages. The answers we get aren’t just wrong; in the case of project estimates, they’re hopelessly optimistic. This bears getting hammered on. The math, the formulas, the computation is fundamentally, intrinsically, wrong, on several fronts.

And the errors are one-sided; they underestimate cost and time.

There’s a long dog-eared list of things that contribute to project failure. Most of them have to do with human factors or natural disasters like hundred-year storms – hard problems to deal with. Here, on the other hand, is an easy problem to deal with – just fix the math.

Errors Don’t Average Out

One problem is the myth that errors in project estimates “average out.” They don’t.

At the highest level, being early on one project and late on another doesn’t balance the books. There’s rarely more than a minor benefit to being early – not enough to balance the cost of being late. Also, the range of outcomes is between a little early and a lot late.

Within a project, one factor that stands out is that there’s a lower limit to a task’s duration, but no upper limit. This is an unavoidable bias.

There’s another bias related to Parkinson’s Law. If the estimated task duration is over-estimated, the developers will cruise to an easy on-time finish; if it’s under-estimated, they’ll be late. On average, they’ll be late.

The Expected Finish Isn’t Expected

Then there’s the use of expected values for estimating.

The probability of finishing on or before the expected finish for a task is close to 50:50 – a coin flip. That’s what “expected” means: the weighted average of all the values the variable can take on, usually close to the median. If the expected durations were carefully estimated and they’re reasonably reliable, half the activities will finish later than expected. A bit of irony there.

If, as predicted, half the activities are later than expected, the project will be later than expected unless it’s restructured – usually blowing the budget or sacrificing quality.

I’m used to getting some push-back here. My response is that, if you wield such magic that all or the majority of your activities complete on or before their expected finish, or that's what you plan on, you’re sandbagging the meaning of ‘expected’ and pushing it up into the high percentiles.

This is easier to do right, as a transparent, informed and auditable decision, with Probability Management. To avoid confusion we should stick with the standard definition of ‘expected’.

The Concurrency Tax

The error that’s most relevant to projects is what happens when a set of work flows converge on a milestone. The CPM expected milestone date is the latest of the expected finish dates.

On the other hand, the probability that all the work flows will finish before a particular date shrinks exponentially with the number of concurrent work flows. It’s the product of the individual probabilities. To compensate for the reduced probability, to bring the probability back up to what we mean by ‘expected’, the expected milestone date must be later than the latest expected finish. This is true whether by 'expected' you mean 50% or 95%.

Twisting the knife, there’s no valid way to calculate the milestone date given only the expected finishes.

Models with Probability Management

Probability Management is a set of tools and techniques for avoiding the Flaw of Averages and all these problems.

Although it can have a big impact on the quality of the plan and the success of the project, Probability Management has a very narrow role in the plan’s development.

All we’re doing is calculating with distributions instead of the single numbers we would otherwise use. We’re fixing the bad math that plagues conventional estimates.

All of the factors analysis and negotiating that would normally go into producing an expected time or cost for a particular activity still needs to be done. The reliability of a subcontractor, developer productivity, hostile inspectors, scope creep, and everything else that can affect time and cost still need to be factored into the estimate. But, instead of fixating on a single number or three-point spread, we compose a sample distribution that quantifies the uncertainty in the value we’re estimating.

And we use these distributions for calculation and rolling up time and cost. The arithmetic is the same as that we would use in the conventional model, but repeated for hundreds or thousands of different input combinations. That’s the part that cures the Flaw of Averages.

In other words, we’re not changing the way we plan, just the way we compute, and the way we interpret the results.

No comments:

Post a Comment