—Sam Savage. The Flaw of Averages.
There's a Dilbert cartoon from 2003 in which Dilbert is asked for a description of his project and its projected cost. He responds by declaring that, "The Project Uncertainty Principle says that if you understand a project, you won't know its cost and vice versa." I don't think Adams knew he was on to something.
Since the financial meltdown exposed the players' mishandling of uncertainty in their analyses, there have been a slew of books and articles written about uncertainty and a few older books have received renewed attention. A lot of them have found their way to my bookshelf. In just about every one, somewhere in the first few paragraphs, the author refers to the problem with software project estimates in order to demonstrate the need to quantify uncertainty—and it's never mentioned again.
In this article I'll be picking up that thread and I'll start taking it to its logical conclusion. The logic applies to all types of projects, but I'll focus on software development because it's the most difficult kind of project to plan. I'll start by taking a look at why conventional estimates tend to be optimistic even when we take that tendency into account. The next few articles will look at how we can use probability distributions to get better estimates.
The Estimating Problem Stereotype
Let's say we have a project decomposed into some number of tasks. In each case the task has been kept small enough to estimate reliably and every reasonable measure has been taken to inform the estimate. In each case we've taken a conservative tack and the probability of finishing the task within the estimated time is 90%. That's a very high level of confidence, meaning that most of the time, tasks should come in quite a bit early.
If we were perfectly clairvoyant, we'd see that when the project was finished, 90% of the tasks were completed within the estimated time, and the remainder weren't far off the mark. We would also see that the project finished late.
How could such a good start with the parts result in failure for the whole project? A facetious response would be that "the whole is greater than the sum of its parts."
Let's take two of our carefully estimated tasks. For simplicity we'll have them both estimated at five days, and look at the two ways they can be combined.
So how do we get back to our desired 90%? The answer is "insufficient information". We don't know. We need to increase the number of days, but we don't have enough information to guess at how many.
The second case, where the tasks are in series, is even more interesting. With tasks in series, the estimated time to finish the series would be the sum of the individual times, in this case ten days. So what's the probability that the sequence will finish in ten days? The answer is "insufficient information".
There are three possible scenarios, whose probabilities add up to 100%:
- Both tasks finish within five days, and the series definitely finishes within ten days (81%).
- Neither task finishes within five days and the series definitely finishes in more than ten days (1%).
- One task finishes earlier and the other later, we don't know whether the early compensates for the late or not, so the combined result is unknown (18%).
PERT tries to manage this problem by using three numbers, optimistic, pessimistic, and most likely. The "pessimistic" corresponds to our 90% number and the others have even lower task-level probabilities. This results in three wildly unlikely numbers instead of just one.
The way out of this quagmire is Probability Management—calculating with probability distributions to quantify and manage uncertainty.