04 March 2010

Pert Loses at Monte Carlo

It's pretty clear that calculating with averages is unwise; it's particularly true when we're estimating the resources needed for a project.

If all the tasks were strung out in series and just added together, the errors would tend to "average out." We might be safe in assuming that errors in estimating would be high and low and would cancel each other out. If only real projects were that convenient.

Project activities aren't usually strung out like a string of pearls. They look more like Figure 1, one of the classic project plan segments used in descriptions of PERT.

Figure 1. Classic PERT segment

We'll play with this to get a sense of the difference between PERT predictions and Monte Carlo simulation .

There are six activities in the plan, each with an optimistic, most-likely, and pessimistic estimate. Figure 2 shows a spreadsheet used to estimate the results using the PERT formula. The yellow cells are the input and the green cells are the output. The three paths (C, E, F) to the final milestone are each additive, but the finish date is the maximum of those three paths, in this case 221 days.

 Figure 2. PERT Estimate

Some implementations of PERT will use this to derive a shape for the finish date, usually a beta distribution. Two things to remember about this are that the distribution will be anchored to the 221 days and that the uncertainty in the inputs is not accounted for. The beta is more of an incantation than useful information.

By way of comparison, Figure 3 is the histogram and cumulative curve we get from the same inputs converted into shapes and run through a 400-trial simulation.


Figure 3. Monte Carlo Simulation

To do this right, the shape associated with each task would be derived from historical data, expert estimations, or both. For this exercise, however, I wanted to keep the same data as was used for the PERT calculations. I used the three numbers as the min, mode, and max of a triangular distribution with 400 samples.  This is fairly coarse and a good way to signal that we don't know much about the actual shape. It also tends to give too much weight to values close to the mode, so the resulting confidence intervals will be tighter than they would likely be with real data. In any case, the triangular shape has a firm minimum and maximum, which is an advantage over other distributions that might be used here.

The first thing Figure 3 tells us is that the 90% confidence interval runs from about 215 days up to about 255 days. The next thing is that the 221-day PERT finish is at the 25% cumulative probability mark; the odds are 3:1 against meeting that estimate. Even money, where we're as likely to be early as late, is at 230 days.

A lot depends on the client's risk tolerance, but if there's a need to synchronize with other activities, it's likely that we're going to plan for 250 days, six weeks later than the PERT estimate.

This example involves a very small project. If we apply this to a large project, particularly one with a lot of parallel paths, the PERT errors get very interesting.

If you'd like to play with this, there's an Excel spreadsheet at http://smpro.ca/crunch/pertsim.xls

The chart is interactive, so if you change the inputs (in yellow) and press F9, you'll see the data tables and chart change. It may be slow, so wait for it. If you find an input combination that makes the PERT calculation more pessimistic than the Monte Carlo, please let me know because, so far, I don't think it's possible.

11 comments:

  1. Hi Marc,

    Thank you for this great article. You answered the questions I had from your previous post "The Project Uncertainty Principle".
    Do you have a an excel sheet that is able to account for more activities?

    ReplyDelete
  2. Maybe I have missed smth, but when I apply PERT to exactly this example I'm getting that

    - we can be 95% sure to fit in 243 days
    - we can be 99% sure to fit in 253 days

    that seems to be quite in line with your calculations.

    What's the gotcha?

    ReplyDelete
    Replies
    1. Hi Andriy
      Keep in mind that this is a small model with only a little leeway for differences in results.

      Two gotchas: (1) Your PERT numbers are optimistic compared to the Monte Carlo (which we now call SIPmath) results shown in the CPD curve. (2) Normal practice with PERT is to project the expected value with a 50% probability of meeting or beating it.

      Delete
    2. Hi John,
      and thank you for replaying in this really old post?

      Do you mean that you are taking less then 99% probability when you say ' it's likely that we're going to plan for 250 days'?

      Delete
    3. And btw, concerning your second comment.

      Do you have an idea why is 50% confident PERT estimation is a common practice? Does it have smth with understanding the method and the math behind it? Or smth else?

      Delete
    4. Hi Andriy. I suggested 250 days to get a 90% probability of success when other activities depend on the outcome of this one. Obviously, in real life, a manager would decide on a target based on their risk tolerance and the context.

      The 50% mark is the "expected value". It's a tradition that goes back to the days when plans were made with pencil and paper, and simulation was out of the question. We have computers now, so taking a fresh approach is long overdue.

      Delete
  3. Hi John, I found your tool super useful, thanks a lot.
    But I can't understand what is frequency on graph and why we have a small table section with data in the right With fixed days? Which are actually correspond to Days on the graph.

    And when I am pressing F9 one time, Macros runs through the values in 22nd row? Or through the all values in 400 rows? That's a little bit unclear.
    If the second what mean the values in 22nd row

    Thanks in advance

    ReplyDelete
  4. Mr Thibault, your articles have been very enlightening to me, you helped me realize the futility of trying to "fix" PERT/CPM.
    I have now built a free online Monte Carlo simulator for projects, and would really appreciate if you helped me test it's accuracy and gave me your honest feedback about it. You can find it at http://montecarlo.atehortua.com/

    ReplyDelete
    Replies
    1. Hi Vladimir
      Very Nice. The only thing I can think of to improve it is to add a little help. It took me a while to figure out how to edit the activities.

      Delete