The Estimation Game

In trying to put down some words on project estimation I’ve had to come to terms with an internal contradiction which I guess goes all the way back to the origins of software engineering. Logic tells me that because the ultimate goal of a software project is a series of machine instructions that perform some useful task, and because at the lowest level these are almost wholly predictable in their nature (strange microcode bugs notwithstanding, although these are far more rare than they used to be), and because enough analysis should empirically capture precisely what this useful task looks like, predicting how long a project will take should be commonplace, reasonably accurate, and scientifically methodical.

Yet experience and regular news reports tells me that all of that predictability and most of the scientific method and accuracy is a wistful mirage. And I’m not happy with this. It’s the source of so much frustration when doing portfolio planning. You want (and need) accuracy. The amount of budget approved, the people employed, and the time available sit there on the page, specified to three decimal places, and somehow you have to determine a subset from the pile of potential projects that not only fits with these figures, but will also deliver the best bang for buck in the coming year.

Reconciling logic and experience in project estimation is like trying to understand the Chewbacca Defence - it just doesn’t make sense. I’ve read lots of papers about how to estimate and I believe most of them. I’ve looked at models to construct and analyse requirements and can not fault their mathematical basis. The problem is the real world. The more sane and scientific the approach, the more bureaucratic and process-centric it appears, the less the business likes it. They just want to say what they want, for you to magically know what that means, and for accurate project estimates to appear on the slide deck. Here, look at the monkey. Look at the silly monkey.

Despite this contradiction, the question “how long is a project?” has to be answered in some form because it’s this that drives the scheduling. It may be tempting to just start the highest priority projects first, but if they take more than a year there may be no tangible output during that year, whereas a balanced portfolio would give you some visible results and some progress towards future delivery.

And at this point we’re not concerned with the philosophical debate of when is a project done?, which hangs in the air during that twilight period after go-live, as questions are asked about which bugs are really bugs, which can be worked around, and which can be allowed to dissolve into business-as-usual. This is about the beforetime. Before the project is a project. Before the people come with their tools and their plans. Before the arguments and the meetings, the changes of direction and design. When the project is just an idea. A twinkle in the eye of a wistful staff member, who dared to say “wouldn’t it be great if..”

Because someone heard that and replied, “Yes. It would be great. But how long’s it gonna take?”

Recently, I wrote about annual planning and the McFarlan Matrix. I showed, I hope, that it can help you assess the kind of company you are, and therefore the kind of projects you should be taking on. If you don’t use some sort of structured approach then the default, and unfortunately the quite common, practice is to resource projects on the basis of first-come-first-served or who-shouts-loudest. Meaning you’ll do stuff, just not necessarily the right stuff.

So let’s say we’ve got ten ideas in the strategic innovation category. The CEO and the board agree that all are worth a shot this year and they recognise that we can’t do all of them at the same time. Constraints of people, skills, time, budget and amount of change humans can handle at once, tell us that we should only bite off what we can chew. So we need to scope all ten and see which ones are chewable - and because we’re agile, we remain conscious of the fact that planning isn’t the same as delivery, so we also need to do this quickly.

Let’s take a walk through that wistful mirage.

What do you want?

Logic : I’d say the most logical place to begin sizing a project is in understanding the problem we’re trying to solve. If we can wrap our arms around this then surely we’ll start to see the plan emerging. Focusing on the problem is good practice because it stops us jumping to the solution too early. Possibly my top anti-pattern of all time, one that I feel like having tattooed on my forehead to save having to repeat it so often, is projects that centre themselves around what they perceive ‘the solution’ to be (often naming themselves after it). As soon as the focal point becomes an answer, not the question, you are doomed.

Reality : What anyone wants is nearly always the wrong basis for a estimating a project, because what most of us want remains unfettered by annoyances like reason and rationality. As Steven Lott said:

I had a customer claim that they required 24x7 availability. But they would not consider any hardware changes, and the hardware they had purchased couldn’t provide the level of availability they were asking for. When I brought this up, they claimed that 24x7 wasn’t really a “requirement” it was more of a “goal.” My follow-up question was “Where’s the line? What’s the least availability that you’ll tolerate before you sue me.” They chuckled nervously, and said that any talk of lawsuits was irrelevant.

It isn’t irrelevant. “Required” means required, as in “if the system doesn’t do this, you don’t get paid.”

If 24x7 availability is a requirement, then “Architect gets a Bentley” is also a requirement. There’s no business justification for either position. They were clearly a 12x5 operation, and could justify requiring 18x6 to cover weekends and west-coast timezones. They could not show a business reason for 24x7 any more than I could show a business reason for a Bentley. It may have been my goal, but I couldn’t justify it as a contractual requirement.

In order to find the line, perhaps we’d be better asking what’s actually needed.

What do you really need?

Logic : If you really really need it, then we’ve got to build it. This is the core of the project. If there’s time we can embellish this with what you want, but if we agree what you need in order to remain competitive we can estimate around that.

Reality : With some careful politicking and canvassing we may be able to itemise what’s needed, but not in enough detail for us to size the project required to deliver it. Every time we ask a question another level of detail is revealed. The sirens of Analysis Paralysis are dancing around our boat, drawing us towards the rocks. We fight them off with a clearly communicated MoSCoW list but this just makes it worse.

Because it turns out that not all must-haves are really must haves to all stakeholders. Depending on who you talk to in the business you get a different view. Financial controls are must-haves to finance, but not to marketing. At the project level it becomes hard to define exactly what we mean by must. Ideally it should be do-this-or-we-die, as in, if this feature is not present in the final application, we will go out of business. It’s clear that too many must-haves could blow the schedule and yet we haven’t gone out of business despite not having done this project before now, so are any requirements must-have?

Except for legal requirements for example, or features fundamental to the nature of the application (an e-commerce application must-have an ability to collect payment) must can be a pretty relative term.

How are we going to build it?

Logic : Time is ticking away so let’s pare the project down to a core of must-haves, even if some of them are contested. Now we can model a couple of likely solution options and estimate around these.

Reality : How long any one solution option might take, for estimation purposes, depends on an army of elusive factors: the maturity of the technology, our procurement process, it’s appropriateness for the task, the skills available in it and more than anything else, who will actually be available to do it.

Who’s going to do it?

Logic : If we assign a core team of our best people to this task, we’ll crack it faster and get this sucker delivered.

Reality : Every project sponsor wants the same core team of our best people, the ones you get may not have the experience or skills, they might be sick at a critical time, they may be pay-check players who only work their contracted hours and no more.

When do you need it by?

I covered this last time because it turned out to be a bigger topic than would sit easily here.

Logic dictates that this is a key question to ask, the reality is that most projects have this predefined and that the date is not one that will permit a strategic option to be considered. It may even be that the quickest and dirtiest option can’t be done in the time, or will fail to respond to a changing requirement at a critical moment.

The Four Candles Project

Bugger it. Let’s say you have a set of reasonable business sponsors. They tell you what they want. You filter out what they need. There’s a perfect and mature technology for it that you understand well. You have a core team of your best people ready to go. You can hit the desired date. You’re set, right? The estimate, the business need and the project all in perfect harmony? Can we stick this one in the portfolio for board approval without fear that we’ll blow the delivery?

A man walks into a hardware store and asks for “four candles”.

The shop assistant verbally confirms the request, climbs a ladder, fishes around in a big cardboard box on the top shelf, comes back down and lays four candles on the counter top.

“There you go”, he says.

“No”, the customer says, “Fork handles. Handles for forks.”

In September 1976 the famous Four Candles sketch by the Two Ronnies was broadcast. It perfectly sums up what it’s like to gather requirements. Even after understanding the basis of the gag, when you are alert to possible misinterpretations, when the customer says:

“Got any plugs? A rubber one, bathroom”

It’s hard to spot where the confusion can come from. Turns out he wants 13 amp power socket plugs, not drainage-prevention sink plugs. And the list goes on - sore tips/saw tips, hose/hoes/o’s, p’s/peas, windscreen washers, car washers, dishwashers, floor washers, back scrubbers, lavatory cleaners, floor washers, half inch washers, etc.

In case you aren’t familiar with the routine, and for those who’d like to briefly relive it (now that I’ve ruined it by over analysis), you can watch it here.

The point is it’s easy to misunderstand a requirement, it’s common, and happens even when you are consciously looking for it. And if that’s the fundamental reality, have we got any hope of ever estimating how long a project will take?

Arise Sir Agile of Snowbird?

Maybe I’m just making a rambling and rather dull case for agile techniques. Well, sort of. There’s no doubt that an agile mindset recognises all of the above and addresses it by adopting a “no surprises” approach to solving each problem. But we’re not at delivery yet, so although we can prioritise individuals, interactions and customer collaboration very nicely, it’s a bit harder to respond to change when our ‘working software’ is, in essence, a plan.

Or is it? Maybe we should first ask ourselves what we really need here. We think we need a plan for the year, but maybe that’s what we want and can’t have (with any degree of certainty). As Scott Ambler says “you can accurately plan in detail only for nearby tasks”. So I don’t see agile coming to the rescue for Portfolio Planning, but it does remind us that accuracy at this stage is illusory because we are entering the Cone of Uncertainty and (therefore) any time spend garnishing this illusion with detail is wasted.

There is one important feature of agile project management that helps construct portfolios though - velocity metrics. At the end of each iteration on an agile project you have, to different degrees, delivered some things, failed to deliver some other things, and debated changes to things you have yet to deliver. Ideally mostly the first and next-to-nothing of the other two. This is the project’s velocity and each iteration it transparently lets everyone know how work is progressing, estimating future ability by using actual past performance. Velocity Charts don’t claim 100% accuracy over what will happen, but they do show what’s most likely.

Precisely, as it happens, what we need to construct a portfolio.

Summary

When you start a new financial year some sense of direction is a necessity. Best value for the IT spend available requires that this direction is underpinned by a balanced portfolio of projects that come with some degree of delivery confidence. This confidence can only follow an analysis of each potential project, with the proviso that analysis is at once a splendidly engrossing activity, potentially misleading, and not delivery.

In this analysis some form of high-level requirement gathering is useful but only in a very broad sense, because requirements are an abstraction, and not a very good one, of what work is required and therefore how long it will take. All requirements, however solid, are candidates for change, which calls into question the value of any plan over anything but the shortest time-scales. There can be no preconceived accuracy.

But all is not lost. The plan accuracy of projects already delivered is 100%. If you know anything about a potential project you have something to work with, because you can assess it against what’s been done before. The more you learn and record the more accurate this will be. What we need now is a simple way to model this, so we can churn out some quick answers.

And that’s for next time.

Notes

1. Pretty much all the major pitfalls are summarised nicely in the Ten Deadly Sins of Software Estimation on Steve McConnell’s Construx site.

2. Despite careful planning, the estimated delivery date for this article was delayed by approximately three weeks due to the birth of my new daughter, Meryn. Like so many best laid plans, she arrived two weeks early and I delivered three weeks late.