Why is estimating hard?

1. You haven't done it before. Not exactly the same thing. You think you've done it before, but you haven't. If you had actually done it before, you would probably be faster this time, not slower.

2. Other things besides you have changed, or will change. Coworkers, company, customers, tools, economic realities, the weather. Either it's already changed, or it's going to change. That has an impact on deliverability, even for the exact same work. (but it's not the same work, because you haven't done it before)

3. You have changed, or will change. You're literally a different person since the last time you did it, and you haven't taken that into account, and now things take a different amount of work/time.

4. You're fallible. You just remembered wrong, guessed wrong, fucked up, forgot, etc.

5. Estimating is a skill. After a long career, you can look back at what you did and what happened, and look at what you've got now, and make an estimate. The more experienced you get, the better your estimates get. That doesn't mean they're accurate, because you can't control #2-4. But they'll certainly be closer.

My favorite is when I actually have a very strong feeling about an estimate, and I tell my boss, and he doesn't want to believe it, so he arbitrarily tries to find a way to get a better estimate. I end up right, he conveniently forgets my original estimate.

This is actually a really good summary of how projects should be estimated. A lot of people I know just focus on the core task or the minimum viable product, ignoring everything around it.

I especially find it frustrating that many PMs think that the instant a system goes into production, the project is finished and everyone should be putting tools down and walking away. In reality, the post-go-live support can take a surprising amount of time, but is rarely accounted for.

One particular customer, a telco, had PMs that started in the telephony space and were especially stuck in this mindset. Imagine digging a trench and laying down some fibre for a customer. The second the hole is filled and the fibre is lit up, the project is done. There is zero ongoing maintenance! Fibre doesn't need monthly patching, or regular replacement. It lasts decades. Sure, it might be cut by a backhoe, but that's break-fix, not project work.

Now picture the same mentality applied to general IT projects. Build it, mark the project as closed, and walk away. Never patch anything. Never upgrade. Never consolidate. Just leave things precisely as-built forever or until it breaks...

... or gets hacked and makes national headlines.

I’ve come to the recent conclusion that estimating is not so much hard, but uneconomical.

The amount of time/effort required to create a reliable estimate would probably double the cost of the completed project (ie, even after taking into account the overshoot on the informal estimate). The number of failed projects as a percentage of all projects would fall, but less projects would even start.

We conflate “can’t estimate” with “not willing to do all the work necessary to create an accurate estimate” because we know that it’s easier and more economical to just jump into the code, than it is to do a deep, formal analysis.

What’s really happening is that we are giving up accurate estimates to reduce overall cost.

Unfortunately, businesses only see the overshoot on time (which they incorrectly equate with project cost) and not the actual undershoot on budget relative to a project that has been fully planned.

(Note that I’m not really talking about waterfall. More “plan to build it twice, because you will anyway”)

The article description is "A deep dive on why projects always overrun and a framework to improve future estimation"

This was a personal investigation into my faulty estimation skills, off the back of a small project which became a medium project, which became a large project with no shortage of surprises, overruns and pain.

The blog post was born from an honest and thorough postmortem where it turns out most of the work was simply not expected – and so wasn't accounted for.

I then went a stage further and attempted to outline more general reasons for this and to visualise how might look in terns of time and effort. It's not meant to be scientific, but is certainly an interesting way to look at things.

Anyway, I hope someone finds it useful.

I really like this article. I think impressive how informative it manages to be in such a short space, and how the visual work helps tell the story. I’m definitely going to use this (with full credits) from time to time when the discussions on how a team needs to work with estimations.

What I wish it also included was the not-so-academic parts of estimations. Like how we’re often operating in systems where it is much better to over than underestimate, because it’s much better to never let a client know that you didn’t actually spend that X time compared to letting them know that you need X more. Or how estimating things you’ve done before can also be hard because people don’t actually deliver Y “story points” every week, because sometimes their children get sick, they sleep poorly, whatever. Or how even the most senior developer can spent 3 hours on something silly.

But I guess a lot of that plays into a different sort of discussion. One where we talk about whether a lot of the “control” parts of the work process are actually necessary or just wasted resources. And I don’t want to sound like I think it’s wasted resources, because it can be both and often it’s a combination. Personally I tend to avoid working in areas with too much “control”, if you estimate by anything less than a day, then I’d like not want to work for you. Not because I can’t do it, but because I hate spending time on things that aren’t “the work”. I’m not sure how to include that sort of metric into the discussion, but I think it may be relevant. Because I doubt I’m the only one, and while completely anecdotal, I wouldn’t want to invest in any of the places I wouldn’t want to work because they tend not to do so hot in the long term. So maybe looking at overhead is important? But my sample size is way too small to mean much.

A nag popup where the dismiss button just changes itself instead of dismissing the window? Oof, that's a new one.
Great article!

Microsoft Press' "Software Estimation: Demystifying the Black Art" is a good deep dive into the subject.

Found it via a talk by Jakob Persson at Drupalcon many years ago, he recommended a template-based spreadsheet approach where you estimate a project by splitting it into components and giving each a "confidence" level from 1-5, then adding fixed padding for various items like project management (20%), deployment (5%), buffer for unforeseen problems (20%) etc that essentially came down to "2-3x the initial estimate" but with some calculations to back it up, and made estimations systematically easy.

His slides are here:

The sheet template is here: - have used variations of it with great success for many years beyond when I shifted away from Drupal work.

Noting that his more recent blog articles now recommend a value-based pricing approach for the consulting model (which I agree with and also shifted to in my freelance days).

Good to see more writing on estimation for engineers, Agile's story points model is nice for teams doing the work, but in many kinds of work when you're running a team there's someone above you who signs the checks that wants to know what something's going to (approximately) cost before they sign off on it.

I wish companies would do this kind of analysis within their own projects. Probably not every project since doing this analysis would add overhead, but perhaps a small sampling of arbitrary projects.

How are we supposed to optimize things if we don't know where the time and effort is going to? How are we supposed to estimate, especially when a huge chunk of the extra time may be caused by an inefficient part of the process, caused by specific people (say, who like to schedule useless meetings), or caused by external factors?

Instead we see people choosing arbitrary things to blame, and business goes on as usual.

Here are the factors proposed by Code Complete (2nd edition):

- baseline: a program, designed by and destined to the same entity (programmer = customer) - a product (3x): designed by an entity destined to another entity (customer). - a system (x3): 2 programs communicating with each other - a system product (x3x3=x9): a system of 2 products

It's kind of ridiculous that people still believe that they can get better at estimating work they've never done before.

The amount of hours lost by teams estimating work is insane.

I have some thoughts on the vast differences between working on a team delivering firmware that is indefinitely the lifeblood of the hardware, vs working on a team writing hardware test tools that get used by people for one cycle (1-2 years) and nobody bats an eye when they're broken/unusable or difficult to extend to the next generation of hardware...

Clean, readable, beautiful C with well thought out descriptive names for everything, comments that get anyone up to speed on the code section. Self-descriptive, pleasant to work with.


MVP Python repo that has a super wide directory structure with poorly thought out grouping that takes 5x as long to memorize. When trying to understand a code section, told: "Just hit tab complete on the command line". Okay, that tells me the relationships between objects, classes, methods, containers, etc... But how does it work? Obfuscated not just from the users but the devs themselves.


The author seems to argue that estimates are off because of "unknown knowns", and can be improved by adopting an estimation framework that makes these items more explicit.

While this is certainly valid I would add that in my experience the most influential causes of major shedule disruption were the "unknown unknowns", wich are inherrent in any new creation, but which are exacerbated in software because of the extremely high and fractal dimensionality of digital assets. The smallest deviation from assuption can have an unbounded impact on the plan, often popularly refered to as a 'butterfly effect'.

Love the graphic, mind sharing how you made it?
Good related Freaknomics podcast about the planning fallacy:

Timeboxing (simplifying designs and eliminating requirements) is a far more important skill than estimation. Shrinking a schedule has much more impact than predicting it correctly
I always use the 80 20 rule for estimating stuff:

1. Everything takes at least 80 days (or hours if I‘m generous).

2. Multiply the result from step 1 by 20.

Clients will ignore high estimates and take the unrealistic one.
How do I get into these odd jobs as a professional developer?
I was asked to estimate how much time remained to complete all the components for a 3d game engine to make it perform excellently according to the design in my head. This was back in the days when you rendered every pixel on the CPU using every clever technique you could think of for performance, and you also needed similar geometry calculations as you do nowadays for higher level modelling and physics.

All the components, math, data structures, algorithms and APIs fitted neatly together but the big leaps in performance would only come when they were mostly done. One of those things where it's fast when all the parts are in harmony and slow otherwise. But hard to explain convincingly to others (especially those lacking algorithm or low-level performance experience) without a working demo. Even the APIs for other developers using the engine made little sense to them, until built, and then they liked them a lot. Sometimes you've just got to show rather than tell.

So I sat down and estimated systematically, going through each little item on the list as I imagined doing them: X hours here, X days there. Just wrote down everything that came to mind, and estimated each one independently. Let's just grind through this estimate one item at a time. I didn't pad it or try to make it fit any overall expectation. I just did the honest exercise of writing down each item to build and each X that seemed to fit that item.

To my surprise that was an intensive exercise that 2 weeks full time, just to write down all the moving parts in a list, which, somehow, all fitted together as a "simple" design in my head. It never seemed so large that it could possibly take that long to write down a list of one-liners. No wonder I had trouble explaining the whole scheme.

Was that a fantasy? Turns out it was accurate.

The sum of those independently estimated tiny parts came to about 2 years. Ouch! A lot longer than the design seemed in my head, which seemed closer to a few months, but somehow always slowed by life experiences, tired weeks, distractions from other projects' requests and trade show demos. Because of all those "unpredictable" factors I didn't believe my estimate was really going to be accurate, and figured that I must just be overestimating, adding too much padding, or not combining overlapping parts of items enough. I wasn't very exprienced with large projects!

And then I went ahead and worked on the game engine until it was working and performing well. I buried and ignored that document and just got on with each thing that made sense to me to work on as I saw fit.

The real surprise for me, the lesson which has stuck with me ever since is, that estimate and plan proved to be reasonably accurate even though I didn't believe it, didn't think hard about it when I wrote the items one after another, and completely ignored the document after writing it.

It took 2 years to get the components working really well together as my design intended, and by then just about everything on the list was done. I was unhappy it had taken so long, but had to admit that all the small items really did end up adding up as I'd written down earlier, written down, and the design really did work and perform well when the parts came together.

The list of things to build proved reasonably accurate, despite surprises on the way such as complete changes of target platform and new hardware coming out.

I learned so many lessons from that exercise and seeing it play out. One of them is, sometimes an honest but low level "add up the parts" estimate can turn out surprisingly accurate but is a huge grind to produce. Another is, perhaps I'm not as bad at estimating as it seems when I'm asked to throw out an estimate quickly without giving it that kind of detailed examination. So now, I'll consider doing an exercise like that, if there's time, and if I don't like the answer, I should treat it as a harsh fact rather than something malleable by wishing.

Another is, like many programmers, I'll sometimes imagine a design that seems simple to me on the face of it, imagined almost magically fast visually, yet it turns out to have a huge number of details when examined properly that take weeks just to list and years to implement, yet marvellously something so complex does end up working pretty much as conceived in the first place, including big leaps of performance, good features and useful APIs. That ability to conceive of something in a flash that actually works well yet is vastly more complex than it seems on the surface is a kind of superpower, if you're looking to push the state of the art or solve stuck problems, but it's also a problem for what are hopefully obvious reasons. I have to be careful to not answer questions like "can we do X" with "[thinks then] sure, just do Y and Z this way and it will work" because it never sounds like years of work when summarised, but it can be. And I have to be really careful with side projects, where simple ideas correspond to months of work and a few favourite ideas corresponds to centuries as I zoom in, like a macro that contains much more when expanded then you'd expect. Life is literally too short to implement most of the things I think of that I'd like to build, and I have a disappointed feeling that some of them would actually work, but I'll never be able to implement them to find out for sure.

Another lesson from that particular project was: I was asked if I could add a significant capability to the engine: change it from quasi-3d a bit like Wolfenstein/Doom to full 3d like Descent but including surface physics. Having just done that estimate I was loathe to rip up a perfectly good plan that already seemed daunting, and add a huge, onerous change, breaking countless invariants and assumptions. So I said no. Years later after leaving the job I looked back with new insight and realised I should have said yes, taken two weeks of low-pressure vacation to let my imagination conceive how to make the more versatile thing work (at every level, from rendering up to physics), and I would have ended up with a simpler design than the original that would have shipped more quickly. The more powerful, general solution was much simpler to reason about, just as performant, and had more value to the business long and short term, as well as being more satisfying to use.

All I needed to do was rip up some working assumptions and take a look with fresh eyes, to get something simpler and more powerful at the same time. So now, if I asked, I remember that "no" as a mistake. I think it's often best to evaluate "go bold" changes with an open mind.