Estimates pass through the three stages of truth...

All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident.
—
Arthur Schopenhauer, German philosopher (1788 – 1860)

The older I get the more I appreciate this quote.   It even works for the little things like when I'm asked to give an estimate on a software project.

Retrospectives are as important for "successes" as they are for failures

The generation before me still believed that children should be "seen and not heard."  I was heard from plenty.

Today children are told, "You can do anything, be anything you want."  I was in my twenties for most of the '80s, and I saw this change happen.  Before the '80s generation kids were told, "get a job" and the ones that didn't were told, "you can't live here anymore ya hippie."  I don't ever remember someone saying, "you can do anything you want" when I was a kid.  Now I over hear kids being told that all the time.

We do plans and estimates so we can tell the story

Why do the people that work on a software project create project plans and estimates?

We create plans and estimates so that we can tell the story of the solution and, through the telling of the story, we can practice our craft.   When we do it as an ensemble or team, we do it so we can work and practice our craft together, bringing all our individual skills into one cast of characters and producers.  We do it to bring joy and amazement to our audience and we use our stories to help solve their problems.

Business needs plans and the tools of plans, such as estimates and budgets.  As software story tellers we often sell our skills to make our living and those that manage the money - whether it's clients or employers - need to use it wisely and responsibly. They want a plan to show that they are not wasting the money and are able to account for it responsibly.

Not all software is created for money.  Many open source efforts exist simply because of a desire to tell that software's story.  Those projects usually have far less elaborate plans. Ironically, they are also some of the most successful stories ever told.

The need to plan, to organize budgets, time and resources is not to be dismissed. Some methods of funding a project cannot happen without an estimate or plan. This is the professional part of being a modern day story teller.  The bulk of this message deals with the craft and the means by which the craft accommodates the need for estimates and plans.

To be professional, we need to do our best to help the business with it's plans, estimates, forecasts and resourcing, but it's not what drives us.  It's not what fuels our creative juices or helps us hone our craft. For that we leave the profession behind and turn to the work of the craftsmen.

Defer risks in your estimate

One tactic that is useful for keeping the cost of a project plan low is to externalize as many of the risks as possible from the main flow.

Identify risks and for each risk identify a trigger which will indicate when the risk is happening and the team will need to deal with it. Once you have this list of risks and triggers, associate a cost for dealing with each risk. Sometimes the cost will be in dollars, sometimes in days, often it's both.

You cannot just list the risks.  Each risk must have a trigger signifying when it is happening and that it needs to deal with it.   It is the triggers that get written into the contract.  Let's look at a fictitious example.

Urgency is a trap that keeps us from improving our processes

Ask someone why they don't change their process to be "better" and they will often tell you, "I just don't have time."

I see it a bit differently.  It's not time that's the problem.  It's urgency.

If it's worth improving it's worth improving now.  But it's not the potential gains I'm worried about. I'm worried about inaction because you are paralyzed by a lack of urgency.  One of the best ways to get this sense of urgency back is to think in terms of small short experiments that can be done in "this sprint" or in the next week.

Try not to confuse an estimate with a plan

Let's suppose that a manager request an estimate from a technical team.  "Manager" in this sense is anyone who wants the estimate.  It can be an upper-level manager or an account person or perhaps even the client if you're very cozy with them.

The technical team will look at the requested feature(s) and discuss the effort, risks, and experience levels required to do the work and offer an estimate.  The estimate might be in the form of user story points if your team has an established velocity or it could be in days or hours if you have a reasonable work breakdown.

The trend I've noticed is that in response to the estimate, the managers will say something very strange.  It'll usually go something like this, "Nope.  That's too high."  (If it were too low they'd pee their pants and be quiet.)   "Too high!  What are you talking about?"  The manager will then respond with the damnedest comment, "Nope - too high.  For us to get the work your estimate needs to be $X number of dollars and finish in Y number of days."

Technical teams are being asked for an estimate where what is wanted is a plan.

You think you're better than those silly story points?

Suppose you have a team which estimated a set of new features and was expecting the work to take them 3 months (90 days) then, like any good software project, they launched in 5 months (150 days).

The 90 days represented how long they thought the work would take.  The 150 days represented how long the work actually took.  In the end, the 90 days worth of work took 150 days.

If we mapped the days at the beginning of the project to actual days we would find out that a "day" before the project began was equal to 1.7 days of actual time [150/90=1.7].

The notion that if you only had your estimate in days and not those silly story points it would "be better or more useful" is nonsense.   Knowing that the project was scheduled for 90 days doesn't help you a wink when it comes to doing the work.  "It'll take as long as it takes."  We are hoping that 1 project-day is equal to 1 actual-day.

Velocity converts to time, not story points

Suppose for a moment that you have a collection of features you want to build into a product. 

You sit down with your team, brainstorm some user stories and then score them.  When you are done you total up the story points and there are 100 story points.

Suppose then that I tell you this project is scheduled to be delivered in 3 months.   Hmm, that's interesting.

Now suppose that you are also consulting on another project.  You have a collection of features you want to add to this other product.  This project has an entirely different team.  Like the first team these folks have been working together for a while and many of the team members go back to version 1.0 when the product was launched.

You sit down with this team, brainstorm some user stories and score them.  When you are done you total up the story points and there are 100 story points.

You look at the project plan and see that the project is scheduled to be delivered in 9 months.

Both are 100 points but one is a 3 month forecast and the other a 9 month forecast.  Is one team's assessment of the user stories right and the other wrong?

It's likely that both are correct.

Estimates with Risk Assessments, sounds boring, do it anyway...

So Ed sent an email which explained the math around calculating variance.  That's useful, but probably overkill for our purposes but it's worth going over why.

In fact, what we are trying to do is well understood via "Standard Deviation."  Everyone is familiar with Standard Deviation.  In charts it looks like this:



If we take this bell curve and put the earliest date a task could possibly be completed as the point 3 standard deviations to the left and the longest a task could possibly take as the point 3 standard deviations to the right, we would have a risk assessment showing a range of likely completion dates for our task assuming it's risk follows a natural bell curve (as we'll see in a moment that the hump for most software tasks is to the right of center).

All things being equal, if you estimated a completion date for a task as right in the middle, you would have 50% chance of being right and a 34.1% to 84.1% chance of finishing within 1 standard deviation of the middle (median) date.   How many days or hours is that?  Well, that depends on the width of the graph which is found by estimating the earliest possible start date and putting it on the left of the graph and the longest possible completion date on the right.  Then draw your bell curve between the two.  Simple.

Of course you should consume feeds, but I can't get there fast enough...

It's really hard for me to separate data from view on the first go around. 

I grew up on the web during the heady days of server side MVC.  Server side MVC is still heavily used in the way that steam engines were used into the 1900.   Still, I find it very challenging to create a useful feed and a useful browser side view all in one go.

Agile needs front-end planning model to move away from the current back-end reactionary model...

Here is an (edited) conversation I had over chat with a friend today.

The point of this conversation is that I'm exploring the idea that what Agile is missing is front loaded change feedback.  Something along the lines of, "Ok, we can do that but it'll cost you $5."  The current model works like this, "OK, we did the work, that was days/weeks ago, and it cost you $5."

This is not about controlling change or costs.  It's about visibility.