Are you models skillful...

If you get a chance, you should consider watching this TED Talk

I'm not asking you to watch it to increase your belief in the hoax that is global climate change, but rather to hear his argument about the models themselves.

He takes the position of asking if the models are "skillful."  In other words, do the models offer us more than if we didn't have the models at all.


He shows that we can compare the models with observation to get a sense of their skill.

Then he posits that we would much prefer observations of the future than predictions made from models. No doubt, but we don't have and can't get observations of/from the future. At least, not for some time. So, let's look at what the models say predicatively. Then he runs the models forward after showing that they accurately predict the past.

His basic argument is to get agreement on whether the models are skillful, then shows you a predictive use for them.

This, I think, was one area where I failed on a large project in 2012.

I had written a Project Planner tool that did some of what Juval showed in the AMC.  By using the product backlog and user stories the tool stepped outside the normal Agile tools and looked primarily at forecasting. I had built a fair number of features into the tool for changing the shape of how the team could approach the backlog.  PivitolTracker has some of these features built in and it's predictive capabilities are the closest I've seen in a commercial product to my "Project Planner" tool. I talked to the creator of PivitolTracker about this a few weeks ago - great stuff.

I don't believe I ever convinced management of that team that the model was skillful.  They always believed that they could somehow outsmart the data, if they only looked hard enough or were more clever. They never fully grokked that I had places in the model to adjust for being clever [changing the earliest start date of epics, changing the velocity, eliminating stories above a certain size threshold, filtering based on tags to include or eliminate tagged stories, the ability to save runs and then load and rerun them].

It's just stunning to think about the failure of the planning tool. I believe that how I feel now about that 2012 project is how the entire population will feel about the people of today in the year 2,150 or so. The data was ignored at great cost and peril.

I had put logging into the planning tool to showed who used it. I had gone to everyone with any need for project related data at all and given them one on one lessons on how the tool worked (it was super easy). I made the tool securely available inside and outside the corporate network. I used it during planning meetings projected on the overhead in team meetings.

In the end, not a single person from the project management group ever used the tool even once. The head of the project ran 3 scenarios on one day and never used it again.

After watching the TED talk above, I think this failure was because I never convinced them that this tool was skillful. Perhaps they didn't understand it and viewed it as some automated voodoo. Perhaps they didn't think it could accurately make predictions. Perhaps the thought that it did was just too devastating? If the reason was that they didn't think it was skillful, then that's on me. If they thought it was skillful but unacceptable, that's on them (that is, after all, kinda the reason you do forecasting!).

The problem is that it's now two years later and I'm not entirely sure which one it was. Convincing people that your model is skillful is a valuable approach.  No arm twisting, no cajoling, just do the models show themselves to be skillful. You're not trying to get them to choose a method - "your way" vs "my way." You are saying, we all agree that having this model is better than not having it. Project Design is the same.

By the way, funny anecdote. On the project above the PMs kept changing the dates I had on the team calendar for Agile meetings. I called them up and chewed them out and said, "leave my appointments alone." I got a call later from the head of Project Management.  Basically their boss's boss. I told him they couldn't touch my meetings and he proceeded to tell me I couldn't talk to him that way. I said, let's talk face-to-face and met him in a conference room.

I drew out my plan on the whiteboard and then I asked him, "What's your problem with me doing 'scrum-master' duty as I see best?" I'll never forget what he said, "You're too much of a purist, we don't do pure Agile." I didn't even blink, I fired back, "Great, but I've got a plan which I've laid out here and it follows the best suggestions of Agile (sprint planning, team estimation, retrospectives). Describe to me your plan?" There was a long pause. I saw him go from amazement to confusion to anger to acceptance all in that long pause, there was an almost imperceptible hunch in his shoulders and a look of resignation and he said, "Ok, we'll go with your plan."  His people never changed one of my appointments again without checking with me.

See, they didn't have a plan. To Juval's point during the AMC, if we do things the way he suggests we ned up having maps of the world. The other folks are just sailing where-ever the wind is taking them. They think the dragons on the edge of the maps are landmarks. We don't.

Still, we end up explaining and justifying our plans to people that don't even have a plan. Imagine how easy it is to shoot us down.  In the case of the conversation above, we were in the heat of it. The project was moving fast and I had called his bluff. Had he continued to try and submarine me, his lack of a plan would have been exposed to the entire organization, not just the two of us holed up in a conference room.

In the end, he got fire, his boss got fire, the two underlings that had been changing my appointments got fired. How?  Organizationally people at the very top stepped in and said we want to see the plan.  I showed them mine and they said, "what else you got" and the head of PMs tried to use it as a power grab and completely hosed the operation getting themselves fired in the process.

BTW, the reason they wanted more after seeing my plan is that it accurately predicted the completion date and effort required (both predicatively and what was to become historically) and that was JUST UNACCEPTABLE.  "It cannot take that much time and require that much effort. In the end it's JUST A WEB SITE and we build those every day."  It did.