Variation in software estimation

Last week I visited Warsaw, Poland for the first edition of Agile By Example.

The organization did a great job setting up this conference.  For instance, they opened a call for solutions on their website, where people could list the topics they are interested in.  This helped to create a balanced program with blocks of two sessions on one topic.  After each block there was a 15 minute slot available for the audience to asks questions to the speakers, which sometimes led to interesting discussions on stage.

Last but not least, they found quite an interesting venue. The Centralny Basen Artystyczny used to be a swimming pool and has been renovated into an event venue.

The audience was actually sitting inside the swimming pool!

I was invited to do a talk on Agile Estimating & Planning.

The session was based on these 4 questions:

• Why do we need estimating & planning?
• How do we estimate & plan?
• Why do Agile projects need a different style of estimating & planning?
• How do we estimate & plan in Agile projects?

Part of the second question ‘How do we estimate and plan?’, I explained the differences between guessing, counting and measuring by asking the audience to estimate the number of people in the room using the techniques guessing and counting.

I collected the results afterwards and visualized them in this graph.

What are my conclusions from this data?

1. There’s a lot of variation between people’s estimates.
   While it seems not very hard to estimate the number of people inside the room, knowing that this was not a very big conference, estimates ranged from 400 to 50.
2. Guessing has a higher variation than counting.
   You can see on the graph that the range of estimates was much narrower when we used counting techniques.
3. Most of us over-estimated.
   This doesn’t compare to estimating the duration of tasks in software development, where research has shown that we typically under-estimate 20% to 30%.
4. Measuring gives us a pretty accurate estimate.
   By measuring the number of forms that I collected afterwards, I had a pretty good estimate on the number of people present.  Although I still couldn’t be 100% sure it was accurate since some people might not have participated.

When I think of the reasons why there’s so much variation in these estimates, I realize that I might not have been clear about the scope of what we were estimating.

Estimating the number of people present can be understood in different ways:

•  Estimate the number of people sitting in the swimming pool
• Estimate the number of people sitting in the swimming pool & the ones sitting on the sides
• Estimate the number of people sitting in the swimming pool & the ones sitting on the sides & the people at the registration desk in the back.

This confirms that when you’re estimating, the scope should be clearly understood by all.

We often have these issues in teams when we’re estimating software features & tasks.  But for me,the underlying message stays the same.

Count or measure if you can, only use guessing as your last option since it is often the least accurate.

Tags: Agile, Conferences, Estimating, software development, sprint planning, story points