Thursday, 31 March 2016

Why is Iteration Better than Specification?

If you've spent time working in Agile environments, this might seem like Agile 101 - sprints are better than Waterfall, for well-known reasons - so why, on a blog which tries to delve a little deeper into how and why Agile methodologies work, would would this need to be explained?  Well, if you're one of those people that already "gets" Agile, this doesn't need to be explained.  But lots of people haven't really been exposed to Agile methodologies, especially if they don't work in or with an Agile team, so this explanation is for them.

Firstly, a little primer on Agile.  Unlike the traditional Waterfall methodology where you specify, build, test, document and release a whole piece of work, Agile takes an iterative approach.  In practical terms this means splitting up the available time into periods called sprints, which last between 1 and 4 weeks (2 weeks is probably the most common, as it provides a good balance between the focus of a short period and enough time to get meaningful work done).  In each sprint the team works on items taken from the backlog of work with the aim of completing enough items to release an increment of finished software to the customer.  The backlog is analogous to the specification that is found in Waterfall, with the caveat that in the backlog the specification has been broken down into much smaller tasks (this is called backlog grooming) to allow each sprint to take on part of the work.  The backlog is also ordered so that the highest priority work is at the top, and will be the next work done by the team.

At the end of the sprint, the completed increment of work is shown to the stakeholders, and they can use it to see if it meets their needs.  Any changes they need are added to the backlog, given an appropriate priority, and will be done by the team in a future sprint (the higher the priority of a work item on the backlog, the sooner it will be done).

In a Waterfall project all of the dev/test/doc is done up front based on a complete specification, and the finished software is delivered to the customer as an entire piece.  The customer then performs some kind of review or user acceptance testing, critical bugs are fixed by the dev team, and then the software is accepted (or not).  Normally at this point the software will enter a maintenance mode where changes or fixes are done on an "as and when" basis by the dev team, usually in return for an annual maintenance payment by the customer.

A sensible question from a Waterfall perspective is therefore: Isn't grooming the backlog, working in sprints and constantly re-prioritising the work an administrative overhead that wastes time?  Time that could be better spent developing the software from a complete we do in Waterfall...?

It's a fair question, although the answer is a definite "No", as I'll now attempt to justify. 

The primary aim of an Agile development methodology is to increase quality.  That means the quality of the actual work, as measured by unit test coverage, number of known but unfixed bugs, etc, AND the quality of the finished product, as measured by the customer's satisfaction with the functionality.  We've looked at this before with regards to the classic problem of "Good,
Cheap, Quick: Pick 2" so I won't rehash the whole thing, but I will quote from that post:

"This means that in Scrum the overall solution requirements are not fixed at the start of the process but rather become a variable to be managed through the life of the project.  The 2 fixed points of Scrum are the increment delivery date - the end of each sprint - and the quality of the increment .... de-scoping allows quality to be prioritised and the commitment from a mature team with a known velocity allows the Definition of Done to be met at the end of each sprint and thus meet the increment delivery date.  As more increments are delivered, the overall solution requirements can change as much as is necessary to meet changing customer demands.  So rather than Good-Cheap-Quick referring to the total deliverable at the end of the project, it relates to each individual sprint.


This all means that the Good-Cheap-Quick conundrum is somewhat less of a burden in Scrum, because the ability to de-scope covers Good, the ability to re-prioritise for each sprint enables efficiency, and therefore covers to a certain extent Cheap, and the short length of the increments covers Quick."

Meanwhile (taken from the same article):

"Contrast this with a Waterfall process, where the overall solution requirements are a fixed point around which quality rotates. Nominally the time to delivery is also fixed, but it might (and often does) change as technical debt is accumulated, requirements change, staffing issues arise, etc. In Waterfall the increment is the entire solution, and over that time period there are much more likely to be changes which affect the time to delivery; in Scrum, because the increment is only 2 - 4 weeks any changes which occur will only affect the next increment, not the current one."

So whilst the grooming, sprint ceremonies and re-prioritisation might seem like an unnecessary overhead from the outside, what they actually do is constantly push the team to develop exactly what the customer wants, in a time-frame agreed with the customer, in a situation where the worst-case scenario is that a single sprint's work might not be exactly what the customer wanted.  Contrast this with Waterfall, where there can easily be a year or more between specification sign-off and delivery of the finished product.

In that time span any small misconceptions about the customer's needs can be the basis for fundamental decisions which can take an application to a place that was never envisioned - or wanted - by the customer.  Or it might be that several misconceptions cause several parts of the application to be functionally irrelevant for the customer. Or the customer may change their mind about what functionality is most important, or needed.  Legislation may be created that affects the required functionality.  No matter the cause, that's expensive rework and customer dissatisfaction all day long.  In sprints, if the customer doesn't like something, that's fine, either remove the functionality if it's not needed or amend it if it's not quite right.  But with only 2 weeks between sprint reviews the customer will never be more than a fortnight away from seeing the latest iteration of the software, and the software will never be more than a fortnight away from being pushed back onto the right track if it's starting to divert away from what is needed.

As the Systems Theorist John Gall has said:

"Complex systems designed from scratch will never work in the real world, since they haven't been subject to environmental selection forces while being designed,"

In other words, building one big honkin' system straight out of the gate has a pretty good chance of failing as soon as it meets the real world.  And before you say, "Well, we've thought of everything", let me pre-empt you:  No, you haven't.  And even in the very unlikely event that you have, by the time you've built whatever it is, "everything" will have changed in ways you can't predict.  That's the biggest problem with an upfront monolithic specification, aka Waterfall.

Sprints still use specifications, it's just that each iteration uses a new specification that's been created in response to the user pressures of an evolving system.  In Agile, these specifications are called user stories, and whilst they have a different format (e.g. "As a....I that....", acceptance criteria, conditions of satisfaction, etc) they fulfil the same role: giving the team the guidance they need to build something that the customer wants.

It should be fairly easy to see that in Waterfall the risk of delivering something that doesn't satisfy the customer's needs (a low quality product) is inherently greater than in Agile due simply to the amount of time in Waterfall between specification and delivery.  Iteration reduces this risk dramatically, thus iterations are better than specification.