Saturday, 28 February 2015

Targeting Documentation Using Personas

Personas are imaginary users that fulfils a specific role in the use of a software product, and they are used by usability and user experience designers to infer what a real person might need.  For example, for a Financial application there may be personas for Accounts Payable clerks, Accounts Receivable clerks, Financial Controllers, Finance Directors, and so on.  The personas will include key information about levels of knowledge (of both the product and the domain) and common tasks that are undertaken by the persona at regular intervals (e.g. entering invoices, scheduling batch payments, producing reports, etc) to allow the designers to produce designs which are more user-centric, rather than being driven by the designer's own understanding of the requirements which might be inaccurate or incomplete.

As a secondary benefit, these personas can be used during grooming and planning to help the scrum team understand why the requirements are what they are, e.g. it's important that certain fields are grouped together because the persona that most often uses this form will always have to fill those fields in and grouping the fields together will make them more efficient.  Using personas to help the scrum members stand in the shoes of a real (statistically-averaged) user means that the scrum members are more likely to immediately "see" why certain design aspects will make things easier for the users, and thus they are more likely to produce an increment that meets those needs.

This brings us to the role of personas in targeting documentation.  Writers are no more immune from the fallacy of "me-centred design" than designers or developers are: making decisions based on what we would want or we think our users would want.  Documentation is often created and maintained by a smaller team, proportionally, than the team that creates and maintains the product; it is (unfortunately) not unusual for there to be software designers and UX/usability specialists working on the artifacts for a sprint, but for there to be no corresponding documentation designers or UX specialists.  Writers themselves are often expected to fulfil these roles, either singularly or as a team.  Just think about that for a moment.  On a large piece of software the help file is likely to be amongst the most accessed pieces of the functionality, and yet a lot of companies do the equivalent of expecting the developers to make all of the software design decisions.  That's both worrying and depressing!

If this situation is familiar to you and you are not a documentation UX specialist, the personas that are used by the software UX/usability experts in your company can help you avoid making the basic mistakes that prevent your readers from easily accessing the information they need from your documentation. If you don't have access to UX/usability personas, or you want to learn more about creating and using them specifically for documentation, I recommend Communicating the User Experience: A Practical Guide for Creating Useful UX Documentation by Richard Caddick.  As well as personas this book covers other useful tools and practises for designing user-centric documentation and will help you when you are maximising documentation ROI.

< /LeonardNimoy >

This is a blog about agile documentation, and as such not the appropriate or preferred forum for sharing my personal feelings about events in my life or events in the world at large.  However, occasionally something happens which intersects the personal and professional, and the passing of Leonard Nimoy is one such event.

Star Trek in general, and Spock in particular, was a significant inspiration to my love of philosophy and science, loves which continue to this day.  I would not have have been half as interested in computer science and technology without my weekly doses of Star Trek, TNG, Voyager and DS9.  Spock's rational viewpoint was a role model for how I tried to view the world and tackle the problems I faced, and I know I'm not alone in that.  My life would probably have taken a different course without that role model, and this blog, modest though it is, wouldn't have happened.

Thank you Leonard.  Live long and prosper, wherever you are.

Normal service will be resumed in the next post.

Scrum Works Better in Startups

Inspired by this article, I've been thinking about how agile works better in startups than in mature companies.  The article in question argues that Kanban is often more appropriate for startups due to the time-to-market imperative and the rapid pace of change, and I'm not disagreeing with that because a) she makes a great point, and b) I'm not going to argue with an agile maven like Abby Fichtner. 

Rather, it occurs to me that the posts I've written previously suggest that scrum is difficult to implement in practise because of the unicorn at the heart of scrum, which tends to lead towards an inevitable Water-Scrum-Fall situation.  This could imply that I think scrum doesn't work.  This is not true.  Scrum is a great methodology and although its application is hindered by the lack of multi-functional teams, that doesn't mean that it can't still be implemented well and have many benefits.

However, my experience comes
mainly from working with mature products and teams, or new products and teams within mature companies.  The larger and more mature a company gets, the more specialists they employ.  No tech startup hires an agile coach as one of their first employees; it's normally 2 or 3 coders with an idea and easy access to deliverable fast food.  Agile coaches come after another developer, an accountant, a saleswoman, a marketing guy, a tester, another couple of developers, and about 50 other people, because that's the point when the company stops being naturally agile and starts needing help to understand and proceduralise the things that make them good at what they do.

In a more mature company that's got hundreds or thousands of staff you'll find increasing levels of specialisation.  In the technical writing team you might have editors, sub-editors, API specialists, tools experts, learning and education experts, and so on, on top of a pool of staff writers who actually document the output from development.  And then there's the non-technical writers who provide marketing copy, sales brochures, company reports, blog posts, social media content, and all other manner of written communications.  In a startup, the founders will do all of that as and when necessary, and as they hire staff the marketing guy will produce the marketing copy, the saleswoman will put together the sales brochure and the developers will provide some release notes or instructions on installing the product.  After that, as the company grows further, someone in development will gradually start to take responsibility for documentation and eventually become the technical writer, or the company will hire someone specifically to write it.  At that point, scrum starts to become more difficult.

In a startup, generalists are highly valued because there are many jobs to do and not many people to do them.  The founders and early employees have to do everything from development to sales to documentation to marketing to testing to paying the wages and they normally lack proscriptive standards for these things.  Therefore, an agile development methodology like scrum or Kanban is ideal: it allows them maximum agility and they are already a fully multi-functional team where the emphasis is on quick, iterative delivery.  As a company grows though, employees become more specialised to increase efficiency and quality.  This is an economy of scale that basically turns any suitably large company into a form of production line.  Processes, procedures and standards provide not only the structure of the company which analysts can constantly study for new ways to increase efficiency (read: profits), but also the conceptual conveyor belt along which the product passes from conception through delivery and implementation to maintenance and support.

But scrum is the antithesis of this production line ethos with its proscriptive standards and linear flow.  Scrum suggests, nay, demands a multi-functional approach to production within the scrum team, and that cannot happen if the scrum team is made up of specialists.

The more specialised the roles in your company are, the less effective scrum will be

This highlights an inherent tension that might help explain why so many companies end up implementing some version of Water-Scrum-Fall and not getting the results they expected.  Specialism - division of labour - is a clearly defined economic tool to help companies scale up their production to make it more efficient and, ultimately, increase profits.  This is one of the reasons why Waterfall is considered to be an efficient model.  Scrum is a methodology that demands generalists.  Therefore, whilst scrum can be more efficient and produce higher-quality deliverables in a small team, the staff needed for it are at odds with the staff needed for the traditional division-of-labour, process-driven methodologies.  So large companies have specialists who work in a (conceptually) linear process, but if the company tries to implement scrum they need to employ generalists that don't fit with the perceived conception of efficiency. 

In a startup though, specialisms are an anathema, and the entire company - which might be as small as 1 person - is composed of generalists who are agile by default.  Agile is not so much the right methodology for a startup to use, it's the only methodology that can be used.  Really, agile development as a methodology is the re-creation and formalisation of a tech startup mentality for larger companies, and larger companies have moved away from the generalised skill set needed to fuel that mentality in favour of a division-of-labour efficiency.  Is agile trying to recapture a mentality that's inevitably lost in the paradigm shift to being a "proper" company?  Is it possible for a large company to be agile in this way?  I'm not sure it is.

I'd be interested in hearing from anyone that seen a startup move from being a small agile operation to a larger company with increasing specialisation.  Did
your company keep itself agile?  How did it do it?  Were generalists still valued as highly?  Thoughts and experiences are welcome in the comments.....

Wednesday, 25 February 2015

Documentation Can't Complete Until Testing Completes

To some writers, the fact that we can't complete the documentation task for a development work item until the testing task has been completed  seems so obvious that it's unworthy of explanation.  However, for those people who aren't immersed in writing documentation it isn't always so apparent.  Here's an explanation for those people.

Scrum is a methodology for cross-functional teams to produce increments of a deliverable that gradually build up into the finished product over a number of sprints.  As part of this methodology, the cross-functional team members work on the sprint backlog tasks in priority order until they are all done, and then move on to the next sprint (I've used the analogy of carnivorous beetles swarming over a carcass elsewhere as a nicely graphic image to emphasise this).  However, due to the lack of multi-functional teams (a.k.a. the unicorn at the heart of scrum) most teams end up working in some form of Water-Scrum-Fall, that is, a relatively linear progression from development to testing to documentation, done in sprints with appropriate ceremonies and artifacts***. 

In this linear situation, the developer will complete the development task, the tester will test the work, and if they find a problem they will refer it back to the developer, who will fix the problem and send it back to testing, and this process will continue iteratively until the work passes and the testing task is complete. (Some referrals might be put into the release backlog if they do not affect the acceptance criteria of the work item, but that doesn't affect the documentation in this situation). 

The reason that the documentation task cannot be completed before the testing task is completed is that it is ONLY when the testing task has been completed that no more changes will be made to that piece of work.

At this point the standard argument against you is: "Well, the developer will have followed the acceptance requirements, and wire frame, and conditions of satisfaction and [any other artifacts provided] and so the writer can complete the documentation once the developer has completed their task." 

This is true in theory, just not in practise.  Let me explain by way of an example.  The designer mandates that 4 fields need to be added in a row because users will always fill in field A, then field B, then field C, then field D.  A wire frame is produced that shows the fields as:

FIELD A        FIELD B        FIELD C        FIELD D 

At task implementation time, the developer realises that at a lower resolution these fields will not all be visible without scrolling, and, as scrolling is frowned upon, they actually put the fields as:


The tab flow goes ABCD, the natural flow is correct (for the European market where this software is sold) and it ensures that the developer has met both the user requirements and the development rules.  However, if the writer takes a screen shot at this point in time, checks that the fields do what they are expected to do as per the artifacts and sets their Documentation task to done, what happens if the tester realises that your company stopped supporting that low resolution last year, so actually it's ok for the developer to revert back to a row in line with the designer's original plan?  The developer makes the changes, the tester passes the testing task....and the documentation is wrong, but the writer doesn't know it. 

The more developers and/or scrum teams you have, the more likely it is that changes that materially affect the documentation will be made to a piece of work after the Development task has been completed for the first time. To be clear, it is NOT ok to assume or even mandate that either the developer or tester tells the writer about this, in the hope that that will solve the problem and make your team more "swarm of beetles" and less "line of ants".  Developers and testers don't know what will materially affect the documentation, so they'll have to tell the writer about lots of things that don't matter to the writer, which wastes time for all parties, and, more importantly, this puts the onus onto people to to catch and solve problems which should be prevented from ever happening by the system itself.  And that system should include writers not being able to complete their tasks before testing have completed theirs.

*** - Now, if you DO work in a multi-functional team, you can develop a piece of work and then complete the documentation task before testing having finished. But be honest, you don't work in a
multi-functional team.  Don't feel bad, neither does anyone else.

Factors that Determine if Documentation Should be in the Definition of Done

When we talked about whether documentation should be in the Definition of Done, there were 6 factors that spoke directly to the answer for teams that weren't multi-functional (which is the vast majority of teams that practise scrum). That article focused on what combination of those factors would allow you to add documentation to the Definition of Done (DoD); this article looks at those factors in a bit more depth and explains why they are so important.

Let's remind ourselves of the factors which need to be looked at:

1 - Can anyone other than the designated writer(s) complete the documentation tasks?
2 - What proportion of documentation can be done quickly by the writer, e.g documenting a bug (as opposed to complicated enhancements, for example)?
3 - Is the quality of the input to the documentation tasks reliably high and consistent?
4 - Is the number of deliverables less than 2 (or if 2 or more, can they be single-sourced)?
5 - Are there proscriptive standards and templates for the documentation that is being produced?
6 - Are complicated enhancements front-loaded by the team to allow the writer to work on them as soon as possible?

Don't forget that for single- or semi-functional teams the question of whether documentation is part of the sprint deliverables is not a factor; this is about whether you can deliver documentation at the end of each sprint, and if you can't then you (or rather your company) shouldn't commit to providing documentation as a sprint deliverable.

1 - Can anyone other than the designated writer(s) complete the documentation tasks?

The answer to this question determines - for the specific purpose of working out if documentation should be in the
DoD - whether the team is single-functional or semi-functional.  If there are, for example, 6 developers, 2 testers and 1 writer, and only the writer can produce documentation, the team is single-functional.  If one or more developers or testers can produce documentation to the same standard as the designated writer then the team is semi-functional. Note that adding another specialist writer to the team won't make the team semi-functional, because documentation can't complete until testing completes, so although this will provide more manpower to get the documentation done, it won't change the inherent problem that non-multi-functional teams face - the need to work in a linear process within each sprint.  If one or more developers or testers can produce documentation, they are able to take on the documentation tasks as well as tasks in their own speciality earlier in the sprint.  This is what makes the team internally agile. The more developers and testers that can produce documentation to the correct standard, the closer the team is to being multi-functional (for the purposes of providing documentation).

2 - What proportion of documentation can be done quickly by the writer, e.g documenting a bug (as opposed to complicated enhancements, for example)?

This is the key metric that determines how much work the writer can do before testing have completed their tests, and how much will have to be done after testing have completed.  The development team should provide information for each bug in the form of "This is what was happening", "This is why it was a problem", "This is what happens now". Once the developer has done this work and sent it for testing, it is rare for this information to change (i.e. testing might find an issue with the developer’s fix, but the "This is what happens now" information rarely changes), so the writer can document a bug before it's been tested and move on with a low chance of rework later on. However, if a writer is working on enhancements, especially UI changes, then there is a much higher chance that any issues found in testing will affect the documentation, therefore the chance of rework is much higher, therefore the chance of the writer completing their tasks before the end of the sprint is much lower.  For a single-functional team this proportion needs to be very high to allow documentation to be part of the
DoD, otherwise documentation will be a likely point of failure when determining if a sprint completed successfully.

One thing to bear in mind:  Because of the inherent fluidity of new work and the fact that a large piece of development might be done over more than one sprint, and it is easier to document it as a whole once it’s completed (so you can fully understand it, take screen shots, work on it as one piece of work rather than task-switching all the time, etc), this factor is something to think long and hard about.  If you have a lot of enhancements to document, especially ones that tie together to make a few large enhancements, you really should opt for documenting outside of the sprint cycle, no matter what your answers are to the other factors. 

If documentation IS part of your
DoD, you’ll be at the end of the accordion, and if development or testing don’t hit their deadlines within the sprint then the writer will get squeezed and squeezed until either quality or completion  - or both - become an elusive dream. And if development and testing DO hit their deadlines then they’re twiddling their thumbs for several days while you write the documentation. It’s an intractable problem that is best solved by removing documentation from the DoD and working in sprint bundles to make development and testing more efficient, and to minimize rework for the writer, but if documentation is part of the sprint deliverable then it's something that has to be managed as much as possible.

3 - Is the quality of the input to the documentation tasks reliably high and consistent?

By "high" we mean "All the information that the writer needs to document as much of the work as is possible before development and testing have completed, for example, wire frames showing the expected field positions, a complete explanation of the functionality and what it will be used for, a complete listing of all parameters that affect the new work or are affected by it, and any other information that the writer needs to provide to the customers."

By "consistent" we mean "Every work item that is in the sprint has this information in it, every time, without fail."

If the quality of the input is not both of these things then the writer will have to spend a lot more time finding the information out, which can only be done after testing have completed, and again, the writer is much more likely to fail to complete their tasks by the end of the sprint.  It's also much more inefficient in general terms; the information is all there, but the writer will have to get it manually from developers, designers, the Product Owner, etc, so this is good practise anyway.

4 - Is the number of deliverable less than 2 (or if 2 or more, can they be single-sourced)?

Deliverables might mean traditional documents such as help files and release notes, but equally it could be editing help messages and other on-screen text, writing API documentation, and so on.  The point of concern here is two-fold: Firstly, will the writer have to enter the same information in more than one deliverable in any given work item (unless it can be single-sourced, in which case the practical answer is "No") and secondly, Will the writer have to complete (format and prepare for release) more than 1 deliverable at the end of the sprint?  If the answer to either of these is Yes, then the writer has to either add the same information into multiple deliverables, or different information into different deliverables.  Either way, there is an overhead in these scenarios that adds more required time onto the documentation.

5 - Are there proscriptive standards and templates for the documentation that is being produced?

What is one of the most difficult and time-consuming things for most people to do?  Making a decision.  What is a guaranteed way to ensure inconsistent and therefore poor quality documentation? Letting people make individual decisions about formatting, language, and so on.  Having incomplete - or missing - proscriptive standards and templates therefore both costs more time AND lowers quality.  Even if a writer is good at making consistent decisions, there is still the possibility - the probability, in fact - of errors or inconsistencies being made from time to time, and as the decisions will have to be made every time they work on a documentation task there is an in-built bottleneck and inefficiency there. Anything which prevents the writer from writing should be seen as an impediment by the Scrum Master and as waste (in LEAN terminology) by the company.  So perhaps a more apt question would be: Why WOULDN'T you have proscriptive standards and templates?

6 - Are complicated enhancements front-loaded by the team to allow the writer to work on them as soon as possible?

Because documentation can't complete until testing completes and because, as noted in #2 above, there is a much higher chance of rework with an enhancement, any enhancements that require significant documentation should be front-loaded by the developers if possible.  This is not always possible, because development need to start getting work to testing, and complicated enhancements often take longer to do than bugs, so bugs get done first to ensure that testing have work to do as soon as possible in the sprint.  Bugs can also be "quick wins" for development.  There is an inherent tension here, because bugs can be normally be documented quickly (especially if the quality of the input is high and consistent and you have proscriptive standards) and also can be estimated more easily.  If you have 10 bugs left, for example, you will know that all things being equal it will take you 2.5 hours to document them (or whatever your rule to thumb tells you - mine says 15 minutes per bug as an average).  But if you have 10 enhancements, all with a different number of story points then you can still estimate them, but it's something you have to work out as opposed to being something you know.  I know that 15 minutes is enough time on average to document a single bug because over many years of experience 15 minutes has proven to be a reliable estimate for me in many different situations.  But I can only estimate how long it will take me to document an enhancement.  That estimation may be more accurate as I learn the product and participate in the grooming to properly understand the work that will be done, but I will never have the level of confidence in it that I do with my bug estimates.  The larger and/or more complex the enhancement, the less accurate your estimate will be.  And as development for a single-functional team is linear (i.e. each sprint follows a Waterfall pattern) and documentation comes at the end, it's much easier to tell if you will complete on time if you only have bugs left to document.  The more enhancements that are left until the end of the sprint to document, the greater the chance of you failing to complete your documentation tasks and being the point of failure.  This is doubly annoying because it looks like the writer has failed, but actually it's the system that has allowed the whole team to fail by not understanding what each function within the team needs, and when they need it.

Once you've understood these factors you can use them to help make your decision on whether documentation should be in the
DoD.  Good luck!

Should Documentation be in the Definition of Done?

Welcome to the thorniest, most contentious question is the world of agile documentation.

If you've discussed this with many people, you'll have probably heard views that range from "Who cares?" (thank you cynical developer-types) to "It ABSOLUTELY MUST/MUST NOT be in the Definition of Done!! Anything else is madness!!!" (thank you dogmatic autodidacts) and everything in between (thank you normal people). 

So let's get one things clear from the outset: There is no objectively-right-in-every-situation answer to this, only an answer which is right for your situation.

This is why you will hear such a broad and contradictory range of opinions on the subject.  That's not to say that some of those opinions aren't correct for your situation, but what works in one scrum team situation might not work in another, even within the same department, let alone the same division or the same company.

There's a good reason that there is no one-size-fits-all answer, and it's entirely to do with the unicorn at the heart of scrum.  Truly multi-functional teams are like unicorns - very desirable and yet seemingly a myth - and as such it isn't possible for most team members to pick up any given task.  Teams range from the entirely "single-functional", where a developer can only develop, a tester can only test and a writer can only write, through "semi-functional", where, for example, some developers and writers can also test, some testers and developers can also write and some testers and writers can also develop, up to truly multi-functional, where any member of the team can pick up any task. 

For a truly multi-functional team, the decision on whether documentation should be in the Definition of Done (DoD) is based simply on whether documentation is part of the deliverable due at the end of each sprint.  If it is, then it's just another task and documentation is in the
DoD, if it isn't part of the deliverable then it isn't in the DoD***. 

Outside of truly multi-functional teams though, there are several factors that need to be taken into account:

1 - Can anyone other than the designated writer(s) complete the documentation tasks?
2 - What proportion of documentation can be done quickly by the writer, e.g documenting a bug (as opposed to complicated enhancements, for example)?
3 - Is the quality of the input to the documentation tasks reliably high and consistent?
4 - Is the number of deliverables less than 2 (or if 2 or more, can they be single-sourced)?
5 - Are there proscriptive standards and templates for the documentation that is being produced?
6 - Are complicated enhancements front-loaded by the team to allow the writer to work on them as soon as possible?

(See here for a more in-depth discussion of why these things are required if you're not sure.)

For the purposes of this discussion:

    If #1 is a NO, then the team is a "single-functional" team.
    If #1 is a YES, then the team is a "semi-functional" team.

For a single-functional team:

    If #2 < 75% then documentation should not be in the DoD.
    If #2 = 75% - 85% then documentation should only be in the
DoD if #3 - #6 all = YES.
    If #2 >= 86%  then documentation should only be in the
DoD if at least 3 out of #3 - #6 = YES.

For a semi-functional team:

    If #2 =< 50% then documentation should not be in the
    If #2 =< 75% then documentation should only be in the
DoD if #3 - #6 all = YES.
    If #2 >= 76%  then documentation should only be in the
DoD if at least 3 out of #3 - #6 = Yes

If you can't answer YES to at least 3 out of #3 - #6, then for a single- or semi-functional team documentation should not be in the
DoD, no matter what the answer to #2 is.  This is because those 4 questions determine whether your processes and standards will provide enough support to the writer to allow them to get the documentation done without a significant lag at the end of sprint.  If less than 3 are YES, then there are too many possible problems for the writer(s) to overcome.  (There's no absolute guarantee that a writer will face problems, but in the real-world situations to which this applies you shouldn't assume everything will be smooth sailing all the way - that's just unrealistic.)

 A note about the numbers: No sprint should be more than 80% full, to allow for contingency.  If #2 <= 85% for single-functional teams or <= 75% for semi-functional teams, then you are getting too close to using up all of your writing contingency by asking the writers to work on complicated work items that are likely to take longer, and have a higher expectation of rework if the writers works on them before testing has completed.  These figures are therefore partly a matter of logic and partly a matter of realism.  In the spirit of scrum, you should adjust them based on your own experience within a mature sprint team with a settled velocity.

You may have noticed that for both single- and semi-functional teams I have nowhere mentioned whether documentation is part of the sprint deliverable.  If you did notice, well done and gold star to you!  This is a key point: If your team does not have the skills, processes and standards in place to allow documentation to be completed in-sprint, then it should not be part of the sprint deliverable.  This is an issue that should be decided before agreeing a contract with a client. 

In summary, development is a process and some things simply have to be done in order. Agile helps with that, but, unless you have a truly multi-functional team, it doesn’t change it. Therefore, it's fair to say that to the question of whether documentation should be in the
DoD, there isn’t a right answer, just answers that have varying degrees of usefulness for your situation. Hopefully the factors listed above will help you make that decision.

*** - There are other good reasons why documentation might or might not be in the
DoD in this circumstance - every team, product and company is different, after all - but in principle this is the sole reason, and there's no need to muddy the waters by talking about the possible differences in business process or contractual obligations.

Monday, 23 February 2015

The Power of Scrum Doesn't Lie in the Scrum Team

Part of the reason that agile development is so attractive to decision makers is that it gives the impression that more work can be done to a higher quality, with greater customer satisfaction, without increasing resources. And in one sense that's correct; after all, why would people use scrum if it didn't have at least some of these benefits?

But the rationale behind this impression sometimes seems to be that implementing scrum within the development team will produce the results and that everyone else can continue working in the same way as before - in other words, the power of Scrum lies in making development work as a scrum team, because that is where the efficiencies are to be made.

This is not correct.

A scrum team can indeed be a powerful force for productivity.  If a multi-functional team of people attack the user requirements in an organised and efficient way, then like a team of carnivorous beetles they could swarm over a backlog, devouring whatever work was in front of them and passing out sections of the deliverables until the bare bones of the sprint are picked clean and they can move onto the next one. In this scenario there certainly are efficiencies to be made and more will be produced without having to increase resources.

(On a side note, multi-functional teams don't exist in reality, but that's for another post.)

And what will the scrum team produce?  They'll produce what the Product Owner has said is the highest priority, a collection of tasks that together make up a deliverable iteration.  But so what?  You can have the most efficient, skilled, experienced demolition team in the world, but if they knock down the wrong building, who cares how efficiently or cleanly they did it? The productivity of the scrum team is absolutely irrelevant to the measurement of the most important aspect of scrum: Does the customer get a more appropriate deliverable than if you'd used waterfall? 

(I totally get that the increased productivity of a scrum team, as measured in its velocity over time, is an important factor in choosing to "go scrum" because this velocity is seen as improving time to market, which is a key goal for many companies.  My point is that it doesn't matter how quickly Betamax got to market, what mattered was that people didn't want it.)

By contrast with scrum, Waterfall's great strength is its attention to detail, the fact that everything is locked down at the start.  Yes, this means that by definition it is not agile and therefore if the specification changes this can cause serious budget and time overruns, and in theory developing in sprints ameliorates this problem almost entirely.  But agility in the scrum team on its own won't produce a better deliverable for the customer.  Remember that the scrum team is the means of production, not the decision maker about what actually gets produced.

The power of scrum lies in the ability of the Product Owner to change the direction of the scrum team's production by changing the priority of work items on the backlog.

So if you're having problems giving customer what they want with Waterfall because the specifications keep changing, that isn't going to stop just because the team produces in iterations.  It will stop when the analysts, project managers, designers and
Product Owner work together in an effective and efficient manner to identify these specification changes as they happen and re-prioritise the backlog in real time accordingly.  If the backlog isn't looked after by the Product Owner then the scrum team can produce wonderful code that does entirely the wrong thing, which is hardly an efficiency improvement.  To paraphrase a slightly ruder saying, "Rubbish In, Rubbish Out".  A scrum team that isn't directed to produce the right things will produce the wrong things, and the deliverable to the customer will not be any more appropriate than it was with Waterfall.  

This leaves us with a significant and often overlooked point: it's the quality of the client-company communication and how that is handled by the Product Owner which drives the power to the scrum team.  And if THAT'S handled well, then quality gets better, deliverables get more appropriate and the scrum team starts to make worthwhile efficiency savings.

What is Water-Scrum-Fall (and is it inevitable)?

Water-Scrum-Fall is a compound term formed from Waterfall and Scrum. It is used to describe a situation where a company runs their development teams in an agile (scrum) fashion with scrum ceremonies, artifacts and roles, but follows a sequential waterfall or quasi-waterfall methodology both outside of the scrum team and within the scrum team itself.  The outcome for the scrum team is that each sprint becomes a mini-waterfall with lots of documentation input and a set output at the end.  In essence, it's waterfall software delivery using an incremental scrum-style release cycle rather than either a pure "big project" waterfall approach or a pure "agile prioritisation" scrum approach.

This of course is a generalisation; there is no set definition of Water-Scrum-Fall and each company that uses some flavour of it will implement it to fit their own circumstances and foibles.  But the definition above covers the basic truth of the matter.  There is a fuller explanation by Dave West, the man who coined the term, here and a short but interesting piece on the causes and the fact that Water-Scrum-Fall is a reality we have to face by the DevOps evangelist Sanjeev Sharma here.

What's interesting about these articles is that both authors focus on the business reasons for Water-Scrum-Fall, and both argue that a key factor is that the business functions outside of the scrum team are not agile, and therefore continue with their waterfall methodologies.  I don't doubt their analysis, and for the business processes and structures outside of development I'm sure they're correct, but their analysis doesn't explain why development teams continue to produce deliverables in a linear Dev-Test-Doc process within each sprint. They are missing a fundamental point about scrum that means a sequential Water-Scrum-Fall process within the scrum teams themselves is so common:

To be agile you must have a multi-functional scrum team, but multi-functional scrum teams don't exist.

Scrum is a great idea in theory, but so is the star ship Enterprise.  We can't build a functioning star ship without a warp drive, or it's equivalent, and we can't build a functioning scrum team without a multi-functional team, and those basically don't exist.  I've described this problem elsewhere as the unicorn at the heart of scrum, and in essence it boils down to this: It takes time and effort to become competent, and stay competent, at any one of the individual roles in a scrum team; it is almost impossible to become and stay competent at Development AND Testing AND Documentation AND anything else you have in your scrum team.  This is not a question of will, or training budgets, or management buy-in.  It's a question of logistics.  There's a fuller analysis of why there aren't more multi-functional scrum teams here if you want to explore this topic a bit more.

Whilst Waterfall has many flaws, it works because it takes into account the essentially linear nature of development work.  Testing can't complete until Development have completed and Documentation can't complete until Testing have completed.  Different functions can overlap, but testing cannot complete their testing of the software until development have finished development - it's a matter of physics, not choice. Development HAS to finish before testing, testing HAS to finish before documentation, and documentation HAS to finish last. This is not going to change simply by moving to a different methodology because the factor that's missing isn't methodological, it's functional. Or, more precisely, it's multi-functional. 

Water-Scrum-Fall is not caused by a failure to implement scrum correctly or completely.  It is caused by scrum being a theory that requires something that doesn't exist in the real world - a multi-functional team.  

The lack of multi-functional teams means that it is inevitable that scrum teams will move in a linear fashion, because they simply cannot operate in any other way.  If each individual member of the team is single-functional, then the testers must wait for development to finish and writers must wait for testers to finish.  Yes, of course testers and writers can use some time at the beginning of the sprint to write test plans, produce skeleton release notes, and any other tasks they can start or complete before seeing the software, but the meat and drink of their work must happen after development.  A tester cannot fully test and a writer cannot fully document without having the software.  At the end of the process, when testers are testing the last few development items, the developers have nothing to do.  When the writers are documenting the last few work items the developers and testers have nothing to do.

Let us be clear: This is a fundamental problem with the scrum theory.  There are ways of ameliorating this to a certain extent, but you cannot change the sequential nature of Dev-Test-Doc just by changing the methodology to scrum.  So if you don't have multi-functional team - and you almost certainly don't - then a linear in-sprint system is the only realistic way to operate.  This means that Water-Scrum-Fall is an inevitability for most of us, both in and out of the scrum team, and we need to find ways of making that work.


Why Aren't There More Multi-Functional Teams?

Multi-functional teams are The Unicorn at the Heart of Scrum - desirable, unattainable and quite possibly mythical. Why have so few people ever seen - let alone been in - a multi-functional team? Or, to put it another way, what is it about a multi-functional team that makes it so hard to create?  There are several parts to the answer:

1.  It takes a long time to master the skills to perform each task. 

Let's assume that the backlog contains development, testing and documentation tasks.  Firstly, being a competent writer can take years of training and experience, as well as aptitude.  To be a competent developer or tester you need the same.  Let's say you could master the art of creating software deliverables to a decent standard in 3 years.  Then you move on to Testing, and take another 3 years, and finally Documentation over another 3 years.  This makes the assumption that you have an aptitude for each of these areas and can pick up the skills quickly and easily, but it still takes 9 years.  That's a long time to become competent.

To make matters worse, within each of these areas - development, testing, documentation - there are specialities that require additional learning and experience.  For example, security, automated testing and documenting APIs are all specialisms within development, testing and documentation respectively.  These take additional time to learn and master.    And on top of this is the fact that each of these areas is constantly evolving with new technologies and methodologies.  You learnt about information mapping when you started as a writer, but now you need to know about DITA.  Best practise for testing plans has been significantly updated.  HTML5 is a different development paradigm to AJAX.  And so on, and so on, ad infinitum.

So it takes a long time to become basically competent in each area, and you have to constantly keep up-to-date with new developments, tools and best practises in each of these areas.  This is a difficult thing to do in 1 of these 3 areas, let alone all 3 areas, especially if you are working 5 days a week on sprint tasks.

2. How many people can really do development, testing and documentation to a high enough level?

Being a member of a
multi-functional team requires each member of that team to be able to competently perform at least 2, and preferably 3, of the roles required (assuming a simple model of development, testing and documentation tasks within a sprint).  In theory this should be possible because some of the aptitudes overlap: creativity is important in both development and documentation, an eye for detail is required for both testing and documentation, a technical mindset is needed to both develop and test a piece of software, and so on. But, as someone who has worked as both a developer and a writer, I can say without doubt that development and writing are absolutely poles apart in terms of aptitude and key skills. Likewise, having asked a friend who started off as a developer and became a tester, those two trades require a very different mindset and set of talents.  Apart from technical skills, the interpersonal and soft skills for a writer need to be better than those of a developer, because a writer needs to spend time talking to developers, testers, analysts, designers, SMEs, and so on, whereas it's fair to say that the cliché of a developer who sits in a corner and never talks to anyone - and doesn't want to - is a cliché because everyone who reads this blog will have met very good developers who do exactly that.  A writer couldn't get away with that and still do their job well, but a developer can.  On the other hand, a developer who isn't comfortable with applied maths will be missing a key component of the skill set, but most writers can get away without being particularly mathematical.  There are similar difference between developers and testers, and testers and writers.

So finding people who can fulfil all of these roles at anything higher than a basic level can be difficult.

3. Do these people WANT to do all of these things?

Even if you can find several people who have all the aptitudes to be
multi-functional, do they want to be?  They might not be interested in doing all of it.  Here's a cast-iron guarantee: Poll 100 developers to ask if they want to do documentation as part of their job, and 80%+ will say "No", even if they can do it and do it well.  Besides, if you spend the best part of a decade learning all of the required skills, all you have to show from it at the end is that you are a jack-of-all-trades but a master of none. Does that sound like it will provide the career progression people want? No, and especially because Scrum implementations vary from company to company, and often from team to team, so "multi-functional" can mean different things for different teams.  If you move to a team that does analysis, design and development then 2 of the 3 skills you have are not relevant, meaning you are no longer multi-functional. 

So even if you can find people who can fulfil all of these roles at a high enough level, they may not want to perform them.

In conclusion then, if your company is likely to use the same technology for a decade, AND you're lucky enough to have an aptitude for development AND testing AND documentation, AND you want to be a generalist rather than a specialist, AND you don't have aspirations of career progression outside of that team/department AND you work for a company that will allow (and pay) for the training time to get you up to speed on each of these 3 areas then congratulations, you might become a member of a truly
multi-functional team.  Otherwise, you won't.  And that's why there aren't many multi-functional teams out there, and why you aren't likely to see or join one.

The Unicorn at the Heart of Scrum

The Scrum development methodology was designed to provide a flexible framework, to be used by multi-functional teams to work together to complete all of the tasks that are required to produce a functionally-ready product increment.  Each increment builds on the previous increment to eventually produce a complete solution that meets the customer requirements.  So far, so simple, and, let's be honest, quite enticing.  It's certainly a more dynamic and empowering vision than the "small cog in a big project" feeling that Waterfall provides.

But once you start applying this enticing theory in practise, most companies run up against a fundamental problem.  Scrum is deliberately designed to allow any (or at least most) team members to pick up any task that is Ready.  In this way the "
multi-functional team passes the "ball" [sprint backlog tasks] back and forth on the way to the "goal line" [completion of the Sprint]".

The fundamental problem with Scrum is that multi-functional teams don't exist in practise.

Peter Measey, CEO of the agile training and coaching company Radtac, and a man with over 20 years experience of working with agile teams (agile methodologies have been around a lot longer than most people realise) once told me during a training session that he has only ever seen 1 truly
multi-functional team. 1. Uno. Less than 2.  And given that no-one else I've ever met has ever seen one at all, it does seem like a truly multi-functional team is something of a unicorn - desirable, unattainable and quite possibly mythical.

I've addressed the question of why there aren't more
multi-functional teams in a different post, but in brief, it's mainly because the specialisations that you need to master in each area - e.g. development, testing, documentation - are too varied, too large and too constantly evolving and updating for it to be realistic for most people to master all of them.  So if it's unrealistic to expect a multi-functional team, how does this impact your Scrum implementation?

Well, primarily it means that Water-Scrum-Fall, where jobs are done largely in a linear fashion within a sprint, is a very common implementation of Scrum.  And of course, for us writers, it means that we are at the end of the queue for work.  Testing can't complete until Development completes, and Documentation can't complete until Testing completes (which is something else I'll expand on in a later post).  So the writers sit at the end of the sprint accordion, and if development and testing don't finish in plenty of time then the writers' time to complete their tasks will get squeezed as the accordion closes and the sprint will fail.  Therefore this unicorn feeds directly in to the answers to possibly the most difficult Scrum question for agile writers: Should Documentation be in the Definition of Done?

Because of this unicorn at the heart of the Scrum theory, and to prevent the Water-Scrum-Fall problem that arises from it, it is important to recognise an important and often overlooked truth about Scrum: The power of Scrum doesn't lie in the Scrum team. Instead, the power of Scrum, and its ability to smooth the linear Waterfall-style progression within each sprint, is dependent on the following:

1. The quality and timeliness of the user stories and associated artifacts such as wire frames that are produced by analysts and designers who are working outside of the Scrum team;

2. The provision of proscriptive standards that allow testers and writers to produce skeletons and outlines for consistent testing plans and documentation deliverables, based on the artifacts and information from the ceremonies, before seeing the software.

3. The prioritisation by the Product Owner of UI elements that can be front-loaded in the sprint by development so that testers and writers can see the software as soon as possible.

4. And, most importantly of all, high quality client-company communication and constant feedback by the Product Owner to the scrum team in the form of real-time backlog prioritisation.

If these 4 elements are present then Water-Scrum-Fall can at least be made more Scrum than Waterfall, even if there will always be an element of linear progression in the absence of Scrum's unicorn.

Sunday, 22 February 2015

Does a Writer Need to be Able to Read and Write Code?

In a word: Maybe.  In several words, maybe, but it depends on what you mean by "code".  Technically speaking, "code" means a compiled language like C#, Java or C++, whereas "markup" means tagging languages like HTML, SGML or XML and "scripting" means code that can be interpreted, for example by a browser, like JavaScript, Perl or Python.  There is also SQL, which is a language specifically used for querying databases.  Although this comes under the "code" branch of the family tree of languages, I'll be treating it as a separate skill here, for reasons I'll explain later on.  Some of these things a writer must be able to read and write, some they probably should be able to, and some they'll never have to unless they want to.

You Must Be Able to Read and Write Markup

A common requirement for Technical Writers is the ability to at least read, and normally edit, markup.  This is because:

a) The Darwin Information Typing Architecture (DITA) standard is used by many companies, and that is an XML data model, and;

b) Most modern content authoring tools such as MadCap Flare, or document processing tools such as Adobe Framemaker, use markup to structure the content, whilst showing the content to the writer in a WYSIWYG front end. 

Being able to interpret and edit markup is therefore an necessary skill in many, if not most, technical writing jobs.  Cascading style sheets (CSS) are used to define the format of the content; being able to manipulate HTML without being able to manipulate CSS is like being able to draw a picture but not be able to colour it in, so if you can read and edit markup, you'll probably need to read and edit CSS as well.  So in the case of markup (and CSS), the answer to this question is yes, as a Technical Writer you should be able to at least read and interpret HTML, XML and CSS, and preferably edit it as well.

(If this is something you can't do, there are free on-line lessons for HTML, XML and CSS)

You Probably Should be Able to Read and Write SQL

SQL (Structured Query Language, often pronounced "sequel") is the standard language for interrogating relational databases.  There are as many flavours of SQL as there are relational database products but the 3 most widely known are probably Microsoft SQL Server, Oracle and MySQL.  Each of these has a slightly different syntax, but nothing which deviates hugely away from the ANSI standard. Being able to read and write at least basic SQL (in any flavour) will allow you to interrogate a database, amend data and produce SQL code samples if you need to.  The requirement to do these things varies enormously between jobs, but it's a common enough requirement that it's worth learning.  Luckily, the basics of SQL such as retrieving all data from a table where column x = value y, inserting or deleting a row in a table, and finding out the lengths and data types of all the columns in a table are very easy to learn and there are good tutorials available.

There are a lot of different coding languages and frameworks, and which one(s) you'll come across depends very much on the industries and companies you work for, but SQL is used anywhere that relational databases are used.  That makes this skill portable regardless of what coding languages you might need to get to grips with (see below), hence SQL gets its own section here.

You Don't Have to Read and Write Code and Scripts. Unless You Want To.

For code and scripting languages, the situation is different.  If you are creating release documentation for an application, it is a lot less likely that you will need to be able to read, let alone write, code or scripts. Code isn't information rich and it's not the best way to understand the functionality that's been developed (that would involve demonstrations and asking questions). So why would reading code be helpful?  The answer lies in the necessity for developer-to-developer documentation, and this means APIs.

An API (Application Programming Interface) is a set of functions that can be used by programmers to access the operations that are held internally in an application***.  To document them properly, you have to be able to identify the inputs, outputs and upshots, that is, what needs to be passed in, what will be passed back out, and anything else that the API function will do if it's accessed.  And if you want to find that information out, you're going to need to read code and/or scripting languages.  API documentation is a speciality within technical writing and many writers, even if they document software for a living, won't ever have to do it. But it is a very valuable string to your bow because (as of 2015) there is a strong demand for API writers, and with the trend towards the Internet of Everything, the need for API documentation that allows developers to get products talking to each other in a meaningful way is only going to get higher. 

The drawback (for some people) of documenting APIs is the emphasis on the "Technical" part of the "Technical Writer" job title.  A lot of Technical Writers are people who love using the written word to de-mystify the mysterious and elucidate the complex, so they see themselves as writers who specialise in technology, rather than technologists who write.  If you're documenting APIs though, you need to be a technologist AND a writer, because aside from documenting the inputs, outputs and upshots, you'll have to write code samples using the existing code line as a guide.  Someone else's code can be quite opaque to the initiated, never mind the uninitiated, and if there's a lack of rigorous code comments then hoo boy, look out, that's a world of pain right there.  API writers love to work through this pain and produce coherent documentation from it, so if that sounds like you then congratulations, you're in demand! As long as you can read and write the code.

*** - For those of you that have API experience, you're right, it's not quite that simple!  But this will do as a simple explanation for those people who've never met APIs before.

Documentation IS Important in Scrum

As part of the Manifesto of Agile Software Development, to which scrum is an adherent, there is a list of things which are valued.  These valued things include "Working products over comprehensive documentation"  The manifesto goes on to say, "That is, while there is value in the items on the right [processes and tools] we value the items on the left [individuals and interactions] more".

I've heard this principle expressed in the following ways with regards to documentation:

1 - This means there are no formal specifications.
2 - Documentation deliverables aren't important.
3 - There's no need for code comments or testing plans.
4 - There's no need to record anything because scrum says development of the product is the most important thing.

At the risk of going all "Psychology 101" on you, there is a definite temptation to believe that some people are saying some of these things more as an expression of their hopes, because they don't like proscriptive processes and/or writing documentation, rather than a genuine belief that this is what is intended.  There are also a few "end of the bell curve" types who do genuinely believe that code comments and testing plans have no merit, and that recording anything is somehow tantamount to KGB-style surveillance that will be used against them maliciously if the company decides to restructure.  These people are are not particularly valued by their team mates.

However, for the majority of people, these are expressions of concern about scrum's suitability to cover certain statutory or legal requirements, and the provision of a process framework for things that need to be agreed in writing or recorded for good reason.

Scrum is designed as a series of guidelines rather than a specific, one-size-fits-all framework that covers every eventuality.  Each company is different, with different working patterns, different statutory and legal requirements, and different processes and procedures.  Scrum is deliberately flexible to allow it to be implemented in different ways to suit the myriad circumstances that can be found in the real world. However, there are agile principles for a reason, and they are there to guide the practical implementation rather than to proscribe a set way of working that everyone must adhere to***.

So let's correct these misunderstandings:

1 - This means there are no formal specifications.

    Wrong.  There don't HAVE to be formal specifications if you don't want them, but there is nothing in scrum that stops you having them.  Many companies continue to feed formal specifications into a scrum process very successfully.  The difference in scrum is that these specifications are broken down until they reach a granularity whereby each task that is required to complete the specification can be completed in 1 day or less.  These tasks then form (part or all of) the backlog, and are prioritised by the Product Owner. 

Sidebar: Specifications are often replaced with User Stories in scrum.  These user stories can either be created in concert with the customer or internally as part of the process that feeds work into the backlog.  User stories are broken down into smaller and smaller chunks until they reach the size of work items, which will consist of a day's task for each area of competence.  For example, a work item will contain a development task, a testing task and a documentation task, each of which will take a day or less to complete.  The Acceptance Criteria and Conditions of Satisfaction that are part of each work item are agreed during grooming and apply to all the tasks within the work item.

2 - Documentation deliverables aren't important.

    Wrong.  Documentation deliverables form part of the product, as do any other deliverables.  By their very nature as deliverables, they are required and (normally) part of the contract for what will be provided from the supplier to the customer. Having said that, if the customer has a direct role in the specification of the software, for example where a piece of bespoke work is being delivered, documentation can potentially be delivered at release time, rather than sprint by sprint.  This is because the customer will be a stakeholder, and as such the increment that is being delivered should be demonstrated to them (and any other stakeholders) in the review ceremony at the end of each sprint.  Due to the short nature of sprints and the granularity of the tasks the customer should already know what they are getting in the sprint, and so the documentation deliverable will be less time-critical.  However,  it is only the urgency that decreases in this situation, not the importance.

3 - There's no need for code comments or testing plans.

    Wrong.  Code comments are an essential part of any sensible development process.  Staff come and go, either to other companies or other projects, but the code line remains for as long as the product does.  If there is code, someone will have to maintain it.  Even if the developer who wrote the code is the one who maintains it, if they haven't looked at that section of code for any great length of time then without code comments they will still have to work out what the code does, why it does it, what work item or bug number it relates to, and any other relevant information that they need.  These comments can also be directly linked to unit testing, in that they provide the information that allows a unit test to be written or amended.  Testing plans allow the same traceability for what, why, when and how, with the same potential link to automated testing runs, and, depending on your regulatory framework, can be an important part of your company's ISO/BS/CMMI compliance.  There is nothing in scrum that says code comments and testing plans aren't important or needed.

4 - There's no need to record anything because scrum says development of the product is the most important thing.

    Wrong.  Firstly, scrum methodology does not override regulatory, statutory or contractual obligations around auditing and record keeping.  Secondly, as a corollary to the comments above about code comments and testing plans, keeping a record of decisions made and work done is a sensible part of running a good business.  The backlog should be the record of what's happening, when, why and who has done it.  This means that whilst scrum ceremonies do not need to have audit-able minutes, there is no requirement to stop producing a record of what decisions where made, when they were made, why they were made, and who made them, nor is there any push to stop entering time sheet information (which is often vital for the company to work out its final billing to a customer) or other standard records that might be kept.  It IS true that scrum tries to remove the all-encompassing documentation of every little thing that is said and done for the purposes of back covering, but that is as much about having a decent working environment for your staff as it is about removing an onerous and often pointless exercise in blame avoidance. 

In conclusion, what this principle is saying is that large, formal, immutable documents that form the basis of the work and act as a record of what was agreed and what was eventually done do not have the same level of importance to the scrum team or customer as a working increment of software delivered at the end of each sprint that has been created in line with the agreed priorities.

And that's something that we should all be able to agree on.

*** - Obviously the ceremonies, roles and artifacts are proscribed, but they provide a framework within which any implementation that uses them can be described as scrum or a variant thereof.  There are as many ways of operating in this framework, especially in the small details, as there are companies that implement it.

Welcome to the Agile Documentation blog

Hello, and welcome to the Agile Documentation blog.

This blog is written by an agile writer, for other agile writers, about writing documentation in an agile environment. You can expect a few posts about agile issues in general, and occasional posts on other things in the fun and glamorous world of Technical Writing, but mainly the aim is to shine a light on some of the bigger - and murkier - questions that affect writers who work in or with scrum teams.

Comments are welcome, as is feedback, constructive criticism, suggestions, and anything else.