Saturday, 30 April 2016

Measuring Technical Documentation

A list of things you can measure about technical documentation.

When measuring these things, you need to find your baseline figure - what the figure is now - and aim for each figure to go up (e.g. documentation coverage) or down (e.g. number of bugs) over time.  This can be measured on any timescale you want, but probably the most useful would be one of monthly, yearly, by sprint, or by release.

This list is not intended to be exhaustive or authoritative (inspect and adapt!). Not all of these measurements will be relevant to every situation. How you measure these things is not covered here. There are other measurements you can perform for writing teams or individual writers; these are not included here.

The terminology should be understood as follows:

 - A documentation set is a vertical slice, i.e. the documents that cover a complete product.
 - A documentation type is a horizontal slice, i.e. a type of document that will be in most if not all documentation sets, such as a help file, configuration guide, etc.

Completeness:

  • All documentation sets are complete, i.e. contain every required documentation type;
  • All document types are complete, i.e. contain every required section.

Coverage:

  • Every product has a documentation set;
  • Every documentation set covers all aspects of the product;
  • Every document type that is relevant to the product is included in the documentation set;
  • Each document covers every aspect that a user would find in a default installation.
Notes:
  1. Coverage doesn't have to include ALL documentation for a product, as some documentation - e.g. training material, sales and marketing literature - does not come under the heading of technical documentation.  Where this boundary lies is a question for your company to answer.

Consistency:

  • Every document set contains the same document types (if needed for the product that the documentation set covers);
  • Every document type follows the same formatting and layout guidelines;
  • Every document follows the same formatting and layout guidelines;
  • A global glossary defining all terms is available and complete;
  • Style guidelines defining writing style are available and complete;
  • Every document uses the same terms in the same way, i.e. adheres to the global glossary;
  • Every document is written in the same style, i.e. adheres to the style guidelines;
  • Every document is clearly labelled with related release(s), product(s) and version.
Notes:
  1. Where there are multiple delivery mediums, the formatting and layout guidelines should provide the same look and feel as much as possible.
  2. There may be multiple glossaries for different markets, e.g. English vs American English, or different industries where standard terms are different.

Bugs:

  • The total number of reported documentation bugs;
  • The number of reported bugs per documentation set;
  • The number of reported bugs per document type;
  • The number of reported bugs per document;
  • The average time to correct any documentation bug;
  • The average time to correct bugs per documentation set;
  • The average time to correct bugs per document type;
  • The average time to correct bugs per document.
Notes:
  1. "Average time" means "from the time the bug is accepted onto the backlog to the time the fix has passed testing".  This can be measured in days, sprints or releases.

Peer-review:

  • The percentage of total documentation (%oD) that has been peer-reviewed;
  • The %oD per documentation set that has been peer-reviewed;
  • The %oD per document type that has been peer-reviewed;
  • The %oD per release that has been peer-reviewed;
  • The %oD that has been reviewed by technical staff;
  • The %oD per documentation set that has been reviewed by technical staff;
  • The %oD per document type that has been reviewed by technical staff;
  • The %oD per release that has been reviewed by technical staff;
  • The %oD that has been reviewed by customer-facing staff;
  • The %oD per documentation set that has been reviewed by customer-facing staff;
  • The %oD per document type that has been reviewed by customer-facing staff;
  • The %oD per release that has been reviewed by customer-facing staff.

Notes:

  1. "Peer reviewed" means other professional technical communicators - writers, editors, documentation managers.  They will check for technical communication errors/issues/omissions.
  2. "Technical staff" means architects, designers, developers, testers.  They will check for technical errors/issues/omissions.
  3. "Customer-facing staff" - means those staff who install, configure and support the product for the customers. They will check for understandability, ambiguity, and clarity both for themselves and on behalf of the customers.

Technical:

  • All documentation is written using a recognised Help Authoring Tool, e.g. MadCap Flare, Adobe FrameMaker, and/or a recognised technical communication methodology, e.g. topic-based authoring, DITA;
  • Different versions of documentations, e.g. for different markets or industries, are created from base versions;
  • Standardised templates are used when creating new pages or new documents;
  • All in-progress documentation is stored in a versioned repository, e.g. TFS , GitHub, alongside the release branch it relates to;
  • All released documentation is stored in a backed-up documentation management tool, e.g. SharePoint, Documentum

Technical Debt:

  • The total number of known technical debt issues across all documentation;
  • The number of known technical debt issues per documentation set;
  • The number of known technical debt issues per document type;
  • The number of known technical debt issues per document;
  • The average time to correct any documentation technical debt;
  • The average time to correct technical debt per documentation set;
  • The average time to correct technical debt per document type;
  • The average time to correct technical debt per document.

Notes:

  1. "Average time" means "from the time the debt is accepted onto the backlog to the time the debt has been paid".  This can be measured in days, months, sprints or releases.

Access:

  • All documentation (including the glossary) is available for viewing and/or download on a public website (this may be limited to customers only);
  • All documentation (including the glossary and style guidelines) is available for viewing and/or download on an internal intranet/portal/document management system;
  • All web-based documentation adheres to W3C accessibility guidelines;
  • All documentation relating to servers is available in simple formats such as RTF or HTML/XML/XHTML (because a lot of servers won't have Office or a PDF reader installed).

Bug reporting:

  • Documentation bugs are reported in the same way and in the same system as software bugs;
  • Documentation bugs are added to the backlog in the same way as software bugs;
  • Documentation bugs are given the same type of weighting and value as software bugs.

Customer feedback:

  • External customer feedback is requested after every release (potentially every major release if you're releasing incrementally);
  • Internal customer feedback is requested after every release (potentially every major release if you're releasing incrementally);
  • Negative/constructive feedback is treated as a bug;
  • Negative/constructive feedback is treated as more important than what is "right", e.g. if there is multiple feedback suggesting that more screen shots are needed, this overrides the standards that say screen shots should be kept to a minimum;
  • Negative/constructive feedback is treated as more important that a writer's sense of pride.

Customer feedback suggests that:

  • Every documentation set is useful;
  • Every documentation type is useful;
  • Every document is useful;
  • Every documentation set is accurate;
  • Every documentation type is accurate;
  • Every document is accurate;
  • Every documentation set is delivered in a timely fashion;
  • Every documentation type is delivered in a timely fashion;
  • Every document is delivered in a timely fashion.

Notes:

  1. "Timely fashion" may mean before delivery of the software, or at the same time as the software.  This is dependent on the customer needs, i.e. internal customers will often want all the documentation before the software is released to customers, and customers may want the release notes before the software is delivered, but not need the help file until they receive the software.

Any suggestions for omissions from this list are welcome.