Introduction
Documentation is a frequent source of heated discussion when it comes to Agile practices. Discussions can become rather animated as people from different backgrounds and experiences bring their unique viewpoints to the table. Some people say that Agile and documentation cannot coexist because they are incompatible, comparing them to oil and water. On the other hand, other people believe that documentation is an essential and crucial component of the Agile process.
Maybe you are one of the believers that Agile says to avoid documentation completely. We're here to debunk this myth.
The approach to documentation that Agile and Extreme Programming take is sometimes misunderstood or misinterpreted. So, in this post, we'll go into the essence of XP and expose the true nature of Agile documentation as it relates to this powerful and dynamic methodology.🚀📝
The Truth About Documentation in Agile
Let’s start by stating the truth very clearly: in no way does Agile Manifesto, XP, or any other Agile-related methodology suggest avoiding documentation completely.
I mean, that wouldn’t make any sense, right? Some documentation is just needed, sometimes. Think about audits, support, onboarding, tracking decisions, brainstorming - there simply are good reasons to document stuff, sometimes.
The only real take about documentation is:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
[…]
Working software over comprehensive documentation
[…]That is, while there is value in the items on the right, we value the items on the left more.
The Agile Manifesto states very clearly that “there is value” in documentation, but working software is considered more important - and hopefully, this is something we all agree on.
The problem is that, over time, some people misunderstood this sentence and spread the idea that all documentation should be avoided. This is simply not true and is one of the things that contributed to making people skeptical about Agile.
Still, most people underestimate that statement - and I think that to clarify it we need to bring eXtreme Programming's take on documentation when describing the principle of “mutual benefit”:
Extensive internal documentation of software is an example of a practice that violates mutual benefit: I am supposed to slow down my development considerably so some unknown person in the potential future will have an easier time maintaining this code.
This is the big problem with extensive documentation in most organizations, IMHO: while some consider the cost of documentation, almost no one considers the ROI of that cost - that most of the time is very low.
Writing technical documentation has a lot of issues:
-
writing good technical documentation is hard, for multiple reasons:
-
even writing well in general is hard
-
writing good technical documentation is a different job than writing code
-
-
async written communication is generally less efficient than sync conversations (and the fact that writing well is hard doesn’t help)
-
it is completely detached from the codebase, so you need to invest effort to be sure either you don’t have duplicate info between code and doc or to keep that duplicated info in sync
I’m sure most of you feel related to those problems, but you probably think that documentation is still important and we just have to do it.
I agree, we need certain types of documentation, but not all.
The heuristics we should care about
From all the ways to think about software documentation, I like the approach of splitting it into two main categories:
-
Product documentation → here we have all the documentation we write for users (both internal and external), but also all the documentation related to the system - for example, API public documentation and Architecture documentation fall into this category
-
Process documentation → here we have all the documentation we produce during the process of development = for example, user stories and meeting notes fall into this category
We can safely say, IMHO, that Product documentation is the most important one to invest in - because it will last over time; Process documentation instead, is only useful in the process, then it becomes mostly useless.
This is one of the keys to approaching documentation in an Agile (and smart) way: consider trade-offs like we always do in code (or should do!). If the Product doc is more important, we want to invest more time in it compared to the Process doc.
Talking about the mutual benefit example we discussed earlier, the solution is not to don’t write documentation at all - XP simply suggests a different approach:
-
Invest time in writing automated tests instead of documenting the code what it should do and how to use it - this way not only benefit will be mutual (for both the current dev and the future one) but in general the code will be easier to read and maintain
-
Invest time in refactoring instead of documenting the code to explain what it should do in detail - this way, we can remove accidental complexity and improve readability, giving the current dev more satisfaction and fewer defects, and also an easier-to-understand code for the future dev
-
Invest time in names to make them explicit and coherent, so that the current dev works faster and future dev will easily read it
The point is not to avoid documentation, but to handle it in a professional and smart way - not writing any doc or writing too much doc without thinking are just 2 faces of the same, problematic coin.
Basically, we should handle the documentation the same way we treat code and tests: the less it is, the better; the more automated it is, the better; the less duplication there is, the better; and so on.
Here are the principles and heuristics that we can follow to face documentation in an effective and sustainable way:
-
Code should be self-documenting → the right place to read what the code does is the code itself: instead of writing documentation to describe it, refactor it to make it readable
-
Tests are living documentation → tests are still code, and they basically are examples of usages of the code under tests: make tests readable so that they can actually be useful as documentation
-
Automate all the documentation possible → writing documentation is painful, and keeping it updated with changes is even more painful - be sure to always ask yourself if a piece of documentation can be automated, and pick your tools to enable the automation chances; documentation is costly, we need to reduce that cost
-
Keep documentation close to what it describes → instead of having a single place for documentation, make sure you document something very close to where you use it - a centralized wiki makes a lot of sense for company-wide and product documentation, but, for example, document your codebase in the repository itself
-
Write just enough documentation → “just enough” means be sure you write all that truly matters, in a clear way, but no more; it also means be sure to cover all that matters
-
Write documentation “just in time” → before documenting, wait. Produce documentation when it’s needed, not before - a similar approach to creating an abstraction: do not do it immediately, wait until the last responsible moment when you are sure you need it and most information is now steady and not changing continuously.
Just as clean code and solid architecture contribute to a healthy project, so does comprehensive and well-maintained documentation - and missing/wrong/bad documentation also contributes to technical debt, so again, “just enough”: write all that’s important, but no more.
This synergy can help us steer clear of the ever-dreaded tech debt, maintaining a system that's truly agile, adaptable, and ready for the future.
Until next time, happy coding! 🤓👩💻👨💻
Dan’s take 🙋🏻♂️
I’m pretty good at documenting stuff - at least, this is how I feel about it, due to some self-reflection but also the feedback I got in my career.
The truth is: that I hate documentation - so I had to learn how to handle it in order to avoid writing too much.
At first, I did some research, because you know - it’s not like you can stop doing something just because you don’t like it, it is not professional. I wanted to understand how people were handling documentation.
In the first years of my career, when I still wasn’t into Agile, all I could find was a lot of tips on how to write and maintain a big documentation. I’ve never been able to make it work, and I’ve never seen people that have that approach to documentation making it work properly.
Documentation is just part of the Agile and XP revolution that I faced years ago: in the same way that I discovered that there actually are ways of developing software successfully, there also are ways to handle documentation successfully!
Differently from code, where we can identify a lot of practices to approach development in an Agile way (TDD, CI, etc..) - on documentation we mostly have principles and ideals.
Today, I want to give you an example of what we set up in my previous team, where I was tech lead and we tried to handle documentation in the most lightweight and agile way we could think of.
-
Distributed documentation
In a similar way to distributed services and microservices, we tried to approach documentation in a distributed way: we tried to keep the documentation always as close as possible to the thing we wanted to document.
Everything related to the codebase was in the README file in the repository - things like a description of the project, setup requirements, and domain knowledge that it’s worth an explanation of why it’s implemented that way.
We had a 2 layer Wiki (made with Notion) - layer 1 was company-side documentation, used for organization stuff like describing the company the business, etc; layer 2 was the products, each product had a dedicated sub-wiki where documenting all the things related to that product (POCs, External APIs or Hardware tools doc, Meeting notes, etc).
We also had an ADR repository: we used the Any Decision Record interpretation, and there we tracked all company-wide tech decisions, such as the format for money and date in our internal APIs.
-
Prefer automation
We were a small team of a small startup, so it was fundamental for us to consider the impact of documentation on our productivity - and being able to automate documentation was a game changer for us.
Here is the best example of it: we didn’t need a Swagger documentation API for external users - our API was only used by the front end. At first, we automated Swagger doc from code - then, we evolved it to create the Typescript types library and automatically share it from BE to FE. This way, we automated the creation of 90% of boilerplate code - we only needed to implement the client class using that type, we had a standard way of handling exceptions and the only thing that remained to share was special error cases. Another future improvement we planned was automating the entire SDK instead of only the types of Requests and Responses, but we didn’t achieve that before I left.
-
Pick tools that enable automation
It’s fundamental to choose your tools with automation in mind: for example, we choosed Notion because thanks to its APIs we had the chance to automate some kind of documentation - for example, we automated the publication of approved ADRs, and when I left we were working in automating release notes using Notion as CMS.
Having tools such as Notion, with official API included in the plan. will always enable you to automate stuff as you need - and most of time even without implementing due to an existing integration via Zapier/Make.
-
Consider the trade-offs
In the same way, we need to consider trade-offs when implementing something in the code, we should do the same with documentation.
The simplest example: is if you need to invest 1 hour to document a change that requires 15 minutes of implementation, you should ask yourself if it’s worth it. Probably, you can reduce that time - probably, you are documenting way more than “just enough” needed. Be sure to document everything that’s worth it, especially the whys and constraints that lead to a certain decision.
But be 100% sure that everything you write in documentation is worth the time.
-
Shared responsibility
Writing documentation is not the responsibility of a single specific person, even if he was the one working on the related part. Everything related to the product should be a team responsibility, following Agile principles - and documentation makes no exception.
Be sure that automation is the priority for everyone when it comes to documentation, and that the written documentation that you still need to write is written by a different member every time so that the pain is shared.
This is just an example, something that worked pretty well for us in a context where documentation was only internal and we were driving it as a best practice - so basically the company trusted us in most decisions.
In your context, you could of course face more constraints (stakeholder needs, user-facing documentation required, etc) - but following the same principles and heuristics shared in this article, you should be able to handle it successfully without making developers suffer!
Documentation is a pain 99% of the time, so be sure to face it in a smart way or it will lead you to hard times. Avoid documenting what can be written in code, avoid handling manually what can be automated, and in general treat documentation as something that has a high cost: carefully consider every investment in it.
Go Deeper 🔎
📚 Books
-
Agile Documentation: A Pattern Guide to Producing Lightweight Documents for Software Projects - Documentation is the castor oil of programming. Managers think it is good for programmers, and programmers hate it!
-
The Product is Docs: Writing technical documentation in a product development group - This book provides a broad perspective on the essential aspects of creating technical documentation in today's product development world. It is a book of opinions and guidance, collected as short essays.