Do you know the Black Swan Theory from N.N. Taleb? And did you realise it can be linked to software engineering and a customer-oriented approach like Lean/Agile? In fact, agile has the potential to work better than other approaches, even when it’s executed less than properly.
Read this intriguing blog, written by Berk Ülsoy – Practice Manager ‘Open Source, DevOps Cloud & Open Network Infrastructure’ at Devoteam – and discover why agile principles fit better.
To many people, software engineers seem to be constantly focusing on bits and bytes, while being disconnected and isolated from the hassles of real life. One can sit in front of a screen for hours, trying to solve a problem in an appropriate way… until a product manager comes over and asks something ‘very urgent’… or until that lucrative contract falls… or until…
Or maybe, software engineering is not isolated from the hassles of real life at all. Maybe it lives right in the centre of it, impacted by wing flaps of a small butterfly?
Brewing their experiences in software engineering over decades, the thought leaders of our industry authored the ‘Agile Manifesto’ to present a set of values and principles that give organisations a better chance to stay relevant to their customers, with sustainable quality in a fluctuating environment. Many consider the combination of lean software development and agile our best chance of survival. It is not perfect, but we do not have a better alternative yet.
But why has agile the potential to work better than other approaches, even when it is executed less than properly?
In this post, I will try to answer this question with the help of Nassim Nicholas Taleb’s infamous The Black Swan Theory.
The Black Swan Theory
Wikipedia includes following information about N.N. Taleb: Nassim Nicholas Taleb is a Lebanese-American (of Antiochian Greek descent) essayist, scholar, mathematical statistician, and former option trader and risk analyst, whose work concerns problems of randomness, probability, and uncertainty. His 2007 book The Black Swan has been described by The Sunday Times as one of the twelve most influential books since World War II.
Black swan is a metaphor for events that are totally surprising from the perspective of the observer (but not necessarily from the perspective of the originator), have major impact and are often rationalised after they occurred, as if they could be well predicted.
Some black swan events according to Taleb are the invention of the internet, personal computer, 9/11, World War 1, or the collapse of the Soviet Union.
What’s more, Taleb introduces two categories or lands in which random events fall: Mediocristan and Extremistan.
Mediocristan is a land with a lot of variations without none of them being extreme. Evolution belongs to Mediocristan, tons of genetic mutations are happening all the time, but nobody develops wings overnight. In Mediocristan, variations fit into normal distribution and no event can have enough impact to change distribution substantially. Most of the time, we think and act as if we live in Mediocristan.
Extremistan on the other hand has low variations with some of them being extreme (fat-tailed events). Therefore, the impact of a black swan event is consequential.
The world that we as humans created (business, economy, life) operates in Extremistan. Many of our problems are caused by our belief that we live in Mediocristan and have safe assumptions, forecasts or precise plans. It is not our reality. That is why I would say:
Software engineering, as integral and ever-growing part of the business and economy, also lives in Extremistan.
Impact of Black Swans in software engineering
Theory shows its marvel on a large scale and large impact events. But let’s try and see if we can scale it down to a single industry, company or a bunch of people and check if this principle also applies to much smaller areas.
The area of impact is a single company or team, rather than the whole world, societies or countries. Many events can be consequential for a team or a company. In the worst case, the company goes bankrupt or a team is disbanded.
Now let’s think about some events that heavily impact our engineering efforts:
- An important customer requesting an immediate feature (a surprise for the observer, well-known by the customer)
- A new regulation in a country we end up having customers (surprise for the observer, not for the author)
- That big customer who pays the majority of the bills, is going out of business or is changing its partner (again, not a surprise for the customer)
- Drastic changes in roadmaps due to fluctuating business requirements
- Serious events in the world shaking the businesses from its core (political instabilities, economic issues, pandemics)
Could we predict any of those as a member of a company? Probably not. If an event would be perfectly predictable, we could proactively engage them before they even happened. That level of predictability would contradict the fact that we live in Extremistan.
But that is not the case. We indeed must accept that “we do not know what we do not know”. Moreover, we do not have control on events like that. We can only survive through them, gaining some advantage if we play it properly.
Claiming events like the above mentioned and many more are predictable, meaning that software engineering is pretty much a deterministic area. However, it is described as an empirical domain, rather than deterministic.
Agile embraces the unknown exactly like this. It does not deny the fact that we cannot know and predict everything. While waterfall-based approaches treat the world like a controlled environment with a strong self-esteem and long-term detailed plans; a lean/agile mind inherently knows that the map is not the territory and positions itself to survive through consequential events.
Therefore, an agile mind is closer to how the software world actually works.
Containing the damage
What can we do when a consequential event happens?
We can either pick a more flexible, opportunistic, customer-oriented approach like lean/agile, or something else that sits on the opposite side of the bar.
The cost of containing damage can skyrocket if we follow a path that is inherently rigid and in denial of realities.
- Reduce risks as feasible, appropriate as possible, to contain the damage when the risk realises
- Be prepared and have the flexibility to make course corrections, to react on events without losing time
Both of these points map to many of the lean/agile principles.
Conclusion
When we read The Black Swan, we are exposed to a clear portrayal of the world as it brutally exists. It is definitely not a recipe book. It teaches us how to look at the world in a different, much better way.
On the other hand, agile principles start with the assumption that our problem domain is complex, non-deterministic and empirical. We often do not question it, but I believe that internalising this assumption is important to see why those principles work better. If we dig into failed agile transformations, more often than not, we will see that the organisation still tried to deal with the world as if it was living in Mediocristan.
If the world and the business we live in is part of Extremistan, it also implies that there will be some black swan events. If those events satisfy the criteria and happen consequential enough for our existence (team, company, product, etc.) we will have to deal with them somehow. Since lean/agile principles naturally prepare us better than other approaches that deal with such events, we can conclude that lean/agile is a very good way to survive in Extremistan.
If you have questions our you want more info, please reach out to the author Berk Ülsoy. Continue reading by consulting our Distributed Cloud offer.