How Complexity Domain Impacts Software Development Process

Should your software product development team be agile?

The answer I almost always hear (especially in Silicon Valley) seems to be "of course, yes", but the right answer should be: "it depends." These days there seems to be a bit of religion and marketing brainwash around development teams' requirement for "agility" and the Scrum process in particular, and yet for all of the talk most teams in my experience were not actually agile and many of them should not have tried to be.

I submit that one of the greatest factors impacting how a team should structure their software development process is "complexity domain": a concept pioneered by the brilliant Dave Snowden with his Cynefin framework. This post describes complexity domain and how it forms a decision model around how to structure a team's software development process.

1. What is Complexity Domain?

Complexity Domain refers to the environment in which the team is building their product. Dave Snowden's Cynefin Framework describes this most clearly through the following classification of different kinds of complexity environments shown below:

For now don't worry too much about deciphering the nuances of the framework except just to understand the four primary categories. The ability of the team to interpret cause and effect best determines the complexity domain they are operating in.

More specifically as follows:

1. Complex: In this domain, the relationship between cause and effect is unclear. If we are building a new kind of product with unknown user reaction this is the domain that fits best.

2. Complicated: Here the relationship between cause and effect is solvable through careful analysis or specialized, expert knowledge.

3. Simple: The relationship of cause and effect should be obvious, at least to those skilled in the field of study.

4. Chaotic: There is no apparent relationship of cause and effect. Hence, you can't tell WTF is going on and potentially shit is hitting the fan.

Cynefin is both a description of complexity environments but also serves as a decision making framework that will help determine the best approach to how a team should approach their work... and, as I will argue later, their specific development process.

For additional clarification on the Cynefin Framework, Dave Snowden does a great job explaining in more concrete detail in his video below:

The beauty of Cynefin is the ability to have a simple way of determining how best to approach a team's product development process. Snowden refers to this as "requisite applicability" which is to say:

The danger is that too often teams are used to a specific approach and therefore keep taking their hammer to the world. As Snowden puts it: "we will interpret the situation based upon our personal preference for action."

Ok. So what?

So bear with me for a little bit longer, but for now let's revisit the Cynefin framework image one more time (I've added yellow highlighted areas to focus on):

Based on the complexity domain, Snowden argues an optimal approach for each domain. Note the yellow highlighted text (Chaotic skipped in the interest of focus/time); this describes how the team should approach their work.

• Simple: In the Simple case, the team should know how best to respond just by understanding the situation. Hence, the first step is to "Sense" the problem, "Categorize" it, and then "Respond" accordingly with a known "best practice" solution.

• Complicated: As mentioned earlier, experts in the field can figure out what to do with enough time and analysis. Therefore the process of actions should be to first "Sense" the problem, "Analyze" how to solve it or determine what to build, and then "Respond" with the appropriate solution or product design.

• Complex: In this space, because of the lack of ability to determine cause and effect we should conduct a number of experiments to see what works. Hence the decision model is "Probe", "Sense", then "Respond." Based on experiments conducted: "if an experiment succeeds we amplify, if an experiment starts to fail we dampen it." This is that magic area of "emergence" where new solutions will emerge based on experimentation/Darwinism.

Let's now cut to the chase. This means we can infer optimal approaches based on the complexity domain we are faced with. In particular, I submit that based on how users should approach their work according to this framework we can conclude that particular well known software development process practices apply as follows:

2. Complex vs. Simple Domains

Why is This True?

Well let's for the moment look at the Complex vs. Simple domains. We need to select a development process that supports the ideal work approach described:

• Complex: Probe -> Sense -> Respond, vs.

• Simple: Sense -> Categorize -> Respond

Review of Scrum:

Scrum is the most popular form of agile software development and seems to address the work approach of the Complex domain the best. There are 2 underlying key principles to be aware of that support: Probe -> Sense -> Respond:

1. Iteration (Probe, Sense)

2. Incrementalism (Respond)

Iteration:

• This principle assumes that we will screw up and will need a lot of changes and re-work to develop what we need.

• So the idea is to have a planned rework strategy or as Snowden puts it "safe fail experiments."

• In Scrum, iterative development is built into the work process.

Incrementalism:

• This philosophy supports building the product in incremental pieces instead of trying to build the full product; hence, the Scrum concept of sprints.

• Further, each sprint should deliver a PSPI (potentially shippable product increment). Therefore, at the end of every sprint there should be confidence that everything delivered works, has been tested, and there is no remaining uncompleted work or outstanding technical debt.

• Even further, the advantage of sprint is to delay the planning process based upon new information or information we obtain from reviewing the output of the latest sprint. Without this, Scrum loses much of its advantage.

As you can see in the diagram above, Scrum supports the Complex work approach because we are able to incrementally build product and conduct a number of safe fail experiments (iterate) which we can then amplify or dampen after each sprint.

Review of Waterfall:

In Waterfall, as most are probably already aware, there is a fixed process of product development that typically follows the following model:

Here, there is no incremental delivery. There may be some iteration or change requests but that then means just extending the process and introducing some delay.

The key advantages of waterfall approach however include:

• It is very simple and efficient.

• If we have the right product design without too much iteration required, this type of approach should yield the fastest and most efficient delivery.

• Further, management can make easy to understand go/no-go project decisions based upon fairly clear milestones of product delivery.

In the Simple complexity domain there is a single well understood solution, hence, there is no need to have the characteristics of Scrum such as iteration or incremental delivery. Supporting a system for amplification/dampening through testing and discussion/reflection around what to do next just introduces communication and process inefficiencies when it is not necessary. Further, this type of work flow process is one that many software developers still don't have much experience with.

Finally, one additional way to understand matching the right development process to complexity domain is based on the concept of product/market information that is gained as the project is being developed. The amount of planning & product specification work is front-end loaded in Waterfall as opposed to spread out over the project in a Scrum/agile approach.

• In a Complex domain we expect that product/market information will be steadily gained over time and as we conduct safe fail experiments across sprints

• In a Simple domain because the solution is well known we expect that we already know the majority of the product/market information ahead of time

What we don't want is a mismatch between our software development process approach and the expected product/market information we expect to have based on complexity domain. See below:

In the above diagram, I depict:

• Planning (Waterfall): Waterfall planning process where specifications are front-end loaded in the design phase

• Product/Market Knowledge (Complex Domain): Expected product/market knowledge acquisition typical of a Complex domain gathered over time

In this situation because we complete most of our planning and product specification ahead of obtaining product/market information, there will likely be significant re-work, iteration, and massive cost of changes.

3. Complicated Domain:

Finally, let's now take a look at the Complicated domain. Here, several different solutions can be successfully employed and therefore the appropriate response can be situational. This is an area where the people and team should be the focus rather than a specific process. Here is Snowden on this topic:

Here, make your choice based on the specific situation and the expertise of the team that you have.

Summary:

That's it folks. Thanks for reading through and I hope these thoughts can be helpful when you are designing the right software development process for your team!