Are Some Software Development Challenges Too Complex for Agility?

Synopsis
When the concepts of agile software development are introduced to an organization, a common response is, “Sure -- that will work with a simple application, but ours is a very complex ‘Enterprise’ system”. Could it be that core agile software development principles are too simplistic to tackle the really tough situations? I recently found myself in a position in which it appeared that this might be so.

The Problem
The problem that confronted me was that I had a legitimate stakeholder requirement, but I was limited in my implementation approach by a highly-interdependent system of systems. To compound matters, the optimal approach that would allow me to mitigate risk to the system while implementing the specified requirement was precluded by a firm stakeholder-imposed constraint. It didn’t take long for the agile-averse to begin chanting, “See how complicated it is? That’s why an agile approach won’t work here. This requires REAL architecture and design”.


So there I was with a justifiable stakeholder requirement, limited by a tightly-coupled architecture, shackled by a constraint that wouldn’t go away, and playing to an antagonistic crowd. It was a fine situation I had gotten myself into, and I was certain that I could get out – I just wasn’t sure how. I tried to think of an analogous set of circumstances that would help provide clarity while distancing myself from the details specific to my situation, but nothing quite captured the essence. After having wrestled with this for the better part of an afternoon and into the night, I decided to call it a day. I flipped on the television for a few minutes of decompression, and happened across a very interesting documentary…

The Story
The story was that of a man who had lived most of his 50 plus years with a medical condition that had severely diminished his ability to breathe, eat, and see, and which had finally progressed to the point that it was life-threatening. He had seen many doctors over his lifetime, and had always opted to live with the condition, but with the realization that his life was in danger, he was ready to seek treatment. The documentary team followed his journey to seek two medical opinions.

The first stop was a highly-qualified surgeon who had years of experience treating similar cases. After examining the man, the surgeon reported that he believed that he could help the man via a major surgical procedure, and that several reconstructive procedures would follow. He went on to inform the man that in order to perform the procedure, a blood transfusion would be required during surgery. For personal reasons, however, the man would not agree to have a blood transfusion. Faced with that fact, the surgeon told the man that he would not be able to help him.

The second opinion was rendered by a team of surgeons at a different institution. Upon evaluation of the man’s condition, they too reported that they would be able to perform the required surgery. However, the procedure they would perform would also require a blood transfusion. Again, the man would not consent. This time, though, instead of simply sending the man on his way, the team got together and came up with a second option. The second proposal was to perform a series of less-invasive procedures. This would provide vast improvement in the man’s ability to breathe, eat, and see without requiring any blood transfusions. The tradeoffs would be that they would not be able to accomplish all that they might have with the major procedure, and they would not be able to do some of the reconstructive surgery. The documentary concluded with the man still having not made a decision between a) going forward with the surgery team’s second proposal, or b) continuing to live with the condition.

The Light Comes On
The team of surgeons had been able to propose a way to meet the man’s most important needs while respecting his fundamental constraint, but balanced that with the reality that the solution would be an incremental delivery of a functionally-sufficient solution, and not a solitary “perfect” solution. It was a fantastically agile approach. As I watched the story unfold, I saw parallels between this and my situation, and it reminded me of some core principles:

Good teams will, and can be trusted to, make good decisions
As this story and standard practice in innovative medical institutions around the world indicate, the best opportunity for providing optimal medical choices derives from team collaboration. This principle works in software development, too, and I believe that the best solutions come from teams, and not gurus or heroes. As with the first surgeon in the story, the limited perspective of a single “expert” can result in too quickly locking onto the trees and missing the forest.

All of the constraints can’t be fixed
There’s certainly nothing new in this statement, but stakeholders will still try to control all the parameters (many project managers, incredibly, will nod yes and then set off to plan it so). Just as the man was left at the end of the story, stakeholders will have to make difficult choices at times. One of our most important responsibilities is to give them the best, most straightforward information possible to assist in those choices.

The desired benefit needs to be kept center-stage, even when it deviates from the “specifications”
Often, the desired benefit can be realized by some means other than the feature as specified by the stakeholders. In this man’s case, the team was able to maintain focus on his most critical needs and present a feasible option for meeting them, despite an immutable constraint. Sometimes there’s a less-expensive and/or more feasible route to the benefit other than the feature as specified. There may be situations in which the oft-cited “80/20” rule comes into play, whereby, for instance, we can offer a solution that would give the stakeholders 80% of what they’re asking for with a reasonable amount of effort, while the other 20% would cost dramatically more.

Problem Solved
Though I had allowed the problem I was facing to be cast as a perplexing dilemma, I realized that the path to its solution was really no different from the one I routinely take with problems I would consider less complex. This wasn’t an issue for one person to figure out, nor was the goal to deliver “to spec”. The resolution to my predicament, then, was to go back to the stakeholders and say, “Let’s put this to the team, realizing that they will propose options that will present the benefit you desire, but that the implementation may not be exactly as you have specified. Understand that the constraint you have imposed may limit our options, but also be reassured that you will be able to see incremental progress early and often, and that we can change direction in between increments as you see fit.”

Final Thoughts
Unquestionably, architecture and design are still important in agility. Dean Leffingwell’s “architectural runway” concept is an excellent approach to this, in my view. Moreover, we certainly need domain and technical expertise if we expect to produce valuable software. And, yes, there are situations in which strict adherence to specifications is required. But a hallmark of agility is that it deals with complexity by simplifying it -- not by fighting complexity with complexity. And now we are seeing industries that were once deemed largely off-limits to agility because of regulatory constraints embracing agility. So, are there software development situations that are too complex for the application of core agile principles? Until someone proves otherwise, I say “no”.