Why is discussing architecture so hard?

In my previous post I outlined a few of the ways that application architecture fits with the Agile values.  In this post, I attempt to outline some of things that make it tough to get everyone on the same page architecturally.

As an architect I've noticed a curious thing over the years.  Most of the issues I face in software have to do with the lack of sound architectural principles or the tangled mess of monolithic style applications strung together by not following SOLID object-oriented principles.  (I include much of what I've written in the past in that mess.)  However the first thing you hear when you mention architecture is a fear that the application will be "over architected."  (This is also often confused with the term "over engineered.")  I suggested creating one class this week in a meeting and was met with, "well we don't want to over architect this."  Yikes!

Why the fear of over architecting when so much software is so poorly designed in the first place?   It didn't make sense until I had a conversation with my friend Eric.  He mentioned that if you can't visualize the architectural solution in your own head, then it just becomes an impediment.   You just can't get down that path from where you are.  And it may make you feel stupid.

No one wants to feel stupid, especially not smart people and software people are almost all smart.  No one likes to feel as if they don't know how to do something.  Talking to someone who is not an architect about architecture has a tendency to make them feel stupid.  I've noticed that even software architects don't go around talking too much about architecture.

Looking back on it, most of the time when I've heard the fear of "over architect-ing" mentioned it was followed by comments like "I can do this faster without (that)" or "that's just in the way" or "I don't understand why we need that."   I could never reconcile that with the promise of design patterns (remember: a design pattern is a solution to a problem in a context).  Applying design patterns within a solution should make you go faster, not slower.  They certainly shouldn't be more "in the way" than they are helpful.  In fact, they should allow you to talk about and refactor large chunks of functionality with ease.  More importantly, they should communicate what they do and how it might be tested.

I've watched the same developer that expressed concern about too much architecture struggle with the lack of structure.  With chaos that is barely manageable.  I've watched them struggle to go fast with so many tangled dependencies and interwoven responsibilities plaguing their objects.  I've watched them struggle with where to put concepts and ideas as the requirements change.   I've watched them be forced to use debugging to get their code into submission as the immediate feedback from QA is a string of bugs they didn't expect but which were completely predictable.

While watching all this, I'm convinced that these developers don't connect their struggles with the cause.  They'll often say things like, "Well that changed," or "no one could have predicted that" or if they are talking to non-technical people they will just mention the technical reasons for the issue.   A break when reading a feed will be blamed on it's structure, and while technically correct, the rejected architectural components which would have made adjusting to the change is rarely brought up.   

I think part of the problem is that when you first introduce most architectural style changes they do get in the way.  Most architecture is meant to prevent a problem that you don't have yet.  When you are building a house you often have a stage where you put up a ridge support of some kind.  These "ridge" beams tend to run the entire length of the building and are usually extremely heavy.  Putting up the ridge beam tends to be very hard to do.  Often it's even followed by a little celebration.

After you put in the ridge beam, it looks sorta stupid.  Here you have this huge beam, basically floating in the air supporting nothing.  Even after the walls are put on, it seems a waste.  But everyone knows it's not.  They know it holds the house together and that when all the roofing, insulation and heating components are installed, a sturdy beam like that will be a benefit, a structural requirement even.

Well, the first day you decide to move a set of classes into a control layer, or a domain layer, or a persistence layer, or a UI layer, or a services layer the added infrastructure is going to look and feel like that ridge beam sitting out there for no reason.   At this point (for the first few days or a few weeks on larger projects) it is extra, you probably could go faster without adding it.   But unlike the ridge beam, not everyone can envision it's necessity down the road.   The comment above that "I don't understand why we need it" is a killer.  And it's not just the architecture that's at risk, the success of the project is at risk.  If the efforts to put structure behind the chaos are undermined, the chaos to follow will bring the software to it's knees like a building without a ridge beam.

So that's the chasm we are trying to cross.  On one hand, Agile, RAD, and solid architecture go hand in hand.  On the other hand, some developers aren't ready for design patterns, application layering, and constant refactoring.  It threatens them because before you can refactor, you have to see at least a hint of where you want to go.  Before you can layer, you need to understand the sort of things that might share a layer - you also have to understand how the layers will stitch together and why the constrains they allow are a good thing.  You have to understand that layering is one of our few lines of defense against the chaos of dependencies.   As communicators of architecture the architects need to understand that talking about it can make smart people feel stupid.   It's tough.

It may be tough, but if we want to be RAD, do Agile, while still creating quality software we have to find a way to cross the chasm.

Suggestions?

Where does application architecture fit in the Agile/Rad landscape?

The Manifesto for Agile Software Development states:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

Lets look at these one at a time.

Individuals and interactions over processes and tools

Developers are human and humans are social animals.   From a process point of view, when we come together as a group to review architecture, I've noticed that it's one of the most interactive parts of the software development process.  The debates are lively and the conversations are usually friendly and robust.  So, having architectural discussions is a great way to stress individual opinion and group interaction.

A primary tool in architecture development is the design pattern.  Design patterns were inspired by Christopher Alexander, the father of the modern pattern language view of architecture.  By establishing and using design patterns, advocates (...) claim that ordinary people of ordinary intelligence can use it to successfully solve very large, complex design problems.  When you use a design pattern, you are applying an architectural solution.

Working software over comprehensive documentation

This one is interesting.  It says you should spend your time writing and creating working software rather than creating comprehensive documentation.  However, it is not an excuse to not create ANY documentation.  Some of the easiest and most useful documentation can be architectural.  So, if you're trying to minimize the time spent on documentation, make an architectural drawing.  Do it quickly, include just what's important to convey your ideas and constraints, then build it in working software.

Better yet, since working software is your goal, build it and document it once, and then reuse it.   Reuse has gotten a bad name in the last decade or so because it's so hard to do it right.  However, with the NuGet packages, extension methods, and test driven development, we are finally in the position to say that if you are not creating the software you deliver by plugging-in, stitching together, and importing templates, components, and binaries from a familiar and trusted supplier (yourself in many cases), then you are costing someone time and money.

RAD originally meant (in 1991) the use of heavy prototyping as an emphasis on developing software (see Martin, et al).    However today it tends to refer to highly component-ized development using frameworks like Rails, VB, Lightswitch, etc.  (Ruby, by the way is not a RAD language, Rails is a RAD framework because of the level of scripting.)

Since our goal is to create working software we need to focus our effort.  In software we often do that by focusing on one responsibility at a time.  In object-oriented programming, this is know as the Single Responsibility Principle and it is a fundamental principle for creating software using the SOLID guidelines.

You can write software without following these principles, but it won't be object-oriented and it won't be as maintainable as it could be.   It's also highly unlikely that it will be easy to change.  In other words - it won't be a form of RAD.   By creating clarity and focus, you create a landscape where you have individuals interacting to create working software over documentation.   Proper architecture helps make that possible.   Take a quick look at the SOLID guidelines and how many of them help guide good structure and style (architecture):

Initial	Stands for (acronym)	Concept
S	SRP	Single responsibility principle the notion that an object should have only a single responsibility.
O	OCP	Open/closed principle the notion that “software entities … should be open for extension, but closed for modification”.
L	LSP	Liskov substitution principle the notion that “objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program”. See also design by contract.
I	ISP	Interface segregation principle the notion that “many client specific interfaces are better than one general purpose interface.”[5]
D	DIP	Dependency inversion principle the notion that one should “Depend upon Abstractions. Do not depend upon concretions.”[5] Dependency injection is one method of following this principle.

Customer collaboration over contract negotiation

One of the biggest values that architecture (and experience) bring in this area is that knowing the architecture helps the technical people to estimate better.  Estimation will remain as flexible as feature scope, but the closer the technical people can come to providing the customer with realistic expectations for quality, cost, features, and schedule the better the collaboration with the customer will be.

Responding to change over following a plan

When good architectural principles (layering, components, and design patterns) are applied, then followed with sound software development practices (Lean, Agile, Scrum) and SOLID object-oriented principles are followed, the software stays flexible to change.   That's not to say that some changes won't take a long time, but it does mean that they will take a fraction of the time to change when compared to a product where these practices and principles were not followed.

These thoughts on architecture are continued in my next post about why discussing application structure and style is so hard.