The Wizard of Oz-itechture: Decision Making in Software Architecture

According to research on Software Architecture, decision making processes were not created until the early 2000s.  The topic was not covered in research papers until 2003^[1]. But in system engineering, various tools have been used since the 1940s to aid the decision making process and these tools can carry over to Software Architecture. I wrote an earlier blog post was about using decision tools in architecture^[2]  I also worked with Axiomatic Design^[3] as a student and used it on several personal software projects. Axiomatic Design tools help decouple functional requirements with design so that a change in a design parameter only affects a single requirement.  

However, I have observed that the decision making comes down the the interaction of the various personalities in the process, regardless of the most detailed documentation.    I’ve observed a Wizard of Oz behind a black curtain making decisions on architecture.  Here’s how it works: you follow the detailed process to the letter, and for some reason that you don’t see, your architecture doesn’t get done.  Someone behind a black curtain is making the architecture decisions.  

Case Study 1:  Just Say No

We had a feature that customers had been screaming to get for years.  We had a solution that didn’t work very well.  It was over ten years old, was refactored to the maximum.  It was also complex to work with it.  When if failed, customers could distinguish the source of failure.  My boss called it a “two-headed state machine”.  

When we first introduced a new architecture for this problem, there was strong push back against it.  “It will take too long”, said one seasoned lead architect.  Customers ended up using their own software and hardware, which was a financial loss to us.

At that same time, some development on another feature was done to make our new idea more feasible. And yet we still  heard “It will take too long” from a new seasoned lead architect.  

So we prepared a truck-load of documentation from our customers and detailed use cases of how it would make our customer’s life easier.  It got approved and was about to start development. Then we heard “It’s a security risk”.  This came from our boss’s boss’s boss.  She didn’t read the documentation or history of this idea, and how little it would take to implement it.  

Our understanding of the process was that this decision should have been done by a large group of engineers that meets regularly.  Their decision can be overturned without a whimper.  

Case Study #2 - Bullheads

Years ago, we had to make a major decision about what infrastructure to use for an API.  It included a set of standards and technologies and would affect our products for years to come.  

We had over fifty requirements and many options to figure out how they met those requirements.  I used a decision matrix to evaluate the options with our team.  This matrix laid out all the options on one axis and all the requirements on the other.  We filled in how well each option met each requirement.  Our team had built in bias: we had a set of team members that wanted one way, and another set of team members that was already biased another way.  When we finalized the matrix, we found two options that were in just a few points of each other.  The team whose favorite lost was upset.

Perhaps there was another non-functional requirement that we missed?  I asked them but they couldn’t come up with one.  It seemed arbitrary that their favorite way would lose by a few points out of almost 1000 points total tallied.

Amazingly a few years later, a third option which we considered, won the contest.  A bull headed engineer had a set of emergency meetings, and pushed this way through. And sadly for me, it was the right decision.

So much for spending weeks doing a DSM.  

Case Study #3 - Group Think

Most companies I’ve worked for have a review board to review design and architecture.   In this scenario, the decision is made by a team of people.  If the team doesn’t agree there is a process to raise the issue to a higher level person who makes the final call.  The team then must live with that decision.

While this works well in my experience, it can be not as altruistic as it sounds.  First, people can canvas members before hand to make sure things go the way they want in the review.  I’ve heard “Hi Chris, I know Bill will be against my design today.  Can we review and will you support my design?”.  Let’s all gang up on Bill when he objects.

Recent research is that group-think can settle into teams.^[4]   To have a functioning group, you should get people you don’t like because they think differently.  Years ago, I had two people on my team. One was a nice person, but did not contribute very much to our work.  The other had a prickly personality, but questioned everything we did, and was sharp as a tack.  Ms. Prickly ended up being far more valuable by disrupting a lot of things we were doing that no longer worked.  With a more diverse team, we disagreed more but work through some difficult design problems and came out better for it.  

Case Study #4: I’m with the boss

I had a nasty feature to design. It had a lot of moving parts and some difficult security challenges.  But I was loath to overdesign it.  After we had presented an initial design to the team that was going to be implementing it with us,  the lead architecture burst into one of our meetings.  He talked so rapidly no one else could talk.  He presented a new design and announced “Oh and this has been approved by ”.  Everyone went silent.  Most in the room wouldn’t go against what a VP designed, and those of us that would, well, this person wasn’t in the room for us to debate.  

When I tried to debate the fast talker, I was shut down.  “ said that is way to do it”.  

I had a fight on my hands, but this feature got dropped. I’m preparing to present my case to the Big VP when it returns.  

Conclusion

Be clear about how you make decisions, and learn what really goes on behind the curtain (in other words - what do you not see about decisions).  Document decisions.  Understand why they are made and make sure up to your Big VP to the developers and testers - they all understand.  And I don’t know why but whenever I document something well, we never have to go back and look at why (which drives me crazy but I guess it works).  

---

[1] See https://www.researchgate.net/publication/291185587_Decision_Making_in_Software_Architecture/download

[2] https://abstractsoftware.blogspot.com/2010/08/architectural-decisions-or-on-purpose.html

[3] https://en.wikipedia.org/wiki/Axiomatic_design

[4] https://player.fm/series/worklife-with-adam-grant/how-astronauts-build-trust - listen how you need people you don’t like or trust on your team to think differently than you.

Further Reading: Good Traits of a Software Architect

Interface Segregation Principle

This is another principle developed by Robert Martin who I have had the great pleasure to see speak several times at conferences.  He is a very lively speaker and you return to coding wanting to do your best work after you see him speak.  As for a skilled software craftsman, he is at the top of his game. 

ISP says that no client should be forced to depend on methods it doesn't use.  Martin came up with this when he was consulting with Xerox.  He found they had a job class that was used for every single task in the system.   A stapling job shouldn't have to know anything about a print job and all the methods for printing.

OK people I work for Xerox and there nothing called  Staple Job.  Stapling is an attribute on a job.  Print jobs can have an attribute that says staple me.  But so can Copy jobs.  So I'm not sure where he saw this but maybe it was a looong time ago. 

In practice when working with Microsoft Windows for example, there are what I call "Big A!& Classes".  They are huge with a bazillion attributes and methods.  Often I don't care about 9/10ths of them. I search through the documentation for what I need.  This is one of many interfaces I have used and dare say I have created that violate this principle. 

However, when I create an interface, I literally fight people to the bone to keep it simple.  More functionality is desired and interfaces get new methods and before you know it, you have a massive class with more methods and data then any sane person can keep straight.  I have to argue quite a bit to keep it simple - deadlines are tight and people don't want to do the extra work for this principle. 

On the flip side, I think this principle could be considered "too pure" for many software engineers.  And engineers I talk to about this principle find it too difficult to practice in a real word setting.  They understand it intellectually but don't see the benefits.   Testing is one huge benefit.  If you have a massive class for a job you would have to implement the print functions even for a staple job (which doesn't exist, just saying…).  Even if it was a no-op, you'd have to test it.

Testing takes up over half of our development activities.  So if you can save on that, ISP is worth it! 

Single Responsibility or Changing Growing Responsibilities Principle

Architecture is full of examples of items designed to do more than one thing.  Your smart phone is a flash light.  From my last examples, a chair is also a clothes hanger.  A seat cushion is a flotation device. 

Robert Martin said in his book Agile Software Development, Principles, Patterns, and Practice (http://en.wikipedia.org/wiki/Robert_Cecil_Martin),  that every class should have a single responsibility over a part of the function.  The function-part should be encapsulated and the class should do that and only that. 

I understand the need for this as it makes the class easy to understand.  It also could reduce the introduction of bugs.  Have you ever worked with "spaghetti code"?  You make a change and it causes a bug in another area you couldn’t have imagined in your wildest dreams.   If a class has two different things it does, and you change one thing, it could affect the other piece and thus break something.  Martin gives an example a class that compiles AND prints a report.    If the content of the report changes, the class must change.  If the format of the print must change, then the class changes.  Martin argues that there is no reason for these two things to be coupled because if the format of the print changes, the content does not change and thus should not need to be recompiled and tested. Martin says a class should have only one reason to change (not two, like in this example).  

In an imperfect world, needs of software change over time.  An example: My manager is crazy over this particular module we have to is used to access and configure accounting data.  Over time it has evolved to access security data as well.  This drives him crazy because it violates the single responsibility principle (although he doesn't call it that and doesn't know that Robert Martin invented the term).  I however, think this is OK, and have argued this fact with him.   Given the schedule pressures we were under to combine these two pieces, which are closely related in our system, is not horrific.  Not perfect but not catastrophic either.  

Form Follows Function and Function can also follow Form

We as engineers know this old saying and if you are an SDM graduate, you may remember hearing or saying it classes.   But where did that saying come from?  It can be attributed to a book by Peter Blake called "Form follows Fiasco".  It was a critique of modern architecture which the author said was too focused on form and that the function should be the utmost important in any design.  This is what I always thought "Form follows function" meant - but some interpretations say that if the function is met in the purest, most simple way, the form follows and it will be beautiful.  (1)  .  The chair designs in Denmark represent this principle.  Software, which can so easily get complex, is another example (see some actual code later). 

So if we look at the chair picture from my last blog post, in the last row, that chair looks uncomfortable to me.  If the purpose of the chair is comfort, then without a solid back and full arm rests, this chair doesn't fit the bill.  However, if the chair is to made to fit completely underneath a table, then the design goals are different and conflict with the comfort goal.  To make a chair fit underneath the table, some comfort will have to be sacrificed.

So depending on the goals of the design, what form  will meet the design criteria?  That is the question to ask.  Each design criteria might not be met perfectly but the stakeholders need to decide which one can be sacrificed.  It was more important for the chair to fit underneath the table then for it to be perfectly supportive of the back.  After all people will only be sitting here for short bouts at the dinner table. 

Also what is beautiful in software code? I had my 84 year old mother look at some code and I asked her if she thought it was beautiful. She thought all of it was … very strange.  She could see no beauty in it whatsoever.  (To be fair, she also never liked furniture from Denmark, even when it was all the rage here in the US in the late 1960s). 

Here's an example of some weather data I was working with recently from a web service that needed to be parsed:

The data was in JSON (Javascript Object Notation) format which is a widely used  by web services to return data described with text, and is human readable.  To parse this, we could created one massive class with all the items needed inside. 

However, one could break everything into subsclasses like this design, such as wind, coordinates, etc into separate classes.  Is that cleaner?  Easier to maintain?  Easier to understand?  Beautiful form?  I happen to like all in one, the first case, because I can see it all in one file.  But if I changed wind, I would end of changing the weather class.  That can be a problem to introduce bugs into a larger system. 

Back to my Danish furniture, I found a chair that was brilliantly designed as clothes hanger. 

Is this chair a good clothes hanger?  Yeah, it's pretty good.  It won't function as a full closet but it can hang a few things.  Is it the most comfortable chair in the world?  No but it is supportive and someone can sit in it, even when clothes are hanging on it.   And it also keeps beauty in the curved lines and rounded edges.  Notice that the form does indeed follow function.  This form is not just of a chair that you can sit on.  It's a chair that you sit on and hang clothes on. 

Another example is a seat cushion that serves as a flotation device.  This does two things.  It isn't the best seat cushion in town and it isn't the best flotation device in town, but it works as both.  Tradeoffs occur.  Can you imagine what would happen to this chair if the designer kept iterating to make the chair more comfortable and to hang more clothes?

So while I agree function dictates form, the simplest form and often the most beautiful form will come from meeting the a minimalist set of functions.  

Each Iteration is Better

That's the theory any way.  An iteration is defined as "A process of repeating a set of operations until a specific result is achieved".   Iteration in software works because we can typically build something quickly for stakeholders to review.  From that feedback, we can revise the design.   For stakeholders, sometimes they don't know exactly what they want, so iterating allows them to change their minds without waiting until the final product to discover it wasn't what they wanted at all. 

This funny picture shows all the stakeholders involved in a product and how it can go oh-so-wrong.  Iterating through the project can make this picture not so funny-sad.  Yes the customer will explain the project like the first picture and the first iteration might deliver the second picture.  "But it doesn't swing", says the customer.  OK, we'll fix it to make it swing.  There's the third picture.  But oh, I need it to go around in a circle too!  The fourth picture won't happen because the programmer was already delivering the first two pictures, right?  And the descriptions don't matter, because we are using an Agile development process where "Working Software over comprehensive documentation" is what is at value.  There wasn't a separate installation, the product was delivered in phases so the seventh picture shouldn't occur with iterations.  How software gets supported - remember, all the stakeholders should give input into iterations, including support staff.  Iterations would make this series converge on the last picture.  With each iteration, the first picture would  evolve into the last picture - and with hooks on the rope so it can be more easily replaced by support staff! 

With iterations,  each design cycle should narrow all the possibilities of change so that the project can move forward.    For iterating to work with software or really any product design, all the stakeholders must have a clear picture of the final product.  In practice, this doesn't always work.  For example, the customer has one vision,  and several iterations are completed, and then a new vision emerges.  This is often considered a positive, as iterating is a discovery process for all involved.  However, most stakeholders involved need some kind of finish criteria and often a date is involved as well as money.  If you take the customer on a journey to figure out what they want,   it should be clear that there you can't predict when this could end. 

I've been involved in several projects where convergence didn't happen.  One was a case where the product was documentation to describe an existing architecture.  I couldn't get stakeholders to agree on what the architecture was.  I iterated over it several times and was honing in on something and slowly but surely bringing stakeholders along.  But one stakeholder, who was only marginally involved decided at the last minute that the architecture description was all wrong and that I needed to take a completely different view point.  His idea wasn't wrong, it was actually an interesting take on our system,  but I was running out of time.  It was frustrating.  Since I could not get stakeholders to agree, I drew up a financial statement showing what the company had already spent on this effort and that convergence was not going to occur, and I recommended they give up the effort. 

Here's another twist on iterations.  You need a team that trusts each other and is honest, while still being respectful of course.  One the stakeholders was the president of the company.  He wrote up an architecture a few years earlier and had several people review it.  Everyone said it was good and correct.  I realized that no one had the gumption to give him the feedback they were giving me.   I guess I should feel lucky.  His architecture description was not very good.   So even though it looked like his documentation converged, it truly didn't.  It wasn't used by anyone (except him), there were things incorrect with it, and it wasn't a useful description.  

Iterations often don't converge in real projects because we run out of time and thus we are stuck with what we've got.  I observed non-convergence in iterations when I was in Copenhagen recently, and I visited several furniture design museums.  I was fascinated to see these modern furniture designers used iteration. 

But when I saw these chairs, they all looked wonderful - beautiful designs in their own.  They didn't seem like incomplete versions of something.  In part this is the nature of furniture design versus software.  You have to build a chair and then build another chair for each iteration, versus just the legs.  But in this case, it seems like the furniture designer was not heading towards an end goal. 

I have a hard time figuring out what the designer's goals were with the first row of chairs.  It seems like he started with something more traditional/colonial, made it smaller and more simple, and then made some radical changes with the last chair.  This last chair doesn't look all that comfortable, but it is simple and beautiful.  In the second row is that each iteration - something was taken out, to make the chair more simple.  Each chair in the is functional and beautiful. 

A design that meets all its goals perfectly will not always lead to the simplest design.  And yet the two most simplest chairs are arguably the most beautiful.  From my experience,  the simplest software is the most beautiful as well.

My rule for iterating is stop before you made it too perfect.  Meet a minimum set of goals which may deliver the best form.    

Architectural Niceties

"I'm not going to slip schedule for architectural niceties"

This is what my very-long-time-ago boss said to me when I suggested we make a change.  He was a great guy - very technical and had experience shipping and supporting all kinds of hardware and software products.  But the words stung.  I recalled these words on another project where I was designing something for another group in the enterprise.  They needed the design done yesterday so the schedule pressure was high.  They were already behind schedule.  When I suggested a change that would increase the robustness of the design, but would be more time, the systems engineer flatly rejected it.  "The engineer working on it only knows this one way on how to do things.  Besides we need to finish this."

Oh. 

Certainly, designing for today and not tomorrow is something I adhere to in practice.  In Kent Beck's Extreme Programming (http://en.wikipedia.org/wiki/Extreme_programming) he describes four values, one of which is courage. It does take courage to design and code for today and not for tomorrow.  We engineers love "what if" scenarios.  However, it requires a lot of courage to refactor something - to throw it away and rewrite it when it no longer works or is holding back progress in the overall system.  Management also needs courage to allow engineers to do this refactoring as sometimes it doesn't add any new functionality but allows the system to more agile to other required changes. 

One project I worked on we had management approval to rewrite a whole swath of code that was generated by an obsolete tool.  The tool and the company that sold it were gone but the code we had in our system remained.  When functionality had to be added we coded around this generated code creating solutions that were downright awful.  Finally we had enough and we wiped out the whole thing and rewrote it.  It took months with several engineers involved.  It was completely worth it.  And we had the courage to do it.

In another project we found out we did the wrong design.  Unfortunately we already released the interfaces and they could not be changed. So we added in another wedge of code that was somewhat duplicative but was the design we should have done.  I recently was talking with another engineer and she pointed out the problems in this code.  I cringed.  I remember having to do the work and was just horrified at the solution I created.  But hey, we were under schedule pressure.  We had to fix this and fix it quickly. 

So as much as I want to believe I take the architectural high road, I don't.  I realize I have to have the courage to refactor these things.  Off to talk with my boss…it is a different person so maybe he will appreciate architectural niceties. 

Is Software Architecture Creative?

With my recent focus on building architecture and its comparisons to software architecture, a reader wrote me and said the he often found the field of software uncreative versus building architecture which he found very creative. I was surprised to hear this from anyone because I find software architecture inherently creative. I find living life creative though. So what do others think?

The Software Engineering Institute surveyed leaders in the field of software and asked them to define the skills of a software architect. This page collects those submissions. Out of the 37 or so entries, only eight listed the word “creative” in the description of skills needed.

Again I was surprised. I expected to find creativity as a key trait everyone looked for in a software architect. So maybe my reader was correct? Why do I find it so creative then?

Like any hard work, sometimes I get in a rut. Peter Cripps writes in his blog about keeping creativity flowing:. He has three points that I use in my everyday work.

First, collect the pieces as they come. Architecture in my experience is never designed all up front. We do the major framework and then we implement, then we change it, then we implement that new change, then we add something new to the framework, and we implement and test that. Make sure you don’t ignore that new information when it is fed back to you as the architect. Don’t gold plate your architecture.

Second, stop talking and do it. This of course is the heart of agile. Architects do get blamed for living in an ivory tower. This is why am I still writing code. Not only do I love to write code, but I can actually implement some of my designs. All that said, I know that I get criticized for thinking big picture when there are fires raging. I feel obligated to do that and I don’t consider it a waste of time. If that was all I ever did that would be a problem, but I believe in a combination of hands on work and pie-in-the-sky thinking.

Third, decide the time is now. This means using the information you have at the moment to set a direction and make a decision. I sometimes manage the risk of doing this but making the area most unknown the most flexible in the architecture. But I have also had to completely change directions in my design. Most managers hate when I do this. They feel it wastes time. Of course if I could always be right the first time, that would be ideal. But that is not realistic. Please, managers, allow your people to make mistakes!

So while building architecture is obviously artistic and creative, software architecture must also be. What happens sometimes with buildings is that they are so creative, the architect misses the function of the building. (Dare I point out some buildings at MIT that are like this? Not in this blog post!). In the field of software architecture, we strive to deliver functionality with a form that has the properties we desire over time – like maintainability, readability, extensibility – and other “ilities” that we compromise with depending on the system we are building. To me this is as creative as painting and it is work I could do for a long time.