Is Software Architecture Creative?

With my recent focus on building architecture and its comparisons to software architecture, a reader wrote me and said the he often found the field of software uncreative versus building architecture which he found very creative. I was surprised to hear this from anyone because I find software architecture inherently creative. I find living life creative though. So what do others think?

The Software Engineering Institute surveyed leaders in the field of software and asked them to define the skills of a software architect. This page collects those submissions. Out of the 37 or so entries, only eight listed the word “creative” in the description of skills needed.

Again I was surprised. I expected to find creativity as a key trait everyone looked for in a software architect. So maybe my reader was correct? Why do I find it so creative then?

Like any hard work, sometimes I get in a rut. Peter Cripps writes in his blog about keeping creativity flowing:. He has three points that I use in my everyday work.

First, collect the pieces as they come. Architecture in my experience is never designed all up front. We do the major framework and then we implement, then we change it, then we implement that new change, then we add something new to the framework, and we implement and test that. Make sure you don’t ignore that new information when it is fed back to you as the architect. Don’t gold plate your architecture.

Second, stop talking and do it. This of course is the heart of agile. Architects do get blamed for living in an ivory tower. This is why am I still writing code. Not only do I love to write code, but I can actually implement some of my designs. All that said, I know that I get criticized for thinking big picture when there are fires raging. I feel obligated to do that and I don’t consider it a waste of time. If that was all I ever did that would be a problem, but I believe in a combination of hands on work and pie-in-the-sky thinking.

Third, decide the time is now. This means using the information you have at the moment to set a direction and make a decision. I sometimes manage the risk of doing this but making the area most unknown the most flexible in the architecture. But I have also had to completely change directions in my design. Most managers hate when I do this. They feel it wastes time. Of course if I could always be right the first time, that would be ideal. But that is not realistic. Please, managers, allow your people to make mistakes!

So while building architecture is obviously artistic and creative, software architecture must also be. What happens sometimes with buildings is that they are so creative, the architect misses the function of the building. (Dare I point out some buildings at MIT that are like this? Not in this blog post!). In the field of software architecture, we strive to deliver functionality with a form that has the properties we desire over time – like maintainability, readability, extensibility – and other “ilities” that we compromise with depending on the system we are building. To me this is as creative as painting and it is work I could do for a long time.

The Permanence (or Not) of Software Architecture

There is a series of rants in software blogs about software architecture not being the same as building architecture. (http://bexhuff.com/software-architecture-is-not-building-architecture , http://triebert.nl/downloads/072005_swarm_concluding_paper.pdf ). The basic theory is the buildings are permanent where software is ever-changing.

I’m reading a book that throws this theory on its head. How Buildings Learn is subtitled “What happens after they’re built”. I live in a section of Massachusetts that has many old mills. The mills, beautiful sturdy buildings were falling down. An effort was made to rework them and now we have condos, restaurants, and businesses in these beautiful red brick buildings.

And although I agree that software architects are not building architects, we’ve been building for thousands of years and we’ve been creating software for 60 years at most. The book’s author agrees with the software people that buildings are not designed to adapt, but all buildings “adapt anyway, however poorly, because the usages in and around them are changing constantly.”

This line could also be applied to software. How many cases can you think of where the software was designed to do one thing but was called on to do another in later revisions? My team spends a lot of time figuring out what will be or could be desired in the future so we don’t break any needed extensibility. This is the Open-Closed Principle which states: Software entities like classes, modules and functions should be open for extension but closed for modifications.

Several famous building architecture quotes from the book can be examined in the software world. “Form ever follows function”, said by Lois Sullivan, a Chicago high rise designer in 1896 implies that we can always determine function. In software we can’t. In fact with Agile processes, we focus only on the needs at hand, not what could be (because it may never come to be).

Winston Churchill said “We shape our buildings, and afterwards our buildings shape us”. The author points out that this goes on forever. Yes we shape our buildings/software, then they shape us, and then we shape them again, etc. This reminds me of putting software in the marketplace, getting feedback, and then reworking the software to meet the needs of the customer.

“Flow, continual flow, continual change, continual transformation” is not part of the Agile Manifesto although it sounds like it could be. It’s how a Pueblo Indian, Rina Swentzel, describes her home village.

The book points out that more is being spent on changing buildings than building new ones. Most software engineers I know work on older systems modifying them versus building something brand new. The changes in old buildings are caused by three forces: technology, money, and fashion. In software, technology is a huge force in the design. No one would argue that. Money/ schedule/ resources also have more effect on software architecture than anything else in my experience. How many times have you not been able to do it right because of time/money/resource constraints?

Fashion is defined in the book as “non-functional stylistic dynamism”. There are certainly these pressures on software architecture – government regulations, software processes that go in and out of fashion, design patterns, etc. all effect software.

Three types of buildings are mentioned in the book – commercial, domestic, and institutional. I think there are also three types of software bodies – web, start-up and/or open-source, and larger companies/institutions. Commercial/web applications adapt quickly and radically due to intense competitive pressure. Domestic buildings/ start-up code bases or open source change steadily due to direct feedback from the family/customers and designers. Institutional buildings/software bases are slower to change there is reluctance and delay but they do eventually change.

But what in a building or software architecture project actually lasts or does not change? What makes it become loved? In a building, it is age. We have a module in our system that has been around since the beginning and has not changed much. I’m not sure it is loved or admired. But I secretly admire it - there isn’t much in the software world that survives the way an old grand building survives. The module is simple, extensible, and easy to learn.

So while I’ve always believed that Software Architecture is not permanent, I’m beginning to think we can learn more from the building architects. I will be blogging further abut many of the chapters in this book which I found on: http://www.accreditedonlinecolleges.com/blog/2011/20-educational-architecture-books-anyone-can-enjoy/

Why Software Architecture Fails

I have worked on very old software systems. Some are over twenty years old. Many software engineers I know work on older systems. What happens is subsystems get replaced over times by new subsystems, fundamentally changing the architecture. Sometimes when the architectural decay is so great, the entire system must be replaced.

Not so in this system I worked on. It is a work horse, as complex as it is now, with hundreds of engineers contributing to it.

There was one piece of this system I remember well. It was a communications system that allowed subsystems to exchange requests and data with ease. It has had very little change over the many years it has existed. The architecture stood the test of time. Even new users praised its simplicity. We added over the years many new subsystems and they all use this communications system.

But this is a rarity in the software world. George Fairbanks, in his book Just Enough Architecture, discusses a software system that manages hosted e-mail servers. Customers call support for help and support must search log files to see what happened so they can help the customer. The first version was simple and a grep style search was performed on log files on each of the servers. But as the system grew these searches took longer, slowed down the servers, and engineers were required to do the search. This architecture had failed. So a new architecture for logs was needed. The next version of the architecture had the log files copied to a central server every few minutes and the data was loaded into a central database. Support techs could access this database via a web interface. But the central database got slow when the amount of log data grew even more and sometimes the centralized server failed, losing data. The architecture had failed. So a new system was created by saving the log data into a distributed file system and parallelizing the indexing of log data. Ten commodity machines were used rather than one single powerful server using a third party file system. This system finally was able to accommodate their growing data needs and has remained in place.

In his book Why Buildings Fall Down, Mathys Levy says “A building is conceived when designed, born when built, alive while standing, dead from old age or an unexpected accident.” At first glance, this doesn’t seem to apply to software. Software doesn’t die from old age. It can run over and over again and not wear out. That said, software and the design behind it does seem to have a lifecycle. And when the load on it becomes too heavy, change must come or the system will not run, like the e-mail server example.

The book many sections that when read, appear to be talking about software. It has a chapter on redundancy and theorizes that all structural failures are due to a lack of redundancy. There is a redundancy failure with the e-mail server system as well as shown with centralized server.

Most software systems are not made to withstands what the book describes as Big Bangs. Whether directly or accidentally, software architectures make tradeoffs based on risks. If the risk of receiving huge amounts of log data is low, as it was in the beginning of the e-mail server project, then a simple architecture will work. But as the risk grows higher, a system to withstand the big-bang of data is required. Most buildings were also not created to withstand a big bang. An analysis of why the World Trade Towers fell on 9/11 (http://en.wikipedia.org/wiki/Collapse_of_the_World_Trade_Center) reveals that the building was not created to endure the heat of the fire and impact.

Resonance failures of when bridges break are similar to infinite loops in software. (http://www.youtube.com/watch?v=3mclp9QmCGs ). When this kind of loop occurs in a system, it will eventually shut down, just as the bridge broke.

Many buildings are covered with facades or what the book calls “Structural Dermatology”. Software also has a design pattern called a façade. A facade is an object that provides a simplified interface to another software subsystem. Like a building façade improves the look and feel of a building, the software façade can redesign a poor interface into a great interface. Building facades in New York City were causing a lot problems as they were crumbling down, water damages was creeping in, and lack of skilled artisans were not able to maintain and repair the facades. Software facades can also be just as problematic as the original and need to be skillfully added for the pattern to work.

Software can technically last forever. Buildings when they are erected are believed at that time to last forever. But in both cases they often do not. Neglect, abandonment, and replacement often dismantle buildings as pointed out in the book. Neglect, abandonment, and replacement also dismantle software architectures. As with buildings, it is often economically more feasible to demolish what is there and recreate something new.

The book concludes with five factors that the longevity of a structure. They are:

1. Structural theories – this is the mathematics around load and structure.

2. Calculation techniques – the ability to compute various scenarios in the design process.

3. Material properties – how do the building blocks of the architecture work under various scenarios?

4. Communication procedures – how do all the people working on the architecture understand all aspects of the architecture?

5. Economic factors – what can do within a particular budget?

In many ways, this is identical to the issues we deal with in software architecture.

1. Structural theories – what are the design patterns we are using?

2. Calculation techniques – what are our performance metrics?

3. Material properties – what are our assumptions about what data will be flowing through our system?

4. Communications procedures – how do we work together as a team? Do we all understand the systems architecture?

5. Economic factors – what can we do within the given time we have with the people we have?

There are more similarities to buildings than I could have imagined.

For more information on building architecture: http://www.accreditedonlinecolleges.com/blog/2011/20-educational-architecture-books-anyone-can-enjoy/

Book Review: Just Enough Software Architecture by George Fairbanks

Software architecture tends to be so hard to define which makes it extraordinarily hard to read and even harder to write about. Thank goodness this is improving and the George Fairbanks new book Just Enough Software Architecture is a great improvement on software architecture books I have read in the past. Many of the books I have read in the past have taken tremendous discipline to get through – not so with this one!

The book focus “just enough” is a prevalent theme which makes it different from many other architecture books that are often written to promote a particular set of tools or methodology. Here Fairbanks lays out many ideas from the field and elaborates on the ones he finds most useful. He tends to have an agile slant which I appreciate. His main thesis is that architecture work should be risk driven, and he a chapter an on example that he actually worked on (it wasn’t invented for the book).

The area of the book I found most useful was on models, which is over the half the book. My favorite chapter of the book was “Using Architectural Models”. This was full of real world recommendations that I will be able to put to use immediately. I create “mini-models” almost daily as we try to sort through various issues. He also had a great section on informal and formal analysis techniques, the informal techniques being especially useful.

I think this book will be a valuable book for a college level course but also to software practitioners both architects and developers. There are not many textbooks that are useful to software architects practicing in the field. But this book is useful and it will provide material for many blog posts to come.

Abstraction Amplifies

One of the important uses of abstraction in software architecture is the ability to isolate a problem into its essence without cluttering it with details. In Just Enough Software Architecture, George Fairbanks says that abstraction provides insight and leverage. Mary Shaw created this diagram (from George Fairbanks book and reproduced here)

When a problem is complex we software architectures abstract the problem, abstract the solution, and then apply the abstraction to the real world. Less complex problems can be solved directly from the real world problem to the real world solution.

In my day to day work, I often have problems that I need to bring to a larger group. Without abstracting the problem, the multi-disciplinary team would collapse. Some of the business minded people would glaze over with programming details and some of the developers would dive deep into the details without looking at all the options.

Different abstractions can be made for different groups. This is related to views of the system. I was once asked to derive an architecture of an existing system. I started with all the inputs and outputs of the system, based on the hardware ports. This abstraction worked for many problems but some problems it did not. I needed another abstraction for how data flowed through the system. These two models would have been overwhelming if I looked at them together but by focusing on the problem at hand, I could create simple yet orthogonal abstractions of the same system.

A good rule of thumb when modeling from Fairbanks book is “question first and model second”. Otherwise you will not have the right abstraction as I discovered. And when others can read my model, this increases common understanding.

My designs are based in a process “The Universal Principles of Design” call Design By Committee. It is based on consensus building, group decision making, and extensive iteration. It works best when there are diverse members on the committee – like business people, managers, testers, and developers. To me this is an ideal committee – one that can vet out any issues in the design.

Everyone is a Software Architect

When I first starting working in the field of software I don’t remember the job title “Software Architect”. We did have managers, who were very technical – still writing code, and leaders, who we vetted design with. These people also set technical direction but they weren’t called architects. I worked in a large open lab area, in 1985, and looking back we were practicing XP. We worked in pairs and I have no idea how the work was organized. My manager would give me a project /problem and off I would go either with my office mate or someone else.

In that job, we were building a very large CAD/CAM system that never saw the light of day unfortunately. It was a project that could have benefitted from a software architect. In this case, the product was dealing with some fierce market forces that should have been understood and managed by a business team and an architect.

Another project I worked for was with a brand new manager who was eager to make his mark. He picked a new technology and went with it boldly. I remember thinking how fearless he was. It did not pay off. The project failed disastrously with this new technology not delivering the performance to be competitive. And then I thought, too late, shouldn’t we have challenged the manager? If we did, we may have saved his job, the project, and the company a boatload of money.

When I became an architect myself, I realized that many people wanted me to tell them what to do. That doesn’t work well for me. First of all, I do not know all the answers. Second, I rely on other people to give me data so that we can make an informed decision.

In Just Enough Software Architecture, George Fairbanks talks about distributed architecture skills. If developers don’t have architectural skills, they give the architect poor data for their models. This is true for me in my daily work. I rely on developers to give me information about the system they implement on. But beyond this, I also believe that architects should be experiencing development. I don’t think they can be doing as much development as a pure developer/software engineer/programmer, but I do think it’s important for me to experience the environment we have to deliver to. Although architecture is a big picture activity, it can be greatly influenced and also affect the minutia of the system.

I believe having someone responsible for architecture is an effective organizational model. But that person must make sure the entire team has a mind for architecture because otherwise the architect may not have influence. Architectural principles are not easy tested and if broken by implementers, the result is not found until it is causing problems.

So why wasn’t I an architect in those early projects? I truly was not; I was just trying to do the day to day work assigned to me. But I was starting to develop an architect’s mind and if I had spoken up in either of those early projects, the results of those disastrous projects may have ended differently. I only hope the developers I work with will do the same for me.