The Abstract Software Architect

Is Software Architecture Creative?

2011-12-22T12:38:00.004-05:00

With my recent focus on building architecture and its comparisons to software architecture, a reader wrote me and said the he often found the field of software uncreative versus building architecture which he found very creative. I was surprised to hear this from anyone because I find software architecture inherently creative. I find living life creative though. So what do others think?

The Software Engineering Institute surveyed leaders in the field of software and asked them to define the skills of a software architect. This page collects those submissions. Out of the 37 or so entries, only eight listed the word “creative” in the description of skills needed.

Again I was surprised. I expected to find creativity as a key trait everyone looked for in a software architect. So maybe my reader was correct? Why do I find it so creative then?

Like any hard work, sometimes I get in a rut. Peter Cripps writes in his blog about keeping creativity flowing:. He has three points that I use in my everyday work.

First, collect the pieces as they come. Architecture in my experience is never designed all up front. We do the major framework and then we implement, then we change it, then we implement that new change, then we add something new to the framework, and we implement and test that. Make sure you don’t ignore that new information when it is fed back to you as the architect. Don’t gold plate your architecture.

Second, stop talking and do it. This of course is the heart of agile. Architects do get blamed for living in an ivory tower. This is why am I still writing code. Not only do I love to write code, but I can actually implement some of my designs. All that said, I know that I get criticized for thinking big picture when there are fires raging. I feel obligated to do that and I don’t consider it a waste of time. If that was all I ever did that would be a problem, but I believe in a combination of hands on work and pie-in-the-sky thinking.

Third, decide the time is now. This means using the information you have at the moment to set a direction and make a decision. I sometimes manage the risk of doing this but making the area most unknown the most flexible in the architecture. But I have also had to completely change directions in my design. Most managers hate when I do this. They feel it wastes time. Of course if I could always be right the first time, that would be ideal. But that is not realistic. Please, managers, allow your people to make mistakes!

So while building architecture is obviously artistic and creative, software architecture must also be. What happens sometimes with buildings is that they are so creative, the architect misses the function of the building. (Dare I point out some buildings at MIT that are like this? Not in this blog post!). In the field of software architecture, we strive to deliver functionality with a form that has the properties we desire over time – like maintainability, readability, extensibility – and other “ilities” that we compromise with depending on the system we are building. To me this is as creative as painting and it is work I could do for a long time.

The Permanence (or Not) of Software Architecture

2011-08-30T20:22:00.001-04:00

There is a series of rants in software blogs about software architecture not being the same as building architecture. (http://bexhuff.com/software-architecture-is-not-building-architecture , http://triebert.nl/downloads/072005_swarm_concluding_paper.pdf ). The basic theory is the buildings are permanent where software is ever-changing.

I’m reading a book that throws this theory on its head. How Buildings Learn is subtitled “What happens after they’re built”. I live in a section of Massachusetts that has many old mills. The mills, beautiful sturdy buildings were falling down. An effort was made to rework them and now we have condos, restaurants, and businesses in these beautiful red brick buildings.

And although I agree that software architects are not building architects, we’ve been building for thousands of years and we’ve been creating software for 60 years at most. The book’s author agrees with the software people that buildings are not designed to adapt, but all buildings “adapt anyway, however poorly, because the usages in and around them are changing constantly.”

This line could also be applied to software. How many cases can you think of where the software was designed to do one thing but was called on to do another in later revisions? My team spends a lot of time figuring out what will be or could be desired in the future so we don’t break any needed extensibility. This is the Open-Closed Principle which states: Software entities like classes, modules and functions should be open for extension but closed for modifications.

Several famous building architecture quotes from the book can be examined in the software world. “Form ever follows function”, said by Lois Sullivan, a Chicago high rise designer in 1896 implies that we can always determine function. In software we can’t. In fact with Agile processes, we focus only on the needs at hand, not what could be (because it may never come to be).

Winston Churchill said “We shape our buildings, and afterwards our buildings shape us”. The author points out that this goes on forever. Yes we shape our buildings/software, then they shape us, and then we shape them again, etc. This reminds me of putting software in the marketplace, getting feedback, and then reworking the software to meet the needs of the customer.

“Flow, continual flow, continual change, continual transformation” is not part of the Agile Manifesto although it sounds like it could be. It’s how a Pueblo Indian, Rina Swentzel, describes her home village.

The book points out that more is being spent on changing buildings than building new ones. Most software engineers I know work on older systems modifying them versus building something brand new. The changes in old buildings are caused by three forces: technology, money, and fashion. In software, technology is a huge force in the design. No one would argue that. Money/ schedule/ resources also have more effect on software architecture than anything else in my experience. How many times have you not been able to do it right because of time/money/resource constraints?

Fashion is defined in the book as “non-functional stylistic dynamism”. There are certainly these pressures on software architecture – government regulations, software processes that go in and out of fashion, design patterns, etc. all effect software.

Three types of buildings are mentioned in the book – commercial, domestic, and institutional. I think there are also three types of software bodies – web, start-up and/or open-source, and larger companies/institutions. Commercial/web applications adapt quickly and radically due to intense competitive pressure. Domestic buildings/ start-up code bases or open source change steadily due to direct feedback from the family/customers and designers. Institutional buildings/software bases are slower to change there is reluctance and delay but they do eventually change.

But what in a building or software architecture project actually lasts or does not change? What makes it become loved? In a building, it is age. We have a module in our system that has been around since the beginning and has not changed much. I’m not sure it is loved or admired. But I secretly admire it - there isn’t much in the software world that survives the way an old grand building survives. The module is simple, extensible, and easy to learn.

So while I’ve always believed that Software Architecture is not permanent, I’m beginning to think we can learn more from the building architects. I will be blogging further abut many of the chapters in this book which I found on: http://www.accreditedonlinecolleges.com/blog/2011/20-educational-architecture-books-anyone-can-enjoy/

Why Software Architecture Fails

2011-07-24T21:46:00.002-04:00

I have worked on very old software systems. Some are over twenty years old. Many software engineers I know work on older systems. What happens is subsystems get replaced over times by new subsystems, fundamentally changing the architecture. Sometimes when the architectural decay is so great, the entire system must be replaced.

Not so in this system I worked on. It is a work horse, as complex as it is now, with hundreds of engineers contributing to it.

There was one piece of this system I remember well. It was a communications system that allowed subsystems to exchange requests and data with ease. It has had very little change over the many years it has existed. The architecture stood the test of time. Even new users praised its simplicity. We added over the years many new subsystems and they all use this communications system.

But this is a rarity in the software world. George Fairbanks, in his book Just Enough Architecture, discusses a software system that manages hosted e-mail servers. Customers call support for help and support must search log files to see what happened so they can help the customer. The first version was simple and a grep style search was performed on log files on each of the servers. But as the system grew these searches took longer, slowed down the servers, and engineers were required to do the search. This architecture had failed. So a new architecture for logs was needed. The next version of the architecture had the log files copied to a central server every few minutes and the data was loaded into a central database. Support techs could access this database via a web interface. But the central database got slow when the amount of log data grew even more and sometimes the centralized server failed, losing data. The architecture had failed. So a new system was created by saving the log data into a distributed file system and parallelizing the indexing of log data. Ten commodity machines were used rather than one single powerful server using a third party file system. This system finally was able to accommodate their growing data needs and has remained in place.

In his book Why Buildings Fall Down, Mathys Levy says “A building is conceived when designed, born when built, alive while standing, dead from old age or an unexpected accident.” At first glance, this doesn’t seem to apply to software. Software doesn’t die from old age. It can run over and over again and not wear out. That said, software and the design behind it does seem to have a lifecycle. And when the load on it becomes too heavy, change must come or the system will not run, like the e-mail server example.

The book many sections that when read, appear to be talking about software. It has a chapter on redundancy and theorizes that all structural failures are due to a lack of redundancy. There is a redundancy failure with the e-mail server system as well as shown with centralized server.

Most software systems are not made to withstands what the book describes as Big Bangs. Whether directly or accidentally, software architectures make tradeoffs based on risks. If the risk of receiving huge amounts of log data is low, as it was in the beginning of the e-mail server project, then a simple architecture will work. But as the risk grows higher, a system to withstand the big-bang of data is required. Most buildings were also not created to withstand a big bang. An analysis of why the World Trade Towers fell on 9/11 (http://en.wikipedia.org/wiki/Collapse_of_the_World_Trade_Center) reveals that the building was not created to endure the heat of the fire and impact.

Resonance failures of when bridges break are similar to infinite loops in software. (http://www.youtube.com/watch?v=3mclp9QmCGs ). When this kind of loop occurs in a system, it will eventually shut down, just as the bridge broke.

Many buildings are covered with facades or what the book calls “Structural Dermatology”. Software also has a design pattern called a façade. A facade is an object that provides a simplified interface to another software subsystem. Like a building façade improves the look and feel of a building, the software façade can redesign a poor interface into a great interface. Building facades in New York City were causing a lot problems as they were crumbling down, water damages was creeping in, and lack of skilled artisans were not able to maintain and repair the facades. Software facades can also be just as problematic as the original and need to be skillfully added for the pattern to work.

Software can technically last forever. Buildings when they are erected are believed at that time to last forever. But in both cases they often do not. Neglect, abandonment, and replacement often dismantle buildings as pointed out in the book. Neglect, abandonment, and replacement also dismantle software architectures. As with buildings, it is often economically more feasible to demolish what is there and recreate something new.

The book concludes with five factors that the longevity of a structure. They are:

1. Structural theories – this is the mathematics around load and structure.

2. Calculation techniques – the ability to compute various scenarios in the design process.

3. Material properties – how do the building blocks of the architecture work under various scenarios?

4. Communication procedures – how do all the people working on the architecture understand all aspects of the architecture?

5. Economic factors – what can do within a particular budget?

In many ways, this is identical to the issues we deal with in software architecture.

1. Structural theories – what are the design patterns we are using?

2. Calculation techniques – what are our performance metrics?

3. Material properties – what are our assumptions about what data will be flowing through our system?

4. Communications procedures – how do we work together as a team? Do we all understand the systems architecture?

5. Economic factors – what can we do within the given time we have with the people we have?

There are more similarities to buildings than I could have imagined.

For more information on building architecture: http://www.accreditedonlinecolleges.com/blog/2011/20-educational-architecture-books-anyone-can-enjoy/

Book Review: Just Enough Software Architecture by George Fairbanks

2011-06-12T21:10:00.000-04:00

Software architecture tends to be so hard to define which makes it extraordinarily hard to read and even harder to write about. Thank goodness this is improving and the George Fairbanks new book Just Enough Software Architecture is a great improvement on software architecture books I have read in the past. Many of the books I have read in the past have taken tremendous discipline to get through – not so with this one!

The book focus “just enough” is a prevalent theme which makes it different from many other architecture books that are often written to promote a particular set of tools or methodology. Here Fairbanks lays out many ideas from the field and elaborates on the ones he finds most useful. He tends to have an agile slant which I appreciate. His main thesis is that architecture work should be risk driven, and he a chapter an on example that he actually worked on (it wasn’t invented for the book).

The area of the book I found most useful was on models, which is over the half the book. My favorite chapter of the book was “Using Architectural Models”. This was full of real world recommendations that I will be able to put to use immediately. I create “mini-models” almost daily as we try to sort through various issues. He also had a great section on informal and formal analysis techniques, the informal techniques being especially useful.

I think this book will be a valuable book for a college level course but also to software practitioners both architects and developers. There are not many textbooks that are useful to software architects practicing in the field. But this book is useful and it will provide material for many blog posts to come.

Abstraction Amplifies

2011-05-30T18:22:00.004-04:00

One of the important uses of abstraction in software architecture is the ability to isolate a problem into its essence without cluttering it with details. In Just Enough Software Architecture, George Fairbanks says that abstraction provides insight and leverage. Mary Shaw created this diagram (from George Fairbanks book and reproduced here)

When a problem is complex we software architectures abstract the problem, abstract the solution, and then apply the abstraction to the real world. Less complex problems can be solved directly from the real world problem to the real world solution.

In my day to day work, I often have problems that I need to bring to a larger group. Without abstracting the problem, the multi-disciplinary team would collapse. Some of the business minded people would glaze over with programming details and some of the developers would dive deep into the details without looking at all the options.

Different abstractions can be made for different groups. This is related to views of the system. I was once asked to derive an architecture of an existing system. I started with all the inputs and outputs of the system, based on the hardware ports. This abstraction worked for many problems but some problems it did not. I needed another abstraction for how data flowed through the system. These two models would have been overwhelming if I looked at them together but by focusing on the problem at hand, I could create simple yet orthogonal abstractions of the same system.

A good rule of thumb when modeling from Fairbanks book is “question first and model second”. Otherwise you will not have the right abstraction as I discovered. And when others can read my model, this increases common understanding.

My designs are based in a process “The Universal Principles of Design” call Design By Committee. It is based on consensus building, group decision making, and extensive iteration. It works best when there are diverse members on the committee – like business people, managers, testers, and developers. To me this is an ideal committee – one that can vet out any issues in the design.

Everyone is a Software Architect

2011-05-16T20:43:00.001-04:00

When I first starting working in the field of software I don’t remember the job title “Software Architect”. We did have managers, who were very technical – still writing code, and leaders, who we vetted design with. These people also set technical direction but they weren’t called architects. I worked in a large open lab area, in 1985, and looking back we were practicing XP. We worked in pairs and I have no idea how the work was organized. My manager would give me a project /problem and off I would go either with my office mate or someone else.

In that job, we were building a very large CAD/CAM system that never saw the light of day unfortunately. It was a project that could have benefitted from a software architect. In this case, the product was dealing with some fierce market forces that should have been understood and managed by a business team and an architect.

Another project I worked for was with a brand new manager who was eager to make his mark. He picked a new technology and went with it boldly. I remember thinking how fearless he was. It did not pay off. The project failed disastrously with this new technology not delivering the performance to be competitive. And then I thought, too late, shouldn’t we have challenged the manager? If we did, we may have saved his job, the project, and the company a boatload of money.

When I became an architect myself, I realized that many people wanted me to tell them what to do. That doesn’t work well for me. First of all, I do not know all the answers. Second, I rely on other people to give me data so that we can make an informed decision.

In Just Enough Software Architecture, George Fairbanks talks about distributed architecture skills. If developers don’t have architectural skills, they give the architect poor data for their models. This is true for me in my daily work. I rely on developers to give me information about the system they implement on. But beyond this, I also believe that architects should be experiencing development. I don’t think they can be doing as much development as a pure developer/software engineer/programmer, but I do think it’s important for me to experience the environment we have to deliver to. Although architecture is a big picture activity, it can be greatly influenced and also affect the minutia of the system.

I believe having someone responsible for architecture is an effective organizational model. But that person must make sure the entire team has a mind for architecture because otherwise the architect may not have influence. Architectural principles are not easy tested and if broken by implementers, the result is not found until it is causing problems.

So why wasn’t I an architect in those early projects? I truly was not; I was just trying to do the day to day work assigned to me. But I was starting to develop an architect’s mind and if I had spoken up in either of those early projects, the results of those disastrous projects may have ended differently. I only hope the developers I work with will do the same for me.

More Defining of Software Architecture

2011-05-01T20:31:00.001-04:00

I’ve been reading a lot of books on software architecture recently and I’m amazed at how many definitions there are of “Software Architecture”. This blog post will look at the ones I’ve come across and critique them.

First is from the IEEE that I’ve been using a lot and had assumed it was being used every where (no it is not):

Software architecture is the fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution.
(IEEE 1471-2000)

I use this one a lot but this one doesn’t mention dynamic behavior although “relationships” although that could be interpreted in different ways. The elaboration of this definition defines various qualities of the system like performance which is related to dynamic behavior. Perhaps dynamic behavior is too closely related to functionality where the architecture is not?

From Wikipedia:

^·The software architecture of a system is the set of structures needed to reason about the system, which comprise software elements, relations among them, and properties of both. ^[1] The term also refers to documentation of a system's software architecture. Documenting software architecture facilitates communication between stakeholders, documents early decisions about high-level design, and allows reuse of design components and patterns between projects.^[2]

The one objection I have to this definition because software architecture designs are often not early in my experience. Some decisions must be made early but others need feedback from implementation before they can be made.

From the book Software Architecture in Practice (2nd edition), Bass, Clements, Kazman; Addison-Wesley 2003:

The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them.

Again – we’re missing the dynamic behavior of the system.

In Just Enough Software Architecture, George Fairbanks says that “Architecture is mostly orthogonal to functionality”. He compares architectures to skeletons and points out that some architectures are more suited to certain functionality than others. Perhaps this is why the dynamic functional behavior is often left out.

One of my favorite definition isn’t from software domains , but is from A Taxonomy of Decomposition Based on Structures, Behaviors, and Goals, Koopman, Design Theory and Methodology ’95:

·System Architecture should contain goals /requirements artifacts, and structure and behavior artifacts based on those goals

This has more to do with documenting software architecture but it covers my main three points that I like to see. It doesn’t say anything about a set of rules like the IEEE definitions does. Rules governing the architecture are very useful in my experience and can evolve over time. The rules govern the teams design decisions and given meaning to any assumptions and constraints. If those constraints change, often the rules can change.

The Software Engineering Institute comprised this web site to answer the question: http://www.sei.cmu.edu/architecture/start/definition-form.cfm. I’ll be adding this last definition.

How Software Architecture Reduces Complexity

2011-04-18T20:14:00.002-04:00

In the systems I have worked on in my career, none have been more complex than the software systems I see in the market now. There is constant pressure to add new features and new features get added at a dizzying rate. Some features are so complex that I can barely understand how they are to be used. And it’s not just one company that produces this complexity, software systems seem to be becoming more feature loaded. The systems are just plain bigger with more lines of code than when I started in the industry 25 years ago.

I’m reading a book right now titled “Just Enough Software Architecture and I’ll have a review in a few weeks. But in the first chapter, George Fairbanks discusses ways to deal with software complexity:

Partitioning: This is what the book Universal Principles of Design refers to as Modularity – A method of managing complexity that involves dividing large systems into multiple, smaller self contained systems. My experience with making systems modular is that I often can’t modularize completely in the first iteration of the system. I need to gain knowledge of that system and I tend to break it down into smaller pieces once I learn more. This approach works well when using an iterative process.

Knowledge: This is applying experience of prior problems to the current one. This reminds me of using design patterns. Although this may not be what Fairbanks had in mind, but I liken this to “Design by Committee” as described in the book Universal Principles of Design. This is a group process based on consensus building, group decision making, and extensive iteration. Recently I was involved in an architecture decision we had made that was found to be incorrect. But we weren’t sure. We took an iteration to do some tests, do some research, and then we all reconvened. We applied our group knowledge with our different perspectives to solve the issue, along with iterative research. We came to a final answer that I feel a lot more confident about than if I had done it myself.

Abstraction: Abstraction allows software designers to hide the lower level details when working with the architecture. The definition in Thefreedictionary.com says it means “a concept or idea not associated with any specific instance”. We use abstraction all the time when discussing our design and architecture. It is the only we can talk about the complex systems we work with and is key to communicating software architecture to all the stakeholders involved.

I’m not sure how I would add another category but I find communication to be just as important as the others. Architecture is about communicating what the structure and behavior of the system is, and how the structure and behavior meets the goals. That message must be evangelized to the developers and the business people.

Software Architecture is everyone’s business. The developers often don’t want to get involved with abstractions and the business people often don’t want to or can’t understand the partitioning, but the knowledge of that group, with a software architects guidance, can produce the strongest architecture with the least complexity. This is not easy to do and to get a diverse group to agree but it pays off when problems arise - the group owns the architecture.

Architecture: A Competitive Advantage

2011-03-21T21:38:00.001-04:00

I recently attended a seminar with James Utterback, an MIT Sloan professor who wrote the book Design Inspired Innovation. He started the talk by something that was clear to me but I never noticed: that today many companies have access to the same technology but it is the design that distinguishes them and makes their products different. He also points out that great products can enhance our lives and he points to the Applie Ipod, an innovative wheel chair, and a new saddle that protects a horses back and makes the rider more comfortable.

Design and architecture is starting to be seen as a commodity and also collaboration between customers, suppliers, manufacturing, national labs, universities, and other open sources of knowledge. An example of how this works was presented by a speaker of the seminar, the founder of Vigix This company has only a handful of employees and contracted out the design of their product to IDEO, the manufacturing to Flextronic, and had a number of other agreements to other specialty companies. Typically these resources were only available to larger companies, but this knowledge is becoming more open and the expertise available for purchase.

His talk and his papers also emphasized the need for simplicity in design. An example is the Google search interface versus the Yahoo search interface. The Google interface remains amazingly simple and the theory is that it became more useful because of that simplicity (I contend it also had superior search functionality to the other engines and that was a big part of its success). He also talked about how companies make simple products complex over time (I’m a witness and shamefully a contributor to that in the products I’ve worked on).

Systems, he says, are greater than the individual components that make them up. Simple ideas may create new combination of component interaction and may not even require new technology. This is architecture. An example of how a system is greater than the sum of its parts: An Olympic skier who won the overall event but doing well in each of the individual events but not winning any of them.

He defines product architecture as a map in the abstract of subsystems and their interface or connections. Products today must be built modularly so that they can be integrated easily but adapted to individual tastes. This is how product platforms are born. The Black and Decker power tools, built around a singular battery power supply are an example. The book states that the interfaces these days should be open rather than proprietary as it gives the customer more flexibility (although it does leave the firm wide open to the competition).

After attending this talk, I give even more importance to the job of the Software Architect. I also see the trend for more contracting out of this. Perhaps we need a Software-only IDEO. Start-up anyone?

MIT Systems Thinking Webinar

2011-02-20T19:11:00.002-05:00

Please use the attached link to view a playback of the webinar I presented at MIT on Systems Architecture:

Systems Architecture Webinar at MIT

This series is open to the public and they have many intersting topics for architects. The next one is on System Interaction and Complexity Metrics.

Book Review: Universal Principles of Design

2011-01-30T21:17:00.001-05:00

I’ve always thought of Software Design and Software Architecture as different from other design disciplines. After all, building a house is way different then constructing software. And I also read a lot of books on Software Architecture, and in these books, the authors often point out the differences, such as:

- We have only been building software for fifty years but we have been building houses for several thousands of years.

- Software is more malleable and can be modified easier than a building.

- Design patterns for software don’t look much like design patterns in other disciplines.

And yet I have discovered a book about design that applies to software in a big way. It is title Universal Principles of Design, by Lidwell, Holden, and Butler. This book contains 216 design principles, with each described on a few pages or less, with diagrams and quotes from other sources for each principle. The book is brilliant.

Here are some of my favorite principles that I use in my work:

The 80/20 Rule

“A high percentage of effects in any large system are caused by a low percentage of variables”.

There are many examples you probably heard about with this rule. Some listed in the book are:

- 80 percent of the town’s traffic is on 20 percent of its roads

- 80 percent of a company’s revenue comes from 20 percent of its products

And my favorite:

- 80 percent of a product’s usage comes from 20 percent of its features.

What if we could focus our efforts only on that 20 percent of the features? Productivity and success will follow.

Another favorite:

Mental Model

“People understand and interact with systems and environments based on mental representations developed from experience”.

A great example of this given in the book is the interaction with ABS brakes. With a conventional braking system, the drier should pump brakes on a wet slick surface, but with ABS brakes, the car will properly stop and make a turn when slamming or steering while braking. If you are a driving used to conventional brakes, you may not understand this when using ABS brakes.

The book says that designers often have accurate system models but inaccurate user interaction models. Users have accurate user interaction models but inaccurate systems models. Designers need to obtain user interaction models to create designs. They can obtain this by testing and interacting with their designs as users. One of my goals in the next year is to be a tester for all my designs.

And my favorite so far is:

Ockam’s Razor

“Given a choice between functionality equivalent designs, the simplest design should be selected.”

A great example of this is the Google search interface. Nothing could be simpler and it continues as the main interface to the search engine.

Another application is with interfaces. When I design interfaces I strive to make the fewest methods possible. It is a struggle to do this but each method is carefully evaluated.

I plan to read and reflect on a principle a day with this book and will blog on other principles in the future. I feel like I found a treasure cove of ideas for my daily work.

Less Is More

2011-01-15T21:02:00.003-05:00

I discovered this web site completely dedicated to Agile Software Architecture. In fact, when I search the web I can’t find anyone that thinks that Software Architecture is dead or gone with the Agile Software movement. For once, the world agrees with me.

The web site outlines what we would like to achieve with Agile Software Architecture:

Deliver working solutions
Maximize stakeholder value
Find solutions which meet the goals of all stakeholders
Enable the next effort
Manage change and complexity

In my experience the first objective is the most important. I have always wanted to deliver a working solution as soon as possible to test out the ideas of the architecture. In fact, I often work on the implementation before the architecture is completely documented or modeled.

The web site also lays out a set of principles that align with the Agile Manifesto but are specific to architecture:

My favorite of these is #3, Less is More. In my work, I consider the expense of everything. I once had a colleague who was a master requirements writer. She wrote hundreds of requirements for each feature where I would write about a dozen or two for similarly sized features. From my experience, each requirement had a high cost. A test was created for each requirement on each of our product platforms. If that requirement failed, error reports would be generated. If the requirement was truly necessary, then these things (tests, error reports) were indeed necessary. But what I found was when there were too many requirements, very few people would read them and thus their usefulness went downhill.

With documentation and requirements, I like to use advice that Robert C. Martin gave at a seminar I attended: “Rip content out until it hurts!”. After all, you can always put it back in.

Long Projects

2010-12-12T14:41:00.005-05:00

I recently found out that a project I started years ago was just cancelled. I was devastated. How could this happen? We did everything right with this project! It even had a code name that indicated it would be a success.

This was a technology disruptive project. It would change the way we do our work and work with our business partners. It had strong backing from many management tiers. These managers were committed. They knew the perils of supporting a multi-year project like this and also meeting their commitments to deliver a steady stream of new products in the meantime.

From the beginning there were constant challenges because we could not get enough engineers on the project. I was moved off the project in a cascade of reorganizations. But as I watched from afar, many talented engineers were moved on the project.

And yet, I smelled the trouble from afar. The project communications were stellar. I was impressed with the progress. They were creating deliverable systems and getting all the key stakeholders involved. More and more of a working system was delivered. But every time I asked how much time before it would be delivered I was gold “two years”. Another year passed and I asked the questioned again and got “two years”.

That “two years” is kiss of death. In fact “two” of anything is the kiss of death. We have an engineer in another organization who we have been waiting months on for a design. Every month he tells us it will be done in “two weeks”.

After six months we gave up and now we did the design ourselves. It’s not as solid a design because he is an expert in this area but we are learning. We’ll have to build the design expertise ourselves.

Long projects notoriously fail. Here is some research from objectmentor.com:

The project I started on was about four years old.

I recalled when we made the key decision. We were going to do something that was more malleable and could be done slowly over time or something disruptive, which was ultimately chosen. At the time, it appeared the disruptive technology would be more robust and indeed that is still the case. But ultimately, the business could not sustain the investment to make this disruptive change.

What does this all mean for architecture? Architects have to make decisions that the organization can follow through on. I had my doubts seeing our organization not follow through on other disruptive changes. And yet I felt at the time, that this environment was different. In the end, however, business needs usually win. And software architecture may have to work with business demands versus the most elegant architecture.

Design Patterns in Everyday Architecture

2010-11-13T20:59:00.006-05:00

Software Design Patterns were a sensation when this book about them was published. I became aware of this book in 2001 when a team I worked with was using design patterns. The book promoted software design patterns similar to using a cookbook in other engineering disciplines like building architecture or electronics. The authors define a Design Pattern:

A design pattern systematically names, motivates, and explains a general design that addresses a recurring design problem in object-oriented systems. It describes the problem, the solution, when to apply the solution, and its consequences. It also gives implementation hints and examples. The solution is a general arrangement of objects and classes that solve the problem. The solution is customized and implemented to solve the problem in a particular context.

It all made sense. And yet…

I have a confession to make. I haven’t used Software Design Patterns in a very long time. I think I developed an aversion to them. In the last project where we actively used them as a team, we developed a massively BDUF (Big Design Up Front) and the project was an utter failure, ultimately being canceled and many people on the team being laid off. Since then, while I am a proponent of thinking about the project up front, I also know that the team can and will get lost in oodles of documentation. Our team did and ultimately that documentation was of little value to our stakeholders.

Our team used to work like this: we would see a problem that needed to be solved and then look through the Design Patterns book for a solution.

The problem with doing that is this: often the pattern didn’t precisely solve our solution. Since we worked on embedded systems, object oriented solutions often not used to save memory. Our patterns seemed to be a lot simpler then what was presented in the book.

I found a lot of criticism of design patterns on the internet. One person felt that design patterns only existed because of language restrictions and that in languages like Perl, the patterns are built in and ready for use. (This poor guy got a ton of backlash for not believing in Design Patterns).

The biggest problem I had with design patterns is that the ones I wanted to use weren’t in the book. That said there is an effort to create a larger repository of design patterns. Since I’m working mostly with XML for my work, I plan to explore this XML repository.

My experience with Design Patterns left me running away from them. While they have some obvious detractions, it’s worth it to look at them again and see if I can find a use for them in my daily architecture work, where decisions are not usually made up front but on a daily basis.

Software Architects: The Best Job

2010-10-16T19:57:00.002-04:00

I was very happy to see this article on CNN’s Money web site: http://money.cnn.com/magazines/moneymag/bestjobs/2010/snapshots/1.html

This article ranks the Software Architect as the #1 ranked job in America!

I couldn’t agree more with the ranking of course. Like the person profiled, I work on business strategy some days and other days I’m deep in code. And I’m everywhere in between.

There is a projected 34% job growth in this area over the next ten years.

Agile Processes – So What’s the Architecture?

2010-10-06T21:35:00.004-04:00

About seven years ago I attended a Best Practices conference for Software Development. The seminars were on RUP (Rational Unified Process ) and CMMI (Capabilities Maturity Model ). At that time, I was working with my own company’s internal product release processes that were waterfall http://en.wikipedia.org/wiki/Waterfall_model in nature. There were a handful of seminars on agile development.

Fast forward two years: I’m attending the same Best Practices conference। And the seminars couldn’t be more different. Agile development was the subject of the vast majority of topics presented with titles like: Agile Development and XXX (you fill in the XXX). There were lively roundtables with book authors on agile. One poor fellow asked skeptical questions about agile methods and was roasted by the roundtable leaders.

And one seminar that stood out but itself because Agile was not in the title: Documenting Software Architectures with one of the authors of the same titled book। This seminar was packed and one of the most well attended of the conference.

But don’t software architecture methods and agile methods directly conflict with each other? Don’t they have different values?

Being a software architect and systems engineer at my current company, I have observed and been part of BDUF (Big Design Up Front) and features that are YAGNI (You Ain’t Gonna Need It)। I’m guessing that poor fellow that was roasted in the roundtable was also part of this effort. And yet those efforts sometimes have some incredible forethought. The foundation was laid down to allow for robust change in the future. As someone who has worked with existing architectures, I have great respect for the foresight the earlier designers had when that serendipity occurs “Wow, this is already in place and all we have to do is….”.

During a business trip with the SDM at MIT, we were visiting a software company in Dublin, Ireland that was an early user of agile methods in 2001. One of the students asked the lead software engineer how they will deal with legacy issues given the way they were using agile. The lead designer didn’t know and thought it would be a problem in the future.
Fast forward and it’s 2010। My company has rolled out agile methods and we’ve all been using them for several years now. We show improvements in productivity and reliability. And what has happened to architecture? It’s more important than ever.

Like most companies, our software systems are large and complex। An overall framework is needed at the beginning of a project and decisions that are the hardest to change during the lifecycle of the project are made.

Many architectural decisions are made during the project and the key for the architect is to communicate those changes to all stakeholders. See my previous blog on communication tools for architectural decisions.

At that conference, Agile zealots accused architects of being software dinosaurs and architects accused Agile zealots of being cowboys। I don’t really see the contradiction in the two. I’ve come from using CMMI to using Agile. Both are disciplined processes and Agile is more disciplined and constrained with the focus on the final product. Complex systems have architecture whether it is done iteravely or not. It just smart to make some of the overarching decisions before coding begins. The architect’s role continues as someone who manages key decisions and tests the code to make sure it is executing those decisions.

When those decisions are not made up front, it can be disastrous for the project (at least some that I’ve been involved in)।

I wonder there are any software developers that don’t use Agile method? Maybe not. And I also wonder if there is another better process in our future. Regardless, classes like Documenting Software Architectures will always be interesting because we all need to know “What’s the architecture?” Architecture occurs whether you have someone working on it or not. It’s better to be aware up front what it is and how it meets the business requirements of the project.

Systems Tools: SysML

2010-09-12T22:09:00.002-04:00

Systems Engineering is about specifying functionality, where that functionality resides in the system, and there what interfaces are required – is by definition a complex activity that involves an array of players and a multitude of considerations. As if the pressure of efficiently and accurately developing systems in the face of tight budgets isn’t enough, competitive considerations are forcing companies to develop increasingly sophisticated systems faster.

(Systems Engineering Best Practices White Paper, IBM/Rational Software, December 2009, by Hoffman, Sibbald, and Chard)

Where I work, systems engineers are also system architects. They, with the rest of the development team, make day to day decisions that evolve software architecture. I can’t speak for other professions, but the disciplines are deeply entwined in the world of software.

One tool that I’ve been looking at adopting is based on an extension to the Unified Modeling Language (UML) called SysML. SysML has views that deal with multiple aspects of the system – functional and behavioral, structural, performance, and slew of other models like cost and safety.

INCOSE http://www.uml-sysml.org/documentation/sysml-tutorial-incose-2.2mo (International Council on System Engineering) defines SysML as:

A graphical modeling language in response to the UML for Systems Engineering RFP developed by the OMG, INCOSE, and AP233a UML Profile that represents a subset of UML 2 with extensions
Supports the specification, analysis, design, verification, and validation of systems that include hardware, software, data, personnel, procedures, and facilities
Supports model and data interchange via XML Metadata Interchange (XMI®) and the evolving AP233 standard (in-process)

The four key diagrams of SysML structure (similar to a class diagram in UML), requirements (new to SysML), behavior (uses standard UML behavior diagrams), and parametrics (which are used to express system constraints).

Requirements can be mapped to structure and behavior as parametrics can be. Moreover, models can be customized by extending SysML with mechanisms called stereotypes.

I plan to use SysML in the near future and report back buy my success with model driven development has been poor. I’m not sure if I and the teams I work with lack the discipline and technical expertise to generate such a complex model of a system or that it simply doesn’t work.

Have any of you in the blogosphere used SysML?

IEEE 1471 – What’s required for Software Architecture?

2010-09-08T22:05:00.004-04:00

I recently read through the IEEE Std 1471-2000. In one of the last sections it gives an example of what is needed for an architectural description (an AD) to meet the standard’s requirements. A large group of industry representatives worked on this standard for several years and this standard was approved in 2000.

The first section is the basic information about the architecture. This includes the data of issue, the organization issuing the architecture, change history, a summary, the scope and context, a glossary and references. This is standard for the technical documents we write in my organization.

The second section must identify “architecturally relevant stakeholders”. This would include users, acquirers, developers, and maintainers (at a minimum). With this section, the concerns of the architecture are also included: the purpose, the appropriateness of the system in filling the purpose, the feasibility of constructing a system, the risks to all the stakeholders, and the general maintainability of the system. What concerns are most important to each of the stakeholders?

An example of this section would be a stakeholder that takes the software system being built and uses it in another system. This stakeholder is interested in the interfaces to the system and not much the internal design. But the developer /stakeholder is very interested in the internal design.

The third section describes viewpoints which are required. The specific views for an architecture are not specified but the standard says the views must be “selected, defined, and analyzed for coverage and that the rationale be given for their selection”. Each viewpoint must have a name, the stakeholder interested in that view, the concerns addressed, the modeling technique or method used to construct the view, the source information (like author, date, etc.).

The viewpoints should cover the stakeholders concerns. Note my blog entry where I lay out the minimum viewpoints I need to define an architecture.

The fourth section lays out more requirements about views. Each view will be only one viewpoint. Hmmm…this seems to favor Unified Modeling Language (UML). Check out Object Process Methodology which has one diagram with many views.

The next section requires that the views must be consistent with each other. This can be hard to do in practice if your tools don’t do this for you. Peer reviews can also help here.

Section six specifies that the architectural rationale must be included. This seems most important in evolving systems. See my blog post on recording architectural decisions – this is a practice we do in my organization on a daily basis.

Reading this specification reminded me of a time long ago when I was asked to come up with an architecture of a very large system. I came up with several views and assembled of team of cross functional designers from across the system. I spent months working on the diagrams trying to get agreement. I could not get convergence – even on what views to include. If I had this standard in hand at the time, it might have given me the framework to work on defining the views better in relation to the stakeholders.

I recommend reading the specification – it’s short.

Ch-Ch-Ch-Changes: Software Interface and Software System Dynamics

2010-08-29T21:20:00.009-04:00

When a software system is created, there are typically interfaces to that system so that third party developers can write programs that use the functionality of that system. These external interfaces, once released, must be supported practically forever. Once a third party developer uses them, we the system designers, must make sure any changes we make to this interface be both forward and backward compatible.

Forward and backward compatibility put considerable constraints on the interface designer, especially when changes are to be made. The definition is:

Forward compatibility or upward compatibility (sometimes confused with extensibility) is the ability of a system to gracefully accept input intended for later versions of itself.

If we use the same sentence structure, we can define backward compatibility as:

Backward compatibility or downward compatibility is the ability of a system to gracefully accept input intended for earlier versions of itself.

As an example, think of some of a software library like the Boost C++ Libraries. If a change needs to be made in an interface, it must be done very carefully to ensure at least backward compatibility. If a method must be deprecated, it must not be done immediately. The method is supported but it is documented that support for this method will be removed in the future. And it is often years before that happens.

Much of my work is on existing systems so I have the difficult task of making sure my changes to external interfaces are both forward and backward compatible. This is difficult to do intellectually and I often miss things that break compatibility. I’ve built up what our team has learned into a set of standards we follow to assure that we can keep this compatibility.

But our interfaces are getting more complex and we are still missing things.
This is causing an unfortunate dynamic on our team. The first behavior is tremendous fear. Team members fear that what they release will not be forward and backward compatible so they get in the mode of carefully trying to find every problem imaginable.

Here’s a systems dynamics model of our current problem. The models were developed using Vensim and there is a free trial download of the software at their website.

The first picture is of a simple model. Note that the development rate is constant if the rate of problems found equals the rate of problems fixed.

Note: The graphed results of a simulation of this model are shown in the picture of the model (see the flat rate of software productivity and the rate of problems found and fixed).

Now we’ll add the fear factor to the model. When fear occurs, more problems, whether real or imaginary, are discovered. The software development rate goes down.

When this occurs, schedule pressure starts occurring (note that this dynamic is not shown in the model). Schedule pressure creates more fear which causes more problems to found, real and imaginary, both of which need to be analyzed. Schedule pressure might create an increase in productivity (“you must fix more bugs – please work as much as you can to do this!”). But it can’t match the fear factor and the software development rate decreases even more.

My theory is that if we create a more supportive team environment, we can counter the fear and get our development rate back up. The team will have the following operating conditions:
1. All problems raised are considered and accepted respectively
2. Problems are classified and analyzed by a cross disciplinary team (people with variety of expertise) and decision is made – do we do this change or not
a. This should decrease the churn we have from changes – we probably won’t do as many changes as we have in the past.
3. This team will also recommend the best fix for the problem so the rate of problems fixed will increase.
Voila – software productivity is back up and even increases.

I’m trying this experiment on our team and will report back the results in a few months. I feel confident that this supportive team model will change our dynamics.

Architectural Decisions – Accidental or On Purpose?

2010-08-25T20:56:00.014-04:00

“Every interesting software-intensive system has an architecture. While some of these architectures are intentional, most appear to be accidental.” - Grady Booch (http://www.informit.com/articles/article.aspx?p=471929 )

Last week I wrote about what software architecture is and that I rely on a software architecture consisting of goals which map to structure and behavior. But in my everyday work in the field of software design, I rarely get the luxury of creating such a complete mapping.

What typically happens is that I join a project that already has had the vision established and may not have any artifacts explaining that vision. I have to do some archeology in the code to discover what the original vision was. And more often than not, there are several conflicting visions in place.
Typically where I work, we make design and architecture decisions on a daily basis. And as we iterate through our work and develop code and test our solutions, we find changes we need to make. To make sure these decisions are accidental and that they merge with our team’s vision of our architecture, we meet weekly to review changes. We often argue.

Part of my job as a software architect is to examine and analyze various points of view from team. There are two tools I use that are invaluable to getting the team to converge on a solution (and sometimes convergence is not possible, and a decision has to be made, but in the majority of cases, convergence does occur).

The first tool is a decision matrix. This is from the book “Getting Started in Project Management” by Martin and Tate (http://www.amazon.com/Getting-Started-Project-Management/dp/0471135038 ) - double click the image to get a close up:

This works well but is very quantitative and the human nuances of decision making are often lost in this format. We also sometimes break the rule above “Do not change the numbers to affect the selection of a “favored” solution.”

My favorite format is to use a Mind Map (http://en.wikipedia.org/wiki/Mind_map). And the tool I use is Mind Manager (http://www.mindjet.com/ ) but there are free tools available for download and templates available for Visio.
Here are two the templates I work with:

I just noticed that I have hundreds, yes, hundreds of these maps in place. We refer back to them when we forget why we made a certain decision and when someone new to the team questions the decision. Sometimes we re-evaluate our decision and use our map as a launch point to see why our thinking as changed.

In the article Booch says:

"Thus, having an accidental architecture is not necessarily a bad thing, as long as
• the decisions that make up that architecture are made manifest and
• the essential ones are made visible as soon as they are instituted and then are allowed to remain visible throughout the meaningful life of that system."

By using these tools you can achieve, and thrive, with an accidental architecture. Let me know if these tools are useful to you or what tools you use.

Software Architecture Definition?

2010-08-18T21:05:00.003-04:00

When I was a student in the MIT Systems Design and Management Program, I had to take a course titled “Systems Architecture”. The course was about overall systems and not specifically on software. In that course, there was a paper I read that forever shaped my view on what software architecture should be. The paper is titled “A Taxomony of Decomposition Strategies Based on Structures, Behaviors, and Goals” by Phillip J. Koopman (DE-Vol. 83, 1995 Design Engineering Technical Conferences, Volume 2, ASME 1995). In this paper, Koopman describes architecture as containing structures, behaviors, and goals along with a variety of decomposition strategies. In my own work in software architecture, I find that I must tie my structure and behavior specifications to business goals and when I don’t have all three pieces, my architecture often doesn’t withstand the rigors of peer review from a wide audience.

How does this definition of architecture components stand up to other definitions? The standard on software architecture - IEEE 1471-2000 - says
Software architecture is the fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution.

The standard defines the following bold terms:

A system is a collection of components organized to accomplish a specific function or set of functions. The term system encompasses individual applications, systems in the traditional sense, subsystems, systems of systems, product lines, product families, whole enterprises, and other aggregations of interest. A system exists to fulfill one or more missions in its environment. [IEEE 1471]

The environment, or context, determines the setting and circumstances of developmental, operational, political, and other influences upon that system. [IEEE 1471]

A mission is a use or operation for which a system is intended by one or more stakeholders to meet some set of objectives. [IEEE 1471]

A stakeholder is an individual, team, or organization (or classes thereof) with interests in, or concerns relative to, a system. [IEEE 1471]

This definitions correlates to Koopmans article in the following ways: the goals of the architecture are the mission; the behavior maps to the relationships and the principles, and the structure maps to the fundamental organization.

In practice I see many designs that show mapping and decomposition with behavior and structure (think UML class diagrams and sequence diagrams) or a mapping between goals and behavior (UML Use Cases and Sequence Diagrams) but very architecture artifacts that successfully map all three.

I will be looking at other definitions of architecture in future blog posts.

Polymorphism: A Place in Systems Engineering?

2010-07-25T19:15:00.000-04:00

This article originally appeared in the Fall 2009 edition of MIT SDM Pulse newsletter.

Polymorphism: A Place in Systems Engineering?

As a software architect and a systems engineer, I’ve been working with larger systems that include software, hardware, and mechanical subsystems. I work on designing software and systems for a multifunction device that scans, prints, and faxes documents. In my work, I often try to translate principles of software design to larger systems.

Once such principle is polymorphism. How does polymorphism work in systems engineering? A key principle in software engineering is that software objects should have only one function; polymorphism allows for abstract software objects to have more than one function (more on the meaning of abstract below). In the non-software world, some objects-that-you –can-hold-in-your-hands can also have more than one function. System engineers often want to fit many functions into a single piece of hardware and polymorphism is a pattern embraced by system designers.

So what the heck is polymorphism? If you are not a software engineer, you may not be familiar with the term but you are probably familiar with the principle. The basic idea is from a Greek root meaning “having multiple forms”. In object-oriented programming, polymorphism is the ability to execute a function specific to a context. The classic text-book example is with shape objects. A Shape is a general object and a Circle and Rectangle are more specific objects that inherit from a Shape. A Circle is a Shape and a Rectangle is a Shape.

In C++ here is a partial definition of a Shape object. The “virtual” function that is set to “0” means this class is an Abstract Base Class or an ABC. This object can never be instantiated or created in executable code but exists for other objects to inherit from.

class Shape {

public:

Shape();

virtual void draw()=0;

};

Now let’s define a rectangle and a circle class. These objects inherit from the Shape class. Inheritance is an “is a” relationship. That is, a Rectangle “is a” Shape and a Circle “is a” Shape.

class Rectangle: public Shape {

public:

   Rectangle(int width, int height);

   void draw();

private:

   int width;

   int height;

};

class Circle: public Shape {

public:

   Circle(int radius);

   void draw();

private:

   int radius;

};

The Rectangle and the Circle object would have their own implementations or code for the draw function. Each shape has its own formula or algorithm for drawing their shapes.

So why is this useful? In a program, the Shape object can be used to draw any shape without knowing what exact shape it is at compile time. For example, the line of code

Shape->draw()

will draw correctly (with some help from the compiler). This is why C++ is a polymorphic language. You don’t need to write this kind of code:

if (Circle) then

draw a circle

else if (Rectangle) then

draw a rectangle

So this is useful with software – less code and it is easier to understand the abstraction. How is this useful in larger systems? Polymorphism can work with a vehicle that can fly, drive over roads, and float in the water. Imagine that you the driver, could drive this vehicle in the same way without know how the vehicle when over these different mediums. You just knew how to drive and it worked the same way no matter where you were. That’s polymorphism and that’s useful to the customers of my product – which is the machine that has multiple functions.

Another example of polymorphism is from my days at MIT in the SDM program. In systems engineering one day, we talked about a seat cushion in a boat. This seat cushion serves as two functions – as padding for a seat in one context and as a floatation safety device in another (this cushion has straps on it). The design of this seat cushion has been influenced by the fact that it has two functions. However it’s basic core function or Abstract Base Class is to lift a human – whether it be on a seat or in the water.

Here’s an example of how polymorphism can save Unit Manufacturing Cost (UMC). Let’s return the vehicle example. If you want a vehicle that drivers over land and floats in water you could buy two vehicles, one that runs over land and another that floats and moves in water. Or us engineers could design you one vehicle that does both. [i]

The multipurpose vehicle is more expensive than a single purpose vehicle and should be less expensive than two vehicles (one for land, one for water). That is the challenge with polymorphism. If our customers did indeed want a multi-function device and if we could reuse systems to perform multiple functions in a cost effective manner, then polymorphism works for our business.

Polymorphism has a role in product platform design. For example, a platform has a set of common functionality. How that functionality is implemented will vary on what system the platform is implemented on. Another example from my Product Design and Development class is a power tool which had a product platform of a power supply. This power supply fit onto a variety of power tools and could be switched between power tools. Like the shape and the draw function, the basic functionality of obtaining power to do work is the “base class”. The function of doing the work varies from power tool to power tool.

But polymorphism isn’t always the best idea. In product design, designing hardware to have multiple functions often has tradeoffs. For example, if you had to design a seat cushion that did not have to be flotation device, you would not be as constrained as when you had to design it as two things. Here’s where software and hardware design differs. Notice in the shape example, the Shape is a distinct object and the Circle and Square objects are distinct. Each one serves a purpose. There is no tradeoffs with polymorphism as far as the function of the object goes (there is memory cost). The Shape object is never actually created in software, but used as an abstract reference. During run time of the code, the shape object will actually always be either a Circle or a Square. Each object is distinct with only one purpose. With objects-you-can-hold, the object is two things at once. For example, the seat cushion is something that provides padding and floats a person in water. The tradeoffs are in the physical object, where in software, there is no tradeoff in the objects themselves.

Polymorphism does have an expense associated with it and should be used only when you need different functions in different contexts. The base functionality should be the same in both contexts – to draw a Shape, to drive a vehicle, to life up a person, to power a tool for work. Polymorphism is a powerful design pattern, popular is software, that can be used in larger systems to build in flexibility in product design and one that will be increasingly used in our design push to build smaller, cheaper, better products for our customers.

[i] Incidentally, such a vehicle was designed and built during WWII and many of them are still in use today with Boston Duck Tours: http://www.bostonducktours.com/duck_history_main.html