Forensic Architecture and other lessons from SOA land.

Forensic Architecture! What is it? I use this term to refer to the process of describing the architecture of something after it has been built. It is a largely frustrating, cumbersome process yet I seem to have made a career out of it. Let me explain.

Note: These are my own views and lessons I have learned. I have greatly enjoyed working with many talented minds in the past and am looking forward to working with more in the future. The sum of all these experiences has made me what I am today for better of worse and I am grateful to everyone I have met on this journey.

In a perfect world of the idealist software engineer, software architecture is written in response to a documented set of requirements. The sequence of events in a perfect world usually involves the following steps:

1. A problem is identified.
2. The problem is documented.
3. A solution is proposed.
4. Specific requirements for the solution are documented.
5. An architecture is proposed to solve the problem.
6. An iterative process of mapping requirements to solutions takes place until all issues are resolved.
7. The final solution is built and solves the problem.

These are somewhat generic and there are many variations to this sequence of events.

Lesson #1: In the real world, this process rarely happens without some complications. The following sections discuss some real-world projects I have learned from and their associated complications. Each section briefly explains the project and then provides an introspective analysis of the architecture process and the lessons learned during that process. The purpose of these case studies is to illustrate how vastly different architecture processes can be and to try and learn from the past.

Example #1: ebXML

ebXML, which stands for Electronic Business using eXtensible Markup Language, is a modular suite of specifications that enables enterprises of any size and in any geographical location to conduct business over the Internet. The overall standards process for developing ebXML (circa 1999–2001) was somewhat flawed and marked a serious departure from the architectural idealism described above, yet the product itself was rather revolutionary and advanced for the time. ebXML is in my mind the first large scale web and XML based Service Oriented Architecture (SOA). The organization and overall process used to create the ebXML specification was the root cause of the problem. No one is at fault as there were external pressures to get the work done fast and we all agreed, myself included, to adhere to the schedule we created. In retrospect, there were great lessons to be learned.

The following six ebXML working groups, each with a very specific job to do, started work at exactly the same time and were supervised by an organizing committee:

• Requirements
• Technical Architecture
• Business Process Specification Schema
• Core Components
• Messaging
• Registry

Lesson #2: In an ideal world, the Requirements group would have completed its work before any other group started or perhaps before those groups were formed.

Lesson #3: In the same idealistic manner, the Technical Architecture group would have started and completed its work based on the requirements, and then a series of other groups would have been formed to complete the balance of the work. The groups should have logically been formed in alignment with the components of the architecture.

As stated above, these two ideal practices probably never happen in the real world ;-)

Instead, all six groups started work at the same time with mostly aligned but somewhat differing ideas of what ebXML should be, given that no comprehensive documentation existed prior to this activity. Although this sounds like a very funny Dilbert comic, it actually happened.

Eventually, most of the work fit within the work completed by the Technical Architecture group; however, there were gaps and issues in the solution. The unfulfilled requirements led a lot of follow-on work, including to the formation ofthe Universal Business Language (UBL), and several web services standards that evolved out of the ebXML model. The entire experiment was similar to the Monty Python skit, “The 100-Yard Race for People with No Sense of Direction.” At the start of this skit, several racers are standing in a straight line, ready to sprint down a racetrack. When the starter pulls the trigger, each competitor runs as fast as he can in completely different directions, each eagerly trying to complete his race.

Other Lessons learned

1. Never try to document a moving target in a static architecture specification.
2. Something not documented in architecture is not universally understood, nor can it be implied to exist within the scope.
3. Alignment, understanding, and agreement on a specific model or first order of logic are a good way to get any project off to a good start.
4. Solving a problem before you know what the problem is can be suboptimal.
5. Splitting a large thing into several smaller things is a good approach.
6. Keeping unnecessary dependencies out of a design of multiple components is generally a good thing for longevity.

ebXML was a ground-breaking effort that at one time enjoyed participation from most major software vendors. It was a joint initiative of the United Nations Centre for Facilitation of Commerce, Electronic Business and Trade (a subgroup of the United Nations Economic Commission for Europe) and OASIS. ebXML was the first post-XML, post-Internet effort to build a truly global electronic business and commerce infrastructure based on open standards and protocols.

Although the project eventually lost support from most major software vendors, a lot of its functionality and patterns live on today within something known as web services. Web services, although itself a term that does not concretely define a set collection of specifications, is generally accepted as one of the future building blocks for advanced information systems connected to the Internet.

Example #2: W3C Web Services Architecture Working Group

Post-ebXML, several skilled architects went on to create a piece of work called the W3C Web Services Architecture. During the effort, there were disparities in terms of the scope of the W3C Web Services Architecture recommendation and its recommended level of abstraction. The result was a hybrid document that is both part abstract model and part reference architecture, while also mentioning certain implementation constraints. During its development, there was also no clear definition of web service, and the web services versus Representational State Transfer (REST) debate came up many times.

The W3C Web Services Architecture Working Group was working on many complex interdependent issues concurrently, which in retrospect was not an ideal process. Some people were examining very concrete technologies such as the Simple Object Access Protocol (SOAP) at the same time the requirements document was declared to be a “living” (dynamic) document. Other disparities included describing some architectural concepts quite abstractly in the concepts and relationships section, followed by very specific mentions of technologies such as SOAP and HTML (albeit without versions).

Although this layered approach is somewhat normal, the content of the final recommendations seemed to have large gaps in terms of concepts and their link to concrete entities.

Lessons learned

1. Try to create consensus on what is achievable before starting any architecture work.
2. Clearly define the rules and constraints for abstraction and what constitutes a normative part of the solution before creating a table of contents.
3. Consider splitting knowledge into multiple sets of artifacts, each at a different level of abstraction, to capture knowledge for different audiences. For example, architects and modelers require different levels of detail than those writing code.

Example #3: The United Nations (UN/CEFACT) eBusiness Architecture

Another architecture that is worth studying is the eBusiness Architecture specification for the United Nations (UN/CEFACT) group. This modern SOA work tried to reconcile the ebXML and web services standards into a single architecture specification, and tried to add a modeling methodology on top. The Techniques and Methodologies Working Group managed to cobble together a candidate architecture specification for global electronic commerce that reflected the thinking at the time.

The final work produced by the working group was focused far too much on a narrow audience, namely those who model processes of international trade using the Unified Modeling Language (UML heads for short, of which I am one). It also tried to reconcile a large and very diverse set of information into a single, short document. As a result, significant portions of content were omitted, which made the document very confusing to readers who did not have the background knowledge to correctly interpret it. Although it was also an early example of a large infrastructure or IT system built in alignment with the principles of SOA, the document failed to produce a level of detail that was sufficient enough to facilitate implementations, and instead was largely used by authors of other specifications as a common basis for their work.

Lessons learned

1. Clearly define the audience for each artifact in advance.
2. Do not mix together disparate views of architecture unless absolutely necessary.
3. A layered approach to documenting the structure of a system is sometimes easier to understand than one large view that tries to incorporate multiple domains of knowledge.

Example #4: The OASIS Reference Model for SOA

The OASIS Service Oriented Architecture Reference Model Technical Committee (say it ten times really fast) formed in early 2005. This group included more than 200 members and observers to define SOA as an architectural paradigm, starting with a diverse set of beliefs and a blank piece of paper. At the time, SOA was an unspecified concept and each person had a slightly different view of what it was. Over the course of 18 months, the Technical Committee managed to distill the common beliefs of the participants and the public and build a reference model to capture the core lexicon of SOA.

In the process, several opinions were collected, and although they differed vastly at the concrete level, they were generally aligned at some abstract level. During the process, it became common for the architects involved to look beyond the examples cited and to recognize concepts and axioms at a higher level. It was a very iterative process, yet very productive in the end. Although some vendors attempted to define SOA in alignment with their own products or offerings, they were unsuccessful. The final delivered specification was a reference model, completely abstract of any implementation, and therefore reusable in multiple domains. This was deemed a success by most participants however it too had its share of complications due to the sheer number of disparate opinions of what SOA was. Additionally, since the term SOA itself was such a hot topic in the industry, there was a large degree of political pressure from some interests who had already heavily invested in their own definitions of SOA.

Lessons learned

1. Creating any model or architecture specification generally requires a means to find a common ground.
2. Politics places pressure on documenting architecture. It is best to think freely from forces and constraints that may lead to decisions based on a preconceived notion of what the solution is.
3. Meetings of standards bodies involving large numbers of software architects seem to be rife with repeated consumption of fermented vegetable product beverages containing high levels of alcohol and caffeine.

The latter should generally be regarded as the best lesson learned and I am grateful for all the people I have worked with on all of these projects, no exceptions.

I keep these lessons in mind as I am embarking on yet another forensic architecture journey to write a book on Web 2.0 for O'Reilly Media with Web 2.0 luminaries Tim O'Reilly, Simon St. Laurent, James Governor and Dion Hinchcliffe with great help from Steve Weiss. In accordance with the above lessons, we have tried to avoid introducing any dependencies or material in the book that would affect its durability or lessen its utility in the future. By doing that, hopefully we will not exclude anyone, nor will we arrogantly make our definition of Web 2.0 the only definition that applies.




By documenting architectural, design, and social patterns, this book is part of a framework that may define some aspects of Web 2.0 and share knowledge with others. To be durable, the patterns in this book are not tied directly to any specific implementation or work. In some cases, we mention well-known implementations for illustrative purposes only.

Where does that leave us (as a community) with respect to sharing knowledge about Web 2.0? It provides the groundwork for a methodology to derive knowledge from examples. This is the stepping-off point where we can start to truly evaluate the distinctions of Web 2.0 as it is, compared with the Internet of today and tomorrow.

Hey ho - let's go!