chris horn's blog

A VP friend at major global corporation – let me call him K-san, rather than give his complete name and afflliation - put it to me thus last week: “The fundamental issue that we have with SOA is skills shortage. With CORBA, it was much easier: there was just one, relatively cohesive, family of standards, and we could recruit to these. In the good days of J2EE, equally so. Now today, SOA is just too ill-defined, and a common set of skills are just not there in the market”.

Wow. Despite all the excitement, all the momentum, all the vendor investment in Service Oriented Architectures (SOA), we in the enterprise software industry are failing to produce an adequate talent pool for our customers. SOA is an architectural style in which complex enterprise systems are composed from business logic components, structured as loosely coupled software services. Because SOA is an architectural style, rather than a specific technology, some (many ?) apparently find it difficult to make a commitment to SOA, and to find staff.

Why is SOA relevant ? One frequently cited justification is – at long last – a common basis for dialogue between business executives and IT specialists. The fundamental building block for SOA is a specific business function, usually encapsulated as a software service. A business function may also be a step purely carried out by humans, with no direct machine intervention. Business processes can be choreographed as appropriate sets of interacting (business level) software services, and human steps if necessary.

A key business value which results from SOA is encapsulating, or wrapping, each business logic component as a (usually, software) service. Other parts of the system can then make use of each such service, regardless of the physical location (in-house or off-premises) or specific technology used to implement that particular service. In turn, a business judgment that a service could be profitably re-located or re-implemented can be executed upon, without a cascade of changes being required on those other components which use that service.

A second, perhaps less obvious, business value is the reduction in effort to implement new business services. This can arise both because business components can be choreographed to work together in concert, and new routines can then be quickly defined to provide new services; and also because new business components can be developed and added into the portfolio of available software services, re-using where appropriate some of the functionality already available elsewhere in the portfolio.

From my own personal experience, the epitome of a practical enterprise scale SOA initiative is that undertaken by Credit Suisse First Boston, built on products from IONA and other vendors, starting in the late 1990s under the leadership of Stefan Murer and Martin Prater. I am sure there are others. The CSFB experience is well summarized in online presentations by Hermann Schlamann and Keith Tice. In some of their SOA metrics – see the presentations - CSFB state they have over 900 business logic components implemented as software services, used by over 150 composite enterprise applications, with an average of 3.7 services used per client application, 43% re-use and development savings of over 12,000 mandays. I understand that more recent figures show even high re-use figures and savings.

But back to my VP friend K-san. In the good old days, you could use CORBA to do SOA – and CSFB did do so. Today, there is confusion about SOA technologies and standards, yielding a perceived show stopper: lack of skilled staff.

I take it as a given that staff are professional: everybody does the best job they can, given their education, training, abilities and knowledge. My own view is that it is hardly the fault of the people: it is the fault of the process. What is it about our approach to SOA that makes it difficult to find people ? How can we change our thinking about SOA so that it becomes tractable, pragmatic, and harder for corporations to make expensive mistakes ?

In chatting to Larry Alston about this, he reminded me of the “people to process” transition made by Henry Ford. Henry Ford did not invent the automobile: the problem was that up until Henry arrived on the scene, cars were being hand-crafted by extraordinarily skilled, and expensive, workers. Henry Ford focused on process: he empowered ordinary mortals to reliably build cars by creating the correct processes, and paid them high wages to do so.

Today’s manufacturing processes are derivative of the innovation of Lean Manufacturing. As K-san effortlessly reminds me, Lean is of course associated with Toyota, and specifically the founder Sakichi Toyoda, his son Kiichiro, the external consultant Shigeo Shingo; the engineer Ohno Taiichi, and the HR executive Isao Kato. The story goes that, in the 1950s, a delegation from Toyota visited the Ford manufacturing plants in Michigan. Although Ford was a global industry leader at the time, the Toyota people were apparently not particularly impressed by what they saw at Ford. In particular they observed large quantities of inventory sitting idle, and noted that the production across the entire plant did not flow smoothly, with staff occasionally just waiting around.

By contrast, during their visit they happened to be shopping in a local supermarket, and observed the simple idea of an automatic drink re-supplier: once a customer takes a drink, another is automatically ordered and replaces it. The supermarket deferred re-ordering and re-stocking goods until customers made purchases.

On return to Japan, Toyota further developed the basic concepts of what is now known as the Toyota Production System, which greatly reduced lead-time and wastage, whilst simultaneously improving quality. The basic ideas of “Lean Manufacturing” were subsequently taken into Toyota’s R&D and design shops, yielding “Lean Design”. Today, Toyota use the tagline ( at least here in Ireland, where we have no car manufacturing of our own and thus import global brands): “The best built cars in the World.”

How do we turn ordinary mere mortals in the software industry into powerfully productive professionals for SOA ? Is it just a matter of education, training, abilities and knowledge ? What about process ? What can we in the software industry learn from what has gone before ?

Lean thinking has had a dramatic impact on the global software community. Significant momentum in Lean Software Development and Lean Product Development developed back in 2003 when Mary and Tom Poppendieck published their excellent book on the topic. The Poppendieck’s studied the lessons of Lean Manufacturing, and particularly Lean Design, and applied it to software development. Elimination of waste, delayed decision making, fast execution, integral integrity, learning amplification, team empowerment are the key factors which they identified in driving a holistic view of the process.

Agile software development is related, but had a different starting point. Rather than a heritage in manufacturing systems, Agile has its nemesis in software development projects and by developers themselves. It is particularly associated with Kent Beck, Ron Jeffries, and Ward Cunningham (now at eclipse.org and a committer to the Eclipse Spaces project which I mentioned last week), during their work on development of a payroll system for Chrysler in 1996, leading to the Extreme Programming (XP) movement. However Wikipedia also notes that the "practice of test-first development, planning and writing tests before each micro-increment" was used as far back as NASA's Project Mercury, in the early 1960s.

Thad Sheer provides an excellent explanation of the differences in philosophy between Lean and Agile approaches in his 2005 blog post. Lean projects need not necessarily be Agile, and vice versa. In summary, he suggests that Lean usually gets more initial corporate support than Agile since it uses concepts to which most executives can relate, rather than terminology more appropriate to software developers. He also states: “There’s a lot more bang for the buck vis-à-vis the bottom line with a partially completed Lean initiative than a partially completed Agile initiative. Once an organization is LEAN, becoming AGILE is a step that developers can take on their own without requiring much management backing.”

In IONA, we started looking seriously at Extreme Programming back in about 2002, when we hosted Kent Beck in person several times to meet with our software development organization. I personally recall that Ciaran Dynes drove much of the initiative within IONA, as one of our engineering managers, and I am grateful for his leadership. Ciaran went on to build our Beijing based development centre – using XP - and is currently our Artix product manager, based out of Dublin.

So Chris, what about SOA, and do Lean and/or Agile apply ? Well, interestingly there is an analysis published by IBM which in its first part summarises various approaches to both Lean and Agile software development; and in its second part discusses that these approaches are not incompatible to SOA. The take-away is “Oil and Water do mix!”

Ooof. Personally, I find such a take-away divisive: it implies that the world of SOA and the world of Lean/Agile are fundamentally confrontational, but that perhaps with sufficient management effort they can probably be brought together. To be fair, the folks at IBM say they were responding to a previous article suggesting that SOA and Lean/Agile actually are incompatible!. All of this is hardly comforting to corporate strategists thinking about SOA adoption, and I suspect would not be greeted with relief by my VP friend K-san.

I think I have a differing view. I believe that a successful SOA results from Lean thinking. I discuss this in my next posting.