chris horn's blog

The Times Higher Education/QS World University Rankings, published in last October, showed Trinity College Dublin breaking into the global top 50 universities, ranked at 49 - up from 53. TCD became the first Irish third-level institution to make it into the elite global top-50 group. The rankings also delivered very good news for University College Dublin (UCD), now ranked 108 worldwide, and up from 177 last year.

Last wednesday, both universities announced an intention to further enhance their world class positioning, by integrating together their fourth level (ie postgraduate) activities, and merging the activities of their respective technology licensing offices into a single unit. The text of the press release is here.

The most significant aspect of the announcement, in my view, is the focus on innovation as an equal core part of their mission as universities alongside both research and education. They have set themselves an ambitious target of creating at least 300 new companies of scale, and of high value, by 2019. They will jointly offer new facilities for pre-competitive research and design, prototyping and process innovation - to help harness and commercialise new ideas, knowledge and inventions. They also promise to prioritise the establishment of a wider support framework of educational, legal, financial, technical, management and marketing capabilities and support needed to set good new business ideas on their way.

I strongly believe that an overall national policy objective should primarily be fostering dynamic young companies based in Ireland. While I believe that foreign multinational investments into Ireland have served us extremely well over the last forty years or so, I fervently believe that sustainable wealth and employment should be based on a vigorous breeding ground of young companies. Perhaps in the past, there has been an over-dependence on relatively rapid creation of employment by major multinational investment, and a sense that our indigenous companies were "not playing at Croker". There was a sense perhaps that the multinationals were the "A" team, and the indigenous technology companies were just a "cottage industry".

In fact, I think that's absolutely right: the indigenous technology sector in Ireland has been pretty much a cottage industry, and long may it continue to be so. We need many many more "cottages". In my view, the indigenous technology sector in Silicon Valley is also in effect a cottage industry. A vibrant community of innovative small companies, each with a relatively short life-cycle, but with re-cycling of its engineers and business leaders, is actually the foundation of Silicon Valley.

If the TCD-UCD partnership is to succeed, I believe it should do so by creating many small innovative companies. I hope most of them don't last too long: a few years each, and certainly less than a decade each. Most of these companies should exit and be bought out, and then both the human capital and financial capital re-cycled into replacement new ventures. By re-cycling, competence, expertise, wisdom, capability, and wealth are all nurtured and grown. A few of these companies may perhaps emerge to become global champions, but fostering global champions should not in itself, in my view, be a primary national strategy. Rather global champions are a by-product of a successful innovation environment.

Its perhaps ironic that an Irish Green Party Minister is in Silicon Valley this week and currently positioning Ireland as the potential Silicon Valley of Europe: I wonder does he agree with me that re-cycling (not of materials, but of human and fiscal capital) is at the core of the Valley's success.

Technology advances, and so a company founded on one exciting new technical development has only a limited window in time to be successful: it - the people and the capital behind it - should then move on. A company which overgrows itself can find itself limited in its strategic options: too small to be a sustainable global top ten player, but too big to be easily acquired.

Stating that our national technology enterprise policy should be akin to a vibrant cottage industry is probably heresy to some! But I believe that it is precisely because of the large number of small, relatively short lived, companies in Silicon Valley that giants of the Valley like HP, Oracle, Google, eBay and Cisco have emerged. These larger companies have emerged for a number of reasons: there are bounteous estate of dynamic young companies to acquire; R&D can be augmented by acquisition; emerging markets can be pump-primed by dynamic young start ups; experienced human capital can be sourced who have lived through at least one successful business expansion; and seed and investment capital can demonstrably work.

One danger inherent in the TCD-UCD announcement is the public perception that innovation is primarily the outcome of University led research. Ordinary mere mortals - not the elite fourth level and PhD types - might not expected to produce new companies of value. There is a further danger that there may be a public perception that a major - no, the major - reason for tax payers funding the universities is precisely to create new companies and new employment.

On the contrary, I believe that in principal anyone can be innovative. In my view, the primary source for innovation is intelligent insight into a market opportunity. Thus innovation is primarily (not solely of course) led by market intelligence: a deep understanding of a market opportunity that in general leads to an evolution from where the market already is today, rather than a complete revolution. In general, revolutionary new ideas take considerable marketing investment and muscle to become successful; incremental step forwards from what the market already understands are in general easier to introduce and commercialise.

Being successful in the global marketplace requires a careful holistic orchestration of various activities: market and opportunity analysis; technology assembly and integration; field testing and early, reference, customers; go-to-market capability and channel reach; funding; and leadership. I refer to this orchestration as an "innovation score", in the same sense that a musical score synchronises concurrent orchestral activities into an overall harmonious effect.

The technology underpinning innovation need not always be developed in Ireland: innovation scoring requires technology assembly and integration, identifying and selecting the correct technologies, regardless of their global origin. Tax payer funded Irish based science may as a result not be exploited in Ireland: equally, successful young Irish companies should not limit their source of technology to Irish based science. Most of all, a successful innovation may simply bring together pieces of technology which already exist, but which have never been put together before in such an innovative way.

At a recent event in the Science Gallery, a composite map of the world was shown at night. It clearly shows the industrialised world against the darkness of the third world.

The presenters, the Lebone group (pronounced "le-bone-nay"), met as undergraduate engineers at Harvard University. They observed that the coupling of two existing technologies - microbial fuel cells and standard high efficient light emitting devices (LEDs) - could provide light in rural Africa. The same technology can also be used to re-charge mobile phones. Their product is used by simply inserting two wires into rotting compost or dung, and the microbes bio-electrochemically generate small amounts of direct current which is then used to power highly efficient LEDs, or to re-charge mobiles. The product has been field-tested in Tanzania over several months, and the project is now supported by the World Bank.

This example shows how an innovative, pragmatic solution was derived by understanding a market problem (off-the-grid power generation throughout much of the third world), identifying existing technologies (microbial fuel cells and new generation LEDs) invented and researched elsewhere, bringing these technologies together into a new product, field testing and then initiating a new business venture. The successful innovation score did not require extensive scientific research to be carried out by the innovators.

Technology can come from latent and unexploited intellectual property within major corporations. As just one example, Hewlett Packard has just closed their second global call for their Innovation Research Programme, in which they requested proposals from innovators outside of HP anywhere on the planet, to license specific intellectual property from HP in areas such as analytics, cloud computing, intelligent infrastructure, digital commercial print and so on. This enables HP to test new emerging technology markets via young dynamic partner companies.

In the current global economic climate, it is not unusual for major corporations to in effect out-source their R&D. Acquiring young companies who are each proving that a new innovation is successful in an early stage market, but which do not have the global channel reach, may be a less expensive way to enhance their product portfolio and innovate, than funding internal R&D and product teams. As a further consequence, internal R&D and product development teams have increasing difficulty getting internal budget, increasing the possibility of spin-outs from the corporation, and increasing the possibility of licensing intellectual property out of the corporation. HP has completed 116 acquisitions in its life time: it is now actively licensing intellectual property out of HP.

An insightful understanding of the current state of a market can come from emerging global industry standards, particularly when there is end user pull. IONA Technologies was successful precisely because we personally participated in such an initiative. For a time IONA was able finesse the global industry by using global end user feedback and experience of IONA products to improve these standards, creating global industry leadership and a "catch-up" game for other global vendors.

I've probably written enough in this blog article. In summary, I very much welcome the TCD-UCD link up, and fervently hope that it results in more commercial research being attracted. But I also fervently hope that we can create a small company culture, with reasonably frequent re-cycling of human and financial capital, and from which a small number of larger companies may eventually emerge. I also fervently hope that we can foster "innovation scores" in which anyone with insight, skill and sheer determination can build a great company, whether or not they happened to gain a PhD from UCD-TCD (or anywhere else).

Like some other teenagers of my generation in the seventies, one of my hobbies at the time was faithfully constructing scale models from plastic kits from Airfix, Revell, Historex etc. As I developed my skills and interest, I took great pride in adding extra levels of detail – particularly for ship kits. Eventually I converted particular kits so as to model an aircraft, ship or vehicle not directly available as a standard kit, by moulding balsa, crafting acetate sheet and so on – for example building a twin engine, triple tail fin Avro Manchester from the Airfix kit of the successor four engine Lancaster. The monthly Airfix Magazine was a great source of designs and examples, and I still have a shelf full of back copies here in my office at home.

When I was an engineering student, I took a similar interest in building my own, albeit fairly simple, analogue electronic circuits – the usual things such as oscillators, radios, fire/smoke detectors, etc. Of course, as with converting plastic kits, having a good stock of spare parts - scavenged from broken electronic circuits – was useful. I used to keep them sorted into empty margarine tubs on my shelves.

In more recent years, I became more and more involved in computers, from the digital gates and circuits up to microcode, assembler, compiler/interpreter systems and finally full systems and applications. I was fascinated when they became affordable as home systems, and played around adding extension cards and interfaces to my first PC. I was initially really impressed by Dell when I saw their range of alternative pre-configured systems for direct order across the internet. Dell used to have a plant in Bray just a few miles away from my home, and I have bought many systems over the years from them for home, school and charitable use.

But I’ve often thought there must be quite a few people out there like me who aren’t quite happy with a specific system they buy from Dell (or indeed any other supplier). Although I carefully chose a configuration before buying, still afterwards I frequently tailor my purchase further with additional cards, expansions and options. I also wonder whether somebody out there has already built the perfect system for creative digital photography, or multi-screen home media management, or whatever. Or whether anyone would be interested if I could publish my own designs and configurations. Or whether any configurations would be commercially interesting, for example for specialist systems such as creative digital media design.

Reflecting on my fads above, the common aspects are a desire to build interesting variations of basic designs; from additional parts either built myself, or stock-piled somewhere (not necessarily in margarine tubs!), or purchased elsewhere; to look at designs published by others, and perhaps even to publish my own.

Well, that’s what Cloudsmith is doing for the world of software. One way to think of Cloudsmith is as a derivation of Dell’s world, for software. You can browse an online catalogue at Cloudsmith and see what “distros” – configurations - of software components are available. If you want to chose any particular one, you click it and it “materializes” – ie downloads its parts, and then automatically assembles them together – onto your machine. And like getting a system from Dell, the different components of your configuration will usually come from different “repos” (repositories - think Dell sub-suppliers) around the world. So rather than just a single download of a binary file, a materialization will usually automatically fetch multiple files, from multiple places, and assemble them automatically for you on your machine as a complete system. The distro is thus “virtual”: it is not monolithic (like an old download) and its parts are not hosted at Cloudsmith (any internet repo can contribute).

If you like what you find, you can send others the “Cloudlink” you used. When they click it, the same components will materialize from the cloud of components available in multiple repos across the internet, and assemble and install on their machine too. So, you can materialize something specific without having to yourself connect to the Cloudsmith site and without having to search the online catalogue for it – just get the Cloudlink for it from someone else via an email or blog or whatever, and the magic will happen when you click.

Whats kinda nice about the Redhats of this world is that they pre-assemble a large collection of software components for you as a full Linux system. Whats nice about Cloudsmith is that if you build your own interesting set of software components, then you can publish that configuration at Cloudsmith, and reliably make it available for other people to use. It need not of course be as complex as a full operating system: just a nice application or tool or subsystem for a particular use which you have put together, using your own design or as an interesting derivation from somebody else's published distro. You can publish novel and interesting things that you’ve done, for others to take a look at and possibly use – no longer are you or they tied just to the pre-assembled varieties which the software equivalents of Dell provide. And if you have components from a new repo which Cloudsmith does not yet know about, you yourself can simply add that repo to Cloudsmith’s map of the world – it is the Cloudsmith software equivalent to introducing Dell to a new sub-supplier.

Why hasn’t a service like Cloudsmith existed up to now ? One reason is that although there has been a proliferation of useful re-useable software components developed and published around the world in many online repos, there has equally been a proliferation of software version control, make and build systems. It is difficult to justify re-engineering everything globally to use the same version control and build technology. IMHO it is impossible to impose one build or version control technology, or one IDE, on everyone: and it is short-sighted to expect everyone else to use the one that you happen to think is best. However, with some thought and help, it is possible for systems like Cloudsmith to automatically, on the fly, interpret the meta-information in all these version control and build system technologies, and thus automatically derive a global perspective and capability across many different repos.

But how do you know that somebody else’s published configuration is any good, and trustworthy? Well how does a plastic modeler or an electronics hobbyist know that somebody else’s design is any good ? Part of the answer is the popularity of the design – how many others are using it, and what do they say about it – and part of it is the reputation of the designer. Social community sites on the internet in general work because most people are ethical, and many people can watch, observe, comment and mend where necessary information which is incorrect.

Assembling and publishing new configurations of components is fun. Is it however just for hobbyists and software hot-rodding ? I believe that many commercial organizations will find interest in discovering and contributing to software configurations and assemblies that add real value in their business domains. I think enterprise managers will find reassurance not only in the popularity of certain distros, but also because they can be reassured that precisely the same distro (and bill of materials) is reliably installed on every machine under their control, and because component updates can be notified and controlled. I think vendor product managers and technology strategists will find interest in the concept of being able to build their own private “cloudspaces” for their licensed customer communities, to manage and distribute software in a controllable way.

Now, let me put my hand up and confess that all of this isn’t yet as smooth as we would like at Cloudsmith just right at the moment: it is an alpha version for the community to experiment with and give us feedback on. The materialization wizard, the wizard to build a cloudlink are IMHO both pretty good and straightforward to use. Private and public cloudspaces are ready to use. But today, the publishing wizard and the wizard to register a new repo are IMHO a bit clunky: we’re improving them right now and we expect better versions within a couple of weeks. Our online documentation is being improved. However, if you’re interested, contact us and we’ll talk you through online and get you involved in the Cloudsmith community.

There’s an interesting interview with Niall McCullough, architect, yesterday in the Irish Times weekend magazine, about the new version of his book “Dublin: An Urban History” (unfortunately the Irish Times online is only premium paid-for content so I can only give you this url to the article). There’s also an interesting web site, giving additional histories of Dublin and the patterns which have shaped it at www.reflectingcity.com.

One of the things which I had not realized about Georgian Dublin is that the buildings, which of course are a part of our heritage, were apparently in general not built to last! The article says: “Based on a land-lease system, the terraces and squares were designed to stand for the lifetime of the lease, usually between 40 and 100 years, whereupon they would be torn down and built again.” Perhaps this is one reason why the Georgian Society has had so many challenges in trying to preserve the best of Georgian Dublin for us and for future generations.

Earlier in the last week, I was in Liaoning province in north east China, for Sli Siar, for discussions relating to the “Five Points, One Line” strategy to re-invigorate one of the old industrial parts of China. As you probably know, Chinese government policy (until recently) is that land is only available for lease, and is owned by the State. The lengths of leases are set by national law, and vary between 40 and 70 years, depending on the land use: residential, commercial, industrial etc. In fact, China has been through various land reform policy changes: the 1946 reform in which land was expropriated from the landlords and equitably distributed to individual households in rural villages; the collectivisation period during the Great Leap Forward in the 1950s in which land was grouped into shared communes; and the current system introduced in the late1970s during Deng Xiaoping’s reforms. I say “the current system”, because in fact just last month, the National People’s Congress passed a new property law, after many years of deliberations, which recognizes the status of private property including land. In my own experience, urban planners and developers today in modern China in general expect their buildings and developments to survive the length of leases of the land on which they are built.

Then on Friday, having traveled back to Europe the previous day, I was with some folks from IONA, visiting one of our customers in the financial services sector, in Zurich. The meeting was with their senior architect – let me call him Tom. Tom is responsible for their group-wide, global IT architecture, and the meeting was to discuss SOA strategy. One of the very interesting points he and I discussed was that there is a sense in the enterprise IT industry, that at long last, we collectively in the enterprise software industry are “building to last”.

As an aside, I do wonder whether “architect” is the appropriate title for somebody like Tom. I tend to think of civil architects as professionals who design buildings. Rather somebody like Tom is really an “urban planner” – he presides over the current and future infrastructure of an entire software city, on which individual applications – buildings – are built.

Anyway. With previous middleware technologies – DCE, DCOM, CORBA, J2EE, etc – and with all due respect to those thousands of technologists world-wide who worked to create these technologies, I think there was always an expectation amongst senior architects (urban planners ?) and IT visionaries, that each of these technologies would fade in time. Sure, each might be strong enough to last for a decade or so, but business logic and applications that were built to exploit any one of these technologies were constructed in the expectation that a more modern, better middleware technology, would emerge within at most a decade. It was perhaps like the relatively short land leases of Georgian Dublin I mentioned above: build your artifacts in the expectation of re-building them a few years later. And so the middleware world proved to be,

Tom postulated that, at long last, enterprise software architects (urban planners ?) can be like the Victorians of over a century ago: laying down infrastructure – whether it be urban water supply, underground and metro train networks, or even sewage pipe networks (what is the best analogy for middleware ? – I leave it to your personal prejudice!) – that will last for a hundred years or so. Is SOA the end of middleware as we know it ? Isn’t SOA good enough to give us a stable infrastructure for at least a hundred years ?

Well, my view is yes, I agree that it is but with one proviso: one has to construct a SOA based (urban-like, city) environment in the expectation that middleware technologies will in fact continue to evolve and change. SOA may mark the end of middleware as we know it, yes Tom, but its chief contribution in this context is its meta-level. In the same way that metadata in a database allows one to reason and manipulate the underlying data, so should a SOA system enable one to reason and manipulate the underlying middleware.

SOA capabilities in frameworks like Artix and its open source companion Celtix allow a de-coupling between business logic and services, and the underlying middleware. In particular, future middleware technologies can be inserted into such frameworks. The meta-level capabilities enable dynamic re-configuration, including for example interface versioning, data versioning, retooling and end-point guardianship.

With SOA, we have indeed reached the end of middleware as we know it, and can now enable enterprise applications which can be built to last.

In my previous post, I discussed a key challenge facing executives, such as the real VP, K-san, who in principle wish to generate value by adopting SOA across their enterprise. I summarized how Lean Manufacturing and Lean Design gives Toyota the confidence to claim the tagline (here in Ireland at least!): “The best built cars in the World”. I also noted the emergence of Agile thinking in software development thinking.

SOA should impact an entire enterprise’s computing capability as well as business architecture, and needs to be more holistic than specific software development practices such as XP or Scrum. Lean principles can be applied to SOA, whether or not Lean and Agile techniques are also applied to the actual software development of services in SOA. While I personally advocate that software development should be agile and lean, some enterprises may chose not to do so whilst still nevertheless applying Lean thinking to their architecture. The macro principles do not necessarily imply micro ones. The focus of what I call Lean Software Services - LeSS - is empowering people within an organization to focus on their key responsibilities and opportunities, without being overwhelmed by technology choices.

SOA is more than just another technology fad. It is not an issue merely for the CIO: it impacts the CEO and the Board room. It fundamentally impacts the business architecture of an enterprise, as tasks and services undertaken both directly by humans (alone), machines (alone), and humans and machines (acting together), are composed into business choreographies. The composite processes which result are fundamentally different from the classical stove pipes and “silo-ed” applications which are sometimes found as a consequence of history and immediacy: specific business support is needed, specific applications are purchased or built to address these urgent requirements, and once operational these applications become valuable – if not critical – to business operations. Another cause of software silos is merger and acquisition activity, which sometimes result in businesses being urgently combined by operating two software business stacks side by side. It is not that unusual to see operators – for example in a call centre – using multiple physical screens to interact with entirely separate backend systems. SOA is fundamentally and dramatically different: it creates enterprise business value by horizontal impact across the entire business and IT architecture.

SOA embraces a number of software strategies:

· user interfaces, including both traditional screens and new (eg handheld) devices;

· the choreography of software services, and thus the composition of various business logic components to implement business process flows across the enterprise;

· software services, which are the software implementations of those business logic components;

· and finally the underlying substrate of hardware, operating software, databases and middleware.

Separate to each of these four strata, but having influence on all four of them, is the governance of the architecture. Governance decisions arise from both operational and executive concerns: they include for example variation of operational capacity, operational fail-over and backup policies, but also executive led audit trails, authentication policies, and authorization procedures.

In considering Lean principles and SOA, one of the key Lean strategies is the reduction and elimination of waste. Henry Ford introduced “dock to factory floor” inventory avoidance, in which incoming materials were not warehoused but used immediately on the production floor. Toyota subsequently adopted the practice as part of their Just In Time inventory policy. The Poppendiecks interpreted waste elimination in software development as the minimization of unreleased software: code in development, and not yet in production. Partially done work – whether in manufacturing or software development – represents sunk cost but as yet unrealized value.

The Poppendiecks advocate a policy of delaying commitment – postponing writing any software – until there is clear evidence of precisely what is needed. Fuzzy and uncertain requirements lead to wasted effort and probable “bloatware”. Folklore asserts that the Pareto Principle in general applies to software services: 80% of users use only the same set of 20% of the features of a software system, and that by implication 80% of the features are of marginal value. I personally am unaware of any specific research data that supports the folklore, but it’s a good attention grabber for any executive! Kent Beck and his colleagues in the XP world also advocate cautious commitment until customer expectations and “stories” are well understood.

Folklore also has it that the cost of changing a software program increases (exponentially) with time. Delaying commitment therefore appears to destroy value. Lean Design and Agile/XP on the contrary increase value by delaying commitment, benefiting from modern software technologies which have reduced the costs of change.

Consider a fully flexible airline ticket, allowing you to select any flight over the next month. It is more valuable, and costs more, than a fixed ticket for a specific flight. Think also of stock options: an option to buy or sell something in the future has value right now. Delaying commitment can create value, not destroy it: see the excellent paper by Hakan Erdogmus and John Favaro who apply financial option analysis – including Black-Scholes option valuation - to software development using XP.

Naturally, if you want full flexibility for an airline ticket, you go buy yourself an aircraft of your own. The longer the time you have before an option to do something (fly, buy a stock, develop a software program) lapses, the more valuable the option is: I guess that’s one reason private aircraft are so expensive.

Of course, in the same way that delaying a decision can improve the discounted cash flow (DCF) value of a software implementation, poor subsequent execution can destroy the value by delaying positive cash flow. The Poppendiecks conjunct delayed commitment with rapid execution: once a decision is actually (eventually) made, it is put into practice extremely quickly. It is no good having an expensive private aircraft if it is out of service when you eventually decide to use it.

Seems a bit of a digression from SOA, perhaps ? Bear with me. As I noted above, the focus of LeSS is empowering people in the organization to work on their key responsibilities and opportunities, without being overwhelmed by technology choices. The highest value comes from delaying a decision to the opportune time, and then executing immediately. And then, perhaps later on, making a new decision, and executing immediately.

In fact, the ultimate value results if the decision can be orthogonal to the environment it affects. That is, the decision can be taken at any time, and re-considered at any further time, without restriction from the environment. Your private aircraft should be available any time you want it. To quote from Wikipedia: “Orthogonality guarantees that modifying the technical effect produced by a component of a system neither creates nor propagates side effects to other components of the system. The emergent behaviour of a system consisting of components should be controlled strictly by formal definitions of its logic and not by side effects resulting from poor integration, i.e. non-orthogonal design of modules and interfaces. Orthogonality reduces testing and development time because it is easier to verify designs that neither cause side effects nor depend on them.” A classic example from hardware design is that any computer instruction can be applied to any memory location, without restriction.

A good architecture – particularly a SOA – meets or exceeds expectations for run-time stability and performance, while minimizing the investment – time, financial, and human – to create, modify and maintain it. Orthogonality is an excellent strategy.

As we reflect on the four SOA layers I mentioned above (user interface, choreography, software services, and substrate) together with governance, LeSS insists that all are mutually orthogonal, yielding a separation of concerns. If this is achieved, staff are empowered to focus on their key responsibilities and opportunities, without being overwhelmed by technology choices. Learning can be amplified without corruption or qualification arising from other layers. Decisions can be delayed until appropriate information is known for a specific activity – whether designing a new screen, building a new choreography routine, or implementing a new software service – and then executing quickly, without recourse to factors arising in orthogonal layers. Integrity can be strengthened by building testing, assertions and governance without risk of stress cracks being induced from adjacent layers. In summary, the principles of Lean Design as advocated by the Poppendiecks and others, can be applied to SOA.

It goes further. My VP friend, K-san, noted in my previous blog posting of his concern of the lack of a talent pool for SOA. In particular, what technologies and industry standards do software developers need to implement SOA components ? What skills should I train my staff for, and what skills should I hire against ? Using LeSS, the answer is that software developers coding business logic as software services can pretty much use whatever technology with which they are comfortable – C#, Java, C++, C, Ruby, even Cobol; along with, as appropriate to the selected technology, message definitions, copybooks or IDL. The choreography of the software services which they implement, and the infrastructure to support them, should be – and are, is a LeSS environment - orthogonal concerns.

These are bold claims. Can the SOA strata be orthogonal ?

Let us start on the user interface and choreography layers. The LeSS claim is that user interface specialists and choreographers should independently design and implement. User interface specialists focus on the ergonomics of screen layouts, devices and the human-machine interface. Their primary attention is ensuring efficient dialogue with the software infrastructure, and minimizing misunderstanding resulting from inadequate layouts, poorly designed graphics and inconsistent command activation. Choreographers are concerned with business process flows across a set of collaborating business logic components, implemented as software services. The routines which they design ensure that business procedures are safely implemented and, further, that these can be rapidly extended and modified as business needs dictate. The routines in turn are implemented by a business process engine, of which several alternatives exist – both open source and vendor produced. In practice, most business process engines, certainly of which I am aware, do not impact very much on the user interfaces offered by the software services which they choreograph. Orthogonality at this particular boundary does not appear an issue.

Business logic components implemented as software services frequently require user input and guidance, and present information. In SOA, can the user interface and software services strata also be orthogonal ? In practice, user interfaces designed for browser access today separate “look’n’feel” from actual dialogue, by using for example CSS. Indeed there is a trend to further separation of business logic implementation in SOA, from presentation and dialogue to end users, by new client side tools based on Ajax technology – for example “html scraping”, data mashups and personalized portals. These browser based tools are orthogonal to the enterprise software services with which they interact.

Let us move on. Choreography tools, such as business process engines, ensure appropriate process flows between specific software services implementing business logic. Each software service implements a prescribed interface encapsulating its functionality – indeed this is a primary motivation for SOA, as I noted in my previous post. Thus, so as to control the process flows across software services, a business process engine has to understand the interface definitions. This begins to get at the root cause of my VP friend K-san’s issue: what technology should be used for interface definitions ? Can the choreography and software service strata really be orthogonal ?

A similar issue arises at the boundary between the software services and the underlying enterprise substrate (of middleware, databases, operating systems and hardware): can these two strata also be orthogonal ? Programming language portability used to be a major issue a couple of decades ago: it arguably is not today, due to dynamic languages (such as Ruby and PHP), bytecode interpreters (eg for Java), and platform virtualization technologies (such as VMware and Xen). On the other hand, middleware substrates and software application programming are usually strongly mutually coupled, and are hardly ever orthogonal. Business logic is written to exploit a specific middleware technology, whether it be message definitions for MQ, Tibco Rendezvous or JMS; or FML for Tuxedo; or IDL for CORBA; copybooks for Cobol; or J2EE EJBs or .Net components and WSDL.

LeSS strongly argues that software developers writing business logic to implement software services should be able to do so regardless of specific middleware technologies. Of course, each software project does have to make some choice: a C# developer selects Windows and .Net for her substrate; a mainframe programmer selected Cobol, copybooks and CICS; and a Java developer perhaps 10 years later choses Linux and JMS. Different components may be implemented in different technologies, for reasons relating to history and/or skills availability. It is the role of the middleware substrate to allow such decisions to be independently made, concurrently and also across time.

Please note that I am not arguing for complete transparency of the distributed infrastructure to the software developer. I can recall certain research projects, and certain commercial products back in the early 90s – lets not name them for fear of embarrassment – advocating that any (fine-grained) software interface should be capable of remote invocation, so that a software program can be provisioned arbitrarily across a network of machines! Rather, LeSS asserts that each software service should explicitly identify (at least) one interface as available for use by other software services (regardless of their technology), and quite probably from a remote machine. However the technology – MQ message definitions, CORBA IDL, WSDL, whatever – chosen to do so, may vary from software service to software service.

Please note too that I am not advocating universal availability of all technologies across the entire substrate of SOA: for example, it does not make sense to make MQ and CORBA and JMS and Tuxedo available everywhere across the enterprise. Instead, using Lean principles, a specific technology should be made available only when and specifically where needed: just in time. In particular, making a specific technology available at the core of the system – for example in a central hub – may be a prime target for the (Lean principle of) eliminating waste. When software is available, but infrequently used, it may be an example of sunk cost and poorly realized value. LeSS advocates provisioning and installation of substrate software – e.g. a messaging system like JMS – only at those specific software services (end-points) which need it at this time.

What of the governance of a SOA system ? Lean Manufacturing systems pay extraordinary attention to ensure smooth flow across the factory infrastructure even in the face of fluctuating demand and micro-orders from customers. Resource hubs are in general a source of challenge and problems: de-centralisation and dynamic allocation are fruitful tactics to minimize contention and to anticipate demand waves. LeSS likewise emphasizes that software business logic should be orthogonal from dynamic provisioning of capacity and capability. In general, dependence on central software hubs (usually on dedicated servers) is a poor tactic, even more so if a source of sunk cost and poorly realized value.

Drivers of governance and substrate changes can be very many: for example, security policy improvements; fail-over enhancements; change in choice of messaging protocol so as to reduce cost; change in a data format so as to modernize and improve integration; and re-factoring of a service interface definition schema so as to improve re-use, add clarity and enhance development agility. Versioning of service interfaces is common in the evolution of practical SOA implementations. Such policy enhancements in the governance, and improvements to operation of the enterprise substrate, should not require human intervention in the business software.

My friend K-san was concerned about the lack of skills for SOA: the technology choices available overwhelm a clear selection across the enterprise, leading to risk in staff selection and skill sets. LeSS empowers staff in the organization to focus on their key responsibilities and opportunities, without being overwhelmed by technology choices. Each project team can chose to use the technology with which they are most familiar. Separation of concerns, and orthogonality, can be achieved – today - by appropriate selection of off the shelf products.

Lean principles are being applied by many software development teams worldwide to eliminate waste, delay decision making until facts emerge, execute fast, make integrity inherent, amplify learning, empower teams and drive a holistic view of their system.

Lean principles can also be applied at the macro-level, to entire enterprise architectures. Learning can be amplified without corruption or qualification arising from other layers. Decisions can be delayed until appropriate information is known for a specific activity – whether designing a new screen, building a new choreography routine, or implementing a new software service – and then executing quickly, without recourse to factors arising in orthogonal layers. Integrity can be strengthened by building testing, assertions and governance without risk of stress cracks being induced from adjacent layers. The principles of Lean Design as advocated by the Poppendiecks and others, can be applied to SOA.

Separation of concerns, supported by technology which enables orthogonal decision making, and just in time commitment, is the key to Lean Software Services. In adopting SOA, ensure your chosen supplier or vendors provide you with orthogonal, just in time choices throughout the entire architecture. Then empower your teams to use technology with which they are familiar.

A VP friend at major global corporation – let me call him K-san, rather than give his complete name and afflliation - put it to me thus last week: “The fundamental issue that we have with SOA is skills shortage. With CORBA, it was much easier: there was just one, relatively cohesive, family of standards, and we could recruit to these. In the good days of J2EE, equally so. Now today, SOA is just too ill-defined, and a common set of skills are just not there in the market”.

Wow. Despite all the excitement, all the momentum, all the vendor investment in Service Oriented Architectures (SOA), we in the enterprise software industry are failing to produce an adequate talent pool for our customers. SOA is an architectural style in which complex enterprise systems are composed from business logic components, structured as loosely coupled software services. Because SOA is an architectural style, rather than a specific technology, some (many ?) apparently find it difficult to make a commitment to SOA, and to find staff.

Why is SOA relevant ? One frequently cited justification is – at long last – a common basis for dialogue between business executives and IT specialists. The fundamental building block for SOA is a specific business function, usually encapsulated as a software service. A business function may also be a step purely carried out by humans, with no direct machine intervention. Business processes can be choreographed as appropriate sets of interacting (business level) software services, and human steps if necessary.

A key business value which results from SOA is encapsulating, or wrapping, each business logic component as a (usually, software) service. Other parts of the system can then make use of each such service, regardless of the physical location (in-house or off-premises) or specific technology used to implement that particular service. In turn, a business judgment that a service could be profitably re-located or re-implemented can be executed upon, without a cascade of changes being required on those other components which use that service.

A second, perhaps less obvious, business value is the reduction in effort to implement new business services. This can arise both because business components can be choreographed to work together in concert, and new routines can then be quickly defined to provide new services; and also because new business components can be developed and added into the portfolio of available software services, re-using where appropriate some of the functionality already available elsewhere in the portfolio.

From my own personal experience, the epitome of a practical enterprise scale SOA initiative is that undertaken by Credit Suisse First Boston, built on products from IONA and other vendors, starting in the late 1990s under the leadership of Stefan Murer and Martin Prater. I am sure there are others. The CSFB experience is well summarized in online presentations by Hermann Schlamann and Keith Tice. In some of their SOA metrics – see the presentations - CSFB state they have over 900 business logic components implemented as software services, used by over 150 composite enterprise applications, with an average of 3.7 services used per client application, 43% re-use and development savings of over 12,000 mandays. I understand that more recent figures show even high re-use figures and savings.

But back to my VP friend K-san. In the good old days, you could use CORBA to do SOA – and CSFB did do so. Today, there is confusion about SOA technologies and standards, yielding a perceived show stopper: lack of skilled staff.

I take it as a given that staff are professional: everybody does the best job they can, given their education, training, abilities and knowledge. My own view is that it is hardly the fault of the people: it is the fault of the process. What is it about our approach to SOA that makes it difficult to find people ? How can we change our thinking about SOA so that it becomes tractable, pragmatic, and harder for corporations to make expensive mistakes ?

In chatting to Larry Alston about this, he reminded me of the “people to process” transition made by Henry Ford. Henry Ford did not invent the automobile: the problem was that up until Henry arrived on the scene, cars were being hand-crafted by extraordinarily skilled, and expensive, workers. Henry Ford focused on process: he empowered ordinary mortals to reliably build cars by creating the correct processes, and paid them high wages to do so.

Today’s manufacturing processes are derivative of the innovation of Lean Manufacturing. As K-san effortlessly reminds me, Lean is of course associated with Toyota, and specifically the founder Sakichi Toyoda, his son Kiichiro, the external consultant Shigeo Shingo; the engineer Ohno Taiichi, and the HR executive Isao Kato. The story goes that, in the 1950s, a delegation from Toyota visited the Ford manufacturing plants in Michigan. Although Ford was a global industry leader at the time, the Toyota people were apparently not particularly impressed by what they saw at Ford. In particular they observed large quantities of inventory sitting idle, and noted that the production across the entire plant did not flow smoothly, with staff occasionally just waiting around.

By contrast, during their visit they happened to be shopping in a local supermarket, and observed the simple idea of an automatic drink re-supplier: once a customer takes a drink, another is automatically ordered and replaces it. The supermarket deferred re-ordering and re-stocking goods until customers made purchases.

On return to Japan, Toyota further developed the basic concepts of what is now known as the Toyota Production System, which greatly reduced lead-time and wastage, whilst simultaneously improving quality. The basic ideas of “Lean Manufacturing” were subsequently taken into Toyota’s R&D and design shops, yielding “Lean Design”. Today, Toyota use the tagline ( at least here in Ireland, where we have no car manufacturing of our own and thus import global brands): “The best built cars in the World.”

How do we turn ordinary mere mortals in the software industry into powerfully productive professionals for SOA ? Is it just a matter of education, training, abilities and knowledge ? What about process ? What can we in the software industry learn from what has gone before ?

Lean thinking has had a dramatic impact on the global software community. Significant momentum in Lean Software Development and Lean Product Development developed back in 2003 when Mary and Tom Poppendieck published their excellent book on the topic. The Poppendieck’s studied the lessons of Lean Manufacturing, and particularly Lean Design, and applied it to software development. Elimination of waste, delayed decision making, fast execution, integral integrity, learning amplification, team empowerment are the key factors which they identified in driving a holistic view of the process.

Agile software development is related, but had a different starting point. Rather than a heritage in manufacturing systems, Agile has its nemesis in software development projects and by developers themselves. It is particularly associated with Kent Beck, Ron Jeffries, and Ward Cunningham (now at eclipse.org and a committer to the Eclipse Spaces project which I mentioned last week), during their work on development of a payroll system for Chrysler in 1996, leading to the Extreme Programming (XP) movement. However Wikipedia also notes that the "practice of test-first development, planning and writing tests before each micro-increment" was used as far back as NASA's Project Mercury, in the early 1960s.

Thad Sheer provides an excellent explanation of the differences in philosophy between Lean and Agile approaches in his 2005 blog post. Lean projects need not necessarily be Agile, and vice versa. In summary, he suggests that Lean usually gets more initial corporate support than Agile since it uses concepts to which most executives can relate, rather than terminology more appropriate to software developers. He also states: “There’s a lot more bang for the buck vis-à-vis the bottom line with a partially completed Lean initiative than a partially completed Agile initiative. Once an organization is LEAN, becoming AGILE is a step that developers can take on their own without requiring much management backing.”

In IONA, we started looking seriously at Extreme Programming back in about 2002, when we hosted Kent Beck in person several times to meet with our software development organization. I personally recall that Ciaran Dynes drove much of the initiative within IONA, as one of our engineering managers, and I am grateful for his leadership. Ciaran went on to build our Beijing based development centre – using XP - and is currently our Artix product manager, based out of Dublin.

So Chris, what about SOA, and do Lean and/or Agile apply ? Well, interestingly there is an analysis published by IBM which in its first part summarises various approaches to both Lean and Agile software development; and in its second part discusses that these approaches are not incompatible to SOA. The take-away is “Oil and Water do mix!”

Ooof. Personally, I find such a take-away divisive: it implies that the world of SOA and the world of Lean/Agile are fundamentally confrontational, but that perhaps with sufficient management effort they can probably be brought together. To be fair, the folks at IBM say they were responding to a previous article suggesting that SOA and Lean/Agile actually are incompatible!. All of this is hardly comforting to corporate strategists thinking about SOA adoption, and I suspect would not be greeted with relief by my VP friend K-san.

I think I have a differing view. I believe that a successful SOA results from Lean thinking. I discuss this in my next posting.