chris horn's blog

How do you change an entire nation ?

It’s a topical and fascinating discussion to have over a coffee, and clearly is preoccupying a number of people right now. You may be aware elsewhere from this blog that I have been a frequent visitor to China since 2000: the incredible social shepherding of this enormous nation must be one of human history’s extraordinary moments. I found George Packer’s book on the American administration’s work in Iraq both entirely credible and frightening. My trip to KwaZulu Natal province in South Africa last November with UNICEF brought home to me the frustration of a nation challenged to put its wealth and talent to work in rebuilding after a brutal regime. Closer to home, it is intriguing to watch the recent developments in Northern Ireland as Ian Paisley of the DUP and Gerry Adams of Sinn Fein sit down together to form a devolved government.

Changing an entire enterprise IT infrastructure should not be as demanding as changing an entire nation, but sometimes I’m sure some of us wonder. The introduction of a Service Oriented Architecture, and the management of a Service Oriented Architecture, at enterprise scale, and almost certainly globally, gives justifiable cause for reflection.

As you may have seen if you follow IONA, we announced our first product in the SOA repository and registry sector last week. While some observers expressed strong interest, nevertheless certain others believed that IONA is entering a somewhat crowded space, with various other existing pure play and bundled repository/registry products already out there.

The essential difference, IMHO, between a SOA repository and a SOA registry is in fact largely historical, and most vendors in the space – including now IONA – offer a combination. A SOA registry is (traditionally) used to register WSDL interface definitions and to catalogue available services which implement particular interfaces. It is particularly useful at development time to guide software engineers to discover and re-use existing definitions. A SOA repository is (again, traditionally) a runtime store in which certain meta-data about WSDL interfaces and services, and certain governance policies, are recorded, and made available for interrogation by applications and by the underlying middleware substrate.

In a SOA environment, the distinction between a development time phase and a runtime phase is arguably a little artificial – and as a result, the industry is moving to the combination of repository and registry functions within the same products. A new service is evolved over its lifetime within the SOA, and must both be designed and maintained within the context of other operational services. Certainly, a new service must be deployed with care, adequate testing, and appropriate “sand-boxing” so that it does not disrupt the operations of other services; but equally it cannot be developed completely in isolation to the operational effectiveness of the current system. SOA systems are inherently loosely coupled, and thus full testing of the interdependency between services usually ultimately happens carefully in an operational setting.

The governance of SOA systems is a key theme of the vendors – again, including IONA – in this space. Combined registry and repository products are positioned as assisting the judicious management of both the development and operation of a SOA. No doubt “SOA governance” has been chosen carefully as a catchphrase by the SOA industry, resonating as it does in some business executives’ minds against the backdrop of Sarbanes-Oxley and other recent corporate legislation worldwide.

The pragmatic challenge for any SOA architect concerned about the governance of enterprise systems is of course that governance policy mechanisms are almost certainly already widely fragmented across the entire software system. Some policy mechanisms may already be embedded in application code and business logic. Some are in stored procedures, relating to the probity of particular databases and their data management applications. Some are described by business processes, and orchestrated by business process engines. And now, SOA registry/repository vendors come along and pitch “SOA governance” using their respective products.

In a SOA system, how do you implement policy ? How do you affect changes in policy ?

Let’s return for just a second to the analogy of governing a sovereign state. There is both the integrity of the state, and strategic management of it, to consider. By integrity, I mean both a legal framework by which the state operates, including an appropriate set of laws, perhaps set against the framework of a constitution; and an appropriate enforcement mechanism by which the state and its citizens can reasonably assured that laws are obeyed. By strategic management, I mean the set of policies – laws but also fiscal and other incentives - which the state chooses to adopt to, for example, grow its economy, educate its children and its citizens, and care for the health of its population. However, strategic management cannot occur unless the fundamental integrity of the state is assured; equally, an assurance of integrity alone does not necessarily lead to the long term economic and social success of a nation.

From this analogy, I believe it may be useful to draw a distinction between the integrity of a SOA system, and the business policies and processes chosen to be implemented using it. I believe that integrity should be defined and enforced using a “SOA Guardian” – I deliberately introduce a new term – whilst business policy implementation is rightly the domain of business process engines and orchestration.

Interestingly, business process mechanization and SOA service orchestration has probably been given more attention, priority and emphasis, by the SOA industry at large than SOA integrity issues. BPEL and WS-CDL are probably the best known business process mechanization initiatives, for which a number of process engines exist, including in open source form. Indeed, orchestrating business services defined using standardized interfaces is one of the key selling points of SOA, if not the very essence of SOA for some protagonists. Perhaps business process orchestration and choreography has had more attention than SOA integrity, since there may be a perception amongst SOA practitioners that line of business managers, and corporate executives, can relate most to corporate business process enhancements, rather than the more complex holistic concept of SOA.

But “salus populi suprema est lex”: Cicero said that the ultimate law is the welfare of the people. If the people are unwell, how they can they be governed ? If an enterprise system is impaired, what benefit is business process orchestration ?

“SOA Governance”, if it denotes anything at all, should encompass both SOA integrity and strategic management of business services orchestration: I believe that that is in fact what line of business managers, and corporate executives, expect from the SOA industry.

SOA integrity is a critical prerequisite to leveraging a SOA for business process orchestration. The boundary between the two is naturally dithered to produce effects which are actually not really of substance: business process orchestration can be overloaded to attempt to define and enforce integrity; integrity functions can be strained to implement business policy rules. In the spirit of lean software services, I argue however for a separation of concerns: SOA integrity should be focused as a complete baseline from which business policies and procedures can be easily defined, rapidly deployed, and safely evolved.

A SOA Guardian should be driven by rules relating to the integrity and unimpaired operation of the entire system. It certainly includes security concerns, and the authentication and authorization of access from principals to SOA hosted services, based for example on LDAP directories. However it also includes performance concerns, such as load management and brisk responsiveness. It knows the configuration of every service, with technical details such as the size of the allocated thread pools, binding and addressing information, runtimes and containers across the (almost certainly, highly heterogeneous) SOA deployment: it can be used to re-boot any failed service or even ultimately the entire enterprise. It includes transactional integrity, fault and outage management, so that the temporary loss of one or more SOA hosted services does not cause catastrophic failure. It includes management of change, including management of schema and version changes of services and interfaces. In particular, it includes the planning and execution of technology change, so that old technologies and services can be phased out, rationalized and consolidated, without adversely affecting operations.

It is clear that a SOA Guardian has both – forgive me folks, I’m an engineer by background! – sensors and actuators. That is, it collects and presents operational information relating to the integrity of the system. Equally, it is the mechanism for implementing and enforcing changes to the integral operation of the system. It is clear that the Guardian itself needs to be sound and cohesive: a Guardian system will itself likely be federated and fault-resilient. Because of the heterogeneous nature of most SOA environments, a SOA Guardian may well use combined repository and registry products from other vendors in order to administer details of certain proprietary environments. A SOA Guardian is thus much more than the emerging generation of combined repositories and registries.

Having a repository to record decisions and rules relating to the integrity of the system is certainly useful: the library of a parliament records the laws made therein. Having a console from which operational metrics can be gathered is also important: laws should be made in the context of the society they influence, and it is the role of elected representatives to reflect the views of the populace and the current state of society. But a SOA Guardian also includes the enforcement mechanisms to impose change on a system. Some “SOA Governance” products, surprisingly, apparently are weak or even non-existent in this regard.

For a SOA Guardian to enforce a change, it should clearly be desirable that the entire SOA system not be brought to a halt. Changes in security integrity are already routinely dynamically enforceable in most systems. Changes in configuration to improve performance can sometimes be dynamically implemented in some systems. Changes in fault avoidance and outage management can likewise be dynamically enforced in some systems. Changes in schema and interface versions likewise. Changes in technology are perhaps the most difficult to dynamically implement.

A SOA Guardian, given the correct meshing with the middleware substrate, should nevertheless be able to dynamically deploy all such changes to the integrity of an operational SOA, including changes not hitherto envisaged or previously planned. There should nevertheless be no need to update application business logic or business orchestration in making changes to the integrity of the substrate.

Provisioning changes in telecommunications infrastructures is both a science and an art, based on operational experience. Vendors with experience of such realtime provisioning may emerge to become one source of what I have called SOA Guardians.

How do you change an entire nation ? How do you successfully effect evolution of an entire system ? My view is “carefully”, using small incremental and rapidly evaluated steps, along with a strong separation of concerns. The integrity of a SOA implementation is different from the orchestration of business services to exploit a SOA deployment.

“SOA governance” tools may (perhaps, for some vendors even deliberately) confuse the issue. I encourage the industry to instead separate the concept of SOA guardianship from SOA orchestration. A combined repository/registry tool is a step towards a SOA guardian: dynamic binding, late decision, immediate execution and enterprise wide deployment, is also needed for complete integrity and agility.