I wrote this as part of my work for Cloudsmith. It is a follow up of my earlier posting a few weeks ago.
In summary, Cloudsmith lets you browse and find useful bundles of software components which work together – software playlists – and then download ones of interest. Each one can contain components from different software repositories, and Cloudsmith knows where to go, and how to get to them.
--
Eric S. Raymond wrote a seminal paper in 1997, The Cathedral and the Bazaar, contrasting how Linux emerged from a loosely structured, highly collaborative community or "bazaar" with the traditional approach to developing software (open source or proprietary), in which a select group of cathedral-builders controlled every aspect of design and technology.
Most engineers strive to build at least one great “building” during their career, a monument, a shrine, and a testament to their skill. Today, even "cathedrals" are made from parts found at the bazaars - a huge and growing marketplace for open source components, in which thousands of developers promote parts that many other developers combine into new products. The output of many bazaars -- projects and communities such as the Eclipse Foundation, the Apache Foundation, Google Code, SourceForge, etc. - support and publish the efforts of component development teams. Popular components turn up in multiple bazaars, sometimes as identical copies, other times with subtle variations.
Among the challenges development teams, and their co-worker product management and product marketing teams, face when operating within this new ecosystem are:
* What range of components is currently available? Which bazaars have them; what is their status and quality; how popular are they; where can updates and fixes be found; and so on.
* What works with what? What components, and combinations of components, are available? How do the pieces all fit together, and which bazaars have them?
* How popular is this combination of components compared to that alternative one? How do we know when and if we should update a selection of components, as new versions of the constituent parts emerge?
* How can we build playlists which combine components we built ourselves, with components found in public bazaars and that change in ways we don't control? How can we move to the new version of a public component without breaking what we already have? And how can we keep what we found in the bazaar from getting so intertwined with what we built that we can no longer separate them? What is the best strategy to manage change, when your organisation and your team are increasingly mixing public software components with your proprietary assets?
* Who is going to support us when we use some unique combination which we assembled from public bazaars? Is there anyone out there doing something similar we can learn from?
* We fix and extend components we find in the bazaar, and sometimes create entirely new component playlists of our own. How do we share our work with other developers in our organisation or (assuming our corporate policy allows it) contribute things back to the bazaar for the public good? And assuming we've shared it, how do we know who is using it, and for what?
It is, of course, no longer just an issue of providing a stable, managed foundation on which you and your colleagues can build. There is heightened corporate awareness reaching all the way to the audit committees of publicly quoted companies, due to the multiplicity of software licensing policies. The issue of knowing if, when and how public software assets are being used inside a corporation has become a high concern.
The ability to tailor software should be its value rather than its risk. But in todays world, isn't software componentisation paradoxically slower than it could be, precisely due to the changes, improvements and proliferation offered by the community?
Eric Raymond describes how extremely useful software can result from open collaboration, despite the absence of a clear lead architect directing the project. Today’s software repositories illustrate this principle on a grand scale - they are collections of really good and useful components developed, published, maintained and extended, sometimes by individuals and sometimes by organized teams of collaborators, in a process that can seem almost anarchic compared to conventional internal development.
As bazaars of developed, and contributed, software components have matured, the complexity of fitting together appropriate combinations have increased, as has ensuring that things do not break as each component is maintained.
One example is Eclipse, which is a common integration platform for many components. The recent Ganymede release lists nine application frameworks, six toolsets for embedded and device development, six toolsets for enterprise development, five language IDEs, and five aspects of its rich client platform. All of these elements, in principle, can be used in any combination of choice, although there are seven different official Ganymede packages are listed. Forty-five additional different project downloads are listed. And nine different distributions from member organisations are promoted. It shows an impressive level of community momentum and collective activity, but which of all of the alternatives do you really need for your particular project?
Actually, it is even more complex, because each bazaar stacks up components from its own shelves with components it finds in other bazaars. And you are often building not just one cathedral, but several based on a common set of blueprints. Perhaps you want to develop using Seam rich client Java toolkit? Then you might need a playlist of the Eclipse Classic IDE, JBoss Tools, Seam Core, JBoss AS, and PostgreSQL (with thanks to Stefan Daume for suggesting this particular playlist). But to do so, you may need to visit the Eclipse, JBoss, Seam and Postgres bazaars to put this all together -- unless you can happen to find somebody else who has already done this for you. If you want to build an email spam filter, then maybe a playlist of MySQL, qpsmtpd, my qpsmtpd custom modules, php pages (status), and open flash chart run-time files might be just the job (with thanks to Bjorn Freeman-Benson for this playlist).
Finding out what software components are available is a modest challenge: you can use raw Google, or Google CodeSearch, or Koders, or Krugle, or Codase, or something similar. The more significant challenge is finding out what works with what else to form a useful playlist; then how to get hold of the right version of each these pieces from each of the right bazaars concerned; how popular is this specific playlist of components; and how to get notified if any of the pieces are subsequently changed. If you want to be civic-minded, you might also want to find out how best to contribute original or derivative works back to the remainder of your organisation or community at large.
Our industry is maturing: we really soon should reach the equivalent levels of professional practice as our colleagues in other engineering disciplines, such as electronics hardware and civil engineering. There now is - perhaps at long last - a substantial number of re-usable, well-engineered, components available to all of us, being extended and improved on a daily basis. We should all be able to build cathedrals, and other artifacts, from the components we find. But the vast range of components, coupled with the fluidity of material - software - with which to work, has presented our industry with some new challenges,and which are not as apparent in other engineering disciplines.
Showing posts with label SOA. Show all posts
Showing posts with label SOA. Show all posts
Thursday 18 September 2008
Tuesday 30 October 2007
The Long Tail Wags
What kicked of this blog entry for me were some reflections I had as I chaired a one day conference, with a preceding half day workshop, a couple of weeks ago, which was organized in Galway by Fidelity Investments, on the subject of Web 2.0, and how commercial organizations – such as Fidelity – can benefit from and add value to the global internet community.
A core theme of Web 2.0 is the collective wisdom that results from the network effects of sharing across the community. The wisdom of the crowd is sometimes more than the sum of the individuals therein. Wikipedia is one prime example, since collective knowledge can trigger insights for individuals, which can then augment the collective wisdom: the added knowledge which would not arise if the group did not collaborate together. I recently wrote a blog entry about a similar phenomenon in multi-disciplinary research. Group and individual reasoning can positively feed off each other.
One of the dichotomies of the Web 2.0 phenomenon is on one hand, the power of the group and the collective knowledge of the crowd; and yet on the other the significance of each individual. Yes, collective knowledge such as Wikipedia and del.icio.us (and of course Google!) have emerged, but blogs written and commented by individuals are still a fundamental force. Some may argue that indeed the information in specific blogs written by certain individuals is more valuable, accurate and reliable than that in shared wiki repositories maintained by the amorphous net community. There is room for both individuals and the crowd.
In the race to be successful on the web, the focus is on attracting and retaining users, eyeballs and clicks. For the promoters of a web site - whether a commercial venture, or just a worthy cause for society at large - building and growing a “market” is key. Monitoring usage patterns, listening and reacting to user feedback, all builds momentum: hopefully a tipping point is passed, and the network effect of market momentum reinforces the popularity and acceptance. However: does a mass market strategy play only to the crowd, and not to the individuals therein ?
For Web 2.0 , Chris Anderson introduced the term “long tail” to emphasise deploying:
“customer-self service and algorithmic data management to reach out to the entire web, to the edges and not just the center, to the long tail and not just the head”.
As noted by an Amazon employee quoted by Wikipedia: “We sold more books today that didn't sell at all yesterday, than we sold today of all the books that did sell yesterday” -- read that slowly to yourself again if you hadn’t come across it before..
Further, in his discussion on technology market dynamics, Christensen noted:
“Simply put, when the best firms succeeded, they did so because they listened responsively to their customers and invested aggressively in the technology, products, and manufacturing capabilities that satisfied their customers' next-generation needs. But, paradoxically, when the best firms subsequently failed, it was for the same reasons--they listened responsively to their customers and invested aggressively in the technology, products, and manufacturing capabilities that satisfied their customers' next-generation needs ….But the problem established firms seem unable to confront successfully is that of downward vision and mobility, in terms of the trajectory map….”
Success – at least in some industries, such as the disk drive and earth moving machinery industries which Christensen documents – can equally lay the foundation for failure. Group and individual behaviour can play off each other, creating emerging forces from below.
Combining the Anderson’s exhortation to reach out to the edge, with the warning by Christensen not to be outflanked by emerging disruptive technologies from below, it would seem wise to ensure that a Web 2.0 market strategy explicitly recognises the potential of the individual, as well as the mass market of the group. Even more important is to react to dynamics in the market and to changing tastes. The observation is: that in a global market of mass consumerism, the ability to cater for the vast number of changing individual personal tastes and desires – to personalise and dynamically tailor your products and services – may be much more commercially – and socially – valuable, than volume plays of standard products and services to an anonymous amorphous group.
The long tail changes: it wags. The universe of micro-markets in the long tail is worth addressing – this is
Ajit Jaokar was one of the contributors at the Fidelity event in
- the web as a platform;
- harnessing collective intelligence;
- data as the next Intel inside;
- the end of the software release cycle;
- software above the level of a single device; and
- rich user experiences.
I thought about each of these six facets as I reflected on the dynamics of the micro-markets, and list some (not all!) of my observations below: how does Web 2.0 address the wagging long tail ?
One common way of reaching out to the long tail, and monitoring its movements, is “data as the next Intel inside”. Google exploits what its itself calls the ”uniquely democratic nature of the web” to derive data on the click activity of millions of users to drive its PageRank algorithm. I suspect that social networking sites, such as MySpace, Facebook, Bebo and LinkedIn, could likewise exploit their data about networks of people. I’ve always wondered in passing how data regulators, such as the Irish Data Protection Commissioner, view such click gathering and social networking activities…
As Web 2.0 embraces beyond “the level of a single device” – which was Ajit’s theme of his presentation - I observe that telecommunications operators, such as Vodafone, O2 and T-Mobile, collect and record quantities of data, ironically in a large part due to regulatory requirements of data retention legislation in jurisdictions such as Ireland. These vast data archives capture about social communication patterns of populations – it is not just the social networking sites which have such data! There are opportunities, and privacy risks, to exploit these vast patterns to provide commercial value. One can imagine intelligence being generated by mobile phone operators to enable targeted marketing by third parties and content providers…
Quietly collecting mouse click activity, or analyzing phone call patterns and triangularising the locations of their owners, are data driven ways of using to drive for the long tail. Importantly, the data is dynamic and therefore so also is the derived collective intelligence: as trends emerge, or change, so can the offerings tailored for particular sets of individuals. But it seems to me that these approaches may wear a clandestine cloak: there are more explicit ways of using collective intelligence for the wagging long tail.
A successful web site needs stickiness to retain, and track, its audience, including the wagging long tail. A social networking site such as those above in part generates its stickiness because of the network effect of having ones friends, co-workers and colleagues using the same site: most people think it is too cumbersome to re-register oneself and encourage one’s cohort elsewhere. IMHO, the resulting stickiness is in fact rather superficial, rather than systemic: it consequently threatens any purported fiscal value for the site. The SIOC project at DERI in
Another strategy to capture the long tail wags is “harnessing collective intelligence” – another of O’Reilly’s Web 2.0 facets. One explicit way is to allow individuals to define their own collections of interesting items, and then to share and discuss them with other like-minded souls. A simple way to do this is just by tagging items: for example, photos tagged with “
Rather than expecting people to work out the correct search and query expression across a set of tag values for what they actually want, another way is to let people explicitly define and share their own collections. Hobbyists – such as stamp collectors – have been doing this for years, and a craft worker likewise learns what the set of right tools are for the job from the experiences of his trade. Enabling individuals – and micro-markets and even the crowd - to share what works well together when, is a positive tactic for managing the wagging long tail, and which I wrote about recently in the context of software configurations and Cloudsmith.
When doing a little research for this blog entry, I came across Dan Bricklin’s interesting blog entry on When the Long Tail Wags The Dog. His theme is that “must have” items have more value than those which are less likely to fit the job at hand; and that general purpose items are likely to have most value since they can entertain the dog as well as its long tail. While I don’t disagree, my thoughts about the dynamics of the long tail, and how to work with within it, are a slightly different emphasis.
Reinforcing Bricklin, being relevant to the individuals and small groups in the long tail requires customization and tailoring of general purpose offerings. In the enterprise software space, software vendors have traditionally worked with systems integrators to tailor more appropriate solutions to different niches of the market. With componentization in the software industry, in principle useful aggregations of different constituent parts can more likely be used to tailor specific solutions. Collective intelligence can be harnessed by explicitly sharing these aggregations, as I indicated above.
However, once delivered, installed and put into use, assemblies of software components in production environments have in the past usually been relatively static. The challenge is that the long tail wags: micro-markets and even mass markets change, and production systems need to more easily do so too. One promise of the dynamic module environment of OSGi is to enable dynamic evolution: if collective intelligence can be dynamic as trends emerge, or change, and then suggest new offerings tailored for particular sets of individuals, then perhaps also can production software assemblies in business and enterprise environments be dynamically adjusted. This is a theme which we are working upon in IONA, particularly in the context of our highly dynamic plug-in architecture for Artix and our open source FUSE offerings – see Eric Newcomer’s blog.
But if the world is dynamic, and contains a universe of micro-markets as well as a mass, can offerings not only be tailored so as to be a good fit, but also be priced attractively for each ? What when micro-markets change ? As I wrote about, one industry which faces these challenges is the mobile/cell phone operators, for whom intelligent and responsive bundling of services (SMS and MMS and call rates and roaming charges etc) is competitively critical. The ability to generalise this approach into the world of software components and packages is of interest and relevance to the wagging long tail. LeCayla is one company building a common approach to the issue of dynamically rolling out new charge and billing rates so that software vendors can competitively foster specific micro-markets.
Let me summarise: the internet as a platform enables a global market to be addressed. Web 2.0 uses collective intelligence to play in both the mass markets, and the long tail of numerous micro-markets. There is room for both individuals and the crowd, and group and individual reasoning can positively feed off each other. – creating the ever lurking possibility of being disrupted from below. Scalably addressing both the mass market and the long tail is however not the complete issue: markets change, the long tail wags, and scalably addressing dynamic markets is even harder – but possible.
Footnote: my 7 month old Alsatian, Charlie, has yet to grow into his long long tail. He wags it a lot.
Monday 20 August 2007
Cloudsmith goes live!
As some of you may know, I regularly holiday just outside Roundstone in
If you haven’t yet been to the west of
From ground level, the profusion of little stone walls can appear as a complex pattern, perhaps even fractal-like. But viewed from a
I came back to
Well, also while I was away in
- Cloudsmith keeps meta-data about assemblies of software components.
- Software components can be sourced from any number of public and private repositories worldwide. Of course, components from private repositories are only available to those duly authorized to use them.
- Cloudsmith does not store the components themselves: but it knows where they are worldwide and how it can access them in the appropriate repository formats.
- A software publisher – an individual, project, or company – can register one or more specific software component assemblies with Cloudsmith.
- A software consumer – an individual, project, or company – can search and browse for available assemblies; and can readily download and install any particular one – “materialize” in Cloudsmith-speak – onto his local machine (or indeed another machine if appropriately authorized).
In effect, Cloudsmith is building a global map of software components (in various forms: source, binary, versioned, and optionally with test suites, documentation and license agreements). Professional software developers - individually or in a community project or working on a commercial offering – can publish interesting new assemblies of components, sourced across one or more repositories.
One of the neatest capabilities of Cloudsmith is a Cloudlink. A Cloudlink is simply a URL: it can be sent in an email, or given in a blog or whatever. When a Cloudlink is clicked, the software assembly which it denotes is then materialized without further intervention, onto the local machine. This gives a very simple download mechanism: publish a Cloudlink, and anyone clicking on it within a recent-vintage web browser can download your software. In practice, when a Cloudlink is clicked, behind the scenes the Cloudsmith site is contacted, and it resolves the differences between the assembly of software components identified by the Cloudlink, and those already available on the local machine, and then fetches (as appropriate from various repositories worldwide) and downloads the missing components.
Cloudlinking in turn enables “virtual distributions”. A software publisher can create a virtual distro, whose components reside across multiple (eg open source) projects and repositories: materializing a virtual distro requires nothing more than a web browser.
If your project is looking for a simple way to make its software available to the worldwide community; if your project is itself using software from multiple sources and multiple projects; if you want to keep your community regularly updated with patches and extensions; if you want to manage installation and distribution processes; then Cloudsmith should be worth taking a look.
Software components, and configurations and assemblies of them, are very malleable. It is relatively easy to define new interesting configurations, as well as new components. Looking at the world wide activity, and the multitude of repositories and projects, it is easy to become overwhelmed. It is possible to detect patterns, and different styles of construction, but sometimes it can be very confusing to see overall themes, to understand how other people are using configurations, and what changes have occurred.
I’m reminded of
Cloudsmith is giving clarity to the construction of assemblies of software components.
Thursday 17 May 2007
Professional software business management
I was on a long haul flight last weekend, from Hong Kong to San Francisco – one of those wonderful flights where you land before you take off – on the way to the next installment of the Leadership For growth programme being run by Stanford Graduate Business School and Enterprise Ireland Hong Kong airport, which I read on an occasional basis, the issue with Tony Blair on the front cover, to pass the time. There’s an interesting article on business schools, and how they are beginning to regain their lost vogue from earlier this decade.
I was the Chairperson of the Irish Management Institute a few years back, which was a little strange because I have no formal background whatsoever in business management, economics or finance! One of the raging discussions we had at the board level of the IMI was what should be the future of executive education in
Then, by coincidence, Monday’s Financial Times had a supplement on Business Education, including its international ranking of the top global executive education schools. It carried several very interesting articles suggesting that the top schools have changed their product from business education to business advice, and almost to management consulting. Customisation of curricula and classes lead to faculty not so much teaching, but instead providing insight in a discussion about specifically how to address issues within a client company, and/or specifically how to apply a particular idea or theory within a client company.
One of the things I had asked myself during my years at the IMI was whether there can be such a thing as a management profession. A profession, by definition, implies some core knowledge, which may be expanded and refined over time by appropriate research and in the light of experience; a way of asserting that an individual has attained a particular level of competence in that knowledge, and therefore can be admitted to the profession; and a code of ethics particularly as to service to the public, including appropriate disciplinary actions if these should be broken (The Economist article makes similar comments). As a professional engineer in
But can there be a management or business profession ? Is there a body of knowledge, an admissions procedure, a code of ethics and a disciplinary mechanism ? Can there be a guardian organization for the profession ? In fact should there not be one, so as to protect the public and including shareholders and investors ? However would such an organized profession stifle innovation and entrepreneurship ? As The Economist observes, Bill Gates dropped out of university and would presumably never have made the grade to become a “professional business manager”.
Hmmm. So what is executive education all about ? Can business management ever become a profession ?
I hesitate to comment further in general for all industries, but I do have some views more specifically as applies to the software industry.
In the mid 90s I had the sincere pleasure of having John Cullinane on the board of directors of IONA. John, as I am sure you know, founded and ran the first software company to file an IPO, the first billion dollar software company, and the first company to do a Super-Bowl ad! His company Cullinet was well known during the 1970s and 1980s. John has written an excellent summary of some of his lessons from those years, which I believe are as applicable today to software companies as they were then, in his book “The Entrepreneurs Survival Guide: 101 Tips for Managing in Good Times and Bad”.
One of the things John said to me early on as a board member at
I guess if what I suggest above is true, then in principle two rival companies with very similar product offerings, and very similar strategies, and of very similar sizes, in principle should be unable to out-execute each other. That is a controversial claim, since execution is key to the success of any software company: but I do believe that a professional experienced software CEO is unlikely to make mistakes in execution, since what is needed in execution is actually now reasonably understood across the industry.
Competitive advantage then in the software industry is increasingly unlikely to come from execution alone. Instead, in my view, advantage comes from strategic insight and analysis, from new products and new business models. Advantage comes from understanding the current state within a particular segment of the industry, and leveraging that to introduce new products and services, perhaps in new ways, and which add sufficient value to motivate the market to invest and customers to buy.
“Success is 10 per cent inspiration and 90 per cent perspiration” said
An accepted body of knowledge will not, in my view, stifle innovation. Bill Gates would not have suffocated had such a pragmatic tome been available to him.
Perhaps I’m just representing a personal bias. I get excited by discussions on the state and direction of the industry, and where the current leverage points and opportunities are. I get less excited by discussions about operational issues, which of course are important and critical, but reasonably obvious in what needs to be done. Innovation in the industry creates competitive advantage; sheer execution is increasingly unlikely to do so.
I’m open to counter-persuasion. Flame suit on. What do you think ?
Sunday 22 April 2007
Built To Last
There’s an interesting interview with Niall McCullough, architect, yesterday in the Irish Times weekend magazine, about the new version of his book “Dublin: An Urban History” (unfortunately the Irish Times online is only premium paid-for content so I can only give you this url to the article). There’s also an interesting web site, giving additional histories of
One of the things which I had not realized about Georgian Dublin is that the buildings, which of course are a part of our heritage, were apparently in general not built to last! The article says: “Based on a land-lease system, the terraces and squares were designed to stand for the lifetime of the lease, usually between 40 and 100 years, whereupon they would be torn down and built again.” Perhaps this is one reason why the Georgian Society has had so many challenges in trying to preserve the best of Georgian Dublin for us and for future generations.
Earlier in the last week, I was in Liaoning province in north east
Then on Friday, having traveled back to Europe the previous day, I was with some folks from IONA, visiting one of our customers in the financial services sector, in
As an aside, I do wonder whether “architect” is the appropriate title for somebody like Tom. I tend to think of civil architects as professionals who design buildings. Rather somebody like Tom is really an “urban planner” – he presides over the current and future infrastructure of an entire software city, on which individual applications – buildings – are built.
Anyway. With previous middleware technologies – DCE, DCOM, CORBA, J2EE, etc – and with all due respect to those thousands of technologists world-wide who worked to create these technologies, I think there was always an expectation amongst senior architects (urban planners ?) and IT visionaries, that each of these technologies would fade in time. Sure, each might be strong enough to last for a decade or so, but business logic and applications that were built to exploit any one of these technologies were constructed in the expectation that a more modern, better middleware technology, would emerge within at most a decade. It was perhaps like the relatively short land leases of Georgian Dublin I mentioned above: build your artifacts in the expectation of re-building them a few years later. And so the middleware world proved to be,
Tom postulated that, at long last, enterprise software architects (urban planners ?) can be like the Victorians of over a century ago: laying down infrastructure – whether it be urban water supply, underground and metro train networks, or even sewage pipe networks (what is the best analogy for middleware ? – I leave it to your personal prejudice!) – that will last for a hundred years or so. Is SOA the end of middleware as we know it ? Isn’t SOA good enough to give us a stable infrastructure for at least a hundred years ?
Well, my view is yes, I agree that it is but with one proviso: one has to construct a SOA based (urban-like, city) environment in the expectation that middleware technologies will in fact continue to evolve and change. SOA may mark the end of middleware as we know it, yes Tom, but its chief contribution in this context is its meta-level. In the same way that metadata in a database allows one to reason and manipulate the underlying data, so should a SOA system enable one to reason and manipulate the underlying middleware.
SOA capabilities in frameworks like Artix and its open source companion Celtix allow a de-coupling between business logic and services, and the underlying middleware. In particular, future middleware technologies can be inserted into such frameworks. The meta-level capabilities enable dynamic re-configuration, including for example interface versioning, data versioning, retooling and end-point guardianship.
With SOA, we have indeed reached the end of middleware as we know it, and can now enable enterprise applications which can be built to last.
Wednesday 4 April 2007
SOA Guardianship and SOA Governance
How do you change an entire nation ?
It’s a topical and fascinating discussion to have over a coffee, and clearly is preoccupying a number of people right now. You may be aware elsewhere from this blog that I have been a frequent visitor to
Changing an entire enterprise IT infrastructure should not be as demanding as changing an entire nation, but sometimes I’m sure some of us wonder. The introduction of a Service Oriented Architecture, and the management of a Service Oriented Architecture, at enterprise scale, and almost certainly globally, gives justifiable cause for reflection.
As you may have seen if you follow
The essential difference, IMHO, between a SOA repository and a SOA registry is in fact largely historical, and most vendors in the space – including now
In a SOA environment, the distinction between a development time phase and a runtime phase is arguably a little artificial – and as a result, the industry is moving to the combination of repository and registry functions within the same products. A new service is evolved over its lifetime within the SOA, and must both be designed and maintained within the context of other operational services. Certainly, a new service must be deployed with care, adequate testing, and appropriate “sand-boxing” so that it does not disrupt the operations of other services; but equally it cannot be developed completely in isolation to the operational effectiveness of the current system. SOA systems are inherently loosely coupled, and thus full testing of the interdependency between services usually ultimately happens carefully in an operational setting.
The governance of SOA systems is a key theme of the vendors – again, including
The pragmatic challenge for any SOA architect concerned about the governance of enterprise systems is of course that governance policy mechanisms are almost certainly already widely fragmented across the entire software system. Some policy mechanisms may already be embedded in application code and business logic. Some are in stored procedures, relating to the probity of particular databases and their data management applications. Some are described by business processes, and orchestrated by business process engines. And now, SOA registry/repository vendors come along and pitch “SOA governance” using their respective products.
In a SOA system, how do you implement policy ? How do you affect changes in policy ?
Let’s return for just a second to the analogy of governing a sovereign state. There is both the integrity of the state, and strategic management of it, to consider. By integrity, I mean both a legal framework by which the state operates, including an appropriate set of laws, perhaps set against the framework of a constitution; and an appropriate enforcement mechanism by which the state and its citizens can reasonably assured that laws are obeyed. By strategic management, I mean the set of policies – laws but also fiscal and other incentives - which the state chooses to adopt to, for example, grow its economy, educate its children and its citizens, and care for the health of its population. However, strategic management cannot occur unless the fundamental integrity of the state is assured; equally, an assurance of integrity alone does not necessarily lead to the long term economic and social success of a nation.
From this analogy, I believe it may be useful to draw a distinction between the integrity of a SOA system, and the business policies and processes chosen to be implemented using it. I believe that integrity should be defined and enforced using a “SOA Guardian” – I deliberately introduce a new term – whilst business policy implementation is rightly the domain of business process engines and orchestration.
Interestingly, business process mechanization and SOA service orchestration has probably been given more attention, priority and emphasis, by the SOA industry at large than SOA integrity issues. BPEL and WS-CDL are probably the best known business process mechanization initiatives, for which a number of process engines exist, including in open source form. Indeed, orchestrating business services defined using standardized interfaces is one of the key selling points of SOA, if not the very essence of SOA for some protagonists. Perhaps business process orchestration and choreography has had more attention than SOA integrity, since there may be a perception amongst SOA practitioners that line of business managers, and corporate executives, can relate most to corporate business process enhancements, rather than the more complex holistic concept of SOA.
But “salus populi suprema est lex”:
“SOA Governance”, if it denotes anything at all, should encompass both SOA integrity and strategic management of business services orchestration: I believe that that is in fact what line of business managers, and corporate executives, expect from the SOA industry.
SOA integrity is a critical prerequisite to leveraging a SOA for business process orchestration. The boundary between the two is naturally dithered to produce effects which are actually not really of substance: business process orchestration can be overloaded to attempt to define and enforce integrity; integrity functions can be strained to implement business policy rules. In the spirit of lean software services, I argue however for a separation of concerns: SOA integrity should be focused as a complete baseline from which business policies and procedures can be easily defined, rapidly deployed, and safely evolved.
A SOA Guardian should be driven by rules relating to the integrity and unimpaired operation of the entire system. It certainly includes security concerns, and the authentication and authorization of access from principals to SOA hosted services, based for example on LDAP directories. However it also includes performance concerns, such as load management and brisk responsiveness. It knows the configuration of every service, with technical details such as the size of the allocated thread pools, binding and addressing information, runtimes and containers across the (almost certainly, highly heterogeneous) SOA deployment: it can be used to re-boot any failed service or even ultimately the entire enterprise. It includes transactional integrity, fault and outage management, so that the temporary loss of one or more SOA hosted services does not cause catastrophic failure. It includes management of change, including management of schema and version changes of services and interfaces. In particular, it includes the planning and execution of technology change, so that old technologies and services can be phased out, rationalized and consolidated, without adversely affecting operations.
It is clear that a SOA Guardian has both – forgive me folks, I’m an engineer by background! – sensors and actuators. That is, it collects and presents operational information relating to the integrity of the system. Equally, it is the mechanism for implementing and enforcing changes to the integral operation of the system. It is clear that the Guardian itself needs to be sound and cohesive: a Guardian system will itself likely be federated and fault-resilient. Because of the heterogeneous nature of most SOA environments, a SOA Guardian may well use combined repository and registry products from other vendors in order to administer details of certain proprietary environments. A SOA Guardian is thus much more than the emerging generation of combined repositories and registries.
Having a repository to record decisions and rules relating to the integrity of the system is certainly useful: the library of a parliament records the laws made therein. Having a console from which operational metrics can be gathered is also important: laws should be made in the context of the society they influence, and it is the role of elected representatives to reflect the views of the populace and the current state of society. But a SOA Guardian also includes the enforcement mechanisms to impose change on a system. Some “SOA Governance” products, surprisingly, apparently are weak or even non-existent in this regard.
For a SOA Guardian to enforce a change, it should clearly be desirable that the entire SOA system not be brought to a halt. Changes in security integrity are already routinely dynamically enforceable in most systems. Changes in configuration to improve performance can sometimes be dynamically implemented in some systems. Changes in fault avoidance and outage management can likewise be dynamically enforced in some systems. Changes in schema and interface versions likewise. Changes in technology are perhaps the most difficult to dynamically implement.
A SOA Guardian, given the correct meshing with the middleware substrate, should nevertheless be able to dynamically deploy all such changes to the integrity of an operational SOA, including changes not hitherto envisaged or previously planned. There should nevertheless be no need to update application business logic or business orchestration in making changes to the integrity of the substrate.
Provisioning changes in telecommunications infrastructures is both a science and an art, based on operational experience. Vendors with experience of such realtime provisioning may emerge to become one source of what I have called SOA Guardians.
How do you change an entire nation ? How do you successfully effect evolution of an entire system ? My view is “carefully”, using small incremental and rapidly evaluated steps, along with a strong separation of concerns. The integrity of a SOA implementation is different from the orchestration of business services to exploit a SOA deployment.
“SOA governance” tools may (perhaps, for some vendors even deliberately) confuse the issue. I encourage the industry to instead separate the concept of SOA guardianship from SOA orchestration. A combined repository/registry tool is a step towards a SOA guardian: dynamic binding, late decision, immediate execution and enterprise wide deployment, is also needed for complete integrity and agility.
Tuesday 20 March 2007
Lean Software Services
In my previous post, I discussed a key challenge facing executives, such as the real VP, K-san, who in principle wish to generate value by adopting SOA across their enterprise. I summarized how Lean Manufacturing and Lean Design gives
SOA should impact an entire enterprise’s computing capability as well as business architecture, and needs to be more holistic than specific software development practices such as XP or Scrum. Lean principles can be applied to SOA, whether or not Lean and Agile techniques are also applied to the actual software development of services in SOA. While I personally advocate that software development should be agile and lean, some enterprises may chose not to do so whilst still nevertheless applying Lean thinking to their architecture. The macro principles do not necessarily imply micro ones. The focus of what I call Lean Software Services - LeSS - is empowering people within an organization to focus on their key responsibilities and opportunities, without being overwhelmed by technology choices.
SOA is more than just another technology fad. It is not an issue merely for the CIO: it impacts the CEO and the Board room. It fundamentally impacts the business architecture of an enterprise, as tasks and services undertaken both directly by humans (alone), machines (alone), and humans and machines (acting together), are composed into business choreographies. The composite processes which result are fundamentally different from the classical stove pipes and “silo-ed” applications which are sometimes found as a consequence of history and immediacy: specific business support is needed, specific applications are purchased or built to address these urgent requirements, and once operational these applications become valuable – if not critical – to business operations. Another cause of software silos is merger and acquisition activity, which sometimes result in businesses being urgently combined by operating two software business stacks side by side. It is not that unusual to see operators – for example in a call centre – using multiple physical screens to interact with entirely separate backend systems. SOA is fundamentally and dramatically different: it creates enterprise business value by horizontal impact across the entire business and IT architecture.
SOA embraces a number of software strategies:
· user interfaces, including both traditional screens and new (eg handheld) devices;
· the choreography of software services, and thus the composition of various business logic components to implement business process flows across the enterprise;
· software services, which are the software implementations of those business logic components;
· and finally the underlying substrate of hardware, operating software, databases and middleware.
Separate to each of these four strata, but having influence on all four of them, is the governance of the architecture. Governance decisions arise from both operational and executive concerns: they include for example variation of operational capacity, operational fail-over and backup policies, but also executive led audit trails, authentication policies, and authorization procedures.
In considering Lean principles and SOA, one of the key Lean strategies is the reduction and elimination of waste. Henry Ford introduced “dock to factory floor” inventory avoidance, in which incoming materials were not warehoused but used immediately on the production floor.
The Poppendiecks advocate a policy of delaying commitment – postponing writing any software – until there is clear evidence of precisely what is needed. Fuzzy and uncertain requirements lead to wasted effort and probable “bloatware”. Folklore asserts that the Pareto Principle in general applies to software services: 80% of users use only the same set of 20% of the features of a software system, and that by implication 80% of the features are of marginal value. I personally am unaware of any specific research data that supports the folklore, but it’s a good attention grabber for any executive! Kent Beck and his colleagues in the XP world also advocate cautious commitment until customer expectations and “stories” are well understood.
Folklore also has it that the cost of changing a software program increases (exponentially) with time. Delaying commitment therefore appears to destroy value. Lean Design and Agile/XP on the contrary increase value by delaying commitment, benefiting from modern software technologies which have reduced the costs of change.
Consider a fully flexible airline ticket, allowing you to select any flight over the next month. It is more valuable, and costs more, than a fixed ticket for a specific flight. Think also of stock options: an option to buy or sell something in the future has value right now. Delaying commitment can create value, not destroy it: see the excellent paper by Hakan Erdogmus and John Favaro who apply financial option analysis – including Black-Scholes option valuation - to software development using XP.
Naturally, if you want full flexibility for an airline ticket, you go buy yourself an aircraft of your own. The longer the time you have before an option to do something (fly, buy a stock, develop a software program) lapses, the more valuable the option is: I guess that’s one reason private aircraft are so expensive.
Of course, in the same way that delaying a decision can improve the discounted cash flow (DCF) value of a software implementation, poor subsequent execution can destroy the value by delaying positive cash flow. The Poppendiecks conjunct delayed commitment with rapid execution: once a decision is actually (eventually) made, it is put into practice extremely quickly. It is no good having an expensive private aircraft if it is out of service when you eventually decide to use it.
Seems a bit of a digression from SOA, perhaps ? Bear with me. As I noted above, the focus of LeSS is empowering people in the organization to work on their key responsibilities and opportunities, without being overwhelmed by technology choices. The highest value comes from delaying a decision to the opportune time, and then executing immediately. And then, perhaps later on, making a new decision, and executing immediately.
In fact, the ultimate value results if the decision can be orthogonal to the environment it affects. That is, the decision can be taken at any time, and re-considered at any further time, without restriction from the environment. Your private aircraft should be available any time you want it. To quote from Wikipedia: “Orthogonality guarantees that modifying the technical effect produced by a component of a system neither creates nor propagates side effects to other components of the system. The emergent behaviour of a system consisting of components should be controlled strictly by formal definitions of its logic and not by side effects resulting from poor integration, i.e. non-orthogonal design of modules and interfaces. Orthogonality reduces testing and development time because it is easier to verify designs that neither cause side effects nor depend on them.” A classic example from hardware design is that any computer instruction can be applied to any memory location, without restriction.
A good architecture – particularly a SOA – meets or exceeds expectations for run-time stability and performance, while minimizing the investment – time, financial, and human – to create, modify and maintain it. Orthogonality is an excellent strategy.
As we reflect on the four SOA layers I mentioned above (user interface, choreography, software services, and substrate) together with governance, LeSS insists that all are mutually orthogonal, yielding a separation of concerns. If this is achieved, staff are empowered to focus on their key responsibilities and opportunities, without being overwhelmed by technology choices. Learning can be amplified without corruption or qualification arising from other layers. Decisions can be delayed until appropriate information is known for a specific activity – whether designing a new screen, building a new choreography routine, or implementing a new software service – and then executing quickly, without recourse to factors arising in orthogonal layers. Integrity can be strengthened by building testing, assertions and governance without risk of stress cracks being induced from adjacent layers. In summary, the principles of Lean Design as advocated by the Poppendiecks and others, can be applied to SOA.
It goes further. My VP friend, K-san, noted in my previous blog posting of his concern of the lack of a talent pool for SOA. In particular, what technologies and industry standards do software developers need to implement SOA components ? What skills should I train my staff for, and what skills should I hire against ? Using LeSS, the answer is that software developers coding business logic as software services can pretty much use whatever technology with which they are comfortable – C#, Java, C++, C, Ruby, even Cobol; along with, as appropriate to the selected technology, message definitions, copybooks or IDL. The choreography of the software services which they implement, and the infrastructure to support them, should be – and are, is a LeSS environment - orthogonal concerns.
These are bold claims. Can the SOA strata be orthogonal ?
Let us start on the user interface and choreography layers. The LeSS claim is that user interface specialists and choreographers should independently design and implement. User interface specialists focus on the ergonomics of screen layouts, devices and the human-machine interface. Their primary attention is ensuring efficient dialogue with the software infrastructure, and minimizing misunderstanding resulting from inadequate layouts, poorly designed graphics and inconsistent command activation. Choreographers are concerned with business process flows across a set of collaborating business logic components, implemented as software services. The routines which they design ensure that business procedures are safely implemented and, further, that these can be rapidly extended and modified as business needs dictate. The routines in turn are implemented by a business process engine, of which several alternatives exist – both open source and vendor produced. In practice, most business process engines, certainly of which I am aware, do not impact very much on the user interfaces offered by the software services which they choreograph. Orthogonality at this particular boundary does not appear an issue.
Business logic components implemented as software services frequently require user input and guidance, and present information. In SOA, can the user interface and software services strata also be orthogonal ? In practice, user interfaces designed for browser access today separate “look’n’feel” from actual dialogue, by using for example CSS. Indeed there is a trend to further separation of business logic implementation in SOA, from presentation and dialogue to end users, by new client side tools based on Ajax technology – for example “html scraping”, data mashups and personalized portals. These browser based tools are orthogonal to the enterprise software services with which they interact.
Let us move on. Choreography tools, such as business process engines, ensure appropriate process flows between specific software services implementing business logic. Each software service implements a prescribed interface encapsulating its functionality – indeed this is a primary motivation for SOA, as I noted in my previous post. Thus, so as to control the process flows across software services, a business process engine has to understand the interface definitions. This begins to get at the root cause of my VP friend K-san’s issue: what technology should be used for interface definitions ? Can the choreography and software service strata really be orthogonal ?
A similar issue arises at the boundary between the software services and the underlying enterprise substrate (of middleware, databases, operating systems and hardware): can these two strata also be orthogonal ? Programming language portability used to be a major issue a couple of decades ago: it arguably is not today, due to dynamic languages (such as Ruby and PHP), bytecode interpreters (eg for Java), and platform virtualization technologies (such as VMware and Xen). On the other hand, middleware substrates and software application programming are usually strongly mutually coupled, and are hardly ever orthogonal. Business logic is written to exploit a specific middleware technology, whether it be message definitions for MQ, Tibco Rendezvous or JMS; or FML for Tuxedo; or IDL for CORBA; copybooks for Cobol; or J2EE EJBs or .Net components and WSDL.
LeSS strongly argues that software developers writing business logic to implement software services should be able to do so regardless of specific middleware technologies. Of course, each software project does have to make some choice: a C# developer selects Windows and .Net for her substrate; a mainframe programmer selected Cobol, copybooks and CICS; and a Java developer perhaps 10 years later choses Linux and JMS. Different components may be implemented in different technologies, for reasons relating to history and/or skills availability. It is the role of the middleware substrate to allow such decisions to be independently made, concurrently and also across time.
Please note that I am not arguing for complete transparency of the distributed infrastructure to the software developer. I can recall certain research projects, and certain commercial products back in the early 90s – lets not name them for fear of embarrassment – advocating that any (fine-grained) software interface should be capable of remote invocation, so that a software program can be provisioned arbitrarily across a network of machines! Rather, LeSS asserts that each software service should explicitly identify (at least) one interface as available for use by other software services (regardless of their technology), and quite probably from a remote machine. However the technology – MQ message definitions, CORBA IDL, WSDL, whatever – chosen to do so, may vary from software service to software service.
Please note too that I am not advocating universal availability of all technologies across the entire substrate of SOA: for example, it does not make sense to make MQ and CORBA and JMS and Tuxedo available everywhere across the enterprise. Instead, using Lean principles, a specific technology should be made available only when and specifically where needed: just in time. In particular, making a specific technology available at the core of the system – for example in a central hub – may be a prime target for the (Lean principle of) eliminating waste. When software is available, but infrequently used, it may be an example of sunk cost and poorly realized value. LeSS advocates provisioning and installation of substrate software – e.g. a messaging system like JMS – only at those specific software services (end-points) which need it at this time.
What of the governance of a SOA system ? Lean Manufacturing systems pay extraordinary attention to ensure smooth flow across the factory infrastructure even in the face of fluctuating demand and micro-orders from customers. Resource hubs are in general a source of challenge and problems: de-centralisation and dynamic allocation are fruitful tactics to minimize contention and to anticipate demand waves. LeSS likewise emphasizes that software business logic should be orthogonal from dynamic provisioning of capacity and capability. In general, dependence on central software hubs (usually on dedicated servers) is a poor tactic, even more so if a source of sunk cost and poorly realized value.
Drivers of governance and substrate changes can be very many: for example, security policy improvements; fail-over enhancements; change in choice of messaging protocol so as to reduce cost; change in a data format so as to modernize and improve integration; and re-factoring of a service interface definition schema so as to improve re-use, add clarity and enhance development agility. Versioning of service interfaces is common in the evolution of practical SOA implementations. Such policy enhancements in the governance, and improvements to operation of the enterprise substrate, should not require human intervention in the business software.
My friend K-san was concerned about the lack of skills for SOA: the technology choices available overwhelm a clear selection across the enterprise, leading to risk in staff selection and skill sets. LeSS empowers staff in the organization to focus on their key responsibilities and opportunities, without being overwhelmed by technology choices. Each project team can chose to use the technology with which they are most familiar. Separation of concerns, and orthogonality, can be achieved – today - by appropriate selection of off the shelf products.
Lean principles are being applied by many software development teams worldwide to eliminate waste, delay decision making until facts emerge, execute fast, make integrity inherent, amplify learning, empower teams and drive a holistic view of their system.
Lean principles can also be applied at the macro-level, to entire enterprise architectures. Learning can be amplified without corruption or qualification arising from other layers. Decisions can be delayed until appropriate information is known for a specific activity – whether designing a new screen, building a new choreography routine, or implementing a new software service – and then executing quickly, without recourse to factors arising in orthogonal layers. Integrity can be strengthened by building testing, assertions and governance without risk of stress cracks being induced from adjacent layers. The principles of Lean Design as advocated by the Poppendiecks and others, can be applied to SOA.
Separation of concerns, supported by technology which enables orthogonal decision making, and just in time commitment, is the key to Lean Software Services. In adopting SOA, ensure your chosen supplier or vendors provide you with orthogonal, just in time choices throughout the entire architecture. Then empower your teams to use technology with which they are familiar.
Subscribe to:
Posts (Atom)