chris horn's blog

What kicked of this blog entry for me were some reflections I had as I chaired a one day conference, with a preceding half day workshop, a couple of weeks ago, which was organized in Galway by Fidelity Investments, on the subject of Web 2.0, and how commercial organizations – such as Fidelity – can benefit from and add value to the global internet community.

A core theme of Web 2.0 is the collective wisdom that results from the network effects of sharing across the community. The wisdom of the crowd is sometimes more than the sum of the individuals therein. Wikipedia is one prime example, since collective knowledge can trigger insights for individuals, which can then augment the collective wisdom: the added knowledge which would not arise if the group did not collaborate together. I recently wrote a blog entry about a similar phenomenon in multi-disciplinary research. Group and individual reasoning can positively feed off each other.

One of the dichotomies of the Web 2.0 phenomenon is on one hand, the power of the group and the collective knowledge of the crowd; and yet on the other the significance of each individual. Yes, collective knowledge such as Wikipedia and del.icio.us (and of course Google!) have emerged, but blogs written and commented by individuals are still a fundamental force. Some may argue that indeed the information in specific blogs written by certain individuals is more valuable, accurate and reliable than that in shared wiki repositories maintained by the amorphous net community. There is room for both individuals and the crowd.

In the race to be successful on the web, the focus is on attracting and retaining users, eyeballs and clicks. For the promoters of a web site - whether a commercial venture, or just a worthy cause for society at large - building and growing a “market” is key. Monitoring usage patterns, listening and reacting to user feedback, all builds momentum: hopefully a tipping point is passed, and the network effect of market momentum reinforces the popularity and acceptance. However: does a mass market strategy play only to the crowd, and not to the individuals therein ?

For Web 2.0 , Chris Anderson introduced the term “long tail” to emphasise deploying:

“customer-self service and algorithmic data management to reach out to the entire web, to the edges and not just the center, to the long tail and not just the head”.

As noted by an Amazon employee quoted by Wikipedia: “We sold more books today that didn't sell at all yesterday, than we sold today of all the books that did sell yesterday” -- read that slowly to yourself again if you hadn’t come across it before..

Further, in his discussion on technology market dynamics, Christensen noted:

“Simply put, when the best firms succeeded, they did so because they listened responsively to their customers and invested aggressively in the technology, products, and manufacturing capabilities that satisfied their customers' next-generation needs. But, paradoxically, when the best firms subsequently failed, it was for the same reasons--they listened responsively to their customers and invested aggressively in the technology, products, and manufacturing capabilities that satisfied their customers' next-generation needs ….But the problem established firms seem unable to confront successfully is that of downward vision and mobility, in terms of the trajectory map….”

Success – at least in some industries, such as the disk drive and earth moving machinery industries which Christensen documents – can equally lay the foundation for failure. Group and individual behaviour can play off each other, creating emerging forces from below.

Combining the Anderson’s exhortation to reach out to the edge, with the warning by Christensen not to be outflanked by emerging disruptive technologies from below, it would seem wise to ensure that a Web 2.0 market strategy explicitly recognises the potential of the individual, as well as the mass market of the group. Even more important is to react to dynamics in the market and to changing tastes. The observation is: that in a global market of mass consumerism, the ability to cater for the vast number of changing individual personal tastes and desires – to personalise and dynamically tailor your products and services – may be much more commercially – and socially – valuable, than volume plays of standard products and services to an anonymous amorphous group.

The long tail changes: it wags. The universe of micro-markets in the long tail is worth addressing – this is Anderson’s observation. In addition, changes in a micro-market may in due course influence the mass market (and blind side you) – this is in essence Christensen’s observation.

Ajit Jaokar was one of the contributors at the Fidelity event in Galway, and he reminded the audience of Tim O’Reilly’s characterization of the Web 2.0 phenomenon:

• the web as a platform;
• harnessing collective intelligence;
• data as the next Intel inside;
• the end of the software release cycle;
• software above the level of a single device; and
• rich user experiences.

I thought about each of these six facets as I reflected on the dynamics of the micro-markets, and list some (not all!) of my observations below: how does Web 2.0 address the wagging long tail ?

One common way of reaching out to the long tail, and monitoring its movements, is “data as the next Intel inside”. Google exploits what its itself calls the ”uniquely democratic nature of the web” to derive data on the click activity of millions of users to drive its PageRank algorithm. I suspect that social networking sites, such as MySpace, Facebook, Bebo and LinkedIn, could likewise exploit their data about networks of people. I’ve always wondered in passing how data regulators, such as the Irish Data Protection Commissioner, view such click gathering and social networking activities…

As Web 2.0 embraces beyond “the level of a single device” – which was Ajit’s theme of his presentation - I observe that telecommunications operators, such as Vodafone, O2 and T-Mobile, collect and record quantities of data, ironically in a large part due to regulatory requirements of data retention legislation in jurisdictions such as Ireland. These vast data archives capture about social communication patterns of populations – it is not just the social networking sites which have such data! There are opportunities, and privacy risks, to exploit these vast patterns to provide commercial value. One can imagine intelligence being generated by mobile phone operators to enable targeted marketing by third parties and content providers…

Quietly collecting mouse click activity, or analyzing phone call patterns and triangularising the locations of their owners, are data driven ways of using to drive for the long tail. Importantly, the data is dynamic and therefore so also is the derived collective intelligence: as trends emerge, or change, so can the offerings tailored for particular sets of individuals. But it seems to me that these approaches may wear a clandestine cloak: there are more explicit ways of using collective intelligence for the wagging long tail.

A successful web site needs stickiness to retain, and track, its audience, including the wagging long tail. A social networking site such as those above in part generates its stickiness because of the network effect of having ones friends, co-workers and colleagues using the same site: most people think it is too cumbersome to re-register oneself and encourage one’s cohort elsewhere. IMHO, the resulting stickiness is in fact rather superficial, rather than systemic: it consequently threatens any purported fiscal value for the site. The SIOC project at DERI in Galway underlines the benefits, and business consequences, of seamless federation of online communities. Stickiness should not be driven by re-registration inertia, but rather by community value and impact - which is one reason why I believe Ammado will succeed.

Another strategy to capture the long tail wags is “harnessing collective intelligence” – another of O’Reilly’s Web 2.0 facets. One explicit way is to allow individuals to define their own collections of interesting items, and then to share and discuss them with other like-minded souls. A simple way to do this is just by tagging items: for example, photos tagged with “Galway” on Flickr will appeal to certain people. But tagging rather quickly loses its impact when trying to find items with a combination of tags, requiring tedious trials of various search criteria; and tag synonyms are a problem in a global world: try “football” in del.icio.us as an example – or even “Galway Hooker” in Flickr!

Rather than expecting people to work out the correct search and query expression across a set of tag values for what they actually want, another way is to let people explicitly define and share their own collections. Hobbyists – such as stamp collectors – have been doing this for years, and a craft worker likewise learns what the set of right tools are for the job from the experiences of his trade. Enabling individuals – and micro-markets and even the crowd - to share what works well together when, is a positive tactic for managing the wagging long tail, and which I wrote about recently in the context of software configurations and Cloudsmith.

When doing a little research for this blog entry, I came across Dan Bricklin’s interesting blog entry on When the Long Tail Wags The Dog. His theme is that “must have” items have more value than those which are less likely to fit the job at hand; and that general purpose items are likely to have most value since they can entertain the dog as well as its long tail. While I don’t disagree, my thoughts about the dynamics of the long tail, and how to work with within it, are a slightly different emphasis.

Reinforcing Bricklin, being relevant to the individuals and small groups in the long tail requires customization and tailoring of general purpose offerings. In the enterprise software space, software vendors have traditionally worked with systems integrators to tailor more appropriate solutions to different niches of the market. With componentization in the software industry, in principle useful aggregations of different constituent parts can more likely be used to tailor specific solutions. Collective intelligence can be harnessed by explicitly sharing these aggregations, as I indicated above.

However, once delivered, installed and put into use, assemblies of software components in production environments have in the past usually been relatively static. The challenge is that the long tail wags: micro-markets and even mass markets change, and production systems need to more easily do so too. One promise of the dynamic module environment of OSGi is to enable dynamic evolution: if collective intelligence can be dynamic as trends emerge, or change, and then suggest new offerings tailored for particular sets of individuals, then perhaps also can production software assemblies in business and enterprise environments be dynamically adjusted. This is a theme which we are working upon in IONA, particularly in the context of our highly dynamic plug-in architecture for Artix and our open source FUSE offerings – see Eric Newcomer’s blog.

But if the world is dynamic, and contains a universe of micro-markets as well as a mass, can offerings not only be tailored so as to be a good fit, but also be priced attractively for each ? What when micro-markets change ? As I wrote about, one industry which faces these challenges is the mobile/cell phone operators, for whom intelligent and responsive bundling of services (SMS and MMS and call rates and roaming charges etc) is competitively critical. The ability to generalise this approach into the world of software components and packages is of interest and relevance to the wagging long tail. LeCayla is one company building a common approach to the issue of dynamically rolling out new charge and billing rates so that software vendors can competitively foster specific micro-markets.

Let me summarise: the internet as a platform enables a global market to be addressed. Web 2.0 uses collective intelligence to play in both the mass markets, and the long tail of numerous micro-markets. There is room for both individuals and the crowd, and group and individual reasoning can positively feed off each other. – creating the ever lurking possibility of being disrupted from below. Scalably addressing both the mass market and the long tail is however not the complete issue: markets change, the long tail wags, and scalably addressing dynamic markets is even harder – but possible.

Footnote: my 7 month old Alsatian, Charlie, has yet to grow into his long long tail. He wags it a lot.

There’s an interesting interview with Niall McCullough, architect, yesterday in the Irish Times weekend magazine, about the new version of his book “Dublin: An Urban History” (unfortunately the Irish Times online is only premium paid-for content so I can only give you this url to the article). There’s also an interesting web site, giving additional histories of Dublin and the patterns which have shaped it at www.reflectingcity.com.

One of the things which I had not realized about Georgian Dublin is that the buildings, which of course are a part of our heritage, were apparently in general not built to last! The article says: “Based on a land-lease system, the terraces and squares were designed to stand for the lifetime of the lease, usually between 40 and 100 years, whereupon they would be torn down and built again.” Perhaps this is one reason why the Georgian Society has had so many challenges in trying to preserve the best of Georgian Dublin for us and for future generations.

Earlier in the last week, I was in Liaoning province in north east China, for Sli Siar, for discussions relating to the “Five Points, One Line” strategy to re-invigorate one of the old industrial parts of China. As you probably know, Chinese government policy (until recently) is that land is only available for lease, and is owned by the State. The lengths of leases are set by national law, and vary between 40 and 70 years, depending on the land use: residential, commercial, industrial etc. In fact, China has been through various land reform policy changes: the 1946 reform in which land was expropriated from the landlords and equitably distributed to individual households in rural villages; the collectivisation period during the Great Leap Forward in the 1950s in which land was grouped into shared communes; and the current system introduced in the late1970s during Deng Xiaoping’s reforms. I say “the current system”, because in fact just last month, the National People’s Congress passed a new property law, after many years of deliberations, which recognizes the status of private property including land. In my own experience, urban planners and developers today in modern China in general expect their buildings and developments to survive the length of leases of the land on which they are built.

Then on Friday, having traveled back to Europe the previous day, I was with some folks from IONA, visiting one of our customers in the financial services sector, in Zurich. The meeting was with their senior architect – let me call him Tom. Tom is responsible for their group-wide, global IT architecture, and the meeting was to discuss SOA strategy. One of the very interesting points he and I discussed was that there is a sense in the enterprise IT industry, that at long last, we collectively in the enterprise software industry are “building to last”.

As an aside, I do wonder whether “architect” is the appropriate title for somebody like Tom. I tend to think of civil architects as professionals who design buildings. Rather somebody like Tom is really an “urban planner” – he presides over the current and future infrastructure of an entire software city, on which individual applications – buildings – are built.

Anyway. With previous middleware technologies – DCE, DCOM, CORBA, J2EE, etc – and with all due respect to those thousands of technologists world-wide who worked to create these technologies, I think there was always an expectation amongst senior architects (urban planners ?) and IT visionaries, that each of these technologies would fade in time. Sure, each might be strong enough to last for a decade or so, but business logic and applications that were built to exploit any one of these technologies were constructed in the expectation that a more modern, better middleware technology, would emerge within at most a decade. It was perhaps like the relatively short land leases of Georgian Dublin I mentioned above: build your artifacts in the expectation of re-building them a few years later. And so the middleware world proved to be,

Tom postulated that, at long last, enterprise software architects (urban planners ?) can be like the Victorians of over a century ago: laying down infrastructure – whether it be urban water supply, underground and metro train networks, or even sewage pipe networks (what is the best analogy for middleware ? – I leave it to your personal prejudice!) – that will last for a hundred years or so. Is SOA the end of middleware as we know it ? Isn’t SOA good enough to give us a stable infrastructure for at least a hundred years ?

Well, my view is yes, I agree that it is but with one proviso: one has to construct a SOA based (urban-like, city) environment in the expectation that middleware technologies will in fact continue to evolve and change. SOA may mark the end of middleware as we know it, yes Tom, but its chief contribution in this context is its meta-level. In the same way that metadata in a database allows one to reason and manipulate the underlying data, so should a SOA system enable one to reason and manipulate the underlying middleware.

SOA capabilities in frameworks like Artix and its open source companion Celtix allow a de-coupling between business logic and services, and the underlying middleware. In particular, future middleware technologies can be inserted into such frameworks. The meta-level capabilities enable dynamic re-configuration, including for example interface versioning, data versioning, retooling and end-point guardianship.

With SOA, we have indeed reached the end of middleware as we know it, and can now enable enterprise applications which can be built to last.

This month is the 10^th anniversary of the IONA IPO n Nasdaq. I can remember vividly being asked by a smart Harvard MBA during the roadshow: “are you guys trying to be a Microsoft or an Oracle ?” Very presumptuous I thought, but the core of the question was our go-to-market strategy for our middleware technology.

Until the IPO, we had had a shrink-wrapped package approach, enabling inbound telesales in IONA and a number of regional distribution channel partners – the “Microsoft approach”. We believed we had introduced a new leverage point into the industry: apparently, nobody had thought about shrink-wrapping middleware before – up until us, middleware had been an enterprise sale with a large amount of systems integrator involvement.

The MBA’s question was whether we would, after the IPO, build a more conventional enterprise field sales force – the “Oracle approach”, and attempt to increase our average selling price beyond its then current sub 10,000USD level.

I had thought about this business strategy issue at length. The stock answer to the question was – audaciously – “both!” It was my belief that we were seeding the enterprise market with a low cost, packaged Trojan horse. However, having seeded opportunities in our customer base, there could well be an opportunity to up-sell additional enterprise products to the basic platform. Some IONAians may remember my BMW theme: “we first sell them a 3-series over the phone, but later they’ll want a 5 or even 7 series which they will want to buy from us face to face. And they must be all part of the same brand” (and as an aside, BMW didn’t yet have a 1 series back in the 90s).

So that’s what we did. We built a field organization after the IPO to sell our high end products, and continued to use the telesales operation to prime the pump. Some MBA types and business faculty now call this “ambidextrous” I think (see for example O’Reilly/Tushman; Celly/Han/Nia; Organizational Science. Another aside: Charles O’Reilly is one of the faculty on the Enterprise Ireland Leadership for Growth programme).

Sometime later, I had the privilege of being invited to join Sanjay Vaswani’s Centre for Corporate Innovation (CCI), which is a number of chapters of CEOs of high technology companies, most of them public, across the US. I believe I was the first European CEO to join. Each quarterly meeting typically consists of a roundtable discussion, followed by a presentation and discussion led by a guest speaker. At one such event, Clayton Christiansen spoke to us about his then latest book “The Innovators Dilemma” – I’m sure you’ve read it. As I listened, I realized that our approach at IONA followed Christiansen’s model: we disrupted the incumbent market from below by a low-cost high-value displacement, and then introduced add-ons as we focused on the enterprise. Shrink wrapped middleware was our successful attack on the incumbents. Christiansen’s general observation and warning was that a company had to be wary of itself being displaced from below by a new entrant – i.e. a company had to continue to be ambidextrous. I was sufficiently and perhaps naively confident that our ability to innovate and “eat our own dog food” would protect us. In retrospect, we were late catching an emerging technology wave in the late 90s – but that’s another story for another blog entry sometime.

Roll on to today. Is open source the weapon to displace incumbent enterprise software by disruption from below ? Is IONA today a closed source enterprise software company, or an open source aggressor ? I guess, as before, the audacious answer is: “Both!”.

Let me explain. I believe that enterprise customers – Fortune 2000 and federal – seek vendors whose competence and sustainability they can trust, to understand and deliver mission critical systems. The full cost of enterprise software is naturally not only the actual license cost but also the investments needed to integrate it with other corporate applications and enterprise data. There is the cost of staff training, not only for the user screens, menus, features and functions, but also for the training of the IT operations staff so as to ensure smooth and efficient operations, and no critical outages. Before making all these investments, a customer wants to ensure that the solution is likely to be valid for some considerable time, while still being maintained and evolved as needs dictate.

In general, risks arise if the system becomes uniquely tailored for that particular customer. Forking of the product source code base by the vendor – a separate derivative version of the source code for each customer – is dangerous, and makes the vendor vulnerable to unviable maintenance costs and skills shortages. It is even worse, of course, should the vendor, or an acquirer of the vendor, drop the product line, or even go out of business completely. A source code escrow account, negotiated at the time of the product commitment, and triggered by a dramatic change in the vendor’s business, is sometimes used to mitigate risk of vendor viability. Nevertheless, very few customers are comfortable with the prospect of having to adopt a substantial foreign code base as a result of a nuclear meltdown.

An enterprise customer in general seeks a vendor who understands the customer’s industry and specific business. The vendor has either verifiably delivered similar solutions before or, if the customer is prepared to be an early adopter, is willing to extend and modify the product – without forking the code base – and work with the customer to deliver competitive business value, despite the risks involved. The professionalism and competence of the sales engagement, particularly the comprehension of the solution by the pre-sales team, are thus vital to establish trust and reassurance.

How does open source play in this arena ? Naked open source – source code downloaded from an internet repository – is usually of little value in an enterprise, except for low value routine commodity utilities such as document format converters. However a number of software vendors package open source with service such as product distribution, maintenance, support and training. Comparatively few of these companies currently have a field organization which is capable of professional sales engagement to rigorously define and commit to the likely business value.

Forking of the source code base is as fraught with risk as it is with a closed source vendor, and clearly should be discouraged and avoided. Having multiple distinct copies of the same open source technology is not helpful. However in my view, the bigger risk is the loss of talented committers to the code base. Although the source code is open and available to all, in most open source projects in practice there is in each case a small number of committers who do the real work. If those committers stop maintaining and extending that code base, then the project quickly stagnates. A committer may stop because of a change in her job responsibilities or the pressure of the day job. But more likely, a committer may stop on a particular project because she feels she has learnt enough from that project, and there is a more exciting new project and new technology starting elsewhere in the open source community.

In my experience, enterprise customers are always keen to understand a vendor’s product roadmap. An appreciation of what likely new releases, with what additional functionality, over the next couple of years, help reassure that the vendor is committed and has a vision for its competitive development, but also help the customer to plan with respect to internal projects and priorities. However, it is in general quite rare to be presented with a road map for an open source project, going out over a couple of years or so, because in general there is no long term responsibility for the project.

In some, but certainly not all, open source projects, the legal ownership of the intellectual property can be opaque. Sometimes, projects include components which were partially supported by the public funds, and jurisdictions can have different perspectives – for example many European Union funded projects typically have an expectation, as a first call, of commercial exploitation. Sometimes, projects include dependencies, or even “cut’n’paste” of code originating from other projects or even from text books. Sometimes projects are forks of others. And sometimes, a closed source vendor will cast doubt on the legitimacy of some open source, as most recently Steve Ballmer did as a result of the Novell partnership. These factors tend to cause enterprise customers, particularly those publicly quoted and under the current regulatory environment of Sarbennes-Oxley and 404, to reflect on the appropriate use of some open source. I am aware of enterprises which proactively take steps to purge open source from their organisations unless there are legally enforceable contracts in place from a vendor warrantying the ownership and responsibility of the associated source code.

Where does this all leave enterprise customers and open source ? Enterprises with a strong technical capability will occasionally embrace open source and assume responsibility for maintenance and extension themselves if necessary. Occasionally, an enterprise will contribute its own open source efforts to work with the community at large. But most enterprises, to the extent that they have a development organization at all, prefer to dedicate their internal resources to helping their business units gain competitive differentiation and advantage.

Now, I hope I haven’t left you with the impression that I reject open source for the enterprise. My emphasis is rather what an open source project should do if it is to be successful in the enterprise. And those thoughts are driving the current “ambidextrous” strategy of IONA, with both our closed source Artix product, and our open source Celtix project. In particular, the commercial investment in Celtix by IONA provides a reasonable guarantee of long term commitment; a product roadmap and direction; a supply of skilled committers; clear legal and IP ownership, with contracts as enforceable as those for closed source; consulting and training services; and the option, should you want it, of commercial grade maintenance and enterprise support – including a free 30day trial period of support. We are also building community support for Celtix initially by contribution to ObjectWeb and subsequently contributing to Apache’s CXF project. Other open source initiatives are also considering Celtix at this time.

Why does IONA provide both open source and closed source forms of similar technology ? Artix has certain enterprise features, including for example support for high end and mainframe technologies, which are currently absent from Celtix. It is an application of the old 80/20 rule: 80% of the customers will use 20% of the potential of the technology – and by implication 20% will need at least 80%. Celtix and Artix play to each segment. And one further consequence is that we are seeing some early adopters of Celtix move on to also subsequently adopt Artix. To re-emphasise above: “we first sell them a 1-series or 3-series over the internet, but later they’ll want a 5 or even 7 series which they will want to buy from us face to face. And they are all part of the same brand”.