I wrote this as part of my work for Cloudsmith. It is a follow up of my earlier posting a few weeks ago.
In summary, Cloudsmith lets you browse and find useful bundles of software components which work together – software playlists – and then download ones of interest. Each one can contain components from different software repositories, and Cloudsmith knows where to go, and how to get to them.
--
Eric S. Raymond wrote a seminal paper in 1997, The Cathedral and the Bazaar, contrasting how Linux emerged from a loosely structured, highly collaborative community or "bazaar" with the traditional approach to developing software (open source or proprietary), in which a select group of cathedral-builders controlled every aspect of design and technology.
Most engineers strive to build at least one great “building” during their career, a monument, a shrine, and a testament to their skill. Today, even "cathedrals" are made from parts found at the bazaars - a huge and growing marketplace for open source components, in which thousands of developers promote parts that many other developers combine into new products. The output of many bazaars -- projects and communities such as the Eclipse Foundation, the Apache Foundation, Google Code, SourceForge, etc. - support and publish the efforts of component development teams. Popular components turn up in multiple bazaars, sometimes as identical copies, other times with subtle variations.
Among the challenges development teams, and their co-worker product management and product marketing teams, face when operating within this new ecosystem are:
* What range of components is currently available? Which bazaars have them; what is their status and quality; how popular are they; where can updates and fixes be found; and so on.
* What works with what? What components, and combinations of components, are available? How do the pieces all fit together, and which bazaars have them?
* How popular is this combination of components compared to that alternative one? How do we know when and if we should update a selection of components, as new versions of the constituent parts emerge?
* How can we build playlists which combine components we built ourselves, with components found in public bazaars and that change in ways we don't control? How can we move to the new version of a public component without breaking what we already have? And how can we keep what we found in the bazaar from getting so intertwined with what we built that we can no longer separate them? What is the best strategy to manage change, when your organisation and your team are increasingly mixing public software components with your proprietary assets?
* Who is going to support us when we use some unique combination which we assembled from public bazaars? Is there anyone out there doing something similar we can learn from?
* We fix and extend components we find in the bazaar, and sometimes create entirely new component playlists of our own. How do we share our work with other developers in our organisation or (assuming our corporate policy allows it) contribute things back to the bazaar for the public good? And assuming we've shared it, how do we know who is using it, and for what?
It is, of course, no longer just an issue of providing a stable, managed foundation on which you and your colleagues can build. There is heightened corporate awareness reaching all the way to the audit committees of publicly quoted companies, due to the multiplicity of software licensing policies. The issue of knowing if, when and how public software assets are being used inside a corporation has become a high concern.
The ability to tailor software should be its value rather than its risk. But in todays world, isn't software componentisation paradoxically slower than it could be, precisely due to the changes, improvements and proliferation offered by the community?
Eric Raymond describes how extremely useful software can result from open collaboration, despite the absence of a clear lead architect directing the project. Today’s software repositories illustrate this principle on a grand scale - they are collections of really good and useful components developed, published, maintained and extended, sometimes by individuals and sometimes by organized teams of collaborators, in a process that can seem almost anarchic compared to conventional internal development.
As bazaars of developed, and contributed, software components have matured, the complexity of fitting together appropriate combinations have increased, as has ensuring that things do not break as each component is maintained.
One example is Eclipse, which is a common integration platform for many components. The recent Ganymede release lists nine application frameworks, six toolsets for embedded and device development, six toolsets for enterprise development, five language IDEs, and five aspects of its rich client platform. All of these elements, in principle, can be used in any combination of choice, although there are seven different official Ganymede packages are listed. Forty-five additional different project downloads are listed. And nine different distributions from member organisations are promoted. It shows an impressive level of community momentum and collective activity, but which of all of the alternatives do you really need for your particular project?
Actually, it is even more complex, because each bazaar stacks up components from its own shelves with components it finds in other bazaars. And you are often building not just one cathedral, but several based on a common set of blueprints. Perhaps you want to develop using Seam rich client Java toolkit? Then you might need a playlist of the Eclipse Classic IDE, JBoss Tools, Seam Core, JBoss AS, and PostgreSQL (with thanks to Stefan Daume for suggesting this particular playlist). But to do so, you may need to visit the Eclipse, JBoss, Seam and Postgres bazaars to put this all together -- unless you can happen to find somebody else who has already done this for you. If you want to build an email spam filter, then maybe a playlist of MySQL, qpsmtpd, my qpsmtpd custom modules, php pages (status), and open flash chart run-time files might be just the job (with thanks to Bjorn Freeman-Benson for this playlist).
Finding out what software components are available is a modest challenge: you can use raw Google, or Google CodeSearch, or Koders, or Krugle, or Codase, or something similar. The more significant challenge is finding out what works with what else to form a useful playlist; then how to get hold of the right version of each these pieces from each of the right bazaars concerned; how popular is this specific playlist of components; and how to get notified if any of the pieces are subsequently changed. If you want to be civic-minded, you might also want to find out how best to contribute original or derivative works back to the remainder of your organisation or community at large.
Our industry is maturing: we really soon should reach the equivalent levels of professional practice as our colleagues in other engineering disciplines, such as electronics hardware and civil engineering. There now is - perhaps at long last - a substantial number of re-usable, well-engineered, components available to all of us, being extended and improved on a daily basis. We should all be able to build cathedrals, and other artifacts, from the components we find. But the vast range of components, coupled with the fluidity of material - software - with which to work, has presented our industry with some new challenges,and which are not as apparent in other engineering disciplines.
Showing posts with label rmap. Show all posts
Showing posts with label rmap. Show all posts
Thursday 18 September 2008
Sunday 18 February 2007
Lego instruction sheets and Buckminster RMAPs
In those “black and white days”, as my now 10 year old daughter calls them – the days before colour TV and lcd flat panels – there wasn’t a lot of plastic, and wonderful kits like Lego™ had only just begun appearing. Now my own kids can build towns from Lego kits, with consistent scale across all the buildings, unless of course they choose to make one building much bigger than the others. The roads all smoothly meet up at the junctions, and Lego people and animals and freight all appear wonderfully realistic as scenarios are enthusiastically played out for Dad’s benefit.
What is even more wonderful about toys such as Lego is that children’s fantasies can be cultivated, as they themselves design and construct their own buildings and vehicles and systems, using their imagination to mimic and extend their worlds. They are not limited to the pre-designed model described by each Lego instruction sheet or booklet. Indeed, my teenage son shares designs with his friends, sketching out their own instructions, and photo-copying them for each other. I googled to see whether there is a web site somewhere where people can share their designs and instructions with each other – I could find a few (such as http://staff.science.uva.nl/~leo/lego/ and http://www.telepresence.strath.ac.uk/jen/lego/ and Lego’s own http://factory.lego.com) but no single aggregation: now, there’s an idea if someone hasn’t already done it.
In 1992, when I started building Orbix v1.0 - our CORBA implementation - with John Moreau and Bridget Walsh, the software community was talking about the novel concept of object oriented programming, and some proponents certainly drew parallels with construction kits such as Lego. If civil engineers build using pre-fabricated components, and kids build complex buildings and systems from re-usable Lego bricks, then why can’t the software industry build using a portfolio of re-usable objects and components ? A decade plus later, I guess today we pretty much have learnt this lesson, and much of our complex software artefacts are constructed using re-usable components and common design patterns.
In the “black and white” days, toys were complete. A dolls house was a dolls house. A toy garage was a toy garage. You couldn’t re-assemble it to build something closer to your imaginary perfect garage. A decade ago, software applications and indeed operating systems were likewise monolithic. Sure, you could configure products like SAP R/3 and SVR4, but you couldn’t delve inside them, learn from them, and build systems which were closer to the perfection you craved for your organisation. Today, open source, and software componentisation, has changed the industry, and so you can now assemble and build tailored – and even generally useful – software yourself.
However, now the interesting thing, I think anyway, is where is our – I mean the software world’s – equivalent of those Lego instruction sheets and booklets, those photo-copied personally designed instructions for new models ? Software design patterns are not really the same thing: a software design pattern gives a “how-to” elegantly solve a commonly occurring scenario, regardless of what you are actually building; a Lego instruction sheet, whether direct from Lego or invented by one of your friends, tells you how to build a complete and finished model.
That’s why I got very enthusiastic when Mitch Sonies and Henrik Lindberg approached me last year to talk about Buckminster and Cloudsmith. Buckminster is open source technology which they have donated to the Eclipse foundation -- although Buckminster is not dependent on Eclipse, a “headless” version also exists. A specific software configuration is captured in an RMAP and CQUERY. I like to think of a CQUERY as akin to a Lego instruction sheet, with the RMAP saying which Lego blocks are in which containers and bags. An CQUERY captures, both explicitly but also implicitly by transitive closure, all the necessary software components – other completed models and even individual bricks – needed to build a fully complete system, or component. Given a CQUERY and RMAP, Buckminster then can actually go ahead and build – or “materialize” in Buckminster parlance – the artefact so described, for you on your machine.
Now, you might say, gee Chris, ‘make’ systems have been doing that even since before you were a developer back in the 90s. Well, Buckminster is a bit more than that. What if you want to build something that depends on components from different software repositories: some repositories inhouse in your company or on your own private computer, and some remote across the internet. What if these repositories use different formats – CVS, maven, clearcase ? What if there are version dependencies between the components ? What if there are alternative repositories for the same component ?
Some of the Eclipse projects are looking very hard at Buckminster, and I understand that some have already adopted it. I think there will be quite a bit of discussion about Buckminster at the forthcoming eclipsecon conference. But I don’t want you to think of Buckminster as an Eclipse-based Java-centric tool. Because it isn’t. Back to the Lego analogy – as well as Lego, not only is there Duplo from the same company, but also construction kits from Young Architects and Brio and Imagibrick and others. In our software industry, we’re similarly plagued by different core technologies, leading to bumps and cravesses as you try to butt together one completed system with another. Buckminster at least solves the problem of defining and materializing useful configurations of software – regardless of the technologies – Java, PHP, C# or whatever - used.
One of the most frustrating things about the software industry today, I think, is that although it is much easier now to build interesting new artefacts and systems from re-usable components, it is quite difficult to communicate these configurations and fit them together with other peoples’. Linux distribution vendors, such as RedHat and Ubuntu, do a fine job selecting a specific configuration of several thousand software components, and warranting to a greater or lesser degree that they should all work together. But, sometimes creative developers have different configurations that they seek, or want to define, share and publish to the world. Can interesting new virtual distributions be rapidly defined, communicated and materialized ?
Subscribe to:
Posts (Atom)