I wrote this as part of my work for Cloudsmith. It is a follow up of my earlier posting a few weeks ago.
In summary, Cloudsmith lets you browse and find useful bundles of software components which work together – software playlists – and then download ones of interest. Each one can contain components from different software repositories, and Cloudsmith knows where to go, and how to get to them.
--
Eric S. Raymond wrote a seminal paper in 1997, The Cathedral and the Bazaar, contrasting how Linux emerged from a loosely structured, highly collaborative community or "bazaar" with the traditional approach to developing software (open source or proprietary), in which a select group of cathedral-builders controlled every aspect of design and technology.
Most engineers strive to build at least one great “building” during their career, a monument, a shrine, and a testament to their skill. Today, even "cathedrals" are made from parts found at the bazaars - a huge and growing marketplace for open source components, in which thousands of developers promote parts that many other developers combine into new products. The output of many bazaars -- projects and communities such as the Eclipse Foundation, the Apache Foundation, Google Code, SourceForge, etc. - support and publish the efforts of component development teams. Popular components turn up in multiple bazaars, sometimes as identical copies, other times with subtle variations.
Among the challenges development teams, and their co-worker product management and product marketing teams, face when operating within this new ecosystem are:
* What range of components is currently available? Which bazaars have them; what is their status and quality; how popular are they; where can updates and fixes be found; and so on.
* What works with what? What components, and combinations of components, are available? How do the pieces all fit together, and which bazaars have them?
* How popular is this combination of components compared to that alternative one? How do we know when and if we should update a selection of components, as new versions of the constituent parts emerge?
* How can we build playlists which combine components we built ourselves, with components found in public bazaars and that change in ways we don't control? How can we move to the new version of a public component without breaking what we already have? And how can we keep what we found in the bazaar from getting so intertwined with what we built that we can no longer separate them? What is the best strategy to manage change, when your organisation and your team are increasingly mixing public software components with your proprietary assets?
* Who is going to support us when we use some unique combination which we assembled from public bazaars? Is there anyone out there doing something similar we can learn from?
* We fix and extend components we find in the bazaar, and sometimes create entirely new component playlists of our own. How do we share our work with other developers in our organisation or (assuming our corporate policy allows it) contribute things back to the bazaar for the public good? And assuming we've shared it, how do we know who is using it, and for what?
It is, of course, no longer just an issue of providing a stable, managed foundation on which you and your colleagues can build. There is heightened corporate awareness reaching all the way to the audit committees of publicly quoted companies, due to the multiplicity of software licensing policies. The issue of knowing if, when and how public software assets are being used inside a corporation has become a high concern.
The ability to tailor software should be its value rather than its risk. But in todays world, isn't software componentisation paradoxically slower than it could be, precisely due to the changes, improvements and proliferation offered by the community?
Eric Raymond describes how extremely useful software can result from open collaboration, despite the absence of a clear lead architect directing the project. Today’s software repositories illustrate this principle on a grand scale - they are collections of really good and useful components developed, published, maintained and extended, sometimes by individuals and sometimes by organized teams of collaborators, in a process that can seem almost anarchic compared to conventional internal development.
As bazaars of developed, and contributed, software components have matured, the complexity of fitting together appropriate combinations have increased, as has ensuring that things do not break as each component is maintained.
One example is Eclipse, which is a common integration platform for many components. The recent Ganymede release lists nine application frameworks, six toolsets for embedded and device development, six toolsets for enterprise development, five language IDEs, and five aspects of its rich client platform. All of these elements, in principle, can be used in any combination of choice, although there are seven different official Ganymede packages are listed. Forty-five additional different project downloads are listed. And nine different distributions from member organisations are promoted. It shows an impressive level of community momentum and collective activity, but which of all of the alternatives do you really need for your particular project?
Actually, it is even more complex, because each bazaar stacks up components from its own shelves with components it finds in other bazaars. And you are often building not just one cathedral, but several based on a common set of blueprints. Perhaps you want to develop using Seam rich client Java toolkit? Then you might need a playlist of the Eclipse Classic IDE, JBoss Tools, Seam Core, JBoss AS, and PostgreSQL (with thanks to Stefan Daume for suggesting this particular playlist). But to do so, you may need to visit the Eclipse, JBoss, Seam and Postgres bazaars to put this all together -- unless you can happen to find somebody else who has already done this for you. If you want to build an email spam filter, then maybe a playlist of MySQL, qpsmtpd, my qpsmtpd custom modules, php pages (status), and open flash chart run-time files might be just the job (with thanks to Bjorn Freeman-Benson for this playlist).
Finding out what software components are available is a modest challenge: you can use raw Google, or Google CodeSearch, or Koders, or Krugle, or Codase, or something similar. The more significant challenge is finding out what works with what else to form a useful playlist; then how to get hold of the right version of each these pieces from each of the right bazaars concerned; how popular is this specific playlist of components; and how to get notified if any of the pieces are subsequently changed. If you want to be civic-minded, you might also want to find out how best to contribute original or derivative works back to the remainder of your organisation or community at large.
Our industry is maturing: we really soon should reach the equivalent levels of professional practice as our colleagues in other engineering disciplines, such as electronics hardware and civil engineering. There now is - perhaps at long last - a substantial number of re-usable, well-engineered, components available to all of us, being extended and improved on a daily basis. We should all be able to build cathedrals, and other artifacts, from the components we find. But the vast range of components, coupled with the fluidity of material - software - with which to work, has presented our industry with some new challenges,and which are not as apparent in other engineering disciplines.
Showing posts with label eclipsecon 2007. Show all posts
Showing posts with label eclipsecon 2007. Show all posts
Thursday 18 September 2008
Monday 12 March 2007
Eclipse Spaces
Part of the dilemma with the global software community today is that there are so so many open source repositories and projects. I guess amongst the best known community owned initiatives are Apache, eclipse and SourceForge. But there are very many others, as a quick Google will reveal: for example HotScripts, NetLib, RubyForge, and CodeHaus. Most of course are primarily specific to a particular technology but, nevertheless, large scale projects frequently span several technologies.
How can you easily and quickly put together your own project which relies on software components from several different source code repositories ? If your project works out to be cool, how can you easily and quickly share it with other people who may be interested in it ? Now, Linux distribution vendors such as redhat and ubuntu do a great job packaging together a large set of Linux based components – in ubuntu’s case, over 16,000 according to their web site. Great. But what if I myself want to package together some components which I found useful, in a configuration which those Linux distro-vendors don’t currently offer ?
The problem is generally exacerbated if the components which you want to select come from different repositories and projects, having different repository and dependency technologies – CVS, Maven, Subversion, Ant&Ivy, whatever. The problem is also made more difficult because of version dependencies on your installation environment and other components.
In an earlier blog entry, I wrote how assembling software today is reminiscent of childhood experiences with toy building kits, such as Lego. However, there is some danger that in assembling software components from different sources, you end up with a large pile of individual Lego bricks on the floor! There should be a better way of defining and tracking component assemblies across technologies and repositories.
I was excited to see that last week at eclipsecon, AOL’s Lucas McGregor announced the Eclipse Spaces technology project. I guess the essential concept is simple: just in the same way that MySpace provides online storage for friends to share videos and photos and news and events, there is now an opportunity for developers to informally and easily share online interesting component assemblies with one another. The eclipse project process of course offers a formal, process driven approach to collaboratively developing new software between companies, and I guess Apache has something similarly formal between individuals. But Eclipse Spaces suggests something less rigid, in which components which do not necessarily conform to the precise terms of the eclipse.org license, can be informally assembled as personal and individual projects, especially if they come from other repositories outside of eclipse. And since Xdrive is free, this would be a wonderful added service for AOL to provide to the global software community.
I was also delighted to see that Eclipse Spaces is one application of Buckminister’s publishing and materialisation technology for software component assemblies. Henrik Lindberg, Thomas Hallgren and Filip Hrbek are included as the initial set of committers to Eclipse Spaces, all three from Cloudsmith. I’m also delighted to see Malcolm Sparks on the list too, an old IONAian who joined IONA when Ejbhome was acquired in 1999. And Bjorn Freeman-Benson and Ward Cunningham, both from the eclipse foundation. And finally Dennis O’Flynn, the eclipse Corona project lead from Compuware. Looks like a very solid team.
As a global community, we need a far simpler and more elegant way to quickly build, publish, maintain, acquire, materialise and track interesting aggregations of software components, even if they are cross technology. I’ll watch Eclipse Spaces with interest
Subscribe to:
Posts (Atom)