Open source network management comparison 2007
Mark Hinkle from Zenoss sent me a link to an interesting document he prepared yesterday.
I think Mark may be over egging Zenoss “clear market leadership” but without any doubt their growth over the last year has been impressive.
Perhaps the most impressive thing to take away is that all of the projects featured have grown over the last year.
Popularity: 53% [?]
Jack Hughes's wry observations of the tech industry from the bottom looking up. 
Subscribe now with RSS
This will span two comments.
Downloads are such a poor way of ranking the success of an open source project, at least when you go out and try to claim “I’m the best”. Sure, a lot of downloads does indicate interest in the software, but does it really translate into use?
For example, Mark would claim that Zenoss is just as good if not better than Nagios, based on his little graph. However, if I google “zenoss” vs. “nagios”, Nagios has 6 times the number of references. Not that Google hits are a better indication of popularity, but it definitely doesn’t jive with the graph.
Next, not every project uses Sourceforge for downloads. At OpenNMS we put them up as a courtesy, but we recommend using yum or apt to install OpenNMS. You go to http://yum.opennms.org/repofiles/, download the proper repo RPM, and then “yum install opennms”. You’re done, but it uses our servers and mirrors and not those on Sourceforge, thus those downloads are not counted. Hyperic does the same thing, and since Nagios is included in so many distros I would guess that less than half of the Nagios downloads come from Sourceforge. I think it is even more so for something like NetSNMP.
Quote | Posted January 3, 2008, 4:55 pmFinally, Sourceforge downloads are not normalized. Zenoss has so many versions, paks, plugins and documents on Sourceforge I couldn’t figure out the minimum number of files I’d have to download just to get it running, but it would be considerably more than one.
Now I’m definitely biased, but I’m thinking that Mark is being a little disingenuous constantly touting the Zenoss Sourceforge ranking. Downloads? When the numbers are one step away from being arbitrary? I met Bill Karpovich at last August’s LinuxWorld, and he’s a really nice guy and I respect what he is doing with the company he founded. A number of people I know like Zenoss, so I can’t imagine that “downloads” is the best they can do.
We do things differently at OpenNMS. If you want some numbers to compare, look at the number of developers on the project. Better yet, subtract those developers whose job is to work on the project. At OpenNMS we have 25, and subtracting the six accounts that are used by opennms.com you still end up with 19. Open source is about the community and adoption, not downloads.
Quote | Posted January 3, 2008, 4:56 pm@Tarus - Thanks for the comment. And apologies for the problems with the blog. We’re investigating
You’re right though…downloads are a very far from perfect performance metric, interesting though nevertheless.
I don’t think open source is a zero sum game…everybody seems to be winning.
Quote | Posted January 3, 2008, 5:12 pmWell as usual Tarus makes some good points but colored with his own biases. Pardon my lengthy reply as I prefer not to have words put into my mouth nor my sincerity questioned.
In this forum especially, downloads are relevant as Jack’s riffed on the topic before:
http://www.openxtra.co.uk/blog/open-source-network-management-download-comparison/).
For Tarus’ benefit I quickly counted only the Zenoss Core downloads from SF it’s still 198,899 which doesn’t include the downloads from all sources like SVN, VMware, rBuilder Online, Debian apt repositories and other venues. Though a visit to the OpenNMS download page on SourceForge is no less confusing to me. I count 32 packages there including things called jrrd, jicmp, and iplike, which are included in my analysis for OpenNMS total downloads.
The rapid adoption of Zenoss is something that I am proud of given the fact that the open source project was launched less than two years ago. Growth year over year is exponential. We are relatively young compared to open source veterans like Nagios or OpenNMS from whom I have learned a lot.
As a measure of success I personally like the SF.Net activity ranking better than downloads rating because it measures “vitality” (I recall Tarus proclaiming to a room full of people at BarCamp Raleigh in 2006 the OpenNMS SF activity ranking). Vitality includes forum activity (I think the 14,000+ forum postings in 18 months is a good indicator), file releases (release early, release often in true open source style) plus the tens of thousands community members subscribed to and participating in our forums and mailing lists. I also like the fact that #zenoss is often packed with users helping each other.
Also I am happy to see the success of the OpenNMS in recruiting open source developers. Zenoss too has had code contributions in the last two months alone from at least twelve community members but really it’s not a contest. Nor is it relevant who gets paid to contribute. I suspect that some of those contributions where made while those community members where being paid by someone, probably while they were doing network management for their employer. Even the humble beginnings of the program that became OpenNMS project where funded by venture capital from Oculan who started development of the program that is now the OpenNMS before Tarus’ set it free. What’s important is that useful open source software is being produced and used.
By the way I think it’s notable that I am far from a OpenNMS hater. I think that it’s a good project and I feel that while I am the chief cheerleader for Zenoss Core, I also have had a bit to do promoting OpenNMS and open source systems management beyond my own project. You see I was in charge of the judging when OpenNMS won the Product Excellence Award for Best Network Monitoring Tool in 2005. I also published a glowing article on their success in LinuxWorld Magazine written by my friend Maria Winslow. Plus there’s my deep dark secret… in my Windows to Linux migration book published by Thompson in 2006, I licensed that case study from Maria to talk about the value that could be provided by open source systems management tools highlighting OpenNMS. Of course that happened in my pre-Zenoss days
Bottom line, the more open source software that gets developed and used by people the better. And as Jack points out EVERBODY seems to be winning. Just as Tarus isn’t shy about blowing his projects horn neither am I. I think there’s plenty of opportunity for my friend Ethan at Nagios, Tarus and his gang at OpenNMS, and Zenoss.
So Tarus, rather than spend any more time disagreeing, can we just knock back a couple of pints and help educate those proprietary software users on the value of open source systems management and let them choose whichever solution they care to?
Quote | Posted January 4, 2008, 4:14 amGrrr, the “quote” feature is broken, so …
Mark writes:
“For Tarus’ benefit I quickly counted only the Zenoss Core downloads from SF it’s still 198,899 which doesn’t include the downloads from all sources like SVN, VMware, rBuilder Online, Debian apt repositories and other venues. Though a visit to the OpenNMS download page on SourceForge is no less confusing to me. I count 32 packages there including things called jrrd, jicmp, and iplike, which are included in my analysis for OpenNMS total downloads.”
Exactly my point. Even if Sourceforge was the only place to download this software, a serious analysis would still have to include not only the number of files required to install the product but number of releases per year, among other things. In raw numbers a project with 3 required files and 10 releases would appear to have many more downloads than a project with 1 file and one release even if they were downloaded by the same number of people. My complaint is that you made no such analysis, yet felt compelled enough to publish the numbers anyway.
“As a measure of success I personally like the SF.Net activity ranking better than downloads rating because it measures “vitality” (I recall Tarus proclaiming to a room full of people at BarCamp Raleigh in 2006 the OpenNMS SF activity ranking).”
I sure did, and it obviously gave you some ideas (grin). My point was, look, we have a pretty high activity ranking - we’re not dead. I no case did I compare our activity ranking with another management project.
Why? Because the system can be gamed. Ross Turk showed me how do to it. Just by logging in to Sourceforge once a day can greatly increase your activity numbers. Using their bug tracker, which Zenoss does and OpenNMS does not, is another way.
Please note that I am seriously not knocking Zenoss here. I just hate marketing, and you touched a nerve with this graph. There are people out there that would look at something like this an infer a relationship of quality, and since the basis for the graph is flawed I feel compelled to speak out.
“can we … help educate those proprietary software users on the value of open source systems management and let them choose whichever solution they care to?”
Certainly. But I don’t see OpenView on your graph. I don’t see Tivoli. All I see are open source projects. Perhaps it would be best to lead by example?
Quote | Posted January 4, 2008, 3:08 pmTarus, I understand your point on marketing however, I think in this case it’s evangelizing (maybe a shade of grey) success of Open Source Management not Systems Management in general. It’s right there in the title of the graph and the document. Like it or not our projects can and do benefit from marketing that helps create large thriving communities to help add features, distribute and improve our software in other ways. I am sure they also drive more business for the OpenNMS Group and I know they do for Zenoss Inc. Plus if you have something cool why not tell the world. If you had the cure for cancer would you just leave the syringes in a cup on your desk with a “Take One” sign?
To your point:
“Certainly. But I don’t see OpenView on your graph. I don’t see Tivoli. All I see are open source projects. Perhaps it would be best to lead by example?”
Well, I don’t include them because I don’t have access to the numbers and to tell you the truth the just aren’t interesting. There is a huge difference between proprietary and open source. I would venture to guess that if the numbers were public the size of our users bases would be as big or bigger than many of the big proprietary players simply because of the barrier to entry (expensive license fees and installation complexity). I talk frequently to Tivoli expert John M. Willis (www.johnmwillis.com) who notes that the Tivoli community is very active just like our communities but to participate you must write a nice fat check to IBM. It’s an exclusive club. I like the idea of take our software and use it and if you think we can help you out then write me a check. If not that’s cool too we still derive other non-monetary benefits from community participation.
Whether it’s Nagios, OpenNMS, Hyperic, or Zenoss. What’s interesting is that we can all provide high quality software by leveraging our communities where barrier to participation is very low. I believe this makes open source software in general a more attractive way to consume software than traditional models. And I am not afraid to tell that to anyone that will listen.
Quote | Posted January 4, 2008, 6:12 pmMark, you seem to be missing my point here. My complaint is not with Zenoss or open source, etc. I am also not trying to open up a debate about how open source projects should behave. My complaint is simply with the graph you sent to Jack and the words “clear market leadership”. I am assuming that is what you wrote to Jack when you sent it to him. Without those words I don’t have a problem. Yes, it is nice to see a lot of downloads, but if you really wanted to make an analysis of them that was meaningful, meaningful enough to reflect on the market, it takes a little more work.
I’m being a bit geeky and anal-retentive here, so for everyone else who has long tired of this discussion I hear that Brittany Spears has done something again - go google it.
First off, anything dealing with downloads that is limited to Sourceforge is inherently flawed, due to the reasons I stated above. Many projects, (including as you mention, Zenoss) distribute their software outside of Sourceforge.
But for the sake of argument, let’s assume that the only place to download this software is Sourceforge, so how would one go about making a meaningful analysis of the download data?
Well, first, let’s pick a file from each project that every user would have to download if they wanted to install it. For Zenoss I chose zenoss-core (both rpms, src rpms, and vmware image). For OpenNMS I chose both opennms-core and the opennms-installer (which contains opennms-core), the vmware image and source code downloads for our latest unstable release 1.3.9 (I am way too lazy to do it for each distro in 1.2). For Hyperic HQ I chose their installer and their source code images. For Nagios, NetSNMP and Zabbix I just chose the main tarball (which is all they have).
So now that we have somewhat normalized the download numbers to a discreet set of files, we’re faced with another problem. Downloads of new software spike when the files are released and then drop off pretty sharply. It would seem that projects with a pretty steady download rate would indicate a higher level of interest, whereas those that spike and drop off would indicate that the core group of users is very interested in the product but that it is not gaining many new ones. What if we normalized the numbers into “downloads per day”? Thus it would seem that every project would want to maximize their downloads per day number.
This is what I get for Zenoss:
Zenoss 1.1.2, released 23Apr07, 12597 downloads, 48.83 dl/day
Zenoss 2.0.6, released 11Sep07, 13700 downloads, 117.09 dl/day
Zenoss 2.1.2, released 19Dec07, 7938 downloads, 441.00 dl/day
All in all, very impressive numbers. For OpenNMS:
OpenNMS 1.3.9, released 28Nov07, 2543 downloads, 65.21 dl/day
So, higher than Zenoss 1.1.2 (which is to be expected since that is an older release) but much lower than the other two Zenoss releases.
However, a good portion of Zenoss downloads are of the VMWare image. I could argue that a serious deployment of a network management solution would not be done using VMWare, so those might just be people trying it out but not running it. Of course, there are a number of valid arguments against that (hence yet another problem with using downloads as a metric), but just for grins let’s remove the VMWare download numbers from both Zenoss and OpenNMS:
Zenoss 2.0.6, released 11Sep07, 6891 downloads, 58.90 dl/day
Zenoss 2.1.2, released 19Dec07, 4073 downloads, 226.28 dl/day
OpenNMS 1.3.9, released 28Nov07, 1751 downloads, 44.90 dl/day
Still pretty impressive for 2.1.2, but that is the newest release of any software in this analysis. You can see that OpenNMS and Zenoss 2.0.6 are much closer now, but again the usefulness of these numbers is questionable.
But the real fun starts when you add in the other projects:
NetSNMP 5.3.2, 11Dec07, 1629 downloads, 62.65 dl/day
Hyperic HQ 3.1.4, 16Nov07, 6229 downloads, 122.14 dl/day
Zabbix 1.4.4, 17Dec07, 4282 downloads, 214.10 dl/day
Nagios 2.10, 21Oct07, 24720 downloads, 321.04 dl/day
As you can see, Nagios is still king, with an amazing number of downloads per day even though the software is months old, and Zabbix is doing well too, although their release is about as new as the latest Zenoss.
I mentioned in the comments above that in the last half of 2007 most of the downloads of OpenNMS have been from servers other than Sourceforge in order to support our yum and apt repos, so I’m happy with how well we’re doing despite that, although we pretty much came in last.
I hope I have once and for all illustrated my complaint with your graph above. The raw download numbers aren’t very useful. There are lies, damn lies and statistics, but at least let’s put a little analysis into it. (grin)
I promise I won’t reply to this thread again, and you can have the last word if you choose.
Quote | Posted January 6, 2008, 11:16 pmBoys, Boys,
We all know there is only one true number that needs to be $counted$.
johnmwillis.com
Quote | Posted January 7, 2008, 1:36 am@John - most things are about money in the end I suppose. It’s nice to see companies trying to do things differently, seeing where the value for them and the customer really lies.
2007 may be the year Tarus thanks his lucky stars he doesn’t have VC money. I doubt that venture money will flow quite as vigorously this year as it did over the last few years especially for businesses not showing signs of cash flow.
Quote | Posted January 8, 2008, 11:30 am