What features made CentOS successful
As you likely know, CentOS is a free rebuild of Red Hat Enterprise (RHEL). I wrote a praise on RHEL which pretty much explains why RHEL is great.
I don't think it'll surprise you that there are a couple of RHEL rebuilds:
These rebuilds ("clones" as they are sometimes named) generally take the source rpms published by Red Hat, remove Red Hat's branding (required by Red Hat legal) and recompile the packages. The end result is usually free to use.
However even "just" recompiling is quite some work even if it can be done by individuals which enough time and skill. CentOS gained a lot of users quickly and always had a couple (about 2-6) core maintainers who did a lot of work. As a consequence the creators of Whitebox Linux and Tao Linux (both were RHEL clones as well) recommended switching to CentOS when they discontinued their work.
CentOS always had the major advantage that they promised the same support period as RHEL and strived for maximum compatibility with RHEL. CentOS did not add additional packages or patches unless it was unavoidable (e.g. adding yum to update a system in CentOS 4 instead of using the Red Hat Network).
And last but not least CentOS had a community behind so the project was present on all major Open Source conferences and fares which might have increased its popularity as well.
Chronicles of a Downfall
However even with all that popularity problems began to mount up. CentOS' downfall did not start due to a major catastrophe but was more a slow but steady process. During that time the CentOS core team members failed to react appropriately and so things got worse.
Dag Wieers leaving
In June 2009 Dag Wieers left the CentOS project because of his level of frustration with the CentOS development team. This in itself didn't have a big impact on the project. However it was a clear warning sign. When a highly skilled, well known contributor leaves a project not for the lack of time but because they are not satisfied with the project itself, it means that there is something seriously wrong.
Dag is an experienced RPM packager, providing additional RPMs for Fedora and CentOS for a long time (he's also one of the guys behind RPM Fusion. From personal experience I can say that he's a nice guy, easy to deal with and for sure someone with good 'social' skills.
Now while there were no changes in the project, some of the problems Dag was frustrated about became public just a month later…
Disrupted communication within the CentOS core team
CentOS was founded by Lance Davis in 2004. However Lance stopped contributing a few years afterwards. This resulted in an exceptional situation of an Open Letter to Lance Davis in July 2009.
The public learned that Lance was the sole owner of the CentOS.org domain as well as the CentOS Paypal account. For several years he did not even disclose to other core developers how much money was available and the actual project ("project" being the people doing marketing and engineering) did not benefit at all from monetary donations.
Lance did not answer neither email nor calls even from the inner circle of CentOS developers. Tim Verhoeven even wrote: "We tried to contact Lance multiple times over a period of more then a year."
Ralph Angenent, another CentOS core member, wrote: "This means that the project depends on one person in too many ways. Add to that a person who doesn’t answer calls, isn't available as meetings, doesn't publish things he promised to do - we have a problem. And this is unacceptable. We as a project have to be more transparent. And this is one of the things blocking this."
As many users mentioned this is unacceptable and not only a failure of Lance but IMHO also a project failure because keeping quiet for long means that problems are just brushed under the carpet.
Also Dag Wieer's comment on the whole story is an interesting read and he disclosed that Lance's position and absence was one of the things which drove him to leave the CentOS project.
End of the story was that Lance showed up in a IRC meeting two days later and the issues were resolved. However at least to me as an outsider transparency did not really get better…
Release delays and missing security updates
RHEL clones can publish their updates only after Red Hat naturally. For CentOS regular updates are done in a quite timely manner. However every few months Red Hat releases 'minor updates' (similar to Windows service packages) like 5.1, 5.2, … Many CentOS pay a lot of attention to the time it takes CentOS to get their release out as the update also contains security fixes (comparions with release dates of Scientific Linux are misleading as the latter project continues to publish security fixes even when the full update is not yet released).
Wikipedia has the complete release history with the slack time relative to RHEL. Generall you can see that the slack time increases over time both for CentOS 4 and 5 which means CentOS became slower to follow up on Red Hat (the quick release of CentOS 4.9 is likely because Red Hat doesn't change much anymore for RHEL 4.x at this point in the release cycle).
This culminated in a delay of 85 days to release CentOS 5.6 which is almost 3 months without security updates for a big install base! How is that for an enterprise (CentOS) distribution?
Actually Karanbir Singh mentioned that critical/important security fixes will be published for CentOS 5.5 as updates but in that case he failed by his own standards:
- Firefox and Thunderbird: CVE-2011-0051, CVE-2011-0053, CVE-2011-0053, CVE-2011-0055, CVE-2011-0056, CVE-2011-0057, CVE-2011-0058, CVE-2011-0061, CVE-2011-0062
- libtiff: CVE-2011-0192, CVE-2011-0282
- krb5: CVE-2011-0281
- glibc CVE-2011-0536
- …
As you can see on the CentOS archive there were no updates after the release of RHEL 5.6 (13.01.2011). There was even a LWN article about that: CentOS 5, RHEL 5.6, and security updates.
Compared to that the 242 days of delay to publish CentOS 6.0 is not that significant because there were no existing CentOS 6 users waiting for security updates. Though the enormous time tells something about CentOS' capacities.
Lack of Collaboration
Red Hat publishes their source RPMs which is really nice because makes rebuilding RHEL possible. However rebuilding is still not trivial because of Red Hat's packaging errors (missing build requirements for example) and non-obvious build system requirements. Scientific Linux documented the rebuild problems they encountered. Also rebuilding from scratch (and keeping compatibility with RHEL) requires to follow a specific, non-documented build order.
But despite of that core members just deny that there is any "secret sauce" (1, 2, 3, 4, …).
Questions about privately rebuilding RHEL are frowned upon on the CentOS devel mailing list (for example 1) or people are put of with superficial answers (1, 2).
In the end there is a lot of fear (of unofficial rebuilds) and of giving away their own position:
Red Hat did not tell me how to build it. (...) Why should I tell someone how to build a replacement OS to CentOS.
(Johnny Hughes). Also FUD tactics are used for excuses not to publish things.
There is one statement which summarizes the philosophy pretty well:
Our goal is not a reproducable system so YOU can build software, it is for US to produce software. If you are looking for a distribution that teaches YOU to build things, get Gentoo or Linux From Scratch."
Uh, wait. I always though that sharing knowledge and enabling people to do things on their own was at the very heart of the Free Software Movement? Well, this brings us to the next problem…
CentOS is only a community of users
Many people critizing the CentOS core team because of their attitude towards the community don't understand how Karanbir and Johnny see the CentOS 'community'.
what do you think is your association with CentOS ? do you use it ? if so, you are a part of the community already. None ever said it was BUILT by a bunch of random driveby community members. Its built for a community and around a community.
Community because all the QA team, the forum moderators, the graphics team, the mailing lists and the wiki are all members of the community providing help. (…) CentOS is for the community ... it is not BUILT buy the community.
As you can see, even though CentOS is officially the 'Community Enterprise OS', the term 'community' only means a community of mere users.
"My way or the highway" attitude - failure to understand the essence of voluntary contributions
Karanbir Singh, November 2010:
Lots of people will argue that open source works in a way where people do what they want to do, so you cant tell them what needs doing - and they will do what they want, when they want. Its what many imagine is the 'fun' in the open source way. Fortunately, or unfortunately we dont have that luxury. What comes down the pipe needs to be addressed, sometimes its what we want to do - and sometimes its what needs doing because that's the issue on hand. The process we have in place is mostly finite, with a specified origin and a specified delivery expectation. We need to join those dots. And if people don't want to help with that joining-the-dots effort, they are never going to be a part of the process.
So when people imply that there are lots of potential-contributers who would want to get involved and help etc : What fantasy world are they looking at ? I, or one, would like to get in on that action please.
To me this post illustrates a severe lack of misunderstanding how voluntary contributions work. The first two sentences are mostly correct. However the conclusions he draws from this ("if people don't want to help with that joining-the-dots effort, they are never going to be a part of the process.") are just shortsighted.
The essence of voluntary work is motivation. Potential contributors will have all kinds of motivation when the first try to help. It is crucial to allow them to work how they want initially. Over time some will stick with the project and develop a sense of duty so they care also about non-fun stuff. Others will contribute only one small thing but even the little things will add up as they were shared.
However if you set a process in stone and throw some more or less boring tasks with much explanation over the fence (as Karanbir did initially) it's very unlikely to attract new contributors. So in a way Karanbir is right: There are not many potential contributors – for his definition of potential contributor as someone who is willing to even boring tasks without much questioning.
This ties in "nicely" with an attitude that many people offering help want to do their own/don't care for "the project".
That attitude is absolutely killing. Getting new volunteers in a free software project is usually quite hard. The project members have to work hard (in a social sense, not technically!) to get new developers constantly. Any obstactles in that process will stop people from contributing very quickly.
Well, so it does come as a surprise that there is very few attempts are being make to speed up the release process with the help of outsiders (1) and that there are still significant delays in the release process.
At the same time I find interesting that Dag doubts that QA is the main bottleneck. Which in turn means getting more people involved in actually building CentOS might be a good thing…
Also CentOS chose a way of working which requires extra work to pass on knowledge about the build process. As Johnny wrote even in case of a missing build requirement they don't modify the source rpm.
While I certainly like that CentOS tries not to change the upstream sources I think this is a clear exaggeration: In the end the missing package is use in the build one way or another. The alternative (add the missing BuildRequires line which Red Hat should've added) is quicker (no need to change the build root/restart the build process), passes the knowledge to others (as it is recorded in the source rpm) and looks just like the right approach to the problem. Of course filing bugs in the Red Hat Bugzilla should always be done.
Another limiting factor is that CentOS is very strict about not publishing anything before they are sure that all branding was removed (Karanbir's statement) unlike Scientific Linux which also publishes betas and the like.
A good example for the lack of understanding is a conversation between Douglas McClendon and Karanbir Singh where Karanbir does not accept any of Douglas' more significant proposals related to feedback and generally dismisses most of his ideas pretty strictly ("if you believe that, you are about as far away from CentOS as can be.", "No, that's a serious waste of time and effort.", "Whats so hard to understand about that ?").
Analysis of the failures
I think the current situation can be attributed to the three following project failures:
- Showing the secret sauce only to core members Details of the development process are completely intransparent: How packages are rebuilt, which special handholding is required for which package is not documented because of an attitude "this process works, we need help elsewhere". Many people and companies are interested in this (so there they are potential contributors) but the project doesn't grab them.
- No strategic community building: There is no strategic effort to recruit new team members constantly which is required to keep a project alive in the long run.
- Lack of trust and fear of uncontrolled work: You have to complete predefined QA tasks or something similar before you can even start thinking about fixing (maybe very small) issues. You can't even work on your own and just submit the results because you lack information from the very beginning unless you are a CentOS team member.
Is this the End of CentOS?
Now even with all my ranting about the CentOS project I don't think it is dead already. Certainly some users switched while waiting on CentOS 5.6/6.0 (mostly to Scientific Linux I think). The bad news got some media coverage which has some impact on CentOS' image but the install base is still huge. There are still people doing fares and conferences for CentOS. Karanbir Singh is still doing package rebuilding.
So the project is as alive as a year ago. Its weaknesses are just more obvious than before.
The most important thing keeping CentOS strong at the moment is that there is no real competitor right now. What is needed is a real community-based, open and transparent RHEL clone which does not add any features (like CentOS, even better if there is no such thing as "CentOS extras").
Even with such a contender CentOS will remain an important distribution as long as the key people still invest time to maintain CentOS. So currently it's are bit premature to talk about a 'downfall' of CentOS but there is for sure an eclipse…