Before You Get Mad About The CentOS Stream Change, Think About…

Posted By: fatherlinuxPosted On:
Before You Get Mad About The CentOS Stream Change, Think About…

Change is hard. Explaining that change in a way that makes sense is hard. Not getting frustrated when someone is explaining change to you, or while explaining it is hard. We are all human, so please be patient. But, before you get angry, please read this. I’ve been surfing Twitter, Reddit, and HackerNews threads just like many others. I’ve gathered some major buckets of complaints which seem to come up, and I’d like to address each of them.

Before I start, full disclosure, I work at Red Hat and I’ve been here 9.5 years. I’ve never missed a paycheck. Never. Not one time. It’s always been there on payday. That makes me happy, but just because you work somewhere doesn’t mean you check your brain at the door. I am constantly questioning what we are doing, and coming to my own conclusions. A work relationship is like any other relationship, and it must be evaluated from time to time. Like any rational human being, I do this every now and then. I’m also known for saying my mind, even when the truth is difficult to hear. I think this makes me a decent diplomat for tough conversations. That said, these are my opinions and thoughts, not Red Hat’s.

Let’s have a tough conversation here. There are a lot of complaints and some optimism about the CentOS to CentOS Stream change. Let’s dissect each of what I consider legitimate complaints:

#1 This Was Communicated Poorly

As I said in the introduction, messaging is hard. We are all human, even the people at Red Hat. Do I think it’s possible that this could have been communicated better? Sure. That said, I’ve seen a lot of people tossing hate toward Rich Bowen (CentOS Project shifts focus to CentOS Stream) as well as Chris Wright (CentOS Stream: Building an innovative future for enterprise Linux) who wrote the two main pieces which communicated this change.

That’s not fair. These are both stand up guys. I mean seriously. I met Rich many moons ago because I had read and used his Apache docs. I went up to him and said thank you with nervousness. This guy believes in open source, more than most. He not only uses, he contributes. Big time. Same with Chris. I’ve never seen anyone who gets open source more than Chris. Seriously, when you listen to him speak you can discern how deeply he thinks about it. He is committed to it. If Red Hat ever changed their stance on open source, I am 100% sure that both of these guys would leave. There’s no conspiracy here, not with these guys. I know them both personally, and I believe in both of them. When these guys get up and say something, it’s true to the best of their knowledge at the given time. But, the world changes, and so do strategies. 

Furthermore, this is a big change, and many people were involved, from both the community and from Red Hat. In fact, I’m writing this blog because having a bias towards contribution, and I believe it’s all of our responsibility to try and help communicate this change effectively. Will Red Hat live and learn from this messaging? Yes, I believe so. 

My only ask is, please extend some goodwill to Red Hat and see how this change goes. Give it a chance. I believe it will actually be better in the long run, and more sustainable.

 

#2 Things Changed, I Don’t Like Change

This brings me to the next most common gripe I hear. I get it it. Please try to separate your anger that is a direct result of unwanted change, from the other things discussed in this blog. Every time I’ve ever went through a re-org at work, changed roles, changed companies, bought a house, had a child or remodeled a room in my house, I’ve sort of had this “who moved my cheese” feeling. Everything was one way, now it’s another. Being annoyed by change is legitimate, but ask yourself what the benefits might be?

 

Already, some people have proposed some benefits to this change, and I agree. I’m going to summarize the benefits I’ve seen from three different sources [1][2][3] and add some of my own thoughts:

  1. It makes RHEL development more transparent and reliable [1] – historically, once a RHEL release snapped from Fedora it became a black box. Red Hat tries to solve this by creating Alpha and Beta releases which we share with customers and partners. We spend a tremendous amount of time/energy/money (as a PM I have to answer bunches of questions about Alpha/Beta), but literally almost nobody uses them. It’s expensive and painful. CentOS Stream will change this. Now any CentOS Stream user will have access to and be able to participate in this process. This is amazing.
  2. It provides a way for ISVs and developers to contribute fixes and features [1] – Even when partners and customers would use Betas/Alphas it was difficult for them to contribute changes back. None of the branches/repositories were public. Remember, the source code at git.centos.org is basically read only, downstream code from RHEL. That’s how Red Hat complies with the GPL. Technically we go above and beyond because we are only legally required to provide code to customers, and not required to provide code for BSD/Apache/etc licensed code, only attribution. Nonetheless, this was not a community, this was a code repository. The RHEL code was more akin to Android’s read-only, versus Kubernetes’ or the Linux kernel’s read-write community. With CentOS Stream, this all changes.
  3. It provides a way for the community to provide feedback [1] – Partners have special people dedicated to them called partner managers, and historically these partner managers help partners provide feedback and drive changes to the RHEL repositories. Community CentOS users (and even Fedora users) never had a way to drive changes into RHEL minor releases. Fedora drives changes into major releases, but there was a black box with minor releases. This gave RHEL the perception of being insulated from the upstream community, and it was. With CentOS Stream, this all becomes transparent. It’s a noble goal, and I genuinely believe this is better.
  4. Moving to a rolling stream is a consequence of moving to a cloud native world [2] – this was an interesting one and I hadn’t thought about this. This is a consequence of the world of containers. I always understood the value of containers to control when you absorb updates into your application dependencies, but it also means that a rolling stream of OS updates just doesn’t matter anymore. Now that containers control when the updates occur, why even care about whether it’s a rolling stream or released with version numbers. Do a “podman build” after the minor release drops in RHEL and don’t update again until the next dot release – boom, you have something quite similar to downstream CentOS Stream. A rolling stream lets us move faster and freer.
  5. Don’t underestimate the value of working directly with Red Hat engineering [3] – As Neal points out, CentOS was always a community driven project. Red Hat had extended it’s brand to the CentOS brand, and perhaps mistakenly given the perception that this was a well funded project when in fact it never was. Like every community driven project I have ever seen, there are more wants and desired than time and energy. Maintenance of community driven projects is not wholly unlike non-profits where a sort of depression permeates because it feels like you are never making enough progress, and worse you feel like you’re not being appreciated by the people who benefit from the charity. CentOS stream will be directly integrated with Red Hat engineering providing a direct line to RHEL. These are people paid to do this work, in service of making RHEL better. This aligns incentives and makes it much more sustainable, but more importantly, it creates a natural communication channel between Red Hat engineering and the community which frankly never existed. This will keep RHEL engineering more in touch with the realities of user needs, and provide the community a sense of connectedness. This is a very positive move.
  6. Don’t underestimate the power picking up a thousand new smart, highly motivated power users with a bias towards contribution, not consumption [4] – I haven’t seen anyone highlight this value yet. There are thousands of Solutions Architects, Consultants, Engineers, Product Managers, Product and Technical Marketing Managers, and Evangelists at Red Hat. None of them had an incentive to talk about CentOS much less contribute to it. I haven’t used CentOS in 9 years because I like getting a pay check. I was annoyed when customers wanted to talk to me about CentOS. I never contributed. I didn’t hate it, but I always viewed it as a wash – it garnered users of Red Hat technology, but these users very rarely contributed. I’ve never even seen a CentOS user file an issue for CRI-O, Podman, Buildah, Skopeo, or anything else I work on at Red Hat. These users weren’t even on my radar. About a month ago, I installed CentOS Stream so that our team could collaborate with a customer on some Podman development. I didn’t have to feel dirty about pointing them to a downstream rebuild. With an upstream rebuild, the feedback is all directly useful to RHEL – feature requests, bugs, etc. This is powerful. 

[1]: Thoughts on CentOS Stream by Jim Perrin (@BitIntegrity)

[2]: Unpopular opinion: CentOS switching to Stream is not a bad thing at all by Kristian Köhntopp (@isotopp)

[3]: Don’t underestimate the value of a direct relationship with RHEL engineering by Neal Gompa (@Det_Conan_Kudo)

[4]: My own thoughts.

#3 The Life Cycle Commitment Was Changed

The complaint is that the CentOS wiki originally said that CentOS 8 would be supported until 2029, and that this seems particularly egregious since CentOS 6 was recently deprecated. I totally understand being upset by this if you already had your heart set on using this, but think about why you’re upset? Let’s delve into this from an open source product management perspective. To the best of my knowledge, I am the first person to attempt to talk about open source in the context of creating and capturing value (Blogs/Presentations: The Delicate Art Of Product Management With Open Source) – two concepts that are quite common in the start-up world. 

Let’s talk about this in the first principles of product management. Something free was meeting your needs (market problem). That solution was not lowering the development cost of RHEL (bottom dark red block), and it was also putting price pressure on Red Hat’s ability to capture some of the value that we create. This is a problem long term. Here’s a dirty secret with technology companies, it’s not about making money, it’s about out growing your competitors. Making steady money is fine for a gravel company, but it’s death for a technology company. You have to grow or die. Red Hat reported 69 quarters of consecutive growth before being folded into IBM’s earnings, and continues to grow within the Cloud Business Unit at IBM. This isn’t coincidence. This isn’t luck. We have to keep focusing on growth. We grow or die, just like any technology company.

 

 

Everyone knows that CentOS puts price pressure on RHEL. Let’s all look each other in the eye and discuss it. Many CentOS users speculate that that’s why Red Hat made this change, but let me explain how this isn’t a conspiracy. If successful, CentOS will lower the development cost of RHEL by spreading it across the community. With more eyeballs, all bugs are small. But, it will not remove the price pressure. That’s right, if CentOS Stream is successful, we will still have to compete with ourselves to make money. Why? Because CentOS Stream will likely be ridiculously stable just like CentOS was. That’s fine. That means that as a Product Manager on the RHEL team, I need to be laser focused on providing my customers with what they need. Every. Single. Day. I have to be focused on velocity of the value our team (Podman and container images) provides to RHEL and OpenShift customers.

Challenge accepted. I’ll still make sure RHEL provides value that CentOS Stream doesn’t. For me, it has nothing to do with hamstringing CentOS Stream and everything to do with focusing on creating value in the Buyer’s Problems boxes (light red, and pink boxes in above drawing)!

Could Red Hat have waited until CentOS 9 to make this life cycle change? Sure, I agree that would have ruffled less feathers, but we need these people in the CentOS Stream community. Proprietary engineering is a zero sum game, and the only way to change that is to do community driven development where the cost is socialized among the people who benefit (see also: Why Red Hat is investing in CRI-O and Podman). Any complaints about the stability of CentOS Stream absolutely positively must be looked at in the light of spending money stabilizing downstream CentOS. Without downstream to worry about, upstream will get better. The zero sum portion of the cost of engineering CentOS moves upstream, and becomes socialized. That effectively doubles the chance that CentOS Stream will be great.

#4 The Marks Are Owned By Red Hat

This is a touchy subject. One could make the argument that Red Hat should have never touched CentOS. Over beer, I could be convinced of this argument. I was in sales at the time that Red Hat acquired the CentOS trademarks. Some people hated it, some loved it. I always felt neutral about it. I felt like it was the best existing development platform for OpenStack, which was a problem at the time. The Red Hat contingent working on OpenStack needed something more stable than Fedora because they weren’t interested in doing Operating System development, they were interested in building a platform on top of the operating system. That was a tough problem to solve. Fedora wasn’t the answer because Fedora isn’t the upstream for RHEL minor releases (aka dot releases). I’m sure some people thought about releasing an upstream to RHEL, but it would have taken years and OpenStack was moving at the speed of light. So, Red Hat acquired CentOS.

Again, let’s all look each other in the eye and have a talk like grown ups about this. Once Red Hat touched the CentOS brand, it could never set it free again. There would always and forever be a perception that “Red Hat owns it” – this would forever benefit the CentOS brand and forever incur a cost for Red Hat. Before you slam your laptop shut in anger, let me explain.

First let’s talk about the value that Red Hat gave to the CentOS brand. Before Red Hat acquired the marks, and started paying the contributors, there were always worries that the project might fail one day. Remember in 2009 when Greg Davis disappeared for a while and people freaked out? Yeah, that’s what I’m talking about. There were many similar incidences where updates were slow, rebuilds of new versions took longer than expect, etc, etc. But, when Red Hat took over, all of those worries went away. That made a lot of Red Hat customers feel a lot less worried about running development, test, and even production workloads on CentOS for things they didn’t feel they needed support on.

There’s a big problem that almost nobody understand. The RHEL update stream has a huge value proposition, and most people assume the CentOS update stream is built by the same people, or at least people that can communicate with the people that build it. Let’s be honest. But, the updates in CentOS were never produced by the RHEL engineering team. In fact the packager on my team never, ever talked to the CentOS people, and the CentOS people never talked to him. There was no transfer of knowledge going on. The CentOS updates were produced by a few community members who rebuilt them. Doesn’t matter, the world perceived it as “updates from Red Hat” – I know it with all of my heart. That could never be undone. Never. If we let the marks go, some other people would fire up a rebuild and capture that value in the community. It’s fair that Red Hat didn’t want to give up that value capture.

Second, when Red Hat acquired the CentOS brand, we automatically incurred a cost – forever. Even if it’s just answering questions about how we gave CentOS back to the community and don’t patch it. I remember when I was a Solutions Architect and the CentOS acquisition occurred, there was an expectation from my customers that I would tell them about the value of CentOS. If you think about that, it’s completely insane. Why would a sales person come in and tell you about the value you can capture with a free project? It’s fine to capture the value, but asking Red Hat about it is a cost we incur. Forever and ever. We will always have to answer questions. This is a weird one, and to be honest, I don’t think anybody could foresee how expensive that is. If there’s a cost incurred, there needs to be value captured. With CentOS upstream it’s probably net-neutral from a value perspective. It will lower RHEL development costs, but incur sales costs. That’s better than a net loss.

I genuinely understand why this one makes people mad, but I also understand why Red Hat would have heartburn giving these back to the community. Open source is a tough business.

#5 CentOS Stream Support is Only 5 Years

This is an interesting one. It’s a fair critique, and to be honest, one I hadn’t internalized this when we made this announcement. Let’s be fair, this is basically identical to what Canonical does with Ubuntu LTS. You pay for the later year updates. This has a very good logic to it. If you are running something for that long, it’s probably a cost/benefit analysis and not some upstream development. That means you’re making money off of it. In the parlance of product management, we discuss these concepts as creating value and capturing value. What you might not realize is that whatever workload you are running on Downstream CentOS is creating value for your business, especially in years 6-10. Moreover, you’re likely charging for the service you run on downstream CentOS and capturing value. This is a leaky value chain that isn’t sustainable.

I mean, I’ll admit it, I’m a lazy sysadmin at heart and have had blogs, wikis, and ticket systems run for ten years before, so this isn’t 100% true. Not everyone running old-ass shit is making money, but you get the point. This is a key area of value capture for RHEL and it doesn’t feel wrong to me to charge money here. It feels fair. Neither Ubuntu nor SUSE Enterprise Linux have freely available, highly promoted downstream rebuilds which are supported for 10 years, built and paid for by the companies that sell the upstream products. This actually feels like madness right? 

Tangential to the life cycle complaint, that Red Hat should have waited until RHEL 9 to make this change, I get why people are mad about having an expectation changed. That said, providing maintenance for CentOS Stream 8 until 2024 is still pretty damn good for $0. Also, remember, when you run something for $0, you take full and complete responsibility for it. With open source, you have the power to do whatever you want with it. You’re not trapped. That’s the power of open source, and also the responsibility.

This five year support debate is really about value creation and value capture – what should be given away for free, and what should be charged for. That’s strictly a business decision, even though I know it’s annoying. I’m not some hardcore Libertarian that likes to get on the “responsibility soap box” but I see the problem in the CentOS space and it’s even more egregious in the Linux container space. Companies are consuming container images for $0 without realizing they need to have the expertise to build/rebuild them if the upstream project ever disappears or decides go a different direction. One time, I talked to a partner that wanted to move to UBI, but they didn’t know how to move a bunch of their software because they just consumed container images from upstream without knowing how to rebuild them. They were selling this solution to their customers. Let that sink in. This is a serious problem in the cloud native world. You must ruthlessly track and gauge the risk you incur with by consuming upstream dependencies.

When you consume community open source software like CentOS, you become a product manager like me whether you realize it or not. I don’t mean to be insensitive, but consuming CentOS for $0 instead of buying RHEL means you are responsible for the life cycle even if somebody else was making your life easier by doing it for you. You are only in a bind if you are paying $0 and have no contributors. 

#6 CentOS Stream Will Be Less Stable Than RHEL

There’s a perception that CentOS Stream will be less “stable” than downstream CentOS. This could be rephrased as, “this pisses me off because I always relied on paying RHEL users to absorb these changes, test them, and file bugs for me!” In reality this move basically swaps who gets changes first, but the changes themselves were never “unstable.” Assuming these changes are unstable is essentially a conspiracy theory. Here’s why.

If Red Hat was upsetting RHEL users with a bunch of unstable live testing, how would Red Hat even have paying customers? The fact is, every dot release, RHEL users have been absorbing these same changes without a problem for years and years and years. How is this possible? Do you want the blue pill or the red pill? 

The truth is Neo, with RHEL, there never really were dot releases. Not in the sense of traditional Unix numbering. RHEL has always been quite similar to a rolling release within a major version. It’s just that few people understand it. For years, we’ve encourage customers and partners not to park on dot releases. We always tell them that there is forward compatibility built into the architecture, but they never listen. I genuinely think this comes from how other software vendors have trained them. Many vendors don’t adhere to Semantic Versioning. RHEL does. If you don’t feel like reading that Semantic Versioning, or SemVer is the three digit version number that is so commonly used in software. It works like this:

Given a version number MAJOR.MINOR.PATCH (ex Podman 2.0.5):

  • MAJOR version when you make incompatible API changes,
  • MINOR version when you add functionality in a backwards compatible manner, and
  • PATCH version when you make backwards compatible bug fixes.

RHEL has four levels of compatibility for different pieces of software. If you’ve never heard about this, or don’t understand, check out: Red Hat Enterprise Linux 8: Application Compatibility Guide. All of the key pieces of software in RHEL like the kernel, glibc, openssl, are in compatibility level 1 or 2 which means they are binary compatible within a major release. Stated another way, changes to compatability level 1 or 2 software are in the category of patches, not major or minor change.s The only reason the RHEL dot release increments is because some of the higher level pieces of software in the compatibility level 3 and 4 can change. RHEL dot releases are nothing more than a bunch of patches bundled up on certain day, ran through quality engineering (QE) testing, and tagged together.

This doesn’t change with CentOS Stream. It will follow these exact same rules.

How could it not? CentOS Stream wouldn’t be an upstream for RHEL dot releases if deviated from these rules.  The stability of RHEL never came from testing, it came from this model of software release with the compatibility levels. Think through this carefully. Human and automated QE has never caught more than a small percentage of bugs with any software. The magic of QE test plans, and automated test plans is that the velocity to change can move quicker, break things, fix things, and add a new test so that we don’t break that exact same thing again. There’s an implicit assumption that the things that break were the weakest link, and we need to verify those weakest links. Stated another way, you can’t test your way of crappy code (design, bad programmers, etc). It doesn’t work that way. None of this changes with CentOS Stream.

RHEL’s stability has always come from high quality engineering (open source), back ports, and ABI/API compatibility for level 1 and 2 software. The magic of RHEL stability has always come from the engineering model of how we do things. Our competitors have never been able to achieve the same thing, because they refuse to do the same level of work. Back porting is. Not. Fun. Let’s all be honest. But, it’s super convenient to consume. 

The argument being made is that CentOS Stream won’t be stable because it isn’t tested. That’s patently false. Don’t listen to me, listen to a CentOS contributor named Carl George, “Yes, it is. In fact, large portions are tested twice. Everything released into CS8 has already been through RHEL8’s internal tests as well as the public t_functional test suite.” CentOS Stream is effectively tested the same way RHEL is being tested before it’s released to customers. The RHEL Alphas/Betas are rarely used, and were never the key ingredient to stability. The stability has always come from the architecture and quality of Red Hat engineering.

That won’t change when we swap places in the upstream/downstream. You’re anger likely stems from not understanding this, which is even more frustrating because it means you were consuming, but not contributing.

#7 You’re Not Looking For Opportunities to Improve This Ecosystem

RHEL and CentOS have become tied at the hip from an ecosystem perspective. This especially became true when Red Hat acquired CentOS, but you’re not looking for opportunities to improve this. Let me provide an example from my perspective. 

The Container Tools team in the Red Hat engineering (Podman, Buildah, Skopeo, CRI-O, Red Hat Universal Base Image) releases our main application stream (container-tools:rhel8) about once every 12 weeks. We move faster than most subsystem teams in RHEL. Our upstream projects are still young tools and adding features very quickly. Our customers are dying to get the latest versions of Podman. Often, our customers are happy to get access to new versions of Podman even just a weeks earlier, especially when there is some killer feature they want. Historically, I had to point our RHEL customers to Fedora (which I love), but the configuration is often different than RHEL (ex. cgroups V2 on by default). Worse, our team doesn’t commit to Betas. Since we release every 12 weeks, the version on the Beta would already be old. This means, my customers had to wait until the RHEL dot release, which is painful. 

Before the release of RHEL 8.3, I was able to share Podman 2.2.5 configured on a box that looked pretty much identical to RHEL (no cgroups V2, same registry config, etc) weeks before it dropped in RHEL 8.3. This gave us user feedback that will be used for our next 12 week release, as well as RHEL 8.4. Historically, we had to wait for feedback until RHEL 8.4 dropped. This was very painful because the upstream Podman team is moving so fast and the version on was never up to date. With CentOS, our problem is solved for both our team and our customers.

Furthermore, let me propose an idea. A place every project struggles, is with docs. I am constantly giving user stories to our documentation people, and they are constantly having to stack rank them, prioritize them and complete the pieces that they can. Imagine if we can move our containers guide (BUILDING, RUNNING, AND MANAGING CONTAINERS) upstream and have the CentOS Stream community contribute docs? That would be amazing and socialize the development cost for my team. This is a concrete example of the types of ideas that will come down the pipe if we all work together.

Look for the opportunity, not the challenges. 

 

Conclusion

This change will be difficult for some, and I have empathy for that. Know that we are listening. I’m listening and available on Twitter @fatherlinux. I’m sure there’s still going to be a lot more anger, but hopefully this perspective helps people see the benefits, and perhaps better understand why these changes were made. Give it a year, and lets see how this things turn out. It’s really up to all of us to make this work. I want to leave you with a something funny, and something sad.

Let’s start with funny. Apparently, somebody spent the time to make this website: centos.rip. I’m not going to lie, I literally laughed out loud when I saw it. I mean, like really hard. I’m fine with people trolling Red Hat about this change. All is fair in love and open source. This is funny.

Now, let’s talk about something sad. There’s a change.org petition (Do not destroy CentOS by using it as a RHEL upstream), which I think is on the line because this is for things like wildlife being killed or rain forest being destroyed, but fine. I don’t have any heartburn with people being angry and expressing it, but I do have a problem with people comparing this change to the Charlie Hebdo Shooting.  The Je Suis CentOS meme I’ve seen floating around is just disrespectful to the people that were murdered.  It feels like people are just trying to find things to go viral.

Right now, children are getting blown up in Syria.  If you’re in an emotional state where you feel the urge to make comparisons between a free software change and mass murder, please spend your energy and money on something that really helps the world. I urge you to donate to The Free Burma Rangers. They deliver food and water to people in need, even in war zones where normal NGOs won’t and can’t operate. This is a problem on the scale of Charlie Hebdo, or global warming, or the deforestation of The Amazon. Don’t make these comparisons lightly.

Finally, I’m not trying to dismiss your anger, but moving to CentOS Stream is literally three commands (How do I migrate my CentOS Linux 8 installation to CentOS Stream?). I understand that it’s annoying to have to migrate servers, even if moving is super easy, but put this in context. This CentOS change will cause you to do something that you might not want to do, but there are a lot of potential positives and I hope you will focus on that and try to help make it better. 

69 comments on “Before You Get Mad About The CentOS Stream Change, Think About…”

  1. At least this change isn’t as bad as children being blown up in Syra? Your conclusion was amazing. I would just like to say that this may be the worst explanation from a company employee regarding a controversial decision that I’ve ever seen.

    Reply

    1. I was not making a comparison between this change and children getting blown up in Syria. I’m saying that people creating memes like Je Suis CentOS is just disrespectful to the people that died, and trying to remind people that problems that bad are happening right now, as we have the luxury to have this debate about a free software change. I have not problem with people being mad at Red Hat. I have a problem with righteous indignation stemming from a sense of entitlement with a blatant lack of self awareness for how lucky we are.

      I’ve cleaned up the last paragraph to try and better reflect what I think. I wrote it quickly and I don’t think what I was trying to say was coming through.

      Reply

  2. The last paragraph is dirty pool. Almost all of our (professional computer touchers in the United States) troubles pale in comparison to those faced in war zones. Whatever pressures Red Hat is feeling also don’t matter in comparison.

    Reply

    1. Correct. That’s my point. Comparing a free software change to the Charlie Hebdo massacre is just disrespectful to the people that died. Troll Red Hat, fine. Get mad at Red Hat fine. You can’t control if you’re angry, but you an absolutely control how you respond to it. Have respect for real problems. This truly isn’t as bad as children getting blown up in Syria, so I’m encouraging people to realize when they make these comparisons. It’s a display of righteous indignation that seems to stem from a spoiled sense of entitlement. I absolutely appreciate that my children are not getting blown up, and I do donate money to that caus, so I figured I would use it as an opportunity to highlight it.

      Reply

  3. Thanks for taking the time to write this article. I am genuinely curious to know how much time it took to gather your thoughts, research the referenced articles, and then type/post. Basically how much time do you have in it total?

    As far as an opinion on the article, it’s still sinking in. It was a good read, little dramatic, which helps communicate how personal it is.

    “If successful, CentOS will lower the development cost of RHEL by spreading it across the community.”

    Does Red Hat have resources (paid employees) who’s full time job is to contribute to CentOS Stream? If so, is that because it is upstream and can provide value to RHEL? Were there no resources assigned to CentOS because it was downstream and this provided no value or would result in a net-loss?

    I have been closely following the Red Hat jobs/careers posting for several months and in hindsight can see that positions may have been posted to work on CentOS stream team, if there is one, with out explicitly mentioning it.

    Are there other projects upstream or otherwise that Red Hat assigns employees to? If so, which ones? I’m part of the OKD community and believe there is at least one person fully assigned and maybe another partially assigned.

    I’m not saying it’s a bad thing to have these folks in place, but more clarity/transparency from Red Hat would be beneficial, and help alleviate any future conspiracy theories.

    Reply

    1. I’ll try to address each of your questions. I work in the RHEL product team, so I understand how RHEL is built, tested, released and used by customers. Even with this knowledge, it took me a couple of days to gather my thoughts and publish. Even for people internal at Red Hat, even in the product teams, we each have to internalize this and figure out how it affects us and our products. Obviously, for me, this change is quite close to home.

      Red Hat does pay tons of people for upstream projects, ranging from Kubernetes, to Fedora, CebtOS and thousands of smaller projects.

      I’m not specifically aware if we are hiring more people for CentOS Stream, but I can find out. It definitely doesn’t seem unreasonable or unlikely.

      Red Hat is pretty transparent about hiring people who’s job it is to work upstream. We even have an entire program office dedicated working on and in upstream communities.

      Their old name was Open Source and Standards (OSAS). I think theyncha he’s their name, but like anyone, I struggle with change 🙂

      Reply

  4. Well, I do understand RHEL perspective.

    It makes total sense.

    FOR RHEL. NOT CENTOS.

    You use centos as a test environment for RHEL features, so centos would also receive security patchs first, right? WRONG. You’re a guinea pig, no nice security patchs for you, only testing stuff.

    If a stream rolling release is sooo good, RHEL would use it. You don’t make an incredible revolutionary product and hand over to the next guy. Just to the fall guy.

    You rent a house today with a 10 year contract and next day the owner come and say GTFO. You can’t trust someone like that, you lose respect.

    RHEL base, if much, bears 1% in the market share and will likely decrease and certainly lose relevance.

    I just hope RHEL dies alongside CentOS stream.

    Reply

    1. When you rent a house, you are paying money. RHEL is a house. When you use CentOS, you are sleeping on your beat friend’s couch and eating his favorite pop tarts. If he asks you to throw money in for food every now and then, you don’t get mad. You don’t have a contract.

      If RHEL and Stream dies, so does Rocky Linux. I don’t want to sound cold, but it is. This is business. People don’t buy software/support because they get an emotional reward from it. They do it because it provides concrete value.

      IMHO, CentOS Stream will be better than CentOS and this is a “who moved my cheese” comment.

      Reply

  5. To be honest I quit reading midway, because first part was already quite “moot”.

    The only reason why many companies use CentOS is to piggyback on Redhat’s stability. You change CentOS into something else, no enterprise user will be interested. You say its not fair, but eh I am pretty sure this is also an advantage for Redhat. We have no need for any support other than what is available publicly. But still we do have some products which we pay Redhat for. And dont forget as a CentOS user we also do bug reports for RHEl…

    All the internal issues you mentioned, you could have left CentOS as it is and started Redhat Stream, or Fedora Stream and you would have achieved exactly the same. Because at the end that is exactly what CentOS is going to be. So be a man and just admit you guys killed CentOS for no good reason for its users.

    I am not mad at Redhat nor IBM, this is not the first time a RHEL clone got killed and probably wont be the last.

    I have one criticism towards users of CentOS, when Redhat acquired CentOS, that was the moment that the community should have came up with an alternative, because deep down you know this was going to be a conflict of interest. So I hope we all learned our lesson and dont let it happen again.

    And there’s nothing to be angry about, there’s nothing Redhat nor IBM can do to prevent yet another RHEL clone.

    disclaimer: I am responsible for a fleet of ~5000 baremetal servers with CentOS.

    Reply

    1. +1. My thoughts exactly. Why touch CentOS? Could of just created RHEL Stream instead and you would achieved the same goal.

      Reply

      1. Read my other comments. There is a cost zero sum game. The computer and human resources for downstream CentOS are significant and like the users complaining, we can’t just come up with new budget out of thin air to put the same investment in upstream.

        Reply

  6. I wanted to add, I am thankful for Redhat for all the things the company has done. I started using Redhat in 97 and pretty much during my whole career Redhat played a big role. But you cant blame people for feeling betrayed right now, starting by acquiring CentOS, getting acquired by IBM and then kill CentOS as it is. I wouldnt be surprise people will get suspicious now regarding the other redhat products. And this is sadly not a good thing for Redhat and us as users.

    Reply

    1. Alex,

      I also started using RHL back in the 90s — before there was RHEL! We be old! 🙂 I wouldn’t be where I am today without Red Hat, so am also thankful.

      This feels like a betrayal of trust b/c it’s taken right out of the Embrace-Extend-Extenguish playbook: embrace CentOS Board, extend with CentOS Stream, extinguish CentOS Linux. Whose to blame? That’s what I still want to know…

      Jp

      Reply

      1. I think that’s a fair criticism, and I definitely understand the sentiment. I just don’t think it’s extinguished. In fact, I think it will burn much brighter, not be extinguished. Lets look back at this in a year and see where CentOS Stream goes.

        Reply

  7. I’ve only touched CentOS lightly a few times. My choice of server OS is OpenBSD. I’ve used Fedora frequently. But my Linux distro of choice is Slackware. Granted, it’s been at least four years since the 14.2 release. And CentOS users are complaining?!?? If there’s anyone who should squawk, it’s us Slackware users. The repositories are still active. But trying to stay true to Slackware is almost comparable to the Trump supporters not budging on the outcome of the 2020 Election.

    Reply

    1. All fair comments, and I see why people, especially in 2020, are suspect of changes like this. This was NOT driven by IBM. They literally play no tactical role in our product and project decisions.

      This is all public knowledge, but IBM’s biggest push is on OpenShift to support a large ecosystem of software. They are not driving changes in CentOS, etc.

      Reply

  8. this line sums it up

    Here’s a dirty secret with technology companies, it’s not about making money, it’s about out growing your competitors. Making steady money is fine for a gravel company, but it’s death for a technology company.

    yes very very very true

    Reply

  9. Hi,
    Thanks for the detail explanation and covering redhat. As one of the user correctly said, redhat would have created a RHEL stream as upstream project and leave the CentOS as it is.

    So, surely it’s no secret from Redhat/IBM to get rid of a solid OS so that more RHEL subscriptions can be sold.

    Thanks.

    Reply

    1. It’s more about moving the community that sticks around upstream. If there’s going to be hemmorhaging, it’s best to make the change fast and crisp. IMHO, I think it’s 6 of one and half a dozen of another. Waiting 9 more years to make this change wod eat up a ton of resources from Red Hat.

      Reply

  10. The biggest question still has not been addressed or answered. Would you utilize centos steam as a production product in your organization? Everything leans toward development and testing, but none of this inspires use of centos in a corporate production environment. So now companies using centos in production will hit a fork in the road: pay to switch to RHEL or switch to another product.

    Reply

    1. I can’t answer that for you. I never could. IMHO, running “production workloads” – aka workloads that either make money or save your company money – on CentOS was crazy. That hasn’t changed one way or another.

      If you can’t answer that question after reading this blog, I’m genuinely concerned that you should have never a made a decision to run on CentOS.

      That said, if you “though” CentOS was good enough before, you’ll likely think CebtOS Stream is good enough now. It’s three commands. Try it out. I think you’ll be surprised. Our customers have been absorbing this change first for years, and they never complained.

      Reply

      1. We use CentOS in production. We paid for RHEL for several years but never used support because we support ourselves. We would gladly pay a small sum to use RHEL without support but with updates (access to the repositories).

        Reply

        1. That’s already possible today. If you talk to a sales rep there are ridiculous deals for update only support and Developer subs that are “lightly supported” – that said, they are going to announce more streamlined versions of these. I don’t even know the details yet, but stay tuned (my focus is on Containers, not our SKUs and support).

          Reply

          1. How about for customers who dont utilize support, but have thousands of machines? A developer sub wont due. If I turn around and ask management for 4.8 million dollars so we can license everything, do you think I wouldn’t be laughed out of the room?
            We use RHEL on our backend infra, but we dont use it in remote facilities because the sheer number of them would hurt us in licensing costs. Watch what happens, everyone will move to Rocky and/or Cloud Linux. The engineers who were running/learning your product to support customers around the world will start to fizzle out. You closed the doors to us at work, why would we experiment on our own time?

            Why would running production workloads on CentOS be crazy? A lot of us are Red Hat Certified Engineers, do you think Red Hat support is more capable? Outside of that, there is no difference. Its a rebuild.

          2. You’re assumptions are way off. Thousands of developer subs might cost a few thousand dollars a year even with today’s programs. The fact that you don’t know that is the one thing I’ll blame Red Hat for.

            Furthermore there are all kinds of pricing models for HPC and remote fleets.

            I respect RHCEs a lot. I did my RHCA back in the day, and I’m still proud of that. It’s not about being better then RH support or worse, it’s about the relationship that gives you access to drive change. Engineers are not always good at assessing tail risk, that’s why we have finance and procurement people in big companies.

    1. This was never a lifecycle commitment from Red Hat. This was a life cycle commitment from a community project, not Red Hat. To get a commitment, you also have to be committed. I’m sure you have a significant other and understand how this works.

      Reply

      1. This statement only means your article on CentOS Stream is as bogus as the EOL offered by RedHat for CentOS 8.

        Since you talked about that person’s relationship I must remind you an important aspect in a healthy relationship. Trust. It is earned when action meets words.

        Reply

        1. That life cycle “commitment” was a page in a wiki written by a CentOS community member who may or may not have been paid by Red Hat, so it’s way more nuanced than your making it sound. That said, you can still move to Stream 8 for 4 more years of maintenance. Then, you can use the LEAP tool to upgrade to Stream 9, in place. I understand the trust thing, but imagine these two scenarios:

          1. We move everyone over to Stream 12 months from now, and spend the next 8 years of the life cycle focusing on making it better
          2. We wait nine more years, trying to support both, then move over in RHEL 9, and 10, and 11 which will all be in maintenance during the CentOS 8 “life cycle” as you refer to it.

          Which one sounds like it will succeed. I’m not sure if you noticed, but RHEL sped up with the release of RHEL 8. It’s now moving way faster than it ever has. RHEL moved to a 3 year life between major versions come out. That means that there will literally be four concurrent releases, maybe 4 or 5 with EUS/ELS support. That’s an insane amount of maintenance. And, you expect us to extend that to the free downstream rebuild? For 8 more years? This is a zero sum game from an engineering perspective. If e do that, I predict that CentOS Stream would be under massive pressure and would probably have stability problems in some of the releases. I have a sigh of relief that we are moving to Stream 8 as soon as possible. This will conserve resources and give Red Hat the ability to move faster while at the same time provide stability. If we waited until 2029, we would die on the vine. This is a cloud native world, and we have to move faster.

          Reply

        2. As someone in the reddit comments outlined “implied contract”. There 100% was an implied contract based on the fact that CentOS and RHEL had a very established pattern of the same EOL date.

          Reply

  12. @fatherlinux, thanks for the thoughtful post. I’ve been considering this situation for a day, and it seems to me that you and Red Hat have overlooked something important. You’re killing the goose that lays golden eggs, not just for Red Hat but for the entire RH ecosystem.

    When I started my career, IBM and Microsoft were the default answers in most companies. Today, Red Hat is the default answer for almost any serious discussion. That didn’t happen by accident, and it didn’t happen solely because Red Hat’s solutions are technically superior.

    It happened because of what CentOS is, and has done, for Red Hat.

    The idea that a teenager could install an enterprise-grade OS on a computer in his bedroom, play with it, and learn it was critical to the success of Red Hat. Free-as-in-beer software attracted young developers to Red Hat, and that led to the much larger understanding of free-as-in-speech that underlies most modern software.

    It also hooked them on the Red Hat environment, tools, and way of thinking. Many hobby toolsets are built and tested on CentOS. Some of those hobbies become major players, which in turn reinforce Red Hat’s position at the center of the ecology. Centos Stream is useful, but it cannot fill this role.

    The pricing argument doesn’t hold water for me; those sales were never going to happen, and they won’t happen now. Either someone rebuilds CentOS under another name, or the hobbyists will use debian or Ubuntu.

    Only time will tell us whether Red Hat’s prediction that the OS no longer matters will come true before the next generation of coders decides that Red Hat no longer matters.

    Reply

    1. I appreciate your thoughts, but strongly disagree on cause and effect. CentOS did NOT cause RHEL.

      A teenager can literally install an enterprise OS on their computer in their bedroom. CentOS Stream changes none of that. In fact that teenager is now a RHEL contributor giving me (old Scott McCarty) ideas about how to make RHEL better now. That never happened with downstream CentOS.

      Who on earth could CentOS Stream not fill that roll? That makes no sense to me.

      We agree on “those sales” – I stated that in the article. CentOS Stream will still put price pressure on RHEL. Let’s be honest with that. I’m totally fine with that. It’s even healthy for the market place and Red Hat OMs like me. We can’t get lazy.

      The OS matters more than ever, and we need more contributors, even if that sacrifices users who are mad.

      Reply

    2. Teenagers in their basement can still sign up for a completely free Redhat Developer account and download and install the real RHEL Server distribution completely free. Not rebuilt RHEL, the actual RHEL served directly from the Redhat network. I know, because I’ve done this (even though I’m not a teenager and I don’t live in a basement).

      The more I’ve thought about this the more convinced I am that this change is a total non-issue. I DON’T work for Redhat. But CentOS Stream is literally just the latest RHEL except that it’s about one week ahead of RHEL. The old CentOS was at its worst up to three months behind! One week ahead or two months behind, which would you prefer? I know I would prefer the former. If you have some use case that’s not satisfied by what Redhat offers (you need to be on an exact RHEL point release long term, you need more than five years support, you can’t use free RHEL on a free Developer account) then you should really be paying Redhat.

      And if the community needs someone to fill the role of the old CentOS, they’re welcome to do that, and it looks like they already are working on it. But that will happen on their own dime. Redhat doesn’t owe anyone this.

      Reply

      1. I’ve also had this same exact realization as I’ve wrapped my brain around this change. It basically feels like three commands, and no more issue. What am I missing? People had a warm and fuzzy being downstream from paying users. That’s the main epiphany I’ve had.

        Reply

          1. I have used Debian and Ubuntu. I still do use them. But they are no substitute for this particular use case. Debian has three years of security updates and two additional years of “long-term support.” Ubuntu provides 2.5 years of “hardware and maintenance updates”, 2.5 additional years of “maintenance updates”, and three additional years of “extended security maintenance.” Note that the three additional years of ESM for Ubuntu require either a monetary payment or (free) registration and subscription with usage restrictions in the free registration case (namely, personal use only), very similar to what Redhat provides with a Redhat Developer subscription which gets you access to RHEL and its 10-year support lifespan. There is nothing in the .deb world that compares to the previously offered 10-year CentOS support life. The new(ish) CentOS Stream offering is very comparable to Ubuntu LTS: it’s five years of updates using CI delivery.
            In summary, the only thing that changes is that Redhat’s registration-free offering now matches Debian/Ubuntu instead of greatly exceeding it. If you’re willing to submit to registration, Redhat gives you 10 years, Ubuntu 8 years, and Debian 5 years. Also, there still exist third-party downstream rpm-based alternatives (Oracle Linux, Rocky Linux) with the full 10 years of update support. The 10 year support lifespan isn’t anything that deb-based distributions have ever come close to achieving.

  13. You’ve re-defined what a rolling release distro is to fit the RH announcement about CentOS 8.

    If we are to believe that RHEL really is (now, via v8) a rolling release distro, users would expect that is has:

    1. no . versioned releases
    2. no pinning to release option
    3. no version specific patches
    4. only support for latest
    5. world updates & upgrades
    6. update early & update often

    AFAIK, RHEL8 still does (1), (2), (3): it has major.minor versions, allows version pinning, releases version specific patches. You’d probably argue that RH strongly encourages & facilitates users to do (5) and (6), and that’s fine. But I don’t think someone can argue that RHEL has (4) without ignoring EUS / LTS and overlapping major releases. More importantly, (1)-(3) fly at the face of how users manage rolling release distros.

    Arch is my daily driver, and I’ve ran Gentoo in a previous life. Both are considered de facto rolling release distros that we can compare to point release distros, like RHEL/CentOS. The comparison is stark when it comes to how users maintain systems, like upgrades. There’s a reason that the Gentoo package manager update command (emerge) includes the parameter “@world” — because rolling distro users are updating every package globally, and every package to latest.

    Pinning a rolling release distro is either a) not-possible, and/or b) an anti-pattern. You might agree on (b) even for RHEL but CentOS still made it possible (and expected?) to pin versions, and users have re-stated their uses cases for doing this pinning. The mechanism for them doing this on CentOS is yum --releasever=N.N and evidently RHEL8 still has subscription-manager release --set=N.N, even if discouraged. This pinning flies at the face of rolling releases. Fundamentally, it’s just not accurate to even say “download Arch v10.1” because speaking like that implies a point release. A users can, however, “download RHEL8.3”. Going beyond fundamental definitions, rolling release distros also make pinning difficult. Gentoo attempts to help some users maintain older packages out of sync with latest but Arch goes so far as to state partial, non-global upgrades are “not supported”. Some would say Arch is the purer rolling release distro — however, just from poking around on this, I did find that Arch’s official package repo includes a LTS of the kernel. I’ve not used it, but I’m also not aiming for stability or LTS support.

    Finally, the support — commercial or community support — of rolling releases is expected on latest, and latest only. No LTS, no EUS. If a user falls behind, they’re expected to upgrade; and because there are no point releases, that means all packages to latest in repo. And to beat the horse dead: “latest” for rolling distros does not mean the largest minor/dot release of a major release because those don’t exist.

    I don’t think you’ll find a lot of folks defending your re-definition of RHEL as a rolling release. Supposedly Stream really is rolling but I’ve not tried it to know, and am all ears on that. For RHEL, I don’t think customers will go for a rolling release because they expect:

    1. standardizing (SOE) on . versioned releases
    2. pinning to release options
    3. version specific patches
    4. support on older releases (LTS/EUS)

    Perhaps RHEL9 will get rid of (1)-(4)? And become really rolling 🙂

    Reply

    1. So, as a former, and long time, Gentoo user, I apareciste tour thorough responde 🙂

      Inconrir with almost everytjing You said, but for the point of this conversation it’s too philosophical.

      Technically, RHEL isn’t a rolling release because you do have to upgrade between major versions. Historically, this was a migration but 7+ is now pretty much an in place upgrade, but I digress.

      It’s definitely always been an anti-pattern to lock versions, but yes we let people, and we even sell them EUS. We don’t want to, but they literally demand it. This is how the world works 🙂

      Sigh, I have contemplated proposing that we should move to a full rolling release in RHEL 9, but I’m not convinced the world is ready for it. Especially, after this announcement.

      Also, by your definition, stream basically isn’t a rollingnreleas either. All the major versions of software like the kernel or glibc are locked and back ported. So, it’s not a rolling release, but effectively the same from an update perspective. A yum update always works. That’s all I meant by casually throwing rolling around.

      Reply

    2. Either way, my technical brain now went down the rat hole. I changed that section around to not refer to RHEL as a rolling stream. I want to be as precise as I can. Instead, I highlighted the Application Compatibility Guide and the levels of compatibility RHEL offers combined with the SemVar system so that people can understand what this change really means.

      Reply

    1. Rich was pretty famous in the Apache community. He wrote the most insanely good docs for some of the Apache modules. When I first met Rich, I had never really work with upstream contributors before. I was basically star struck. I was just some red neck sysadmin from Akron, OH, and to me it was kind of like meeting a famous person to me. That’s all I meant. Rich is a super good guy.

      Rich is very much a typical Red Hatter in that I know he will call balderdash when he smells something not right.

      Internally at Red Hat, we fight like cats and dogs about things we believe in, or don’t believe in. There is always a debate going on about something. It’s the nature of open source. One time, when I was fairly new, Lee Congdon our old CIO told everyone we were moving to Google Docs, and people put up a digital petition on a local server. They were mad becasue it wasn’t open source, and it had 700 people within hours or a day. Lee ended up reversing course. A few years later we moved anyway, but their opinions mattered. I’d never worked at a place where you could literally have an anarchy-like revolt. I used to joke that if you did that at IBM, you’d get fired, but so far so good, lol. Not a single Red Hatter that I know has been fired, and we still debate everything.

      That’s all I meant with that reference. Rich would not sell the world on some evil plan. He’s not that guy. Period.

      Reply

  16. I understand Red Hat / IBM, after all, everyone wants to make money, it couldn’t be different with this companies. However, you changed the rules of the game in the middle of the game and this is very dirty… very dirty. Centos 8 should go until 2029, no matter if it’s free or not, or was there a notice on his license saying that he could or could not go until 2029 ??? I believe there was not.
    Another unfortunate point when trying to do all this justification – there is already something with the same purpose as the centos stream (Fedora), it could very well control the beta and alpha by version numbers, you don’t need to kill an entire distribution for that purpose.
    The only point I agree with you is to wait and see how the story will end in a year, we have good options like Oracle Linux, but if I have to pay for Red Hat / IBM, so be it – which I don’t want and having to read another long text and discover that in the end there is none.
    And again… DO NOT CHANGE THE RULES OF THE GAME IN THE MIDDLE OF THE GAME.

    Reply

    1. As I’ve said other places. RHEL has sped up massively with 3 years between major versions and six months between minor versions. That means the 3-6 major versions are in some phase of maintenance at any given time. Waiting until 2029 to shift focus to stream would have been death by a thousand paper cuts. Then, CentOS would have went away anyway, because we would be put out of business.

      Reply

  17. The CentOS change may be a significant strategic mistake.

    The tone of RedHat employees I would say sees the status quo CentOS as an unreasonable burden delivering no benefit to RedHat, and thus they need to become free development and test for RedHat. Meanwhile, mere mortal users need to be paying or even if permitted to do it freely, must be tracked through registration. RHEL clones frustrate as it is freeloading on RedHat efforts, which may be true to some extent, but those freeloaders do provide value. The reaction to the disappointment in CentOS fail to consider “What is in it for the users?”

    This is a repeat of 2004, where RedHat in the face of being *the* Linux distribution decided freeloaders should be testers (Fedora) and stable users should pay for it (RHEL). One major consequence of that act was Ubuntu suddenly making huge inroads out of nowhere, as the RH users had to go *somewhere*. I would argue the bleeding of mindshare was only staunched by CentOS and projects like it. To the extent some had not yet bailed on RedHat yet, that was a life raft to keep RedHat relevant in the onslaught of Ubuntu (who now have the majority share, but CentOS is big enough to command mindshare). If RedHat had perhaps done the same change, but let ‘Enterprise’ be cost and registration free, then Ubuntu may not have happened, and RedHat would be the vast majority of Linux installs today. As it stands, I’ve been in Canonical sales presentations, and they make a compelling argument about how they have much more share than RedHat, so technically they seem to be in a compelling position to be listened to, and perhaps in practice the more authoritative.

    At this pace, I expect the community that RedHat wants to exploit will evaporate, moving wholly over to Ubuntu where things are a bit more straightforward (free distro is also the paid distro) with greater pressure. Sure Rocky Linux may pick up the slack, but the CentOS scenario certainly has people thinking a move wholesale away form the RedHat ecosystem is a safer bet for long term strategy.

    RedHat does a lot and employs a great deal of people, but they should not dismiss how much CentOS keeps their paid offering relevant. So long as Canonical, SuSE, and Oracle are participating, there are other viable options and with less than 2% of hosting share without the free user bump, it’s hard to imagine people continuing to think of RedHat as an authority.

    The sane thing would be to adopt the Oracle model, RedHat would be free for self-support, and subscriptions are just for support. RedHat has more credibility, the free user bump would proudly bear the RedHat name directly, and Canonical would no longer have sales presentations showing how embarrassingly small RedHat’s share is. Without that 19% free user chunk, I would expect RedHat to see even their 1.8% share erode over time.

    Reply

    1. I welcome the competition from SUSE, Canonical, and Oracle. FYI, you have to pay Canonical to get LTS updates after year 5. We actually moving to be more inline with them on this. Let’s look back in a year, and see how CentOS Stream does. I just don’t people realize that not much has changed. The maintenance has been cut to 5 years, but you are grossly underestimating the challenges that CentOS has presented in our development pipelines. I’d have to write a whole different blog entry to explain. Stream removes this tension and makes every Red Hatter a potential Stream contributor. That wasn’t true before. It was a tiny team, kind of fire walled off from everyone.

      Reply

      1. The thing is while there may be more contribution to what is now called ‘CentOS’, the value was not the CentOS brand so much as it was uncomplicated access to a well tested base, the product that the CentOS name meant to 99.9% of the userbase. No need to register to download, ability to mirror freely, yum repositories without doing rhn registration, no overly complicated stratification of what could and could not be installed. To say not much changes would contradict the statements issued that say CentOS stream isn’t production appropriate, that it’s testing for RHEL, and if you need stability contact a RedHat sales rep (instead of probably going elsewhere).

        If you are moving to be more inline with the competition on this, that would be a fantastic thing to actually say either before or concurrent with the “no more stable CentOS” announcement. If this is what is truly coming in 1H 2021, a lot of headache could have been avoided by coming out saying what RedHat plans to do.

        I don’t think anyone is attached to the CentOS *brand* and can recognize that RedHat expending resource to do RHEL *and* do more effort to rebrand every release and make it free is a silly situation for all involved. RedHat endorses simple access to installation and repositories, but under a weird different name that lacks a clear upsell to support like a free RHEL would. The simple answer would be “ok, registration and rhn is now an optional part of the experience for paying customers, but RHEL 8.4 and up shall be freely downloadable with plain old yum repositories for the would-be CentOS users, for the verbatim CentOS experience without the brand confusion and easy path to upgrade to paid support”.

        The language thus far has alluded to potential free access for certain use cases, but those terms seem to imply potentially awkward limitations and does not recognize that RHN and channels, and registration is part of why CentOS is many cases is easier to manage. It would be nice to have had clarity from the onset.

        Reply

        1. These are fair critiques. I and many others at Red Hat have been griping about Subscription Management forever. That said there are actually a lot of improvements now days, I just think a lot of people aren’t aware as it takes time for information to disseminate. Check out Subscription Watch: https://access.redhat.com/documentation/en-us/subscription_central/2020-04/html/getting_started_with_subscription_watch/con-what-is-subscriptionwatch_assembly-about-subscriptionwatch-ctxt. I think SW along with low/no cost subs will solve a lot of this. Believe me, as Product Manager for RHEL, I wish there was some way to “not have” subscription management at all. When you move into the world of containers this also gets a lot better with UBI (my baby): https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image

          The challenge with subscription management has always been that there’s no way to protect price pressure. There are horror stories I’ve heard about Canonical, and even SUSE. Look at the layoffs. Seriously, nobody notices. SUSE recently acquired Rancher (Kubernetes thing), then layed a bunch of people off. Some of these people were friends of friends of mine. We’ve never had layoffs like that at Red Hat, because our revenue stream has always grown nice and steady. This is because we found a fair, consistent way to monetize enough of the value that we create.

          That said, I fully admit the world is/has changed and subscription management is painful in cloud and containers. We are working on it, furiously, I assure you. It’s not easy.

          Reply

      2. > I’d have to write a whole different blog entry to explain.

        Please do. But also note, this is their own fault. No one asked Red Hat to acquire the CentOS project.

        Reply

  18. You mentioned something about people having to pay for LTS (for Canonical) after year five. I don’t expect an operating system to be running on the same version for more than 5 years. I expect that a better version of the operating system is out by year four or maybe on year five. Canonical doesn’t force you to sign up for a subscription to download the next version of the operating system, unless you want paid support. Generally in place upgrades are easy and don’t cause many issues, Personally as a DevOps/Failover Systems Engineer I prefer to handle all of my own issues as much as I can instead of putting my problems on others, which would cost me more money than I make, so I really don’t like support contracts. for the past 5 years I had primarily focused on CentOS, but as soon as they announced stream I started making a lot of my projects in Ubuntu. This just puts the nail in the coffin for me on any sort of hope of using any red hat software. A lot of people pointed out that this community should have realized that as soon as this turned into Stream, that we all should start moving over to Ubuntu or other OSes

    Reply

    1. I’m confused by your comment. Stream has maintenance for 5 years, just like LTS Ubuntu. You are never forced to buy a contract. You can upgrade in place from 7 to, and will be able to upgrade in place from 8 to 9. They are literally almost exactly the same. The difference with Stream is, you will be influencing the next dot release. Take a look at the RHEL 8 ACG and internalize what Stream really means. This is not nearly as earth shattering as most people think it is.

      The RHEL and therefore CentOS Stream ACG is ridiculously more stable than Ubuntu. If you want more control, this actually gives you that, not less.

      Reply

      1. The messaging is very different.

        Value judgements about how well they do at it aside, Ubuntu does not proclaim that free users are getting the beta version of the OS, but only paid users get to stick with a stable version. They proclaim the free edition as being identical to the production appropriate version.

        Really, RedHat would fare so much better in the internet reaction if they had just said “RHEL will replace CentOS and be as easy to access and update, and RHEL Stream is now available for people who want in on the development of the OS”. The CentOS brand is not a particularly valuable one to try to make mean something new, and only complicates RedHat’s ability to capitalize on the success of that segment of users. No one is going to shy away from the RedHat brand but remain firecely loyal to the CentOS brand. It wasn’t the brand, but the sensibilities that were valued.

        Reply

        1. Correct about the messaging, I am not here to contribute to a development cycle. I was using CentOS as a fairly stable, well tested by the Community, Operating System to deploy applications.. I don’t want to be a tester, and I don’t want to pay for support, I know my way around a terminal, and the community was GREAT about help already. I wasn’t strictly referring to CentOS Stream, but was kind of mentally adding the RHEL subscription model to it because, moving CentOS to an upstream, feels like a money grab, to push the OS that people considered stable, to be in the development, ie the beta part of the cycle, therefore making sure that they 1. Get bugs, 2. don’t get support for them, and then 3.Urged to ugprade to RHEL for support and bug fixes. It totally screams of a ploy to move more people to a paying RHEL subscription.

          Reply

          1. I get it, but this isn’t YouTube, this is Enterprise Software. We’re not a guy talking into a camera, we have like 5000 engineers that need a paycheck. The CentOS team is tiny, maybe 10ish people (I’m not 100% sure on size), but theynare just revising the work of a thousand people. The container subsystems team alone has 20-50 people (depending on how you slice it) that need a paycheck.

            We need to accelerate RHEL growth. You need a free operating system that is perceived to be as stable as RHEL, one of the key value propositions that compete with RHEL. You don’t want to test, we need testers.

            If our wills are at odds, then they are at odds. Personally, the level of testing that you are imaging and that I’m imaging are on differently planes of existence. Paying RHEL users were already doing this “testing” as you call it, so forgive me if I don’t feel that bad making the CentOS community look at bits first, instead of paying customers.

          2. Also, if you’re not here to contribute, no worries. Run three commands and enjoy four more years of free maintenance. Nothing changes for you other than year 6-10 maintainance (you’ll have to do an I place upgrade). I think that’s more than fair.

        2. This comment is exactly my sentiment. You keep saying that this actually brings Red Hat more in line with the model of companies like Ubuntu, but the elephant in the room is that it’s a CentOS Stream is simply RHEL Beta now, and Ubuntu is just Ubuntu. That’s a massive difference whether you’d like to admit it or not.

          Reply

          1. Yeah, it means we’ll be able to innovate even faster than Ubuntu. Thank you for your usage.

  19. I am opposed to this change but appreciate you providing your viewpoint. I actually like the idea of Stream and think it serves a good niche.

    Your article talks about the benefits to RH you get from Stream by distribution of effort through Stream. That’s fair. However, you’re essentially forcing people off their existing release into 1. A licence purchase, or 2. into becoming a tester.

    What was the harm in keeping the CentOS downstream available and allowing Stream to prove its value through merit?

    You’ve mentioned the stability that we will likely achieve but I notice your article completely ignores the security update process.

    What will this look like for RHEL and Stream? What about embargoed fixes? Will Stream be (is it?) some hybrid of upstream features and downstream security errata?

    Reply

    1. To be fair, I didn’t mean to ignore the security aspect. I keep adding to the article as I think about things, and that’s still sort of on my to-do list. I agree that there is a downgrade in security patch SLA from RHEL, but it’s about the same as downstream CentOS was anyway. Forgive me because I’m responding off the top of my head and brainstorming this as I write.

      The harm in keeping downstream was paying people’s salaries. Wild guess, call it 10 people that costs us $1.5M to $2M a year for the CentOS rebuild team. Plus infrastructure costs for computers, software maintenance, etc. I don’t know the full cost for the CentOS program but call it $3M/year.

      So, just like CentOS users are complaining, we don’t have a magical $3M budget to double the investment and put that much work into Stream and keep paying for downstream.

      Now multiply that $3M by 9 more years, the hypothetical time when we could have gotten rid of CentOS 8 without making people mad.

      By 9 years from now, the cloud world will have changed so much we won’t even recognize it. Plus, it would have cost us a hypothetical $18M in costs based on my wild guess. It would have cost $36 for the same investment upstream and downstream. Worse, this is a rare pool of talent.

      These numbers might look small for an AWS or Azure, but I assure you those are NOT tiny numbers for us.

      Just like CentOS users are complaining that they can’t go get budget out of nowhere, we have the exact same business realities.

      So, that makes it a zero some game. It’s our business reality, or these free users business reality. Which one are you going to choose?

      Mind you, I’m a product manager so I’m used to doing back of the napkin calculations like this, but I don’t run the CentOS stuff, so my numbers are a wild guess.

      Reply

  20. How does CentOS Stream fit in for communities like the HPC community (but I’m sure others from what I’ve read in forums) that rely on downstream out-of-tree kernel modules? Think popular projects like OpenZFS, Lustre, Intel Omni-Path, Mellanox OFED, IBM Spectrum Scale that target builds at RHEL/CentOS point releases (and these things break between RHEL point releases). Where are we to go now that CentOS will be Stream only and thus impossible to reliably use with these tools at all (the coming RHEL 7.4 kernel that is in Stream now isn’t supported by these projects yet and won’t be until it lands in RHEL or even weeks after that).

    For RHEL to stay relevant in these communities generous free RHEL licensing will need to be provided and GUARANTEED explicitly by RHEL to stay that way (so that there is never any future confusion) for these organizations and their employees to use, no bait and switch after 2-5 years. Not to mention the thousands of folks who run homelab environments based on CentOS (that exceed the current developer subscriptions) and also rely on tools like OpenZFS and the like.

    Reply

    1. This is a good piece of feedback and I will bring it back to the RHEL team. A peer on my team, Jeffery Katcher, who is a new hire, is actually dedicated to HPC. I will bring this up to him. It may already be on his radar, but I’ll make sure.

      Reply

  21. This blog has been very informative, thanks for providing so much insight in the internal thought processes at RHEL. As a home user of the CentOS distribution I was under the impression that the stability and overall benefits would vanish.

    After taking the time to read this I am confident to perform an upgrade in 6 months to CentOS Stream.

    Reply

  22. We run a mixed environment of RHEL and CentOS. CentOS for dev/test and HPC clusters of machines, and RHEL for everything else. We benefit from the consistency between the 2 versions to reuse tooling, configuration, and packaging. We gladly support RedHat by buying RHEL, but paying such high subscription prices for 100’s of HPC servers we don’t want any support for isn’t palatable. If RedHat had a lower subscription fee, we would have no problem with this move.

    Reply

    1. There will be more streamlined low/no cost options for these use cases.

      I think Stream will be fine for many of these use cases as well. It’s just not as different as everybody thinks.

      Reply

  23. Dear Scott,

    Thanks for the clarity and the will to explain the change from a RHatter perspective, very appreciated. A lot has been said already and I don’t want to extend myself (I’m planning an article on my site for that). I only have a question:

    How far from reality is stating CentOS Stream is a Release Candidate for RHEL?

    I ask this because, my appreciation is lots of people are merely consuming CentOS and are mad because their perceived quality is gone. A quality which, I believe, was never there in the same amounts their perception is. Would that “Release Candidate” phrasing help them understand the move? It could make things worse, and I would have preferred to read your explanation or RHEL releases instead of a “cool” Rolling Release model, but this may be another way to put it.

    Waiting to hear from you.

    Albert

    Reply

    1. I tend to agree, it’s not “becoming beta software” which is how people perceive it, which admittedly annoys me. There’s this perception that they were getting something perhaps even more stable than RHEL which frustrates me. CentOS was a “blind rebuild” that included any bugs RHEL had. Calling CentOS Stream a Release Candidate with good visibility is a pretty good description, I like it.

      Reply

    2. As an example, many of these CentOS users have no idea and honestly don’t care about how much work the Podman team does to stabilize it before cutting the RHEL branches and releasing. Stream will get all of this work. It’s not like we just do rolling branch updates, hmthat would be insanity. Each cut of Podman is specifically chosen and QE’ed and patched often many times before we even do a build that’s public (even in the RHEL alphas/betas).

      We don’t just do PRs against a rolling branch and auto build RPMs. There is a human process in between. That’s why we don’t break RHEL customers.

      Reply

Leave a Reply

Your email address will not be published.