RHEL and all those patches to their kernels
Have you ever looked at the .spec file for a RHEL kernel RPM? I mean, *really* looked at it? Holy patches batman! As of this writing, the RHEL 5.2 kernel has about 1600 patches. For the longest time I was always just outright amazed (and annoyed) by all those patches.
Today I spent some time doing the tedious work of going through a large chunk of them to see what they really all do – sometimes the name of the patch just doesn’t give it justice. I was also looking for some elusive security issues that have been “hush hush”.
I have to say, Brad from grsecurity was right; there are plenty of things such as NULL pointer exceptions that have security ramifications in the linux kernel that aren’t publicly disclosed. They are buried deep in the commit logs of the kernel, and not labeled as a security issue. I found evidence of this looking through these patches. Everything like this that I’ve seen so far has been sent and accepted upstream – but damn – makes me want to keep tabs on all new patches to the RHEL kernels.
I figured out pretty fast into my research that about 120 of the patches went in cleanly into my 2.6.24.7 kernel sources, and none conflicted with the grsecurity patch. Most of them, it seems, are “ugly hacks” or vendor hardware fixes that haven’t gone upstream yet.
About 200 of the patches applied cleanly to the 2.6.24.7 sources with the -R switch – these seem to be patches that got sent upstream and went into the main kernel verbatim (and haven’t changed since). Pretty impressive to have that kind of influence on a moving target like the Linux kernel.
The rest of the patches (still over a thousand) are a messload of back-ports, RHEL-specific enhancements that didn’t go upstream, changes to device drivers that either went upstream and have been modified since or no longer apply (or have been re-implemented).
To me it looks like RedHat reviews every single commit to git in the upstream linux kernel to see if it applies to their own kernels, they have a suite of regression testing for each patch, and commit it if there are no problems. They also appear to write their own patches when they have a specific problem (or accept patches directly from the hardware vendor where the problem happens). This all sounds like a lot of work – but those 1600 patches in the RHEL kernel were made over a long period of time. If you really think about it, hire a few engineers to review the upstream commits daily, and that’s all it takes. Then you just send over the interesting patches to go through the weekly regression testing on the development, so-to-be production RHEL kernels.
My impression before this research has always been “how inconvient, all these patches” when I would try to apply a patch or two to the RHEL kernel sources. But putting things in perspective now – this is no different from the upstream kernel which uses git. It’s just a hell of a lot more obvious because all the patches make the kernel.spec file huge, and git easily hides them all after each subsequent release.
The moral of the story? Well there are a few:
1) Treat the kernel source + patches as a whole: don’t try to insert a patch before any of them, and don’t take out a patch unless you *really really* need to; and if that’s the case, you’re probably better off grabbing the latest upstream kernel source and starting from there.
2) Not all known security issues are flashed across the heavens via CVE or other vulnerability database – so use a vendor kernel if you don’t want to the inconvenience of staying up-to-date with the latest upstream vanilla kernel … all those recompiles and reboots.
3) Linux Vendors really do know what they are doing – and this is why they charge money for their support.
So, I’m not amazed by all the patches anymore; I’m impressed. Also I’m no longer annoyed by them (well, for the most part). Amazing how your perspective on things changes when you get a taste of the other side of the fence.