Archive
You are currently browsing the cormander blog archives
for June, 2008.
By cormander
Woah that was a close one. Not sure on the details but the server hosting the VPS that ravencore.com and cormander.com was on died a horrible death. The hosting provider was able to bring the VPS back online on another machine, but it was on another subnet so I had to repoint DNS.
Now that they’re back up – I’ve got a lot of cleaning up to do. For example – my mysql server was damaged in the crash and so far the only database I wasn’t able to recover so easily was the wiki, so the wiki is down for now. But that seems to be the only gaping problem at the moment.
By cormander
Time is a resource that we all wish we had more of; the problem though is we’re seeing the glass half-empty. The truth is we all get the same amount of time each day. Time is continuous and replenishing. We always have the same amount of time, no more, no less.
It all boils down to how we spend it. Like this past week I haven’t made a single blog post. Why? I could give the excuse “I wanted to but I didn’t have the time” when the answer really should be “I thought about it but never actually sat down and spent the time to do it”. How did the time get spent, then? I spent it thinking about wanting to post, instead of posting. I thought about it on the way to work, while at lunch, on the way home from work, and laying in bed late at night. Bad timing – or in other words – I spent the time in the wrong place.
Our minds know what we need to do, and what we want to do. The problem is – it bugs us at the most inconvenient times. I could have been thinking about my financial plans while driving – instead of what I want to blog about … and spent the time at home blogging what I wanted to say instead of sitting on the couch thinking about my financial plans.
Enter the greatest time management tool of all time – a TODO list. Never underestimate the power of a TODO list!! (when correctly used, of course). If I *knew* that “write a blog post about XYZ” today was on my TODO list, and I *knew* that I always do as much as I can on my list, I wouldn’t worry about it on the drive home from work. If it entered my mind – it would immediately be dismissed with “It’s on my TODO list, I won’t forget” and I’ll go onto the next thought that isn’t on my TODO list.
A TODO list is spending time to save time. If you think of something you can’t do right now, but can do later, write it down. Then later do it. That’s all there is to it. If you write down and then actually read *AND* do it later, things will start to not constantly bug you. You will get things done.
That is, of course, you prove to yourself that you do things on your TODO list. If you put it on your list and then never do it, you’ll find that your mind will continue to bug you anyway. It won’t leave you alone because it knows that just because it’s on your TODO list, it won’t get done. Why? Because you have a bad track record.
A little effort and a lot of self-control – when combined with a TODO list – will do you a lot of good.
Oh – and for those of you who still think time is limited – you’ve then got too many things on your plate. A constant go-go-go life isn’t healthy in the long run. I’ve decided to take it down a natch and focus on my family for a while – a reason why RavenCore hasn’t gotten much progress lately.
I used to do the time-management pretty well, but have been failing at it recently because I haven’t been keeping up on my TODO list. Time to motivate myself to get back into it. This post is my first step toward that goal.
By cormander
So I try to boot my 2.6.25.7 kernel that I built from last night, and it panics on my CentOS 5 box (but not my fedora core 9 box). In the output just before the panic, the sata and such drivers are loaded but the initrd is unable to see the drive which I found to be very odd.
After a conversation with Scott Shinn he points me in the direction of SYSFS changes in the 2.6.25 kernel. I hacked the fedora core 9 initrd to boot on the CentOS 5 machine, and it booted normally. Long story short, I recompiled the kernel with this option enabled in the config:
CONFIG_SYSFS_DEPRECATED_V2=y
Turns out that things like udev and nash croak on the latest kernel if they aren’t the latest version of themselves (which they obviously aren’t on an enterprise os). This config option turns on the “deprecated” sysfs system calls and fixes the issue. Thankfully the kernel with this option on still seems to work when booting a fedora core 9 machine.
This particular issue is a pretty good reason to continue to stick with 2.6.24.7 for my “stable” kernel – I’d hate to have to use deprecated systemcalls in a later kernel, they’re no longer maintained, not guaranteed to work, and once they start to conflict with other kernel internals, they are removed altogether.
This isn’t the first time I’ve had to deal with something like this when upgrading the linux kernel to a whole new version tree, hense the title of this article being “wack-a-mole”. You narrow in on one problem and smack it, and pretty soon you find yourself with two or three more sticking back up at you in other locations that you have to hunt down as fast as you can.
By cormander
So far things are going great. I have setup several different build chroots and created a solid build system that is growing in complexity. Basically I get a source RPM and run it through my scripts and it builds out RPMs for each distribution I have setup, which currently include:
CentOS 4 and 5, Fedora 8 and 9, all for both i386 and x86_64
At the end it informs me of whether or not it succeeded on each one, and saves all the rpmbuild output in a log file. Oh – and it autoinstalls RPM build dependencies from the source RPM into each chroot if they are missing, so I don’t get silly build failures.
Tonight I ran the latest grsecurity kernel through the build tree and I’ll run it through regression testing in the morning. Since this is the first time I’ve built a i686 kernel on a x86_64 machine I’m going to do a little more testing with it to make sure it’s good to go.
So far the gradm source RPM did pretty well on all 8 builds, but paxctl needs a little work. The pax guy was right – that sub-project is bleeding from many wounds, and has compile problems when you start dealing with cross-distribution builds. I’ll have to revamp the code if I can find the time in the near future.
Hopefully the grsecurity yum repo will be full of many more packages by this weekend.
By cormander
One of my favorite TV shows of all time is Firefly. From that show, one of the best quotes is from the episode War Stories where Mal says:
“About fifty percent of the human race is middle men and they don’t take kindly to being eliminated”
This is SO TRUE especially when it comes to the hosting business. Consider the following situation where a guy puts up a website. He got the website domain and hosting from a friend (customer A) who has an account with a web design company (customer B). This design company buys hosting accounts from a shared hosting company (customer C) which basically sells accounts through cpanel or plesk. Their control panels are on VPS servers they get from a VPS server provider (customer D) who rents server space from an HSP (customer E) which actually turns out to be a company that co-locates at a datacenter (source provider).
Oh, and I forgot to mention that the company who owns the datacenter probably hasn’t paid off their mortgage yet so technically they’re customer F and the bank is the source provider. But you can take it even further where the bank is customer G and the source provider is the mega-corporation which owns the bank.
Sheesh.
By cormander
Today I got not one, but two new servers! One is a AMD64 3200 2GB RAM machine I’ll be using as a build server, and to host all of the RPMs I create. The other is a xen domU virtual machine with 1GB RAM and access to 4 CPUs that I’ll be moving my website(s) to. I’ve just about finished provisioning them (customizing to my liking) and then begins the tedious task of moving everything around. At least I’ll finally have the RavenCore demo back online – I put a xen kernel on the build machine and I have access to a few more IP addresses.
Now I just have to decide what to do with the current hosting I have – free from a company in London. I’ll probably just make it a mirror of my build/download server – and call it europe in DNS or something.
I’ve got big plans for the build server; first and foremost the building of x86_64 grsecurity kernel RPMs. My preliminary builds show I need to make some tweaks to the spec file and some of the file path names, but I’ll have the details sorted out soon. Once they are built and pass regression testing, I’ll update the repos with them. And that will slowly become true for every other package I maintain.
By cormander
A little while ago I made a post about RHEL and all those patches to their kernels. Since that post I started thinking about the patches, and started to experiment with the theory of back-porting.
I currently maintain a list of Linux CVE issues and this is where my experiment begun. If you really think about it – that isn’t a whole lot. Sure these are security specific issues, but I know for a fact that this isn’t a complete list of patches to security issues. But just security fixes asside – exactly how many patches that go into later versions could also apply to old ones?
Turns out, a lot. I mean a LOT. “Holy patches batman!” as Tim might say. About 40% of the 2.6.25 tree (including all patches except ones from release candidate 1) apply cleanly to the 2.6.24.7 kernel. A few reference data structures or functions that don’t exist in 2.6.24.7, and thus cause build errors, so those are removed during the build process. No interfaces change (because the bulk of those happen in the release canidate 1’s, which have thousands of patches in themselves, any anything after that is likely to fail to apply).
Anyway, that’s about 1400 patches! Regression tests with the Linux Test Project after it’s built show no problems. I’ve been running it since I first built it – haven’t seen any issues. Who knows how many potential security problems I just fixed?
So, I updated my grsecurity RPM yum repo 2.6.24.7 branch of the grsecurity kernel with these 1400 some odd patches. When the 2.6.26 linux kernel is released as stable, I’ll have a few thousand more to review. Luckily I have the bulk of it scripted.
I’m sure there are some patches that could still apply to the older kernel with a few tweaks – but I currently don’t have to resources to review each one of the 2100 patches manually for this purpose. I’m also quite certain that this kernel won’t build on arches other then x86 – but all you’d have to do to get it to build is determine which patch caused the build error, which is very easy to do, and reverse it. When I get a 64bit machine to play with, I’ll build the kernels for x86_64. Same goes for other arches – testing will happen when I have the hardware available.
And for version control – since this is the linux kernel we’re talking about – I’m using git. Here is the git repository for this project.
By cormander
For the first few years of my carrier my form of revision control was “make a backup of it every now and then” which obviously has its flaws. I struggled to get familiar with CVS without much success; it seemed that every time I sat down to try to learn it, I ended up doing something else.
Same story when I tried to learn subversion instead – I for some reason had a hard time wrapping my brain around it. It wasn’t until a year ago that when I was required to use subversion for a job that I got anywhere with it. It seems that money is plenty of motivation to finally grasp something you’ve been mentally putting off 
Both subversion and CVS are classified as a client/server revision control system; I’ll spare you my opinion of it. My friend Tim Post introduced me to the world of “distributed” version control not too long ago with a tool called mercurial. I instantly fell in love with it – the ability to track changes in your own local repo and then later push and merge upstream was all I ever dreamed for. This certainly would have solved a lot of problems we had at my last job while using subversion.
I have heard of git before but never really paid much attention to what it was. It wasn’t until after my discovery of mercurial that I payed enough attention to figure out what git was – another distributed revision control system. It was originally coded by Linus Torvalds to maintain the Linux kernel code, but is available for any project you wish to manage.
I’ve wanted to get into linux kernel development for some time now – but haven’t really been very serious about it. Since I’ve now been familiar with mercurial I decided to give git a spin to see how it was in comparison. At first I was turned off by how much more complicated git was, but used it anyway to track my changes to the linux kernel, since that’s what git was made for. This past week it has grown on me. It didn’t take long before I decided that I’ll use it on everything, unless that project is already on mercurial.
Here is a link to my git repository. Right now it just has my grsecurity kernel tree with a few days of history on it, but I plan on putting everything I got into it (except SRCE, since that’s already under mercurial and controlled by Tim Post).
By cormander
I’m an avid user of CentOS and really like having an enterprise operating system for free. Don’t get me wrong – I’d use the upstream vendor in a heartbeat if I could – but since it’s the service they charge money for, and I don’t need the service, I currently don’t buy their products.
Two things that have always irked me about CentOS, though, are these:
1) 100% binary compatibility
2) there are still traces of “redhat” in the signatures of the binary files
Regarding the first one – it’s more of a “bitter sweet” thing then anything. There are several real good reason why they do it, and I don’t at all contest it; it just makes my life a little more inconvenient when I want to install software that isn’t included. For this there are things like rpmrepo.org which takes most of the pain away, but it nonetheless still irks me I usually end up grabbing a source RPM from the latest fedora distribution or rawhide and recompiling it on my system. Perhaps I’d like gentoo linux because they recompile everything – but then again, I don’t want to recompile EVERYTHING.
Regarding the second one – this has to do with the compiler (gcc) and the package manager (rpm). Compiled into gcc is the version number with “Red Hat” (and the RPM release of the gcc rpm) inside of it; and when it builds binaries it puts that version string inside it. Also, inside the rpm library there are default macros; one of them is %_vendor which is still set to “redhat” on the CentOS systems which gets put inside the RPM file’s signature, and often is passed to ./configure for things that are platform-specific (gcc, glibc, kernel, etc).
If you look at any binary on a CentOS system you’ll see one or both of these trace elements of the upstream vendor.
I decided to see if I could have these removed from the binaries I build, so I downloaded the gcc source RPM and rebuilt it with the version to not contain “Red Hat”. After it was done (it took over 6 hours!) I rebuilt a package and had a look at the binary with a hex editor.
Success! Well, mostly. Of the 18 or so “GCC” version strings embedded in the binary, three of them still said “Red Hat”. I decided to get creative and run strace on the build process and saw it using ld against these files:
/usr/lib/crti.o
/usr/lib/crt1.o
Those contain the “Red Hat” string and are part of the glibc-devel package. I went to rebuild that from the source RPM but got this build error:
/tmp/ccIC829t.s: Assembler messages:
/tmp/ccIC829t.s:58: Error: suffix or operands invalid for `fnstsw'
And this is where I am stuck for the moment. Almost there, but not 100%.
Once I’m done with this I’ll see what the CentOS guys have to say. I doubt the above changes take away from the “100% binary compatibility” but I could be wrong.
