yet another linux engineer
By cormander
I’ve been getting complaints that my blog posts make no sense. These people like to stay in touch with me and know what I’m up to, but considering these complainers aren’t technical people the majority of my posts aren’t what they had in mind.
So, any post that I deem “readable” by normal people, I’ll tag with the new “plain” category.
Oops, I must have offended everyone. I just called technical people “not normal” and non-technical people “plain”. That’s not how I meant it, so if it bothers you, get over it.
By cormander
Now I’m not sure if this is from bad coding on part of the hudson FTPPublisher plugin or from java.net’s URI.create, but:
Connecting to rpm.cormander.com
FATAL: null
java.lang.IllegalArgumentException
at java.net.URI.create(URI.java:842)
at java.net.URI.resolve(URI.java:1028)
at com.zanox.hudson.plugins.FTPPublisher.perform(FTPPublisher.java:131)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:56)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildStep(AbstractBuild.java:379)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildStep(AbstractBuild.java:367)
at hudson.model.Build$RunnerImpl.post2(Build.java:183)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:352)
at hudson.model.Run.run(Run.java:958)
at hudson.model.Build.run(Build.java:112)
at hudson.model.ResourceController.execute(ResourceController.java:93)
at hudson.model.Executor.run(Executor.java:118)
Caused by: java.net.URISyntaxException: Illegal character in path at index 7: Vanilla Kernel 32bit
at java.net.URI$Parser.fail(URI.java:2809)
at java.net.URI$Parser.checkChars(URI.java:2982)
at java.net.URI$Parser.parseHierarchical(URI.java:3066)
at java.net.URI$Parser.parse(URI.java:3024)
at java.net.URI.<init>(URI.java:578)
at java.net.URI.create(URI.java:840)
... 11 more
Emphasis on the: Illegal character in path at index 7: Vanilla Kernel 32bit
Well, assuming we start counting from zero, which we do in an array index (and strings are still arrays in addition to being objects in java), the illegal character is a space. What, can’t convert a space to a %20 for me? Renaming my project to Vanilla%20Kernel%2032bit would look kind of ugly.
I ended up renaming it to kernel-32bit, and that has fixed the problem… sort of. Then there came this:
Connecting to rpm.cormander.com
file:/var/build/workspace/kernel-32bit/kernel-32bit
ERROR: Failed to upload files
hudson.util.IOException2: remote file operation failed
at hudson.FilePath.act(FilePath.java:645)
at hudson.FilePath.act(FilePath.java:633)
at hudson.FilePath.list(FilePath.java:1038)
at com.zanox.hudson.plugins.FTPPublisher.perform(FTPPublisher.java:144)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:56)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildStep(AbstractBuild.java:379)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildStep(AbstractBuild.java:367)
at hudson.model.Build$RunnerImpl.post2(Build.java:183)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:352)
at hudson.model.Run.run(Run.java:958)
at hudson.model.Build.run(Build.java:112)
at hudson.model.ResourceController.execute(ResourceController.java:93)
at hudson.model.Executor.run(Executor.java:118)
Caused by: java.io.IOException: Expecting Ant GLOB pattern, but saw '/home/hudson/rpmbuild/TARGETS/kernel.i686/*rpm'. See http://ant.apache.org/manual/CoreTypes/fileset.html for syntax
at hudson.FilePath.glob(FilePath.java:1059)
at hudson.FilePath.access$000(FilePath.java:155)
at hudson.FilePath$24.invoke(FilePath.java:1040)
at hudson.FilePath$24.invoke(FilePath.java:1038)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1783)
at hudson.remoting.UserRequest.perform(UserRequest.java:103)
at hudson.remoting.UserRequest.perform(UserRequest.java:47)
at hudson.remoting.Request$2.run(Request.java:236)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Long story short, turns out ant doesn’t like full path names, which means the rpmbuild stuff needs to exist within the hudson workspace. A security mesure I would assume. Anyway, I set the %_topdir for the hudson user to be %(pwd)/rpmbuild and manually created that directory (and the necessary subdirectories within it) and built the RPM and ftp’d it over just fine, sort of.
I was using the ravencore project to test since it only takes a minute to do, rather than two hours. This is what the file on the remote ftp server ended up being:
./ravencore/2009-06-29_09-59-47/file:/var/www/.hudson/jobs/ravencore/workspace/rpmbuild/RPMS/noarch/ravencore-0.3.6-0.1246291028.noarch.rpm
I really expected it to be:
./ravencore/ravencore-0.3.6-0.1246291028.noarch.rpm
I’m gonna have to try out the scp plugin instead to see if I can avoid this.
By cormander
The past week I’ve been talking about Hudson, and not just here. Quite frankly I think some of my buddies are sick of me talking about it. For this (and other) reasons, I exposed the build system at this URL:
http://build.cormander.com/
I granted anonymous view access to most of it. I currently have ravencore in there, and my grsecurity stack setup in there and building successfully. Pretty soon I’ll be putting rootsh and other projects I manage in there, as well as things like python26 for CentOS, Xen 3.4 for CentOS, and any other thing that comes to mind that I have built and made use of.
And as far as this saving my world goes, it’s because it reduces the amount of effort I have to put into this stuff, which means I can do more of it.
By cormander
Sometimes it’s just easier to have a 32bit system to build 32bit packages on, but we don’t always have that option. And creating a virtual machine may just be too much overhead. After all, you just want some 32bit packages to be built on your 64bit system and not have to worry about multi-arch problems at build time. Here is how I usually solve the problem.
First, fool rpm into thinking this is a 32bit system:
mv /etc/rpm/platform /etc/rpm/platform.orig
echo i686-redhat-linux > /etc/rpm/platform
Then make your target directory. In this case I’m doing Fedora 11:
mkdir -p /var/distro/Fedora11.i386
cd /var/distro/Fedora11.i386
Now you’re in the directory you’re going to install in. This is important as I reference this directory throughout this tutorial as $(pwd). If you cd out of this directory in the middle of these steps, you’re bound to run into problems.
Setup some basic items:
mkdir -p dev proc sys root etc var/{log/yum,lib/rpm}
mount -t proc none proc
mount -t sysfs none sys
cp /etc/resolv.conf etc/
You don’t need an fstab, but no harm in creating one. You’ll actually avoid some meaningless errors by creating it:
cat << EOF > etc/fstab
/dev/xvda2 / ext3 defaults 1 1
/dev/xvda1 /boot ext2 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
EOF
Create your /dev/null device and give it the mark of the beast:
mknod dev/null c 1 3
chmod 666 dev/null
For Fedora 11:
rpm --root $(pwd) --import http://mirrors.kernel.org/fedora/releases/11/Fedora/i386/os/RPM-GPG-KEY-fedora
wget http://mirrors.kernel.org/fedora/releases/11/Fedora/i386/os/Packages/fedora-release-11-1.noarch.rpm
rpm --root $(pwd) --nodeps -ivh fedora-release-11-1.noarch.rpm
For CentOS 5:
rpm --root $(pwd) --import http://mirrors.kernel.org/centos/5/os/i386/RPM-GPG-KEY-CentOS-5
wget http://mirrors.kernel.org/centos/5/os/i386/CentOS/centos-release-5-3.el5.centos.1.i386.rpm
rpm --root $(pwd) --nodeps -ivh centos-release-5-3.el5.centos.1.i386.rpm
Now install. This may take a while, depending on your internet connection speed.
yum --installroot=$(pwd) -y install kernel rootfiles passwd vim-enhanced wget strace grub \
openssh-server openssh-clients gcc patchutils diffutils gcc make rpm-build rpmdevtools sudo yum
Now wipe out the rpm db, it’ll get rebuilt next time it’s used:
rm -f var/lib/rpm/__db.00*
Lastly, don’t forget to restore your rpm arch configuration:
mv /etc/rpm/platform.orig /etc/rpm/platform
Now you can do this:
setarch i686 chroot $(pwd)
Anything you build while in this chroot is guaranteed to be completely 32bit and have no 64bit related build problems.
By cormander
So I decided that since Hudson is so cool that I’d try to build something as big as the linux kernel inside it. I put the rpmbuild command for my grsecurity kernel inside and watched my builds fail until I got it just right. I ended up with a small git repo to revision control the spec file, kernel config, and a new Makefile to tie it all together.
Long story short, I now have automated builds of my spec file which is now capable of building the grsecurity kernel, the PaX kernel, and the vanilla kernel, all depending on how you call it: make grsec, make pax, and make vanilla respectively. I have each setup with both 32bit and 64bit builds, each takes two hours to do. So I just commit a change to apply a new grescurity or kernel patch, and 12 hours later I have all the RPMs fresh out of hudson and I don’t have to so much as bat an eyelash for it. Unless, of course, a build fails. If one fails, they’re all liable to fail for the same reason. So I’m gonna need some more build slaves so it doesn’t take as long in the future.
I know, I know, I’m not “using hudson how it is intended” because hudson itself is a build system, and rpmbuild is a seperate build system, so I’m having a build system calling a build system so each build has to “take it from the top”. But I don’t care about what it was designed to do, I care about what it can do.
And it rocks.
By cormander
At work, we use Hudson (https://hudson.dev.java.net/) to manage all the various projects for the software development teams. It’s primarily a tool for java, but can do general purpose builds as well.
Well, I decided to take a crack at it myself, and I really like it. I pushed a few commits up to the ravencore git repo to make builds work in hudson today. Now I just will have to configure it to build a snapshot each time it sees a new commit, and export those snapshots via the web.
It’ll at least give the illusion that ravencore is an active project, right?
Seriously, though, I feel like I’m slowly moving toward that developer “sweet spot”. I’ve got full revision history with a tool I really like (git), build scripts to automate things, and now a build management system.
By cormander
“If minimum wasn’t good enough, then they wouldn’t call it satisfactory.”
This is either from the mouth of someone with too much to do, or that of someone who is really, really lazy.
By cormander
Yesterday was my first day on the “Tools & Automation” team, where I proceeded to kill a big project. It was the whole “First day on the job and I’m already killing projects”. But now, on the second day, we find out that the team lead (manager) is leaving.
“Get the hell out of this team before something else happens!” said one of the other team members to me about this situation (jokingly, of course).
Gotta love coincidences.
By cormander
Starting on Monday, I’ll have moved to a different department in the company: Tools & Automation. So my new job title; Tools & Automation System Engineer, or TASE; as in, I shock people.
I should start carrying around a Taser to be funny.
By cormander
“The road to hell is paved with good intentions. Churches are full of good intentions. Ergo, church is the way to hell.”
I love her so much.
