Hello,

My name is Corey Henderson, and I'll be giving a presentation on security hardening for distribution kernels.

Before I begin, I'd like to mention that my trip here was sponsored by my employer, IdenTrust. They are a certificate authority, recognized by global financial institutions, government agencies and departments, and commercial organizations around the world. And of course, all the major web browsers

Products include:

* a unique legally and technologically interoperable environment for authenticating and using identities worldwide.
* Identity credentials under the ACES program, certified by the General Services Administration
* Identity credential under the Department of Defense External ECA program.
* personal SSL certificates
* SSL certificates that are CAB Forum-compliant

I also have a bunch of SWAG here with me, feel free to come pick up a button after the presentation. See mine? I'm Albert Einstein, and I'm coming from a TOR ip address. Do you believe me? You shouldn't. But if I had a valid IdenTrust certificate that said I was, then you could believe me.

Back to the presentation.

Now, when I say distribution kernels, I mean the binary kernels that come pre-built and pre-packages by your distribution; RedHat, CentOS, Debian, Ubutnu, etc. Sure, you can always get the sources for your kernel and rebuild it yourself, but not only is that not interesting or viable to operations people, it's not any fun.

NEXT SLIDE -> 2, In This Presentation

In this presenation I'll lightly talk about some of the current security systems available in the linux kernel that vendors enable in the kernel.

I'll cover three methods of altering your distribution's kernel at runtime, but in the interest of staying focused more on the operational side of things, I'll skip the nitty gritty coding details invovled.

Finally, I'll demonstrate a linux security module I wrote that uses one of the methods I'll cover.

NEXT SLIDE -> 3, What is Linux Missing?

Ok, what is linux missing? Because I wouldn't be talking about adding security features to distribution kernels if they already had everything we wanted. This isn't a complete list, just some of the heavy hitters. I'll talk about some more features later on too.

If you're familiar at all with grsecurity, you'll know that it has a lot of really heavy duty security features that aren't in any of the distributions. Hell, most of the stuff hasn't gone into the mainline kernel, and I don't expect it to either. So grsecurity will keep on keeping on as a patch to the linux kernel, and for those of us who use distribution kernels, probably won't get to use it in a broad deployment, if at all.

So we have these bullet items; non executable memory, address space layout randomization, and exploit dections and auto response. Unfortionatly I won't be showing how to implement these specific features into distribution kernels. The code involved is way over my head. But I thought I'd mention them to see if I can motivate any developer crazy enough to attempt it.

Suffice it to say, the linux kernel is missing a lot of really good security features.

NEXT SLIDE -> 4, Give credit where credit is due

Now I don't want to, at any point, sound like I'm ripping on the distributions that work hard to package us up good linux kernels. They do a good job. And I'd like to give them credit for the things to do, and have, back ported into their kernel after they have gone mainline. So I'll list a few of them here:

<say list>

I should mention that all of these came out of the grsecurity project. They were cherry picked by kernel developers and pushed upstream, and eventually trickeled down to distribution kernels. So there is hope. Anything we can demonstrate as a good, usable, non-headache inducing security feature is likely to make it into the mainline kernel, and eventually make it into the distribution kernels.

But, I'm an impatient guy. If you're anything like me, you want your extra security features, and you want them right now. And you don't want the headache of recompiling your kernel, maintaining it, etc.

NEXT SLIDE -> 5, Current Security Frameworks

So you've got SELinux, AppArmor, and the other LSMs like SMACK, TOMOYO, etc. The guy presenting after lunch in this very room is going to talk about his new LSM called CaitSith. At least I hope I pronounced that right. The name reminds me of Final Fantasy 7. A cool guy by the way, I've found over email, and I look forward to meeting him in person.

And I can tell you from personal experience that a lot of system administrators, as the very first thing they do when they install a system, is turn it off. That's assuming it wasn't already disabled upon installation with their bootstrapped install.

While I'll advocate the use of these systems, what good are they if they're disabled? No good at all. Sorry to answer my own rehtoricle questions, but it is what it is.

So my question to you is, why do you not use them? Don't need to answer out loud.

NEXT SLIDE -> 6, Current Security Frameworks (cont.)

So to summarize why the people I know to turn off their LSM:

<read bullets>

Not sure why you don't use them, maybe you have a different reason. But the end result's the same. When turned off, they do you no good.

Now, these all may seem like good reasons to turn off the LSM, but in some situations these reasons can be used as a good reason to turn them on. If you want to be the bastard operator from Hell, while gaining the job security of handling something no one else in the organization wants to touch, then these are reasons to turn on the LSM. You just have to have management that is willing to put up with the added complexity.

Now I'll mention that the LSM framework has no add-on capability, meaning you can't insert a module you built yourself without recompiling the whole kernel. I understand why the kernel developers did this, believe me, but I gotta say; doesn't that make the whole idea of a "MODULE" pointless?

NEXT SLIDE -> 7, We want more options

So to summarize, we don't have any of the heavy hitting security features from grsecurity. We can't insert a new LSM module that wasn't built with the kernel itself, and now we have a list of even more features some of us may want. These are some of the easier features to implement;

<read list>

And, by the way, it should be no suprise that this list is also features from grsecurity. Except the randomized kernel symbol addresses, although you almost get that with grsecurity anyway because you have to build your kernel and that randomizes the addresses. Just doesn't randomize them beyond creating that specific kernel image.

NEXT SLIDE -> 8, Why we don't have options

Back to what I said about not ripping on the distributions for the lack of security features. I'd like to re-enforce that statement. It's not rosey situation for them. I understand their plight, and why they don't add these features.

<read list>

NEXT SLIDE -> 9, Methods of adding features

Huh, finally, okay. To the meat of the topic. I've rambled on long enough.

There are various methods of adding code to your running kernel. I'll lightly cover three of them:

<read list>

You also have the more shady ways such as writing to /dev/mem and /dev/kmem, but I'm not going that in-depth here. When you're hijacking a system like this, you don't need to do it in a way that hides your tracks, so creating kernel modules is just fine. As the administrator, you have control over the system anyway.

NEXT SLIDE -> 10, Kprobes

Linux comes with a nice framework built-in on inserting code at the start and end of kernel symbols. It's called kprobes. I've mostly only seen it used for debugging the kernel, kernel modules, etc. One example of this is a module called "netlog", you can find it on github. I think I had a fork of it at some point, I might still. I'll have to check.

Anyway, redirecting code execution paths completly with kprobes is a little more tricky than I'd like, and out of the gate it's mainly used for planting instructions at the start and end of kernel functions, which is mainly for debugging. That and not all kernels are going to even have kprobes enabled. So while it's nice and safe and supported by the kernel developers, it's not always a viable option.

If you want to read more about how kprobes works, you can read the linux kernel documentation, and I also have a link to a great article published in phrack magazine. Sorry the slide cuts it off, so if you get a copy of my slides you can click the link. Or you can just google "phrack kprobes" and find it that way.

And that's all I have to talk about on kprobes. I mainly just want to make you aware it exists so you can research it yourself, if you are interested. I want to stay focused on the operational side of things. This isn't the kind of thing you can toy around with unless you can write code and have at least a basic understanding of kernel space.

NEXT SLIDE -> 11, Ksplice

Anyone ever heard of ksplice? Rebootless kernel updates. It was started a while back by a few college students that were hacking at the kernel's innards and got the bright idea that it's possible to patch security flaws on a system without having to reboot it. It's a novel idea, and really what sparked my interest in this topic to begin with. So these guys started a company called Ksplice, and were recently bought out by Oracle.

Ksplice is unique in the way it works. They deal with binary diffs rather than the traditional kernel hacking of intercepting kernel functions. They just patch the functions in-place. To summarize, it:

* takes a compiled kernel as input
<read rest of bullets>

Again, go to these links if you want more details on how to use the ksplice raw utilities yourself. They focus on patching security holes, but the technology can also be used to add security features. Or making a pretty scary rootkit. It's all up to you.

NEXT SLIDE -> 12, Oracle Violates the GPL

Now I'll touch on a side topic for a moment. Since Oracle's aquisition of Ksplice, they have not released the sources to ksplice. Sure they have a silly tarball burried somewhere on their website that contains source code that is a few years old, essentially a copy of their git repository the last time it was committed to. But it doesn't work for newer kernels. They've obviously made changes in how they do things, and aren't sharing that with us. Believe me, I've asked them, only to be ignored. They allow you to ask for the source of a given patch they produce, but refuse to release how that patch is compiled into their rebootless updates. And if you're a client of theirs and ask for that source, they may just offer you a discount to get you to stop asking.

You can read some of the posts on my blog about this if you want more information on the details of the GPL violation. I just wanted to call out Oracle publically that their refusal to release the updated source code to ksplice is a violation of the GPL.

Bottom line, be careful about licensing issues when you're hot-patching the linux kernel. It's GPL code you're changing, so if you release your module publically, you must also abide by the GPL and release the sources of your method as well with the GPL license.

NEXT SLIDE -> 12, Hijack kernel functions

Now to the last method I'll cover. Kernel function hijacking is overal a pretty simple process.

<read list>

The implementation details of how it works on the code level can be found on my blog. The post is almost a year old but it's all still correct. I'm actually working on a different method of hijacking that involved replacing kernel symbol function references instead, but it follows the same basic process. I'll give you a URL to github at the end of this presentation that has the code to do it.

NEXT SLIDE -> 12, Hijacking not only for bad guys

At this point, some of you may be thinking; "gosh, hijacking functions. That sounds like a bad thing to do".

<read list>

Kernel developers may argue that you should send the security features to the mainline kernel to have them included. And I'm all for that. But as I said earlier, I'm just a really impatient guy. And, like how grsecurity has shown us, some features just simply won't be accepted for one reason or another. So I'll just avoid the political nonsense and say, "here is your security feature. Sorry it's not already the kernel, and sorry I have to use tricky methods to insert the code, but here it is anyway".

NEXT SLIDE -> 13, Trusted Path Execution

Now to the pilot project that I put together over a year ago to highlight that this method works and is safe. One of my favorate features of grsecurity is trusted path execution, and is something that as far as I know, has twice been submitted to the upstream kernel the past 10 years and so far hasn't made it. Maybe I'll attempt a submission myself. But in the mean time, I'll implement it via function hijacking.

<read slide>

While this won't stop all exploit attempts, it does add a layer of security to the system. In the case of an exploit that gives an attacker non root access to a system, either via social engineering or a network program that has a nasty bug, the attacker will have a difficult time escalating their access to the system, because all kernel privilage escalation bugs that I know about require you to execute code on the system, and that is exactly what this feature prevents.

Simply put, if the administrator didn't put that binary there, and set permissions and ownership correctly, then the system will refuse to execute it. That really limits what an attacker can do. And if they do attempt to execute something and get denied, your Intrusion Detection System alarms should start to go off. You do run an IDS system, right?

NEXT SLIDE -> 14, A typical exploit

Now here's what an exploit might look like. The attacker gains shell access through a web application, or something, and can execute commands. They download a kernel exploit and run it to gain full root access. I apologize for the overly simplified example.

NEXT SLIDE -> 15, A typical exploit (cont.)

Now enter the trusted path execution security feature. The attacker gains access, but when they go to execute the exploit, it fails. They get permission denided. And if they're a script kiddie, they'll be lost for words, and go and try and exploit someone else. Otherwise, like I said, a log message will go out and if you run a properly configured IDS, you'll be alerted to the attempt.

NEXT SLIDE -> 16, TPE isn't in Linux

Like I mentioned before, TPE never made it into the mainline kernel, and as such, isn't in distributions. If you want the feature, you've had to use grsecurity.

But not anymore. I'll now show you a kernel module I wrote that implements the TPE security feature into distribution kernels. So far I've only tested it on various RedHat/CentOS and Ubuntu systems, but it works just fine and I'd expect it to work on most linux system, if only to need slight code modifications to get it to compile with the different kernel versions.

NEXT SLIDE -> 17, tpe-lkm

Introducing tpe-lkm. Or as it's called in the elrepo yum repository, kmod-tpe.

<read bullets>

NEXT SLIDE -> 18, tpe-lkm (cont.)

Here's a slide that shows the simple installation of the security module, and a test to see that it's working.

<read slide>

Now this slide is missing a step, the module doesn't get loaded upon installation anymore. That was a call made by the elrepo guys. So, after install, run modprobe tpe. Rebooting your system will auto load the module, so you don't need to worry about having to get it to load on boot yourself.

And due to how Redhat/CentOS package their kernels, there is no need to update the module when you update the kernel. It uses the stable kernel ABI and will get linked to the new kernel upon installation automatically. This feature curtious of the elrepo guys, and I appriciate their help in getting this module packaged into an RPM, and hosted in their yum repository.

NEXT SLIDE -> 19, tpe-lkm (cont.)

Now, with every security feature comes the possibility that things will break. So, if you're super paranoid about that, you can enable what is called "softmode" to have the trust path checked but not enforced. Things can still execute, but their would-be denial is still logged. As you can see from this slide, the program executed just fine, and a message that it would have been denied was logged.

This should make it easy for you to work out the kinks in your system, permission and ownership issues, etc before you enable the full protection.

NEXT SLIDE -> 20, tpe-lkm (cont.)

Now the module also comes with a few non-TPE related features, that are also from grsecurity, that I put into the module as proof of concept. More security features can be added, I just haven't done so, as they're out of scope with trusted path execution. Like I said before, this is an operational presentation, any developer who wants to take a whack at implementing more security features can use this module as a starting point.

<read slide>


DO DEMO IF I HAVE TIME


NEXT SLIDE -> 21, Get involved

As promised, here is the github link to the source code of the tpe-lkm code, which uses the hijack kernel functions method to implement its security into the system. Also, feel free to email me if you have any questions.

NEXT SLIDE -> 22, Questions