[merged] Serious Kernel Exploit - Affects x86_64 (including default rhel5)

[linuxperts]

Sure, check out:
http://wiki.openvz.org/Download/kern...5/028stab070.5

Thanks Waseem.

[Dave W]

This is why we run ksplice on our openvz nodes. It really does turn these things into a non-event.

[CodyRo]

This is why we run ksplice on our openvz nodes. It really does turn these things into a non-event.

In this case if the exploit code was ran previously it would have left a backdoor in memory allowing it to be compromised after the ksplice update. The only fix for this currently is to reboot the machine.

In addition to this we (and some others) have had issues with ksplice on OpenVZ nodes which causes the CPU to spike and take ages to install a single update. Imagine rebooting using an older kernel and having to apply 30+ of these..

Needless to say relying solely on ksplice doesn't seem like a good idea especially for something as critical as this (in a shared environment at least) .

**EDIT** Oh and we love ksplice, just saying it's not the end all.

[Dave W]

Well of course you test for an exploit like this but we found that all of our 64 bit ksplice nodes were already patched by ksplice.

[DigitalLinx]

In this case if the exploit code was ran previously it would have left a backdoor in memory allowing it to be compromised after the ksplice update. The only fix for this currently is to reboot the machine.

In addition to this we (and some others) have had issues with ksplice on OpenVZ nodes which causes the CPU to spike and take ages to install a single update. Imagine rebooting using an older kernel and having to apply 30+ of these..

Needless to say relying solely on ksplice doesn't seem like a good idea especially for something as critical as this (in a shared environment at least) .

**EDIT** Oh and we love ksplice, just saying it's not the end all.

It does not leave a backdoor, it just changes the LSM value, its just a way to fingerprint if the server was exploited previously using the public exploit.

The LSM value is reset upon reboot, it is also possible for the attacker to restore the LSM value after he exploited the vulnerability, just because the ksplice tool says 'no backdoor found' does not mean you're safe or not exploited previously.

[Steven]

Well of course you test for an exploit like this but we found that all of our 64 bit ksplice nodes were already patched by ksplice.

If a vps was exploited prior to the ksplice patch (it was in the wild for a couple days before ksplice had a patch) then you could still be vulnerable.

[Steven]

I think that's just a version they patched themselves and not the official update from Red Hat.

As stated earlier in the thread, it was a dev kernel from centos themself.

[SPaReK]

Official CentOS updated kernel is now available. Should be making it to repositories soon. Or directly at:

http://centosq4.centos.org/centos/5....el5.x86_64.rpm

[LynxUser]

GOOD GOOD GOOD news

Be back shortly if system fails to boot

[CodyRo]

GOOD GOOD GOOD news

Be back shortly if system fails to boot

Hah, we haven't had any issues. If anyone is curious Facebook mirror has the new kernel synced.

[LynxUser]

I swear its not me ! But its like a jinx, Every kernal I do the damn box does not boot up lol, I have to phone the DC to get me online... They even know me by my voice now it happened that many times

Anyhow, Booted fine this time.. Looking all good so far and things are stable, I was more worried about mysql issues, Since I applied the patch originaly I could hardly keep the sql online, Only issue is with WHM setups.

This seemed to do the trick tho... Fingers crossed.

[Steven]

Its in yum now.

[pueblosnet]

I'm moving accounts to a new sever with 32bits kernel

[Dave W]

You are downgrading over this?

[CodyRo]

Its in yum now.

That would be dependent on the repositories you're using w/ CentOS.. not all mirrors have it yet.

[pueblosnet]

yes, welcome back to 32bits!

[VPSForge-Ray]

Good thing we havent upgraded all of our servers to 64bit. Less work fixing this if any of our servers is exploited

[hostineuro]

I can't think of any reason why it wouldn't, I was playing around with it on a CentOS 5.5 VM in VirtualBox which was completely vulnerable, so I don't think virtualization would really get in the way at all.

Apparently openvz is safe unless exploit is used on HW node. It was tested by some members from openvz forum.

[Boris A Dolgov]

I'm moving accounts to a new sever with 32bits kernel

It is very strange - to use 32bit system in 2010

[pueblosnet]

yes, and secure

[hostineuro]

yes, and secure

Secure until new exploit for 32bit kernel

[DigitalLinx]

Apparently openvz is safe unless exploit is used on HW node. It was tested by some members from openvz forum.

Wrong, the openvz breakout code was intentionally removed from ABftw.c

[wdaher]

Wrong, the openvz breakout code was intentionally removed from ABftw.c

Right, exactly.

More generally, once you're executing arbitrary code in the kernel, you already have a potential breakout exploit on your hands -- you just need to be a little bit clever about getting it to work.

This is just one of the properties inherent to Virtuozzo/OpenVZ: if all of your containers are sharing the same kernel, and one container can do arbitrary things to the kernel, it can affect all the other containers.

[LynxUser]

I can only imagen openVZ servers, Everyone will be affected rather than just the one user, so my theory is, If you have a vps, You run the ksplice and it says its clean.. But reality the main system could be compromised and not just the one user so it won't show on that account ?

Correct me if I'm wrong.

[wdaher]

I can only imagen openVZ servers, Everyone will be affected rather than just the one user, so my theory is, If you have a vps, You run the ksplice and it says its clean.. But reality the main system could be compromised and not just the one user so it won't show on that account ?

Correct me if I'm wrong.

A few comments:
First, on OpenVZ, all of the containers share one kernel, so if a change was made in the kernel, it would affect all the containers. So, no, that's not quite right.

But the situation is subtle: as folks have mentioned, the Ksplice test tool looks for backdoors left by the high-profile exploit code, ABftw.c. The unmodified version of this exploit doesn't work on OpenVZ (it's had that portion of the code removed), so our detector tool also won't do anything useful there.

In general, no one can write a checker that's 100%: if an attacker has root on the system, they can do arbitrarily clever things to hide themselves. For example, a sufficiently clever attacker could modify ABftw.c not to leave those backdoors (or could modify it to work on OpenVZ), and then the Ksplice tool would not detect that the system has been compromised.

So in general, if you suspect your system has been compromised, you should treat it as such. The diagnostic tool we provided isn't a general rootkit checker or a tool that says "Yes, your systems are totally free of any and all attackers". It's answering a very specific question: Has someone already run ABftw.c (without any modifications) on this system?

Did that answer your question?

[feliper]

Hi, my server hacked for this problen, how to I solved it? I use CENTOS 5.5 x86_64 standard with cPanel/WHM.

[crucialx]

Hi, my server hacked for this problen, how to I solved it? I use CENTOS 5.5 x86_64 standard with cPanel/WHM.

Best practice is do a complete re-install and restore, update your kernel, and restore from backups.

[LynxUser]

Hi, my server hacked for this problen, how to I solved it? I use CENTOS 5.5 x86_64 standard with cPanel/WHM.

Move user backups / re-install server -restore backups - This is the only way you know your safe.

[feliper]

OK, and I have other 30 servers with this config, i executed comand:

[email protected] [~]# yum upgrade
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding Packages in global exclude list
Finished
Setting up Upgrade Process
Resolving Dependencies
There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them.
The program yum-complete-transaction is found in the yum-utils package.
--> Running transaction check
---> Package kernel.x86_64 0:2.6.18-194.11.3.el5.centos.plus set to be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================================== =========================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================================== =========================================================================================================================
Installing:
kernel x86_64 2.6.18-194.11.3.el5.centos.plus centosplus 21 M

Transaction Summary
====================================================================================================================================================== =========================================================================================================================
Install 1 Package(s)
Upgrade 0 Package(s)

Total download size: 21 M
Is this ok [y/N]: y
Downloading Packages:
kernel-2.6.18-194.11.3.el5.centos.plus.x86_64.rpm | 21 MB 00:01
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : kernel 1/1

Installed:
kernel.x86_64 0:2.6.18-194.11.3.el5.centos.plus

Complete!
[email protected] [~]#




This solved this problen?

[Dougy]

yes, welcome back to 32bits!

That's silly. There is plenty of exploits which leave 32 bit open too.

yes, and secure

not quite bud

[Steven]

OK, and I have other 30 servers with this config, i executed comand:

[email protected] [~]# yum upgrade
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding Packages in global exclude list
Finished
Setting up Upgrade Process
Resolving Dependencies
There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them.
The program yum-complete-transaction is found in the yum-utils package.
--> Running transaction check
---> Package kernel.x86_64 0:2.6.18-194.11.3.el5.centos.plus set to be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================================== =========================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================================== =========================================================================================================================
Installing:
kernel x86_64 2.6.18-194.11.3.el5.centos.plus centosplus 21 M

Transaction Summary
====================================================================================================================================================== =========================================================================================================================
Install 1 Package(s)
Upgrade 0 Package(s)

Total download size: 21 M
Is this ok [y/N]: y
Downloading Packages:
kernel-2.6.18-194.11.3.el5.centos.plus.x86_64.rpm | 21 MB 00:01
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : kernel 1/1

Installed:
kernel.x86_64 0:2.6.18-194.11.3.el5.centos.plus

Complete!
[email protected] [~]#




This solved this problen?

You will need to install 2.6.18-194.11.4 and get off the centos plus branch.

[Steven]

For what its worth - Redhat enterprise 4 / Centos 4 is potentially exploitable, not by the public exploit, but someone with some knowledge could probably do it.

From the Redhat Advisory:

Red Hat Enterprise Linux 4 and Red Hat Enterprise MRG
The Red Hat Enterprise Linux 4 and Red Hat Enterprise MRG kernels do not include a backport of the upstream git commit 42908c69; therefore, those kernels do not include compat_mc_getsockopt(). We plan to backport the missing compat_alloc_user_space() sanity checks in future Red Hat Enterprise Linux 4 and Red Hat Enterprise MRG updates.

Note: Red Hat Enterprise Linux 4 and Red Hat Enterprise MRG are not affected by the publicly-circulated exploit.

Redhat 4 / Centos 4 does not have compat_mc_getsockopt(), but it does not mean the exploit couldn't take place in another function that utilizes compat_alloc_user_spac if someone was to find it.

Redhat will release a patch, but it sounds like its low priority. I don't know about other people, but we will be manually patching our Rhel4/Centos4 customers until its fully resolved. Better to be safe than sorry.

[feliper]

You will need to install 2.6.18-194.11.4 and get off the centos plus branch.

how to install this version? u cam help-me?

[Steven]

how to install this version? u cam help-me?

If your not using any of the special features in the centos-plus kernel you can run:

yum install kernel-2.6.18-194.11.4.el5

[feliper]

not work:

[email protected] [~]# yum install kernel-2.6.18-194.11.4.el5
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
addons | 951 B 00:00
base | 2.1 kB 00:00
centosplus | 1.9 kB 00:00
extras | 2.1 kB 00:00
update | 1.9 kB 00:00
Excluding Packages in global exclude list
Finished
Setting up Install Process
No package kernel-2.6.18-194.11.4.el5 available.
Nothing to do
[email protected] [~]#

[dclardy]

Does anyone have any steps to see if you were hacked? I ran the check tool after I updated kernel and rebooted. Need some help here just to be sure. The two major rootkit checkers show nothing, but I know they are pretty much useless.

[Steven]

Does anyone have any steps to see if you were hacked? I ran the check tool after I updated kernel and rebooted. Need some help here just to be sure. The two major rootkit checkers show nothing, but I know they are pretty much useless.

Every hack is different, there is no clear steps to finding a hack that involves a root compromise.

[feliper]

If your not using any of the special features in the centos-plus kernel you can run:

Steven, can u help-me? Not work u info for me. Look result:

[email protected] [~]# yum install kernel-2.6.18-194.11.4.el5
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
addons | 951 B 00:00
base | 2.1 kB 00:00
centosplus | 1.9 kB 00:00
extras | 2.1 kB 00:00
update | 1.9 kB 00:00
Excluding Packages in global exclude list
Finished
Setting up Install Process
No package kernel-2.6.18-194.11.4.el5 available.
Nothing to do
[email protected] [~]#

[UNIXy]

It's disappointing to see places online monetizing this security flaw. If you're one of those places, keep in mind that people remember you for how you helped them and not for how much you charged them.

Regards
Joe / UNIXY

[dclardy]

Does anyone have the name of someone who could check out server for a reasonable price? I am not a security expert, and I am pretty sure there is nothing wrong.