HOWTO: Upgrading OpenSSL

The Prohacker

Verified User
Joined
May 23, 2003
Messages
142
Title: Upgrading OpenSSL on Redhat Servers
Revision: 1.0.5
Date: June 28th, 2003
Time: 2:50am CST
Description: Guide for installing and upgrading OpenSSL services on a typical Redhat server to close the very common OpenSSL hole that is plaguing many servers today.
Difficulty: 6/10

------------------------------------------------------------

Disclaimer:
THIS HOWTO IS PROVIDED BY NULL-ROUTE DOT COM "AS IS" AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL NULL-ROUTE DOT COM. OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

------------------------------------------------------------

Introduction:
Many servers today even have a common hole that allows anyone local access to a server as the apache user. The main part of the exploit was found to be in mod_ssl 2.8.12 but many underlying faults lay in OpenSSL installation and libraries bellow version 0.9.6j. Upgrading mod_ssl to version 2.8.14 generally slows an intruder down but will not stop someone who is determined if they know about the older OpenSSL libaries.

This howto will explain how to properly remove older symlinks made by the Redhat installer and give details on the installation of a secure version of OpenSSL.

NOTE: This guide is for installing OpenSSL 0.9.6j. At the time of this writing I felt the 0.9.7x series was still to new for a production environment. But using 0.9.7x should work fine in all instances.

After upgrading the OpenSSL binary and libraries any other program that depends on them will also need to be recompiled to take advantage of the new libraries unless they were compiled with dynamic library support (most applications are compiled statically).

This howto assumes the reader has a general knowledge of Linux and the command line interface for installing applications and has root access to the server. Readers with minimal knowledge of Linux or the processes of compiling and installing software should not attempt this without proper supervision from a system administrator or in a production environment.

Pre-Installation:
Before beginning installation you should verify that your server is indeed open to this exploit because of an older OpenSSL installation. To verify at command line type:

Code:
openssl version
It should return something like this:

Code:
[root@grep root]# openssl version
OpenSSL 0.9.6b [engine] 9 Jul 2001
[root@grep root]#
If your version is bellow 0.9.6j. Then you are most likely open to these exploits and should continue on with this upgrade. If you show version 0.9.6j there is still a very good chance that you are open to this exploit and you must run another check to see if Apache has been compiled with an older version of OpenSSL.

To run this check visit this script: http://null-route.com/httpv.php
Note: This script is privately hosted by myself and will not log any server addresses you enter into it.

After running the second check you should get a result looking like this:


Server domain.com: Apache/1.3.27 (Unix) mod_ssl/2.8.14 OpenSSL/0.9.6b PHP/4.3.2 mod_perl/1.27 FrontPage/5.0.2.2510


As you can see the server is running OpenSSL 0.9.6b and would be open to many exploits even though mod_ssl is running the latest version.

Installation and upgrading:
To begin installation we will need to download OpenSSL 0.9.6j source tar ball to the server we will be upgrading. I generally put all my source files in /usr/src to keep them for later incase I need them.

Code:
# cd /usr/src
# wget [url]http://www.openssl.org/source/openssl-0.9.6j.tar.gz[/url]
# tar -xvzf openssl-0.9.6j.tar.gz
# cd openssl-0.9.6j/
You should now be in /usr/src/openssl-0.9.6j/

We need to now remove all old symlinks to older version of OpenSSL before we begin.

Code:
# rm /usr/lib/libssl.so.0
# rm /usr/lib/libssl.so
# rm /lib/libssl.so.2
# rm /usr/lib/libcrypto.so.0
After removing all old symlinks to old versions of OpenSSL we can begin configuring and compiling the new version.
NOTE: On older systems this may take several hours. For a P4 2.4Ghz equiped with 1gb RAM it took me around 20 minutes to fully configure/compile/install OpenSSL. Your build times will vary depending on equipment and current load of the server.

We now begin by configuring our new installation of OpenSSL with shared libary support and threading support turned off to increase speed with mod_ssl.

Code:
# ./config no-threads shared
After config has ran we will now make/compile our build of OpenSSL. We will use one command, make, but will run it 3 times with differnt operators. Run these commands in this order. These will take some time:

Code:
# make
# make test
# make install
NOTE: make test requires BC math functions to be installed on your server to test the OpenSSL install.

After make install has been ran we will now export our libary path for OpenSSL:

Code:
# echo "/usr/local/ssl/lib" >> /etc/ld.so.conf
# ldconfig -v
We also need to recreate the symlink /lib/libssl.so.2 and /usr/lib/libcrypto.so.0 to our new OpenSSL libary.

Code:
# ln -s /usr/lib/libssl.so.0.9.6 /lib/libssl.so.2
# ln -s /usr/lib/libcrypto.so.0.9.6 /usr/lib/libcrypto.so.0
At this point you are done upgrading OpenSSL but your system isn't secure. You will need to recompile the other applications that used the old OpenSSL libaries at build time to fully seucre your system. Applications such as OpenSSH and Apache+mod_ssl are the ones you will absolutely need to recompile.

NOTE: I will try to do a howto covering the upgrading of OpenSSH within a few days to aid in the proccess.

NOTE: If after buiding OpenSSL openssl version still reports an older version you will need to run make clean and add --prefix=/usr to the config command I used earlier..

Code:
# ./config --prefix=/usr no-threads shared
# make
# make install
Closing:
If you have found any glaring typos, or outdated info in this document, please let me know.
The Prohacker, prohacker[at]null-route.com
 
Last edited:

exposed

Verified User
Joined
Jul 22, 2003
Messages
53
Upgrading was gaving some problem with some older software i have and now i want back to the default openssl rpm from redhat.

Installing the rpm with rpm -ihv goes ok but when i do openssl version i still see's the newest version i updated manual (tar.gz) any idea how i can return to the basic redhat openssl rpm again?
 

The Prohacker

Verified User
Joined
May 23, 2003
Messages
142
To go back you need to install the main openssl rpm and the deval rpm....

I beleive for RH 7.3 and 8:
openssl-0.9.6b-32.7.i386.rpm
openssl-devel-0.9.6b-32.7.i386.rpm
 

thoroughfare

Verified User
Joined
Aug 11, 2003
Messages
575
Hi,

I tried upgrading OpenSSL using the latest version (http://www.openssl.org/source/openssl-0.9.7c.tar.gz) but I've ran into problems.

If I do:

locate libsssl.so.0.9.7c

then it returns nothing, and when I do:

locate libssl

the only thing that looks similar to the file 'libssl.so.0.9.7c' from:
# ln -s /usr/lib/libssl.so.0.9.6 /lib/libssl.so.2

is 'libssl.so.0.9.6b'.

Shouldn't there exist a file 'libssl.so.0.9.7c'?

I can't restart the httpd service because of this, and nothing that relies on libssl is working.

I'd gratefully appreciate any advice.

Many thanks,
Matt
 

thoroughfare

Verified User
Joined
Aug 11, 2003
Messages
575
Actually no, it's httpd still won't restart... and when I tested wget, I got this:

Code:
[root@xxxxxxxxxx customapache]# wget [url]http://files.directadmin.com/services/customapache/build[/url]
--19:22:23--  [url]http://files.directadmin.com/services/customapache/build[/url]
           => `build.2'
Segmentation fault
Help! :)

Matt
 

thoroughfare

Verified User
Joined
Aug 11, 2003
Messages
575
Some progress... when I run

Code:
[root@xxxxx logs]# httpd
I get this:

Code:
Syntax error on line 1506 of /etc/httpd/conf/httpd.conf:
Invalid command 'SSLEngine', perhaps mis-spelled or defined by a module not included in the server configuration
Is this because I need to update the OpenSSL engine too?

cheers,
M
 

ProWebUK

Verified User
Joined
Jun 9, 2003
Messages
2,326
Location
UK
Try running the upgrade again. It seems there was a problem during the upgrade. If this fails i suggest you use another copy of OpenSSL, still stable and secure but hopefully wont give the problem.

Chris
 

thoroughfare

Verified User
Joined
Aug 11, 2003
Messages
575
Fixed it... kinda :)

I used openssl-0.9.6k.tar.gz instead of 0.9.7c, followed the install instructions, and httpd works now :)

Although I may be completely wrong, I think the problem was to do with this: http://www.openssl.org/support/faq.html#MISC6

...version 0.9.7c doesn't include a seperate OpenSSL engine, so it probably didn't exist when httpd was looking for it.

On the other hand, if I type 'httpd' at the terminal just by itself, I still the error:

Code:
Syntax error on line 1506 of /etc/httpd/conf/httpd.conf:
Invalid command 'SSLEngine', perhaps mis-spelled or defined by a module not included in the server configuration
Still not sure what's causing this but at least httpd is happy now :)
 

jasonyates

Verified User
Joined
Sep 24, 2003
Messages
79
Hey,

I get that error as well:

Syntax error on line 1508 of /etc/httpd/conf/httpd.conf:
Invalid command 'SSLEngine', perhaps mis-spelled or defined by a module not incl
uded in the server configuration
Jason
 

ProWebUK

Verified User
Joined
Jun 9, 2003
Messages
2,326
Location
UK
I think i know your problems....


the error line should be something like:

LoadModule ssl_module /lib/libssl.so.X.X.Xa

change the X.X.Xa

with the version stated on the compiled version, eg:

OpenSSL 0.9.7c

I recommend you backup yout httpd.conf before making changes firstly also.

Chris
 

thoroughfare

Verified User
Joined
Aug 11, 2003
Messages
575
Hi,

The line in httpd.conf said:

Code:
LoadModule ssl_module         modules/libssl.so
So I located all instances of files called libssl.so, removed the most of them and replaced them with symlinks, linking to the /usr/local/ssl/lib/libssl.so file which is linked to the actual libssl.0.9.6 file in the same directory. So now all instances of libssl.so are symlinks apart from the actual file itself.

I'm still getting the error though, and now httpd doesn't restart again :)

Good job this isn't a production server :)

Matt
 
Last edited:

thoroughfare

Verified User
Joined
Aug 11, 2003
Messages
575
OK, I found two instances of 'SSLEngine' in httpd.conf, one in the general section of the file towards the beginning, and the second was here:

Code:
<VirtualHost 123.12.12.123:443>
        ServerName localhost
        ServerAdmin [email]webmaster@xxxxxxx.xxx[/email]
        DocumentRoot /var/www/html
        ScriptAlias /cgi-bin/ /var/www/cgi-bin/

       SSLEngine on
       SSLCertificateFile /etc/httpd/conf/ssl.crt/server.crt
       SSLCertificateKeyFile /etc/httpd/conf/ssl.key/server.key
</VirtualHost>
And it seemed it's been these lines that are causing the problem, so commented out just the last 3 (SSLEngine, SSLCertificate*) and now httpd is restarting fine.

I left the first instance of SSLEngine uncommented. So why is one instance causing problems but not the other?

Cheers,
Mattt
 

ProWebUK

Verified User
Joined
Jun 9, 2003
Messages
2,326
Location
UK
you did make backups of the files you removed now.... didnt you? :D
 

thoroughfare

Verified User
Joined
Aug 11, 2003
Messages
575
Erm, nope... didn't think I'd need to... I mean they're all supposed to be the same file, so I just pointed them all at the actual file.

I've got a backup image of my filesystem if I need it.

Plus, I can always just reinstall OpenSSL right?

Sorry if I seem super-dumb at all this, I've been working with PCs since the age of 8 but I've only touched linux in the last year (got RH 8 on my 2nd HDD) and I've never adminned a server before :D Gotta start somewhere tho :D (Before a few days ago, I didn't even know how to use vi, or any other Linux command for that matter.) :D

Matt
 
Last edited:

ProWebUK

Verified User
Joined
Jun 9, 2003
Messages
2,326
Location
UK
ok possibly try the latest openSSL or recompile the one you just installed, if you have problems restarting apache check the error location and the line, take a look in the file, go the correct line (press ctrl + c to check your current line - in pico) and check what i stated above........ also check the any paths on the specified line or just above exist.

Chris
 

thoroughfare

Verified User
Joined
Aug 11, 2003
Messages
575
Hi,

I have been using service httpd restart; I used httpd before because it proved helpful in finding out what was stopping httpd from restarting when service httpd restart wasn't outputting any errors but was still failing.

I've been researching my problem and it seems there are others who've experienced the same, so I'm going to work on it for a while and post later with a fix hopefully ;)

Thanks for everyone's help so far...

Matt :D
 
Top