"Invalid Certificate" Error

pmet

Verified User
Joined
Oct 9, 2015
Messages
18
Hi Guys,

First of all, let me introduce myself as an experience admin with more than 10 years of experience.

I'm having a problem install any SSL after successfully cloning a server using R1soft CDP. My ailing (old) server has been decommissioned after the cloning and and I've updated my new primary IP address using ipswap.sh in the scripts directory. My secondary ip address have been migrated to my new server so there are no changes required. So, long story short, my cloning has been very successful and directadmin and all hosted websites are working fine.

However, I'm running into a very very strange problem with the SSL. I'm not able to install any SSL, be it Godaddy, Thwate or Comodo. I know how to install SSL and have been doing it on directadmin for over 7 years so please don't discount me on that. The error message I got on the admin site (2222) was:

Could not execute your request

Details

Certificate is Invalid

The error I got from the debug mode /usr/local/directadmin/directadmin b2000 was:

2FGbPb5sVJmRBkiUrFN03KYBzeFIK5gV4JzdtBWAWCsd5Guf8%2Ftm2%0D%0A9mNVz%2FSYSpsbQ2PzSUqUxSxdZI4KZed6VYJw%2BXE9XHTD95EAVxLnJINhkw%2F8igcM%0D%0AM0w5blbrLaB1keu5tRVuOSzv81oqZwslpdAn3ifAt5rA1J2NasLQWBfwscoJKaR%2F%0D%0AN1ztWTKYPL8iZA%3D%3D%0D%0A-----END+CERTIFICATE-----%0D%0A&submit=Save
unable to write certificate
19668:error:09072007:pEM routines:pEM_write_bio:BUF lib:pem_lib.c:595:

I've checked all over and all files and directories in /usr/local/directadmin/ and /etc/pki and they have the exact same ownership and permission as on my old server. I've also checked /tmp and /var/tmp and there are all ok. I've also checked the output of ps aux | grep directadmin and the results were several directadmin instances running as nobody.

I did a strace on the directadmin instances using "strace $(/sbin/pidof directadmin |sed 's/\([0-9]*\)/\-fp\1/g') 2> test" and saw somewhere along the line where the error occured but it's not making any sense to me:

[pid 6262] read(0, "-", 1) = 1
[pid 6262] read(0, "-", 1) = 1
[pid 6262] read(0, "-", 1) = 1
[pid 6262] read(0, "-", 1) = 1
[pid 6262] read(0, "\n", 1) = 1
[pid 6262] fstat64(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
[pid 6262] mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7fc7000
[pid 6262] write(1, "Modulus=BC8F2FF6684B70E316570153"..., 1024) = -1 EPIPE (Broken pipe)
[pid 6262] --- SIGPIPE (Broken pipe) @ 0 (0) ---
[pid 6262] write(2, "unable to write certificate\n", 28) = -1 EBADF (Bad file descriptor)
[pid 6262] write(2, "6262:error:09072007:pEM routines"..., 70) = -1 EBADF (Bad file descriptor)
[pid 6262] exit_group(1) = ?

Now, let me provide more details on various modes of SSL installation. If I use "Create your own self signed certificate", there would be no problem and I've checked that the cert and key files are updated in /usr/local/directadmin/data/users/username/domains with the respective self-signed cert and key. There's also seeming no problem when I create a CSR, with the new private key created and updated in the key file. The problem comes when I select "Paste a pre-generated certificate and key" and save. The above error will be produced no matter what I paste into the box. Even directadmin's own self-signed certificate would produce the same error when I paste the cert and key pair into the "pre-generated certificate and key" box.

I suspected directadmin was having a problem verifying any SSL certificate. So I went in and run "cat username.com.cert | openssl x509 -modulus" as the diradmin user and compared it with the results from "cat username.com.key | openssl rsa -modulus". The outputs are conclusive that openssl is working properly and the cert/key pair is matching.

I'm running out of idea as to what might have gone wrong with the SSL checking mechanism behind the directadmin admin panel but the strace output gives me a feeling that some files might be inaccessible and that resulted in an error of "Bad file descriptor" which interrupted the SSL verification process.

Lastly, I'm not able to apply any valid SSL certificate to any of my hosted domain names created in this hierarchy:

Admin --> Reseller -- User --> domains

Sorry for the long post as I'm desperate and could use any available help or clues. Please help me and thank you all for your time!
 
Can anyone provide some pointers to what openssl might be writing to the disk or memory?
 
write(1) is stdout, back to DA.
write(2) is stderr... which DA has closed for that scenario because 2>&1 is added to the end of the call, routing stderror to stdout.
So.. somewhere along the lines, the stdout pipes has broken.

Running DA in debug mode changes how DA closes it's top-level pipes, so try debug mode and see what that does:
http://help.directadmin.com/item.php?id=293

Debug level 500 should be enough.. you'd be looking for 2 lines that start with:
singleCertValid
it should output the return code and all text output by the script.

Also, adding this to the directadmin.conf will pipe anything that might be uncaught, which could apply in this case:
pipe_log=/var/log/directadmin/pipe.log
but it's for stderr, so I don't think it will may any difference.. but worth doing in case something is missed (goes to /dev/null by default)

As for the strace, that's handy.. possibly more info from higher up would help... I can read it somewhat.

John
 
Hi John,

Thank you for the guide. I'm still not sure what's wrong with DA that it's unable to accept the pipe from openssl. I did a debug mode before if you refer to my first post above but went ahead to do it again this time with "./directadmin b500". Below's the output (obviously I had to censor out some info with ***):

Accepting Connections on port 2222
0: Accept-Encoding: gzip, deflate
1: Accept-Language: en-us
2: Accept: text/html, application/xhtml+xml, */*
3: Cache-Control: no-cache
4: Connection: Keep-Alive
5: Content-Length: 4084
6: Content-Type: application/x-www-form-urlencoded
7: Cookie: session=HDZ4hDZLDs61P2IJjOJkFazfuwLMhRmweL0KZYfkOljnlz7UNODdzCGr8aDn1zjg
8: Host: ***.com:2222
9: Referer: https://***.com:2222/CMD_SSL?DOMAIN=***.com
10: User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)
Post string: domain=***.com&action=save&country=&province=&city=&company=&division=&name=www.***.com&email=&type=paste&certificate=-----BEGIN+RSA+PRIVATE+KEY-----%0D%0AMIIEpQIBAAKCAQEAvI8v9mhLcOMWVwFTClXmV9Z5TI9FOpaY2k2TUOryCH9nNxxV%0D%0AE4ekSzE5cyJoKtfg6FP5eNDoAk545QffKqiWaISvMXhmtt%2Ba1EiiAJumpxsmplS3
***
%0D%0AHQvNa2xVVnGDtgJq%2F6NckJhU5Orx6C9vNaXYFG%2FLv%2BOosIykvKP5B7v1AFLGQgsG%0D%0AYGpedD7d8mL%2F975%2BLfCcw5omtDGdzaeZgN%2BrPQFMDjbSz3J1EVcyiCI%3D%0D%0A-----END+RSA+PRIVATE+KEY-----%0D%0A-----BEGIN+CERTIFICATE-----%0D%0AMIIFFjCCA%2F6gAwIBAgIRAKnpU%2F%2BvN%2BxzaYx6YIrDfiMwDQYJKoZIhvcNAQEFBQAw%0D%0AcjELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G
***
%0D%0A9mNVz%2FSYSpsbQ2PzSUqUxSxdZI4KZed6VYJw%2BXE9XHTD95EAVxLnJINhkw%2F8igcM%0D%0AM0w5blbrLaB1keu5tRVuOSzv81oqZwslpdAn3ifAt5rA1J2NasLQWBfwscoJKaR%2F%0D%0AN1ztWTKYPL8iZA%3D%3D%0D%0A-----END+CERTIFICATE-----%0D%0A&submit=Save
unable to write certificate
19668:error:09072007:pEM routines:pEM_write_bio:BUF lib:pem_lib.c:595:
writing RSA key
unable to write key
19673:error:09072007:pEM routines:pEM_write_bio:BUF lib:pem_lib.c:595:

Also, adding the line to directadmin.conf did nothing as no log was generated either in debug or daemon mode.

According the the output from strace, I can deduce that the pipe back to DA was broken for some reasons but I don't get it. How can the pipe which is an integral part of Linux be broken? I've no problem piping commands logged in as root though.
 
On the same note, could directadmin be writing to a temp dir which it doesn't have access to? I checked /var/tmp and it seems to be ok:

[root@hosting directadmin]# ls -alh /var/tmp
total 16K
drwxrwxrwt 2 root root 4.0K Oct 3 22:31 .
drwxr-xr-x 21 root root 4.0K Jun 4 2014 ..


What else should I be checking?
 
Are you sure permissions are right in /usr/local/directadmin

Have you ran the fix permissions script in /usr/local/directadmin/scripts
 
@pmet:

so it looks like it's openssl related.
The "type=paste" let's us dig in further to call what DA calls.

Following the code all the way down to where DA does the check, this is the command used by DA:
Code:
/usr/bin/openssl x509 -modulus
and then you'd paste in, just the certificate, into stdin.
Ctrl-d should close the stdin file descriptor so openssl closes (don't press it twice, or you close the sshd file descriptor, and you get booted out of ssh)

Also looking at DA's call, if you're testing the server IP (eg: set as an Admin on a server IP), then it will run as root.
But if you're logged into DA on an owned IP, then it will be run as "diradmin"... so this might be related to the permission issue..
again, I'm suspect of /tmp permissions but really not where where openssl saves anything.

Other guesses would be to check /dev/null in case openssl is trying to write to that,. eg:
crw-rw-rw- 1 root root 1, 3 May 19 13:14 /dev/null

John
 
Are you sure permissions are right in /usr/local/directadmin

Have you ran the fix permissions script in /usr/local/directadmin/scripts

I've done it. Permissions are all set correctly and the cert is still invalid. All errors remain the same.
 
@pmet:

so it looks like it's openssl related.
The "type=paste" let's us dig in further to call what DA calls.

Following the code all the way down to where DA does the check, this is the command used by DA:
Code:
/usr/bin/openssl x509 -modulus
and then you'd paste in, just the certificate, into stdin.
Ctrl-d should close the stdin file descriptor so openssl closes (don't press it twice, or you close the sshd file descriptor, and you get booted out of ssh)

Also looking at DA's call, if you're testing the server IP (eg: set as an Admin on a server IP), then it will run as root.
But if you're logged into DA on an owned IP, then it will be run as "diradmin"... so this might be related to the permission issue..
again, I'm suspect of /tmp permissions but really not where where openssl saves anything.

Other guesses would be to check /dev/null in case openssl is trying to write to that,. eg:
crw-rw-rw- 1 root root 1, 3 May 19 13:14 /dev/null

John

Hi John,

Yes, I assumed it was an openssl problem too. I tried running openssl twice, once with rsa and the other with x509. Results from the rsa and x509 came out fine with their modulus outputs matching. I ran it first with root and a second time with diradmin without any difference. However, as opposed to what you mentioned, I didn't have to use CTRL-D; it just kicked my out of stdin after I pasted the cert and pressed ENTER. I'm not sure if there's anything wrong with that though. Below's the output:

[root@hosting admin]# /usr/bin/openssl x509 -modulus
-----BEGIN CERTIFICATE-----
MIIFFjCCA/6gAwIBAgIRAKnpU/+vN+xzaYx6YIrDfiMwDQYJKoZIhvcNAQEFBQAw
cjELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G
A1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxGDAWBgNV
...
9mNVz/SYSpsbQ2PzSUqUxSxdZI4KZed6VYJw+XE9XHTD95EAVxLnJINhkw/8igcM
M0w5blbrLaB1keu5tRVuOSzv81oqZwslpdAn3ifAt5rA1J2NasLQWBfwscoJKaR/
N1ztWTKYPL8iZA==
-----END CERTIFICATE-----
Modulus=BC8F2FF6684B70E3165701530A55E657D6794C8F453A9698DA4D9350EAF2087F67371C551387A44B31397322682AD7E0E853F978D0
...
-----BEGIN CERTIFICATE-----
MIIFFjCCA/6gAwIBAgIRAKnpU/+vN+xzaYx6YIrDfiMwDQYJKoZIhvcNAQEFBQAw
cjELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G
A1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxGDAWBgNV
...
hW3DCnI5MrKr/GbPb5sVJmRBkiUrFN03KYBzeFIK5gV4JzdtBWAWCsd5Guf8/tm2
9mNVz/SYSpsbQ2PzSUqUxSxdZI4KZed6VYJw+XE9XHTD95EAVxLnJINhkw/8igcM
M0w5blbrLaB1keu5tRVuOSzv81oqZwslpdAn3ifAt5rA1J2NasLQWBfwscoJKaR/
N1ztWTKYPL8iZA==
-----END CERTIFICATE-----
[root@hosting admin]# /usr/bin/openssl rsa -modulus
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEAvI8v9mhLcOMWVwFTClXmV9Z5TI9FOpaY2k2TUOryCH9nNxxV
E4ekSzE5cyJoKtfg6FP5eNDoAk545QffKqiWaISvMXhmtt+a1EiiAJumpxsmplS3
S1Buw3nq94knodHCT0CKHaIURShdxHtlI0ypsYjSlmHbGom2XZ0i+yfOuK9l3g1F
...
r4hGmU8CgYEAl3gjk8hwoLwx/XyAttI2FUwDNgDVOtFjaHgOB9qBnDNGc/DlqPBo
HQvNa2xVVnGDtgJq/6NckJhU5Orx6C9vNaXYFG/Lv+OosIykvKP5B7v1AFLGQgsG
YGpedD7d8mL/975+LfCcw5omtDGdzaeZgN+rPQFMDjbSz3J1EVcyiCI=
-----END RSA PRIVATE KEY-----
Modulus=BC8F2FF6684B70E3165701530A55E657D6794C8F453A9698DA4D9350EAF2087F67371C551387A44B31397322682AD7E0E853F978D0
...
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEAvI8v9mhLcOMWVwFTClXmV9Z5TI9FOpaY2k2TUOryCH9nNxxV
E4ekSzE5cyJoKtfg6FP5eNDoAk545QffKqiWaISvMXhmtt+a1EiiAJumpxsmplS3
S1Buw3nq94knodHCT0CKHaIURShdxHtlI0ypsYjSlmHbGom2XZ0i+yfOuK9l3g1F
...
r4hGmU8CgYEAl3gjk8hwoLwx/XyAttI2FUwDNgDVOtFjaHgOB9qBnDNGc/DlqPBo
HQvNa2xVVnGDtgJq/6NckJhU5Orx6C9vNaXYFG/Lv+OosIykvKP5B7v1AFLGQgsG
YGpedD7d8mL/975+LfCcw5omtDGdzaeZgN+rPQFMDjbSz3J1EVcyiCI=
-----END RSA PRIVATE KEY-----


I ran set_permissions.sh and checked the following:

crw-rw-rw- 1 root root 1, 3 Oct 26 18:04 /dev/null
drwxrwxrwt 5 root root 96K Oct 26 18:26 tmp


Permission seems to be fine. But I'm still getting the "Invalid cert" error.

Any other ways to debug this pesky issue?
 
Can you create a ticket?
https://tickets.directadmin.com

Attach/upload a .txt file with the certificate/key pair and I'll try running them through on one of our test boxes to see if they behave differently here.
Also include which OS you're running.

John
 
Ok, I created an account and posted the following in the ticket:

Hi John,

I'm relaying the information from this thread:
https://forum.directadmin.com/showthread.php?t=52114

The following certificate/key pair is for another domain, but you get the idea. I've even reissued the certificate via Comodo more than 5 times to no avail. Also, the self-signed cert generated from directadmin would work the first time, but when I copy/paste the respective cert/key pair into the textbox on the SSL page, directadmin would reject it with "Invalid Certificate" error. It clearly isn't the problem with the cert but DA or openssl itself. So anyway, here's one of the cert/key I've mentioned:

[root@hosting nonono]# openssl rsa -modulus
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA4TrtFWgphEF7kUcDlS1s7hUwPojEA4ALD9Cp31v6qnxVSUfV
c+DMnELwuY9l/CP3yMhcvHZI1QDJyBDIrVZgfXEU0EtvSsKZTD9yGDVPuxHk+aoF
ihiOgp1/U2/Jz+5z4NPYru0FT6WjkZhvvL1xZE28DSDBW7hnMTpNVpiGTT8O63yj
...
21ITdDkCgYEAjNFrZPA9rZodZqVqF8QCLp9QPH2uxiXqFiz+IlZvKlliV9namqKv
mfqilhQ/4qA9BFd0Rzur015zpnqL3InqmW3eGM64lncxQikItanyyfAaJ1MYsURs
Sou4T4sLWXitdYpE+O5NWD8/CnbrLrgIWUIEn8IAmIQnj0R9e5J69bs=
-----END RSA PRIVATE KEY-----
Modulus=E13AED15682984417B914703952D6CEE15303E88C403800B0FD0A9DF5BFAAA7C554947D573E0CC9C42F0B98F65FC23F7C8C85CBC7
...
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA4TrtFWgphEF7kUcDlS1s7hUwPojEA4ALD9Cp31v6qnxVSUfV
c+DMnELwuY9l/CP3yMhcvHZI1QDJyBDIrVZgfXEU0EtvSsKZTD9yGDVPuxHk+aoF
ihiOgp1/U2/Jz+5z4NPYru0FT6WjkZhvvL1xZE28DSDBW7hnMTpNVpiGTT8O63yj
...
21ITdDkCgYEAjNFrZPA9rZodZqVqF8QCLp9QPH2uxiXqFiz+IlZvKlliV9namqKv
mfqilhQ/4qA9BFd0Rzur015zpnqL3InqmW3eGM64lncxQikItanyyfAaJ1MYsURs
Sou4T4sLWXitdYpE+O5NWD8/CnbrLrgIWUIEn8IAmIQnj0R9e5J69bs=
-----END RSA PRIVATE KEY-----
[root@hosting nonono]# openssl x509 -modulus
-----BEGIN CERTIFICATE-----
MIIFbDCCBFSgAwIBAgIRAIMf5fl6/6+AAy/BbrcX74MwDQYJKoZIhvcNAQELBQAw
gZAxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTYwNAYD
...
BYHYHhZGHluS/OkrvHb6N11zdcimQToiCT6piZCHuZgEUXQV2yw7N8F3cEBtfC8Q
IiCUmuODz+JjlNn6ObSXv7v8xHr4BpwWhHHPp17cng3dFWa+F4LSKkhJNfiVdfy8
OafXHhiYd8xZwJaAyo+WMbAtJGmZ4bVQgzt0C77XB9V2tQAeY0uDNkPAFcTiQ27W
-----END CERTIFICATE-----
Modulus=E13AED15682984417B914703952D6CEE15303E88C403800B0FD0A9DF5BFAAA7C554947D573E0CC9C42F0B98F65FC23F7C8C85CBC7
...
-----BEGIN CERTIFICATE-----
MIIFbDCCBFSgAwIBAgIRAIMf5fl6/6+AAy/BbrcX74MwDQYJKoZIhvcNAQELBQAw
gZAxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTYwNAYD
...
BYHYHhZGHluS/OkrvHb6N11zdcimQToiCT6piZCHuZgEUXQV2yw7N8F3cEBtfC8Q
IiCUmuODz+JjlNn6ObSXv7v8xHr4BpwWhHHPp17cng3dFWa+F4LSKkhJNfiVdfy8
OafXHhiYd8xZwJaAyo+WMbAtJGmZ4bVQgzt0C77XB9V2tQAeY0uDNkPAFcTiQ27W
-----END CERTIFICATE-----

Please let me know.

Thanks!
 
Hi John,

Thanks for the fix! Can you share what was changed in this fix so I can bring it over manually?
 
It's explained in the id=1799. Fairly sure it was just a timing issue with how DA was incorrectly waiting for the openssl output, but a pause in the output threw things off (2 different chunks of data, once instant, the other about 1" later)
I totally scrapped that method (piping the data from stdin from DA), and changed it to stdin in usnig the "openssl < /file.txt" method.
So the new DA binaries are needed for the fix.
 
Hi John,

Can you let me know what's being entered into stdin when "openssl rsa -modulus" is called?
 
The stdin for RSA -modulus would be the key itself, everything from/including -----BEGIN RSA PRIVATE KEY----- all the way to the end of -----END RSA PRIVATE KEY-----.
 
Hi John,

I've 3 questions for you and would appreciate your clarifications:

1. Sorry, just tested and it still isn't fixed with an openssl reinstall. I'm not in the position to upgrade directadmin for now, is there another fix which I can apply manually?

2. Also, it seems that you may be right that it's a timing issue afterall. All the tests which I've done did nothing to fix it and it's still throwing invalid cert error despite using a newly purchased cert. Any idea why this may happen on the cloned system but not on the parent?

3. You mentioned that if I run directadmin in debug level 500, I should find something with "singleCertValid" along the line. However, I don't see it in any of the output (I've tried with debug level from 500 to 2000 to 10000). Is it because I'm still using an old version of directadmin? I'm running v.1.31.1

Thanks!
 
Last edited:
Realized that I can't reply to post so I edited my original message above. Thx!
 
Back
Top