Httpd service down after customization

[twoflies]

Dear forum members,

I'm hoping you could help us with a major problem we are now experiencing with DirectAdmin on a VPS. The VPS provider support is not reachable in weekends, but a solution is urgently needed. As a service to several small companies and a soccer team we are managing a VPS (93.180.66.223) with DirectAdmin installed.

On a Wordpress website (www.vergelijkenmobiel.nl) we had to import some large product feeds. A timeout caused the process to be stopped. In order to set a higher timeout value in the htaccess I needed to set AllowOverride to All via the httpd.conf. As an admin user I went to the form "Httpd.conf Customization for vergelijkenmobiel.nl" and added "AllowOverride All". After saving I refreshed the website and got the message the server was unreachable. In DirectAdmin I went to the Service Monitor and noticed the httpd service was stopped. Clicking Start, Restart or Reload did not work. With all websites down I tried restarting DirectAdmin. A message displayed saying DirectAdmin could be down for a few minutes while restarting. However DirectAdmin did not start anymore, and also SSH, FTP and mail are now down. The only thing I can do right now is restart the VPS via the VPS console, but this does not have any effect at all. I cannot view any logfiles or change any settings.

How could this have happened? How is it possible that restarting DirectAdmin brings down FTP and SSH? What can I do right now? I am desperate for an answer, please help!

Kind regards,

Emiel

 

[Richard G]

Hi Emiel.

I'm Dutch too and might be able to help you, unless you can do it yourself.
First thing to do is to look if you can still reach your server via SSH (Putty) as root. This way you can read out the logfiles to see what exactly was happening.
Normally such setting and restarting the vps or directadmin should not bring down everything.
However, if the server isn't even reachable via SSH, I think you have to wait until support is present again.

If you are interested in my help (I'm prepared to have a short look for free for you) you can send me a pm or email. But it might be tomorrow afternoon until I'm backup online.

 

[twoflies]

Hello Richard,

Thank you for your quick reply. At this moment there is no SSH service, so fixing it without physical access will not be possible I guess. My biggest question right now is how could a simple edit to the configuration and a restart of DirectAdmin result in the whole system going down?

Meanwhile I have received response from support at DirectAdmin, they note the following:

Anytime ssh goes down it usually signifies something bigger than just a simple httpd.conf setting.
Typically, it would be a network issue... sometimes a lack of memory (especially on a VPS, if there is no swap)... or perhaps a disk corruption issue.
We've seen many VPSs run out of ram, so that would be my main suspect.. but couldn't say sure without logging in to confirm.

If you're able to get ssh up, I'd be happy to try and login to see what's going (IP, root pass, and admin pass).. but at a minimum, ssh would need to be running and accessible.
If ssh and DA are not working, then there isn't anything we can really do remotely.

I would love to hear from other people around that experienced similar problems, what caused it and how was it fixed? With the same configuration, would it be save to make changes to the settings in future?

 

[Richard G]

what caused it and how was it fixed?

As DA support and I also said, there is no way of knowing or discovering what caused the problem, without being able to see into logfiles.
Normally an "Allowoverride all" setting should not get the whole system down. Even if it would take httpd and DA down, still other services like network and SSH should still be working.
That's why DA also thinks it's a memory or hardware problem.
Normally this kind of changes should be save in the future.

Best way is to wait until you can access your VPS again. I see also DA supprot would be happy to have a look for you what went wrong. When investigated and a cause is found, then you can also see what to do to prevent it in the future.

 

[zEitEr]

Hello,

According to whois, your IP belongs to www.pcextreme.nl. In case it is the copmany from which you buy a VPS (a KVM based I guess), and if to believe this FAQ article https://community.pcextreme.nl/wiki/Premium_Virtual_Private_Servers#VPS_Console :

De VPS console kunt u zien als het controlepaneel van uw server. Op dit controlepaneel kunt u de status van uw VPS bekijken, uw VPS uitzetten, starten en rebooten, (her-) installeren. De VPS console is eigenlijk een softwarematige vervanging van de behuizing van uw server.

and Google translate:

The VPS console you can see if the control panel of your server. In this control panel, you can view the status of your VPS, your VPS off, start and reboot, (re-) install. The VPS console is basically a software replacement for the housing of your server.

Then you might want to use the console to check your VPS status, and if it's possible even to login there.

 

[twoflies]

Thanks Richard and zEitEr for your replies. I was indeed able to access the VPS console, where I could (re)start the VPS itself, but that did not have any effect on DA or any other services.

However I received a reply from PCextreme, they suggested logging in via VNC instead of SSH. That worked. It appeared that the restart of Debian was hanging on the system password. After entering the system password all services and DirectAdmin were started, with the exception of httpd.

Instead of trying to start httpd via DA I tried via SSH:

root@server:/etc/init.d# httpd -k start
Syntax error on line 55 of /usr/local/directadmin/data/users/mobile/httpd.conf:
Wrapper /fcgi/mobile/public_html/fcgid.sh cannot be accessed: (2)No such file or directory

So httpd couldn't start because the complete folder mobile was missing in fcgi. The problem was resolved by copying another users folder and setting right permissions. After this I could start the httpd service.

cd /fcgi/
cp -R svgg mobile
chown -R mobile:mobile mobile/

I'm very glad the problem has been resolved, however what I know want to know is how could the folder have been deleted? As mentioned I believe the problem started after I added "AllowOverride All" to the httpd.conf customization for the mobile user. That customization is still needed, however I'm not feeling comfortable trying to add it again. If someone over here has an idea how this could have happened I'd like to hear it. I'm currently also in touch with DA support about it, if the answer is found I'll off-course share it here.

Kind regards,

Emiel

 

[zEitEr]

AllowOverride All

The directive by itself can not break your apache and make your server unresponsive either. You must had done something prior to that, that could break apache, and as soon as you restarted apache, the things happened.

Note, to increase timeouts, there is no need to tweak AllowOverride at all. You should either edit php.ini to set greater execution time, or update /etc/httpd/conf/extra/httpd-default.conf/etc/httpd/conf/extra/

# cat /etc/httpd/conf/extra/httpd-default.conf | grep -i timeout
# Timeout: The number of seconds before receives and sends time out.
Timeout 300

P.S. For importing of large amount of data, you might need to enable ignore_user_abort in your PHP application.

ignore_user_abort — Set whether a client disconnect should abort script execution

http://php.net/manual/en/function.ignore-user-abort.php

FALSE by default. If changed to TRUE scripts will not be terminated after a client has aborted their connection.

http://www.php.net/manual/en/misc.configuration.php#ini.ignore-user-abort