Can a linux server auto-restart on crash?

[modem]

Is it possible to have a CentOS 3.8 server reboot itself when it reaches a point where the server has an "out of memory" error pertaining to a process that maxes out all memory?

My recent server issue was NOT related to missing RPM's, but rather SpamAssassin 3.2.0 seemed to run amok causing an "Out of memory" error on the server console and locked it hard so that even a CTRL-ALT-DEL wouldn't restart it. The engineer had to restart it by powering off and restarting that way.

I am wondering... is there anyway to set a setting in CentOS (or any linux distro) that would allow for a server to reboot itself in case of a memory dump, crash, etc like Windows does with it's bluescreens?

 

[nobaloney]

You can monitor memory and when it reaches a certain point, restart your server, but it's not as easy as you might think because you can't just use the memory usage output of something such as top.

You can use top to look at server load, which is generally a result of your server running out of memory and using swap memory, and then send you the output of top in an email when it reaches a certain point, so you can decide what you want to do.

While you could have it reboot the server, we don't; we like to be able to log in and do forensics before the server gets so bad.

We use a combination of internal monitoring, external monitoring and remote reboot switches.

Jeff

 

[modem]

Thanks for that. I wasn't sure if there was some sort of script or anything to cause a server to reboot on a memory dump or crash like Windows XP/2k3 has.

I mostly rely on monitoring for reboots as the servers are in Seattle WA and I'm in Virginia thus walking on site is a bit prohibitive.

 

[rohit]

Could you please give more info on your findings on SpamAssassin 3.2.0 as what you might reckon is causing the Memory Leaks?

 

[modem]

Actually I believe SA 3.2.0 *might* be the problem, or it might be an associated module. I got Exim 4.67 installed and that's working great, I downgraded to SA 3.1.8 and that's working perfectly normal now. I'm investingating if the HTML:arser 3.55 had anything to do with it as well...

 

[nobaloney]

You can find a link here to a little script we wrote to restart httpd if/when server load gets too high. Note that it should only be used as a temporary workaround.

Jeff