Memory Leak

[IT_Architect]

I've been battling a memory leak since the beginning of December. The only reason that I am bringing it up here is because there is one significant event that happens at 00:40:00, which restores my RAM. (See ClipA attached) Nothing else short of a reboot will do that. Restarting any of the processes from within DA does not restore the ram. The only thing in crontab at that time are these DirectAdmin processes, and the time it happens favors the second one.

30 0 * * * root echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue
40 1 1 * * root echo 'action=reset&value=all' >> /usr/local/directadmin/data/task.queue

My thought is that by learning what DA does at that time, I may be able to determine the source of the leak.

I've tried about everything else. The php.ini doesn't load Zend or anything else anymore. I've tried lots of things with Apache, including MaxMemFree, KeepAliveTimeout, MaxKeepAliveRequests, but nothing I do in Apache seems to make any difference. At some point the CPU will go to 100% / 460 to 811 load, and stay there. I can log in and restart Apache, and after a time or two of that, I usually get past it, but never get much free back like I do at 00:40 or a reboot. What happens at 3? A backup starts at 1:30, finishes at 3, at which point the snapshot is removed.

The software I'm currently running is:
FreeBSD FreeBSD 7.2-RELEASE-p2 #0: Wed Jun 24 00:14:35 UTC 2009 [email protected]:/usr/obj/usr/src/sys/GENERIC amd64
3 Gigs available RAM locked in.
Apache 2.2.17 Prefork CLI
Php 5.2.17
MySQL 5.0.77
Exim 4.71
dovecot 1.2.16
ProFTPd 1.3.3d
Named 9.4.3

Lowest traffic is at 3:00 and highest is 14:30 to 15:30 Eastern.

It's a head scratcher!

Thanks!

 

[IT_Architect]

What DA does at 00:40 each morning was indeed the key to solving the problem. No matter what kind of restart I did to Apache, it made little difference, but DA at 00:40 had the secret sauce. The answer to the riddle is the traffic on that website has grown by 85% in 11 months. By 00:40, my Apache log files are several gigs in size. I deleted the log files, and did an Apache restart, and bingo! Gigs of free RAM instantly available. And why the abrupt drop in free memory at 3:00? I have a theory for that too. Backup takes a snapshot prior to starting. Writes to disk are written to separate area for the duration of the snapshot. After the backup completes, the removal of the snapshot merges the changes with the normal file system. This caused a rapid increase in size because it posts the accumulation of the log file entries during the snapshot period over a matter of seconds. The line would have been gradual like it is for the remainder of the day had a snapshot not been taken.

What to do. Turning off logging for now by adding:
CustomLog /dev/null combined
ErrorLog /dev/null
at the bottom of the httpd-includes.conf. That doesn't work.
I also read that if you kill the virtual host logging, it writes it to the main file. That doesn't help.

- I do need my AWStats from time to time, mostly to chase down abusers.
- There is gap of 10 minutes between when stats run and the log gets truncated, meaning it loses something less that 10 minutes of stats each time. If I rotate the logs when they get to a certain size, I lose a variable amount of data in the stats. If I do it say every hour, the loss wouldn't be entirely consistent, but would more accurately reflect traffic levels.
- Running stats exacts its own penalty on the server, and now it would be happening during peak times as well as slower periods making the time the stats run, variable, but it meets the requirements better than anything else I can think of.
- I wonder if there is a way that both of the processes could be combined to run serially so the log would be truncated immediately after the stats run completes? The most important thing at this point is to stabilize the site, and the easiest way to do that is have it run every hour. Even during busy times, it would take at least 3 hours to build 2 gigs.
- I'll have to try to come up with a method to monitor how the restarts affect users while delivering up to 66 dynamic pages a second.

The more I write, the less I like the sounds of this. I don't want to restart Apache at all, and I don't want it handling large access logs. I believe we're concluding that a logging server is the way to go. That would also allow consolidation of statistics for the domain across servers as well. If you've done anything like this before, I'd be interested in knowing your experiences, and what you ultimately ended up doing that worked best.

PS: Here's a little gold mine that should help: Apache: Log Files