Slow memory leak... ideas?

I have a server running FreeBSD 5.4 and DirectAdmin. Everything is running at default, and the only web site on there right now is my own (very light usage at the moment). I have noticed that the memory seems to leak at a rate of around 100M a day, until I eventually have to reboot the server. Has anyone seen something like this before? Any ideas on how to fix it?

On mrtg I see mym mem usage rise slowly upto midnight then goes down again to a low level and repeats every day, I think is due to open log files getting bigger and consuming memory, however there is no mem leak needing a reboot.

Remember that most tools show how much memory is free, but not what is cached, so it could be that most of your mem is cached wich doesn't affect anything

It does when the swap mem usage crawls up with it