/var full

CrazyMouse

Verified User
Joined
Jul 24, 2004
Messages
53
Location
The Netherlands
My /var is everytime full. It is 250mb

I must delete logs to empty my var.. how do I make my /var bigger for all my logs?? Or is there something else I must do?
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
First:

You can delete all the files in /var/log that end with a number or serious of numbers. You should not delete any other files in /var/log, if you can help it.

If you do delete the files in /var/log that are NOT followed by numbers they will continue to grow and take up space anyway, until you stop and restart the syslogd daemon.

Of course if you delete any log files you'll lose the information in them; if you feel you may need them later you may decide to move them to a storage directory on another filesystem, but if you do, it'll become your responsibility to maintain them.

Now let's discuss the problem: Yours is not the first server to have been created with a /var partition of only 250 MB.

In order to help other people avoid this problem, can you please tell us:

1) What OS and version you're running ?

2) Did your server come with a 250 MB partition, or did you create it yourself?

3) If a data center did the install it would help to know who they are so we can educate them.

Now that that's out of the way I'll start by saying that it's a serious problem, and that there's no easy way out of it...

Your DA server not only saves it's logs in /var, but also all incoming and outgoing email; on a busy server that can become many gigabytes very quickly, depending on whether or not your clients save any pop3-downloaded email on the server.

For exmaple, i just checked one of our servers; the /var usage is at 2.3 gigabytes.

You can move subdirectories to other file systems, but it's easy to make mistakes, and some subsystems require you stop and start certain services and/or daemons while doing so. And then you'll have a non-standard system that's no longer taking advantage of the reasons you've got multiple filesystems in the first place.

There are programs available that will enable you to resize partitions on the fly, but I don't recommend them for several reasons. Among them are:

1) They're dangerous to use; if they fail for any reason you'll have an unusable and unrecoverable system.

2) Unless the partition you want to enlarge is right below the partition you want to reduce, it can become very hard to figure out the steps you need to take to get the job done right, and you greatly increase the chances you'll end up with an unusable and unrecoverable system.

You can add another drive and do some moving of data and remounting, so you can put your /var partition on another drive. This is the method we use for our clients when at all possible.

You can save your system, rebuild it from scratch, and restore it. This causes a lot of downtime and if you're not careful you can lose some client data and end up spending a lot of time restoring the rest.

Of course if you're renting the system you can just rent another, partitioned properly (there have been a few threads on the subject) and start moving over domains at your leisure. Then stop renting the original server when there's no more customer data on it.

While this is the most expensive solution, I recommend it highly as the best solution for a rented server.

Jeff
 

jmstacey

Verified User
Joined
Feb 12, 2004
Messages
4,107
Location
Colorado
Another option is to setup logrotate to rotate the logs at extremely close intervals. This may help for the short term but the size problem will still plague you eventually when you server gets more busy. Depending on your Operating System it may be possible to readjust the partition sizes without having to reinstall it.
 

CrazyMouse

Verified User
Joined
Jul 24, 2004
Messages
53
Location
The Netherlands
wauw, thanks for the big help!

I understand. But I already have some big sites on my server. And my /var file will everytime going full. And I have new costemers waiting to join the server. But with this problem. I let them wait.

The install is a standart FreeBSD 4.9 I read on this forum that is was oke. So it make by it self the /var 250 mb


the options that jmstacey says, will not be a good option I think, becouse. If there will me more and more sites.. So in the end. I will have the same problem again.

The only think, without many downtime, will me to make the /var bigger. if it posseble. otherwise, I most re-install all the things. I hope not

so I will study, today the website from freeBSD, if it is possible. the make /var bigger.


1 thing, here in holland, If you rent a server, you will rent it at least for a hole year, (for the most data center). (its with many things here) so If, i wanne move all data to a new server, for backup. that won't be a option. Becouse, I will stuck on it for a year. And I don't need a server. for this time. to mutch cost.
 

l0rdphi1

Verified User
Joined
Jun 22, 2003
Messages
1,471
I ran a quick search of this thread for the term mysql and didn't turn up anything. So here's another option: Depending on how large your /var/lib/mysql directory is, you could symlink that over to /home/mysql.

On previous servers this fixed my /var filling up problems.

Edit: I see Jeff has already covered this. Oh well :)

Phi1.
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
CrazyMouse said:
The install is a standart FreeBSD 4.9 I read on this forum that is was oke. So it make by it self the /var 250 mb
Does a "standard" FreeBSD install recommend such a small partition? The recommendation on the DA site, while oversimplified in my opinion, to run virtually everything under the same partition would work better than partitions that are too small.
The only think, without many downtime, will me to make the /var bigger. if it posseble.
I don't know if it's possible with FreeBSD or not; the tools I've seen that allow you to resize partitions are for linux. But of course I don't spend a lot of time looking at FreeBSD, so perhaps I've missed something. I hope you can find something.
otherwise, I most re-install all the things. I hope not
Reinstalling is a good option, but of course you end up with a lot of downtime, and you may have to tweak sites for some time to get everything back the way you want it.
1 thing, here in holland, If you rent a server, you will rent it at least for a hole year, (for the most data center). (its with many things here) so If, i wanne move all data to a new server, for backup. that won't be a option. Becouse, I will stuck on it for a year.
How about renting a temporary server elsewhere while you rebuild your server in Holland?

Is your current server rented? Or one you built? If you rented a server that came with DA, then if they didn't follow DA's instructions for parititoning and the server is unusable, it's their fault and they should be willing to somehow deal with you so you can continue to rent from them. Otherwise the server is unusable, and at least here in the U.S. that's a good reason to break a contract.

I wish you good luck.
l0rdphi1 said:
Edit: I see Jeff has already covered this. Oh well :)
I try :) .

We've run into these problems on lots of systems over the years, so while I don't catch everything, I have a lot of experience to draw on, and I do try.

Jeff
 

CrazyMouse

Verified User
Joined
Jul 24, 2004
Messages
53
Location
The Netherlands


this it the standart thing from FreeBSD

I found it on

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/install-steps.html

and this it the instal note from DA
Step 1: Do you meet the system requirements?
- Clean OS install: Redhat 7.2, 7.3, 8.0, 9.0 or FreeBSD 4.8, 4.9. (FreeBSD 5.x currently in beta)
- At least one external IP address (NAT/LAN-based systems will NOT work)
- SSH installed, gcc installed (these are usually standard with 99% of systems)


so nothing that say, please make /var BIG :)
I will now search over the internet, if I can fix the problem. I will let you know.

And This server is from my self. And placed in a dataCenter in holland.
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Here's the install note I found here:

We suggest a simple partition structure of:

/boot 40 meg
swap 2 x memory
/ rest of drive
I agree that that's not clear enough; I hereby urge the DA staff to clarify partition sizes a bit.

FreeBSD can recommend small /var patition because FreeBSD ports system creates symlinks to /usr for just about everything.

However DA does NOT use the FreeBSD ports system.

Jeff

Jeff
 
Last edited:

jmstacey

Verified User
Joined
Feb 12, 2004
Messages
4,107
Location
Colorado

Wunk

Verified User
Joined
Sep 25, 2003
Messages
121
You could just boot in single user mode for a bit, and do something like the following:

Make a directory called var on a partition with plenty of space (IE: /home), mkdir /home/var

Now copy all the contents from the real var to /home/var

Unmount var and remove it from your /etc/fstab

Now: FIRST CHECK IF /VAR IS REALLY UNMOUNTED typing mount and looking in the /var dir if it's empty..

It is ?, woo, now remove /var and issue ln -s /home/var /var

You'll have /var symbolically linked to /home/var where PLENTY of space exists..


Mind that these are some nice guidelines, change where needed to suit your system and ALWAYS make backups :)

Addition:
You could mount the old 250MB var partition to /tmp for example if that wasn't done yet either and set the options to nosuid, noexec, rw to secure your server a little more..
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Wunk said:
You could just boot in single user mode for a bit, and do something like the following:

Make a directory called var on a partition with plenty of space (IE: /home), mkdir /home/var
[/qote]
Good solution, Wunk.

The reason I didn't recommend it is that I tried this once on a Plesk server. And completely broke outgoing email from qmail.

I don't believe it'll break anything on a DA system, but not having tried it, I didn't want to recommend it.

Has anyone tried this?

(And of course then you have a /var that could fill an entire /home partition with, for example, run-away logs.)

Jeff
 

CrazyMouse

Verified User
Joined
Jul 24, 2004
Messages
53
Location
The Netherlands
I make a symbolically link for my /log/httpd/domeins to /etc/home/logs/httpd/domeins as a test.

becouse its a live server. it will be active after the cron jop is runing. so atnight 0.30 than it will swits. and works fine. but, there is a other log that will be bigger and bigger. yesterday I had 130 mb free. now 119 mb.


Becouse I dont have the time now. I will try the thing from Wunk
in the weekend. Becouse I will have plenty of time to reinstall the server. Its 1 houre drive from my house to the server.
 

Wunk

Verified User
Joined
Sep 25, 2003
Messages
121
jlasman said:
Good solution, Wunk.

The reason I didn't recommend it is that I tried this once on a Plesk server. And completely broke outgoing email from qmail.

I don't believe it'll break anything on a DA system, but not having tried it, I didn't want to recommend it.

Has anyone tried this?

(And of course then you have a /var that could fill an entire /home partition with, for example, run-away logs.)

Jeff [/B]
Take into account though that plesk already breaks if you sneeze too hard in the direction of the server.. :D

The only part that would change somewhat, is that /var and /home would in this case share user quota's, so a filling mailbox will munch away web quota too (shared quota in this case instead of seperate quota's voor mail and web)
 
Last edited:

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Wunk said:
The only part that would change somewhat, is that /var and /home would in this case share user quota's, so a filling mailbox will munch away web quota too (shared quota in this case instead of seperate quota's voor mail and web)
Considering the way some clients store mail, that could be good :) .

Jeff
 
Top