var is full, how do I fix this problem?

Q

questions

Verified User
Joined
Oct 24, 2009
Messages
131
Code: 
/var: write failed, filesystem is full

# df -h
Filesystem       Size    Used   Avail Capacity  Mounted on
/dev/twed0s1a    869G    219G    580G    27%    /
devfs            1.0K    1.0K      0B   100%    /dev
/dev/twed0s1d    1.9G    183M    1.6G    10%    /tmp
[B][COLOR="Red"]/dev/twed0s1e     24G     22G    -38M   100%    /var[/COLOR][/B]
procfs           4.0K    4.0K      0B   100%    /proc



# du -skh * | sort -n
1.0M	named
2.0K	account
2.0K	audit
2.0K	empty
2.0K	games
2.0K	heimdal
2.0K	logs
2.0K	mail
2.0K	preserve
2.0K	rwho
2.0K	tmp
4.0K	crash
4.0K	msgs
6.0K	du.txt
8.0K	at
 11M	db
 20M	spool
 22K	yp
 86K	run
114K	du
116K	backups
[B][COLOR="Red"]125M	log
209M	installatron[/COLOR][/B]
272K	lib
[B][COLOR="Red"]462M	www[/COLOR][/B]
620K	cron
# cd www
# du -skh * | sort -n
6.0K	cgi-bin
 13M	manual
 26K	htdocs
184K	error
282K	build
[B][COLOR="Red"]448M	html[/COLOR][/B]
542K	icons
# cd html
# du -skh * | sort -n
  0B	phpMyAdmin
  0B	roundcube
  0B	squirrelmail
2.0K	allard.php
2.0K	du.txt
2.0K	index.html
2.0K	redirect.php
4.4M	phpPgAdmin
5.1M	webmail
 12M	roundcubemail-0.5.2
 18M	phpMyAdmin-3.4.1-all-languages
 68K	daplugin
[B][COLOR="Red"]408M	squirrelmail-1.4.21[/COLOR][/B]
# cd squirrelmail-1.4.21
# du -skh * | sort -n
1.5M	help
2.0K	README
2.0K	configure
2.0K	index.php
2.0K	install
2.0K	version
4.0K	ReleaseNotes.locales
 10K	TRANSLATORS
 12K	ChangeLog.locales
 16K	COPYING.locales
 17M	locale
 38K	contrib
 62K	include
 78K	po
144K	themes
192K	class
228K	config
252K	images
[B][COLOR="Red"]385M	data[/COLOR][/B]
412K	doc
426K	src
778K	plugins
912K	functions
# cd log
# du -skh * | sort -n
  0B	dmesg.yesterday
  0B	pf.today
2.0K	cron
2.0K	debug.log
2.0K	ipfw.today
2.0K	ipfw.yesterday
2.0K	lpd-errs
2.0K	mount.today
2.0K	ppp.log
2.0K	security
2.0K	slip.log
2.0K	xferlog
4.0K	auth.log.2.bz2
6.0K	auth.log.1.bz2
6.0K	auth.log.3.bz2
6.0K	rkhunter.log.old
6.0K	setuid.today
8.0K	auth.log
 [B][COLOR="Red"]12M	exim[/COLOR][/B]
 14K	messages.1.bz2
 16K	messages.2.bz2
 16K	messages.3.bz2
 18K	messages.0.bz2
[B][COLOR="Red"] 23M	httpd[/COLOR][/B]
 24K	messages
[B][COLOR="Red"] 26M	proftpd[/COLOR][/B]
 28K	lastlog
 28K	messages.4.bz2
[B][COLOR="Red"] 28M	maillog[/COLOR][/B]
 30K	messages.5.bz2
 33M	directadmin
 34K	setuid.yesterday
 40K	wtmp
 50K	dmesg.today
 64K	maillog.0.bz2
 96K	auth.log.0.bz2
112K	mysql
118K	rkhunter.log
188K	userlog
368K	wtmp.0
416K	wtmp.1
528K	wtmp.2
928K	wtmp.3
 
No easy answer, but here are some suggesstions:

You can delete all logfiles ending in 1, 2, 3, and 4. You just end up without the log information if you need it.

But I see a greater issue, because your directory listing for /var doesn't seem to contain anywhere near 22 GB of data. Investigate.

Jeff
 
how do i investigate if the du reading doesn't show the usage?


nobaloney said:
No easy answer, but here are some suggesstions:

You can delete all logfiles ending in 1, 2, 3, and 4. You just end up without the log information if you need it.

But I see a greater issue, because your directory listing for /var doesn't seem to contain anywhere near 22 GB of data. Investigate.

Jeff
Click to expand...
 
# cat mount.today
/dev/twed0s1a / ufs rw 1 1
devfs /dev devfs rw 0 0
/dev/twed0s1d /tmp ufs rw 2 2
/dev/twed0s1e /var ufs rw 2 2
procfs /proc procfs rw 0 0
Click to expand...

found a thread about this and it says that perhaps there is a mounted remote filesystem open somewhere?

http://www.monkey.org/openbsd/archive/misc/0407/msg00107.html

I was experimenting with doing some remote backups on a fusefs filesystem mount (not involving var directory) at some time in the past...
 
Found another thread stating:

Unfortunately, if you aren't able to catch it when it has the high usage, it's difficult for us to state why it's alerting. It could be high session activity (those sessions do clear on their own) for some site during the times it hits the high disk space level. It could be MySQL having the temporary MyISAM tables filling it up. It could be EAccelerator writing files to it (if you use that, it writes some temporary files there). There are many reasons it could be happening. If you aren't certain, you can simply resize it to be larger. Otherwise, you'll have to watch it closely until you do catch it and then provide the details on what you are seeing.

There are also some scripts that will periodically clear tmp such as tmpwatch, but you need to be careful not to cron clear /tmp as EasyApache writes to /tmp during build, so you cannot simply clear files every 15 minutes or something, since you'll then break EasyApache from recompiling properly.

Thanks.
Click to expand...

http://forums.cpanel.net/f5/drive-critical-dev-loop0-var-tmp-97-full-190772.html
 
Found this with a possible solution but I'm not sure how to do it or if syslogd is the problem in my case... I am running a socket proxy/SSH tunnel for Firefox through this server, perhaps its trying to log this activity?

I figured out the issue. Based on your suggestion that it may be an open file, I searched for open files in /var (using lsof /var) and found that syslogd had several very large files open that I had removed, so I restarted that and now the partition shows the correct usage. Thank you for the suggestion.
Click to expand...

http://forums.devshed.com/linux-help-33/var-partition-full-but-not-really-633446.html
 
does anyone have any ideas why my var is full but not full at the same time?
 
I could be reading your post (#1) incorrectly. But I'm not experienced enough in FreeBSD to offer to log in for a fee to look for the problem. First I'd suggest you don't consider anything with the word partition in it unless a FreeBSD expert tells you it may be a viable direction to take; My recollection is that FreeBSD doesn't use partitions and I don't know if slices act the same way or not.

I've moved the thread to a FreeBSD section of the forum; hopefully it'll get more experienced eyes there.

Jeff
 
I could be reading your post (#1) incorrectly. But I'm not experienced enough in FreeBSD to offer to log in for a fee to look for the problem. First I'd suggest you don't consider anything with the word partition in it unless a FreeBSD expert tells you it may be a viable direction to take; My recollection is that FreeBSD doesn't use partitions and I don't know if slices act the same way or not.

I've moved the thread to a FreeBSD section of the forum; hopefully it'll get more experienced eyes there.

Jeff

<[email protected]> +1 951 643-5345
Third-Party DirectAdmin administration and support
Dedicated Servers, Dedicated Reseller Accounts
NoBaloney Internet Services
P.O. Box 52200
Riverside, Calif. 92517
Click to expand...

yes they are the same partitions outlined here:

http://www.directadmin.com/install.html

freebsd is almost identical to linux except for a few commands, tools, and principals.... this problem is happening to people across the *nix rainbow.
 
my research has lead me to be 95% sure that this is due to files that are open but in limbo, not written at all or not written properly, but i dont have the skill to track them down and close them... or find the source of the problem....
 
isn't it great how talking to yourself fixes the problem... i restarted mysql and now i'm down to just 4% usage of /var

# /usr/local/etc/rc.d/mysqld restart
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]
# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/twed0s1a 869G 217G 583G 27% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/twed0s1d 1.9G 130K 1.8G 0% /tmp
/dev/twed0s1e 24G 796M 21G 4% /var
procfs 4.0K 4.0K 0B 100% /proc
 
this is not just a freebsd problem so you can put it back into a category outside freebsd
 
Your MySQL logs rotation might need to get reviewed, as it seems your mysql logs do not get flushed on its rotation, check /etc/newsyslog.conf and post here results for

Code: 
grep -i mysql /etc/newsyslog.conf

you might need to add a path to mysql pid file in the line with mysql log(s), or some other actions might be required.
 
I had same issue. %98 full but none of file are such big. After a server reboot it is reduced to %4. What do you think was the cause that I should check?
 
You must log in or register to reply here.
Back
Top