Please HELP, Disk Usage Almost Full

[ivanm]

Hello to everyone, i hope you can help me to solve this issue. I'm using Hetzner VPS and i have just upgraded to 160GB space, it was 80GB.
Before upgrade it was showing something like 72 out of 75 GB used. Now it's showing 137 out of 149 GB used. I know this is not normal and believe has to do something with this https://docs.directadmin.com/operat...html#disk-usage-is-showing-0-00-or-is-too-low

I have tried this "How to enable EXT4 quota feature", but none of the commands worked.

Appreciate your help!

 

[Stije]

First install ncdu and check if you really use this much of space.

You have root access and your disk is EXT4?

 

[ivanm]

Yes, i have root access and yes EXT4. I have 3 files under /var/log: mail.err, mail.log and syslog. Each one is 37GB large, that's insane

 

[Richard G]

Each one is 37GB large, that's insane

Insane indeed. You sure you are running Directadmin on that vps?
If yes, check your mail.err because we never had such file and it most likely will point out what is causing the problem.

I have 3 files under /var/log

I hope you mean 3 big files, because with Directadmin running there should be a bunch more files.

 

[ivanm]

Insane indeed. You sure you are running Directadmin on that vps?
If yes, check your mail.err because we never had such file and it most likely will point out what is causing the problem.


I hope you mean 3 big files, because with Directadmin running there should be a bunch more files.

Yes i meant 3 big files, that's correct. I'm sure that i'm running directadmin on my vps server
I will check mail.err and let you know. Thank you!

 

[Richard G]

Oke also check file date of the maillog file and first entry, because it should use logrotate every 7 days (default).

 

[Stije]

So you rub Debian/Ubuntu or something.

Just do this via ssh:

echo "" >  /var/log/syslog
echo "" >  /var/log/mail.err
echo "" >  /var/log/mail.log

This will empty the logs. Because they are to big to read. Then follow the logs to see what is filling it up:

#follow syslog
tail -f /var/log/syslog

#follow mail.err
tail -f  /var/log/mail.err

follow mail.log
tail -f  /var/log/mail.log

You should see what is filling up the logs and you can stop it.

 

[ivanm]

So you rub Debian/Ubuntu or something.

Just do this via ssh:

echo "" >  /var/log/syslog
echo "" >  /var/log/mail.err
echo "" >  /var/log/mail.log

This will empty the logs. Because they are to big to read. Then follow the logs to see what is filling it up:

#follow syslog
tail -f /var/log/syslog

#follow mail.err
tail -f  /var/log/mail.err

follow mail.log
tail -f  /var/log/mail.log

You should see what is filling up the logs and you can stop it.

Thanks for this man! It's Ubuntu btw, After running these commands to empty logs, now it's showing the available space correctly. And after running #follow commands, on syslog is just showing many firewall TCP_IN Blocked lines...no idea what they are. On the other two log files, not showing anything yet.

 

[Richard G]

You checked the mail.err log before clearing? Because it's of little use to clean the logs without knowing what is causing it, because then they soon will be filled again.
Same for the logrotate check I pointed out.

 

[ivanm]

Okay so I have attached all 3 log files from directadmin log viewer, you can check and let me know what should I do. Again within one day they are now 19 gb each. Why does my logrotation doesn't work for example getting rid of these daily, if it's enabled by default?
Thanks guys!

 

[Richard G]

Well... if you masked the mail.err.txt then I don't know what is going wrong exactly but it seems the be a permission problem.
In the mail.log I seen an attack, but I don't spend all time to go through all log files, that's your job.

Seems there is more wrong on your system if it has a permission issue on Dovecot and also logrotate is not working for your logfiles.

You could start with this as root via SSH:

cd /usr/local/directadmin/scripts
./set_permissions.sh all

This should fix permissions issues. Clear the mail.err and see if it will fill again. If yes, then you might want to send in a support ticket. Or if this is not possible, wait for somebody else to help figuring this out or hire somebody to fix things for you.

As for logrotate, you can see if this is running if other files are rotating correctly, like the syslog.

 

[ivanm]

Seems like the issue is gone now. No more large file sizes in those 3 files. Thanks, everyone!