Disk usage on partition /home exceeding 90% threshold

R

RADRaze2KX

Verified User
Joined
Jan 4, 2024
Messages
35
Hey there,
I recently got done banning some IPs that were flooding a client's website's "Contact us" form... some ~85K messages were sent from around a dozen IPs.

During this time, I started receiving notifications that state:
"This is an automated message notifying you that the following partition(s) have exceeded the 90% threshold:

97% - /home"

I've already run:
du -h -x / | sort -h -r | head -50
27G /
18G /usr
14G /usr/share
13G /usr/share/cagefs-skeleton
5.1G /var
4.1G /usr/share/cagefs-skeleton/opt
4.0G /opt
3.6G /usr/share/cagefs-skeleton/opt/alt
3.6G /opt/alt
3.4G /var/lib
3.2G /usr/share/cagefs-skeleton/var
3.0G /usr/share/cagefs-skeleton/usr
2.9G /var/lib/mysql
2.9G /usr/share/cagefs-skeleton/var/lib/mysql
2.9G /usr/share/cagefs-skeleton/var/lib
1.7G /usr/local
1.3G /usr/share/cagefs-skeleton/usr/lib
1.3G /usr/share/cagefs-skeleton/lib
1.3G /usr/local/directadmin
1.3G /usr/lib
915M /usr/share/cagefs-skeleton/usr/lib/firmware
915M /usr/share/cagefs-skeleton/lib/firmware
915M /usr/lib/firmware
835M /usr/local/directadmin/custombuild
735M /var/log
725M /usr/lib64
711M /usr/share/cagefs-skeleton/usr/lib64
711M /usr/share/cagefs-skeleton/lib64
461M /usr/local/directadmin/custombuild/mysql_backups
404M /usr/share/cagefs-skeleton/usr/local
404M /usr/bin
368M /usr/share/cagefs-skeleton/usr/share
358M /var/lib/rpm
348M /usr/local/directadmin/custombuild/cache
338M /var/lib/mysql/pcxprt_wp533
338M /usr/share/cagefs-skeleton/var/lib/mysql/pcxprt_wp533
324M /usr/local/directadmin/plugins
323M /var/lib/mysql/venueatt_wp108
323M /usr/share/cagefs-skeleton/var/lib/mysql/venueatt_wp108
322M /usr/share/alt-mod-passenger
297M /usr/share/alt-mod-passenger/build
235M /usr/share/cagefs-skeleton/opt/cloudlinux/venv
235M /usr/share/cagefs-skeleton/opt/cloudlinux
234M /usr/share/cagefs-skeleton/opt/cloudlinux/venv/lib/python3.11/site-packages
234M /usr/share/cagefs-skeleton/opt/cloudlinux/venv/lib/python3.11
234M /usr/share/cagefs-skeleton/opt/cloudlinux/venv/lib
233M /usr/share/alt-mod-passenger/build/buildout
233M /opt/cloudlinux/venv/lib/python3.11/site-packages
233M /opt/cloudlinux/venv/lib/python3.11
233M /opt/cloudlinux/venv/lib

and
cd /
du -h -x --max-depth=1 ./

However, neither of these seem to revealing why there's a large chunk of 384GB being used.
36M ./etc
5.0M ./root
5.1G ./var
18G ./usr
4.0G ./opt
0 ./media
0 ./mnt
0 ./srv
19M ./tmp
44K ./man
25M ./.gem
12K ./share
0 ./dir
27G ./

When I run "du -h -x /home | sort -h -r | head -50" I do see some media-heavy sites hitting ~6-8GB each but that's like 3 of 42. Is there a way I can see overall disk usage through DA in visual form, like qdirstat would provide?
 
Bash: 
[root@server home]# df -h -T
Filesystem                     Type      Size  Used Avail Use% Mounted on
devtmpfs                       devtmpfs  7.7G     0  7.7G   0% /dev
tmpfs                          tmpfs     7.7G  3.6M  7.7G   1% /dev/shm
tmpfs                          tmpfs     7.7G  193M  7.6G   3% /run
tmpfs                          tmpfs     7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/mapper/cloudlinux_45-root xfs        70G   16G   55G  23% /
none                           tmpfs     7.7G  6.5M  7.7G   1% /var/lve/dbgovernor-shm
/dev/vda1                      xfs      1014M  371M  644M  37% /boot
/dev/mapper/cloudlinux_45-home xfs        54G   52G  1.9G  97% /home
tmpfs                          tmpfs     1.6G     0  1.6G   0% /run/user/0

Bash: 
[root@server home]# du h --max-depth=1 /
du: cannot access 'h': No such file or directory
338852  /boot
3592    /dev
52782880        /home
du: cannot access '/proc/2273/task/2478417/fd/637': No such file or directory
du: cannot access '/proc/2488329': No such file or directory
du: cannot access '/proc/2491625/task/2491625/fd/4': No such file or directory
du: cannot access '/proc/2491625/task/2491625/fdinfo/4': No such file or directory
du: cannot access '/proc/2491625/fd/4': No such file or directory
du: cannot access '/proc/2491625/fdinfo/4': No such file or directory
0       /proc
197288  /run
0       /sys
34208   /etc
5032    /root
5301388 /var
10421648        /usr
0       /media
0       /mnt
0       /srv
24736   /tmp
44      /man
25036   /.gem
12      /share
0       /dir
69134724        /

Bash: 
[root@server home]# lsblk -a
NAME                   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                     11:0    1 1024M  0 rom
vda                    252:0    0  384G  0 disk
├─vda1                 252:1    0    1G  0 part /boot
└─vda2                 252:2    0  127G  0 part
  ├─cloudlinux_45-root 253:0    0   70G  0 lvm  /
  ├─cloudlinux_45-swap 253:1    0    4G  0 lvm  [SWAP]
  └─cloudlinux_45-home 253:2    0 53.1G  0 lvm  /home
 
Last edited by a moderator:
Ah I see... for some reason the drive is partitioned into 2 segments and the one all the data is on is only 127G... I wonder what happened. I'm not very good with Linux, is there a way I can extend or re-arrange the partitions without needing to reinstall?
 
I remember what happened. I started the VPS with a smaller drive. It was only 128GB, and I extended it. Vultr took the server offline briefly and when it came back up, it was a larger drive but I didn't bother to check HOW the additional space was allocated.

*groan*

Can someone with more Linux experience verify these commands for me, assuming I have ~40GB of space to allocate (meaning I'll need to resize about 30GB):
resize2fs /dev/cloudlinux_45/root 30G

lvreduce -L 30G /dev/cloudlinux_45/root

lvextend -L +40G /dev/cloudlinux_45/home

resize2fs /dev/cloudlinux_45/home
 
First of all, your server has an unlocated disk space, something like 256GB - you might add this space to your LVM.
Secondly, XFS can NOT be reduced live - you will need to boot a rescue mode for this kind of an operation.

These operations can potentially bring to a data loss, so I won't give instructions on how to complete it by your own. You can find guides in Internet, if you want it.

If you don't want to risk, you can hire me for this.
 
Thank you for the info about XFS. I'll take a snapshot on Vultr and test a restore first then take it offline and resize the partitions using gparted from a live disc.
 
I got the partition resized to take up the remainder of the disk:
# lsblk -a
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 384G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 383G 0 part
├─cloudlinux_45-root 253:0 0 70G 0 lvm /
├─cloudlinux_45-swap 253:1 0 4G 0 lvm [SWAP]
└─cloudlinux_45-home 253:2 0 53.1G 0 lvm /home

I verified the server boots successfully and is accessible from backend and some client sites. I then verified the available space in the volume group, extended the volume, and verified the size. I rebooted, and all is well. Thank you so much!
 
You must log in or register to reply here.
Back
Top