Maildir showing wrong usage

xerox

Verified User
Joined
Jul 16, 2019
Messages
151
Hello,

I discovered that the /home/(user)/Maildir/new is showing 4gb of data but if I go inside the dir it shows like 9000 small 2-10kb files which equals approx 90 mb.
So, I am little confused whats going.

What it is and how to fix this issue?


Thanks.
 
Hi,

Post the commands that you used to calculate disk space usage both for the "/home/(user)/Maildir/new" and from the inside of that folder.
If you did everything correctly, my first guess is that there is a broken/corrupt file.
In that case, I would try using lsof to look for open files that are deleted, but the file descriptors are still not closed.
 
Here is the command:
Code:
[root@server Maildir]# du -a /home/(user)/Maildir/new/ | sort -n -r | head -n 20
4148048 /home/(user)/Maildir/new/
36      /home/(user)/Maildir/new/1560857983.M376823P27845.server.domain.com,S=33639,W=34515
16      /home/(user)/Maildir/new/1606338057.H754739P11414.server1.domain.com
16      /home/(user)/Maildir/new/1606335648.H46333P31063.server1.domain.com
16      /home/(user)/Maildir/new/1606335603.H579201P31006.server1.domain.com
16      /home/(user)/Maildir/new/1606335568.H935609P30860.server1.domain.com
16      /home/(user)/Maildir/new/1559742304.M123700P30861.server.domain.com,S=15109,W=15386
12      /home/(user)/Maildir/new/1610668321.H382840P30064.server1.domain.com
12      /home/(user)/Maildir/new/1610668261.H307125P29425.server1.domain.com
12      /home/(user)/Maildir/new/1610668201.H985243P29172.server1.domain.com
12      /home/(user)/Maildir/new/1610668141.H779567P29027.server1.domain.com
12      /home/(user)/Maildir/new/1610668081.H640023P28376.server1.domain.com
12      /home/(user)/Maildir/new/1610668021.H443199P28101.server1.domain.com
12      /home/(user)/Maildir/new/1610667961.H294241P28012.server1.domain.com
12      /home/(user)/Maildir/new/1610667902.H161690P27293.server1.domain.com
12      /home/(user)/Maildir/new/1610667841.H828164P27057.server1.domain.com
12      /home/(user)/Maildir/new/1610667781.H758231P26979.server1.domain.com
12      /home/(user)/Maildir/new/1610667721.H540375P26246.server1.domain.com
12      /home/(user)/Maildir/new/1610667661.H328245P26041.server1.domain.com
12      /home/(user)/Maildir/new/1610667602.H141426P25881.server1.domain.com
[root@server Maildir]#
 
Ok, "du" or "ls" will probably not show you a corrupt/broken file.
example:

~]# ll -ah /tmp/
total 8.0K
drwx-wx-wt 2 root root 160 Jan 22 12:47 .
dr-xr-xr-x 23 root root 4.0K Nov 18 18:21 ..
-rw-r--r-- 1 root root 0 Nov 18 18:58 firewall.lock
-rw-r--r-- 1 root root 0 Nov 18 18:27 networks.lock
-rw-r--r-- 1 root root 0 Nov 18 18:22 quota.lock
-rw-r--r-- 1 root root 468 Jan 22 12:47 .status
-rw-r--r-- 1 root root 0 Nov 18 18:28 tasks.lock
-rw-r--r-- 1 root root 0 Jan 17 04:44 .tmp.optimize_db

~]# du -a /tmp/
4 /tmp/.status
0 /tmp/.tmp.optimize_db
0 /tmp/firewall.lock
0 /tmp/tasks.lock
0 /tmp/networks.lock
0 /tmp/quota.lock
4 /tmp/

But lsof might help, try using:
lsof /home/(user)/Maildir/new/

You should see something like this ("deleted"), if you have possibly corrupt files.

~]# lsof /tmp/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 1298 elasticsearch mem REG 144,131 125693 (deleted)/tmp/x_data_elasticsearch/1298 (stat: No such file or directory)
java 1298 elasticsearch 0r REG 144,131 1 125553 (deleted)/tmp/sh-thd-1605711703
php74-cgi 27543 manager 3u REG 144,131 0 414419134 (deleted)/tmp/.ZendSem.G73z7R


Try to kill processes that are using those files and see if the files "disappear" when you check them with lsof again.
But if you see a lot of those files, maybe a server reboot will be a faster solution.

Hope this helps.
 
So, the both commands did not return any result.

lsof /home/(user)/Maildir/new/

and

lsof /tmp/
 
Interesting. Can you give the output for these commands, I want to see the sizes and inode counts for these folders:

du -sh /home/(user)/Maildir/
du -sh /home/(user)/Maildir/new/

and then

du --inodes -sh /home/(user)/Maildir/
du --inodes -sh /home/(user)/Maildir/new/
 
Ok, so here is output...
Code:
[root@server1 ~]# du -sh /home/user/Maildir/
4.0G    /home/user/Maildir/

[root@server1 ~]# du -sh /home/user/Maildir/new
4.0G    /home/user/Maildir/new

[root@server1 ~]# du --inodes -sh /home/user/Maildir/
751K    /home/user/Maildir/

[root@server1 ~]# du --inodes -sh /home/user/Maildir/new
751K    /home/user/Maildir/new

P.S I am using Centos 7. Also rebooted the server before i gave you the commands.
 
Last edited:
Thanks. I see you wrote that you found 9000 files (a.k.a. inodes) in that directory.

I discovered that the /home/(user)/Maildir/new is showing 4gb of data but if I go inside the dir it shows like 9000 small 2-10kb files which equals approx 90 mb.

But according to the "du --inodes -sh" command, which calculates the amount of files (a.k.a inodes) in the directory, there are about 751000 files.

[root@server1 ~]# du --inodes -sh /home/user/Maildir/new
751K /home/user/Maildir/new

I think you might have just miscalculated the amount of files you have in that directory. According to the output of the command, you have a lot more files there.
 
This could be probably issue in WinSCP when it calculated the wrong amount.
So, what to do with all those files, as some of them could be real emails.

Should i migrate my emails to new email account, then remove all the data from the account which has lot of files from new/ and then put those emails back to the right account or there is another solution?

I am still confused about the amount of data the folder has because there is 183 emails in this account when look with thunderbird.

Looks like in every 10-15 minutes there is new entry in that folder. So there must be something gone haywire.

 
Last edited:
Normally there should only be email files in user mailbox directories.
From the directory "/home/user/Maildir/new" I assume that this is a system mailbox for your USER.
Regular mailboxes should usually be in "/home/USER/imap/DOMAIN/USERNAME_of_mailbox/Maildir/new/".

Usually when the system mailbox is receiving a lot of messages, this could mean that a cron task is failing and sending messages to that mailbox.
Also, the systems mailbox may receive various other messages from directadmin services, for example someone is brute focring the server, or failed emails from other mailboxes are being delivered to the system mailbox.

I would recommend to open and read a few files in "/home/user/Maildir/new" using a text editor. You might find useful information inside the files.
 
OK, so here is what i found out.

Because the program which used the curl was removed then it caused errors and those are that error.

Now I am not sure, is there no other option than creating a new account and moving my emails in thunderbird to this account and then removing all the contents on /new/ directory and after that dropping the emails back to the right account?
Is there another way to it ?

Error contains:
Code:
Could not open input file: /home/user/domains/domain.com/public_html/demo/artisan
 
Ok so firstly, I would recommend to use a normal mailbox for regular mailing.
Use the system mailbox for troubleshooting purposes only.

As far as I see, you have 2 problems:

1. Something is generating a lot of emails to your system mailbox (most likely a cron job, if that is the only problem).
2. You need to clean all those useless email files, after you fix problem no. 1.


I would start by going into that USER DirectAdmin -> then open your "Cron jobs" and check if there are cron jobs that are trying to run something on the website, which is located inside folder "/home/user/domains/domain.com/public_html/demo/".

From experience I can say that many people remove websites, but often forget to remove cron jobs, or change something in their website and then the cron job sends tons of errors to the system mailbox.


You need to find and fix problem no. 1 first.
 
Sorry if i didnt make myself clear but I already did removed the cronjob and found out what the error was.

P.S. there was not program in that dir left.

I am now figuring out what is the best way to remove those cron errors without removing real emails.
 
Last edited:
That's ok. There are several ways to remove a lot of files at once.

If the emails in that folder are all not important, you can just delete the entire folder "rm /home/user/Maildir/new -Rf" and then recreate the folder again "mkdir /home/user/Maildir/new". Also set the correct permissions (check the ownership permissions and the the correct access permissions read, write, exexute). Check the permissions before deleting the folder, of course.

If you have a small amount of emails that are very important, I suggest just copying them to another mailbox manually. Via Thunderbird or command line.
Then do the above to delete the junk.


But if you have a lot of important emails and moving them manually would take too long, I would suggest this strategy:
Find a common pattern inside the emails that you need to delete, for example:
"Could not open input file: /home/user/domains/domain.com/public_html/demo/artisan" could be the pattern.

First test if the search by pattern works as you expect:
grep -Hir "Could not open input file: /home/user/domains/domain.com/public_html/demo/artisan" /home/user/Maildir/new/

It should return something like this:
/home/user/Maildir/new/very_long_file_name_123435,S:Could not open input file: /home/user/domains/domain.com/public_html/demo/artisan

You only need the full filename so add "cut -d: -f1", where -d with the ":" will cut the line at the delimiter ":" and "-f1" will return the first portion of the line, in other words the filename.
grep -Hir "Could not open input file: /home/user/domains/domain.com/public_html/demo/artisan" /home/user/Maildir/new/ | cut -d: -f1

You should see something like this (the filename may have symbols such as ","):
/home/user/Maildir/new/very_long_file_name_123435,S

Ultimately, to delete all those files add "-exec rm -f {} \;"
grep -Hir "Could not open input file: /home/user/domains/domain.com/public_html/demo/artisan" /home/user/Maildir/new/ | cut -d: -f1 -exec rm -f {} \;


Just please test all of the above commands in a test environment first (like in a VirtualBox VM). Just to be 100% safe.
Keep in mind that you might run into various errors, if the command can't handle so many files at once, or permissions errors. You might need to repeat the command or modify it to work with a smaller list of files.
 
I made backup and removed the contents of /new/.

Interesting but I still see all the emails listed in all the email addresses. So whats the /new/ folder is needed anyway :D
 
Back
Top