Compress mail manually (dovecot_compress.sh)

xerox

Verified User
Joined
Jul 16, 2019
Messages
137
Hello,

CustomBuild 2 have option to compress new emails, but i would like to know how to manually compress mail which is already existing in inbox and sent items.

I did not see any explanation how to do it manually.

Screenshot:
cmail.jpg

Ideas ?
Thanks a lot.
 

system-admin

Verified User
Joined
Aug 1, 2019
Messages
59
Hello Xerox,

You can run following script to compress existing mails/boxes:

/usr/local/directadmin/scripts/dovecot_compress.sh
 

xerox

Verified User
Joined
Jul 16, 2019
Messages
137
Hey! Thanks for the info, i totally missed that file in that dir.
 

Dannik

Verified User
Joined
Jan 7, 2009
Messages
70
Location
Netherlands
Hi all,

Am I correct the script only compresses files in the ./cur directory and not all other directories?
I use many subfolders for my mail and they ar stored like ./INBOX.folder1 /.INBOX.folder2 just next to ./cur.

I checked out the script and I expected all directories should be processed (and compressed), but ./cur is the only folder which is compressed after running the script.

I run the script using:
Bash:
/usr/local/directadmin/scripts/dovecot_compress.sh /home/user/imap/domain.com/email/Maildir
The result is
Bash:
Checking for directories in /home/user/imap/domain.com/email/Maildir...
Size before compression: 16M
Compressed /home/user/imap/domain.com/email/Maildir/./cur...
Size after compression: 6.1M

But when I do
Bash:
du -h Maildir/
The result is something like the following:
Bash:
4.0K    Maildir/.INBOX.IsGeenSpam/new
4.0K    Maildir/.INBOX.IsGeenSpam/tmp
4.0K    Maildir/.INBOX.IsGeenSpam/cur
20K     Maildir/.INBOX.IsGeenSpam
12K     Maildir/cur/tmp
6.1M    Maildir/cur
4.0K    Maildir/.Drafts/new
4.0K    Maildir/.Drafts/tmp
4.0K    Maildir/.Drafts/cur
20K     Maildir/.Drafts
4.0K    Maildir/.INBOX.Thermen/new
4.0K    Maildir/.INBOX.Thermen/tmp
672K    Maildir/.INBOX.Thermen/cur
692K    Maildir/.INBOX.Thermen
4.0K    Maildir/.INBOX.Sent/new
4.0K    Maildir/.INBOX.Sent/tmp
556K    Maildir/.INBOX.Sent/cur
576K    Maildir/.INBOX.Sent
4.0K    Maildir/.Archives.2010/new
4.0K    Maildir/.Archives.2010/tmp
12M     Maildir/.Archives.2010/cur
12M     Maildir/.Archives.2010
4.0K    Maildir/.Archives.2013/new
4.0K    Maildir/.Archives.2013/tmp
38M     Maildir/.Archives.2013/cur
38M     Maildir/.Archives.2013
4.0K    Maildir/.INBOX.Marktplaats/new
4.0K    Maildir/.INBOX.Marktplaats/tmp
13M     Maildir/.INBOX.Marktplaats/cur
13M     Maildir/.INBOX.Marktplaats
4.0K    Maildir/.INBOX.Facebook/new
4.0K    Maildir/.INBOX.Facebook/tmp
322M    Maildir/.INBOX.Facebook/cur
324M    Maildir/.INBOX.Facebook
4.0K    Maildir/.INBOX.Trash/new
4.0K    Maildir/.INBOX.Trash/tmp
4.0K    Maildir/.INBOX.Trash/cur
24K     Maildir/.INBOX.Trash

etc.
So there are many more folders next to ./cur, but they're not being compressed. Am I expecting the wrong result or should I compress every folder separately?

Regards,
Danny
 

lordlex

Verified User
Joined
Aug 17, 2008
Messages
50
Location
Romania
Am I correct the script only compresses files in the ./cur directory and not all other directories?
That's right...
So there are many more folders next to ./cur, but they're not being compressed. Am I expecting the wrong result or should I compress every folder separately?
You can't without modify the script... it checks if you provide a path ending with /Maildir, and all folders inside your maildir folder doesn't contain that but just cur folder.

Also, the script doesn't delete the tmp folder created inside cur folder.

Regards,
Dan
 

wavoe

Verified User
Joined
Nov 21, 2007
Messages
75
Location
Belgium
@Dannik , @lordlex

I also started to use this script a few weeks ago and had some info from here
But indeed experiencing the same like you.

I'm busy to find out if I can make it better.
The first loop I want to change already to something like:
Code:
find . -regextype egrep -regex '.*/(cur|new)' -print0 | while read -d $'\0' directory; do {
Like this I want to find all the cur and new folders.

If you change the script already on that line, I'm experiencing for the moment issues when the folder name has a space in it.
And something else I think I saw is that all emails are set to read state.
Once I make some progress I will share it.
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
10,282
Location
LT, EU
@Dannik , @lordlex

I also started to use this script a few weeks ago and had some info from here
But indeed experiencing the same like you.

I'm busy to find out if I can make it better.
The first loop I want to change already to something like:
Code:
find . -regextype egrep -regex '.*/(cur|new)' -print0 | while read -d $'\0' directory; do {
Like this I want to find all the cur and new folders.

If you change the script already on that line, I'm experiencing for the moment issues when the folder name has a space in it.
And something else I think I saw is that all emails are set to read state.
Once I make some progress I will share it.
Why not to just use ./dovecot_compress.sh all ? :)
 

wavoe

Verified User
Joined
Nov 21, 2007
Messages
75
Location
Belgium
Why not to just use ./dovecot_compress.sh all ? :)
Uhm I'm currently not looking at compressing all configured email addresses but I'm looking at having the email items *also in the (sub) folders*?
@smtalk if I'm correct this shell is only compressing the emails in the real inbox folder, and as an example not the Trash folder?
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
10,282
Location
LT, EU
Uhm I'm currently not looking at compressing all configured email addresses but I'm looking at having the email items *also in the (sub) folders*?
@smtalk if I'm correct this shell is only compressing the emails in the real inbox folder, and as an example not the Trash folder?
It's compressing all the mails in all the folders.
 

Dannik

Verified User
Joined
Jan 7, 2009
Messages
70
Location
Netherlands
Thanks @smtalk , I didn't know all was an option 🙃. The shell script doesn't mention it in the usage...
Bash:
#VERSION=0.0.2
# This script is written by Martynas Bendorius and DirectAdmin
# It is used to gzip all emails in Maildir directory
# Official DirectAdmin webpage: http://www.directadmin.com
# Usage:
# ./dovecot_compress.sh </home/user/imap/domains/domain.com/email/Maildir>

So do I cd into /home/user/imap/domain.com/email/Maildir
and then run:
/usr/local/directadmin/scripts/dovecot_compress.sh all
?

or just run
./dovecot_compress.sh /home/user/imap/domains/domain.com/email/Maildir all


I'll be awaiting the results of @wavoe :)
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
10,282
Location
LT, EU
/usr/local/directadmin/scripts/dovecot_compress.sh all

Code:
# ./dovecot_compress.sh 
Usage:
./dovecot_compress.sh /home/user/imap/domains/domain.com/email/Maildir
or
./dovecot_compress.sh all
you gave #0: ./dovecot_compress.sh

It's mentioned :)
 

Dannik

Verified User
Joined
Jan 7, 2009
Messages
70
Location
Netherlands
I believe you, but I also think there must be a newer script circulating then what's available on my server... :rolleyes:

Bash:
if [ $# -lt 1 ]; then
        echo "Usage:";
        echo "$0 /home/user/imap/domains/domain.com/email/Maildir";
        echo "you gave #$#: $0 $1";
        exit 0;
fi

Bash:
[[email protected] Maildir]# /usr/local/directadmin/scripts/dovecot_compress.sh
Usage:
/usr/local/directadmin/scripts/dovecot_compress.sh /home/user/imap/domains/domain.com/email/Maildir
you gave #0: /usr/local/directadmin/scripts/dovecot_compress.sh
 

wavoe

Verified User
Joined
Nov 21, 2007
Messages
75
Location
Belgium
Ah well spotted, I see I have suddenly version 0.0.3 now :LOL:

@Dannik I recently did the update of DA to 1.60
When I started testing I also had this 0.0.2 version.
 

Dannik

Verified User
Joined
Jan 7, 2009
Messages
70
Location
Netherlands
Ok. That must be the thing then.
I will wait a week or so before updating. Like now, I have seen before that after releasing an update, 1 or more patches are released quite quickly to resolve some bugs. It's normal, I know (I work for a software developing company and bugs are quite common), but I want to limit the risks my customers suffer from those bugs 🙃
Besides that, I won't have the time the next few days to test the compression anyway...
 

wavoe

Verified User
Joined
Nov 21, 2007
Messages
75
Location
Belgium
It's compressing all the mails in all the folders.
I'm not yet that much further yet but to give an update:
I created again a test account with some emails, mainly the biggest folders are inbox itself and the sent folder.
Using the dovecot_compress.sh (v0.0.3) is not compressing "all" the folders as I was expecting.

So on the left the Maildir folder before using the dovecot_compress.sh, on the right when it finished.
1580940038206.png
And just a detail: Size before 1.2GB (which is 721 MB + the tmp folder with compressed email of 471 MB) :cool:

Just sharing my experience now, if I have a free moment I would like to deep digger into the dovecot_compress.sh
 

Dannik

Verified User
Joined
Jan 7, 2009
Messages
70
Location
Netherlands
If I'm not mistaking you forgot the all option, like smtalk mentioned...? However I'm still not sure if this works the selected mailbox or al mailboxes on the server. Haven't had any time to figure that out myself 🙃
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
10,282
Location
LT, EU
If I'm not mistaking you forgot the all option, like smtalk mentioned...? However I'm still not sure if this works the selected mailbox or al mailboxes on the server. Haven't had any time to figure that out myself 🙃
@wavoe I think so too :) It wasn't "all" option, was it? If not, you just compressed the mail folder you specified, and it worked as intended.
 

wavoe

Verified User
Joined
Nov 21, 2007
Messages
75
Location
Belgium
@wavoe I think so too :) It wasn't "all" option, was it? If not, you just compressed the mail folder you specified, and it worked as intended.
A part of the dovecot_compress.sh:
Bash:
if [ "${1}" = "all" ]; then
        cat /etc/virtual/*/passwd | cut -d: -f6 | sort | uniq | while read line; do {
                doCompressMaildir "${line}/Maildir"
        }
        done
else
        doCompressMaildir "${1}"
fi

For me this means that using "all" option will compress all email accounts that exists on the server.
And it's just like I'm doing manually, placing the location of the Maildir folder.
Meaning that I come back on the issue what I mentioned earlier.

And just an example of the output of cat /etc/virtual/*/passwd | cut -d: -f6 | sort | uniq
Code:
...
/home/testemail1/imap/testemail1.com/info
/home/testemail2/imap/testemail2.com/info
/home/testemail3/imap/testemail3.com/info
/home/testemail4/imap/testemail4.com/info
/home/testemail5/imap/testemail5.com/info
...
This is not what I want (yet), I just want to be sure that every email folder is getting compressed.
Hope someone can confirm what I describe here as I have mixed feelings as it's not the first time that I try to explain this.
 

smtalk

Administrator
Staff member
Joined
Aug 22, 2006
Messages
10,282
Location
LT, EU
A part of the dovecot_compress.sh:
Bash:
if [ "${1}" = "all" ]; then
        cat /etc/virtual/*/passwd | cut -d: -f6 | sort | uniq | while read line; do {
                doCompressMaildir "${line}/Maildir"
        }
        done
else
        doCompressMaildir "${1}"
fi

For me this means that using "all" option will compress all email accounts that exists on the server.
And it's just like I'm doing manually, placing the location of the Maildir folder.
Meaning that I come back on the issue what I mentioned earlier.

And just an example of the output of cat /etc/virtual/*/passwd | cut -d: -f6 | sort | uniq
Code:
...
/home/testemail1/imap/testemail1.com/info
/home/testemail2/imap/testemail2.com/info
/home/testemail3/imap/testemail3.com/info
/home/testemail4/imap/testemail4.com/info
/home/testemail5/imap/testemail5.com/info
...
This is not what I want (yet), I just want to be sure that every email folder is getting compressed.
Hope someone can confirm what I describe here as I have mixed feelings as it's not the first time that I try to explain this.

Sorry, you're right :) Double-checked the code and found the bug. Please check if the following 0.0.4 version does the trick (it should):
Code:
wget -O /usr/local/directadmin/scripts/dovecot_compress.sh https://custombuild.eu/dovecot_compress.sh

We could include it into the next release of DA then.
 
Top