Compress mail manually (dovecot_compress.sh)

smtalk · Jun 3, 2020

myH2Oservers said:
How about using/supporting that format?

Dovecot does not support it

Until 2.3.11. So, when it's released we could modify the script, but at this time it'd just make all emails unreadable by dovecot.

smtalk · Aug 16, 2020

Just a FYI that dovecot 2.3 is released (zstd is used by default with it when compressing using latest version of the script).

factor · Oct 22, 2020

How does one uncompress?
or use gzip instead of zstd?
Uninstall zstd?

smtalk · Oct 24, 2020

Emails? "find" command would need to be used for this, to find all the emails in Maildir directories and execute decompression command. Do you see any reason to do it?

factor · Oct 24, 2020

smtalk said:
Emails? "find" command would need to be used for this, to find all the emails in Maildir directories and execute decompression command. Do you see any reason to do it?

The script corrupted a bunch of email. So I am looking for away to fix them. Or undo the compression. It basically created a bunch of no subject emails. When you try to delete them in RC you can’t.

seems to be this issue

Compress mail manually (dovecot_compress.sh)

Sorry, you're right :) Double-checked the code and found the bug. Please check if the following 0.0.4 version does the trick (it should): wget -O /usr/local/directadmin/scripts/dovecot_compress.sh https://custombuild.eu/dovecot_compress.sh We could include it into the next release of DA then...

forum.directadmin.com

smtalk · Oct 24, 2020

bdacus01 said:
The script corrupted a bunch of email. So I am looking for away to fix them. Or undo the compression. It basically created a bunch of no subject emails. When you try to delete them in RC you can’t.

seems to be this issue

Compress mail manually (dovecot_compress.sh)

Sorry, you're right :) Double-checked the code and found the bug. Please check if the following 0.0.4 version does the trick (it should): wget -O /usr/local/directadmin/scripts/dovecot_compress.sh https://custombuild.eu/dovecot_compress.sh We could include it into the next release of DA then...

forum.directadmin.com

Is that the latest version of the script/DA? If yes, I'd suggest running the script again, you may run it for a single maildir if you wish, and see if it solves it

I've seen some clients who had older version and got zstd emails gzippped (because of no zstd support in that version of the script).

If no luck - I may connect to the server and check what's happening there.

factor · Oct 24, 2020

smtalk said:
Is that the latest version of the script/DA?

cat /usr/local/directadmin/scripts/dovecot_compress.sh
#!/bin/sh
#VERSION=0.0.4

smtalk said:
I've seen some clients who had older version and got zstd emails gzippped

I did run it once before zstd support
So the mail-in theory would have to been gziped then zstd compressed.

I will try to run for one mailbox

smtalk · Oct 24, 2020

It should be 0.0.5.

factor · Oct 24, 2020

How do we get latest? Is it in custombuild?

factor · Oct 24, 2020

I just did ./build update

no luck

smtalk · Oct 24, 2020

It comes together with latest release of DirectAdmin. What's the version you run?

factor · Oct 24, 2020

Ah I only update weekly. let me look

you mean ./build update_da?

cb versions
Latest version of DirectAdmin: 1.61.5
Installed version of DirectAdmin: 1.61.5

still
04

Is there a manual way of updating it?

I may have used the wget link thinking it was best...

factor · Oct 24, 2020

I found it in update tar

factor · Oct 24, 2020

ok I put in the script reran it. I didnt really help

It seems to create blank "No subject emails" OR they are corrupt older emails?

doveadm indexing shows

Code:

doveadm(tech@my domain.com): Error: Mailbox Sent: UID=2: read(zlib(/home/admin/imap/my domain.com/tech/Maildir/.Sent/cur/1603476378.M66804P736843.myserver.my domain.com,S=578,W=594:2,S)) failed: Cached message size larger than expected (578 > 378, box=Sent, UID=2) (read reason=mail stream)
doveadm(tech@my domain.com): Error: Mailbox Sent: Cache /home/admin/imap/my domain.com/tech/Maildir/.Sent/dovecot.index.cache: Deleting corrupted cache record uid=2: UID 2: Broken physical size in mailbox Sent: read(zlib(/home/admin/imap/my domain.com/tech/Maildir/.Sent/cur/1603476378.M66804P736843.myserver.my domain.com,S=578,W=594:2,S)) failed: Cached message size larger than expected (578 > 378, box=Sent, UID=2)
doveadm(tech@my domain.com): Error: Mailbox Sent: UID=2: read(zlib(/home/admin/imap/my domain.com/tech/Maildir/.Sent/cur/1603476378.M66804P736843.myserver.my domain.com,S=578,W=594:2,S)) failed: Cached message size larger than expected (578 > 378, box=Sent, UID=2) (read reason=)
2/2

var mail log

Code:

Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457445><SF3x1G2y0oVf2MRm>: Error: Mailbox Trash: Saving mail: save: read(zlib(/home/admin/imap/mydomain.com/tech/Maildir/cur/1603288859.M824688P3389109.myserver.mydomain.com,S=13637,W=13845:2,S)) failed: Cached message size larger than expected (13637 > 2864, box=INBOX, UID=16337)
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457445><SF3x1G2y0oVf2MRm>: Error: Mailbox Trash: copy: i_stream_read(zlib(/home/admin/imap/mydomain.com/tech/Maildir/cur/1603288859.M824688P3389109.myserver.mydomain.com,S=13637,W=13845:2,S)) failed: Cached message size larger than expected (13637 > 2864, box=INBOX, UID=16337)
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457445><SF3x1G2y0oVf2MRm>: Logged out in=164 out=1200 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
Oct 24 12:14:37 myserver dovecot[457240]: imap-login: Login: user=<[email protected]>, method=PLAIN, rip=my ip, lip=my ip, mpid=457447, TLS, session=<ocr01G2y1oVf2MRm>
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457447><ocr01G2y1oVf2MRm>: Error: Mailbox INBOX: UID=16337: read(zlib(/home/admin/imap/mydomain.com/tech/Maildir/cur/1603288859.M824688P3389109.myserver.mydomain.com,S=13637,W=13845:2,S)) failed: Cached message size larger than expected (13637 > 2864, box=INBOX, UID=16337) (read reason=mail stream)
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457447><ocr01G2y1oVf2MRm>: Error: Mailbox INBOX: Cache /home/admin/imap/mydomain.com/tech/Maildir/dovecot.index.cache: Deleting corrupted cache record uid=16337: UID 16337: Broken physical size in mailbox INBOX: read(zlib(/home/admin/imap/mydomain.com/tech/Maildir/cur/1603288859.M824688P3389109.myserver.mydomain.com,S=13637,W=13845:2,S)) failed: Cached message size larger than expected (13637 > 2864, box=INBOX, UID=16337)
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457447><ocr01G2y1oVf2MRm>: Error: Mailbox INBOX: UID=16337: read(zlib(/home/admin/imap/mydomain.com/tech/Maildir/cur/1603288859.M824688P3389109.myserver.mydomain.com,S=13637,W=13845:2,S)) failed: Cached message size larger than expected (13637 > 2864, box=INBOX, UID=16337)
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457447><ocr01G2y1oVf2MRm>: FETCH failed: Internal error occurred. Refer to server log for more information. [2020-10-24 12:14:37] in=287 out=1588 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0

smtalk · Oct 24, 2020

Feel free to send me access and I’ll check what’s going on (free of charge, of course).

factor · Oct 24, 2020

sent in ticket and pm

factor · Oct 24, 2020

All:
I wanted to come back and report. @smtalk worked on the issues I had today with the script. He discovered a few new bugs due to the script zstd compressing the emails over and over. Sometimes 3 to 5 times. He was able to improve the script so it can decompress email now as well. I had several hundred compressed, compressed and more compressed. Needless to say, he was able to work it all out. All mails are all back and fully operational.

He will be adding the updated 0.0.6. in the next release of DA.

Active8 · Oct 25, 2020

bdacus01 said:
He will be adding the updated 0.0.6. in the next release of DA

Don't have to wait for DA update , just do this :

Code:

wget -O /usr/local/directadmin/scripts/dovecot_compress.sh https://custombuild.eu/dovecot_compress.sh

factor · Oct 25, 2020

Active8 said:
Don't have to wait for DA update , just do this :

Code:

wget -O /usr/local/directadmin/scripts/dovecot_compress.sh https://custombuild.eu/dovecot_compress.sh

This is true as well for now. It's not always updated with the latest version.. Which is how I got in the mess I was in.

Tafaz · Oct 29, 2020

@smtalk today i was trying to compress an email account i got since 2008 and i noticed that some directories were skipped because the way your script search email files to compress don't consider files like this 1212677781.P1511Q2.mail.hostname.com:2,S because it doesn't have the size part in file name like S=12341 so i changed the find criteria to match it in this way

Original:

find . -type f -name "*,S=*" ! -name "*,*:2,*,*Z*"

New:

find . -type f -regextype posix-extended -regex "\./[0-9]+\..*:2,S" ! -name "*,*:2,*,*Z*"

Hope this could be useful

Compress mail manually (dovecot_compress.sh)

Administrator

Administrator

Verified User

Administrator

Verified User

Administrator

Verified User

Administrator

Verified User

Verified User

Administrator

Verified User

Verified User

Verified User

Administrator

Verified User

Verified User

Verified User

Verified User

Verified User