Compress mail manually (dovecot_compress.sh)

Just a FYI that dovecot 2.3 is released (zstd is used by default with it when compressing using latest version of the script).
 
How does one uncompress?
or use gzip instead of zstd?
Uninstall zstd?
 
Emails? "find" command would need to be used for this, to find all the emails in Maildir directories and execute decompression command. Do you see any reason to do it? :)
 
smtalk said:
Emails? "find" command would need to be used for this, to find all the emails in Maildir directories and execute decompression command. Do you see any reason to do it? :)
Click to expand...
The script corrupted a bunch of email. So I am looking for away to fix them. Or undo the compression. It basically created a bunch of no subject emails. When you try to delete them in RC you can’t.

seems to be this issue

Compress mail manually (dovecot_compress.sh)

Sorry, you're right :) Double-checked the code and found the bug. Please check if the following 0.0.4 version does the trick (it should): wget -O /usr/local/directadmin/scripts/dovecot_compress.sh https://custombuild.eu/dovecot_compress.sh We could include it into the next release of DA then...
forum.directadmin.com forum.directadmin.com
 
bdacus01 said:
The script corrupted a bunch of email. So I am looking for away to fix them. Or undo the compression. It basically created a bunch of no subject emails. When you try to delete them in RC you can’t.

seems to be this issue

Compress mail manually (dovecot_compress.sh)

Sorry, you're right :) Double-checked the code and found the bug. Please check if the following 0.0.4 version does the trick (it should): wget -O /usr/local/directadmin/scripts/dovecot_compress.sh https://custombuild.eu/dovecot_compress.sh We could include it into the next release of DA then...
forum.directadmin.com forum.directadmin.com
Click to expand...
Is that the latest version of the script/DA? If yes, I'd suggest running the script again, you may run it for a single maildir if you wish, and see if it solves it :) I've seen some clients who had older version and got zstd emails gzippped (because of no zstd support in that version of the script).

If no luck - I may connect to the server and check what's happening there.
 
smtalk said:
Is that the latest version of the script/DA?
Click to expand...
cat /usr/local/directadmin/scripts/dovecot_compress.sh
#!/bin/sh
#VERSION=0.0.4

smtalk said:
I've seen some clients who had older version and got zstd emails gzippped
Click to expand...
I did run it once before zstd support
So the mail-in theory would have to been gziped then zstd compressed.

I will try to run for one mailbox
 
It comes together with latest release of DirectAdmin. What's the version you run?
 
Ah I only update weekly. let me look

you mean ./build update_da?

cb versions
Latest version of DirectAdmin: 1.61.5
Installed version of DirectAdmin: 1.61.5

still
04

Is there a manual way of updating it?

I may have used the wget link thinking it was best...
 
ok I put in the script reran it. I didnt really help

It seems to create blank "No subject emails" OR they are corrupt older emails?
1603559993146.png

doveadm indexing shows
Code: 
doveadm(tech@my domain.com): Error: Mailbox Sent: UID=2: read(zlib(/home/admin/imap/my domain.com/tech/Maildir/.Sent/cur/1603476378.M66804P736843.myserver.my domain.com,S=578,W=594:2,S)) failed: Cached message size larger than expected (578 > 378, box=Sent, UID=2) (read reason=mail stream)
doveadm(tech@my domain.com): Error: Mailbox Sent: Cache /home/admin/imap/my domain.com/tech/Maildir/.Sent/dovecot.index.cache: Deleting corrupted cache record uid=2: UID 2: Broken physical size in mailbox Sent: read(zlib(/home/admin/imap/my domain.com/tech/Maildir/.Sent/cur/1603476378.M66804P736843.myserver.my domain.com,S=578,W=594:2,S)) failed: Cached message size larger than expected (578 > 378, box=Sent, UID=2)
doveadm(tech@my domain.com): Error: Mailbox Sent: UID=2: read(zlib(/home/admin/imap/my domain.com/tech/Maildir/.Sent/cur/1603476378.M66804P736843.myserver.my domain.com,S=578,W=594:2,S)) failed: Cached message size larger than expected (578 > 378, box=Sent, UID=2) (read reason=)
2/2

var mail log
Code: 
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457445><SF3x1G2y0oVf2MRm>: Error: Mailbox Trash: Saving mail: save: read(zlib(/home/admin/imap/mydomain.com/tech/Maildir/cur/1603288859.M824688P3389109.myserver.mydomain.com,S=13637,W=13845:2,S)) failed: Cached message size larger than expected (13637 > 2864, box=INBOX, UID=16337)
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457445><SF3x1G2y0oVf2MRm>: Error: Mailbox Trash: copy: i_stream_read(zlib(/home/admin/imap/mydomain.com/tech/Maildir/cur/1603288859.M824688P3389109.myserver.mydomain.com,S=13637,W=13845:2,S)) failed: Cached message size larger than expected (13637 > 2864, box=INBOX, UID=16337)
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457445><SF3x1G2y0oVf2MRm>: Logged out in=164 out=1200 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
Oct 24 12:14:37 myserver dovecot[457240]: imap-login: Login: user=<[email protected]>, method=PLAIN, rip=my ip, lip=my ip, mpid=457447, TLS, session=<ocr01G2y1oVf2MRm>
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457447><ocr01G2y1oVf2MRm>: Error: Mailbox INBOX: UID=16337: read(zlib(/home/admin/imap/mydomain.com/tech/Maildir/cur/1603288859.M824688P3389109.myserver.mydomain.com,S=13637,W=13845:2,S)) failed: Cached message size larger than expected (13637 > 2864, box=INBOX, UID=16337) (read reason=mail stream)
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457447><ocr01G2y1oVf2MRm>: Error: Mailbox INBOX: Cache /home/admin/imap/mydomain.com/tech/Maildir/dovecot.index.cache: Deleting corrupted cache record uid=16337: UID 16337: Broken physical size in mailbox INBOX: read(zlib(/home/admin/imap/mydomain.com/tech/Maildir/cur/1603288859.M824688P3389109.myserver.mydomain.com,S=13637,W=13845:2,S)) failed: Cached message size larger than expected (13637 > 2864, box=INBOX, UID=16337)
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457447><ocr01G2y1oVf2MRm>: Error: Mailbox INBOX: UID=16337: read(zlib(/home/admin/imap/mydomain.com/tech/Maildir/cur/1603288859.M824688P3389109.myserver.mydomain.com,S=13637,W=13845:2,S)) failed: Cached message size larger than expected (13637 > 2864, box=INBOX, UID=16337)
Oct 24 12:14:37 myserver dovecot[457240]: imap([email protected])<457447><ocr01G2y1oVf2MRm>: FETCH failed: Internal error occurred. Refer to server log for more information. [2020-10-24 12:14:37] in=287 out=1588 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0
 
Feel free to send me access and I’ll check what’s going on (free of charge, of course).
 
All:
I wanted to come back and report. @smtalk worked on the issues I had today with the script. He discovered a few new bugs due to the script zstd compressing the emails over and over. Sometimes 3 to 5 times. He was able to improve the script so it can decompress email now as well. I had several hundred compressed, compressed and more compressed. Needless to say, he was able to work it all out. All mails are all back and fully operational.

He will be adding the updated 0.0.6. in the next release of DA.
 
bdacus01 said:
He will be adding the updated 0.0.6. in the next release of DA
Click to expand...
Don't have to wait for DA update , just do this :
Code: 
wget -O /usr/local/directadmin/scripts/dovecot_compress.sh https://custombuild.eu/dovecot_compress.sh
 
Active8 said:
Don't have to wait for DA update , just do this :
Code: 
wget -O /usr/local/directadmin/scripts/dovecot_compress.sh https://custombuild.eu/dovecot_compress.sh
Click to expand...
This is true as well for now. It's not always updated with the latest version.. Which is how I got in the mess I was in.
 
@smtalk today i was trying to compress an email account i got since 2008 and i noticed that some directories were skipped because the way your script search email files to compress don't consider files like this 1212677781.P1511Q2.mail.hostname.com:2,S because it doesn't have the size part in file name like S=12341 so i changed the find criteria to match it in this way



Original:

find . -type f -name "*,S=*" ! -name "*,*:2,*,*Z*"



New:

find . -type f -regextype posix-extended -regex "\./[0-9]+\..*:2,S" ! -name "*,*:2,*,*Z*"

Hope this could be useful
 
You must log in or register to reply here.
Back
Top