Exim 4.95

[mxroute]

Good conversation. I happened across this line length limit today by customer report. Even Roundcube is capable of breaking this 998 limit right now. I'm looking back at the only announcement I can find for it here: https://lists.exim.org/lurker/message/20210928.211934.67f92669.en.html

I'm proposing that enforcing a 998 character limit was not a hill that exim meant to die on. Rather, they merely intended to add the message_linelength_limit transport variable and, reasonably, defaulted it to the RFC defined maximum. With so many applications and hosts not enforcing this, it seems to me that increasing this is not only sane, but I'm going to go as far as to suggest that it's what the exim project expected people to do. Unless and until this moves to something highly enforced and something highly adopted by developers, leaving it at the undefined default does not seem like a sane choice.

 

[mxroute]

I've been informed via ticket that they do not want to define the variable and they do want to leave it at the default. It seems that they are under the impression that this 998 limit is something rolling out industry wide, but it isn't. I was simply pointed to the feedback form. I'll wipe my hands clean of it, this doesn't impact me due to heavy customization, but I do suspect this will increasingly result in user complaints for hosts here as they update to 4.95. So if you suspect that might be you, or if it already is, you may want to toss a vote on here: https://feedback.directadmin.com/b/feature-requests/increase-message-linelength-limit-default

 

[Ohm J]

@mxroute
should make option with directadmin.conf or edit with custombuild 2.0 better than increase default value for newer install

 

[sparek]

I'm proposing that enforcing a 998 character limit was not a hill that exim meant to die on. Rather, they merely intended to add the message_linelength_limit transport variable and, reasonably, defaulted it to the RFC defined maximum.

I think you're right.

should make option with directadmin.conf or edit with custombuild 2.0 better than increase default value for newer install

Probably something like @kam821 proposed in the post 64 where include files are present in the transports. Or at the very least add an explicit:

message_linelength_limit = 998

in the respective transports so that a post exim_conf hook can be written to change this to a desired value, i.e:

sed -i "s/^message_linelength_limit = 998$/message_linelength_limit = 4096/g" /etc/exim.conf


I still believe the RFC should be updated to define a higher value or more mail services need to be aware of this RFC defined value. But it's obvious that there are a slew of other mailers that do not operate within this confine, not just Microsoft. And when Exim 4.95 becomes more readily adopted there's going to be a slew of missed mails.

 

[Richard G]

I'm proposing that enforcing a 998 character limit was not a hill that exim meant to die on.

So why did they now changed their mind suddenly?

Anyway, I don't agree that exim.conf should be adjusted for that. RFC's are to be obeyed. Is the used software which needs to be adjusted or the users who are complaining.

So that means that I will vote for your suggestion, but will comment that it should be a switchable option and not default active, because then you would break RFC by default, which should never be the case.
If it's possible via a custom file, that would be very nice for those who want to use this.

 

[mxroute]

So why did they now changed their mind suddenly?

Anyway, I don't agree that exim.conf should be adjusted for that. RFC's are to be obeyed. Is the used software which needs to be adjusted or the users who are complaining.

So that means that I will vote for your suggestion, but will comment that it should be a switchable option and not default active, because then you would break RFC by default, which should never be the case.
If it's possible via a custom file, that would be very nice for those who want to use this.

Why would they need to change their mind? Who said that the exim project intended for this value to not be set by users and that they intended to enforce this default as an absolute value across the internet? I didn't see them say that. They only added a variable and, naturally, it has a default. That isn't a statement that they think you shouldn't override it, otherwise it would have been a hard coded global with no override.

There's an RFC for carrier pigeons. It may feel noble to fall on one's sword to enforce an RFC from 2001, but when no one else is that's a hill you're going to die on. Hotmail isn't. Gmail isn't. Outlook isn't. Roundcube isn't. At this point, who is complying with that old RFC other than people who strangely decided that they wouldn't define the variable in their transport on an installation of exim 4.95? No one else will be rejecting email for this, customers will take notice and react appropriately. Variables are to be configured when they don't match common needs, not respected for their defaults.

The decision to enforce this and not allow a reasonable override solution is a mistake and miscalculation by the DA team.

 

[sparek]

It's a give and take from both sides.

In order to communicate you have to rely on defined standards. Now, if you and I want to define our own method of communicating, that's fine. But if someone else wants to join the conversation... unless they know what method we are using to communicate, then they won't be able to communicate with us.

At some point a governing body has to be present that administers communication methods. Otherwise you wind up with a hodge-podge of different communication methods. Bob and Tom use one method of communicating. Aaron and Seth use another. Bob and Aaron use another one. Tom and Seth use another one. And pretty soon, everybody is communicating with different people with a different communication method.

That's why I do believe following RFCs is the proper way to approach thing.

However, it's obvious that nobody is adhering to this particular RFC regarding email line lengths. You can debate the reasons why (Didn't know it existed? Didn't think it applied any longer? Didn't care?). Using the logwrite warn code provided in post 62 by @kam821 I've collected 1812 entries of emails being sent to our servers since yesterday that had lines exceeding 998 characters. (For those that may be wondering, Average length was 3740 characters, Max was 2515948 characters). That means that had these servers been upgraded to Exim 4.95 and had the default 998 character limit applied, they would have been rejected. Conversely though... 819 of these 1812 (45%) were flagged as spam or viruses, and probably a lot of the remaining were spam the end user just didn't have SpamAssassin enabled for whatever reason. So enforcing this could be a decent spam deterrent.

The give and take part is that the IETF and those responsible for RFCs need to be aware of this. They need to either realize that they did a very poor job of advertising RFC 5322 as being relevant. Or they need to realize that the limits imposed in RFC 5322 are no longer relevant. If there is a reason for the 998 character limit, then explain it so that mail server providers and email client developers can implement this.

Obviously mail servers and email clients have been operating outside of the 998 character limit for a long, long time now, as it just reared it's head when Exim started enforcing this default value. And as far as I know, nothing in email broke with this character limit being longer than 998 characters.

I would prefer for RFC 5322 to be updated to set a larger max value - maybe something in the 4000 character range? (183 of the 1812 messages I caught were over 4000 characters). Otherwise, it's going to be a hodge podge of different max character lengths. Some may have 2000 characters limits, some may have 800000 characters limits. To be followed by "well, my message worked when I was with hosting company XYZ".

 

[Richard G]

The decision to enforce this and not allow a reasonable override solution is a mistake and miscalculation by the DA team.

It might have existed a lot longer, maybe they editted it out of exim.conf but I don't know. I did see a message going back to 2015 about somebody on CP complaining that they got this "message to long" problem when using spamexperts as their external mail provider.

I don't agree with you about others, what others do is their issue. RFC's are standards. If we all just ignore standards we don't like, we would be back to chaos possibility's as in the early years.
As Sparek says, it's going to be a hodge podge of different max character lengths.
As I see it, RFC's are a kind of laws. You also just don't break laws because you don't like them. You might run a red light if you're sure nobody's on the road and nobody sees you. Not even if the police only starts giving tickets for it after lots of years.
But it shouldn't be a default behavior, that is what I ment.

However, it's obvious that nobody is adhering to this particular RFC regarding email line lengths.

I like to have some doubts about that, having been Googling about it since my last post. Spamexperts, 2015, Mailgun 2019, I've seen messages on Stackoverflow from 2020. There was no Exim 4.95 that time.
So I'm wondering if it's indeed something Exim did not used before but others did, or if it's something DA took care of. I don't know.

Anyway.....

I would prefer for RFC 5322 to be updated to set a larger max value

that's the way it should really be done. As with other RFC's (not only mail) which got a bit too old and need updating.

 

[Active8]

I am really confused, apparently this is not new in Exim
Saw older posts on internet posted few year ago with the same "problem"

Exim limiting number of emails and cc

550 maximum allowed line length is 998 octets, got 1014 Getting this error in emails bouncing back. Is there a setting in exim to solve this problem? Tried to google it up but cannot find any sol...

serverfault.com

Exim4 line length in Debian Stretch – “Mail delivery failed: returning message to sender” | boltblog

Or did I missed something? if not why is this suddenly a problem ?

 

[sparek]

The message_linelength_limit directive was added in Exim 4.95 as per:

https://raw.githubusercontent.com/Exim/exim/master/doc/doc-txt/NewStuff

After reading the verbage on that - I'm in agreement with @mxroute that the intent was to just introduce the message_linelength_limit directive and it has to have a default value so they went with the RFC 5322 max value of 998.

Prior to Exim 4.95 - none of this was enforced in Exim.

Other MTAs may have handled this differently. I think someone earlier in this thread said Postfix does line folding if a line exceeds 998 characters. And I suppose that's an option, but Exim's probably not going to incorporate this themselves since they tend to have a strict "do not touch" policy when it comes to a message's body. And I agree with them, the issue should probably be handled by MUAs if lines are presented as longer than 998 characters.

Other MTAs may have been rejecting messages that exceed this 998 character limit, I really don't know. Exim and Postfix are probably (?) the two most popular MTAs. So I'm not sure what others might be doing.

Obviously messages have been funneling through Exim with longer than 998 characters for quite some time, and - as far as I know - this has never created any problems. That's why I'm tending to think that the 998 character limit is just an arbitrary limit the IETF set for RFC 5322 and it would serve them well to increase this. Otherwise, MUAs need to be made aware of this 998 character limit - something they probably weren't aware of because Exim had not been enforcing it and Postfix was doing line folding on it's own - and they need to update their code to cut lines that are 998 characters.

I do see discussions prior to Exim 4.95 (some dating back to 2004) that seem to indicate Exim was rejecting messages back then. Although reading through those discussions it's really difficult to ascertain what exactly is going on. It's not really clear to me that Exim is the one doing the rejecting (people may think it's Exim that's rejecting the message). Perhaps the Debian package of Exim was some how enforcing this by default?

Having a look at:

#828801 - Legitimate mail can contain long lines - Debian Bug report logs

bugs.debian.org

(Note: dated 2017-06-27)

seems to indicate that the Debian package of exim4-config 4.87-3 added an ACL to reject messages that exceed 998 characters. So if you're seeing mentions of this problem in prior than Exim 4.95 versions, I would suspect that it's Debian (or maybe another distro based Exim package) Exim package that is enforcing this within the confines of it's configuration ACL.

Exim 4.95 added this enforcement directly into the transports and defaulted to the RFC 5322 max length.

At least that's my understanding.

 

[mxroute]

I don't agree with you about others, what others do is their issue. RFC's are standards. If we all just ignore standards we don't like, we would be back to chaos possibility's as in the early years.

That won't get you very far when your client says "I can send this email from Gmail to Hotmail, fix it or I leave."

The number of RFCs we all don't follow are through the roof. Spend some time in them, you'll realize we're all violating an RFC or a hundred somewhere, and if you don't violate them you'll end up running your own section of the internet that doesn't work with anyone else's use case. Heck, the number of RFCs probably already violated by all of our exim.conf files are surely up there as well. If you want to die on that hill, go read every RFC and adjust all of your systems to comply with every single one of them. I promise you, it will be a disaster, you will have no customers left. The rest of the internet will not rise to the occasion to save you.

RFC is not a hill to die on unless you have the influence to be the leader in the market. It doesn't do you any good to try to lead and then have no customers, a leader with no followers is just a strange guy standing on the street corner shouting things. That's an output of zero. If you want to be the one to set the standards, you need to have enough influence that customers will actually tell the giant corporations to follow your lead. Gmail doesn't even have a person customers can reach out to in order to make such a demand, but you do, so guess who will get it and which one will lose a customer. The answer isn't Gmail. Because who are you to suddenly make such arbitrary demands, for no clearly stated purpose other than "It was written in a paper well over a decade ago" against a giant? I promise, they will not care how passionately you respect RFCs, they are very busy handling their day to day concerns that actually impact their customers at large.

The purpose of DirectAdmin is to make it easier for less technical people to use, and web hosts to provide services, for the purpose of functioning on the broader internet. If DA fails to do that, it will be replaced on the market. People will not give up Outlook, Roundcube (which even ships with DA), Gmail, Hotmail, and all of these other things while lifting DirectAdmin up as the new authority for standards across the internet. It will sooner lose it's users than accomplish such a mission. And again, with no users, the influence by attempting to lead when no one follows is an output of zero.

I'm pretty passionate about this, it's a mistake and it's based on a faulty idea. Exim had reasons for every default value, but that doesn't stop us from having a huge config file full of adjustments and reconfigurations of those default variables, because that's what every one of our exim.conf files are. If the defaults were sane, we wouldn't need one.

Anyway, sorry if I sound harsh. I love you, but I'm steamrolling this one because I believe what I'm saying, and I don't appreciate the added overhead (I can only apply it to my external transport without editing exim.conf, local emails don't go through that transport) with nothing to gain when it could be demolished in one little update, without harming anyone. Make no mistake, not setting this to a different value from the default will only create problems, doing so will remove them and not create new ones.

 

[sparek]

I'm pretty passionate about this, it's a mistake and it's based on a faulty idea. Exim had reasons for every default value, but that doesn't stop us from having a huge config file full of adjustments and reconfigurations of those default variables, because that's what every one of our exim.conf files are. If the defaults were sane, we wouldn't need one.

Still, it would be nice to have a nice RFC link to point back to when a customer complains that their message that has lines that are 80,000 characters in length doesn't work on our server but works when they were with Web Hosting Company A. "Why did it work with Web Hosting Company A?" ... 'Well son, they used an insanely large number for their message_linelength_limit' ... "Why can't you use an insanely large number for your message_linelength_limit?" ... 'Because we think 4000 is a reasonable limit' ... "Have anything to back that up?" ... 'Well here's a link to RFC 5322 that says it should be 988 characters, now... we're breaking that rule, just not as much as Web Hosting Company A' ... "Does it really matter how much you break a rule if you break the rule?"

But... you are right, we all tend to break RFCs that we see fit to break. It's just that most of us break the same RFCs with mostly the same values. So we just kind of sweep those RFCs under the rug ... "nothing to see here!"

 

[mxroute]

Still, it would be nice to have a nice RFC link to point back to when a customer complains that their message that has lines that are 80,000 characters in length doesn't work on our server but works when they were with Web Hosting Company A. "Why did it work with Web Hosting Company A?" ... 'Well son, they used an insanely large number for their message_linelength_limit' ... "Why can't you use an insanely large number for your message_linelength_limit?" ... 'Because we think 4000 is a reasonable limit' ... "Have anything to back that up?" ... 'Well here's a link to RFC 5322 that says it should be 988 characters, now... we're breaking that rule, just not as much as Web Hosting Company A' ... "Does it really matter how much you break a rule if you break the rule?"

Aye, for that @kam821's solution is looking promising for a long term solution.

 

[Richard G]

That won't get you very far when your client says "I can send this email from Gmail to Hotmail, fix it or I leave.

And that is the whole problem. Every little thing is adjusted everywhere because a customer complaints about something. If you teach the customers it's rules and everybody would do that, the customer could do what he/she wants but would run into the same issue everywhere.
But due some people who always want to give the customer what he/she wants, rules are bent, software is adjusted, RFC's are violated. And then if such customer comes at another hoster, who obeyes the RFC's and rules and want to keep the mail on internet as should be, they get a bunch of xxx from the customer.
Luckily I don't have to live from my servers, and I have customer friendlyness and customer service very high. But certain things... well... if they don't like it, goodbye, I don't care.
But that is globally spoken. But imho it happens too often that I see hosters (even here) want to make odd changes because of some customer with odd wishes imho.

Having this said. Exactly because Exim did not apply this rule for many years, which also caused many applications to be developped without knowledge of this rule, is the reason that I would opt in for an option to be able to customize it. I've not been against it.

Make no mistake, not setting this to a different value from the default will only create problems, doing so will remove them and not create new ones.

I'm also passionate about this, because I very much disaggree with you that RFC's are not a hill to die on. RFC's are created to finally finish the rubbish out there and make things better. To me, that is law. And if everyone would obey them, we had a lot less issues.

Still, I don't see any difference if you can set this to a different value from the default via a custom file, because in fact it's a customization.

However, I changed my mind about having it as default or not.
In this specific case it might indeed be better to change it by default to unlimited.

Reason for this change of mind is that I did some more investigation and these 2 combinations made me change my mind:
a.) Exim has waited too long, so lots of software do not obey the RFC limit anymore for years which could indeed cause troubles by default. Normally it's not that bad, but we've seen not only Outlook but also CSF and other applications having problems with this.
b.) Maybe even morge important. The issue @sparek was referring to (everybody has it's own limits) is already happening. I had a look and found that Postfix uses a 2048 limit and MS Exchange even uses an 8000 limit. Which removes any reason for any lower limit or limit at all.

This would indeed cause heavy issues by default. So to be compatible with everybody, the total limit in Exim should be gone again by default c.q. set to what it was in the previous version. Maybe a limit at 8000 to prevent again issues with the big MS.

Still.... this should not be a DA or only Exim discussion. This should be a MTA discusion on RFC level. And then probably MS would again force the 8000 to everyone. I couldn't find the line length limit that Gmail is using.

The purpose of DirectAdmin is to make it easier for less technical people to use

For customers and partly for admins. Admins should be admins, not click and play wannebe's. But that is a totally different discussion, which we best don't discuss in this thread too...

 

[mxroute]

Still, I don't see any difference if you can set this to a different value from the default via a custom file, because in fact it's a customization.

That would be acceptable. As it stands right now, you can't. Not unless you custom code your system to repeatedly redo it on updates. Which is fine, but not for something that's required to function as is expected and as has been expected for years where nothing else external changed to compensate.

 

[Richard G]

That would be acceptable.

I was suggesting that from the beginning. I was never for keeping things as they were, I was only suggesting to do it by customisation file.

Not unless you custom code your system to repeatedly redo it on updates.

Files like exim.variables.conf.custom redo changes on updates, so that would not be problem I guess.

But due to more insight, I rather agree now that the limit should be removed by default, at least to be able to be compatible with other MTA's like Postfix and Exchange which also ignored the RFC.

Edit: fixed some typo's.

 

[Richard G]

Anyway, sorry if I sound harsh. I love you, but I'm steamrolling this one

Just to be sure. There is no need to be sorry. You didn't sound harsh to me (although sometimes it can be good to be a bit harsh or blunt in a discussion). You defended your arguments with passion, without getting rude or personal, sometimes I also like to discuss with passion about some things. There is nothing wrong with that. I love and respect you too, not a tiny bit less then before.
So please don't change they way you discuss and defend your arguments with passion if needed. I like it!

 

[tomputer]

The 998 character limit is due to limitations in many implementations
that send, receive, or store IMF messages which simply cannot handle
more than 998 characters on a line.

I'm wondering if these limitations are really relevant today? Which kind of application/device today is unable to handle more than 998 characters.

I think the real problem is not Exim but the legacy rfc5322 rule (from 2008), which needs to be updated or removed.

 

[Richard G]

I think the real problem is not Exim but the legacy rfc5322 rule (from 2008), which needs to be updated or removed.

That is what we are saying too, especially cince other MTA's also ignore this RFC. Better is to update this RFC.

However, we need a solution for this sudden appeared issue now, it would take too long to wait for an RFC update (if ever) in this case.

 

[ikkeben]

It doesn't make sense at all to have standards ore even RULES LAWS whatever if not all related and stakeholders are compliant, also the control and real live practice.

If such conflicts as here it shouldn't be on our backs / shoulders getting into problems with it ?