Exim 4.95

[smtalk]

We do not refuse to add any easier ways to change the default and we're open for suggestions Would additional include line in exim.conf to change the parameter be enough?

 

[cjd]

@smtalk I think that would be completely acceptable for myself. Let those that need to 'fix' it for their non compliant customers till they get things sorted out, or for external issues out of your control.

 

[sparek]

We do not refuse to add any easier ways to change the default and we're open for suggestions Would additional include line in exim.conf to change the parameter be enough?

I think the easiest solution for DirectAdmin to deploy in regards to this, would be to explicitly set the message_linelength_limit option in the remote_smtp and remote_smtp_forward_transport transports.

At least, from what I gather, that is the only two transports that this would be affecting? I'm not completely sure if that is right.

Since this option is not explicitly set, it's going to the default value 998, correct?

So just add a:

message_linelength_limit = 998

some where in each of the affected transports.


Then a simplier post exim_conf script can be written to regex replace message_linelength_limit = 998 with whatever value each of us desires.

sed -i "s/message_linelength_limit = 998/message_linelength_limit = 4096" /etc/exim.conf

An include file in the affected transports might also be a solution. But just explicitly listing the option with the default value would seem to be the easiest.

 

[sparek]

For anybody that might be interested - there's now a discussion about this on the Exim mailing list. Subject is message has lines too long for transport

 

[cjd]

I see on their dev list there are problems also with exim exceeding it's own line length limit and causing a problem with some bounce messages, that is a problem. I also saw the above mentioned message threads, there are a couple conversations. Looks like some work still needs to be done on exim's end to also fix some unforeseen problems due to this change.

 

[kam821]

I updated exim_line_folding.
1) Both compile and runtime maximum line length versions are combined into one, generic source code with boolean switch (Config::CompileTimeLength), if you set false, you can specify maximum line length at runtime by parameter.
2) Both runtime implementations - charconv and strtoll are also combined into the same source code, charconv is used if header is available, otherwise - strtoll is used.
3) Folding line sequence is now placed after whitespace (space, tab, etc.) position if this is possible to avoid breaking line in the middle of e.g. Message-ID.
4)
- Both the version with compile time and runtime line length can be used without parameters, in this case, the version with the runtime line length will use the default value (Config::MaxLineLength), exactly the same as compile-time version does.
Usage: /path/to/exim_line_folding
- For version with runtime line length, you can set the maximum line length as the first parameter, but this is optional:
Usage: /path/to/exim_line_folding 998
5) I put a toggle (Config::UseBuiltinIsSpace) to decide, if standard library std::isspace function or self-written equivalent routine is used to determine, if character is whitespace.
I don't know why, but the std::isspace slows down entire execution significantly, it's possible that the compiler doesn't inline std::isspace but instead compiles code into a function call.
6) Performance is nearly as good as it was before.
7) Replacing existing whitespace instead of inserting folding line sequence after whitespace in the future may be a good idea.

Compilation:
1) Save source code to e.g. exim_line_folding.cpp file
2) Change CompileTimeLength to false if you want to compile runtime line length version.
3) Run: g++ -std=c++20 -Wall -Wextra -Werror -pedantic -O3 -Wno-attributes -o exim_line_folding exim_line_folding.cpp
(if you can't compile with c++20, the code should compile with -std=c++17 as well)
4) Move exim_line_folding binary to destination folder you want, it may be /usr/bin, /usr/local/bin, /opt, etc.
5) Make sure, that binary has executable flag set (chmod).

Integration with Exim:

Exim 4.95

I rather comply to RFC's if this is an RFC thing. I've also updated to 4.95 a couple of weeks ago and so far 1 only had 1 line with those "too long" statements on 1 of 3 servers. It's better to teach customers than to violate RFC rules imho.

forum.directadmin.com

#include <iostream>
#include <optional>
#include <string>
#include <string_view>

#ifdef __has_include
#define has_include(x) __has_include(x)
#else
#define has_include(x) 0
#endif

#if has_include(<charconv>)
#include <charconv>
#else
#include <cstdlib>
#endif

using namespace std::string_view_literals;

struct Config
{
    static inline constexpr auto CompileTimeLength = true;
    static inline constexpr auto FoldLineSequence  = "\n\t"sv;

    static inline constexpr std::size_t MaxLineLength  = 998;
    static inline constexpr std::size_t LineBufferSize = 4096;

    static inline constexpr auto UseBuiltinIsSpace = false;

    static_assert(Config::MaxLineLength >= 2, "Config::MaxLineLength must be at least 2");
};

enum ReturnCode : int
{
    Success = 0,
    Error
};

#if has_include(<charconv>)
static std::optional<std::size_t> stringToSize(std::string_view str) noexcept
{
    std::size_t value{};

    const auto result = std::from_chars(str.data(), str.data() + str.size(), value);

    if (result.ec == std::errc{})
        return value;

    return std::nullopt;
}
#else
static std::optional<std::size_t> stringToSize(std::string_view str) noexcept
{
    char* end;
    const auto value = std::strtoll(str.data(), &end, 10);

    if (str.data() != end && value >= 0)
        return static_cast<std::size_t>(value);

    return std::nullopt;
}
#endif

static constexpr bool isCharWhitespace(char c) noexcept
{
    if constexpr (Config::UseBuiltinIsSpace) {
        return std::isspace(static_cast<unsigned char>(c));
    } else {
        switch (c) {
            case ' ':
            case '\t':
            case '\r':
            case '\f':
            case '\v':
            case '\n':
                return true;
            default:
                return false;
        }
    }
}

static void foldLines(std::istream& inStream, std::ostream& outStream, std::size_t maxLineLength)
{
    std::string line;
    line.reserve(Config::LineBufferSize);

    while (std::getline(inStream, line)) {
        for (std::size_t prevPos = 0, curPos = maxLineLength; curPos < line.length(); ) {
            if (curPos == line.length() - 1 && line[curPos] == '\r')
                break;

            const auto findInsertPosition = [curPos, prevPos](std::string_view str) {
                if (isCharWhitespace(str[curPos]))
                    return curPos;

                for (auto pos = curPos - 1; pos > prevPos; --pos) {
                    if (isCharWhitespace(str[pos]))
                        return pos + 1;
                }

                return curPos;
            };

            const auto insertPos = findInsertPosition(line);

            line.insert(insertPos, Config::FoldLineSequence);

            prevPos = insertPos + Config::FoldLineSequence.length();
            curPos  = prevPos + maxLineLength - 1;
        }

        outStream << line << '\n';
    }

    outStream << std::flush;
}

int main(int argc, char* argv[])
{
    std::ios_base::sync_with_stdio(false);
    std::cin.tie(nullptr);

    if constexpr (Config::CompileTimeLength) {
        if (argc != 1) [[unlikely]] {
            std::cerr << "Usage: "sv << argv[0];
            return ReturnCode::Error;
        } else {
            foldLines(std::cin, std::cout, Config::MaxLineLength);
        }
    } else {
        const auto maxLineLength = [argc, argv]() -> std::optional<std::size_t> {
            switch (argc) {
                case 1: return Config::MaxLineLength;
                case 2: return stringToSize(argv[1]);
                default: [[unlikely]] {
                    std::cerr << "Usage: "sv << argv[0] << " [max_line_length]"sv << '\n';
                    std::exit(ReturnCode::Error);
                }
            }
        }();

        if (!maxLineLength) [[unlikely]] {
            std::cerr << "Usage: "sv << argv[0] << " [max_line_length]"sv << '\n';
            std::cerr << "max_line_length must be a positive number"sv << '\n';
            return ReturnCode::Error;
        } else if (*maxLineLength < 2) [[unlikely]] {
            std::cerr << "Usage: "sv << argv[0] << " [max_line_length]"sv << '\n';
            std::cerr << "max_line_length must be at least 2"sv << '\n';
            return ReturnCode::Error;
        } else {
            foldLines(std::cin, std::cout, *maxLineLength);
        }
    }

    return ReturnCode::Success;
}

 

[sparek]

I see on their dev list there are problems also with exim exceeding it's own line length limit and causing a problem with some bounce messages, that is a problem. I also saw the above mentioned message threads, there are a couple conversations. Looks like some work still needs to be done on exim's end to also fix some unforeseen problems due to this change.

I think the issue is that so many mail service providers and mail clients either ignored this RFC or did not know it existed and when Exim introduced the setting in 4.95 and subsequently defaulting to the RFC value, this magnified the issue.

The fact that so many mail service providers and email client ignored or otherwise disregarded this RFC and email worked fine for years, tells me that the 998 limit is a relic of a long ago era. The RFC needs to be updated to a larger value or an explanation needs to be given as to what breaks when this limit is not enforced.

 

[kam821]

I think the issue is that so many mail service providers and mail clients either ignored this RFC or did not know it existed and when Exim introduced the setting in 4.95 and subsequently defaulting to the RFC value, this magnified the issue.

The fact that so many mail service providers and email client ignored or otherwise disregarded this RFC and email worked fine for years, tells me that the 998 limit is a relic of a long ago era. The RFC needs to be updated to a larger value or an explanation needs to be given as to what breaks when this limit is not enforced.

Most SMTP clients respect RFC standard and have line-folding implemented.

Many mail providers (including Gmail) also have line-folding implemented for their submission service.
It is submission server responsibility to avoid sending broken/non-compliant messages to external mailserver.
If some e-mail client is braindead (e. g. doesn't generate Message-ID), message should be fixed before being forwarded.

Some SMTP servers (e.g. Postfix) have line-folding functionality implemented, Exim unfortunately doesn't.
For a long time there was a hack, that was used to enforce RFC line limit (used e.g. in Debian's Exim package).
4.95 simply replaced this hack with the proper solution.

As an administrator of the SMTP server, you can do what should and can be done - implement missing line folding.
Cutting corners by some e-mail clients/providers (especially the big ones) and non-compliance with the standard, that is well defined and available for a long time, cannot be simply explained by the fact, that this is relic of a past.

If this is indeed obsolete, there should be proposal, standard should be updated and changes implemented, arbitrariness is not a proper way do to things.
In my opinion the Exim team should implement a built-in line folding and add variable to switch between rejecting and 'adjusting' messages that do not meet the condition.
Only then it makes sense to set default variable value at 998 and enforce it.

 

[sparek]

Well, gmail (209.85.222.49) is sending emails to my servers with lines over 998 bytes.

I'm not sure how those gmail users are sending mail through gmail's server and those users may be bypassing something within gmail that is adhering to this limit.

But presumably gmail (and others) has been sending messages with lines over 998 bytes for quite some time. And as far as I am aware of, this didn't cause any issues.

Non-native line folding in Exim is not yet a bridge I'm willing to cross. Maybe I will at some point. Maybe I can be convinced to do such at some point in time. But as it stands now, I'm a bit more comfortable with raising the message_linelength_limit variable since this would seem to better duplicate the behavior of Exim versions < 4.95.

 

[Richard G]

Would additional include line in exim.conf to change the parameter be enough?

Yes but exim.conf can be overwritten, right?
That's why I earlier suggested or rather asked, if this for example could be catched by a line we could add in exim.variables.conf.custom or exim.strings.conf.custom?
But any solution would be very nice.

It would at least be a very good workaround/solution until the MTA devs agree that they should change the RFC which nobody seems to obey to, except Exim suddenly.

 

[mxroute]

We do not refuse to add any easier ways to change the default and we're open for suggestions Would additional include line in exim.conf to change the parameter be enough?

Absolutely. However, I still think as you see more users upgrade to 4.95 you may want to consider a higher default simply to reduce your inbound tickets. It will take a while for people to update, longer for customers to notice, and the tickets won't all roll in at once. However, if you set a higher default it will only cause less problems and generate less tickets. From where I sit, that's pretty much the best criteria for making a universal decision. Remember as you go into it that the default for the value if not defined, is not a value being enforced anywhere. That's why increasing the default will break precisely nothing.

I'm all too familiar with the impact of sweeping changes and how much it can cost down the line even if it seemed like nothing up front. In this case, doing nothing is the sweeping change that has the cost.

 

[Yoshua]

As a fact....No they didn't raise the prices.
They are obligated to help and report in tickets, which has priority. The forum is (as with others) mostly not a place for priority support.

Richard, you know as well as I do, that before in this forum we were more in contact with the developers, they helped us with specific problems, we helped them to create a better panel and among the users we helped each other.
That disappeared since JBMC Software decided to change the course of its company, with the change of the licenses, the change of the support, the Pro-Pack, among other things.
Before we had few feature updates, but they were stable, the service updates were applied once tested and if something went wrong, they fixed it instantly thanks to the reports and help of well-known users in this forum, who unfortunately left us.
Now they take out the updates without being tested, they update the services without knowing their problems and being reported, they ignore us.
This is why now the forum is dead, and as you can see in other threads, I am not the only one who thinks this.

 

[Active8]

Richard, you know as well as I do, that before in this forum we were more in contact with the developers, they helped us with specific problems, we helped them to create a better panel and among the users we helped each other.

Naild it ! this is my toughs for an very long time good to see that other users feel the same

with the change of the licenses, the change of the support

Yep, despite many time we see them on the forum active, they do not participate actively
I understand you must earn money but please find a balance in it

they update the services without knowing their problems and being reported, they ignore us.

Agree

This is why now the forum is dead

Its more quiet yes , many active forum members like Alex have abandoned .
I am happy that users like @Richard G still tries to help people out , no offense to other who really tries to help also

 

[smtalk]

That's why increasing the default will break precisely nothing.

What would you suggest for the default value?

 

[Richard G]

Richard, you know as well as I do, that before in this forum we were more in contact with the developers

Yes but they did not raise prices as you said. However, they had to change some priority's.
But this is a discussion for the off-topic section.

And you know as well as I do that they are only a few people, and the sudden price change at CP caused a very big run of CP users to DA so staff here got overrun. And as a result of that everything including license policy (not prices) changed. They had to do that business wise, it wasn't fun, but if they hadn't done it, it would have been bad for their business.
Yes the Pro Pack is something special, but nothing you or an can't do without because the functions exist in another way.

Before we had few feature updates, but they were stable, the service updates were applied once tested and if something went wrong, they fixed it

That I can agree to. Updates were more often without any issues but we also had to wait a long time for nice stuff to be added. This happens faster now. I don't think they don't test updates before they come out. They also still bring out the beta's which users can test.
As for Exim, that is not their fault, that is Exim's fault and one does normally not override RFC's. So that nothing was done in the beginning was understandable. But smtalk is here now, right?

Its more quiet yes , many active forum members like Alex have abandoned .

Correct, we lost a very good one, Jeff, when he died. Also Alex has started doing other things and Sellerone is busy with other things. So collegue support is a lot lower, but since they aren't here anymore (or like before), the forum lost a lot of knowledge and helping hands.
Thank you for the compliment @Active8 but indeed there are also others helping. For me it's more easy maybe, because I don't have to live from it and I also see it as a hobby and have too much free time.

What would you suggest for the default value?

How about the Postfix default of 2048? I don't know.

 

[Driesp]

I suppose upping the default limit is not a real solution.
Exim enforced the RFC for a reason.
If you up the limit you breach the RFC anyway, it will never be about how much you breach it.
And even if you up the limit, it won't ever be high enough.

I would suggest holding off the update until exim provides a real solution.
I'ts already posted in this thread, a real fix would be exim escaping the string into smaller pieces.

Directadmin could provide 4.95 optionally and default to 4.94.2 for now.

Kr
Dries

 

[mxroute]

I suppose upping the default limit is not a real solution.
Exim enforced the RFC for a reason.
If you up the limit you breach the RFC anyway, it will never be about how much you breach it.
And even if you up the limit, it won't ever be high enough.

I would suggest holding off the update until exim provides a real solution.
I'ts already posted in this thread, a real fix would be exim escaping the string into smaller pieces.

Directadmin could provide 4.95 optionally and default to 4.94.2 for now.

Kr
Dries

They created a new variable. This was not a statement that they intended to see it's default value enforced. If all defaults were sane, we wouldn't have config files. No email service is enforcing a limit that matches the default value of this new variable. Like so many other RFCs that we all defy every day, this one has no inherent value in production. If you enjoy lecturing end users about how things should be instead of how they are, leaving it at the default may have value, just don't be surprised when they replace you. It's not a very new problem in IT, end users being tortured by admins for the benefit of no one.

What would you suggest for the default value?

I'd say somewhere excessive, even if it's 999,999. Since exim never split the lines before like postfix does, and since there are no complaints I can find of email services rejecting mail due to line length being too long, it's reasonable to assume that virtually no one out there is sending mail that contains lines long enough to generate a problem. This would basically keep things the closer to the same, rather than actually being a change. Though, personally, 8000 is proving to be an adequate (if not excessive) limit for my users and they're a pretty good sample.

Anyone running into a problem due to the line length limit being too high would have run into the same issue before the variable had been added, solidifying the idea that an excessively high limit merely maintains the status quo.

 

[This is it]

They created a new variable. This was not a statement that they intended to see it's default value enforced. If all defaults were sane, we wouldn't have config files. No email service is enforcing a limit that matches the default value of this new variable. Like so many other RFCs that we all defy every day, this one has no inherent value in production. If you enjoy lecturing end users about how things should be instead of how they are, leaving it at the default may have value, just don't be surprised when they replace you. It's not a very new problem in IT, end users being tortured by admins for the benefit of no one.



I'd say somewhere excessive, even if it's 999,999. Since exim never split the lines before like postfix does, and since there are no complaints I can find of email services rejecting mail due to line length being too long, it's reasonable to assume that virtually no one out there is sending mail that contains lines long enough to generate a problem. This would basically keep things the closer to the same, rather than actually being a change. Though, personally, 8000 is proving to be an adequate (if not excessive) limit for my users and they're a pretty good sample.

Anyone running into a problem due to the line length limit being too high would have run into the same issue before the variable had been added, solidifying the idea that an excessively high limit merely maintains the status quo.

8000 may not be long enough,
I tested it, and some people do make long read replys in textaera boxes without enter ;(
ok this is not often the case but it happens. why not make it 100k if it doesnt matter?

 

[Richard G]

8000 may not be long enough,

Why not? Long reply's will be wrapped normally. Even Microsoft is using 8000 so using more than them?

 

[sparek]

Just to back up @mxroute here.

Across all of our servers, I've managed to collect 7084 log incidents of the line length exceeding 998 bytes.

The average length of these miscreants is 4238 bytes.

544 of these 7084 incidents are larger than 4238 bytes. Or about 7.6% of the total.

If you more or less double this to 8192 (I don't really know why I'm using multiples of 1024) you get 157 out of 7084 or 2.22%.

Up to 9216 you get 155 - meaning two fell between 8192 and 9216 - 2.19%

The full data set looks like:

> 1024 - 93.61%
> 2048 - 43.63%
> 3072 - 15.77%
> 4096 - 8.15%
> 5120 - 5.86%
> 6144 - 4.19%
> 7168 - 3.01%
> 8192 - 2.22%
> 9216 - 2.19%
> 10240 - 2.12%
> 11264 - 1.33%
> 12288 - 1.04%
> 13312 - 0.85%
> 14336 - 0.72%
> 15360 - 0.64%
> 16384 - 0.59%
> 17408 - 0.56%
> 18432 - 0.55%
> 19456 - 0.55%
> 20480 - 0.54%

This shows a pretty clear convergence happening around 8192 and then another one at around 15360 or 16384 and another one at 18432. So 8192 would appear to be a valid starting point. @mxroute had stated an 8000 byte limit and this pretty much backs him up.

Earlier I had suggested a 4096 byte limit, but it would appear that 8192 is a bit more sound after crunching the numbers a bit more.

Now, having said all of that... I don't know if it's DirectAdmin's place to raise this value on it's own. This leads into the "are you a server administrator or are you expecting your control panel to be your server administrator?" debate. Since Exim itself is already controlled by DirectAdmin, more specifically exim_conf - this prevents server administrators from using DirectAdmin AND having full control of their Exim configuration (otherwise server administrators could edit their exim.conf file directly without fear of any DirectAdmin update chewing those changes up). DirectAdmin does a good job of trying to cover this by offering various include files all throughout the exim.conf. The exim.conf is littered with various .include_if_exists statements where server administrators can add their own code, effectively inserting that code into the exim.conf file where the include statement is made. And it also allows DirectAdmin to maintain a stable exim.conf.

But... what this issue magnifies... a server administrator is held hostage by where those .include_if_exists statements are. There isn't a server administrator's definable .include_if_exists in the necessary Exim transports to modify this behavior.

And when the next configuration option that server administrators want to customize comes out - what are the chances that an .include_if_exists statement will be in the necessary spot for server administrators to make changes?

It's a damned if you do, damned if you don't proposition.

And it's not feasible for DirectAdmin (or any control panel) to explicitly include every directive and configuration option with it's default value.

But to use this case as an example, IF DirectAdmin provided exim.conf had explicitly defined:

message_linelength_limit = 998

in the necessary transports - then I don't think any of the long winded responses in this thread would have any relevancy. Simply regex replace this statement and value with whatever we want in a post exim_conf hook.