Solved Rsync connection setup very slow to 1 server only, transfer fast.

[Richard G]

Often I transfer some file from one server to the others. Originating server can be different.

I use this command for example:
rsync -avzHpWP -e 'ssh -p 12345' somefile.conf [email protected]:/somedir/

On all servers I use the same authorized_keys file to login via SSH, same port, same sshd_config file.

The connection being made to all servers except one is within 3-5 seconds.
However, the connection to 1 server is veeeeeerrrrrrryyyyy slooooooowwwww.... Connection setup from any server to that server often takes 30 seconds or more.

The transfer of the file or files is the same speed as from or to any other server.

The other way around, so from the problem server to any other server, the connection setup is very fast, even faster than some of the others.

So it's only the incoming rsync connection setup wich takes a lot of time. Any clue on what could be causing this?

 

[cjd]

Possible DNS lookup issue timing out on that server? Try disabling rsyncd reverse dns lookup. Or in ssh daemon disable DNS lookup.

 

[Richard G]

DNS lookups works fine. If I do a nslookup on anything on that server, it responses immediately.
UseDNS no on all servers in the sshd_config file.

 

[Richard G]

Maybe interesting, I also have this on all servers:

#  /etc/ssh/sshd_config.d/  which will be automatically included below
Include /etc/ssh/sshd_config.d/*.conf

# If you want to change the port on a SELinux system, you have to tell
# SELinux about this change.
# semanage port -a -t ssh_port_t -p tcp #PORTNUMBER
#
Port 12345
AddressFamily inet
#ListenAddress 0.0.0.0
#ListenAddress ::
AuthorizedKeysFile      .ssh/authorized_keys
ClientAliveInterval 120
ClientAliveCountMax 30
UseDNS no

I don't want to connect to SSH via ipv6 so that's why I use the inet AddressFamily.
Firewalls are also the same, port 12345 (masked port) is opened incoming and outgoing on all servers ipv4.

 

[Ohm J]

Disable firewall and directly connection via ssh without rsync still have same problem ?

 

[Richard G]

@jamgames2 You are correct. With firewall enabled on the receiving server, problem persists.
With firewall disabled on the receiving server, connection is made instantly.

And that is very odd, because this specific sending server is even present in the /etc/csf/csf.allow file which is connected with the ignore file so it should bypass the firewall anyway.

So now I'm even more confused.

 

[cjd]

Try disabling CFS's LF_LOOKUPS and maybe also ST_LOOKUP if it is enabled.

 

[Ohm J]

@Richard G
Some option of CSF Firewall doesn't ignored. Mostly ignored option just filter when there have some Temp Block or Perm Block. other just mess up.

 

[Richard G]

Some option of CSF Firewall doesn't ignored.

I've got IGNORE_ALLOW = "1" and if my ip is in the csf.allow then I can reach everything even if the port is closed, no matter which port. So that should be even better than other servers.

So it's very odd, maybe some other setting. Because same is happening when I try from other servers too which are not present in the csf.allow file.

@cjd I don't have ST_LOOKUP enabled. I tested with LF_LOOKUPS disabled, but same result.

It's also happening with to other servers incoming, but 2 two others not. So I'm going to compare firewall settings again, see if there is a difference anywhere. Although I doubt that since they are setup the same mostly.

 

[Richard G]

Found it!

In spite of the fact that I told sshd_config to only connect to ipv4, for some reason csf will check if port 12345 is also allowed on ipv6 anyway.
As soon as I added that port in the ipv6 section of CSF, connections are going fast again.

@jamgames2 you were partly right. Bypassing the firewall does take place, but not instantly, firewall does check ports anyway (closed or open) before letting the ip bypass.
And for ip's not in the csf.allow it's the same. Seems ipv6 is checked first before sshd_config (which only allows ipv4 connections) is checked. Which must be causing the delay.

 

[zEitEr]

Does your hostname server.domain.com resolve to IPv6 as well as IPv4? If it does, then your rsync/ssh client tries IPv6 first. And that's most likely the culprit.

 

[Richard G]

Yep, nowadays it does also resolve to ipv6. So we're having the same idea. Ipv6 is checked first causing the delay.

 

[Stije]

So if you use rsync -4 it works good? Else it could also be conntracking

 

[Richard G]

I don't know as I didn't try with -4. I always use just rsync and that was always working fine on all servers.
Probably because at first I didn't use ipv6 and since we were using ipv6 the port was always opened in ipv6 too.

Since it always worked without specifying I was looking for the cause and found it. It was not in my intention to use a seperate -4 option, it just had to work like on the other servers.

 

[Stije]

I know but it would also exclude that it is really ipv4 / ipv6. As in you could test with -4 to see if it is correct right?

I am interested as i have a lot of rsyncs and i wander if this really is the cullprit I use a lot of servers with ip4 and ip6 and i do not have this problem. Or i can speed up my rsync apparantly Just wandering where it comes from

 

[Richard G]

As in you could test with -4 to see if it is correct right?

Yes but then I would have to change things back, why can't you test it yourself?

I use a lot of servers with ip4 and ip6 and i do not have this problem.

But do you use the same config as me, where you have set SSH to be limited to use ipv4? Because if you don't have that, then it's logical you don't have the issue I had.