Solved Errors on mysqldump during CB update of 1 site. Reproducable.

[Richard G]

during mysql update, CB not using the command same as admin backup.

Yes that's what I also said. DA is using a different method. But during update the mysqldump command is used and there WF blocks things.

NP user can make backups himself then. It's clear to me that this is a WF issue, so I don't worry about it anymore then.

Sleep well, thank you!

 

[sysdev]

I get the impression that the error is in the password.
The password uses a . and a ! so as example (i masked it) it looks like 8P!S3D.6DF in the wp-config.php file.
Now when I use this line:
mysqldump -uuser_wp979 -p8P!S3D.6DF user_wp979 >/home/user/backups/dump.sql
the result is:
-bash: 8P!S3D.6DF: event not found

So the only way to do this via commandline (which mysql upgrade also uses) is using the -p without the password and then fill in the password after pressing enter.
mysqldump -uuser_wp979 -p user_wp979 >/home/user/backups/dump.sql
then fill in password after pressing enter and then this error occurs.

Enter password:
mysqldump: Error 2013: Lost connection to server during query when dumping table `wp_wfconfig` at row: 226

however some dump is made anyway. I didn't test if it would be the same when I would pack it with .tar.zst like DA does.

So it seems as if the password is causing the issue, it shouldn't give that "event not found error" either so that might also be causing the issue with the mysqldump that the mysql upgrade uses. While DA probably uses a different method to provide the password, which makes things work correctly.

Maybe this sheds some more light on the issue? Unless when using DA the Wordfence doesn't care and so WF is the issue.

Maybe add single quotes around passwords because using #/! etc expands first in the shell.

 

[Richard G]

because using #/! etc expands first in the shell.

Thank you for the tip, but there are 2 thoughts with that idea.
His passwords does not start with ! and we have more passwords with a ! or # in them which don't give issues.
Second thing is that even if this would work in shell, it still would not work during the custombuild update of MariaDB as this works automatically.

Since it's happening as of a certain table, it think it's Wordfence blocking the mysqldump command somehow.

 

[sysdev]

Thank you for the tip, but there are 2 thoughts with that idea.
His passwords does not start with ! and we have more passwords with a ! or # in them which don't give issues.
Second thing is that even if this would work in shell, it still would not work during the custombuild update of MariaDB as this works automatically.

Since it's happening as of a certain table, it think it's Wordfence blocking the mysqldump command somehow.

Well, in post #10 this expanding gives an error. It doesn't have to start with a !, just be somewhere unescaped in a command that user a subshell for it's execution. But, it would error on every backup then.

For a client to loose the connection it can have many causes. Different client libs, timeout's, dump queries using too much memory and OOM'ing the connection.

Check:

- does it happen during a high load? Try another backuptime
- enough ram? Check OOM messages
- did mariadb completely crash at that moment or is only the connection killed?
- check max_allowd_packet size. Increase it and check again.
- is the table too large so it triggers timeouts?
- Is it the same table every time? Export it, delete it, and try the backup again. Recreate it again afterwards. (or delete and reinstall wordfense completely)
- Do you have enough /tmp/ disk space? /tmp/ might gets filled when using large temporary tables leaving not enough space for huge mysql dumps.

Hope it helps!

 

[Richard G]

- does it happen during a high load? Try another backuptime

Thanks for your insights, but as you could read, it only happens with the mysqldump command, either when custombuild is doing the backup when upgrading MariaDB or when issueing the command manually.
I always to CB updates at early night time and we almost never have a high load then.

Also when using a single quotes around the password, when password provided on the commandline, same error on row 226 (always 226) and when using with single quotes when only -p is used and password given when prompted it's an access denied error.

- enough ram? Check OOM messages

I don't know where I can find these, but we have 64 GB RAM in the server, never have issues with too little ram. This is from this moment when usage is a bit higher.

               total        used        free      shared  buff/cache   available
Mem:        65350660    10360988     1888568      713164    54537624    54989672
Swap:       33520636           0    33520636

- did mariadb completely crash at that moment or is only the connection killed?

Neither, only on this database the connection is lost (says Mariadb) during custombuild backup of the database, after this database there are a lot of other databases also being backed up by CB without any issues.

It does -never- happen when creating a backup via the user backup option or via the admin backup/transfer option.
To me this means, there is no issue with ram or tables too large otherwise it would also happen during backups made from DA itself.
The database is even fairly small (uncompressed like 87 MB) and we got way bigger on that server and ram and space more than enough.

So this all, and since backup from within DA is working correctly, it strenghts me in the thought that it must be Wordfence, I can't think of anything else which would only block mysqldump backups.
Users are responsible for backups themselfs on our servers anyway and the user has backups.

 

[Ohm J]

Directadmin use different arguments of musqldump between admin backup and dump during install/update.

if you want to know which arg, just try use the admin backup and search command in top/process monitor.