Solved Mysql don't start anymore

[Richard G]

Mysql was running and suddenly yesterday evening there was no access to localhost anymore for roundcube and others.

So I rebuilded roundcube, did not work, restarted mysql and it went down.

2023-08-10 14:30:02 0 [Warning] InnoDB: Log scan aborted at LSN 616003072
2023-08-10 14:30:02 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-08-10 14:30:02 0 [Note] InnoDB: Starting shutdown...
2023-08-10 14:30:02 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-08-10 14:30:02 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-08-10 14:30:02 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-08-10 14:30:02 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2023-08-10 14:30:02 0 [ERROR] Aborting
2023-08-10 14:31:15 0 [Note] Starting MariaDB 10.6.14-MariaDB-log source revision c93754d45e5d9379e3e23d7ada1d5f21d2711f66 as process 741
2023-08-10 14:31:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
2023-08-10 14:31:16 0 [Note] InnoDB: Number of pools: 1
2023-08-10 14:31:16 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-08-10 14:31:16 0 [Note] InnoDB: Using Linux native AIO
2023-08-10 14:31:16 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2023-08-10 14:31:16 0 [Note] InnoDB: Completed initialization of buffer pool
2023-08-10 14:31:16 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=615948759,615948759
2023-08-10 14:31:16 0 [ERROR] InnoDB: Corrupted page identifier at 615948760; set innodb_force_recovery=1 to ignore the record.
2023-08-10 14:31:16 0 [Note] InnoDB: Dump from the start of the mini-transaction (LSN=615948760) to 100 bytes after the record:
 len 101; hex ffffffff0000350080833100c31604ffb48f82963747008073aab204ff3400003d970300824035000081e6ab48008073aa72804000003a00808432ffffffff0000b700ffffffff0000350080843100c31604ffb48f82963747008073b2f204ff3400003d96; asc       5   1         7G  s    4  =    @5     H  s r @  :   2              5   1         7G  s    4  = ;

Edit:
it's on a VPS, enough memory free:

               total        used        free      shared  buff/cache   available
Mem:        30805992     2052612    27014200       45848     1739180    28337060
Swap:        4194300           0     4194300

disk also enough free:

[root@srv100: /var/log/mysql]# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             15G     0   15G   0% /dev
tmpfs           3.0G  516K  3.0G   1% /run
/dev/sda3       785G  153G  592G  21% /
tmpfs            15G     0   15G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda2       2.0G   79M  1.8G   5% /boot
tmpfs           3.0G     0  3.0G   0% /run/user/0

Debian 11 Mariadb 10.6

The /etc/my.cnf file

[mysqld]
max_allowed_packet=64M
local-infile=0
bind-address=127.0.0.1
# skip-name-resolve=1

max_connections = 300

sql_mode=
#datadir=/var/lib/mysql
# socket=/var/lib/mysql/mysql.sock
#tmp_table_size=512M
#table_definition_cache=3000
#table_open_cache=4000
#max_heap_table_size=512M
#innodb_buffer_pool_size=6144M
#innodb_buffer_pool_instances=4
#innodb_log_file_size=768M

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

log-error=/var/log/mysql/mysqld.log
slow_query_log
slow_query_log_file=/var/log/mysql/mariadb-slow.log

#
# include all files from the config directory
#
#!includedir /etc/my.cnf.d

[client]
socket=/usr/local/mysql/data/mysql.sock

Can this still be fixed? @fln or anybody? @Zhenyapan ?

 

[wtptrs]

Try to set

innodb_force_recovery=1

in my.cnf and try to start MySQL. If you're able to remove the corrupted table(s), you can unset the my.cnf value.

Some more info:

Warning
Only set innodb_force_recovery to a value greater than 0 in an emergency situation, so that you can start InnoDB and dump your tables. Before doing so, ensure that you have a backup copy of your database in case you need to recreate it. Values of 4 or greater can permanently corrupt data files. Only use an innodb_force_recovery setting of 4 or greater on a production server instance after you have successfully tested the setting on a separate physical copy of your database. When forcing InnoDB recovery, you should always start with innodb_force_recovery=1 and only increase the value incrementally, as necessary.

 

[Richard G]

I did that, but didn't help, gave an error that innodb was not recognized or something like that.

 

[Richard G]

Seems Debian crap to me, I already had issues with this database ad localhost connections when nothing was in there and VPS was freshly installed with only DA on it.

 

[ericosman]

Found this on the internet:

This error also happens when your Database data is corrupt. You may fix this issue by moving your Db data files (ib_logfile0 and ib_logfile1) mentioned below to another location. ib_logfile0 and ib_logfile1 are system tablespace for the InnoDB infrastructure. These files contains several classes for information vital for InnoDB. You may read about these files here.
Before following below steps please keep a copy of files (ib_logfile0 and ib_logfile1) so you may restore your data in case it is lost:
Follow below steps:

• Login to server via SSH with root access.
• Navigate to /var/lib/mysql.
• If you see files like, ib_logfile0 and ib_logfile1, rename or move them to some other folder.
• Stop and start the MySQL service by running sudo service mysql stop and sudo service mysql start

These files will be recreated after you restart the server and the issue will be fixed hopefully.
Thanks

 

[Richard G]

Yes I found that too, but I'm afraid to loose more, so I wanted to see if somebody with a big mariadb knowledge says this is a correct procedure.

Also I can't stop the service anymore because it's already stopped.

 

[Richard G]

Just tried it, and MariaDB seems running again, but now I encounter this issue:

[root@srv100: /var/lib/mysql]# mysql -u root -p
Enter password: 
ERROR 2002 (HY000): Can't connect to local server through socket '/run/mysqld/mysqld.sock' (2)

I tried to solve this to change the root password according to the docs.

I might be able to fix this by removing the mailutils installation.

 

[ericosman]

Try : sudo service mysql start

internet says:





I once had this problem and solved it by installing mysql-server, so make sure that you have installed the mysql-server, not the mysql-client or something else.
That error means the file /var/run/mysqld/mysqld.sock doesn't exists, if you didn't install mysql-server, then the file would not exist. So in that case, install it with
sudo apt-get install mysql-server

But if the mysql-server is already installed and is running, then you need to check the config files.
The config files are:
/etc/my.cnf
/etc/mysql/my.cnf
/var/lib/mysql/my.cnf

In /etc/my.cnf, the socket file config may be /tmp/mysql.sock and in /etc/mysql/my.cnf the socket file config may be /var/run/mysqld/mysqld.sock. So, remove or rename /etc/mysql/my.cnf, let mysql use /etc/my.cnf, then the problem may solved.

 

[Richard G]

Nope, mailutils was not installed. But bsd-mailx was installed. I removed that packages but same error.

ERROR 2002 (HY000): Can't connect to local server through socket '/run/mysqld/mysqld.sock' (2)
[root@srv100: /etc]#

I only have /etc/my.cnf I don't have any /etc/mysql/my.cnf or /var/lib/mysql/my.cnf.
The /var/run/mysqld/mysqld.sock did not exist, neither did it yesterday and then it did not cause any problems. The problems suddenly started yesterday evening.

There is 1 mysql.sock which is in /var/lib/mysql where it should be.

Website with databases are running now but nothing can be changed or backed-up because there is no access to localhost.

2023-08-10 15:07:58 0 [Note] Starting MariaDB 10.6.14-MariaDB-log source revision c93754d45e5d9379e3e23d7ada1d5f21d2711f66 as process 9020
2023-08-10 15:07:58 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
2023-08-10 15:07:58 0 [Note] InnoDB: Number of pools: 1
2023-08-10 15:07:58 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-08-10 15:07:58 0 [Note] InnoDB: Using Linux native AIO
2023-08-10 15:07:58 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2023-08-10 15:07:58 0 [Note] InnoDB: Completed initialization of buffer pool
2023-08-10 15:07:58 0 [Note] InnoDB: 128 rollback segments are active.
2023-08-10 15:07:58 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-08-10 15:07:58 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-08-10 15:07:58 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-08-10 15:07:58 0 [Note] InnoDB: 10.6.14 started; log sequence number 618398579; transaction id 104780
2023-08-10 15:07:58 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-08-10 15:07:58 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-08-10 15:07:58 0 [Note] Server socket created on IP: '127.0.0.1'.
2023-08-10 15:07:58 0 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '10.6.14-MariaDB-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
2023-08-10 15:07:58 0 [Note] InnoDB: Buffer pool(s) load completed at 230810 15:07:58
2023-08-10 15:10:48 33 [Warning] Access denied for user 'admin_wp484'@'localhost' (using password: YES)

 

[Richard G]

Oke I found the cause, just don't know how to fix this yet. It's again the same Debian crap as I had before. I NEVER had this with RH alikes, so this really is a pain in the but.

Look at this:

[root@srv100: /etc]# mysql -u da_admin -p  
Enter password: 
ERROR 2002 (HY000): Can't connect to local server through socket '/run/mysqld/mysqld.sock' (2)

Now we specify the host 127.0.0.1:

[root@srv100: /etc]# mysql -h 127.0.0.1 -u da_admin -p         
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 6
Server version: 10.6.14-MariaDB-log MariaDB Server

tadaa and we're in.

Which is odd because it has bee working the whole time correctly with the bind-address=127.0.0.1 setting in the /etc/my.cnf file.

Now I tried in my.cnf the setting:
bind-address=0.0.0.0
and tried removing the bind setting, but it doesn't help.

From the logfile:

2023-08-10 15:39:07 0 [Note] Server socket created on IP: '0.0.0.0'.
2023-08-10 15:39:07 0 [Note] Server socket created on IP: '::'.

Anybody knows how I can fix this in Debian?

 

[Richard G]

Seems no mariadb-server package is installed either but I don't know if that is needed in Debian 11.

 

[pat0]

Script is searching socket in /run/mysqld/mysqld.sock and you have one here /var/lib/mysql/mysql.sock
Just create symlink from /var/lib/mysql/mysql.sock to /run/mysqld/mysqld.sock and you will be fine.

 

[Richard G]

yes but that is a workaround not a fix.
I had it working before without any symlink needed. I will use that as a temp fix, but i'm still looking for the correct permanent fix for this. Symlink should not be needed.

 

[Richard G]

Also, roundcube does not work, and rebuilding does not fix it.

 

[Richard G]

Tried to rebuild roundcube, threw an error. So tried to drop Roundcube database and then:

can't rmdir './da_roundcube', errno: 39 "Directory not empty"

Pfffff.

A build roundcube gives:

ERROR: Error in DDL upgrade 2013011000: [1005] Can't create table `da_roundcube`.`system` (errno: 184 "Tablespace already exists")

 

[sysdev]

Seems like you have a different socket path in one of your mysql config files. Maybe a rogue ~/my.cnf somewhere? A host : port type of connection in one file and a socket=/bla/bla somewhere else?

 

[Richard G]

No. Nowhere unfortunately. There is only 1 my.cnf on the whole server an only 1 mysql.sock.

 

[smtalk]

Just tried it, and MariaDB seems running again, but now I encounter this issue:

[root@srv100: /var/lib/mysql]# mysql -u root -p
Enter password:
ERROR 2002 (HY000): Can't connect to local server through socket '/run/mysqld/mysqld.sock' (2)

I tried to solve this to change the root password according to the docs.

I might be able to fix this by removing the mailutils installation.

Just use:

mysql --defaults-extra-file=/usr/local/directadmin/conf/my.cnf

 

[Richard G]

Just use:

I fixed it by creating the symlink, for the time being, however normally mysql -u root -p should work.

Also, this does unfortunately not fix the roundcube issue either (post #15). I struggled with this database on newly installed VPS from the beginning. Odd that it stopped working correctly and refusing localhost again since last evening, without doing any changes.

 

[pat0]

If custombuild was still a normal sh script and not compiled into directadmin binary, you would have the option to rename database in it to da_roundcube2 or something and it will be it, but now only way to do that is manual install...