Huge problem after reboot server

[Yzord]

I have a vps which runs Debian 6 and Directadmin which did his job perfect for 4 months. Last night all of a sudden the server hanged and a forced reboot was required. Since then, nothing works anymore. Why? Mysql won't run.

I've been busy since 3 AM last night to try to fix this, but as of now, i did not get any step further then a faulty server. The problem is that mysql and directadmin constantly restarting its service. No matter what i try...nothing helps, not even a bit.

First i have this:

2014:01:19-15:02:16: mysqld restarted
2014:01:19-15:03:01: directadmin started
2014:01:19-15:03:06: directadmin restarted
2014:01:19-15:03:11: mysqld started
2014:01:19-15:03:16: mysqld restarted
2014:01:19-15:04:01: directadmin started
2014:01:19-15:04:06: directadmin restarted
2014:01:19-15:04:11: mysqld started
2014:01:19-15:04:16: mysqld restarted
2014:01:19-15:05:01: directadmin started
2014:01:19-15:05:06: directadmin restarted
2014:01:19-15:05:11: mysqld started
2014:01:19-15:05:16: mysqld restarted
2014:01:19-15:06:01: directadmin started
2014:01:19-15:06:06: directadmin restarted
2014:01:19-15:06:11: mysqld started
2014:01:19-15:06:16: mysqld restarted
2014:01:19-15:07:01: directadmin started
2014:01:19-15:07:06: directadmin restarted
2014:01:19-15:07:11: mysqld started
2014:01:19-15:07:16: mysqld restarted
2014:01:19-15:08:01: directadmin started
2014:01:19-15:08:06: directadmin restarted
2014:01:19-15:08:11: mysqld started
2014:01:19-15:08:17: mysqld restarted
2014:01:19-15:09:01: directadmin started
2014:01:19-15:09:06: directadmin restarted

Which is causing the problem. So i checked some logs and found all this:

140119 15:34:01 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140119 15:34:01 [Note] Plugin 'FEDERATED' is disabled.
/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)
140119 15:34:01 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
140119 15:34:01 InnoDB: Initializing buffer pool, size = 8.0M
140119 15:34:01 InnoDB: Completed initialization of buffer pool
140119 15:34:01 InnoDB: Started; log sequence number 0 44233
140119 15:34:01 [ERROR] /usr/sbin/mysqld: Can't find file: './mysql/host.frm' (errno: 13)
140119 15:34:01 [ERROR] Fatal error: Can't open and lock privilege tables: Can't find file: './mysql/host.frm' (errno: 13)
140119 15:34:01 mysqld_safe mysqld from pid file /var/lib/mysql/safemail.nl.pid ended

Which is strange, cause my databases, which are made by DA, are located in /home/mysql. Also /usr/sbin/mysqld (i guess he wants a folder) is not there. A file named mysqld is there but when i open that it is all scrambled.
So someone gave me the advice to rebuild mysql trough the command

cd /usr/local/directadmin/custombuild/
./build set mysql_inst yes
./build set mysql_backup yes
./build update
./build mysql

The output of that was:

Stopping mysqld ...
mv: cannot stat `/etc/mysql/my.cnf': No such file or directory
Giving mysqld a few seconds to start up...
Looking for 'mysql' as: /usr/local/mysql/bin/mysql
Looking for 'mysqlcheck' as: /usr/local/mysql/bin/mysqlcheck
FATAL ERROR: Upgrade failed
*** Linking libmysqlclient.so.16 to libmysqlclient.so.18
Restarting MySQL.
It is recommended to do "./build php n" after MySQL update.

Yes, it's true that there is no /etc/mysql/my.cnf because i have another folder in /etc which is called mysql.moved where indeed my.cnf is located. The content of that one is:

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/english
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address = 127.0.0.1
#
# * Fine Tuning
#
key_buffer = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover = BACKUP
#max_connections = 100
#table_cache = 64
#thread_concurrency = 10
#
# * Query Cache Configuration
#
query_cache_limit = 1M
query_cache_size = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# Here you can see queries with especially long duration
#log_slow_queries = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
#binlog_do_db = include_database_name
#binlog_ignore_db = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem



[mysqldump]
quick
quote-names
max_allowed_packet = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer = 16M

#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

Which also does not look like the my.cnf i need. When i check mysql.log, then this file is empty.

I have been working on this already 15 hours, without a single step forward. I really don't know it anymore. Is there anyone who wants to help me?

 

[Yzord]

I was reading this forum and someone said you never do an upgrade trough apt-get but with the custombuild. And it is possible i did that. Can this be the problem and if so, is it possible to fix this?

 

[Arieh]

I don't have a solution, but you could check with

dpkg --get-selections | grep mysql

To see if you have mysql installed through apt-get. If so, you see output. If you get nothing it's not installed with apt-get.

 

[Yzord]

I don't have a solution, but you could check with

dpkg --get-selections | grep mysql

To see if you have mysql installed through apt-get. If so, you see output. If you get nothing it's not installed with apt-get.

The output is:

root@server:~# dpkg --get-selections | grep mysql
libmysqlclient-dev install
libmysqlclient16 install
mysql-common

Guess i am screwed. I am really desperate now. Would pay for solution. Anyone?

 

[nobaloney]

I can't offer to help because I don't have enough Debian experience.

Before doing that check to see if you've got two versions of MySQL running startup scripts, and if so, which one is running. Maybe you need to disable that one and make sure the other one is running, then restart your system.

Then once you're sure, remove the one that sholdn't have been installed, and make sure it won't run again.

I think a good place to look would be to see if yu've got both apt-get and CustomBuild installing MySQL. If so, then you probably should remove the version installed by apt-get and reinstall using MySQL.

Note I don't know enough about your setup to know that this won't delete all your databases. I do not guarantee this solution to not completely break your system. It may. I strongly suggest a complete backup before doing it.

I understand you likely cannot backup your databases from DirectAdmin. You should try doing that from the command line, and if that won't work, you probably want to at a minimum save all the database directories so you can restore them later.

But ... you probably want to hire a MySQL expert with Debian experience to assure your data.

Jeff