Mysqld is down

K

kbtechno

Verified User
Joined
Sep 7, 2021
Messages
16
Hi I have a problem with my server.
I have fresh installed everthing and optimize the server.
I am using
Directadmin
Openlitespeed
Cloudlinux

Problem is with MYSQL it is stop and restart again.
I just getting this logs
Code: 
[ERROR] mysqld got signal 11 ;
Sorry, we probably made a mistake, and this is a bug.
Your assistance in bug reporting will enable us to fix this for the next release.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.6.20-MariaDB-cll-lve source revision: f00711bba2cd383825d0be1867f7d7d7f641c9e4
key_buffer_size=27262976
read_buffer_size=131072
max_used_connections=93
max_threads=802
thread_count=93
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1793002 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f153000a8c8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f16281f5000 thread_stack 0x49000
2025-02-27 16:17:56 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x560c696e8f4e]
/usr/sbin/mariadbd(handle_fatal_signal+0x478)[0x560c69197e68]
/lib64/libc.so.6(+0x3e730)[0x7f1c5f43e730]
/usr/sbin/mariadbd(+0xdd52de)[0x560c695402de]
/usr/sbin/mariadbd(+0xe75025)[0x560c695e0025]
/usr/sbin/mariadbd(+0xe0ff84)[0x560c6957af84]
/usr/sbin/mariadbd(+0xd4330e)[0x560c694ae30e]
/usr/sbin/mariadbd(_ZN7handler18ha_index_next_sameEPhPKhj+0x2d5)[0x560c691a0cb5]
/usr/sbin/mariadbd(+0x8358a1)[0x560c68fa08a1]
/usr/sbin/mariadbd(_Z10sub_selectP4JOINP13st_join_tableb+0x1f0)[0x560c68f93d30]
/usr/sbin/mariadbd(+0x814078)[0x560c68f7f078]
/usr/sbin/mariadbd(_Z10sub_selectP4JOINP13st_join_tableb+0x22d)[0x560c68f93d6d]
/usr/sbin/mariadbd(+0x814078)[0x560c68f7f078]
/usr/sbin/mariadbd(_Z10sub_selectP4JOINP13st_join_tableb+0x22d)[0x560c68f93d6d]
/usr/sbin/mariadbd(_ZN4JOIN10exec_innerEv+0x103a)[0x560c68fc0b1a]
/usr/sbin/mariadbd(_ZN4JOIN4execEv+0x37)[0x560c68fc0e77]
/usr/sbin/mariadbd(_Z12mysql_selectP3THDP10TABLE_LISTR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x106)[0x560c68fbef46]
/usr/sbin/mariadbd(_Z13handle_selectP3THDP3LEXP13select_resultm+0x169)[0x560c68fbf6f9]
/usr/sbin/mariadbd(+0x7d8a85)[0x560c68f43a85]
/usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x4540)[0x560c68f53bc0]
/usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x3ee)[0x560c68f5569e]
/usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x14f5)[0x560c68f57f15]
/usr/sbin/mariadbd(_Z10do_commandP3THDb+0x13f)[0x560c68f59e9f]
/usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x395)[0x560c6906bae5]
/usr/sbin/mariadbd(handle_one_connection+0x5d)[0x560c6906be3d]
/usr/sbin/mariadbd(+0xc8cbd2)[0x560c693f7bd2]
/lib64/libc.so.6(+0x89d22)[0x7f1c5f489d22]
/lib64/libc.so.6(+0x10ed40)[0x7f1c5f50ed40]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f1530015330): SELECT SQL_CALC_FOUND_ROWS  wp_posts.ID
           FROM wp_posts  INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id )  INNER JOIN wp_postmeta AS mt1 ON ( wp_posts.ID = mt1.post_id )
           WHERE 1=1  AND (
  wp_postmeta.meta_key = '_elementor_template_type'
  AND
  (
    ( mt1.meta_key = '_elementor_template_type' AND mt1.meta_value = 'popup' )
  )
) AND wp_posts.post_type = 'elementor_library' AND ((wp_posts.post_status <> 'trash' AND wp_posts.post_status <> 'auto-draft' AND wp_posts.post_status <> 'vacation'))
           GROUP BY wp_posts.ID
           ORDER BY wp_posts.post_date DESC
           LIMIT 0, 1
Connection ID (thread ID): 1933227
Status: KILL_SERVER
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off,cset_narrowing=off
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units   
Max cpu time              unlimited            unlimited            seconds 
Max file size             unlimited            unlimited            bytes   
Max data size             unlimited            unlimited            bytes   
Max stack size            8388608              unlimited            bytes   
Max core file size        unlimited            unlimited            bytes   
Max resident set          unlimited            unlimited            bytes   
Max processes             254991               254991               processes
Max open files            32768                32768                files   
Max locked memory         8388608              8388608              bytes   
Max address space         unlimited            unlimited            bytes   
Max file locks            unlimited            unlimited            locks   
Max pending signals       254991               254991               signals 
Max msgqueue size         819200               819200               bytes   
Max nice priority         0                    0                  
Max realtime priority     0                    0                  
Max realtime timeout      unlimited            unlimited            us      
Core pattern: "/var/crash/%e-%t.core"
Kernel version: Linux version 5.14.0-503.21.1.el9_5.x86_64 ([email protected]) (gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), GNU ld version 2.35.2-54.el9) #1 SMP PREEMPT_DYNAMIC Sun Jan 12 09:45:05 EST 2025
2025-02-27 16:18:02 0 [Note] Starting MariaDB 10.6.20-MariaDB-cll-lve source revision f00711bba2cd383825d0be1867f7d7d7f641c9e4 server_uid 21tU7ERLM7oPwGeLYgYA9Ueg7Uk= as process 3417930
2025-02-27 16:18:02 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2025-02-27 16:18:02 0 [Note] InnoDB: Number of pools: 1
2025-02-27 16:18:02 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2025-02-27 16:18:02 0 [Note] InnoDB: Using Linux native AIO
2025-02-27 16:18:02 0 [Note] InnoDB: Initializing buffer pool, total size = 25769803776, chunk size = 134217728
2025-02-27 16:18:02 0 [Note] InnoDB: Completed initialization of buffer pool
2025-02-27 16:18:02 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1057572410686,1057572413588
2025-02-27 16:18:02 0 [Note] InnoDB: To recover: 1208 pages
2025-02-27 16:18:02 0 [Note] InnoDB: 128 rollback segments are active.
2025-02-27 16:18:02 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2025-02-27 16:18:02 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2025-02-27 16:18:02 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2025-02-27 16:18:02 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2025-02-27 16:18:02 0 [Note] InnoDB: 10.6.20 started; log sequence number 1057616709250; transaction id 109457680
2025-02-27 16:18:02 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-02-27 16:18:02 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2025-02-27 16:18:02 0 [Note] Server socket created on IP: '0.0.0.0'.
2025-02-27 16:18:02 0 [Note] Server socket created on IP: '::'.
2025-02-27 16:18:02 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.6.20-MariaDB-cll-lve'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
2025-02-27 16:18:05 0 [Note] InnoDB: Buffer pool(s) load completed at 250227 16:18:05
2025-02-27 16:18:07 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
2025-02-27 16:18:07 0 [Note] InnoDB: FTS optimize thread exiting.
2025-02-27 16:18:07 0 [Note] InnoDB: Starting shutdown...
2025-02-27 16:18:07 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2025-02-27 16:18:07 0 [Note] InnoDB: Restricted to 389376 pages due to innodb_buf_pool_dump_pct=25
2025-02-27 16:18:07 0 [Note] InnoDB: Buffer pool(s) dump completed at 250227 16:18:07
2025-02-27 16:18:08 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2025-02-27 16:18:08 0 [Note] InnoDB: Shutdown completed; log sequence number 1057616795497; transaction id 109457922
2025-02-27 16:18:08 0 [Note] /usr/sbin/mariadbd: Shutdown complete
2025-02-27 16:18:08 0 [Note] Starting MariaDB 10.6.20-MariaDB-cll-lve source revision f00711bba2cd383825d0be1867f7d7d7f641c9e4 server_uid 21tU7ERLM7oPwGeLYgYA9Ueg7Uk= as process 3418064
2025-02-27 16:18:08 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2025-02-27 16:18:08 0 [Note] InnoDB: Number of pools: 1
2025-02-27 16:18:08 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2025-02-27 16:18:08 0 [Note] InnoDB: Using Linux native AIO
2025-02-27 16:18:08 0 [Note] InnoDB: Initializing buffer pool, total size = 25769803776, chunk size = 134217728
2025-02-27 16:18:08 0 [Note] InnoDB: Completed initialization of buffer pool
2025-02-27 16:18:08 0 [Note] InnoDB: 128 rollback segments are active.
2025-02-27 16:18:08 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2025-02-27 16:18:08 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2025-02-27 16:18:08 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2025-02-27 16:18:08 0 [Note] InnoDB: 10.6.20 started; log sequence number 1057616795497; transaction id 109457922
2025-02-27 16:18:08 0 [Note] Plugin 'FEEDBACK' is disabled.
2025-02-27 16:18:08 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2025-02-27 16:18:08 0 [Note] Server socket created on IP: '0.0.0.0'.
2025-02-27 16:18:08 0 [Note] Server socket created on IP: '::'.
2025-02-27 16:18:08 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.6.20-MariaDB-cll-lve'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
2025-02-27 16:18:11 0 [Note] InnoDB: Buffer pool(s) load completed at 250227 16:18:11
2025-02-27 16:19:08 298 [ERROR] mariadbd: Table './etds_wp430/wpcv_options' is marked as crashed and should be repaired
2025-02-27 16:19:08 298 [Warning] Checking table:   './etds_wp430/wpcv_options'
2025-02-27 18:46:35 56162 [Warning] Access denied for user 'username_here'@'localhost' (using password: YES)


It is happening randomly. I am not able to fix this issue. I don't know what is wrong with mysql.
 
Last edited by a moderator:
Hello,

Just a guess, I did not find any evidence to it, and still. The MySQL server gets probably killed by OOM_Killer, and your server ran out of free RAM. In order to confirm or discard this version you will need to check /var/log/messages for possible lines related to OOM Killer and Out of memory.

And since you are using CloudLinux you might open a ticket with their support team and let them to troubleshoot the issue without an additional cost.
 
Hi,

Upon check logs i found these


Feb 28 05:32:08 s1 kernel: oom_kill_process.cold+0xb/0x10
Feb 28 05:32:09 s1 kernel: oom_kill_process.cold+0xb/0x10
Feb 28 05:32:10 s1 kernel: oom_kill_process: 3 callbacks suppressed
Feb 28 05:32:10 s1 kernel: oom_kill_process.cold+0xb/0x10
Feb 28 05:32:11 s1 kernel: oom_kill_process.cold+0xb/0x10
Feb 28 05:32:11 s1 kernel: oom_kill_process.cold+0xb/0x10
Feb 28 05:32:12 s1 kernel: oom_kill_process.cold+0xb/0x10
Feb 28 05:32:13 s1 kernel: oom_kill_process.cold+0xb/0x10
Feb 28 05:32:13 s1 kernel: oom_kill_process.cold+0xb/0x10
Feb 28 05:32:14 s1 kernel: oom_kill_process.cold+0xb/0x10
Feb 28 18:01:44 s1 kernel: oom_kill_process.cold+0xb/0x10
Feb 28 18:01:46 s1 kernel: oom_kill_process.cold+0xb/0x10
Feb 28 18:01:46 s1 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/lve1062,task_memcg=/lve1062,task=lsphp,pid=208905,uid=1062
Feb 28 18:01:44 s1 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/lve1062,task_memcg=/lve1062,task=lsphp,pid=207857,uid=1062
Feb 28 18:01:44 s1 kernel: Memory cgroup out of memory: Killed process 207857 (lsphp) total-vm:422364kB, anon-rss:176912kB, file-rss:20120kB, shmem-rss:0kB, UID:1062 pgtables:624kB oom_score_adj:0
Feb 28 18:01:46 s1 kernel: lsphp invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Feb 28 05:32:13 s1 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/lve1139,task_memcg=/lve1139,task=lsphp,pid=4055838,uid=1139
Feb 28 05:32:13 s1 kernel: Memory cgroup out of memory: Killed process 4055838 (lsphp) total-vm:236908kB, anon-rss:80068kB, file-rss:7580kB, shmem-rss:0kB, UID:1139 pgtables:288kB oom_score_adj:0
Feb 28 05:32:14 s1 kernel: lsphp invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Feb 28 18:01:44 s1 kernel: lsphp invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Feb 28 05:32:14 s1 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/lve1139,task_memcg=/lve1139,task=lsphp,pid=4055830,uid=1139

I am not sure which one is related with mysql

there is different kind of oom kill proccess
 
Hi,

Feb 28 03:15:22 s1 kernel: [ 123456.789] Out of memory: Killed process 1234 (mysqld) total-vm:1024000kB, anon-rss:512000kB, file-rss:1024kB, shmem-rss:0kB
Feb 28 03:15:22 s1 kernel: [ 123456.790] oom_reaper: reaped process 1234 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Feb 28 03:45:55 s1 mysqld[5678]: [Warning] InnoDB: page_cleaner: 1000ms intended loop took 2578ms. The settings might not be optimal.
Feb 27 22:10:15 s1 systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Feb 27 22:10:15 s1 systemd[1]: mariadb.service: Failed with result 'signal'.Feb 27 22:10:16 s1 systemd[1]: mariadb.service: Service RestartSec=100ms expired, scheduling restart.
Feb 27 22:10:16 s1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.

These details.
 
Hi,
2025-02-28 5:32:06 435484 [Warning] Aborted connection 435484 to db: 'oks_w8' user: 'oks_w8' host: 'localhost' (Got an error reading communication packets)
2025-02-28 5:32:07 435520 [Warning] Aborted connection 435520 to db: 'oks_w8' user: 'oks_w8' host: 'localhost' (Got an error reading communication packets)
2025-02-28 5:32:07 435502 [Warning] Aborted connection 435502 to db: 'oks_w8' user: 'oks_w8' host: 'localhost' (Got an error reading communication packets)
2025-02-28 5:32:07 435524 [Warning] Aborted connection 435524 to db: 'oks_w8' user: 'oks_w8' host: 'localhost' (Got an error reading communication packets)
2025-02-28 5:32:08 435577 [Warning] Aborted connection 435577 to db: 'oks_w8' user: 'oks_w8' host: 'localhost' (Got an error reading communication packets)

is these warnings is serious
 
install program like htop an watch your resources , see if there any hiccups
 
Hi,
Htop is already installed.
it is not like something happening everytime.
Its just happened and mysql get restart
 
kbtechno said:
is these warnings is serious
Click to expand...
G

Solved Thread 'Got an error writing communication packets'

Have one database, which mysqldump cannot complete its job. The error:
mysqldump: Error 2013: Lost connection to server during query when dumping table `xxxxx_options` at row: 399
Click to expand...
In /var/log/messages, the error was recorded:
[Warning] Aborted connection 179 to db: 'xxxx_db1' user: 'da_admin' host: 'localhost' (Got an error writing communication packets)
Click to expand...

I tried to increase max_allowed_packets, net_write_timeout, net_read_timeout, time_wait... but error is still appearing. Meanwhile, the db can be exported with phpMyAdmin, so the MariaDB server config should be fine, and the...
 
My cnf:

[mysqld]
max_allowed_packet = 256M
innodb_buffer_pool_size = 24G
join_buffer_size = 512K

interactive_timeout=300
net_read_timeout=60
net_write_timeout=60

tmp_table_size=64M
max_heap_table_size=64M

table_definition_cache=1000

local-infile = 0
max_connections = 800
innodb_file_per_table

key_buffer_size=26M

myisam_sort_buffer_size = 64M

log_error=/var/log/mysqld.log
general_log=0
general_log_file=/var/log/mysql/mysql.log

[client]
socket = /var/lib/mysql/mysql.sock
 
Do you have any swap ?

Code: 
innodb_buffer_pool_size = 24G
This is bad config, that's why it's randomly cause the problem.

For the safety value must calculate by Max memory divide by 8, 64 / 8 = 8G
Or 10% of maximum memory.

And you must tuning "innodb_log_file_size" when modify the pool size.

Since the information still not clear which process cause the problem, just create support ticket to cloudlinux.
 
We checked both the MariaDB and system logs and found multiple errors related to DB table corruption, which could explain the reason for the MariaDB service crashing. Also, we found entries related to a segmentation fault:

Feb 27 16:17:55 s1 kernel: BUG: Bad page map in process mariadbd pte:800000078789b867 pmd:afa36b067<br>
Feb 27 16:17:55 s1 kernel: CPU: 1 PID: 1207258 Comm: mariadbd Kdump: loaded Tainted: G B W O ------- --- 5.14.0-503.21.1.el9_5.x86_64 #1<br>
Feb 27 16:17:56 s1 kernel: mariadbd[1207258]: segfault at 10f0 ip 0000560c695402de sp 00007f16281f0da0 error 4 in mariadbd[560c68dac000+9ea000] likely on CPU 1 (core 0, socket 0)<br>
Feb 27 16:17:57 s1 systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV<br>Feb 27 16:17:57 s1 systemd[1]: mariadb.service: Failed with result 'signal'.<br>
Feb 27 16:18:02 s1 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.<br>Feb 27 16:18:02 s1 systemd[1]: Stopped MariaDB 10.6.20 database server.


The message "BUG: Bad page map in process mariadbd" suggests a problem with the memory management of the MariaDB process. This could be due to a bug in the MariaDB software, an issue with the kernel, or a hardware problem.


Cloudlinux Replied
 
kbtechno said:
We checked both the MariaDB and system logs and found multiple errors related to DB table corruption, which could explain the reason for the MariaDB service crashing. Also, we found entries related to a segmentation fault:
Click to expand...


try and run MySQL with the lower innodb_buffer_pool_size value as already suggested then
 
Hi,

Yes I have changed tinnodb_buffer_pool_size value to 16gb.
Also we run complete hardware test which is absolutely fine.

and now i am monitoring the sever and will update you here
 
You must log in or register to reply here.
Back
Top