Apache crash during DA tally

stefantriep

Verified User
Joined
Dec 15, 2012
Messages
70
Location
Haarlem, The Netherlands
Since 1 week Apache crashes during DA Tally at 0.10 AM and all websites are offline (HTTP 500). I've got it almost everyday now.

Yesterday got this error in apache error log:
*** glibc detected *** /usr/sbin/httpd: free(): invalid pointer: 0x000000000488fa00 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75e66)[0x7f0b1b42ae66]
/etc/httpd/lib/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f0b1bdc127d]
/usr/sbin/httpd[0x4bf88e]
/usr/sbin/httpd[0x4bff4b]
/lib64/libpthread.so.0(+0xf710)[0x7f0b1b758710]
/lib64/libc.so.6(+0x75f76)[0x7f0b1b42af76]
/lib64/libc.so.6(+0x78948)[0x7f0b1b42d948]
/etc/httpd/lib/libapr-1.so.0(apr_allocator_destroy+0x1d)[0x7f0b1bdc127d]
/usr/sbin/httpd[0x4bf88e]
/usr/sbin/httpd[0x4bfca9]
/usr/sbin/httpd[0x4bfef4]
/usr/sbin/httpd(ap_mpm_run+0xbf7)[0x4c0b47]
/usr/sbin/httpd(main+0x9e4)[0x4302e4]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f0b1b3d3d5d]
/usr/sbin/httpd[0x42f479]
======= Memory map: ========
00400000-00503000 r-xp 00000000 08:03 927456 /usr/sbin/httpd
00702000-00709000 rw-p 00102000 08:03 927456 /usr/sbin/httpd
00709000-0070f000 rw-p 00000000 00:00 0
0266e000-0488b000 rw-p 00000000 00:00 0
0488b000-04abc000 rw-p 00000000 00:00 0
7f0b16e53000-7f0b16e5f000 r-xp 00000000 08:02 393368 /lib64/libnss_files-2.12.so
7f0b16e5f000-7f0b1705f000 ---p 0000c000 08:02 393368 /lib64/libnss_files-2.12.so
7f0b1705f000-7f0b17060000 r--p 0000c000 08:02 393368 /lib64/libnss_files-2.12.so
7f0b17060000-7f0b17061000 rw-p 0000d000 08:02 393368 /lib64/libnss_files-2.12.so
7f0b17061000-7f0b17093000 r-xp 00000000 08:02 393317 /lib64/libidn.so.11.6.1
7f0b17093000-7f0b17292000 ---p 00032000 08:02 393317 /lib64/libidn.so.11.6.1
7f0b17292000-7f0b17293000 rw-p 00031000 08:02 393317 /lib64/libidn.so.11.6.1
7f0b17293000-7f0b172a9000 r-xp 00000000 08:02 393261 /lib64/libgcc_s-4.4.7-20120601.so.1
7f0b172a9000-7f0b174a8000 ---p 00016000 08:02 393261 /lib64/libgcc_s-4.4.7-20120601.so.1
7f0b174a8000-7f0b174a9000 rw-p 00015000 08:02 393261 /lib64/libgcc_s-4.4.7-20120601.so.1
7f0b174a9000-7f0b17591000 r-xp 00000000 08:03 529207 /usr/lib64/libstdc++.so.6.0.13
7f0b17591000-7f0b17791000 ---p 000e8000 08:03 529207 /usr/lib64/libstdc++.so.6.0.13
7f0b17791000-7f0b17798000 r--p 000e8000 08:03 529207 /usr/lib64/libstdc++.so.6.0.13
7f0b17798000-7f0b1779a000 rw-p 000ef000 08:03 529207 /usr/lib64/libstdc++.so.6.0.13
7f0b1779a000-7f0b177af000 rw-p 00000000 00:00 0
7f0b177af000-7f0b178f5000 r-xp 00000000 08:03 804156 /usr/local/lib/libxml2.so.2.7.6
7f0b178f5000-7f0b17af4000 ---p 00146000 08:03 804156 /usr/local/lib/libxml2.so.2.7.6
7f0b17af4000-7f0b17afe000 rw-p 00145000 08:03 804156 /usr/local/lib/libxml2.so.2.7.6
7f0b17afe000-7f0b17aff000 rw-p 00000000 00:00 0
7f0b17aff000-7f0b17b3b000 r-xp 00000000 08:03 804334 /usr/local/lib/libxslt.so.1.1.28
7f0b17b3b000-7f0b17d3a000 ---p 0003c000 08:03 804334 /usr/local/lib/libxslt.so.1.1.28
7f0b17d3a000-7f0b17d3c000 rw-p 0003b000 08:03 804334 /usr/local/lib/libxslt.so.1.1.28
7f0b17d3c000-7f0b17d61000 r-xp 00000000 08:03 794286 /usr/local/lib/libpng12.so.0.50.0
7f0b17d61000-7f0b17f60000 ---p 00025000 08:03 794286 /usr/local/lib/libpng12.so.0.50.0
7f0b17f60000-7f0b17f61000 rw-p 00024000 08:03 794286 /usr/local/lib/libpng12.so.0.50.0
7f0b17f61000-7f0b17ff1000 r-xp 00000000 08:03 793560 /usr/local/lib/libfreetype.so.6.11.2
7f0b17ff1000-7f0b181f0000 ---p 00090000 08:03 793560 /usr/local/lib/libfreetype.so.6.11.2
7f0b181f0000-7f0b181f6000 rw-p 0008f000 08:03 793560 /usr/local/lib/libfreetype.so.6.11.2
7f0b181f6000-7f0b18254000 r-xp 00000000 08:03 802094 /usr/local/lib/libcurl.so.4.3.0
7f0b18254000-7f0b18453000 ---p 0005e000 08:03 802094 /usr/local/lib/libcurl.so.4.3.0
7f0b18453000-7f0b18456000 rw-p 0005d000 08:03 802094 /usr/local/lib/libcurl.so.4.3.0
7f0b18456000-7f0b1846c000 r-xp 00000000 08:02 393341 /lib64/libnsl-2.12.so
7f0b1846c000-7f0b1866b000 ---p 00016000 08:02 393341 /lib64/libnsl-2.12.so
7f0b1866b000-7f0b1866c000 r--p 00015000 08:02 393341 /lib64/libnsl-2.12.so
7f0b1866c000-7f0b1866d000 rw-p 00016000 08:02 393341 /lib64/libnsl-2.12.so
7f0b1866d000-7f0b1866f000 rw-p 00000000 00:00 0
7f0b1866f000-7f0b18699000 r-xp 00000000 08:03 794739 /usr/local/lib/libmcrypt.so.4.4.8
7f0b18699000-7f0b18898000 ---p 0002a000 08:03 794739 /usr/local/lib/libmcrypt.so.4.4.8
7f0b18898000-7f0b1889c000 rw-p 00029000 08:03 794739 /usr/local/lib/libmcrypt.so.4.4.8
7f0b1889c000-7f0b188a1000 rw-p 00000000 00:00 0
7f0b188a1000-7f0b18b99000 r-xp 00000000 08:03 529323 /usr/lib64/libmysqlclient.so.18.1.0
7f0b18b99000-7f0b18d98000 ---p 002f8000 08:03 529323 /usr/lib64/libmysqlclient.so.18.1.0
7f0b18d98000-7f0b18e6b000 rw-p 002f7000 08:03 529323 /usr/lib64/libmysqlclient.so.18.1.0
7f0b18e6b000-7f0b18e70000 rw-p 00000000 00:00 0
7f0b18e70000-7f0b18e82000 r-xp 00000000 08:03 804363 /usr/local/lib/libexslt.so.0.8.17
7f0b18e82000-7f0b19082000 ---p 00012000 08:03 804363 /usr/local/lib/libexslt.so.0.8.17
7f0b19082000-7f0b19083000 rw-p 00012000 08:03 804363 /usr/local/lib/libexslt.so.0.8.17
7f0b19083000-7f0b19ab1000 r-xp 00000000 08:03 396093 /usr/lib/apache/libphp5.so
7f0b19ab1000-7f0b19cb1000 ---p 00a2e000 08:03 396093 /usr/lib/apache/libphp5.so
7f0b19cb1000-7f0b19d75000 rw-p 00a2e000 08:03 396093 /usr/lib/apache/libphp5.so
7f0b19d75000-7f0b19d97000 rw-p 00000000 00:00 0
7f0b19d97000-7f0b19db4000 r-xp 00000000 08:02 393426 /lib64/libselinux.so.1
7f0b19db4000-7f0b19fb3000 ---p 0001d000 08:02 393426 /lib64/libselinux.so.1
7f0b19fb3000-7f0b19fb4000 r--p 0001c000 08:02 393426 /lib64/libselinux.so.1
7f0b19fb4000-7f0b19fb5000 rw-p 0001d000 08:02 393426 /lib64/libselinux.so.1
7f0b19fb5000-7f0b19fb6000 rw-p 00000000 00:00 0
7f0b19fb6000-7f0b19fcc000 r-xp 00000000 08:02 393394 /lib64/libresolv-2.12.so
7f0b19fcc000-7f0b1a1cc000 ---p 00016000 08:02 393394 /lib64/libresolv-2.12.so
7f0b1a1cc000-7f0b1a1cd000 r--p 00016000 08:02 393394 /lib64/libresolv-2.12.so
7f0b1a1cd000-7f0b1a1ce000 rw-p 00017000 08:02 393394 /lib64/libresolv-2.12.so
7f0b1a1ce000-7f0b1a1d0000 rw-p 00000000 00:00 0
7f0b1a1d0000-7f0b1a1d2000 r-xp 00000000 08:02 393415 /lib64/libkeyutils.so.1.3
7f0b1a1d2000-7f0b1a3d1000 ---p 00002000 08:02 393415 /lib64/libkeyutils.so.1.3
7f0b1a3d1000-7f0b1a3d2000 r--p 00001000 08:02 393415 /lib64/libkeyutils.so.1.3
7f0b1a3d2000-7f0b1a3d3000 rw-p 00002000 08:02 393415 /lib64/libkeyutils.so.1.3
7f0b1a3d3000-7f0b1a3dd000 r-xp 00000000 08:02 393300 /lib64/libkrb5support.so.0.1
7f0b1a3dd000-7f0b1a5dc000 ---p 0000a000 08:02 393300 /lib64/libkrb5support.so.0.1
7f0b1a5dc000-7f0b1a5dd000 r--p 00009000 08:02 393300 /lib64/libkrb5support.so.0.1
7f0b1a5dd000-7f0b1a5de000 rw-p 0000a000 08:02 393300 /lib64/libkrb5support.so.0.1
7f0b1a5de000-7f0b1a651000 r-xp 00000000 08:02 393223 /lib64/libfreebl3.so
7f0b1a651000-7f0b1a850000 ---p 00073000 08:02 393223 /lib64/libfreebl3.so
7f0b1a850000-7f0b1a852000 r--p 00072000 08:02 393223 /lib64/libfreebl3.so
7f0b1a852000-7f0b1a853000 rw-p 00074000 08:02 393223 /lib64/libfreebl3.so
7f0b1a853000-7f0b1a857000 rw-p 00000000 00:00 0
7f0b1a857000-7f0b1a859000 r-xp 00000000 08:02 393321 /lib64/libdl-2.12.so
7f0b1a859000-7f0b1aa59000 ---p 00002000 08:02 393321 /lib64/libdl-2.12.so
7f0b1aa59000-7f0b1aa5a000 r--p 00002000 08:02 393321 /lib64/libdl-2.12.so
7f0b1aa5a000-7f0b1aa5b000 rw-p 00003000 08:02 393321 /lib64/libdl-2.12.so
7f0b1aa5b000-7f0b1aa84000 r-xp 00000000 08:02 393293 /lib64/libk5crypto.so.3.1
7f0b1aa84000-7f0b1ac84000 ---p 00029000 08:02 393293 /lib64/libk5crypto.so.3.1
7f0b1ac84000-7f0b1ac85000 r--p 00029000 08:02 393293 /lib64/libk5crypto.so.3.1
7f0b1ac85000-7f0b1ac86000 rw-p 0002a000 08:02 393293 /lib64/libk5crypto.so.3.1
7f0b1ac86000-7f0b1ac87000 rw-p 00000000 00:00 0
7f0b1ac87000-7f0b1ac8a000 r-xp 00000000 08:02 393222 /lib64/libcom_err.so.2.1
7f0b1ac8a000-7f0b1ae89000 ---p 00003000 08:02 393222 /lib64/libcom_err.so.2.1
7f0b1ae89000-7f0b1ae8a000 r--p 00002000 08:02 393222 /lib64/libcom_err.so.2.1
7f0b1ae8a000-7f0b1ae8b000 rw-p 00003000 08:02 393222 /lib64/libcom_err.so.2.1
7f0b1ae8b000-7f0b1af66000 r-xp 00000000 08:02 393295 /lib64/libkrb5.so.3.3
7f0b1af66000-7f0b1b165000 ---p 000db000 08:02 393295 /lib64/libkrb5.so.3.3
7f0b1b165000-7f0b1b16f000 r--p 000da000 08:02 393295 /lib64/libkrb5.so.3.3
7f0b1b16f000-7f0b1b171000 rw-p 000e4000 08:02 393295 /lib64/libkrb5.so.3.3
7f0b1b171000-7f0b1b1b2000 r-xp 00000000 08:02 393257 /lib64/libgssapi_krb5.so.2.2
7f0b1b1b2000-7f0b1b3b2000 ---p 00041000 08:02 393257 /lib64/libgssapi_krb5.so.2.2
7f0b1b3b2000-7f0b1b3b3000 r--p 00041000 08:02 393257 /lib64/libgssapi_krb5.so.2.2
7f0b1b3b3000-7f0b1b3b5000 rw-p 00042000 08:02 393257 /lib64/libgssapi_krb5.so.2.2
7f0b1b3b5000-7f0b1b53f000 r-xp 00000000 08:02 393225 /lib64/libc-2.12.so
7f0b1b53f000-7f0b1b73f000 ---p 0018a000 08:02 393225 /lib64/libc-2.12.so
7f0b1b73f000-7f0b1b743000 r--p 0018a000 08:02 393225 /lib64/libc-2.12.so
7f0b1b743000-7f0b1b744000 rw-p 0018e000 08:02 393225 /lib64/libc-2.12.so
7f0b1b744000-7f0b1b749000 rw-p 00000000 00:00 0
7f0b1b749000-7f0b1b760000 r-xp 00000000 08:02 393249 /lib64/libpthread-2.12.so
7f0b1b760000-7f0b1b960000 ---p 00017000 08:02 393249 /lib64/libpthread-2.12.so
7f0b1b960000-7f0b1b961000 r--p 00017000 08:02 393249 /lib64/libpthread-2.12.so
7f0b1b961000-7f0b1b962000 rw-p 00018000 08:02 393249 /lib64/libpthread-2.12.so
7f0b1b962000-7f0b1b966000 rw-p 00000000 00:00 0
7f0b1b966000-7f0b1b96d000 r-xp 00000000 08:02 393285 /lib64/libcrypt-2.12.so
7f0b1b96d000-7f0b1bb6d000 ---p 00007000 08:02 393285 /lib64/libcrypt-2.12.so
7f0b1bb6d000-7f0b1bb6e000 r--p 00007000 08:02 393285 /lib64/libcrypt-2.12.so
7f0b1bb6e000-7f0b1bb6f000 rw-p 00008000 08:02 393285 /lib64/libcrypt-2.12.so
7f0b1bb6f000-7f0b1bb9d000 rw-p 00000000 00:00 0
7f0b1bb9d000-7f0b1bba4000 r-xp 00000000 08:02 393404 /lib64/librt-2.12.so
7f0b1bba4000-7f0b1bda3000 ---p 00007000 08:02 393404 /lib64/librt-2.12.so
7f0b1bda3000-7f0b1bda4000 r--p 00006000 08:02 393404 /lib64/librt-2.12.so
7f0b1bda4000-7f0b1bda5000 rw-p 00007000 08:02 393404 /lib64/librt-2.12.so
7f0b1bda5000-7f0b1bdd6000 r-xp 00000000 08:02 133630 /etc/httpd/lib/libapr-1.so.0.5.1
7f0b1bdd6000-7f0b1bfd5000 ---p 00031000 08:02 133630 /etc/httpd/lib/libapr-1.so.0.5.1
7f0b1bfd5000-7f0b1bfd7000 rw-p 00030000 08:02 133630 /etc/httpd/lib/libapr-1.so.0.5.1
7f0b1bfd7000-7f0b1c0d2000 r-xp 00000000 08:03 797133 /usr/local/lib/libiconv.so.2.5.1
7f0b1c0d2000-7f0b1c2d1000 ---p 000fb000 08:03 797133 /usr/local/lib/libiconv.so.2.5.1
7f0b1c2d1000-7f0b1c2d3000 rw-p 000fa000 08:03 797133 /usr/local/lib/libiconv.so.2.5.1
7f0b1c2d3000-7f0b1c2f8000 r-xp 00000000 08:02 131903 /etc/httpd/lib/libexpat.so.0.5.0
7f0b1c2f8000-7f0b1c4f7000 ---p 00025000 08:02 131903 /etc/httpd/lib/libexpat.so.0.5.0
7f0b1c4f7000-7f0b1c4fa000 rw-p 00024000 08:02 131903 /etc/httpd/lib/libexpat.so.0.5.0
7f0b1c4fa000-7f0b1c51e000 r-xp 00000000 08:02 131909 /etc/httpd/lib/libaprutil-1.so.0.5.3
7f0b1c51e000-7f0b1c71d000 ---p 00024000 08:02 131909 /etc/httpd/lib/libaprutil-1.so.0.5.3
7f0b1c71d000-7f0b1c71f000 rw-p 00023000 08:02 131909 /etc/httpd/lib/libaprutil-1.so.0.5.3
7f0b1c71f000-7f0b1c75c000 r-xp 00000000 08:03 789116 /usr/local/lib/libpcre.so.0.0.1
7f0b1c75c000-7f0b1c95b000 ---p 0003d000 08:03 789116 /usr/local/lib/libpcre.so.0.0.1
7f0b1c95b000-7f0b1c95c000 rw-p 0003c000 08:03 789116 /usr/local/lib/libpcre.so.0.0.1
7f0b1c95c000-7f0b1c9df000 r-xp 00000000 08:02 393337 /lib64/libm-2.12.so
7f0b1c9df000-7f0b1cbde000 ---p 00083000 08:02 393337 /lib64/libm-2.12.so
7f0b1cbde000-7f0b1cbdf000 r--p 00082000 08:02 393337 /lib64/libm-2.12.so
7f0b1cbdf000-7f0b1cbe0000 rw-p 00083000 08:02 393337 /lib64/libm-2.12.so
7f0b1cbe0000-7f0b1cd98000 r-xp 00000000 08:03 524989 /usr/lib64/libcrypto.so.1.0.1e
7f0b1cd98000-7f0b1cf98000 ---p 001b8000 08:03 524989 /usr/lib64/libcrypto.so.1.0.1e
7f0b1cf98000-7f0b1cfb3000 r--p 001b8000 08:03 524989 /usr/lib64/libcrypto.so.1.0.1e
7f0b1cfb3000-7f0b1cfbf000 rw-p 001d3000 08:03 524989 /usr/lib64/libcrypto.so.1.0.1e
7f0b1cfbf000-7f0b1cfc3000 rw-p 00000000 00:00 0
7f0b1cfc3000-7f0b1d025000 r-xp 00000000 08:03 538812 /usr/lib64/libssl.so.1.0.1e
7f0b1d025000-7f0b1d224000 ---p 00062000 08:03 538812 /usr/lib64/libssl.so.1.0.1e
7f0b1d224000-7f0b1d228000 r--p 00061000 08:03 538812 /usr/lib64/libssl.so.1.0.1e
7f0b1d228000-7f0b1d22f000 rw-p 00065000 08:03 538812 /usr/lib64/libssl.so.1.0.1e
7f0b1d22f000-7f0b1d245000 r-xp 00000000 08:03 789510 /usr/local/lib/libz.so.1.2.3
7f0b1d245000-7f0b1d444000 ---p 00016000 08:03 789510 /usr/local/lib/libz.so.1.2.3
7f0b1d444000-7f0b1d445000 rw-p 00015000 08:03 789510 /usr/local/lib/libz.so.1.2.3
7f0b1d445000-7f0b1d465000 r-xp 00000000 08:02 393340 /lib64/ld-2.12.so
7f0b1d46e000-7f0b1d491000 rw-s 00000000 00:04 5889263 /dev/zero (deleted)
7f0b1d5d3000-7f0b1d651000 rw-s 00000000 00:04 5889259 /dev/zero (deleted)
7f0b1d651000-7f0b1d65d000 rw-p 00000000 00:00 0
7f0b1d662000-7f0b1d663000 rw-p 00000000 00:00 0
7f0b1d663000-7f0b1d664000 rw-p 00000000 00:00 0
7f0b1d664000-7f0b1d665000 r--p 0001f000 08:02 393340 /lib64/ld-2.12.so
7f0b1d665000-7f0b1d666000 rw-p 00020000 08:02 393340 /lib64/ld-2.12.so
7f0b1d666000-7f0b1d667000 rw-p 00000000 00:00 0
7fff1f238000-7fff1f24d000 rw-p 00000000 00:00 0 [stack]
7fff1f3f8000-7fff1f3f9000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

Running DirectAdmin 1.46.3, Got Apache 2.2.29, MySQL 5.6.21 and Php 5.5.19

Anyone got the same problem or can help? Thanks!
 
Last edited:
Are you still having this error?

Do you get any error when you restart apache yourself?

What OS version are you running?

Jeff
 
Unfortunately no ideas from me. Hopefully someone else will have some ideas.

Jeff
 
Hi Jeff,

Running CentOS 6.6 64 bits. Got no errors when restarting apache.
Only at tally this problem occurs, but not always the glibc error.

Stefan

Hello,

Is that thread also yours: http://forum.directadmin.com/showthread.php?t=50492 ?
I see users are different and software versions differ. Anyway one and the same issue reported from two different users now with Apache 2.2.
What if you upgrade to apache 2.4?
 
Back
Top