Super slow (30 seconds) load for PHP 5.6 / DirectAdmin / NetData / Litespeed / Ubuntu

ozzWANTED

Verified User
Joined
Mar 10, 2015
Messages
45
Hi,
We have a website in our hosting. That became super slow. 30 seconds of PHP waiting for something.
It becomes great for guest sometimes. And for some logged-in users it is terrible.

We have (for given website in server):
1. NetData
2. PHP 5.6+Litespeed Enterprise (4 workers)
3. Ubuntu Server 16.04 LTS
4. MariaDB (separate VPS instance).
5. DirectAdmin
6. VPS is fast NVMe.

I was able to get some notes on sync-flood yesterday, and got some thought that maybe US-based Statcounter that API calls exist in core with probably no timeout in PHP can be causing slow down.

But now I have issues with my logged in user on website.

On localhost we probably would use xDebug to look for that function.

But what to do on live site. Is there a way to have some extension for DirectAdmin or NetData to show me what is a slowdown (which PHP function), like SolarWinds AppOptics. But as SolarWinds were hacked themselves, I'm not sure if I want to go with their solution, could I found/buy something self-hosted.

The below is what AppOptics (SolarWinds) replied:
---
AppOptics APM is the right tool to identify where the slowness is occurring. I would recommend enabling Code Profiling on APM as well to help isolate the problem. The NetData agent may cause some problems if run alongside our APM Agent so be aware of possible resource conflicts between these tools.

You can sign up for a free trial account at my.appoptics.com

Using APM:

https://documentation.solarwinds.co...ppoptics/content/kb/apm_tracing/using_apm.htm

PHP Agent System requirements:

https://documentation.solarwinds.co...content/kb/apm_tracing/php/support-matrix.htm
---
 

Attachments

  • WebDnD is very slow (Dec 18 12-45 AM).jpg
    WebDnD is very slow (Dec 18 12-45 AM).jpg
    569.1 KB · Views: 306
@Arieh - is there is any way that this can be achieved via some custom NetData extension (even if bought). Or DirectAdmin plugin.
And AppOptics says that there can be issues with NetData if both used. Can I find a MINIMAL TOOL just for APM?
 
Not sure, I have looked at the netdata website and don't see an apm that can do this in that list. It seems this APM for PHP is quite a new thing. Datadog may be possible to install on all servers.
 
Are other sites on that same server fast enough at those slow times for that website?

IF so

PHP FPM?
Look for settings for that user then.

Also the how many connections / sessions same time are alowed, and if error /warning somewhere "the kind wait times ( retry's )script runs over the error and goes on. "

Sometimes scripts needs external things and then wait times, retry and so on.

A limit somewhere set for that user, or on the side of the external things needed.


If only for that website / scripts, try test on other server and look in log files for errors /warning then , if you are not responsable for those things, they should do that.

And some more of such things loaded external:
"


 
Last edited:
Not sure, I have looked at the netdata website and don't see an apm that can do this in that list. It seems this APM for PHP is quite a new thing. Datadog may be possible to install on all servers.
While in here it says it is not that new thing:

https://docs.moodle.org/dev/Profiling_PHP

PHP has two types of profiler:
  • XHProf is a standard designed for capturing profile traces on live servers. It was originally designed by engineers at Facebook and is suitable for production environments.
  • XDebug is well known. and understood, and its trace output is supported by tools like KCachegrind. It is not recommended for production environments.
Tideways (replacement for XHProf):
Tideways is a PHP7-compatible replacement for XHProf, and support for this is built into Moodle. (Tideways is a large, paid-for service, which helps to identify a range of issues with production servers. Whilst the service itself is a paid-for service, the instrumentation tooling is both Open Source, and free.)
When the XHProf PHP extension is installed, the "Profiling" section is displayed in "Site administration / Development". And when the profiling setting (profilingenabled) is enabled, a new link "Profiling runs" is also displayed in the Development tab. From this section, apart from access to the profiling runs, you can also import .mpr (Moodle profiling) files.

Actively maintained (by tideways, requires PHP7):

Actively maintained (by PHP.net, now requires PHP7 as well):
https://pecl.php.net/package/xhprof (continously updated)
https://github.com/phacility/xhprof (appears these guys are forgot to update)


Tideways - Self-hosted XHProf (free & opensource)​

We maintain a fork of the original XHProf extension that provides PHP 7 support and a completely rewritten code-base for less overhead.

You can install tideways_xhprof PHP Extension by compiling it from its open-source codes. Head over to the Github repository and download the source from there. You can install the extension using these steps:

Bash:
git clone "https://github.com/tideways/php-xhprof-extension.git"
cd php-xhprof-extension
phpize
./configure
make
sudo make install

Load the extension in your php.ini with
Apache config:
extension=tideways_xhprof.so

As for integration of Tideways into your application you can use the low-level API to enable and disable the Profiler, much like XHProf worked:

PHP:
<?php
tideways_xhprof_enable();

// your application code

$data = tideways_xhprof_disable();
file_put_contents(
    sys_get_temp_dir() . "/" . uniqid() . ".yourapp.xhprof",
    serialize($data)
);

This stores the trace in your temporary directory which the default UI uses to look for data. Install the xhprof_lib and xhprof_html directories from this repository into your webfolder and navigate to xhprof_html/index.php to see a list of trace.

If you want to use some improved open-source UIs take a look at the following projects:
---

XHGui​

A graphical interface for XHProf profiling data that can store the results in MongoDB or PDO database.

Application is profiled and the profiling data is transferred to XHGui, which takes that information, saves it in MongoDB (or PDO database), and provides a convenient GUI for working with it.

XHGUI profile.png

System Requirements​

XHGui has the following requirements:

  • PHP version 7.0 up to 7.3
  • If using MongoDB storage, see MongoDB requirements
  • If using PDO storage, see PDO requirements
  • To profile an application, one of the profiling PHP extensions is required. See Profiling a Web Request or CLI script. The extension is not needed to run XHGui itself.
-----

So all this sounds cool,but questions are here these:
  1. How in reality it will integrate with PHP7 + Litespeed + DirectAdmin + MariaDB.
  2. How can we run this on same production (live) VPS.
  3. Will the Litespeed won't be broken because of that, will the site speed will not be impacted (at least not less than +30%)?
  4. Will we actually be able to detect exact slowness of one of 100 websites in server we have. Meaning is the is a way to find botleneck PHP function with this.
  5. We have PHP 5.6 / PHP 7.4 / PHP 8+JIT on our server.

I actually even created a ticket on XHGui asking is the is a way we can sponsor this as DirectAdmin community, meaning I've self would probably agree to put 1500 USD on this, and I believe at least 5-10 other DirectAdmin users also could do the same.

As of now, we cannot compare with these hi-end and super expensive cloud enviroments like AWS or Azure when it has all this integrated.
And having a PROFESSIONAL PHP PROFILING WITH GRAPHICAL UI that would give data PER WEBSITE, that would be awesome.

>> @ikkben:
>> Are other sites on that same server fast enough at those slow times for that website?
Yes, they are. And often the issue is with specific users on exact site, i.e. those that has more data and longer story with their profile.
 
Last edited:
@ozzWANTED

If
>> @ikkben:
>> Are other sites on that same server fast enough at those slow times for that website?
Yes, they are. And often the issue is with specific users on exact site, i.e. those that has more data and longer story with their profile.

Then somewhat clear is the using scripts and settings there (Forum/blog whatever) so is it your job to get that clear?

If server has enough resources, but they are at some limit of their use for...

As written look at the data then for those users only and the data / scripts and php-pfm and more settings for those users and site.
If they are using other external parts for example connections to other servers to update change their data ..

If only bigger profiles has problem look in the forum / support for that (forum/blog / software they are using for that, maybe it is possible to have it separated only when changes , and if not cache, but probably some limits in connections or other limits and such slowing down.


IF PHP Fusion is used here for slow TTFB ?

How can I reduce the Waiting (TTFB) time in PHP-Fusion 9.xx​


and many many more

Also make a copy of that all if you have to solve it, and then test on other server with the same credentials / settings as those slow user has, you can then figure out without disturbing live ...

The last i write above makes more sense if you already know it is that site and those users, then don't go for testing on live box anymore, but first a copy test setup, is better to start with . There you can do a lot without disturbing other server users and sites!

Also it makes total no sense to load on "LT" based sites and server scripts out of USA as i did point out above, for performance.

Ok i gues you have to support that haha ;)
"https://webdnd.com/ozzwanted,u1"
 
Last edited:
Look at versions of that PHP Fusion and support for the php versions, could be some outdated things that are not working anymore as expected, also if you have updated mariadb , and after that slow take a look at some there could be warnings and and.

Mariadb version in how strict mode for example. while this is on separate VPS...
 
Look at versions of that PHP Fusion and support for the php versions, could be some outdated things that are not working anymore as expected, also if you have updated mariadb , and after that slow take a look at some there could be warnings and and.

Mariadb version in how strict mode for example. while this is on separate VPS...
Thats is custom made version, as it has lot of customizations done.

But at this moment it is the fact that I cannot even LOAD A PAGE at all with my user. I mean i get a timeout. So something is wrong in some function / broken.

I'm considering to install Blackfire.io - that looks easiest to run as a SaaS and it has Chrome extension.
But I'm not sure how this would work with DirectAdmin - will it work, and will Chrome extension will help me to debug that even if I cannot load the page at all with my user:

Also will it be easy to uninstall and will it slow down server.
 
And I do have no idea how you thing we can run Statcounter and Google scripts from LT. I mean yes, we can download these JS files, but then after if Google or Statcounter will change that - how we can update that without getting forgotten to do that.
Also there is a backend PHP call to statcounter for visualizing data, and that is always giving +2 seconds on load time. Yes, we use data caching and call it every 30 minutes, but still even that 1 API call for 1 external resource is not Google, and someone every 30 mins is not happy.

But still, the main problem now that I want to X-RAY the server (current website) with my user logged in to know what is wrong (where it stucks).
And I want to be able to do this for other websites as well.

As this is related to my user (and probably other users), it is either has some code issue in data parsing, or in Database Query/Result.
 
Did you looked at the mariadb vps and updates and so

Take a look in history when it worked and when not, only guessings from me but mariadb updates for example could break some older in php 5.6.4x things that worked with some older mariadb versions , and now not as wanted or should

( the external scripts not causing this could be, but for testing purposes you could timely switch them of.)
 
@ikkeben , I mean we have the idea how to debug, but the problem is how to x-ray securely and fast the production server's website one-by-one.
As well as there is question about XHProf (either native, or tideways) +XHGui integration for real-time monitoring that would add no extra cost to server.

And these questions are here as well:
  1. How in reality it will integrate with PHP7 + Litespeed + DirectAdmin + MariaDB.
  2. How can we run this on same production (live) VPS.
  3. Will the Litespeed won't be broken because of that, will the site speed will not be impacted (at least not less than +30%)?
  4. Will we actually be able to detect exact slowness of one of 100 websites in server we have. Meaning is the is a way to find botleneck PHP function with this.
  5. We have PHP 5.6 / PHP 7.4 / PHP 8+JIT on our server. Will it be able to work with all those versions.
 
@ozzWANTED

Iam not clear enough, you write in start post became superslow.

So the first thing to look for is not debug, but what is changed between not slow and superslow period .

Or was it already for longer time slow?

Then you have for that site php 5.6.4 and external Mariadb vps connection, that combi if mariadb is updated could be ...

But ok maybe i don't understand your way and order to do things. ;)

Also you don't test live production updates to main new version (as php 5.6.4 to 7x) normally in my opinion, if it is your site and you choose to go for ok. ;)

The way for only one site bottleneck is normally a test with copy of all that one site , and or try some minor default stuff as firewall settings and and, but bigger things tryouts could break also other sites maybe..

You already have the bottleneck namely that 1 site so why not get it appart, to have no risk with your tests and tryouts for the other sites and users?

I think almost no one could say upfront with the info's from now, what work and how in your "custom made" and behaviours.
For the debugers monitors (XHProf (either native, or tideways) +XHGui integration) themself you have to ask them i guess.
 
Last edited:
If you're really worried about harming production environment, you could also consider making a full copy/backup of it and putting it live on a new testing environment that has the APM tools you need. If you would be worried it may send e-mails to clients for example, make sure to close it off with firewalls or disable mail etc. If it's in the combination of the dataset + software, you should be able to reproduce it anywhere. If not, you will know it's in the environment.
 
Back
Top