Is 404.shtml some default error document?

[Anne]

Hi All,

For weeks I'm searching for a strange error in my error logs that is only popping up now and then. The error is that 404.shtml cannot be found. This is logic, because I don't have a 404.shtml file at all. The only problem is, I can't find who is telling to use this?

I did not have a custom error document in my .htaccess for example. Also I did not have this in my website code. To be sure I scanned every file, plugin or extension of this website.

I remember from long ago that CPanel for example had this as the default error document, but I can't remember or find anything about DirectAdmin has this configured anywhere?

It happens a few times a day, it can be triggered from loading a page, but also from an image or a .css file. All of them exist of course, so should not trigger a 404.

Any tips where I could search more to find this?

 

[Daniel_Dog]

404.shtml is the default 404 file for DirectAdmin websites.
DirectAdmin does not have a default 404.shtml in its domain web page template.
And thus giving the error/warning you described above. (It will always fall back to the error page that is configured by your web service in cases it can not find the default.)

 

[Richard G]

for a strange error in my error logs

Exactly which error logs are you referring to?

DirectAdmin does not have a default 404.shtml in its domain web page template.

In which web page template do you mean?
Because DA has it in templates:

/usr/local/directadmin/data/templates/default/404.shtml

Also it creates automatically a 404.shtml document in every public_html on account creation.

 

[Anne]

The 404.shtml was indeed there when the account was created. I remove all those files and install my own website.

The website has a custom error handling log system. So it also log 404 errors and the not found 404.shtml file is one of it. This is not such a big deal, but I would like to find how and why the 404.shtml was triggered?

For example, I see in my log that is was triggered while loading stylesheet.css. At that moment, the server was not overloaded or something, so why could it not load the css file and try to serve a 404.shtml instead?
That was the reason I went searching where the 404.shtml was configurated in the hope to find the cause.

 

[Richard G]

I can't help you with that. If you have working custom error logs created for your website, then the 404.shtml must be triggered by the webserver itself, before your custom error log can be triggered.

So it might be that it is due to some custom change, something different in the virtual host or something else. I have no clue.
Maybe if you temporarily restore the 404.shtml that it makes a nicer better log to what is triggering it.

The only thing I can think of is to review exact paths in the logfiles and verify what all lines say at the exact same time in all logfiles, so compare the apache access_log and error_log and also the domains log and error log.
Maybe if you compare the lines of all these logs that you can get a clue on to why your custom error log isn't triggered.

Maybe this thread might also be useful:

Custom error pages not working

I have made some custom error pages that I want new users to have instead instead of the default ones. These are located under the default folder of the reseller. The files in there are copied nicely to the new user's domain folder. Now, when I type a wrong link I get the correct custom...

forum.directadmin.com

 

[Anne]

Thank you Richard for your response and ideas.

I think:

Maybe if you temporarily restore the 404.shtml that it makes a nicer better log to what is triggering it.

Is an excellent idea! How I did not came up to that. Perfect way to check what is really going on there. I will do this.

If it's worth sharing for the public I will post it, else this topic might just sink down for good.

 

[Richard G]

It's certainly worth sharing for the public so it's very nice if you would post the solution if you find it.

 

[Anne]

Hi,

It took me forever, but I narrow it down.

I found a way to reproduce it. When I go to www.mywebsite.com/an-image.jpg that works fine, when I add a slash at the end www.mywebsite.com/an-image.jpg/ it forwards to www.mywebsite.com/404.shtml/

When I remove the following lines in my .htaccess, it will not forward to the (not existing) .shtml file and just show the normal 404 error page that is handled by the website.

# Force a trailing slash.
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*[^/])$ https://%{HTTP_HOST}/$1/ [L,R=301]

It does not make sense to me.

 

[Richard G]

when I add a slash at the end www.mywebsite.com/an-image.jpg/ it forwards to www.mywebsite.com/404.shtml/

Which is logical, because if you add a slash at the end, you're in fact pointing to a directory which does not exists thus generating the 404.

Very good you found where it came from and share the solution you found!

So you removed the lines in .htaccess, I'm not sure but my guess is that it also might work when only removing the trainling slash from the second line which seems not to belong there normally.

However I don't know where those lines came from, Mostly these kind of lines look like this:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

 

[ikkeben]

Probably while 2 diferent one with 301 to https and other without https?

"https://"

 

[Richard G]

No that shouln't matter, I also use https in such line, works fine.
If not https was present you would get a different error notice.

 

[Anne]

Hi,

But still I don't really get 'who/what' is telling to use the 404.shtml file to be honest. All other errors including 404's are normally handled by the website/application.

About the lines in the htaccess, I've put them there myself . I want the url to always end with a trailing slash.

No the http or https does not seems to matter indeed.

 

[ikkeben]

No that shouln't matter, I also use https in such line, works fine.
If not https was present you would get a different error notice.

No i mean the topic starter removed those 301 out of htaccess, so the normal call without https is then another then the rewrite 301 to https with that htaccess.

So if no forced ssl and no redirect in panel itself then two diferent locations and files for those. ( under private and under public)

404 is a default apache and more in all the worldwideweb, you can change the content of that file and do some own redirects i guess.

U images don't end with / those are for "directory's not filenames

HTTP 404 - Wikipedia

en.wikipedia.org

To have it complete:

How do I create a custom 404 error page using HTML and CSS?

Answer: The 404 page is a staple of the internet. Technically, a 404 (Not Found) page is an HTML Status Code. The HTTP 404, 404 Not Found, or 404 Page Not Found error message is a Hypertext Transfer Protocol (HTTP) standard response code, in computer network communications, to indicate that the ...

www.quora.com

 

[Anne]

Ok, I think I figure it out.
Let's break it down.

My htaccess is basically this:

# Checking domain, https, etc etc......
# ............
# ............

# Force a trailing slash.
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*[^/])$ https://%{HTTP_HOST}/$1/ [L,R=301]

# Rewrite conditions.
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

RewriteRule . index.php

So everything that is not a file or a directory will end up in index.php. So if anything goes wrong, the website/application will also server the 404 page.

But if I got a file and I add a trailing slash (www.mywebsite.com/existing-file.js/), the mod rewrite will STILL see it as a valid file, so this condition will return false:

RewriteCond %{REQUEST_FILENAME} !-f

And because it returns true, it will not end up with index.php. And because of the rewrite lines before that, also the condition returns true, it will even try to add a trailing slash that of course also returns in a wrong url. So my assumption that it was because of these lines was not correct.

And in the end, when everything fails, I guess the 404.shtml page is served. This must be somewhere be configured in the apache config I guess.

In my option, the line "RewriteCond %{REQUEST_FILENAME} !-f" should return true if the file url has a trailing slash. Not sure if this can be called a bug, maybe it's expected behavior.

So, there you have it

BTW, I fixed it with a rewrite that looks for files that are valid but contains a trailing slash. I rewrite them to a url without that slash:

RewriteCond %{REQUEST_FILENAME} -f
RewriteCond %{REQUEST_URI} !\/+[^\.]+$
RewriteRule ^(.+)/$ /$1 [R=301,L]

Good solution do you think?

 

[Richard G]

Sorry. I'm not native English, but I still don't understand why you are forcing a trailing slash in the .htaccess file.

Anyway, have a look here for a better example.

.htaccess Rewrite to Force Trailing Slash at the end

I have the following code in my htaccess file: # Force Trailing Slash RewriteCond %{REQUEST_FILENAME} !-f RewriteRule ^[^/]+$ %{REQUEST_URI}/ [L,R=301] That seems to work fine when I go to www.my...

stackoverflow.com

Like this one:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)([^/])$        /$1$2/ [L,R=301]

they are using a second variable $2.

I don't know if your solution is good or theirs is better, I don't do these kind of redirects because I don't know what benefit it would have for my sites.

 

[ikkeben]

Partly seo. i guess

If you can reach site under with slash and without , they say double content, but i thinks search engines know now, you can handle it in their tools

and canonical maybey some ...

It is ok to use for having one version only if done in right way but hmm https://websiteseochecker.com/blog/trailing-slash-seo-why-it-matters/

 

[Anne]

Hi both,

Indeed old seo habbits. I also filter double slashes and all kind of stuff. It use to be a problem for search engines, but they have become much smarter nowadays. But I do like very strict URL's and try to avoid any unwanted variations.

So far the problem is gone now. No side effects seen. It's one of these very small problems, but always better to fix in the end for peace of mind

BTW Richard, your rule does work too. I'm also not an expert in these. I can't tell if there are big differences between them. I think mine has the benefit that it always rewrite to https, I guess that's why I use this, but you know, years ago. Most of the time there are like 10 different options to write these rules.

What I learned from this is that this line (RewriteCond %{REQUEST_FILENAME} -f) does find a valid file, because it checks on filename only. Unless that the url is not valid because there is a slash at the end.