500 Error in one specific site case

 
Post new topic   Reply to topic    Aprelium Forum Index -> General Questions
View previous topic :: View next topic  
Author Message
Kalbintion
-


Joined: 16 Jul 2019
Posts: 5

PostPosted: Tue Jul 16, 2019 8:41 pm    Post subject: 500 Error in one specific site case Reply with quote

I recently have started development on a website with two others, one of them being the author of the site initially for quite a few years before reaching out to myself and another for aid in continuing development of the site. The site in question is for a game for creating builds for items and equipment found in said game. Game in question, if you're curious is Warframe.

The website works flawlessly using Abyss Web Server X1 v2.12.1 except in one particular case. When a user visits a build page, they are able to export said build so they can go back to it. However, under Abyss Web Server it will error with a 500 internal server code and leave an extra php-cgi.exe running indefinitely. After quite a few hours of attempting to debug, as both other developers reported no problems with the build links running under different environments (PHP Version & Web server), I installed IIS locally and attempted to test it there. It works without an internal server error. I haven't a clue why and none of my debug testing or knowledge can give any hint as to what is going on with Abyss specifically.

I ruled out it being the site code based on:

  • It worked randomly for 10 minutes after messing with the code in non-changing ways (inserting spaces, tabs, brackets, etc) through Abyss
  • Two other developers have no issues with it under different environments (One running IIS & PHP 7.2.7, another Apache & PHP 7.1.3)
  • All of my debugging of the code indicates it should be loading without problems


I ruled out it being PHP based on:

  • Trying out different versions of PHP (5.4.4, 7.1, 7.2, 7.3.5)
  • Matching php.ini with another developers that had these build URLs working


Some additional server details and links below.

Following two links are the base website:
Abyss: http://tfk.zapto.org/ddfd452bae7136ae985229ca66b30d65/
IIS: http://tfk.zapto.org:88/ddfd452bae7136ae985229ca66b30d65/

Following two links would be an exported blank build
500'ing URL (via Abyss): http://tfk.zapto.org/ddfd452bae7136ae985229ca66b30d65/Warframes/Builder/Ash/t_30_0200020020__f-f-f-f-f-f-f-f-f-f-f-f-f-f-f-f-f-f-f-f_0/en/1-0-1/0
Working URL (via IIS): http://tfk.zapto.org:88/ddfd452bae7136ae985229ca66b30d65/Warframes/Builder/Ash/t_30_0200020020__f-f-f-f-f-f-f-f-f-f-f-f-f-f-f-f-f-f-f-f_0/en/1-0-1/0

Following is the re-write rule used by Abyss:
Code:
Type: Global
Virtual Path Regular Expression: ^/ddfd452bae7136ae985229ca66b30d65/(.+)/?$
Case Sensitive: Checked
Conditions:
   REQUEST_FILENAME is not a file
   REQUEST_FILENAME is not a directory
Apply to subrequests too: Checked
If this rule matches: Perform an internal redirection
Redirect To: /ddfd452bae7136ae985229ca66b30d65/index.php?p=$1
Append Query String: Unchecked
Escape Redirection Location: Checked
Next Action: Stop matching
Advanced Parameters: None

Following is the re-write rule used by IIS:
Code:
Requested URL: Matches the Pattern
Using: Regular Expressions
Pattern: ^ddfd452bae7136ae985229ca66b30d65/(.+)/?$
Ignore Case: Unchecked
Conditions: Match All
   {REQUEST_FILENAME} Is Not a File
   {REQUEST_FILENAME} Is Not a Directory
Track capture groups across conditions: Unchecked
Server Variables: None
Action Type: Rewrite
Rewrite URL: ddfd452bae7136ae985229ca66b30d65/index.php?p={R:1}
Append Query String: Unchecked
Log rewritten URL: Unchecked
Stop processing of subsequent rules: Unchecked

Following is the original .htaccess the above was converted from
Code:
RewriteEngine on
RewriteCond %{HTTP_HOST} ^www\.(.+) [NC]
RewriteRule ^(.*) http://%1/$1 [R=301,NE,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)/?$ index.php?p=$1
RewriteCond %{HTTP_REFERER} !^http://(.+\.)?warframe-builder\.com/ [NC]
RewriteCond %{HTTP_REFERER} !^$
RewriteRule .*\.(jpe?g|gif|bmp|png|css|js)$ - [F]


It appears to me that the problem lies within Abyss itself or the configuration thereof as it isn't working there and is on IIS after doing much testing switching PHP versions, code changes, and the last resort was trying another web server where it consequently worked, and any help in understanding what is going on would be appreciated. This is the first major issue I've had at all with Abyss after many years of using it. I do not know what information is going to be needed to help figure this out, but do let me know if something is needed about the configuration.
Back to top View user's profile Send private message
admin
Site Admin


Joined: 03 Mar 2002
Posts: 1295

PostPosted: Fri Jul 19, 2019 7:38 pm    Post subject: Re: 500 Error in one specific site case Reply with quote

Kalbintion,

Thank you for the detailed report.

We tried browsing the URL and clicking "Export". But we got a JS error:

ReferenceError: ShortenURL is not defined[Learn More] builder-warframes.min.js:1:28238
exportBuild http://tfk.zapto.org/ddfd452bae7136ae985229ca66b30d65/web/dist/js/builder-warframes.min.js:1:28238
init/< http://tfk.zapto.org/ddfd452bae7136ae985229ca66b30d65/web/dist/js/builder-warframes.min.js:1:4644
w.event.dispatch http://tfk.zapto.org/ddfd452bae7136ae985229ca66b30d65/web/libs/jquery.min.js:1:41464
w.event.add/y.handle http://tfk.zapto.org/ddfd452bae7136ae985229ca66b30d65/web/libs/jquery.min.js:1:39529


Anyway, our guess is that the PHP backend script may take sometime to execute. Is that the case? If so, error 500 could be related to the timeouts imposed by the FastCGI parameters.

If the script takes more than 30 seconds to execute, special configuration should apply to let both PHP (in FastCGI/CGI mode) and Abyss Web Server knows about that and wait for it longer.

At the PHP level, you should probably increase the timeout parameter is php.ini: the first is by default set to 30 (seconds):

max_execution_time = 30

http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time

If your scripts may read very long POSTed data (for example large uploaded files), you may also need to increase the parameter max_input_time http://www.php.net/manual/en/info.configuration.php#ini.max-input-time (which is also in php.ini.)

After each change to php.ini, do not forget to restart Abyss Web Server to ensure that all PHP instances are reloaded and take into account php.ini changes.

At the Web server level, the parameter controlling timeout is the "I/O timeout" parameter for FastCGI which is set to 30 seconds by default:

http://www.aprelium.com/data/doc/2/abyssws-win-doc-html/hosts-configuration.html#HOSTS-SCRIPTING-FASTCGI-IOTIMEOUT

Increasing this parameter should also make Abyss Web Server wait longer for PHP scripts before bailing out and considering them as "dead" or "unresponsive".

We'll be waiting for your feedback.

If the problem persists, please send us by email (support AT aprelium) your abyss.conf as well as your php.ini files for review.
_________________
Follow @abyssws on Twitter
Subscribe to our newsletter
_________________
Forum Administrator
Aprelium - https://aprelium.com
Back to top View user's profile Send private message
Kalbintion
-


Joined: 16 Jul 2019
Posts: 5

PostPosted: Fri Jul 19, 2019 10:09 pm    Post subject: Reply with quote

My apologies about the JS error, it was actually being worked on when you posted, as the links are live development environments of mine. Bad/Great/Weird timing on that. That error shouldn't be a problem anymore for attempting to Export as the branch I was working on is essentially finalized to be pushed into master. But I digress...

Quote:
Anyway, our guess is that the PHP backend script may take sometime to execute. Is that the case? If so, error 500 could be related to the timeouts imposed by the FastCGI parameters.

That was one of my thoughts, as one of the original lines being sent into php_errors.log as I configured it to output to the file was saying it was hitting the 30 second execution limit. I changed it 1200 seconds, restarted Abyss, and the 500 error was still occurring. However, with the increased execution time, there was no log in the php_errors file about it hitting max execution time and was empty outside of some known PHP notices. You can actually view these logs still via the github gists I made for the other two devs about this issue.

Initial 500 Logs: https://gist.github.com/Kalbintion/8d7becd56777a98fd728e9209c191eee
With debug lines added to every class/method to see how the execution order was happening: https://gist.github.com/Kalbintion/7591e542d288e6ae1611c903526660aa
With debug lines of the base url loading (/Warframes/Builder/Ash) with no attached build data: https://gist.github.com/Kalbintion/d0b9ad9ae5e40bd4bbef61e77fb075fa

Do note that the second gist contains a lot of information about the exact start of a page load to everything that happens inbetween until it finally outputs one final line and the server stops with the 500 code (not shown in log), along with included file outputs. Unfortunately I am unable to easily replicate these logs for any additional testing as I reverted these changes as development must go on, and that I found this issue didn't exist under IIS.

Quote:
At the Web server level, the parameter controlling timeout is the "I/O timeout" parameter for FastCGI which is set to 30 seconds by default:


Was unaware of this setting being there. Double checking it was set to 30 seconds, default, it was indeed. After changing it to 90 seconds (and PHP back to 1200s) and using the url: http://tfk.zapto.org/ddfd452bae7136ae985229ca66b30d65/Warframes/Builder/Ash/t_30_0200020020_565-4-3-574-0-3-611-2-5-653-5-3-709-3-3-992-1-3_574-13-992-9-611-9-709-7-565-5-653-6-f-f-f-f-f-f-f-f_0/en/1-0-1/ - An exported build with a random assortment of mods put onto it - I was unable to load the page, still getting a 500 response code and nothing outputted into php_errors about execution time being hit. That URL works with no issues under IIS (to test, add port 88 to url). Tried once more setting I/O Timeout to 1200s to match the PHP ini setting, and still getting a 500 code. The rest of the site works fine with these settings, it is only this one particular case of it attempting to load an exported build.


Do note: This site has been around for many years before I started touching it at all and this issue existed on Abyss (production site works without problem) on the original files sent by the original website developer that were direct copies from the live website version, it would be very odd and highly unlikely that hitting execution time limitations would of been missed due to the settings in place of the live server that can have hundreds of connections to it at any one time.

Quote:
If the problem persists, please send us by email (support AT aprelium) your abyss.conf as well as your php.ini files for review.

Email has been sent with subject line "Forum Thread: 500 Error in one specific site case"

For php info outside of the file however, it can be viewed at:
- Abyss: http://tfk.zapto.org/php.php
- IIS: http://tfk.zapto.org:88/php.php

One Final Note: I will be out of town from July 20th to July 28th and may not be able to properly test or respond in a timely fashion during that time frame. This thread has not been forgotten and will be checked on as soon as possible.

Note Edit: I was just informed that the location I was going to go to has had a severe storm that knocked out power and downed trees. In the event power is restored by tomorrow I will be leaving still during that time frame, but if not the duration of my departure will be next week, July 28th to August 3rd.


Last edited by Kalbintion on Sat Jul 20, 2019 3:14 pm; edited 1 time in total
Back to top View user's profile Send private message
pkSML
-


Joined: 29 May 2006
Posts: 952
Location: Michigan, USA

PostPosted: Sat Jul 20, 2019 3:44 am    Post subject: Reply with quote

I have a suggestion.

Sometimes I've had PHP issues and 500 errors with nothing logged in the error log.

Go to your scripting parameters in the Abyss console and add the error_log directive.

For my FastCGI (Local - Pipes) interpreter, I have an argument like this:
Code:
-d error_log=/path/php_errors.log


Note: I'm running Linux, so adjust for Windows paths accordingly.

If you'd rather just edit your php.ini file, you can set the error_log directive there instead. Note that this will affect ALL of your hosts that rely on this php.ini configuration whereas specifying the argument in Abyss will only log errors for that host in the file given in the argument.

Also note that you won't see PHP errors anymore in the FastCGI error log. They will now be logged to the file specified in the interpreter argument.
_________________
Stephen
Need a LitlURL?


http://CodeBin.yi.org
Back to top View user's profile Send private message Visit poster's website
Kalbintion
-


Joined: 16 Jul 2019
Posts: 5

PostPosted: Sat Jul 20, 2019 12:04 pm    Post subject: Reply with quote

Thanks for your suggestion pkSML, it actually had me double check on the php.ini file and it appears I didn't reset error_log like i normally have it set. During my version changing and testing, having kept it on PHP 7.3.5 because newer version, I forgot to ensure this directive was changed. I have changed it back to what I preferred and still receive nothing in the log file about hitting max execution time limit when attempting to visit the url that 500's mentioned in the previous post.


Note: To reiterate, I will be out of town from July 20th to July 28th and may not be able to properly test or respond in a timely fashion during that time frame. This thread has not been forgotten and will be checked on as soon as possible.

Note Edit: I was just informed that the location I was going to go to has had a severe storm that knocked out power and downed trees. In the event power is restored by tomorrow I will be leaving still during that time frame, but if not the duration of my departure will be next week, July 28th to August 3rd.
Back to top View user's profile Send private message
pkSML
-


Joined: 29 May 2006
Posts: 952
Location: Michigan, USA

PostPosted: Sun Jul 21, 2019 6:22 pm    Post subject: Reply with quote

So I presume you can confirm your timeouts in PHP and Abyss are sufficiently large in your present setup?

Also note that the script itself may be setting timeouts which will override the php.ini value using ini_set() (though if a timeout is the issue, you'd expect to find something in the log about it).

What error messages (if any) are you getting in the logfile specified by error_log?
_________________
Stephen
Need a LitlURL?


http://CodeBin.yi.org
Back to top View user's profile Send private message Visit poster's website
Kalbintion
-


Joined: 16 Jul 2019
Posts: 5

PostPosted: Sun Jul 21, 2019 6:54 pm    Post subject: Reply with quote

pkSML, The timeouts are indeed sufficiently large, perhaps overly so. I am receiving nothing in the error log when the timeouts are at 1200 seconds (20 minutes). As for possible ini_set() lines, Agent Ransack, a file searching program, reports a few being used. One for turning on/off error reporting which is explicitly turned on (to report errors) for the dev environment and some others through the use of PHPMailer for sendmail_from and magic_quotes_runtime settings. PHPMailer, as far as I'm aware however, is completely unused due to the site being ran in the EU and personal information stripped when the GDPR was enforceable (including email addresses).

Though, I need to ask, if it was absolutely the execution timeout being the problem, why do two other devs under different environments have absolutely no issue with their default timeouts being 30 seconds? And why would I have no issue running the site under IIS locally, on the same machine as Abyss, with default server/PHP settings (excluding URL rewrite mentioned before) and only have this problem with the site running under Abyss? It does not make any sense to me immediately on why a timeout setting would be the problem or aid in diagnosing.
Back to top View user's profile Send private message
pkSML
-


Joined: 29 May 2006
Posts: 952
Location: Michigan, USA

PostPosted: Wed Jul 24, 2019 2:24 am    Post subject: Reply with quote

Kalbintion wrote:
pkSML, The timeouts are indeed sufficiently large, perhaps overly so. I am receiving nothing in the error log when the timeouts are at 1200 seconds (20 minutes). As for possible ini_set() lines, Agent Ransack, a file searching program, reports a few being used. One for turning on/off error reporting which is explicitly turned on (to report errors) for the dev environment and some others through the use of PHPMailer for sendmail_from and magic_quotes_runtime settings. PHPMailer, as far as I'm aware however, is completely unused due to the site being ran in the EU and personal information stripped when the GDPR was enforceable (including email addresses).

Though, I need to ask, if it was absolutely the execution timeout being the problem, why do two other devs under different environments have absolutely no issue with their default timeouts being 30 seconds? And why would I have no issue running the site under IIS locally, on the same machine as Abyss, with default server/PHP settings (excluding URL rewrite mentioned before) and only have this problem with the site running under Abyss? It does not make any sense to me immediately on why a timeout setting would be the problem or aid in diagnosing.


Yah, maybe the timeouts aren't the issue. How long does it take to get the 500 error? You can find out in the Chrome Inspector (F12).

That's really strange if you're getting absolutely nothing in your php_errors.log.

Without any concrete errors, we're just grasping at straws.

Is error_reporting set to 2047 (E_ALL)? That will affect how much gets reported in the logs. (Perhaps a production-ready script turns off error_reporting somewhere???) You can find out what the error_reporting level is by adding this code to the PHP script.

Code:
echo "<h2>Error Reporting: " . error_reporting() . "</h2>";


Is your CPU going crazy during processing?

Perhaps you could post a screenshot of the PHP scripting parameters page in the console for the affected host.
_________________
Stephen
Need a LitlURL?


http://CodeBin.yi.org
Back to top View user's profile Send private message Visit poster's website
Kalbintion
-


Joined: 16 Jul 2019
Posts: 5

PostPosted: Wed Jul 24, 2019 8:37 pm    Post subject: Reply with quote

pkSML wrote:
Yah, maybe the timeouts aren't the issue. How long does it take to get the 500 error? You can find out in the Chrome Inspector (F12).

Directly related to what I have the php timeout set to. Whatever it is set to, it takes that amount of time for it to 500 on me.

pkSML wrote:
That's really strange if you're getting absolutely nothing in your php_errors.log.

I can get errors to show in the log if the timeout is set at 2 minutes or below, and the only error is saying it hit max execution time, going at 20 minutes however, it doesnt send it to the log. May be a weird thing with that long of a timeout.

pkSML wrote:
Without any concrete errors, we're just grasping at straws.

Is error_reporting set to 2047 (E_ALL)? That will affect how much gets reported in the logs. (Perhaps a production-ready script turns off error_reporting somewhere???) You can find out what the error_reporting level is by adding this code to the PHP script.

Error reporting is set to 22527: E_ALL & ~E_DEPRECATED - You can find a link to my phpinfo() output in the second reply post. This is not modified in any of the site files, the only thing the site has for a toggle, that I mentioned before, is turning all error reporting on or off, to which it is on.

pkSML wrote:
Is your CPU going crazy during processing?

It is not actually, its minimal processing, roughly the same amount of CPU is used while just loading one of the pages normally.

pkSML wrote:
Perhaps you could post a screenshot of the PHP scripting parameters page in the console for the affected host.

Base: https://gyazo.com/fec3cf1993767c2f10112626753b1f66
FastCGI: https://gyazo.com/b7d32f6508b9df6e004b68b7294ab70b
CGI And ISAPI are defaulted. They have not been changed.
Interpreter: https://gyazo.com/0b964f692d9e75b3259cb31b8b6fa28f
Adv Interpret: https://gyazo.com/381c65c79feab8d95895c4511b80acbe

(Note: the timeouts on the Abyss side and PHP side have been reset to 30 seconds, more info on why below)



I have been in contact with Aprelium via email. They apparently are not able to replicate the 500 error by connecting to the web server hosted on my machine directly. And after some initial delays for the page being sent, I have reduced the timeout from 20 minutes to 30 seconds so in the event they were to get error 500 it wouldn't have them wait that long. I have had a few other people test it, one of whom lives half-way across the US from me, and they were able to get error 500. I am not sure why Aprelium themselves are able to connect to the server with a blank build url when the two devs I'm working with, myself, and the friend I mentioned are getting 500 errors while connecting to my server hosted locally for the same link. They, Aprelium, have requested a file be sent to them, one I mentioned in the first post, that I had cosmetically changed and had the build urls magically work for 10 minutes before starting to fail again, that I sent along with some additional files from the website. I'm hoping with them having all this info and access they are able to guide me into what to look at in the long run or find the actual problem itself.

I do appreciate you wanting to help, pkSML, along with Aprelium, in this issue.

Note on being gone: I will be gone from July 28th to August 3rd as a few of my other posts mention, they were unable to get power back up until yesterday. Just as before, if this problem isnt resolved before I leave, I may be delayed on responding and/or testing in a timely fashion.
Back to top View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Aprelium Forum Index -> General Questions All times are GMT + 1 Hour
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB phpBB Group