Issue reading from a CGI application

 
Post new topic   Reply to topic    Aprelium Forum Index -> FastCGI/CGI
View previous topic :: View next topic  
Author Message
thinkr
-


Joined: 24 Oct 2013
Posts: 15
Location: California

PostPosted: Thu Oct 24, 2013 6:20 am    Post subject: similar problem Reply with quote

My CGI system (calls an executable, not an interpreter) for uploading pictures via a normal HTML form has worked fine for months using Abyss X2 version 2.7. Then I upgraded to version 2.9 and started getting the above problem, yet for me it chops at exactly 4095 bytes every time. (The exact same CGI program works with Abyss X2 version 2.7 and not version 2.9.)

I did a lot of experimenting with my code (nearly identical to the work-around code posted above by mdoigny). Here is what I discovered so far...

- the content_length environment variable always contains the correct length of what should have been the full content (several parameters plus the picture)

- yet the total bytes returned by the ReadFile function always equals 4095 (7 iterations at 512 bytes plus one iteration at 511 bytes)

- every attempt to get it to perform more iterations failed

- I was previously using long variables instead of dwords, yet switching back to dwords made no difference

Just to verify some assumptions I prefilled with dashes (instead of spaces) the buffer variable and prefilled with periods (instead of spaces) the output variable, this way I could see what went where in the areas not filled by actual content. The dashes never showed anywhere (good), and actual content filled through position 4095, while periods only always filled positions 4096 through to the full length of what should have been the actual content.

Clearly 4096 through to the end had no activity from the ReadFile function on any of my many dozens of tests (as these bytes retained the prefill character).

I find it an interesting coincidence that the new feature to restrict HTTP lines defaults to 4096 characters. So I tried setting different values. Higher values were rejected, and lower ones had no effect. I tried clearing it from abyss.conf - again, no effect.

Both the working (version 2.7) system and the non-working (version 2.9) system run on Windows 7 machines with plenty of memory. I rarely ask for help. I've lost 2.5 days to this so far, can't afford to lose anymore. Any ideas?
_________________
If you don't question it then you don't know it.
Back to top View user's profile Send private message
aprelium-support
-


Joined: 20 Feb 2009
Posts: 356

PostPosted: Sat Oct 26, 2013 9:05 am    Post subject: Re: similar problem Reply with quote

thinkr,

The 4096 limit in Header lines is not related to uploads. It is in effect in the header of the HTTP request not in the body of the request.

Post your code for review. That's easier than having us do guess work about how your CGI implementation is behaving and what might be wrong in it.

Also try testing the same upload with another script (PHP/Perl for example) to verify if the problem is on Abyss Web Server's side or on your side. Doing the tests always with the same "actors" won't help you determine who's buggy. ;)
_________________
Support Team
Aprelium - http://www.aprelium.com
Back to top View user's profile Send private message Visit poster's website
thinkr
-


Joined: 24 Oct 2013
Posts: 15
Location: California

PostPosted: Mon Oct 28, 2013 6:22 am    Post subject: cgi code Reply with quote

This code has worked perfectly for the past 2 years on Abyss X2 version 2.7...

FUNCTION webdataStdIn() AS STRING

DIM hInput AS LONG
DIM iRead AS LONG
DIM iResult AS LONG
DIM sBuffer AS STRING
DIM sOutBuffer AS STRING

dim c as dword
dim j as dword
c = VAL(ENVIRON$("CONTENT_LENGTH"))
sOutBuffer = SPACE$(c): j = 1

hInput = GetStdHandle(%STD_INPUT_HANDLE)
IF hInput THEN
DO
sBuffer = SPACE$(512)
iResult = ReadFile(hInput, BYVAL STRPTR(sBuffer), _
LEN(sBuffer), iRead, BYVAL %NULL)

'- If there was an error, return nothing
IF iResult = 0 THEN
EXIT DO

'- We're done if iRead is 0
ELSEIF iRead = 0 THEN
EXIT DO

'- Otherwise, accumulate the buffer
ELSE

MID$(sOutBuffer, j, iRead) = mid$(sBuffer,1,iRead)
j = j + iRead

END IF

'- Bail if there's nothing left to read.
IF iRead < LEN(sBuffer) THEN
EXIT DO
END IF
LOOP

END IF
FUNCTION = sOutBuffer

END FUNCTION


Yet, the same code does not work right in Abyss X2 version 2.9 (truncates everything at 4095 bytes).
_________________
If you don't question it then you don't know it.
Back to top View user's profile Send private message
aprelium-support
-


Joined: 20 Feb 2009
Posts: 356

PostPosted: Sat Nov 09, 2013 8:59 am    Post subject: Re: cgi code Reply with quote

thinkr,

The code looks fine. But have you made a test with PHP or with another CGI implementation.

The fact that POSTs get truncated to 4095 bytes cannot go unnoticed for so many users that we do not think this would have been reported soon. By the way, this forum is hosted on 2.9 and most of the POSTs you make here are larger than 4096 bytes...

By the way, your code looks like it is in some sort of BASIC. What language and/or what compiler are you using?
_________________
Support Team
Aprelium - http://www.aprelium.com
Back to top View user's profile Send private message Visit poster's website
thinkr
-


Joined: 24 Oct 2013
Posts: 15
Location: California

PostPosted: Mon Nov 11, 2013 10:49 pm    Post subject: Power Basic Reply with quote

Hi, yes, its Power Basic 7, great compiler, been using it for a number of years. And I agree that it is likely not Abyss, but rather some sort of conflict that I don't understand having to do with an interaction between the newer version of Abyss (x2 2.9) and/or the latest version of Windows 7, and/or some OS tweak I did for greater performance. I can't think of anything else that's different between the two servers. The OS tweaks are well documented things like disabling 8.3 filenames, increasing the NTFS area, disabling file indexing, etc.

I intentionally did not upgrade windows 7 on the old server (the one that works fine), just in case something like this would happen, however, the new server is fully upgraded.

I'm sure it is something super simple, probably some configuration in windows. However, all my tests indicate it is lower than the readfile function that I tie into (see above code), making it very hard to track down.

I have never used PHP or any other CGI system, only my own homemade CGI. If you have a link to some steps I could follow to test it then that might help.

I find the number 4095 is 100% consistent, it always truncates exactly there, not 4096. This seems the biggest clue. Googling on 4095 produces a couple entries, both dead ends. However, I did find quite a bit of documentation on a variety of things that have in the past in other systems caused this type of problem. I did my best to track down and understand each one to no avail.

I was hoping you'd tell me that my code isn't looking for some new variable with a response code to process, something necessary to newer versions of windows or abyss. This seems likely to me.

Here's one more difference: On the new server Abyss hosts 25 million files, whereas the old server is hosting about 20 million files. These files are distributed across many folders to boost performance, and works fine on both. (The only thing not working on the new server is the http request -- trancation at 4095 bytes -- and the code for this is exactly the same on both servers.)
_________________
If you don't question it then you don't know it.
Back to top View user's profile Send private message
thinkr
-


Joined: 24 Oct 2013
Posts: 15
Location: California

PostPosted: Thu Nov 14, 2013 7:55 am    Post subject: work-around implemented successfully Reply with quote

The way the code was written presumes that all chunks except the last are 512 bytes long. Once it hits a chunk less than 512 bytes it stops. Every copy of this code that I've seen in other forums works this way, presuming that the chunk size is a constant up to the last chunk, so I figured it was a safe assumption. However, for some reason, my newest Windows 7 machine always sends me 511 bytes for the 8th chunk, and 510 bytes for the 24th chunk. The rest are all 512 bytes as expected. So the way the code was written it would always stop at the 8th chunk (4095 bytes).

I would have caught this earlier but one of my tests had a bug and gave the wrong results (the one that tests past the end to see if there was more data). Today I redid all the tests.

Despite the odd 8th & 24th chunk sizes the data itself is in order, no bytes missing.

So to fix this I changed the test on iRead (to see if it is less than the buffer length) to a test on j (to see if it matches c). Now it works fine.

Any idea why the 8th and 24th chunk sizes are different on the new server?

More importantly, is readfile supposed to process variable chunk sizes, or is this an indication of a security problem of some sort?
_________________
If you don't question it then you don't know it.
Back to top View user's profile Send private message
aprelium-support
-


Joined: 20 Feb 2009
Posts: 356

PostPosted: Sat Nov 16, 2013 10:14 am    Post subject: Re: work-around implemented successfully Reply with quote

thinkr wrote:
More importantly, is readfile supposed to process variable chunk sizes, or is this an indication of a security problem of some sort?


Yes, ReadFile may have such a behavior. Never assume that there is an issue if it gives you an underfull buffer.

Please refer to Microsoft's documentation when dealing with low level API functions such ReadFile. Testing for the EOF is not correct in your code. From http://bit.ly/18CL4d4 :

Quote:
The ReadFile function checks for the end-of-file condition (EOF) differently for synchronous and asynchronous read operations. When a synchronous read operation gets to the end of a file, ReadFile returns TRUE and sets the variable pointed to by the lpNumberOfBytesRead parameter to zero.

_________________
Support Team
Aprelium - http://www.aprelium.com
Back to top View user's profile Send private message Visit poster's website
aprelium-support
-


Joined: 20 Feb 2009
Posts: 356

PostPosted: Sat Nov 16, 2013 10:20 am    Post subject: Re: work-around implemented successfully Reply with quote

According to our developers team, there was little changes in that module (CGI/FastCGI) since 2.7. We have just redone some tests in PHP using simple scripts (in CGI mode) and we were able to "upload" files of several hundred KB without issues.

If you'd like a faster exchange regarding that issue, please contact our technical support.
_________________
Support Team
Aprelium - http://www.aprelium.com
Back to top View user's profile Send private message Visit poster's website
thinkr
-


Joined: 24 Oct 2013
Posts: 15
Location: California

PostPosted: Sun Nov 17, 2013 5:35 am    Post subject: working fine Reply with quote

Thanks for your help on this, it is working fine now that the code allows for variable chunk sizes. This same coding mistake is in sample code in several forums. Obviously consensus cannot be trusted, so from now on I'll refer to the specification. I plan on upgrading to asynchronous down the road (for performance) after reading through Miscrosoft's specifications.
_________________
If you don't question it then you don't know it.
Back to top View user's profile Send private message
aprelium-support
-


Joined: 20 Feb 2009
Posts: 356

PostPosted: Sat Nov 23, 2013 10:37 am    Post subject: Re: working fine Reply with quote

thinkr,

Glad to know that our references to the official MS documents helped you. With MS stuff, we learnt the hard way that one should never assume that a long time behavior is always guaranteed.

Regarding moving to async, it may not be the panacea since CGI (as a specification) is synchronous and time. Asynchronous is probably useful only if you're writing a low level server with hundreds/thousands of clients.
_________________
Support Team
Aprelium - http://www.aprelium.com
Back to top View user's profile Send private message Visit poster's website
thinkr
-


Joined: 24 Oct 2013
Posts: 15
Location: California

PostPosted: Mon Nov 25, 2013 1:13 am    Post subject: yep Reply with quote

Actually that's exactly what I'm doing. The basic system is running now, and next I plan on upgrading to maximize performance for lots of users.

Ok, in reading your response a new question came to mind. I was planning all along to upgrade later the HTTP requests involving file uploads to assign each to a separate Windows process running asynchonously. My compiler is all set with a command for it and I've planned my programming to not conflict with these particular files uploading over time in parallel. And then as I learned about asynchronous chunk transfer I assumed that this mode would be required for such parallel uploads to occur (to both change the compiler to assign a new thread to each upload, and the ReadFile access parameters to handle asynchronous chunks). Then just now realized that that was an assumption. Does one require the other? Or are these independent considerations?

I know I could find the answer if I first read a mile of specifications, but then again, with my luck Microsoft will not make this particular point clear (seems to happen to me a lot), and it seems you may have the answer off the top of your head.
_________________
If you don't question it then you don't know it.
Back to top View user's profile Send private message
aprelium-support
-


Joined: 20 Feb 2009
Posts: 356

PostPosted: Tue Dec 03, 2013 9:57 pm    Post subject: Re: yep Reply with quote

thinkr,

Each upload could be handled by its own thread running your Web application. No need to go in the complexity of async transfers as offered by Microsoft unless you expect the same FastCGI application to handle several thousands of clients. And even with that figures, the bottleneck would be reached elsewhere and will made your optimizations of reading uploads almost unnoticeable.
_________________
Support Team
Aprelium - http://www.aprelium.com
Back to top View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    Aprelium Forum Index -> FastCGI/CGI All times are GMT + 1 Hour
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB phpBB Group