Broken pipe error processing large file - help!

[holdech]

I have a PHP application that reads from a flat file and posts to a PostgreSQL database. Everything is running on a Win XP pro machine.

It works fine for files around 100,000 records or less.

Above that, I get this in the access.log:
127.0.0.1 - - [24/Nov/2005:14:05:22 -0500] "GET /scraper/scrapeBigFF.php HTTP/1.1" 200 224 "" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
127.0.0.1 - - [24/Nov/2005:14:22:32 -0500] "GET /scraper/scrapeBigFF.php HTTP/1.1" 500 246 "" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

And this in the cgi.log:
CGI: [C:\PHP\php-cgi.exe ] URI: /scraper/scrapeBigFF.php Broken pipe

I've tried closing theDB connection and reconnecting at 100,000 record increments, but the same result.

I assume the "broken pipe" is a php error? Is there a setting in the php.ini that I need to tweak for large files?

Thanks

Chris

[AbyssUnderground]

Broken pipe is a script error. You need to find out where the error is occuring in the script. Do this by turning php errors in the php.ini on.
_________________
Andy (AbyssUnderground) (previously The Inquisitor)
www.abyssunderground.co.uk

[aprelium]

holdech,

Please read http://www.aprelium.com/forum/viewtopic.php?t=7961 for a similar problem.

You may also find the second part of http://www.aprelium.com/forum/viewtopic.php?p=37831#37831 interesting.
_________________
Support Team
Aprelium - http://www.aprelium.com

[holdech]

Thanks for the info.

The file is read, not uploaded - so that's not the problem.
It seems to gag when processing more than 100,000 inserts into PostgreSQL.
As long as I load less than that, all is fine. Must be related to the server to db communications.

In answer to the other post - it's not a script error. The script runs just fine. It's not a file data issue either as I can process all 2 million records in 100K chunks.

I also tried reading the entire file into an array, using file(). That part works fine - can echo all 2 million records back to the browser. I think this would localize the problem to the database connection (pipe).

Chris

[holdech]

You may want to keep this one for reference.

When php is running as a cgi, it's using a socket to communicate with the web server.

This will timeout in 30 seconds unless you send "something" back to the web server in under 30 seconds i.e. just "echo" something to the browser.

I had this problem because the db server was active for more than 30 seconds processing the file I was importing - so nothing was sent to the web server socket and it timed out - hence the "broke pipe".

Note that this is NOT the script "max execution" time - which I also had to set higher than it's default 30 seconds.

There is no setting in php.ini that can be tweaked to fix this. Their socket timeout parm relates to not getting a reply vs no request.

Question - is their a socket timeout in Abyss that can be tweaked?

Anyway - file this one for the next guy that has a database intensive script and is getting a 500 error with a broken pipe.

Chris

[aprelium]