[Babase] Row limits with Import button and Upload program

Karl O. Pinc kop at meme.com
Thu Apr 10 14:55:35 EDT 2008


On 04/10/2008 10:45:41 AM, kfenn wrote:
> Hi Karl,
> 
> Catherine M reported having trouble uploading more than ~20,000 rows  
> of data via the Import button in phpPgAgmin.  The data were going  
> into a new table she created in her schema and was trying to  
> populate.  I've been working with the weatherhawk data  (about 35,000  
> rows of data) and ran into the same problems (I can get about 20,000)  
> in at one time, but after that I get the error:
> 
> Import error: File could not be uploaded to the server
> 
> I have also tried using the UPLOAD program just for fun, but it seems  
> to max out at the same point.  However, you can upload an entire  
> table if you do it in chunks of 20,000 rows.
> 
> Do you know anything about row/data limitations when importing or can  
> you suggest another approach for uploading lots of data?  Maybe we  
> should just be creating the table with the data all in one step  
> rather than creating the table "shell" and populating it?

The problem is in the webserver configuration.  There's a number
of default limits that I've just upped.

You should now be able to upload a 100MB file, unless it takes
more than 200MB when encoded to transfer over the network or
unless the uploading php program takes more than 128MB of total
memory.

Please try it and see if it works.

I can increase these limits if necessary.

For the record, this is what shows up in the logs when the limits
are too small:

[client 128.112.21.181] PHP Fatal error:  Allowed memory size of  
8388608 bytes exhausted
(tried to allocate 97 bytes) in  
/disk/srv/www/include/lib_upload.php.inc on line 566, ref
erer: https://papio.biology.duke.edu/programs/upload/

[client 128.112.21.181] PHP Warning:  upload_max_filesize of 2097152  
bytes exceeded - fil
e [source=whawk_test_upload_30Knulls.txt] not saved in Unknown on line  
0, referer: https:
//papio.biology.duke.edu/phpPgAdmin/tblproperties.php?subject=table&table=whawk2&action=i
mport&server=localhost%3A5432%3Aallow&database=babase&schema=kfenn

See the /etc/httpd/conf.d/Xbabase.conf file.
(And also /etc/httpd/conf.d/XphpPgAdmin.conf.)
Note that these files are in the source control system under
/www/.

Karl <kop at meme.com>
Free Software:  "You don't pay back, you pay forward."
                  -- Robert A. Heinlein



More information about the Babase mailing list