Featured post
Bad performance function in PHP. With large files memory blows up! How can I refactor? -
i have function strips out lines files. i'm handling large files(more 100mb). have php memory 256mb function handles strip out of lines blows 100mb csv file.
what function must this:
originally have csv like:
copyright (c) 2007 maxmind llc. rights reserved. locid,country,region,city,postalcode,latitude,longitude,metrocode,areacode 1,"o1","","","",0.0000,0.0000,, 2,"ap","","","",35.0000,105.0000,, 3,"eu","","","",47.0000,8.0000,, 4,"ad","","","",42.5000,1.5000,, 5,"ae","","","",24.0000,54.0000,, 6,"af","","","",33.0000,65.0000,, 7,"ag","","","",17.0500,-61.8000,, 8,"ai","","","",18.2500,-63.1667,, 9,"al","","","",41.0000,20.0000,,
when pass csv file function got:
locid,country,region,city,postalcode,latitude,longitude,metrocode,areacode 1,"o1","","","",0.0000,0.0000,, 2,"ap","","","",35.0000,105.0000,, 3,"eu","","","",47.0000,8.0000,, 4,"ad","","","",42.5000,1.5000,, 5,"ae","","","",24.0000,54.0000,, 6,"af","","","",33.0000,65.0000,, 7,"ag","","","",17.0500,-61.8000,, 8,"ai","","","",18.2500,-63.1667,, 9,"al","","","",41.0000,20.0000,,
it strips out first line, nothing more. problem performance of function large files, blows memory.
the function is:
public function deleteline($line_no, $csvfilename) { // function strips specific line file // if line stripped, functions returns true else false // // e.g. // deleteline(-1, xyz.csv); // strip last line // deleteline(1, xyz.csv); // strip first line // assigna o nome ficheiro $filename = $csvfilename; $strip_return=false; $data=file($filename); $pipe=fopen($filename,'w'); $size=count($data); if($line_no==-1) $skip=$size-1; else $skip=$line_no-1; for($line=0;$line<$size;$line++) if($line!=$skip) fputs($pipe,$data[$line]); else $strip_return=true; return $strip_return; }
it possible refactor function not blow 256mb php memory?
give me clues.
best regards,
the problem blowout file
function brings entire file in memory. overcome need read file line line, write line deleted temporary file , rename temporary file.
public function deleteline($line_no, $csvfilename) { // temp file name in current working directory..you can use // other directory /tmp $tmpfilename = tempnam(".", "csv"); $strip_return=false; // open input file reading. $readfd=fopen($csvfilename,'r'); // temp file writing. $writefd=fopen($tmpfilename,'w'); // check fopen errors. if($line_no==-1) { $skip=$size-1; } else { $skip=$line_no-1; } $line = 0; // read lines input file 1 one. // write lines except line deleted. while (($buffer = fgets($readfd)) !== false) { if($line!=$skip) fputs($writefd,$buffer); else $strip_return=true; $line++; } // rename temp file input file. rename($tmpfilename,$csvfilename); return $strip_return; }
- Get link
- X
- Other Apps
Comments
Post a Comment