Import large amounts of CSV data with high performance in PHP

1. Controller writing

//Get the parameters of the request
$arrayParams = Request::all();
$objCsvFile = $arrayParams['csv_file'];
$strRealPath = $objCsvFile->getRealPath();//tmp path, here you can save it to your own predetermined path first, and then read it

//****************The focus is on this step********************//
$glob = CommonUtilFunction::readPathCsvFile($strRealPath);
//********************************************//

$intRowNum = 0;
while($glob->valid()) {
$arrayNewLineData = [];
$intRowNum++;
if (1 === $intRowNum) {
//The first line is skipped, usually the title
$glob->next();
continue;
}
$arrayLineData = $glob->current();

// handle empty string empty line
/**
* Generally, csv has two kinds of line data that can be considered as blank lines
* The first type ',,,,,,,,,,,,,,,,,,,,,,,,,,,', like this pure comma without any data
* The second ' ', is a real blank line, nothing
* Returns a uniform array[] when processing is complete
*/
$arrayLineData = CommonUtilFunction::dealCsvLineData($arrayLineData);
// skip blank lines
if (true === empty($arrayLineData)) {
$glob->next();
continue;
}

// own code logic
...

// avoid unexpected errors
unset($arrayNewLineData);
$glob->next(); // Process the next line of data
}

2.yield reading data and processing blank lines

/**
* @description iterator to read csv file
* @param $strCsvPath
* @return \Generator
*/
public static function readPathCsvFile($strCsvPath)
{
if ($handle = fopen($strCsvPath, 'r')) {
while (!feof($handle)) {
yield fgetcsv($handle);
}
fclose($handle);
}
}


/**
* @description processes c single-line information
* @param $arrData
* @return \Generator
*/public static function dealCsvLineData($arrData = [])
{
$arrAfterData = [];
if (false === empty($arrData)) {
//Remove spaces before and after each string
foreach ($arrData as &$colData) {
//Detect the corresponding encoding format csv file format Shift-JIS
$strEncodeType = mb_detect_encoding($colData, ['UTF-8', 'Shift-JIS']);
//If you think utf-8 format does not need to be transcoded, shift-jis format needs to be converted to utf8 format
if ('SJIS' === $strEncodeType) {
//jis=>utf8
$colData = mb_convert_encoding($colData, 'UTF-8', 'Shift-JIS');
}
$colData = trim($colData);
}
// remove blank lines
$isEmptyRow = true;
foreach ($arrData as $item) {
if ('' !== $item) {
$isEmptyRow = false;
break;
}
}
if (false === $isEmptyRow) {
$arrAfterData = $arrData;
}
}
return $arrAfterData;
}

in conclusion

Using yield can greatly reduce server overhead, and the pressure is on the database. The upper limit has not been tested, but 10,000 data is very easy.