Hi,
I have huge CSV files that I need to import a database. For this approach, I have to write my own import function. The code works fine, except... it takes time to import files (sometimes an hour!)
Basically, the process is
Code:
1 ) open file
2 ) read file - fgets()
3 ) split each line into tokens
4 ) for each read token
4.1 ) check if there are any "bad characters" which will cause SQL errors while executing SQL queries
4.1.1 ) if there is, remove it
4.2 ) construct SQL query
4.3 ) update database
I have to perform step 4 since I had errors because of bad characters before. But evaluating each character really takes time.
To improve the overall performance, I was thinking to map the file into memory before fgets() but mmap() is not portable
any recommendations are welcome....