Thread: fastest way of processing a file

Agreed. You need to profile the thing to work out where the time is being spent. One way to essentially do that is to comment out a certain part and see how much faster it becomes.
So yes please comment out the sqllite calls and time that.

Go to my website and take my csv file parser. It might be fast enough.

If, as some of us suspect, the SQL is the elephant in the room, then try this.

Code:

void tryit ( ) {
  const char *word = "TestElement";

  //!! pick some numbers which represent your data
  const int numLines = 1000000, numFields = 100;

  for ( int line = 0 ; line < numLines ; line++ ) {
    for ( int field = 0 ; field < numFields ; field++ ) {
      sqlite3_bind_text(stmt, 1, word, -1, SQLITE_STATIC);
    }
    if (sqlite3_step(stmt)!=SQLITE_DONE)
        return sqlite3_errcode(db);

    sqlite3_reset(stmt);
  }
}

If this still takes an hour, then you're wasting your effort saving a couple of minutes on improving (although there is some very low hanging fruit to be picked).

Your efforts might be better employed looking for more efficient use of SQL to begin with.
Importing CSV would be so common that there should be an efficient way of doing it.

I was experimenting with fnprintf's code on the previous page, and a 20,500 line data file, as the OP suggested.

Most assuredly, there is no input bottleneck with this or other reasonable code. fnprintf's program (as tweaked a bit to get input from a a file), handles it in less than half a second.

Code:

//fnprintf's code 
#include <stdio.h>
#include <string.h>
#include <time.h>
 
int main (void)
{
   char *p, csv[100]; 
   char rmc[] = ".:;`~*\\?=)(/&%+^'!\"|}][{$ #@\t";
   char pstr[30][100];
   char ostr[30][100];
   int i=0,c=0;
   clock_t timer;
   FILE *fpIn,*fpOut;
          
   fpIn=fopen("data20500.txt", "r");
   fpOut=fopen("dataOut20500.txt","w");
     
   if(!fpIn ) { // || !fpOut) {
      printf("Error opening file\n");
      return 1;
   }
   timer=clock();

   while(fgets(csv, sizeof(csv),fpIn)) {
      i=0;
      p = strtok (csv, ",");
      while (p)
      {
         strcpy (pstr[i], p);
         i++;
         p = strtok (NULL, ",");
      }
      c = i;
    
      for (i = 0; i < c; i++)
      {
         p = strtok (pstr[i], rmc);
         strcpy (ostr[i], "");
         while (p)
         {
            strcat (ostr[i], p);
            strcat (ostr[i], " ");
            p = strtok (NULL, rmc);
         }
      }
      for (i = 0; i < c; i++)
      {
         fprintf (fpOut,"%d: %s\n", i, ostr[i]);
      }
   }  
   timer=clock()-timer;
   printf("Elapsed time: %f seconds\n",(double)timer/CLOCKS_PER_SEC);
 
   fclose(fpIn);
   fclose(fpOut);
	
   return 0;
}