Thread: Best route to the result

I have a 100mb file which contains 10 million entries, I need to append 3 characters to the end of each line.

I know I can do this in DOS.... but way too slow....

Does anyone have an idea how long this would take to complete in C..... Is C the best program language for this?

I have read I can use strcat to do this, but is this the best route for dealing with such large volumes....?

Any help appreciated....

Depends mostly on IO speed. Appending three characters to a line (even if it's ten million lines) shouldn't be too slow. Read one line at a time, strcat the characters and write.

@Adak, thanks for helping...

1st txt file contains 10million lines of

AAAAA
AAAAB
AAAAC
AAAAD
etc

2nd file contains 5 3 character codes
XXX
YYY
ZZZ

output required is 3 new text files containing
FILE1
XXXAAAAA
XXXAAAAB
XXXAAAAC
etc

FILE2
YYYAAAAA
YYYAAAAB
YYYAAAAC
etc

So in effect I need to append 3 characters to the front of each line in a file of 10milion, in a reasonably effecient manner...

I can do this in my sleep in DOS (yes Windows) but output is limited to 200 per minute, hoping C will complete task in an hour or 2?

Thanks for any help...

Only 200 per minute with a bat file? You're in for a VERY pleasant surprise with C.

About the system you'll be running this on:

cpu is?:

Amount of memory?:

Your compiler is?:

Your operating system is Windows XP?

Write the prefix to the new file
Read a line from the existing file
Write the line from the existing file to the new file

Repeat.

By the way.... you are not "appending" your are "Prepending"...

As it turns out, this should be fast, all around. It appears the 5 letters per row data, is a 5 letter permutation list. Since I didn't have enough sample data to work with, I generated my own list, up through AZZZZ - that took just a couple seconds.

Which made me think that maybe re-generating the entire permutation file, WITH these extra 5 char's we're trying to prepend, would take less time than coding up a prepending program with the file handling the OP wanted, etc.

In any case, this is only slightly tested for accuracy, and not optimized, or tested against other algorithms or data structures. I avoided using strcat, because I thought it would be a slow down, and wasn't necessary. It's fast enough, imo - about 2.2 Million records prepended, in about 2.5 seconds.

As Brafil mentioned earlier in the thread, it's running speed is largely bound by the IO throughput.

Code:

/* 
prepends 3 char's from perms3.txt file, (which has 5 rows of char's), 
each row having 3 char's and a newline we don't use. These are 
written out into the front of each row before the 5 chars 
(6 counting the newline), in the perms6.txt file, are written out
to five sequentially numbered files, (allperm1.txt - allperm5.txt).

It's not optimized, or tested against other algorithms, but it's fast. 
Try it and C. ;)

*/

#include <stdio.h>

typedef struct {
  char char3[3];
  char newline;
}record;

int main(void) {
  FILE *fpin3, *fpin6, *fpout;
  int i;
  const char *filename6="perms6.txt"; //reminder to take the newline char #6
  const char *filename3="perms3.txt";//leave the newline behind
  char fileOut[]="allperm1.txt";
  record rec;
  char char6[6];
  unsigned long int count = 0;

  fpin3 = fopen(filename3, "rt");
  fpin6 = fopen(filename6, "rt");
  if(fpin3 == NULL || fpin6 == NULL) {
    printf("\nError opening input files");
    return 1;
  }
  if((fpout =fopen(fileOut, "wb"))== NULL) {
    printf("\nError opening output file");
    return 1;
  }
  printf("\n\n\n");
  for(i=0;i<5;i++) {
    fread(&rec, sizeof(rec), 1, fpin3);
    
    while(fread(char6, 6, 1, fpin6) >0) {
      fwrite(rec.char3, 3, 1, fpout);     //fpout or stdout (for debug)
      fwrite(char6, 6, 1, fpout);         //ditto
      ++count;
      //getch();
    }
    if(fileOut[7]=='5')
      break;
    rewind(fpin6);
    fclose(fpout);
    printf("\n closing file %s", fileOut);
    fileOut[7]++; //increment the file number in the name
    printf("\n opening file %s", fileOut);
    if((fpout =fopen(fileOut, "wb"))== NULL) {
      printf("\nError opening output file");
      return 1;
    }
  }
  fcloseall();
  printf("\n\n %lu\n\t\t\t    press enter when ready", count);
  i=getchar();
  return 0;
}

@Adak and Brewbuck.....

Thanks great help, trialling Adaks version at the mo, is massivley quicker than I thought it would be.

I take your point about generating the whole file from within the program, hadnt really considered that approach, and well out of my skills at the moment.... I suspect maybe one for me to try in the future!!

Also going to amend code to have an append version as I think this will be useful for me shortly.

Thanks again....

@Adak...

Got it working to append rather than preappend....

The program/terminal requires a key to be pressed to exit the terminal window, is there a way round that? I woudl like the progrsm to run create the files then exit.

I have searched, looked at break, return and exit but none seem to work?

Thanks again...

Remove or REM out the i=getchar() line of code. Second from the last line. By REM I mean "REMark" by putting either a // in front of the first letter of the line, or by surrounding the line, like so:

/* i=getchar(); */

How long is the longest number of permutations you need? 8 char's is still pretty quick, but around 15, it REALLY begins taking up more time than you'd probably like.

May I ask what you want these permutations for?