file reading problem

**but** · 04-03-2016

ok guys - sorry for bothering you again and again with the same issue, but i seem lost - every time i take one step forward, its two steps behind.

the problem is this - as simply as i can put it. In fact i already have put it (here) and for a while it seem to work but now it doesnt !!

there is a file with 1000 sets of data-set. Each data set has about 12344 rows and 5 columns. I need only column 3, 4 and 5.

Also the first two lines of each data-set and the last line is redundant - in between the first 2 useless lines and the last useless line, are 12344 lines - then again 2 lines of bogus, 12344 useful rows, then the last bogus line and on and on for 1000 times.

I have to write a program that reads the previous file in such a way, so as to ignore the first two lines, then ignore the first 2 rows and read in the next 3 rows, a total of 12344 times. Then this entire procedure is to be repeated 1000 times. Finally this new "sussed out" data has to be written into a new file, but only every 4th line.

To that end i wrote the following program - with lots and lots of help from you guys - but its not working and giving me "segmentation error" and/or 'core dump"

Code:

main (int argc, char **argv)
{
 int i, j;
 double x[1000][12344], y[1000][12344], z[1000][12344];
 char line[1024];
 FILE *fp;
 FILE *fp2;
 FILE *fp3;
 fp = fopen ("abcd.gro", "r"); //  reading   //
 for (j=0; j<1000; j++)
     {
       fgets(line, sizeof line, fp);
       fgets(line, sizeof line, fp);
       for (i=0; i<12344; i=i+1)
        {
         fscanf(fp, "%*d%*d%lf%lf%lf", &x[j][i], &y[j][i], &z[j][i]);  
        }
     fgets(line, sizeof line, fp);
      }
 fp2 = fopen ("xyz.gro", "w"); // printing xyz //
 for (j=0, j<1000, j++)
{
for (i=0; i<12344; i=i+4)
     {
     fprintf (fp2, "%2.5lf%8.5lf%8.5lf\n", x[j][i], y[j][i], z[j][i]);
     }


fclose(fp);
fclose(fp2);
return 0;
}

see if someone can clarify things for me one final time.

**algorism** · 04-03-2016

If you're just trying to write the data to another file, you don't need to store it. Just open both your input and output file and write the data as you go along.

Code:

#include <stdio.h>
#include <stdlib.h>

int main (void) {
    char line[4096]; 
    int i, j;
    double x, y, z;
    FILE *fpin, *fpout;

    fpin = fopen ("abcd.gro", "r");
    if (fpin == NULL) {
        fprintf(stderr, "Can't open input file\n");
        exit(EXIT_FAILURE);
    }

    fpout = fopen("xyz.gro", "w");
    if (fpout == NULL) {
        fclose(fpin);
        fprintf(stderr, "Can't open output file\n");
        exit(EXIT_FAILURE);
    }

    for (i = 0; i < 1000; i++) {
        fgets(line, sizeof line, fpin);
        fgets(line, sizeof line, fpin);
        for (j = 1; j <= 12344; j++) { // counting from 1
            fscanf(fpin, "%*f%*f%lf%lf%lf", &x, &y, &z);
            if (j % 4 == 0)
                fprintf(fpout, "%.5f %.5f %.5f\n", x, y, z);
        }
        fgets(line, sizeof line, fpin);
    }

    fclose(fpin);
    fclose(fpout);

    return 0;
}

**but** · 04-05-2016

Thanks a million. now the code works - I do have one doubt tho - i do not understand why you counted form 1 and why (because of the 1 i am guessing) you used a <= in the following -

for(j = 1; j <= 12344; j++) { // counting from 1

**algorism** · 04-05-2016

Originally Posted by but

i do not understand why you counted form 1 and why (because of the 1 i am guessing) you used a <=

It starts at 1 so that j % 4 == 0 is true for every 4th line. Alternatively it could start at 0 and the mod could be j % 4 == 3.

And yes, it uses <= 12344 because it starts at 1. Alternatively it could have been j < 12345.

**but** · 04-05-2016

I see thanks for the explanation - but this problem is hydra headed. So now it has reared its head again -

here is the code i am running - based on your code -

Code:

 for (j=0; j<1000; j++)
      {
       fgets(line, sizeof line, fp);
       fgets(line, sizeof line, fp);


       for (i=0; i<13968; i=i+1)
            {        
             fscanf(fp, "%*f %*f %lf %lf %lf", &x, &y, &z);
             if (i%4 ==3)
             fprintf(fp2, "%.5f %.5f %.5f\n", x, y, z);
             } 
       fgets(line, sizeof line, fp);
      }

The remarkable thing is that in the output, only the first group - of 13968/4 sets of x, y, z, is being written correctly in the xyz file. After that the y and the z is written correctly but the x value is screwed up. Instead of printing the x value, it keeps printing the "i" value - so the x values go - 4.000 then 8.000, then 12.000 etc

So why does it get the first group right but not the subsequent groups - mind you the y and z values are correctly written - its only in the x values that the drama starts.

copied below is the last few of the first 13968 data then the first few of the next set, so you can see the issue.

3.53500----- 8.22600 -----8.39600
3.96500----- 8.23500 -----8.37100
3.73900 -----8.35700 -----8.26700
4.18400 -----8.36900 -----8.26700
4.00000 -----5.14600 -----9.14800
8.00000 -----4.92500 -----9.29600
12.00000 ---5.39300 ------9.26200

**algorism** · 04-05-2016

Originally Posted by but

this problem is hydra headed. So now it has reared its head again

I thought the beast was finally slain!

Post your entire program. Believe it or not, the problem could be pretty much anywhere.

Could you upload your data file? I'm assuming it's less than a 2 gigs. You can use wetransfer.com (I'm not advocating it, but it's free for files up to 2 gigs.) I think that once you're on the transfer page and have "added" the file you need to click the bottom left button and choose "link" and then hit the "transfer" button to get a link to the file. Then post the link here.

**jimblumberg** · 04-05-2016

Post a sample of the actual input file as well. It could be that your file is not as described as well.

Jim

**but** · 04-05-2016

ok sure - i have to be afk for a few hours but later today i shall upload the whole code and the first 2 or 3 sets of the data file - no need for 1000 sets. cheers.

**algorism** · 04-05-2016

Originally Posted by but

ok sure - i have to be afk for a few hours but later today i shall upload the whole code and the first 2 or 3 sets of the data file - no need for 1000 sets. cheers.

Unless you're certain that the problem occurs within the first 2 or 3 data sets (are you?) then you have to upload the whole thing.

**but** · 04-05-2016

the problem does occur soon as the first set is complete and we move into the 2nd - from there on instead of printing the x value, it writes the x number (instead of writing the 4th x, 8th x etc in the file it writes 4, 8, 12 etc). the y and z values are printed correctly.

Here is the code - the j goes to only 10 (0-9) instead of 1000 - since i am uploading only the first 10 sets of data.

Code:

#include <math.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <signal.h>
/* 
Compile as:
          cc allFrames.c -o Frame -lm
*/
main (int argc, char **argv)
{
 int i, j;
 double x, y, z;
 char line[4096];
 FILE *fp;
 FILE *fp2;
 fp = fopen ("abcd.gro", "r"); //  reading   //
 fp2 = fopen ("xyz.gro", "w"); // writing//
 for (j=0; j<10; j++)
     {
       fgets(line, sizeof line, fp);
       fgets(line, sizeof line, fp);
       for (i=0; i<13968; i=i+1)
        { 
        fscanf(fp, "%*f %*f %lf %lf %lf", &x, &y, &z);
        if (i%4 ==3)
           fprintf(fp2, "%.5f %.5f %.5f\n", x, y, z);
          }
     fgets(line, sizeof line, fp);
      }
fclose(fp);
fclose(fp2);


return 0;
}

i have the file but cant upload- says its too large - at 5.5ish mb

**but** · 04-05-2016

https://www.wetransfer.com/downloads...6020049/53c652

thats the file

**algorism** · 04-05-2016

I can't replicate your problem.
The output looks perfect to me.
Each 13968-line group becomes a group of 3492 lines.
They seem to have the correct values.
I don't see (with my scanning program) any place where the first column (i.e., x) is 4.0, then 8.0, then 12.0.
There's not even two lines in a row where x is 4.0 and then 8.0.

**but** · 04-05-2016

can you pls show me a screen shot of the output you got - this is what i got file reading problem-issue-jpg

see what happens to the forst column from after the blue dot - thats when the 2nd set of data starts.

**algorism** · 04-05-2016

Try this:

Code:

#include <stdlib.h>
#include <stdio.h>

int main (void) {
    int i, j;
    double x, y, z;
    char line[4096];
    FILE *fp, *fp2;

    fp = fopen ("abcd10.txt", "r");
    fp2 = fopen ("xyz.gro", "w");

    for (j=0; j<10; j++) {
        fgets(line, sizeof line, fp);
        fgets(line, sizeof line, fp);
        for (i = 0; i < 13968; i++) {
            fgets(line, sizeof line, fp);
            if (sscanf(line, "%*f %*f %lf %lf %lf", &x, &y, &z) != 3) {
                printf("bad line: j=%d i=%d\n[%s]\n", j, i, line);
                return EXIT_FAILURE;
            }
            if (i % 4 == 3)
                fprintf(fp2, "%.5f %.5f %.5f\n", x, y, z);
        }
        fgets(line, sizeof line, fp);
    }

    fclose(fp);
    fclose(fp2);
 
    return 0;
}

I changed it to first read in the line with fgets and then scan the line with sscanf, which I should have done in the first place since that's basically always the proper thing to do.

Now that I think about it, it was the combination of fgets and fscanf that screwed it up. The fscanf leaves the newline at the end of the line, so the next fgets only reads that and you end up falling a line behind. Dumb mistake on my part. So it wasn't actually printing i somehow, but instead it was printing the second column as x (the one that's counting upwards) and in fact your y and z would've been incorrect too since they would have the values that should have been in x and y, respectively.