Error while writing into a File; after certain file size

Hi all,

I am executing a C program that writes into a file. I get the following error after the file reaches certain size ( < 0.4 GB).

10 [main] anj3 4872 _cygtls::handle_exceptions: Error while dumping state (
probably corrupted stack)
Segmentation fault (core dumped)

It always happens when the size of the file is too big. Is it because of too big a file if so how can I let my program know the file needs to be Big enff.


If not what else could be wrong.

Note: I use a loop that executes at least 4000 times and each time it writes 1000000 or more chars into the file.

Quote:

---

Originally Posted by anjum

Note: I use a loop that executes at least 4000 times and each time it writes 1000000 or more chars into the file.

---

So, actual memory usage does not increase between the first iteration and the nth?

Do you have any recursion?


Quzah.

Same iteration but after some 2000 iteration it just halts with the error.
If I stop the loop right before the halting point ( by changing the condition) no error occurs and the program continues with the code after the loop.

IS there any file size limitation in C or the operating system
I am using cygwin and vista

I believe Problem with my program is due to excessive use of memory.


I have more than 2000 iteration. In each iteration I open at least 2 files. Windows vista does not close these files anticipating they will be used in future.

I close them after every iteration using fclose() method. Is there anything I can do to tell OS to remove the file from run time memory.

I tried using sleep() method after very 1000 iterations to kind of stop the memory usage. But it is does not seem to work well for my problem. I do not understand why this problem happens. The weird thing is I get the error message

[see the error in the first message]


at different time during differ run. Meaning something outside the program is causing it.


Please help. I just want to know what I am doing wrong

Quote:

---

Originally Posted by anjum

I believe Problem with my program is due to excessive use of memory.

I have more than 2000 iteration. In each iteration I open at least 2 files. Windows vista does not close these files anticipating they will be used in future.

I close them after every iteration using fclose() method. Is there anything I can do to tell OS to remove the file from run time memory.

---

If it's closed properly, that should be done. Are you sure they are? Why don't you post some code with the fopen() and fclose() calls in it?

The following is the my code partially. The first while loop is the one that I want to execute way more than 2000 times. But I kind of get stuck at about 2350 or 2260 (I do not know why the difference)

Code:

---


      loop: while(!feof(rf))
        {
                fgets(line,380,rf);
               
        //        Reading a line from pdb list file printf(line);
                        strncpy(filename,line,4);
                        filename[4] ='\0';
                        strncpy(profile,line,5);
                        profile[5] = '\0';
                        strcpy(pdbName,profile);
                        strcat(profile,".profile");
                        strncpy(chain,line+4,1);
                        strcat(filename,".pdb");
                        char st[100] = cullpdb;
                        strcat(st,filename);                        //        Getting the file name of the pdb entry
                        char st4[100] = psiblast;
                        strcat(st4,profile );
                        st4[strlen(psiblast)+1+13]='\0';
                        st[strlen(cullpdb)+1+8]='\0';
                        //printf("Profile File Name = %s", st4);


  // These are two files I open in every iteration
                        if((fpro = fopen(st4,"r"))==0)             
                        {
                                printf("Profile File Not found %s", st4);
                                goto loop;
                               
                                }
                                        if((f = fopen(st,"r")) == 0)
                        { printf(" File %s is not in Directory", &st);
                                goto loop;
                        }
                        else
                        {
                                        strcpy(pdb_list[pdbn],pdbName);
                                        pdbn++;
                                }
                               
                               
                               
                                        fprintf(fp,"1  ");
                                        featureNum = 1;
                //Profile data converted into features for SVM
                        while(!feof(fpro))
                        {
                                fgets(proLine,380,fpro);
                                int k=0;
                                char str3[100];
                               
                                for(i=0; i < strlen(proLine); i++)
                                {
                                               
                                                if(proLine[i] != ' ')
                                                {
                                                        str3[k] = proLine[i];
                                                        k++;
                                                }
                                                else
                                                {
                                                        str3[k]='\0';
                                                        j = atoi(str3);
                                                        if(j!=0)
                                                        {
                                                                fprintf(fp,"%d:%d ", featureNum,j);
                                                       
                                                                }
                                                        featureNum++;
                                                        k=0        ;
                                                }                                       
                                                         
                                }
                       
                               
                        }
                       
       
                n=0;
                modelFlag = 0;
               
                        while(!feof(f))
                        {
                               
                                                fgets(pline,380,f);
         
                                                strncpy(st2,pline,5);
                                                st2[5] = '\0';
                                                if(strcmp(st2,"MODEL") ==0)
                                                {
                                                                if(modelFlag ==0)
                                                                                modelFlag = 1;
                                                                else
                                                                                goto Dist;
                                                }
                                                strncpy(st2,pline,4);
                       
                                                st2[4] = '\0';
                       
                                                if(strcmp(st2, "ATOM") == 0)
                                                {
                                                        strncpy(st3,pline+13,2);
                                                        st3[2] ='\0';
                                                        strncpy(seq_chain,pline+21 ,1);
                                                        seq_chain[1]='\0';
                                        if((strcmp(st3,"CA") ==0) && (strcmp(seq_chain,chain) ==0))
                                                        {
                                                                                                       
                                                                strncpy(xc,pline+26,12);
                                                                strncpy(yc,pline+38,8);
                                                                strncpy(zc,pline+46,8);
                                                                xc[12] ='\0'; yc[8]= '\0'; zc[8] = '\0';
                                                                                                       
                                                        ca[n].x = atof(xc); ca[n].y = atof(yc); ca[n].z =atof(zc);
                                                                n++;
                                                        }
                                                }
                       
                        }
                       
                       
                Dist:       
                        for(i=0;i<n;i++)                                       
        // Make number 4 to n for N*N distance matrix or it will be N*4 matrix
    {
        for(j=i+1;j<n;j++)
        {
          dist = sqrt((ca[i].x -ca[j].x)*(ca[i].x-ca[j].x) +  (ca[i].y -ca[j].y)*(ca[i].y-ca[j].y) +  (ca[i].z -ca[j].z)*(ca[i].z-ca[j].z));
          fprintf(fp,"%d:%.3lf ", featureNum,dist);
          featureNum++;
         
        }
      }
      fprintf(fp,"# %s %s", pdbName,seqn);
                        printf(".");
                        printf("\nDone with one vector %d pdb ID is %s Feature Number is %d",posn,pdbName,featureNum);
                        if(maxF<featureNum)
                                maxF = featureNum;
                                avgF = avgF+ featureNum;
                        posn++;
                        if(posn>C1000*1000)
                        {
                                sleep(200);
                                C1000++;
                        }
                        //scanf("%c",st2);
                        fclose(f);                      // I am closing BOth files here
                        fclose(fpro);
                        fprintf(fp,"\n");
                }

---

Code:

---

                        if((fpro = fopen(st4,"r"))==0)             
                        {
                                printf("Profile File Not found %s", st4);
                                goto loop;
                               
                                }
                                        if((f = fopen(st,"r")) == 0)
                        { printf(" File %s is not in Directory", &st);
                                goto loop;
                        }
                        else

---

If you ever execute this code, then you will get a handle-leak, because fpro is still open.

--
Mats

I closed the fpro file. I still get the same problem after 2387 iterations




388 [main] a 5060 _cygtls::handle_exceptions: Error while dumping state (pro
bably corrupted stack)
Segmentation fault (core dumped)

And there isn't any chance that for example "n" gets out of range for the arrays you have?

There are no memory allocations and nothing else that seems to "leak", so there's no reason why this should fail.

--
Mats

GASP! Is that a "goto" I see???

Quote:

---

Originally Posted by Bladactania

GASP! Is that a "goto" I see???

---

Kernel hackers are allowed to do that ;)

Code:

---

strcpy(pdb_list[pdbn],pdbName);
                                        pdbn++;

---

If this is a stack variable and you have enough of them (I see a few cantidates), you will run out of stack eventually.

Quote:

---

Originally Posted by MK27

Code:

---

strcpy(pdb_list[pdbn],pdbName);
                                        pdbn++;

---

If this is a stack variable and you have enough of them (I see a few cantidates), you will run out of stack eventually.

---

Good catch.

--
Mats

Code:

---

while(!feof(rf))
{
    fgets(line,380,rf);

    ...

while(!feof(fpro))
{
    fgets(proLine,380,fpro);

    ...

while(!feof(f))
{
    fgets(pline,380,f);

    ...

---

Avoid using feof in this manner to control your loops. A better choice is to use the return result directly from the fgets call:

Code:

---

while(fgets(line,380,rf) != NULL)
{
    ...

while(fgets(proLine,380,fpro) != NULL)
{
    ...

while(fgets(pline,380,f) != NULL)
{
    ...

---

Quote:

---

Originally Posted by MK27

Kernel hackers are allowed to do that ;)

---

Yeah, so why is random dude using it? ;)

Hontestly, I stopped reading when I ran into it. It's not worth the effort to chase down the horrible path his code is taking. There are some paths that I just don't care where they lead.


Quzah.

Quote:

---

Originally Posted by quzah

Yeah, so why is random dude using it? ;)

Hontestly, I stopped reading when I ran into it. It's not worth the effort to chase down the horrible path his code is taking. There are some paths that I just don't care where they lead.


Quzah.

---

Actually, it's perfectly possible, in this case, to avoid the goto, using continue instead. It's not BAD use of goto, however, so I'd say it's not much worse than using continue. Refactoring the code entirely to remove both the goto any a replacement continue is possible, but require a fair bit of work.

--
Mats