Error while writing into a File; after certain file size

This is a discussion on Error while writing into a File; after certain file size within the C Programming forums, part of the General Programming Boards category; Hi all, I am executing a C program that writes into a file. I get the following error after the ...

  1. #1
    Registered User
    Join Date
    Apr 2009
    Posts
    5

    Unhappy Error while writing into a File; after certain file size

    Hi all,

    I am executing a C program that writes into a file. I get the following error after the file reaches certain size ( < 0.4 GB).

    10 [main] anj3 4872 _cygtls::handle_exceptions: Error while dumping state (
    probably corrupted stack)
    Segmentation fault (core dumped)

    It always happens when the size of the file is too big. Is it because of too big a file if so how can I let my program know the file needs to be Big enff.


    If not what else could be wrong.

    Note: I use a loop that executes at least 4000 times and each time it writes 1000000 or more chars into the file.

  2. #2
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by anjum View Post
    Note: I use a loop that executes at least 4000 times and each time it writes 1000000 or more chars into the file.
    So, actual memory usage does not increase between the first iteration and the nth?
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  3. #3
    ATH0 quzah's Avatar
    Join Date
    Oct 2001
    Posts
    14,826
    Do you have any recursion?


    Quzah.
    Hope is the first step on the road to disappointment.

  4. #4
    Registered User
    Join Date
    Apr 2009
    Posts
    5

    Unhappy

    Same iteration but after some 2000 iteration it just halts with the error.
    If I stop the loop right before the halting point ( by changing the condition) no error occurs and the program continues with the code after the loop.

    IS there any file size limitation in C or the operating system
    I am using cygwin and vista

  5. #5
    Registered User
    Join Date
    Apr 2009
    Posts
    5

    Unhappy

    I believe Problem with my program is due to excessive use of memory.


    I have more than 2000 iteration. In each iteration I open at least 2 files. Windows vista does not close these files anticipating they will be used in future.

    I close them after every iteration using fclose() method. Is there anything I can do to tell OS to remove the file from run time memory.

    I tried using sleep() method after very 1000 iterations to kind of stop the memory usage. But it is does not seem to work well for my problem. I do not understand why this problem happens. The weird thing is I get the error message

    [see the error in the first message]


    at different time during differ run. Meaning something outside the program is causing it.


    Please help. I just want to know what I am doing wrong

  6. #6
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by anjum View Post
    I believe Problem with my program is due to excessive use of memory.

    I have more than 2000 iteration. In each iteration I open at least 2 files. Windows vista does not close these files anticipating they will be used in future.

    I close them after every iteration using fclose() method. Is there anything I can do to tell OS to remove the file from run time memory.
    If it's closed properly, that should be done. Are you sure they are? Why don't you post some code with the fopen() and fclose() calls in it?
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  7. #7
    Registered User
    Join Date
    Apr 2009
    Posts
    5

    Unhappy

    The following is the my code partially. The first while loop is the one that I want to execute way more than 2000 times. But I kind of get stuck at about 2350 or 2260 (I do not know why the difference)




    Code:
           loop: while(!feof(rf))
    	{
    		fgets(line,380,rf);
    		
    	//	Reading a line from pdb list file printf(line);
    			strncpy(filename,line,4);
    			filename[4] ='\0';
    			strncpy(profile,line,5);
    			profile[5] = '\0';
    			strcpy(pdbName,profile);
    			strcat(profile,".profile");
    			strncpy(chain,line+4,1);
    			strcat(filename,".pdb");
    			char st[100] = cullpdb;
    			strcat(st,filename);			//	Getting the file name of the pdb entry
    			char st4[100] = psiblast;
    			strcat(st4,profile );
    			st4[strlen(psiblast)+1+13]='\0';
    			st[strlen(cullpdb)+1+8]='\0';
    			//printf("Profile File Name = %s", st4);
    
    
        // These are two files I open in every iteration
    			if((fpro = fopen(st4,"r"))==0)              
    			{
    				printf("Profile File Not found %s", st4);
    				goto loop;
    				
    				}
    					if((f = fopen(st,"r")) == 0)
    			{ printf(" File %s is not in Directory", &st); 
    				goto loop; 
    			}
    			else
    			{
    					strcpy(pdb_list[pdbn],pdbName);
    					pdbn++;
    				}
    				
    				
    				
    					fprintf(fp,"1  ");
    					featureNum = 1;
    		//Profile data converted into features for SVM
    			while(!feof(fpro))
    			{
    				fgets(proLine,380,fpro);
    				int k=0;
    				char str3[100];
    				
    				for(i=0; i < strlen(proLine); i++)
    				{
    						
    						if(proLine[i] != ' ')
    						{
    							str3[k] = proLine[i];
    							k++;
    						}
    						else
    						{
    							str3[k]='\0';
    							j = atoi(str3);
    							if(j!=0)
    							{
    								fprintf(fp,"%d:%d ", featureNum,j);
    							
    								}
    							featureNum++;
    							k=0	;
    						}					
    					  		
    				}
    			
    				
    			}
    			
    	
    		n=0;
    		modelFlag = 0;
    		
    			while(!feof(f))
    			{
    				
    						fgets(pline,380,f);
       	
    						strncpy(st2,pline,5);
    						st2[5] = '\0';
    						if(strcmp(st2,"MODEL") ==0)
    						{
    								if(modelFlag ==0)
    										modelFlag = 1;
    								else
    										goto Dist;
    						}
    						strncpy(st2,pline,4);
    			
    						st2[4] = '\0';
    			
    						if(strcmp(st2, "ATOM") == 0)
    						{
    							strncpy(st3,pline+13,2);
    							st3[2] ='\0';
    							strncpy(seq_chain,pline+21 ,1);
    							seq_chain[1]='\0';
    					if((strcmp(st3,"CA") ==0) && (strcmp(seq_chain,chain) ==0))
    							{
    													
    								strncpy(xc,pline+26,12); 
    								strncpy(yc,pline+38,8);
    								strncpy(zc,pline+46,8);
    								xc[12] ='\0'; yc[8]= '\0'; zc[8] = '\0';
    													
    							ca[n].x = atof(xc); ca[n].y = atof(yc); ca[n].z =atof(zc);
    								n++; 
    							}
    						}
    			
    			}
    			
    			
    		Dist: 	
    			for(i=0;i<n;i++)					
    	// Make number 4 to n for N*N distance matrix or it will be N*4 matrix
         {
            for(j=i+1;j<n;j++)
            {
               dist = sqrt((ca[i].x -ca[j].x)*(ca[i].x-ca[j].x) +  (ca[i].y -ca[j].y)*(ca[i].y-ca[j].y) +  (ca[i].z -ca[j].z)*(ca[i].z-ca[j].z));
               fprintf(fp,"%d:%.3lf ", featureNum,dist);
               featureNum++;
               
            }
           }
           fprintf(fp,"# %s %s", pdbName,seqn);
    			printf(".");
    			printf("\nDone with one vector %d pdb ID is %s Feature Number is %d",posn,pdbName,featureNum);
    			if(maxF<featureNum)
    				maxF = featureNum;
    				avgF = avgF+ featureNum;
    			posn++;
    			if(posn>C1000*1000)
    			{
    				sleep(200);
    				C1000++;
    			}
    			//scanf("%c",st2);
    			fclose(f);                       // I am closing BOth files here
    			fclose(fpro);
    			fprintf(fp,"\n");
    		}

  8. #8
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Code:
    			if((fpro = fopen(st4,"r"))==0)              
    			{
    				printf("Profile File Not found %s", st4);
    				goto loop;
    				
    				}
    					if((f = fopen(st,"r")) == 0)
    			{ printf(" File %s is not in Directory", &st); 
    				goto loop; 
    			}
    			else
    If you ever execute this code, then you will get a handle-leak, because fpro is still open.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  9. #9
    Registered User
    Join Date
    Apr 2009
    Posts
    5

    Unhappy

    I closed the fpro file. I still get the same problem after 2387 iterations




    388 [main] a 5060 _cygtls::handle_exceptions: Error while dumping state (pro
    bably corrupted stack)
    Segmentation fault (core dumped)

  10. #10
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    And there isn't any chance that for example "n" gets out of range for the arrays you have?

    There are no memory allocations and nothing else that seems to "leak", so there's no reason why this should fail.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  11. #11
    Registered User
    Join Date
    Feb 2009
    Posts
    278
    GASP! Is that a "goto" I see???

  12. #12
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by Bladactania View Post
    GASP! Is that a "goto" I see???
    Kernel hackers are allowed to do that
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  13. #13
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Code:
    strcpy(pdb_list[pdbn],pdbName);
    					pdbn++;
    If this is a stack variable and you have enough of them (I see a few cantidates), you will run out of stack eventually.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  14. #14
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by MK27 View Post
    Code:
    strcpy(pdb_list[pdbn],pdbName);
    					pdbn++;
    If this is a stack variable and you have enough of them (I see a few cantidates), you will run out of stack eventually.
    Good catch.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  15. #15
    Registered User hk_mp5kpdw's Avatar
    Join Date
    Jan 2002
    Location
    Northern Virginia/Washington DC Metropolitan Area
    Posts
    3,801
    Code:
    while(!feof(rf))
    {
        fgets(line,380,rf);
    
        ...
    
    while(!feof(fpro))
    {
        fgets(proLine,380,fpro);
    
        ...
    
    while(!feof(f))
    {
        fgets(pline,380,f);
    
        ...
    Avoid using feof in this manner to control your loops. A better choice is to use the return result directly from the fgets call:
    Code:
    while(fgets(line,380,rf) != NULL)
    {
        ...
    
    while(fgets(proLine,380,fpro) != NULL)
    {
        ...
    
    while(fgets(pline,380,f) != NULL)
    {
        ...
    "Owners of dogs will have noticed that, if you provide them with food and water and shelter and affection, they will think you are god. Whereas owners of cats are compelled to realize that, if you provide them with food and water and shelter and affection, they draw the conclusion that they are gods."
    -Christopher Hitchens

Page 1 of 2 12 LastLast
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Newbie homework help
    By fossage in forum C Programming
    Replies: 3
    Last Post: 04-30-2009, 04:27 PM
  2. Adventures in labyrinth generation.
    By guesst in forum Game Programming
    Replies: 8
    Last Post: 10-12-2008, 01:30 PM
  3. Inventory records
    By jsbeckton in forum C Programming
    Replies: 23
    Last Post: 06-28-2007, 04:14 AM
  4. Post...
    By maxorator in forum C++ Programming
    Replies: 12
    Last Post: 10-11-2005, 08:39 AM
  5. Possible circular definition with singleton objects
    By techrolla in forum C++ Programming
    Replies: 3
    Last Post: 12-26-2004, 09:46 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21