Reading a buffer line by line, while using system read

This is a discussion on Reading a buffer line by line, while using system read within the C Programming forums, part of the General Programming Boards category; Hello, the goal of my programe is to read from a log file "A" (comma delimited text file) which is ...

  1. #1
    Registered User
    Join Date
    May 2008
    Posts
    4

    Reading a buffer line by line, while using system read

    Hello,
    the goal of my programe is to read from a log file "A" (comma delimited text file) which is generated when events occurs while the system is running and to extract some fields from that log file "A" and create a comma delimited text file "B" consists of the extracted fields from the parent log file "A".
    I have opted the way of "Reading the file in to a buffer" after some seconds. using system I/O "read (filehandle, buffer, buffersize) "
    and then extracting the fields from the buffer by getting one line at a time and process/extract the required fields from the line and putting the extracted fields in to the destination file (comma delimited).

    I am facing difficulty while reading the buffer line by line so i can extract the required fields from the each line one by one from the buffer when it got filled in a read operation.

    Please help me for this, and also comment on effeciency of the programe implemented in this way.

  2. #2
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Exactly what difficulty are you having, and what exactly are you doing (post the code).

    To comment on the efficiency, we would really have to see the code - there's no way I can tell if you are doing it more or less efficiently than for example using fgetc().

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  3. #3
    Registered User ssharish2005's Avatar
    Join Date
    Sep 2005
    Location
    Cambridge, UK
    Posts
    1,682
    Yes it is difficult to just say, where you are making a mistake or perhaps to talk about the efficiency. But here is an easy way dealing this.

    1. When you say records, do see any inconsistency between them in a value like any extra field compared from one records to an another.
    2. Cant you read your records from the file, like in chunks and fill it into structure and then access the data members of that structure?
    3. How many fields do you have in a record?
    4. You can also use fread and fwrite, indeed these two function trap the kernel to call read and write respectively.

    ssharish

  4. #4
    Registered User
    Join Date
    May 2008
    Posts
    4
    Dear mats and ssharish, thanks for the quick reply,
    1. When you say records, do see any inconsistency between them in a value like any extra field compared from one records to an another.
    the fields are consistant in all records that is all records have the same number and type of fields
    2. Cant you read your records from the file, like in chunks and fill it into structure and then access the data members of that structure?
    I am trying to do so that get the data in chunk and fill the required fields in the structure
    3. How many fields do you have in a record?
    Each and every record consists of 23 fields, but i am interested in only first 4 to 6 fields that is why i dont want to traverse through all other fields, some times file size reaches to 80MB and sometimes it remains in 600KB, it actually depends on the events occur in the system, i am getting difficulty in fgets (perhaps memory leak or overflow).

    Code:
    #define BUFFSIZE 512
    char Rd_buff[BUFFSIZE];
    int main(int argc, char* argv[])
    {
    int inhandle, outhandle, seek, C_read=0, T_read = 0;
    char linestore[512];
    char *ptr;
    ptr = Rd_buff;
    inhandle =  open("sample.txt", O_RDONLY);
    seek = lseek(inhandle, C_read, SEEK_SET);
    outhandle =  open("Destination.txt", O_CREAT | O_WRONLY | O_APPEND, S_IWRITE);
    while( (C_read = read(inhandle, Rd_buff, BUFFSIZE)) > 0){	
    T_read = T_read + C_read;
    seek = lseek(inhandle, T_read, SEEK_SET);  
    
    		while ( (fgets(linestore, sizeof(linestore), ptr) ) != NULL){
    		printf("Line by Fgets  = %s", linestore);
    		}       
    
    write(outhandle, Rd_buff, C_read);
    }}
    Last edited by Hammad Saleem; 05-27-2008 at 03:38 AM.

  5. #5
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Code:
    		while ( (fgets(linestore, sizeof(linestore), ptr) ) != NULL){
    This line does not work - ptr is a char pointer that points to your buffer, not a pointer to a FILE struct. There is no way you can make this work in the way you have it right now.

    If you want to use stdio.h functions, you need to open the file using fopen, and use the pointer to that file. If you want to use the low-level read functions, you need to write your own line-parsing code (and bear in mind that lines may straddle whatever block-size you choose to use). You will probably find it easiest to write a function that reads a character from the buffer, and if the buffer is empty, read in another buffer-full (also bear in mind that the buffer may not be completely full on the last attempt).

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  6. #6
    Registered User
    Join Date
    May 2008
    Posts
    4
    What is the effecient way of doing such work either using low-level functions or use of standard functions like fopen, fread etc?
    while using the fopen & fread a stream is established, is it effecient to access the file for each character further i want ot access after some time delay for example after each 30 seconds, that is why i adopted for the buffer so that i hav the chunk of data and i can then process it.

  7. #7
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    There is a principle in programming called "Don't optimize prematurely".

    Get the code (all parts of it) working correctly first, then spend time making it run faster.

    The EASIEST solution is to just open the file with fopen() and use fgets() to read the file.

    Don't quite understand what you are asking for when you talk about time-delay - can you explain what you are trying to achieve?

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  8. #8
    Registered User
    Join Date
    May 2008
    Posts
    4
    thank you dear for the help i will come again after writing the function ( reading character from the buffer) while using system calls for filling the buffer.
    by time delay i mean "lets suppose that 10 events are recorded per second in the system then i am of view that i will get data of 100 events in the buffer after 10 seconds so read system call will read the file after 10 seconds each time ".
    Mean while any good and simple tutorial for Debugging? i am using Visual studio 6.0 for the development.

  9. #9
    Woof, woof! zacs7's Avatar
    Join Date
    Mar 2007
    Location
    Australia
    Posts
    3,459
    > Mean while any good and simple tutorial for Debugging? i am using Visual studio 6.0 for the development.
    Start by upgrading the compiler

    I hear VS Express 2008 is good (and it's free), and comes with a nice debugger.

  10. #10
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by Hammad Saleem View Post
    thank you dear for the help i will come again after writing the function ( reading character from the buffer) while using system calls for filling the buffer.
    by time delay i mean "lets suppose that 10 events are recorded per second in the system then i am of view that i will get data of 100 events in the buffer after 10 seconds so read system call will read the file after 10 seconds each time ".
    Mean while any good and simple tutorial for Debugging? i am using Visual studio 6.0 for the development.
    If it's console application, you could do something like this:
    Code:
    while(some_stop_condition)  // E.g. user input, number of loops or some other "way" to stop the app. 
    {
        displayStuff();
        Sleep(10000);   // Wait for 10s. 
    }
    I don't see why you need to mess about with read/write rather than stdio.h functions when you have such a slow delivery of data tho' - it makes absolutely no sense.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. "sorting news" assignment
    By prljavibluzer in forum C Programming
    Replies: 7
    Last Post: 02-06-2008, 06:45 AM
  2. Replies: 4
    Last Post: 06-13-2005, 10:03 AM
  3. Read only one line using seekg
    By RedZippo in forum C++ Programming
    Replies: 3
    Last Post: 03-31-2004, 11:10 PM
  4. Validating the contents of a char buffer
    By mattz in forum C Programming
    Replies: 3
    Last Post: 12-09-2001, 06:21 PM
  5. How do I read file line by line?
    By Blurr in forum C Programming
    Replies: 1
    Last Post: 09-22-2001, 01:32 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21