Thread: Performance issue!

  1. #1
    Registered User
    Join Date
    Dec 2005
    Posts
    136

    Performance issue!

    Hi All,

    In my C program i am using very large file(approx 400MB) to read parts of it frequently. But due to large file the performance of the program goes down very badly. It shows high I/O usage and I/O wait time.
    My question is, What are ways to optimize or tune I/O on linux or how i can get better performance?

    I am using
    Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
    2.6.9-42.EL, x86_64 GNU/Linux
    S_ccess is waiting for u. Go Ahead, put u there.

  2. #2
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    How are you reading the file, and how is the file opened.

    Is it a local file, or do you access it over the network?

    To me, 400MB is not a huge file (yes, of course, typing 400MB of data in by hand would take a long time [at least if you can't copy'n'paste most of it]). If you have a few gigabytes of memory, it should be possible to cache that file (or the regions that are frequently accessed at least).



    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  3. #3
    Registered User
    Join Date
    Dec 2005
    Posts
    136
    reading the file with read function.
    Actually the files are of database. One file is there which keeps the index information and based on the index (Using binary search method to search index with the help of key value) data is fetched from data file.

    Size of index file is around 100 MB and size of data file is around 300 MB.
    Yes it is a local file.

    When i see there is high I/O wait then reading goes damn slow. Is there any way through which i can prevent I/O(swap In/Out) for these files. Or tune my machine?
    S_ccess is waiting for u. Go Ahead, put u there.

  4. #4
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    read() or fread()?
    How do you open the file? (fopen() or open(), and what arguments?)

    Is there only one process accessing the file, or many processes?

    Are you mixing reads and writes, or only doing reads?

    Do you open & close the file for each request, or is there "one open at start of day, and close at end of day"?

    If you have enough free memory in the system, it should automatically cache the file, unless you have specifically requested it should NOT.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  5. #5
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    Hardware speed up's include:

    1) Use a RAID drive configuration
    2) Have the data on a physically separate HD from the system files.
    3) Use a faster drive for the data drive - SSD if you can justify the expense

    Software:
    1) Use fread() in the natural big chunk size that matches well with the architecture of
    that particular system.

    2) Don't use linked lists or other largely sequential data structures- use arrays, instead.

    3) Use malloc and slower functions (strcmp, strlen, sizeof, etc. sqrt), as little as possible

    4) Keep your data that you're processing, together, together in your program. Locality of reference, is critical to avoid flushing the cache memory.

    5) Look to your algorithms - that's the biggest improvement you can make, if it's non-optimal. Don't optimize the code, before the algorithm is really right.

    6) Profile and test the program and functions therein. Any program that has a lot of I/O is likely to have the I/O as the bottleneck. So when you are reading data, read big chunks of it, when you're writing data, again - big chunks. Hopefully, again on separate physical HD's (not just different logical drives).

    The worst thing is to have read 100 bytes here, write 50 bytes over here, and the other 50 bytes in another file, over there, then reading another 100 bytes, especially on the same physical hard drive.

    A recent program of mine processes 75,000 records, in two data files of 28 Megs, including a multi-level merge of some 7,400 records. For the various sorting, I thought I'd use Quicksort from the C standard library.

    Because the sorting is in small classification groups, Quicksort was slower than a bubblesort, in my tests. <shock> I love Quicksort, but for small groups it can't match selection or bubblesort. (no, the data was not partially sorted)

    The reason, and the reason that program is finished in about 5 seconds, is because it takes full advantage of the HD buffer and arrays. Big reads, big writes, and it uses fscanf() for much of the data input into the array of structs. Way faster than fgets(), or something like that.

    You may want to check your HD also, and just see that in fact, it's giving you the sustained throughput that it should be. Case in point, I recently had to back up an entire HD, so I hooked it up to the mobo, using the ATA cable going to/from the optical drive. The HD's were identical in size and make, with a large disk buffer.

    The data *crawled*, from one drive to the other! The reason was the backup HD had been connected to the slave connector of the ATA cable, and it was set for the slowest device on the ATA cable - an 8X optical drive. Incredibly slow! I haven't seen a HD move data that slow since x86-80286 cpu days.

    Another factor was that these HD's apparently use their large disk buffer to act like they have a fine throughput rate. Once you're past the data in the buffer, it's *much* slower.

    Good luck, sorry to make this so long.

  6. #6
    Registered User
    Join Date
    Nov 2006
    Location
    japan
    Posts
    126

    Wink check phisical place or your files

    You might want to check also the place of your files.
    Make sure that all files that are frecuently read are in the same place. Not only in the same HD, I would put them in the same folder. And if you can next to the executable.

    Also ... it depends on your algorithm but, Maybe you should consider using fseek() or equivalents to go to the exact position of your file. I think fseek is a little bit faster.
    Mac OS 10.6 Snow Leopard : Darwin

  7. #7
    Registered User
    Join Date
    Oct 2008
    Posts
    1,262
    Quote Originally Posted by Adak View Post
    3) Use malloc and slower functions (strcmp, strlen, sizeof, etc. sqrt), as little as possible
    Since when is sizeof slow? And since when is it a function?

  8. #8
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by nacho4d View Post
    You might want to check also the place of your files.
    Make sure that all files that are frecuently read are in the same place. Not only in the same HD, I would put them in the same folder. And if you can next to the executable.

    Also ... it depends on your algorithm but, Maybe you should consider using fseek() or equivalents to go to the exact position of your file. I think fseek is a little bit faster.
    Placement on hard-disk and location in relation to the executable is unlikely to affect thing if the files involved are 400MB worth of data.

    Adak makes several good suggestions, although only some of them deal with I/O optimization.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  9. #9
    Registered User
    Join Date
    Sep 2006
    Posts
    8,868
    Maven, there is a much faster way (possibly - depends on your data), to do your search, than to use the nice binary search technique.

    I'm a big fan of binary search, but I have seen a better technique, also. Actually, I was on the losing end of a debate on what was the fastest searcher, so I tested it myself.

    If you send me a link to some representative data, I'll see if this indexing search can work faster for you, as well.

    It can be *much* faster, but that depends on the data and how this index can be smartly built for that data.

  10. #10
    Algorithm Dissector iMalc's Avatar
    Join Date
    Dec 2005
    Location
    New Zealand
    Posts
    6,318
    Quote Originally Posted by maven View Post
    reading the file with read function.
    Actually the files are of database. One file is there which keeps the index information and based on the index (Using binary search method to search index with the help of key value) data is fetched from data file.

    Size of index file is around 100 MB and size of data file is around 300 MB.
    Yes it is a local file.

    When i see there is high I/O wait then reading goes damn slow. Is there any way through which i can prevent I/O(swap In/Out) for these files. Or tune my machine?
    Do you read at least the index file in only once and hold it in RAM? 100MB isn't that much.
    Have you considered using a B-Tree instead of a binary tree? These are what directory structures use because they minimise the number of disk accesses or updates required.
    Is the data or index file something that would compress easily? If you can even halve the space of the data or index file you could significantly reduce the IO.
    If you dont want to hold the whole 400MB in RAM at once, how about having a cache of say the 10 most recently accessed chunks of the file? Even caching 5% of it could make a huge difference depending on data access patterns.
    Finally, have you considered memory-mapped IO?
    Last edited by iMalc; 03-16-2009 at 12:31 PM.
    My homepage
    Advice: Take only as directed - If symptoms persist, please see your debugger

    Linus Torvalds: "But it clearly is the only right way. The fact that everybody else does it some other way only means that they are wrong"

  11. #11
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Hey -- just a shot in the dark, but since this is linux/GNU: if the file size is fairly constant you could mmap()
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  12. #12
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by MK27 View Post
    Hey -- just a shot in the dark, but since this is linux/GNU: if the file size is fairly constant you could mmap()
    Whether it's mmap()'d or not, the kernel should be trying to cache as many file blocks as possible.

    Sequentially reading from a file should be fast, unless the filesystem is totally fragmented. I suspect the problem comes from seeking like crazy. If you have a gig of RAM, then the whole file might end up in cache, but if you have considerably less than that, seeking is going to create a serious hit, no matter how you access the file.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  13. #13
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Quote Originally Posted by brewbuck View Post
    Whether it's mmap()'d or not,
    Right...a file in RAM is a file in RAM. Just ignore me
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  14. #14
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by MK27 View Post
    Right...a file in RAM is a file in RAM. Just ignore me
    Well it does make a difference, just not one that I think would be significant to performance. When a file is read normally, the blocks are cached in the "buffer cache." If the file is mmap()'d, the blocks are cached in the page cache, and associated with the file through backing store. So the specific cache which is used is different, but it would be a bug if the performance was signficantly different.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  15. #15
    Crazy Fool Perspective's Avatar
    Join Date
    Jan 2003
    Location
    Canada
    Posts
    2,640
    Binary search on disk is very slow. In many cases, a linear scan of your data may be faster than binary search. Disk drives are fast for sequential reads and slow for random access. Binary search has way too much random access for disk based algorithms.

    400mb is tiny, buffer it in memory.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. float calculation issue
    By George2 in forum C# Programming
    Replies: 1
    Last Post: 05-26-2008, 04:56 AM
  2. Performance and footprint of virtual function
    By George2 in forum C++ Programming
    Replies: 8
    Last Post: 01-31-2008, 07:34 PM
  3. File map performance
    By George2 in forum C++ Programming
    Replies: 8
    Last Post: 01-04-2008, 04:18 AM
  4. Binary Search Trees Part III
    By Prelude in forum A Brief History of Cprogramming.com
    Replies: 16
    Last Post: 10-02-2004, 03:00 PM
  5. inheritance and performance
    By kuhnmi in forum C++ Programming
    Replies: 5
    Last Post: 08-04-2004, 12:46 PM