File output

This is a discussion on File output within the C++ Programming forums, part of the General Programming Boards category; I am writing output to a text file. However these files are getting quite large. I am writing information line ...

  1. #1
    Registered User
    Join Date
    Jul 2005
    Posts
    41

    File output

    I am writing output to a text file. However these files are getting quite large. I am writing information line by line and finding files get to about 1MB. How can I minimise the size of these files. I have thought that changing the seperation may work but uses the same space. How can I reduce the size of these files? I dont want the files to be hard to read by user either. Any idea?

    Files are something like this

    Position A
    44.44
    434.34
    343.434
    43.43
    4343.434
    Position B

    Thanks

  2. #2
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Portugal
    Posts
    7,456
    Well, if these are text files you want to be read by an human, the only way to reduce the size of the file is to write less to it.

    You may need to provide more details here. Are all those numbers needed?

    "Position A" could probably be reduced to "Pos. A" or even further to "A". But without information on what those numbers mean, I don't think anyone can help you much.
    The programmer’s wife tells him: “Run to the store and pick up a loaf of bread. If they have eggs, get a dozen.”
    The programmer comes home with 12 loaves of bread.


    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  3. #3
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,532
    Well whether you write it as
    44.44
    434.34

    or
    44.44 434.34

    Isn't going to make that much of a difference.
    Anything else is really going to impact on readability.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  4. #4
    Registered User whiteflags's Avatar
    Join Date
    Apr 2006
    Location
    United States
    Posts
    7,689
    Well, since you tried changing how the file is represented, maybe you should consider whether or not everything in the file needs to be there. I mean, the file you showed us is meaningless to me, maybe you're logging too much.

    To be honest, one megabyte is not that bad. I could easily produce a megabyte document in Word. If you really want smaller files, consider using a compression algorithm/tool, and decompressing the file when you read it. You can learn one such algorithm from the site: http://cprogramming.com/tutorial/com...y/huffman.html
    but there are many of these.

  5. #5
    Registered User
    Join Date
    Jul 2005
    Posts
    41
    Thanks guys

    I have tried minimising the characters in the file. I am away from my office machine with the code on it so could not post an actual example. Yes a tab character "\t" uses the same space as newline "\n". I dont think compression is the way forward...its actually scientific research and I am generating thousands upon thousands of these files so they add up really.

    I guess I am stuck.

  6. #6
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Portugal
    Posts
    7,456
    Then do as Citizen suggested. Store those files compressed and provide the user with a "reader" program that decompresses the file and displays the contents.
    The programmer’s wife tells him: “Run to the store and pick up a loaf of bread. If they have eggs, get a dozen.”
    The programmer comes home with 12 loaves of bread.


    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  7. #7
    pwns nooblars
    Join Date
    Oct 2005
    Location
    Portland, Or
    Posts
    1,094
    Our access_log and error_log generate way more than 1mb per hour depending on how many hits we get and if we are currently working on the code and have more debug output turned on. On days when a lot of work is going to PHP, the error_log file can grow to 50m+ easy in a couple hours. 1m is almost nothing by newer standards... You can pick up a 80 gig harddrive used for 20-30 bucks off craigslist. If you are worried about how much logging you are doing. Then get one of those. 80gig ~=80,000 of your log files (no one needs to go into a gig is 1024 meg.. just easier to round).

  8. #8
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,046
    You could gzip the files. If the files are quite repetitive it can save a lot of space.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 3
    Last Post: 03-04-2005, 01:46 PM
  2. Unknown Memory Leak in Init() Function
    By CodeHacker in forum Windows Programming
    Replies: 3
    Last Post: 07-09-2004, 09:54 AM
  3. archive format
    By Nor in forum A Brief History of Cprogramming.com
    Replies: 0
    Last Post: 08-05-2003, 07:01 PM
  4. System
    By drdroid in forum C++ Programming
    Replies: 3
    Last Post: 06-28-2002, 10:12 PM
  5. Simple File Creation Algorithm
    By muffin in forum C Programming
    Replies: 13
    Last Post: 08-24-2001, 03:28 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21