Move to specified line in CSV file

Hi,

I have a csv file and I want to move to a specific line and then begin overwriting data from that line onwards.

I know from certain variable the specific line I want to move to.

For example,

Say I had a csv file with 5 lines of data, I may wish to move to line 3 and then begin to re-write data on line 3 and onwards.

simple eg) csv file
1,3,4
2,3,5,6,7
1,2,3
3,4,5
6,7,8

Move to 3rd line (1,2,3) and begin to edit from there onwards. Everything on and after line 3 is irrelevant and will be overwritten so it can as well just be deleted from that point onwards.

So, again. I do not know what will be the data on line 3 (in this example) but I do know that line 3 is where I want to start editing from.

Any ideas/suggestions?

many thanks!!

The easiest (and safest) thing to do is to
- read the lines you want to keep, then write them to a new temp file
- write your new data to the new temp file.

When you're done, then
- rename the old file to say file.bak
- rename the temp new file to whatever you want

The problems with your approach are
- what happens in case of program/machine failure - your file is partly trashed
- what happens if you write less data than was already there - you have trash at the end of the file.

Thanks for your reply.

I take your points.

However, I do not want to keep a temp file since this particular CSV file is likely to be pretty big (up 200,000+ lines) and take up some time and space.

problems
- You mention about a program failure. What would happen in this case to the data already written to the file? Surely I could move back to a specfifc line and begin to re-write?
- To overcome the problem with trash at the end of the file. Could I not move to a specified line (which I know to be less than the end of file position) and then delete everything after this point before I begin to re-write?

> However, I do not want to keep a temp file since this particular CSV file is likely to be pretty big (up 200,000+ lines) and take up some time and space.
A 20MB file for a program running on a machine with processors in the GHz range, with RAM in the GB range and disk space also in the GB range is going to be no more than a blip in the grand scheme of things.

> - You mention about a program failure. What would happen in this case to the data already written to the file?
Maybe the file system sees the file as corrupt (or at least inconsistent) and just deletes it.

> Surely I could move back to a specfifc line and begin to re-write?
In theory, you can record each tellg() and then read the corresponding line. Later, you can use either seekg() or seekp() to go back to a particular line and then start reading or writing.

> Could I not move to a specified line (which I know to be less than the end of file position) and then delete everything after this point before I begin to re-write?
Would you know how much to "delete", by say overwriting it with something meaningless like newlines or spaces?
Some file systems provide a "truncate" feature, which may just work for you (but it isn't portable).

O_o

How important is your data?

Very Important: Don't even try to update files "in place"; let your file system and hardware do their jobs by writing out a new file.

Otherwise: Feel free to do an "in place" update.

Soma

Thanks both for the advice and help!

I need to re-think my approach: Is there no way to save your files in c++? If I could 'save' them periodically then that would suffice and in the event of a failure I could always have certainty as to where I'm at.

So, what happens then when we write to a file? Is it 'saved' straight away as soon as the command is given or does it all get written when the file is closed??

thanks again

It is not 'saved' until the filestream is closed

Quote:

---

So, what happens then when we write to a file? Is it 'saved' straight away as soon as the command is given or does it all get written when the file is closed??

---

The data is not actually written until one of several conditions is met.

1. You manually flush the stream with fflush() or endl;

2. The output buffer becomes full.

3. The file is closed.


Jim

ostream::flush - C++ Reference
This ensures that any data you have output is consistent with the environment.

fsync(2) - Linux man page
This ensures that the environment flushes any data to the real storage.
It's only after such a call (or closing the file) that the data is likely(*) to be permanently stored.

If you're just appending data to the file, you should probably try just opening the file in "append" mode, write what you need to and then close it again.



(*) Hard disks also have caches, but you have to assume that the OS device drivers correctly manage these.

Thanks again for replies and advice; it is much appreciated!

It seems as if the close function just won't do for me... because it is likely that the program will be closed in a proper manner and not abruptly terminated.

That being said, I have tested this and will require to overwrite from a specific point in the file.

Thus far, I am able to move in the CSV file to the exact location from where I want to overwrite by using:

Code:

---

                                  f.seekp(ios::beg);
                                  for(long int i = 0; i < temp_v; ++i)
                                  {
                                          f.ignore(numeric_limits<streamsize>::max(),'\n');
                                  }

---

Doing so gets me to the line that I now want to start overwriting from. But now, my question is, how do I set the stream pointer to this position???

thanks

tellg() tells you where the 'get' (reading) pointer has got to.
seekp() will then move the 'put' (writing) pointer to that position.

Hi thanks,

Basically this is what I have. Note: f is declared as fstream

Code:

---

f.open(Name_v, ios::binary|ios::in|ios::out);
long int posl;

f.seekp(ios::beg);
for(long int i = 0; i < temp_v; ++i)
{
  f.ignore(numeric_limits<streamsize>::max(),'\n');
}
posl = f.tellp();
f.seekp(posl);

---

When I check the file. It does not overwrite from the point where it moves to (temp_v).
Instead it just continues to append from the end of the file as if I didn't do any of the moving to a specific line stuff.

Any ideas?

cheers

You must use the correct open mode if you want to overwrite the information. The default (out) will always append to the end of the file. See this link for the open modes: ofstream.open. You may want to try ate.

Jim

O_o

Just in case you've not been made aware; this is not going to be as easy as your code implies that you think it will be done.

To update a file "in place" means lots of bookkeeping and manual buffering.

For example, with a simple naive attempt garbage will be left on the next line when the updated line has fewer bytes than the old line.

An alternative example, with a simple naive attempt replacing a line will consume parts of valid data when the updated line has more bytes than the old line.

[Edit]
With a note because your posts imply you think there is a performance issue:

You'll have to rewrite every byte of the file after an updated line that has fewer or more bytes than the old line.

In other words, if you need to update the first line of the file with data that has only has a one byte difference you'll still have to read and rewrite the entire file to account for that difference from the point that difference occurs.
[/Edit]

[Edit]
Also, have you considered just journaling updates and writing multiple updates at once on occasion?

So, you would write the updates lines to a new file and read the file along with the old file.
[/Edit]

Soma

how is this to be used? are you planning to implement on a single machine with a known capability? If so you 'could' do that based on the scope of that machine, and forget about some of the other stuff, i mean you could maybe buffer the whole data and parse it out, but that is still pretty poor in my opinion, and if the data is read from a network connection, then bin that idea. I think you have seen some of the best suggstions, temp files so the source data is only read etc.

Thanks for suggestions!!

Okay, so how about this:

- I copy the file content in to memory (up until the line that I specify).
- Delete the file (can i do this independent of platform?)
- open up the file again of the same name.

I dont want to rename because I am sending program over a network and it gets complicated. I dont want to open a new file because I want to continue to use the file handle I already have created for collecting data.

Any ideas or examples for me to do this?
Or alternative ideas?
the easier the better!!

thanks

Quote:

---

Originally Posted by strokebow

- Delete the file (can i do this independent of platform?)

---

There's the remove function from <cstdio>, but...

Quote:

---

Originally Posted by strokebow

- open up the file again of the same name.

---

Why not just use an open mode that truncates the file?

Thanks very much laserlight

I like this idea of truncating the file. How would I go about that?

thanks

You could refer to jimblumberg's post #13.

Quote:

---

Originally Posted by jimblumberg

You must use the correct open mode if you want to overwrite the information. The default (out) will always append to the end of the file. See this link for the open modes: ofstream.open. You may want to try ate.

Jim

---

Thanks for reply.
Ok, so I use ios::ate instead of ios::out

But I am not sure how that will allow me to truncate the file?

So you read the portion of the file that you want to preserve using an ifstream. Now close this ifstream, and using the same file name create an ofstream. If you don't provide the optional third parameter to your ofstream creation it will erase the file contents when it open the file. Now just write the old information to this ofstream.

Code:

---

ifstream ins("MyFileName); // Open a file for reading.
// Check that it opened correctly and read the information you want to preserve into vectors.

ins.close(); // Close the file.
ofstream out(MyFileName); // Open a file for writing, erase current contents.
// Check that it opened correctly and write the new information.

---

Jim

Did you read the page that was linked?

Quote:

---

Originally Posted by jimblumberg

So you read the portion of the file that you want to preserve using an ifstream. Now close this ifstream, and using the same file name create an ofstream. If you don't provide the optional third parameter to your ofstream creation it will erase the file contents when it open the file. Now just write the old information to this ofstream.

Code:

---

ifstream ins("MyFileName); // Open a file for reading.
// Check that it opened correctly and read the information you want to preserve into vectors.

ins.close(); // Close the file.
ofstream out(MyFileName); // Open a file for writing, erase current contents.
// Check that it opened correctly and write the new information.

---

Jim

---

Cheers Jim. I'll give that a go now!

Thanks also laserlight for the heads up

Also, is it really necessary to open the ifstream file?

Couldn't I just store the data in to memory?

I thought the information was contained in a file.

Quote:

---

I have a csv file and I want to move to a specific line and then begin overwriting data from that line onwards.

---

How do you intend to read the information from this file? An ifstream reads from an existing file.

Jim

How would you plan to obtain and store the data from the existing file without opening it for reading?

Points taken :-)

Hey, thanks to everyone.

Got it sorted. Works nicely. Ended up implementing as Jim suggested. Cheers Jim!!!!