Thread: C file-manipulation in Windows

> Do you happen to know what the overhead is? For a 100 MByte file, how big is the ECC file(s) that go with it?
You can choose your recovery size.

Code:

$ par2create -c1 p2c_test1.par2 kubuntu-20.04.2.0-desktop-amd64.iso 
$ par2create -c100 p2c_test2.par2 kubuntu-20.04.2.0-desktop-amd64.iso 
$ ls -lh kubuntu-20.04.2.0-desktop-amd64.iso p2c*
-rw-rw-r-- 1 sc sc 2.5G Apr  3 14:56 kubuntu-20.04.2.0-desktop-amd64.iso
-rw-rw-r-- 1 sc sc  40K Jun  7 03:59 p2c_test1.par2
-rw-rw-r-- 1 sc sc 1.3M Jun  7 03:59 p2c_test1.vol0+1.par2
-rw-rw-r-- 1 sc sc  40K Jun  7 04:02 p2c_test2.par2
-rw-rw-r-- 1 sc sc 1.3M Jun  7 04:02 p2c_test2.vol000+01.par2
-rw-rw-r-- 1 sc sc 2.6M Jun  7 04:02 p2c_test2.vol001+02.par2
-rw-rw-r-- 1 sc sc 5.2M Jun  7 04:02 p2c_test2.vol003+04.par2
-rw-rw-r-- 1 sc sc  11M Jun  7 04:02 p2c_test2.vol007+08.par2
-rw-rw-r-- 1 sc sc  21M Jun  7 04:02 p2c_test2.vol015+16.par2
-rw-rw-r-- 1 sc sc  41M Jun  7 04:02 p2c_test2.vol031+32.par2
-rw-rw-r-- 1 sc sc  47M Jun  7 04:02 p2c_test2.vol063+37.par2

The minimal single block is like 0.05% of the data size.

Originally Posted by Salem

> Do you happen to know what the overhead is? For a 100 MByte file, how big is the ECC file(s) that go with it?
You can choose your recovery size.

8< [8< = Scissors = text snipped.]

The minimal single block is like 0.05% of the data size.

I've tried reading such documentation as there is, but it isn't clear to me. One of my priorities is that I need to deal with disk corruption, which can result in the loss of blocks of data OR just a byte here and there.

This lead me to conclude, some time ago, that the best ECC system can't beat just storing a copy of the data under ECC protection ... provided that copy is not corrupt, and does not become corrupt, of course!

To me, Parchive resembles the way a RAID array works. Is that likely to offer what I'm after? I'm not sure.

But thanks for all the info nonetheless! It's all grist for the mill, as they say.

Here's an example.

1. Create a minimal recovery set for some file.

Code:

$ cp 20200903_055944.mp4 dummy.mp4
$ par2create -c1 p2c_test1.par2 dummy.mp4 
Block size: 85456
Source file count: 1
Source block count: 2000
Recovery block count: 1
Recovery file count: 1

Opening: dummy.mp4
Computing Reed Solomon matrix.
Constructing: done.
Wrote 85456 bytes to disk
Writing recovery packets
Writing verification packets
Done

2. Some major league bit-rot. This could be anywhere within the file, not necessarily contiguous damage either.

Code:

$ dd if=/dev/urandom of=dummy.mp4 count=100 conv=notrunc
100+0 records in
100+0 records out
51200 bytes (51 kB, 50 KiB) copied, 0.00818186 s, 6.3 MB/s

3. Check it.

Code:

$ par2verify p2c_test1.par2
Loading "p2c_test1.par2".
Loaded 4 new packets
Loading "p2c_test1.vol0+1.par2".
Loaded 1 new packets including 1 recovery blocks
Loading "p2c_test1.par2".
No new packets found

There are 1 recoverable files and 0 other files.
The block size used was 85456 bytes.
There are a total of 2000 data blocks.
The total size of the data files is 170905919 bytes.

Verifying source files:

Target: "dummy.mp4" - damaged. Found 1999 of 2000 data blocks.
85328 bytes of data were skipped whilst scanning.
If there are not enough blocks found to repair: try again with the -N option.

Scanning extra files:


Repair is required.
1 file(s) exist but are damaged.
You have 1999 out of 2000 data blocks available.
You have 1 recovery blocks available.
Repair is possible.
1 recovery blocks will be used to repair.

4. Fix it.

Code:

$ par2repair p2c_test1.par2
Loading "p2c_test1.par2".
Loaded 4 new packets
Loading "p2c_test1.vol0+1.par2".
Loaded 1 new packets including 1 recovery blocks
Loading "p2c_test1.par2".
No new packets found

There are 1 recoverable files and 0 other files.
The block size used was 85456 bytes.
There are a total of 2000 data blocks.
The total size of the data files is 170905919 bytes.

Verifying source files:

Target: "dummy.mp4" - damaged. Found 1999 of 2000 data blocks.
85328 bytes of data were skipped whilst scanning.
If there are not enough blocks found to repair: try again with the -N option.

Scanning extra files:


Repair is required.
1 file(s) exist but are damaged.
You have 1999 out of 2000 data blocks available.
You have 1 recovery blocks available.
Repair is possible.
1 recovery blocks will be used to repair.

Computing Reed Solomon matrix.
Constructing: done.
Solving: done.

Wrote 170905919 bytes to disk

Verifying repaired files:

Target: "dummy.mp4" - found.

Repair complete.
$ sha1sum 20200903_055944.mp4 dummy.mp4
1b97ee7b451f551e663f80bdda8627f87a9d03bc  20200903_055944.mp4
1b97ee7b451f551e663f80bdda8627f87a9d03bc  dummy.mp4

> To me, Parchive resembles the way a RAID array works. Is that likely to offer what I'm after? I'm not sure.
Each addition raises the bar on failures that eventually result in data loss.
I suppose it's where you choose to draw the line.

> [I keep one master collection and two full copies on different discs.]
Please tell me all 3 copies are not in your house.
If you're going to this much trouble, one of them has to be off-site at all times.
It doesn't matter how many copies you have if your house burns down.

A nearby relative's is probably fine, unless city killer asteroids are in your threat model.
Society is a mess, but hey, my music collection is safe.

Originally Posted by Pattern-chaser

[I keep one master collection and two full copies on different discs.]

Originally Posted by Salem

Please tell me all 3 copies are not in your house.

I'm afraid they are, but you are right to point out how silly this is. If my house burns down, I'm going to get blisters grabbing a USB drive on the way out!