I'm writing a tool which processes data. The amount of data can be huge, also multiple files.
So what I need to do is the following: 1) read from all files, for example, 1024 samples, 2) process the 1024 samples from all files and 3) repeat (go back to 1)
To optimize this I thought threads could help. So the file IO is in one thread and the processing in an other one. Which means the io thread makes sure there are always enough samples available for processing so the processing can continue without waiting.
The questions I have are: If using pthreads is the io not blocking all other threads ? and should I use 1 thread per file or should I deal with all files in 1 thread.
If you have other ideas / suggestions how to optimize this I would really like to hear them!