Threads example required

[baxy]

Hi,

well even after all the advice you guys gave me I am still struggling to make a functional piece of code. (thought i could do it on my own but it turns out i cannot) so I am asking for some help if anyone has time to spare on this.

The thing i need to do is to thread a function that does some serious data processing, after which it returns some coordinates (integer values). What i wish to do, is to have these int values after processing in one vector that i can query latter.

As an example consider:

#include <vector>
#include <threads>
#include <mutex>

std::mutex mtx;

void ThreadedFunction (vector <int>& vec){
  mtx.lock()
   int z = 0;
   for(int i=0; i<1000; i++){
      // z = some processing...
      vec.push_back(z);
    }
   mtx.unlock();
}


int main (){


  vector<int> vv;
  vector<vector<int>> v(thr,vv);  // thr = number of threads

/* Launch threads */
   for (int i = 0; i < thr; ++i) 
      threads.push_back(thread(ThreadedFunction, ref(v[i])));

/* Join the threads with the main thread */
   for(auto &t : threads)
      t.join();

/* Join v[0..thr-1]*/

  return 0;
}

I always get some memory corruption error or segmentation fault or something. However, if i use valgrind everything goes fine but only one thread at a time.

How to do this properly so that threads are executed in parallel and vector filled withought clashes?

[jiggunjer]

I suppose this is a good chance to start practicing threads. I'll have a look when I get home later.

[rogster001]

a function that does some serious data processing

So don't bother messing about with threads

[baxy]

So don't bother messing about with threads

ok , but how do i go about utilizing my computer resources then. since now i am only using one thread to do the task and i have 23 left. and since my processing time is long i thought the best way to increase the speed would be to thread the process. unfortunatly input data splitting is not an option. maybe forking would be but shared memory problem seams more difficult that with threads. isn't it??

[rogster001]

If your problem is cpu intensive - depends on power, then give it all of the cpu you can - dont worry about your browser or something wanting it too - if you booked time on a supercomputer to run your program would you worry about doing other things on that machine at the same time?

[EDIT] I see your point, the process may only be on one thread full stop. I dont know the technicalities of how that works, it is probably an OS management thing but by that token I think your program would get all of the resource it can from the OS on that thread, but I am speculating a little there. It would probably consume all the available overall processing time for one core.

[baxy]

Ok after a day of googling i understand that threading is not the way to go. so here is my attempt with forking:

#include <iostream>
#include <vector>

#include <unistd.h>
#include <sys/wait.h>
#include <sys/types.h>
#include <signal.h>



using namespace std;
int main (){

  pid_t pid;
  int status;
  vector<pid_t> pidarray(2);
  vector<int> vec(10,0);
  vector<vector<int>> a(2,vec);
  for (int i = 0; i< 2; i++){
    pid = fork();

    if (pid == pid_t(0)) {
        for(int z = 0; z <  10;z++){a[i][z] = z;}
        cout << "child " << pid << endl;
        exit(0);
    }
    else if (pid > pid_t(0)){
        pidarray[i] = pid;
        cout << "parent " << pid << endl;
    }else{
        cerr << "fsiled\n";
        exit(1);
    }
  }

  for(int i =0; i<2;i++)
     waitpid(pidarray[i], &status, 0);

  cout << endl;
  for(int i =0; i< 2; i++){
   cout << endl;
   for(int z = 0; z <10;z++){cout << a[i][z]<< endl;}
  }


  return 0;

}

so the aim here is to have my a vector filled with two vectors of ints from 0 to 10. The above example does not work since once a process is branched out a new memory footprint is created so when it is terminated the memory and thus my soooo needed numbers are erased. My questio n is how to save them !!

Shouls I post this as a new post since this is not threding anymore??

thnx

[MutantJohn]

void ThreadedFunction (vector <int>& vec){
  mtx.lock()
   int z = 0;
   for(int i=0; i<1000; i++){
      // z = some processing...
      vec.push_back(z);
    }
   mtx.unlock();
}

Hmm... This doesn't look like threading will help you in any way, shape or form. Assuming the mutex locks the vector reference and prevents other threads from writing to it, this will be slower than a single-threaded version. Especially because you're also using

std::vector<T>::push_back

which will dynamically reallocate the vector once you overflow the initial allocation.

I would suggest allocating the entirety of the vector at once. This may require over-allocating but memory is cheap compared to faster CPUs, if you're looking at cost-efficient ways of solving performance bottlenecks. And then you can write to the vector in chunks where each thread is given a range of unique indices that do not conflict with the write indices of other threads. This way you wouldn't have to lock the vector reference either. You would just have to trust that you're not overwriting data previously written by other threads. This also means you'd no longer be using push_back.

[algorism]

It sounds like you might want threads. The idea would be to give each thread a different index into the vector's data (or possibly a C-style in array if a vector doesn't work) so that they won't get in each other's way. Then you don't need the mutex and you can get the benefit of multiple threads.

size_t size = 1000;
vector<int> v;              // preallocate elements
v.reserve(size);            // preallocate elements
size_t num_threads = 8;
size_t chunksize = size / num_threads;
for (size_t i=0; i<num_threads; ++i) {
    size_t i_end = (i+1)*chunksize > v.size() ? v.size() : (i+1)*chunksize;
    threads.push_back(thread(func, v, i*chunksize, i_end));
}

Then the threaded function would only access the vector within a particular range.

void func(vector<int>& v, size_t i_begin, size_t i_end) {
    for (size_t i = i_begin; i < i_end; ++i)
        v[i] = 1;
}

[MutantJohn]

It sounds like you might want threads.

Yes. I should've been more specific in my post.

What I meant was, the current way in which you (the OP) are using threads is not "good". By locking the vector, you're effectively reducing the assignment to a single-threaded implementation even though you're using multiple threads.

It's like this, you want to increment a number 1,000,000 times. So you use threads and atomicAdd(). By doing so, you effectively serialize the execution of the instruction set of each thread.

Threads can indeed help you in this case but you do not need a mutex and you can no longer use push_back().

[Elysia]

You really should be using a different variable for each thread, then reduce it after all threads are finished. Atomics are still a performance drain, even though it's still not as much as locks.

And people, please, stop using push_back.
Instead of:
threads.push_back(thread(func, v, i*chunksize, i_end));
do:
threads.emplace_back(&func, v, i*chunksize, i_end);

[jiggunjer]

I don't know why you get those errors, the following works fine for me:

#include <iostream>
#include <vector> //replaced with array
#include <array>
#include <thread>
#include <mutex> //don't need


//std::mutex mtx;


using std::cout;	using std::endl;
using std::vector;	using std::array;


const unsigned int chunksize{ 1000 }; //magic number
typedef array< int, chunksize > data_array;


void ThreadedFunction(data_array& arr){	
	int z = 0;
	for (auto i = 0; i<chunksize; i++){
		// z = some processing...
		//(!) without a common resource like cout
		arr[i] = z; 
	}
}


int main()
{
	const int thr{ 10 }; // number of threads
	data_array arr;
	array<data_array, thr> data; 
	data.fill(arr);
	array<std::thread,thr> threads;


	/* Launch threads */
	for (int i = 0; i < thr; ++i)
	{		 
		threads[i] = std::thread(ThreadedFunction, std::ref(data[i]));
	}


	/* Join the threads with the main thread */
	for (auto& t : threads)	{t.join();}


	/* check sizes*/
	for (auto b : data) {cout << b.size() <<endl;}
}

[Aslaville]

The thing i need to do is to thread a function that does some serious data processing, after which it returns some coordinates (integer values). What i wish to do, is to have these int values after processing in one vector that i can query latter.

I don't know why you get those errors, the following works fine for me.

OP didn't mention the problem he/she is facing in the first place. Could it be merging the vectors(The resulting dataset) ? IDK

OH: Poor reading skills

[Aslaville]

I always get some memory corruption error or segmentation fault or something. However, if i use valgrind everything goes fine but only one thread at a time.

O.o, That's a bit ambiquous.

Could you be having a linking error ? On some setups, you could get away without '-pthread' flag while sometimes your program will segfault.