threads and mem usage

[MK27]

I've written a threaded process logger (in C++) and I'm kind of disappointed with the way it eats memory. I guess I was presuming something (wrongly) about threads and resource sharing.

The idea is simple: the logger runs as a daemon server. When it receives a request to monitor a new process, it launches a thread to do so. So all the threads are identical as far as code goes -- they are the same function.

When it starts, it has a respectably small virtual size, ~15mB, with no threads, just the main server process waiting. The stack is less than 1mb. Then each thread adds 70mb to the total stack size. Pretty soon, the process logger is using more memory than most of the processes it is monitoring, lol.

What's the deal with that? I might as well just run a whole bunch of independent processes. I thought part of the point of multi-threading was to save resources vs. using separate processes, but I seem to be wrong.

I guess the best model would be to use a single process server without threads that handles multiple requests -- the reason I didn't do that in the first place was laziness, just launching threads seemed easier. So there is not much code to change, but thought I'd get some input first.

[MK27]

Okay, live and learn. This is it in a nutshell:

#include <iostream>
#include <unistd.h>
#include <pthread.h>

using namespace std;

void *test (void *x) {
	while (1) sleep(1);
	return NULL;
}

int main(int argc, char *argv[]) {
	cout << getpid() << endl;

	for (int i = 0; i < 10; i++) {
		sleep(5);
		pthread_t id;
		pthread_create(&id, NULL, test, NULL);
		pthread_detach(id);
		cerr << i << endl;
	}


	return 0;
}

This starts out with a stack size <200 kB. Then each thread adds 8MB...yeeeck.

[Elysia]

Check stack size? And why not use Boost instead of pthreads?

[MK27]

Check stack size? And why not use Boost instead of pthreads?

That is the stack size I'm referring to, you can see it yourself if you compile the code in post #2 and watch it in top with "S" (data+stack) on. Apparently that simple "test()" function requires 8MB as a thread -- imagine if it actually had something in it.

Since it is a process logger, it is in no way portable, so it makes more sense to use pthreads as pthreads will (most likely) be in use on any system where I want to use it, whereas Boost probably is not. I'm also guessing Boost just wraps pthreads on POSIX systems (but I didn't check).

[Codeplug]

Linux Thread Memory Usage | EmptyCrate
Virtual Threads: Understanding memory usage on Linux
http://ktown.kde.org/~seli/memory/analysis.html

But I wouldn't be too concerned about virtual memory. You have to touch it before it becomes resident.

gg

[Elysia]

Since it is a process logger, it is in no way portable, so it makes more sense to use pthreads as pthreads will (most likely) be in use on any system where I want to use it, whereas Boost probably is not. I'm also guessing Boost just wraps pthreads on POSIX systems (but I didn't check).

Boost is a portable C++ library which does not depend on external dependencies. Granted, I did not check if Boost Thread requires some library.
There is no runtime library that has to exist on the target machine (except for a few libraries in the boost collection; but you can use static linking for those).
Boost allows for creating static libraries of its requirements allowing for easy porting to virtually any platform without the need for dependencies.

[MK27]

Linux Thread Memory Usage | EmptyCrate
Virtual Threads: Understanding memory usage on Linux
http://ktown.kde.org/~seli/memory/analysis.html

But I wouldn't be too concerned about virtual memory. You have to touch it before it becomes resident.

gg

Yeah, the resident size is quite small -- I've had one running for a few days with 8 threads and it's only 1300 kB.

Unfortunately, it's for use on OpenVZ server slices -- mostly to track mem usage, lol -- and what OpenVZ appears to count seems to be best approximated by the "writable/private" stat from pmap, which is still 73MB. That would appear to be very closely related to the per-thread stack allocation of 8mb* (eg, the simple do-nothing code from post #2 has a "writable/private" of 82MB with 10 idle threads). OpenVZ doc's refer to this as "allocated memory":

In practice, a lot of applications do not use the memory very efficiently and, sometimes, allocated memory will never be used later. For example, Apache Web server at start time it allocates about 20–30%% more memory that it will ever use. Some multi-threaded applications are especially bad at using their memory, and their rate of allocated to used memory may happen to be 1000%.

Eek, that seems a little unfair. But I can't have a profiling tool on a 1 GB slice using 15% of the memory, so I will have to take the threads out

later: looks like you can use ulimit on an OpenVZ slice, which should make the problem mostly moot:
inportb » Slash OpenVZ Memory Usage by Shrinking Thread Stack
More later: this works in general and for the basic per thread allocation but appears not to help enough with my app

* on the other hand, one of the processes I was most concerned about -- node.js -- reports 5 threads but a "writable/private" of 14MB (smaller than it's resident size). I guess that is something to be ultimately happy about, since it means in addition to being a truly great piece of software, node makes a nice guest under OpenVZ. Unfortunately, the other key component -- couchdb -- is a terrible pig. Mebbe I will bug the node people to find out how they have kept the "allocated" memory so low.

[Codeplug]

You could try this if changing the system default isn't ideal: pthread_attr_getstacksize

gg

[MK27]

You could try this if changing the system default isn't ideal: pthread_attr_getstacksize

gg

Thanks much, very glad to see that exists.

"Pthread_attr_setstacksize" is in the nodejs source so I presume that's how they accomplished that there.