Trying to use the libgtop examples

Hello,


I'm working in a monitoring system for a database application. I need to measure usage of CPU, network, disk, partitions, memory, etc. I cannot handle the overhead of parsing this information from proc, so I found the libgtop. I am currently using the libgtop-2.28.1 version. After ./configure, make and make install it, I tried a small program in Eclipse to start learning the basics. I copied the procarg.c example and build it, having problems.


Code:

Code:

---

#ifdef HAVE_CONFIG_H
#  include <config.h>
#endif

#include <glib-2.0/glib.h>
#include <libgtop-2.0/glibtop.h>
#include <libgtop-2.0/glibtop/procargs.h>

#include <stdlib.h>
#include <unistd.h>




static void show_args(pid_t pid, unsigned max_len)
{
  glibtop_proc_args buf;
  char ** const pargv = glibtop_get_proc_argv(&buf, pid, max_len);
  char **parg = pargv;
  printf("<%ld>\t", (long)pid);
  while(*parg)
    {
      printf("\"%s\" ", *parg);
      parg++;
    }
  putchar('\n');
  g_strfreev(pargv);
}


int main(int argc, char **argv)
{
  glibtop_init();
  show_args(getpid(), 0);
  show_args(getpid(), 15);
  while(*++argv)
    {
      pid_t pid = strtol(*argv, NULL, 10);
      show_args(pid, 0);
      show_args(pid, 15);
    }
  glibtop_close();
  return 0;
}

---

Errors:

Code:

---

**** Build of configuration Debug for project GTopTest ****




make all
Building file: ../src/GTopTest.cpp
Invoking: GCC C++ Compiler
g++ -Igtop -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/GTopTest.d" -MT"src/GTopTest.d" -o"src/GTopTest.o" "../src/GTopTest.cpp"
In file included from ../src/GTopTest.cpp:13:
/usr/local/include/libgtop-2.0/glibtop.h:29: error: expected constructor, destructor, or type conversion before ‘typedef’
/usr/local/include/libgtop-2.0/glibtop.h:117: error: ‘glibtop_sysdeps’ does not name a type
/usr/local/include/libgtop-2.0/glibtop.h:118: error: ‘glibtop_sysdeps’ does not name a type
/usr/local/include/libgtop-2.0/glibtop.h:119: error: ‘pid_t’ does not name a type
/usr/local/include/libgtop-2.0/glibtop.h:124: error: expected initializer before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop.h:128: error: expected constructor, destructor, or type conversion before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop.h:131: error: expected constructor, destructor, or type conversion before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop.h:136: error: expected constructor, destructor, or type conversion before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop.h:141: error: ‘G_END_DECLS’ does not name a type
In file included from ../src/GTopTest.cpp:14:
/usr/local/include/libgtop-2.0/glibtop/procargs.h:38: error: ‘guint64’ does not name a type
/usr/local/include/libgtop-2.0/glibtop/procargs.h:39: error: ‘guint64’ does not name a type
/usr/local/include/libgtop-2.0/glibtop/procargs.h:44: error: ‘glibtop’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:44: error: ‘server’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:44: error: ‘glibtop_proc_args’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:44: error: ‘buf’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:45: error: ‘pid_t’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:45: error: expected primary-expression before ‘unsigned’
/usr/local/include/libgtop-2.0/glibtop/procargs.h:45: error: initializer expression list treated as compound expression
/usr/local/include/libgtop-2.0/glibtop/procargs.h:48: error: ‘glibtop_proc_args’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:48: error: ‘buf’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:48: error: ‘pid_t’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:48: error: expected primary-expression before ‘unsigned’
/usr/local/include/libgtop-2.0/glibtop/procargs.h:48: error: initializer expression list treated as compound expression
/usr/local/include/libgtop-2.0/glibtop/procargs.h:65: error: variable or field ‘_glibtop_init_proc_args_s’ declared void
/usr/local/include/libgtop-2.0/glibtop/procargs.h:65: error: ‘glibtop’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:65: error: ‘server’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:68: error: ‘glibtop’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:68: error: ‘server’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:68: error: ‘glibtop_proc_args’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:68: error: ‘buf’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:69: error: ‘pid_t’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:69: error: expected primary-expression before ‘unsigned’
/usr/local/include/libgtop-2.0/glibtop/procargs.h:69: error: initializer expression list treated as compound expression
/usr/local/include/libgtop-2.0/glibtop/procargs.h:79: error: ‘glibtop’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:79: error: ‘server’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:79: error: ‘glibtop_proc_args’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:79: error: ‘buf’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:80: error: ‘pid_t’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:80: error: expected primary-expression before ‘unsigned’
/usr/local/include/libgtop-2.0/glibtop/procargs.h:80: error: initializer expression list treated as compound expression
/usr/local/include/libgtop-2.0/glibtop/procargs.h:83: error: ‘glibtop_proc_args’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:83: error: ‘buf’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:83: error: ‘pid_t’ was not declared in this scope
/usr/local/include/libgtop-2.0/glibtop/procargs.h:83: error: expected primary-expression before ‘unsigned’
/usr/local/include/libgtop-2.0/glibtop/procargs.h:83: error: initializer expression list treated as compound expression
In file included from ../src/GTopTest.cpp:16:
/usr/include/unistd.h:28: error: expected constructor, destructor, or type conversion before ‘extern’
make: *** src/GTopTest.o Error 1

---

This is my first post here in this forum. I tried to be very specific to helo you to help me If I disagree with any rule just tell me, I really need to solve this and didn't finish reading the FAQ yet.


Thanks in advance,


Pedro Dusso

Quote:

---

Originally Posted by pmdusso

Code:

---

make all
Building file: ../src/GTopTest.cpp
Invoking: GCC C++ Compiler
g++ -Igtop -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/GTopTest.d" -MT"src/GTopTest.d" -o"src/GTopTest.o" "../src/GTopTest.cpp"
In file included from ../src/GTopTest.cpp:13:
/usr/local/include/libgtop-2.0/glibtop.h:29: error: expected constructor, destructor, or type conversion before ‘typedef’
...

---

---

Your code looks like C. libgtop was probably built as a C library. Your Eclipse configuration, however, is trying to compile C++. C and C++ deal with library interfaces differently, so you can't necessarily link a C library to a C++ executable without doing something extra. Try changing your file to GTopTest.c and configuring Eclipse to compile it as a C program (using gcc) instead of C++ (using g++).

Quote:

---

Originally Posted by pmdusso

I need to measure usage of CPU, network, disk, partitions, memory, etc. I cannot handle the overhead of parsing this information from proc, so I found the libgtop.

---

Just so you know, altho I have not confirmed this by looking at the glibtop source, this concern with "the overhead of parsing /proc" is chimeric, because that is what glibtop has to do anyway. Note this contradiction in the glibtop documentation:

Quote:

---

Linux has a very nice file /proc filesystem, but reading and parsing file /proc is very slow and inefficient.

---

from: White paper

Makes it sound as if glibtop therefore does not use this method, and has something more magical to offer! However, on the next page:

Quote:

---

The LibGTop server is a setgid/setuid binary which contains all the system dependent code which needs special privileges. It is only build if it's required on the current system (for instance, the Linux kernel provides all the required data via its file /proc filesystem so we do not need the server at all)

---

from: http://developer.gnome.org/libgtop/s...verview-server

ROTFL. Pretty much confirms that in fact, glibtop on linux gets its information by "reading and parsing" the proc filesystem ("inefficient" or not).

Also, having written a specialized process monitor for use on linux servers*, I don't believe there is any choice in the matter: either you read and parse the data in proc, or you use a library that does it for you. There is no other source of this information. You could write a kernel module, but that is essentially what "proc" already is.

Anyway, who would consider reading and parsing the very succinct, intended for fast, simple, single pass parsing info in /proc "slow and inefficient"? Compared to what? Osmosising it somehow? There is not even a hardware bottleneck, since this filesystem is not on disk -- it's a kernel interface.

I'd also bet money that if you wrote 50 lines of custom code to get the data you need, it will be several times faster than the facilities provided by a generic API like libgtop, which must do the same thing anyway. Plus then you have one less dependency and one less shared library that has to be loaded into memory even if no one else is using it. Ie, "reading and parsing" proc will require less overhead than anything else, not more. And proc is very well documented -- the man page even gives you the data type for individual fields.

* BTW, that monitor works on multiple processes with a slow sample rate (in minutes) and provides a lot of information:

Code:

---

Date  Time State Prio  __Utime__ Threads VirtualSz ResidentSz Shared_Mem Data+Stack Priv&Write MinorFaults  Major
10- 2  8:01  S    20      5:13    25  605.86MB  105.79MB    34.14MB  272.59MB  273.52MB      37535    351
10- 2  8:11  S    20    1:11:51    24  697.45MB  167.08MB    35.31MB  364.11MB  365.04MB      73542    355
10- 2  8:21  S    20    2:42:40    24  720.26MB  200.74MB    35.57MB  387.05MB  387.97MB      185796    361
10- 2  8:31  S    20    3:23:32    23  719.03MB  191.21MB    35.58MB  385.82MB  386.74MB      198228    361
10- 2  8:42  S    20    3:52:02    23  711.21MB  194.18MB    35.58MB  378.00MB  378.93MB      200289    361
10- 2  8:52  S    20    5:25:47    26  806.39MB  248.98MB    35.73MB  473.71MB  474.62MB      306259    366
10- 2  9:02  S    20    6:44:44    24  870.87MB  311.19MB    35.72MB  538.18MB  539.11MB      362095    366
10- 2  9:12  S    20    8:07:10    24  826.95MB  297.68MB    35.66MB  494.30MB  495.23MB      432884    366

---

Using a minuscule amount of resources of its own. Sampling a few single line, <1kB "files" and applying sscanf() is a very very minor event (getting the "priv&write" field is a little more complicated, but still...). What "overhead" are you concerned with?

Thanks for the reply, anduril462.

I did not know this. I thought, as C++ was made to be fully compatible with C, that it handled the libraries in same way. Well, I created a C project in Eclipse and the results, if you are interested, are:

Code:

---



**** Build of configuration Debug for project GTopTest_C ****
make all
Building file: ../src/GTopTest_C.c
Invoking: GCC C Compiler
gcc -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/GTopTest_C.d" -MT"src/GTopTest_C.d" -o"src/GTopTest_C.o" "../src/GTopTest_C.c"
In file included from ../src/GTopTest_C.c:17:
/usr/local/include/libgtop-2.0/glibtop.h:29: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘typedef’
/usr/local/include/libgtop-2.0/glibtop.h:117: error: expected specifier-qualifier-list before ‘glibtop_sysdeps’
/usr/local/include/libgtop-2.0/glibtop.h:124: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop.h:128: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop.h:131: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop.h:136: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
In file included from ../src/GTopTest_C.c:18:
/usr/local/include/libgtop-2.0/glibtop/procargs.h:28: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘G_BEGIN_DECLS’
/usr/local/include/libgtop-2.0/glibtop/procargs.h:38: error: expected specifier-qualifier-list before ‘guint64’
/usr/local/include/libgtop-2.0/glibtop/procargs.h:44: error: expected ‘)’ before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop/procargs.h:48: error: expected ‘)’ before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop/procargs.h:65: error: expected ‘)’ before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop/procargs.h:68: error: expected ‘)’ before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop/procargs.h:79: error: expected ‘)’ before ‘*’ token
/usr/local/include/libgtop-2.0/glibtop/procargs.h:83: error: expected ‘)’ before ‘*’ token
../src/GTopTest_C.c:20: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘int’
make: *** [src/GTopTest_C.o] Error 1

---

Reading the answer of MK27, maybe a I have a new path to follow. But, suppose I want to use the glibtop (or any other C lib) with my C++ project, how should I proceed? Can you send me some links/references to study? It is possible to have this "mix" (is it good from the software engineering point of view?)?

Thanks again,

Pedro

Thanks for the long reply MK27,

This is a very good analysis. I got in touch with the writer of htop, and he answered me saying he also use just the /proc. My fear about the overhead using is was based:
- first in the white paper you also read, exactly from the

Quote:

---

/proc is very slow and inefficient

---

part;
- second, from my believe it was a file and the I/O times for opening and closing it could be too much for real time reporting. What does mean the /proc be a "kernel interface" and why this mean no bottleneck?

The white paper of glibtop is very bad but it give some insights, mostly about how to difficult is to get this kind of information in different UNIX systems. If I am working in a linux environment, apparently it worth to write a parser in C++. But this will not reduce the portability of my project? I know this is not a technical question anymore and more a decision project, but I would like to hear you opinion.

Right know I totally agree with your last paragraph and will not bet money against you :) I gonna start reading some documentation about the proc in the meanwhile.

Thanks very much!

Pedro

Quote:

---

Originally Posted by pmdusso

I did not know this. I thought, as C++ was made to be fully compatible with C, that it handled the libraries in same way. Well, I created a C project in Eclipse and the results, if you are interested, are:

---

A common misunderstanding. C++ is almost a superset of C, thus they are close to source-compatible, but not 100%. That means a lot of valid C code can be compiled by a valid C++ compiler, but not all of it. We have a few links discussing some of the major differences: Cprogramming.com FAQ > C and C++? and Differences Between C and C++ - Cprogramming.com.

That is very different, however, from a C library being binary compatible with a C++ program. The libraries are essentially in different format, due to things like C++ name mangling.

Quote:

---

Reading the answer of MK27, maybe a I have a new path to follow. But, suppose I want to use the glibtop (or any other C lib) with my C++ project, how should I proceed? Can you send me some links/references to study? It is possible to have this "mix" (is it good from the software engineering point of view?)?

---

Yes, it's possible, and there is nothing wrong with it. I've only done it once or twice some time ago, so I don't remember much about what's involved. I think all you really need is the extern "C" feature of C++ to tell the compile to link against C libraries/functions. Take a look at these: [32] How to mix C and C++ ..Updated!.., C++ FAQ and C-library not linking using gcc/g++ - Stack Overflow. I think you would just wrap any place you #include <libgtop-2.0/...>

Quote:

---

Originally Posted by pmdusso

second, from my believe it was a file and the I/O times for opening and closing it could be too much for real time reporting. What does mean the /proc be a "kernel interface" and why this mean no bottleneck?

---

You're getting file and disk I/O a little confused. The kernel interface is basically the system calls that handle file I/O. That is, things like open, read, write and close. Those work the same for all files of any type, whether regular files, special ones like those in /proc, pipes, sockets, or whatever else. What makes disk I/O slow is the need to go all the way to the disk, possibly spin it up, find the right sector, and read it back. Special file systems like /proc live entirely in memory, so reading from them only involves reading from RAM, which is much quicker.

Quote:

---

The white paper of glibtop is very bad but it give some insights, mostly about how to difficult is to get this kind of information in different UNIX systems. If I am working in a linux environment, apparently it worth to write a parser in C++. But this will not reduce the portability of my project? I know this is not a technical question anymore and more a decision project, but I would like to hear you opinion.

---

Your program should be portable across all flavors of Linux. If you're writing your own /proc parser, then it will only work on Linux. If it is just interacting with the gtop server, then it should port easily to most/all Posix systems.

Quote:

---

Originally Posted by pmdusso

This is a very good analysis. I got in touch with the writer of htop, and he answered me saying he also use just the /proc.

---

I am quite sure that WRT linux, the only standard kernel->userspace source for the kind of stats you find in /proc, is /proc itself. There are no system calls, etc. If you wanted to get really really fancy, you could write a kernel module to provide customized reporting for selected processes, but I'm dubious as to what advantages (beyond fanciness) that would provide.

Quote:

---

My fear about the overhead using is was based:
- first in the white paper you also read, exactly from the part;

---

Hmm -- the reason it caught my eye was I've seen the same thing said before, and it kind of made me do a double take. I wouldn't be surprised if that libgtop doc has had a sort of viral effect. Someone should drop them a line about that because:

Quote:

---

- second, from my believe it was a file and the I/O times for opening and closing it could be too much for real time reporting. What does mean the /proc be a "kernel interface" and why this mean no bottleneck?

---

It's as anduril described. Here's some simple benchmark code:

Code:

---

#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>

int main(int argc, const char *argv[]) {
        char path[32], data[2048];
        int tps = sysconf(_SC_CLK_TCK); 

        sprintf(path, "/proc/%s/stat", argv[1]);
        int fd = open(path, O_RDONLY);

        if (fd < 0) {
                perror("open");
                return -1;
        }

        if (read(fd, data, 2048) == -1) {
                perror("read");
                return -1;
        }
        close(fd);

        char name[1024];
        long unsigned int utime, virt;
        long int rss;
        sscanf(data, "%*d %s %*c %*d %*d %*d %*d %*d %*u %*u %*u %*u %*u %lu"
                " %*u %*d %*d %*d %*d %*d %*d %*u %lu %ld", name, &utime, &virt, &rss);

        printf("%s active %lus, virtual size: %ld kB, resident pages: %ld\n",
                name, utime/tps, virt/1024, rss);     

        return 0;
}

---

This just follows the manual page for proc. It looks to me like an effort has been made to keep the information formatted in a consistent and compatible way over time, so this should work for any and all linux systems that have a proc filesystem. Run that with a pid as the first arg, eg:

./a.out 1541
(firefox-bin) active 302s, virtual size: 717608 kB, resident pages: 51178

To test, comment out lines 20 & 31 (the console output), and put a "for(i=0; i<1000000; i++" loop around everything in main().

time ./a.out 1541

real 0m13.479s
user 0m3.832s
sys 0m9.532s


The files in proc are special and will not be mem-cached, so all of those reads are real and reflect current info from the kernel.

So on a single 2.2Ghz processor, /proc/[pid]/stat can be open, read, closed, and the data parsed into variables ~74,000 times per second, using a nominal amount of memory. Certainly contrived and simplistic -- presumably you want to do more reporting than just this -- but by who's standard is this "very slow and inefficient"?

Ie, I think that particular piece of "information" from the libgtop white paper is completely bogus and I have no idea how it got to be there at all.

Quote:

---

how to difficult is to get this kind of information in different UNIX systems.

---

Well, this is certainly what libgtop will be good for. If you have to make this portable beyond linux, I would definitely use something like that.

But if you don't, it is almost certainly easier and more efficient to just parse the proc data yourself. If you do go that route, here's a couple of caveats from my experience:

- You must use the low level I/O calls -- open, read, close. You cannot use the stream based interface (fopen, fgets, etc). I did this in C++ at one point and I *think* the ifstream stuff from there is okay.

- When reading, read in <= 1kB blocks. If you read in blocks of page size (usually 4 kB) or more, you will experience errors. Most of the proc files are always smaller than that anyway, but some (eg, 'maps') can be bigger. You cannot stat() a proc file to get its size in advance -- you must simply read until you have the info you need or you reach eof.