Thread: Help with Linux and performance

  1. #1
    Internet Superhero
    Join Date
    Sep 2006
    Location
    Denmark
    Posts
    964

    Help with Linux and performance

    Here is the scenario:

    Me and a friend have a headless server set up that we access with ssh and scp. The server is an old desktop machine with a Q6600 and 4GB RAM, with Ubuntu server 11.10. We use it as a Minecraft server, which is a _very_ heavy application, that needs to run smooth as silk to be enjoyable for the clients.
    Once a day a cron job activates a small program which take a snapshot of our game server and saves it as a PNG. This is also a _very_ heavy process, AFAICT this process uses ray-tracing to generate the image, which is a 13000*13000 PNG file, it takes about half an hour to finish up. So the thing is, the main Minecraft application gets laggy and unresponsive during this period, which is strange since Minecraft is completely single-threaded and only uses a single core when running. If i fire up htop while both processes are running i see one core screaming along at 100% and then 3 other cores doing nothing, shouldn't the operating system be handling this alot better?
    These are 2 seperate programs, one is a java application, the other is written in C++, why do they have to fight over the same core?

    Also, we need some heavy compression for the PNG files themselves, what compression software should we use? It seems it's either bzip2 or gzip, but which has the highest compression rate, and are there any flags i can pass to them to make them compress even further?

    If you haven't noticed im pretty new with Linux, but it's been quite enjoyable so far.
    How I need a drink, alcoholic in nature, after the heavy lectures involving quantum mechanics.

  2. #2
    Registered User
    Join Date
    Dec 2011
    Posts
    6
    I can't help you with your first question unfortunately, but I believe I've read a few times that bzip2 has the best compression rate.

  3. #3
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    That's definitely not right if they are 2 different processes. I'm assuming your kernel was compiled with proper SMP support? (only a problem if you compiled your own kernel, since all kernels in distributions are now SMP)

    Maybe the screenshot process loads up the Minecraft process?

    What does "top" output look like when both are running?

    PNG is compressed already, so there's no point compressing it further with another lossless algorithm. The only way to compress further (effectively) is to convert it to some lossy format, like JPG.

  4. #4
    Make Fortran great again
    Join Date
    Sep 2009
    Posts
    1,413
    cyberfish is right about what he said about compressing pngs to begin with, but for argument's sake, 7-zip (LZMA) has better compression than bzip2 AFAIK.

  5. #5
    &TH of undefined behavior Fordy's Avatar
    Join Date
    Aug 2001
    Posts
    5,793
    Have you tried to manually set the processor affinity?

    taskset

  6. #6
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    cyberfish is right about what he said about compressing pngs to begin with, but for argument's sake, 7-zip (LZMA) has better compression than bzip2 AFAIK.
    Yeah 7-zip is the best general purpose compression algorithm with reasonable run time.

    Nowadays I only use 7-zip and gzip. 7-zip for most things, and gzip for very fast compression (eg. moving files on LAN).

  7. #7
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    At first it sounded as if your kernel was configured wrong, since that is not normal linux SMP behaviour. But if you have multiple cores shown in htop, then presumably SMP is enabled....

    Are you sure it is not some other resource? Eg, does the raytracing process write to disk constantly? The same disk needed by the minecraft server? Presumably, since it is doing some kind of state snapshot, it is either communicating with the server directly, or else actually accessing the same files on disk. If these files were only being read by both processes, presuming enough free RAM, that would not be a problem, because linux will use free RAM for file caching. But obviously that changes if the file is written to, because it requires an actual disk write.

    BTW, I've never experienced this because I never fill disks to capacity, but apparently you will get performance degradation if the HD is over 75% capacity.

    What about RAM itself? Ray tracing can use a lot of that, I believe. If things get to the point where the swap is active, everything will slow down.

    Try deprioritizing the ray-tracer a bunch with nice:

    nice - Linux Command - Unix Command

    The nice value is used by the kernel scheduler. Look at the PRI and NI fields in htop. You want the PRI of the ray tracer to be higher than anything else, maybe substantially (higher values are lower priority). Try setting it as high as possible (nice 19). You can affect this with the F7 and F8 key in htop too.
    Last edited by MK27; 12-16-2011 at 07:09 AM.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  8. #8
    Internet Superhero
    Join Date
    Sep 2006
    Location
    Denmark
    Posts
    964
    We've told the snapshot app to not use more than 1GB of RAM, this can be specified with a command-line parameter. Yes, the snapshot app accesses the world-data in the Minecraft directory, and this data is constantly changing while the Minecraft server is running.
    I'm gonna try manually assigning the proc affinity, and if that fails, ill try to deprioritize the snapshot app.
    How I need a drink, alcoholic in nature, after the heavy lectures involving quantum mechanics.

  9. #9
    Internet Superhero
    Join Date
    Sep 2006
    Location
    Denmark
    Posts
    964
    Okay so taskset was exactly what we wanted, thank you for the help guys. I think we're going to use imagemagick to compress the png to a jpg, and then to reduce it in size (The final render is actually 41000x21000, not 13kX13k as i said earlier, it took me 1½ hour to transfer it to my desktop with winSCP yesterday o.O)
    How I need a drink, alcoholic in nature, after the heavy lectures involving quantum mechanics.

  10. #10
    Registered User
    Join Date
    Dec 2006
    Location
    Canada
    Posts
    3,229
    iotop will tell you how much IO/disk bandwidth each process is using.

    "free -m" tells you how much RAM is in use. How much swap, etc.

  11. #11
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    If you have two compute-intensive processes running on a multicore system and they are both running on the same core, then your kernel is buggy. Upgrade! These aren't hyperthreaded Intel cores are they?
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  12. #12
    Internet Superhero
    Join Date
    Sep 2006
    Location
    Denmark
    Posts
    964
    No its an old Q6600, no hyperthreading at all. The OS is a bog standard Ubuntu Server 11.06 installation, i doubt that the kernel is buggy to be perfectly honest. Anyways, we solved it by using taskset to force one of the apps to run on a different core :-)
    How I need a drink, alcoholic in nature, after the heavy lectures involving quantum mechanics.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Pthreads performance
    By C_ntua in forum C Programming
    Replies: 42
    Last Post: 06-17-2008, 11:29 AM
  2. Program Performance
    By Hypercase in forum C++ Programming
    Replies: 2
    Last Post: 09-28-2004, 05:33 PM
  3. Best performance.
    By Benzakhar in forum Game Programming
    Replies: 3
    Last Post: 01-14-2004, 10:34 AM
  4. Performance Tests?
    By genghis in forum C++ Programming
    Replies: 5
    Last Post: 01-30-2003, 03:49 AM