Which compiler are you using for C?

> -Xmx512m
I'm guessing this means 512 Megabytes.

As far as I know, the C runtime will continue to allocate memory until either the 2GB limit for user address space is used up, or the real+swap space is used up (whichever is first).