You have two major sources of memory in TC, the stack, and the heap.
If you can't get enough memory out of the heap, then try putting your other variables onto the stack, and see how that works.
Alternatively, put your 2D array onto the stack by making it non-dynamic, and try to free up the rest of the stack, as much as possible.
Also, you can change your program to allow it to work with just a portion of the whole image, at any one time. Maybe do a quarter of it at a time?
Bottom line is, TC (and I'm a big TC user still), is not well suited for larger memory demands. Move the program over to MS Visual C Express (it's free), or one of the other 32 bit C compilers.
That's what I do with my larger programs. I love TC, but ...