I was wondering what tools people use when debugging problems with programs or interactions between multiple programs/systems?
Also, if you could give a brief description of what kind of problems the tools are best at detecting, that would be great. I'm trying to get better at finding the cause of problems, especially those that involve multiple programs or systems (mostly UNIX, but Windows tools are welcome too).
Some tools that people have already suggested to me are: pfiles, ptree, pstack, netstat, strace, dtrace, tcpdump.
I've used netstat and ptree before, but I'm not too familiar with the others.
ptree shows you the currently running processes & PIDs in a tree of process -> sub-processes...
netstat shows you all the currently open ports...
Obviously looking at the log files for the programs you're debugging it a good place to start, and there's also debuggers like gdb (assuming you have access to the program's source code).