Generally, trying to include only the header files that contain declarations/definitions for what you actually need in your code is best practice, I believe. This also reduces compile time when you get larger projects, as long as you are using a makefile or some such. It will only compile the files that have been modified.
Knowing a bit more about your program and how you are structuring it might help people with giving you some advice.