GNU Auto build tools

This is less a question about C++ coding, as it is using the GNU Auto build tools with it.
I have my project all set up using these tools, but when I compile I realize that the libraries I need to link with are linking with EVERY source file in my project. Now most of them need to be linked with these libraries, but some source files have no code that needs it, so wouldn't this just be making the file larger than it needs to be?
How can I specify which specific source files need to be linked with the libraries?
I assume this needs to be done in my src/Makefile.am file, but I don't know how exactly. I have all the libraries stored in a Makefile variable, so if I just know the macro or variable I need to use I can easily set it up. Thanks.

> but when I compile I realize that the libraries I need to link with are linking with EVERY source file in my project
Huh?

The compilation should be at it's most basic
gcc -c file.c

> I have all the libraries stored in a Makefile variable,
Did you add them to say CPP_FLAGS, and not say LD_FLAGS ?

Ok, here's my Makefile.am in my src directory:

Code:

---

bin_PROGRAMS = merc
merc_SOURCES = merc.cc info.cc controls.cc listbox.cc window.cc main.cc
AM_CXXFLAGS = $(Xmms2Client_CFLAGS) $(Gtkmm_CFLAGS) -Wall -g
merc_LDADD = $(Xmms2Client_LIBS) $(Gtkmm_LIBS)

---

Now only merc.cc and main.cc use the Xmms2Client_LIBS, and the rest only need the Gtkmm_LIBS. Yet when I compile, both libraries are linked for every source file. Is this normal and supposed to happen, or is this just making my object files for each source file larger than it needs to be? I'm not sure if this is even an actual "problem" or not, but it just seems to me if every source file is being linked to libraries it doesn't need, it'd make object code larger than it should be.