Thread: Symbols classing during linking

  1. #1
    Kung Fu Kitty Angus's Avatar
    Join Date
    Oct 2008
    Location
    Montreal, Canada
    Posts
    115

    Symbols clashing during linking

    I have this problem that seems to come down to calls to the libjpeg library from different shared libraries. Here's a snip of my valgrind report:

    ==25374== Invalid write of size 4
    ==25374== at 0x5025FC3: jpeg_CreateDecompress (in /usr/local/lib/libPdfOptimizeAPI.so)
    ==25374== by 0x59AAAA3: DCTStream::init() (in /usr/lib64/libpoppler.so.4.0.0)

    libpoppler is one library that is supposed to have nothing to do w/libPdfOptimizeAPI. I've run ldd on both libs, and they don't mention each other, but Poppler does mention libjpeg.so.62. So it seems that Poppler links libjpeg as a shared, but PDFOptimize links it statically. Whatever the case, it seems that Poppler is calling PdfOptimize, which it has no business doing. How do I stop this?
    Last edited by Angus; 05-12-2009 at 03:15 PM. Reason: typo in the title

  2. #2
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by Angus View Post
    I have this problem that seems to come down to calls to the libjpeg library from different shared libraries. Here's a snip of my valgrind report:

    ==25374== Invalid write of size 4
    ==25374== at 0x5025FC3: jpeg_CreateDecompress (in /usr/local/lib/libPdfOptimizeAPI.so)
    ==25374== by 0x59AAAA3: DCTStream::init() (in /usr/lib64/libpoppler.so.4.0.0)

    libpoppler is one library that is supposed to have nothing to do w/libPdfOptimizeAPI. I've run ldd on both libs, and they don't mention each other, but Poppler does mention libjpeg.so.62. So it seems that Poppler links libjpeg as a shared, but PDFOptimize links it statically. Whatever the case, it seems that Poppler is calling PdfOptimize, which it has no business doing. How do I stop this?
    I think what's happening here is that the symbol information is clashing between the two modules. I bet it's really inside libpoppler, not libPdfOptimizeAPI, and Valgrind is just reporting it wrong.

    Try running "ldd your_program" to get a map of where the shared objects are loading -- whichever address range contains the address "0x5025FC3" is the real library which is invoking jpeg_CreateDecompress().

    EDIT: If libPdfOptimize is linking libjpeg statically, and it is dynamically exporting those symbols, then the dynamic linker would resolve the call to jpeg_CreateDecompress() from the first convenient location, which in this case happens to be libPdfOptimize. The problem is that libPdfOptimize should not be dynamically exporting those symbols. But I think it's harmless.
    Last edited by brewbuck; 05-12-2009 at 03:27 PM.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  3. #3
    Kung Fu Kitty Angus's Avatar
    Join Date
    Oct 2008
    Location
    Montreal, Canada
    Posts
    115
    My instincts tell me that valgrind is not reporting anything wrong. It's pretty good at getting things right, and exercises due diligence in labeling apocryphal information.

    Ok, I ran ldd on the executable, but I didn't get anything in the same ballpark as 0x5025FC3. This is what I did get, though:
    linux-vdso.so.1 => (0x00007fffecffe000)
    libPdfOptimizeAPI.so => /usr/local/lib/libPdfOptimizeAPI.so (0x00007f9be45c0000)
    libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007f9be437d000)
    libjpeg.so.62 => /usr/lib64/libjpeg.so.62 (0x00007f9be4159000)
    libpoppler.so.4 => /usr/lib64/libpoppler.so.4 (0x00007f9be3d62000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f9be3a56000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f9be3800000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f9be35e8000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f9be328f000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9be3073000)
    libstdc++.so.5 => /usr/lib64/libstdc++.so.5 (0x00007f9be2d91000)
    libidn.so.11 => /usr/lib64/libidn.so.11 (0x00007f9be2b5e000)
    libssl.so.0.9.8 => /usr/lib64/libssl.so.0.9.8 (0x00007f9be2910000)
    libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00007f9be2593000)
    libldap-2.4.so.2 => /usr/lib64/libldap-2.4.so.2 (0x00007f9be234f000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f9be2146000)
    libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f9be1f1a000)
    libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f9be1c78000)
    libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f9be1a52000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f9be184e000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f9be164b000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f9be1434000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f9be1230000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f9be101a000)
    libxml2.so.2 => /usr/lib64/libxml2.so.2 (0x00007f9be0cbc000)
    libfreetype.so.6 => /usr/lib64/libfreetype.so.6 (0x00007f9be0a36000)
    libfontconfig.so.1 => /usr/lib64/libfontconfig.so.1 (0x00007f9be0804000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9be4c49000)
    liblber-2.4.so.2 => /usr/lib64/liblber-2.4.so.2 (0x00007f9be05f4000)
    libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00007f9be03d9000)
    libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f9be01d0000)
    libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f9bdffa6000)
    You'll have to tell me what any of that means, because I don't know. But I suspect that those hex values are about relocation or something, while 0x5025FC3 is a more concrete address.

    So what can I try now?

    EDIT: would a call to objdump tell you anything? You'll have to tell me what options to use, because I don't understand half of them
    Last edited by Angus; 05-13-2009 at 08:36 AM.

  4. #4
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by Angus View Post
    My instincts tell me that valgrind is not reporting anything wrong. It's pretty good at getting things right, and exercises due diligence in labeling apocryphal information.
    I didn't know that Valgrind could handle 64-bit executables. It wouldn't surprise me if it's a bit buggy. The fact that it reports a function from libpoppler at an address nowhere near where libpoppler is loaded makes me suspicious.

    You'll have to tell me what any of that means, because I don't know. But I suspect that those hex values are about relocation or something, while 0x5025FC3 is a more concrete address.
    "ldd" loads and dynamically links the executable but does not actually run it. It just reports the locations where the libraries loaded, as well as any missing dependencies.

    EDIT: would a call to objdump tell you anything? You'll have to tell me what options to use, because I don't understand half of them
    Maybe... The problem is that before the image is loaded, none of the dynamic symbols are resolved and so you can't see their addresses.

    Really though, apart from Valgrind seemingly reporting the wrong address, I think everything is fine. What I think is going on is libPdfOptimize was linked statically with libjpeg, and when it was turned into a DSO the linker was told to export ALL available symbols dynamically, including the ones from libjpeg. So when the dynamic linker tries to resolve a call to a JPEG function, it finds it inside the libPdfOptimize library because it is mentioned before libjpeg in the executable's DYNAMIC section. You could try to fix that by listing libjpeg before libPdfOptimize on the link line.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  5. #5
    Kung Fu Kitty Angus's Avatar
    Join Date
    Oct 2008
    Location
    Montreal, Canada
    Posts
    115
    Do you have any other reasons you would think valgrind buggy? I've used it for a few years now, and found it very reliable.

    If you think everything is fine then perhaps I should point out that this Invalid write causes a seg fault: this project is unusable, and I think this bizarre linking is at the heart of it.

    And I don't mention libjpeg when I link, because I don't use it--the .so libraries I link to do.

  6. #6
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    Don't put valgrind on a pedestal; if you google "valgrind gtk+" you will find another (very broad set of) case(s) where it produces erroneous or misleading output.

    On the other hand, if you are having a real problem and you think it has helped locate that, then maybe there is a real problem (it appeared in your previous posts that you were just "looking for one where one has not yet been found").
    Last edited by MK27; 05-13-2009 at 11:11 AM.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  7. #7
    Kung Fu Kitty Angus's Avatar
    Join Date
    Oct 2008
    Location
    Montreal, Canada
    Posts
    115
    Also, I've tried running my project w/ou the optimizer library, and the Poppler code performed perfectly. So even if the optimizer is not at fault (and I think it highly likely that it is, and is causing Poppler calls to be misdirected) then the optimizer is definitely key to the problem.

    EDIT: Oh yes, and I keep forgetting, at the end of my build I always see this: warning: libstdc++.so.5, needed by /usr/local/lib/libPdfOptimizeAPI.so, may conflict with libstdc++.so.6. Who knows, that could be part of the problem.
    Last edited by Angus; 05-13-2009 at 11:35 AM.

  8. #8
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    Quote Originally Posted by Angus View Post
    Also, I've tried running my project w/ou the optimizer library, and the Poppler code performed perfectly. So even if the optimizer is not at fault (and I think it highly likely that it is, and is causing Poppler calls to be misdirected) then the optimizer is definitely key to the problem.
    Well, like I said, try listing libjpeg explicitly on your link line, before libPdfOptimize. You aren't using it explicitly, but the libraries are. The IJG JPEG library hasn't changed in years, so I doubt there is actually a bug there. I blame libPdfOptimize for doing something as silly as statically linking a library which is commonly available as a dynamic.

    As far as Valgrind being buggy, its bugs are few and far between but they do exist. I trust it for the most part, but I've used too many dynamic analysis tools and seen too many weird things to ever think that a tool like that could be perfect.

    It's never appropriate to blame the tools first. But that doesn't mean they can't have problems. I think Valgrind is reporting a real problem, but the backtrace is obviously... weird.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  9. #9
    Kung Fu Kitty Angus's Avatar
    Join Date
    Oct 2008
    Location
    Montreal, Canada
    Posts
    115
    UN-BE-LIEVABLE! Explicitly linking libjpeg worked. I still think someone should be shot for making that measure necessary. The optimizer does have a number of other problems unbecoming of a 1200$ library.
    Thanks for your help! I could have been at this a long time if I'd waited for the vendor to fix it.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Linux Putting my children to sleep with futex!?
    By Abs in forum Linux Programming
    Replies: 18
    Last Post: 02-12-2009, 06:43 PM
  2. Replies: 7
    Last Post: 02-02-2009, 07:27 AM
  3. unresolved external symbols...linking errors in VC++
    By rammohan2b in forum C++ Programming
    Replies: 2
    Last Post: 01-22-2009, 02:19 AM
  4. Trouble with Windows/DirectX programming
    By bobbelPoP in forum Windows Programming
    Replies: 16
    Last Post: 07-08-2008, 02:27 AM
  5. Strange error?
    By MrLucky in forum C++ Programming
    Replies: 5
    Last Post: 02-04-2006, 03:01 PM

Tags for this Thread