I realize I am rather late to the party, and I do not know if it helps any, but the book
Linkers and Loaders by John Levine covers the structure of ELF32 in detail. It is an older book, so it doesn't really discuss ELF64 , but it gives a reasonable description of the ELF format itself, and even discusses how an ELF file would be generated, though only in general terms. The code examples from the book can be found for free on
Levine's own book page.
While the book isn't exactly cheap, you can probably find a copy of it in any university library, if you have access to one.
The
OSDev wiki page on the ELF format also covers the format itself in detail.
You can go over the
ELF64 specification as well, if you like.