I've been looking around and it was a surprisingly complicated matter. Most things seem to be for Java and C#. What I've found is:
zlib can't handle zip files by itself. I checked contrib/minizip like the FAQ said and... ouch. Looks like you have to deal with the zip headers yourself, and decode the rest with zlib.
7z's libraries I read could handle both rar and zip, but evidently the testprogram that came with lzma couldn't.
unrarlib is heavily outdated.

I use MinGW and preferably the library(ies) should be DLLs. I'm thinking of making some that can read comic book archives, which are just normal archives of different types with renamed extensions.