As you say, round trips through a void pointer (for anything except a pointer to member) are a specific exception, identified in the standard. That still, however, doesn't allow you to dereference a pointer to char as if it was an int.
Unfortunately, your search will be more difficult than finding a clause that says this is undefined behaviour, because this is one of those things that the standard leaves undefined, by omission, so you won't find a specific clause saying it is undefined.
As per Section 1.3.12 "Undefined behavior" of ISO/IEC 14882 (aka the 1998 C++ standard)
I've added the bold for emphasis.behavior, such as might arise upon use of an erroneous program construct or erroneous data, for which this International Standard imposes no requirements. Undefined behavior may also be expected when this International Standard omits the description of any explicit definition of behavior.
If you do a hunt through, you will find all sorts of things (such as the clauses that result in a round-trip through a void pointer being valid) about what is allowed. It is the omissions that are important (and I assume the omissions occurred because it would have been very difficult to comprehensively articulate all possible cases of undefined behaviour).
Incidentally, I can't claim credit (if credit it is) for this interpretation. Several mainstream compiler vendors, in reference to all C and C++ standard since 1989, have interpreted the standard that way. And I have yet to see anyone find a convincing interpretation of any clause in the standards that allows a rebuttal. Given that history, I might still be wrong in my interpretation, but I won't hold my breath in anticipation of being proven wrong.
