...and thus the advice to always use non-blocking sockets with select(). I'd give the same advice here, since that's really what you want: Mark the socket non-blocking, and forget select. Just call accept(). If you get EWOULDBLOCK, then obviously there's nobody there.
Under Linux, select() may report a socket file descriptor as "ready for reading", while nevertheless a subsequent read blocks. This could for example happen when data has arrived but upon examination has wrong checksum and is discarded. There may be other circumstances in which a file descriptor is spuriously reported as ready. Thus it may be safer to use O_NONBLOCK on sockets that should not block.