select() destroys the fds bitset and you never reinitialize it. You have to do it every time through the outermost loop.

There might be other problems -- I haven't looked carefully.