I have been working on an application that is basically a server, a client library and a handful of clients. The interface between the clients and the server is functional, meaning that the clients don't worry about the socket connections they just call API functions to read or write information to the server. The client library would then send/recv a request/response over the already connected socket, and return the result to the calling client. The purpose of the API is to allow other developers to build clients quickly and easily without too much trouble. This already works quite well. I am using a local domain stream socket if the client is on the same host as the server and if it's on another machine it uses TCP/IP.
I would like to add the capability for the server to send messages to the clients asynchronously. Something like event or data change notification. It's causing me some grief because of resource contention issues surrounding the socket. I'm trying to keep this thing as efficient as possible so I'm trying to avoid excessive system calls.
I have a couple of ideas. I don't know if either is any good or which is better so I was hoping to get some advice on the potential pitfalls or benefits of each one.
First is to start another thread from within the library and have it handle communication. The server already requires pthreads so from a dependency perspective this isn't that big of a deal, but it seems like bad form to start threads from within a library. Maybe not. It seems to me that I'll wind up having to use a bunch of system calls and signals to coordinate all this.
The other idea is to simply open a second socket and give the client developer a select/poll kind of interface to the asynchronous messaging. This would probably still require some multi-threading in most clients but it would be at the discretion of the client developer not hidden away inside the library.
I guess a third option is to do both. :-) I'm basically looking for advice from those who have gone before.
I would not use threads as it means you add an 'hidden' mechanism to the client code (it implies the client is not in full control of its resources and may lead to various problems). From this point of view, a select()-like routine is more attractive but 1/ it constrains the client code to an event-based design (your API has to offer a way to setup handlers and one of your API call does the select() call) 2/ even if the client is event-based, your API may miss some routines to support 'unforeseen' events (if the select() is done in one of your API calls).
However you can also adopt a X11-like solution: propose an event-based design but without using directly select() or poll(), simply by proposing a getNextEvent()-like call and a way to get back the file descriptors to use select() if the client wants to. This is the simplest solution and it has no performances impact (the client controls everything). Be sure to make your API routines thread-safe, though (if the client wants to use threads)...or be explicit they are not (or propose 2 versions to avoid performance issues for single-threaded clients, as in the libc).