Note: if this thread belongs elsewhere I apologize.

I am starting out a new project on my own involving what is basically swarm computing and would like to get the general opinion of the forum members on what the best approach to intermachine communication. I have methods of my own but do not consider them optimal and would like to see what others have done in the regard.

There are two levels of messaging I want to address here:
1. General IPC, or sending small, well-defined messages between machines running various OS (Win, Lin and custom OS) so things like endianness cannot be assumed. The messages should be 1K or less. To date what I have been doing is a small socket server and the messages have all been small XML-based examples. I am doing the parsing on the other end so if something doesn't adhere to the XML standard I can deal with that. In this case the messages are of a type "start doing X, here are a few parameters" or broadcast msgs of the type "I am machine XXXX and have capacity to do YYYY work, here are my available services".

Often these smaller msgs will take the form of:
Code:
<message>
     <command>"doJobX"</command>
     <data>somePayload</data>
</message>
The weak points here are many, not the least of which is the parsing chores on the receiving end..

2. This involves moving heavier blocks of data between machines such as files of data. Sadly the approaches I have taken in the past have involved to SSH file stransfers between machines. It works and is secure after a fashion but would like to know if anyone has any other suggestions for this or if there is some kind of standard for doing this sort of thing of which I am unaware. One approach I have tried in the past is network sharing the files with the sibling machines but that brings its own problems to the table, not the least of which is it only works well if all machines are running the same OS (or some variant thereof). Safe key exchange is the achilles heel of the SSH approach for new machines entering the "swarm".

This post is supposed to be a conversation starter with those who have done this before so I can avoid re-inventing the wheel (my wheels are described above).

Thoughts? More details on requirements?

Peace...