Implementing a simple transfer protocol

I'm on my quest on building a digital scope with a microcontroller on the serial port and c++.

The code on the uc is written, and will need various adjustments, now comes the real hard part - implementing a PC side software.

I'm thinking of how to send and process the data. Basically, I send at first two bytes, one which will tell the IC how many pins will be read, and then another byte for "start".

Then, the uc sends ascii 65 (packet start), then two characters for each pin's value, then ascii 90 (packet end).

I'm thinking of enhancing the protocol. Surely, there will be missed bytes, which will render this bad. How exactly can I detect when there will be skipped bytes, and how should the software react?

For the moment, I'm thinking of adding a "time" value to each sample, so in case a few samples are missed, the current sample is added a bit ahead of the last good one, and the missed data can be marked as bad, or it can be interpolated.

I'm also thinking - should I buffer all this data? I'm thinking the best way to do it this way is a circular/ring buffer. Another idea is using a fixed size deque as a buffer. And there's also a standard class for the last one.

Moved to Device Communication

OK, so are you expecting the values 65 and 90 to occur in the data-stream? If so, you may want to prefix and modify those data items in the protocol - that way, if data IS lost, you can sync back up on the next packet, rather than getting lost for a long time because you are indeed getting the correct data in, but your code can't find the start/end markers correctly.

E.g use 66 as a marker for the next character is one of my special ones, modified so it doesn't look special any more - the simplest way to encode it is to add one, so I'll use that method:
Original data: 44 18 65 93 90
Becomes: 65 44 18 66 66 93 66 91 90

Also having a length in the data-packet is a good idea unless all packets are of identical length.

Do you acknowledge packets on either end, e.g. if you send a packet from the device, does the device wait for a reply saying "I Got that!".

Using a sequence number in the packet is also a good idea, as well as some form of checksum (basic checksum of just adding the data together works fine), so if some spike or disruption causes the data to be corrupted, you can tell, and discard the broken packet.

--
Mats

Thanks Mats! To be honest, I realised I have a lot of stuff to figure out before I can actually start doing this. It's the first time I'm writing this kind of software :)
I think I should have asked for a more general guideline in designing the protocol.

I'm aiming for speed (at least now). And i'm storing/buffering the data. Using an 'escape' character for the special characters seems like a good idea, but it also uses a some 'bandwidth'.

I was thinking of ack's too, but only as a secondary option. It might eat up too much time. After all, I'm trying to make an oscilloscope here, and it should easily be able to put up with 1khz.

I can change those 65 and 90 values, and i can even make a more complex protocol on the device side - since i'm programming that too. I used 65 and 90 since they represent "A" and "Z", a pretty way of saying 'start' and 'done'.

So. uh. if it were you to build this protocol from scratch, how would you do it?
(I'm not asking for ultra details here, just a guideline :))
Thanks!

Well, as I was part of a team designing "Home-made protocol", I suppose I can share some knowledge. These are more random ramblings than a formal design document, but it may help you somewhat.

If you are using this locally [PC is standing within some feet/a few meters of the other device], it should be reasonably OK to expect very few missed characters (I've used a 486 as a line-listener using 2 serial ports to send/receive/log simultaneously, and that didn't have a problem to keep up with 115200 on both serial ports, and it didn't miss anything, so a modern PC processor should have lots of processing power to spare).

The reason I suggested that you use a distinctive start/end character is simply that it makes life easier for your protocol to recover. The method of escaping certain characters is not adding much overhead unless packets frequently contain those characters - if the data is "random", then there is of course a chance that every character/byte needs escaping, but it's unlikely to be a common case unless you choose something "silly" as your escape or start/end characters.

As to what features you want in the protocol itself, it all depends on how reliable you want it, and how often you expect errors. Sequence numbering the packets is a way to detect that a packet went missing. You can then either ask for that packet to be resent, or ignore it. Likewise, checksum will make the protocol stronger, but if it's unlikely that the packets are broken, then it's not adding a big benefit.

If your packets are always the same length, then you can safely ignore having a length as part of the packet. As an alternative, you could also have a packet-type that implicitly indicates the length. The benefit of knowing the length is similar to other protocol enhancements in that it gives you the chance to verify that the packet you got is complete and therefore less likely to be incorrect.

There are essentially some choices:
- Checksum or no checksum. Checksum adds extra overhead in the data as well as needing to be calculated.
- Make start & end markers unique or not - the former means that you NEED to ensure that the end-marker is never in the actual data. If they are not unique, then I don't really see the point of having them at all, as you can't distinguish an end-marker from some random data in the data-packet.
- Sequence numbers or no sequence numbers. Adds overhead, but benefit is that you can tell if you missed a packet.
--- If you implement sequence numbers, you may also have a "resend the previous packet" command.

Some comments to your first post, as to how to deal with the data. On the receiving side (PC), you probably just read the data into a large enough buffer to hold a packet and send it to another thread for processing.

On teh sending side, you probably want to build packets and send them from a queue of some sort (deque sounds about right, stick a packet on the back, remove packets when sent from the front). You may want to have a scheme where you use the serial interrupt to pick off the packet from the queue - when the current packet is empty, pick up the next one.

Please feel free to ask questions if you have any.

--
Mats

Uh well, I don't get one thing - since a packet is formed from a few bytes (using 9600 - for now - 8, N, 1, btw), how exactly would I know what is dropped from the data stream?

For example, I could simply send data without any headers - raw info. If I have two pins scanned, that's four bytes. The problem is that, if one byte is missed, weird values will end up (part from the current packet, part from the next packet). So, the PC can send from time to time a "sync" - where the device will send the bytes in a given order.

Another thing I'm thinking of is using header, but simply counting them. For example, if the device is only scanning two pins, it can alwasy send a sync byte, and then the four bytes of data. Of course, these will be a bit modified (as you suggested, by altering the data in a known fashion and adding an escape char for example) so nothing will interfere with the sync byte.

The PC side will wait for a sync byte, and it will discard everything if the fifth character after a sync byte is not a sync byte. Where should I process this? The receiving thread should simply put out everything in a buffer, or process the data, and only put the sampled values?

Another thing - about the deque on the sending side, do you think that's necessary? I was talking about a queue on the receiving side. The sending device is a PIC microcontroller which can't operate higher than 4Mhz.. and a bunch of this cycles are eaten up by the sending algo, and another bunch by the A/D conversion, so I really don't think things will need to queue up here :) what do you think?

Quote:

---

Originally Posted by _izua_

Uh well, I don't get one thing - since a packet is formed from a few bytes (using 9600 - for now - 8, N, 1, btw), how exactly would I know what is dropped from the data stream?

---

If you don't have a way of knowing what data to expect, then you don't know if you dropped something - that's the whole point of having "marker" characters.

Quote:

---

For example, I could simply send data without any headers - raw info. If I have two pins scanned, that's four bytes. The problem is that, if one byte is missed, weird values will end up (part from the current packet, part from the next packet). So, the PC can send from time to time a "sync" - where the device will send the bytes in a given order.

---

That's one possible solution. But using a sync at each end of the packet is probably a better choice.

Quote:

---

Another thing I'm thinking of is using header, but simply counting them. For example, if the device is only scanning two pins, it can alwasy send a sync byte, and then the four bytes of data. Of course, these will be a bit modified (as you suggested, by altering the data in a known fashion and adding an escape char for example) so nothing will interfere with the sync byte.

The PC side will wait for a sync byte, and it will discard everything if the fifth character after a sync byte is not a sync byte. Where should I process this? The receiving thread should simply put out everything in a buffer, or process the data, and only put the sampled values?

---

I would receive the data in one thread, including the verification and decoding of packet information.

Quote:

---

Another thing - about the deque on the sending side, do you think that's necessary? I was talking about a queue on the receiving side. The sending device is a PIC microcontroller which can't operate higher than 4Mhz.. and a bunch of this cycles are eaten up by the sending algo, and another bunch by the A/D conversion, so I really don't think things will need to queue up here :) what do you think?

---

The point of having a queue at the sending side is that you can start sending the data before it's complete - rather than preparing the whole packet and then starting the transmission. Keeping to a simple scheme using a circular buffer of 2^n would also work (using a 2^n size means that you can use AND to reset to the beginning of the circular buffer, rather than a bunch of compares and/or divide(modulo) operations - which is generally slower.

--
Mats