I'm thinking about writing a USB webcam (USB video class) driver for an embedded platform. Probably just going to implement things used by my webcam (only MJPEG payload, etc).

I have very little experience with USB, but I have skimmed over the specifications and it doesn't seem overly complex and mostly make sense.

The UVC document is fairly long... but I'm hoping I won't have to implement most things, as all I need to do is read MJPEG frames from the webcam, and write them directly to SD card. No decoding required.
USB.org - Approved Device Class Document Download

Has anyone done anything like this before? What's the most challenging part? Anything easily overlooked?

Thanks