File storage for embedded

[Elysia]

Also, I get the feeling you can compress the heck out of a lot of that data. The readings from IMU are smoothly varying plus some amount of noise. If you can be lossy you can compress that incredibly well, even if lossless you could probably cut it at least in half. If you can compress it enough, you can store it in RAM.

Yep, I've thought of this. It is something to investigate.
But it is unlikely it can be stored in RAM for very long. 50 KB/s is 180 MB/h. Multiply that by 24 and you've got an insane number. I doubt it's possible to store it in RAM all the time, but buffering it would certainly be possible.

You still need some way to get the data off the device. If not SD or other removable media, then what? USB?

USB is part of the plan, if only for being able to upgrade the firmware. Aside from that, nothing is really determined. But keep in mind that the system should run for a week+ before the data is emptied.

Can you power this thing with a solar panel?

Energy harvesting is a topic in itself. But I think this will be inside, so that probably won't help much. But yes, energy harvesting in some form is being looked into. But no concrete details yet, so I have to make a plan for worst case scenario.

Before you really start worrying about if your storage subsystem will consume too much power you'll need to determine the power consumption of the rest of your system. What are the power requirements of your IMU? Your microprocessor? Your display screen if any? Can you run just these systems without the external memory for the length of time required?

I don't know the power of other parts of the system yet. The CPU will probably be 30 - 90 microamps / MHz. I plan on trying to find out how much frequency we need for the CPU, but it's still not finished since we don't have the code or, more importantly, the data to work on yet.
I just want a general approach and techniques whose power consumption I can test. Can't make a real plan without testing the entire system, but you need to plan ahead first.

Can your IMU actually process and transmit it's data at the data rate you're talking about? Can you tell us a little more about your IMU?

Apparently it can sample at around 3 kHz or 10 kHz, depending on what exactly it's sampling. I'm not 100% sure about that, though.
We're also testing several different ones. but I can try to get the exact models and data sheet specs for the ones we're going to test right now.

As far as the external storage, you can possibly get by with using a large RAM buffer, only writing to the external storage when your internal buffer reaches it's limit. This can greatly reduce your power consumption by keeping the external storage powered most of the time.

I was thinking this, as well. I think this probably is going to be the best approach.
I'm thinking to buffer the data in the CPU embedded ram, then flush it to external ram and finally flush it to storage or wireless modem.
For this purpose, I would need a quick bus that would be powered on only when I need to flush everything. AMBA sounds nice, but I dunno how many smaller microprocessors support it. I've only one so far (I think?), but it's way too power hungry (the cpu, that is). I'm only familiar with Atmel and TI processors, though.


tl;dr:
I don't know the power consumption of the rest of the system. I need to plan ahead on strategies, buses, RAM and the like to ensure we can get the proper parts and that they will work with each other.

[brewbuck]

Have you looked at SnapDragon?

Snapdragon 800, 600, 400, 200 Processor Specs | Mobile Technology | Qualcomm Snapdragon Processors

[Elysia]

No. These things are high-power, high performance processors. I need a low-power, low performance. I love to be proved wrong, but I can't see the power consumption on their homepage.
50 MHz is probably more than I'll ever need, and I need consumption to be measured in microamps per MHz.

[Elkvis]

arduino/netduino?

[brewbuck]

No. These things are high-power, high performance processors. I need a low-power, low performance. I love to be proved wrong, but I can't see the power consumption on their homepage.
50 MHz is probably more than I'll ever need, and I need consumption to be measured in microamps per MHz.

The SnapDragon can be massively down-clocked, which is what happens when a smartphone sleeps for instance. You ain't going to be running it at 900 MHz... The specs quoted on Qualcomm's site are the maximums when the thing is fully ripped. I don't know if it meets your power requirements either, but I wouldn't discount it immediately. If you want to use WiFi, Bluetooth, or GPS you'll probably save power by going with a SoC that already has those things onboard.

I'm not trying to pimp SnapDragon, it's just what I've been working with recently and so it's fresh in my mind.

[Elysia]

I'm not trying to pimp SnapDragon, it's just what I've been working with recently and so it's fresh in my mind.

It's alright to pimp things. I don't mind.
Well, let's try to contact them and see what happens.

arduino/netduino?

We actually have some of these already.
But we're probably going full custom in the end, so good for testing, but not so much for the final product.

[gemera]

The other kind of thing I'd like to ask is about file systems. Does anyone know what alternatives are out there and if they licenses, etc? My only requirement is that it probably should run on Windows. I'd like to avoid filesystems that are not native to windows (eg ext* family), if possible. I'm not sure if there are good existing solutions (ie controllers or such) for writing to common filesystems out there? Writing custom code to write systems is ... unpleasant, to say the least. Equally unpleasant is having to write a driver for Windows.
I'd just like an overview of what's out there and what alternatives exist.

Thanks.

Don't know if I'm stating the obvious here - but Linux has a layer of abstraction built into its filesystem so it's not just compatible with the ext* family but a long list of filesystems including FAT32, nfs, etc.

[Elkvis]

We actually have some of these already.
But we're probably going full custom in the end, so good for testing, but not so much for the final product.

you can get raw arduino chips to integrate into whatever device you choose to build.

[Elysia]

Got a few raspberries which we're going to use to build the first prototype.
I just wish it was easy to find components and some bloody software to simulate everything. These stupid data sheets are so lacking when it comes to crucial information (such as power consumption).

[brewbuck]

Got a few raspberries which we're going to use to build the first prototype.
I just wish it was easy to find components and some bloody software to simulate everything. These stupid data sheets are so lacking when it comes to crucial information (such as power consumption).

Welcome to embedded development, heh. Most of my knowledge is hard-won from experience.

Depending on what kind of simulation you want, I might have a few suggestions. Or maybe I'll be useless, who knows.

The idea of compressing the IMU output is still floating through my head. Have you looked at that? If you can compress it (without using too much CPU power) that might save power by reducing time on the bus and the number of bytes you have to write to storage.

[Elysia]

The idea of compressing the IMU output is still floating through my head. Have you looked at that? If you can compress it (without using too much CPU power) that might save power by reducing time on the bus and the number of bytes you have to write to storage.

Yes, I have thought about that. Since it's mostly repeating text (a lot of similar bit patterns hopefully), it should compress well, so just taking the contents of a memory and compressing it before sending it might save power in the end.
But I'm not at a point where I can test that. But maybe we're getting there. I'll at least see how much compression can be gained.

The problem is basically that I have no idea how much power the system will consume. If I could have estimates of everything, it would be so much easier.
How much power would the CPU use? The IMU? The I2C bus? The SPI/AMBA bus? The SD card/wireless modem? There is no hard data anywhere, and that's why it would be so nice with some simulation.