File storage for embedded

[Elysia]

I am looking at writing to some kind of storage in an embedded system and thought I'd ask if anyone has any advice here since I am unfamiliar with the technologies that exist in this area for embedded.

The thing here is that this embedded system will run on battery and must run for a week to a month, so power consumption is crucial. I took a look at SD cards, but the power consumption is horrible (at least ~20 mA) so that's out of the question. Question is if there are any other low consumption alternatives out there?

The other kind of thing I'd like to ask is about file systems. Does anyone know what alternatives are out there and if they licenses, etc? My only requirement is that it probably should run on Windows. I'd like to avoid filesystems that are not native to windows (eg ext* family), if possible. I'm not sure if there are good existing solutions (ie controllers or such) for writing to common filesystems out there? Writing custom code to write systems is ... unpleasant, to say the least. Equally unpleasant is having to write a driver for Windows.
I'd just like an overview of what's out there and what alternatives exist.

Thanks.

[jimblumberg]

I am looking at writing to some kind of storage in an embedded system

What do you consider an embedded system? Today embedded systems start at small microprocessors with memory in the range of tens of bytes to several gigabytes, many times there is no actual operating system.

The thing here is that this embedded system will run on battery and must run for a week to a month, so power consumption is crucial.

The thing you need to consider is how often do you need to write to this "file system" no just the power consumption. After all one way of saving power is to power off unneeded subsystems when they are not in use. The less time they are running the less power they use.

My only requirement is that it probably should run on Windows.

Why? What exactly are you trying to accomplish? Do you actually need any operating system at all?

If you must use an existing file system, but still need simple, consider Fat16. There are many open source implementations for this file system that are available for use and it is available for many different processors and this file system can still be read by Windows.

Without knowing more information it's hard to point you to any one solution. Also just stating that the embedded product must run from a battery for a month or more doesn't mean much, remember batteries run the gamut from a "watch" to a series of large capacity batteries that can power a normal PC for multiple years.

Jim

[Elysia]

It's a university project and the client wants it to run on battery all the while sampling necessary data which can later be processed and viewed.
So:
- Yes, it needs to be an embedded system. That's a requirement.
- It needs to run on battery and it should be a week to a month.
- The data that is collected should be able to be viewed later. It can be on any device: laptop, tablet, phone. I gather easiest is just Windows. That's what I am familiar with. Besides, the embedded system has a budget. The less we spend on that, the better.
- The weight of the system should be around 300 g.
- We probably don't need an OS on the embedded system.
- You make a good point about how often it writes to the SD card. At the moment, I'm guessing it will write around 50 KB/s. But that's guesswork without any backup (it's a calculation from sampling an IMU @ 1 kHz).

I'm going to redo some calculations, but it would be good if I knew more about the power consumption characteristics of SD cards. Setup consumption, for example.

[Epy]

Some of the smaller embedded boards I've looked at recently have a hardwired flash chip on the board which can be used for storage (as an alternative to SD).

Edit: Also, is the SD card really that horrible with respect to the rest of the system? I would think that the CPU would be the main draw of power, unless you had a really small CPU like a PIC or an AVR chip.

[Elysia]

Some of the smaller embedded boards I've looked at recently have a hardwired flash chip on the board which can be used for storage (as an alternative to SD).

Dunno if we're going to use an embedded board as a base or just design the whole things ourselves. That's too much up in the air.

Edit: Also, is the SD card really that horrible with respect to the rest of the system? I would think that the CPU would be the main draw of power, unless you had a really small CPU like a PIC or an AVR chip.

The processors I've looked at have 30-90 microamps / MHz in power consumption and I doubt we need so much in terms of MHz.

[Elysia]

Looking further, it looks like I2C can only handle about 400 KB/s, which is way too low to manage any good power consumption with SD cards. So maybe another good low power bus out there with higher bandwidth would be good? If so, are there any good/recommended candidates? I only wish I knew a good source for this stuff.

[brewbuck]

An embedded system that can run for a week and access an SD card matches the description of my smartphone (as long as it's kept unconnected and the screen is off, it'll go about 5 or 6 days). You haven't described the typical power use profile of your application or how often you might need to access the SD card. You do not have to keep the SD powered up unless you are using it.

I don't have charts of per-byte or per-byte-per-second energy consumption across various storage devices. I haven't seen it called out specifically in any specs I've looked at. Collecting that information would be a good exercise, but you may need to be empirical about it. If you look at a representative dev board like the BeagleBone Black, these typically have either SD or MMC or both (in the case of the BBBlack), but they are not particularly designed for low-power.

For filesystem, FAT32 is widely supported and Windows-compatible. It is reasonably simple, which reduces code complexity and hopefully power use.

What are you doing? 50 kb/sec is just about the rate of monaural audio data at 44.1 khz, so I'm going to wildly guess that you're sampling audio.

[SirPrattlepod]

AN10916: FAT library EFSL and FatFs port on NXP LPC1700 | www.LPCware.com

[SirPrattlepod]

For filesystem, FAT32 is widely supported and Windows-compatible. It is reasonably simple, which reduces code complexity and hopefully power use.

I've come across FAT12 and FAT16 embedded but I suppose FAT32 might be an option as well. As you said the FAT filesytems are widely supported and probably the route I'd take

[Elysia]

An embedded system that can run for a week and access an SD card matches the description of my smartphone (as long as it's kept unconnected and the screen is off, it'll go about 5 or 6 days). You haven't described the typical power use profile of your application or how often you might need to access the SD card. You do not have to keep the SD powered up unless you are using it.

If I am a little unforthcoming about details, it's because I don't know how much I can say. Some information is confidential.
This much I know. The goal is to sample an IMU @ 1 ~kHz and a GPS for now. This data needs to be processed either on-board or later (depending on how feasible it is).

If it's on-board, then I don't know how much data it will spit out. That is too hard to estimate right now.
If it's not processed on-board, then the raw data needs to possibly be stored for a week up to a month. The card would then be removed from the embedded system, plugged into a computer and batch processed by an application on the PC.
Another option is to transmit the data continuously via mobile wireless technologies to some server that processes the data.

In the end, the data from the IMU and GPS needs to be processed and the system should run for a week to a month without charge. That is the goal.
All this should run on a battery (probably Li-Ion or Li-Po). The entire system should weight something around 300 g, because it needs to be mounted, so the battery can be around 10-13 Wh (roughly 100 g according to some sources). This is a rough initial estimate.

I don't have charts of per-byte or per-byte-per-second energy consumption across various storage devices. I haven't seen it called out specifically in any specs I've looked at. Collecting that information would be a good exercise, but you may need to be empirical about it. If you look at a representative dev board like the BeagleBone Black, these typically have either SD or MMC or both (in the case of the BBBlack), but they are not particularly designed for low-power.

We're doing initial information gathering. What is feasible? What might we need to test? Empirical and statistical hard quantitative data is not something we have or can get right now.
I need a little info to answer the above questions. Is it feasible? Or not? Do we maybe need something else than a I2C bus? Do we need something else than an SD card?

[brewbuck]

We're doing initial information gathering. What is feasible? What might we need to test? Empirical and statistical hard quantitative data is not something we have or can get right now.
I need a little info to answer the above questions. Is it feasible? Or not? Do we maybe need something else than a I2C bus? Do we need something else than an SD card?

I don't know. I'm not sure how any of this can be answered when you don't even know if you're making a board or using something that's already available. As far as fast buses, AMBA is pretty standard and SD parts are available that use AHB.

[Elysia]

I don't know. I'm not sure how any of this can be answered when you don't even know if you're making a board or using something that's already available.

And I don't know either. That's the fun part, isn't it?
Everything will be fleshed out as everything goes along. But for now, good components whose interfaces match each other would be a good idea.
I'll take a look at the power characteristics of AMBA.

[anduril462]

Consider SPI as an alternative to I2C for on-board flash. Don't know how they compare on speed and power, but you can look into it. Depending on the route you go, some microcontrollers have built-in flash you can use for non-volatile storage, so you don't even need to worry about protocol, selecting a chip, etc. Though I doubt you would get much beyond 1MB there. At ~1kHz, that's not much. Another alternative is (depending on your situation) memory mapped storage. If you have enough ram, you can just let the data sit there (maybe using SRAM for power savings), or only saving to flash as needed. You could see if you can design it so that you can detect low battery and only dump to flash when you drop below some threshold. Just some (very poorly thought out) thoughts.

[brewbuck]

Also, I get the feeling you can compress the heck out of a lot of that data. The readings from IMU are smoothly varying plus some amount of noise. If you can be lossy you can compress that incredibly well, even if lossless you could probably cut it at least in half. If you can compress it enough, you can store it in RAM.

You still need some way to get the data off the device. If not SD or other removable media, then what? USB?

Can you power this thing with a solar panel?

[jimblumberg]

Before you really start worrying about if your storage subsystem will consume too much power you'll need to determine the power consumption of the rest of your system. What are the power requirements of your IMU? Your microprocessor? Your display screen if any? Can you run just these systems without the external memory for the length of time required?

Can your IMU actually process and transmit it's data at the data rate you're talking about? Can you tell us a little more about your IMU?

As far as the external storage, you can possibly get by with using a large RAM buffer, only writing to the external storage when your internal buffer reaches it's limit. This can greatly reduce your power consumption by keeping the external storage powered most of the time.

Jim