File storage for embedded

This is a discussion on File storage for embedded within the Tech Board forums, part of the Community Boards category; I am looking at writing to some kind of storage in an embedded system and thought I'd ask if anyone ...

  1. #1
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,612

    File storage for embedded

    I am looking at writing to some kind of storage in an embedded system and thought I'd ask if anyone has any advice here since I am unfamiliar with the technologies that exist in this area for embedded.

    The thing here is that this embedded system will run on battery and must run for a week to a month, so power consumption is crucial. I took a look at SD cards, but the power consumption is horrible (at least ~20 mA) so that's out of the question. Question is if there are any other low consumption alternatives out there?

    The other kind of thing I'd like to ask is about file systems. Does anyone know what alternatives are out there and if they licenses, etc? My only requirement is that it probably should run on Windows. I'd like to avoid filesystems that are not native to windows (eg ext* family), if possible. I'm not sure if there are good existing solutions (ie controllers or such) for writing to common filesystems out there? Writing custom code to write systems is ... unpleasant, to say the least. Equally unpleasant is having to write a driver for Windows.
    I'd just like an overview of what's out there and what alternatives exist.

    Thanks.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  2. #2
    Registered User
    Join Date
    May 2010
    Posts
    2,753
    I am looking at writing to some kind of storage in an embedded system
    What do you consider an embedded system? Today embedded systems start at small microprocessors with memory in the range of tens of bytes to several gigabytes, many times there is no actual operating system.

    The thing here is that this embedded system will run on battery and must run for a week to a month, so power consumption is crucial.
    The thing you need to consider is how often do you need to write to this "file system" no just the power consumption. After all one way of saving power is to power off unneeded subsystems when they are not in use. The less time they are running the less power they use.

    My only requirement is that it probably should run on Windows.
    Why? What exactly are you trying to accomplish? Do you actually need any operating system at all?

    If you must use an existing file system, but still need simple, consider Fat16. There are many open source implementations for this file system that are available for use and it is available for many different processors and this file system can still be read by Windows.

    Without knowing more information it's hard to point you to any one solution. Also just stating that the embedded product must run from a battery for a month or more doesn't mean much, remember batteries run the gamut from a "watch" to a series of large capacity batteries that can power a normal PC for multiple years.

    Jim

  3. #3
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,612
    It's a university project and the client wants it to run on battery all the while sampling necessary data which can later be processed and viewed.
    So:
    - Yes, it needs to be an embedded system. That's a requirement.
    - It needs to run on battery and it should be a week to a month.
    - The data that is collected should be able to be viewed later. It can be on any device: laptop, tablet, phone. I gather easiest is just Windows. That's what I am familiar with. Besides, the embedded system has a budget. The less we spend on that, the better.
    - The weight of the system should be around 300 g.
    - We probably don't need an OS on the embedded system.
    - You make a good point about how often it writes to the SD card. At the moment, I'm guessing it will write around 50 KB/s. But that's guesswork without any backup (it's a calculation from sampling an IMU @ 1 kHz).

    I'm going to redo some calculations, but it would be good if I knew more about the power consumption characteristics of SD cards. Setup consumption, for example.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  4. #4
    Epy
    Epy is offline
    Fortran lover Epy's Avatar
    Join Date
    Sep 2009
    Location
    California, USA
    Posts
    963
    Some of the smaller embedded boards I've looked at recently have a hardwired flash chip on the board which can be used for storage (as an alternative to SD).

    Edit: Also, is the SD card really that horrible with respect to the rest of the system? I would think that the CPU would be the main draw of power, unless you had a really small CPU like a PIC or an AVR chip.
    Last edited by Epy; 09-24-2013 at 03:24 PM.

  5. #5
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,612
    Quote Originally Posted by Epy View Post
    Some of the smaller embedded boards I've looked at recently have a hardwired flash chip on the board which can be used for storage (as an alternative to SD).
    Dunno if we're going to use an embedded board as a base or just design the whole things ourselves. That's too much up in the air.

    Quote Originally Posted by Epy View Post
    Edit: Also, is the SD card really that horrible with respect to the rest of the system? I would think that the CPU would be the main draw of power, unless you had a really small CPU like a PIC or an AVR chip.
    The processors I've looked at have 30-90 microamps / MHz in power consumption and I doubt we need so much in terms of MHz.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  6. #6
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,612
    Looking further, it looks like I2C can only handle about 400 KB/s, which is way too low to manage any good power consumption with SD cards. So maybe another good low power bus out there with higher bandwidth would be good? If so, are there any good/recommended candidates? I only wish I knew a good source for this stuff.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  7. #7
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,239
    An embedded system that can run for a week and access an SD card matches the description of my smartphone (as long as it's kept unconnected and the screen is off, it'll go about 5 or 6 days). You haven't described the typical power use profile of your application or how often you might need to access the SD card. You do not have to keep the SD powered up unless you are using it.

    I don't have charts of per-byte or per-byte-per-second energy consumption across various storage devices. I haven't seen it called out specifically in any specs I've looked at. Collecting that information would be a good exercise, but you may need to be empirical about it. If you look at a representative dev board like the BeagleBone Black, these typically have either SD or MMC or both (in the case of the BBBlack), but they are not particularly designed for low-power.

    For filesystem, FAT32 is widely supported and Windows-compatible. It is reasonably simple, which reduces code complexity and hopefully power use.

    What are you doing? 50 kb/sec is just about the rate of monaural audio data at 44.1 khz, so I'm going to wildly guess that you're sampling audio.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  8. #8

  9. #9
    11DE784A SirPrattlepod's Avatar
    Join Date
    Aug 2013
    Posts
    485
    Quote Originally Posted by brewbuck View Post
    For filesystem, FAT32 is widely supported and Windows-compatible. It is reasonably simple, which reduces code complexity and hopefully power use.
    I've come across FAT12 and FAT16 embedded but I suppose FAT32 might be an option as well. As you said the FAT filesytems are widely supported and probably the route I'd take

  10. #10
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,612
    Quote Originally Posted by brewbuck View Post
    An embedded system that can run for a week and access an SD card matches the description of my smartphone (as long as it's kept unconnected and the screen is off, it'll go about 5 or 6 days). You haven't described the typical power use profile of your application or how often you might need to access the SD card. You do not have to keep the SD powered up unless you are using it.
    If I am a little unforthcoming about details, it's because I don't know how much I can say. Some information is confidential.
    This much I know. The goal is to sample an IMU @ 1 ~kHz and a GPS for now. This data needs to be processed either on-board or later (depending on how feasible it is).

    If it's on-board, then I don't know how much data it will spit out. That is too hard to estimate right now.
    If it's not processed on-board, then the raw data needs to possibly be stored for a week up to a month. The card would then be removed from the embedded system, plugged into a computer and batch processed by an application on the PC.
    Another option is to transmit the data continuously via mobile wireless technologies to some server that processes the data.

    In the end, the data from the IMU and GPS needs to be processed and the system should run for a week to a month without charge. That is the goal.
    All this should run on a battery (probably Li-Ion or Li-Po). The entire system should weight something around 300 g, because it needs to be mounted, so the battery can be around 10-13 Wh (roughly 100 g according to some sources). This is a rough initial estimate.

    I don't have charts of per-byte or per-byte-per-second energy consumption across various storage devices. I haven't seen it called out specifically in any specs I've looked at. Collecting that information would be a good exercise, but you may need to be empirical about it. If you look at a representative dev board like the BeagleBone Black, these typically have either SD or MMC or both (in the case of the BBBlack), but they are not particularly designed for low-power.
    We're doing initial information gathering. What is feasible? What might we need to test? Empirical and statistical hard quantitative data is not something we have or can get right now.
    I need a little info to answer the above questions. Is it feasible? Or not? Do we maybe need something else than a I2C bus? Do we need something else than an SD card?
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  11. #11
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,239
    Quote Originally Posted by Elysia View Post
    We're doing initial information gathering. What is feasible? What might we need to test? Empirical and statistical hard quantitative data is not something we have or can get right now.
    I need a little info to answer the above questions. Is it feasible? Or not? Do we maybe need something else than a I2C bus? Do we need something else than an SD card?
    I don't know. I'm not sure how any of this can be answered when you don't even know if you're making a board or using something that's already available. As far as fast buses, AMBA is pretty standard and SD parts are available that use AHB.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  12. #12
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,612
    Quote Originally Posted by brewbuck View Post
    I don't know. I'm not sure how any of this can be answered when you don't even know if you're making a board or using something that's already available.
    And I don't know either. That's the fun part, isn't it?
    Everything will be fleshed out as everything goes along. But for now, good components whose interfaces match each other would be a good idea.
    I'll take a look at the power characteristics of AMBA.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  13. #13
    Registered User
    Join Date
    Nov 2010
    Location
    Long Beach, CA
    Posts
    5,498
    Consider SPI as an alternative to I2C for on-board flash. Don't know how they compare on speed and power, but you can look into it. Depending on the route you go, some microcontrollers have built-in flash you can use for non-volatile storage, so you don't even need to worry about protocol, selecting a chip, etc. Though I doubt you would get much beyond 1MB there. At ~1kHz, that's not much. Another alternative is (depending on your situation) memory mapped storage. If you have enough ram, you can just let the data sit there (maybe using SRAM for power savings), or only saving to flash as needed. You could see if you can design it so that you can detect low battery and only dump to flash when you drop below some threshold. Just some (very poorly thought out) thoughts.

  14. #14
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,239
    Also, I get the feeling you can compress the heck out of a lot of that data. The readings from IMU are smoothly varying plus some amount of noise. If you can be lossy you can compress that incredibly well, even if lossless you could probably cut it at least in half. If you can compress it enough, you can store it in RAM.

    You still need some way to get the data off the device. If not SD or other removable media, then what? USB?

    Can you power this thing with a solar panel?
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  15. #15
    Registered User
    Join Date
    May 2010
    Posts
    2,753
    Before you really start worrying about if your storage subsystem will consume too much power you'll need to determine the power consumption of the rest of your system. What are the power requirements of your IMU? Your microprocessor? Your display screen if any? Can you run just these systems without the external memory for the length of time required?

    Can your IMU actually process and transmit it's data at the data rate you're talking about? Can you tell us a little more about your IMU?

    As far as the external storage, you can possibly get by with using a large RAM buffer, only writing to the external storage when your internal buffer reaches it's limit. This can greatly reduce your power consumption by keeping the external storage powered most of the time.

    Jim

Page 1 of 2 12 LastLast
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Reading and writing to an embedded file
    By HLMetroid in forum C# Programming
    Replies: 4
    Last Post: 01-01-2009, 11:03 PM
  2. Txt file and data storage
    By radnik in forum C++ Programming
    Replies: 52
    Last Post: 06-24-2008, 09:16 AM
  3. Embedded file (C++ and SQL)
    By Opel_Corsa in forum C++ Programming
    Replies: 4
    Last Post: 12-11-2006, 06:30 AM
  4. File storage
    By lyx in forum Windows Programming
    Replies: 4
    Last Post: 09-27-2003, 02:56 PM
  5. Using string type in class for file storage
    By johnnyd in forum C++ Programming
    Replies: 9
    Last Post: 03-11-2003, 11:11 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21