Thread: Debian on a system with Compact Flash, without Hard Disk

  1. #1
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251

    Debian on a system with Compact Flash, without Hard Disk

    I have been told that Debian on an industrial PC with a Comapct flash and without hard disk is not a great idea cause the OS writes often on the main disk and this would hit with flash limited write amount
    is that correct?
    I am preparing a prototype for driving cameras and analyzing images. I do not need much stuff to run. Just my threads, a few servers like ssh or ftp and very probably also Eclipse to avoid cross compiling.
    Cannot I just install the minimal needed so that probably the OS will not write much on flash and corrupt it?

    Thank you

  2. #2
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    I'm sure there is a flash-filesystem you can use in Linux, but I wouldn't worry about it too much - modern flash devices are writable [guaranteed/typical] a million times or more - so even if you write every second to the same sector, it would last a few weeks (around 12 days).

    CF cards cost near nothing, so for now it wouldn't be a problem, right?

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  3. #3
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251
    Quote Originally Posted by matsp View Post
    I'm sure there is a flash-filesystem you can use in Linux, but I wouldn't worry about it too much - modern flash devices are writable [guaranteed/typical] a million times or more - so even if you write every second to the same sector, it would last a few weeks (around 12 days).

    CF cards cost near nothing, so for now it wouldn't be a problem, right?

    --
    Mats
    well so and so
    I would like to better investigate this problem

  4. #4
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    The other point is of course that on such a system, I expect you'd be using a RAM disk to store temporary data, etc [and you probably don't want to "syslog" to your CF device either - at least not once you've got a working system - and even boot-debug etc, I'd prefer to send the console output to a serial port than to rely on what was stored by syslog in the filesystem - it's usually missing the very last dozen lines or so, which is the part where it crashes.].

    So what do you expect to write to the CF device very much?

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  5. #5
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251
    Quote Originally Posted by matsp View Post
    The other point is of course that on such a system, I expect you'd be using a RAM disk to store temporary data, etc [and you probably don't want to "syslog" to your CF device either - at least not once you've got a working system - and even boot-debug etc, I'd prefer to send the console output to a serial port than to rely on what was stored by syslog in the filesystem - it's usually missing the very last dozen lines or so, which is the part where it crashes.].

    So what do you expect to write to the CF device very much?

    --
    Mats

    My process would save some important data (a few files)say in average 100 times a day
    For sure I would install a quite big RAM disk (I should know more or less how to do that, even if I did only on a disk with PC)
    The problems are two:
    (1) how to configure the OS so that its processes save the least (no logs etc.)
    (2) Can Debian be installed on a disk less system or do I have to look for some not very common and tested minimal Linux distributions?

    Another doubt. Maybe I can find some alternative to a flash, like some USB device?
    My problem is that this prototype has to run with relatively high vibrations so an hard disk is not a great idea

  6. #6
    Cat without Hat CornedBee's Avatar
    Join Date
    Apr 2003
    Posts
    8,895
    Are there external solid-state disks? SSDs are writeable very often.
    All the buzzt!
    CornedBee

    "There is not now, nor has there ever been, nor will there ever be, any programming language in which it is the least bit difficult to write bad code."
    - Flon's Law

  7. #7
    Registered User
    Join Date
    Oct 2006
    Posts
    3,445
    Quote Originally Posted by CornedBee View Post
    Are there external solid-state disks? SSDs are writeable very often.
    there are INTERNAL solid state disks, made for IDE, SATA, SCSI, and SAS. the SATA, SCSI, and SAS types are usually intended for server applications, meaning it's likely that they can be written to a nearly infinite number of times, especially with todays servers requiring database inserts and updates at a high rate. I have seen SATA and IDE SSDs for as little as a few hundred dollars (US) for up to 64GB.

  8. #8
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    > Another doubt. Maybe I can find some alternative to a flash, like some USB device?

    That's not a doubt, it's a question.

  9. #9
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,057
    Flash is pretty much the way to go if you need non-volatile storeage. It's cheap and pretty fast (especially for reading).

    I was reading an article on flash the other day. It said that most flash chips actually have about 20%-50% more space than they say they have. As soon as part of the flash goes bad, after the million-some writes are up or whatever, the chip will silently mark it as so and use some of the spare space instead. That's what it said, anyway.

    So I really wouldn't worry about it too much. If you write in the same place over and over, it will just get replaced when it gets fried. If you write to the whole disk frequently, well, you're getting a lot of use out of the device.

    For what it's worth -- how many times have you scanned your hard drive and discovered bad sectors on it?

    (2) Can Debian be installed on a disk less system or do I have to look for some not very common and tested minimal Linux distributions?
    Absolutely. I myself have a bootable Debian system on a USB flash pendrive. It's amazing what you can fit into a gigabyte.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

  10. #10
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251
    ok great
    And what about the corruption if somebody switches the system off abruptly (in my case (where the device will be installed) it happens very often)?
    i have been told the flash can be corrupted if the system was writing in that moment? Is that correct? can ext3 system recover the file system?
    What about solid state hard drives instead?

  11. #11
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by mynickmynick View Post
    ok great
    And what about the corruption if somebody switches the system off abruptly (in my case (where the device will be installed) it happens very often)?
    i have been told the flash can be corrupted if the system was writing in that moment? Is that correct? can ext3 system recover the file system?
    What about solid state hard drives instead?
    If this happens "often", then you probably need some sort of preventative measure - such as a "power-off that isn't immediate".

    There is no real difference between flash and real hard-disks here - if you pull the power on a hard disk when data is being written, you end up with inconsistent data on the disk. Same with flash - you will have some random data in the flash. If that is some IMPORTANT data, then that could be fatal, but modern filesystems do have backup mechanisms to recover that USUALLY works.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  12. #12
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251
    Quote Originally Posted by matsp View Post
    If this happens "often", then you probably need some sort of preventative measure - such as a "power-off that isn't immediate".

    There is no real difference between flash and real hard-disks here - if you pull the power on a hard disk when data is being written, you end up with inconsistent data on the disk. Same with flash - you will have some random data in the flash. If that is some IMPORTANT data, then that could be fatal, but modern filesystems do have backup mechanisms to recover that USUALLY works.

    --
    Mats
    I will explain myself better. For the data being written in that moment of course they are very probably lost. But if one has implemented some smart double recording he can prevent that (he can retrieve data in the last old version).
    What I am talking of is more general all-flash corruption (you do not retrieve also all other data on flash) : file system corruption!

  13. #13
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by mynickmynick View Post
    I will explain myself better. For the data being written in that moment of course they are very probably lost. But if one has implemented some smart double recording he can prevent that (he can retrieve data in the last old version).
    What I am talking of is more general all-flash corruption (you do not retrieve also all other data on flash) : file system corruption!
    Flash is no more (or less) vulnerable to this [assuming the filesystem is indeed duplicating crucial data - and most Linux/Unix filesystems do in one way or another - even FAT does that]. The flash device itself is not "destroyed" during a power-down during writing - it will just contain incomplete/incorrect data in that area. Don't you think you would have heard about it if this happened easily, considering the number of iPod's that are out there - must be some people who accidentally shut down the power on their iPod [or run out of power or whatever] whilst the device is writing to the flash-device.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  14. #14
    Registered User Codeplug's Avatar
    Join Date
    Mar 2003
    Posts
    4,981
    >> And what about the corruption if somebody switches the system off abruptly
    Journaling file systems are designed for this to make corruption much less likely...like JFFS2 for example.

    gg

  15. #15
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251
    Quote Originally Posted by Codeplug View Post
    >> And what about the corruption if somebody switches the system off abruptly
    Journaling file systems are designed for this to make corruption much less likely...like JFFS2 for example.

    gg
    thank you
    I have been told only some linux partitioning have journaling (like ext3). Is that correct?

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Strange hard disk
    By GanglyLamb in forum Tech Board
    Replies: 20
    Last Post: 03-01-2003, 04:05 AM
  2. hard disk space and spyware
    By oldmancan in forum A Brief History of Cprogramming.com
    Replies: 5
    Last Post: 01-23-2003, 01:24 AM
  3. truely Cleaning a hard disk
    By xds4lx in forum A Brief History of Cprogramming.com
    Replies: 29
    Last Post: 01-23-2003, 12:37 AM
  4. how to store a node on hard disk
    By ALLRIGHT in forum C Programming
    Replies: 3
    Last Post: 05-13-2002, 10:11 AM
  5. program to check hard disk transfer rate
    By shadow99er in forum C Programming
    Replies: 3
    Last Post: 03-01-2002, 05:04 PM