linux writing often on main disk

This is a discussion on linux writing often on main disk within the Linux Programming forums, part of the Platform Specific Boards category; I had a little argue with my boss: he thinks that for our puropuses (embedded Vision Systems using GigE cameras) ...

  1. #1
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251

    linux writing often on main disk

    I had a little argue with my boss:
    he thinks that for our puropuses (embedded Vision Systems using GigE cameras) buildroot crosscompiling together with uclibc on an embedded system with flash and no hard disk is fine
    I think instead that an industrial PC without hard disk, with two flash and a standard linux distribution like Debian installed in minimum form (only those applications that are strictly necessary to us like ssh, ftp) is better.
    His arguments are that standard linux distribution writes often on disk so can block our system for many msec (we have a required response time of about 25 msec that means we need not to be hanged by the OS for more than 25 msec) and can damage the compact flash
    My arguments are :
    (1) a standard linux distribution uses NPTL (while buildroot + uclibc uses linuxthreads) -> much more efficient multithreading and more easy to program as it is 99% POSIX compliant
    (2) standard linux with glibc has commercial libraries available for opur cameras system

    My questions are:
    (1) Is his argument about linux writing often to main file system correct?
    (2) Is my argument about NPTL vs. linuxthreads correct?

    Or also after some study might be emdebian could the right agreement between the two opinions?

    Thank you
    Last edited by mynickmynick; 04-02-2009 at 03:38 AM.

  2. #2
    Cat without Hat CornedBee's Avatar
    Join Date
    Apr 2003
    Posts
    8,893
    (2) Is my argument about NPTL vs. linuxthreads correct?
    Yes.

    (1) Is his argument about linux writing often to main file system correct?
    I don't think any standard linux distro writes to disk more often than any other linux.
    All the buzzt!
    CornedBee

    "There is not now, nor has there ever been, nor will there ever be, any programming language in which it is the least bit difficult to write bad code."
    - Flon's Law

  3. #3
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    re: linux writing to disk: I read a little while ago that because linux, like all *nix, maintains "access times" for the filesystem, this means every disk read must also be complimented by a disk WRITE in order to update the atime. Supposedly all this adds up, so in heavy use situations where the atime is not considered crucial (since the m(odification)time is seperate from the atime, you might wonder when it ever would really be important) you can start linux with the "noatime" kernel parameter which means the kernel no longer writes an atime to disk every time a file is accessed. And this means something like a 10-20% improvement in speed where the disk access is constant.

    Since you are dealing with significance on the order of 25ms and are worried about time wasted with (basically irrelevent) writes to disk, I thot I'd mention that. Google "noatime" and some articles will appear.
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  4. #4
    Cat without Hat CornedBee's Avatar
    Join Date
    Apr 2003
    Posts
    8,893
    Gentoo by default introduces all mount lines with noatime. I'm booting my laptop to see if Ubuntu does it too, but I'm convinced it does, just as I'm convinced every other distro out there does it by default. atime is an extremely specialized feature, and I'd be very surprised to find anything that enables it by default.

    Edit: Ubuntu mounts with relatime, which means "update atime if it's older than mtime or ctime", or in other words, "update atime on write, or when accessing after someone accessed it with noatime". In other words, it's mostly a noop.
    Last edited by CornedBee; 04-02-2009 at 09:12 AM.
    All the buzzt!
    CornedBee

    "There is not now, nor has there ever been, nor will there ever be, any programming language in which it is the least bit difficult to write bad code."
    - Flon's Law

  5. #5
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    You may want to put something in place to log where the writes to disk are coming from - for example which process and what file (filehandle and PID should be enough).

    I guess having a kernel-side buffer inside the write system call, that stores a count of each PID/FILE combination, and whenever you feel it's a good time, dump it to syslog, serial port or some such.

    If it's a regular thing, then it would most likely happen quite often.

    One solution may be to move that file to a tmpfs (ram-disk)...

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  6. #6
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,236
    Why would anything be writing to disk if nothing is running except your code?

    Disk writes don't just appear from nowhere, some process has to request them.

    And even if atime updating is turned on, the filesystem is cached, including inodes. I think your boss is inventing a problem where none exists.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  7. #7
    spurious conceit MK27's Avatar
    Join Date
    Jul 2008
    Location
    segmentation fault
    Posts
    8,300
    I think a number of distros are using relatime now but certainly not all of them (eg, neither fedora nor debian do it by default), and most of them do not use noatime, since my original source for this was a recent article by Mr. Linus Torvalds (or rather interviewing him), where he says that most large commercial installations (which I guess is the opposite of what the OP is doing, so mebbe this all irrelevent) in fact use neither, tho he suggests they should.

    Here's a link to a linux foundation article and tests on solid state disks where the author claims that "What the results show is that metadata-heavy workloads, such as make clean, do result in almost twice the amount data written to disk." That's the biggest difference scenario.

    http://beta.linuxfoundation.org/news...oatimerelatime

    This could even be the bird that half whispered in that bosses ear, starting all the fuss...
    C programming resources:
    GNU C Function and Macro Index -- glibc reference manual
    The C Book -- nice online learner guide
    Current ISO draft standard
    CCAN -- new CPAN like open source library repository
    3 (different) GNU debugger tutorials: #1 -- #2 -- #3
    cpwiki -- our wiki on sourceforge

  8. #8
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251
    Quote Originally Posted by brewbuck View Post
    Why would anything be writing to disk if nothing is running except your code?

    Disk writes don't just appear from nowhere, some process has to request them.

    And even if atime updating is turned on, the filesystem is cached, including inodes. I think your boss is inventing a problem where none exists.

    He is sponsoring his platform (he had already developed and used for other purpouses, in this case with very little adaptaments)

  9. #9
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251
    Thank you everybody for help

  10. #10
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251
    Quote Originally Posted by CornedBee View Post
    Yes.


    I don't think any standard linux distro writes to disk more often than any other linux.
    I apologize I was not detailed and precise:
    actually his complete argument is that his embedded linux has the main file system on ramdisk and only a subdir as flash
    This would eliminate compact flash progressive corruption (still actually I guess reached in so many years it is not very relevant) and would reduce the time loss
    But I remember I made some tests on Debian with RAMdisk and I found out that the time for writing to ramdisk was only 20% less than writing on hard disk
    Plus as I said our applications would write so little that the compact flash would be corrupted in xxx years so..

  11. #11
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Quote Originally Posted by mynickmynick View Post
    Thank you everybody for help
    Does this mean that you have found a solution, and if so, can you share some of what you've found with us?

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

  12. #12
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251
    unfortunately not
    I have to work now with this buildroot crosscompiling tools that I don't like
    It even has only alpha C++ support
    That means I will program only in C with no C++
    I will in the future investigate on using emdebian instead and maybe without cross compiling.
    May be compiling directly on the target or on a copy of the target

  13. #13
    Captain Crash brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,236
    Quote Originally Posted by mynickmynick View Post
    But I remember I made some tests on Debian with RAMdisk and I found out that the time for writing to ramdisk was only 20% less than writing on hard disk
    That just means Linux's filesystem caching is about 80% efficient.
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  14. #14
    Alessio Stella
    Join Date
    May 2008
    Location
    Italy, Bologna
    Posts
    251
    Quote Originally Posted by brewbuck View Post
    That just means Linux's filesystem caching is about 80% efficient.
    what you mean? I don't know chaching mechanism in the case of file system.. Is the writing on hard disk delayed after a wriitng on RAM?

  15. #15
    Kernel hacker
    Join Date
    Jul 2007
    Location
    Farncombe, Surrey, England
    Posts
    15,677
    Yes, Linux is generally using any spare memory (memory that isn't currently used by applications or the kernel itself) as a cache for disk operations, so writing to disk will actually just write to memory, and later on write to disk proper.

    --
    Mats
    Compilers can produce warnings - make the compiler programmers happy: Use them!
    Please don't PM me for help - and no, I don't do help over instant messengers.

Page 1 of 2 12 LastLast
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. ReadyBoost for Linux?
    By cyberfish in forum Tech Board
    Replies: 0
    Last Post: 06-07-2009, 04:52 AM
  2. C99 and int main()
    By cwr in forum C Programming
    Replies: 8
    Last Post: 09-19-2005, 06:54 AM
  3. Writing Linux Drivers
    By Jperensky in forum Linux Programming
    Replies: 1
    Last Post: 03-20-2002, 08:30 AM
  4. void or int for main?:)
    By bigtamscot in forum C Programming
    Replies: 13
    Last Post: 09-27-2001, 03:11 AM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21