What mean filedescriptor error-codes?

Hi,
I have written the following code:

Code:

#include <stdio.h> int main(void) { char *myfile = "/home/username/Desktop/testfile"; FILE *fd; fd=fopen(myfile, "r"); printf("%i\n",fd); fclose(fd); return 0; }

Executed I get this here: 134520840, which looks like a valid filedescriptor.

If I try now the same with syscalls:

Code:

#include </usr/include/linux/unistd.h> #include <stdio.h> int main(void) { char *myfile = "/home/username/Desktop/testfile"; int fd; fd = syscall(__NR_open,myfile,"O_RDONLY") ; printf("%i\n",fd); return 0; }

I always get -1 and if I use &myfile in the syscall I get 3 ? What are -1 and 3 meaning?

Can someone see the mistake in my second code?

bye

> Executed I get this here: 134520840, which looks like a valid filedescriptor.
It isn't, it's just a pointer mushed into an integer.

I'd be surprised if the flags were passed as a string, and not as a direct copy of the flags passed to the regular open() call at the API level.

okay I did so now like I would do the same in assembler:

Code:

fd = syscall(__NR_open,myfile,00,04000) ;

in asm I do this an it works:

Code:

mov eax,5 mov ebx, myfile mov ecx,00 mov edx,04000 int 80h

But in my C-code I now always get 3 printed out, which is an error-code or? 3 is no pointer to a filedescriptor..

3 sounds like a perfectly valid file descriptor to me.
0 being stdin
1 being stdout
2 being stderr
3 being the next lowest available integer for a file descriptor.

I'll say it again so you're clear, a FILE* is NOT the same thing as a file descriptor. It is a file handle.

Ahh thank you I thought FILE would be a fd type or something like this. 3 was really the correct fd. This here is working fine for me:

Code:

#include </usr/include/linux/unistd.h> #include <string.h> int main(void) { char *myfile = "testfile",buffer[100]; int fd; fd = syscall(__NR_open,myfile,00,04000) ; syscall(__NR_read,fd,buffer,100); syscall(__NR_write,1,&buffer,strlen(buffer)); return 0; }

But I have one last question. The syscall_write expects:

Quote:

eax 4
ebx Device descriptor.
ecx Pointer to the buffer containing the data to be written.
edx Number of bytes to be written.

A pointer is an adress pointing to the value of a variable. Now the &buffer above in my code snippet gives us the adress where the content of buffer starts on the stack. But why is buffer without the & also working exactly the same way? So the code above also works without the & before the buffer in the syscall_write?

And why is *buffer not working? I mean k buffer is no pointer typ or, but at least a compiler-error should be given for using it as such or?

greets

Quote:

Originally Posted by lilcoder

I mean k buffer is no pointer typ or, but at least a compiler-error should be given for using it as such or?

The name of an array becomes a pointer to it's first element -> buffer is a pointer to char -> *buffer is the same as buffer[0] -> not a pointer -> compile error
Kurt

Quote:

Originally Posted by ZuK

The name of an array becomes a pointer to it's first element

Ahh so buffer and &buffer are exactly the same? Because also &buffer points to the first element of this char array.

Quote:

Originally Posted by ZuK

*buffer is the same as buffer[0] -> not a pointer -> compile error

But is * not the symbol for the pointer-type? And next to that I don't get a compiler error for using it.

Quote:

Ahh so buffer and &buffer are exactly the same? Because also &buffer points to the first element of this char array.

Not exactly. buffer is an array, &buffer is the address of the array, which is a pointer. They may both point to the same place in memory, but they are not the same.

Quote:

But is * not the symbol for the pointer-type?

Yes, but the meaning of * depends on context.

Code:

int *x = NULL; // * used such that x is a pointer. int a = *b; // * used such that *b is an int.

Quote:

And next to that I don't get a compiler error for using it.

How did you use it?

Code:

int *x = NULL; // * used such that x is a pointer.

makes sense

Code:

int a = *b; // * used such that *b is an int.

Aha this makes a containing the hexcode of the first field of b[]:

Code:

char *b="test"; int a; a = *b; printf("%i",a) ;

Prints: 116 so is it that char *b="test" tells push test on the stack and make b a pointer to it. While *b points to the content which start with 116, b and &b are just the memory adresses ?

Quote:

Originally Posted by laserlight

How did you use it?

In this code:

Code:

#include </usr/include/linux/unistd.h> int main(void) { char *myfile = "testfile",buffer[100]; int fd,buffer_len; fd = syscall(__NR_open,myfile,00,04000) ; syscall(__NR_read,fd,buffer,100); syscall(__NR_close,fd); for ( buffer_len=1; buffer[buffer_len] != '\0'; buffer_len++); syscall(__NR_write,1,*buffer,buffer_len); return 0; }

I don't get any errors for *buffer. Normally compiled with gcc -o test test.c and gcc version 4.1.2 . The program simply doesn't work but compiles nice. *buffer exchanged with buffer or &buffer and the program works fine.

Quote:

Originally Posted by lilcoder

I don't get any errors for *buffer.

That is because syscall is a function that takes a variable number of arguments. That's why the compiler can't check the type of the arguments.
Kurt

ah okay thank you for your answers :)

Your mistake is believing that a FILE object and a file descriptor have anything to do with each other. They don't. The only problem I see here is attempting to print a FILE pointer as an integer -- why would you do that?

EDIT: I'm still clueless as to why you are calling system calls like that. Why not just call open()? Calling a syscall directly is not guaranteed to behave anything like the manual says it should.

And you should DEFINITELY close the file after it's been opened. The C runtime keeps track of regular files, and I expect there to be some safeguards on system calls too, but just in case.

[I agree with brewbuck - you are gaining ABSOLUTELY NOTHING by doing this, but risking that your code will be incompatible in a future version of Linux, as well as that the C library is "fixing" bugs that are in some varieties of Linux].

--
Mats

> for ( buffer_len=1; buffer[buffer_len] != '\0'; buffer_len++);
The ; at the end renders this for loop useless.

> syscall(__NR_write,1,&buffer,strlen(buffer));
read doesn't append a \0, so it's wrong to use strlen() to go looking for it.
Since read() returns a result (which you're ignoring), perhaps you could pick that up and use that to determine how much to write.

Ditto what others have said, you've gained nothing over using the standard POSIX API.

Quote:

Originally Posted by Salem

> for ( buffer_len=1; buffer[buffer_len] != '\0'; buffer_len++);
The ; at the end renders this for loop useless.

Isn't this just a "strlen()" re-implemented.

Althouhg, of course, there's no guarantee from read that there's a zero at the end of the buffer - there could be, but that depends on the content of the filel and/or the content fo buffer before the read. If you read a string from a binary file, it would be OK. If it's a text file and there's more than 100 bytes in the file, it will fail to have a zero at the end. [And of course, the string could be EMPTY, in whcih case starting at 1 could give the wrong result.

--
Mats

Mmm, so it is.

Hi,

Quote:

Originally Posted by matsp

And you should DEFINITELY close the file after it's been opened.

So do I: syscall(__NR_close,fd);

Quote:

Originally Posted by Salem

> for ( buffer_len=1; buffer[buffer_len] != '\0'; buffer_len++);

As said this is strlen without including string.h .

Quote:

Originally Posted by matsp

read doesn't append a \0, so it's wrong to use strlen() to go looking for it.
Since read() returns a result (which you're ignoring), perhaps you could pick that up and use that to determine how much to write.
[..]
If it's a text file and there's more than 100 bytes in the file, it will fail to have a zero at the end. [And of course, the string could be EMPTY, in whcih case starting at 1 could give the wrong result.

I can't follow that? I created a 182kb textfile and have resized my buffers and everything works fine. What do you mean with there's not always \0 representing the eof? What else? Or how could I achivie this by using syscalls? You say read is giving something back? I mean how works strlen:

Code:

static inline size_t strlen(const char * s) { int d0; register int __res; __asm__ __volatile__( "repne\n\t" "scasb\n\t" "notl %0\n\t" "decl %0" :"=c" (__res), "=&D" (d0) :"1" (s),"a" (0), "0" (0xffffffffu) :"memory"); return __res; }

I don't understand this asm+C mixture totally but the line starting with :"1" looks for me like checking for wheter \a or \0 depending on if it's a string or a file?

@ the why-question for this all: I'm just playing around :D I found a paper explaining how to use syscalls in kernelmodules, so I wanted to learn how to invoke them from C and just wanted to play around a bit with them. Next to the fact that calling the syscall directly should be faster or?

greets

If you have a 10 byte file, then 10 bytes is all that will get copied to your buffer.

The 11th character will NOT be a \0 or some representation of end of file or anything.

The only way to determine how much data was written to the buffer is to look at the return result of the read syscall.

> but the line starting with :"1" looks for me like checking for wheter \a
It's nothing of the sort, but I'm not going to explain the intricacies of the GNU assembler syntax. You'll need to google something for yourself.
Suffice to say, what it does do is provide a kind of bridge between the C part of the code and the asm part of the code (where to read values from, and where to store them).

Okay I did now like suggested and a few other things, so I think now there shouldn't be any errors:

Code:

#include </usr/include/linux/unistd.h> #include </usr/include/asm/stat.h> int main(int argc, char *argv[1]) { int fd,len; struct stat mystat ; fd = syscall(__NR_open,argv[1],00,04000) ; if (fd==-1) { syscall(__NR_write,1,"No such file\n",13); return 1; } syscall(__NR_fstat,fd,&mystat); char buffer[mystat.st_size]; len = syscall(__NR_read,fd,buffer,mystat.st_size); syscall(__NR_close,fd); syscall(__NR_write,1,buffer,len); return 0; }

thanks for all youre help!

> char buffer[mystat.st_size];
Variable length arrays, and declarations in the middle of statements are both outside the scope of C89.

Also, if you have a file of many MB in size, you've just blown away the stack.

Quote:

Originally Posted by Salem

> char buffer[mystat.st_size];
Variable length arrays, and declarations in the middle of statements are both outside the scope of C89.

Really? I thought this would be a good way of reading files with unknown length.

Okay I got now behind, that I have to use gcc -pedantic to get these kind of warnings you are telling about.

Quote:

Originally Posted by Salem

Also, if you have a file of many MB in size, you've just blown away the stack.

I didn't realize that because I thought there's a max size arrays could be and after that it would simply stop allocating memmory.

But what is then the best or better correct way of reading a file with unknown size? Getting its size and reading then block for block?

greets

Quote:

Originally Posted by lilcoder

But what is then the best or better correct way of reading a file with unknown size? Getting its size and reading then block for block?

Yep, pretty much. The standard way would be to read the file one line at a time, do your processing and then read the next line in.

QuantumPete

Quote:

Originally Posted by QuantumPete

Yep, pretty much. The standard way would be to read the file one line at a time, do your processing and then read the next line in.

But a line has not definite size or? So a 10 MB file could have just one line and never contain \n ? I think I'm gonna try to accomplish this now with sys_readahead:

Quote:

Populates the page cache with data from a file so that subsequent reads from that file will not block on disk I/O.

Arguments
eax 225
ebx File descriptor identifying the file which is to be read.
ecx Starting offset from which data is to be read. This value is effectively rounded down to a page boundary and bytes are read up to the next page boundary greater than or equal to (ecx+edx)
edx Number of bytes to be read.

But it seems I can't access this page cache then, there no variable I can give it and readv also expects a certain number of buffers, so also this seems useless to read a file of unkown size..

Use a fixed sized buffer, and read that amount.
The constant BUFSIZ in stdio.h is a good example size.

Say for argument that the OS restricts the max working set of your process to 50MB and you try to read a 100MB file in one go. What you end up with is the first half of the file written back out to the swap file and the second half in memory.

Then when you come to write it out again, the first half is read from swap and the 2nd half is written to swap. And so on.

> But what is then the best or better correct way of reading a file with unknown size?
Reading fixed sized buffers in a while loop, and detecting when end of file is reached.

Code:

char buff[BUFSIZ]; int n; while ( (n=read(fd,buff,BUFSIZ)) > 0 ) write(1,buff,n);

> So a 10 MB file could have just one line and never contain \n ?
I've seen XML files like this before now.
Also binary files have no concept of 'line', but have some other structure instead.

Yes the while loop is a good solution and works really well:

Code:

#include </usr/include/linux/unistd.h> #define BUFSIZE 4092 int main(int argc, char *argv[1]) { char buffer[BUFSIZE]; int fd,len; if (argc!=2) { syscall(__NR_write,1,"Usage: macro [filename]\n",24); return 0; } fd = syscall(__NR_open,argv[1],00,04000) ; if (fd==-1) { syscall(__NR_write,1,"No such file\n",13); return 0; } while ((len=syscall(__NR_read,fd,buffer,BUFSIZE))>0) { syscall(__NR_write,1,buffer,len);} syscall(__NR_close,fd); return 0; }

And if you take your code and read a file of (say) 100MB, how long does it tak, and how long does "cat inputfile" take [inputfile obviously should be replaced by the name of your 100MB file.]

Use "time", such as "time cat inputfile" to show you how fast it was.

Just to see - I haven't tried it myself, I'm just curious.

--
Mats

Hi okay I did a ~60MB Zip-file for this test now.

Code:

time cat ind.zip [..] real 2m31.340s user 0m00.000s sys 2m30.089s time ./macro ind.zip [..] real 2m34.212s user 0m00.004s sys 2m33.358s

But I dunno how relyable this test is. Not pretty much or what do you think?

I took here the best clocktick for 1000 tries, one time for sys_write and the other time for printf. I think this is a better method than the above:

Code:

#include </usr/include/linux/unistd.h> #include <stdio.h> #include </usr/src/linux-headers-2.6.20-16/include/asm-i386/msr.h> int main(void) { unsigned long ini, end, now, best, tsc; int i; char buffer[4]; #define measure_time(code) \ for (i = 0; i < 10000; i++) { \ rdtscl(ini); \ code; \ rdtscl(end); \ now = end - ini; \ if (now < best) best = now; \ } /* time rdtsc (i.e. no code) */ best = ~0; measure_time( 0 ); tsc = best; /* time an empty read() */ best = ~0; // measure_time( syscall(4,1,"test1\n",6) ); measure_time( printf("test2\n") ); /* report data */ printf("rdtsc: %li ticks\nsyscall():%liticks\n", tsc, best-tsc); return 0; }

Now I get for the syscall_write:

Quote:

rdtsc: 192 ticks
syscall():3396ticks

and for printf:

Quote:

rdtsc: 192 ticks
syscall():4092ticks

Which is an expected result because calling syscalls should of course be faster than calling the libC first which then calls the syscalls.

Well, my point was rather that cat is faster than your code (admittedly by only a fraction of the total time) - despite cat being portable and using C runtime library calls.

I think it would be fairer to use fputs() or fwrite() rather than printf() to compare the time for your application. The printf call is MUCH more complex than a simple write to a file - even if at the end it is what it does - for example, printf does essentially this:

Code:

int printf(const char *fmt, ...) { .... while(*fmt) { if (*fmt == '%') { *fmt++; switch(*fmt) { case 's': // a bunch of code. ... break; case 'd': // a bunch of code. ... break; ... // Lots more case labels. default: } } else { // output char from *fmt } fmt++; } }

Also, if you use /dev/null as your output device, e.g. "time cat blah.zip > /dev/null", you don't have the scrolling of the screen to take into account. The same applies to printf & write of course.

As you've just proven, a big van with load is slower than an ordinary car - but there's really should be no one that will be surprised by this.

--
Mats

fputs doesn't seem much faster than printf. It's best result was 4032ticks over 1000 tries..

I'll redo the time-test this night with closed X-server and no network-connection to bypass any irregularities.

With processor clock times now measuring <1nS and disk seek times still well above 1mS, there is somewhere between 6 and 7 orders of magnitude of difference between them.

Tinkering with the code when you're dealing with physical devices just isn't going to work. You'll never get your 2 minutes down to say 5 seconds without changing your storage technology.

For sure, you may see some improvement if you manage to get lucky and your small file is resident in some cache (or contiguous within the same cylinder on disk). But any sufficiently large file will always be playing catchup with the processor. As soon as you end up with a disk seek, that's it.

Those awesome disk transfer times the ads throw at you (hundreds of MB/Sec) are only really sustainable when the heads don't move. It's the seek time which kills you.

Quote:

Originally Posted by lilcoder

fputs doesn't seem much faster than printf. It's best result was 4032ticks over 1000 tries..

I'll redo the time-test this night with closed X-server and no network-connection to bypass any irregularities.

Was that "viewing the output" or "disposing it in /dev/null"?

--
Mats

@salem: Yes, indeed it's limited by hd-speed and with me even more because my HD is full-encrypted. But I think I can get better than cat. Maybe my BUFSIZE isn't that good choosen as I thought because I thought taking exactly one mempage should be the best case. Maybe it's not. Or it was simply some other process while the syscall test which stole memmory which didn't run during the cat test.

@matsp: this was now the printing way. If I do this speedtest code I posted with > /dev/null also the result will be dumped. Okay I could write it to a file before.. but all tests show, that syscall_writing to the screen is a bit faster than printf or fputs. It's simply the time to the libC which is not with using syscalls.

Quote:

Originally Posted by lilcoder

@salem: Yes, indeed it's limited by hd-speed and with me even more because my HD is full-encrypted. But I think I can get better than cat. Maybe my BUFSIZE isn't that good choosen as I thought because I thought taking exactly one mempage should be the best case. Maybe it's not. Or it was simply some other process while the syscall test which stole memmory which didn't run during the cat test.

@matsp: this was now the printing way. If I do this speedtest code I posted with > /dev/null also the result will be dumped. Okay I could write it to a file before.. but all tests show, that syscall_writing to the screen is a bit faster than printf or fputs. It's simply the time to the libC which is not with using syscalls.

Yes, but you are also spending most of the time scrolling the display, rather than performing system call - which is like measuring the speed of a car in Central London (or NYC) during rush-hour and saying "Cars do 12mph maximum". Use the same car somewhere a bit less full of traffic, and you are going to get a different result!

There is indeed some overhead in using the C library - this is not a new discussion. But the C library isn't there to stop you from doing what you want and doing it reasonably efficiently - it is there to help you get the job done with relative ease.

--
Mats

Quote:

Originally Posted by lilcoder

@salem: Yes, indeed it's limited by hd-speed and with me even more because my HD is full-encrypted. But I think I can get better than cat. Maybe my BUFSIZE isn't that good choosen as I thought because I thought taking exactly one mempage should be the best case. Maybe it's not. Or it was simply some other process while the syscall test which stole memmory which didn't run during the cat test.

Unlikely that some other process steals 3-4 seconds out of two minutes, but certainly encrypted hard disk content will make a difference, and but it's the same for both.

As for buffer size, try a size that is roughly half the L2 cache of your processor.

--
Mats