Directory crawler problems

This is a discussion on Directory crawler problems within the C Programming forums, part of the General Programming Boards category; Im try to make a directory crawler. Lets say this is the working directory for the program Code: Master Folder ...

  1. #1
    Registered User
    Join Date
    Jul 2002
    Posts
    913

    Directory crawler problems

    Im try to make a directory crawler.

    Lets say this is the working directory for the program
    Code:
    Master Folder
    	Category 1(folder)
    		Folder 1
    		Folder 2
    	Category 2(folder)
    		Folder 3
    		Folder 4
    	file
    	file2
    Id like to start in "Master Folder". From there i would like to enter every sub folder. So the program first sees "Category 1". It enters it and then prints all the folders in it.

    Here what ive tried:
    Code:
    #define     PORTAGE_DIR     "."
    
    #include <stdio.h>
    #include <dirent.h>
    #include <sys/types.h>
    #include <sys/stat.h>
    
    int main() {
        /* Holds info for the main portage directory. */
        DIR             *master;
        struct dirent   *master_dirent;
        struct stat     master_stat;
        /* Holds info for the crawler which goes in every category folder. */
        DIR             *crawl;
        struct dirent   *crawl_dirent;
        struct stat     crawl_stat;
        
        master = opendir(PORTAGE_DIR);
    
        if(master != NULL) {
            /* Read in every category folder. */
            while( (master_dirent = readdir(master)) != 0 ) {
                if( stat(master_dirent->d_name, &master_stat) == 0 ) {
                    /* Check to see if its a folder. */
                    if( S_ISDIR(master_stat.st_mode) != 0 ) {
                        /* It is a director, enter and crawl. */
                        
                        /* Open folder. */
                        crawl = opendir(master_dirent->d_name);
                        
                        while( (crawl_dirent = readdir(crawl)) != 0 ) {
                            if( S_ISDIR(crawl_stat.st_mode) != 0 )
                                printf("%s\n", crawl_dirent->d_name);
                        }
                        
                        closedir(crawl);
                    }
                }
            }
            
            closedir(master);
        } else {
            perror("Error: ");
        }
    
        return 0;
    }
    This gives me back nothing. I checked the FAQ and a couple of posts, im sure its right in front of me but i dont see the problem.

    Thanks

    <edit>
    Id like thsi to work on Linux and Windows, if that cant be done than just Linux/Unix. And this is being done with GCC.

  2. #2
    Registered User
    Join Date
    Jul 2002
    Posts
    913
    Well right now im cheating in Windows(no mp3s any where else).

    In my working dir I made 2 folders. Heres what the program gave me.
    .
    ..
    New Folder
    New Folder (2)
    .
    ..
    .
    ..
    .
    ..
    Whats with the periods?

    There are folders in the New Folders that arent listed, whered I goof?

    Thanks

  3. #3
    Been here, done that.
    Join Date
    May 2003
    Posts
    1,161
    Each directory has 2 dirs:
    . represents the current directory
    .. represent the parent directory.

    In your code, simply test for "." and ".." and ignore them.
    Definition: Politics -- Latin, from
    poly meaning many and
    tics meaning blood sucking parasites
    -- Tom Smothers

  4. #4
    Registered User
    Join Date
    Jul 2002
    Posts
    913
    I always thought the dots were from the OS/shell to make things easier...

    Why doesnt my code list the directories? I dont get it.

    Thanks

  5. #5
    Been here, done that.
    Join Date
    May 2003
    Posts
    1,161
    Originally posted by mart_man00
    I always thought the dots were from the OS/shell to make things easier...
    Nope, it's part of the way directories work.

    Why doesnt my code list the directories? I dont get it.
    It's wrong...

    Thanks
    Welcome
    According to your message above, you have 2 dirs (not including the . & ..) and your program listed 2 dirs. It seems to work based on the info given.

    If it's not, details would be helpful, as well a the current code if it's changed.
    Definition: Politics -- Latin, from
    poly meaning many and
    tics meaning blood sucking parasites
    -- Tom Smothers

  6. #6
    Registered User
    Join Date
    Jul 2002
    Posts
    913
    Code:
    Start dir
    	Folder 1
    		Folder 2
    Is my layout.

    The program starts in 'Start Dir'. It goes into every directory there. So it enters 'Folder 1' Then it basicly does a dir/ls of only folders. So it never enters 'Folder 2', it just lists it.

    That why i only go 2 deep.

  7. #7
    End Of Line Hammer's Avatar
    Join Date
    Apr 2002
    Posts
    6,231
    The FAQ examples (walker) use recursion, which makes directory processing relatively easy.
    When all else fails, read the instructions.
    If you're posting code, use code tags: [code] /* insert code here */ [/code]

  8. #8
    Registered User
    Join Date
    Jul 2002
    Posts
    913
    I did do the unthinkable and checked out the FAQ(actually, I looked at old vVv post first...).

    I only want to go to level, so Im thinking im just ripping off the vVv's one(he answer some of my stat questions before, it could come in handy in th FAQ).

    Plus im not seeing the problem with my code(so im not getting something right, like a concept).

    I did fix one part,
    Code:
    #define     PORTAGE_DIR     "."
    
    #include <stdio.h>
    #include <dirent.h>
    #include <sys/types.h>
    #include <sys/stat.h>
    
    int main() {
        /* Holds info for the main portage directory. */
        DIR             *master;
        struct dirent   *master_dirent;
        struct stat     master_stat;
        /* Holds info for the crawler which goes in every category folder. */
        DIR             *crawl;
        struct dirent   *crawl_dirent;
        struct stat     crawl_stat;
        
        master = opendir(PORTAGE_DIR);
    
        if(master != NULL) {
            /* Read in every category folder. */
            while( (master_dirent = readdir(master)) != 0 ) {
                if( stat(master_dirent->d_name, &master_stat) == 0 ) {
                    /* Check to see if its a folder. */
                    if( S_ISDIR(master_stat.st_mode) != 0 ) {
                        /* It is a director, enter and crawl. */
                        
                        /* Open folder. */
                        crawl = opendir(master_dirent->d_name);
                        
                        while( (crawl_dirent = readdir(crawl)) != 0 ) {
                            if( stat(crawl_dirent->d_name, &crawl_stat) == 0 ) {
                                if( S_ISDIR(crawl_stat.st_mode) != 0 ) {
                                    printf("%s\n", crawl_dirent->d_name);
                                }
                            }
                        }
                        
                        closedir(crawl);
                    }
                }
            }
            
            closedir(master);
        } else {
            perror("Error: ");
        }
    
        return 0;
    }
    It starts in the working directory. From there it does a ls. If it a directory it enters it. From there it does anything ls. If its a folder its name is printed on the screen. When there nothing left it brakes and goes back to the starting posistion. From there it repeats itself.

    I dont see the problem yet.

    Thanks

  9. #9
    Registered User
    Join Date
    Jul 2002
    Posts
    913
    Code:
    #define     PORTAGE_DIR     "."
    
    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <dirent.h>
    #include <sys/types.h>
    #include <sys/stat.h>
    
    int main() {
        /* Holds info for the main portage directory. */
        DIR             *master = opendir(PORTAGE_DIR);
        struct dirent   *master_dirent;
        struct stat     master_stat;
        /* Holds info for the crawler which goes in every category folder. */
        DIR             *crawl;
        struct dirent   *crawl_dirent;
        struct stat     crawl_stat;
        
        char *open = NULL;
    
        if(master != NULL) {
            /* Read in every category folder. */
            while( (master_dirent = readdir(master)) != 0 ) {
                if( stat(master_dirent->d_name, &master_stat) == 0 ) {
                    /* Check to see if its a folder. */
                    if( S_ISDIR(master_stat.st_mode) != 0 ) {
                        /* It is a director, enter and crawl. */
                        
                        if( nodots(master_dirent->d_name) == 0 ) {
                            open = malloc( strlen(PORTAGE_DIR) + strlen(master_dirent->d_name) - 1 );
                            sprintf("%s%s", open, PORTAGE_DIR, master_dirent->d_name); //crashes here
                            
                            crawl = opendir(open);
                        
                            while( (crawl_dirent = readdir(crawl)) != 0 ) {
                                if( stat(crawl_dirent->d_name, &crawl_stat) == 0 ) {
                                    if( S_ISDIR(crawl_stat.st_mode) != 0 ) {
                                        printf("%s\n", crawl_dirent->d_name);
                                    }
                                }
                            }
                        
                            closedir(crawl);
                        }
                    }
                }
            }
            
            closedir(master);
        } else {
            perror("Error: ");
        }
    
        return 0;
    }
    
    int nodots(char *src) {
        if( strcmp(src, ".") == 0 )
            return 1;
        if( strcmp(src, "..") == 0 )
            return 1;
            
        return 0;
    }
    Im now trying to see if the directories are the dots(for lack of a better name). My idea was to just sprintf it, but it crashes there. Printf letsme print it as a string, whats wrong here?

  10. #10
    Registered User
    Join Date
    Jul 2002
    Posts
    913
    Code:
    if(master != NULL) {
            /* Read in every category folder. */
            while( (master_dirent = readdir(master)) != 0 ) {
                if( stat(master_dirent->d_name, &master_stat) == 0 ) {
                    /* Check to see if its a folder. */
                    if( S_ISDIR(master_stat.st_mode) != 0 ) {
                        /* It is a director, enter and crawl. */
                        
                        if( nodots(master_dirent->d_name) == 0 ) {
                            open = malloc( strlen(master_dirent->d_name) + 1 );
                            sprintf(open, "./%s", master_dirent->d_name); 
                            
                            crawl = opendir(open);
                        
                            while( (crawl_dirent = readdir(crawl)) != 0 ) {
                                if( stat(crawl_dirent->d_name, &crawl_stat) == 0 ) {
                                    if( S_ISDIR(crawl_stat.st_mode) != 0 ) {
                                        if( nodots(crawl_dirent->d_name) == 0 ) {
                                            printf("%s\n", crawl_dirent->d_name);
                                        }
                                    }
                                }
                            }
                        
                            closedir(crawl);
                        }
                    }
                }
            }
            
            closedir(master);
        }
    Now i get a blank screen. I was trying to get the 3rd level.

  11. #11
    Been here, done that.
    Join Date
    May 2003
    Posts
    1,161
    If you want to crawl thru all directories, no matter how deep, you really have to use a recursive function to do it easily and maintanably. It also makes your function more compact.

    Also, if you change your nodots() function to isdir() and return 0 for a file and a dot dir, you can remove an if statement.
    Definition: Politics -- Latin, from
    poly meaning many and
    tics meaning blood sucking parasites
    -- Tom Smothers

  12. #12
    Registered User
    Join Date
    Jul 2002
    Posts
    913
    I was trying to avoid recursive since i know how many level i want to go. I think ill try it now tho, it might be better.

    Ill add you if idea too, i didnt think about that one.

    Thanks

  13. #13
    Registered User
    Join Date
    Jul 2002
    Posts
    913
    GCC doesnt like me or my function args

    Code:
    int isdir(struct dirent **check_dirent, struct stat *check_stat) {
    	if( stat(&check_dirent->d_name, &check_stat) == 0 ) {
    		if( S_ISDIR(&check_stat.st_mode) != 0 ) {
    			if( strcmp(&check_dirent->d_name, ".") != 0 ) {
    				if( strcmp(&check_dirent->d_name, "..") != 0 ) {
    					return 0;
    				}
    			}
    		}
    	}
    
    	return 1;
    }
    
    int main() {
    	DIR             *master = opendir(".");
    	struct dirent   *master_dirent;
    	struct stat     master_stat;
    
    	isdir(&master_dirent, &master_stat);
    
    	return 0;
    }

  14. #14
    End Of Line Hammer's Avatar
    Join Date
    Apr 2002
    Posts
    6,231
    >>GCC doesnt like me or my function args
    Which ones...? Some errors messages from your compiler would help.
    When all else fails, read the instructions.
    If you're posting code, use code tags: [code] /* insert code here */ [/code]

  15. #15
    Registered User
    Join Date
    Jul 2002
    Posts
    913
    Im getting back
    new_crawl.c: In function `isdir':
    new_crawl.c:8: request for member `d_name' in something not a structure or union

    new_crawl.c:8: warning: passing arg 2 of `stat' from incompatible pointer type
    new_crawl.c:9: request for member `st_mode' in something not a structure or union
    new_crawl.c:10: request for member `d_name' in something not a structure or union
    new_crawl.c:11: request for member `d_name' in something not a structure or union
    I thought i was giving a pointer to a structure.

Page 1 of 3 123 LastLast
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Finding files in a directory
    By smarta_982002 in forum C Programming
    Replies: 1
    Last Post: 01-25-2008, 09:10 AM
  2. Building a tree from a directory structure
    By geekoftheweek in forum C Programming
    Replies: 2
    Last Post: 11-26-2007, 02:15 AM
  3. About current working directory
    By George2 in forum Linux Programming
    Replies: 3
    Last Post: 05-02-2007, 10:34 PM
  4. fprintf in directory
    By rkooij in forum C Programming
    Replies: 9
    Last Post: 03-09-2006, 03:23 AM
  5. problems with my first attempt at a Qt application...
    By Captain Penguin in forum C++ Programming
    Replies: 0
    Last Post: 10-01-2002, 09:14 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21