Any regex people here?

I want to make a list of all links from an index.html page. All I want is the URL and I'll live with the assumption that there's double quotes around it - for now. Here's what I have so far:

Code:

#include <stdlib.h>
#include <string.h>
#include <regex.h>

#define MAX_STRING_SIZE 1024

int main() {
  int rc;
  regex_t * myregex = calloc(1, sizeof(regex_t));
  regmatch_t matches[3];
  FILE *fp;
  char line[MAX_STRING_SIZE];
  
  if(myregex == NULL)
     return 1;

  fp = fopen("/var/tmp/index.html", "r");
  rc = regcomp(myregex, "href\\s*=\\s*(\")*(.*?\")([^\"]+)", REG_EXTENDED);
  while(fgets(line, MAX_STRING_SIZE, fp) != NULL) {
     if(regexec(myregex, line, 3, matches, 0) == 0) {
        printf("String: %s\n", line + matches[2].rm_so);
     }
  }
  free(myregex);
  return 0;
}

My output shows me the URL, but it also shows me the rest of the line of text as well. I keep trying to tell it (I think) to stop after the first double quote after the URL, but no matter what I keep getting the rest of the line.

Thank you in advance for helping out.

What's the last group for?

It's for my lack of trying everything else I could possibly think of.

Really, I've stumbled so many times with regex. I've read and reread books and examples and everytime I come across a situation where they seem like the best strategy, I create them very carefully and with a lot of research, then they just never work.

Any suggestions are so welcome.

"Sometimes when people have a problem, they think, 'I know, I'll use a regular expression.' Now they have two problems."

Can't remember where I got that quote from.

Anyway, this Perl syntax regex should pick up link HREFs. I'll leave the C string escaping to you.

Code:

href\s*=\s*['"]([^"']*)

I don't know about the POSIX regex API, since I've never used it.

I tested that with Perl and yes it does work, however the POSIX API must be different because it doesn't work in C.

And I like your quote.

Is there any other method you'd use to scan through an html file looking for all links?

Thank you for your help.

Try:

Code:

href[:space:]*=[:space:]*['|"]([^"|']*)

Also, you probably shouldn't calloc your regex_t struct. Use a stack variable, pass a pointer to it to the functions, and use regfree when you're done:

Code:

regex_t re;
regcomp(&re, ...
regexec(&re, ...
regfree(&re);

I'd wack up a simple HTML parser, or use one of the hundreds out there...

Or simply search for <a ...></a>

Is there any other method you'd use to scan through an html file looking for all links?

Search for "href" as text (strstr for example), then look for symbol, equal sign and quotes - gather up what's between the quotes [you probably should remember what quote it started with and match it at the end - but I don't think quotes are valid within URL's, so I guess it won't make much difference].

--
Mats

rags_to_riches - I tried your regex and it still pulls everything to the end of the line. Should I use some non-greed qualifier? Like "?".

I originally thought about using strstr, but thought I should use regex and once and for all learn the darn thing.

I might go back to strstr with strtok as my method.

Thank you all for your ideas, comments and suggestions.

What did we all do before there were forums like this? I know I struggled for weeks before finding answers. This almost seems like cheating. Almost.

Thank you all.

Use Boost.Regex (C++) or PCRE (C) to use Perl-syntax regular expressions. It's a lot saner than POSIX syntax, IMO.

Thanks CornedBee.

I'll search for how to use PCRE(C) and see if it works easier than what I'm using now. It's really been frustrating.

It's because this line:

Code:

printf("String: %s\n", line + matches[2].rm_so);

is wrong. You are not taking into account the end offset of the match (rm_eo). This works:

Code:

#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <regex.h>

#define MAX_STRING_SIZE 1024

int main() {
   int rc;
   regex_t myregex;
   regmatch_t matches[2];
   FILE *fp;
   char line[MAX_STRING_SIZE];
   char match[MAX_STRING_SIZE];

   if (NULL != (fp = fopen("./index.html", "r"))) {
      rc = regcomp(&myregex,
                   "href[:space:]*=[:space:]*['|\"]([^\"|']*)",
                   REG_EXTENDED);
      while(fgets(line, MAX_STRING_SIZE, fp) != NULL) {
         if(regexec(&myregex, line, 2, matches, 0) == 0) {
            if (matches[1].rm_so != -1) {
               size_t match_len = matches[1].rm_eo - matches[1].rm_so;
               strncpy(match,
                       &line[matches[1].rm_so],
                       match_len);
               match[match_len] = '\0';
               printf("String: %s\n", match);
            }
         }
      }
      fclose(fp);
   }
   regfree(&myregex);
   return 0;
}

rags_to_riches - you are fantastic!

Thank you for your help. (help nothing. you did it for me)

I had tried it with the ending offset but I did my math in reverse so I got some weird characters all over my screen so I knew I went into some deep memory area.

Hopefully this thread gets indexed on all the search engines because your code will definitely be useful to many, many others.

Thank you again.

Is there a way to grab all href's on one line?

I'm coming across one line of html that has multiple href's on it and this is only collectinng the first one.

Any ideas?

It's worth $100 for the help. (not to me, to you)

It's simple. When you find a match on a line, take the substring starting at the end offset and try again, until you no longer find matches.