Thread: libcurl experience and question

  1. #1
    Jack of many languages Dino's Avatar
    Join Date
    Nov 2007
    Location
    Chappell Hill, Texas
    Posts
    2,332

    libcurl experience and question

    Well, I've written my first "big" (relatively speaking) program with libcurl - it sends a request to a classified ad site (CL) looking for old Mustangs for sale. It works pretty good. I wrote it in C. It's about 13 screens big in my IDE (Xcode).

    My first test was to hardcode five different cities, loop through them, and pass a URL to search for "mustang". This takes on average about 15-20 seconds depending on network delays. Each time it gets a whole page, it sucks out of the ad data, then parses each ad, validating it's likelihood of actually being a mustang ad, then gets the positional price info, it is exists, and then determines the year. This is kinda tricky because the year might be 1967, '67, 67, and so on. So I look for the best fit first, then get looser in my tests. No regexs, just blunt force parsing in C - the old "divide and conquer" approach.

    The output is then sorted from lowest to highest price and I create an XML file, using XLINK for the hyperlinks to show the detail ad. I wasted about 5 hours trying to get the hyperlinks to work in Safari and IE 6, then tried Firefox and they worked like a charm.

    I used the curl_easy_... functions, but I'm going to write another version that uses the curl_multi interface, and with that version, also search every city in the country. Kinda neat! libcurl sure makes it easy.

    I do have a question that I'm sure I could dig out of researching, but figured I would ask here. (I'm obviously too lazy to manually search for a mustang... )

    One of the sites always failed to respond, so my program just sat there, waiting synchronously, and after a few minutes I killed it. Is there timeout value I can specify for curl_easy_perform to not just sit there and wait?

    Thanks.
    Mainframe assembler programmer by trade. C coder when I can.

  2. #2
    Registered User
    Join Date
    Sep 2004
    Location
    California
    Posts
    3,268
    libcurl has 2 timeout values which can be set via curl_easy_setopt. One is CURLOPT_CONNECTTIMEOUT which is the timeout when connecting to the web server. The other is CURLOPT_TIMEOUT which is the read timeout. This is how long you stay connected to the webserver waiting for data to come. If you don't get any data in the specified amount of time, the timeout will expire.
    bit∙hub [bit-huhb] n. A source and destination for information.

  3. #3
    Jack of many languages Dino's Avatar
    Join Date
    Nov 2007
    Location
    Chappell Hill, Texas
    Posts
    2,332
    Ah - makes sense. Thanks bithub. I need to go through all the setopts to learn more. Thanks!
    Mainframe assembler programmer by trade. C coder when I can.

  4. #4
    Jack of many languages Dino's Avatar
    Join Date
    Nov 2007
    Location
    Chappell Hill, Texas
    Posts
    2,332
    CURLOPT_CONNECTTIMEOUT and CURLOPT_TIMEOUT do not seem to have an effect. I set this:
    Code:
    	curl_rc = curl_easy_setopt(easyhandle, CURLOPT_NOSIGNAL, 1L ) ; /* since unix  */ 
    	if (curl_rc) { 
    		printf("*************RC from setopt NOSIGNAL = %d\n", curl_rc) ; 
    		return 8 ;
    	}		
    			
    	curl_rc = curl_easy_setopt(easyhandle, CURLOPT_CONNECTTIMEOUT, 5L ) ; /* 5 seconds */ 
    	if (curl_rc) { 
    		printf("*************RC from setopt CONNECTTIMEOUT = %d\n", curl_rc) ; 
    		return 8 ;
    	}		
    
    	curl_rc = curl_easy_setopt(easyhandle, CURLOPT_TIMEOUT, 10L ) ; /* 10 seconds */ 
    	if (curl_rc) { 
    		printf("*************RC from setopt TIMEOUT = %d\n", curl_rc) ; 
    		return 8 ;
    	}
    and in my loop, it's been sitting there for (on "galveston") ~34 minutes.
    Any other ideas?

    Code:
    [Session started at 2009-10-26 13:13:57 -0500.]
    Listings for abilene...
    Date: >Oct 13<, Year: >1967<, Price: >$6000<, Price (int): >6000<, MM/DD: 10/13, Descr: >67 Mustang GT/CS clone<
    Listings for amarillo...
    Listings for austin...
    Date: >Oct 15<, Year: >1967<, Price: >$14500<, Price (int): >14500<, MM/DD: 10/15, Descr: >1967 Ford Mustang<
    Date: >Oct  4<, Year: >1967<, Price: >$8000<, Price (int): >8000<, MM/DD: 10/04, Descr: >67 Mustang Coupe<
    Date: >Oct  3<, Year: >1967<, Price: >$18500<, Price (int): >18500<, MM/DD: 10/03, Descr: >1967 Mustang Hardtop <
    Listings for beaumont...
    Date: >Oct 20<, Year: >1967<, Price: >$2500<, Price (int): >2500<, MM/DD: 10/20, Descr: >1967 mustang <
    Date: >Oct 20<, Year: >1967<, Price: >$1<, Price (int): >1<, MM/DD: 10/20, Descr: >==1967 FORD MUSTANG! RUNS AND DRIVES!ONLY ONE AROUND!====<
    Listings for brownsville...
    Listings for collegestation...
    Listings for corpuschristi...
    Listings for dallas...
    Date: >Oct 25<, Year: >1967<, Price: >$6500<, Price (int): >6500<, MM/DD: 10/25, Descr: >1967 mustang sale/trade<
    Date: >Oct 24<, Year: >1967<, Price: > <, Price (int): >0<, MM/DD: 10/24, Descr: >1967 mustang<
    Date: >Oct 24<, Year: >1967<, Price: >$6700<, Price (int): >6700<, MM/DD: 10/24, Descr: >1967 mustang<
    Date: >Oct 23<, Year: >1967<, Price: >$5000<, Price (int): >5000<, MM/DD: 10/23, Descr: >Trade for 64-67 Mustang<
    Date: >Oct 23<, Year: >1967<, Price: >$5400<, Price (int): >5400<, MM/DD: 10/23, Descr: >1967 Ford Mustang Coupe<
    Date: >Oct 23<, Year: >1967<, Price: >$29000<, Price (int): >29000<, MM/DD: 10/23, Descr: >1967 Mustang convertible !! CONVERTIBLE !!<
    Date: >Oct 22<, Year: >1967<, Price: >$11000<, Price (int): >11000<, MM/DD: 10/22, Descr: >1967 Mustang<
    Date: >Oct 22<, Year: >1967<, Price: > <, Price (int): >0<, MM/DD: 10/22, Descr: >67 mustang 351 windsor<
    Date: >Oct 21<, Year: >1967<, Price: >$3200<, Price (int): >3200<, MM/DD: 10/21, Descr: >1967 ford Mustang Coup<
    Date: >Oct 20<, Year: >1967<, Price: >$6500<, Price (int): >6500<, MM/DD: 10/20, Descr: >1967 Mustang ***Must See***<
    Date: >Oct 18<, Year: >1967<, Price: >$9800<, Price (int): >9800<, MM/DD: 10/18, Descr: >Beautiful Red 1967 Ford Mustang Coupe (Must See)<
    Date: >Oct 17<, Year: >1967<, Price: >$800<, Price (int): >800<, MM/DD: 10/17, Descr: >1967 Ford Mustang<
    Date: >Oct 16<, Year: >1967<, Price: >$1<, Price (int): >1<, MM/DD: 10/16, Descr: >67 Mustang Parts Car<
    Date: >Oct 13<, Year: >1967<, Price: >$3500<, Price (int): >3500<, MM/DD: 10/13, Descr: >1967 Ford Mustang coupe running and driving<
    Date: >Oct 13<, Year: >1967<, Price: >$9800<, Price (int): >9800<, MM/DD: 10/13, Descr: >Beautiful Red 1967 Mustang Coupe (Must See)<
    Date: >Oct 12<, Year: >1967<, Price: >$5000<, Price (int): >5000<, MM/DD: 10/12, Descr: >1967 mustang v8 302  67 eleanor front end<
    Date: >Oct 12<, Year: >1967<, Price: >$4500<, Price (int): >4500<, MM/DD: 10/12, Descr: >1967 Mustang<
    Date: >Oct 11<, Year: >1967<, Price: > <, Price (int): >0<, MM/DD: 10/11, Descr: >1967 Mustang<
    Date: >Oct  9<, Year: >1967<, Price: > <, Price (int): >0<, MM/DD: 10/09, Descr: >1967   mustang    fastback $2500 project car !! FASTBACK!!<
    Date: >Oct  9<, Year: >1967<, Price: >$2500<, Price (int): >2500<, MM/DD: 10/09, Descr: >67 Mustang<
    Listings for delrio...
    Listings for elpaso...
    Listings for galveston...
    Mainframe assembler programmer by trade. C coder when I can.

  5. #5
    Jack of many languages Dino's Avatar
    Join Date
    Nov 2007
    Location
    Chappell Hill, Texas
    Posts
    2,332
    No, I take that back. My app ended silently and wrote nothing to the output file. I must have a logic error.
    Mainframe assembler programmer by trade. C coder when I can.

  6. #6
    Jack of many languages Dino's Avatar
    Join Date
    Nov 2007
    Location
    Chappell Hill, Texas
    Posts
    2,332
    No, I take back my take back. I must have killed my app. (I thought I was wrong, but I was wrong.) It still blocks on galveston.
    Mainframe assembler programmer by trade. C coder when I can.

  7. #7
    Jack of many languages Dino's Avatar
    Join Date
    Nov 2007
    Location
    Chappell Hill, Texas
    Posts
    2,332
    I commented out the NOSIGNAL option and it seems to timeout as expected now. Since I'm not multi-threading, having signals fire, I'm guessing, is an OK thing there.
    Mainframe assembler programmer by trade. C coder when I can.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Programming Experience
    By shoutatchickens in forum A Brief History of Cprogramming.com
    Replies: 40
    Last Post: 04-04-2008, 01:00 PM
  2. database performance question
    By onefootswill in forum C# Programming
    Replies: 4
    Last Post: 12-28-2007, 06:27 PM
  3. constructor overloading question
    By rozner in forum C++ Programming
    Replies: 6
    Last Post: 10-06-2006, 09:54 PM
  4. Alice....
    By Lurker in forum A Brief History of Cprogramming.com
    Replies: 16
    Last Post: 06-20-2005, 02:51 PM
  5. Yet another memory question
    By Dohojar in forum C Programming
    Replies: 5
    Last Post: 03-16-2002, 01:47 PM