URL help!

[Liz320]

I'm fairly new at this (i.e., one intro programming course in C, 4 years ago and I've been re-learning for the past week), so bear with me. I have a LARGE data collection project where I need to grab 30 years worth, in hourly increments of only certain variables (i.e., day/hour, temperature, pressure, wind speed and several latitude/longitude coordinates), from a climate data website, make calculations, and output to another file as raw data for analysis using a completely different program.

I know that getting the data would require ‘wget’ and changing certain components of the URL (which I understand is called 'parsing a URL'?) many, many, many, many, many… times! After browsing a few tutorials I'm still at a loss as to how to do this. I know what loops are and that they somehow apply, but I’m not exactly sure how they would work here.

Here's one of those URLs to give you an idea...

"http://nomads.ncdc.noaa.gov/dods/NCEP_NARR_DAILY/200901/20090101/narr-a_221_20090101_0000_000.ascii?tmpprs[0][3][94][327]"

I'd need to adjust... "200901" and "20090101", "tmpprs" and each of these four components, "[0][3][94][327]" to represent different values.

Again I'm fairly new at this ‘language’ (i.e., speak slowly please...lol!) Any suggestions are greatly appreciated!!!


- Thanks!

[root4]

I would advise to split the problem in 2: write a script in any higher level language (shell or python or whatever) to retrieve your data from the website and store it in files, and write your calculation program (in C but you can consider alternatives here too) which takes those files in input and output the result. Trying to embed the network part in your c program, as you are beginner, will cost you a lot of time for no real interest.

[Liz320]

I would advise to split the problem in 2: write a script in any higher level language (shell or python or whatever) to retrieve your data from the website and store it in files, and write your calculation program (in C but you can consider alternatives here too) which takes those files in input and output the result. Trying to embed the network part in your c program, as you are beginner, will cost you a lot of time for no real interest.

Thanks and I agree on this approach. Sorry for not being more specific, but I'm concerned about working with this URL. How would I get the values to change in the URL, hundreds or more like thousands of times? I'm not sure how to assign them in order to get what I need out into the file, or what that would entail when they are separate parts of a URL.

[Salem]

curl can fetch URLs with wildcards
cURL - How To Use

[Liz320]

I think I'm just looking for a way to parse the link. Anyone have any suggestions?

Thanks,

[Salem]

For instance

for ( y = 1970 ; y <= 2009 ; y++ ) {
    sprintf( buff, "http://nomads.ncdc.noaa.gov/dods/NCEP_NARR_DAILY/%4d01/%4d0101/narr-a_221_20090101_0000_000.ascii?tmpprs", y, y );
}

More loops, more combinations.

[Liz320]

For instance

for ( y = 1970 ; y <= 2009 ; y++ ) {
    sprintf( buff, "http://nomads.ncdc.noaa.gov/dods/NCEP_NARR_DAILY/%4d01/%4d0101/narr-a_221_20090101_0000_000.ascii?tmpprs", y, y );
}

More loops, more combinations.

Thank you!!!!!!!!!!!!!!!!!