Downloading a full year of HRRR srf variables about pybkb_v2

[NicWayand]

OK, so this hack worked for future reference:

            # 1) Open the Metadata URL and read the lines
            try:
                # Hack to not certify (https://stackoverflow.com/questions/19268548/python-ignore-certicate-validation-urllib2/28048260#28048260)
                ctx = ssl.create_default_context()
                ctx.check_hostname = False
                ctx.verify_mode = ssl.CERT_NONE
                idxpage = urllib2.urlopen(idxfile, context=ctx)
                lines = idxpage.readlines()
            except:
                print "\n   ERROR!!! Does the .idx file exist: %s \n" % idxfile
                continue

[blaylockbk]

Hi Nic,

I'm glad you are finding the HRRR archive useful.

I'm not sure why the HRRR_S3.py script hangings, and I think the URLError: has to do with the version of urllib you are using. I re-wrote a download script that I hope will simplify things for you and others:
https://github.com/blaylockbk/pyBKB_v2/blob/master/demos/hrrr_variable_from_pando.py
All I did was reorganized the loops

I'd suggest reading all the comments so you can tailor the function parameters for your needs. Let me know if you still get the URLError, and I'll try to find how I got around that earlier.

From what it sounds like you are trying to download, you might try running the main download function like this...

# Download multiple variables from date range
sDATE = date(2016, 1, 1)   # Start date
eDATE = date(2017, 1, 1)   # End date (exclusive)
days = (eDATE-sDATE).days
DATES = [sDATE + timedelta(days=d) for d in range(days)]

# Variable strings must be part of a line in the .idx file
variables = ['TMP:2 m', 'DPT:2 m', 'APCP:surface']

for variable in variables:
    for DATE in DATES:
        # This is the main download function. We are
        # simply looping over all the days and variables
        # we want to get.
        download_HRRR_variable_from_pando(DATE, variable,
                                          hours=[0, 6, 12, 18],
                                          fxx=[8],
                                          model='hrrr',
                                          field='sfc',
                                          outdir='./')

[NicWayand]

Thank you for the updated function! But when I run the example you provided I get multiple:
ERROR!!! Does the .idx file exist: https://api.mesowest.utah.edu/archive/HRRR/oper/sfc/20170310/hrrr.t18z.wrfsfcf00.grib2.idx

Which is strange, because the file does exist at that url.

Maybe its an issue with my urllib2?? I am using python 2.7.12.

EDIT: Yes the underlying issue is still a URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
Will try to figure out why.

[NicWayand]

Closing issue as it was urllib2 bug on my end.

[blaylockbk]

ah, yes. That was the same solution I've used before. Good to hear you got it working. Let me know if you need any other help.