openeemeter / eeweather Goto Github PK

Describe the bug
Calls to methods to pull ISD temperature data raise an exception:
RuntimeError: Could not connect to ftp.ncdc.noaa.gov

To Reproduce
Steps to reproduce the behavior:

1. call .load_isd_hourly_temp_data() on a station
   Exception encountered:

        host = "ftp.ncdc.noaa.gov"
        for i in range(n_tries):
            # attempt anonymous connection
            try:
                ftp = ftplib.FTP(host, timeout=timeout)
                ftp.login()  # default u='anonymous' p='anonymous@'
                logger.info("Connected to {}.".format(host))
                return ftp
            except ftplib.all_errors as e:
                logger.warn(
                    "Failed attempt ({} of {}) to connect to {}:\n{}".format(
                        i + 1, n_tries, host, e
                    )
                )
    
>       raise RuntimeError("Could not connect to {}.".format(host))
E       RuntimeError: Could not connect to ftp.ncdc.noaa.gov.

../../envs/ohm/lib/python2.7/site-packages/eeweather/connections.py:50: RuntimeError
----------------------------------- Captured stderr call -----------------------------------
2019-11-25 16:01:29,508 WARNING: Failed attempt (1 of 5) to connect to ftp.ncdc.noaa.gov:
530-Unable to set anonymous privileges.
530 Login incorrect.
2019-11-25 16:01:30,036 WARNING: Failed attempt (2 of 5) to connect to ftp.ncdc.noaa.gov:
530-Unable to set anonymous privileges.
530 Login incorrect.
2019-11-25 16:01:30,600 WARNING: Failed attempt (3 of 5) to connect to ftp.ncdc.noaa.gov:
530-Unable to set anonymous privileges.
530 Login incorrect.
2019-11-25 16:01:31,107 WARNING: Failed attempt (4 of 5) to connect to ftp.ncdc.noaa.gov:
530-Unable to set anonymous privileges.
530 Login incorrect.
2019-11-25 16:01:31,511 WARNING: Failed attempt (5 of 5) to connect to ftp.ncdc.noaa.gov:
530-Unable to set anonymous privileges.
530 Login incorrect.```

**Expected behavior**
Normally returns data from NOAA or returns an expected error code

fetch_tmy3_hourly_temp_data always returns the same csv's data because CSVRequestProxy caches but does not use the url as a key.

I have a fix in a branch and can open a pull request: https://github.com/andrew-mcharg-tendril/eeweather/tree/fix_csv_request_proxy

Describe the bug
Trying to follow the basic example, I execute
import eeweather
ranked_stations = eeweather.rank_stations(35, -95)

What results is this traceback and OSError:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\chris.mills\Miniconda3\envs\openee\lib\site-packages\eeweather\ranking.py", line 231, in rank_stations
site_climate_zones = get_lat_long_climate_zones(site_latitude, site_longitude)
File "C:\Users\chris.mills\Miniconda3\envs\openee\lib\site-packages\eeweather\geo.py", line 120, in get_lat_long_climate_zones
from shapely.geometry import Point
File "C:\Users\chris.mills\Miniconda3\envs\openee\lib\site-packages\shapely\geometry\__init__.py", line 4, in <module>
from .base import CAP_STYLE, JOIN_STYLE
File "C:\Users\chris.mills\Miniconda3\envs\openee\lib\site-packages\shapely\geometry\base.py", line 18, in <module>
from shapely.coords import CoordinateSequence
File "C:\Users\chris.mills\Miniconda3\envs\openee\lib\site-packages\shapely\coords.py", line 8, in <module>
from shapely.geos import lgeos
File "C:\Users\chris.mills\Miniconda3\envs\openee\lib\site-packages\shapely\geos.py", line 145, in <module>
_lgeos = CDLL(os.path.join(sys.prefix, 'Library', 'bin', 'geos_c.dll'))
File "C:\Users\chris.mills\Miniconda3\envs\openee\lib\ctypes\__init__.py", line 348, in __init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found

I get this in both the python interpreter and in jupyter lab, and on two different computers using the steps that follow.

To Reproduce
Steps to reproduce the behavior:

1. I have created a fresh conda environment with python=3.6 and then activated it.
2. Ran pip install eeweather.
3. Started the python interpreter or created a new notebook in jupyter lab.
4. import eeweather
5. ranked_stations = eeweather.rank_stations(35, -95)

Expected behavior
As discussed in the documentation: https://eeweather.readthedocs.io/en/latest/basics.html

Screenshots
If applicable, add screenshots to help explain your problem.
Here's what's installed in my environment.

Desktop (please complete the following information):

• OS: Windows 10 (one computer has Pro edition, Version 1909, and the other has Home edition, Version 1903)

Additional context
Is it simply that Windows isn't supported, so I'm better off using Linux?

Good morning,

I tried using the match_zcta function in eeweather (I'm using Python 3.6.5):

import eeweather
print(eeweather.__version__)
zcta = '70001'
result = eeweather.match_zcta('70001')
result

Which resulted in:

0.2.1
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-857f81446776> in <module>()
      2 print(eeweather.__version__)
      3 zcta = '70001'
----> 4 result = eeweather.match_zcta('70001')
      5 result

AttributeError: module 'eeweather' has no attribute 'match_zcta'

It looks like the match_zcta function, while referenced in the documentation, doesn't actually exist in the package/the master branch. I'm wondering if match_zcta exists in another branch, and maybe hasn't been merged into master yet?

Describe the bug

Looks like the url to the TMY3 link is broken as listed under fetch_tmy3_hourly_temp. File name is stations.py

url = (
"http://rredc.nrel.gov/solar/old_data/nsrdb/"
"1991-2005/data/tmy3/{}TYA.CSV".format(usaf_id)
)

To Reproduce
Steps to reproduce the behavior:

1. Go to Scripts/Tutorial
2. Click on "Run all'
3. Scroll down to 'Cell 20 or 21.'
4. See error

Expected behavior
Jupyter Notebook Error: Could not find http://rredc.nrel.gov/solar/old_data/nsrdb/1991-2005/data/tmy3/722880TYA.CSV.

Screenshots

Desktop (please complete the following information):

• OS: [e.g. macOS].
  Irrelevant

Additional context
Add any other context about the problem here.

The CLI command examples (eeweather --help) use underscores while the actual command names use hyphens.

Describe the bug
When I try to fetch weather data as follows, sqlalchemy gives me the following exception:

import eeweather

ranked_stations = eeweather.rank_stations(35, -95)
station, warnings = eeweather.select_station(ranked_stations)

start_date = datetime.datetime(2016, 6, 1, tzinfo=pytz.UTC)
end_date = datetime.datetime(2017, 9, 15, tzinfo=pytz.UTC)
station.load_isd_hourly_temp_data(start=start_date, end=end_date)

The exception stacktrace:

---------------------------------------------------------------------------
ArgumentError                             Traceback (most recent call last)
[<ipython-input-83-3690950de639>](https://localhost:8080/#) in <cell line: 3>()
      1 start_date = datetime.datetime(2016, 6, 1, tzinfo=pytz.UTC)
      2 end_date = datetime.datetime(2017, 9, 15, tzinfo=pytz.UTC)
----> 3 station.load_isd_hourly_temp_data(start=start_date, end=end_date)

8 frames
[/usr/local/lib/python3.10/dist-packages/eeweather/stations.py](https://localhost:8080/#) in load_isd_hourly_temp_data(self, start, end, read_from_cache, write_to_cache, fetch_from_web, error_on_missing_years)
   1449             Whether or not to write newly loaded data to cache.
   1450         """
-> 1451         return load_isd_hourly_temp_data(
   1452             self.usaf_id,
   1453             start,

[/usr/local/lib/python3.10/dist-packages/eeweather/stations.py](https://localhost:8080/#) in load_isd_hourly_temp_data(usaf_id, start, end, read_from_cache, write_to_cache, error_on_missing_years, fetch_from_web)
    843                 pass
    844     else:
--> 845         data = [
    846             load_isd_hourly_temp_data_cached_proxy(
    847                 usaf_id,

[/usr/local/lib/python3.10/dist-packages/eeweather/stations.py](https://localhost:8080/#) in <listcomp>(.0)
    844     else:
    845         data = [
--> 846             load_isd_hourly_temp_data_cached_proxy(
    847                 usaf_id,
    848                 year,

[/usr/local/lib/python3.10/dist-packages/eeweather/stations.py](https://localhost:8080/#) in load_isd_hourly_temp_data_cached_proxy(usaf_id, year, read_from_cache, write_to_cache, fetch_from_web)
    713 ):
    714     # take from cache?
--> 715     data_ok = validate_isd_hourly_temp_data_cache(usaf_id, year)
    716 
    717     if not fetch_from_web and not data_ok:

[/usr/local/lib/python3.10/dist-packages/eeweather/stations.py](https://localhost:8080/#) in validate_isd_hourly_temp_data_cache(usaf_id, year)
    482 def validate_isd_hourly_temp_data_cache(usaf_id, year):
    483     key = get_isd_hourly_temp_data_cache_key(usaf_id, year)
--> 484     store = eeweather.connections.key_value_store_proxy.get_store()
    485 
    486     # fail if no key

[/usr/local/lib/python3.10/dist-packages/eeweather/connections.py](https://localhost:8080/#) in get_store(self)
    110     def get_store(self):  # pragma: no cover
    111         if self._store is None:
--> 112             self._store = KeyValueStore()
    113         return self._store
    114 

[/usr/local/lib/python3.10/dist-packages/eeweather/cache.py](https://localhost:8080/#) in __init__(self, url)
     51         if not has_sqlalchemy:  # pragma: no cover
     52             raise ImportError("KeyValueStore requires sqlalchemy.")
---> 53         self._prepare_db(url)
     54 
     55     def __repr__(self):

[/usr/local/lib/python3.10/dist-packages/eeweather/cache.py](https://localhost:8080/#) in _prepare_db(self, url)
     73 
     74         eng = create_engine(url)
---> 75         metadata = MetaData(eng)
     76 
     77         tbl_items = Table(

[/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/schema.py](https://localhost:8080/#) in __init__(self, schema, quote_schema, naming_convention, info)
   5438         """
   5439         if schema is not None and not isinstance(schema, str):
-> 5440             raise exc.ArgumentError(
   5441                 "expected schema argument to be a string, "
   5442                 f"got {type(schema)}."

ArgumentError: expected schema argument to be a string, got <class 'sqlalchemy.engine.base.Engine'>.

The cause is at line 73 in /eeweather/cache.py:

(The stacktrace above shows the line number 74, but the actual code's line number is 73)

To Reproduce
The codes in the description section explains pretty well.

Expected behavior
The weather should be fetched.

Desktop (please complete the following information):

• OS: Google Colab

Additional context
eeweather==0.3.24

Describe the bug
Calls to the method to pull ISD temperature data raise an exception

To Reproduce
Steps to reproduce the behavior:

1. call .load_isd_hourly_temp_data() on a station.
   Exception encountered: Could not connect to ftp.ncdc.noaa.gov.

Expected behavior
Normally returns data from NOAA or returns an expected error code.

When rebuilding the database the file scripts/create_ca_climate_zone_geojson.sh tries to download http://ww2.energy.ca.gov/maps/renewable/CA_Building_Standards_Climate_Zones.zip but that climate zone file doesn't exist at that location anymore. I found a copy of the file at https://community.esri.com/ccqpr47374/attachments/ccqpr47374/coordinate-reference-systemsforum-board/1814/1/CA_Building_Standards_Climate_Zones.zip but this is not a stable link. I suggest hosting your own version of this (and other source files) in the same place you've started hosting the TMY3 data.

To Reproduce
Steps to reproduce the behavior:

1. Delete eeweather/resources/metadata.db
2. docker-compose build
3. `docker-compose run --rm --entrypoint "bash -c" eeweather bash
4. (from inside the shell) eeweather rebuild-db
5. That rebuild-db command will fail because it can't find this file.

Expected behavior
When you rebuild the database all of the source files are found and rebuild-db completes successfully.

I tried installing this package with Python 3.6.5 in a brand-new conda environment, and got a ModuleNotFoundError error. Here is the traceback:

Python 3.6.5 |Anaconda, Inc.| (default, Apr 26 2018, 08:42:37)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import eeweather
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/anaconda3/envs/py3test/lib/python3.6/site-packages/eeweather/__init__.py", line 19, in <module>
    from .api import (
  File "/anaconda3/envs/py3test/lib/python3.6/site-packages/eeweather/api.py", line 3, in <module>
    from .connections import metadata_db_connection_proxy
  File "/anaconda3/envs/py3test/lib/python3.6/site-packages/eeweather/connections.py", line 5, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

Here's the code I used to get here:

conda create --name py3test python=3.6
source activate py3test
pip install eeweather

I'm wondering if this needs to be updated with requests?

# setup.py
REQUIRED = [
    'click',
    'pandas',
]

It looks like the latest version of eeweather has pinned a version of pyproj that is not compatible with python 3.7.

The issue is elaborated in this ticket:

pyproj4/pyproj#136

Would it be possible to pin a later version of pyproj?

Thanks!

I am consistently getting a FileNotFoundError on running docker-compose run --rm eeweather rebuild-db on Windows 11. This follows:

• Updated docker build as per dockerfile-dependency-upgrade
• Correct cli installation of ogr2ogr (i.e. GDAL), wget, plus mapshaper.
• Downloading of all primary source files running ~eeweather/scripts/download_primary_sources.sh

Nevertheless I still get: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp_wiu3k1t/cb_2016_us_zcta510_500k.json'. Is this replicated on anyone else's machine trying to rebuild the database? It looks like this is down to a temporary file issue deleting itself after being created, but it is causing real issues with rebuilding the database on my machine.

P.S. this is in service of adding a UK facility to eeweather.

I would like to convert multiple zipcode into Latitude and longitude and then get the data for climate zones. How can I do it for 50,000 zipcodes

In exploring the functionality, it seems that the majority of the stations included are in the U.S. and Australia. It would be great if data for weather stations in other regions of the world could be included. I am particularly interested in the UK and other portions of Europe but it would be great to explore the possibilities of expansion to cover as much of the world as possible.

Describe the bug
I briefly got a runtime error about the NCDC FTP server (ftp.ncdc.noaa.gov) not responding. When I went to check on the web, I discovered that NCDC changed their name to NCEI, and they now show a link to download data from their ftp server at ftp://ftp.ncei.noaa.gov/pub/data/noaa/

NOTE: this turned out to be a transient error, and the next time I ran the function I got a successful response with data. However, it might be safer for the future if we change the ftp address to match their new one.

To Reproduce
Steps to reproduce the behavior:

1. Run station.load_isd_hourly_temp_data(start_date, end_date)

Expected behavior
Data comes back with no errors

The select_station function tries to create an ISDStation instance for each weather station in the rank_stations result. When creating each ISDStation instance, the instance creation attempts to call the self._load_medata function, which attempts to caste numerical metadata values as floats. When they're missing from the metadata database, it causes a type error.

Example traceback:

Traceback (most recent call last):
  File "/.../controllers.py", line 280, in fetch_weather_data
    station, warnings = eeweather.select_station(ranked_stations, coverage_range=(start_date_utc, end_date_utc))
  File "/.../test_env/lib/python3.6/site-packages/eeweather/ranking.py", line 365, in select_station
    station = ISDStation(usaf_id)
  File "/.../test_env/lib/python3.6/site-packages/eeweather/stations.py", line 1013, in __init__
    self._load_metadata()
  File "/.../test_env/lib/python3.6/site-packages/eeweather/stations.py", line 1043, in _load_metadata
    self.elevation = float(metadata['elevation'])  # meters
TypeError: float() argument must be a string or a number, not 'NoneType'

I'm trying to confirm compatibility and operation of eeweather and eemeter using OpenEE docs.

C:\Users\micha>python
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import eeweather
import datetime
import pytz
ranked_stations = eeweather.rank_stations(35, -95)
station, warnings = eeweather.select_station(ranked_stations)
station
ISDStation('720627')
ranked_stations.loc[station.usaf_id]
rank 1
distance_meters 32692.7
latitude 35.283
longitude -95.1
iecc_climate_zone 3
iecc_moisture_regime A
ba_climate_zone Mixed-Humid
ca_climate_zone None
rough_quality low
elevation 183.2
state OK
tmy3_class None
is_tmy3 False
is_cz2010 False
difference_elevation_meters None
Name: 720627, dtype: object
start_date = datetime.datetime(2016, 6, 1, tzinfo=pytz.UTC)
end_date = datetime.datetime(2017, 9, 15, tzinfo=pytz.UTC)
tempC = station.load_isd_hourly_temp_data(start_date, end_date)
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\micha\AppData\Local\Programs\Python\Python36\lib\site-packages\eeweather\stations.py", line 1458, in load_isd_hourly_temp_data
error_on_missing_years=error_on_missing_years,
File "C:\Users\micha\AppData\Local\Programs\Python\Python36\lib\site-packages\eeweather\stations.py", line 853, in load_isd_hourly_temp_data
for year in range(start.year, end.year + 1)
File "C:\Users\micha\AppData\Local\Programs\Python\Python36\lib\site-packages\eeweather\stations.py", line 853, in
for year in range(start.year, end.year + 1)
File "C:\Users\micha\AppData\Local\Programs\Python\Python36\lib\site-packages\eeweather\stations.py", line 715, in load_isd_hourly_temp_data_cached_proxy
data_ok = validate_isd_hourly_temp_data_cache(usaf_id, year)
File "C:\Users\micha\AppData\Local\Programs\Python\Python36\lib\site-packages\eeweather\stations.py", line 484, in validate_isd_hourly_temp_data_cache
store = eeweather.connections.key_value_store_proxy.get_store()
File "C:\Users\micha\AppData\Local\Programs\Python\Python36\lib\site-packages\eeweather\connections.py", line 120, in get_store
self._store = KeyValueStore()
File "C:\Users\micha\AppData\Local\Programs\Python\Python36\lib\site-packages\eeweather\cache.py", line 52, in init
raise ImportError("KeyValueStore requires sqlalchemy.")
ImportError: KeyValueStore requires sqlalchemy.

After receiving this error, I installed SQLAlchemy. The ImportError stopped, but I received the following when writing the next two example lines.

tempC.head()
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'tuple' object has no attribute 'head'
tempF = tempC * 1.8 + 32
Traceback (most recent call last):
File "", line 1, in
TypeError: can't multiply sequence by non-int of type 'float'

I'm expecting to get tempC and convert to tempF outputs, like the example http://eemeter.openee.io/basics.html#loading-sample-data
Should SQLAlchemy be included in installation with eeweather, if it doesn't work without it?

OS: Windows 10
eemeter 2.7.0
eeweather 0.3.20
matplotlib 3.1.1
pandas 0.23.4
patsy 0.5.1
pip 19.1.1
python-dateutil 2.8.0
pytz 2019.1
SQLAlchemy 1.3.5

I'm new, so nothing is too obvious... Thanks.

It appears that eeweather is currently capable of finding weather stations for Canadian locations. I'm wondering if it might be possible for a future release of eeweather to include this data in the metadata.db file. We're currently using postalcodes-ca for retrieving lat / lon for Canadian locations and having the functionality of getting the weather station (and possible climate zone) would be stellar.

Thanks!

This could be very useful if it could look up both the 2006 and 2009 IECC climate zones for a location. I am about 80% sure those are the only years where the zones changed and that 2012 and 2015 have not changed them.