immm-sfa / diyepw Goto Github PK

Set license and disclaimer year to 2021

Hi there! I am using the package on windows and blank lines are created between each records of the resulting EPW file.

Let's document how the user can specify any options they want to customize when creating weather files (max-records-to-interpolate, max-records-to-impute)

as in #20 (comment)

Currently, if a TMY3 or ISD Lite file is missing, it will automatically be downloaded from EnergyPlus or NOAA. The code needs to be updated so that the default behavior instead will be to require user interaction for this download to take place. If the user does not wish to download the file, instructions for downloading it manually will be given.

Concurrently, the file download method itself will be made public so that users can invoke it to download the files manually.

Can you change the copyright to Battelle Memorial Institute. I can see it on several html pages in the docs directory, but I assume there is some origin for this text that the pages are being built from and I'm not sure where it is.

• https://github.com/IMMM-SFA/diyepw/blob/410439531b8d4d6f805fd762ba20842720bd29aa/docs/_build/html/search.html
• https://github.com/IMMM-SFA/diyepw/blob/410439531b8d4d6f805fd762ba20842720bd29aa/docs/_build/html/index.html
• https://github.com/IMMM-SFA/diyepw/blob/410439531b8d4d6f805fd762ba20842720bd29aa/docs/_build/html/py-modindex.html

It also looks like version 0.0.1 is hard coded into the docs (see index.html above). Can we make this consistent with the version number that's showing on pypi?

The main README.md comes from the im3py project, but doesn't accurately describe this project. It should be replaced with instructions for installing and executing the application. Maybe move the README.md from scripts/ into the root directory as a starting point?

We need to make a determination as to what license DIYEPW will be made available under.

The outputs/ directory currently contains results generated by a previous run. They could possible serve as a starting point for a test suite, comparing current outputs with those expected outputs.

Amanda writes:

"The function replace_series_value() is probably unnecessary. I tried to use the .replace method but it wasn’t working and I figured it would be faster to write a function than to keep troubleshooting. So you can fix this if you know how, or just delete the commented out lines (# var_series.replace( …) and leave it as is, since it works and shouldn’t be slowing us down."

As noted in the doc comment of replace_series_value(), this may be related to an issue in Pandas that prevents replace() from working reliably with slices containing heterogenous data types. Just take a look and remove one or the other solution.

Reviewer suggestion: openjournals/joss-reviews#3313 (comment)

I am working on a project where I would like to obtain some AMY files for Hawaii. It seems ISD-lite catalogs only include the continental U.S. Can diyepw be expanded to include Hawaii and other international locations?

Reminder link documentation introduction to diyepw to the JOSS paper contingent on publication with DOI.

When attempting to process the three ISD files for WMP 725776, an exception is generated because there is no EPW file present for that WMO in Energy_Plus_TMY3_EPW/

I think that we can solve this either by getting the missing EPW and adding it, or by removing those ISD files.

Figure out how to wrap a link in a footnote within the YAML block for author within Markdown for the JOSS paper compile.

See #37

The below WMOs are missing data for 2020 at present, which causes the 2019 files to fail in create_amy_epw_files.py because we can't append data from the subsequent year.

I think our options are to just accept the missing files, or to treat a missing year as equivalent to a year that just contains empty rows, which would mean we'd have valid data for most of the year but then a few hours of NAs the evening of December 31st.

The files currently affected are:

722520-12907-2019
722740-23160-2019
725750-24126-2019
723630-23047-2019
747185-03144-2019

@amandadsmith - What is your preference for how we resolve this?

In the main loop in create_amy_epw_files.py, we have a try/catch block that is occasionally catching an error in processing certain files. Look into what is causing each error and see whether additional handling could fix whatever the issues are.

Running this command:
epwfile = diyepw.create_amy_epw_file(725290,2021,allow_downloads=True)
generates a weather file where every other line is blank.

This causes common epw parsers such as the following to fail:
data,meta = pvlib.iotools.read_epw(epwfile)

Environment:
python 3.8.12 [MSC v.1916 64 bit (AMD64)]
diyepw 1.2.2
pvlib 0.9.3

I'm getting the following error from energy plus when I try to use weather files generated from 2021 data with diyepw, however I don't run into issues when I use files generated by the program for previous years (2018-2020). I'm trying to isolate if the problem is originating from diyepw or possibly the weather station data.

************* Beginning Zone Sizing Calculations
** Severe ** Invalid Month Day date format=5/ 2

** Severe ** ProcessEPWHeader: Invalid Typical/Extreme Periods End Date Field(WeatherFile)=5/ 2

** Severe ** Invalid Month Day date format=12/31

** Severe ** Data Periods must be of the form or , (WeatherFile) found=12/31

** Warning ** Weather file location will be used rather than entered (IDF) Location object.
** ~~~ ** ..Location object=GOLDEN-NREL
** ~~~ ** ..Weather File Location=Golden-NREL CO USA customized weather file WMO#=724666
** ~~~ ** ..due to location differences, Latitude difference=[0.00] degrees, Longitude difference=[0.00] degrees.
** ~~~ ** ..Time Zone difference=[0.0] hour(s), Elevation difference=[100.00] percent, [1829.00] meters.
** Severe ** GetNextEnvironment: No location specified, program will terminate.
** Fatal ** GetNextEnvironment: Errors found in Weater Data Input. Program terminates.

I'm calling the package as follows:
diyepw.create_amy_epw_files_for_years_and_wmos([2018,2019,2020,2021],[725650,724695,724660,724666],allow_downloads=True, amy_epw_dir=download_dir)

We've got great information being shown at console, e.g.

727676-24013-2019: The longest set of missing records for Sea_Level_Pressure is 49, but the max allowed is 48

but we're not capturing that in create_amy_epw_files_output. Can we find a way to deliver this information in the CSV file? For the above example, upon seeing this info I would probably prefer to go ahead and have it create the AMY file for 727676-24013-2019, but I might miss it scrolling through all the "Processing" and "Problem processing" messages.

Paper

• L21: "Its output is a weather file"
• L25-26: The separation of diyepw and diyepw-scripts seems to be outdated: Implement cli within main repo and remove the diyepw-scripts repo IMMM-SFA/diyepw#51
• L51f: Could you add another sentence on the functionality/background of LAF (apart from the limitations)?
• references: many links are duplicated, could you remove the duplicates?
• references: could you round out the citations of numpy, pandas, xarray? I.e. with DOI following instructions at https://numpy.org/citing-numpy/, https://pandas.pydata.org/about/citing.html, http://xarray.pydata.org/en/stable/getting-started-guide/faq.html#how-should-i-cite-xarray

I am looking at an error that occurs in create_amy_epw_files.py when processing file 727530-14914-2017 (and 725500-14942-2019, but the example below uses dates from the first).

The data file includes entries from 01:00 January 1, 2017 up to 23:00 December 31, 2017 (note that it's missing an entry for 00:00 January 1). However, the function clean_noaa_df() creates one extra row at the end of the year for 00:00 January 1, 2018, containing all NAs.

Later, when we append the next year's data (727530-14914-2018), which does include an entry for 00:00 January 1, we end up with two entries for the same time. We ultimately shift the timestamps, so that both entries end up timestamped at 18:00 December 31, 2017.

Our code doesn't notice that anything is wrong until we call convert_to_station_pressure(), which in its first line attempts to convert its first argument (Pa) into a float, which fails because the timestamp that we use as an index to get that argument resolves to a list containing two items.

Questions:

• Why does the 2017 data file not have an entry for 00:00 January 1?
• Why does clean_noaa_df() create an extra record for the first hour of the following year? Presumably either this behavior or the 00:00 January 1 entry is inconsistent between files, because otherwise this issue would occur for every file.

In create_amy_epw_files:
max_records_to_interpolate = 6
max_records_to_impute = 48

In analyze_noaa_data:
max_rows_missing = 700
max_consec_rows_missing = 48

Let the user define these values, with the current values as defaults.

This is from the JOSS review thread: openjournals/joss-reviews#3313
Functionality issues reproduced below for convenience.

Functionality

• diyepw.get_wmo_station_location(725300) results in an error

  Exception: Missing WMO station info file at /home/USER/miniconda3/envs/diyepw/lib/python3.9/site-packages/diyepw/data/Weather_Stations_by_County.csv
  

  I have used pip to install diyepw from PyPI. Do you ship the data folder with the pip installation? I think the paths in the package data might not be up to date (

  diyepw/setup.py

  Line 25 in 63e8e9a

  package_data={'diyepw': ['data/**/*', 'test/files/**/*']},

  ). It works if I install directly from the repository with pip install -e .. Could you check that?

• Is it intentional to not implement the defaults mentioned in https://diyepw.readthedocs.io/en/latest/README.html#using-diyepw-to-generate-amy-epw-files for max_records_to_interpolate and max_records_to_impute in the function create_amy_epw_file and similarly for create_amy_epw_files_for_years_and_wmos?

• When running the CLI, e.g. create_amy_epw_files_for_years_and_wmos --help, I get an error:

  Traceback (most recent call last):
    File "/home/ws/sp2668/miniconda3/envs/diyepw/bin/create_amy_epw_files_for_years_and_wmos", line 5, in <module>
    from diyepw.scripts.create_amy_epw_files_for_years_and_wmos import create_amy_epw_files_for_years_and_wmos
  ModuleNotFoundError: No module named 'diyepw.scripts'
  

  Do you know what I might be doing wrong here, or whether this is a bug?

• The jupyter notebook autocomplete shift + TAB only shows the parameters, not the corresponding explanations. It would be nice to have them show up there. Would that be simple to fix? (minor)

• Any reason why analyze_noaa_isd_lite_files has compulsory arguments, but analyze_noaa_isd_lite_file does not? I would have expected both to function widely analogously. (minor)
  @BenjaminStuermer Please advise if you remember.

• The versions in setup.py are mostly fixed. Any chance you can loosen these constraints? (minor)

Raise more helpful errors if the NOAA or EnergyPlus sites are unavailable or return unexpected content.

Running the code with Python 3.6 in the OSX environment will fail to download files with the error msg: SSL: CERTIFICATE_VERIFY_FAILED

The error is because python 3.6 on OSX has no certification and cannot validate SSL connection.
The temporary fix is to run "/Applications/Python\ [VERSION]/Install\ Certificates.command" in the terminal, before using the diyepw package.

I have proposed splitting this project into two separate repositories:

• The DIYEPW Python package, which would contain everything in the current repository's lib/diyepw/ directory.
• The DIYEPW Script toolkit, which would contain everything else currently in the repository.

The expected convention for a repository containing a Python package is that the package itself makes up the entire contents of a repository. The scripts and associated input files are not part of the package - they were specifically requested to be written as command-line tools, which means a user interacts with them in a fundamentally different way than with package code.

If we split them, users who are interested in using the prepared command-line scripts can clone the appropriate repository, and users who are interested in embedding DIYEPW capabilities into their own Python scripts can install the package. If they remain in the same repository, then we have an odd situation where users who want the command-line scripts also have to download the entire package code, even though they should be installing it via a package manager rather than locally.

I'd like to make sure all three of us are on board with this solution (or some alternative) before I proceed, since it's a fairly substantial change to the project.

This is from the JOSS review thread: openjournals/joss-reviews#3313
Functionality issues reproduced below for convenience with comments from @amandadsmith

Documentation

• The first paragraph of the README/documentation directly uses the abbreviations EPW and WMO. For the latter, could you mention what the abbreviation stands for, for the former, could you link to an explanation of the file format and add some background where these files are commonly used?

• EnergyPlus Weather
• World Meteorological Organization
  File format defined: https://bigladdersoftware.com/epx/docs/8-3/auxiliary-programs/energyplus-weather-file-epw-data-dictionary.html
  Note: Can use material from the paper itself for explanation in docs

• In general, the introduction is quite minimal. Sure, it doesn't need to be long, but given the documentation/README will be the primary point of entry for prospective users, I think it would be helpful to copy over some of the background, context, statement of need from the paper.
  Note: Can use material from the paper itself for explanation in docs. Reach out to @amandadsmith for content as needed

• Particularly, the data source NOAA ISD Lite could be featured more prominently.
• Could you link to a website that lists the available codes for the WMOs? Actually, it would be useful to have such a reference also in the docstring of functions like create_amy_epw_files_for_years_and_wmos. I found a link in the tutorial, but currently it's somewhat hidden.

I am not aware of a website that has a current, dependable list. @cdburley referred me to a journal paper for this. We may need to check with him about whether we should qualify this link in some way if we're not confident it lists all the available codes.

• Please also add a page like "Contributing" with community guidelines (one of the boxes I need to tick above)
• You could link to the documentation in the About section of the Github repository (top right) (minor)

There is a variable called ´max_records_to_impute´ in create_amy_epw_files.py that is meant to define a limit to how many records will be imputed during a script run. It currently has no effect.

Reviewer suggestion to use Click package: openjournals/joss-reviews#3313 (comment)

All references to data files within the package need to be rewritten to use pkg_resources instead of hard-coded relative path references. For example, to get the contents of a NOAA ISD Lite catalog:

pkg_resources.resource_string("diyepw", "data/noaa_isd_lite_catalogs/2001")