haydenth / ish_parser Goto Github PK

Providing a license will make it much easier for other projects to include this nice little library.

Hi Tom, thanks for writing ish_parser. It was exactly what I was looking for.

Would you please upload v0.0.5 to PyPI? It looks like the most recent version there is 0.0.4.

The AU1 code has all sorts of goodies about the present weather conditions. Add support for this into the tool.

This code doesn't adhere to PEP8, especially two spaces instead of four. Please see here: http://legacy.python.org/dev/peps/pep-0008/

This conversion fails for Distances in METERS and always returns None or 'Missing', and also millimeters convert to inches * 10. I propose the fixes below:

#This is inches per centimeter:
INCH_CONVERSION_FACTOR = 1/2.54

def get_inches(self):
    ''' convert the measurement to inches '''
    if self._obs_value in self.MISSING:
      return 'MISSING'
    if self._obs_units == self.MILLIMETERS:
      return round(self.INCH_CONVERSION_FACTOR * self._obs_value / 10, 4)
    if self._obs_units == self.METERS:
      return round(self.INCH_CONVERSION_FACTOR * self._obs_value * 100, 4)

I am using your library with Pandas. Performance is not that good (it takes 1-2 seconds to process a full year).
The reasons for this are:

• operations are performed sequentially while it could be partially vectorised.
• everyhting is decoded even though you don't need everything

The way I see things:

• use pandas.read_fwf for the mandatory sections
• use apply method for the remaining part of the string (additional fields + remarks).

Usually, you know what information you are trying to get (and probably not every field that is present).
The idea would be to provide a list of desired fields. Based on that list, we could perform only the necessary decoding and return a Pandas Dataframe (or a list of records)

That would increase speed a lot.

Are you interested in such evolution for your library ?

Thanks,
Vincent

I have a question about interpreting a reports sky summation cover observation. If I am looking at the data

coverage                                      OVERCAST - 8/8 coverage
secondary_coverage                    Missing
height                                          300
characteristic                               Missing
Name: (1981-01-28 03:22:00+00:00, sky_cover, 1), dtype: object

and I use get_numeric to convert the coverage data to float

a.coverage.get_numeric()
Out[363]: 4.0

What does 4.0 mean?

Thanks for the great package!

Not much of an 'issue', but...
The usage example with gzip could be better.

import ish_parser
import gzip

ish_filename = 'path/to/a/ish/file.gz'

# Read content
parser = ish_parser.ish_parser()
with gzip.open(ish_filename, 'rb') as gzstream:
    for line in gzstream:
        ishp.loads(line.decode('utf-8'))

# get the list of all reports
reports = parser.get_reports()
print(len(reports))

I'm currently working on code to fetch the data for a particular station over a particular range.
Also converting some results (just air_temperature for now) into pandas dataframes.
If your interested, I can submit that code to you if it ever gets done to a 'clean enough' standard.

Oh, and thanks for writing a parser for the insane mess... I was not looking forward to that.

Currently, this doesn't supporting parsing any of the remarks sections of the file. It basically stops after the additional data section, and does nothing with remarks. This is a feature we should add at some point, especially for stripping things out like METARs, etc.