haydenth / ish_parser Goto Github PK
View Code? Open in Web Editor NEWParser for NOAA ISH Files
License: MIT License
Parser for NOAA ISH Files
License: MIT License
Providing a license will make it much easier for other projects to include this nice little library.
The AU1 code has all sorts of goodies about the present weather conditions. Add support for this into the tool.
I am using your library with Pandas. Performance is not that good (it takes 1-2 seconds to process a full year).
The reasons for this are:
The way I see things:
Usually, you know what information you are trying to get (and probably not every field that is present).
The idea would be to provide a list of desired fields. Based on that list, we could perform only the necessary decoding and return a Pandas Dataframe (or a list of records)
That would increase speed a lot.
Are you interested in such evolution for your library ?
Thanks,
Vincent
Error getting in prod
WARNING:root:unable to load report, error: Cannot find code AW7 in string 0472723060137222023090820237+35892-078782FM-16+0133KRDU V0302305N0206A018295MN0008055N5+02225+02065999999ADDAA101001731AU107000025AU230020025AU320070035AU400002015AU500000215AW1185AW2305AW3335AW4635AW5905AW6955AW7965GA1075+018295095GA2085+076205999GD13991+0182959GD24991+0762059MA1101425099835MW1635MW2905OC103405OD149903391240REMMET20609/08/23 15:23:02 SPECI KRDU 082023Z 23040G66KT 1/2SM +TSRAGR FG SQ BKN060CB OVC250 22/21 A2995 RMK AO2 PK WND 24066/2021 RAB13GRB23 TSB06 FRQ LTGICCG OHD TS OHD MOV N GR LESS THAN 1/4 P0007 T02220206 (JBA)EQDR01 407WNS038D01 0ADE735 (213).
Currently, this doesn't supporting parsing any of the remarks sections of the file. It basically stops after the additional data section, and does nothing with remarks. This is a feature we should add at some point, especially for stripping things out like METARs, etc.
This code doesn't adhere to PEP8, especially two spaces instead of four. Please see here: http://legacy.python.org/dev/peps/pep-0008/
This conversion fails for Distances in METERS and always returns None or 'Missing', and also millimeters convert to inches * 10. I propose the fixes below:
#This is inches per centimeter:
INCH_CONVERSION_FACTOR = 1/2.54
def get_inches(self):
''' convert the measurement to inches '''
if self._obs_value in self.MISSING:
return 'MISSING'
if self._obs_units == self.MILLIMETERS:
return round(self.INCH_CONVERSION_FACTOR * self._obs_value / 10, 4)
if self._obs_units == self.METERS:
return round(self.INCH_CONVERSION_FACTOR * self._obs_value * 100, 4)
Traceback (most recent call last):
File "/data2/noaa/parse_noaa_wind.py", line 67, in <module>
if 'METAR' not in str(report.report_type) and 'SYNOP' not in str(report.report_type):
File "/home/tom/.local/lib/python3.10/site-packages/ish_parser/ReportType.py", line 45, in __str__
return self.MAP[self._obs_value]
KeyError: 'MESOW'
I have a question about interpreting a reports sky summation cover observation. If I am looking at the data
coverage OVERCAST - 8/8 coverage
secondary_coverage Missing
height 300
characteristic Missing
Name: (1981-01-28 03:22:00+00:00, sky_cover, 1), dtype: object
and I use get_numeric
to convert the coverage data to float
a.coverage.get_numeric()
Out[363]: 4.0
What does 4.0 mean?
Thanks for the great package!
Hi Tom, thanks for writing ish_parser
. It was exactly what I was looking for.
Would you please upload v0.0.5 to PyPI? It looks like the most recent version there is 0.0.4.
Not much of an 'issue', but...
The usage example with gzip could be better.
import ish_parser
import gzip
ish_filename = 'path/to/a/ish/file.gz'
# Read content
parser = ish_parser.ish_parser()
with gzip.open(ish_filename, 'rb') as gzstream:
for line in gzstream:
ishp.loads(line.decode('utf-8'))
# get the list of all reports
reports = parser.get_reports()
print(len(reports))
I'm currently working on code to fetch the data for a particular station over a particular range.
Also converting some results (just air_temperature for now) into pandas dataframes.
If your interested, I can submit that code to you if it ever gets done to a 'clean enough' standard.
Oh, and thanks for writing a parser for the insane mess... I was not looking forward to that.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.