avwx-rest / avwx-engine Goto Github PK
View Code? Open in Web Editor NEWAviation Weather parsing engine. METAR & TAF
License: MIT License
Aviation Weather parsing engine. METAR & TAF
License: MIT License
When attempting to manually update a station's METAR, as can be seen in the example from the documentation site, a crash occurs.
>>> from avwx import Metar
>>> ksfo = Metar('KSFO')
>>> report = 'KSFO 031254Z 36024G55KT 320V040 1/8SM R06/0200D +TS VCFC OVC050 BKN040TCU 14/10 A2978 RMK AIRPORT CLOSED'
>>> ksfo.update(report)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/avwx/base.py", line 120, in update
report = self.service.fetch(self.icao, timeout=timeout)
File "/usr/local/lib/python3.7/site-packages/avwx/service/scrape.py", line 101, in fetch
return aio.run(self.async_fetch(station, timeout))
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
return future.result()
File "/usr/local/lib/python3.7/site-packages/avwx/service/scrape.py", line 109, in async_fetch
return await self._fetch(station, url, params, timeout)
File "/usr/local/lib/python3.7/site-packages/avwx/service/scrape.py", line 79, in _fetch
resp = await client.get(url, params=params)
File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1456, in get
timeout=timeout,
File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1282, in request
request, auth=auth, allow_redirects=allow_redirects, timeout=timeout
File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1313, in send
request, auth=auth, timeout=timeout, allow_redirects=allow_redirects
File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1342, in _send_handling_redirects
request, auth=auth, timeout=timeout, history=history
File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1378, in _send_handling_auth
response = await self._send_single_request(request, timeout)
File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1414, in _send_single_request
timeout=timeout.as_dict(),
File "/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 183, in request
await self._add_to_pool(connection, timeout=timeout)
File "/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 309, in _add_to_pool
await self._connection_semaphore.acquire(timeout=timeout.get("pool", None))
File "/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py", line 203, in acquire
await asyncio.wait_for(self.semaphore.acquire(), timeout)
File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
if timeout <= 0:
TypeError: '<=' not supported between instances of 'str' and 'int'
I am getting this crash on both MacOS and Raspberry Pi OS after a fresh install of avwx.
Hi,
I use the API from an Azure Function (similar to AWS Lambda) to provide my travelling colleagues with some flight weather info before they drive to the airport.
However, like 4 out of 10 times the API comes back with an empty response and only after retrying several times I get an actual response with weather info.
As this is used as a Slackbot, retrying multiple times is obviously not great as it destroys user experience.
Invoke-RestMethod -Method Get -Uri "http://avwx.rest/api/metar.php?station=eddk&format=JSON&options=translate,info" -Verbose
VERBOSE: GET http://avwx.rest/api/metar.php?station=eddk&format=JSON&options=translate,info with 0-byte payload
VERBOSE: received 1114-byte response of content type application/json
This is a successful response, but then like 40% of all times I get a 0-byte response back.
Any idea as to why this happens or what I can do to prevent this?
On http://www.bom.gov.au/aviation/forecasts/taf/#20 I can get Australian METARs/TAFs, but unfortunately they're not available trough NOAA.
I couldn't find a clear documented API (they have 'data feeds', and most probably have an API to retrieve METAR/TAF via the page above).
I couldn't quickly figure out what the internal structure for AVWX-Engine is like, so after I get to solving that puzzle for myself, I might be able to send a PR. In the mean time, there's this issue ;)
Hiya!
Not really an issue, more of a query.
Could I use my API key to print a METAR to a Discord webhook daily?
Hello, I have the following error using the TAF Scrapper: NOAA_Scrape server returned 404. Is it an error of the library?
Looking for METAR or TAF for airport with ICAO "ZBAD" the API returns an error: "ZBAD does not publish reports"
In stations.json
everything seems correct:
"ZBAD": {"city": "Beijing", "country": "CN", "elevation_ft": 98, "elevation_m": 30, "iata": "PKX", "icao": "ZBAD", "latitude": 39.509945, "longitude": 116.41092, "name": "Beijing Daxing International Airport", "note": null, "reporting": true, "runways": [{"ident1": "11L", "ident2": "29R", "length_ft": 12467, "width_ft": 0}, {"ident1": "17L", "ident2": "35R", "length_ft": 12467, "width_ft": 0}, {"ident1": "17R", "ident2": "35L", "length_ft": 12467, "width_ft": 0}, {"ident1": "01L", "ident2": "19R", "length_ft": 11155, "width_ft": 0}], "state": "13", "type": "large_airport", "website": null, "wiki": "https://en.wikipedia.org/wiki/Beijing_Daxing_International_Airport"}
And on ADDS I can retrieve the METAR and TAF just fine: https://aviationweather.gov/metar/data?ids=zbad&format=raw&hours=0&taf=on&layout=on
KSFO = "International Airport", so does KLAX. They look to be defined wrong in stations.json.
Current validation via Flask-RESTful are deprecated. They need to be replaced. Voluptuous has worked well in the passed for input and output validation.
Secondary reason is Flask-RESTful extension is not compatible with Quart conversion
Hi there,
I was trying to follow the basic example on the README of how to use this package. But I got the error:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6091107: character maps to <undefined>
(Full stack trace below)
python -m venv venv
source ./venv/Scripts/activate
pip install avwx-engine
python
>>> import avwx
>>> metar = avwx.Metar('KJFK')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\current\base.py", line 65, in __init__
super().__init__(icao)
File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\base.py", line 61, in __init__
self.station = Station.from_icao(icao)
File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\station.py", line 158, in from_icao
info = copy(_STATIONS[ident.upper()])
File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\structs.py", line 29, in __getitem__
self._load()
File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\structs.py", line 25, in _load
self.data = json.load(self.source.open())
File "C:\Python38\lib\json\__init__.py", line 293, in load
return loads(fp.read(),
File "C:\Python38\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6091107: character maps to <undefined>
Am I doing something wrong?
Many European airports do not have TAF available using avwx-engine and avwx.rest.
For example:
>>> import avwx
>>> taf = avwx.Taf('EDDM')
>>> taf.update()
False
and
curl --request GET \
--url 'https://avwx.rest/api/taf/EDDM?options=&airport=true&reporting=true&format=json&onfail=cache' \
--header 'authorization: Bearer *******************'
Returns a cached report from 2020-03-30T17:51:54.476000Z
.
The same result has been tested on EGLL
, EDDF
, EBBR
, LFPG
, LEMD
.
I get up to date reports for these locations using NOAA's https://aviationweather.gov/taf/data?format=raw&ids=EDDM
I think a quick(ish) refactor of the parsing modules could pay dividends in the future. Currently the parsing logic looks through a string and extracts based on string manipulation. Then in the same context, the parsing function determines what type of translation should be used for each of the plucked strings (lets call them ‘atoms’).
What I propose is a lightweight framework that predefines ‘atoms’ as a regular expressions. This allows for multi-worded atoms to be searched for with the same amount of effort as any normal encoded/non-encoded atom. Each atom can be predefined and a translator is responsible for translating/decoding each atom into whatever. This way, each module can just build the atoms it expects to find in whatever it’s going to translate, add them to translators that decode even the most complicated ones (this will allow for those awful ones like tornados and what not), and then in the parser modules, you’d never have to change anything and can add as many handlers as you want.
An example in the remarks could be
PEAK_WIND = r”””
\bPK WND
(?<direction>\d{3})
(?<velocity>\d{2,3})
/
(?<hours>\d{2})? # rarely present
(?<minutes>\d{2})
\b”””
Using that, some class...
class PeakWindAtom(Atom):
pattern = PEAK_WIND
search(str) -> match or None
__contains__(str) -> bool
would be used by
class PeakWindHandler(AtomHandler):
atom: some Atom
<<abstract>> can_translate(str) -> bool : atom in is string
<<abstract>> translate(str) -> str: decode the data
would be used by
class RemarksParser(Parser):
- handlers: List[handlers]
+ parse(remarks_string) -> Dict[str, str]: raw-decoded pairs
+ add_handler
The patterns and atoms could be reused. And simple ones like ACFT MSHP could be build using a simple factory (so the API doesn’t have to change for simple ones or complex ones). All of your existing decoding logic could stay in place, or slowly be changed to regex group dicts for easier changes.
No tests would need to be changed as far as I can tell. Added tests will be in their own module that you can place wherever you like best before any merge.
I think starting with the remarks module is a good place to start. Before I’d get started and submit a PR, is this something you’d be open to?
When a smaller airport like KSQL is closed for the night, the METARs are aging beyond the query window avwx-engine uses (2 hours I believe). E.g. at this point KSQL's last METAR is 149 minutes old and is not returned because it is outside the query window.
Suggested behavior would be to get the last available METAR, even if older than 2 hours.
File "/home/user/metar/venv/lib/python3.6/site-packages/avwx/init.py", line 89, in update
report = self.service.fetch(self.station)
File "/home/user/metar/venv/lib/python3.6/site-packages/avwx/service.py", line 56, in fetch
report = self._extract(resp.read().decode('utf-8'), station)
File "/home/user/metar/venv/lib/python3.6/site-packages/avwx/service.py", line 101, in _extract
raise self.make_err(raw)
avwx.exceptions.InvalidRequest: Could not find report path in NOAA response
example: "WSMV31 VRMM 020506 VRMF SIGMET 2 VALID 020500/020900 VRMM- VRMF MALE FIR EMBD TS FCST WI N0524 E07800-S0215 E07748-S0600 E07500-S0600 E06807-S0007 E06800-N0457 E07000-N0524 E07800 TOP FL420 MOV E2KT INTSF="
error: "invalid literal for int() with base 10: 'E2KT'"
I came across a NOTAM that was throwing an error:
A3475/22 NOTAMN
Q) LIMM/QFAXX/IV/NBO/A/000/999/4537N00843E005
A) LIMC B) 2205182200 C) PERM
E) REF AIP AD 2 LIMC 1-12 ITEM 20 'LOCAL TRAFFIC REGULATIONS'
BOX 2 'APRON' PARAGRAPH 2.1 'ORDERLY MOVEMENT OF AIRCRAFT ON
APRONS' INDENT 4 'SERVICES PROVIDED' POINT C) 'FOLLOW-ME ASSISTANCE
PROVIDED ON PILOT'S REQUEST AND MANDATORY IN CASE' ADD THE FOLLOWING
IN CASE:
- GENERAL AVIATION AIRCRAFT UP TO ICAO CODE B (MAXIMUM WINGSPAN 24
METERS) AND HELICOPTERS ARRIVING AND DEPARTING FROM STANDS 301 TO 320
AND FROM 330 TO 336.
ARR TAXI ROUTE: AFTER TWR INSTRUCTIONS VIA APN TAXIWAY P-K TO
INTERMEDIATE HOLDING POSITION (IHP) K9 WHERE FOLLOW-ME CAR WILL
BE WAITING.
DEP TAXI ROUTE: AFTER TWR INSTRUCTIONS AND WITH FOLLOW-ME
ASSISTANCE VIA APN TAXIWAY N-K TO IHP K8
The body of the NOTAM (under tag E)
) contains text that is the same as a tag, the C)
in POINT C) 'FOLLOW-ME
.
avwx/current/notam.py
doesn't allow for this to happen.
I solved the error by checking that the tag's content is being set for the first time (in theory, arbitrary text should only appear after the header, so this should be relatively safe).
My proposed edit below:
if tag == "Q":
if qualifiers is None:
qualifiers = _qualifiers(item, units)
elif tag == "A":
if station is None:
station = item
elif tag == "B":
if start_text == "":
start_text = item
elif tag == "C":
if end_text == "":
end_text = item
elif tag == "D":
if schedule is None:
schedule = item
elif tag == "E":
if body == "":
body = item
elif tag == "F":
if lower is None:
lower = core.make_altitude(item.split()[0], units, repr=item)[0]
elif tag == "G":
if upper is None:
upper = core.make_altitude(item.split()[0], units, repr=item)[0]
However, one issue remains because the original string is being sliced every time the tags are found, the NOTAM body is cut just before the C)
in the text.
body='REF AIP AD 2 LIMC 1-12 ITEM 20 'LOCAL TRAFFIC REGULATIONS'\nBOX 2 'APRON' PARAGRAPH 2.1 'ORDERLY MOVEMENT OF AIRCRAFT ON\nAPRONS' INDENT 4 'SERVICES PROVIDED' POINT'
I'm no RegEx wizard, so there might be a more succinct way of solving this problem by altering the RegEx that matches the tags.
i maintain a discord bot that uses this library, and it would be really usful for there to be a way to lookup a station by the iata code, instead of just by the icao code.
Hi, when retrieving a METAR for AU BOM station YCNK, there is no TAF issued as this is not an airport with a published approach. Hence there is only a METAR. How can we retrieve just the metar data only when no TAF exists? I see from service.py that the BOM service looks for TAF to then pull the METAR data. Can this dig a bit deeper?
Currently only "NBH", "NBS", and "NBE" are supported. These three support forecasts out to 192 hours. Adding support for "NBX" would support forecasts out to 264 hours.
The NBX format is documented here: https://vlab.noaa.gov/web/mdl/nbm-textcard-v4.1
any ICAO code starting with "LL" is = Israel
LLJR is not an exception.
See more at : https://en.wikipedia.org/wiki/Atarot_Airport
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.