avwx-rest / avwx-engine Goto Github PK

View Code? Open in Web Editor NEW

88.0 88.0 30.0 68.29 MB

Aviation Weather parsing engine. METAR & TAF

License: MIT License

Python 100.00%

avwx-engine's People

Contributors

Stargazers

Watchers

avwx-engine's Issues

Crash when overriding normal METAR fetch

When attempting to manually update a station's METAR, as can be seen in the example from the documentation site, a crash occurs.

>>> from avwx import Metar
>>> ksfo = Metar('KSFO')
>>> report = 'KSFO 031254Z 36024G55KT 320V040 1/8SM R06/0200D +TS VCFC OVC050 BKN040TCU 14/10 A2978 RMK AIRPORT CLOSED'
>>> ksfo.update(report)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/avwx/base.py", line 120, in update
    report = self.service.fetch(self.icao, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/avwx/service/scrape.py", line 101, in fetch
    return aio.run(self.async_fetch(station, timeout))
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/avwx/service/scrape.py", line 109, in async_fetch
    return await self._fetch(station, url, params, timeout)
  File "/usr/local/lib/python3.7/site-packages/avwx/service/scrape.py", line 79, in _fetch
    resp = await client.get(url, params=params)
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1456, in get
    timeout=timeout,
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1282, in request
    request, auth=auth, allow_redirects=allow_redirects, timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1313, in send
    request, auth=auth, timeout=timeout, allow_redirects=allow_redirects
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1342, in _send_handling_redirects
    request, auth=auth, timeout=timeout, history=history
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1378, in _send_handling_auth
    response = await self._send_single_request(request, timeout)
  File "/usr/local/lib/python3.7/site-packages/httpx/_client.py", line 1414, in _send_single_request
    timeout=timeout.as_dict(),
  File "/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 183, in request
    await self._add_to_pool(connection, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 309, in _add_to_pool
    await self._connection_semaphore.acquire(timeout=timeout.get("pool", None))
  File "/usr/local/lib/python3.7/site-packages/httpcore/_backends/asyncio.py", line 203, in acquire
    await asyncio.wait_for(self.semaphore.acquire(), timeout)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    if timeout <= 0:
TypeError: '<=' not supported between instances of 'str' and 'int'

I am getting this crash on both MacOS and Raspberry Pi OS after a fresh install of avwx.

empty responses

Hi,

I use the API from an Azure Function (similar to AWS Lambda) to provide my travelling colleagues with some flight weather info before they drive to the airport.

However, like 4 out of 10 times the API comes back with an empty response and only after retrying several times I get an actual response with weather info.
As this is used as a Slackbot, retrying multiple times is obviously not great as it destroys user experience.

Invoke-RestMethod -Method Get -Uri "http://avwx.rest/api/metar.php?station=eddk&format=JSON&options=translate,info" -Verbose
VERBOSE: GET http://avwx.rest/api/metar.php?station=eddk&format=JSON&options=translate,info with 0-byte payload
VERBOSE: received 1114-byte response of content type application/json

This is a successful response, but then like 40% of all times I get a 0-byte response back.

Any idea as to why this happens or what I can do to prevent this?

Support for Australian METARs

On http://www.bom.gov.au/aviation/forecasts/taf/#20 I can get Australian METARs/TAFs, but unfortunately they're not available trough NOAA.
I couldn't find a clear documented API (they have 'data feeds', and most probably have an API to retrieve METAR/TAF via the page above).

I couldn't quickly figure out what the internal structure for AVWX-Engine is like, so after I get to solving that puzzle for myself, I might be able to send a PR. In the mean time, there's this issue ;)

Print METAR to webhook

Hiya!

Not really an issue, more of a query.

Could I use my API key to print a METAR to a Discord webhook daily?

Error scrapper

Hello, I have the following error using the TAF Scrapper: NOAA_Scrape server returned 404. Is it an error of the library?

ZBAD airport

Looking for METAR or TAF for airport with ICAO "ZBAD" the API returns an error: "ZBAD does not publish reports"

In stations.json everything seems correct:

"ZBAD": {"city": "Beijing", "country": "CN", "elevation_ft": 98, "elevation_m": 30, "iata": "PKX", "icao": "ZBAD", "latitude": 39.509945, "longitude": 116.41092, "name": "Beijing Daxing International Airport", "note": null, "reporting": true, "runways": [{"ident1": "11L", "ident2": "29R", "length_ft": 12467, "width_ft": 0}, {"ident1": "17L", "ident2": "35R", "length_ft": 12467, "width_ft": 0}, {"ident1": "17R", "ident2": "35L", "length_ft": 12467, "width_ft": 0}, {"ident1": "01L", "ident2": "19R", "length_ft": 11155, "width_ft": 0}], "state": "13", "type": "large_airport", "website": null, "wiki": "https://en.wikipedia.org/wiki/Beijing_Daxing_International_Airport"}

And on ADDS I can retrieve the METAR and TAF just fine: https://aviationweather.gov/metar/data?ids=zbad&format=raw&hours=0&taf=on&layout=on

Add avt7 service for china

Example:
http://www.avt7.com/Home/AirportMetarInfo?airport4Code=ZJQH

Incorrect airport names

KSFO = "International Airport", so does KLAX. They look to be defined wrong in stations.json.

Replace validators with voluptuous

Current validation via Flask-RESTful are deprecated. They need to be replaced. Voluptuous has worked well in the passed for input and output validation.

Secondary reason is Flask-RESTful extension is not compatible with Quart conversion

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6091107: character maps to <undefined>

Hi there,

I was trying to follow the basic example on the README of how to use this package. But I got the error:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6091107: character maps to <undefined>

(Full stack trace below)

Steps to Reproduce

Created a new directory using Git Bash on Windows 10
python -m venv venv
source ./venv/Scripts/activate
pip install avwx-engine
started a python repl with python

typed:

>>> import avwx
>>> metar = avwx.Metar('KJFK')

Immediately got an error

Full Stack Trace

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\current\base.py", line 65, in __init__
    super().__init__(icao)
  File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\base.py", line 61, in __init__
    self.station = Station.from_icao(icao)
  File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\station.py", line 158, in from_icao
    info = copy(_STATIONS[ident.upper()])
  File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\structs.py", line 29, in __getitem__
    self._load()
  File "C:\Users\morga\dev\avwx-test\venv\lib\site-packages\avwx\structs.py", line 25, in _load
    self.data = json.load(self.source.open())
  File "C:\Python38\lib\json\__init__.py", line 293, in load
    return loads(fp.read(),
  File "C:\Python38\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 6091107: character maps to <undefined>

Am I doing something wrong?

TAF not available at many airports

Many European airports do not have TAF available using avwx-engine and avwx.rest.

For example:

>>> import avwx
>>> taf = avwx.Taf('EDDM')
>>> taf.update()
False

and

curl --request GET \
  --url 'https://avwx.rest/api/taf/EDDM?options=&airport=true&reporting=true&format=json&onfail=cache' \
  --header 'authorization: Bearer *******************'

Returns a cached report from 2020-03-30T17:51:54.476000Z.

The same result has been tested on EGLL, EDDF, EBBR, LFPG, LEMD.

I get up to date reports for these locations using NOAA's https://aviationweather.gov/taf/data?format=raw&ids=EDDM

Refactor parsing

I think a quick(ish) refactor of the parsing modules could pay dividends in the future. Currently the parsing logic looks through a string and extracts based on string manipulation. Then in the same context, the parsing function determines what type of translation should be used for each of the plucked strings (lets call them ‘atoms’).

What I propose is a lightweight framework that predefines ‘atoms’ as a regular expressions. This allows for multi-worded atoms to be searched for with the same amount of effort as any normal encoded/non-encoded atom. Each atom can be predefined and a translator is responsible for translating/decoding each atom into whatever. This way, each module can just build the atoms it expects to find in whatever it’s going to translate, add them to translators that decode even the most complicated ones (this will allow for those awful ones like tornados and what not), and then in the parser modules, you’d never have to change anything and can add as many handlers as you want.

An example in the remarks could be

PEAK_WIND = r”””
\bPK WND 
(?<direction>\d{3})
(?<velocity>\d{2,3})
/
(?<hours>\d{2})?  # rarely present
(?<minutes>\d{2})
\b”””

Using that, some class...

class PeakWindAtom(Atom):
    pattern = PEAK_WIND
    search(str) -> match or None
    __contains__(str) -> bool

would be used by

class PeakWindHandler(AtomHandler):
    atom: some Atom
    <<abstract>> can_translate(str) -> bool : atom in is string
    <<abstract>> translate(str) -> str: decode the data

would be used by

class RemarksParser(Parser):
    - handlers: List[handlers]
    + parse(remarks_string) -> Dict[str, str]: raw-decoded pairs
    + add_handler

The patterns and atoms could be reused. And simple ones like ACFT MSHP could be build using a simple factory (so the API doesn’t have to change for simple ones or complex ones). All of your existing decoding logic could stay in place, or slowly be changed to regex group dicts for easier changes.

No tests would need to be changed as far as I can tell. Added tests will be in their own module that you can place wherever you like best before any merge.

I think starting with the remarks module is a good place to start. Before I’d get started and submit a PR, is this something you’d be open to?

Metars for airports without constant observation

When a smaller airport like KSQL is closed for the night, the METARs are aging beyond the query window avwx-engine uses (2 hours I believe). E.g. at this point KSQL's last METAR is 149 minutes old and is not returned because it is outside the query window.

Suggested behavior would be to get the last available METAR, even if older than 2 hours.

File "/home/user/metar/venv/lib/python3.6/site-packages/avwx/init.py", line 89, in update
report = self.service.fetch(self.station)
File "/home/user/metar/venv/lib/python3.6/site-packages/avwx/service.py", line 56, in fetch
report = self._extract(resp.read().decode('utf-8'), station)
File "/home/user/metar/venv/lib/python3.6/site-packages/avwx/service.py", line 101, in _extract
raise self.make_err(raw)
avwx.exceptions.InvalidRequest: Could not find report path in NOAA response

74332561 6

SIGMET error

example: "WSMV31 VRMM 020506 VRMF SIGMET 2 VALID 020500/020900 VRMM- VRMF MALE FIR EMBD TS FCST WI N0524 E07800-S0215 E07748-S0600 E07500-S0600 E06807-S0007 E06800-N0457 E07000-N0524 E07800 TOP FL420 MOV E2KT INTSF="

error: "invalid literal for int() with base 10: 'E2KT'"

Allowing for NOTAM body with field delimiters, like `C)`

I came across a NOTAM that was throwing an error:

A3475/22 NOTAMN
Q) LIMM/QFAXX/IV/NBO/A/000/999/4537N00843E005
A) LIMC B) 2205182200 C) PERM
E) REF AIP AD 2 LIMC 1-12 ITEM 20 'LOCAL TRAFFIC REGULATIONS'
BOX 2 'APRON' PARAGRAPH 2.1 'ORDERLY MOVEMENT OF AIRCRAFT ON
APRONS' INDENT 4 'SERVICES PROVIDED' POINT C) 'FOLLOW-ME ASSISTANCE
PROVIDED ON PILOT'S REQUEST AND MANDATORY IN CASE' ADD THE FOLLOWING
IN CASE:
- GENERAL AVIATION AIRCRAFT UP TO ICAO CODE B (MAXIMUM WINGSPAN 24
METERS) AND HELICOPTERS ARRIVING AND DEPARTING FROM STANDS 301 TO 320
AND FROM 330 TO 336.
ARR TAXI ROUTE: AFTER TWR INSTRUCTIONS VIA APN TAXIWAY P-K TO
INTERMEDIATE HOLDING POSITION (IHP) K9 WHERE FOLLOW-ME CAR WILL
BE WAITING.
DEP TAXI ROUTE: AFTER TWR INSTRUCTIONS AND WITH FOLLOW-ME
ASSISTANCE VIA APN TAXIWAY N-K TO IHP K8

The body of the NOTAM (under tag E)) contains text that is the same as a tag, the C) in POINT C) 'FOLLOW-ME.
avwx/current/notam.py doesn't allow for this to happen.

I solved the error by checking that the tag's content is being set for the first time (in theory, arbitrary text should only appear after the header, so this should be relatively safe).

My proposed edit below:

        if tag == "Q":
            if qualifiers is None:
                qualifiers = _qualifiers(item, units)
        elif tag == "A":
            if station is None:
                station = item
        elif tag == "B":
            if start_text == "":
                start_text = item
        elif tag == "C":
            if end_text == "":
                end_text = item
        elif tag == "D":
            if schedule is None:
                schedule = item
        elif tag == "E":
            if body == "":
                body = item
        elif tag == "F":
            if lower is None:
                lower = core.make_altitude(item.split()[0], units, repr=item)[0]
        elif tag == "G":
            if upper is None:
                upper = core.make_altitude(item.split()[0], units, repr=item)[0]

However, one issue remains because the original string is being sliced every time the tags are found, the NOTAM body is cut just before the C) in the text.

body='REF AIP AD 2 LIMC 1-12 ITEM 20 &apos;LOCAL TRAFFIC REGULATIONS&apos;\nBOX 2 &apos;APRON&apos; PARAGRAPH 2.1 &apos;ORDERLY MOVEMENT OF AIRCRAFT ON\nAPRONS&apos; INDENT 4 &apos;SERVICES PROVIDED&apos; POINT'

I'm no RegEx wizard, so there might be a more succinct way of solving this problem by altering the RegEx that matches the tags.

lookup station by IATA code?

i maintain a discord bot that uses this library, and it would be really usful for there to be a way to lookup a station by the iata code, instead of just by the icao code.

METAR with no TAF

Hi, when retrieving a METAR for AU BOM station YCNK, there is no TAF issued as this is not an airport with a published approach. Hence there is only a METAR. How can we retrieve just the metar data only when no TAF exists? I see from service.py that the BOM service looks for TAF to then pull the METAR data. Can this dig a bit deeper?

Add support for NOAA NBM NBX files

Currently only "NBH", "NBS", and "NBE" are supported. These three support forecasts out to 192 hours. Adding support for "NBX" would support forecasts out to 264 hours.

The NBX format is documented here: https://vlab.noaa.gov/web/mdl/nbm-textcard-v4.1

LLJR

any ICAO code starting with "LL" is = Israel
LLJR is not an exception.
See more at : https://en.wikipedia.org/wiki/Atarot_Airport

avwx-rest / avwx-engine Goto Github PK

avwx-engine's People

Contributors

Stargazers

Watchers

Forkers

avwx-engine's Issues

Steps to Reproduce

Full Stack Trace

Recommend Projects

Recommend Topics

Recommend Org

Jobs