Python 3.8 and stream issues about pynmea2

[Knio]

The API to the library accepts strings, and not bytes. This is intentional. It will probably never accept bytes, unless someone has some good arguments or a proprietary protocol requires some low-level binary data.

You not wanting to properly decode your inputs is not a good argument.

It is your responsibility to know what encodings you need for your own devices and own IO, and to perform them. This is not because I'm lazy, it's because the library cannot know where your data comes from or how it was encoded. The common interface for passing string data is strings.

Please do some research about strings vs bytes best practices. Here's the first google result that I found that I will quote from: https://medium.com/better-programming/strings-unicode-and-bytes-in-python-3-everything-you-always-wanted-to-know-27dc02ff2686

There is no string without encoding
Having a string without knowing its encoding makes no sense at all.

You simply can’t interpret and decode a string unless you know its encoding.

The bytes type has no inherent encoding, so you must know the encoding if you want to try and decode it, as we saw a few paragraphs above. Again: you can’t pretend to decode something unless you know its encoding.

This library could be used in many different contexts. Sometimes directly connected to a device over a serial port, where the encoding is probably ascii, but also reading logs from a file, streaming over the internet, etc, and I have no way of knowing what encodings were used. Your proposed fixes with ascii encoding in the library are pretending to decode it, and while it may work for you most of the time on a serial device, it is incorrect and will cause bugs for other people and other use cases.

The Unicode Sandwich
“Bytes on the outside, unicode on the inside, encode/decode at the edges.”

pynmea2 here is the inside, and deals strictly with strings. The edges, and how you use the library, are up to you.

@vobject:

There's another workaround that doesn't require you to modify the library sources. Instead, convert the serial output to string before passing it to pynmea2:

ser = serial.Serial("/dev/ttyAMA0", baudrate=9600, timeout=0.5)
while True:
    data = ser.readline().decode("ascii", "ignore")
    if data.startswith("$GPGGA"):
        msg = pynmea2.parse(data)
        print(msg)

This isn't a workaround, but is correct usage and is basically how I would recommend you use the library (but without "ignore"). However, don't just blindly guess ascii, without actually know what encoding you're using.

It works ok for GGA, but some other types still fail.

Please provide examples, and I can help debug them, but without any data, I can only speculate that your IO layer is buggy (the "ignore" in your decoding is a red flag to me that your input data is corrupt, and you are masking device errors. Or you're wrong about the encoding and it's not ascii)

This is not just my opinion, it is the accepted practice in Python 3 of how to do IO. Note the examples in pyserial all encode/decode the bytes directly at the IO edge after reading/writing them to the device, how built in file IO operates on bytes and makes the user specify an encoding or do the encoding themselves, etc.

[sterwen]

Hello,

I have already complained several times about that bug, but my correction proposal was rejected. See issue #87 that has been closed rejected. But the problem is still here as you experience it.
This is a pure bug so the knio reaction is a mystery for me. The workaround is in the issue #87

[vobject]

@sterwen is right, the bytes vs. str handling in the library is broken for Python 3.

There's another workaround that doesn't require you to modify the library sources. Instead, convert the serial output to string before passing it to pynmea2:

ser = serial.Serial("/dev/ttyAMA0", baudrate=9600, timeout=0.5)
while True:
    data = ser.readline().decode("ascii", "ignore")
    if data.startswith("$GPGGA"):
        msg = pynmea2.parse(data)
        print(msg)

It works ok for GGA, but some other types still fail.

[Geordie-Jon]

pyserial reads and sends bytes
clearly no good reason to use a stream reader, with .next() explicitly - More strangeness than I like