cann't read unicode correctly about pymem

[wkingnet]

update: Problem has been solved

I wrote a small piece of code to convert the UTF16 encoding, maybe you can use it in the future too.

I think UTF8 or other encodings are applicable, you only need to replace ('utf-16') in the code

temp = b''
while pm.read_bytes(address, 1) != b'\x00':
    temp = temp + pm.read_bytes(address, 1)
    address += 1
print(temp.decode('utf-16'))

[wkingnet]

update: Problem has been solved

I wrote a small piece of code to convert the UTF16 encoding, maybe you can use it in the future too.

I think UTF8 or other encodings are applicable, you only need to replace ('utf-16') in the code

temp = b''
while pm.read_bytes(address, 1) != b'\x00':
    temp = temp + pm.read_bytes(address, 1)
    address += 1
print(temp.decode('utf-16'))

this code still have a problem.

result = b''
while pymem_instance.read_bytes(address, 1) != b'\x00':
    result = result + pymem_instance.read_bytes(address, 1)
    print(result)
    address += 1

If the string is all composed of UTF16 is OK. But if the string is composed of a mixture of UTF16 and ASCII, it will still read errors.

The reason for the error is that pymem's read_byte() automatically saves the bytes as ASCII codes, Although what I created in python is a byte variable.

I took two screenshots, the difference is only the CE display text encoding.

[srounet]

Thank you for the reporting, the value is returned as a c_char.

https://docs.python.org/3/library/ctypes.html#ctypes.create_string_buffer

Maybe there should be an alternative that does not break everything within the read_bytes function ?

[wkingnet]

Thank you for the reporting, the value is returned as a c_char.

https://docs.python.org/3/library/ctypes.html#ctypes.create_string_buffer

Maybe there should be an alternative that does not break everything within the read_bytes function ?

If read_bytes function has been using binary to save data like b'\x07\x86\xF8\x66', then everything will be fine. Because you can use decode('GBK/UTF8/16/LADIN') to decode into any kind of encoding

[wkingnet]

update code, now the code can correctly handle UTF16 and ascii mixed encoding

result = ""
address  # a memory address
while True:
    _temp = pymem.read_bytes(address, 2)
    if _temp == b'\x00\x00':
        break
    else:
        try:
            _temp = _temp.decode('utf-16')
        except UnicodeDecodeError:
            logger.warning(f'UTF-16 decode error, replace with empty str')
            _temp = ascii(_temp)
        finally:
            result += _temp
            _temp_address += 2

[StarrFox]

closing this assuming it was fixed by #121

please comment if not