'utf8' codec can't decode byte 0x85 in position 5: invalid start byte about idarling

[NeatMonster]

Glad to hear it.

[NeatMonster]

There was an issue with the encoding under Windows that should be fixed by the commit 5471abd. Could you check if your issue still occurs @hacksysteam? Thank you.

[hacksysteam]

Hi @NeatMonster ,

I checked with the latest version and it seems that the issue still persists.
This is mostly when I map two variables in one variable and rename the variable name.

[ERROR] 'utf8' codec can't decode byte 0x85 in position 4: invalid start byte	(sockets:_notify_write)
Traceback (most recent call last):
  File "C:/Users/IEUser/AppData/Roaming/Hex-Rays/IDA Pro/idarling\idarling\shared\sockets.py", line 187, in _notify_write
    line = json.dumps(self._write_packet.build_packet())
  File "C:\Python27\lib\json\__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "C:\Python27\lib\json\encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Python27\lib\json\encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 4: invalid start byte

Seems the issue is with this line.

https://github.com/IDArlingTeam/IDArling/blob/master/idarling/shared/sockets.py#L187

line = json.dumps(self._write_packet.build_packet())

[NyaMisty]

Could you please give us your IDA version and your system language?

[hacksysteam]

IDA 7.0 and EN-US. It's Windows 10 Enterprise Evaluation VM downloaded from https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/

[NyaMisty]

Oh sorry for the late reply. Could you please share the idb and steps to reproduce the problem with us?
Also are there any logs before the exception raised?

[NeatMonster]

I can't seem to reproduce the issue @hacksysteam. As @NyaMisty mentioned, could you paste the output from IDA's console, as well as give us more detail about what you did prior to the seeing the exception?

[hacksysteam]

Hi @NyaMisty and @NeatMonster ,

Thanks for the response. I will not be able to share the IDB. However, I will let you know the steps to reproduce it.

1. I map two variables into one.
2. Rename the variable

[23:50:12][DEBUG] Sending packet: UserLvarSettingsEvent(lvar_settings={'stkoff_delta': 0L, 'lvvec': [{'flags': 0, 'll': {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 24L, 'atype': 3, 'stkoff': 24L, 'reg1': 24}}, 'cmt': u'', 'type': ('\n=\x04#\x85A', None, None), 'name': u'this'}, {'flags': 0, 'll': {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u'var'}, {'flags': 0, 'll': {'defea': 6442657687L, 'location': {'reg2': 0, 'ea': 64L, 'atype': 3, 'stkoff': 64L, 'reg1': 64}}, 'cmt': u'', 'type': (None, None, None), 'name': u'sum_rlen_llen'}, {'flags': 0, 'll': {'defea': 6442657715L, 'location': {'reg2': 0, 'ea': 48L, 'atype': 3, 'stkoff': 48L, 'reg1': 48}}, 'cmt': u'', 'type': (None, None, None), 'name': u'new_bstr'}, {'flags': 0, 'll': {'defea': 6442657727L, 'location': {'reg2': 0, 'ea': 56L, 'atype': 3, 'stkoff': 56L, 'reg1': 56}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u'gcblock'}, {'flags': 0, 'll': {'defea': 6442657748L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': ('\n=\x04#\x85A', None, None), 'name': u''}, {'flags': 0, 'll': {'defea': 6442657752L, 'location': {'reg2': 0, 'ea': 80L, 'atype': 3, 'stkoff': 80L, 'reg1': 80}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u''}, {'flags': 0, 'll': {'defea': 6442657812L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}, 'cmt': u'', 'type': (None, None, None), 'name': u'sourcebstr'}, {'flags': 0, 'll': {'defea': 6442657821L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': (None, None, None), 'name': u'bstrlen'}, {'flags': 0, 'll': {'defea': 6442657879L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': (None, None, None), 'name': u'pGcBlockVar'}], 'ulv_flags': 1, 'lmaps': [({'defea': 6442657684L, 'location': {'reg2': 0, 'ea': 120L, 'atype': 3, 'stkoff': 120L, 'reg1': 120}}, {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}), ({'defea': 6442657690L, 'location': {'reg2': 0, 'ea': 32L, 'atype': 3, 'stkoff': 32L, 'reg1': 32}}, {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 24L, 'atype': 3, 'stkoff': 24L, 'reg1': 24}})], 'sizes': []}, ea=6442657656)
[DEBUG] Sending packet: UserLvarSettingsEvent(lvar_settings={'stkoff_delta': 0L, 'lvvec': [{'flags': 0, 'll': {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 24L, 'atype': 3, 'stkoff': 24L, 'reg1': 24}}, 'cmt': u'', 'type': ('\n=\x04#\x85A', None, None), 'name': u'this'}, {'flags': 0, 'll': {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u'var'}, {'flags': 0, 'll': {'defea': 6442657687L, 'location': {'reg2': 0, 'ea': 64L, 'atype': 3, 'stkoff': 64L, 'reg1': 64}}, 'cmt': u'', 'type': (None, None, None), 'name': u'sum_rlen_llen'}, {'flags': 0, 'll': {'defea': 6442657715L, 'location': {'reg2': 0, 'ea': 48L, 'atype': 3, 'stkoff': 48L, 'reg1': 48}}, 'cmt': u'', 'type': (None, None, None), 'name': u'new_bstr'}, {'flags': 0, 'll': {'defea': 6442657727L, 'location': {'reg2': 0, 'ea': 56L, 'atype': 3, 'stkoff': 56L, 'reg1': 56}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u'gcblock'}, {'flags': 0, 'll': {'defea': 6442657748L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': ('\n=\x04#\x85A', None, None), 'name': u''}, {'flags': 0, 'll': {'defea': 6442657752L, 'location': {'reg2': 0, 'ea': 80L, 'atype': 3, 'stkoff': 80L, 'reg1': 80}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u''}, {'flags': 0, 'll': {'defea': 6442657812L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}, 'cmt': u'', 'type': (None, None, None), 'name': u'sourcebstr'}, {'flags': 0, 'll': {'defea': 6442657821L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': (None, None, None), 'name': u'bstrlen'}, {'flags': 0, 'll': {'defea': 6442657879L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': (None, None, None), 'name': u'pGcBlockVar'}], 'ulv_flags': 1, 'lmaps': [({'defea': 6442657684L, 'location': {'reg2': 0, 'ea': 120L, 'atype': 3, 'stkoff': 120L, 'reg1': 120}}, {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}), ({'defea': 6442657690L, 'location': {'reg2': 0, 'ea': 32L, 'atype': 3, 'stkoff': 32L, 'reg1': 32}}, {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 24L, 'atype': 3, 'stkoff': 24L, 'reg1': 24}})], 'sizes': []}, ea=6442657656)	(sockets:send_packet)
123  defaultdict(<type 'collections.defaultdict'>, {'lvar_settings': {'stkoff_delta': 0L, 'lvvec': [{'flags': 0, 'll': {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 24L, 'atype': 3, 'stkoff': 24L, 'reg1': 24}}, 'cmt': u'', 'type': ('\n=\x04#\x85A', None, None), 'name': u'this'}, {'flags': 0, 'll': {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u'var'}, {'flags': 0, 'll': {'defea': 6442657687L, 'location': {'reg2': 0, 'ea': 64L, 'atype': 3, 'stkoff': 64L, 'reg1': 64}}, 'cmt': u'', 'type': (None, None, None), 'name': u'sum_rlen_llen'}, {'flags': 0, 'll': {'defea': 6442657715L, 'location': {'reg2': 0, 'ea': 48L, 'atype': 3, 'stkoff': 48L, 'reg1': 48}}, 'cmt': u'', 'type': (None, None, None), 'name': u'new_bstr'}, {'flags': 0, 'll': {'defea': 6442657727L, 'location': {'reg2': 0, 'ea': 56L, 'atype': 3, 'stkoff': 56L, 'reg1': 56}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u'gcblock'}, {'flags': 0, 'll': {'defea': 6442657748L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': ('\n=\x04#\x85A', None, None), 'name': u''}, {'flags': 0, 'll': {'defea': 6442657752L, 'location': {'reg2': 0, 'ea': 80L, 'atype': 3, 'stkoff': 80L, 'reg1': 80}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u''}, {'flags': 0, 'll': {'defea': 6442657812L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}, 'cmt': u'', 'type': (None, None, None), 'name': u'sourcebstr'}, {'flags': 0, 'll': {'defea': 6442657821L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': (None, None, None), 'name': u'bstrlen'}, {'flags': 0, 'll': {'defea': 6442657879L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': (None, None, None), 'name': u'pGcBlockVar'}], 'ulv_flags': 1, 'lmaps': [({'defea': 6442657684L, 'location': {'reg2': 0, 'ea': 120L, 'atype': 3, 'stkoff': 120L, 'reg1': 120}}, {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}), ({'defea': 6442657690L, 'location': {'reg2': 0, 'ea': 32L, 'atype': 3, 'stkoff': 32L, 'reg1': 32}}, {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 24L, 'atype': 3, 'stkoff': 24L, 'reg1': 24}})], 'sizes': []}, 'ea': 6442657656L, 'tick': 1205, 'type': 'event', 'event_type': 'user_lvar_settings'})
[23:50:12][WARNING] Invalid packet being sent: UserLvarSettingsEvent(lvar_settings={'stkoff_delta': 0L, 'lvvec': [{'flags': 0, 'll': {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 24L, 'atype': 3, 'stkoff': 24L, 'reg1': 24}}, 'cmt': u'', 'type': ('\n=\x04#\x85A', None, None), 'name': u'this'}, {'flags': 0, 'll': {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u'var'}, {'flags': 0, 'll': {'defea': 6442657687L, 'location': {'reg2': 0, 'ea': 64L, 'atype': 3, 'stkoff': 64L, 'reg1': 64}}, 'cmt': u'', 'type': (None, None, None), 'name': u'sum_rlen_llen'}, {'flags': 0, 'll': {'defea': 6442657715L, 'location': {'reg2': 0, 'ea': 48L, 'atype': 3, 'stkoff': 48L, 'reg1': 48}}, 'cmt': u'', 'type': (None, None, None), 'name': u'new_bstr'}, {'flags': 0, 'll': {'defea': 6442657727L, 'location': {'reg2': 0, 'ea': 56L, 'atype': 3, 'stkoff': 56L, 'reg1': 56}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u'gcblock'}, {'flags': 0, 'll': {'defea': 6442657748L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': ('\n=\x04#\x85A', None, None), 'name': u''}, {'flags': 0, 'll': {'defea': 6442657752L, 'location': {'reg2': 0, 'ea': 80L, 'atype': 3, 'stkoff': 80L, 'reg1': 80}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u''}, {'flags': 0, 'll': {'defea': 6442657812L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}, 'cmt': u'', 'type': (None, None, None), 'name': u'sourcebstr'}, {'flags': 0, 'll': {'defea': 6442657821L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': (None, None, None), 'name': u'bstrlen'}, {'flags': 0, 'll': {'defea': 6442657879L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': (None, None, None), 'name': u'pGcBlockVar'}], 'ulv_flags': 1, 'lmaps': [({'defea': 6442657684L, 'location': {'reg2': 0, 'ea': 120L, 'atype': 3, 'stkoff': 120L, 'reg1': 120}}, {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}), ({'defea': 6442657690L, 'location': {'reg2': 0, 'ea': 32L, 'atype': 3, 'stkoff': 32L, 'reg1': 32}}, {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 24L, 'atype': 3, 'stkoff': 24L, 'reg1': 24}})], 'sizes': []}, ea=6442657656)
[WARNING] Invalid packet being sent: UserLvarSettingsEvent(lvar_settings={'stkoff_delta': 0L, 'lvvec': [{'flags': 0, 'll': {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 24L, 'atype': 3, 'stkoff': 24L, 'reg1': 24}}, 'cmt': u'', 'type': ('\n=\x04#\x85A', None, None), 'name': u'this'}, {'flags': 0, 'll': {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u'var'}, {'flags': 0, 'll': {'defea': 6442657687L, 'location': {'reg2': 0, 'ea': 64L, 'atype': 3, 'stkoff': 64L, 'reg1': 64}}, 'cmt': u'', 'type': (None, None, None), 'name': u'sum_rlen_llen'}, {'flags': 0, 'll': {'defea': 6442657715L, 'location': {'reg2': 0, 'ea': 48L, 'atype': 3, 'stkoff': 48L, 'reg1': 48}}, 'cmt': u'', 'type': (None, None, None), 'name': u'new_bstr'}, {'flags': 0, 'll': {'defea': 6442657727L, 'location': {'reg2': 0, 'ea': 56L, 'atype': 3, 'stkoff': 56L, 'reg1': 56}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u'gcblock'}, {'flags': 0, 'll': {'defea': 6442657748L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': ('\n=\x04#\x85A', None, None), 'name': u''}, {'flags': 0, 'll': {'defea': 6442657752L, 'location': {'reg2': 0, 'ea': 80L, 'atype': 3, 'stkoff': 80L, 'reg1': 80}}, 'cmt': u'', 'type': ('\n=\x04#\x84p', None, None), 'name': u''}, {'flags': 0, 'll': {'defea': 6442657812L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}, 'cmt': u'', 'type': (None, None, None), 'name': u'sourcebstr'}, {'flags': 0, 'll': {'defea': 6442657821L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': (None, None, None), 'name': u'bstrlen'}, {'flags': 0, 'll': {'defea': 6442657879L, 'location': {'reg2': 0, 'ea': 8L, 'atype': 3, 'stkoff': 8L, 'reg1': 8}}, 'cmt': u'', 'type': (None, None, None), 'name': u'pGcBlockVar'}], 'ulv_flags': 1, 'lmaps': [({'defea': 6442657684L, 'location': {'reg2': 0, 'ea': 120L, 'atype': 3, 'stkoff': 120L, 'reg1': 120}}, {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 16L, 'atype': 3, 'stkoff': 16L, 'reg1': 16}}), ({'defea': 6442657690L, 'location': {'reg2': 0, 'ea': 32L, 'atype': 3, 'stkoff': 32L, 'reg1': 32}}, {'defea': 6442657656L, 'location': {'reg2': 0, 'ea': 24L, 'atype': 3, 'stkoff': 24L, 'reg1': 24}})], 'sizes': []}, ea=6442657656)	(sockets:_notify_write)
[23:50:12][ERROR] 'utf8' codec can't decode byte 0x85 in position 4: invalid start byte
Traceback (most recent call last):
  File "C:/Users/IEUser/AppData/Roaming/Hex-Rays/IDA Pro/idarling\idarling\shared\sockets.py", line 188, in _notify_write
    line = json.dumps(self._write_packet.build_packet())
  File "C:\Python27\lib\json\__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "C:\Python27\lib\json\encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Python27\lib\json\encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 4: invalid start byte
[ERROR] 'utf8' codec can't decode byte 0x85 in position 4: invalid start byte	(sockets:_notify_write)
Traceback (most recent call last):
  File "C:/Users/IEUser/AppData/Roaming/Hex-Rays/IDA Pro/idarling\idarling\shared\sockets.py", line 188, in _notify_write
    line = json.dumps(self._write_packet.build_packet())
  File "C:\Python27\lib\json\__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "C:\Python27\lib\json\encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Python27\lib\json\encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 4: invalid start byte

I suspect the mapping the variables is the cause. But you will know better

[NeatMonster]

@hacksysteam Thank you for sharing the offending event. I did some more digging, this time it should actually be fixed. Could you try the master branch and re-open this issue if necessary? Thanks!

[hacksysteam]

Yep! Seems to work now. Thank you very much.