idarlingteam / idarling

Collaborative Reverse Engineering plugin for IDA Pro & Hex-Rays

Warning

This project is no longer under active development and the more featured and up-to-date fork is probably something more interesting for new comers. Also, IDA has announced an official support for collaborative reverse engineering session and one could also wait for this.

Overview

IDArling is a collaborative reverse engineering plugin for IDA Pro and Hex-Rays. It allows to synchronize in real-time the changes made to a database by multiple users, by connecting together different instances of IDA Pro.

The main features of IDArling are:

• hooking general user events
• structure and enumeration support
• Hex-Rays decompiler syncing
• replay engine and auto-saving
• database loading and saving
• interactive status bar widget
• user cursors (instructions, functions, navbar)
• invite and following an user moves
• dedicated server using Qt5
• integrated server within IDA
• LAN servers discovery
• following an user moves in real time

If you have any questions not worthy of a bug report, feel free to ping us at #idarling on freenode and ask away.

Releases

This project is under active development. Feel free to send a PR if you would like to help! :-)

It is not really stable in its current state, please stayed tuned for a first release of the project!

Installation

Install the IDArling client into the IDA plugins folder.

• Copy idarling_plugin.py and the idarling folder to the IDA plugins folder.
  • On Windows, the folder is at C:\Program Files\IDA 7.x\plugins
  • On macOS, the folder is at /Applications/IDA\ Pro\ 7.x/idabin/plugins
  • On Linux, the folder may be at ~/ida-7.x/plugins/
• Alternatively, you can use the "easy install" method by copying the following line into the console:

import urllib2; exec(urllib2.urlopen('https://raw.githubusercontent.com/IDArlingTeam/IDArling/master/easy_install.py')).read()

Warning: The plugin is only compatible with IDA Pro 7.x on Windows, macOS, and Linux.

The dedicated server requires PyQt5, which is integrated into IDA. If you're using an external Python installation, we recommand using Python 3, which offers a pre-built package that can be installed with a simple pip install PyQt5.

Usage

Open the Settings dialog accessible from the right-clicking the widget located in the status bar. Show the servers list by clicking on the Network Settings tabs and add your server to it. Connect to the server by clicking on it after right-clicking the widget again. Finally, you should be able to access the following menus to upload or download a database:

- File --> Open from server
- File --> Save to server

Thanks

This project is inspired by Sol[IDA]rity. It started after contacting its authors and asking if it was ever going to be released to the public. Lighthouse source code was also carefully studied to understand how to write better IDA plugins.

• Previous plugins, namely CollabREate, IDASynergy, YaCo, were studied during the development process;
• The icons are edited and combined versions from the sites freeiconshop.com and www.iconsplace.com.

Thanks to Quarkslab for allowing this release.

Authors

• Alexandre Adamski <[email protected]>
• Joffrey Guilbon <[email protected]>

idarling's Issues

Switch to a msgpack-like format

See pull request #17 for more information.

As we don't want to use an external dependency, we can implement ourselves the useful bits.

Figure out why the plugin is initialized twice

The plugin is initialized twice in my case, leading to ugly hacks, we need to figured out why...
It can be noted that the plugin is initialized once if no IDB is loaded and twice if an IDB is loaded.

'module' object has no attribute 'SO_REUSEPORT'

windows7 ida7.0
'module' object has no attribute 'SO_REUSEPORT'

self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
AttributeError: 'module' object has no attribute 'SO_REUSEPORT'

'NoneType' object has no attribute 'setEnabled'
File "D:/Tools/IDA 7.0/plugins\idarling\shared\discovery.py", line 130, in stop
self._read_notifier.setEnabled(False)
AttributeError: 'NoneType' object has no attribute 'setEnabled'

Bottom Corner Menu doesn't display on IDA Mac

On IDA 7 for Mac, the bottom corner menu doesn't render. I can right click on it just fine to use it for now.

Images attached

Right click:

(Good work on this by the way so far. Really enjoying it :))

Thread-safety consideration

We all know that IDA is still a single-threaded application.
I wonder if we should use the thread-safety decorator like this:
https://github.com/gaasedelen/lighthouse/blob/master/plugin/lighthouse/util/ida.py
http://www.williballenthin.com/blog/2015/09/04/idapython-synchronization-decorator/

Possible issues caused by this:

• Already decompiling a function error
• Cursor sync causing crash
• Frequent unknown reason crash
• Racing condition to local types list

Latency when sending packet

Some latency is introduced during normal use of the plugin.

It can be seen after a certain period of use that the events received are not those sent.
In addition, if no action is sent on the client side, nothing is received on the server side, however, when an action is performed, the client-side logs indicate sending the right action while an earlier action is received by the server. This bug makes me think of a form of congestion on the side of the event loop Qt/IDA.

This bug is all the more serious because if for a while, a user performs actions and a second remains on hold, this form of latency will introduce desynchronization. Once the waiting user will re-perform actions, his ticks will be those of the server, unfortunately the server ticks will be lower than those of the active client, a desynchronization will be detected, breaking the plugin.

Add a CI test

Well, currently we are relying on manually test, and constantly we failed to find a bug caused by several commits earlier. Even the bugfix commits will sometimes breaking the existing functionalities.
Maybe we can have an AppVeyor to check the functionality.

Break down huge source files

Currently the events.py, hooks.py are a little long, and will get longer and longer as new hooks being introduced.
Maybe we can consider refactor them into several folders? If so, It will be easier to both add new feature and doing some changes.

Fix user experience broken by local type adding

The recent addition of local type has disrupted the creation of structure and structure members.
When adding a member to a structure, the structure is automatically collapsed.
A fix is needed.

Where's requirements.txt?

Hello,

I wanted to try the plugin but the instructions are incomplete.

Where's the 'plugin' folder and where are requirements.txt ?

Hook necessary on Segment Register Value

This is a crutial hook, especially when a portion of code is not recognized as Thumb mode in ARM for example :). Fix needed.

Ask users before following them

Some users report us follow mode can be painful for the user followed.
Maybe we can add a demand to follow, or we can add a checkbox to disable cursors in the all views ?
cc @NeatMonster

Implement the Secure Remote Password protocol

The Secure Remote Password protocol provides us with a way to authenticate an user and obtain a shared secret that can be used to encrypt the connection. This would be make all servers (even the integrated one) secure by default without requiring the use of a certificate chain. I'm open to suggestions though...

Error when processing non-ascii comments

Traceback (most recent call last):
  File "C:/Program Files/IDA 7.0/plugins\idarling\core\hooks.py", line 495, in _hxe_callback
    self._send_user_cmts(func.startEA)
  File "C:/Program Files/IDA 7.0/plugins\idarling\core\hooks.py", line 538, in _send_user_cmts
    cmts = HexRaysHooks._get_user_cmts(ea)
  File "C:/Program Files/IDA 7.0/plugins\idarling\core\hooks.py", line 532, in _get_user_cmts
    cmts.append(((tl.ea, tl.itp), Event.decode(str(cmt))))
  File "C:/Program Files/IDA 7.0/plugins\idarling\core\events.py", line 70, in decode
    return s.decode(locale.getpreferredencoding())
UnicodeDecodeError: 'gbk' codec can't decode bytes in position 2-3: illegal multibyte sequence
IDAPython: Hex-rays python callback returned non-integer; value ignored.

It seems that the IDA uses UTF-8 on windows, too.

TCP keepalive support

I strongly suggest we implement the tcp keepalive.
De-synchronization are fatal for our plugin, but too often the connections are reset and the client won’t be notified until a long time passed. In these situation, the clients believe they are connected, but in fact they are already reset, leading to the de-sync.

Backup IDBs Feature

Hi @NeatMonster ,

It would be great to have a functionality to have server automatically save the uploaded IDBs to a GIT server or whatever is good.

I had submitted the same request but can't seem to find it in the issue list, so filing it again.

Thanks.

Seems the function comment in Hex-Rays not synchonrized

Comments like this won't be sent to server :(

Implement collisions recovery

Seems that the IDA kernel will delete the type and then re-create it, causing the local_types_changed hook being called multiple times.
When the network is quite good (no packet loss, very low delay), it doesn’t matter and works very well, but when I test it on a normal network(~2% loss) with 3 person, it seems that it will lead to racing conditions, corrupt the local types list.
I tried to fix it with duplicate hook event detection, and the packet will be only sent once now, but still I doubt whether there is a better approach.

1. Currently I send all the local types to the remote server, which will probably not that good when there are too much local types/network access is not stable
2. I’m considering whether to post only the modified part of the local types, which won’t have the issues above. But as the IDA uses ordinal to reference the types, so there will be higher risk of inconsistency.

I’m in a dilemma again :(

Bug in painter when pulling an IDB

[DEBUG] Received packet: Subscribe(name=1769727, color=unnamedd, ea=82181, repo=test2, branch=test2, command_type=subscribe, tick=0, type=command, silent=True)
Traceback (most recent call last):
  File "/home/patate/ida-7.1/plugins/idarling/shared/sockets.py", line 262, in event
    self._dispatch()
  File "/home/patate/ida-7.1/plugins/idarling/shared/sockets.py", line 280, in _dispatch
    elif not self.recv_packet(packet):
  File "/home/patate/ida-7.1/plugins/idarling/network/client.py", line 60, in recv_packet
    self._handlers[packet.__class__](packet)
  File "/home/patate/ida-7.1/plugins/idarling/network/client.py", line 90, in _handle_subscribe
    packet.name, packet.color, packet.ea
  File "/home/patate/ida-7.1/plugins/idarling/interface/painter.py", line 125, in paint
    self.paint_database(name, color, address)
  File "/home/patate/ida-7.1/plugins/idarling/interface/painter.py", line 248, in paint_database
    self.paint_function(name, color, address)
  File "/home/patate/ida-7.1/plugins/idarling/interface/painter.py", line 193, in paint_function
    self._set_paint_function(new_func, color)
  File "/home/patate/ida-7.1/plugins/idarling/interface/painter.py", line 64, in _set_paint_function
    function.color = color
TypeError: in method 'func_t_color_set', argument 2 of type 'bgcolor_t'

Track the origin in the netnode

We're saving the repo and branch the database originates from in the netnode. But we do not track the server it came from. We should check if the repo and branch exist on the remote server before subscribing.

Hook "Create struct from selection"

Create a struct from selection doesn't trigger a hook. We need to find a way to hook it (if possible)

Possible way to hook "Change callee address"

We can introduce the NetNode hook

char sub_15031020()
{
  int v0; // ecx
  const char *v1; // rdx
  unsigned int v2; // ebx
  int v4; // eax
  unsigned int v5; // eax
  unsigned int v6; // [rsp+30h] [rbp-428h]
  unsigned int v7; // [rsp+34h] [rbp-424h]
  unsigned int v8; // [rsp+38h] [rbp-420h]
  char v9; // [rsp+40h] [rbp-418h]

  v0 = ph[1];
  if ( v0 == 12 )
  {
    v1 = "$ mips";
  }
  else
  {
    v1 = " $arm";
    if ( v0 != 13 )
      v1 = "$ vmm functions";
  }
  qword_15034080 = (__int64)v1;
  netnode_check(&v7, v1, 0i64, 0i64);
  callui(11i64, &v8);
  v2 = v8;
  if ( (get_flags_ex(v8, 0i64) & 0x600) != 1536 )
    return 0;
  v4 = netnode_altval(v7, v2, 321i64);
  v5 = node2ea((unsigned int)(v4 - 1));
  v6 = v5;
  if ( ph[1] == 13 )
    v6 = v5 & 0xFFFFFFFE;
  qsnprintf(
    &v9,
    1024i64,
    "HELP\n%s\nENDHELP\nEnter the callee address\n\n  <~C~allee:$:500:40:::>\n\n\n",
    "This plugin allows the user to change the address of the called function\n"
    "in constructs like\n"
    "\n"
    "       call esi\n"
    "\n"
    "You can enter a function name instead of its address\n");
  if ( !(unsigned int)KickUI((__int64)&v9) )
    return 1;
  if ( v6 == -1 )
  {
    netnode_supdel(v7, v2, 321i64);
  }
  else
  {
    if ( ph[1] == 13 && !(v6 & 1) )
      get_sreg(v6, 0x14i64);
    v8 = (unsigned __int64)ea2node() + 1;
    netnode_supset(v7, v2, &v8, 4i64, 321);
  }
  auto_mark_range(v2, v2 + 1, 40i64);
  return 1;
}

The Change callee address is provided by the plugin callee.dll & callee64.dll (in windows)
With the code above we can see that that function is achieved by setting netnode values, but sadly there aren't any notifications or hooks for netnode change natively in the IDA SDK.
However as the IDA only runs on x64 platforms from version 7.0, we can use the ctypes module in python to hook the netnode functions (sdk manual) and then sync some specific netnodes.
Also if we made it to sync netnodes, we can actually sync the internal storage of other plugins if needed :).

Approach:

1. Get the target functions' address with ctypes
2. Insert trampolines use a disassembler and an assembler
3. Call the IDAPython_cli_execute_line or IDAPython_extlang_call_func to call the python callback
4. Process the change~

Memory leak of Settings dialogue

In both current master branch and my issue-guis branch, the SettingsDialog is created for everytime you click Settings in menu. However, it is not destroyed after the dialogue got closed. This can be reproduced by adding a singleton class into the __init__ method of class SetingsDialog.

Fix related with this issue will be committed together with other code from #46.

Fix the integrated server

Actually the integrated server is broken, it needs to be fixed.

Annoying "Analyzing..." window

An anonymous user reported this enhancement request.

When a change is synced on the local machine, the "analyzing" window from IDA will come up and last for a few milliseconds. During the analyze, a user can not do anything, which is kind of interruptive for users.

Incompatibility w/ IDASkins (crash on exit)

It seems that using IDAConnect while having the IDASkins plugin installed makes IDA Pro 7.0 crash on exit.
Here is a screenshot of the issue:

Can't send comments data in Pseudocode

I test the comments in disassembly mode, it works fine.
When I try add some comments in Pseudocode mode, the error is shown.
On Windows 10

Traceback (most recent call last):
  File "Z:/tools/IDA/IDA_Pro_v7.0_Portable/plugins\idaconnect\core\hooks.py", line 446, in _hxe_callback
    self._send_user_cmts(func.startEA)
  File "Z:/tools/IDA/IDA_Pro_v7.0_Portable/plugins\idaconnect\core\hooks.py", line 488, in _send_user_cmts
    cmts = HexRaysHooks._get_user_cmts(ea)
  File "Z:/tools/IDA/IDA_Pro_v7.0_Portable/plugins\idaconnect\core\hooks.py", line 482, in _get_user_cmts
    cmts.append(((tl.ea, tl.itp), Event.decode(cmt)))
  File "Z:/tools/IDA/IDA_Pro_v7.0_Portable/plugins\idaconnect\core\events.py", line 54, in decode
    return s.decode(locale.getpreferredencoding())
AttributeError: 'citem_cmt_t' object has no attribute 'decode'
IDAPython: Hex-rays python callback returned non-integer; value ignored.

On OS X 10.13.4

Traceback (most recent call last):
  File "/Applications/IDA Pro 7.0/ida64.app/Contents/MacOS/plugins/idaconnect/core/hooks.py", line 446, in _hxe_callback
    self._send_user_cmts(func.startEA)
  File "/Applications/IDA Pro 7.0/ida64.app/Contents/MacOS/plugins/idaconnect/core/hooks.py", line 488, in _send_user_cmts
    cmts = HexRaysHooks._get_user_cmts(ea)
  File "/Applications/IDA Pro 7.0/ida64.app/Contents/MacOS/plugins/idaconnect/core/hooks.py", line 482, in _get_user_cmts
    cmts.append(((tl.ea, tl.itp), Event.decode(cmt)))
  File "/Applications/IDA Pro 7.0/ida64.app/Contents/MacOS/plugins/idaconnect/core/events.py", line 55, in decode
    return s.decode('utf-8')
AttributeError: 'citem_cmt_t' object has no attribute 'decode'
IDAPython: Hex-rays python callback returned non-integer; value ignored.

Save host config in a better way

Currently, host config is separately saved in several variables, which makes adding config items harder, such as client-side certificate verify, or keep-alive.
Maybe save them in a dict is better? Any recommendation/suggestion?

Implement notifications

We should implement a notification system to show alerts to the user. One example of such notifications might be users joining/leaving the collaborative session.

Switch to black/flake8 + extensions

I would like to format the code using black, but with a line length of 79 (confirming to PEP8).

Maybe we could switch to flake8 to enforce the code-style, and use the following extensions:

• https://github.com/PyCQA/pep8-naming
• https://github.com/PyCQA/flake8-import-order
• https://github.com/PyCQA/flake8-docstrings

I'm not too sure about this though...

---

I want to, at least, remove all the useless comments (e.g. :param arg: the arg) from the code.

Implement auto-import of structure local type

As mentioned in the issue #22, structure synchronized by create struct from selection aren't imported in the remote struct view.
We need to find a way to automatically import them.

Remove paint from function

Currently, the painter paints function with the background color, we must remove paint rather than painting with background color when leaving/saving/uploading an idb.

Auto-analysis & Auto-saving

Implementing some kind of an auto-save is needed to:

• avoid sending all the events since the first upload of the IDB
• avois sending them during a desynchronization when repulling is needed.

Also related, we should figure out what to do during IDA's auto-analysis. During the initial auto-analysis, event sending should probably be disabled. During later analyses, maybe we can keep it running?

Implement servers discovery

We should implement some kind of local server discovery, so that you don't have to manually add a server to the list when you're trying to use the integrated server.

Implement user authentication

Client-side certificate verify is originally supported by SSL, and I already have an implementation(untidy, though) for it. If we need to add it, I can just merge it.

Also, this will be the first authentication method added in the panel mentioned in issue #44

Use QML to define the UI

As we have discussed in issue #23, QML might be a good idea to define the UI, so we can get rid of lots of addWidget in our code. (Of course, just say here if you think there is anything better than QML.)

Use the following checklist when mitigating:

• Find a good directory to contain and load QML files
• For each dialogue:
  • Reproduce UI in QML
  • Rebind slots if necessary
  • Confirm state (i.e. isClicked(), isEnabled()) is same with old code
• If necessary, compile into resource file, and change CI scripts

Renaming a register is not hooked

We must find a way to hook the renaming of registers

Implement user defined color synchronization

There is no hook built-in for when an user is changing the color of a function/basic block/instruction. While the painter takes user-defined colors into account (basically saving it and restoring it when it needs to paint over it), it won't synchronized between users. This will require the patch engine to be implemented.

Implement user authentication

Implementing some kind of user authentication would allow for some partitioning between users and projects so that not everyone is able to access everything on the server. Also, by introducing a permissions system, we could make a project read-only.

Local type issue (TBI)

However, another problem: open the local types list and add a new type:

typedef int myint;

This won't be copied to other IDA copies. When I defined one more typedef:

typedef int iii;

my first definition got copied, but incorrectly:

50 myint Error typedef myint

Implement user cursors

User cursors allow users to follow collaborators working on the same project.

There a 3 kinds of cursors:

• Navband cursors;
• Function cursors;
• Disassembly cursors.

Add authentication window

It is possible to use the client-side cert to auth client, and it is also necessary to set up a split config panel for credential management, so we can add some other auth type like username-password or username-TOTP in the future.

So I will add another panel called "Auth Management", which can be used to add a credential pair with a unique name. And add a drop-down menu in the server edit/add box for select one from the previous mentioned panel.

Status bar widget rendering

When the user is connected to a server, the text of the status bar widget will not always be displayed correctly. This issue occurs on Linux, Windows, and maybe macOS. See the screenshot below.

I've tried debugging this issue before, but couldn't find anything. Will try again...

NoneType for self._repos

Traceback (most recent call last):
File "/home/patate/ida-7.1/plugins/idarling/interface/dialogs.py", line 253, in _new_repo_accepted
if any(repo.name == name for repo in self._repos):
TypeError: 'NoneType' object is not iterable

This issue can be reproduced by launching a seperated server, connecting to it thanks to auto-discovering, and saving idb to remote server.

Implement general settings

We currently only have one menu for anything network-related. A general settings menu is necessary to allow the user to configure things like its username, log level, et cetera.

Radare2 support

Could you please kindly consider the option of supporting radare2 as an instance for this platform? Because - why not? See issue https://github.com/radare/radare2/issues/7410 for the reference. Feel free to open issues in radare2 repo if you need to fix something for your plugin to work.

Fix user name in navbar & add user name in func view

Just add nbytes for navbar and hook display hint for functions

Rearrange widgets in settings window and refactor code related

Currently, some files under idarling/interface are kind of too complex and should be refactored.

Also, the settings window don't have a common save/cancel/reset button, which is kind of awkward for users.

I would refactor those code and make a change to the widgets.

'utf8' codec can't decode byte 0x85 in position 5: invalid start byte

[ERROR] 'utf8' codec can't decode byte 0x85 in position 5: invalid start byte	(sockets:_notify_write)
Traceback (most recent call last):
  File "C:/Users/IEUser/AppData/Roaming/Hex-Rays/IDA Pro/idarling\idarling\shared\sockets.py", line 185, in _notify_write
    line = json.dumps(self._write_packet.build_packet())
  File "C:\Python27\lib\json\__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "C:\Python27\lib\json\encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Python27\lib\json\encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 5: invalid start byte

Not working under terminal version of IDA

Under IDA 7.1 Linux, IDA will crash if start with idat.
We should temporarily disable the plugin if under terminal mode.