Possible way to hook "Change callee address" about idarling

[NeatMonster]

I'm hesitant to implement this because so far we've tried to keep the plugin as unintrusive as possible. Still, we have already resorted to something similar for things that cannot be done in another way, like loading another database.

It would also allow us to hook some events (not even plugin-related) that we wouldn't normally have access to. If we do this, I would like to have some heuristic to find the address of the unexported functions (which doesn't apply here, I know), to avoid having to provide offsets for each IDA version.

I will think about it some more, and maybe give it a try. It should probably be optional as tempering with the IDA library might not always be a good idea.

[NyaMisty]

Oh the hook the IDA library is actually not something new. The plugin called REtypedef achieved the typedef function through hook the demangle function in IDA.wll, and it's in fact quite stable, almost no crash at all.
Also, for the unexported function, it seems that hardcoding the offset for different version of IDA will the best choice, since all the heuristic methods can fail and crash everything :)
However I do understand that hook intrusively is quite dangerous and should be used gingerly. For this, we can choose hooking approaches such as PLT hook or Inline hook wisely to avoid the potential crash.

[NeatMonster]

• We're already using ctypes to call unexported functions:

  IDArling/idarling/interface/actions.py

  Lines 260 to 313 in 412478e

  	idaname = 'ida64' if '64' in appName else 'ida'
  	if sys.platform == 'win32':
  	dllname, dlltype = idaname + '.dll', ctypes.windll
  	elif sys.platform == 'linux2':
  	dllname, dlltype = 'lib' + idaname + '.so', ctypes.cdll
  	elif sys.platform == 'darwin':
  	dllname, dlltype = 'lib' + idaname + '.dylib', ctypes.cdll
  	dllpath = ida_diskio.idadir(None)
  	if not os.path.exists(os.path.join(dllpath, dllname)):
  	dllpath = dllpath.replace('ida64', 'ida')
  	dll = dlltype[os.path.join(dllpath, dllname)]
  	# Close the old database
  	oldPath = ida_loader.get_path(ida_loader.PATH_TYPE_IDB)
  	if oldPath:
  	dll.term_database()
  	# Open the new database
  	LP_c_char = ctypes.POINTER(ctypes.c_char)
  	args = [appName, filePath]
  	argc = len(args)
  	argv = (LP_c_char * (argc + 1))()
  	for i, arg in enumerate(args):
  	arg = arg.encode('utf-8')
  	argv[i] = ctypes.create_string_buffer(arg)
  	LP_c_int = ctypes.POINTER(ctypes.c_int)
  	v = ctypes.c_int(0)
  	av = ctypes.addressof(v)
  	pv = ctypes.cast(av, LP_c_int)
  	dll.init_database(argc, argv, pv)
  	# Create a copy of the new database
  	fileExt = '.i64' if '64' in appName else '.idb'
  	tmpFile, tmpPath = tempfile.mkstemp(suffix=fileExt)
  	shutil.copyfile(filePath, tmpPath)
  	class UIHooks(ida_kernwin.UI_Hooks):
  	def database_inited(self, is_new_database, idc_script):
  	self.unhook()
  	# Remove the tmp database
  	os.close(tmpFile)
  	if os.path.exists(tmpPath):
  	os.remove(tmpPath)
  	hooks = UIHooks()
  	hooks.hook()
  	# Open the tmp database
  	s = ida_loader.snapshot_t()
  	s.filename = tmpPath
  	ida_kernwin.restore_database_snapshot(s, None, None)

  .
• We're also using it as a workaround a bug in the Python bindings:

  IDArling/idarling/core/hooks.py

  Lines 506 to 510 in 57a54ac

  	def read_type_sign(obj):
  	import ctypes
  	import struct
  	buf = ctypes.string_at(id(obj), 4)
  	return struct.unpack('I', buf)[0]

  .
• There is also an official blog post from Hex-Rays about doing so: http://www.hexblog.com/?p=695.
• YaCo also has a pull request waiting allowing something similar: DGA-MI-SSI/YaCo#2.

So what I was trying to say is that if I implement this hook, I should implement all the other hooks missing but accessible this way. But hardcoding the offset of each function is very much against what we aimed for at the beginning of this project, so I don't really know how to go about it.

[NyaMisty]

Well sorry I misunderstood you just now. In this way it's really a big dilemma. Personally, I think that we can use hooking as a temporarily workaround, and wait Hex-Rays' updates.
As for the hardcoded offset, I suggest we give the heuristic solution a try, test it on different platforms, and disable when a new version is not supported or not tested.
Also, hooking method in IDA 7.0 is not that easy. Because the the IDA Library becomes to 64-bit, hook trampolines need to be placed in a place near the original code address (±512MB in exact), the pull request for YaCo won't be ported that easily.

[NeatMonster]

Personally, I think that we can use hooking as a temporarily workaround, and wait Hex-Rays' updates.

I agree with you, we're already doing ugly workarounds using ctypes, so it won't be that much uglier; and we've got no other way of implementing the missing hooks anyway.

I suggest we give the heuristic solution a try, test it on different platforms, and disable when a new version is not supported or not tested.

That could work, but I'm not entirely decided yet.

Also, hooking method in IDA 7.0 is not that easy. [...]

I only see two options:

• use the 12-byte mov rax, <abs64>; jmp rax gadget but that will either make some hooks impossible or complicate relocating the overwritten instructions...
• use the 5-byte jmp <rel32> gadget and have the trampoline in a page nearby by specifying a target address to mmap/VirtualAlloc, but that might fail, I'm not sure...

[NyaMisty]

Oh it's the jmp part, that's quite easy and can be achieved in various way, but we need the trampoline, which contains the overwritten instructions to call the original function. Even using the 5-byte jmp can affect instructions with relocation.
If the trampoline is not in a page near to the original address, then most relocation will fail as they can only reference the address in near 32-bit, causing the IDA to crash.
As for the have the trampoline in a page nearby by specifying a target address to mmap/VirtualAlloc, but that might fail, usually there will be gaps between allocated memory areas, so most time we can manage to find a small page between them ;)

[NeatMonster]

Even using the 5-byte jmp can affect instructions with relocation.

I guess we could integrate Captstone/Keystone and write relocation rules, but that would be overkill for our simple needs IMO. Also, we don't want to have any external dependencies, that's why I wrote a networking engine to replace Twisted (and I couldn't even use QtNetwork as it isn't shipped w/ IDA).

Maybe just manually choosing the location of our hooks to avoid overwriting location-dependent instructions would be enough here. Of course, this might make the heuristics more complicated, but that's fine for now.

most time we can manage to find a small page between them

MinHook got the nice idea of allocating a page that contains 64-bit jump gadgets to the trampoline functions. This way, only the minimal amount of code needs to be stored near the hooked functions. I think I will give it a try, but it isn't my highest priority right now as other things are currently broken. :-/

[NyaMisty]

OK, got it ;)

[NeatMonster]

I started working on this the other day. I've successfully hooked an exported function from the library on a 64-bit instance running under Linux, but now I'm looking for a way to notify the Python code.

I don't see IDAPython_cli_execute_line or IDAPython_extlang_call_func being exported anywhere. It doesn't appear to be in libida64, python64, nor ida64. Did I miss something?

In the meantime, I've been trying to use hook_to_notification_point and invoke_callbacks but I still need to figure out how to pass the arguments before I can give my method a try. :-)

[NyaMisty]

Instead I found that we needn't to use the IDAPython_cli_execute_line or IDAPython_extlang_call_func
The ctypes library supports the wrap for c callback. See https://docs.python.org/2/library/ctypes.html#callback-functions

[NeatMonster]

See docs.python.org/2/library/ctypes.html#callback-functions

Damn, that's as easy as it's going to get. I'm not sure I'll have time to work on this before next week.
@NyaMisty if you're interested in working on it in the meantime, I can give you my PoC code.

[NyaMisty]

Actually I've got a PoC now, but I'm stucked on how to call the original function.
As I said above, the trampoline must be put in a page near the target. So I failed when I tried to build the trampoline because the distance between target and trampoline is > 0xffffffff. Obviously we need some cross-platform approaches to search usable memory regions, which make the life a lot harder.
At last I'd like to work on it with u. ;)

[NeatMonster]

Passing an address hint to mmap(2) worked for me under Linux. And I think the same can be done with VirtualAlloc under Windows.

[NyaMisty]

Well in linux that’s easier than I thought.
But in Windows it won’t be that easy :(
We’ll need to get the target page by search them one by one with VirtualQuery.
I’ll push my code to my fork later.

[NeatMonster]

I got it working under Linux, now I'm working on the Windows version. You're right, we need to use VirtualQuery in order to find a free spot. I'm kind of ripping the code from MinHook. :-)

[NyaMisty]

Good! Also could you please have a look at my still-opening pr?

[NeatMonster]

And... I got in working under Windows. Next up is macOS.

[NeatMonster]

Here are the events we noted are missing:

• Change callee address (#9): missing
• User-defined colors (#35): missing

Here are the events added in DGA-MI-SSI/YaCo#2:

• Regvar: unknown
• Regvar Rename: unknown
• Toggle Sign: unknown
• Mark Comment/Delete Mark: unknown
• Area Comment: unknown
• Function Flags: unknown
• Enum Width: unknown

I would appreciate somebody going over them (maybe @patateqbool) and updating their status to either integrated if the hook has been added natively, or missing if it should be added to the patch engine.

[NeatMonster]

And... It works with almost no modifications on macOS too.

[NeatMonster]

I have added my Proof of Concept code. Further developments will continue in the patcher branch.

[NyaMisty]

Suggestion：we can embed an tiny disassembler to avoid hardcoding the trampoline

[NeatMonster]

@NyaMisty Not sure what you mean by that. Having a disassembler would allow us not to have to specify the size of the relocated instructions, but that's all. That would still be interesting, but I haven't been able to find an instruction-length disassembler engine, nor what I able to find any useful APIs into IDA. The only thing that I can think of is to make IDA decompiles itself on the first launch to recover the sizes.

My plan with the patcher is to allow to specify a new hook with the following code:

def callback(a1, a2, a3, a4, a5):
   print 'callback(%x, %x, %x, %x, %x)' % (a1, a2, a3, a4, a5)

info = {
    'addr': 0xdeadbeef,
    'size': 5,
    'args': [
        (ctypes.c_uint, 'rcx'),
        (ctypes.c_uint, 'rdx'),
        (ctypes.c_void_p, 'r8'),
        (ctypes.c_ulonglong, 'r9'),
        (ctypes.c_ubyte, 'rsp+40'),
    ],
    'func': callback,
}

patcher.hook(info)

The user (of the engine) would specify the address to hook, the size of the instructions to relocate, what are the types of the arguments of the callback and where they are located (in a register or on the stack), and finally the callback function. The preparation of the arguments would be performed behind the scene. Here an assembler would be useful, but because of the small number of instructions we use in the trampoline, we can do it manually with a few utility functions to create jumps, push and pop registers, and read memory.

[NyaMisty]

I think hardcoding the trampoline here(https://github.com/IDArlingTeam/IDArling/blob/patcher/idarling/core/patcher.py#L113) is not that robust :(
I've wrote an universal assembler which will offer us relocation informations which can be found here https://gist.github.com/NyaMisty/4ced56540801a2c758a08562ae8c5236.
And also the corresponding trampoline builder: https://gist.github.com/NyaMisty/2a6b7d6a82c9724a92c29dddd41f15de
Also we needn't trampoline that long in current PoC code, only about 10 bytes are enough.

[NeatMonster]

I think hardcoding the trampoline here(/idarling/core/patcher.py@patcher#L113) is not that robust :(

I don't intend on keeping it that way. As I stated earlier, this is just the PoC code that I wrote earlier.

I've wrote an universal assembler which will offer us relocation informations which can be found here gist.github.com/NyaMisty/4ced56540801a2c758a08562ae8c5236.

Nice, I didn't know about libsplice. It looks like what I was initially looking for. But we should consider this example as an easy one as I'm unsure we will have a symbol for all of our hooks. So, they'll probably necessitate some manual analysis, meaning that we'll be able to choose instructions that don't need fixups, and specifying the size of the relocated instructions isn't that big of a deal anymore.

One more thing, how do we know where the arguments that we'll pass to the callback are located? If those are simply the arguments of the function, that's easy, but if we're hooking right in the middle of a function...

Also we needn't trampoline that long in current PoC code, only about 10 bytes are enough.

Are you sure about that? From my experience, the Python callback is likely to change the value of a lot of registers (as specified in the System V/Microsoft x64 calling convention). To make sure of it, I wrote a callback that performs a repetitive operation over a list. I observed that the ymmXX registers were modified. So in order to avoid that, I'm pushing the general registers that aren't callee saved, and avoiding any complex operations within the callback by scheduling the real callback for later (using execute_sync).

[NyaMisty]

Oh I assume we are hooking at the beginning of the function, as most hook library only implemented this. So we need to hook in the middle of a func?? In that case we would need to check and preserve lots of registers :(
Also we needn't trampoline that long in current PoC code, only about 10 bytes are enough.
I used to hook using Cpp, and most of the hook library do so, but I'm not that sure about Python (Sorry :(
But if we only hook the function prologue, we won't need to preserve those registers.

[NeatMonster]

So we need to hook in the middle of a func??

I will investigate more on this, but it didn't seem far-fetched to have to hook within a function, e.g if we need a value for the callback that isn't simply an input parameter. But as I said, I will have to take a look at where the hooks added by YaCo and the new ones we need, are located. This is my next step on this issue.

[NeatMonster]

We've been told by the Hex-Rays support team that all the missing hooks have been implemented into IDA 7.2. I think having our own patching engine is no longer necessary. I'm keeping this issue open to track the progress on implementing those new hooks (when the new version will be out).