PowerPC architecture with custom plugin refuses to decompile about retdec-idaplugin

[s3rvac]

Thank you for the report.

• Can you please share with us the input binary file that you are trying to decompile?
• Can you try decompiling this PowerPC/ELF file (after unziping)? I was able to decompile it without any problems using the 0.4 version of the IDA plugin.

[BullyWiiPlaza]

Your example binary works fine. However, I realized this is a bit of a special case after all but I can make you replicate it easily. It's therefore no surprise that it's not working as of now.

First, download the following plugin and place it in the IDA plugins folder: http://www.mediafire.com/file/6i9tr6dev117mxt
The source code can be found here: https://github.com/aerosoul94/ida_game_elf_loaders/tree/master/src/wiiu
Now download an example ELF binary here: http://www.mediafire.com/file/u19d4vc1nm6p464
Start IDA and load the homebrew_launcher.rpx ELF using the Wii U RPX/RPL loader plugin you just copied to the plugins folder. It will disassemble. Now mark any function and press CTRL + D. You will get the error from the initial post despite the function being "plain" PowerPC code. I'm sure it's just a minor adjustment to make it work.

Thank you!

[s3rvac]

Yes, the binary file that you provided is indeed decompilable outside of IDA via decompile.sh. @PeterMatula, we will need to tweak the checks in the IDA plugin so the file is also decompilable from there.

[PeterMatula]

retdec-idaplugin tries to check if the input being processed can be decompiled before it sends it to retdec. Problem in this case is the following file format check:

inf.filetype == f_BIN || inf.filetype == f_PE || inf.filetype == f_ELF 
|| inf.filetype == f_COFF || inf.filetype == f_HEX

Even though input file is ELF, because wiiu loader was used, IDA has f_LOADER value in inf.filetype.

I tried to solve it intelligently - find out the real file type and check if it is supported. I failed (see below). So the solution is to let all f_LOADER files go through to retdec. Because we do not really know if they can be decompiled, it is possible the decompilation will fail. Also, we have no experience with processing files loaded by custom loaders - it is quite possible something will go wrong (e.g. if custom loader maps binary data to different addresses than retdec's loader, the whole mechanism of function range selection will fail badly - incorrect data will be decoded and decompiled). Therefore I added warning for the users if f_LOADER is let through. In this case however, it looks quite ok (besides general not so good PPC output quality).

What does not work in idasdk v6.6 (for possible future references):

1. It is possible to get linput_t* using open_linput(path, false) from diskio.hpp. This can be used in other functions below.
2. get_basic_file_type() from loader.hpp returns f_BIN, which is not very helpful (real type is f_ELF).
3. All loaders can be get by build_loaders_list(). Then we can iterate over them and find out their filetype_t load_info_t::ftype. Unfortunately for wiiu there is f_COM_old instead of expected f_ELF. So this is also unreliable.

[PeterMatula]

Normally I would write a regression test to check that this is solved, but since this needs a custom loader plugin, it would not be a good idea - everyone running regression tests would need it, or the test would fail.

However, it works for me. I don't know, if you are able to build the master and check it on your machine. If not, you will have to wait for the next release. If it won't work, reopen.