mandiant / flare-ida Goto Github PK
View Code? Open in Web Editor NEWIDA Pro utilities from FLARE team
License: Apache License 2.0
IDA Pro utilities from FLARE team
License: Apache License 2.0
I think "loc" got lost in the merge - in the previous version, "loc" was defined as "loc = 32" prior to the loop at line 260 which contains this line:
crc_data[loc - 32] = get_byte(func.startEA + loc)
Since the loc definition was removed, the module is throwing "NameError: global name 'loc' is not defined"
Thanks for the great scripts!
Erika
As you probably already know IDA 7.4 ships with Python 3.x support.
Default may still be Python 2.x, but as of this year Python 2 is deprecated and many plugins are being migrated to 3.x at the moment.
Are you considering supporting Python 3 support?
The shellcode_hash_search plugin crashes during execution due to an apparently 64 bit integer size problem with the underlying SQLite DB.
---------------------------------------------------------------------------------------------
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:53:40) [MSC v.1500 64 bit (AMD64)]
IDAPython 64-bit v7.3.0 final (serial 0) (c) The IDAPython Team <[email protected]>
---------------------------------------------------------------------------------------------
shellcode_hash: Starting up
shellcode_hash: Processing current segment only: 0x00000000 - 0x00000207
shellcode_hash: 0x000000a3: ror13AddHash32Dll:0xe553a458 kernel32.dll!VirtualAlloc
shellcode_hash: 0x000000bd: ror13AddHash32Dll:0x0726774c kernel32.dll!LoadLibraryA
shellcode_hash: 0x0000012f: ror13AddHash32Dll:0xc99cc96a dnsapi.dll!DnsQuery_A
shellcode_hash: 0x00000198: ror13AddHash32Dll:0x56a2b5f0 kernel32.dll!ExitProcess
shellcode_hash: 0x000001a4: ror13AddHash32Dll:0xe035f044 kernel32.dll!Sleep
shellcode_hash: 0x000001e4: ror13AddHash32Dll:0xcc8e00f4 kernel32.dll!lstrlenA
shellcode_hash: Exception: Python int too large to convert to SQLite INTEGER
Traceback (most recent call last):
File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 342, in lookForOpArgs
hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 192, in getSymbolByTypeHash
cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER
shellcode_hash: 0x000000bc: ror13AddHash32Dll:0x0726774c kernel32.dll!LoadLibraryA
shellcode_hash: Exception: Python int too large to convert to SQLite INTEGER
Traceback (most recent call last):
File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 342, in lookForOpArgs
hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 192, in getSymbolByTypeHash
cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER
shellcode_hash: 0x00000197: ror13AddHash32Dll:0x56a2b5f0 kernel32.dll!ExitProcess
shellcode_hash: Exception: Python int too large to convert to SQLite INTEGER
Traceback (most recent call last):
File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 342, in lookForOpArgs
hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 192, in getSymbolByTypeHash
cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER
shellcode_hash: Exception: Python int too large to convert to SQLite INTEGER
Traceback (most recent call last):
File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 342, in lookForOpArgs
hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
File "C:\D\flare-ida\python\flare\shellcode_hash_search.py", line 192, in getSymbolByTypeHash
cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER
shellcode_hash: Done
Apparently the problem arises due to the storing of the hashes as INT in the SQLite not as string?
Please ref: https://www.google.com/search?q=64+bit+"Python+int+too+large+to+convert+to+SQLite+INTEGER"
RuntimeError: zrange only iterates from smaller to bigger numbers only: 9531258, 9531255
ERROR:idb2pat:make_func_sigs:Failed to create signature for function at 0x916f20L (golang_org_x_net_html_atom_match)
INFO:idb2pat:make_func_sigs:[ 10372 / 10380 ] golang_org_x_net_html_atom_Lookup 0x916f90L
ERROR:idb2pat:make_func_sigs:zrange only iterates from smaller to bigger numbers only: 9531361, 9531358
Traceback (most recent call last):
File "D:/IDAPro6.8/python/idb2pat.py", line 392, in make_func_sigs
sigs.append(make_func_sig(config, f))
File "D:/IDAPro6.8/python/idb2pat.py", line 240, in make_func_sig
ref_loc = find_ref_loc(config, ea, ref)
File "D:/IDAPro6.8/python/idb2pat.py", line 163, in find_ref_loc
for i in zrange(ea, get_item_end(ea) - config.pointer_size):
File "D:/IDAPro6.8/python/idb2pat.py", line 37, in zrange
raise RuntimeError("zrange only iterates from smaller to bigger numbers only: {:d}, {:d}".format(start, end))
A couple of weeks ago Iโve download the flare-ida and, among the scripts, Iโve tested the msdn_crawler. However, I have faced several problems with the parsing process. Additionally, I have tested the MSDN Annotation (MSDN Crawler) in two distinct environments. Few details follow:
installing zipimport hook
import zipimport builtin
installed zipimport hook
C:\Python27\lib\site.pyc matches C:\Python27\lib\site.py
import site precompiled from C:\Python27\lib\site.pyc
C:\Python27\lib\os.pyc matches C:\Python27\lib\os.py
import os precompiled from C:\Python27\lib\os.pyc
import errno builtin
import nt builtin
C:\Python27\lib\ntpath.pyc matches C:\Python27\lib\ntpath.py
import ntpath precompiled from C:\Python27\lib\ntpath.pyc
C:\Python27\lib\stat.pyc matches C:\Python27\lib\stat.py
import stat precompiled from C:\Python27\lib\stat.pyc
C:\Python27\lib\genericpath.pyc matches C:\Python27\lib\genericpath.py
import genericpath precompiled from C:\Python27\lib\genericpath.pyc
C:\Python27\lib\warnings.pyc matches C:\Python27\lib\warnings.py
import warnings precompiled from C:\Python27\lib\warnings.pyc
C:\Python27\lib\linecache.pyc matches C:\Python27\lib\linecache.py
import linecache precompiled from C:\Python27\lib\linecache.pyc
C:\Python27\lib\types.pyc matches C:\Python27\lib\types.py
import types precompiled from C:\Python27\lib\types.pyc
C:\Python27\lib\UserDict.pyc matches C:\Python27\lib\UserDict.py
import UserDict precompiled from C:\Python27\lib\UserDict.pyc
C:\Python27\lib\_abcoll.pyc matches C:\Python27\lib\_abcoll.py
import _abcoll precompiled from C:\Python27\lib\_abcoll.pyc
C:\Python27\lib\abc.pyc matches C:\Python27\lib\abc.py
import abc precompiled from C:\Python27\lib\abc.pyc
C:\Python27\lib\_weakrefset.pyc matches C:\Python27\lib\_weakrefset.py
import _weakrefset precompiled from C:\Python27\lib\_weakrefset.pyc
import _weakref builtin
C:\Python27\lib\copy_reg.pyc matches C:\Python27\lib\copy_reg.py
import copy_reg precompiled from C:\Python27\lib\copy_reg.pyc
C:\Python27\lib\traceback.pyc matches C:\Python27\lib\traceback.py
import traceback precompiled from C:\Python27\lib\traceback.pyc
C:\Python27\lib\sysconfig.pyc matches C:\Python27\lib\sysconfig.py
import sysconfig precompiled from C:\Python27\lib\sysconfig.pyc
C:\Python27\lib\re.pyc matches C:\Python27\lib\re.py
import re precompiled from C:\Python27\lib\re.pyc
C:\Python27\lib\sre_compile.pyc matches C:\Python27\lib\sre_compile.py
import sre_compile precompiled from C:\Python27\lib\sre_compile.pyc
import _sre builtin
C:\Python27\lib\sre_parse.pyc matches C:\Python27\lib\sre_parse.py
import sre_parse precompiled from C:\Python27\lib\sre_parse.pyc
C:\Python27\lib\sre_constants.pyc matches C:\Python27\lib\sre_constants.py
import sre_constants precompiled from C:\Python27\lib\sre_constants.pyc
C:\Python27\lib\locale.pyc matches C:\Python27\lib\locale.py
import locale precompiled from C:\Python27\lib\locale.pyc
import encodings directory C:\Python27\lib\encodings
C:\Python27\lib\encodings\__init__.pyc matches C:\Python27\lib\encodings\__init__.py
import encodings precompiled from C:\Python27\lib\encodings\__init__.pyc
C:\Python27\lib\codecs.pyc matches C:\Python27\lib\codecs.py
import codecs precompiled from C:\Python27\lib\codecs.pyc
import _codecs builtin
C:\Python27\lib\encodings\aliases.pyc matches C:\Python27\lib\encodings\aliases.py
import encodings.aliases precompiled from C:\Python27\lib\encodings\aliases.pyc
import operator builtin
C:\Python27\lib\functools.pyc matches C:\Python27\lib\functools.py
import functools precompiled from C:\Python27\lib\functools.pyc
import _functools builtin
import _locale builtin
C:\Python27\lib\encodings\cp1252.pyc matches C:\Python27\lib\encodings\cp1252.py
import encodings.cp1252 precompiled from C:\Python27\lib\encodings\cp1252.pyc
Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
C:\Python27\lib\csv.pyc matches C:\Python27\lib\csv.py
import csv precompiled from C:\Python27\lib\csv.pyc
import _csv builtin
import cStringIO builtin
import logging directory C:\Python27\lib\logging
C:\Python27\lib\logging\__init__.pyc matches C:\Python27\lib\logging\__init__.py
import logging precompiled from C:\Python27\lib\logging\__init__.pyc
import time builtin
C:\Python27\lib\weakref.pyc matches C:\Python27\lib\weakref.py
import weakref precompiled from C:\Python27\lib\weakref.pyc
import thread builtin
C:\Python27\lib\threading.pyc matches C:\Python27\lib\threading.py
import threading precompiled from C:\Python27\lib\threading.pyc
C:\Python27\lib\collections.pyc matches C:\Python27\lib\collections.py
import collections precompiled from C:\Python27\lib\collections.pyc
import _collections builtin
C:\Python27\lib\keyword.pyc matches C:\Python27\lib\keyword.py
import keyword precompiled from C:\Python27\lib\keyword.pyc
C:\Python27\lib\heapq.pyc matches C:\Python27\lib\heapq.py
import heapq precompiled from C:\Python27\lib\heapq.pyc
import itertools builtin
import _heapq builtin
C:\Python27\lib\atexit.pyc matches C:\Python27\lib\atexit.py
import atexit precompiled from C:\Python27\lib\atexit.pyc
C:\Python27\lib\pprint.pyc matches C:\Python27\lib\pprint.py
import pprint precompiled from C:\Python27\lib\pprint.pyc
C:\Python27\lib\site-packages\BeautifulSoup.pyc matches C:\Python27\lib\site-packages\BeautifulSoup.py
import BeautifulSoup precompiled from C:\Python27\lib\site-packages\BeautifulSoup.pyc
C:\Python27\lib\__future__.pyc matches C:\Python27\lib\__future__.py
import __future__ precompiled from C:\Python27\lib\__future__.pyc
C:\Python27\lib\sgmllib.pyc matches C:\Python27\lib\sgmllib.py
import sgmllib precompiled from C:\Python27\lib\sgmllib.pyc
C:\Python27\lib\markupbase.pyc matches C:\Python27\lib\markupbase.py
import markupbase precompiled from C:\Python27\lib\markupbase.pyc
C:\Python27\lib\htmlentitydefs.pyc matches C:\Python27\lib\htmlentitydefs.py
import htmlentitydefs precompiled from C:\Python27\lib\htmlentitydefs.pyc
C:\flare-ida-master\MSDN_crawler\extract_til_constant_info.pyc matches C:\flare-ida-master\MSDN_crawler\extract_t
il_constant_info.py
import extract_til_constant_info precompiled from C:\flare-ida-master\MSDN_crawler\extract_til_constant_info.pyc
C:\Python27\lib\subprocess.pyc matches C:\Python27\lib\subprocess.py
import subprocess precompiled from C:\Python27\lib\subprocess.pyc
import gc builtin
import msvcrt builtin
import _subprocess builtin
MSDN crawler based on zynamics msdn-crawler - Copyright 2010
C:\Python27\lib\encodings\ascii.pyc matches C:\Python27\lib\encodings\ascii.py
import encodings.ascii precompiled from C:\Python27\lib\encodings\ascii.pyc
Traceback (most recent call last):
File "MSDN_crawler\msdn_crawler.py", line 413, in <module>
main()
File "MSDN_crawler\msdn_crawler.py", line 398, in main
(file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
File "MSDN_crawler\msdn_crawler.py", line 371, in parse_files
result = parse_file(os.path.join(root, file), const_enum)
File "MSDN_crawler\msdn_crawler.py", line 276, in parse_file
return parse_new_style(file, content, const_enum)
File "MSDN_crawler\msdn_crawler.py", line 183, in parse_new_style
parsed_html.find_all(width='60%')]
TypeError: 'NoneType' object is not callable
โฆ..
'ascii' codec can't encode character u'\xa0' in position 174: ordinal not in range(128) - ERROR: virtualallocex.htm
And the last three lines were:
Parsed 341767 files
Extracted information about 34214 functions
ERROR processing 197 files
MSDN Annotator SUMMARY
----------------------------------
Functions not found
-------------------
1 LCMapStringW
2 SetHandleCount
3 LCMapStringA
4 MultiByteToWideChar
5 WideCharToMultiByte
6 InternetOpenUrlA
7 VirtualAlloc
Iโve already seen someone reporting the same problem, but I wasnโt able to find the appropriate explanation.
Please, could you help me?
Thank you.
AB.
For
mov r8d, 95902B19h
IDAPython returns (MSB is 1)
Python>get_operand_value(here(), 1)
0xffffffff95902b19
This values is then queried in the database and not found resulting in no annotation.
For mov r8d, 8FE060Ch
IDAPython returns 0x8fe060c (MSB is 0) and everything works as expected.
PR suggestion for this incoming.
The script works great on x86 applications but fails on x64 applications.
ERROR:idb2pat:make_func_sigs:Python int too large to convert to C long
Traceback (most recent call last):
File "X:/Software/IDA 6.8/python/flare/idb2pat.py", line 370, in make_func_sigs
sigs.append(make_func_sig(config, f))
File "X:/Software/IDA 6.8/python/flare/idb2pat.py", line 219, in make_func_sig
ref_loc = find_ref_loc(config, ea, ref)
File "X:/Software/IDA 6.8/python/flare/idb2pat.py", line 142, in find_ref_loc
for i in xrange(ea, get_item_end(ea) - config.pointer_size + 1):
OverflowError: Python int too large to convert to C long
ERROR:idb2pat:make_func_sigs:Failed to create signature for function at [...]
I tried running the plugin w the db file from the url https://github.com/mr-tz/flare-ida/blob/master/MSDN_data/msdn_data.zip
the plugin threw this error:
Traceback (most recent call last):
File "C:/bin/flare/annotate_IDB_MSDN.py", line 118, in on_ok_button
IDB_MSDN_Annotator.main(config)
File "C:/bin/flare\IDB_MSDN_Annotator_init_.py", line 521, in main
functions_map = parse_xml_data_files(msdn_data_dir)
File "C:/bin/flare\IDB_MSDN_Annotator_init_.py", line 484, in parse_xml_data_files
additional_functions = xml_parser.parse(xml_file)
File "C:/bin/flare\IDB_MSDN_Annotator\xml_parser.py", line 283, in parse
parser.parse(xmlfile)
File "C:\bin\Python27\Lib\xml\sax\expatreader.py", line 110, in parse
xmlreader.IncrementalParser.parse(self, source)
File "C:\bin\Python27\Lib\xml\sax\xmlreader.py", line 123, in parse
self.feed(buffer)
File "C:\bin\Python27\Lib\xml\sax\expatreader.py", line 217, in feed
self._err_handler.fatalError(exc)
File "C:\bin\Python27\Lib\xml\sax\handler.py", line 38, in fatalError
raise exception
xml.sax._exceptions.SAXParseException: C:/bin/flare\annotate_IDB_MSDN.py:1:2: not well-formed (invalid token)
Failed while executing plugin_t.run():
Traceback (most recent call last):
File "C:/Program Files/IDA 7.0/plugins/stackstrings_plugin.py", line 67, in run
flare.stackstrings.main()
File "C:\Python27\Lib\flare\stackstrings.py", line 276, in main
vw = jayutils.loadWorkspace(filePath)
File "C:\Python27\Lib\flare\jayutils.py", line 341, in loadWorkspace
vw.loadFromFile(filename)
File "C:\Python27\Lib\vivisect_init_.py", line 2265, in loadFromFile
fname = mod.parseFile(self, filename, baseaddr=baseaddr)
File "C:\Python27\Lib\vivisect\parsers\blob.py", line 45, in parseFile
raise Exception('Blob loader requires arch option (-O viv.parsers.blob.arch="")')
Exception: Blob loader requires arch option (-O viv.parsers.blob.arch="
Via Minh Triet Pham Tran
Hello, I think for the flare-ida scripts, you should have a check for the
version of IDA (>=6.5 or <=6.4) because idaapi.require is introduced from
IDA >6.4 as mentioned in this article:
http://www.hexblog.com/?p=749
The scripts including idaapi.require:
https://github.com/fireeye/flare-ida/blob/master/python/flare/annotate_IDB_MSDN.py
https://github.com/fireeye/flare-ida/blob/master/plugins/shellcode_hashes_search_plugin.py
https://github.com/fireeye/flare-ida/blob/master/plugins/stackstrings_plugin.py
https://github.com/fireeye/flare-ida/blob/master/plugins/apply_callee_type_plugin.py
Hi William, I am TQN, the old author of idb2sig plugin.
I need to update my plugin, so I used your script to test the result and compared the resulting .pat
files with IDA flair tool.
After some work, I found some bugs in the old plugin code.
So I have fix those bugs in your code. But it still have a small bug:
find_ref_loc
will return BADADDR
for the following sequence:
48 8D 05 29 06 00 00 lea rax, `vtable for'loader_failure_t+10h
with:
But in byte pattern is 0x0629 (29 06) because of +10h
Hope you have a time and fix it.
Thanks and best regards.
My code:
https://github.com/HongThatCong/idb2sig
Manually executing shellcode_hash_search on IDA7.5 and Python3 shows some logic problems:
In promptForSearchTypes()
if using_ida7api:
if idaapi.ASKBTN_YES == idaapi.ask_yn(idaapi.ASKBTN_YES, str('Search for DWORD array of hashes?')):
self.params.searchDwordArray = True
if idaapi.ASKBTN_YES == idaapi.ask_yn(idaapi.ASKBTN_YES, str('Search for DWORD array of hashes?')):
self.params.searchDwordArray = True
else:
if idc.AskYN(1, str('Search for push argument hash value?')) == 1:
self.params.searchPushArgs = True
if idc.AskYN(1, str('Search for DWORD array of hashes?')) == 1:
self.params.searchDwordArray = True
I presume the first call should ask for push arguments and set searchPushArgs accordingly.
def promptForHashTypes()
hashTypes = self.dbstore.getAllHashTypes()
if len(self.params.hashTypes) == 0:
raise RuntimeError('No hashes selected')
# we used to prompt y/n for each one. too obnoxious, just force all hashes
self.params.hashTypes = hashTypes
This code will always raise an exception as self.params.hashTypes is always 0 in this case
Will do a fork and fix the code and set a pull request if you'd like
Note to self for later...
python/flare/objc2_xrefs_helper.py
can replace selectors erroneously.
Example: 4fe4b9560e99e33dabca553e2eeee510
(OSX XAgent), virtual addresses 0x100014F61
and 0x100014FA5
ID for each is an NSThread object held in r12
and later r13
, and SEL is start
, but objc2_xrefs_helper.py
replaces this with __Keylogger_start
(-[Keylogger start]
). Here is the first case:
100014F58 call cs:_objc_msgSend_ptr ; [NSThread initWithTarget:selector:object]
100014F58 ; Creates NSThread targeting [BootXLoader postThread]
100014F5E mov r12, rax
100014F61 mov rsi, qword ptr cs:__Keylogger_start_ ; Keylogger - (void)start
100014F68 mov rdi, r12
100014F6B call cs:_objc_msgSend_ptr ; [r12 start]
Perhaps objc2_xrefs_helper.py
doesn't realize this selector is only unique within this binary but also pertains to objects whose interfaces are not implemented in this binary?
I was using an earlier commit(Jan. 2016) of idb2pat.py in IDA 6.8. However I came across an error as following:
"idb2pat.py:'NoneType' object has no attribute 'rfind'
Traceback(most recent call last)
File D IDA 6.8\python idaapi py line 601, in IDAPython ExecScript execfile (script, g)
File D: /IDA 6.8/plugins/idb2pat py line 478, in <module>
main()
File D: /DA 6.8/plugins/idb2pat py, line 449, in main
update config(c)
File D: /Da 6.8/plugins/idb2pat py, line 426, in update config
name, extension =oS path.splitext(get input file path())
File "C: \Python2/(Lib\ntpath py line 190, in spliter return genericpath. splitext(p, sep, altsep, extsep)
File C:\Python2/Lib\genericpath py", line 91, in splitext
seplndex= p.find(sep)
AttributeError: None Type object has no attribute find"
I put idb2pat under ./plugin. Is it correct?
I get the FATAL: Bad xdigit: error
error when trying to use sigmake on a file with long lines, such as this one:
https://slexy.org/view/s2UyxIhZQR
Here's the .pat file with quite a few problem lines like that if you want to try it yourself: https://www.dropbox.com/s/ym8mfm0k037gbd4/S05_TestingGrounds-Win64-Shipping.pat?dl=0
It's a pat file generated from a compiled executable with Unreal Engine 4 for Win64 with .pdb supplied. And UE4 uses MSBuild, iirc, if that helps in any way.
I use updated idb2pat from #105 but I get the exact same issue on non-updated idb2pat on IDA 7.2 with all the default settings.
I think the problem is here: 48894C2408B808160100E8D11A2A02482BE0C78424D086000002000000488B84 FF ABF9 60247
. Fourth member with numbers - 60247
. Usually there are 4 digits in most lines but on the lines with that exact error it's always more than 4 digits for the fourth member. I don't know how to fix that, I think I'll go try a version from #98 with fixes, they might have fixed exactly this?
Removing all the lines with more than 4 digits on the fourth member does let sigmake tool finish generating .sig file.
If you run idb2pat.py on IDA version 7.5, you'll get this error:
error: Traceback (most recent call last):
File "<string>", line 478, in <module>
File "<string>", line 455, in main
File "<string>", line 416, in get_pat_file
NameError: global name 'askfile_c' is not defined
Which is caused by newer version of IDA and renaming of the askfile_c
member to ask_file
in the ida_kernwin
type, as explained in this porting guide by hex-rays.
There are many more errors like this one for many members in different types.
One possible solution I see is to have 2 versions of the file now: idb2pat.py and idb2pat_74plus.py for appropriate IDA version.
After updating the file and changing names for all the members(which changed names), I was able to succesfully generate .sig file for a simple test Hello World VC++ solution for Windows from a version with .pdb loaded for the same executable without the .pdb loaded:
#include <iostream>
void PrintHelloWorld()
{
std::cout << "Hello World!\n";
}
int main()
{
PrintHelloWorld();
}
i got the issue when run python script annotate_IDB_MSDN, please help me
Traceback (most recent call last):
File "C:/Program Files/IDAPro6.6/python/flare/annotate_IDB_MSDN.py", line 117, in on_ok_button
IDB_MSDN_Annotator.main(config)
File "C:/Program Files/IDAPro6.6/python/flare\IDB_MSDN_Annotator\__init__.py", line 523, in main
functions_map = parse_xml_data_files(msdn_data_dir)
File "C:/Program Files/IDAPro6.6/python/flare\IDB_MSDN_Annotator\__init__.py", line 486, in parse_xml_data_files
additional_functions = xml_parser.parse(xml_file)
File "C:/Program Files/IDAPro6.6/python/flare\IDB_MSDN_Annotator\xml_parser.py", line 283, in parse
parser.parse(xmlfile)
File "C:\Program Files\IDAPro6.6\lib\xml\sax\expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "C:\Program Files\IDAPro6.6\lib\xml\sax\xmlreader.py", line 123, in parse
self.feed(buffer)
File "C:\Program Files\IDAPro6.6\lib\xml\sax\expatreader.py", line 211, in feed
self._err_handler.fatalError(exc)
File "C:\Program Files\IDAPro6.6\lib\xml\sax\handler.py", line 38, in fatalError
raise exception
xml.sax._exceptions.SAXParseException: C:\Program Files\IDAPro6.6\python\flare\annotate_IDB_MSDN.py:1:2: not well-formed (invalid token)
Thank you very much!
Which files do I need in order to let "MSDN Annotations" run standalone without the other plugins?
I cannot use your script to generate the xml file (MSDN data) successfully because "Windows Software
Development Kit (SDK) for Windows 7 and .NET Framework 3.5 Service Pack 1" was unavailable in Microsoft website years ago. So I straightly used the xml file you provided in
https://github.com/mr-tz/flare-ida/blob/master/MSDN_data/msdn_data.zip
But no constant in this xml file has value. So when I execute the script in IDA, no API constant will rename to macro.
This is a part of the xml file:
<constants>
<constant>
<name>BFT_NTDS_DATABASE</name>
<description>The file is an NTDS database file. This file should be copied to the file identified as BFT_NTDS_DATABASE when the data is restored.</description>
</constant>
<constant>
<name>BFT_LOG</name>
<description>The file is a log file. All log files are copied to the directory identified as BFT_LOG_DIR when the data is restored.</description>
</constant>
<constant>
<name>BFT_PATCH_FILE</name>
<description>The file is a patch file. All patch files are copied to the directory identified as BFT_CHECKPOINT_DIR when the data is restored.</description>
</constant>
</constants>
As you can see, all constants only have <name>
and <description>
. I can't see <value>
in all <constant>
.
Hi FLARE Team,
ApplyCalleeType: Exception caught: print_tinfo expected 8 arguments, got 7
Traceback (most recent call last):
File "C:\Tools\IDA\python\flare\apply_callee_type.py", line 267, in run
typename = idaapi.print_tinfo('', '', 0, 0, idaapi.PRTYPE_1LINE, tinfo, '', '')
File "C:\Tools\IDA\python\idaapi.py", line 36101, in print_tinfo
return _idaapi.print_tinfo(*args)
TypeError: print_tinfo expected 8 arguments, got 7
After looking at idaapi.py
def print_tinfo(_args):
"""
print_tinfo(result, prefix, indent, cmtindent, flags, tif, name,
cmt) -> bool
"""
return _idaapi.print_tinfo(_args)
Thanks.
i got the issue of MSDN Crawler, anyone has this issue when execute MSDN Crawler?
C:\Program Files\IDA 6.4>python msdn_crawler.py "C:\\Program Files\\Microsoft SDKs\\Windows\\v7.0\\Help\\1033" "C:\\Program Files\\IDA 6.4\\tilib.exe" "C:\\Program Files\\IDA 6.4\\til\\pc"
MSDN crawler based on zynamics msdn-crawler - Copyright 2010
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\bc31.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\bc31.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\bc5dos.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\bc5dos.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\bc5w16.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\bc5w16.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\bcb4win.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\bcb4win.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\bcb5win.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\bcb5win.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\geos.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\geos.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\gnuwin.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\gnuwin.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\ms16dos.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\ms16dos.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\ms16win.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\ms16win.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mscor.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mscor.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk_2000.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk_2000.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk_nt.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk_nt.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk_vista.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk_vista.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk_win7.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk_win7.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\mssdk_ws03.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\mssdk_ws03.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\nlm.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\nlm.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\ntapi.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\ntapi.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\ntddk.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\ntddk.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\ntddk64.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\ntddk64.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\vc10.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\vc10.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\vc10_64.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\vc10_64.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\vc6win.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\vc6win.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\vc8amd64.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\vc8amd64.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\vc9.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\vc9.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\w16dos.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\w16dos.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\w16os2.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\w16os2.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\w32dos.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\w32dos.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\w32os2.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\w32os2.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\wdm.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\wdm.til']' returned non-zero exit status 1
WARNING:til_extractor:Error calling tilib.exe with C:\Program Files\IDA 6.4\til\pc\wnet.til -- Command '['C:\\Program Files\\IDA 6.4\\tilib.exe', '-l', 'C:\\Program Files\\IDA 6.4\\til\\pc\\wnet.til']' returned non-zero exit status 1
Traceback (most recent call last):
File "msdn_crawler.py", line 413, in <module>
main()
File "msdn_crawler.py", line 398, in main
(file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
File "msdn_crawler.py", line 357, in parse_files
const_enum = extract_til_constant_info.main(tilib_exe, til_dir)
File "C:\Program Files\IDA 6.4\extract_til_constant_info.py", line 95, in main
for enum_name, enum in enums.iteritems():
UnboundLocalError: local variable 'enums' referenced before assignment
C:\Program Files\IDA 6.4>
Attempting to use shellcode hash search for 64-bit IDBs may elicit the following error (hand-typed, HOPEFULLY free of transcription errors):
Traceback (most recent call last):
File "C:/Program Files/IDA 7.1/python/flare/shellcode_hash_search.py", line 297, in lookForOpArgs
hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
File "C:/Program Files/IDA 7.1/python/flare/shellcode_hash_search.py", line 184, in getSymbolByTypeHash
cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER
Commonly, 64-bit shellcode uses 32-bit hash algorithms with 32-bit registers, so one workaround might be to check that the upper 32 bits are null or all ones (sign-extended negative quantity) and truncate (respecting sign extension) before querying (or raise an exception if the upper 32 bits contain numeric information of any significance. It might be better to promote the db schema and existing hashes to 64-bit quantities and make this work seamlessly for 32- and 64-bit IDBs if possible.
Hi, I read your blog post and can't find the script.
Where it is ?
Thanks
Hello,
I have tried generating the the XML database file using the msdn_crawler.py on a Linux machine (Ubuntu 16.04 x64) and it seems to have failed with the following errors (for both tilib and tilib64)
python ./MSDN_crawler/msdn_crawler.py "/home/<username>/extracted1033/" "/opt/bin/ida-6.95/tilib" "/opt/bin/ida-6.95/til/pc"
MSDN crawler based on zynamics msdn-crawler - Copyright 2010
WARNING:til_extractor:Error calling tilib.exe with /opt/bin/ida-6.95/til/pc/nlm.til -- Command '['/opt/bin/ida-6.95/tilib', '-l', '/opt/bin/ida-6.95/til/pc/nlm.til']' returned non-zero exit status 126
...SNIP...
WARNING:til_extractor:Error calling tilib.exe with /opt/bin/ida-6.95/til/pc/ntapi.til -- Command '['/opt/bin/ida-6.95/tilib', '-l', '/opt/bin/ida-6.95/til/pc/ntapi.til']' returned non-zero exit status 126
Traceback (most recent call last):
File "./MSDN_crawler/msdn_crawler.py", line 413, in <module>
main()
File "./MSDN_crawler/msdn_crawler.py", line 398, in main
(file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
File "./MSDN_crawler/msdn_crawler.py", line 357, in parse_files
const_enum = extract_til_constant_info.main(tilib_exe, til_dir)
File "/opt/bin/flare-ida-master/MSDN_crawler/extract_til_constant_info.py", line 95, in main
for enum_name, enum in enums.iteritems():
UnboundLocalError: local variable 'enums' referenced before assignment
I have also tried with the Windows version of IDA and failed like others have pointed out in other issue threads.
MSDN crawler based on zynamics msdn-crawler - Copyright 2010
Traceback (most recent call last):
File "MSDN_crawler\msdn_crawler.py", line 413, in <module>
main()
File "MSDN_crawler\msdn_crawler.py", line 398, in main
(file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
File "MSDN_crawler\msdn_crawler.py", line 371, in parse_files
result = parse_file(os.path.join(root, file), const_enum)
File "MSDN_crawler\msdn_crawler.py", line 276, in parse_file
return parse_new_style(file, content, const_enum)
File "MSDN_crawler\msdn_crawler.py", line 183, in parse_new_style
parsed_html.find_all(width='60%')]
TypeError: 'NoneType' object is not callable
I tried obtaining the modified version of the crawler and already generated database but it's no longer there.
Any instructions on how to install on OSX? I tried following the general directions for the windows installation, but cant import either the plugins or vivisect, nor run any of them.
This is on ida 7.0
Thanks!
There are some changes in IDA 7.*. For example:
"ADRP Rx, label@PAGE; ADD Rx, Rx, label@PAGEOFF" is replaced by "ADRL Rx, label"
without idaapi.auto_wait(), python script cannot get Xref.
Please update objc2_analyzer.py to support IDA 7.*. Thank you.
In the meantime I have included a
try: except: block
aroundif file.endswith('htm'): file_counter += 1 try: result = parse_file(os.path.join(root, file), const_enum) except: error_files.append(file) if result: results.append(result) print 'ERROR processing %d files' % len(error_files)
Which resulted in:
ERROR processing 21828 files Parsed 329993 files Extracted information about 15263 functions
Does this correspond with your numbers?
I'd also like to create flirt libraries for Go code (64 bit) - however, in first tests, only around 30% matched. I'm not sure where the problem exactly is, but it could be here:
...
if isCode(getFlags(ea)):
for i in zrange(ea, max(ea, 1 + get_item_end(ea) - config.pointer_size)):
if get_long(i) == ref:
return i
That code should find outgoing data or code references so they can be excluded from CRC32 checksums and signature and assumes these to be 32- or 64-bit values depending or architecture, not really sure about it... but it seems to fail with 64-bit instructions like
48 8B 05 F6 FF A3 00 mov rax, cs:main_something
as this is a 32-bit relative offset (0xa3fff6) in 64 bit code. Actually I think all references in 64 bit code is 32 bit relative... In these cases, nothing is put into variable_bytes and the offset is added to crc32, which it probably should not - if I understand the code correctly.
Another small thing I find odd is
# this will be either " :%04d %s" or " :%08d %s"
public_format = " :%%0%dX %%s" % (config.pointer_size)
Shouldn't this be config.pointer_size*2
as each character only covers one nibble, so each byte requires 2 characters? Or is this a specific feature of Ida that these values only cover half the bitsize?
Create an enum & apply it for all identified hash values.
debugging output from Visual Studio (paths shorten):
The script was first invoked from the commadline as:
python.exe msdn_crawler.py C:\TEMP\msdn D:\devapps\IDA\util\idasdk70\bin\tilib70\win\tilib64.exe D:\devapps\IDA\IDA.7.0\til\pc
where C:\TEMP\msdn is expanded MSDN docs from Visual Studio 2017
2 questions:
This line fails because the current parse_decl2 has one less argument now:
https://github.com/fireeye/flare-ida/blob/master/python/flare/apply_callee_type.py#L67
Taking out "blah" still throws an error, and I'm not sure why.
I put plugins in the "IDA 7.1/plugin/" directory, and python files in "IDA 7.1/python" dir; installed dependency vivisect
i'm getting this now:
Failed while executing plugin_t.run():
Traceback (most recent call last):
File "C:/Program Files/IDA 7.1/plugins/stackstrings_plugin.py", line 57, in run
flare.stackstrings.main()
NameError: global name 'flare' is not defined
Any help, much obliged.
Hello, I'm trying to use your script on a MIPSL binary and I'm getting this error.
ERROR:Python int too large to convert to C long
Looks like the zrange implementation doesn't consider the case when (func.endEA - func.startEA) < 32. For REALLY small functions, this seems to return a really long iterator.
Try it zrange(32, 8). Script gets stuck at line 265.
Hello,
Setup:
When running the StructTyper plugin, I receive the following error:
Failed while executing plugin_t.run():
Traceback (most recent call last):
File "C:/Program Files/IDA 7.0/plugins/struct_typer_plugin.py", line 47, in run
idaapi.require('flare.struct_typer')
File "C:\Program Files\IDA 7.0\python\ida_idaapi.py", line 153, in require
m = importlib.import_module(modulename, package)
File "C:\python27-x64\Lib\importlib\__init__.py", line 37, in import_module
__import__(name)
File "C:\Program Files\IDA 7.0\python\flare\struct_typer.py", line 48, in <module>
g_dll = ctypes.windll[idaname + ".wll"]
File "C:\python27-x64\Lib\ctypes\__init__.py", line 437, in __getitem__
return getattr(self, name)
File "C:\python27-x64\Lib\ctypes\__init__.py", line 432, in __getattr__
dll = self._dlltype(name)
File "C:\python27-x64\Lib\ctypes\__init__.py", line 362, in __init__
self._handle = _dlopen(self._name, mode)
WindowsError: (126, 'The specified module could not be found')
The key error seems to be in:
g_dll = ctypes.windll[idaname + ".wll"]
Version 7.0 looks to have dropped the .will files (or I missed something on the installation). I tried changing it to .DLL and now get the following:
Failed while executing plugin_t.run():
Traceback (most recent call last):
File "C:/Program Files/IDA 7.0/plugins/struct_typer_plugin.py", line 47, in run
idaapi.require('flare.struct_typer')
File "C:\Program Files\IDA 7.0\python\ida_idaapi.py", line 153, in require
m = importlib.import_module(modulename, package)
File "C:\python27-x64\Lib\importlib\__init__.py", line 37, in import_module
__import__(name)
File "C:\Program Files\IDA 7.0\python\flare\struct_typer.py", line 70, in <module>
print_type_to_one_line = g_dll.print_type_to_one_line
File "C:\python27-x64\Lib\ctypes\__init__.py", line 375, in __getattr__
func = self.__getitem__(name)
File "C:\python27-x64\Lib\ctypes\__init__.py", line 380, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'print_type_to_one_line' not found
Hi,
I have a weird error inside IDA-Pro. I've installed PySide with pip, as recommended in the official documentation.
The following command runs fine in a terminal:
C:\>python
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from PySide import QtGui
But inside IDA Pro (Windows > Output Window), it fails:
Python>from PySide import QtGui
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: DLL load failed: La proc?dure sp?cifi?e est introuvable.
Though, I'm able to import PySide (still in IDA Pro):
Python>import PySide
I've created following environment variables:
PATH:
[REMOVED];C:\Python27;C:\Python27\scripts;C:\Python27\Lib\site-packages\PySide
PYTHONPATH:
C:\Python27\Lib;C:\Python27\DLLs;C:\Python27\Lib\lib-tk;c:\Python27\Lib\flare;c:\Python27\Lib\vivisect;C:\Python27\Lib\site-packages\PySide
Any idea what I'm missing?
Many thanks in advance for your help.
so I'm using ironstring to print the stack string of a sample, but the problem is the string shown in the summary don't have an address related to them, so i can't find them where in the binary they are easily, and there is no table being printed like shown in the readme, just a load of "DEBUG:root..." stuff and no table, what's the problem?
checked the flare emu python files as well and the default verbose is set to 0 so not sure what these DEBUG:root stuff is all about
here's the end of output :
...
DEBUG:root:target 0040B94C hit
DEBUG:root:we missed our target! bailing out of this function..
DEBUG:root:unmapped 0045D000 to 0045EFFF
DEBUG:root:mapping 00002000 bytes @0045D000
DEBUG:root:running on 0x43C3EE
DEBUG:root:getting paths to 0040E539, 1 of 1 targets
DEBUG:root:run #1, 1 targets remaining: 0040E539 (2 paths)
DEBUG:root:emulating path #1 of 2 from 0043C3EE to 0040E539 via basic blocks: [0, 1, 2, 3]
DEBUG:root:0040E515 is outside of block #0 (0043C3EE -> 0043C3F3), forcing PC to 0040E515
DEBUG:root:0040E538 is outside of block #1 (0040E515 -> 0040E51E), forcing PC to 0040E520
DEBUG:root:extracting stackstrings at checkpoint: 0x40E522, stacksize: 0x4
DEBUG:root:extracting global stackstrings at checkpoint: 0x40E522
DEBUG:root:extracting stackstrings at checkpoint: 0x40E52A, stacksize: 0xC
DEBUG:root:extracting global stackstrings at checkpoint: 0x40E52A
DEBUG:root:0040E538 is outside of block #2 (0040E520 -> 0040E537), forcing PC to 0040E538
DEBUG:root:target 0040E539 hit
DEBUG:root:we missed our target! bailing out of this function..
DEBUG:root:emulating path #2 of 2 from 0043C3EE to 0040E539 via basic blocks: [0, 1, 3]
DEBUG:root:0040E515 is outside of block #0 (0043C3EE -> 0043C3F3), forcing PC to 0040E515
DEBUG:root:0040E538 is outside of block #1 (0040E515 -> 0040E51E), forcing PC to 0040E538
DEBUG:root:target 0040E539 hit
DEBUG:root:we missed our target! bailing out of this function..
DEBUG:root:unmapped 0045D000 to 0045EFFF
DEBUG:root:mapping 00002000 bytes @0045D000
DEBUG:root:running on 0x43C3F8
DEBUG:root:getting paths to 0043C400, 1 of 2 targets
DEBUG:root:getting paths to 0043C406, 2 of 2 targets
DEBUG:root:run #1, 2 targets remaining: 0043C406 (1 paths)
DEBUG:root:emulating path #1 of 1 from 0043C3F8 to 0043C406 via basic blocks: [0, 1]
DEBUG:root:target 0043C400 found on the way to 0043C406
DEBUG:root:0043C406 is outside of block #0 (0043C3F8 -> 0043C400), forcing PC to 0043C406
DEBUG:root:target 0043C406 hit
DEBUG:root:run #2, 2 targets remaining: 0043C400 (1 paths)
DEBUG:root:emulating path #1 of 1 from 0043C3F8 to 0043C400 via basic blocks: [0]
DEBUG:root:target 0043C400 hit
DEBUG:root:unmapped 0045D000 to 0045EFFF
DEBUG:root:mapping 00002000 bytes @0045D000
ironstrings summary
------------------------
Ran successfully on 1296/1298 functions
Found 15 stackstrings
Commented 15 stackstrings
Encountered 2 errors
- Error analyzing function 0x410442: Invalid memory read (UC_ERR_READ_UNMAPPED)
- Error analyzing function 0x412C3E: Invalid memory read (UC_ERR_READ_UNMAPPED)
Recovered 10 unique stackstrings
------------------------
strings..
...
Finished ironstrings stackstring deobfuscation after 99.51 seconds
Error
I get the error below when trying to use StackStrings on a shellcode blob, however it does work (excellently!) when I use it on normal PE files.
Error:
Failed while executing plugin_t.run():
Traceback (most recent call last):
File "C:/Program Files/IDA 7.1/plugins/stackstrings_plugin.py", line 60, in run
flare.stackstrings.main()
File "C:/Program Files/IDA 7.1/plugins\flare\stackstrings.py", line 250, in main
vw = jayutils.loadWorkspace(filePath)
File "C:/Program Files/IDA 7.1/plugins\flare\jayutils.py", line 247, in loadWorkspace
vw.loadFromFile(filename)
File "C:\python27-x64\lib\site-packages\vivisect\__init__.py", line 2140, in loadFromFile
fname = mod.parseFile(self, filename)
File "C:\python27-x64\lib\site-packages\vivisect\parsers\blob.py", line 44, in parseFile
raise Exception('Blob loader *requires* arch option (-O viv.parsers.blob.arch="<archname>")')
Exception: Blob loader *requires* arch option (-O viv.parsers.blob.arch="<archname>")
Cause
I believe the error is because in blob.py parseFile()
tries to work out the architecture based on magic bytes which obviously aren't there, since the architecture is needed for emulation it throws an exception.
Solution
In jaysutils.py when loading a Vivisect workspace if the IDA loader is a Binary one then either ask the user the architecture or base it on info from the IDA API, code in https://reverseengineering.stackexchange.com/a/11398 gives you that. Then explicitly tell Vivisect the architecture.
I seem to have hit an issue with the parsing of the files, tried this on 6.6 on x64, and 6.5 on x86.
C:\Users\luser\Desktop\IDA stuff\flare-ida\MSDN_crawler [master]> python .\msdn_
crawler.py 'C:\\sdk_help\\' 'C:\\Program Files\\IDA 6.5\\tilib.exe' 'C:\\Program
Files\\IDA 6.5\\til\\pc'
MSDN crawler based on zynamics msdn-crawler - Copyright 2010
Traceback (most recent call last):
File ".\msdn_crawler.py", line 413, in <module>
main()
File ".\msdn_crawler.py", line 398, in main
(file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
File ".\msdn_crawler.py", line 371, in parse_files
result = parse_file(os.path.join(root, file), const_enum)
File ".\msdn_crawler.py", line 276, in parse_file
return parse_new_style(file, content, const_enum)
File ".\msdn_crawler.py", line 183, in parse_new_style
parsed_html.find_all(width='60%')]
TypeError: 'NoneType' object is not callable
(This happens after a few minutes of processing). Upon running the same command -v, I get:
Lots of this "Could not retrieve function description...", which I figure it's okay since not all files will be relevant to the script.
[...]
DEBUG:__main__:Error: Could not retrieve function description from file C:\\sdk_
help\\abff2e90-4c42-4c07-816f-efde05343e03.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abff323b-e6c6-45e0-93bd-eeb68bca80e0.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abff3c41-301f-4273-9103-8e6197ba41fe.htm
Traceback (most recent call last):
File "c:\python27\lib\logging\__init__.py", line 842, in emit
msg = self.format(record)
File "c:\python27\lib\logging\__init__.py", line 719, in format
return fmt.format(record)
File "c:\python27\lib\logging\__init__.py", line 464, in format
record.message = record.getMessage()
File "c:\python27\lib\logging\__init__.py", line 328, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Logged from file msdn_crawler.py, line 118
DEBUG:__main__:Error: Could not retrieve function description from file C:\\sdk_
help\\abff3c41-301f-4273-9103-8e6197ba41fe.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abffd0fe-d047-4670-a728-eea8253f3f2d.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_activate.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_getautohidebar.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_getstate.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_gettaskbarpos.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_new.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_querypos.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_remove.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_setautohidebar.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_setpos.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_setstate.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abm_windowposchanged.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abnormaltermination.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abn_fullscreenapp.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abn_poschanged.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abn_statechange.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abn_windowarrange.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abort.htm
DEBUG:__main__:Error: Could not retrieve function description from file C:\\sdk_
help\\abort.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abortall.htm
DEBUG:__main__:Error: Could not retrieve function description from file C:\\sdk_
help\\abortall.htm
DEBUG:__main__:Parsing C:\\sdk_help\\abortdoc.htm
Traceback (most recent call last):
File ".\msdn_crawler.py", line 413, in <module>
main()
File ".\msdn_crawler.py", line 398, in main
(file_counter, results) = parse_files(msdn_directory, tilib_exe, til_dir)
File ".\msdn_crawler.py", line 371, in parse_files
result = parse_file(os.path.join(root, file), const_enum)
File ".\msdn_crawler.py", line 276, in parse_file
return parse_new_style(file, content, const_enum)
File ".\msdn_crawler.py", line 183, in parse_new_style
parsed_html.find_all(width='60%')]
TypeError: 'NoneType' object is not callable
Just for some 4 eyeing:
I am running windows version of IDA (tried both 64 and 32 bit hosts).
Python is always 32 bit.
I pip-installed "beautifulsoup" (not beautifulsoup4)
I decompressed all the HxS help files to a flat directory, i.e. all htm files in the same directory.
ps. as a sidenote I get tilib.exe errors on different files, on clean installs, on pretty much every version from 6.1 to 6.6 except for 6.5, (tilib.exe version matching that of the IDA distribution) anyone else experiencing this?
When I run the script I get the following error:
_TOKEN_INFORMATION_CLASS: failed to add constant TokenUser=1 (0x1)
_TOKEN_INFORMATION_CLASS: failed to add constant TokenGroups=2 (0x2)
_TOKEN_INFORMATION_CLASS: failed to add constant TokenPrivileges=3 (0x3)
_TOKEN_INFORMATION_CLASS: failed to add constant TokenOwner=4 (0x4)
Feature to ignore all not-yet-identified functions, named sub_xxxx
in idb2pat
Pretty sure it's not me doing something wrong(and creating all those issues I mean). Works on a simple VC++ Hello World project as expected and explained in the articles(One, Two):
On the other hand, in a big UE4 project 0 functions get applied in IDA from .sig file after generating the .sig file with sigmake from idb2pat:
I generate the .sig file using command sigmake -lrsub_ "S05_TestingGrounds-Win64-Shipping - No Xdigit errors(deleted lines with errors).pat" "S05_TestingGrounds-Win64-Shipping - No Xdigit errors(deleted lines with errors).sig"
. -lrsub_
parameter is to exclude functions that have sub_
in them.
After generating the .sig file and then trying to apply the .sig file, that's what I get: 0 functions applied.
Here's the link with the project, pat and sig file, so you can try to generate and apply this sig file onto executable yourself:
https://www.dropbox.com/h?preview=TestingGrounds_DebugSymbols.zip
This is a Unreal Engine 4.26 C++ project created from FPS template which I packaged in UE4 with debug symbols.
The executable, pat and .sig files are in \WindowsNoEditor\S05_TestingGrounds\Binaries\Win64\
folder. The original pat file with xdigit problem is called S05_TestingGrounds-Win64-Shipping - Original.pat
. The pat file with problematic lines deleted which cause xdigit problem is called S05_TestingGrounds-Win64-Shipping - No Xdigit errors(deleted lines with errors).pat
.
Problem for 0 functions applied is somewhere between lines 30000 and 35000 in the .pat file because if I delete all lines after line 30000, it generates valid .sig file and then applies this sig file appropriately on the executable:
After I delete lines 30000 - 35000 in the pat file and then delete all the lines after 50000, it generates valid .sig files and applies the sig file appropriately on the executable:
As you can see on the screenshots, no meaningful logs is generated after applying new FLIRT signature in IDA, just Plan FLIRT signature: Unnamed sample library
in the case of success and failure to apply any function signatures.
I'm running into a few errors when I run sigmake on the generated pat file.
The most common one of which is "FATAL: Name is too long.", but also once in a while, "FATAL: Bad xdigit:"
My current workaround for both errors is just deleting the specified lines from the pat file.
(idb2pat was ran on IDA 7)
Sometimes malware uses for example ror-13 to check process and file names. I suggest to eiter include popular process/file names to existing database with decoy library names "processes" "filesystem" or make a new database located at the same folder.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.