ohjeongwook / binkit Goto Github PK

View Code? Open in Web Editor NEW

73.0 73.0 9.0 23.09 MB

Binary Reverse Engineering Data Science Kit

C++ 2.01% C 97.32% Python 0.67%

binkit's People

Contributors

Stargazers

Watchers

Forkers

crackercat killvxk fengjixuchui fdlucifer zzz9328fb rmallof hongthatcong wwildey

binkit's Issues

Make va_t conformant to the platform

[ ] TODO: Make va_t conformant to the platform

typedef int va_t; -> address length

Find ways to remove match conflicts

Rename Binkit plugin name to Binkit Exporter

Add fuzzy hash

test_dump_basic_blocks - ValueError: Circular reference detected

I:\ReverseEngineering\Tools\Vulnerabilities\DarunGrim\Src\binkit\tests>python tests.py  1>tests.log
FEE.FF.
======================================================================
ERROR: test_dump_basic_blocks (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 89, in test_dump_basic_blocks
    json.dump(basic_block_data_list, fd, indent = 4)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\json\__init__.py", line 179, in dump
    for chunk in iterable:
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\json\encoder.py", line 429, in _iterencode
    yield from _iterencode_list(o, _current_indent_level)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\json\encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\json\encoder.py", line 284, in _iterencode_list
    raise ValueError("Circular reference detected")
ValueError: Circular reference detected

DiffAlgorithms::GetMatchRateInfoArray

MatchRateInfo *DiffAlgorithms::GetMatchRateInfoArray(va_t source_address, va_t target_address, int type, int& match_rate_info_count)

Broken basic block


.text:0045BC65
.text:0045BC65 ; Attributes: bp-based frame
.text:0045BC65
.text:0045BC65 ; void __stdcall jpeg_idct_islow_x86(jpeg_decompress_struct *cinfo, jpeg_component_info *compptr, __int16 *coef_block, char **output_buf, unsigned int output_col)
.text:0045BC65 _jpeg_idct_islow_x86@20 proc near
.text:0045BC65
.text:0045BC65 workspace= dword ptr -120h
.text:0045BC65 cinfo= dword ptr  8
.text:0045BC65 compptr= dword ptr  0Ch
.text:0045BC65 coef_block= dword ptr  10h
.text:0045BC65 output_buf= dword ptr  14h
.text:0045BC65 output_col= dword ptr  18h
.text:0045BC65
.text:0045BC65 push    ebp
.text:0045BC66 mov     ebp, esp
.text:0045BC68 mov     eax, [ebp+cinfo]
.text:0045BC6B mov     eax, [eax+148h]
.text:0045BC71 sub     esp, 120h
.text:0045BC77 add     eax, 80h ; '€'

0045BC65 = 4570213

Improve get_match_data_combinations algorithm

6c833dd1 - 44402c (size: 70)

1820540369 - 4472876

Make match for the instruction hash collision within function

Multiple matches

Within same function, there are multiple instruction hash matches.

.text:0040C78A ; void __thiscall GPath::AddLineTo(GPath *this, float x, float y)
.text:0040C78A ?AddLineTo@GPath@@QAEXMM@Z proc near

0040C78A == 4245386 = RVA: 51082

Too short code block

1820313516 (0x6C7FC7AC) - 4245424, 4245504, 4245509

            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820313516,
                "target": 4245424,
                "match_rate": 100
            },
            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820313516,
                "target": 4245504,
                "match_rate": 100
            },
            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820313516,
                "target": 4245509,
                "match_rate": 100
            },

1820313516 (6C7FC7AC)

"source": 1820313516, 6C7FC7AC

    .text:6C7FC7AC                 cmp     edx, 1
    .text:6C7FC7AF                 jnz     loc_6C7FC85D

4245424 (40C7B0)

"target": 4245424 (40C7B0)

    .text:0040C7B0                 cmp     edx, 1
    .text:0040C7B3                 jnz     loc_40C861

4245504 (40C800)

.text:0040C800 cmp     eax, 2
.text:0040C803 jz      short loc_40C833

4245509 (40C805)

.text:0040C805 cmp     eax, 5
.text:0040C808 jnz     short loc_40C865

Show list table with matches/unidentified/match rates for functions

test_function_match fails

I:\ReverseEngineering\Tools\Vulnerabilities\DarunGrim\Src\binkit\tests>copy /y ..\x64\Debug\*.pyd .
..\x64\Debug\pybinkit.pyd
The process cannot access the file because it is being used by another process.
        0 file(s) copied.

I:\ReverseEngineering\Tools\Vulnerabilities\DarunGrim\Src\binkit\tests>python tests.py TestCase.test_function_match  1>test_function_match.log
F
======================================================================
FAIL: test_function_match (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 373, in test_function_match
    function_matches = self.do_instruction_hash_match()
  File "tests.py", line 369, in do_instruction_hash_match
    self.assertTrue(self.util.compare_function_matches(expected_matches, matches))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 2.850s

FAILED (failures=1)

Use profile cache file to store local IDA sessions

test_do_instruction_hash_match_in_functions - AssertionError

======================================================================
FAIL: test_do_instruction_hash_match_in_functions (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 218, in test_do_instruction_hash_match_in_functions
    self.assertEqual(expected_instruction_matches, instruction_matches)
AssertionError: Lists differ: [{'so[71 chars]20562989, 'target': 4495475, 'match_rate': 100[679 chars]100}] != [{'so[71 chars]20562884, 'target': 4495378, 'match_rate': 100[431 chars]100}]

First differing element 1:
{'source': 1820562989, 'target': 4495475, 'match_rate': 100}
{'source': 1820562884, 'target': 4495378, 'match_rate': 100}

First list contains 4 additional elements.
First extra element 9:
{'source': 1820562904, 'target': 4495398, 'match_rate': 100}

Diff is 845 characters long. Set self.maxDiff to None to see it.

Better assert logic for complex data structures (multi-level dict/list) in test code

test_dump_functions - TypeError: string indices must be integers

======================================================================
ERROR: test_dump_functions (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 137, in test_dump_functions
    expected_function_data['basic_block_addresses'].sort()
TypeError: string indices must be integers

Save/load function match info into/from database (ex. SQLite)

IDASessions::_Load()

Removed following code


BOOL IDASessions::_Load()
{
    ...
    const char *query = "";

    if (ShowFullMatched)
    {
        if (ShowNonMatched)
        {
            query = "SELECT SourceAddress, EndAddress, TargetAddress, BlockType, MatchRate, SourceFunctionName, Type, TargetFunctionName, MatchCountForTheSource, NoneMatchCountForTheSource, MatchCountWithModificationForTheSource, MatchCountForTheTarget, NoneMatchCountForTheTarget, MatchCountWithModificationForTheTarget From "
                FUNCTION_MATCH_INFO_TABLE
                " WHERE TheSourceFileID=%u AND TheTargetFileID=%u";
        }
        else
        {
            query = "SELECT SourceAddress, EndAddress, TargetAddress, BlockType, MatchRate, SourceFunctionName, Type, TargetFunctionName, MatchCountForTheSource, NoneMatchCountForTheSource, MatchCountWithModificationForTheSource, MatchCountForTheTarget, NoneMatchCountForTheTarget, MatchCountWithModificationForTheTarget From "
                FUNCTION_MATCH_INFO_TABLE
                " WHERE TheSourceFileID=%u AND TheTargetFileID=%u AND MatchRate != 0";
        }
    }
    else
    {
        if (ShowNonMatched)
        {
            query = "SELECT SourceAddress, EndAddress, TargetAddress, BlockType, MatchRate, SourceFunctionName, Type, TargetFunctionName, MatchCountForTheSource, NoneMatchCountForTheSource, MatchCountWithModificationForTheSource, MatchCountForTheTarget, NoneMatchCountForTheTarget, MatchCountWithModificationForTheTarget From "
                FUNCTION_MATCH_INFO_TABLE
                " WHERE TheSourceFileID=%u AND TheTargetFileID=%u AND (NoneMatchCountForTheSource != 0 OR NoneMatchCountForTheTarget != 0 OR MatchCountWithModificationForTheSource!=0 OR MatchCountWithModificationForTheTarget !=0 )";
        }
        else
        {
            query = "SELECT SourceAddress, EndAddress, TargetAddress, BlockType, MatchRate, SourceFunctionName, Type, TargetFunctionName, MatchCountForTheSource, NoneMatchCountForTheSource, MatchCountWithModificationForTheSource, MatchCountForTheTarget, NoneMatchCountForTheTarget, MatchCountWithModificationForTheTarget From "
                FUNCTION_MATCH_INFO_TABLE
                " WHERE TheSourceFileID=%u AND TheTargetFileID=%u AND (NoneMatchCountForTheSource != 0 OR NoneMatchCountForTheTarget != 0 OR MatchCountWithModificationForTheSource!=0 OR MatchCountWithModificationForTheTarget !=0 ) AND MatchRate != 0";
        }
    }

    //TODO: FunctionMatchList = m_diffStorage->QueryFunctionMatches(query, SourceID, TargetID);

Don't add import section

DoFunctionLevelMatchOptimizing

Return value filtering

Add unidentified block information to the test results

Improve diff results file format

specification
IDAPython loader -> display graph

test_do_instruction_hash_match_in_functions

I:\ReverseEngineering\Tools\Vulnerabilities\DarunGrim\Src\binkit\tests>python tests.py TestCase.test_do_instruction_hash_match_in_functions  1>test_do_instruction_hash_match_in_functions.log
E
======================================================================
ERROR: test_do_instruction_hash_match_in_functions (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 238, in test_do_instruction_hash_match_in_functions
    instruction_matches = self.do_instruction_hash_match_in_functions(0x6C83948B, 0x004496D9)
  File "tests.py", line 230, in do_instruction_hash_match_in_functions
    for match_data in diff_algorithms.do_blocks_instruction_hash_match(src_function.get_basic_blocks(), target_function.get_basic_blocks()):
AttributeError: 'NoneType' object has no attribute 'get_basic_blocks'

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)

binkit-shell: Add basic command shell interface

list: list current sessions
export: export disassembly from IDA sessions
diff: drive diff session - with specific output filename or auto-generated file name

binkit-shell: Drive export database & comparing from external python script

test_dump_basic_blocks

I:\ReverseEngineering\Tools\Vulnerabilities\DarunGrim\Src\binkit\tests>python tests.py TestCase.test_dump_basic_blocks  1>test_dump_basic_blocks.log
F
======================================================================
FAIL: test_dump_basic_blocks (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 94, in test_dump_basic_blocks
    self.assertEqual(expected_basic_block_data_list_pair, basic_block_data_list_pair)
AssertionError: Lists differ: [[{'a[213 chars]s': [{'type': 0, 'address': 1820573373}, {'typ[9606406 chars][]}]] != [[{'a[213 chars]s': [], 'parents': []}, {'address': 1820312476[8063418 chars][]}]]

First differing element 0:
[{'ad[212 chars]s': [{'type': 0, 'address': 1820573373}, {'typ[4825426 chars] []}]
[{'ad[212 chars]s': [], 'parents': []}, {'address': 1820312476[3282438 chars] []}]

Diff is 31916560 characters long. Set self.maxDiff to None to see it.

----------------------------------------------------------------------
Ran 1 test in 705.597s

FAILED (failures=1)

pMatchResults->Erase

					else
					{
						m_diffDisassemblyStorage->InsertMatchMap(SourceController->GetFileID(),
							TargetController->GetFileID(),
							*source_member_iter,
							*target_member_iter,
							FINGERPRINT_INSIDE_FUNCTION_MATCH,
							current_match_rate);
					}
                    
                    

                    pMatchResults->Erase(match_data.Addresses[0], match_data.Addresses[1]);

Add symbol hash matches

Merge binkit viewer to binkit plugin

test_function_pair_diffs - AssertionError

======================================================================
FAIL: test_function_pair_diffs (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 395, in test_function_pair_diffs
    self.do_function_pair_diff(0x6c7fc779, 0x40c78a)
  File "tests.py", line 391, in do_function_pair_diff
    self.assertEqual(expected_function_diff_list, function_diff_list)
AssertionError: Lists differ: [{'fu[75 chars]ource': 1820313663, 'target': 4245540, 'match_[4875 chars]]}]}] != [{'fu[75 chars]ource_parent': 0, 'target_parent': 0, 'source'[3199 chars]]}]}]

First differing element 0:
{'fun[74 chars]ource': 1820313663, 'target': 4245540, 'match_[1947 chars]6]}]}
{'fun[74 chars]ource_parent': 0, 'target_parent': 0, 'source'[1437 chars]0]}]}

Diff is 23556 characters long. Set self.maxDiff to None to see it.

pybinkit - compile for 2.7.x

test_dump_basic_blocks - AssertionError

======================================================================
FAIL: test_dump_basic_blocks (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 108, in test_dump_basic_blocks
    self.assertEqual(expected_basic_block_data_list, basic_block_data_list)
AssertionError: Lists differ: [[{'a[27 chars]l': 'error_exit', 'instruction_hash': [143, 1,[9606692 chars][]}]] != [[{'a[27 chars]l': '', 'instruction_hash': [143, 18, 122, 18,[8686798 chars][]}]]

First differing element 0:
[{'ad[26 chars]l': 'error_exit', 'instruction_hash': [143, 1,[4825685 chars] []}]
[{'ad[26 chars]l': '', 'instruction_hash': [143, 18, 122, 18,[4366139 chars] []}]

Diff is 54714313 characters long. Set self.maxDiff to None to see it.

----------------------------------------------------------------------
Ran 1 test in 940.372s

binkit viewer communicate with each other to synchronize disassembly view

Load Binkit Exporter using idaapi.load_plugin from Binkit Agent

Remove ControlFlows.SrcBlockEnd field from SQLite DB

Make pybinkit.Binary accept optional filename

            binary = pybinkit.Binary()
            if self.debug_level > 0:
                print('Opening %s...' % filename)
            binary.open(filename)

Remove reference to m_pdisassemblyReader from Functions.cpp

test_instruction_hash_match - AssertionError

======================================================================
FAIL: test_instruction_hash_match (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 190, in test_instruction_hash_match
    self.assertEqual(expected_match_data_list, match_data_list)
AssertionError: Lists differ: [{'source': 1820540761, 'target': 4473268, 'match_rate': [1626967 chars]0}]}] != [{'source': 1820607143, 'target': 4539756, 'match_rate': [1110499 chars]0}]}]

First differing element 0:
{'source': 1820540761, 'target': 4473268, 'match_rate': [118 chars]00}]}
{'source': 1820607143, 'target': 4539756, 'match_rate': [118 chars]00}]}

Diff is 4194703 characters long. Set self.maxDiff to None to see it.

[GUI] synchronize basic block pair view

Call UpdateFunctionAddressesInStorage only when function information is changed

Remove name redundancies in BasicBlocks, Functions class

Missing symbol from *.db file

Binkit command line working

Don't add IDA internal names (loc_, sub_)

test_dump_functions

I:\ReverseEngineering\Tools\Vulnerabilities\DarunGrim\Src\binkit\tests>copy /y ..\x64\Debug\*.pyd .
..\x64\Debug\pybinkit.pyd
The process cannot access the file because it is being used by another process.
        0 file(s) copied.

I:\ReverseEngineering\Tools\Vulnerabilities\DarunGrim\Src\binkit\tests>python tests.py TestCase.test_dump_functions  1>test_dump_functions.log
F
======================================================================
FAIL: test_dump_functions (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 158, in test_dump_functions
    self.compare_function_list(expected_function_data_list_pair[i], current_function_data_list_pair[i])
  File "tests.py", line 136, in compare_function_list
    self.assertTrue(address in current_address_to_function_data_map)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 1.067s

FAILED (failures=1)

Refactor SourceController/TargetController

Remove from DiffAlgorithms -> ClientAnalysisInfo

Place binkit viewer right next to Function window

Draw call graph from JupyterNotebook

Using various packages

test_instruction_hash_match

I:\ReverseEngineering\Tools\Vulnerabilities\DarunGrim\Src\binkit\tests>python tests.py TestCase.test_instruction_hash_match  1>test_instruction_hash_match.log
F
======================================================================
FAIL: test_instruction_hash_match (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 219, in test_instruction_hash_match
    self.assertEqual(expected_match_data_list, match_data_list)
AssertionError: Lists differ: [{'source': 1820312448, 'target': 4244352,[1110514 chars]0}]}] != []

First list contains 5009 additional elements.
First extra element 0:
{'source': 1820312448, 'target': 4244352, 'match_rate': 100, 'child_match_list': []}

Diff is 1993814 characters long. Set self.maxDiff to None to see it.

----------------------------------------------------------------------
Ran 1 test in 2.038s

FAILED (failures=1)

FunctionMatches::DoControlFlowMatch overwrites existing matches with higher mathc rates

    {
        "source": 1820314868,
        "target": 4246735,
        "matches": [
            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820314868,
                "target": 4246735,
                "type": 1,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820314909,
                "target": 4246776,
                "type": 1,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820314933,
                "target": 4246800,
                "type": 1,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 1820314933,
                "target_parent": 4246800,
                "source": 1820315070,
                "target": 4246944,
                "type": 6,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 1820314933,
                "target_parent": 4246800,
                "source": 1820315077,
                "target": 4246944,
                "type": 6,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820315082,
                "target": 4246949,
                "type": 4,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820315094,
                "target": 4246961,
                "type": 1,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 1820315082,
                "target_parent": 4246949,
                "source": 1820315103,
                "target": 4247148,
                "type": 6,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820315123,
                "target": 4246990,
                "type": 1,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 1820315123,
                "target_parent": 4246990,
                "source": 1820315241,
                "target": 4247108,
                "type": 6,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820315272,
                "target": 4247139,
                "type": 1,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 1820315272,
                "target_parent": 4247139,
                "source": 1820315281,
                "target": 4247148,
                "type": 6,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820315301,
                "target": 4247168,
                "type": 1,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 1820315301,
                "target_parent": 4247168,
                "source": 1820315419,
                "target": 4247286,
                "type": 6,
                "sub_type": 0,
                "match_rate": 100
            },
            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820315445,
                "target": 4247312,
                "type": 1,
                "sub_type": 0,
                "match_rate": 100
            }
        ]
    },

1820315094

            {
                "source_parent": 0,
                "target_parent": 0,
                "source": 1820315094,
                "target": 4246961,
                "type": 1,
                "sub_type": 0,
                "match_rate": 100
            },

            {
                "source_parent": 1820315103,
                "target_parent": 4247148,
                "source": 1820315094,
                "target": 4247139,
                "type": 6,
                "sub_type": 0,
                "match_rate": 90
            },

Add Symbol hash dump

Save addresses as RVA

Add image base meta data in SQLite Database
Save all addresses in RVA
Draw tab/graphs based upon image_base + RVA formula