mahlet-inc / hobbits Goto Github PK
View Code? Open in Web Editor NEWA multi-platform GUI for bit-based analysis, processing, and visualization
Home Page: https://mahlet-inc.github.io
License: MIT License
A multi-platform GUI for bit-based analysis, processing, and visualization
Home Page: https://mahlet-inc.github.io
License: MIT License
@gdot709 proposed using uppercase letters for bytes, e.g. t16
== T2
, s128i80
== S16I10
, and t3s32
== t3S4
I think this is easy and good.
If it's not set, could be nice to have the python interrupter for the Python Runner auto-populate if python can be found on the system path. Perhaps a drop down if multiple versions are available.
When I use the Kaitai Struct analyzer plugin on PNGs or BMPs the program closes (I presume it's crashing) with no error messages.
I'm using the pre-built binary for Windows.
As hobbits expands to support more than just digital data streams, it would be good to have short descriptions for each of the views so people new to the tool or new to using a particular view can quickly and easily know what information that view is attempting to display. These descriptions can take many forms:
Such unneeded and undisableable warnings as violations of the badly-designed style guide.
A lot of the plugins seem to misbehave in the Mac release in 0.29.0 and the operators crash the program. When built from source on a Mac, I could not replicate this problem. There is no requirement to support Mac, so this will not be prioritized (but it would be nice to not be publishing garbage).
The largest component of Linux distribution is lib/ directory, which contains 3 or 4 files of each shared library under different names, e.g *.so, *.so.5, *.so.5.6, *.so.5.6.1. Those files are identical, but each takes separate space on disk. And since most of them are too big for gzip to detect their identity, they take separate space in the archive. Replacing them with hard or symbolic links will work the same, but make the binary package almost 4 times smaller and easier to download.
The view portion of the GUI should support a high level of flexibility and usability. Some ideas:
The core library should be cleaned up to stop requiring the Qt widgets module. The functions that relate to the widgets should be broken out into a separate library (hobbits-qtils? hobbits-widgets?)
This will allow CLI things like the hobbits runner to get rid of the widgets module as well.
Now that there's some proof-of-concept stuff for embedded python from #23 , it should just be a matter of some build engineering and some API refactoring and expansion to make a hobbits python package. At first, this will mostly be useful for making tools like the runner, the tester, and the web server ( #47 ).
This includes:
the min and max input container thing seems like it might make more sense in a ParameterDelegate::validate
I'm not sure, but it's worth revisiting before 1.0 is finalized
When I have parsed a bit-container with a KS spec, and noticed a suspicious pattern within some blob, it'd be nice to view which field is that blob in a tooltip by hovering it in the view and go to that field in Extractor by ctrl+click.
My method of black-box reverse engineering of file formats almost always involves generation of similar files (i.e. with incrementally added records) and binary diffing them. I don't know any good tools for binary diffing (there are tools for binary patching, but they are different beasts, and there are tools for "binary" diffing, where "binary" means an executable format (PE/EFL/Mach-O) filled with machine code of a known arch, so CFG are matched and diffed).
So I usually do the following
hd
all the filesAn example of such a preliminary analysis is https://github.com/kaitai-io/kaitai_struct_formats/pull/532/files (LTO was initially analysed the same way).
Unfortunately hexdump misses diffs because the data is fit into grid and the grid is diffed too and interferes.
So we need a binary diffing tool.
Diffing is just a sequence alignment problem, there exist lot of impls of sequence alignment, i.e. python standard library contains a one, so it is not a problem.
What is needeed is
Kaitai is a declarative language for parsing binary formats. We need plugins that will:
I don't have anything specific at the moment, but some thought should be put into making BitContainer metadata better. That means:
Currently, ASCII view seems to replace symbols below 32 and above 126 with dots. But if Hobbits' ASCII tab behaves more like Hiew, more information would be available to the user. For example, it could display symbols from an 8-bit code page for 128-255 and glyphs similar to IBM control symbols for 0-31 and 127. Here are symbol set examples for code pages 437 and 866: https://en.wikipedia.org/wiki/File:Codepage-437.png https://en.wikipedia.org/wiki/File:Octets_in_conformant_CP866_ordered_by_nibbles.png
Compilation fails with:
peakselector.cpp: In member function ‘virtual void PeakSelector::paintEvent(QPaintEvent*)’:
peakselector.cpp:71:18: error: aggregate ‘QPainterPath path’ has incomplete type and cannot be defined
71 | QPainterPath path;
| ^~~~
Including QPainterPath with every QPainter fixes it.
Thanks.
When "Byte Raster" tab is selected, and the image is over 100 pixels wide, the interface responds to user actions much slower. On my 2-core 2GHz Celeron when a 160x160 image is displayed, switching between other visualization tabs takes negligible time, but switching to and from "Byte Raster" takes between 2 and 3 seconds. Changing its zoom takes about 5 seconds. Other user actions are processed about as slowly.
Is there any way to make the interface faster when "Byte Raster" is visible?
Observed in Ubuntu Linux for versions 0.22.0 and 0.22.1.
Having an interpreter inside the Python runner plugin can speed up python processing and might help with other things.
Probably and import/export plugin, but further research is needed
Even the simpest spec consuming 1 byte fails: EOFError: requested 1 bytes, but got only 0 bytes
.
Hi,
I detect a problem when the "split view" is applied after 'import' file or data -> the application crashes ;
the "split view" is possible only before 'import', add other view crashes application after 'import'.
i m using pre-built binaries v0.46.7 (Hobbits (64-bit Windows 2016 msvc2017), Same issue with v0.46.4.
Tested under 2 x Windows 10 pro 64-bit (20H2 & 2004).
Have you already observed this ?
Thanks
Best regards
> /Applications/hobbits.app/Contents/MacOS/hobbits
dyld: Library not loaded: /usr/local/opt/gettext/lib/libintl.8.dylib
Referenced from: /Applications/hobbits.app/Contents/Frameworks/hobbits-cpython/lib/libpython3.9.dylib
Reason: image not found
fish: Job 1, '/Applications/hobbits.app/Conte…' terminated by signal SIGABRT (Abort)
I would suggest using macdeployqt
for mac builds. It packages Qt a bit more nicely.
Mac 11.5.2, hobbits-0.51.1
BitContainers need to be able to hold larger bit arrays. I think it will go something like this:
The current parsing of the kaitai output does not go through lazily parsed chunks. This is described in kaitai-io/kaitai_struct#880
The kaitai struct plugin should do this lazy parsing in order to get closer to functional parity with other kaitai GUIs
Hi,
got some question about plugins in python :
In the JSON file, when I put "decimal" or "integer" type for my parameter, it doesn't change anything, in the both cases it's an integer, is it normal ? Do i miss something ?
Can we put negative values in our integer field ? And so, can we precise a maximum and a minimum value in the JSON file for a specific parameter ?
When I try to add a boolean parameter in my JSON file and execute my plugin in Hobbits, I've always this error :
"Operator Plugin Error: Plugin 'test' reported an error with its processing: Other errors:
Failed to parse arg 4"
(here the arg 4 is of course my boolean, if i put more parameters, it will say 'arg 5' or 'arg 6')
and this is my script in python :
def operate_on_bits(input, output_bits, output_info, perso_var, progress):
if input.bits.size() > 300 and perso_var: # the perso_var is my boolean
output_info.add_highligt("python", "mychunck", 20, 3000)
output_bits.resize(input.bits.size())
for i in range(0, input.bits.size()):
output_bits.set(i, input.bits.at(i)) # this is the only other modification in the default plugin (I removed the 'not')
if progress.is_cancelled():
return
progress.set_progress(i, input.bit.size())
and my JSON file :
{
"name": "test",
"description": "plugin test",
"tags": ["test"],
"script": "main.py",
"type": "operator",
"extra_paths": [],
"parameters": [
{
"name": "perso_var",
"type": "boolean"
}
]
}
This is clearly wrong.
Observed for version 0.22.0 in Ubuntu Linux.
The tests have been useful for catching regressions when things are refactored. Having tests for all features of all core plugins would be great.
It also might be nice to have a separate fast test command/argument that only does the small file tests (the gigantic file tests take a while to run and will usually only catch problems in the BitArray caching system.)
A file generated with the following command
echo 2f fe b8 9b 22 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a4 0f 08 00 1a 06 08 af d0 bc 82 06 | xxd -r -p > file.bin
opened with hobbits with the following ksy:
meta:
file-extension: bla
id: servermod
endian: le
imports:
- /serialization/google_protobuf
seq:
- id: records
type: record
repeat: eos
types:
main_header:
seq:
- id: ip1
type: u1
- id: ip2
type: u1
- id: ip3
type: u1
- id: ip4
type: u1
- id: body_len
type: u4le
record:
seq:
- id: len_record
type: main_header
doc: Size of whole record, including all headers, footers and data
- id: ctype
type: u1
doc: ctype
- id: flow
type: u1
doc: flow
- id: cid
type: u4le
doc: cid
- id: did
type: u4le
doc: did
- id: seq1
type: u8le
doc: seq1
- id: command
type: u2le
doc: command
- id: protobuf
size: len_record.body_len - 24
type: google_protobuf
When you open the file with the struct in kaitai IDE works correctly, but opening in hobbits when you select the protobuf part (the last 10 bytes - 08 af d0 bc 82 06 ) it's highligthed the start of the file.
If you concatenate some times and select the protobuf of any of the registers always it's highligthed the starting bytes of the file.
The command to generate with 3 concats is:
echo 2f fe b8 9b 22 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a4 0f 08 00 1a 06 08 af d0 bc 82 06 | xxd -r -p > file.bin ; echo 2f fe b8 9b 22 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a4 0f 08 00 1a 06 08 af d0 bc 82 06 | xxd -r -p >> file.bin ; echo 2f fe b8 9b 22 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a4 0f 08 00 1a 06 08 af d0 bc 82 06 | xxd -r -p >> file.bin
The hobbits runner should have a server mode that is configured with a bunch of templates and can apply those templates to uploaded data. It could work something like this:
Server is started with a collection of template files. It makes an HTTP REST interface with endpoints for each of the template files. A user can then upload a file to the endpoint and receive a URL where the output(s) can be downloaded and maybe a websocket channel to listen for progress updates.
With those server capabilities, it should be pretty easy to slap on a Web GUI.
When attempting to launch hobbits.app v0.44.1 downloaded from the release page on a Mac, it crashes with the following lines in the crash report:
Dyld Error Message:
dyld: Using shared cache: E2B1A630-68DD-30AC-A036-6AC018B154DA
Library not loaded: /Users/*/libpython3.9.dylib
Referenced from: /Applications/hobbits.app/Contents/MacOS/hobbits
Reason: image not found
Inspecting the runtime library search paths in the hobbits
executable using otool -L hobbits
shows the following:
/Applications/hobbits.app/Contents/MacOS/hobbits:
@rpath/libhobbits-core.1.dylib (compatibility version 1.0.0, current version 1.0.0)
@rpath/libhobbits-widgets.1.dylib (compatibility version 1.0.0, current version 1.0.0)
@rpath/libhobbits-python.1.dylib (compatibility version 1.0.0, current version 1.0.0)
/Users/runner/work/1/a/lib/libpython3.9.dylib (compatibility version 3.9.0, current version 3.9.0)
@executable_path/../Frameworks/libfftw3.3.dylib (compatibility version 10.0.0, current version 10.9.0)
@executable_path/../Frameworks/libfftw3_threads.3.dylib (compatibility version 10.0.0, current version 10.9.0)
@rpath/QtWidgets.framework/Versions/5/QtWidgets (compatibility version 5.14.0, current version 5.14.1)
@rpath/QtGui.framework/Versions/5/QtGui (compatibility version 5.14.0, current version 5.14.1)
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1875.0.0)
/System/Library/Frameworks/Metal.framework/Versions/A/Metal (compatibility version 1.0.0, current version 211.51.0)
@rpath/QtCore.framework/Versions/5/QtCore (compatibility version 5.14.0, current version 5.14.1)
/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.6.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
Using otool -l hobbits
and searching for LC_RPATH
shows the following two rpaths:
path @executable_path/../Frameworks (offset 12)
path @executable_path/../Frameworks/python/lib (offset 12)
However, the library path for libpython3.9.dylib is hardcoded in the executable.
While this can be fixed using install_name_tool as follows:
install_name_tool -change "/Users/runner/work/1/a/lib/libpython3.9.dylib" "@rpath/libpython3.9.dylib" hobbits
This needs to be applied to hobbits-runner
and libhobbits-python.1.dylib
too.
Likewise, libpython3.9.dylib
references libintl.8.dylib
installed by Homebrew instead of the one bundled with the software. Fixing all of these with install_name_tool
allows hobbits.app to function.
I will look at the build scripts and try to see what's going wrong.
While I am testing on an M1 Mac running macOS 11.1, this problem is not limited to Big Sur or Apple Silicon.
Hi,
just an idea to select an area in a view with the mouse ;
first onclick (left) define the start of the area in the view
SHIFT + onclick (left) define the end of the area
After use the 'Extractor' (Operator plugin) to choose 'Extract Section" or other extract operations.
or may be new 'right click' access for extract operations.
would this be technically feasible?
Best regards
It would be nice to have hobbits packaged in a way that let plugin developers update the application/headers/library easily using a package manager. We could provide the following packages:
A few non-trivial things that might need to be addressed:
https://api.kde.org/frameworks/ktexteditor/html/
It has syntax highlighting for yaml and blocks folding. One needs to set mode (not highlight mode, but just mode) into YAML
For Windows probanly QScintilla can be used, but for KDE, I guess, it may make sense to use KTextEditor
Obstacles:
libqt5xml5-dev
package in Debian & Ubuntu. How have they built KF* packages without that? Debian bugSometime I just want to peek or know I just need a small sample of data from an http call.
It would be nice to be able to specify how much data you want and allow response to be interrupted leaving just the data that had been returned thus far.
This stuff seems to work with the changes I made in bcbd804, but it's not as clean and intuitive as I would like it to be. A few things to think about:
The current display plugin interface is pretty simple and flexible, but it needs to be updated to support some new things:
right now, the GUI from the release binaries is pretty ugly and sometimes weirdly different from the "reference" GNOME build on my machine. There should be a consistent style on all platforms.
Based on what I've seen in other GUIs and what I remember doing in the past, this will require a combination of Qt style sheets and some QStyle (?) management within the application. We only need one theme, preferably dark-ish.
Here's an example (but I don't really like how this one looks): https://github.com/ColinDuquesnoy/QDarkStyleSheet
It might also be worth using some open fonts and shipping with them. I noticed that the "Monospace" font has a weird load error on Mac.
And FSF position is that it is not allowed to use GPLed plugins for a non-GPL program to circumvent GPL virality.
If the main program dynamically links plug-ins, and they make function calls to each other and share data structures, we believe they form a single combined program, which must be treated as an extension of both the main program and the plug-ins.
It may make sense to replace FFTW with something else, i.e. https://github.com/clMathLibraries/clFFT
If you don't want to depend on OpenCL, you can use https://cr.yp.to/djbfft/install.html
May i miss something but the old way to make a plugin in C was to install wizards like said in the tutorial, but I'm unable to find it.
Is their something i miss ? Did the way to do that improved ?
I made some researches and found this, from Adam. But i can't find the file in the Code part.
Can someone help me on this point ?
Thanks !
sometimes the hobbits runner hangs after completing a batch. this never seems to happen in the ci tests, but I saw it happen with a kaitai -> extract batch
CMake Error at CMakeLists.txt:83 (include):
include could not find load file:
/home/KOLANICH/hobbits/build/conanbuildinfo.cmake
Would benefit both platforms a lot. The one way is to do it directly as a plugin, another way is to separate "raster" widgets to be reusable in other projects too, like Cutter.
Radare2 is a highly-portable cross-platform reverse engineering framework and a toolkit without dependencies. It has support for analyzing binaries, disassembling code, debugging programs, attaching to remote GDB/LLDB, WinDbg servers, rich plugin system (see r2pm
), and integration with various decompilers. For example, ghidra decompiler plugin - r2ghidra-dec. It is actively developed and can be easily integrated in various open source and commercial products. I believe, it will be highly beneficial to support these and provide a package for install from r2pm
, see the package repository here: https://github.com/radareorg/radare2-pm
For documentation on writing plugins for radare2 see Scripting and Plugins Radare2 Book chapters.
Cutter is a crossplatform Qt/C++ GUI frontend to radare2:
For documentation on writing plugins for Cutter see the official tutorial and the curated list of various popular plugins.
See also rizinorg/cutter-plugins#3
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.