build_mac only builds debug apps but refers to release versions about die-engine

[skolpadda]

Thanks - I thought it was getting further, but is still failing with the capstone library error, and also seems to be stubbornly building the debug app.

I really appreciate your attention to this, btw.

[skolpadda]

Success! That seems to have done the trick - thanks again for your assistance :)

Maybe consider adding release/ debug as a parameter to the build script? Not sure how much work that would be, but just adding a hard-coded CONFIG+=release might be enough. Anyway, really appreciate all your help.

[skolpadda]

Well.. spoke too soon.

I'm unable to open the DiE.app - Mac whines about error -10810. I think there may still be something wrong with the build. I can upload the whole build.log if needed, since I'm not really sure what's going wrong, but here's the end of it:

Check file /srv/repos/detect-it-easy/die-source/build/release/diec: TRUE
Check file /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/die: TRUE
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/diec (No such file or directory)
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/diec (No such file or directory)
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/diec (No such file or directory)
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/diec (No such file or directory)
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/diec (No such file or directory)
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/diec (No such file or directory)
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/diec (No such file or directory)
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/diec (No such file or directory)
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/diec (No such file or directory)
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/diec (No such file or directory)
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't open file: /srv/repos/detect-it-easy/die-source/build/release/die.app/Contents/MacOS/diec (No such file or directory)
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/Frameworks: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/Frameworks: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/Frameworks: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/Frameworks: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/Frameworks: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/Frameworks: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/Frameworks: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/Frameworks: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/Frameworks: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/Frameworks: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/PlugIns/: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/PlugIns/platforms/: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/PlugIns/: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/PlugIns/platforms/: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/PlugIns/: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/PlugIns/: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/PlugIns/imageformats/: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/PlugIns/: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/PlugIns/imageformats/: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/PlugIns/: File exists
mkdir: /srv/repos/detect-it-easy/die-source/release/die_mac_portable/die.app/Contents/PlugIns/imageformats/: File exists
ditto: can't get real path for source '/srv/repos/detect-it-easy/die-source/release/die_mac_portable/.app'
zip warning: /srv/repos/detect-it-easy/die-source/release/die_mac_portable_portable_3.04.zip not found or empty
zip warning: name not matched: __MACOSX/*

zip error: Nothing to do! (/srv/repos/detect-it-easy/die-source/release/die_mac_portable_portable_3.04.zip)

This is odd since there's nothing under release/, and the entries inside build/release have a different case from what seems to be expected.

[skolpadda]

build.log

[skolpadda]

Yes! It built successfully, although there was the same zip error at the end, however I'm unable to open the app:

LSOpenURLsWithRole() failed with error -10810 for the file /System/Volumes/Data/srv/repos/detect-it-easy/die-source/build/release/DiE.app.

I've seen this before with various permission issues but as far as I can tell, the files are set up correctly - executable permissions, no quarantine attributes etc.

Oddly if I unzip the die_mac_portable_portable_3.04.zip file I'm able to successfully run the die.app contained there. It's just the build/release/DiE.app that won't open.

[skolpadda]

This is actually running inside a linux docker container - and if I remove the call to ::convert (and all the subsequent code), everything works fine - and works up to that point in any case.

[skolpadda]

Here's the line that fails:

            record.colText=QApplication::palette().text().color();

I'm guessing something to do with not running in the GUI maybe? Or I haven't initialized the application correctly?

[horsicq]

Great! Just remove this line. It is only for GUI, color for detect.

[horsicq]

Fixed
horsicq/Formats@73692c8
horsicq/SpecAbstract@7e1d444

[skolpadda]

Thanks again for all your hard work! I think I have enough to move forward now.

[horsicq]

Hello! Thanks a lot for your report! If you need debug version just open .pro file in qt creator or use cmake.

cmake -DCMAKE_BUILD_TYPE=Debug ..

[horsicq]

I could create mac_build_debug.sh for debug bundle but I do not know if it really necessary.

[skolpadda]

Thanks for the (quick!) response. My question is more how can I build the release versions?

[horsicq]

Hmm. I always thought that build_mac makes release version. :) Are you sure?

You could use cmake here too. Just change Debug to Release

[skolpadda]

I can upload the full build log but it's pretty huge. Here's the relevant bit though:

ack debug build.log
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -stdlib=libc++ -headerpad_max_install_names -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -mmacosx-version-min=10.12 -Wl,-rpath,@executable_path/../Frameworks -Wl,-rpath,/usr/local/Qt-5.12.10/lib -o ../build/debug/diec consoleoutput.o main_console.o binary_script.o com_script.o die_script.o die_scriptengine.o global_script.o msdos_script.o pe_script.o ne_script.o le_script.o lx_script.o mach_script.o elf_script.o xformats.o subdevice.o xbinary.o xcom.o xmsdos.o xne.o xle.o xpe.o xmach.o xelf.o xandroidbinary.o xdex.o xarchives.o xarchive.o xmachofat.o xsevenzip.o xzip.o xcab.o xrar.o scanitem.o scanitemmodel.o xcapstone.o entropyprocess.o xoptions.o moc_consoleoutput.o moc_binary_script.o moc_com_script.o moc_die_script.o moc_die_scriptengine.o moc_global_script.o moc_msdos_script.o moc_pe_script.o moc_ne_script.o moc_le_script.o moc_lx_script.o moc_mach_script.o moc_elf_script.o moc_xformats.o moc_subdevice.o moc_xbinary.o moc_xcom.o moc_xmsdos.o moc_xne.o moc_xle.o moc_xpe.o moc_xmach.o moc_xelf.o moc_xandroidbinary.o moc_xdex.o moc_xarchives.o moc_xarchive.o moc_xmachofat.o moc_xsevenzip.o moc_xzip.o moc_xcab.o moc_xrar.o moc_scanitemmodel.o moc_xcapstone.o moc_entropyprocess.o moc_xoptions.o -F/usr/local/Qt-5.12.10/lib /System/Volumes/Data/srv/repos/detect-it-easy/die-source/XArchive/3rdparty/zlib/libs/mac/libzlib.a /System/Volumes/Data/srv/repos/detect-it-easy/die-source/XArchive/3rdparty/bzip2/libs/mac/libbzip2.a /System/Volumes/Data/srv/repos/detect-it-easy/die-source/XArchive/3rdparty/lzma/libs/mac/liblzma.a /System/Volumes/Data/srv/repos/detect-it-easy/die-source/XCapstone/3rdparty/Capstone/libs/mac/libcapstone_x86.a -framework QtScript -framework QtCore -framework DiskArbitration -framework IOKit
make[1]: *** [../build/debug/diec] Error 1

With cmake I get:

srv/repos/detect-it-easy/die-source/XCapstone
CMake Error at gui_source/CMakeLists.txt:21 (find_package):
Could not find a package configuration file provided by "QT" with any of
the following names:

Qt5Config.cmake
qt5-config.cmake

Add the installation prefix of "QT" to CMAKE_PREFIX_PATH or set "QT_DIR" to
a directory containing one of the above files. If "QT" provides a separate
development package or SDK, be sure it has been installed.

[horsicq]

Error 1 is a common error. I cannot say anything. Is there errors before the error?

It seems cmake cannot find qt5 on your machine. Can you open qt creator, create a simple project from the first template and run it or debug from qt creator IDE?

[skolpadda]

I'm not using qtcreator, is the build_mac script not enough? I believe the errors are because the script has built this:

XCapstone/3rdparty/Capstone/libs/mac/libcapstone_x86d.a  # x86d

which seems like it should be

XCapstone/3rdparty/Capstone/libs/mac/libcapstone_x86.a  # x86

...but I just want to build the release app in any case.

[horsicq]

Yes. build_mac script should be enough

// which seems like it should be

Hmm. It is really interesting.

Could you please try to compile on your machine this project. https://github.com/horsicq/XOpcodeCalc
It uses build_mac too but does use libcapstone_x86

[skolpadda]

$ find build -type f
build/debug/XOcalc.app/Contents/MacOS/XOcalc
build/debug/XOcalc.app/Contents/Resources/empty.lproj
build/debug/XOcalc.app/Contents/Resources/main.icns
build/debug/XOcalc.app/Contents/Info.plist
build/debug/XOcalc.app/Contents/PkgInfo

[horsicq]

Great! Thanks you a lot for testing. At least You have qt5 but qt5 compiles debug builds.
I will try to research it.

[horsicq]

I guess it should work if you will add to build_tools/mac.sh line 30

$QMAKE_PATH "$1" -spec $X_QMAKE_SPEC CONFIG+=x86_64
-->
$QMAKE_PATH "$1" -spec $X_QMAKE_SPEC CONFIG+=x86_64 CONFIG+=release

[horsicq]

How did you install Qt on your machine? With brew or Qt online installer?

[skolpadda]

These are the steps that worked:

git clone git://code.qt.io/qt/qt5.git
cd qt5
git checkout 5.12.10
./init-repository
 
# create a build directory, configure qt with specific modules, and build/ install
cd ..
mkdir -p qt5.12.10
../qt5/configure -opensource -skip qtwebengine  -nomake tests -nomake examples -prefix /usr/local/Qt-5.12.10
make # Note this may take the better part of a day
sudo make install  # Also takes a good while

[skolpadda]

I just saw this in the build log for XOpcodeCalc:

cd gui_source/ && ( test -e Makefile || /usr/local/Qt-5.12.10/bin/qmake -o Makefile /System/Volumes/Data/srv/repos/detect-it-easy/XOpcodeCalc/gui_source/gui_source.pro -spec macx-clang CONFIG+=x86_64 ) && /Applications/Xcode.app/Contents/Developer/usr/bin/make -f Makefile clean
/usr/local/Qt-5.12.10/bin/qmake -o Makefile gui_source.pro -spec macx-clang CONFIG+=x86_64

it's not passing the CONFIG+=release flag to make?

[skolpadda]

Aha! Had to make clean AND get rid of the Makefiles. Seems a little happier now, will update when things finish building.

[horsicq]

Thanks a lot for your solution! BtW I added hardcoded release flag horsicq/build_tools@e517981

[horsicq]

The whole build log would be nice.

[horsicq]

Thanks a lot for the build log! Qt compiled the project successful and it is a good news. But there is errors in build_mac.sh

// This is odd since there's nothing under release/, and the entries inside build/release have a different case from what seems to be expected.

Yes. You are right. I changed name from die to DiE on macOS but forgot to fix all macOS scripts. :(

[horsicq]

I fixed it: 517e555

After scripts' work in release folder should be file die_mac_portable_portable_3.04.zip

[horsicq]

Great! Thanks a lot for the bugreport and testing. :)

[skolpadda]

No problem - thanks for jumping on this!

While you're here, any idea how I can get the output as seen in the attached screenshot directly from SpecAbstract? Basically I have a JNI wrapper around StaticScan and it looks like getMsRichString() might be helpful but I have no idea what to pass in.

Thanks!

[horsicq]

SpecAbstract::SCAN_OPTIONS scanOptions={0};
SpecAbstract::SCAN_RESULT scanResult=StaticScan::processFile(sFileName,&scanOptions);
static QListXBinary::SCANSTRUCT _listRecords=SpecAbstract::convert(&(scanResult.listRecords));

ScanItemModel model(&_listRecords);

XBinary::FORMATTYPE formatType=XBinary::FORMATTYPE_TEXT;

printf("%s\n",model.toString(formatType).toLatin1().data());

[horsicq]

SpecAbstract::SCAN_OPTIONS scanOptions={0};
scanOptions.bDeepScan=true;
scanOptions.bRecursiveScan=true;

SpecAbstract::SCAN_RESULT scanResult=StaticScan::processFile(sFileName,&scanOptions);

[horsicq]

XBinary::FORMATTYPE_CSV;
XBinary::FORMATTYPE_JSON;
XBinary::FORMATTYPE_TSV;
XBinary::FORMATTYPE_XML;

If you need another formats

[horsicq]

You can create a topic here: https://github.com/horsicq/DIE-engine/discussions
If you will have any questions in the future. :)

[skolpadda]

Unfortunately this line just crashes with no information:

 static QListXBinary::SCANSTRUCT _listRecords=SpecAbstract::convert(&(scanResult.listRecords));

I can open a discussion/ other ticket if needed.

[horsicq]

Just remove static.

use:

QList<XBinary::SCANSTRUCT> _listRecords=SpecAbstract::convert(&(scanResult.listRecords));

[skolpadda]

Same behaviour.

[horsicq]

It is code from console version of nfd https://github.com/horsicq/Nauz-File-Detector/blob/c596263386067a4d7d7c98c7e1272896b7bd817d/console_source/main_console.cpp#L57

Could you try to compile this project an check behaviour https://github.com/horsicq/Nauz-File-Detector

[skolpadda]

That app seems to work fine. It seems like the convert() method isn't even being called from my code - I just get a SIGSEGV and the entire app bombs out.

    SpecAbstract::SCAN_OPTIONS scanOptions = {};
    SpecAbstract::SCAN_RESULT scanResult = {};

    scanOptions.bRecursiveScan = true;
    scanOptions.bDeepScan = true;
    scanOptions.bHeuristicScan = true;
    scanOptions.bAllTypesScan = true;

    StaticScan staticScan;
    staticScan.setData(buf, len, &scanOptions, &scanResult);
    staticScan.process();
    staticScan.stop();

    cerr << "scanned\n";
    QList<XBinary::SCANSTRUCT> _listRecords = SpecAbstract::convert(&(scanResult.listRecords));
    cerr << "converted\n";

I have some diagnostics in the convert() method but they don't get output.

[horsicq]

// That app seems to work fine.
// SIGSEGV

Can you try to copy libs folder(XArchive,XCapstone) from this app build folder to build folder where SIGSEGV is.

[skolpadda]

Same thing - output is below. I should explain I've got a JNI wrapper around your code, but all of this is happening inside C++, not java, so I don't think it's related to the JNI code.

[thread 140490899154688 also had an error][thread 140490898102016 also had an error]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#
SIGSEGV (0xb) at pc=0x00007fc6b655884c, pid=99, tid=0x00007fc696034700
#
# JRE version: OpenJDK Runtime Environment (8.0_312-b07) (build 1.8.0_312-b07)
# Java VM: OpenJDK 64-Bit Server VM (25.312-b07 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libQt5Gui.so.5+0x10884c]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid99.log
#
# If you would like to submit a bug report, please visit:
#   https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&component=java-1.8.0-openjdk
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

Here's the (AFAICT) relevant bit from the error log, but it's on the java side so might not be super useful.

Event: 0.037 Thread 0x00007fa82804c800 Exception <a 'java/lang/NoSuchMethodError': Method sun.misc.Unsafe.defineClass(Ljava/lang/String;[BII)Ljava/lang/Class; name or signature does not match> (0x000000076c987cc8) thrown at [/builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.312.b0
7-2.el8_5.x86_64
Event: 0.037 Thread 0x00007fa82804c800 Exception <a 'java/lang/NoSuchMethodError': Method sun.misc.Unsafe.prefetchRead(Ljava/lang/Object;J)V name or signature does not match> (0x000000076c987fb0) thrown at [/builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.312.b07-2.el8_5.x86_64/o
penjdk/hotspot/s

[horsicq]

// Basically I have a JNI wrapper

// That app seems to work fine.

Hmm. Mb just use modified console version of Nauz-File-Detector with xml or json output?

Call it from Java code https://dzone.com/articles/execute-shell-command-java and handle output?

[skolpadda]

That's not a bad idea, but the reason for doing all this is for the best performance possible - I think starting a separate process may degrade performance somewhat.

[horsicq]

It may be security limits with java in macOS.

[horsicq]

Hmm. Interesting.

[skolpadda]

I've been able to get some diagnostics to display, will let you know when I figure out exactly what's causing the crash.

[horsicq]

ok!

[horsicq]

I will think about it how fix it for java bind.

[skolpadda]

That helped - getting further at least.