backtrace-labs / crashpad Goto Github PK

Is Backtrace supposed to be filling these out, is there something we're not setting. These are really not fun to sort through when intermixed with real attributes that we've recorded. I'm on the latest build of 0.1.0, and so I'd hoped these would go away after we updated to latest.

"device.model": null,
"_request.truncated": null,
"application.session": "00000000-0000-0000-0000-000000000000",
"_compressed": 0,
"lang.version": null,
"lang.name": null,
"gc.heap.used": 0,
"gc.heap.total": 0,
"cpu.process.blocked": 0,
"cpu.process.running": 0,
"cpu.process.count": 0,
"cpu.brand": null,
"cpu.arch": null,
"cpu.boottime": 0,
"cpu.context": 0,
"cpu.softirq": 0,
"cpu.irq": 0,
"cpu.iowait": 0,
"cpu.idle": 0,
"cpu.kernel": 0,
"cpu.nice": 0,
"cpu.user": 0,
"system.memory.vmalloc.chunk": 0,
"system.memory.vmalloc.used": 0,
"system.memory.vmalloc.total": 0,
"system.memory.slab": 0,
"system.memory.writeback": 0,
"system.memory.dirty": 0,
"system.memory.swap.free": 0,
"system.memory.swap.total": 0,
"system.memory.inactive": 0,
"system.memory.active": 0,
"system.memory.swap.cached": 0,
"system.memory.cached": 0,
"system.memory.buffers": 0,
"system.memory.free": 0,
"system.memory.total": 0,
"sched.cs.involuntary": 0,
"sched.cs.voluntary": 0,
"vm.swap.size": 0,
"vm.pte.size": 0,
"vm.shared.size": 0,
"vm.data.size": 0,
"vm.stack.size": 0,
"vm.rss.size": 0,
"vm.rss.peak": 0,
"vm.locked.size": 0,
"vm.vma.size": 0,
"vm.vma.peak": 0,
"descriptor.count": 0,
"uname.release": null,
"_missing_symbol": 0,
"_delete_reason_audit": null,
"_delete_reason": null,
"_deleted": 0

Archiving multiple attachments building an archive , or even compressing a text log file would be valuable. Our log files are all text, and Windows vs. SteamDeck zip compression should probably be inside the crash handler.

I've been trying to evaluate crashpad (along with the backtrace service), and sadly even running the demo doesn't seem to work properly for me.

I've tried on arm-based macOS and on x64 Ubuntu.

What macOS gives out is:

$ ./demo_macos 
Crashpad URL: https://submit.backtrace.io/yolo/c5a3a1675dbcfa1d12fd7efbc7bef9350874654e55ff86478115c8bc073f54dc/minidump
Crashpad handler: ~/Developer/github/backtrace-labs/crashpad/cbuild/handler/handler
Crashpad database: mydb
45
50
1
[57888:8921244:20230623,162221.882787:WARNING in_range_cast.h:38] value -634136515 out of range
[57888:8921246:20230623,162221.890429:ERROR directory_reader_posix.cc:42] opendir mydb/attachments/07afdd26-7591-40b0-9f84-86419e11a0a3: No such file or directory (2)
[57888:8921244:20230623,162221.891080:WARNING crash_report_exception_handler.cc:257] UniversalExceptionRaise: (os/kern) failure (5)
[1]    57885 segmentation fault  ./demo_macos

It seems that there is an issue with finding the minidump file to upload it to the server (the ERROR above).

On Ubuntu:

Crashpad URL: https://submit.backtrace.io/yolo/c5a3a1675dbcfa1d12fd7efbc7bef9350874654e55ff86478115c8bc00000000/minidump
Crashpad handler: /home/myusername/demo_crashpad_cmake/build/crashpad/handler/handler
Crashpad database: .
45
50
[192834:192834:20230623,163531.683169:FATAL spawn_subprocess.cc:227] posix_spawn: No such file or directory (2)
[192833:192833:20230623,163532.829951:WARNING spawn_subprocess.cc:251] intermediate process terminated by signal 5 (Trace/breakpoint trap) (core dumped)
[192833:192833:20230623,163532.830607:ERROR socket.cc:93] sendmsg: Broken pipe (32)
0
Segmentation fault (core dumped)

Any hints would be appreciated. Thank you.

Our game has a debug builds that don't include crashpad in the build, and crashpad_handler.exe isn't deployed alongside the .exe. Only our shipping builds do this. Yet when running the debug build, we get crashpad errors in our logs with errors trying to find attachments from Chrome. This is worriesome that somehow our app even without the crashpad library or handler is somehow triggering crashpad errors.

[1115/142504.058:ERROR:filesystem_win.cc(130)] GetFileAttributes C:\Users\foo\AppData\Local\Google\Chrome\User Data\Crashpad\attachments\c46abb11-db2d-4000-ae60-62c212511087: The system cannot find the file specified. (0x2)
[1115/142504.059:ERROR:filesystem_win.cc(130)] GetFileAttributes C:\Users\foo\AppData\Local\Google\Chrome\User Data\Crashpad\attachments\66f378f7-2f27-47b4-9284-722b89ddb398: The system cannot find the file specified. (0x2)
[1115/142504.059:ERROR:filesystem_win.cc(130)] GetFileAttributes C:\Users\foo\AppData\Local\Google\Chrome\User Data\Crashpad\attachments\201f16ff-7792-4774-8648-26ea43ec91a9: The system cannot find the file specified. (0x2)
[1115/142504.064:ERROR:filesystem_win.cc(130)] GetFileAttributes C:\Users\foo\AppData\Local\Google\Chrome\User Data\Crashpad\attachments\65950f34-8e80-4070-935b-25950d63ef02: The system cannot find the file specified. (0x2) 

The workaround for now, was to go to this folder delete the metadata file from Google/Chrome.

Upstream minichromium now sets OS_MAC instead of OS_MACOSX which handler_main.cc uses to gate the addition of the attachments support to the CrashReportExceptionHandler object.

if I ues crashpad in my app, crashpad can normally generate .dmp file, but the system exception monitor service(com.apple.ReportCrash) cannot generate .ips file.

if I do not use crashpad in my app, the system exception monitor service(com.apple.ReportCrash) can normally generate .ips file.

You can view these .ips files in the console.app

Currently, There are two fixes from https://chromium.googlesource.com/crashpad/crashpad.git repository.

chromium/crashpad@b6f2d06 fix unpaired namespace which occur when cross compile with Android API level more than 23.

chromium/crashpad@9f4741d fix handler attachment.
This fix is more tricky because it conflicts with Backtrace's change.

Addition information:

• I build Android Crashpad using Docker's Ubuntu image.
• I run Android Crashpad's test by mounting android-linux-platform-tool and connect device through Wifi.
• I need to build Android Crashpad for Unreal Engine 4 on Android device.

Hi,
In the past I have been successfully using https://github.com/unidentifieddeveloper/crashpad.git to build crashpad for use in our projects.
But since https://github.com/backtrace-labs/crashpad seems to be more active & supported, I wanted to try and use it instead for building crashpad.

The build went fine, but when I now use crashpad in one of our apps that have been built in a Release config ... then suddenly I see a handler.exe console window being opened alongside the regular application window.
(for Debug builds I don't see this extra window)

We never saw such a handler.exe console window when we used crashpad binaries that were built from https://github.com/unidentifieddeveloper/crashpad.git

Do you have any guess why this would be happening in the backtrace-labs fork ?
Thanks

We've got a bit of an issue and I can only think of bad solutions so I'm hoping I've just overlooked something obvious. We're using the "attachments" argument to 'StartHandler' to attach our log file on a crash. This works great. The log file is always there on disk...it's always useful to us...and it's uploaded successfully every time.

The issue is that there are often needs to add MORE files at a later time....files that did not exist at the time the handler was started and whose name/location might not be predictable early either. These are non-fatal incidents where the process is still healthy. (A GPU error for example). We cannot find a way to add files later.

Some ideas that I considered:

1. Killing the existing handler and starting a NEW handler now that we have more specific file location information. There seems to be no exposed way to kill/restart a handler though.
2. Communicating with the handler to send more information via IPC. This would require customizing the handler source of course which isn't ideal.
3. Disable 'Uploads', write ourselves a private note somewhere to disk about the location/need of the extra files and then add them to the report the next time the app is restarted and THEN upload it. It seems like the only access to a 'Report' implies that its immutable and cannot easily add attachments to it.
4. Let the system upload reports without the extra attached files and then upload them directly to backtrace later. I've not found an API for this but maybe there's some URL we can use to send via CURL on a worker thread if we knew the report's UUID.
5. Add one or more placeholder paths to StartHandler. If we end up needing them later, we can rename the files to align with those paths. If we don't end up needing them, it seems like handler doesn't care that the files don't exist and will just log out an error but still continue just fine. This seems like it would work but it's a bit crude.

Would you guys have any recommendations on this issue?

Thanks!

Hi @rick-bt , we have worked on a basic formula for porting backtrace_crashpad to homebrew but we would appreciate your feedback before going forward and sending a PR to homebrew.

You can try building it on your system:
https://gist.github.com/jviotti/ba790598c0fb880616d18b21047f8246

We want to start simple, however, in near future, we want to extend the formula to have iOS support. It'd also be appreciated if you supported minidump_stackwalk too.

Another query we have is, what format of includes do you prefer?
#include <crashpad/client/crashpad_client.h>
OR
#include <backtrace/crashpad/client/crashpad_client.h>

cc: @jviotti

We set thread names on our own threads via SetThreadDescription, but I don't see names in any of the crashpad code. And when we set thread names, they don't appear in backtrace crash reports.

Right now if you follow the instructions "cmake -G "Visual Studio 16 2019" .." you end up with all of the projects using /MD and /MDd. I've been struggling to find a way to force the project to use /MT and /MTd for Release and Debug builds respectively.

Through the use of the Microsoft CRT memory leak detection mechanism, I was seeing a report in our application with this announcement:

Detected memory leaks!
Dumping objects ->
{3844} normal block at 0x000001F31FDC3970, 48 bytes long.
 Data: <                > 00 00 00 00 CD CD CD CD FF FF FF FF FF FF FF FF 
Object dump complete.

I tracked down the potential source of the leak and it would appear to be this object:

void CommonInProcessInitialization()
{
   ...
  g_non_crash_dump_lock = new base::Lock(); // <---- implementation lacks a delete (use a smart pointer here?)
}

Could you please add C example for us?

We have version 15.7 of VS2017 and this pre-built lib of Crashpad is built with a different version of VS2017. VS2017 does offer binary compatibility with different versions, but that isn't the case if the /GL ( Whole program optimization ) flag is on.

When I try using the library with VS2017 I get an initial C1047 error which is the first indication that something is wrong https://docs.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/fatal-error-c1047?view=vs-2019

I can then workaround that error with the following undocumented linker switch to get a little more information.
/d2:-AllowCompatibleILVersions

Then I get the following errors which give more specifics

6>client.lib(client.crash_report_database_win.obj) : MSIL .netmodule or module compiled with /GL found; restarting link with /LTCG; add /LTCG to the link command line to improve linker performance
6>LINK : fatal error C1900: Il mismatch between 'P1' version '20180423' and 'P2' version '20180214'
6>LINK : fatal error LNK1257: code generation failed

This is the error.message recorded from an abort. But the prior command to this abort was an assert with useful text. Can we override this? And why does this default string contain a formatting token?

error.message samples:
STATUS_FATAL_APP_EXIT: {Fatal Application Exit} %hs <- reported from stack below
EXCEPTION_ACCESS_VIOLATION_WRITE
EXCEPTION_ACCESS_VIOLATION_READ
EXCEPTION_BREAKPOINT

Is this mangled string supposed to be what we see in traces. Most crash reports skip some number of stack levels to report what is at 03.

[ 00 ] ??$FromPointerCast@_KPEAU_EXCEPTION_POINTERS@@@crashpad@@YA_KPEAU_EXCEPTION_POINTERS@@@Z
[ 01 ] raise(int) ( signal.cpp:547 )
[ 02 ] abort() ( abort.cpp:71 )
[ 03 ] VulkanRenderer::CreateTexture(char const *,TextureDescriptor const &) ( VulkanRenderer.cpp:5969 ) <- our code

I compile the demo in /crashpad/examples/linux/demo. And set the url is
std::string url("https://submit.backtrace.io/yolo/20231222_test_demo/minidump");
handler_path is:
/home/crashpad/handler/handler
when I run the ./demo_linux, I don't know how to find the report

Target names such as compat, client and handler are fairly common names developers might choose for their own targets and thus using the backtrace fork is likely to cause issues vs other forks providing the CMake build system where the targets are normally prefixed "crashpad_".

When an abort() or _purecall() is done on the main thread of a native application, we are getting an unhandled exception in our dmp files: 0x40000015: %hs . This is obscuring the actual underlying issue that caused the crash in our application.

This seems to be originating from this CaptureContext call:

From test app

Our team just integrated Backtrace and crashpad, but the api requires all annotations to be passed to StartHandler. There's no way to pass more, there's no StopHandler, and calling StartHandler repeatedly allows us to specify more annotations, but ends up generating 8x more threads and a process every time we call it. Even calling delete on CrashpadClient doesn't cleanup threads.

I don't understand how this is supposed to work. Most apps don't have all the annotations at the very start of main, and don't know where they are going to crash at startup. We generate them over the course of our initialization. Why is thread cleanup so poor, and why isn't there an api for doing this?

bool ReInitializeCrashpad()
{
	// only way to stop the old one is to delete CrashPadClient?
	delete sClient;
	sClient = nullptr;

	// create a new one
	auto& params = sParams;
	sClient = new crashpad::CrashpadClient();
	bool rc = sClient->StartHandler( params.handler, params.db, params.db, params.endpoint, params.annotations, params.arguments, true, true );
	if ( !sClient || !rc )
	{
		return false;
	}
	return sClient->WaitForHandlerStart( 50000 );
}

I try to compile crashpad and folow the steps in https://docs.bugsplat.com/introduction/getting-started/integrations/cross-platform/crashpad/how-to-build-google-crashpad , but the compile error occurred:

D:\code\crashpad\crashpad>gn gen out/Default Done. Made 93 targets from 27 files in 13475ms
D:\code\crashpad\crashpad>ninja -C out/Default ninja: Entering directory `out/Default' [1/514] STAMP obj/build/default_exe_manifest_win.stamp FAILED: obj/build/default_exe_manifest_win.stamp python3 D:/code/crashpad/crashpad/third_party/mini_chromium/mini_chromium/build/win_helper.py stamp obj/build/default_exe_manifest_win.stamp CreateProcess failed: The system cannot find the file specified. [2/514] STAMP obj/snapshot/snapshot_test_ios_data.stamp FAILED: obj/snapshot/snapshot_test_ios_data.stamp python3 D:/code/crashpad/crashpad/third_party/mini_chromium/mini_chromium/build/win_helper.py stamp obj/snapshot/snapshot_test_ios_data.stamp CreateProcess failed: The system cannot find the file specified. ninja: fatal: ReadFile: 句柄无效。

I have been stucked at here more than one day, can you help me ? Thank you.

Hello. I have attempted to compile the CMake-wrapped version of Crashpad provided by Backtrace separately on Windows, aiming to retrieve and save dump files locally. I do not require the use of remote upload services. While the Windows demo can run, it consistently gets stuck at an unknown point and cannot proceed further. If I forcibly terminate the process, it does save a file, but the saved file is 0kb in size. In contrast, when using the crashpad provided by Sentry with the same examples, it executes successfully and the dump file is saved properly. Moreover, the program returns directly without any freezing issues.
I suspect that the issue might be related to the handling of network connectivity, so I registered a Backtrace account and entered the URL parameters. However, even with this, the Backtrace handle still does not operate correctly. Interestingly, when I simply change the handle path to Sentry's while keeping the client as Backtrace's, it successfully uploads the dump file.

Hi, we at Postman are willing to publish crashpad to homebrew, and since this is a well maintained fork of crashpad which is also CMake compatible, it would be great if this fork could be published to homebrew. However, one of the changes required for publishing this fork is adding tags and stable releases recommended by you guys so that it falls under the regulations of Homebrew.

Any responses or suggestions appreciated!

Being able to view fp/simd registers is important. We have extensive use of floating point and simd (SSE/Neon) code.

Getting these messages when we have an abort() as we're shutting down our app. If I manually erase the crash_reports directory, these go away for a little while. But a few minutes later, this looks like it starts searching back in time (here at 9:44:46) and reporting all sorts of missing attachments and reports that don't exist.

We add one attachment Sky.log, and then there is the .dmp file. There are no docs on who's responsible for this directory, but seems like it could slowly fill a users hard drive with 8MB dmp files, and our 1mb logs.

The crash_reports directory just keeps adding attachments (our Sky.log files) and reports (dmp files). Isn't something supposed to erase this. Also StartHandler seems to create this directory which makes it look like the app has had a crash even when it hasn't.

Also this can't delete the directory but I don't think I have any files within it open.

[14560:49932:20231028,094446.297:ERROR filesystem_win.cc:130] GetFileAttributes crash_reports\attachments\a1f8ab84-380c-4fe0-8ac9-0c608cdd36b1: The system cannot find the file specified. (2)
[14560:49932:20231028,094446.298:ERROR filesystem_win.cc:130] GetFileAttributes crash_reports\attachments\ec0ae2b1-e371-4d5c-b27c-e571390db5c4: The system cannot find the file specified. (2)
[14560:49932:20231028,094446.298:ERROR filesystem_win.cc:130] GetFileAttributes crash_reports\attachments\a04ee8cf-52e2-4322-9875-44fbd38e0ad3: The system cannot find the file specified. (2)
[14560:38840:20231028,094446.298:ERROR filesystem_win.cc:130] GetFileAttributes crash_reports\attachments\a1f8ab84-380c-4fe0-8ac9-0c608cdd36b1: The system cannot find the file specified. (2)
[14560:38840:20231028,094446.298:ERROR filesystem_win.cc:130] GetFileAttributes crash_reports\attachments\ec0ae2b1-e371-4d5c-b27c-e571390db5c4: The system cannot find the file specified. (2)
[14560:38840:20231028,094446.299:ERROR filesystem_win.cc:130] GetFileAttributes crash_reports\attachments\a04ee8cf-52e2-4322-9875-44fbd38e0ad3: The system cannot find the file specified. (2)

This is missing a period. Should be reporting13d94c0c-e8f1-4e2e-8a6a-453cedef1930.dmp?
[14560:38840:20231028,094446.299:ERROR filesystem_win.cc:117] GetFileAttributes crash_reports\reports\13d94c0c-e8f1-4e2e-8a6a-453cedef1930dmp: The system cannot find the file specified. (2)
[14560:38840:20231028,094446.300:ERROR filesystem_win.cc:117] GetFileAttributes crash_reports\reports\cc7931d8-dde0-44ad-96a2-8be55455f1d5dmp: The system cannot find the file specified. (2)

This looks bad
[14560:38840:20231028,094446.301:ERROR filesystem_win.cc:48] RemoveDirectory crash_reports\attachments\cc7931d8-dde0-44ad-96a2-8be55455f1d5: The process cannot access the file because it is being used by another process. (32)

[14560:38840:20231028,094446.301:ERROR filesystem_win.cc:130] GetFileAttributes crash_reports\attachments\a1f8ab84-380c-4fe0-8ac9-0c608cdd36b1: The system cannot find the file specified. (2)
[14560:38840:20231028,094446.301:ERROR filesystem_win.cc:130] GetFileAttributes crash_reports\attachments\ec0ae2b1-e371-4d5c-b27c-e571390db5c4: The system cannot find the file specified. (2)
[14560:38840:20231028,094446.302:ERROR filesystem_win.cc:130] GetFileAttributes crash_reports\attachments\a04ee8cf-52e2-4322-9875-44fbd38e0ad3: The system cannot find the file specified. (2)
[14560:38840:20231028,094446.304:ERROR filesystem_win.cc:130] GetFileAttributes crash_reports\attachments\13d94c0c-e8f1-4e2e-8a6a-453cedef1930: The system cannot find the file specified. (2)