microsoft / dumpling

Source code for dumpling.web a web service for collecting and indexing mini dumps, core dumps and macho dumps, along with related files and diagnostic information

dumpling

Source code for dumpling.web a web service for collecting and indexing mini dumps, core dumps and macho dumps, along with related files and diagnostic information

dumpling's Issues

Turn back on dump triaging for Linux and Windows

Running "python dumping.py install --full" on macOS gives an error

When the install --full command is run on macOS, I get this error:

python dumpling.py install --full
downloaded dumpling.py
downloading debugger for client darwin
Traceback (most recent call last):
  File "dumpling.py", line 1128, in <module>
    main(sys.argv)
  File "dumpling.py", line 1123, in main
    cmdProc.Process(config)
  File "dumpling.py", line 537, in Process
    self.Install(config)
  File "dumpling.py", line 566, in Install
    dbgPath = self._dumpSvc.DownloadDebugger(dbgdir)
  File "dumpling.py", line 184, in DownloadDebugger
    response.raise_for_status()
  File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 929, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://dumpling.azurewebsites.net/api/tools/debug?os=darwin

It's fine if there is no debugging tools for the platform, but the script should handle that internally, IMO.

OSError: [Errno 8] Exec format error

We are hitting some problem in the main dumpling script in some of our CI jobs in corefx. It seems to be related to the debugging / triage portion. Here's a relevant stack trace and the error message:

05:56:53     File 
05:56:53     File "/mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/tempHome/.dumpling/dumpling.py", line 1128, in <module>
05:56:53       main(sys.argv)
"/mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/tempHome/.dumpling/dumpling.py", line 1123, in main
05:56:53     File "/mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/tempHome/.dumpling/dumpling.py", line 529, in Process
05:56:53       self.Upload(config)
05:56:53     File "/mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/tempHome/.dumpling/dumpling.py", line 624, in Upload
05:56:53       self.UploadDump(config)
05:56:53     File "/mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/tempHome/.dumpling/dumpling.py", line 644, in UploadDump
05:56:53       self._triage_dump(dumpid, config.dumppath, config)
05:56:53     File "/mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/tempHome/.dumpling/dumpling.py", line 838, in _triage_dump
05:56:53       CommandProcessor._load_debugger(config.dbgpath, dbgcmds)
05:56:53     File "/mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/tempHome/.dumpling/dumpling.py", line 913, in _load_debugger
05:56:53       proc = subprocess.Popen(procArgs) #, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
05:56:53     File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
05:56:53       errread, errwrite)
05:56:53     File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
05:56:53       raise child_exception
05:56:53   OSError: [Errno 8] Exec format error
05:56:53   Debugger command: /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/tempHome/.dumpling/dbg/bin/lldb -o target create --core /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/bin/Linux.AnyCPU.Release/System.Collections.NonGeneric.Performance.Tests/netcoreapp/core.18519 -o plugin load /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/tempHome/.dumpling/dbg/bin/libsosplugin.so -o command script import /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/tempHome/.dumpling/analysis.py -o analyze -i /mnt/resource/j/workspace/dotnet_corefx/master/outerloop_netcoreapp_portablelinux_release/tempHome/.dumpling/triage.ini -o /tmp/tmpIW5nx_ -o exit

My initial guess is that it is not able to locate lldb. It looks like it defaults to /bin/lldb if not specified, but on this distro (RHEL 7.2), the binary is at /usr/bin/lldb.

I believe the effect of this is that we don't get an analyzed stack trace in our results.

Can't find Microsoft.SymStore.Indexers nuget package

Can't restore Microsoft.SymStore.Indexers nuget package from all sources specified in the solution.

Downloading dumps from internal site takes too long and times out

Occasionally, the download request times out (probably for larger files).

cc @DrewScoggins

ImportError: No module named psutil

OSX dumps fail on both OSX12 and OSX13:

2017-12-11 21:07:20,329: INFO: proc(54): run_and_log_output: Output: /Users/dotnet-bot/dotnetbuild/work/de4c4352-a7e5-442b-8594-bda4af6f7551/Work/788f33cf-7855-4704-b595-584097167a0e/Unzip/RunTests.sh: line 87: 82229 Segmentation fault: 11  (core dumped) $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.ComponentModel.Composition.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonosxtests -notrait category=OuterLoop -notrait category=failing
2017-12-11 21:07:20,546: INFO: proc(54): run_and_log_output: Output: Traceback (most recent call last):
2017-12-11 21:07:20,546: INFO: proc(54): run_and_log_output: Output:   File "/Users/dotnet-bot/.dumpling/dumpling.py", line 33, in <module>
2017-12-11 21:07:20,546: INFO: proc(54): run_and_log_output: Output:     import psutil
2017-12-11 21:07:20,546: INFO: proc(54): run_and_log_output: Output: ImportError: No module named psutil

eg https://mc.dot.net/#/user/maryamariyan/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/12cbeed85897de05d63181974602cd977ea30966/workItem/System.ComponentModel.Composition.Tests/wilogs

Should dumpling install this or do we need to update images?

@Anipik @DrewScoggins

Dumpling fails with AttributeError: DumplingConfig instance has no attribute 'dbgpath'

Moved over from https://github.com/dotnet/corefx/issues/21225. It seems like dumpling is successfully uploading the crash dump, but is also trying to do extra triage steps with tools that aren't available. Dumpling should just detect that those tools aren't available and skip that step.

Produced dumps do not have symbols

Currently dumpling does not produce actionable output because symbols are missing and thus debugging using the dump is not feasible. The dumpling also does not point to a build or hash commit which could be used to produce them.

Longer conversation about this: https://github.com/dotnet/corefx/issues/16988

From @schaabs

you can upload any associated symbols when you upload the dumps using the --incpaths flag

From @mellinoe

We archive all of the product binaries in Jenkins, but don't upload them to Dumpling. Jenkins only keeps them around for a week or so, but if this is happening at least once per week anyways, then we should be able to get some matching binaries and dumps, right? We don't archive the test binaries / assemblies, but we could change that, and I think we could still gather useful info without them.

Example dumps which do not contain symbols:
https://dumpling.azurewebsites.net/90?FAILURE_HASH=SIGSEGV_libcrypto.so.1.0.0!BN_num_bits

The list of dumps should be grouped together by one of PR/commit hash/Job/Build#/etc.

From the list of dumps, it is hard to figure out which dump belongs to your PR/change. The list should be grouped together by some sort of id which maps your change to the set of dumps (for various platforms/OSes). This will also make it easier to scroll through the list.
This should work for both CI jobs (where we have either PR# or commit hash), and for official builds where we have a build number.