intel-ai / pysamprof

Simple Python sampling profiler

PROJECT NOT UNDER ACTIVE MANAGEMENT

This project will no longer be maintained by Intel.

Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.

Intel no longer accepts patches to this project.

If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the open source software community, please create your own fork of this project.

Contact: [email protected]

Sampling Profiler for Python

This tool allows to gather statistical profile of CPU usage of mixed native-Python code. Currently supported platforms are Windows and Linux, x86_64 only.

Quick usage guide for Ubuntu 18.04.

Required for Linux:

• python-dev package
• setuptools python package
• autoconf tool
• libtool
• make tool
• unzip tool
• g++ compiler
• pkg-config tool
• cmake tool

Required for Windows:

• python installed
• cmake
• Visual Studio compilers, version depends on which Python version you want to target, see correct MSVC version

Before building do:

• On Linux:
  • git submodule update --init --recursive
  • cd 3rd_party && ./prereq-build.sh && cd ..
• On Windows:
  • Update third-party intel-xed, mbuild, protobuf, protobuf-c and safestringlib submodules (3rd_party folder):
    • git submodule update --init -- .\3rd_party\intel-xed
    • git submodule update --init -- .\3rd_party\mbuild
    • git submodule update --init -- .\3rd_party\protobuf
    • git submodule update --init -- .\3rd_party\protobuf-c
    • git submodule update --init -- .\3rd_party\safestringlib
  • Apply protobuf-c-vs2008-support.patch (step is required for Python 2 target only):
    • cd 3rd_party\protobuf-c && git apply ..\protobuf-c-vs2008-support.patch && cd ..\..
  • If you have installed VS2017 or VS2019, apply next command (optional step) in the same command line prompt window in which next command will be applied (please note, that path to the vcvars64.bat file may vary by version of VS installed):
    • CALL "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Auxiliary\Build\vcvars64.bat" -vcvars_ver=14.0
  • For Python 2.x run:
    • cd 3rd_party && prereq-build-py2.cmd && cd ..
  • For Python 3.5 or 3.6 run:
    • cd 3rd_party && prereq-build-py3.cmd && cd ..
  • For other versions of 3.x (untested) try fixing the file up specifying correct MSVC version

For building do:

• mkdir pyext/build
• cd pyext/build
• On Windows:
  • For Python 2.x:
    • cmake -G "Visual Studio 9 2008 Win64" -DCMAKE_BUILD_TYPE=Release .. -DPYTHON_EXECUTABLE=path\to\python.exe
  • For Python 3.5 or 3.6:
    • cmake -G "Visual Studio 14 2015 Win64" -DCMAKE_BUILD_TYPE=Release .. -DPYTHON_EXECUTABLE=path\to\python.exe
  • Open generated "pysamprof.sln" with VS2008 (if Python 2) or VS2015 (if Python 3), choose "Release" "x64" as solution configuration
  • Build solution
  • Copy Release\pysamprof.pyd and trace_writer\Release\pysamprof-server.exe to desired location
• On Linux:
  • cmake -DCMAKE_BUILD_TYPE=Release .. -DPYTHON_EXECUTABLE=path/to/python
  • make
  • Copy pysamprof.so and trace_writer/pysamprof-server to desired location
• NOTE: specifying -DPYTHON_EXECUTABLE=path/to/python will force which Python version to compile against; if omitted it will select highest available Python on your system.

To use:

• Add path to location which has pysamprof and pysamprof-server inside to PYTHONPATH
• Do import pysamprof then pysamprof.start(target_path), see pyext/test.py as a quick reference

pysamprof's Issues

Ensure Python 3.6 and newer are supported correctly

Publish pysamprof to PyPI

Prerequisite: #21.

This most likely requires both posting a source package and Python wheels for most popular cases (32- and 64-bit versions; Windows, Linux, Mac; Python 3.6 and 3.7).

Build for Ubuntu is broken.

./prereq-build.sh does not do anything.

Add continuous integration on Travis.

All the checkins should be tested by Travis CI.

Add x86 support

Current code only supports x86_64.

To add x86, it is needed to ensure that all things work correctly when compilied for x86 and add probing for x86 (should be trivial as x86_64 already does "far jumps" in terms of x86 which are called "relative jumps" in libprobengine/prober.c, so it's mostly turning off "very far", or "absolute" jumps).

Modularize `_pysamprof.c`

Currently _pysamprof.c is both the interface to Python and the implementation of some internals. It should be only implementing the functions exposed to Python interpreter, and other things should be done in separate modules.

Add Mac target in all build scripts

Part of #5

Track process spawning on Mac

Part of #5.
On Linux it is done by probing and handling fork and exec.
Also correct handling of pre-fork and past-fork is needed.

Build instructions are incomplete.

Build instructions are not working for Window/VS2017

Converter does not work with Python 3.5.5

(cudf) user@host:/pysamprof/pyext$ PYTHONPATH=../../pysamprof/3rd_party/protobuf/python/ python ../../pysamprof/trace_writer/python/raw2pprof.py --raw ./result/trace/30920.0.raw --symbols ./result/trace/30920.0.symraw --out ./result/pprof_out/trace.pprof --resolve
Traceback (most recent call last):
File "../../pysamprof/trace_writer/python/raw2pprof.py", line 16, in
TYPE_UNDERFINEDF, TYPE_SAMPLE, TYPE_FUNCTION_INFO, TYPE_MAPPING = [struct.unpack('>i', tag)[0] for tag in 'UNDF SMPL FNCI MAPP'.split()]
File "../../pysamprof/trace_writer/python/raw2pprof.py", line 16, in
TYPE_UNDERFINEDF, TYPE_SAMPLE, TYPE_FUNCTION_INFO, TYPE_MAPPING = [struct.unpack('>i', tag)[0] for tag in 'UNDF SMPL FNCI MAPP'.split()]
TypeError: a bytes-like object is required, not 'str'

No instructions on how to use the tool

A few lines in the README would help

Implement auto-import of pysamprof

Currently to make profiling a child process possible that child has to manually do import pysamprof.
By using proper pth file in site-packages it should be possible to automatically import pysamprof upon Python interpreter startup (see https://docs.python.org/3.7/library/site.html for reference).

Wrap certain POSIX syscalls

pysamprof uses signals on POSIX to collect stacks at regular intervals.

According to the spec, incoming signal interrupts a blocking wait of certain system calls (like recv).
To make pysamprof less intrusive it needs to probe these system functions and automatically re-launch them if they were interrupted by profiling signal. Note that those functions shouldn't be restarted if they were interrupted by other signals!

Also note that this most likely applies to both Mac and Linux.

Protobuf patch has trailing whitespace triggering warning.

Applying patch in Windows:
aregm@Barmaley MINGW64 /c/Projects/pysamprof (master)
$ cd 3rd_party/protobuf-c && git apply ../protobuf-c-vs2008-support.patch
../protobuf-c-vs2008-support.patch:36: trailing whitespace.

../protobuf-c-vs2008-support.patch:39: trailing whitespace.

warning: protobuf-c/protobuf-c.h has type 100644, expected 100755
warning: protoc-c/c_message.cc has type 100644, expected 100755
warning: 2 lines add whitespace errors.

Create pip-installable package

Make pysamprof a proper pip-installable package, so that end user can type pip install pysamprof and have it installed.

Add Mac support

pysamprof currently neither works nor can be built on a Mac.

To enable Mac support it is needed to:

• Add Mac target in all build scripts
• Add parsing memory regions on a running process (now done by parsing /proc/self/maps on Linux)
• Add thread following (now done by probing pthread_create and friends on Linux)
• Add correct tracking of process spawning (similar to how fork and exec are probed and handled)
• Make sure native stack unwinding works via libunwind or find a Mac-specific replacement

Enable native stack unwinding on Mac

Part of #5.
Make sure native stack unwinding works via libunwind or find a Mac-specific replacement.

Add parsing memory regions on a running process on Mac

Part of #5.

For reference, on Linux it is now done by parsing /proc/self/maps.
Also needs to report loaded modules of a current process (with also intercepting similarly to how dlopen is probed) to know when a new module is added to the address space.

Add thread following for Mac

Part of #5.

On Linux now done by probing pthread_create and friends.

cmake failures on Ubuntu 20.04

While executing cmake -DCMAKE_BUILD_TYPE=Release .. -DPYTHON_EXECUTABLE=/usr/bin/python3 on Ubuntu 20.04 I get the following errors:

-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /usr/bin/python3 (found version "3.8.2")
-- detected CFLAGS for Python extensions: -fPIC
-- detected LDFLAGS for Python extensions: -Wl,-Bsymbolic-functions  -Wl,-z,relro -g -fwrapv -O2
-- libunwind needs lzma on this platform
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.8.so (found version "3.8.2")
-- Configuring done
CMake Error at /nfs/site/home/gashiman/work/pysamprof/common/CMakeLists.txt:33 (add_library):
  Target "pysamprof-common" links to item "-Wl,-Bsymbolic-functions
  -Wl,-z,relro -g -fwrapv -O2 " which has leading or trailing whitespace.
  This is now an error according to policy CMP0004.


CMake Error at /nfs/site/home/gashiman/work/pysamprof/common/CMakeLists.txt:33 (add_library):
  Target "pysamprof-common" links to item "-Wl,-Bsymbolic-functions
  -Wl,-z,relro -g -fwrapv -O2 " which has leading or trailing whitespace.
  This is now an error according to policy CMP0004.


CMake Error at /nfs/site/home/gashiman/work/pysamprof/common/CMakeLists.txt:33 (add_library):
  Target "pysamprof-common" links to item "-Wl,-Bsymbolic-functions
  -Wl,-z,relro -g -fwrapv -O2 " which has leading or trailing whitespace.
  This is now an error according to policy CMP0004.


CMake Error at /nfs/site/home/gashiman/work/pysamprof/common/CMakeLists.txt:33 (add_library):
  Target "pysamprof-common" links to item "-Wl,-Bsymbolic-functions
  -Wl,-z,relro -g -fwrapv -O2 " which has leading or trailing whitespace.
  This is now an error according to policy CMP0004.


CMake Error at /nfs/site/home/gashiman/work/pysamprof/common/CMakeLists.txt:33 (add_library):
  Target "pysamprof-common" links to item "-Wl,-Bsymbolic-functions
  -Wl,-z,relro -g -fwrapv -O2 " which has leading or trailing whitespace.
  This is now an error according to policy CMP0004.


CMake Error at /nfs/site/home/gashiman/work/pysamprof/common/CMakeLists.txt:33 (add_library):
  Target "pysamprof-common" links to item "-Wl,-Bsymbolic-functions
  -Wl,-z,relro -g -fwrapv -O2 " which has leading or trailing whitespace.
  This is now an error according to policy CMP0004.


-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

The problem is in Python configuration files so I ended up editing /usr/lib/python3.8/_sysconfigdata__x86_64-linux-gnu.py and removing all whitespace after -O2 where I found this flag (I am not sure exactly which lines goes into cmake config). But I don't think this is a correct way of handling this problem. Here is an answer to a similar problem https://stackoverflow.com/a/51367125/4584975 explaining how to strip whitespace from cmake config variables. I think it should be added to CMakeLists.txt.

Deprecate Python 2 support

pysamprof now supports both Python versions.
Most likely adding support for newer versions or other platforms (like Mac) won't break Python 2 support, but, as Python 2 is being deprecated in Jan 2020, it makes sense to deprecate its support (and maybe even remove said support to reduce the amount of ifdef-s around the code).