airspeed-velocity / asv Goto Github PK

View Code? Open in Web Editor NEW

844.0 844.0 176.0 4.29 MB

Airspeed Velocity: A simple Python benchmarking tool with web-based reporting

Home Page: https://asv.readthedocs.io/

License: BSD 3-Clause "New" or "Revised" License

Python 82.42% CSS 0.49% JavaScript 14.50% HTML 1.13% C++ 1.33% Shell 0.13%

airspeed-velocity benchmark python

asv's People

Contributors

Stargazers

Watchers

Forkers

keflavich astrofrog mdboom cdeil aragilar danielballan chrisbeaumont adamfeuer minrk ev-br itziakos pitrou rhattersley matthewturk qwhelan giltis pv rgommers argriffing rmcgibbo waylonflinn charris souravsingh hamogu yoavram kshedden thpfaff jhunkeler johntyree pombredanne jakirkham jmemich andyfaff philpep multiloc moorepants arne-cl dowan craig8 ankravch wrwrwr jsoref rscohn2 postelrich pybenchmark qunaibit karenyyng oolongtea alexbw yarikoptic tomaugspurger octobus-net warbo anntzer ovillellas asmeurer vreuter colcarroll qulogic jbrockmendel daerwang ttong1013 abeelen theodoregoetz jayachandra-9 ray-project jni sylvaincorlay astrojuanlu vishalbelsare leandron marmoute isuruf rlishtaba germanjoey msarahan pierreglaser prisae eteq egpbos maxalbert mmohrhard jeremiedbb brianhendee timoroth rok serge-sans-paille kdk danielsoutar miss-jain-16 mtreinish joseph-zhong payno riturajsingh878 luizirber owena11 atilag kevingerman omarwagih1 cottrell

asv's Issues

Allow custom setup commands

I would like to use asv to test for performance regressions in a code of mine (http://www.hyperion-rt.org/) but the issue is that to set it up, one needs to do:

./configure
make
make install
python setup.py install

Is there a way to do this? (the code is 50% Fortran, which gets built with make, and 50% Python, and it is run via the Python interface).

Version the results and config files

So we can upgrade their format later, if possible.

Problem with virtualenv

As part of #43, @astrofrog reported this:

@mdboom - there is one more issue that I'm not sure how to address. When it installs astropy, it detects the system installed version and tries to uninstall it and fails:

Unpacking ./astropy
  Running setup.py egg_info for package from file:///Volumes/Raptor/asv/astropy-benchmarks/astropy
    Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.28.tar.gz
    Extracting in /var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/tmppstoj6
    Now working in /var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/tmppstoj6/distribute-0.6.28
    Building a Distribute egg in /private/var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/pip-npe8zj-build
    RefactoringTool: Skipping implicit fixer: buffer
    RefactoringTool: Skipping implicit fixer: idioms
    RefactoringTool: Skipping implicit fixer: set_literal
    RefactoringTool: Skipping implicit fixer: ws_comma
    [snip]
    /private/var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/pip-npe8zj-build/distribute-0.6.28-py3.3.egg
    Freezing version number to astropy/version.py

    warning: no files found matching 'astropy/utils/misc/data/.hidden_file.txt'
    warning: no previously-included files found matching '*.pyc'
    warning: no previously-included files found matching '*.o'
    no previously-included directories found matching 'docs/_build'
    no previously-included directories found matching 'build'
Requirement already satisfied (use --upgrade to upgrade): numpy in /Volumes/Raptor/Library/Python/3.3/lib/python/site-packages (from astropy==0.3.dev4071)
Installing collected packages: astropy
  Found existing installation: astropy 0.2.5
    Uninstalling astropy:
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/shutil.py", line 522, in move
    os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: '/var/folders/zq/64w93jcj22366sbdv9wqvh6c0000gq/T/pip-y9dtaw-uninstall/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/astropy'

I'm surprised it even knows about the system version since it's supposed to be in a virtualenv, rig

Auto generate the command documentation from the argparse specs themselves

Generalize the repository tool

Right now, we assume you're using git, but there's no reason it couldn't work with other DVCSs, such as Mercurial.

Most of the time spent running the benchmarks is actually spent building and installing the project. This part could be easily parallelized. (The benchmarking itself can not, as that introduces too much indeterminancy.)

Option for log scaling

Maybe you could add an option to show the speeds on a log scale as in some cases we may want to be looking at improvements of more than an order of magnitude?

conda environment gets created twice?

It looks from the log as if the conda environment gets created twice:

mac-robitaille2:wcsaxes-benchmarks tom$ asv run
· Fetching recent changes..
· Creating environments
·· Creating conda environment for py3.4-astropy0.4.0-matplotlib1.3.1-numpy1.8...
· Installing dependencies
·· Upgrading matplotlib=1.3.1 in py3.4-astropy0.4.0-matplotlib1.3.1-numpy1.8...
·· Upgrading numpy=1.8 in py3.4-astropy0.4.0-matplotlib1.3.1-numpy1.8..
·· Upgrading astropy=0.4.0 in py3.4-astropy0.4.0-matplotlib1.3.1-numpy1.8..
· Discovering benchmarks
·· Fetching recent changes..
·· Creating conda environment for py3.4-astropy0.4.0-matplotlib1.3.1-numpy1.8...
·· Upgrading matplotlib=1.3.1 in py3.4-astropy0.4.0-matplotlib1.3.1-numpy1.8..
·· Upgrading numpy=1.8 in py3.4-astropy0.4.0-matplotlib1.3.1-numpy1.8.
·· Upgrading astropy=0.4.0 in py3.4-astropy0.4.0-matplotlib1.3.1-numpy1.8..
·· Uninstalling wcsaxes from py3.4-astropy0.4.0-matplotlib1.3.1-numpy1.8
·· Installing wcsaxes into py3.4-astropy0.4.0-matplotlib1.3.1-numpy1.8..

is this normal?

Add standard .gitignore to quickstart template

.asv-machine should be better associated with the machine

Since it isn't correct in a shared home directory environment.

Show tags in plot

It would be handy to mark where certain version tags are on the plot.

Python 2.6 doesn't work

Due to shortcomings in unittest module. When we stop using the unittest module, this should probably resolve itself.

Have memory benchmarks do something useful.

Right now, they just return sys.getsizeof, which isn't terribly meaningful.

Database editing

We need some basic command line tools for editing the database.

For example, removing all entries from a particular machine, removing entries before a particular date/commit.

Add axes labels

The y axis is "seconds", x axis is "date", but I can't find a way to add those labels in flot...

Add timeout

If a benchmark enters an infinite loop, things will hang. We should have a time limit (project customizable) after which we forcibly terminate the subprocess and move along.

Write documentation

IPython cluster support?

Maybe we can leverage IPython to run this on a cluster... For good results, however, it will need to be on homogeneous hardware, I suppose.

Build and install pythons ourself

Right now, asv assumes you have python executables on the path for all of the versions of Python that need to be tested.

We could build the Pythons ourself, perhaps by delegating to pythonbrew or tox.

Handle resizing the window

When the window is resized, the plots are not...

Syntax highlight code snippet

In the spirit of being a set of static webpages, ideally this would be done in Javascript.

Display source code of benchmark in interface

There should be a way to easily display the source code of a particular benchmark in the web interface.

Normalize some names

There is some inconsistent terminology already:

Line/Graph

Params/Requirements/Permutation

Reduce amount of bootstrap

It should be possible to use the bootstrap customize tool to reduce the amount we include.

Benchmark the installation itself

Just a random thought (feel free to close this if you disagree) - what about benchmarking the installation time for the package?

Use something other than `unittest` for benchmark discovery

unittest was convenient to get things going, but we should probably use a different mechanism to discover benchmarks that's less verbose, such as what py.test does.

Create cron-friendly benchmark command

To facilitate running benchmarks as part of a cron job, we should have a tool that keeps track of the last time it was run and chooses commits since then.

Add "publish on github" command

In one step, do publish and then commit the result to the gh-pages branch.

Create a way to download profiles

It's non-trivial, since profile files are not portable -- they contain paths to source files specific to a particular machine.

How to register a custom Environment?

I have a custom environment that I'd like to use in asv. The plugin is discovered and imported by asv. However, it looks like asv chooses an Environment class by running util.iter_subclasses(Environment), and using the first class for which subclass.mathces(...) is True.

Because my custom environment shows up last in Environment.__subclasses__(), it is never selected. What is the best way around this?

ValueError: Expecting property name enclosed in double quotes

On my system (Python 2.7, MacOS 10.8), the json parser doesn't seem to like the matrix entry:

{
    // The name of the project being benchmarked
    "project": "astropy",

    // The project's homepage
    "project_url": "http://www.astropy.org/",

    // The URL of the source code repository for the project being
    // benchmarked
    "repo": "",

    // The Pythons you'd like to test against.  If not provided, defaults
    // to the current version of Python used to run `asv`.
    "pythons": ["2.7"],

    // The matrix of dependencies to test.  Each key is the name of a
    // package (in PyPI) and the values are version numbers.  An empty
    // list indicates to just test against the default (latest)
    // version.
    "matrix": {
        "numpy": ["1.7"]
    }
}

gives (when doing asv run):

Traceback (most recent call last):
  File "/Users/tom/Library/Python/2.7/bin/asv", line 9, in <module>
    load_entry_point('asv==0.1', 'console_scripts', 'asv')()
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/asv-0.1-py2.7.egg/asv/main.py", line 39, in main
    args.func(args)
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/asv-0.1-py2.7.egg/asv/commands/run.py", line 41, in run
    conf = Config.from_file(args.config)
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/asv-0.1-py2.7.egg/asv/config.py", line 44, in from_file
    d = util.load_json(path)
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/asv-0.1-py2.7.egg/asv/util.py", line 246, in load_json
    return json.loads(content)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 381, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting property name enclosed in double quotes: line 18 column 5 (char 162)

whereas

{
    // The name of the project being benchmarked
    "project": "astropy",

    // The project's homepage
    "project_url": "http://www.astropy.org/",

    // The URL of the source code repository for the project being
    // benchmarked
    "repo": "",

    // The Pythons you'd like to test against.  If not provided, defaults
    // to the current version of Python used to run `asv`.
    "pythons": ["2.7"]
}

gives:

ValueError: No repo specified in asv.conf.json config file.

which is normal.

Use Javascript timestamps everywhere

Since timestamps are platform specific, we should convert all timestamps to Javascript timestamps before storage.

Add a manpage

Clicking on a data point opens up tons of tabs in Safari

At http://mdboom.github.io/astropy-benchmark/ I clicked on a data point in Safari.

This opened up astropy/astropy@0865bcf once and then
http://github.com/astropy/astropy/commit/0865bcf1ce4a6a444b651a3963788223b3bb4603 about 10 times.

I tried again with some other data point and many more tabs opened up and Safari became unresponsive.

In Chrome It seems to work fine.

@mdboom Can you reproduce the issue or should I try to debug?

Add basic test suite

Create way to parameterize benchmarks

It's probably handy to do:

def time_size(n):
    x = [0] * n
time_size.params = [10, 1000, 100000]

Support Autotools build

I'd like to use asv for GammaLib, which is a C++ library with a SWIG Python wrapper.
The installation after git checkout is basically

./autogen.sh
./configure --prefix=...
make
make install

Then a setup script must be sourced and then it's possible to import gammalib.

How much work would it be to support such an install with asv and what needs to be done?
Is there any change that the memory measurements are meaningful if pretty much all of the objects are going to be SWIG-wrapped C++ objects?

Comment code

I mean, come on!

Timing on Mac OSX is wonky

The standard deviation of the results is really high. Perhaps we need to be nice...

Rationalize Results

There should be a ResultSet class (for a single commit hash and environment) and a Results class (for handling all results).

Make simple Python API...

...that mirrors the commandline API.

Hovering doesn't work in log scale

Error when using with Conda: Symbol not found: __PyInt_AsInt

I'm seeing the error below. I think the key line is this one:

ImportError: dlopen(/Users/ketch/pyclaw_asv/env/py2.7/lib/python2.7/lib-dynload/_socket.so, 2): Symbol not found: __PyInt_AsInt

Tried googling it, but couldn't determine the cause. I have no idea if this is caused by something in asv or by something on my own system.

» asv run
· Fetching recent changes
· Creating virtualenvs
·· Creating virtualenv for py2.7
·· Error running /Users/ketch/anaconda/bin/python2.7 /Users/ketch/anaconda/lib/python2.7/site-packages/virtualenv-1.11.6-py2.7.egg/virtualenv.py --no-site-packages env/py2.7
             STDOUT -------->
             New python executable in env/py2.7/bin/python2.7
             Also creating executable in env/py2.7/bin/python
             Installing setuptools, pip...
               Complete output from command /Users/ketch/pyclaw_.../py2.7/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
               Traceback (most recent call last):
               File "<string>", line 1, in <module>
               File "/Users/ketch/anaconda/lib/python2.7/site-packages/virtualenv-1.11.6-py2.7.egg/virtualenv_support/pip-1.5.6-py2.py3-none-any.whl/pip/__init__.py", line 9, in <module>
               File "/Users/ketch/anaconda/lib/python2.7/site-packages/virtualenv-1.11.6-py2.7.egg/virtualenv_support/pip-1.5.6-py2.py3-none-any.whl/pip/log.py", line 8, in <module>
               File "/Users/ketch/anaconda/lib/python2.7/site-packages/virtualenv-1.11.6-py2.7.egg/virtualenv_support/pip-1.5.6-py2.py3-none-any.whl/pip/backwardcompat/__init__.py", line 66, in <module>
               File "/Users/ketch/anaconda/lib/python2.7/urllib2.py", line 94, in <module>
                 import httplib
               File "/Users/ketch/anaconda/lib/python2.7/httplib.py", line 71, in <module>
                 import socket
               File "/Users/ketch/anaconda/lib/python2.7/socket.py", line 47, in <module>
                 import _socket
             ImportError: dlopen(/Users/ketch/pyclaw_asv/env/py2.7/lib/python2.7/lib-dynload/_socket.so, 2): Symbol not found: __PyInt_AsInt
               Referenced from: /Users/ketch/pyclaw_asv/env/py2.7/lib/python2.7/lib-dynload/_socket.so
               Expected in: dynamic lookup

             ----------------------------------------
             ...Installing setuptools, pip...done.
             STDERR -------->
             Traceback (most recent call last):
               File "/Users/ketch/anaconda/lib/python2.7/site-packages/virtualenv-1.11.6-py2.7.egg/virtualenv.py", line 2338, in <module>
                 main()
               File "/Users/ketch/anaconda/lib/python2.7/site-packages/virtualenv-1.11.6-py2.7.egg/virtualenv.py", line 824, in main
                 symlink=options.symlink)
               File "/Users/ketch/anaconda/lib/python2.7/site-packages/virtualenv-1.11.6-py2.7.egg/virtualenv.py", line 992, in create_environment
                 install_wheel(to_install, py_executable, search_dirs)
               File "/Users/ketch/anaconda/lib/python2.7/site-packages/virtualenv-1.11.6-py2.7.egg/virtualenv.py", line 960, in install_wheel
                 'PIP_NO_INDEX': '1'
               File "/Users/ketch/anaconda/lib/python2.7/site-packages/virtualenv-1.11.6-py2.7.egg/virtualenv.py", line 902, in call_subprocess
                 % (cmd_desc, proc.returncode))
             OSError: Command /Users/ketch/pyclaw_.../py2.7/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 1

·· Failure creating virtualenv for py2.7Traceback (most recent call last):
  File "/Users/ketch/anaconda/bin/asv", line 9, in <module>
    load_entry_point('asv==0.1', 'console_scripts', 'asv')()
  File "/Users/ketch/anaconda/lib/python2.7/site-packages/asv-0.1-py2.7.egg/asv/main.py", line 35, in main
    result = args.func(args)
  File "/Users/ketch/anaconda/lib/python2.7/site-packages/asv-0.1-py2.7.egg/asv/commands/__init__.py", line 42, in run_from_args
    return cls.run_from_conf_args(conf, args)
  File "/Users/ketch/anaconda/lib/python2.7/site-packages/asv-0.1-py2.7.egg/asv/commands/run.py", line 103, in run_from_conf_args
    profile=args.profile
  File "/Users/ketch/anaconda/lib/python2.7/site-packages/asv-0.1-py2.7.egg/asv/commands/run.py", line 142, in run
    environments = Setup.run(conf=conf, parallel=parallel)
  File "/Users/ketch/anaconda/lib/python2.7/site-packages/asv-0.1-py2.7.egg/asv/commands/setup.py", line 67, in run
    env.setup()
  File "/Users/ketch/anaconda/lib/python2.7/site-packages/asv-0.1-py2.7.egg/asv/environment.py", line 153, in setup
    self._path])
  File "/Users/ketch/anaconda/lib/python2.7/site-packages/asv-0.1-py2.7.egg/asv/util.py", line 198, in check_call
    display_error=display_error, shell=shell)
  File "/Users/ketch/anaconda/lib/python2.7/site-packages/asv-0.1-py2.7.egg/asv/util.py", line 297, in check_output
    raise ProcessError(args, retcode, stdout, stderr)
asv.util.ProcessError: Command '/Users/ketch/anaconda/bin/python2.7 /Users/ketch/anaconda/lib/python2.7/site-packages/virtualenv-1.11.6-py2.7.egg/virtualenv.py --no-site-packages env/py2.7' returned non-zero exit status 1

Add multiple verbose levels
Move stuff out of lower levels if possible

Minify the generated Javascript

If it really matters...

Add more line colors

The flot default of a cycle of 4 is not enough.

airspeed-velocity / asv Goto Github PK

asv's People

Contributors

Stargazers

Watchers

Forkers

asv's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs