Comments (12)
@lukasheinrich I'm not sure what can be done here, as we're already using Miniconda (which just installs an empty conda environment, right?). The only package that we're installing beyond PyTorch with conda is scipy
conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION pytorch scipy -c pytorch
so I can remove that and just have pip
install it, but I don't know what else can really be done if we still use conda. I can look at seeing if there is an alternative way to get PyTorch in without conda though, but I think that will probably be equally slow.
from pyhf.
Hi @matthewfeickert yeah.. I'm not sure exactly what can be done (and if nothing can be done that's fine).. I was just looking at the travis logs and saw
PREFIX=/home/travis/miniconda
installing: python-3.6.3-h6c0c0dc_5 ...
Python 3.6.3 :: Anaconda, Inc.
installing: ca-certificates-2017.08.26-h1d4fec5_0 ...
installing: conda-env-2.6.0-h36134e3_1 ...
installing: libgcc-ng-7.2.0-h7cc24e2_2 ...
installing: libstdcxx-ng-7.2.0-h7a57d05_2 ...
installing: libffi-3.2.1-hd88cf55_4 ...
installing: ncurses-6.0-h9df7e31_2 ...
installing: openssl-1.0.2n-hb7f436b_0 ...
installing: tk-8.6.7-hc745277_3 ...
installing: xz-5.2.3-h55aa19d_2 ...
installing: yaml-0.1.7-had09818_2 ...
installing: zlib-1.2.11-ha838bed_2 ...
installing: libedit-3.1-heed3624_0 ...
installing: readline-7.0-ha6073c6_4 ...
installing: sqlite-3.20.1-hb898158_2 ...
installing: asn1crypto-0.23.0-py36h4639342_0 ...
installing: certifi-2017.11.5-py36hf29ccca_0 ...
installing: chardet-3.0.4-py36h0f667ec_1 ...
installing: idna-2.6-py36h82fb2a8_1 ...
installing: pycosat-0.6.3-py36h0a5515d_0 ...
installing: pycparser-2.18-py36hf9f622e_1 ...
installing: pysocks-1.6.7-py36hd97a5b1_1 ...
installing: ruamel_yaml-0.11.14-py36ha2fb22d_2 ...
installing: six-1.11.0-py36h372c433_1 ...
installing: cffi-1.11.2-py36h2825082_0 ...
installing: setuptools-36.5.0-py36he42e2e1_0 ...
installing: cryptography-2.1.4-py36hd09be54_0 ...
installing: wheel-0.30.0-py36hfd4bba0_1 ...
installing: pip-9.0.1-py36h6c6f9ce_4 ...
installing: pyopenssl-17.5.0-py36h20ba746_0 ...
installing: urllib3-1.22-py36hbe7ace6_0 ...
installing: requests-2.18.4-py36he2e5f8d_1 ...
installing: conda-4.3.31-py36_0 ...
installation finished.
which might indeed just be the irreducible minimum for conda
from pyhf.
@lukasheinrich Yeah that looks like the normal list of things that come with a new conda environment. I'll still poke at this though before we decide to close it.
I have some other questions about the CI related to speed but we can go over that later (or in another Issue).
from pyhf.
@lukasheinrich I stand corrected, we can still do it in pip
, as long as we run different install instructions for different versions of Python (they package it as different wheels). Do you have any reference YAML files for how to do this?
from pyhf.
PyTorch is now on PyPI. I’ll do some tests to see what needs to be edited in terms of .travis.yml
and then make a PR to see if we get speedup.
from pyhf.
@lukasheinrich There are currently issues in PR #101 with having conditional requirements in setup.py
to deal with PyTorch having different wheels for different versions of Python in non-GPU environments.
$ pip install -U --process-dependency-links -e .[develop]
Obtaining file:///home/travis/build/diana-hep/pyhf
Complete output from command python setup.py egg_info:
error in pyhf setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.
From searching it seems like this is an issue with setuptools
, but using
before_install:
- pip install --upgrade pip setuptools
in .travis.yml
is still causing things to error.
I don't suppose that you've run across something like this before?
from pyhf.
The above seems to be an issue with setup.py
not being compatible with explicit declarations of wheel urls. This in principle shouldn't be an issues, as pip
should now default to wheel versions of packages if they exist. So I've defaulted just to asking pip
to get torch
.
However, there are still errors and differing errors between Python 2.7 and the Python 3.5/3.6 builds in Travis. Python 2.7 fails to install PyTorch with pip
. Python 3.5/3.6 successfully installs PyTorch, however we get a memory error after all the tests successfully pass
=================== 22 passed, 18 warnings in 99.87 seconds ====================
*** Error in `/home/travis/virtualenv/python3.6.3/bin/python': corrupted size vs. prev_size: 0x0000000002161a90 ***
/home/travis/.travis/job_stages: line 57: 4971 Aborted (core dumped) pytest
The command "pytest" exited with 134.
I'll continue to follow up on this later, as we are seeing a multiple minute speedup over the ~10 minutes it takes for the CI with Conda.
from pyhf.
Hi @matthewfeickert, yeah I think it's fine to pre-install torch.. In the end setup.py
just declares the dependencies. That pip
can install them for you is great, but it's not absolutely required. Does it work locally for you (i.e. is it just an issue with travis?)
from pyhf.
I am running Python 3.6, so I'd have to setup a Python 3.5 Conda environment to check. I will follow up on this in Issue #103.
from pyhf.
@matthewfeickert what happened to this? does the fact that we don't do 3.5 anymore help?
from pyhf.
I had let it slide down the list. Let me look at it again tonight, now that Issue #103 is closed as you point out.
from pyhf.
There seems to be more strangeness going on with memory errors, so unless this is viewed as an important issue I'm going to leave it till later.
from pyhf.
Related Issues (20)
- Pruning with logical "and" instead of "or" HOT 2
- Catch 100% normalization uncertainty modifiers
- Remove `InvalidNameReuse` exception in favor of in `InvalidModel`
- Typing for tensor shapes
- `ValueError: The truth value of an array` ... when using `fixed_params` kwarg for `pyhf.infer.hypotest` HOT 4
- Release v0.7.6 checklist
- Using the hessian matrix in optimization HOT 2
- Typo in docs for qmutilde test-statistic
- Implementation of fixed params in scipy minimizer
- Switch to using 'tensorflow' for Apple silicon Macs for TensorFlow v2.16.1+ HOT 2
- docs linkcheck failing for SciPy conference proceedings
- In Python 3.12 xml.etree.ElementTree will raise DeprecationWarning: Testing an element's truth value will raise an exception in future versions
- In Python 3.12 tarfile raises DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. HOT 1
- Forced measurement object schema validation through `ws.model()` HOT 1
- Inconsistent return types of `model.config.suggested_bounds()` when using parameter config
- Treating arbitrarily correlated parameters with pyhf HOT 3
- toms748_scan doesn't work with JAX backend HOT 6
- xml2json breaks with HistoPath including filename HOT 2
- Move documentation pages to PyData Sphinx Theme HOT 1
- Use PyData Sphinx Theme docs version switcher
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyhf.