bluebrain / bbp-nixpkgs

Blue Brain Project nixpkgs configuration - Build a brain with Nix

BBP NIXPKGS

Contains all the nix expressions for the Blue Brain Project.

• See pkgs/top-level/all-packages.nix for a complete list of supported software.

• See pkgs/bbp/* for the list of the specific BBP softwares.

For any information about nix usage in the BBP, please check https://bbpteam.epfl.ch/project/spaces/display/BBPHPC/Nix+Package+Manager

Version Update Workflow

If you have updated software package and would like to deploy new version via nix, follow below steps :

Step 1

Clone bbp-nixpkgs repository :

git clone https://github.com/BlueBrain/bbp-nixpkgs.git

Step 2

Update revision-id for your software package. For example, for ReportingLib you have to edit below section in bbp-nixpkgs/bbp/hpc/neurodamus/default.nix :

  src = fetchgitExternal {
    url = "ssh://bbpcode.epfl.ch/sim/reportinglib/bbp";
    rev = "98f5b5869ad3a2c741e847a658d2bb75986ed05f";
    sha256 = "1a3p7rl2ic1x3g0xhzxh8psbq6z97qvxs60r3zlsc08y83fink4f";
  };

Replace rev with new commit-id. You have to also update sha256 (just update few characters for now).

Step 3

Now execute below command to get correct SHA :

cd bbp-nixpkgs
nix-build ./ -A reportinglib.src

You will get an error message like :

output path ‘/gpfs/bbp.cscs.ch/apps/bgq/nix/nix-root/store/rl1yi89nh-bbp-fb5c1b5’ has r:sha256 hash ‘1m87s31wjyv5kz33pv9s139mbyn3j1bxgj2ivgy1dgcmcc7lr1xq’ when ‘10g0rm3l0qbgk7ylqsa38bpylnz4329ckr7y42rax586ay7zp51g’ was expected

Copy the hash (1m87s31wjyv5k..... in this case) and replace sha256 filled in Step 2.

Step 4

If you have multiple packages to update, update them as above.

Step 5

Now build your software package :

nix-build ./ -A reportinglib

Step 6

Once everything is built successfully, commit the changes and submit a pull request on Github.

Info

Note: Based on a direct fork of the nixos nixpkgs branch https://nixos.org/nixos

Funding

The development of this software was supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology.

Copyright (c) 2016-2022 Blue Brain Project/EPFL

bbp-nixpkgs's Issues

Pickleshare version BluePyOpt

I just saw this line:
https://github.com/BlueBrain/bbp-nixpkgs/blob/master/bbp/nse/bluepyopt/default.nix#L20
It seems it's there to pin the version of pickleshare to a very old version.

Could you upgrade the pickleshare version to >= 0.7.3

The reason this pickleversion of >=0.7.3 is required in bluepyopt is because of this PR:
ipython/pickleshare#23
Older versions can create a race condition causing optimisations to fail from time to time in a stochastic way.

log test from jean-qu'un

Hello,
It will be nice to catch the log of the test for the continuous integration. For this I need the path of the log ... Where is it ?
Best,
Tim

Cannot pip install anything after loading nix/nse/bluepy

Hi,

it seems that it is not currently possible to pip install any package after module loading bluepy in bb5. This is not the case if I load other modules.

module load nix/python/2.7-full
virtualenv blah
source blah/bin/activate
# after this I can still pip install packages
module load nix/nse/bluepy
# every pip install ends up with a KeyError

Exception:
Traceback (most recent call last):
  File "/gpfs/bbp.cscs.ch/home/zisis/blah/lib/python2.7/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/gpfs/bbp.cscs.ch/home/zisis/blah/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 318, in run
    self._warn_about_conflicts(to_install)
  File "/gpfs/bbp.cscs.ch/home/zisis/blah/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 442, in _warn_about_conflicts
    package_set, _dep_info = check_install_conflicts(to_install)
  File "/gpfs/bbp.cscs.ch/home/zisis/blah/lib/python2.7/site-packages/pip/_internal/operations/check.py", line 89, in check_install_conflicts
    state = create_package_set_from_installed()
  File "/gpfs/bbp.cscs.ch/home/zisis/blah/lib/python2.7/site-packages/pip/_internal/operations/check.py", line 39, in create_package_set_from_installed
    retval[name] = PackageDetails(dist.version, dist.requires())
  File "/gpfs/bbp.cscs.ch/home/zisis/blah/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2613, in requires
    dm = self._dep_map
  File "/gpfs/bbp.cscs.ch/home/zisis/blah/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2864, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/gpfs/bbp.cscs.ch/home/zisis/blah/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2873, in _compute_dependencies
    for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
  File "/gpfs/bbp.cscs.ch/home/zisis/blah/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2855, in _parsed_pkg_info
    metadata = self.get_metadata(self.PKG_INFO)
  File "/gpfs/bbp.cscs.ch/home/zisis/blah/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1794, in get_metadata
    raise KeyError("No metadata except PKG-INFO is available")
KeyError: 'No metadata except PKG-INFO is available'

fetchBBPDevpi and firewall maddness

ping @arsenius7

I just found out a big issue with fetchBBPDevpi in fact.

• bbpcode where the package where coming from before is open to the internet with authentication
• The devpi server is not for security reason

This is a major issue because it makes the installation on every site outside BBP (All HBP machines) fails and that is not acceptable.

We will need to fix that in some way.

glm recipie

Hi, the glm-0.9.6.1 recipe seems only to unzip the header files + doc and does not perform actual installation. Could we have upstream version (glm-0.9.8.5) which seems to have fixed it?

Please let me know what is the procedure to update base packages.

glm-0.9.6.1: https://github.com/adevress/nixpkgs-base/blob/5ef8c9ab8df79d867221d48c61f34052aff190e7/pkgs/development/libraries/glm/default.nix

Missing package in nix/python/2.7-full

On bb5, the nix/python/2-7-full module is missing future package:

bokeh 0.12.15 requires futures>=3.0.3, which is not installed.

Nrn patch breaks reproducibility

This patch:

bbp-nixpkgs/bbp/hpc/neuron/nrn.patch

Line 18 in cd3d1b9

- TABLE minf, mtau, hinf, htau, ninf, ntau DEPEND celsius FROM -100 TO 100 WITH 200

breaks the reproducibility of the results of this channel.

Is this change there because of CoreNEURON ?

Could we fix this in some way, because if we install NEURON using nix on other machines it might break the reproducibility of people's scripts.

E.g. the tutorials we are presenting at conferences use this channel.

git pkg does not work

For some reason, I have the following error message when I run git pull with nix git.

$ git pull
No user exists for uid XXXXX
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Maybe future myself will try to debug that...

BGLibPy patch hinders reproducibility

Could you please remove / find a solution for the following patch:

bbp-nixpkgs/bbp/nse/bglibpy/patch_version.patch

Line 509 in d4dfc8f

+ "full-revisionid": "4658871de80563463dc946fed57013e0a72eb137",

This will easily become outdated. As it is already now, it points to the wrong date and the wrong git hash.
It hinders reproducibility for people that store the version number of BGLibPy in their scripts.

I guess an obviously solution would be to just use the devpi server to install bglibpy.

Update pandas

Hi @tristan0x,

For one of my package I need pandas 0.23. I could override the pandas default.nix by creating a new one in the patches folder but I have seen that master branch of nix-pkgs already have it so we could also upgrade the nix-pkgs submodule. What do you think ?

Thanks

szip fails to download/build

Build log:

❯ bbp-nixpkgs mpi-parquet-converters* matwolf@bbpv2 nix-build -A szip
these derivations will be built:
  /nix/store/dmhbx3fw12cmcz8a1pycvbrrc2lzd30c-szip-2.1.1.tar.gz.drv
  /nix/store/m1z3phyrdvpdkdiacw67dwlkykm5y5ss-szip-2.1.1.drv
building path(s) ‘/nix/store/2v0c8yvq96p1bn1isv3aimrgq6kq45gl-szip-2.1.1.tar.gz’

trying ftp://ftp.hdfgroup.org/lib-external/szip/2.1.1/src/szip-2.1.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Warning: Transient problem: timeout Will retry in 1 seconds. 3 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Warning: Transient problem: timeout Will retry in 2 seconds. 2 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Warning: Transient problem: timeout Will retry in 4 seconds. 1 retries left.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (28) Timeout was reached
error: cannot download szip-2.1.1.tar.gz from any mirror
builder for ‘/nix/store/dmhbx3fw12cmcz8a1pycvbrrc2lzd30c-szip-2.1.1.tar.gz.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/m1z3phyrdvpdkdiacw67dwlkykm5y5ss-szip-2.1.1.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/m1z3phyrdvpdkdiacw67dwlkykm5y5ss-szip-2.1.1.drv’ failed

Incomplete neuron version

When I load the nix neuron module (nix/hpc/neuron/7.5-201707 on CSCS viz), it shows me the following version:

NEURON -- VERSION 7.5 master (NO_GIT) 201707

When I do the same in a git version of neuron compiled the normal way, I get something like:
NEURON -- VERSION 7.5 (1469+:60a58922f20e+) 2016-10-24

As you can see the date format is different and we get a reference to the git hash.

For provenance we need to be able to let scripts save the git hash of the exact neuron version.
Also BGLibPy depends on the date format, to switch between certain functionality (that only works in newer versions of Neuron).

Could you make sure that the nix neuron package picks up the correct git hash and date ?

@elisabettai and @christian-oreilly pointed me to this problem.

hpcbench environment module does not ship dependencies

bensh should be usable after executing command module load hpcbench. This is not the case before all hpcbench dependencies are not loaded.

libFLATIndex nix pkg is not working with python distutils

Hi,

I am trying to create the nix recipe of projectionizer, a python package that depend on libFLATIndex. However, even if I specify flatindexer as a dependency in the recipe, it is not being found at install time when the setup.py of projectionizer is being executed:

installing
/tmp/nix-build-python2.7-projectionizer-1.4.8.dev4.drv-0/projectionizer/dist /tmp/nix-build-python2.7-projectionizer-1.4.8.dev4.drv-0/projectionizer
Processing ./projectionizer-1.1.0.dev3-py2-none-any.whl
Requirement already satisfied: voxcell>=2.3 in /nix/store/2x8kafdvl2vbds9ak94kyg519c6xrmax-python2.7-voxcell-2.5.1/lib/python2.7/site-packages (from projectionizer==1.1.0.dev3)
Requirement already satisfied: partd>=0.3 in /nix/store/8v0nzyf8vfkxhyg331c19b8arvd4c4w1-python2.7-partd-0.3.7/lib/python2.7/site-packages (from projectionizer==1.1.0.dev3)
Collecting libFLATIndex>=1.8 (from projectionizer==1.1.0.dev3)
  Could not find a version that satisfies the requirement libFLATIndex>=1.8 (from projectionizer==1.1.0.dev3) (from versions: )

I'll add that during the build libFLATIndex is being listed in the python sys.path at this location:
/nix/store/s9hin8v7lflfpjrxp1niplwcl45yxkzs-flatindexer-1.8.8/lib/python2.7/site-packages and if I do nix-shell -A projectionizer and python -c "import libFLATIndex", it works as expected. So It seems to be just a distutils discovery problem.

My guess is that the flatindexer package is missing the egg-info or dist-info folder that would make it a real python package.

A simpler but equivalent way to reproduce the problem is the following:
Open a nix-shell: nix-shell -p python pythonPackages.pip neurom flatindexer
pip freeze | grep neurom will find neurom while pip freeze | grep -i flat won't find flatindexer

Provides pyapache Python package

@arsenius7 needs this package to use feather-format Python package.
Here are some hints to build the Nix recipe ... https://arrow.apache.org/docs/python/development.html#development