Conda / Cython incompatibility: GCC finding different version of Pysam from that in Conda environment (formerly: possible pysam 0.9.1 compatibility issue) about plastid

[joshuagryphon]

Hey dude!

It's great to hear form you, and thank you for the heads up. I can't replicate this on my system with pysam 0.9.1.4 (in or out of a virtualenv), and I know I have Mac users who are using pysam >= 0.9.1.0 outside of Conda (usually in a virtualenv)

I think this is issue is the same set of events that came up in issue 2, in that case with numpy. The TLDR is that Conda has some strange interactions with pip, causing Conda's sandboxes to leak. Overall, I strongly recommend using virtualenv (which plays well with pip, and to my knowledge has raised no problems with plastid, pysam, or numpy) in place of Conda. In a virtuanlenv, I have been able to build Plastid with every version of Pysam >= 0.8.4

The long answer (if what I think is happening actually is) is this:

1. Conda aims to create Python sandboxes within which various packages can be installed and upgraded. This is a noble goal
2. When pip is processing dependencies for setup (from setup.py), it sees the package versions that are present inside the Conda envrionment, as expected
3. When pip actually runs the compiler needed to build plastid, it starts a subshell that exists outside the Conda sandbox. This can cause builds to fail, e.g. because pip will try to link against a version of pysam and/or numpy that might be lower or higher than the version required in and detected by setup.py. Even if the build succeeds and both versions of pysam and/or numpy meet the minimal requirements, you may still get runtime errors if plastid is linked against a different version of numpy or pysam from the ones it finds in the runtime environment

In contrast, If pip is run inside a virtualenv (not a Conda virtual environment), subshells stay within the sandbox and this problem doesn't happen.

The way (I think) this affected your installation is that:

1. During install, pip detected some version of pysam >= 0.8.4. It must have, because I hard-coded a sys.exit(1) inside setup.py if pysam.__version__ < 0.8.4. This is an atypical thing to do- but I had to do it for backward compatibility with old versions of pip.
2. Within the subshell called by pip , gcc linked against pysam 0.6.x, causing build to fail
3. After pysam was upgraded, plastid probably linked to 0.9.1 (or 0.8.4) in the subshell
4. During plastid import, it found the opposite or 0.8.4 (or 0.9.1) in its runtime environemnt
5. Rolling back to 0.8.4 in your made the import and build-linked versions of pysam match

Issues of this type have come up now several times, and it has always been an issue with Conda breaking pip (or pip breaking Conda, depending on how you look at it). Unfortunately, I'm powerless to fix this, though I will write a FAQ about it on the install page. I don't intend to besmirch Conda, but I strongly suggest using virtualenv instead (detailed here.

Before I close this, could you try creating a virtualenv, and installing plastid in that, following the instructions at http://plastid.readthedocs.io/en/latest/installation.html ?

[calvinjan]

Hi Josh,
That all makes sense. I tried to do what you suggested in the first place
(as per the docs) but my system fails to create the virtualenv, raising
this exception:
ImportError:
dlopen(/Users/calvinjan/virtualenvs/plastid/lib/python2.7/lib-dynload/_io.so,
2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
This is clearly not a plastid issue (and probably still a conda nuisance),
so I didn't want to bug you about it. I'll update you if I get around to
troubleshooting this.
Thanks,
Calvin

On Wed, Aug 31, 2016 at 6:46 PM, Joshua Griffin Dunn <
[email protected]> wrote:

Hey dude!

It's great to hear form you, and thank you for the heads up. I can't
replicate this on my system with pysam 0.9.1.4 (in or out of a virtualenv),
and I know I have Mac users who are using pysam >= 0.9.1.0 outside of
Conda (usually in a virtualenv)

I think this is issue is the same set of events that came up in issue 2
#2, in that case with
numpy. The TLDR is that Conda has some strange interactions with pip,
causing Conda's sandboxes to leak. Overall, I strongly recommend using
virtualenv https://virtualenv.pypa.io/en/stable/ (which plays well with
pip, and to my knowledge has raised no problems with plastid, pysam, or
numpy) in place of Conda. In a virtuanlenv, I have been able to build
Plastid with every version of Pysam >= 0.8.4

The long answer (if what I think is happening actually is) is this:

Conda aims to create Python sandboxes within which various packages
can be installed and upgraded. This is a noble goal
2.

When pip is processing dependencies for setup (from setup.py), it sees
the package versions that are present inside the Conda envrionment, as
expected
3.

When pip actually runs the compiler needed to build plastid, it starts
a subshell that exists outside the Conda sandbox. This can cause builds to
fail, e.g. because pip will try to link against a version of pysam
and/or numpy that might be lower or higher than the version required
in and detected by setup.py. Even if the build succeeds and both
versions of pysam and/or numpy meet the minimal requirements, you may
still get runtime errors if plastid is linked against a different
version of numpy or pysam from the ones it finds in the runtime
environment

In contrast, If pip is run inside a virtualenv (not a Conda virtual
environment), subshells stay within the sandbox and this problem doesn't
happen.

The way (I think) this affected your installation is that:

During install, pip detected some version of pysam >= 0.8.4. It must
have, because I hard-coded a sys.exit(1) inside setup.py if
pysam.version < 0.8.4. This is an atypical thing to do- but I had
to do it for backward compatibility with old versions of pip.
2.

Within the subshell called by pip , gcc linked against pysam 0.6.x,
causing build to fail
3.

After pysam was upgraded, plastid probably linked to 0.9.1 (or 0.8.4)
in the subshell
4.

During plastid import, it found the opposite or 0.8.4 (or 0.9.1) in
its runtime environemnt
5.

Rolling back to 0.8.4 in your made the import and build-linked
versions of pysam match

Issues of this type have come up now several times, and it has always been
an issue with Conda breaking pip (or pip breaking Conda, depending on how
you look at it). Unfortunately, I'm powerless to fix this, though I will
write a FAQ about it on the install page. I don't intend to besmirch Conda,
but I strongly suggest using virtualenv instead (detailed here
http://plastid.readthedocs.io/en/latest/FAQ.html#faq-install-fails-with-prereqs.

Before I close this, could you try creating a virtualenv
https://virtualenv.pypa.io/en/stable/, and installing plastid in that,
following the instructions at http://plastid.readthedocs.io/
en/latest/installation.html ?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#4 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC-58BIC2_1wopgiB70_XgubTWHqSxDjks5qli5tgaJpZM4Juz6t
.