xmipp_volume_align with option --frm does not work after the recent update about xmipp

[DStrelak]

I can confirm that this test fails with

Signal: Segmentation fault (11)
Signal code: Address not mapped (1)
Failing at address: 0xb8

At first glance, the problem is probably caused by some problem at the MetaData processing:

[ 3] lib/libXmippCore.so(_ZN10MetaDataDb4initERKSt6vectorI7MDLabelSaIS1_EE+0x5c)[0x7f9d8e1e5e08]
[ 4] lib/libXmippCore.so(_ZN10MetaDataDb12copyMetadataERKS_b+0x41)[0x7f9d8e1e5e75]
[ 5] lib/libXmippCore.so(_ZN10MetaDataDbaSERKS_+0x2c)[0x7f9d8e1e8e32]
[ 6] lib/libXmipp.so(_ZN19ProgNmaAlignmentVol15createWorkFilesEv+0x9a)[0x7f9d8f55cf96]
[ 7] lib/libXmippParallel.so(_ZN13MpiProgNMAVol15createWorkFilesEv+0x78)[0x7f9d8dfc79ea]
[ 8] lib/libXmipp.so(_ZN19ProgNmaAlignmentVol10preProcessEv+0xc0)[0x7f9d8f55d34c]
[ 9] lib/libXmippCore.so(_ZN20XmippMetadataProgram3runEv+0x85)[0x7f9d8e139657]

[DStrelak]

Also, I have noticed that recompiling xmipp (xmipp-bundle / developers way) causes Scipion2 to fail starting, is this normal?

Xmipp in devel version is now migrated to python 3 (and expected to be used with scipion 3).
If you need python 2 version, use following version: https://github.com/I2PC/xmipp/releases/tag/v3.19.04-Apollo-DEVEL

[DStrelak]

Thanks for spotting those. See #301

[MohamadHarastani]

Also, I have noticed that recompiling xmipp (xmipp-bundle / developers way) causes Scipion2 to fail starting, is this normal?

Xmipp in devel version is now migrated to python 3 (and expected to be used with scipion 3).
If you need python 2 version, use following version: https://github.com/I2PC/xmipp/releases/tag/v3.19.04-Apollo-DEVEL

Thanks for the fast reply. I moved to Scipion3 already and I am happy with it, with few bugs to report. Thanks for all the Scipion/Xmipp team for your efforts and support.
I don't need to stay in Scipion2 once the issue I reported is solved (I heavily depend on xmipp_volume_align --frm). I tested the version in the link and it didn't compile on my computer. Here is the log.txt after executing "./xmipp >> log.txt"
log.txt

[DStrelak]

I tested the version in the link and it didn't compile on my computer.

Thanks for reporting that. I've created separate issue for that, see #295 for a workaround.

[MohamadHarastani]

I tested the version in the link and it didn't compile on my computer.

Thanks for reporting that. I've created separate issue for that, see #295 for a workaround.

Thanks, I confirm that xmipp_volume_align works there (for python2).

[DStrelak]

@MohamadHarastani
What is your python version? I can confirm that the xmipp_volume_align with --frm flag does not work, but I get different errors (for current devel, i.e. python3 version).
I get segmentation fault for Python 2.7.17, and for version 3.8.3 I get:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/home/david/git/scipion3/xmipp-bundle/build/bindings/python:/home/david/git/scipion3/xmipp-bundle/build/pylib:'
  program name = '\U6d6f682f\U61642f65\U2f646976\U696e696d\U646e6f63\U652f3261\U2f73766e\U70696373\U336e6f69\U6e69622f\U7479702f\U0a6e6f68'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = ''
  sys.base_prefix = '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.base_exec_prefix = '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.executable = ''
  sys.prefix = '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.exec_prefix = '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.path = [
    '/home/david/git/scipion3/xmipp-bundle/build/bindings/python',
    '/home/david/git/scipion3/xmipp-bundle/build/pylib',
    '',
    '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python38.zip',
    '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.8',
    '/tmp/build/80754af9/python_1593706424329/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f44e4e00740 (most recent call first):
<no Python frame>

[MohamadHarastani]

@DStrelak
I don't know if I understood the question correctly, if the following doesn't give you an answer, please let me know.

mohamad@mohamad:~$ echo $PYTHONPATH 
/home/mohamad/xmipp-bundle/xmipp/build/bindings/python:/home/mohamad/xmipp-bundle/xmipp/build/pylib:
mohamad@mohamad:~$ echo $SCIPION_HOME 
/home/mohamad/scipion3
mohamad@mohamad:~$ python --version
Python 2.7.17
mohamad@mohamad:~$ python3 --version
Python 3.6.9
mohamad@mohamad:~$ which python && which python3
/usr/bin/python
/usr/bin/python3

The error I get:

mohamad@mohamad:~/Work$ xmipp_volume_align --i1 volume1.spi --i2 volume2.spi --frm
XMIPP_ERROR 60:  Not classified error.
Cannot import sh_alignment.
File: libraries/interface/frm.cpp line: 85

I get the same error on another computer as well

[DStrelak]

Hi @MohamadHarastani ,
I have just pushed the fix. With the example you sent me some time ago, I got identical results.
However, I am a bit worried that there still might be some inconsistencies, as there has been a rather big change between python 2 and 3 in the way of handling integer division. Python 2 performs floor division, while Python 3 performs real division. This also affected numpy and (probably) other packages, so we might see some deviations.

Since you have more experience with this program, can I ask you to test the code in this branch with more examples, and send me those files that return different results (between version 2 and 3)?

[MohamadHarastani]

Hi @DStrelak ,
I have just checked the fix. I analyzed 100 volumes, and I got identical results as previous.
I see that you did new fixes, nevertheless, I will continue testing and report any inconsistency.
Thanks a lot for your efforts!
Mohamad

[DStrelak]

Hi @MohamadHarastani ,
I'm glad that the results did not change.
Never mind those latest changes, I've been just throwing code from one pile to another.
May we consider this fixed?

[MohamadHarastani]

Hi @MohamadHarastani ,
I'm glad that the results did not change.
Never mind those latest changes, I've been just throwing code from one pile to another.
May we consider this fixed?

Yes, we may consider it fixed.
Cheers

[MohamadHarastani]

After some tests of this program, I found two integer division issues here:
https://github.com/I2PC/xmipp/blob/devel/src/xmipp/external/sh_alignment/python/tompy/filter.py#L171
Lines 171 - 172 should be replaced by:
filter_vol[size[0]//2, :, :] = 0
filter_vol[size[0]//2, :, size[2]//2] = 1
The issue can be re-closed after these fixes.

[MohamadHarastani]

Hello again,
I found out that xmipp_volume_align --frm is not working in the new release.

 -bash-4.2$ xmipp_volume_align --i1 volume1.spi --i2 volume2.spi --frm
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/home/guest/xmipp-bundle/xmipp/build/bindings/python:/home/guest/xmipp-bundle/xmipp/build/pylib:'
  program name = '/usr/bin/python\x0a'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python\n'
  sys.base_prefix = '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.base_exec_prefix = '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.executable = '/usr/bin/python\n'
  sys.prefix = '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.exec_prefix = '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho'
  sys.path = [
    '/home/guest/xmipp-bundle/xmipp/build/bindings/python',
    '/home/guest/xmipp-bundle/xmipp/build/pylib',
    '',
    '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python38.zip',
    '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.8',
    '/tmp/build/80754af9/python_1599203911753/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Consequently, xmipp_volumeset_align, xmipp_nma_alignment_vol and there mpi versions are not working neither.

Thanks in advance,

Regards
Mohamad

[MohamadHarastani]

I found a way to run the program on the terminal by activating the conda environment of scipion3. Still, xmipp_volumeset_align and xmipp_nma_alignment_vol are failing without a clear message (at least to me) to debug the problem.
Probably the easiest way to reproduce is to run the continuousflex tests:
scipion3 tests continuousflex.tests.test_workflow_HEMNMA3D

[DStrelak]

Hello again, I found out that xmipp_volume_align --frm is not working in the new release.
...
ModuleNotFoundError: No module named 'encodings'

Hi @MohamadHarastani ,
this looks like a problem of your Python installation (see e.g. SO).
That would also explain why the error disappears when you use Conda environment (that one has its own Python installation).

[MohamadHarastani]

Thanks. I just need to find why the other programs are failing (xmipp_volumeset_align and xmipp_nma_alignment_vol, and their mpi versions).
I will provide an isolated test that reproduces the error asap.
Thanks for the help

[MohamadHarastani]

An isolated test (a minute):
xmipp_volumeset_align -i volumes.xmd -o output.xmd --ref ref.vol
https://filesender.renater.fr/?s=download&token=6db36ed6-ffe4-48d9-9c16-36ae1982e7ad

I tested with switching between the old and new xmipp versions. The old one works (although I get something about the endianness). The new, I get Segmentation fault (core dumped)