GithubHelp home page GithubHelp logo

i2pc / xmipp Goto Github PK

View Code? Open in Web Editor NEW
37.0 10.0 15.0 14.93 MB

Xmipp is a suite of image processing programs, primarily aimed at single-particle 3D electron microscopy.

Home Page: http://biocomputingunit.es/

License: GNU General Public License v3.0

Python 7.47% C++ 90.25% C 0.34% Shell 0.03% MATLAB 0.76% Cuda 1.14%
cryo-em cryoem scipion

xmipp's Introduction

Build Status

Xmipp

drawing

Xmipp is a suite of image processing programs, primarily aimed at single-particle 3D electron microscopy, designed and managed by the Biocomputing Unit located in Madrid, Spain.

The Xmipp project is divided into four repositories. This is the main repository, which contains the majority of the source code for the programs, additional scripts, and tests. Three remaining repositories, XmippCore, XmippViz, and Scipion-em-xmipp, are automatically downloaded during the installation process.

Getting started

To have a complete overview about Xmipp please visit the documentation web. The recommended way for users (not developers) to install and use Xmipp is via the Scipion framework, where you can use Xmipp with other Cryo-EM-related software. Xmipp will be installed during Scipion installation (pay attemption on the -noXmipp flag). The Scipion installer should take care of all dependencies for you, however, you can make its life easier if you have your compiler, CUDA and HDF5 library ready and available in the standard paths. Read below for more details about these softwares requirements. Follow the official installation guide of Scipion for more details.

Installation requirements

Supported OS

We have tested Xmipp compilation on the following operating systems: Ubuntu 18.04, Ubuntu 20.04, Ubuntu 22.04, Centos 7. Visit the OS wiki page for more details.

While compilation and execution might be possible on other systems, it might not be straightforward. If you encounter a problem, please refer to known and fixed issues. Let us know if something is not working.

Hardware requirements

At least 2 processors are required to run Xmipp. In some virtual machine tools only one is assigned, please check that at least two processors are assigned to the virtual machine

Software dependencies

Compiler

Xmipp requires C++17 compatible compiler. We recommend GCC with G++, we have good experience with GCC/G++-8.4 in any case a version >= 8.4 is required. If use GCC/G++-10.3 and CUDA=11 and experience issues, please change the compiler version. If use GCC-11 and experience issues, please visit this. For more details about the compilation proces and installation of gcc, please visit compiler

Cmake

Xmipp requires Cmake 3.16 or above. To update it please visit

Cuda

Xmipp supports Cuda 8 through 11.7. CUDA is optional but highly recommended. We recommend you to use the newest version available for your operating system, though Cuda 10.2 has the widest support among other Scipion plugins. Pay attemption to the compiler - CUDA compatibility. To install CUDA for your operating system, follow the official install guide.

OpenCV

OpenCV is used for some programs: movie_optical_alignment (with GPU support) and volume_homogenizer, however, it is not required. If you installed OpenCV via apt (sudo apt install libopencv-dev), it should be automatically picked up by the Xmipp script

HDF5

We sometimes see issues regarding the HDF5 dependency. Please visit the HDF5 Troubleshooting wiki

Full list of dependencies

sudo apt install -y libfftw3-dev libopenmpi-dev libhdf5-dev python3-numpy python3-dev libtiff5-dev libsqlite3-dev default-jdk git cmake

pip install scons in the enviroment that xmipp will be compiled (scipion3 if you will run Xmipp within Scipion)

Also a compiler will be required (sudo apt install gcc-10 g++-10)

Standalone installation

Standalone installation of Xmipp is recommended for researchers and developers. This installation allows you to use Xmipp without Scipion, however, in the next section it is explained how to link it with Scipion. Xmipp script automatically downloads several dependencies and then creates a configuration file that contains paths and flags used during the compilation. Please refer to the Xmipp configuration guide for more info.

Start by cloning the repository and then navigate to the right directory.

git clone https://github.com/I2PC/xmipp.git xmipp-bundle && cd xmipp-bundle

Refer to ./xmipp --help for additional info on the compilation process and possible customizations.

Next is to compile xmipp. There are to possibilities and in both you will can run Xmipp in Scipion (see linking step)

  • Compile Xmipp by invoking the compilation script, which will take you through the rest of the process:./xmipp that way you will install Xmipp with the dependencies and thier versions that the enviroment you decide, or the default one.
  • Compile Xmipp via Scipion enviroment scipion3 run ./xmipp that way you will install Xmipp with the dependencies and their versions that Scipion decided.

It is important to highlight that this step only compiles Xmipp, but it does not link to Scipion. The linking to Scipion is explained in the next section.

Linking Xmipp standalone to Scipion

Once the Standalone version has been installed, the user can link such installation to Scipion to have the posibility of use Xmipp inside Scipion. Linking with Scipion requires to the repository of scipion-em-xmipp which can be found in the folder src/scipion-em-xmipp. This repository contains the files that Scipion needs to execute Xmipp programs. However, it remains to link the Xmipp binaries with Scipion. To do that we need Scipion installed (see Scipion installation web page) and just launch the next command to link the binaries

scipion3 installp -p ~/scipion-em-xmipp --devel

where scipion-em-xmipp is the folder of the repository, it means src/scipion-em-xmipp. This command should work in most of the cases. However, if you do this and Scipion does not find Xmipp, please visit Linking Xmipp to Scipion Troubleshooting

Using Xmipp

Xmipp is installed in the build directory located in the same directory where the xmipp script is located. To set all necessary environment variables and paths to all Xmipp programs, you can simply source build/xmipp.bashrc

That's all

If you miss some feature or find a bug, please create an issue and we will have a look at it, you can also contact us by [email protected]. For more details, troubleshootings and information visit the wikip page.

If you like this project, Watch or Star it! We are also looking for new collaborators, so contact us if you’re interested

xmipp's People

Contributors

ajimoreno avatar albertmena avatar cossorzano avatar danielmarchan3 avatar danihr avatar darkyenus avatar david-myska avatar davidherreros avatar delarosatrevin avatar dmaluenda avatar dstrelak avatar erneyramirez avatar estrellafg avatar fede-pe avatar fonsecareyna82 avatar horacekj avatar jamesmkrieger avatar javimota avatar jeymendez avatar kiskaj avatar martinsalinas98 avatar melosa11 avatar mmmtnez avatar mohamadharastani avatar oierlauzi avatar pconesa avatar ratolon avatar rmarabini avatar rsanchezgarc avatar vilax avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

xmipp's Issues

Error in High-Res when GPU is activated

From: https://scipion.slack.com/archives/CA77ZNCMC/p1573516719000600

I am running xmipp-highRes on a dataset using GPU, but it has an error
"Parameter incorrect". It is in 'step 6: reconstruct' and the log is as
follow. Do you have any idea if the problem is in my Scipion
configuration. I am using scipion2 and just installed on this machine
two days ago. Also, I am using cuda 8.0, and using the CPU, protocol works very well.

00304: Direct 3D reconstruction method using Kaiser windows as interpolators
00305: =====================================================================
00306: Input selfile : Runs/003545_XmippProtReconstructHighRes/extra/Iter001/images_corrected01.xmd
00307: padding_factor_proj : 2
00308: padding_factor_vol : 2
00309: Output volume : Runs/003545_XmippProtReconstructHighRes/extra/Iter001/volume01.vol
00310: Symmetry file for projections : c1
00311: Use weights stored in the image headers or doc file
00312:
00313: Interpolation Function
00314: blrad : 1.9
00315: blord : 0
00316: blalpha : 15
00317: max_resolution : 0.5
00318: -----------------------------------------------------------------
00319: Available memory: 11015MB
00320: Required memory (at least): 7MB
00321: 0000/???? sec. ------------------------------------------------------------XMIPP_ERROR 54: Parameter incorrect.
00322: Only pointer aligned to the page size can be registered
00323: File: libraries/reconstruction_cuda/gpu.cpp line: 96
00324: Traceback (most recent call last):
00325: File "/home/mohsen/scipion-v2/scipion/pyworkflow/protocol/protocol.py", line 186, in run
00326: self._run()
00327: File "/home/mohsen/scipion-v2/scipion/pyworkflow/protocol/protocol.py", line 237, in _run
00328: resultFiles = self._runFunc()
00329: File "/home/mohsen/scipion-v2/scipion/pyworkflow/protocol/protocol.py", line 233, in _runFunc
00330: return self._func(*self._args)
00331:
File
"/home/mohsen/scipion-v2/xmipp-bundle/src/scipion-em-xmipp/xmipp3/protocols/protocol_reconstruct_highres.py",
line 1270, in reconstruct
00332: self.runJob('xmipp_cuda_reconstruct_fourier ', tmpArgs, numberOfMpi=1)
00333: File "/home/mohsen/scipion-v2/scipion/pyworkflow/protocol/protocol.py", line 1311, in runJob
00334: self._stepsExecutor.runJob(self._log, program, arguments, **kwargs)
00335: File "/home/mohsen/scipion-v2/scipion/pyworkflow/protocol/executor.py", line 70, in runJob
00336: env=env, cwd=cwd, gpuList=self.getGpuList())
00337: File "/home/mohsen/scipion-v2/scipion/pyworkflow/utils/process.py", line 52, in runJob
00338: return runCommand(command, env, cwd)
00339: File "/home/mohsen/scipion-v2/scipion/pyworkflow/utils/process.py", line 67, in runCommand
00340: env=env, cwd=cwd)
00341: File "/home/mohsen/scipion-v2/scipion/software/lib/python2.7/subprocess.py", line 190, in check_call
00342: raise CalledProcessError(retcode, cmd)
00343: CalledProcessError: Command 'xmipp_cuda_reconstruct_fourier -i
Runs/003545_XmippProtReconstructHighRes/extra/Iter001/images_corrected01.xmd
-o Runs/003545_XmippProtReconstructHighRes/extra/Iter001/volume01.vol
--sym c1 --weight --device 1 --thr 1' returned non-zero exit status 255
00344: Protocol failed: Command 'xmipp_cuda_reconstruct_fourier -i
Runs/003545_XmippProtReconstructHighRes/extra/Iter001/images_corrected01.xmd
-o Runs/003545_XmippProtReconstructHighRes/extra/Iter001/volume01.vol
--sym c1 --weight --device 1 --thr 1' returned non-zero exit status 255
00345: FAILED: reconstruct, step 6
00346: 2019-10-31 15:43:03.639203
00347: *** Last status is failed
00348: ------------------- PROTOCOL FAILED (DONE 6/10)

xmipp_volume_align with option --frm does not work after the recent update

There is an issue with xmipp_volume_align, that looks python-binding related.

To reproduce:

xmipp_volume_align --i1 volume1.spi --i2 volume2.spi --frm

gives the following error:
XMIPP_ERROR 60: Not classified error.
Cannot import sh_alignment.
File: libraries/interface/frm.cpp line: 85

I would appreciate your help in this regards.

Also, I have noticed that recompiling xmipp (xmipp-bundle / developers way) causes Scipion2 to fail starting, is this normal?

Kindly,
Mohamad

CL2D producing less classes than requested

Thank you to XMIPP developers for your contributions. We are running CL2D (xmipp_mpi_classify_CL2D) with ~10k particles and requesting 40 classes (--nref 40). However, it produces only 4 classes. When we run it on a subset of ~5k particles, with the same parameters, it is able to produce all 40 classes. This seems counterintuitive, and when I dig deeper in the code, I see that it stops when number of assignment changes is less than 0.5% of total particles.

I was wondering if anyone else has seen this issue and can comment on this. One workaround to get more classes is to reduce the number of iteration, but I'm not sure if this is am optimal solution.
Thanks.

build warning

g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

libraries/reconstruction/nma_alignment_vol.cpp: In member function 'virtual double ObjFunc_nma_alignment_vol::eval(Vector, int*)':
libraries/reconstruction/nma_alignment_vol.cpp:276:9: warning: ignoring return value of 'int fscanf(FILE*, const char*, ...)', declared with attribute warn_unused_result [-Wunused-result]
   fscanf(global_nma_vol_prog->AnglesShiftsAndScore, "%f,", &global_nma_vol_prog->fit_value);
   ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
libraries/reconstruction/nma_alignment_vol.cpp:293:16: warning: ignoring return value of 'int fscanf(FILE*, const char*, ...)', declared with attribute warn_unused_result [-Wunused-result]
          fscanf(global_nma_vol_prog->AnglesShiftsAndScore, "%f,", &global_nma_vol_prog->Best_Angles_Shifts[i]);
          ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Didn't we have this warning before?

"xmipp_compile" doesn't work in Scipion2

If you have the same problem, you can overcome it temporary by creating a directory in "/xmipp-bundle/xmipp/src/xmipp/applications/programs" and placing your code in it, then compiling xmipp (from xmipp-bundle/xmipp excuting ./xmipp). The binary can be found in "/xmipp-bundle/xmipp/src/xmipp/bin". "xmipp_compile" used to work in Scipion1 though. Any idea?
Thanks

Taking CXXFLAGS and LD_LIBRARY_PATH from Scipion

I'm coping an issue derived from a mail at [email protected].

The generated config file overrides any/some preset environment variables, especially problematic when you compile within Scipion and you set something in the Scipion's config.

I couldn't find a way of inheriting CCFLAGS or CXXFLAGS without hacking the code. Allowing additional variables (LD_LIBRARY_PATH, LIBRARY_PATH, C_INCLUDE_PATH) might be a good enhancement for those of us who have multiple versions of core libraries available in a multiuser environment.

Xmipp does not compile without internet connection

Dear Xmippers,

I tried to compile xmipp without internet connection, but it is not possible to compile it because of this error

Checking configuration ------------------------------
Getting Dependencies -------------------------------------
Checkouting micrograph_cleaner_em ...
fatal: unable to access 'https://github.com/rsanchezgarc/micrograph_cleaner_em/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Cannot get dependencies

Kind regards

Vilas

opencv is always True

I had to edit xmipp file otherwise it always creates a new config with opencv=True. Even after this trick, it fails.

In file included from applications/programs/mpi_volume_homogenizer/volume_homogenizer.cpp:26:
applications/programs/mpi_volume_homogenizer/../mpi_volume_homogenizer/volume_homogenizer.h:30:10: fatal error: opencv2/core/core.hpp: No such file or directory

How can I get rid of it? I dont want opencv at all :)

monores has broken backwards compatibility

Users with old "results" of monores are not able to open the results. It complains about a missing MG_Chimera_resolution.mrc (?)

Seems to be related with the viewer and some file renaming.

PSD generation (using Xmipp programs) is very very SLOW

In Scipion, we have some util functions in base class ProtAlignMovies:
averageMovie (using xmipp_movie_alignment_correlation)
computePSD (called twice, using xmipp_ctf_estimate_from_micrograph)

These these two programs are quite slow, even when putting them in a separate thread (to not block precious GPU time, using option --use_worker_thread), sometimes they can't keep up with the data processing and start to accumulate. They also use a lot of RAM, which cause in some cases that the machine use the swap and make everything slower.

I know that these are Scipion functions (used in many motion correction protocols), but since now the implementation rely on these Xmipp programs, I'm reporting the issue here.

Movie alignment is running on multiple GPUs

I have noticed that Movie alignment on Torvalds tends to run on two GPUs at some point.
This is clearly a bug, probably caused by a thread setting 'default' (no 1) instead of 'requested' (no 7) GPU.

image

Xmipp compilation issue with Cuda11

I had a strange error while compiling xmipp due to the cuda version. I don't want cuda programs to run, I just need to compile.
Here is the error:

(base) bash-4.2$ ./xmipp 
Configuring -----------------------------------------
 gcc detected
 g++ -c -w  -mtune=native -march=native -std=c++11 -O3 xmipp_test_opencv.cpp -o xmipp_test_opencv.o -I../ -I/home/guest/anaconda3/include
 g++ -w  -mtune=native -march=native -std=c++11 -O3 xmipp_test_opencv.cpp -o xmipp_test_opencv -I../ -I/home/guest/anaconda3/include 
 ./xmipp_test_opencv
 g++ -c -w  -mtune=native -march=native -std=c++11 -O3 xmipp_test_opencv.cpp -o xmipp_test_opencv.o -I../ -I/home/guest/anaconda3/include
 OPENCV-2 detected without CUDA support
 rm -v xmipp_test_opencv*
 mpicc, mpiCC detected.
 java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64 detected.
Traceback (most recent call last):
  File "./xmipp", line 1583, in <module>
    config()
  File "./xmipp", line 937, in config
    configCuda(new_config_dict)
  File "./xmipp", line 769, in configCuda
    nvccVersion, nvccFullVersion = getCudaVersion("nvcc")
  File "./xmipp", line 1421, in getCudaVersion
    nvccVersion = float(str(tokens[0] + '.' + tokens[1]))
ValueError: could not convert string to float: 'Build cuda_11.0_bu'

A workaround was to give the nvcc version manually in this line.

    # nvccVersion = float(str(tokens[0] + '.' + tokens[1]))
    nvccVersion = 11.0

I don't know why the cuda is giving 11.0_bu, it is the normal installation from Nvidia website.

Later I got another error, it was gone when I deactivated conda, I know it is another story, but is it usually the case to need to deactivate conda when compiling xmipp?

I also find it puzzling that in the recent documentation of Scipion3, cuda 10.1 is recommended, while it is cuda 8.0 in xmipp.

Kind regards,
Mohamad

compilation using CUDA 7.5 fails

If cuda 7.5 is used xmipp compuilation fails with error message:

In file included from cudaUtils.h:5:0,
from benchmarker.h:6,
from advisor.h:4,
from advisor.cpp:1:
cudaAsserts.h: In function ‘const char* cuFFTAdvisor::_cudaGetErrorEnum(cufftResult)’:
cudaAsserts.h:63:10: error: ‘CUFFT_NOT_SUPPORTED’ was not declared in this scope
case CUFFT_NOT_SUPPORTED:

If we change cuda 7.5 by cuda 8.0 compialtion is OK. In the documentation it should be clearly stated theat CUDA 7.5 is not supported.

compilation fails with error "deduced return type only available with -std=c++14 or -std=gnu++14"

when compiled with gcc 4 I get the error attached at the end of this message. The error message mentioned C++14 support being needed. I found that altering xmipp.conf to set -std=c++14 instead of its current c++11 allows the library to build and link.

cheers

Roberto

===================================
g++ -std=gnu++1y -o libraries/reconstruction/movie_alignment_correlation.os -c -std=c++11 -O3 -I/usr/local/scipion_plugin/software/include -I/usr/include/hdf5/serial -fPIC -I/usr/local/scipion_plugin/software/include/python2.7 -I/usr/local/scipion_plugin/software/lib/python2.7/site-packages -I/usr/local/scipion_plugin/software/lib/python2.7/site-packages/numpy/core/include -Iexternal -Ilibraries -I/usr/local/Plugins/xmipp_bundle/src/xmippCore libraries/reconstruction/movie_alignment_correlation.cpp
In file included from libraries/reconstruction/movie_alignment_correlation.h:32:0,
from libraries/reconstruction/movie_alignment_correlation.cpp:27:
libraries/reconstruction/movie_alignment_correlation_base.h:287:42: error: ‘loadGlobalShifts’ function uses ‘auto’ type specifier without trailing return type
auto loadGlobalShifts(MetaData &movie);
^
libraries/reconstruction/movie_alignment_correlation_base.h:287:42: note: deduced return type only available with -std=c++1y or -std=gnu++1y
libraries/reconstruction/movie_alignment_correlation_base.h:356:64: error: ‘computeRating’ function uses ‘auto’ type specifier without trailing return type
auto computeRating(const LocalAlignmentResult &alignment);
^
libraries/reconstruction/movie_alignment_correlation_base.h:356:64: note: deduced return type only available with -std=c++1y or -std=gnu++1y

Extract particles bug

Hi, launching scipion protocol extract particles with gctf CTF (I do not now with others) crashes.

00031: xmipp_transform_filter -i Runs/004636_XmippProtExtractParticles/tmp/20130109_10497_FinalImage_downsampled.xmp -o Runs/004636_XmippProtExtractParticles/tmp/20130109_10497_FinalImage_noDust.xmp --bad_pixels outliers 5.000000
00032: Input File: Runs/004636_XmippProtExtractParticles/tmp/20130109_10497_FinalImage_downsampled.xmp
00033: Output File: Runs/004636_XmippProtExtractParticles/tmp/20130109_10497_FinalImage_noDust.xmp
00034: float() argument must be a string or a number
00035: Problems found converting metadata:
00036: Label id = 132
00037: Attribute = _fitQuality
00038: Value = None
00039: Value type = <type 'float'>
00040: Traceback (most recent call last):
00041: File "/home/rsanchez/app/scipion/pyworkflow/protocol/protocol.py", line 186, in run
00042: self._run()
00043: File "/home/rsanchez/app/scipion/pyworkflow/protocol/protocol.py", line 237, in _run
00044: resultFiles = self._runFunc()
00045: File "/home/rsanchez/app/scipion/pyworkflow/protocol/protocol.py", line 233, in _runFunc
00046: return self._func(*self._args)
00047: File "/home/rsanchez/app/scipion/pyworkflow/em/protocol/protocol_particles.py", line 237, in extractMicrographStep
00048: self._extractMicrograph(mic, *args)
00049: File "/home/rsanchez/app/scipion/software/lib/python2.7/site-packages/xmipp3/protocols/protocol_extract_particles.py", line 218, in _extractMicrograph
00050: micrographToCTFParam(mic, fnCTF)
00051: File "/home/rsanchez/app/scipion/software/lib/python2.7/site-packages/xmipp3/convert/convert.py", line 401, in micrographToCTFParam
00052: ctfModelToRow(ctf, row)
00053: File "/home/rsanchez/app/scipion/software/lib/python2.7/site-packages/xmipp3/convert/convert.py", line 629, in ctfModelToRow
00054: extraLabels=CTF_EXTRA_LABELS)
00055: File "/home/rsanchez/app/scipion/software/lib/python2.7/site-packages/xmipp3/convert/convert.py", line 250, in objectToRow
00056: raise e
00057: TypeError: float() argument must be a string or a number
00058: Protocol failed: float() argument must be a string or a number

Missing NVCC_CXXFLAGS

while compiling xmipp on Ubuntu 14.04 LTS with CUDA 8, NVCC_CXXDLAGS are missing.

xmipp_transform_add_noise and aligned particles

hi,

The protocol xmippAddNoiseParticles is inconsistent when using as input aligned particles.

xmippAddNoiseParticles read the particle and if there is a transformation matrix it applies the in-plane rotation before saving the new image. Finally, the protocol assigns to the new image the same transformation matrix than had the input image. The correct way to do this would be either

  1. Do NOT apply the inplane rotation to the new image and store the input image transformation (my favorite solution)
  2. Apply the in-plane rotation and store a modified transformation matrix. This matrix would be the product of the original transformation matrix by the inverse of the matrix created by euler angles (0,0,injplanerotation)

./xmipp compile fails in python3_migration branch

I have downloaded xmipp script from python3_migration branch in xmipp repo, then:

  1. ./xmipp get_devel_sources python3_migration
  2. ./xmipp config
  3. ./xmipp check_config
  4. ./xmipp compile 8

Now xmippCore.so looks done, but then I get:

Traceback (most recent call last):
  File "./xmipp", line 1437, in <module>
    ok = compile(Nproc)
  File "./xmipp", line 991, in compile
    if not compileModule(Nproc,"xmipp"):
  File "./xmipp", line 944, in compileModule
    shutil.copyfile(CONFIG_FILE_NAME,"src/%s/install/%s" % (module, CONFIG_FILE_NAME))
  File "/usr/lib/python3.5/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: 'src/xmipp/install/xmipp.conf'

many tests require other packages and fail if not available

There are several tests that require other packages and fail if not available. If you run the test individually you get understandable error messages:

 > relion plugin not found. No module named relion.protocols
   Called by /usr/local/scipion_plugin/software/lib/python2.7/site-packages/xmipp3/tests/test_protocols_subtract_projection.py, line 44
   Check the plugin manager (Configuration->Plugins in Scipion manager window) 
   or use 'scipion installp --help' in the command line to install it.

but if you just ask for a list of tests just get errors:


[roberto@clark-cinco/usr/local/scipion_plugin] (devel)>../scipion_plugin/scipion test --show | grep Error
Error loading the test. Please, run the test for more information: pyworkflow.tests.em.protocols.test_protocols_ctf_streaming
Error loading the test. Please, run the test for more information: pyworkflow.tests.em.protocols.test_protocols_ctf_monitor
Error loading the test. Please, run the test for more information: pyworkflow.tests.em.protocols.test_protocol_volume_homogenizer
Error loading the test. Please, run the test for more information: pyworkflow.tests.em.workflows.test_workflow_xmipp_ctf_consensus
Error loading the test. Please, run the test for more information: pyworkflow.tests.em.workflows.test_workflow_modeling
Error loading the test. Please, run the test for more information: pyworkflow.tests.em.workflows.test_workflow_mixed_large
Error loading the test. Please, run the test for more information: pyworkflow.tests.em.workflows.test_workflow_mixed
Error loading the test. Please, run the test for more information: pyworkflow.tests.em.workflows.test_workflow_streaming
Error loading the test. Please, run the test for more information: pyworkflow.tests.em.workflows.test_workflow_initialvolume
Error loading the test. Please, run the test for more information: pyworkflow.tests.em.workflows.test_workflow_high_throughput
Error loading the test. Please, run the test for more information: xmipp3.tests.test_protocols_gpuCorr_fullStreaming
Error loading the test. Please, run the test for more information: xmipp3.tests.test_protocols_gpuCorr_semiStreaming
Error loading the test. Please, run the test for more information: xmipp3.tests.test_protocols_gpuCorr_classifier
Error loading the test. Please, run the test for more information: xmipp3.tests.test_protocols_realignment_classes
Error loading the test. Please, run the test for more information: xmipp3.tests.test_protocols_subtract_projection

It would be nice not to get these confusing error messages since you get the impression that xmipp does not work

xmipp_matrix_dimred gives wrong mapping matrix with PCA and pPCA

Hello,
xmipp_matrix_dimred has an option to save the mapping for linear methods.

   [--saveMapping <fn=>]
           Save mapping if available (PCA, LLTSA, LPP, pPCA, NPE) so that it can be reused later (Y=X*M) 

X is the input matrix, Y is the output matrix and M is the mapping.
The function seems to give correct output matrix (Y) but incorrect mapping matrix (M) for two methods: PCA and pPCA.

Here is a sample (200 samples, 3 features per sample) for testing.
X.txt

The following lines can generate all the matrices after:

xmipp_matrix_dimred -i X.txt -o Y_PCA.txt -m PCA --dout 2 --saveMapping M_PCA.txt --din 3 --samples 200
xmipp_matrix_dimred -i X.txt -o Y_LLTSA.txt -m LLTSA --dout 2 --saveMapping M_LLTSA.txt --din 3 --samples 200
xmipp_matrix_dimred -i X.txt -o Y_LPP.txt -m LPP --dout 2 --saveMapping M_LPP.txt --din 3 --samples 200
xmipp_matrix_dimred -i X.txt -o Y_pPCA.txt -m pPCA --dout 2 --saveMapping M_pPCA.txt --din 3 --samples 200
xmipp_matrix_dimred -i X.txt -o Y_NPE.txt -m NPE --dout 2 --saveMapping M_NPE.txt --din 3 --samples 200

The following Python commands can be used to test the output:

from numpy import loadtxt, matmul, sum
X = loadtxt('X.txt')
M_PCA = loadtxt('M_PCA.txt')
M_LLTSA = loadtxt('M_LLTSA.txt')
M_LPP = loadtxt('M_LPP.txt')
M_pPCA = loadtxt('M_pPCA.txt')
M_NPE = loadtxt('M_NPE.txt')
print('PCA mapping verification (should be zero) ', sum(matmul(X,M_PCA) - loadtxt('Y_PCA.txt')))
print('LLTSA mapping verification (should be zero) ', sum(matmul(X,M_LLTSA) - loadtxt('Y_LLTSA.txt')))
print('LPP mapping verification (should be zero) ', sum(matmul(X,M_LPP) - loadtxt('Y_LPP.txt')))
print('pPCA mapping verification (should be zero) ', sum(matmul(X,M_pPCA) - loadtxt('Y_pPCA.txt')))
print('NPE mapping verification (should be zero) ', sum(matmul(X,M_NPE) - loadtxt('Y_NPE.txt')))

Here is what the output looks like:

PCA mapping verification (should be zero)  5008.6003077752175
LLTSA mapping verification (should be zero)  0.7019146815386402
LPP mapping verification (should be zero)  0.005094921908160148
pPCA mapping verification (should be zero)  58.372971159836005
NPE mapping verification (should be zero)  0.00780677463630014

Another problem is happening when the size of the input is high. Some of the methods fail and give a message "Killed".

Regards,
Mohamad

devtoolset for installation on Centos 7

Hello,

the dependencies page for Scipion installation mentions that while some version g++ and gcc are already installed on a system, it is important to have version 5 for compatibility with certain plugins like Xmipp3.

Unfortunately the Centos 7 repositories don't provide devtoolset-5 anymore. Currently available are devtoolsets 4 and 6-9.

Is there a list available, showing which plugins and which parts of Xmipp3 depend on gcc 5 for compatibility, or is it possible to substitute gcc 5 with a newer or older version?

Thank you for your help,
Andreas

Compiling local changes fails due to 'working branch' is detected while checouting.

With the new refactoring of git stuffs, I realized that if you work in something the command ./xmipp is failing before compiling/installing since the checkout is failing due to 'working branch detected'.

This checkout should only be done when the br flag is provided, if not we understand that you only want to compile/install.

I think that the problem comes from the 'doPull' meaning. I think that I introduced it in order to persist if the br flag was provided or not....

Sorry, I didn't see it yesterday...

Make sure that script checkouts proper branch

Steps to reproduce:

git clone -b v3.19.04-Apollo-DEVEL https://github.com/I2PC/xmipp.git xmipp-bundle
cd xmipp-bundle
./xmipp

Expected result:
xmippCore and other repositories will be downloaded in the tag version

Actual result:
xmippCore and other repositories are downloaded in the devel version, resulting in compilation error

Workaround:

./xmipp
# wait till it checkouts all repositories and start to build, then kill the process
./xmipp git checkout v3.19.04-Apollo-DEVEL
./xmipp

"xmipp_transform_mask" & "xmipp_transform_filter" save wrong wedge mask

Hello,
I need to generate a missing wedge filter (I need the filter itself). I tried these functions "xmipp_transform_mask" and "xmipp_transform_filter". Both are giving strange results for the mask. Here are the commands I used:
xmipp_transform_mask -i volume.vol -o mask.vol --mask wedge -60 60
xmipp_transform_filter -i volume.vol -o filtered.vol --fourier wedge -60 60 --save filter.vol

The first command gives a box in the same size as the input but is not a wedge filter.
The second command gives a volume of different size of the input (my input is of size 64 64 64, and it gives a mask of size 33 64 64). The generated volume has two separate triangular prisms, that does not correspond to a -60 to 60 wedge, although the filtered volume (applying the wedge) seems to be correct somehow.
Any idea? if not, is there another function to generate a correct missing wedge?
Thanks

constexpr functions

xmipp compilation returns error:

================
^
libraries/reconstruction/bspline_helper.h: In instantiation of 'static constexpr T BSplineHelper::Bspline03(T) [with T = double]':
libraries/reconstruction/bspline_helper.cpp:131:39: required from 'static void BSplineHelper::getShift(int, int, int, int, int, int, int, int, int, T&, T&, const T*, const T*) [with T = double]'
libraries/reconstruction/bspline_helper.cpp:103:61: required from 'static std::pair<_FIter, _FIter> BSplineHelper::getShift(const BSplineGrid&, Dimensions, size_t, size_t, size_t) [with T = double; size_t = long unsigned int]'
libraries/reconstruction/bspline_helper.cpp:93:37: required from here
libraries/reconstruction/bspline_helper.h:97:5: error: body of constexpr function 'static constexpr T BSplineHelper::Bspline03(T) [with T = double]' not a return-statement
scons: *** [libraries/reconstruction/bspline_helper.os] Error 1
scons: building terminated because of errors.

if I remove constexpr from the function declaration, then I may compile the code. Using flag "-std=c++14" and gcc version 4.9.2 (Debian 4.9.2-10)

xmipp_image_operate --minus works on non-existent files

xmipp_image_operate program with binary operations silently ignores non-existent files .

Steps to reproduce (both master/devel branch):

xmipp_image_operate -i existing_file.vol --minus non_existing_file.vol -o result.vol
xmipp_image_operate -i existing_file.vol --plus non_existing_file.vol -o result.vol
xmipp_image_operate -i existing_file.vol --divide non_existing_file.vol -o result.vol
etc.

Result:
Input File: existing_file.vol
Output File: result.vol (contains copy of existing_file.vol)

Expected result (or similar):
XMIPP_ERROR 22: It is not a file
FileName::isMetaData: File: 'non_existing_file.vol' does not exist

error when importing from xmipp3.protocols: No module named sklearn.manifold

python /mnt/gauss/data/rsanchez/app/scipion/pyworkflow/apps/pw_project.py last

error when importing from xmipp3.protocols: No module named sklearn.manifold
File "/mnt/gauss/data/rsanchez/app/xmipp_bundle/src/scipion-em-xmipp/xmipp3/protocols/protocol_angular_alignment_sph.py", line 40, in
Check the plugin manager (Configuration->Plugins in Scipion manager window)
or use 'scipion installp --help --checkUpdates' in the command line to check for upgrades,
it could be a versions compatibility issue.

"xmipp_volume_align / xmipp_volume_align _prog" gives wrong SHIFT values!

In the volume_align_prog, the following change was introduced by someone and should be reverted immediately:
lines (438-442):
A = A + Aaux;
A(0,0) -= 1;
A(1,1) -= 1;
A(2,2) -= 1;
A(3,3) -= 1;
These lines should replaced with:
A = A * Aaux;

In the devel branch, the volume_align_prog and its corresponding volume_align_prog_main are in separate folders (/xmipp/src/xmipp/libraries/reconstruction/) and (/xmipp/applications/programs/volume_align_prog/) respectively. However, in the remaining branches, they are combined and only volume_align_prog_main exists.

This separation causes the following warning which is disturbing:
Segmentation fault (core dumped)
or: /bin/xmipp_volume_align: line 3: 6331 Segmentation fault (core dumped) $XMIPP_HOME/bin/xmipp_volume_align_prog "$@"
This warning is not effecting the functionality of this program (tested with --frm option).

-Llib flag is passed twice

I don't know if this affect the performance during the compilation, but -Llib flag is introduced twice in all compilations. I've tryed to find why, but I couldn't find it.

g++ -o bin/xmipp_multireference_aligneability applications/programs/multireference_aligneability/multireference_aligneabililty_main.o -L/home/david/scipionUSER/software/em/xmippSrc-3.19.03b/src/xmippCore/lib-Llib -Llib-L/home/david/scipionUSER/software/lib -L/home/david/scipionUSER/software/lib -L/usr/lib/openmpi/lib -L/usr/local/cuda-8.0/lib64 -L/home/david/scipionUSER/software/em/xmipp/lib -lXmipp -lhdf5 -lhdf5_cpp -lfftw3 -lfftw3_threads -ljpeg -ltiff -lXmippCore

Dimred LPP and LaplacianEigenmap are unstable

These two methods are either very sensitive to input, or produce unstable results.
The tests are not passing on various machines, so they have been commented out.

We should find what is the cause, and fix it, if possible

See also #220

Warnings during compilation

Branch: devel
Compiler: g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Warning:
libraries/reconstruction/volumeset_align.cpp:140:15: warning: ignoring return value of 'int fscanf(FILE*, const char*, ...)', declared with attribute warn_unused_result [-Wunused-result]
fscanf(fnAnglesAndShifts, "%f,", &global_volumeset_align_prog->Matrix_Angles_Shifts[i]); ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
libraries/reconstruction/volumeset_align.cpp:142:8: warning: ignoring return value of 'int fscanf(FILE*, const char*, ...)', declared with attribute warn_unused_result [-Wunused-result]
fscanf(fnAnglesAndShifts, "%f,", &global_volumeset_align_prog->fitness);

Compiling xmipp has hardcoded mpirun with 4 slots

Hi,
While compiling xmipp on a personal laptop, I faced an error as follows:
"There are not enough slots available in the system to satisfy the 4 slots"
I have exactly 4 mpi slots in the processor (Intel® Core™ i7-4500U CPU @ 1.80GHz × 4 ).
This printing 4 times was only to print a sentence, but it ended up breaking the compilation.
I fixed this issue by replacing '4' by '2' in all the lines 692 to 698 here:
https://github.com/I2PC/xmipp/blob/devel/xmipp#L692
Couldn't we test if mpi runs during the installation another way? or turn the error into a warning?

Cheers
Mohamad

bad configuration if packages are not present

Let us imagine that I try to compile xmipp in a computer without mpi libraries.
I download and execute xmipp instalation command:

./xmipp all N=4 br=devel

The script, downloads the code, executes configure and compiles xmipp

As I do not have mpi libraries at some point an error message is printed on screen suggesting to install the missing software. I follow the instruction and re-execute

./xmipp all N=4 br=devel

This time no configuration is executed and since mpi related variables are not properly set the user gets many errors that are difficult to understand.

I suggest to modify the message printed suggesting to install the missing software so it also suggest to run the configuration step: ./xmipp configure && ./xmipp all N=4 br=devel

xmipp install still tries to get nvcc?

I am following the new install from source docs: https://scipion-em.github.io/docs/docs/scipion-modes/install-from-sources.html

After the step showing that scipion is running in ghost mode, I started plugin manager and clicked to install xmipp. The first thing I got printed in console is:

which: no nvcc in (/home/azazello/soft/scipion-test/software/em/xmipp/bin:/home/azazello/soft/scipion-test/software/bin:/usr/lib/jvm/java-8-openjdk/bin:/usr/bin:/usr/local/cuda-8.0/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl)
which: no nvcc in (/home/azazello/soft/scipion-test/software/em/xmipp/bin:/home/azazello/soft/scipion-test/software/bin:/usr/lib/jvm/java-8-openjdk/bin:/usr/bin:/usr/local/cuda-8.0/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl)
which: no nvcc in (/home/azazello/soft/scipion-test/software/em/xmipp/bin:/home/azazello/soft/scipion-test/software/bin:/usr/lib/jvm/java-8-openjdk/bin:/usr/bin:/usr/local/cuda-8.0/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl)

CUDA was set to False in scipion.conf even before scipion installation. Why does xmipp check nvcc?

ctf consensus

Hi,

I tried to use the protocol xmipp_ctf_consensus, it works but the general design is very inefficient (about two orders of magnitude). For example, let us imagine that I want to filter by Defocus range, this task can be achieved with a single query to the metadata and should take a few milisecons. Actually, it takes 30 minutes without providing any output.

The protocol creates a job per CTF, so in general a few thousand of jobs are created. Why? this is a non parallel protocol.

A much more efficient approach would be to create a single job and then perform a single query. Since this is implemented in xmipp I provide the query for xmipp

    mD = MetaData(mdPath)
    mDout = MetaData()
    mDout.importObjects(mD,  MDValueRange(MDL_CTF_DEFOCUSU mindefocus, maxdefocus))

For most of the situations it makes no sense to convert the metadata from scipion to xmipp, I am not very familiar with metadata in scipion but the comamnd "selectObjectsBy" or an
adecuate objectFilter in selectBy can achieve the results with a single query.

hope this help

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.