GithubHelp home page GithubHelp logo

Comments (15)

BaxterEaves avatar BaxterEaves commented on July 25, 2024

We weren't able to reproduce this error on our machines. I'm not sure if this is an Red Hat thing or what. I will explore this issue further. In the mean time, you may want to try to install in an Ubuntu 14.04 VM.

from crosscat.

BaxterEaves avatar BaxterEaves commented on July 25, 2024

I just had success installing crosscat on CentOS-7.0-1406-x86 (I used the minimal install, which required installing git, gcc, and g++). I followed the procedure currently outlined in the readme (which changed since you raised the issue) but replaced the apt-gets with their yum counterparts.

$ sudo yum install -y Cython boost-devel python-setuptools numpy scipy
$ git clone https://github.com/mit-probabilistic-computing-project/crosscat.git
$ cd crosscat
$ python setup.py build
$ sudo python setup.py install

I'm going to go ahead and close this issue, but please let me know if this process doesn't work for you.

from crosscat.

yarden avatar yarden commented on July 25, 2024

Any process that relies on 'sudo' is not workable: I am running this on a cluster with other users, and I don't have these rights. I'm not sure why this installation process would be different anyway - I have all the requirements of crosscat (Cython, numpy, scipy, etc..) in my virtualenv so I don't think that's the problem. It always complains about the lack of *.c files:

building 'crosscat.cython_code.CyclicComponentModel' extension
gcc -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c crosscat/cython_code/CyclicComponentModel.c -o build/temp.linux-x86_64-2.7/crosscat/cython_code/CyclicComponentModel.o
gcc -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/utils.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/utils.o
gcc -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/numerics.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/numerics.o
gcc -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/RandomNumberGenerator.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/RandomNumberGenerator.o
gcc -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/ComponentModel.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/ComponentModel.o
gcc -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/CyclicComponentModel.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/CyclicComponentModel.o
gcc: error: crosscat/cython_code/CyclicComponentModel.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 1

which sounds like a problem with setup.py not calling Cython or Cython not building the extension correctly.

from crosscat.

BaxterEaves avatar BaxterEaves commented on July 25, 2024

From a virtualenv, doing python setup.py build and then pip install . worked for me without sudo (doing pip install . alone did not work).

from crosscat.

yarden avatar yarden commented on July 25, 2024

I tried again, I recloned the repository. I get the same results with python setup.py build as pip install .:

$ git clone https://github.com/mit-probabilistic-computing-project/crosscat.git
Initialized empty Git repository in /ahg/regevdata/users/yarden/software/crosscat/.git/
remote: Counting objects: 15298, done.
remote: Total 15298 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (15298/15298), 7.96 MiB | 3.00 MiB/s, done.
Resolving deltas: 100% (8750/8750), done.
$ cd crosscat/
$ ls
LICENSE  Makefile  NOTICE  README.md  README_STARCLUSTER.md  cpp_code/  crosscat/  debian/  docs/  examples/  git_unittest_hook.sh*  requirements.txt  scripts/  setup.py  starcluster.config  website/  www/
$ python setup.py build
which: no ccache in (/ahg/regevdata/users/yarden/myenv/bin:/home/unix/yarden/bin:/broad/lsf9/9.1/linux2.6-glibc2.3-x86_64/bin:/ahg/regevdata/users/yarden/myenv/bin:/home/unix/yarden/bin:/ahg/regevdata/users/yarden/myenv/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/emacs_24.2/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/mysql_5.6.20/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/perl_5.14.2/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/bioperl_1.6.923/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/sqlite_3.6.19/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/tophat_2.0.10/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/bowtie2_2.0.5:/broad/software/free/Linux/redhat_6_x86_64/pkgs/bowtie_0.12.9:/broad/software/free/Linux/redhat_6_x86_64/pkgs/samtools/samtools_0.1.19/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/zlib_1.2.6/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/bedtools_version-2.16.1/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/scipy_0.13.0-python-2.7.1-sqlite3-rtrees/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/matplotlib_1.3.1-python-2.7.1-sqlite3-rtrees/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/db_4.7.25/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/tcltk8.5.9/bin:/broad/lsf9/9.1/linux2.6-glibc2.3-x86_64/etc:/broad/software/free/Linux/redhat_6_x86_64/pkgs/lsf_drmaa-1.1.1/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/r_3.1.0-bioconductor-2.14/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/gcc_4.9.0/bin:/home/unix/yarden/bin:/broad/tools/NoArch/pkgs/local:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/unix/yarden/regevdata/bedtools2/bin:/home/unix/yarden/regevdata/software/sratoolkit.2.4.2-ubuntu64/bin:/home/unix/yarden/regevdata/software/ncftp-3.2.5/bin:/home/unix/yarden/bin/:/home/unix/yarden/regevdata/bedtools2/bin:/home/unix/yarden/regevdata/software/sratoolkit.2.4.2-ubuntu64/bin:/home/unix/yarden/regevdata/software/ncftp-3.2.5/bin:/home/unix/yarden/bin/:/home/unix/yarden/regevdata/bedtools2/bin:/home/unix/yarden/regevdata/software/sratoolkit.2.4.2-ubuntu64/bin:/home/unix/yarden/regevdata/software/ncftp-3.2.5/bin:/home/unix/yarden/bin/)
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/crosscat
copying crosscat/EngineTemplate.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/starcluster_plugin.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/LocalEngine.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/settings.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/JSONRPCEngine.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/CrossCatClient.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/HadoopEngine.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/IPClusterEngine.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/__init__.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/MultiprocessingEngine.py -> build/lib.linux-x86_64-2.7/crosscat
creating build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/plot_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/validate_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/data_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/useCase_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/general_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/sample_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/file_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/api_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/hadoop_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/geweke_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/inference_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/mutual_information_test_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/diagnostic_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/enumerate_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/timing_test_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/xnet_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/convergence_test_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
creating build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/convergence_analysis/convergence_test.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/convergence_analysis/parse_convergence_results.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/convergence_analysis/generate_convergence_script.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/convergence_analysis/plot_convergence_results.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/convergence_analysis/automated_convergence_tests.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/convergence_analysis/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
creating build/lib.linux-x86_64-2.7/crosscat/jsonrpc_http
copying crosscat/jsonrpc_http/server_jsonrpc.py -> build/lib.linux-x86_64-2.7/crosscat/jsonrpc_http
copying crosscat/jsonrpc_http/test_resume.py -> build/lib.linux-x86_64-2.7/crosscat/jsonrpc_http
copying crosscat/jsonrpc_http/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/jsonrpc_http
copying crosscat/jsonrpc_http/stub_client_jsonrpc.py -> build/lib.linux-x86_64-2.7/crosscat/jsonrpc_http
copying crosscat/jsonrpc_http/test_engine.py -> build/lib.linux-x86_64-2.7/crosscat/jsonrpc_http
creating build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/test_multinomial_impute.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/mixed_state_test.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/test_sample.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/test_missing_value.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/continuous_component_model_test.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/test_pred_prob_and_density.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/setup.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/state_test.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
creating build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/timing_analysis.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/cpp_long_tests.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/test_sampler_enumeration.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/cpp_unit_tests.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/test_log_likelihood.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/geweke_on_schemas.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/test_pred_prob_and_density.py -> build/lib.linux-x86_64-2.7/crosscat/tests
creating build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/test_predictive_confidence.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/test_kl_divergence_quality.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/test_impute_quality.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/synthetic_data_generator.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/quality_test_utils.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/test_component_model_quality.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/test_mixture_inference_quality.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
creating build/lib.linux-x86_64-2.7/crosscat/tests/component_model_extensions
copying crosscat/tests/component_model_extensions/CyclicComponentModel.py -> build/lib.linux-x86_64-2.7/crosscat/tests/component_model_extensions
copying crosscat/tests/component_model_extensions/MultinomialComponentModel.py -> build/lib.linux-x86_64-2.7/crosscat/tests/component_model_extensions
copying crosscat/tests/component_model_extensions/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/tests/component_model_extensions
copying crosscat/tests/component_model_extensions/ContinuousComponentModel.py -> build/lib.linux-x86_64-2.7/crosscat/tests/component_model_extensions
running build_ext
building 'crosscat.cython_code.CyclicComponentModel' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/crosscat
creating build/temp.linux-x86_64-2.7/crosscat/cython_code
creating build/temp.linux-x86_64-2.7/cpp_code
creating build/temp.linux-x86_64-2.7/cpp_code/src
gcc -pthread -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c crosscat/cython_code/CyclicComponentModel.c -o build/temp.linux-x86_64-2.7/crosscat/cython_code/CyclicComponentModel.o
gcc -pthread -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/utils.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/utils.o
gcc -pthread -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/numerics.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/numerics.o
gcc -pthread -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/RandomNumberGenerator.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/RandomNumberGenerator.o
gcc -pthread -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/ComponentModel.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/ComponentModel.o
gcc -pthread -fno-strict-aliasing -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/tcltk8.5.9/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/sqlite_3.7.5/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/db_4.7.25/include -DNDEBUG -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/CyclicComponentModel.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/CyclicComponentModel.o
gcc: error: crosscat/cython_code/CyclicComponentModel.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 1

Is there a recommended Cython version? My cython version is:

$ python -c 'import cython; print cython.__version__'
0.21.1

Also, I noticed the first line in the output is:

which: no ccache in ...

I'm not sure what that means, but maybe it's related to the issue?

from crosscat.

BaxterEaves avatar BaxterEaves commented on July 25, 2024

I get the no ccache message as well. It shouldn't affect crosscat's building--just the time it takes to build. I was originally using cython 0.21.2 but I downgraded to 0.21.1 and was still unable to recreate the issue.

I notice that you are not getting any compiler output after the gcc commands. Your compiler flags are definitely different than mine. Could you please try running with

$ CFLAGS="-O2 -Wall" python setup.py build

and share the output? Also what version of gcc are you running?

from crosscat.

yarden avatar yarden commented on July 25, 2024

When run the command you suggested with CFLAGS set, I get:

which: no ccache in (/broad/lsf9/9.1/linux2.6-glibc2.3-x86_64/bin:/ahg/regevdata/users/yarden/myenv/bin:/home/unix/yarden/bin:/ahg/regevdata/users/yarden/myenv/bin:/home/unix/yarden/bin:/ahg/regevdata/users/yarden/myenv/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/emacs_24.2/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/mysql_5.6.20/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/perl_5.14.2/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/bioperl_1.6.923/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/sqlite_3.6.19/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/tophat_2.0.10/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/bowtie2_2.0.5:/broad/software/free/Linux/redhat_6_x86_64/pkgs/bowtie_0.12.9:/broad/software/free/Linux/redhat_6_x86_64/pkgs/samtools/samtools_0.1.19/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/zlib_1.2.6/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/bedtools_version-2.16.1/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/scipy_0.13.0-python-2.7.1-sqlite3-rtrees/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/matplotlib_1.3.1-python-2.7.1-sqlite3-rtrees/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/db_4.7.25/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/tcltk8.5.9/bin:/broad/lsf9/9.1/linux2.6-glibc2.3-x86_64/etc:/broad/software/free/Linux/redhat_6_x86_64/pkgs/lsf_drmaa-1.1.1/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/r_3.1.0-bioconductor-2.14/bin:/broad/software/free/Linux/redhat_6_x86_64/pkgs/gcc_4.9.0/bin:/home/unix/yarden/bin:/broad/tools/NoArch/pkgs/local:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/unix/yarden/regevdata/bedtools2/bin:/home/unix/yarden/regevdata/software/sratoolkit.2.4.2-ubuntu64/bin:/home/unix/yarden/regevdata/software/ncftp-3.2.5/bin:/home/unix/yarden/bin/:/home/unix/yarden/regevdata/bedtools2/bin:/home/unix/yarden/regevdata/software/sratoolkit.2.4.2-ubuntu64/bin:/home/unix/yarden/regevdata/software/ncftp-3.2.5/bin:/home/unix/yarden/bin/:/home/unix/yarden/regevdata/bedtools2/bin:/home/unix/yarden/regevdata/software/sratoolkit.2.4.2-ubuntu64/bin:/home/unix/yarden/regevdata/software/ncftp-3.2.5/bin:/home/unix/yarden/bin/)
running build
running build_py
copying crosscat/EngineTemplate.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/starcluster_plugin.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/LocalEngine.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/settings.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/JSONRPCEngine.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/CrossCatClient.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/HadoopEngine.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/IPClusterEngine.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/__init__.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/MultiprocessingEngine.py -> build/lib.linux-x86_64-2.7/crosscat
copying crosscat/utils/plot_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/validate_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/data_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/useCase_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/general_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/sample_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/file_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/api_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/hadoop_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/geweke_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/inference_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/mutual_information_test_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/diagnostic_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/enumerate_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/timing_test_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/xnet_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/utils/convergence_test_utils.py -> build/lib.linux-x86_64-2.7/crosscat/utils
copying crosscat/convergence_analysis/convergence_test.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/convergence_analysis/parse_convergence_results.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/convergence_analysis/generate_convergence_script.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/convergence_analysis/plot_convergence_results.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/convergence_analysis/automated_convergence_tests.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/convergence_analysis/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/convergence_analysis
copying crosscat/jsonrpc_http/server_jsonrpc.py -> build/lib.linux-x86_64-2.7/crosscat/jsonrpc_http
copying crosscat/jsonrpc_http/test_resume.py -> build/lib.linux-x86_64-2.7/crosscat/jsonrpc_http
copying crosscat/jsonrpc_http/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/jsonrpc_http
copying crosscat/jsonrpc_http/stub_client_jsonrpc.py -> build/lib.linux-x86_64-2.7/crosscat/jsonrpc_http
copying crosscat/jsonrpc_http/test_engine.py -> build/lib.linux-x86_64-2.7/crosscat/jsonrpc_http
copying crosscat/cython_code/test_multinomial_impute.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/mixed_state_test.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/test_sample.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/test_missing_value.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/continuous_component_model_test.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/test_pred_prob_and_density.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/setup.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/cython_code/state_test.py -> build/lib.linux-x86_64-2.7/crosscat/cython_code
copying crosscat/tests/timing_analysis.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/cpp_long_tests.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/test_sampler_enumeration.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/cpp_unit_tests.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/test_log_likelihood.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/geweke_on_schemas.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/test_pred_prob_and_density.py -> build/lib.linux-x86_64-2.7/crosscat/tests
copying crosscat/tests/quality_tests/test_predictive_confidence.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/test_kl_divergence_quality.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/test_impute_quality.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/synthetic_data_generator.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/quality_test_utils.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/test_component_model_quality.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/quality_tests/test_mixture_inference_quality.py -> build/lib.linux-x86_64-2.7/crosscat/tests/quality_tests
copying crosscat/tests/component_model_extensions/CyclicComponentModel.py -> build/lib.linux-x86_64-2.7/crosscat/tests/component_model_extensions
copying crosscat/tests/component_model_extensions/MultinomialComponentModel.py -> build/lib.linux-x86_64-2.7/crosscat/tests/component_model_extensions
copying crosscat/tests/component_model_extensions/__init__.py -> build/lib.linux-x86_64-2.7/crosscat/tests/component_model_extensions
copying crosscat/tests/component_model_extensions/ContinuousComponentModel.py -> build/lib.linux-x86_64-2.7/crosscat/tests/component_model_extensions
running build_ext
building 'crosscat.cython_code.CyclicComponentModel' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/crosscat
creating build/temp.linux-x86_64-2.7/crosscat/cython_code
creating build/temp.linux-x86_64-2.7/cpp_code
creating build/temp.linux-x86_64-2.7/cpp_code/src
gcc -pthread -DNDEBUG -O2 -Wall -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c crosscat/cython_code/CyclicComponentModel.c -o build/temp.linux-x86_64-2.7/crosscat/cython_code/CyclicComponentModel.o
gcc -pthread -DNDEBUG -O2 -Wall -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/utils.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/utils.o
gcc -pthread -DNDEBUG -O2 -Wall -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/numerics.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/numerics.o
gcc -pthread -DNDEBUG -O2 -Wall -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/RandomNumberGenerator.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/RandomNumberGenerator.o
gcc -pthread -DNDEBUG -O2 -Wall -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/ComponentModel.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/ComponentModel.o
gcc -pthread -DNDEBUG -O2 -Wall -fPIC -Icpp_code/include/CrossCat -I/ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/numpy/core/include -I/broad/software/free/Linux/redhat_5_x86_64/pkgs/python_2.7.1-sqlite3-rtrees/include/python2.7 -c cpp_code/src/CyclicComponentModel.cpp -o build/temp.linux-x86_64-2.7/cpp_code/src/CyclicComponentModel.o
gcc: error: crosscat/cython_code/CyclicComponentModel.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 1
cpp_code/src/CyclicComponentModel.cpp: In member function 'virtual double CyclicComponentModel::get_draw_constrained(int, std::vector<double>) const':
cpp_code/src/CyclicComponentModel.cpp:199:23: warning: variable 'b_a' set but not used [-Wunused-but-set-variable]
   double kappa, a, b, b_a;
                       ^
cpp_code/src/CyclicComponentModel.cpp:245:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
cpp_code/src/utils.cpp: In function 'void construct_cyclic_specific_hyper_grid(int, std::vector<double>, std::vector<double>&, std::vector<double>&)':
cpp_code/src/utils.cpp:428:10: warning: unused variable 'var' [-Wunused-variable]
   double var = calc_sum_sq_deviation(col_data)/N;
          ^
cpp_code/src/numerics.cpp: In function 'double numerics::vonmises_rand(double, double, int)':
cpp_code/src/numerics.cpp:84:20: warning: unused variable 'vmr_a' [-Wunused-variable]
             double vmr_a = vmr;
                    ^
cpp_code/src/numerics.cpp:91:20: warning: unused variable 'vmr_a' [-Wunused-variable]
             double vmr_a = vmr;

My gcc is:

$ gcc --version
gcc (GCC) 4.9.0
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

from crosscat.

BaxterEaves avatar BaxterEaves commented on July 25, 2024

I am going to spend some time today with our packaging. It's probably best to distribute the cython-generated .cpp files. I'll let you know when it's ready.

from crosscat.

riastradh-probcomp avatar riastradh-probcomp commented on July 25, 2024

The GCC version and CFLAGS are irrelevant, and pip is a red herring. What's happening (or, failing to happen) is that Cython is somehow not getting run to compile the .pyx files into .c/.cpp files. The problem is not that Cython is not available -- if that were the case, setup.py would fail altogether because it imports from Cython.Distutils.

For 'source distributions' generated from the real source in Git, i.e. the product of python setup.py sdist, it may be appropriate to include the Cython-generated .c/.cpp files. But that doesn't resolve the question of why Cython isn't getting run for @yarden.

from crosscat.

riastradh-probcomp avatar riastradh-probcomp commented on July 25, 2024

I'm using the version of Cython packaged with Ubuntu 14.04, namely 0.20.1post0. Generally I don't trust pip at all and don't tolerate using it on any of my systems, but I'll see what happens if I make it install Cython through pip in a virtualenv.

from crosscat.

riastradh-probcomp avatar riastradh-probcomp commented on July 25, 2024

I removed my crosscat and cython system packages from apt, and ran

virtualenv /tmp/venv
. /tmp/venv/activate
pip install cython
pip install .

and the installation succeeded. This is from Ubuntu 14.04, and this installed Cython 0.21.2 in the virtualenv. (Without pip install cython first, running pip install . will fail because setup.py imports Cython.Distutils. I don't know enough about how pip works to make this DTRT.)

from crosscat.

yarden avatar yarden commented on July 25, 2024

It looks like maybe pip and Cython specifically have a problem (http://stackoverflow.com/questions/26069080/pip-does-not-acknowledge-cython). However, it does not explain the current issue. I tried doing:

$ pip install --upgrade cython
$ pip install .

and got the same errors:

...
gcc: error: crosscat/cython_code/CyclicComponentModel.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 1

Using sdist to make a non-Cython dependent source distribution would be great.

Python packaging is a broken business, but using a package manager like pip+virtualenv or easy_install is pretty much the only option for doing a complete install. python setup.py does not do a complete install of packages and obviously doesn't let users install the program and its dependencies from pypi or other repositories with a single command. As far as I know, pip+virtualenv is the only game in town for managing multiple sets of potentially conflicting Python dependencies, which inevitably happens.

Update: I upgraded setuptools and now it appears to work:

$ pip install setuptools
Requirement already satisfied (use --upgrade to upgrade): setuptools in /ahg/regevdata/users/yarden/myenv/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
$ pip install --upgrade setuptools
Collecting setuptools from https://pypi.python.org/packages/3.4/s/setuptools/setuptools-12.0.5-py2.py3-none-any.whl#md5=87647eb8380c5da10e63fdd7274d8b49
  Downloading setuptools-12.0.5-py2.py3-none-any.whl (502kB)
    100% |################################| 503kB 437kB/s 
Installing collected packages: setuptools
  Found existing installation: setuptools 0.6rc11
    Uninstalling setuptools-0.6rc11:
      Successfully uninstalled setuptools-0.6rc11

I'm able to run the dha_example.py at least:

$ python examples/dha_example.py www/data/dha.csv --num_chains 2 --num_transitions 2

[6.7, 5.8, 15.9, 87.7, 92.0]
[6.71, 3.24, 15.05, 87.97, 86.27]

[9.5, 5.8, 21.4, 86.7, 89.2]
[9.03, 7.38, 19.78, 87.76, 86.24]

[9.2, 15.2, 20.2, 82.2, 85.3]
[11.19, 10.06, 22.88, 86.14, 84.61]

[9.4, 13.2, 25.1, 92.3, 92.2]
[11.3, 9.92, 23.03, 89.08, 87.63]

[11.3, 9.7, 23.6, 80.8, 86.3]
[11.22, 10.0, 22.79, 86.14, 84.43]

[7.8, 8.4, 19.9, 89.5, 89.4]
[9.08, 7.37, 19.87, 89.13, 87.68]

[8.4, 7.6, 19.5, 89.7, 92.4]
[11.37, 9.83, 23.04, 89.21, 87.71]

[6.5, 2.7, 15.1, 87.1, 85.5]
[7.28, 4.47, 17.08, 87.86, 85.97]

So it looks like old setuptools is the problem.

from crosscat.

riastradh-probcomp avatar riastradh-probcomp commented on July 25, 2024

Problem caused by prehistoric version of setuptools, not by anything in crosscat. We can open another issue for running cython during python setup.py sdist and making python setup.py build DTRT (if possible given how unbelievably brain-damaged the Python build nonsense is).

from crosscat.

yarden avatar yarden commented on July 25, 2024

A simple fix for the problem that I had is to add setuptools >= x in the setup.py requirements section, to ensure old versions aren't used.

About Cython: It's possible to make setup.py sdist do the right thing with Cython - I have a template that does that here, FYI: https://github.com/yarden/MISO/blob/newmiso/setup.py
But that issue is not pressing for me now that Crosscat is working.

from crosscat.

riastradh-probcomp avatar riastradh-probcomp commented on July 25, 2024

FYI, we just pushed changes to make python setup.py sdist run Cython so that python setup.py build from the resulting 'source' distribution need not.

from crosscat.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.