Dependency to source only package cause error about pyflow

[jefaxot]

Hi!
I'm not familiar with rust but it seems to be an issue of messy resolved relative paths.

These dirty changes seem to fix the issue:

diff --git a/src/install.rs b/src/install.rs
index 948e838..807d821 100644
--- a/src/install.rs
+++ b/src/install.rs
@@ -335,17 +335,27 @@ pub fn download_and_install_package(
             // todo moves, only copies. Figure out how to do a normal move,
             // todo, to speed this up.
 
-            let extracted_parent = paths.lib.join(folder_name);
+            let extracted_parent = fs::canonicalize(paths.lib.join(folder_name));
+            let extracted_parent = match extracted_parent {
+                Ok(path) => path,
+                Err(error) => panic!("Problem converting path to absolute path: {:?}", error),
+            };
+            let python = fs::canonicalize(paths.bin.join("python"));
+            let python = match python {
+                Ok(path) => path,
+                Err(error) => panic!("Problem converting path to absolute path: {:?}", error),
+            };
+
 
             replace_distutils(&extracted_parent.join("setup.py"));
 
             // Build a wheel from source.
             //            println!("EX PAR: {:#?} bin: {:#?}", &extracted_parent, &paths.bin);
-            Command::new(paths.bin.join("python"))
+            Command::new(&python)
                 .current_dir(&extracted_parent)
                 .args(&["setup.py", "bdist_wheel"])
                 .output()
-                .expect(&format!("Problem running setup.py bdist_wheel in folder: {:?}. Py path: {:?}", &extracted_parent, paths.bin.join("python")));
+                .expect(&format!("Problem running setup.py bdist_wheel in folder: {:?}. Py path: {:?}", &extracted_parent, &python));
 
             let dist_path = &extracted_parent.join("dist");
             if !dist_path.exists() {

I let you write a better version (it is the first time I modify code in rust).

[David-OConnor]

@lhaze taking a look now. This one's been unfortunately tough to track down, although @jefaxot 's solution's almost there.

[miurahr]

By doing things by hand, it may be what happened under the hood:

~/tmp/proj/ (master #)$ cd __pypackages__/3.8/lib/ifaddr-0.1.6/
~/tmp/proj/__pypackages__/3.8/lib/ifaddr-0.1.6 (master #)$ ls
MANIFEST.in  PKG-INFO  README.rst  ifaddr  ifaddr.egg-info  setup.cfg  setup.py
~/tmp/proj/__pypackages__/3.8/lib/ifaddr-0.1.6 (master #)$ pyflow setup.py bdist_wheel
⬇ Installing ifaddr 0.1.6 ...
/home/miurahr/tmp/proj/__pypackages__/3.8/.venv/bin/python: can't open file 'setup.py': [Errno 2] No such file or directory
~/tmp/proj/__pypackages__/3.8/lib/ifaddr-0.1.6$ ls
~/tmp/proj/__pypackages__/3.8/lib/ifaddr-0.1.6$ cd ..
~/tmp/proj/__pypackages__/3.8/lib (master #)$ ls
ifaddr  ifaddr-0.1.6.dist-info  wheel  wheel-0.33.6.dist-info
~/tmp/proj/__pypackages__/3.8/lib (master #)$ cd ifaddr
~/tmp/proj/__pypackages__/3.8/lib/ifaddr (master #)$ ls
__init__.py  _posix.py  _shared.py  _win32.py  test_ifaddr.py

[David-OConnor]

I think this was fixed between the 0.2.3 release and now. I just release 0.2.4 - let me know if you still get the error.

[David-OConnor]

Nvm; not fixed.

[eferm]

Hi! I believe I'm seeing the same issue when trying to install jupyter.
I'm on MacOS Mojave using pyflow 0.2.5.

$ pyflow new test
Please enter the Python version for this project: (eg: 3.8)
3.8
Created a new Python project named test
$ cd test
$ pyenv local 3.8.2
$ RUST_BACKTRACE=full pyflow install jupyter
Found multiple compatible Python versions. Please enter the number associated with the one you'd like to use:
1: python3.8: 3.8.2
2: python3: 3.8.2
3: python: 3.8.2
1
🐍 Setting up Python...
Installing multiple versions for prompt-toolkit. If this package uses compiled code or importlib, this may fail when importing. Note that your package may not be published unless this is resolved...
Installing these versions: [
    "name: prompt-toolkit, version: 1.0.18, parent: 3",
    "name: prompt-toolkit, version: 3.0.4, parent: 54",
]
⬇ Installing decorator 4.4.2 ...
⬇ Installing attrs 19.3.0 ...
⬇ Installing pyzmq 19.0.0 ...
⬇ Installing pywinpty 0.5.7 ...
thread 'main' panicked at 'Problem running setup.py bdist_wheel: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1188:5
stack backtrace:
   0:        0x1074bfea5 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h253e83cc475a85d0
   1:        0x1074db8e1 - core::fmt::write::h5b2ae886983aa058
   2:        0x1074bf78b - std::io::Write::write_fmt::h25524112e4cce8f5
   3:        0x1074c4963 - std::panicking::default_hook::{{closure}}::h770880a3b7065f3d
   4:        0x1074c468f - std::panicking::default_hook::h8a113dafa31143b6
   5:        0x1074c507b - std::panicking::rust_panic_with_hook::ha2f5dac743ef67b3
   6:        0x1074c4be9 - rust_begin_unwind
   7:        0x1074e0ccc - core::panicking::panic_fmt::h1bfadfdf9956bc6d
   8:        0x1074dafe9 - core::result::unwrap_failed::h4180424c92aae69a
   9:        0x10713d204 - pyflow::install::download_and_install_package::hbff2fe17ab70fc4a
  10:        0x1071220de - pyflow::sync::h76e42fa2c5774708
  11:        0x10712cb11 - pyflow::main::h31895db4f25886b8
  12:        0x1071318e6 - std::rt::lang_start::{{closure}}::hde567998e918cf01
  13:        0x1074c4a88 - std::panicking::try::do_call::hcc2f2b88c6645a8c
  14:        0x1074c780f - __rust_maybe_catch_panic
  15:        0x1074b1b45 - std::rt::lang_start_internal::h5c91a3e7c72ca35f
  16:        0x1071310a9 - main```

[David-OConnor]

I'm suspicious #57 and this are caused by the same bug. When encountering a source-only distro, pyflow builds a wheel, then installs that. Something's failing when attempting to run python setup.py bdist_wheel, but I've been unable to determine what.

[David-OConnor]

Awesome! I'll poke around with it today. You're always welcome to make a PR like this, and I can edit it as required.

[jefaxot]

Probably better to fix it here:

diff --git a/src/util.rs b/src/util.rs
index dd898c0..e0d8236 100644
--- a/src/util.rs
+++ b/src/util.rs
@@ -511,6 +511,12 @@ pub fn find_or_create_venv(
         }
     }
 
+    let vers_path = fs::canonicalize(vers_path);
+    let vers_path = match vers_path {
+        Ok(path) => path,
+        Err(error) => panic!("Problem converting path to absolute path: {:?}", error),
+    };
+
     (vers_path, py_vers)
 }

[David-OConnor]

Added a tweaked version of your code to this commit: df772c1

Added your fix in both places, for Linux and Mac paths (canonicalize seems to break on Windows, and isn't required). I still ran some issues with certain versions of the custom py binaries not being able to build wheels. Might need to recompile those to get this fully fixed. (When you compile Python using the default instructions, it omits certain packages that normally come with Python)

I'll release as 3.6 once tested more thoroughly.

[jefaxot]

I don't think the correction is needed in both install.rs and utils.rs. To be DRY, I would only keep the correction in utils.rs.

[lhaze]

Please consider if the bug can be fixed as fast as it is possible. I'm blocked with this issue. Unfortunately I can't help with solving it.

[David-OConnor]

Current status is @jefaxot's fix work (after some, but not extensive) testing indicates it works on Windows with both system and pyflow-installed Python, and Linux on system-installed Python. It still fails for source dependencies on Linux if using a python installed by pyflow. If I can't sort this out by the end of the day, will make a new release as-is.

edit: the exact error is ModuleNotFoundError: No module named '_ctypes'. This implies the python I built was missing some things that are required to build wheels. Will workaround this by reverting to system python, which should be version insensitive, and all Linux installs should have it. The proper fix is rebuilding the binaries.

[David-OConnor]

0.2.6 release should fix this