poetry add gumbo_client does not work due to psycopg2 about depmap_omics

[pgm]

Ah, I see. The issue is that you get this error when you attempt to add gumbo client as a dependency (but it is not currently a dependency).

Now, I have a partial answer for why this is failing to install when run under poetry vs pip install:

When you install via pip, it's actually downloading a pre-compiled version of that package.

(When I run pip install ..., it reports: Downloading psycopg2_binary-2.9.5-cp311-cp311-macosx_11_0_arm64.whl (2.0 MB))

whereas when you attempt to install it via poetry, it's trying to compile the source code for psycopg2 on your machine. (This actually also works for me because I have postgresql installed on my machine. The pg_config executable that you're missing is part of a postgresql installation.)

What I don't know is why poetry is not using the prebuilt package, and instead trying to built it from source.

If installing postgresql (I'd recommend using https://postgresapp.com/ for the mac) is a viable solution, it'll fix this issue.

[pgm]

I suspect there is more in that error message above the line "This error originates..." as I suspect that the "this" refers to a very long error message that proceeded that line.

Also, any other details you can provide on how to reproduce? What did you run that resulted in this error?

Github actions doesn't seem to have a problem installing psycopg2 as you can see the build-docker is successful each time. That implies its some environmental difference between the github actions and when running locally.

I'm also able to run poetry install locally on my mac without any error.

[qinqian]

Thanks @pgm for the comment. The way to reproduce the error is:

git checkout dev-maf
mamba create -n cds-omics python==3.9
conda activate cds-omics
poetry lock --no-update
poetry install 
poetry add git+https://github.com/broadinstitute/gumbo_client.git

The full errors are:

Updating dependencies
Resolving dependencies... (10.1s)

Writing lock file

Package operations: 1 install, 6 updates, 0 removals

  • Updating platformdirs (2.6.2 -> 3.1.0)
  • Updating cffi (1.14.6 -> 1.15.1)
  • Updating charset-normalizer (2.1.1 -> 3.0.1)
  • Updating setuptools (65.6.3 -> 67.4.0)
  • Updating soupsieve (2.3.2.post1 -> 2.4)
  • Installing psycopg2 (2.9.5): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  /var/folders/5b/71z1tltn429bywpnr2d7gxv40000gq/T/tmp5hyb50i4/.venv/lib/python3.9/site-packages/setuptools/config/setupcfg.py:520: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
    warnings.warn(msg, warning_class)
  running egg_info
  writing psycopg2.egg-info/PKG-INFO
  writing dependency_links to psycopg2.egg-info/dependency_links.txt
  writing top-level names to psycopg2.egg-info/top_level.txt
  
  Error: pg_config executable not found.
  
  pg_config is required to build psycopg2 from source.  Please add the directory
  containing pg_config to the $PATH or specify the full executable path with the
  option:
  
      python setup.py build_ext --pg-config /path/to/pg_config build ...
  
  or with the pg_config option in 'setup.cfg'.
  
  If you prefer to avoid building psycopg2 from source, please install the PyPI
  'psycopg2-binary' package instead.
  
  For further information please check the 'doc/src/install.rst' file (also at
  <https://www.psycopg.org/docs/install.html>).
  
  

  at ~/miniconda3/envs/cds-omics/lib/python3.9/site-packages/poetry/installation/chef.py:152 in _prepare
      148│ 
      149│                 error = ChefBuildError("\n\n".join(message_parts))
      150│ 
      151│             if error is not None:
    → 152│                 raise error from None
      153│ 
      154│             return path
      155│ 
      156│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with psycopg2 (2.9.5) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "psycopg2 (==2.9.5) ; python_version >= "3.6""'.

The reason build-docker is not raising such an error because gumbo_client is neither a dependency for depmap_omics nor depmap_omics_upload.

[pgm]

...also it's worth noting that "conda activate cds-omics" has no effect in the above. poetry creates its own virtual environment.

In other words, you can't use conda and poetry together. They're mutually exclusive.

[pgm]

...Okay, I understand now the issue with the gumbo_client package. It has two different installation scripts which is a source of confusion for these installer tools.

I'll talk to Sarah to get that straightened out.

[qinqian]

Thanks! These explanations are very helpful. I'll use https://postgresapp.com/ for a simple fix now.

As for conda activate cds-omics, I thought poetry install will install anything into the current activated environment by default unless we specify poetry env use. That's why I used both conda and poetry together, I'll look into it to see if we should use poetry alone with other python version tool such as pyenv.

[pgm]

poetry attempts to make life simpler by handing management of virtual environments _and_ dependencies. You can disable it's handling of virtual environments by setting virtualenvs.create = false (see https://python-poetry.org/docs/configuration/ for more info) but by default, "poetry install" is going to create a virtual environment and install the dependencies within them. Similarly you'll need to use that environment by either using "poetry shell" or "poetry run ..." to be able to see the newly installed packages. Maybe we could sit down and go through your setup if that'd be helpful sometime?

…

On Sat, Mar 4, 2023 at 9:08 PM Alvin Qin ***@***.***> wrote: Thanks! These explanations are very helpful. I'll use https://postgresapp.com/ for a simple fix now. As for conda activate cds-omics, I thought poetry install will install anything into the current activated environment by default unless we specify poetry env use. That's why I used both conda and poetry together, I'll look into it to see if we should use poetry alone with other python version tool such as pyenv. — Reply to this email directly, view it on GitHub <#113 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACSUSHA45U4OJMQSMEKXDW2PYRDANCNFSM6AAAAAAVPAGAIE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[qinqian]

Thanks for the detailed explanation! Sure, let's chat today.