[FEATURE] mypy: Module "frozendict" has no attribute "frozendict" about python-frozendict

[Marco-Sulla]

Mysteries of computer science... I think I can close the issue :)

[amyreese]

@Marco-Sulla the problem is that I get frozendict installed automatically as a dependency of the package I'm developing. pip automatically chooses from the published wheels on PyPI with no meaningful input from me. On an x86 runtime, it can install one of the wheels that includes the C ext, but because there are no arm64 wheels published, pip on my M1 Mac will fall back to the published none-any wheel, which of course doesn't include the C ext.

My only option is to either forego the wheels on all platforms (not great), or build in an extra step on arm64 machines that explicitly uninstalls frozendict just to then build and install it again from source in order to get the C ext (even worse), just to work around type checking not seeing frozendict in the pure-python version.

There are three possible ways that would resolve or mitigate this situation:

• Update the way frozendict/__init__.py generates __all__ so that it exports the same set of names, regardless of whether the C extension is available. A static list that includes a consistent "public API" shared by both implementations would be the best option IMO.
• Publish wheels for arm64 Macs so that pip doesn't fall back to none-any wheels on that platform. cibuildwheel already supports this, and it's just a matter of setting up the build job to make it happen.
• Don't publish a none-any wheel, forcing pip to build from source (and therefore build the C ext by default?) when on any platform that doesn't have a prebuilt wheel with the C ext. I'm guessing this potentially breaks installing frozendict on weird platforms that also don't have a compiler available.

My preference would be towards implementing the first two options together, so that type checking is consistent for all installs of frozendict, and also so that I can get a prebuilt version on arm64 Macs. The third option is the mildly-spiteful option that technically works, but is not what I would choose. 😅

I'm happy to offer a PR for the first two options if that helps.

[MrCreosote]

$ python -c "import frozendict; print(frozendict.c_ext)"
True

[MrCreosote]

Not a fan of this workaround, but it does the trick:

from frozendict import frozendict  # type: ignore[attr-defined]

[lurch]

Is this a duplicate of #39 ?

[MrCreosote]

From reading the thread it's not clear to me

[Marco-Sulla]

This should work now after #62. Can you confirm?

[MrCreosote]

I've never actually installed a package outside of pypi, so I'm not totally sure I'm doing this correctly...

(py3.9playground) ~/virtualenvs/py3.9playground$ cat fdict.py 
from frozendict import frozendict
(py3.9playground) ~/virtualenvs/py3.9playground$ grep frozendict Pipfile
frozendict = "*"
(py3.9playground) ~/virtualenvs/py3.9playground$ mypy fdict.py
fdict.py:1: error: Module "frozendict" has no attribute "frozendict"
Found 1 error in 1 file (checked 1 source file)
(py3.9playground) ~/virtualenvs/py3.9playground$ python --version
Python 3.9.6
(py3.9playground) ~/virtualenvs/py3.9playground$ python -c "import frozendict; print(frozendict.__version__)"
2.3.4
(py3.9playground) ~/virtualenvs/py3.9playground$ mypy --version
mypy 0.971 (compiled: yes)
(py3.9playground) ~/virtualenvs/py3.9playground$ pipenv install git+https://github.com/Marco-Sulla/python-frozendict#egg=frozendict
Installing git+https://github.com/Marco-Sulla/python-frozendict#egg=frozendict...
*snip*
(py3.9playground) ~/virtualenvs/py3.9playground$ python -c "import frozendict; print(frozendict.__version__)"
2.3.4
(py3.9playground) ~/virtualenvs/py3.9playground$ mypy fdict.py
fdict.py:1: error: Module "frozendict" has no attribute "frozendict"
Found 1 error in 1 file (checked 1 source file)
(py3.9playground) ~/virtualenvs/py3.9playground$ grep frozendict Pipfile
frozendict = {git = "https://github.com/Marco-Sulla/python-frozendict"}

[MrCreosote]

Pipfile.lock:

"frozendict": {
            "git": "https://github.com/Marco-Sulla/python-frozendict",
            "hashes": [
                "sha256:15b4b18346259392b0d27598f240e9390fafbff882137a9c48a1e0104fb17f78",
                "sha256:25a6d2e8b7cf6b6e5677a1a4b53b4073e5d9ec640d1db30dc679627668d25e90",
                "sha256:389f395a74eb16992217ac1521e689c1dea2d70113bcb18714669ace1ed623b9",
                "sha256:3d8042b7dab5e992e30889c9b71b781d5feef19b372d47d735e4d7d45846fd4a",
                "sha256:3e93aebc6e69a8ef329bbe9afb8342bd33c7b5c7a0c480cb9f7e60b0cbe48072",
                "sha256:3ec86ebf143dd685184215c27ec416c36e0ba1b80d81b1b9482f7d380c049b4e",
                "sha256:4a3b32d47282ae0098b9239a6d53ec539da720258bd762d62191b46f2f87c5fc",
                "sha256:5809e6ff6b7257043a486f7a3b73a7da71cf69a38980b4171e4741291d0d9eb3",
                "sha256:7c550ed7fdf1962984bec21630c584d722b3ee5d5f57a0ae2527a0121dc0414a",
                "sha256:84c9887179a245a66a50f52afa08d4d92ae0f269839fab82285c70a0fa0dd782",
                "sha256:95bac22f7f09d81f378f2b3f672b7a50a974ca180feae1507f5e21bc147e8bc8",
                "sha256:aca59108b77cadc13ba7dfea7e8f50811208c7652a13dc6c7f92d7782a24d299",
                "sha256:b98a0d65a59af6da03f794f90b0c3085a7ee14e7bf8f0ef36b079ee8aa992439",
                "sha256:ccb6450a416c9cc9acef7683e637e28356e3ceeabf83521f74cc2718883076b7",
                "sha256:d722f3d89db6ae35ef35ecc243c40c800eb344848c83dba4798353312cd37b15",
                "sha256:dae686722c144b333c4dbdc16323a5de11406d26b76d2be1cc175f90afacb5ba",
                "sha256:dbbe1339ac2646523e0bb00d1896085d1f70de23780e4927ca82b36ab8a044d3"
            ],
            "ref": "de2d7b9bc75e148301f89c4b706b524f460cdef9"
        },

The ref looks correct

[Marco-Sulla]

It works for me. I don't have pipenv anymore. This is what I've done:

marco@buzz:~/sources$ git clone https://github.com/Marco-Sulla/python-frozendict.git python-frozendict-test
marco@buzz:~/sources$ cd python-frozendict-test/
marco@buzz:~/sources/python-frozendict-test$ ~/sources/cpython_3_9_build/python -m venv venv_test
marco@buzz:~/sources/python-frozendict-test$ . venv_test/bin/activate
(venv_test) marco@buzz:~/sources/python-frozendict-test$ pip install -U pip setuptools wheel
(venv_test) marco@buzz:~/sources/python-frozendict-test$ ./build.sh bdist_wheel
(venv_test) marco@buzz:~/sources/python-frozendict-test$ pip install dist/frozendict-2.3.4-cp39-cp39-linux_x86_64.whl 
(venv_test) marco@buzz:~/sources/python-frozendict-test$ cd test/
(venv_test) marco@buzz:~/sources/python-frozendict-test/test$ python -c "import frozendict; print(frozendict.c_ext)"
True
(venv_test) marco@buzz:~/sources/python-frozendict-test/test$ echo "from frozendict import frozendict" > typed2.py 
(venv_test) marco@buzz:~/sources/python-frozendict-test/test$ pip install mypy
(venv_test) marco@buzz:~/sources/python-frozendict-test/test$ mypy typed2.py 
Success: no issues found in 1 source file

[MrCreosote]

Interesting, I replicated what you did using pipenv, other than installing the built wheel with pip, and the c extension wasn't being used and mypy was very unhappy. I then repeated with plain old venv and was able to reproduce your results.

I assumed that pipenv and venv modify the environment in the same way but perhaps not.

[amyreese]

This is still an issue anywhere where existing published wheels aren't available, eg, M1 Macs using ARM64 runtimes. Or maybe frozendict should be updated to try building with the C extension by default?

Edit: I guess the problem is that on ARM64, pip falls back to using the none-any wheel rather than building from source.

[Marco-Sulla]

@amyreese This is strange, since I followed the same steps described here: #62 (comment) but using

./build.sh py bdist_wheel
pip install dist/*

instead, and the result is correct.

What version of frozendict have you installed?

[Marco-Sulla]

@amyreese Try also:

pip install --no-binary :all: frozendict

[Marco-Sulla]

@amyreese I quote myself:

This is strange, since I followed the same steps described here: #62 (comment) but using

./build.sh py bdist_wheel
pip install dist/*

instead, and the result is correct.

What version of frozendict have you installed?

And I add: can you follow the steps described above?

[ydirson]

If I upgrade my venv with the current version, the "attribute" problem appears to be fixed:

(env) $ mypy scripts/test-mypy-frozendict.py
scripts/test-mypy-frozendict.py:1: error: Module "frozendict" has no attribute "frozendict"  [attr-defined]
Found 1 error in 1 file (checked 1 source file)
(env) $ pip install -U ~/soft/python-frozendict/
Processing /home/user/soft/python-frozendict
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: frozendict
  Building wheel for frozendict (setup.py) ... done
  Created wheel for frozendict: filename=frozendict-2.3.4-cp39-cp39-linux_x86_64.whl size=102123 sha256=dfeaf14e4c98a6057d93019cd421b0249778f8f9da51bbe951eec94df13225e8
  Stored in directory: /home/user/.cache/pip/wheels/dc/d7/5d/09cf55e872d2f9dbd5e62aa4d18bc5f7b5c36a5011580d3a97
Successfully built frozendict
Installing collected packages: frozendict
  Attempting uninstall: frozendict
    Found existing installation: frozendict 2.3.4
    Uninstalling frozendict-2.3.4:
      Successfully uninstalled frozendict-2.3.4
Successfully installed frozendict-2.3.4
(env) $ mypy scripts/test-mypy-frozendict.py
Success: no issues found in 1 source file

However, the .pyi file does not appear to be correct, as the frozendict type inside the module does not get the proper type, and subsequently usage of frozentype type reports error: Module not callable

from frozendict import frozendict
reveal_type(frozendict)

gets mypy to show:

note: Revealed type is "types.ModuleType"