assay on Python 3.8.3 fails testing skyfield about assay

[brandon-rhodes]

Assay is working fine under Python 3.8 on my own system, so we need to uncover how the underlying OS calls are behaving differently on yours. Looking at where that list of paths is coming from that apparently contains a None on your system:

  File "/usr/lib64/python3.8/posixpath.py", line 152, in dirname
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType

— it looks like it comes from the routine list_module_paths() in iteration.py, because that's where main_loop() gets its list main_process_paths. The routine gets the paths like this:

def list_module_paths():
    items = list(sys.modules.items())
    return [(name, module.__file__) for name, module in items
            if (module is not None) and hasattr(module, '__file__')]

So there must be a module in your Python that has a __file__ attribute, but for which its value is None? I have never seen that before. Right next to the line that sets main_process_paths in main_loop(), you could try:

    print('Crazy:', [name for name, path in list_module_paths() if path is None])

— and we would maybe get the name of the module with the unexpected __file__ value. If 3.8 has simply added such modules and they will be normal from now on, then we can tweak list_module_paths() to ignore None values. (But first I'm very interested in finding out where it's coming from!)

As for your second run: it looks like the assert rewriting ran into trouble? Before having you spend any time on it, I'll today or tomorrow go add the CI for assay that I mentioned the other day, to see if there's a 3.8 issue I can squash on this end before incurring any further time of yours. If I can't reproduce it here or in CI, then I'll comment further on this issue for information about what's going on in your system.

[bnavigator]

abuild@greinerT450s:~/rpmbuild/BUILD/skyfield-1.24> python3 -m assay skyfields.tests
Crazy: ['mpl_toolkits']
Traceback (most recent call last):
  File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/abuild/rpmbuild/BUILD/assay-18f320237345813a49173657e7d2d6ca85e9a38a/assay/__main__.py", line 2, in <module>
    main()
  File "/home/abuild/rpmbuild/BUILD/assay-18f320237345813a49173657e7d2d6ca85e9a38a/assay/command.py", line 22, in main
    monitor.main_loop(args.name, args.batch or not isatty)
  File "/home/abuild/rpmbuild/BUILD/assay-18f320237345813a49173657e7d2d6ca85e9a38a/assay/monitor.py", line 45, in main_loop
    file_watcher.add_paths(main_process_paths)
  File "/home/abuild/rpmbuild/BUILD/assay-18f320237345813a49173657e7d2d6ca85e9a38a/assay/filesystem.py", line 80, in add_paths
    paths = set(dirname(path) for path in file_paths) - self.paths
  File "/home/abuild/rpmbuild/BUILD/assay-18f320237345813a49173657e7d2d6ca85e9a38a/assay/filesystem.py", line 80, in <genexpr>
    paths = set(dirname(path) for path in file_paths) - self.paths
  File "/usr/lib64/python3.8/posixpath.py", line 152, in dirname
    p = os.fspath(p)

abuild@greinerT450s:~/rpmbuild/BUILD/skyfield-1.24> ls /usr/lib64/python3.8/site-packages/mpl_toolkits/
axes_grid  axes_grid1  axisartist  mplot3d  tests

mpl_toolkits is shipped with matplotlib. Now why the __init__.py is missing, I have no idea yet (I updated the package myself a few days ago. Maybe my fault...)

[bnavigator]

From the matplotlib package buildlog during install_lib:
[ 108s] Skipping installation of /home/abuild/rpmbuild/BUILDROOT/python-matplotlib-3.3.0-38.1.x86_64/usr/lib64/python3.8/site-packages/mpl_toolkits/__init__.py (namespace package)

https://packaging.python.org/guides/packaging-namespace-packages/

[brandon-rhodes]

Thank you! That solves the mystery. Probably tomorrow I'll have time to get a fix committed.

[brandon-rhodes]

Thanks for letting me know about the dangers of new PEP 420 namespace packages! I've committed both a fix and a test, so hopefully this will stay fixed.

[bnavigator]

Can confirm, the skyfield tests now pass in batch mode.

Without batch, after running all the tests, it still needs a keyboard interrupt to terminate the process. Is this intentional?

As there is no need to rewrite any assertion failures with the current passing skyfield package, I cannot confirm that this has been fixed.

Filing another issue with failures on ARM and PowerPC shortly.

[brandon-rhodes]

Without batch, after running all the tests, it still needs a keyboard interrupt to terminate the process. Is this intentional?

Yes, intentional: without --batch, assay runs in interactive mode where it re-runs the tests immediately when the developer hits Save on an edit to a Python file.

As there is no need to rewrite any assertion failures with the current passing skyfield package, I cannot confirm that this has been fixed.

Assertion rewriting is, happily, tested directly in assay’s own test suite, which now passes under 3.8 following that Python version’s update to the fields supported by raw Python code objects:

21554af