insarlab / pysolid Goto Github PK

pysolid is now available on PyPI (https://pypi.org/project/pysolid/). But pip install pysolid does not work, because a Fortran compiler is required to build the package but is not available via pip, as far as I know. It will be great if pip install could work!

I was running a MintPy installation with conda (conda install -c conda-forge --file ~/tools/MintPy/requirements.txt), and it failed to find the pysolid package in conda-forge for my M1 Mac:

PackagesNotFoundError: The following packages are not available from current channels:

  - pysolid
 
Current channels:

  - https://conda.anaconda.org/conda-forge/osx-arm64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://repo.anaconda.com/pkgs/main/osx-arm64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/osx-arm64
  - https://repo.anaconda.com/pkgs/r/noarch

Something we might try after this PR: you can specify the dependencies dynamically with newer setuptool. I didn't include it since I wasn't sure how it would interact with PySold's current setuptools version pinning

https://github.com/opera-adt/dolphin/blob/main/pyproject.toml#L23-L34

Originally posted by @scottstanie in #69 (comment)

forwarded from Paul Rosen (on trying to install it on an M2 machine):

F2py appears to be skipping code in solid.fort that is not supported, in my case an implied DO in a DATA statement that is common in old fortran. A possible solution is to modify the setup to use –std=legacy in gfortran, but I don’t know how robust that is for all compilers and all machines.

More details

Pip failed until I had Michael Aivazis guide me through what is actually happening with pip and we got the code to rebuild the source with the right pip command line switch (--no-binary).
I ran f2py by hand in verbose mode and it told me that the compiler did not understand a particular implied do array definition so it was skipping the code! So I have no confidence that the code is actually correctly compiled. According to Aivazis, you must use –std=legacy to get old code like this to compile properly, but I have no idea how to do that with the pip infrastructure.

it might be solved now that you can Conda install it even for osx-arm... but it's probably a bad sign for future stability that a new macbook causes the standard compilation to fail.

Open an issue here to keep track of things for pysolid, so that insarlab/MintPy#648 could be focused on mintpy itself.

@pbrotoisworo are you able to use pysolid on Windows? If so, how do you install it, especially how do you compile the Fortran code?

When running test_SET_grid.py the test fails after running the solid_grid function because the output solid.txt file contains empty values (#12).

Sample of file with empty values:

# program solid -- UTC version -- 2018jun01
# year, month, day = 2020 12 25
# hour, minute, second = 14  7 44
# S, N, W, E =   33.479166795   33.800000000 -118.200000000 -117.796666828
# step_lat, num_lat =   -0.009166663    35
# step_lon, num_lon =    0.009166663    44
,  ,  ,  ,  
,  ,  ,  ,  
,  ,  ,  ,  
,  ,  ,  , 
etc...

(python 3.11.3 windows)

when running python -c "import pysolid; print(pysolid.__version__)"
i get 0.3.2 which means pysolid was installed succesfully ?
when i run python PySolid/tests/grid.py i get the following error :

D:\test_gnss\PySolid\tests\grid.py
Traceback (most recent call last):
File "D:\test_gnss\gnss\Lib\site-packages\pysolid\grid.py", line 45, in calc_solid_earth_tides_grid
from pysolid.solid import solid_grid
ImportError: DLL load failed while importing solid: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\test_gnss\PySolid\tests\grid.py", line 33, in
tide_u) = pysolid.calc_solid_earth_tides_grid(dt_obj, atr, verbose=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\test_gnss\gnss\Lib\site-packages\pysolid\grid.py", line 50, in calc_solid_earth_tides_grid
raise ImportError(msg)
ImportError: Cannot import name 'solid' from 'pysolid'!
Maybe solid.for is NOT compiled yet.
Check instruction at: https://github.com/insarlab/PySolid.

We need to add the compilation process into the setup.py so that one could run the following, as an alternative to run f2py and set PYTHONPATH.

cd ~/tools/PySolid
python setup.py build
python setup.py install

Relevant question on StackOverflow: https://stackoverflow.com/questions/14805236/translate-f2py-compile-steps-into-setup-py.

Contributions are welcomed!

I agree with you, that's the timing difference along the orbit! If it's not accounted for, this timing variation will introduce a small ramp along the azimuth direction for a long track (I described this in section III.E.2 in my 2022 TGRS paper, after Dr. Dennis Milber brought it up in our email communications); and for the multiple frames stitching, as in your case here, it looks like a jump in our current implementation!

I intended to write a swath mode, in addition to the current grid and point mode, for this scenario. The implementation will be much easier if we have a true API-like talking between the python and fortran code (#2), instead of the current talking via text file. Therefore, I have not done it yet.

Originally posted by @yunjunz in #54 (comment)

Since solid.for could return numpy array since #56, we could implement this feature.

When using calc_solid_earth_tides_grid through MintPy for two separate runs involving overlapping frames, I notice unexpected behavior.

Specifically, I would expect the overlapping region between the two to be consistent after referencing epochs in the stack to the same reference date and point, but I instead find a non-zero, sub-mm level mean offset (~0.1 mm, and a range of ~0.5mm) and stdev (~0.33 mm):

Here is the residual within the overlapping region:

Please let me know how I can help to debug, if I should clarify anything, and if also perhaps it would be best to share the input files I used on the MintPy end. Many thanks

I want to install this package in centos, and this machine haven't anaconda.
So I have to build and install locally.
But I encountered this error when I followed the steps of readme.

ERROR: Command errored out with exit status 1: /usr/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/pyserver_dev/PySolid/setup.py'"'"'; __file__='"'"'/home/pyserver_dev/PySolid/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-_iefdv6v/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/python3/include/python3.6m/pysolid Check the logs for full command output.

pip check is failing because PySolid's pyproject.toml has setuptools_scm as a requirement via this line:

https://github.com/insarlab/PySolid/blob/main/pyproject.toml#L39

So, it thinks it's a runtime dependency that is missing. We should fix that in PySolid.

Originally posted by @jhkennedy in conda-forge/mintpy-feedstock#11 (comment)

In addition to the x86_64 builds currently supported by pysolid, it would be great if you could support an arm64 build for MacOS. This would enable newer Mac users to use pysolid. Also, since pysolid is a dependency of MintPy, MintPy is not currently installable on arm64 Macs.

A pip-installable version of pysolid would fix my immediate issue, but long-term it would be good to have conda support as well.

I have tried to install pysolid via pip, but ran into low-level errors when trying to build the pysolid wheels. I'm happy to provide the error messages here if they would be useful.

A bug found with a solution suggested by @fastice:

I ran into a problem with pysolid.calc_solid_earth_tides_grid().

Basically I am running multiple threads at the same time from the same directory, which means that the individual threads are overwriting or deleting the solid.txt file.

As simple fix so that files from multiple threads would not conflict would be to make the file names unique with something like.

import uuid

txt_file = os.path.abspath(f’solid.{uuid.uuid1().hex}.txt')
if os.path.isfile(txt_file):
  os.remove(txt_file)

No need to change soon on my account, I found a workaround but others might run into a similar problem in a production environment.

This unique TXT filename should be passed to the Fortran code directly as a function argument.

PySolid currently does not support Python 3.12+ because it relies on numpy.distutils to build the compiled Fortran extension. numpy.distutils has been removed for Python 3.12+ builds of numpy:
https://numpy.org/devdocs/reference/distutils_status_migration.html#distutils-status-migration

Reccomendations there are to upgrade to:

• ❌ setuptools which we cannot do because it doesn't support Fortran
• ❔ Meson, and the meson-python build backend
• ❔ CMake, and the scikit-build-core build backend

I (@jhkennedy) lean towards scikit-build-core out of familiarity, but I haven't used mesons enough to have a well-formed opinion.

With the latest release of numpy 1.22, pysolid now has a reshape error for the pysolid.grid module as below:

python ${HOME}/tools/PySolid/tests/test_SET_grid.py
Test solid Earth tides calculation for a spatial grid at a specific time
PYSOLID: ----------------------------------------
PYSOLID: datetime: 2020-12-25T14:07:44
PYSOLID: SNWE: (33.4666668, 33.8, -118.2, -117.7833335)
SOLID  : calculate solid Earth tides in east/north/up direction
SOLID  : shape: (36, 45), step size: -0.0092 by 0.0092 deg
SOLID  : calculating / writing data to txt file: /root/tools/PySolid/solid.txt
PYSOLID: read data from text file: /root/tools/PySolid/solid.txt
Traceback (most recent call last):
  File "/root/tools/PySolid/tests/test_SET_grid.py", line 34, in <module>
    tide_u) = pysolid.calc_solid_earth_tides_grid(dt_obj, atr, verbose=True)
  File "/root/tools/miniconda3/lib/python3.9/site-packages/pysolid/grid.py", line 101, in calc_solid_earth_tides_grid
    tide_e = fc[:, 0].reshape(length, width)
ValueError: cannot reshape array of size 1614 into shape (36,45)

If I pin the numpy version to <1.22, there is no issue. Thus, this seems to be an f2py module behavior change introduced in numpy 1.22 release.

It seems straightforward to pass array and scalar variables from Fortran subroutine to Python function, as shown here: https://notmatthancock.github.io/2017/02/10/calling-fortran-from-python.html. Thus, we should eliminate the writing/reading of solid.txt file and pass solid.for results directly to point/grid.py.

Relevant link: https://numpy.org/doc/stable/f2py/f2py.getting-started.html#the-quick-and-smart-way.

Contributions are welcomed!

The setup.py file has been recently updated to specify the license as license='GPL-3.0-or-later'.

Anyway in the classifiers section it is is specified:

License :: OSI Approved :: GNU General Public License v3 (GPLv3)

which corresponds to GPL-3.0-only, while GPL-3.0-or-later corresponds to

License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)

Please clarify the licensing.