cnes / swot_simulator Goto Github PK

When specifying the first_date and last_date of a simulation, it is possible that the simulator generates a swath that ends after the specified last_date (see https://github.com/CNES/swot_simulator/blob/master/swot_simulator/orbit_propagator.py#L298)

In my case, I try to simulate data over the MITGCM time span but my plugin raises an error stating that I do not have sufficient interpolation data:
Exception: Out of bounds, mitgcm files spans [2011-09-13T00:00:00.000000000,2012-11-15T14:00:00.000000000]which does not cover the required time period [2012-11-15T13:59:27.280938, 2012-11-15T14:03:52.230859]

How should we handle this case ?

• Should the plugin be returning invalid values when out of bounds
• Should the last swath be omitted from the simulation if its last date is out of bounds
• Should the user tune its first and last date of simulation so that the simulated swath falls in-bounds

Hi Lucile:
Which swotsimulator project has the up-to-date noise models and orbit ephemeris? This one, or the other one:
https://github.com/SWOTsimulator/swotsimulator
-Ed

When I run the simulator with different NSEED values, the swaths produced are identical, as well as the errors generated.

The left swath is generated using nseed =1655997574418 , while the right swath is generated with nseed= 1655997771280.
The errors are the same. How can we fix this? The errors are no longer generated truly randomly.

I would like to use HYCOM (HYCOM + NCODA Global 1/12° Reanalysis (GLBu0.08/expt_19.1)) to simulate SWOT observations, but the data format I downloaded does not meet the requirements of the swot_simulator. However, the manual has few instructions on the input data format.
Can you provide an example of HYCOM for swot_simlator? Just like the example from AVISO in swot_simulator. Many thanks!

Hi,

I ran the Python Module to generate swath and while running these lines:

import swot_simulator.error.generator
error_generator = swot_simulator.error.generator.Generator(parameters, first_date)

I am getting error message like this:

TypeError Traceback (most recent call last)
/tmp/ipykernel_45/2918145375.py in
2
3
----> 4 error_generator = swot_simulator.error.generator.Generator(parameters, first_date)

TypeError: init() missing 1 required positional argument: 'orbit_duration'

I didn't face this issue when I was running a week before. But today I faced this. I don't know how to fix it.

Thanks.

conf.txt
ncom.txt

Trying to make an ssh plugin that reads some netcdf files.
Used aviso.py as a template, which may be a mistake. Error below. I've attached my new 'ncom.py' and my 'conf.py' file. Thanks for your help!!!

jmdaddez@stout[/u/oot/jmdaddez/S-MODE/swot_simulator]$ /home/jmdaddez/anaconda3/bin/swot_simulator conf.py --first-date 2019-01-01 --last-date 2019-01-31

[E - 128.160.8.61 - Jul 20 20:53:16 - launcher] RuntimeError - Traceback (most recent call last):
/home/jmdaddez/anaconda3/lib/python3.8/site-packages/swot_simulator/launcher.py in main
503 try:
504 parameters = settings.Parameters(
----> 505 settings.eval_config_file(args.settings.name))
506
507 # Keep track of the overriden parameters
/home/jmdaddez/anaconda3/lib/python3.8/site-packages/swot_simulator/settings.py in eval_config_file
70 "called sys.exit()") from err
71 except Exception as err:
----> 72 raise RuntimeError(
73 "There is a programmable error in your configuration "
74 f"file:\n\n{traceback.format_exc()}") from err
RuntimeError: There is a programmable error in your configuration file:
Traceback (most recent call last):
File "/home/jmdaddez/anaconda3/lib/python3.8/site-packages/swot_simulator/settings.py", line 62, in eval_config_file
execfile_(filename, namespace)
File "/home/jmdaddez/anaconda3/lib/python3.8/site-packages/swot_simulator/settings.py", line 38, in execfile_
exec(code, _globals)
File "conf.py", line 101, in
ssh_plugin = swot_simulator.plugins.ssh.NCOM("/u/GoM_4dvar/SWOT-cal-val/model/OoT/raw/ssha")
File "/home/jmdaddez/anaconda3/lib/python3.8/site-packages/swot_simulator/plugins/ssh/ncom.py", line 20, in init
loader = data_handler.NetcdfLoader(
File "/home/jmdaddez/anaconda3/lib/python3.8/site-packages/swot_simulator/plugins/data_handler.py", line 200, in init
self.time_delta = self._calculate_time_delta(self.time_series["date"])
File "/home/jmdaddez/anaconda3/lib/python3.8/site-packages/swot_simulator/plugins/data_handler.py", line 102, in _calculate_time_delta
raise RuntimeError(
RuntimeError: Time series does not have a constant step between two grids: {numpy.timedelta64(0,'s'), numpy.timedelta64(172800,'s'), numpy.timedelta64(86400,'s')} seconds
[E - 128.160.8.61 - Jul 20 20:53:16 - launcher] End of processing.

Hello,
I would like to simulate swot data from other model data. The model was constructed using MITgcm using bathymetry from General Bathymetric Chart of the Oceans (GEBCO_14) Grid. Details of the data are as follows. I have tried 'ssh_plugin = swot_simulator.plugins.ssh.MITGCM('path')'. It doen't work. Do you have any suggestions?

`Dimensions: (time: 6940, YC: 600, XC: 840)
Coordinates:

• time (time) datetime64[ns] 2000-01-01T03:00:00 ... 2018-12-31T03:00:00
• YC (YC) float64 -17.83 -17.75 -17.67 -17.58 ... 14.47 14.55 14.63
• XC (XC) float64 -105.0 -105.0 -104.9 -104.9 ... -77.09 -77.05 -77.02
  Data variables:
  ETAN (time, YC, XC) float32 ...
  Attributes:
  CDI: Climate Data Interface version 2.0.1 (https://mpimet.mpg.de/...
  Conventions: CF-1.6
  history: Mon Apr 15 08:48:01 2024: cdo -L -mergetime -apply,-selname...
  CDO: Climate Data Operators version 2.0.0 (https://mpimet.mpg.de...>`

Many thanks!
Lindsey

I noticed that the example AVISO files supplied with the package contain only a single time slice per file. Is this necessary, or can I have multiple time slices within a file?
I am trying to use the swot_simulator with model output data which are organized into netcdf files containing one month of data, so I am receiving the following error:

RuntimeError: Time series does not have a constant step between two grids: {numpy.timedelta64(2678400,'s'), numpy.timedelta64(2419200,'s'), numpy.timedel\
ta64(2505600,'s'), numpy.timedelta64(2592000,'s'), numpy.timedelta64(86659200,'s'), numpy.timedelta64(81475200,'s'), numpy.timedelta64(42163200,'s'), numpy.t\
imedelta64(44668800,'s'), numpy.timedelta64(84153600,'s'), numpy.timedelta64(10540800,'s'), numpy.timedelta64(5270400,'s')} seconds

Attached are my plugin and the ncdump -h from a sample file. I can split my input files into 1-per-day or 1-per-time-slice, but I would like to avoid this if possible.
ncdump.txt
romsnz_dot_py.txt

distributed.worker - WARNING - Compute Failed
Function: simulate
args: ((1, 1, numpy.datetime64('2019-01-01T00:00:00.000000')))
kwargs: {'error_generator': <swot_simulator.error.generator.Generator object at 0x7faba8db4a30>, 'orbit': <swot_simulator.orbit_propagator.Orbit object at 0x7faba8db42e0>, 'parameters': <swot_simulator.settings.Parameters object at 0x7faba8d681c0>, 'logging_server': ('128.160.8.61', 9020, 20)}
Exception: KeyError('simulated_error_troposphere_nadir')

[W - 128.160.8.61 - Oct 18 22:40:00 - worker] Compute Failed
Function: simulate
args: ((1, 1, numpy.datetime64('2019-01-01T00:00:00.000000')))
kwargs: {'error_generator': <swot_simulator.error.generator.Generator object at 0x7faba8db4a30>, 'orbit': <swot_simulator.orbit_propagator.Orbit object at 0x7faba8db42e0>, 'parameters': <swot_simulator.settings.Parameters object at 0x7faba8d681c0>, 'logging_server': ('128.160.8.61', 9020, 20)}
Exception: KeyError('simulated_error_troposphere_nadir')

When launching the latest version of swot_simulator (1.2.4), I have the following error message :
AssertionError: /data/usrcto/ctoh/restricted/modules/anaconda3_robin/envs/swotsim_20220411/lib/python3.10/distutils/core.py

It seems that the error comes from launcher.py, where the software dependencies version are extracted (they are added to the parameters file to remember the config used for simulation). In the software_dependencies() method, setuptools is imported after pip and it seems that it is the source of the error. In my environment (python 3.10, pip 22.0.4, setuptool 62.1.0) :

>>> import pip
>>> import setuptools
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/usrcto/ctoh/restricted/modules/anaconda3_robin/envs/swotsim_20220411/lib/python3.10/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
  File "/data/usrcto/ctoh/restricted/modules/anaconda3_robin/envs/swotsim_20220411/lib/python3.10/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/data/usrcto/ctoh/restricted/modules/anaconda3_robin/envs/swotsim_20220411/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 72, in do_override
    ensure_local_distutils()
  File "/data/usrcto/ctoh/restricted/modules/anaconda3_robin/envs/swotsim_20220411/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 59, in ensure_local_distutils
    assert '_distutils' in core.__file__, core.__file__
AssertionError: /data/usrcto/ctoh/restricted/modules/anaconda3_robin/envs/swotsim_20220411/lib/python3.10/distutils/core.py

setuptools seems to override the distutils library, which pip is using, so pip should be imported after setuptools. I haven't looked further into this but it seems simply changing the order of the dependencies fixes the issue.

I am using linux ubuntu 20.04 LTS with a freshly installed anaconda3, and I installed swot_simulator using conda. (During the installation of swot_simulator, conda spent many many minutes "solving environment".)

I have set ssh_plugin to use the aviso.py plugin and the sample datafile which is provided inside the swot_simulator/data directory. The paths and ssh_plugin were entered into the generated conf.py file template.

The attempt to run,

swot_simulator conf.py

generates the error message from inside the dask call-chain,

ImportError: cannot import name 'dumps_msgpack' from 'distributed.protocol.core' (/home/ezaron/opt/anaconda3/lib/python3.8/site-packages/distributed/protocol/core.py)
[E - 127.0.1.1 - Sep 22 10:56:53 - launcher] End of processing.

Googling the error message, I find a possible workaround to install an older version of the distributed library,
conda install -c conda-forge distributed=2021.4.0, but this did not change the error message.

Suggestions?

After generating a swath based on the example and using the build in AVISO sample data I noticed that the orbital noise added to the swath is far too high.

The left hand image is the 'truth' from the product.simulated_true_ssh_karin and the right hand image is karin SSH with all errors/noise generated product.ssh_karin
These swaths were generated using:

configuration = swot_simulator.settings.template(python = True) 
parameters = swot_simulator.settings.Parameters(configuration) 
parameters.ssh_plugin = swot_simulator.plugins.ssh.AVISO(swot_simulator.DATA) 

After removing the orbital noise error and utilizing the corrected roll the swath looks much more realistic

Where the left is truth and the right is product.ssh_karin
These swaths were generated using:

configuration = swot_simulator.settings.template(python = True) 
parameters = swot_simulator.settings.Parameters(configuration) 
parameters.ssh_plugin = swot_simulator.plugins.ssh.AVISO(swot_simulator.DATA) 
parameters.noise = ['Altimeter', 'Karin', 'BaselineDilation', 'WetTroposphere', 'Timing', 'CorrectedRollPhase'] 

When looked at alone, the swath with only orbital error looks like:

Is it possible I am implementing this error wrong or is the signal from this error too strong?
Thanks!

Hello
I put in conf.py :

ssh_plugin = swot_simulator.plugins.ssh.nemo("/ec/res6/scratch/ar5/ORCA36-T426a-S1/")

but I have this error ; do you know why ?
Thank you,
Clément

[E - 10.100.66.100 - Oct 21 13:42:15 - launcher] RuntimeError - Traceback (most recent call last):
/perm/ar5/miniconda3/envs/SWOTsimu/lib/python3.10/site-packages/swot_simulator/cli/launcher.py in main
228 try:
229 parameters = settings.Parameters(
----> 230 settings.eval_config_file(args.settings.name))
231
232 # Keep track of the overridden parameters

/perm/ar5/miniconda3/envs/SWOTsimu/lib/python3.10/site-packages/swot_simulator/settings.py in eval_config_file
      84                 "called sys.exit()") from err
      85         except Exception as err:
----> 86             raise RuntimeError(
      87                 "There is a programmable error in your configuration "
      88                 f"file:\n\n{traceback.format_exc()}") from err

RuntimeError: There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/perm/ar5/miniconda3/envs/SWOTsimu/lib/python3.10/site-packages/swot_simulator/settings.py", line 76, in eval_config_file
    execfile_(filename, namespace)
  File "/perm/ar5/miniconda3/envs/SWOTsimu/lib/python3.10/site-packages/swot_simulator/settings.py", line 49, in execfile_
    exec(code, _globals)
  File "conf.py", line 104, in <module>
    ssh_plugin = swot_simulator.plugins.ssh.nemo("/ec/res6/scratch/ar5/ORCA36-T426a-S1/")
TypeError: 'module' object is not callable

[E - 10.100.66.100 - Oct 21 13:42:15 - launcher] End of processing.

There are four .nc format files (daily sea level) in the Data folder, which can be used to interpolate the SSH of swot. Four files (e.g. Dt_global_allsat_phy_l4_20190101_20190515.nc, etc.) are available on Github, I want to get more files, where should I download them? Many Thanks ！

Hi, when I executed the first step swot_simulator conf.py, there is an error raised.
AssertionError - Traceback (most recent call last):
D:\anaconda\envs\swot\lib\site-packages\swot_simulator\launcher.py in main
478
479 # Keep track of the overriden parameters
----> 480 copy_parameters(parameters, pathlib.Path(args.settings.name))
481 launch(client, parameters, logging_server, args.first_date,
482 args.last_date)

D:\anaconda\envs\swot\lib\site-packages\swot_simulator\launcher.py in copy_parameters
      450 # -------------------
      451 """)
----> 452             for dependency, version in sorted(software_dependencies()):
      453                 target_stream.write(f"# * {dependency}: {version}\n")
      454             target_stream.write(source_stream.read())

D:\anaconda\envs\swot\lib\site-packages\swot_simulator\launcher.py in software_dependencies
416 mod = sys.modules[module_name]
417 else:
----> 418 mod = importlib.import_module(module_name)
419 version = version_getter(mod)
420 result.append((module_name, version))

D:\anaconda\envs\swot\lib\importlib\__init__.py in import_module
      125                 break
      126             level += 1
----> 127     return _bootstrap._gcd_import(name[level:], package, level)
      128
      129

D:\anaconda\envs\swot\lib\site-packages\setuptools_init_.py in
5 import re
6
----> 7 import _distutils_hack.override # noqa: F401
8 import distutils.core
9 from distutils.errors import DistutilsOptionError

D:\anaconda\envs\swot\lib\site-packages\_distutils_hack\override.py in <module>
----> 1 __import__('_distutils_hack').do_override()

D:\anaconda\envs\swot\lib\site-packages\_distutils_hack\__init__.py in do_override
      75     if enabled():
      76         warn_distutils_present()
----> 77         ensure_local_distutils()
      78
      79

D:\anaconda\envs\swot\lib\site-packages\_distutils_hack\__init__.py in ensure_local_distutils
      62     # check that submodules load as expected
      63     core = importlib.import_module('distutils.core')
----> 64     assert '_distutils' in core.__file__, core.__file__
      65     assert 'setuptools._distutils.log' not in sys.modules
      66

AssertionError: D:\anaconda\envs\swot\lib\distutils\core.py

[E - 2.0.0.1 - Jan 17 20:51:08 - launcher] End of processing.

This was happened on different environments and different computers. I wonder if the code needs to be modified mannally to fix up it. Thanks very much if you could help with us.

Hi, all

I just installed the swot simulator and run the pytest. Three errors were found:

(base) lei@lei-Abelard:~/swot_simu/swot_simulator$ pytest
============================= test session starts ==============================
platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/lei/swot_simu/swot_simulator
collected 0 items / 3 errors                                                   

==================================== ERRORS ====================================
_______________ ERROR collecting tests/test_error_generation.py ________________
ImportError while importing test module '/home/lei/swot_simu/swot_simulator/tests/test_error_generation.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../miniconda3/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_error_generation.py:4: in <module>
    import swot_simulator
swot_simulator/__init__.py:12: in <module>
    from . import version
E   ImportError: cannot import name 'version' from partially initialized module 'swot_simulator' (most likely due to a circular import) (/home/lei/swot_simu/swot_simulator/swot_simulator/__init__.py)
___________________ ERROR collecting tests/test_launcher.py ____________________
ImportError while importing test module '/home/lei/swot_simu/swot_simulator/tests/test_launcher.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../miniconda3/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_launcher.py:4: in <module>
    import swot_simulator
swot_simulator/__init__.py:12: in <module>
    from . import version
E   ImportError: cannot import name 'version' from partially initialized module 'swot_simulator' (most likely due to a circular import) (/home/lei/swot_simu/swot_simulator/swot_simulator/__init__.py)
_________________ ERROR collecting tests/test_random_signal.py _________________
ImportError while importing test module '/home/lei/swot_simu/swot_simulator/tests/test_random_signal.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../miniconda3/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_random_signal.py:6: in <module>
    import swot_simulator
swot_simulator/__init__.py:12: in <module>
    from . import version
E   ImportError: cannot import name 'version' from partially initialized module 'swot_simulator' (most likely due to a circular import) (/home/lei/swot_simu/swot_simulator/swot_simulator/__init__.py)
=========================== short test summary info ============================
ERROR tests/test_error_generation.py
ERROR tests/test_launcher.py
ERROR tests/test_random_signal.py
!!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!
============================== 3 errors in 53.69s ==============================
(base) lei@lei-Abelard:~/swot_simu/swot_simulator$ 

I also tried this on Windows system and the errors are the same.

Can these error be neglected?

Thanks

Lei

Hello,

I would like to re-create 21 days of SWOT passes, I am interested to have the passes location in a certain region during the entire cycle. To do that, I plan to use the AVISO example. I would like to run: swot_simulator conf.py --first-date 20190101 --last-date 20190122

Thus I would like to extend the range definition.
I get this kind of error if I try to run swot_simulator conf.py --first-date 20190102:
IndexError: period [2019-01-02T00:00:00.000000000, 2019-01-03T00:51:26.254174000] is out of range: [2018-12-31T00:00:00, 2019-01-03T00:00:00]

How can I use other dates other than 20190101 for AVISO? Do I have to modify the AVISO plugin in order to do that or is there a simpler way?

Thanks!

Marina