pytroll / pytroll-cspp-runner Goto Github PK

Documentation for pytroll-cspp-runner is very limited. There are some configuration files, but they are lacking comments explaining the meaning of the different configuration options. There are some docstrings, but it's not clear how everything fits together. It would be useful to have some overview documentation explaining how pytroll-cspp-runner works and how it can be configured.

Logging configuration is currently hardcoded in viirs_dr_runner.py:

It would be more user-friendly if this could be configured with a logging config, such as used by trollflow2.

Currently, viirs_dr_runner.py is hardcoded to look for a section [dev] in the configuration file passed on the commandline. This is undocumented and not user friendly. I propose to add a commandline option -C, similar to the one used by trollstalker in pytroll-collectors, that instructs what section from the configuration to use.

I am open to other suggestions on how to improve the status quo.

The cspp-runner is currently hardcoded to use the non-noisy Publisher with multicast. This is rigid and might not fit for everybody. It would be good if the runner was configurable. This could use the new dict_config option in posttroll (see pytroll/posttroll#35).

The implementation of pytroll-cspp-runner could do with a bit of cleanup:

• The example configuration contains settings that appear to be unused (see also #7).
• It's not clear or consistent what configuration needs to be in environment variables and what in the configuration file. CSPP uses $CSPP_SDR_HOME (relevant) and ($CSPP_RGB_HOME) (not relevant?), but the runner also appears to use CSPP_WORKDIR; I don't know if CSPP itself does?
• The main script bin/viirs_dr_runner.py contains references that appear to be specific to SMHI, such as the environment variable SMHI_MODE.
• Much of the functionality that is currently in bin/viirs_dr_runner.py could be in library functions that should be unit tested.
• Much of the code does not follow clean code standards.
• Unit test coverage is low.
• Some module level definitions in bin/viirs_dr_runner.py could be in a configuration file (example _DEFAULT_LOG_FORMAT).
• Some module level definitions in bin/viirs_dr_runn.rpy appear to be unused (example NPP_SDRPROC_LOG_FILE).

There's probably more I'm not noticing right now.

Here the start_time and end_time

https://github.com/pytroll/pytroll-cspp-runner/blob/master/bin/viirs_dr_runner.py#L353

are parsed from the filename which is the last fle in the file list a few lines above?

This gives not the expected start_time in the published message

GitHub Continuous Integration (CI) unit test setup is failing to install mapbox due to missing GDAL_CONFIG or GDAL_VERSION.

Code Sample, a minimal, complete, and verifiable piece of code

I don't have a code sample, but the problem can be seen in the GitHut CI for #14, #15, and #16, where the tests CI / test (3.9, ubuntu-latest, true) (pull_request) are failing to run. The other runs are successful.

Problem description

This failure is problematic, because it means the unit tests aren't run at all and we don't know if they would be successful or not.

Expected Output

pytroll-cspp-runner main should pass with experimental and non-experimental dependencies.

Actual Result, Traceback if applicable

Collecting git+https://github.com/mapbox/rasterio
  Cloning https://github.com/mapbox/rasterio to /tmp/pip-req-build-n4oacao2
  Running command git clone --filter=blob:none --quiet https://github.com/mapbox/rasterio /tmp/pip-req-build-n4oacao2
  Resolved https://github.com/mapbox/rasterio to commit fe02e31c48d5dcfeabc3feed2861a28d87af9400
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [2 lines of output]
      WARNING:root:Failed to get options via gdal-config: [Errno 2] No such file or directory: 'gdal-config'
      ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Error: Process completed with exit code 1.

Versions of Python, package at hand and relevant dependencies

pytroll-cspp-runner main, Python 3.9, experimental dependencies (GitHub main)