bram94 / nlradar

Installation/Setup: If you use Anaconda, then I suggest creating a new environment with python 3.8. In your command line: "conda create -n NLradar python=3.8". Activate that environment with "conda activate NLradar". Then go to directory in which you want to install NLradar. There, clone the git repository with "git clone https://github.com/Bram94/NLradar.git". Then move into the NLradar folder where a setup.py script is located, and run "pip install .". If the preceding steps went correctly, you should then be able to start the viewer from the NLradar/Python_files folder, by running "python nlr.py".

Note: The Unet velocity dealiasing algorithm can run on either the CPU or GPU. The GPU option is used when you have an NVIDIA GPU on your PC, with CUDA and cuDNN installed. If not, the CPU is used, which is however much slower.

Using the viewer: I suggest taking a look at the Help in order to get started. Navigation heavily uses the keyboard, and a list of all keyboard shortcuts can be found at the tab Help/Keyboard.

Note: This is a first release of NLradar. More expansive documentation will follow over time.

Created by Bram van 't Veen (2016-2024).

nlradar's Issues

Issue installing on mac

Hi Bram,

I get the following error in my terminal:

`Last login: Mon Jul 15 22:04:41 on ttys000
/Users/mw/.zshrc:source:112: no such file or directory: /usr/local/share/zsh-autosuggestions/zsh-autosuggestions.zsh
❯ conda create -n NLradar python=3.8
Channels:

• defaults
  Platform: osx-arm64
  Collecting package metadata (repodata.json): done
  Solving environment: done

Package Plan

environment location: /opt/anaconda3/envs/NLradar

added / updated specs:
- python=3.8

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
ca-certificates-2024.7.2   |       hca03da5_0         128 KB
pip-24.0                   |   py38hca03da5_0         2.6 MB
python-3.8.19              |       hb885b13_0        12.5 MB
setuptools-69.5.1          |   py38hca03da5_0        1004 KB
wheel-0.43.0               |   py38hca03da5_0         108 KB
------------------------------------------------------------
                                       Total:        16.3 MB

The following NEW packages will be INSTALLED:

ca-certificates pkgs/main/osx-arm64::ca-certificates-2024.7.2-hca03da5_0
libcxx pkgs/main/osx-arm64::libcxx-14.0.6-h848a8c0_0
libffi pkgs/main/osx-arm64::libffi-3.4.4-hca03da5_1
ncurses pkgs/main/osx-arm64::ncurses-6.4-h313beb8_0
openssl pkgs/main/osx-arm64::openssl-3.0.14-h80987f9_0
pip pkgs/main/osx-arm64::pip-24.0-py38hca03da5_0
python pkgs/main/osx-arm64::python-3.8.19-hb885b13_0
readline pkgs/main/osx-arm64::readline-8.2-h1a28f6b_0
setuptools pkgs/main/osx-arm64::setuptools-69.5.1-py38hca03da5_0
sqlite pkgs/main/osx-arm64::sqlite-3.45.3-h80987f9_0
tk pkgs/main/osx-arm64::tk-8.6.14-h6ba3021_0
wheel pkgs/main/osx-arm64::wheel-0.43.0-py38hca03da5_0
xz pkgs/main/osx-arm64::xz-5.4.6-h80987f9_1
zlib pkgs/main/osx-arm64::zlib-1.2.13-h18a0788_1

Proceed ([y]/n)? y

Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

To activate this environment, use

$ conda activate NLradar

To deactivate an active environment, use

$ conda deactivate

❯ conda activate NLradar
❯ ls
Applications Dropbox Parallels
Applications (Parallels) Google Drive Pictures
Back up (google drive) Library Public
Desktop Mijn Drive Sites
Documents Movies W676478B009F0E1B5D
Downloads Music dev
❯ cd dev
❯ ls
cb fabric
❯ mkdir NLradar
❯ ks
zsh: command not found: ks
❯ ls
NLradar cb fabric
❯ NLradar
❯ git clone https://github.com/Bram94/NLradar.git
Cloning into 'NLradar'...
remote: Enumerating objects: 1435, done.
remote: Counting objects: 100% (122/122), done.
remote: Compressing objects: 100% (88/88), done.
remote: Total 1435 (delta 67), reused 72 (delta 34), pack-reused 1313
Receiving objects: 100% (1435/1435), 36.23 MiB | 4.86 MiB/s, done.
Resolving deltas: 100% (516/516), done.
❯ ls
NLradar
❯ cd NLradar/
❯ ls
Input_files Python_files
LICENSE README.md
NLradar.ico nlr_spec3_pyqt4.spec
NLradar.png nlr_spec3_pyqt5_linux.spec
NLradar_cmd.ico nlr_spec3_pyqt5_windows.spec
NLradar_cmd.png setup.py
NLradar_nieuw_script.iss
❯ pip install .
Processing /Users/mw/dev/NLradar/NLradar
Preparing metadata (setup.py) ... done
Collecting numpy-bufr@ git+https://github.com/Bram94/numpy_bufr.git (from NLradar==0.1)
Cloning https://github.com/Bram94/numpy_bufr.git to /private/var/folders/3g/tmn7h5397cx6msm6z_cv701c0000gn/T/pip-install-n07xdvr8/numpy-bufr_5023ac9b3038428bbced3f0e84ec4b08
Running command git clone --filter=blob:none --quiet https://github.com/Bram94/numpy_bufr.git /private/var/folders/3g/tmn7h5397cx6msm6z_cv701c0000gn/T/pip-install-n07xdvr8/numpy-bufr_5023ac9b3038428bbced3f0e84ec4b08
Resolved https://github.com/Bram94/numpy_bufr.git to commit 6866fc42508b8c194cd2b18b6b0f0d41a5b2d4ce
Preparing metadata (setup.py) ... done
Collecting numpy (from NLradar==0.1)
Downloading numpy-1.24.4-cp38-cp38-macosx_11_0_arm64.whl.metadata (5.6 kB)
Collecting pyproj (from NLradar==0.1)
Downloading pyproj-3.5.0-cp38-cp38-macosx_11_0_arm64.whl.metadata (28 kB)
Collecting pyshp (from NLradar==0.1)
Downloading pyshp-2.3.1-py2.py3-none-any.whl.metadata (55 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.0/56.0 kB 2.4 MB/s eta 0:00:00
Collecting opencv-python (from NLradar==0.1)
Downloading opencv_python-4.10.0.84-cp37-abi3-macosx_11_0_arm64.whl.metadata (20 kB)
Collecting pillow (from NLradar==0.1)
Downloading pillow-10.4.0-cp38-cp38-macosx_11_0_arm64.whl.metadata (9.2 kB)
Collecting pyqt5 (from NLradar==0.1)
Downloading PyQt5-5.15.10-cp37-abi3-macosx_11_0_arm64.whl.metadata (2.1 kB)
Collecting scipy (from NLradar==0.1)
Downloading scipy-1.10.1-cp38-cp38-macosx_12_0_arm64.whl.metadata (53 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.9/53.9 kB 3.6 MB/s eta 0:00:00
Collecting requests (from NLradar==0.1)
Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting gpxpy (from NLradar==0.1)
Downloading gpxpy-1.6.2-py3-none-any.whl.metadata (5.9 kB)
Collecting netcdf4==1.6 (from NLradar==0.1)
Downloading netCDF4-1.6.0-cp38-cp38-macosx_11_0_arm64.whl.metadata (1.6 kB)
Collecting pyopengl (from NLradar==0.1)
Downloading PyOpenGL-3.1.7-py3-none-any.whl.metadata (3.2 kB)
Collecting matplotlib (from NLradar==0.1)
Downloading matplotlib-3.7.5-cp38-cp38-macosx_11_0_arm64.whl.metadata (5.7 kB)
Collecting xmltodict (from NLradar==0.1)
Downloading xmltodict-0.13.0-py2.py3-none-any.whl.metadata (7.7 kB)
Collecting boto3 (from NLradar==0.1)
Downloading boto3-1.34.144-py3-none-any.whl.metadata (6.6 kB)
Collecting pytz (from NLradar==0.1)
Using cached pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB)
Collecting av (from NLradar==0.1)
Downloading av-12.2.0-cp38-cp38-macosx_11_0_arm64.whl.metadata (4.6 kB)
INFO: pip is looking at multiple versions of nlradar to determine which version is compatible with other requirements. This could take a while.
ERROR: Ignored the following versions that require a different python version: 1.11.0 Requires-Python <3.13,>=3.9; 1.11.0rc1 Requires-Python <3.13,>=3.9; 1.11.0rc2 Requires-Python <3.13,>=3.9; 1.11.1 Requires-Python <3.13,>=3.9; 1.11.2 Requires-Python <3.13,>=3.9; 1.11.3 Requires-Python <3.13,>=3.9; 1.11.4 Requires-Python >=3.9; 1.12.0 Requires-Python >=3.9; 1.12.0rc1 Requires-Python >=3.9; 1.12.0rc2 Requires-Python >=3.9; 1.13.0 Requires-Python >=3.9; 1.13.0rc1 Requires-Python >=3.9; 1.13.1 Requires-Python >=3.9; 1.14.0 Requires-Python >=3.10; 1.14.0rc1 Requires-Python >=3.10; 1.14.0rc2 Requires-Python >=3.10; 1.25.0 Requires-Python >=3.9; 1.25.0rc1 Requires-Python >=3.9; 1.25.1 Requires-Python >=3.9; 1.25.2 Requires-Python >=3.9; 1.26.0 Requires-Python <3.13,>=3.9; 1.26.0b1 Requires-Python <3.13,>=3.9; 1.26.0rc1 Requires-Python <3.13,>=3.9; 1.26.1 Requires-Python <3.13,>=3.9; 1.26.2 Requires-Python >=3.9; 1.26.3 Requires-Python >=3.9; 1.26.4 Requires-Python >=3.9; 2.0.0 Requires-Python >=3.9; 2.0.0b1 Requires-Python >=3.9; 2.0.0rc1 Requires-Python >=3.9; 2.0.0rc2 Requires-Python >=3.9; 3.6.0 Requires-Python >=3.9; 3.6.0rc0 Requires-Python >=3.9; 3.6.1 Requires-Python >=3.9; 3.6.1rc0 Requires-Python >=3.9; 3.8.0 Requires-Python >=3.9; 3.8.0rc1 Requires-Python >=3.9; 3.8.1 Requires-Python >=3.9; 3.8.2 Requires-Python >=3.9; 3.8.3 Requires-Python >=3.9; 3.8.4 Requires-Python >=3.9; 3.9.0 Requires-Python >=3.9; 3.9.0rc2 Requires-Python >=3.9; 3.9.1 Requires-Python >=3.9
ERROR: Could not find a version that satisfies the requirement tensorflow==2.10 (from nlradar) (from versions: 2.13.0rc0, 2.13.0rc1, 2.13.0rc2, 2.13.0, 2.13.1)
ERROR: No matching distribution found for tensorflow==2.10
╭─ ~/dev/NLradar/NLradar main 13s  NLradar 22:57:53
╰─❯`

My python version is:

❯ python --version Python 3.8.19 ❯ python3 --version Python 3.8.19

Not an issue just a question

Hi there

Nice product you've got there. I was scouting through the code & I was wondering if there was a specific reason to leverage on the DWD tables for BUFR decoding.

Were the bufr tables provided by meteofrance not enough by any chance? I myself have also been curious about the meteofrance opendata, but the documentation seems very convoluted & hard to come by... It's also far from incomplete & doesn't provide all the information you would need.

I could also see in one of the files: Python_files/nlr_importdata.py that you used an offset for the meteofrance radardata. I was wondering how you came across this, as I can't seem to find anything about it in the documentation, nor can I find it in a BUFR files when decoding one.

I haven't been able to run the install of this projecton my mac though, so I can't run it & put breakpoints in my editor to see the flow of your application. Having problems with tensorflow & the install seems to break on it.

I would have liked to see how you actually downloaded the MF files because if you look at the api documentation of meteofrance, there is 0 mention of downloading archived data in their swagger. It all seems more convoluted & confusing than it should... Especially levaraging on BUFR, which is pretty archaic & less prone to standardisation (even if BUFR is actually intended as such).

Which is also the reason why DWD has moved to h5 instead of their BUFR files, although they apparently still provide the option to use it if I'm not mistaking.

When I download the Meteofrance BUFR files & decode them, I only get 1 product, reflectivity, while the PAM & PAG files normally contain more. I can't seem to find where (how) to get the other products as CC, ZDR, Advection (PAM) & radial velocity (PAG).

I was hoping your codebase could provide a clue. Keep up the good work! Seems like a major undertaking what you're doing. It's quite impressive

Adding additional archive type

Hi!

I'm from Meteopress and I've been chipping at an attempt to add our ODIM format data into NLradar; of course, if I get it working and clear the code up a bit, I'd gladly pull request it back to you, as GPL3.0 more or less require me to. I've however been struggling a little; would you be okay with getting in contact and helping me understand some details of the codebase?

Velocity not working

Hello.

I'm trying to use the Velocity products, but it's impossible. Somehow my velocity products would always ressemble like any of these

And sometimes this appears. If i go on the previous scan, it goes back to something like the 1st screenshot

What also can happen is that when switching to velocity, nothing appears or those screenshots can appear, and when i try to switch some other product like reflectivity or spectrum width, nothing appears for any products

This is the error i get when i launch the program

D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_importdata.py:2069: RuntimeWarning: Mean of empty slice
  c_data = np.nanmean(np.reshape(c_data[:,i], (360, 2, data.shape[1], ratio)), axis=(1,3))
2024-03-05 21:38:29.137318: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2024-03-05 21:38:29.137399: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2024-03-05 21:38:31.310658: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2024-03-05 21:38:31.310754: W tensorflow/stream_executor/cuda/cuda_driver.cc:263] failed call to cuInit: UNKNOWN ERROR (303)
2024-03-05 21:38:31.313254: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-XXXXXXX
2024-03-05 21:38:31.313383: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-XXXXXXX
detected GPUs: []
2024-03-05 21:38:31.325918: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-05 21:38:31.746989: W tensorflow/core/util/tensor_slice_reader.cc:96] Could not open D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\dealiasing\unet_vda\models\dealias_sn16_csi9764.SavedModel: UNKNOWN: NewRandomAccessFile failed to Create/Open: D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\dealiasing\unet_vda\models\dealias_sn16_csi9764.SavedModel : AccÞs refusÚ.
; Input/output error
 get_data, panel 0
Traceback (most recent call last):
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_datasourcegeneral.py", line 796, in get_data
    self.perform_mono_PRF_dealiasing(j)
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_datasourcegeneral.py", line 853, in perform_mono_PRF_dealiasing
    self.data[j] = self.vda(self.data[j], self.nyquist_velocities_all_mps[self.crd.scans[j]], extra_dealias='extra' in self.gui.dealiasing_setting)
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\dealiasing\unet_vda\unet_vda.py", line 74, in __call__
    self.load_model()
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\dealiasing\unet_vda\unet_vda.py", line 49, in load_model
    self.vda.load_weights(gv.programdir+'/Python_files/dealiasing/unet_vda/models/dealias_sn16_csi9764.SavedModel')
  File "C:\Users\[USER]\miniconda3\envs\NLradar\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\[USER]\miniconda3\envs\NLradar\lib\site-packages\tensorflow\python\training\py_checkpoint_reader.py", line 45, in error_translator
    raise errors_impl.OpError(None, None, error_message, errors_impl.UNKNOWN)
tensorflow.python.framework.errors_impl.OpError

And later, i can also have this error

update current
add len(self._bzip2_start_pos)-1
restore_previous_attrs not set_data
list index out of range get_data, panel 0
Traceback (most recent call last):
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_datasourcegeneral.py", line 777, in get_data
    self.source_classes[self.data_source()].get_data(j)
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_datasourcespecific.py", line 856, in get_data
    self.import_class().get_data(filepath, j)
  File "D:\Users\[USER]\Desktop\Meteo\NLradar\Python_files\nlr_importdata.py", line 2371, in get_data
    scan_index = self.dsg.scannumbers_all[i_p][scan][self.dsg.scannumbers_forduplicates[scan]]
IndexError: list index out of range

I really don't know what's causing those issues, what could i have done wrong compared to other installs i did where i had this bug but only once on first launch

I hope this issue report is good, it's one of my first time doing such thing