The salem from fmaussion

I am trying to apply the .subset function on a xarray.core.dataset.Dataset object. However, I get RuntimeError: dataset Grid not understood. I am able to reproduce the example from the documentation (here), however, when providing another data set I get the error.

Here is a snippet of my data: test_data.zip

Here my code:

import xarray as xr
import salem

ds = xr.open_dataset("./data/test_data.nc")
ds
<xarray.Dataset>
Dimensions:                 (nlat: 85, nlon: 63, time: 3)
Coordinates:
  * nlon                    (nlon) float32 -82.375 -82.125 -81.875 -81.625 ...
  * nlat                    (nlat) float32 -20.125 -19.875 -19.625 -19.375 ...
  * time                    (time) datetime64[ns] 2004-12-31 2005-01-31 ...
Data variables:
    precipitation           (time, nlat, nlon) float32 ...
    relativeError           (time, nlat, nlon) float32 ...
    gaugeRelativeWeighting  (time, nlat, nlon) int32 ...
Attributes:
    Grid.GridHeader:  BinMethod=ARITHMETIC_MEAN;\nRegistration=CENTER;\nLatit...
    FileHeader:       AlgorithmID=3B43;\nAlgorithmVersion=3B43_7.0;\nFileName...
    FileInfo:         DataFormatVersion=m;\nTKCodeBuildVersion=1;\nMetadataVe...
    GridHeader:       BinMethod=ARITHMETIC_MEAN;\nRegistration=CENTER;\nLatit...
    history:          2018-05-08 18:27:22 GMT Hyrax-1.13.4 https://disc2.gesd...

type(ds)
xarray.core.dataset.Dataset

ds_subset = ds.salem.subset(corners=((-80, -15.), (-75., -7.)), crs=salem.wgs84)
RuntimeError: dataset Grid not understood.

Any advice?

TypeError when attempting to subset (roi) a DataArray using salem

I'm trying to use salem for masking out a specific region (defined by a shapefile) in my xarray.
Array and shapefile loaded in correctly and are recognized as such.

Yet when attempting to use "roi" on my DataArray, i get a typeerror.
sub = ds.salem.roi(shape=shdf)

Traceback (most recent call last):

  File "<ipython-input-62-53ac7c79df8c>", line 1, in <module>
    sub = ds.salem.roi(shape=shdf)

  File "C:\Anaconda\lib\site-packages\salem\sio.py", line 501, in roi
    mask = self.grid.region_of_interest(**kwargs)

  File "C:\Anaconda\lib\site-packages\salem\gis.py", line 1009, in region_of_interest
    inplace=inplace)

  File "C:\Anaconda\lib\site-packages\salem\gis.py", line 1229, in transform_geopandas
    result = out.geometry.apply(lambda geom: shapely_transform(project, geom))

  File "C:\Anaconda\lib\site-packages\pandas\core\series.py", line 2551, in apply
    mapped = lib.map_infer(values, f, convert=convert_dtype)

  File "pandas/_libs/src/inference.pyx", line 1521, in pandas._libs.lib.map_infer

  File "C:\Anaconda\lib\site-packages\salem\gis.py", line 1229, in <lambda>
    result = out.geometry.apply(lambda geom: shapely_transform(project, geom))

  File "C:\Anaconda\lib\site-packages\shapely\ops.py", line 246, in transform
    [func(*c) for c in geom.exterior.coords])

  File "C:\Anaconda\lib\site-packages\salem\gis.py", line 652, in transform
    x, y = transform_proj(_crs, self.proj, x, y)

  File "C:\Anaconda\lib\site-packages\salem\gis.py", line 1157, in transform_proj
    if proj_is_same(p1, p2):

  File "C:\Anaconda\lib\site-packages\salem\gis.py", line 1123, in proj_is_same
    s1.ImportFromProj4(p1.srs)

  File "C:\Anaconda\lib\site-packages\osgeo\osr.py", line 1113, in ImportFromProj4
    return _osr.SpatialReference_ImportFromProj4(self, *args)

TypeError: in method 'SpatialReference_ImportFromProj4', argument 2 of type 'char *'

It seems as if salem is expecting some spatial reference or proj4 attribute to be avaialble?

My DataArray (too large to share) looks this:

 <xarray.DataArray (year: 34, Latitude: 1318, Longitude: 1555)>
 [69682660 values with dtype=float32]
 Coordinates:
   * year       (year) int64 1984 1985 1986 1987 1988 1989 1990 1991 1992 ...
   * Latitude   (Latitude) float64 32.6 32.6 32.6 32.6 32.6 32.6 32.6 32.6 ...
   * Longitude  (Longitude) float64 -87.94 -87.94 -87.94 -87.94 -87.94 -87.94 ...
 Attributes:
     crs:         +init=epsg:4326
     is_tiled:    0
     nodatavals:  (-3.4e+38, -3.4e+38, -3.4e+38, -3.4e+38, -3.4e+38, -3.4e+38,...
     transform:   (-87.9403950755027, 0.0002694945852365922, 0.0, 32.599951492...
     res:         (0.0002694945852365922, 0.00026949458523998555)

Python ver. 2.7.14
xarray ver. 0.10.3
salem ver. 0.2.1

Speedup vertical interpolation

Same problem here, no solution:

http://stackoverflow.com/questions/27622808/fast-3d-interpolation-of-atmospheric-data-in-numpy-scipy

Syntax Error on python 3.7

salem/salem/sio.py

Line 1058 in b1ac94a

get = dask.async.get_sync

This line creates a SyntaxError on python 3.7. I'm not sure why yet, just wanted to report for now.
I'm sure there will be a lot more issues with python 3.7, just slowly working through there.

Add automated roll() function to DataArray and Dataset

Based on http://xarray.pydata.org/en/stable/generated/xarray.DataArray.roll.html, it should be quite easy to implement a function wich automatically converts [-180, 180] datasets to [0,360] and vice versa

set_shapefile() should accept Point gemoetries as well

Currently:

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-23-4355a26f2ba4> in <module>()
      1 g = salem.mercator_grid(center_ll=(-77.524, -9.5179), extent=(50000, 50000))
      2 sm = salem.Map(g, countries=False)
----> 3 sm.set_shapefile(df)
      4 sm.visualize()

~/Documents/git/salem/salem/graphics.py in set_shapefile(self, shape, countries, oceans, rivers, lakes, **kwargs)
    710             self._collections.append(LineCollection(lines, **kwargs))
    711         else:
--> 712             raise NotImplementedError(geomtype)
    713 
    714     def _find_interval(self, max_nticks):

NotImplementedError: Point

[CANCELLED] Deprecate WRF diagnostic variables

It hurts me to do this because I spend a lot of time in designing an OOP API that allows to compute diagnostic variables out of WRF files which is quite elegant.

However, it's unlikely that we will ever have the time to support all possible diagnostic variables that wrf-python supports. Furthermore, wrf-python is faster because it relies on underlying fortran routines.

Therefore, we will start to deprecate WRF's diagnostic and vertical interpolation tools starting with salem v0.3.0 and focus our efforts on the salem - xarray bindings and the plotting capabilities.

Masking

Setting the correct dtype for the latest version of my push request fails for me, as
dytpes is not imported:

other = kwargs.pop('other', dtypes.NA)

solution would be to directly use numpy.nan as numpy is already imported
other = kwargs.pop('other', np.nan)

Add staggered coordinates subsetting to the accessor

This should be fairly easy. If staggered coordinates is spotted, add a +1 to the subset in both directions.

Send warning when plate carree domain extends more than 360°

Segmentation fault (core dumped)

I got Segmentation fault (core dumped) error when i tried import salem . Could anyone address this issue ?
Thank You

import error

(master version)

Got error on import, Python 3, Windows

import salem
Traceback (most recent call last):

  File "<ipython-input-3-f947a389823b>", line 1, in <module>
    import salem

  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\salem\__init__.py", line 60, in <module>
    from salem.datasets import *

  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\salem\datasets.py", line 45, in <module>
    from salem import utils, gis, wrftools, sio

  File "C:\Users\byers\AppData\Local\Continuum\anaconda3\lib\site-packages\salem\utils.py", line 89, in <module>
    memory = Memory(location=hash_cache_dir + '_joblib', verbose=0)

TypeError: __init__() got an unexpected keyword argument 'location'

'Grid' object has no attribute '_nx'

After trying to update the Docker image, it fails running its tests with the following error:

ERROR: test_invert_and_run (oggm.tests.test_prepro.TestGrindelInvert)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/oggm/tests/test_prepro.py", line 1313, in test_invert_and_run

    gis.glacier_masks(gdir)

  File "/usr/local/lib/python3.5/dist-packages/oggm/utils.py", line 1442, in _entity_task

    out = task_func(gdir, **kwargs)

  File "/usr/local/lib/python3.5/dist-packages/oggm/core/preprocessing/gis.py", line 509, in glacier_masks

    assert nx == gdir.grid.nx

  File "/usr/local/lib/python3.5/dist-packages/salem/gis.py", line 273, in nx

    return self._nx

AttributeError: 'Grid' object has no attribute '_nx'

Haven't had time to investigate yet, as I'm still on vacation, will Update here if I find out more.

Drop py2 support

To anyone listening here: I plan to drop py2 support in the next salem release. The tests aren't passing since a couple of months due to dependencies (the cause is unlikely to be in salem itself, although its not completely impossible), and I don't have the time to invest on it.

This is in line with many other projects anyway!

Add an xarray backend for geotiffs

Currently the whole file is read when creating the dataset, this is no good.

'Dataset' object has no attribute 'salem'

dataFile = 'wrfout_d01_2019-03-2?_00:00:00'
ds = xr.open_mfdataset(dataFile)

ds

<xarray.Dataset>
Dimensions: (Time: 216, bottom_top: 23, bottom_top_stag: 24, soil_layers_stag: 4, south_north: 74, south_north_stag: 75, west_east: 87, west_east_stag: 88)
Coordinates:
XLAT (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
XLONG (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
XTIME (Time) datetime64[ns] dask.array<shape=(216,), chunksize=(24,)>
XLAT_U (Time, south_north, west_east_stag) float32 dask.array<shape=(216, 74, 88), chunksize=(24, 74, 88)>
XLONG_U (Time, south_north, west_east_stag) float32 dask.array<shape=(216, 74, 88), chunksize=(24, 74, 88)>
XLAT_V (Time, south_north_stag, west_east) float32 dask.array<shape=(216, 75, 87), chunksize=(24, 75, 87)>
XLONG_V (Time, south_north_stag, west_east) float32 dask.array<shape=(216, 75, 87), chunksize=(24, 75, 87)>
Dimensions without coordinates: Time, bottom_top, bottom_top_stag, soil_layers_stag, south_north, south_north_stag, west_east, west_east_stag
Data variables:
Times (Time) |S19 dask.array<shape=(216,), chunksize=(24,)>
LU_INDEX (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
ZNU (Time, bottom_top) float32 dask.array<shape=(216, 23), chunksize=(24, 23)>
ZNW (Time, bottom_top_stag) float32 dask.array<shape=(216, 24), chunksize=(24, 24)>
ZS (Time, soil_layers_stag) float32 dask.array<shape=(216, 4), chunksize=(24, 4)>
DZS (Time, soil_layers_stag) float32 dask.array<shape=(216, 4), chunksize=(24, 4)>
VAR_SSO (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
U (Time, bottom_top, south_north, west_east_stag) float32 dask.array<shape=(216, 23, 74, 88), chunksize=(24, 23, 74, 88)>
V (Time, bottom_top, south_north_stag, west_east) float32 dask.array<shape=(216, 23, 75, 87), chunksize=(24, 23, 75, 87)>
W (Time, bottom_top_stag, south_north, west_east) float32 dask.array<shape=(216, 24, 74, 87), chunksize=(24, 24, 74, 87)>
PH (Time, bottom_top_stag, south_north, west_east) float32 dask.array<shape=(216, 24, 74, 87), chunksize=(24, 24, 74, 87)>
PHB (Time, bottom_top_stag, south_north, west_east) float32 dask.array<shape=(216, 24, 74, 87), chunksize=(24, 24, 74, 87)>
T (Time, bottom_top, south_north, west_east) float32 dask.array<shape=(216, 23, 74, 87), chunksize=(24, 23, 74, 87)>
HFX_FORCE (Time) float32 dask.array<shape=(216,), chunksize=(24,)>
LH_FORCE (Time) float32 dask.array<shape=(216,), chunksize=(24,)>
TSK_FORCE (Time) float32 dask.array<shape=(216,), chunksize=(24,)>
HFX_FORCE_TEND (Time) float32 dask.array<shape=(216,), chunksize=(24,)>
LH_FORCE_TEND (Time) float32 dask.array<shape=(216,), chunksize=(24,)>
TSK_FORCE_TEND (Time) float32 dask.array<shape=(216,), chunksize=(24,)>
MU (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
MUB (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
NEST_POS (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
P (Time, bottom_top, south_north, west_east) float32 dask.array<shape=(216, 23, 74, 87), chunksize=(24, 23, 74, 87)>
PB (Time, bottom_top, south_north, west_east) float32 dask.array<shape=(216, 23, 74, 87), chunksize=(24, 23, 74, 87)>
FNM (Time, bottom_top) float32 dask.array<shape=(216, 23), chunksize=(24, 23)>
FNP (Time, bottom_top) float32 dask.array<shape=(216, 23), chunksize=(24, 23)>
RDNW (Time, bottom_top) float32 dask.array<shape=(216, 23), chunksize=(24, 23)>
RDN (Time, bottom_top) float32 dask.array<shape=(216, 23), chunksize=(24, 23)>
DNW (Time, bottom_top) float32 dask.array<shape=(216, 23), chunksize=(24, 23)>
DN (Time, bottom_top) float32 dask.array<shape=(216, 23), chunksize=(24, 23)>
CFN (Time) float32 dask.array<shape=(216,), chunksize=(24,)>
CFN1 (Time) float32 dask.array<shape=(216,), chunksize=(24,)>
THIS_IS_AN_IDEAL_RUN (Time) int32 dask.array<shape=(216,), chunksize=(24,)>
P_HYD (Time, bottom_top, south_north, west_east) float32 dask.array<shape=(216, 23, 74, 87), chunksize=(24, 23, 74, 87)>
Q2 (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
T2 (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
TH2 (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
PSFC (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
U10 (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>
V10 (Time, south_north, west_east) float32 dask.array<shape=(216, 74, 87), chunksize=(24, 74, 87)>

u_1= ds.isel(Time=1)

<xarray.Dataset>
Dimensions: (bottom_top: 23, bottom_top_stag: 24, soil_layers_stag: 4, south_north: 74, south_north_stag: 75, west_east: 87, west_east_stag: 88)
Coordinates:
XLAT (south_north, west_east) float32 dask.array<shape=(74, 87), chunksize=(74, 87)>
XLONG (south_north, west_east) float32 dask.array<shape=(74, 87), chunksize=(74, 87)>
XTIME datetime64[ns] dask.array<shape=(), chunksize=()>
XLAT_U (south_north, west_east_stag) float32 dask.array<shape=(74, 88), chunksize=(74, 88)>
XLONG_U (south_north, west_east_stag) float32 dask.array<shape=(74, 88), chunksize=(74, 88)>
XLAT_V (south_north_stag, west_east) float32 dask.array<shape=(75, 87), chunksize=(75, 87)>
XLONG_V (south_north_stag, west_east) float32 dask.array<shape=(75, 87), chunksize=(75, 87)>
Dimensions without coordinates: bottom_top, bottom_top_stag, soil_layers_stag, south_north, south_north_stag, west_east, west_east_stag
Data variables:
Times |S19 dask.array<shape=(), chunksize=()>

u_1.salem.wrf_zlevel('U', levels=10000.)
AttributeError: 'Dataset' object has no attribute 'salem'

Add topographical shading to map generated by Geogrid simulator

I want to use topographical shading instead of natural_earth when plotting area defind by namelist.wps.

Code:

from salem.utils import get_demo_file
from salem import geogrid_simulator
import matplotlib.pyplot as plt

fpath='namelist.wps'
g, maps = geogrid_simulator(fpath)
maps[0].set_topography(get_demo_file('hef_srtm.tif'))
maps[0].set_lonlat_contours()
maps[0].visualize(title='Domains 1 to 3')

Here's the error:

Traceback (most recent call last):
  File "/home/xin/cg/plot.py", line 7, in <module>
    maps[0].set_topography(get_demo_file('hef_srtm.tif'))
  File "/home/xin/Software/anaconda3/lib/python3.6/site-packages/salem/graphics.py", line 888, in set_topography
    crs=wgs84, margin=10)
  File "/home/xin/Software/anaconda3/lib/python3.6/site-packages/salem/datasets.py", line 181, in set_subset
    raise RuntimeError('subset not valid')
RuntimeError: subset not valid

However, this example works fine:

from salem import mercator_grid, Map, get_demo_file, open_xr_dataset
grid = mercator_grid(center_ll=(10.76, 46.79), extent=(18000, 14000))
smap = Map(grid, countries=False)
z = smap.set_topography(get_demo_file('hef_srtm.tif'))
smap.set_data(z)
smap.set_cmap('topo')
smap.visualize(title='Topography', cbar_title='m a.s.l.')

Error plotting countries spanning both eastern and western hemispheres

Thanks for an excellent package! I found that trying to use salem to plot a map for an individual country fails for a country like Russia which wraps around to exist in both western and eastern hemispheres.

borders = borders.loc[borders['name'].isin(['Russia'])]
arr.salem.subset(shape=borders, margin=2)

No module named 'pandas.indexes' bug with pandas 0.20.1

The most recent pandas version (0.20.1) is resulting in this error:

File "C:\Users\rit\Documents\Projects\\do_diagnose.py", line 239, in plot_cntr_maps
  sm = sub_arr.salem.get_map(cmap=cmap)
File "C:\Users\rit\Anaconda3\lib\site-packages\salem\sio.py", line 528, in get_map
  return Map(self.grid, **kwargs)
File "C:\Users\rit\Anaconda3\lib\site-packages\salem\graphics.py", line 423, in __init__
  self.set_shapefile(countries=countries)
File "C:\Users\rit\Anaconda3\lib\site-packages\salem\graphics.py", line 653, in set_shapefile
  return self.set_shapefile(shapefiles['world_borders'], **kwargs)
File "C:\Users\rit\Anaconda3\lib\site-packages\salem\graphics.py", line 668, in set_shapefile
  shape = sio.read_shapefile_to_grid(shape, grid=self.grid)
File "C:\Users\rit\Anaconda3\lib\site-packages\salem\sio.py", line 126, in read_shapefile_to_grid
  **grid.to_dict())
File "C:\Users\rit\Anaconda3\lib\site-packages\joblib\memory.py", line 483, in __call__
  return self._cached_call(args, kwargs)[0]
File "C:\Users\rit\Anaconda3\lib\site-packages\joblib\memory.py", line 457, in _cached_call
  out, metadata = self.call(*args, **kwargs)
File "C:\Users\rit\Anaconda3\lib\site-packages\joblib\memory.py", line 675, in call
  output = self.func(*args, **kwargs)
File "C:\Users\rit\Anaconda3\lib\site-packages\salem\sio.py", line 95, in _memory_shapefile_to_grid
  shape = read_shapefile(shape_cpath, cached=True)
File "C:\Users\rit\Anaconda3\lib\site-packages\salem\sio.py", line 66, in read_shapefile
  out = pickle.load(f)
ModuleNotFoundError: No module named 'pandas.indexes'

It seems like it is coming from salem, is that something that can be addressed? thanks!

better control of GoogleVisibleMap domain

firstly, thanks for creating this.

I am trying to create a GoogleVisibleMap by following the example here: http://salem.readthedocs.io/en/latest/auto_examples/plot_googlestatic.html#sphx-glr-auto-examples-plot-googlestatic-py

from salem import GoogleVisibleMap, Map
import matplotlib.pyplot as plt

mapextent = [-82, -65, 17, 31] # W, E, S, N

g = GoogleVisibleMap(x=[mapextent[0], mapextent[1]], y=[mapextent[2],
                     mapextent[3]])
ggl_img = g.get_vardata()

fig, ax = plt.subplots(1, 1, figsize=(12,8))
sm = Map(g.grid)
sm.set_rgb(ggl_img)
sm.visualize(ax=ax)
plt.tight_layout()
plt.savefig('test.png')

which creates:

The domain created is approximately [-87, -59, 11, 36] # W, E, S, N.

Am I missing some arguments? e.g. scale for GoogleVisibleMap or factor for Map?

What are appropriate values for scale?

salem/salem/datasets.py

Line 557 in 430b9c2

scale : int

and factor?

salem/salem/graphics.py

Line 413 in 3250249

factor : float

Or is trying to make the image square?

Extending geogrid_simulator domain plots?

Hi,

we find the geogrid_simulator very useful, but since we sometime have very small offshore domains, it would be helpful to be able to increase the map boundary beyond the outermost domain. Otherwise we sometimes end up seeing only ocean. We tried different things in that regard already, but -- being new to the library -- so far nothing really worked. Can you think of a simple solution for implementing this?

Thanks,
Martin

Add issame based on OGR

See pyproj4/pyproj#15 (comment)

Encoding not preserved when creating ROI

The salem roi method of the xarray accessor objects is great and to my knowledge and in this user-friendliness nowhere else available.
I'm using it to clip DEMs to shapefiles and get great results. The only point I'm wondering about is if it's intentional that the encoding of the dataset/datarray and its variables is not preserved. In code language:

In[4]: data = xr.open_dataset(gdir.get_filepath('dem_ts'), group=demtype)
In[5]: data.height.encoding
Out[5]: 
{'_FillValue': nan,
 'chunksizes': None,
 'complevel': 0,
 'contiguous': True,
 'dtype': dtype('float32'),
 'fletcher32': False,
 'original_shape': (1, 1201, 3501),
 'shuffle': False,
 'source': 'some_path\\dem_ts.nc',
 'zlib': False}
In[6]: data = data.salem.roi(shape=gdir.get_filepath('outlines'))
In[7]: data.height.encoding
Out[7]: 
{}
In[8]: data.encoding
Out[8]: 
{}

The Fill_Value might interfere with the value that salem fills the mask with if other is not passed here, but in case you agree that this is not too much of a problem I guess catching the encoding before and just inserting it after making the mask could be a solution. Do you see any other problem?

roi fails on [1,N] or [N,1] 2D masks

Hi @fmaussion,

The ds_country.salem.roi(shape=country_shape) fails if the mask is [1,N] or [N,1].

Specifically I have mask on very course data that makes from switzerland a set of 3 boxes on the same latitude.

The traceback made me realise that it fails in sio.py at the def _salem_grid_from_dataset(ds): function.
The lines are

` # OK, get it
x = ds.variables[x][:]
y = ds.variables[y][:]

# Make the grid
dx = x[1]-x[0]  ### THIS DOESN'T WORK FOR 1D
dy = y[1]-y[0]  ### SAME
args = dict(nxny=(x.shape[0], y.shape[0]), proj=proj, dxdy=(dx, dy),
            x0y0=(x[0], y[0]))
return gis.Grid(**args)`

At the moment I bodged the info I needed from a different mask, so I can't help with a fool-proof solution.

~Laurens

Better support of 0-360 datasets

I'm not sure yet how much is working / not working in the 180-360 range, I have to do some tests

Improve plotting docs

Maybe remove the discussion about Datalevels as a whole

salem_cache should delete the downloaded zip file

just because it saves space

If possible, could develpoed share the example data in the salem DOCS =sfor learning and practice?

salem xarray accessor does not work with latest xarray (xarray==0.11)

When running the example (as described here: https://salem.readthedocs.io/en/latest/xarray_acc.html), I get following error.

import numpy as np
import xarray as xr
import salem

da = xr.DataArray(np.arange(20).reshape(4, 5), dims=['lat', 'lon'],
coords={'lat':np.linspace(0, 30, 4),
'lon':np.linspace(-20, 20, 5)})
da.salem # the accessor is an object

AttributeError Traceback (most recent call last)
in
6 coords={'lat':np.linspace(0, 30, 4),
7 'lon':np.linspace(-20, 20, 5)})
----> 8 da.salem # the accessor is an object
9
10

/usr/local/lib/python3.6/dist-packages/xarray/core/common.py in getattr(self, name)
179 return source[name]
180 raise AttributeError("%r object has no attribute %r" %
--> 181 (type(self).name, name))
182
183 def setattr(self, name, value):

AttributeError: 'DataArray' object has no attribute ‘salem'

Scipy's imresize

Scipy's imresize is deprecated in favor of skimage.transform.resize

I made some tests but the two functions are not equivalent for the spline interpolation (looks like a bug in skimage, needs more investigation).

salem still uses imresize in the meantime

salem import error with xarray 0.12.0: cannot import name 'basestring'

Hi @fmaussion

Just letting you know that the latest update of xarray (15 March, version 0.12.0) leads to a salem import error in the salem sio.py subpackage on line 20:

 ---> 20 from xarray.core.pycompat import basestring

ImportError: cannot import name 'basestring'

It seems that basestring disappeared from xarray.core.pycompat between version 0.11.3 and version 0.12.0, although it is not documented on the xarray doc ("what's new")

Also taking the opportunity to thank you for your amazing work with salem, it's an incredibly useful library ...

Nico

Diagnostic variables

List of possible WRF diagnostics I implemented in IDL and that could be added to Salem:

Incorrect grid inference with irregular latitude spacing

I've been playing with salem for analyzing some GEOS-Chem output which I've inherited from a previous post-doc. One of the first things I've been doing is extracting data from parts of the USA using shapefiles from Natural Earth. But I think I've stumbled across a small bug that I want to highlight for you.

For starters, here's a sample 2D lat-lon dataset. We load it up like normal, using xarray and salem:

import xarray as xr
import salem

data = xr.open_dataset("test.nc")

There's an issue with the latitudes in this dataset. The last and first gridpoints are just 1.5 degrees away from the points separating them, while the rest of the grid is equally spaced at 2 degrees. If I try to use "quick_map" on this dataset, I get this:

It's clipped at just over 40N. Luckily, the shapefile subsetting functions still work okay, in principle, but there's a big problem:

shdf = salem.read_shapefile(salem.get_demo_file('world_borders.shp'))
roi = ozone.salem.roi(shape=shdf)
roi.salem.quick_map()

This map looks okay, but the data is geo-referenced incorrectly; basically, salem has tried to stretch the latitudes 90S to 42N to cover our entire map, even though it stretches from 90S to 90N.

However, if we clip off those problematic first and last latitude points, things work much better:

ozone_subset = ozone.isel(lat=slice(1, -1))
ozone_subset.salem.quick_map()

And the subsetted data looks much better, too

Any ideas on what might be the cause of this? I'm not sure if there's a special, custom grid projection I could've passed to salem to help it figure things out in this case. Luckily, cutting off the problematic latitudes is fine - and I'm not looking at the poles in my current studies.

Grid objects should be serializable to... something

And also, they could have a better __repr__

Propagate crs via coordinates

See pydata/xarray#1271 (comment)

Map gridded data

https://github.com/fmaussion/salem/blob/master/salem/gis.py#L603

this should be on center_grid

Map(Grid) causes Python to crash on Windows

I've been following the "plot over Google map" tutorial in an attempt to plot some data points over a satellite image. However, the Map function fails every time. My code will run until Map is called, hangs for 30-45 seconds, then I get a pop up window that says "Python has stopped working. A problem has caused the program to stop working correctly. Windows will close the program and notify you if a solution is available." I'm running Python 3.7.3 on Windows 10 64 bit. Based on how easy is is to reproduce the error, where it occurs in the code, and the error itself I believe it is an issues with Salem on Windows/Python 3.7 rather than a flaw in my code.

This is the function I am trying to run:

import matplotlib.pyplot as plt
import pandas as pd
import salem
from salem import get_demo_file, DataLevels, GoogleVisibleMap, Grid, Map
from pyproj import Proj

def showImage():
    g = GoogleVisibleMap(x=[X1, X2], y=[Y1, Y2],    #Censored lat/long since it is where I live
                     scale=2,
                     maptype='satellite')
    
    grid = g.grid
    print(grid)
    salem.Map(grid, factor=1, countries=False)

This example is the simplest version I have been able to reproduce the issue with. g was taken from the example given in salem.Grid documentation:

import salem
from salem import Map, Grid
from pyproj import Proj

g = Grid(nxny=(3, 2), dxdy=(1, 1), x0y0=(0, 0), proj="wgs84")
Map(g)

Here's a list of all my installed packages:

Package         Version
--------------- -----------
altgraph        0.16.1
asn1crypto      0.24.0
attrs           19.1.0
bcrypt          3.1.6
certifi         2019.6.16
cffi            1.12.3
cftime          1.0.3.4
chardet         3.0.4
Click           7.0
click-plugins   1.1.1
cligj           0.5.0
colorama        0.4.1
cryptography    2.7
cycler          0.10.0
decorator       4.4.0
descartes       1.1.0
Fiona           1.8.6
future          0.17.1
GDAL            2.4.1
geopandas       0.5.1
idlex           1.18
idna            2.8
imageio         2.5.0
joblib          0.13.2
kiwisolver      1.1.0
macholib        1.11
matplotlib      3.1.0
motionless      1.3.2
munch           2.3.2
netCDF4         1.5.1.2
networkx        2.3
numpy           1.16.4
osmnx           0.10
pandas          0.24.2
paramiko        2.5.0
pefile          2019.4.18
Pillow          6.1.0
pip             19.1.1
py2exe          0.9.2.2
pyasn1          0.4.5
pycparser       2.19
PyInstaller     3.4
PyNaCl          1.3.0
pyparsing       2.4.0
pyproj          2.2.2
python-dateutil 2.8.0
pytz            2019.1
PyWavelets      1.0.3
pywin32-ctypes  0.2.0
requests        2.22.0
Rtree           0.8.3
salem           0.2.4
scikit-image    0.15.0
scipy           1.3.0
setuptools      41.0.1
Shapely         1.6.4.post2
six             1.12.0
urllib3         1.25.3
xarray          0.12.3

Merge similar efforts?

Hello,

I recently became aware of your project. I think there is potential to combine efforts so we don't duplicate each others work. I am wondering if there are any interests in attempting to combine efforts and merge projects together?

Here are some additional projects:

pip install not working

Hello,
I started using your library as I'm working with netcdf data. I usually work with virtualenv and install all needed packages through pip.

I wanted to use salem this way too, the install went perfectly but when I wanted to use salem I had errors when trying to turn a xarray dataset into a salem dataset :

AttributeError: 'DataArray' object has no attribute 'salem'

or when trying to execute the very first example

AttributeError: type object 'xr' has no attribute 'open_dataset'

I then git cloned your repo and installed it with python setup.py install and it works fine then, but I just wanted to voice this issue I had.

Thanks !

Add DOI citation to the docs

Setting colorbar range in salem

Is there a way to specify a colorbar in salem which truncates beyond a certain number. It is possible to do it in matplotlib like this:

cs.set_over('k')
cs.set_clim(0, 200)

Transform does not work with (self-built) xr.DataSet

Say I have this piece of code:

sg = salem.Grid(proj='+datum=WGS84 +k=0.9996 +lat_0=0 +lon_0=8.336274703953947 +no_defs +proj=tmerc +units=m +x_0=0 +y_0=0', dxdy=(100, -100), x0y0=(-12991., 5155212.), nxny=(3, 4))
ds = sg.to_dataset()
ds = ds.assign_coords(time=np.array([datetime.date(2018, 1, 1), datetime.date(2018, 1, 2)], dtype=object))
ds['some_variable'] = (['x', 'y', 'time'], np.empty((3, 4, 2)))

If I now want to transform ds (for simplicity) onto itself, the following happens, even though there is a pyproj_srs attribute:

ds.salem.transform(ds)
Traceback (most recent call last):
File "C:\Program Files\Anaconda3\envs\*\lib\site-packages\IPython\core\interactiveshell.py", line 3296, in run_code
   exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-35-50ce9e33d219>", line 1, in <module>
ds.salem.transform(ds)
File "C:\Program Files\Anaconda3\envs\*\lib\site-packages\salem\sio.py", line 647, in transform
return self._apply_transform(transform, grid, other)
File "C:\Program Files\Anaconda3\envs\*\lib\site-packages\salem\sio.py", line 589, in _apply_transform
rdata = transform(var)
File "C:\Program Files\Anaconda3\envs\*\lib\site-packages\salem\gis.py", line 878, in map_gridded_data
grid = data.salem.grid  # try xarray
File "C:\Program Files\Anaconda3\envs\*\lib\site-packages\xarray\core\extensions.py", line 27, in __get__
accessor_obj = self._accessor(obj)
File "C:\Program Files\Anaconda3\envs\*\lib\site-packages\salem\sio.py", line 391, in __init__
raise RuntimeError('dataset Grid not understood.')
RuntimeError: dataset Grid not understood.

Funnily, calling grid_from_dataset with ds directly works completely fine.
If I try to transform before I assign the variable, it "works", but it destroys all coordinates. The output then is:

<xarray.Dataset>
Dimensions:  ()
Data variables:
*empty*
Attributes:
pyproj_srs:  +datum=WGS84 +k=0.9996 +lat_0=0 +lon_0=8.336274703953947 +no...

Transposing such that 'y' and 'x' are the last dimensions (as requested here) doesn't help, neither does constructing a fresh dataset from scratch.
It's a bit puzzling to me...what's wrong here?

Image tests not working well on travis

One day, if I have time and patience, go back to:

script:
  - if [[ "$CONDA_ENV" == "py35-all" ]]; then
      py.test salem --mpl --cov=salem --cov-report term-missing;
    else
      if [[ "$CONDA_ENV" == "py27-all" ]]; then
        py.test salem --mpl --cov=salem --cov-report term-missing;
      else
        py.test salem --cov=salem --cov-report term-missing;
      fi
    fi

and see what happens.

Fatal error opening moving nest WRF file

I recently started working with moving nest WRF output and discovered that salem is unable to open these files. I get a fatal error due to mismatched coordinate information. Is there a work-around for this? Something must be going on with the projection attributes not being consistent with the embedded xlong and xlat arrays. Would it be possible to add this capability to salem so it can work with moving nest output?

My domains are pretty big, but I can share them if you want. A diff of the netcdf headers shows that the only differences for the projection information between a nest that is static vs. one that is moving within the same mother domain are: the dimension attributes (moving nest is smaller), CEN_LAT, and CEN_LON. The other info, such as TRULELAT1, STAND_LON, etc. are identical. In my case, I am using a lambert conformal projection.

I just updated to the latest-and-greatest master from git as of 17-Oct-2018, salem-0.2.2-14_g6f5e707. And if it helps, here is a traceback from the command ds=salem.open_wrf_dataset("wrfout_d02_2011-05-20_01_30_00"):

In [2]: ds = salem.open_wrf_dataset("/scratch1/scratchdirs/rajkrai/run_movingNest/d02/wrfout_d02_2011-05-20_01_30_00")
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-2-ce2e72d08666> in <module>()
----> 1 ds = salem.open_wrf_dataset("/scratch1/scratchdirs/rajkrai/run_movingNest/d02/wrfout_d02_2011-05-20_01_30_00")

~/.conda/envs/bill/lib/python3.6/site-packages/salem/sio.py in open_wrf_dataset(file, **kwargs)
    993
    994     # add cartesian coords
--> 995     ds['west_east'] = ds.salem.grid.x_coord
    996     ds['south_north'] = ds.salem.grid.y_coord
    997

~/.conda/envs/bill/lib/python3.6/site-packages/xarray/core/extensions.py in __get__(self, obj, cls)
     25             return self._accessor
     26         try:
---> 27             accessor_obj = self._accessor(obj)
     28         except AttributeError:
     29             # __getattr__ on data object will swallow any AttributeErrors

~/.conda/envs/bill/lib/python3.6/site-packages/salem/sio.py in __init__(self, xarray_obj)
    400                 pass
    401
--> 402         self.grid = grid_from_dataset(xarray_obj)
    403         if self.grid is None:
    404             raise RuntimeError('dataset Grid not understood.')

~/.conda/envs/bill/lib/python3.6/site-packages/salem/sio.py in grid_from_dataset(ds)
    338     if hasattr(ds, 'MOAD_CEN_LAT') or hasattr(ds, 'PROJ_ENVI_STRING'):
    339         # WRF and HAR have some special attributes
--> 340         return _wrf_grid_from_dataset(ds)
    341
    342     # Try out platte carree

~/.conda/envs/bill/lib/python3.6/site-packages/salem/sio.py in _wrf_grid_from_dataset(ds)
    229             reflat = reflat[0, :, :]
    230         mylon, mylat = grid.ll_coordinates
--> 231         np.testing.assert_allclose(reflon, mylon, atol=atol)
    232         np.testing.assert_allclose(reflat, mylat, atol=atol)
    233

~/.conda/envs/bill/lib/python3.6/site-packages/numpy/testing/nose_tools/utils.py in assert_allclose(actual, desired, rtol, atol, equal_nan, err_msg, verbose)
   1394     header = 'Not equal to tolerance rtol=%g, atol=%g' % (rtol, atol)
   1395     assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
-> 1396                          verbose=verbose, header=header, equal_nan=equal_nan)
   1397
   1398

~/.conda/envs/bill/lib/python3.6/site-packages/numpy/testing/nose_tools/utils.py in assert_array_compare(comparison, x, y, err_msg, verbose, header, precision, equal_nan, equal_inf)
    777                                 verbose=verbose, header=header,
    778                                 names=('x', 'y'), precision=precision)
--> 779             raise AssertionError(msg)
    780     except ValueError:
    781         import traceback

AssertionError:
Not equal to tolerance rtol=1e-07, atol=0.0001

python3.4: posixio.c:342: px_rel: Assertion `pxp->bf_offset <= offset && offset < pxp->bf_offset + (off_t) pxp->bf_extent' failed.
python3.4: posixio.c:265: px_pgin: Assertion `*posp == ((off_t)(-1)) || *posp == lseek(nciop->fd, 0, 1)' failed.

Or, on Travis:

/home/travis/build.sh: line 57:  1696 Segmentation fault      (core dumped)

I'm not sure how to deal with this yet...

fmaussion / salem Goto Github PK

salem's Introduction

Salem

Documentation

License

About

salem's People

Contributors

Stargazers

Watchers

Forkers

salem's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs