nansencenter / nansat

Scientist friendly Python toolbox for processing 2D satellite Earth observation data.

Branch	Travis CI	Code Coverage	Zenodo DOI
Master
Develop

Nansat is a scientist friendly Python toolbox for processing 2D satellite earth observation data.

The main goal of Nansat is to facilitate:

• easy development and testing of scientific algorithms,
• easy analysis of geospatial data, and
• efficient operational processing.

You can find a detailed description of Nansat in our paper published in Journal of Open Research Software in 2016.

... and you can join the mailing list.

We appreciate acknowledgments of Nansat. Please add a reference to the following paper if you use Nansat in scientific publications:

Korosov A.A., Hansen M.W., Dagestad K.-F., Yamakawa A., Vines A., Riechert M., (2016). Nansat: a Scientist-Orientated Python Package for Geospatial Data Processing. Journal of Open Research Software. 4(1), p.e39. DOI: http://doi.org/10.5334/jors.120

Documentation

You will find complete documentation for Nansat at Read the Docs.

Contributing

You will find information about contributing to Nansat at Read the Docs.

Installation

An easy way to install Nansat requirements on any platform is to use Anaconda (download installer).

# create environment with key requirements
conda create -y -n py3nansat gdal numpy pillow netcdf4 scipy
# activate environment
conda activate py3nansat
# install nansat
pip install nansat
# launch python
python

Another option is to use Docker containers (read about Docker).:

# download image with everything pre-installed and launch ipython
docker run --rm -it -v /path/to/data:/data akorosov/nansat ipython

Example

# download a test file
wget https://github.com/nansencenter/nansat/raw/develop/nansat/tests/data/stere.tif

# import main file opener
from nansat import Nansat

# open a test file
n = Nansat('stere.tif')

# see file content
print(n)

# create RGB with auto-stretched histogram
n.write_figure('stere_rgb.png', [1,2,3], clim='hist')

Tests

Nansat is outfitted with unittests, which you can use to ensure that all functionality works on your platform.

# install testing packages from conda-forge
conda install -c conda-forge nose mock

# run nansat.tests
nosetests nansat

# Run all tests including nansat_integration_tests with coverage
cd <nansat_repository_folder>
nosetests -w . --with-coverage --cover-package=nansat

Nansat works on both Python 2 and Python 3 but automatic testing on TravisCI is done for Python 3.7 only. Fore more information see Read the Docs or notebooks for Nansat lectures

License

The project is licensed under the GNU general public license version 3.

Acknowledgments

Development is supported by the Research Council of Norway as a part of NORMAP project (grant no. 195397/V30).

nansat's Issues

Radarsat-2 can be both left and right looking

In the radarsat-2 mapper we have hard-coded a right-looking configuration. Apparently, this is usually the case but we should check to be sure...

Pixelfunction UVToDirectionFrom returns array where all numbers are the same, when they should be varying

This is a follow-up to issue #12 which wasn't really an issue... In a Hirlam file (at nersc: sat/auxdata/model/met.no/HIRLAM_10kmEurope_20110608_0000.grib) the wind direction calculated using numpy is different from the wind direction calculated from the pixel function. The numpy method gives the correct answer whereas the pixelfunction returns an array where all values are equal and completely wrong.

Nansatmap doesn't create stereographi maps

Problem

projection is taken from proj4 string but not used anywhere

Solution

• use projection in creation of Basemap
• specify lat_0 and lon_0 (from proj4 string) when projection is stere

Problem with _FillValue when resizing

I think we might have a problem with the fill values when resizing. It seems the fill values are also interpolated or used in the average/nearest-neighbor. We need to check further and find out how to solve it...

Nansat and VRT should be independent of mappers (no mapper parameters in Nansat or VRT)

Problem

• Nansat docstring contains too much descriptive information about parameters of each mapper
  VRT contains default values for mappers. Technically it works but for adding new mapper and for adding new (modification) parameter - one has to change three files. This is inefficient.
• Nansat and VRT should be generic - not dependent on mappers.

Solution

• Remove description of parameters from Nansat
• Remove default values from VRT and remove _set_default
• Keep everything in mappers only

Pixelfunction BetaSigmaToIncidence returns complex incidence angles

Setting 'dataType': 6 in dst for the incidence angle band gives erroneous result when the src data is complex. We must have dataType=dtype (see code) for this to become correct, but then the result is complex numbers where the real part is the magnitude and the imaginary part is 0.

I'm checking with other pixel functions if the problem is the same there...

metadata is lost when resizing with eResampleAlg = 1

Metadata before resizing:

In [6]: w.get_metadata()
Out[6]:
{'ACQUISITION_START_TIME': '2012-06-15T06:20:30.320740Z',
'BEAM_MODE': 'FQ11',
'BETA_NOUGHT_LUT': 'lutBeta.xml',
'Conventions': 'CF-1.5',
'FACILITY_IDENTIFIER': 'Not Specified',
'FAR_RANGE_INCIDENCE_ANGLE': '32.0052795',
'FIRST_LINE_TIME': '2012-06-15T06:20:30.458312Z',
'GAMMA_LUT': 'lutGamma.xml',
'GDAL': 'GDAL 1.10.0, released 2013/04/24',
'LAST_LINE_TIME': '2012-06-15T06:20:34.657212Z',
'LINE_SPACING': '5',
'Model wind field': 'hirlam',
'NEAR_RANGE_INCIDENCE_ANGLE': '30.3024979',
'ORBIT_DATA_FILE': '23505_DEF.ORB',
'ORBIT_DATA_SOURCE': 'Downlinked',
'ORBIT_DIRECTION': 'Descending',
'PIXEL_SPACING': '5',
'PROCESSING_FACILITY': 'KSAT',
'PROCESSING_TIME': '2012-11-19T14:23:20.000000Z',
'PRODUCT_TYPE': 'SGF',
'SAR_center_look_direction': '282.702416328',
'SATELLITE_IDENTIFIER': 'RADARSAT-2',
'SENSOR_IDENTIFIER': 'SAR',
'SIGMA_NOUGHT_LUT': 'lutSigma.xml',
'SLANT_RANGE_NEAR_EDGE': '910764.1268146718',
'fileName': '/vsimem/ABE5D2JBDZ.vrt',
'history': 'Tue Sep 03 11:31:21 2013: GDAL CreateCopy( /Volumes/sat/downloads/Radarsat2/seaU/RS2_20120615_062030_0004_FQ11_HHVVHVVH_SGF_202391_7078_7972066_windfield.nc, ... )'}

In [7]: w.resize(0.5,eResampleAlg=1)

And after:

In [8]: w.get_metadata()
Out[8]: {'fileName': '/vsimem/8FKTXD27EH.vrt'}

In [9]:

fontSize should be proportional to Figure size

Problem

It is difficult to guess the output font size of the Figure. Usually one have to test several times. But when the output figure size is not known it is impossible to make a proper fontSize.

Solution

Make fontSize to be proportional to the size of Figure. Add e.g. fontRatio=1 which defines the ratio between the figure size and font size. With default value fontSize is such that caption is readable and fits into legend when figure is displayed at full screen.

Pixelfunction BetaSigmaToIncidence returns incidence angles equal to 0

This happens when the inci should be nan (to avoid division by 0). It might be better to use a predefined _FillValue, which is treated correctly with the getitem() function in the Nansat class.

Make code compliant with PEP

Problem:

• Current docstrings are not completely compliant with pep-0257
• Current code is not completely compliant with PEP8

Solutions:

Nansat package should have a docstring in init.py
Some docstrings should also be improved (shorter and more consise).
use pep8 for checking compliancy

Test/tutorial for all classes of nansat

Problem

We have a lot of classes and a lot of public methods. It is difficult for a user to have an overview of APIs. It is difficult for a developer to test all possible situations that occur in the code and potential error after introducing changes.

Solution

Let's have a testing script for each class in nansat. The test script calls every public method of the class and maybe repeats calls to few methods to illustrate/test their functionality under different circumstances or with different input parameters. Scripts are supplied with class description and calls to methods are supplied with detailed comments. Thus we combine testing and education.

TODO, TODO, TODO-DO-DO-DO:

• split tutorial.py into domain_test.py and nansat_test.py
• add figure_test.py
• update nansatmap_test.py
• update nansatshape_test.py
• add pointbrowser_test.py
• add brief class description to each script
• add comments for each operation
• add a master script all_test.py that calls all other scripts

QUESTIONS:

• What is a good convention for calling testing/tutorial scripts _test.py, _tutorial.py, _tut.py ?
• Should tutorial have a wiki counterpart with nicer formatting of comments?

Unify band name

band name of incidence angle in RS2 is " incidence_angle" and ASA has "incidenceAngle". Unify them "incidence_angle".

Create color GeoTiff

Create color GoeTiff file using export(). Set 'PHOTOMETRIC' option?

Divide resize function into two functions - one for resizing the nansat object, and one for returning a resized numpy array

This is related to issue #33 - for v0.5 we separate the resize function in two, such that the one corresponding to eResampleAlg>0 returns a numpy.array whereas the resize function only has two options; eResampleAlg= -1 or 0

mapper for high resolution band of Landsat8

Landsat8 includes different resolution bands. (band 1-7,9: 30-meters, band 8: 15-meters).
Currently the size of the 8th band is reduced to 50%.
If option is given, return band with full resolution of 8th band.

Domain overlap() and cover_point()

I propose two new functions for the Domain class:

domain.cover_point(lon, lat) 

returns True of the point is within the domain, and False otherwise

some_domain.overlap(another_domain) 

returns a fraction between 0 and 1, indicating the percentage of some_domain overlapped by another_domain (0 for no intersection, and 1 if some_domain is fully covered by another_domain)

Add option to VRT class to save file on disk instead of in memory

When a lot of new bands are added, it can be too heavy on the machine as these are kept in memory. It could therefore be better to optionally write added bands to files on disk.

Make mapper for ASAR WSM Doppler centroid grid

Make a new mapper called "mapper_doppler.py" to read the Doppler centroid grid in ASAR WSM data. It should have the bands given here: https://earth.esa.int/handbooks/asar/CNTR6-6-25.htm#eph.asar.asardf.asarrec.ASAR_Dop_Cen_Grid_ADSR + downscaled raw counts and incidence angle, as well as the geolocation data used in the asar mapper.

I will make the nansat get_mapper sort the mapper list alphabetically, to ensure mapper_asar.py is used as default. The doppler mapper will then only be used if specified in the call to Nansat.init.

When nansat can read the doppler grid, we'll make a new python package that will do the actual processing to sea surface range velocity.

VRT.create_warped_vrt() - has different parameters than in Nansat.reproject

Problem

• After some last commits the pattern of calling VRT.create_warped_vrt from Nansat.reproject() differs significantly from the function definition.
• **kwargs are not used in create_warped_vrt()

Solution

• Restore create_warped_vrt() as it was before.
• Always test (using e.g. tutorial.py) before push

get_transect from Nansat objects where the _FillValue is a real number fails

For a transect which is not smoothed, it is easy to filter the fill-values after running get_transect. If smoothing is applied, the fill-values are now included in the filter and the result becomes wrong. We therefore need to set the invalid elements to numpy.nan before smoothing.

Solution: I propose to add a function invalid2nan in the Nansat class.

Nansat.get_transect() shows very dark image

Problem

When Nansat.get_transect() is used without parameters a very dark figure may be shown.

Solution

To apply histogram stretching as in Figure or vmin/vmax as in plt.imshow

Make function to extract subset based on row/col and/or lon/lat limits

We need a function with the following usage:

>>> nansat.subset(rowmin, rowmax, colmin, colmax),

and also:

>>> nansat.subset(lonmin, lonmax, latmin, latmax)

selecting the smallest rectangle covering the lon-lat limits.

Basically this involves making some methods based around the lines below, as suggested by Anton on nansat-dev mailinglist:

n = Nansat('gcps.tif')
lon,lat = n.get_geolocation_grids()

we, es = 28.5, 29.5
no, so = 71.2, 70.6
mask = (lon > we) * (lon < es) * (lat < no) * (lat > so)

minrow = min(np.nonzero(mask)[0])
maxrow = max(np.nonzero(mask)[0])
mincol = min(np.nonzero(mask)[1])
maxcol = max(np.nonzero(mask)[1])

subset_array = n[1][minrow:maxrow, mincol:maxcol]

Remaining is to edit/update the VRT XML to subset the VRT.
For large datasets, reading full res lat and lon into memory is not good, thus we might consider a faster, memory-saving method.

Slow Nansat.get_transect()

Problem

The band arrays are requested at least twice in the Nansat.get_transect():
first at line 1844: browser = PointBrowser(self[firstBand])
second at line: 1892: data = self[iBand]

Solution

Open the first band only once and use it for both generation of PointBrowser and for querying values

Improve Nansatshape

Nansatshape was created by asuka in July 2013. After check few things should be improved:

• class description
• more user friendly API (less parameters)
• easier structure (less methods)

Python Drivers for data output

Problem

GDAL drivers have limited functionality and we need more dataformats to write to (e.g. HDF, NPZ)

Solution

We need to make drivers for writing different formats or for improvement of existing ones. Drivers can be in python using various python libraries including GDAL but not only. Drivers can be organized in a directory drivers and should have similar interface. Maybe each driver should be a class Driver <- VRT.

Nansat reproject fails for an ncep model wind field used for a SAR image over Svalbard

This should be tested using openwind software. Use the file RS2_20130520_062731_0076_SCWA_HHHV_SGF_260697_5426_8472420.zip for testing.

Nansat.get_transect does not work with 'Ipython --pylab'

With pylab it's possible to show images directly after imshow, no need to type show() afterwards. This causes the get_transect function not to work because the figure handle is lost. Propably it is possible to temporarily change a setting in pylab to make it work as "normal".

Add pixel function for getting sigma0 in VV polarization from HH for ASAR

The positive direction of the angle returned from UVToDirection* pixel functions should follow CF standard

Presently, these pixel functions return the mathematical angle which is defined positive in the anti-clockwise direction from north. In CF-conventions, the positive direction is clockwise from north, so we need to apply this to the pixel functions as well...

Function upwards_azimuth_direction in Domain class needs the orbit direction as required input(?)

I have a radarsat-2 ascending pass acquisition, and get azimuth=169 degrees. The azimuth should be relative to north, and is therefore wrong by 180 degrees. Do you agree?

Checking the function - startlat > endlat, whereas it should be opposite as far as I understand... I will create a quick solution that works for my radarsat-2 data now but this has to be checked properly...

resize with eResampleAlg>0 gives ProjectionError

n0: Some nansat object

In [37]: n0.resize(0.5,eResampleAlg=1)
In [38]: n0

ProjectionError: Empty projection in input dataset!

Same happens for eResampleAlg=2,3,4

Nansat.get_transect() should get smoothed values from pixels

Problem

get_transect only gets values of pixels on the line between the two points. But sometimes it is required to get weighted averages from the pixels on the line and neighbours.

Solution

Smooth the images using e.g. gauss_interpolation as in Nansatmap before extracting pixels. Additional parameters for the method: smooth=3

Make mapper for GLOBCOLOUR L3B data

The mapper should create GDAL-friendly grids from 1D vectors of lon, lat and var

Longitude and latitude seems to be wrong in exported files

I am working with some radarsat images and export them to netcdf. The band arrays seem to be flipped in the netcdf files. Asuka, can you check this? You could try to plot the data on a map before and after exporting. I get different results...

Remove vrt.__modify_warped_XML()

To make the Nansat code simpler and clearer, I propose to remove the function vrt._modify_warped_XML()
This function is only used in one place (also within vrt.py), and it would thus thus clearer if the statements in the function are written explicitly/inline, instead of making an extra function call. The function seems not very reusable, and the name of the function is also not very good; it indicates some modification of xml, but not why or in what sense.

Add bands of all channels in asar mapper

Currently asar mapper reads only 'SPH_MDS1_TX_RX_POLAR' channel.
Some (or all?) of asar data have 'SPH_MDS2_TX_RX_POLAR'' as well.
ASAR mapper should read both of them.

Exporting selected bands

Presently, we have two export functions. Nansat.export exports all the bands, whereas Nansat.export_band exports only one. Isn't it better to just have one export-function where you can either export everything or just selected bands, e.g., like bands=[2,3,4]?

Nansatmap.contour repeats contourf

Problem

Nansatmap.contour and contourf are 95% similar and repeat the same mistakes.

Solution:

1. Make additional function e.g. do_contour() which is called from contour and contourf and which performs most of the job.
2. Replace the doubtfull solution with "validValues=None, ticks=7, decimals=0" to a much simpler one contour(data, interval=None, ...). Let the user generate interval outside Nansatmap.

Nansat.getitem() cannot replace int with nan

When accessing band which has any INT datatype and _FillValue in attributes Nansat.getitem tries to replace _FillValue with np.nan, but that doesn't work with integer numpy arrays and Nansat.getitem crashes. Hence we cannot access data from int bands.

export function fails on second export

I have tried this with two SAR images: first, I do some processing of the original data, add bands, and export to netcdf. This is ok. Then, I do some more processing, add the bands, and export. When reading the last exported netcdf file, all values in the bands that were added in the first round are set to -10000 (the _Fillvalue).

Cropping with Nansat

After a couple of days playing with Nansat I have a question arised:

Is there a way to crop an image with Nansat, process it and then add band without re-projecting to initial state?

As there is a warning in the functionnansat.add_band() the cropping is not possible at the moment:

NB : Adding band is possible for raw (nonprojected, nonresized) images
only. Adding band will cancel any previous reproject() or resize()

But the solution is very simple and straightforward. One should simply crop the image by re-projecting to some specified domainn.reproject(d), and then create the copy of the nansat vrt:

n.reproject(d)
n_cropped = Nansat(n.vrt.fileName)
n_cropped.fileName = n.fileName

Related issue as mentioned by @knutfrode: #38

I think it is worse mentioning that cropping is one of the core features of Nansat and after some discussion with @knutfrode we are surprised Nansat lacks it.
Please, could one of the team ( @akorosov, @asumak or @mortenwh ) try to fix this as you have been most into such issues recently.

P.S. THX @knutfrode for your help and time

SAR images with complex data need to have non-complex bands

Many packages depending on nansat expect some standard SAR bands like sigma0 in VV pol to be in real numbers. It will therefore be useful to adjust the SAR mappers to use the IntensityPixelFunc to provide sigma0 in magnitude with band names sigma0_VV, sigma0_HH, etc. I propose to add "_complex" to the suffix of bands with complex data.

New concept of VRT

Problem

Now we have two VRT objects inside Nansat:

• raw references original file (or several files, or sereval in memory arrays)
• vrt is either a copy of raw or references raw and is WarpedVRT and contains definition of transformation:

TIF <-- raw <-- vrt

The problem arises when we want to for resize with eResampleAlg>0. It creates vrt with wrong geo-reference. We then can add additional VRT layer warped_resized which references raw and vrt which references warped_resized but contains correct geo-reference:

TIF <-- raw <-- warped_resized <-- vrt
But it is unique operation and hard to treat it in other situation, e.g. when adding bands.

Solution

Let's always use at least three VRTs:

TIF <-- rawVRT <-- proVRT <-- useVRT

• rawVRT is always generated by the mapper and never touched
• proVRT contains definition of any spatial transfromation: resize, reproject, roll, subsample, etc
• useVRT is a frontend, it has correct geo-reference. We add bands also here.

There is no spatial transformation between proVRT and useVRT hence it should not be slower for GDAL to have additional VRT.

If we want more trasfromations, e.g. roll and reproject we can add two proVRT

Interpolation does not work when resizing complex and pixelfunction bands

I have a radarsat-2 object of complex data for which I have made pixel function bands to get the sigma0 magnitudes. When resizing the object, both the complex bands and the pixel function bands are wrongly interpolated (also for average and nearest neighbor). However, copying the magnitude bands to numpy arrays and adding those as separate bands before resizing gives correct result. So it seems resizing in general does not work for complex bands and pixel functions. Can anyone confirm this?

I will add a note in the docstring of resize and resize_lite for reference...

Convert from self.d['attribute'] to self.attribute

Problem

The way we store some default values in some classes (e.g. Figure) is cumbersome and hard to read:
self.d['attribute'] = value
It was used to be able to set the default value using string as a parameter name (e.g. Figure._set_defaults()):

Solution

To use class attributes and getattr(), setattr(), hasattr():

class Figure:
    attr1 = 1
    attr2 = 2
    def _set_default(self, dict):
        ....
        setattr(self, dictkey, dictvalue)

Arguement in ASA mapper is not right

ASA_full_incAng in asar_mapper is not right name.

Remove Nansat.write_nansatmap() for Nansat 0.5

Problem

Unlike write_figure(), which is quite simple, write_nansatmap() Is a very complex method. However it seems to be nearly useless. We cannot cover all needs that a user may face using Nansatmap. And with many options we just introduce more bugs.

Solution

Remove write_nansatmap() at least from 0.5 release.

Memory leak in Nansat (obpg_l2 mapper)

Problem:

When opening 1000 Modis L2 files with Nansat the memory consumption by python process raised from 2% to 17%. The files were not kept open but closed immediately after opening. Opening the same files with gdal.Open() gave no change in average memory consumption.

Solution:

• Check with other files (e.g. ASAR)
• Look at obpg_l2 mapper for identification of possible sources of leakage
• Look at VRT destructor
• Find a tool for memory leakage investigation

Nansat in develop cannot open Radarsat ScanSAR_Wide

In [5]: n = Nansat('/media/123684083683EAD5/Radarsat2/RS2_ScanSAR_Wide/RS2_20120415_041249_0046_SCNB_HHHV_SGF_191351_4674_7304602', mapperName='radarsat2')

KeyError Traceback (most recent call last)
/home/natalia/nansat/ in ()
----> 1 n = Nansat('/media/123684083683EAD5/Radarsat2/RS2_ScanSAR_Wide/RS2_20120415_041249_0046_SCNB_HHHV_SGF_191351_4674_7304602', mapperName='radarsat2')

/home/natalia/nansat/nansat.pyc in init(self, fileName, mapperName, domain, array, parameters, logLevel, **kwargs)
175 if fileName != '':
176 # Make original VRT object with mapping of variables

--> 177 self.raw = self._get_mapper(mapperName, **kwargs)
178 # Set current VRT object

179             self.vrt = self.raw.copy()

/home/natalia/nansat/nansat.pyc in _get_mapper(self, mapperName, *_kwargs)
1184 raise Error('Mapper ' + mapperName + ' not in PYTHONPATH')
1185 tmpVRT = mapper_module.Mapper(self.fileName, gdalDataset,
-> 1186 metadata, *_kwargs)
1187 self.mapper = mapperName
1188 else:

/home/natalia/nansat/mappers/mapper_radarsat2.pyc in init(self, fileName, gdalDataset, gdalMetadata, **kwargs)
148 self.dataset.SetMetadataItem('SAR_center_look_direction', str(mod(
149 Domain(ds=gdalDataset).upwards_azimuth_direction( orbit_direction =
--> 150 gdalDataset.GetMetadata()['ORBIT_DIRECTION']) + antennaPointing,
151 360)))
152

KeyError: 'ORBIT_DIRECTION'

Resize to given pixel size

Currently nansat.resize() can resize by a given factor, or to given width/height in pixels.
But we should add possibility to resize to a desired pixel size in meters.

Thus I propose to update nansat.resize() to allow:

s.resize(pixel_size=500)

This would again require a utility function (useful for other purposes):

s.get_pixel_size()

Note that pixel size might be variable for certain map projections, in such cases an average pixel size would be useful.