gher-uliege / divand.jl

DIVAnd performs an n-dimensional variational analysis of arbitrarily located observations

DIVAnd

---

DIVAnd (Data-Interpolating Variational Analysis in n dimensions) performs an n-dimensional variational analysis/gridding of arbitrarily located observations. Observations will be interpolated/analyzed on a curvilinear grid in 1, 2, 3 or more dimensions. In this sense it is a generalization of the original two-dimensional DIVA version (still available here https://github.com/gher-uliege/DIVA but not further developed anymore).

The method bears some similarities and equivalences with Optimal Interpolation or Krigging in that it allows to create a smooth and continous field from a collection of observations, observations which can be affected by errors. The analysis method is however different in practise, allowing to take into account topological features, physical constraints etc in a natural way. The method was initially developped with ocean data in mind, but it can be applied to any field where localized observations have to be used to produce gridded fields which are "smooth".

See also https://gher-uliege.github.io/DIVAnd-presentation/#1

Please cite this paper as follows if you use DIVAnd in a publication:

Barth, A., Beckers, J.-M., Troupin, C., Alvera-Azcárate, A., and Vandenbulcke, L.: DIVAnd-1.0: n-dimensional variational data analysis for ocean observations, Geosci. Model Dev., 7, 225-241, doi:10.5194/gmd-7-225-2014, 2014.

(click here for the BibTeX entry).

Summary of features

• N-Dimensional analysis/interpolation
• Scattered data
• Noise allowed
• Physical constraints can be added
• Inequality constraints can be added
• Topological constraints are handled naturally (barriers, holes)
• Analysis error maps can be estimated
• Periodicity in selected directions can be enforced
• Multivariate data can be used (experimental)
• The output grid can be curvilinear
• Instead of interpolating scattered data you can also peform Kernel Density Estimations with the points.

Installing

You need Julia (version 1.6 or later) to run DIVAnd. The command line version is sufficient for DIVAnd. Inside a Julia terminal, you can download and install the package by issuing:

using Pkg
Pkg.add("DIVAnd")

It is not recommended to download the source of DIVAnd.jl directly (using the green Clone or Download button above) because this by-passes Julia's package manager and you would need to install the dependencies of DIVAnd.jl manually.

Updating DIVAnd

To update DIVAnd, run the following command and restart Julia (or restart the jupyter notebook kernel using Kernel -> Restart):

using Pkg
Pkg.update("DIVAnd")

Note that Julia does not directly delete the previous installed version. To check if you have the latest version run the following command:

using Pkg
Pkg.status()

The latest version number is available from here.

To explicitly install a given version X.Y.Z you can also use:

using Pkg
Pkg.add(name="DIVAnd", version="X.Y.Z")

Or the master version:

using Pkg
Pkg.add(name="DIVAnd", rev="master")

Testing

A test script is included to verify the correct functioning of the toolbox. The script should be run in a Julia session. Make sure to be in a directory with write-access (for example your home directory). You can change the directory to your home directory with the cd(homedir()) command.

using Pkg
Pkg.test("DIVAnd")

All tests should pass without error (it can take several minutes).

INFO: Testing DIVAnd
Test Summary: | Pass  Total
  DIVAnd      |  461    461
INFO: DIVAnd tests passed

The test suite will download some sample data. You need to have Internet access and run the test function from a directory with write access.

Documentation

The main routine of this toolbox is called DIVAnd which performs an n-dimensional variational analysis of arbitrarily located observations. Type the following in Julia to view a list of parameters:

using DIVAnd
?DIVAndrun

see also https://gher-uliege.github.io/DIVAnd.jl/latest/index.html

Example

DIVAnd_simple_example_4D.jl is a basic example in fours dimensions. The call to DIVAndrun looks like this:

fi,s = DIVAndrun(mask,(pm,pn,po,pq),(xi,yi,zi,ti),(x,y,z,t),f,len,epsilon2);

where mask is the land-sea mask, usually obtained from the bathymetry/topography, (pm,pn,po,pq) is a n-element tuple (4 in this case) containing the scale factors of the grid, (xi,yi,zi,ti) is a n-element tuple containing the coordinates of the final grid, (x,y,z,t) is a n-element tuple containing the coordinates of the observations, f is the data anomalies (with respect to a background field), len is the correlation length and epsilon2 is the error variance of the observations.

The call returns fi, the analyzed field on the grid (xi,yi,zi,ti).

More examples are available in the notebooks from the Diva Workshop.

Note on which analysis function to use

DIVAndrun is the core analysis function in n dimensions. It does not know anything about the physical parameters or units you work with. Coordinates can also be very general. The only constraint is that the metrics (pm,pn,po,...) when multiplied by the corresponding length scales len lead to non-dimensional parameters. Furthermore the coordinates of the output grid (xi,yi,zi,...) need to have the same units as the observation coordinates (x,y,z,...).

DIVAndfun is a version with a minimal set of parameters (the coordinates and values of observations, i.e. (x,f), the remaining parameters being optional) and provides an interpolation function rather than an already gridded field.

diva3D is a higher-level function specifically designed for climatological analysis of data on Earth, using longitude/latitude/depth/time coordinates and correlations length in meters. It makes the necessary preparation of metrics, parameter optimizations etc you normally would program yourself before calling the analysis function DIVAndrun.

DIVAnd_heatmap can be used for additive data and produces Kernel Density Estimations.

DIVAndgo is only needed for very large problems when a call to DIVAndrun leads to memory or CPU time problems. This function tries to decide which solver (direct or iterative) to use and how to make an automatic domain decomposition. Not all options from DIVAndrun are available.

If you want to try out multivariate approaches, you can look at DIVAnd_multivarEOF and DIVAnd_multivarJAC

Note about the background field

If zero is not a valid first guess for your variable (as it is the case for e.g. ocean temperature), you have to subtract the first guess from the observations before calling DIVAnd and then add the first guess back in.

Determining the analysis parameters

The parameter epsilon2 and parameter len are crucial for the analysis.

epsilon2 corresponds to the inverse of the signal-to-noise ratio. epsilon2 is the normalized variance of observation error (i.e. divided by the background error variance). Therefore, its value depends on how accurate and how representative the observations are. len corresponds to the correlation length and the value of len can sometimes be determined by physical arguments. Note that there should be one correlation length per dimension of the analysis.

One statistical way to determine the parameter(s) is to do a cross-validation.

1. choose, at random, a relatively small subset of observations (about 5%). This is the validation data set.
2. make the analysis without your validation data set
3. compare the analysis to your validation data set and compute the RMS difference
4. repeat steps 2 and 3 with different values of the parameters and try to minimize the RMS difference.

You can repeat all steps with a different validation data set to ensure that the optimal parameter values are robust. Tools to help you are included in (DIVAnd_cv.jl).

Note about the error fields

DIVAnd allows the calculation of the analysis error variance, scaled by the background error variance. Though it can be calculated "exactly" using the diagonal of the error covariance matrix s.P, it is generally too costly and approximations are provided. All of them are accessible as options via DIVAnd_errormap or you can let DIVAnd decide which version to use (possibly by specifying if you just need a quick estimate or a version closer the theoretical estimate) (see Beckers et al 2014 )

Advanced usage

Additional constraint

An arbitrary number of additional quadratic constraints can be included to the cost function which should have the following form:

J(x) = ∑_i (C_i x - z_i)ᵀ Q_i^-1 (C_i x - z_i)

For every constrain, a structure with the following fields is passed to DIVAnd:

• yo: the vector z_i
• H: the matrix C_i
• R: the matrix Q_i (symmetric and positive defined)

Internally the observations are also implemented as constraint defined in this way.

Additional inequality constraint

An arbitrary number of additional inequality constraints can be included and which should have the following form:

(H_i x > yo_i)

For every constraint, a structure with the following fields is passed to DIVAnd:

• yo: a vector
• H: a matrix

Run notebooks on a server which has no graphical interface

On the server, launch the notebook with:

jupyter-notebook --no-browser --ip='0.0.0.0' --port=8888

where the path to jupyter-notebook might have to be adapted, depending on your installation. The ip and port parameters can also be modified.

Then from the local machine it is possible to connect to the server through the browser.

Thanks to Lennert and Bart (VLIZ) for this trick.

Example data

Some examples in DIVAnd.jl use a quite large data set which cannot be efficiently distributed through git. This data can be downloaded from the URL https://dox.ulg.ac.be/index.php/s/Bo01EicxnMgP9E3/download. The zip file should be decompressed and the directory DIVAnd-example-data should be placed on the same level than the directory DIVAnd.jl.

Reporting issues

Please include the following information when reporting an issue:

• Version of Julia
• Version of DIVAnd
• Operating system
• Full screen output preferably obtained by setting ENV["JULIA_DEBUG"] = "DIVAnd".
• Full stack strace with error message
• A short description of the problem
• The command and their arguments which produced the error

Note that only official julia builds are supported.

In all cases, if we provide a tentative solution, please provide a feedback in all cases (whether it solved your issue or not).

Fun

An educational web application has been developed to reconstruct a field based on point "observations". The user must choose in an optimal way the location of 10 observations such that the analysed field obtained by DIVAnd based on these observations is as close as possible to the original field.

You do not want to use Julia

You should really reconsider and try out Julia. It is easy to use and provides the native interface to DIVAnd.

If you have a stable workflow using python, into which you want to integrate DIVAnd, you might try

https://github.com/gher-uliege/DIVAnd.py

divand.jl's Issues

netCDF scalefactore is NaN resulting in an empty output variable

While trying to figure out the issue #63 I updated and rebuilt DIVAnd. After that I'm getting the following message at the end of the analysis:

[ Info: Creating netCDF file
[ Info: Time step 1 / 1
[ Info: scaled correlation length (min,max) in dimension 1: (200000.0, 200000.0)
[ Info: scaled correlation length (min,max) in dimension 2: (200000.0, 200000.0)
[ Info: scaled correlation length (min,max) in dimension 3: (25.0, 25.0)
[ Info: number of windows: 1
┌ Warning: scalefactore has the value NaN
└ @ DIVAnd C:\Users\ax_li.julia\packages\DIVAnd\gsgE\src\DIVAnd_adaptedeps2.jl:72

The resulting nc file has just metadata, the actual output variable is empty. Also tested on a piece of code which worked correct before the update.

s.obsout and moddim

In my lon-lat-depth analysis using DIVAndrun, observations between the last longitude grid 358 and the first longitude grid 2 were masked out and returned NaN residuals even by setting moddim = [360, 0, 0]. But this does not seem to affect the gridded results as I don't see any obvious discontinuity across 0. Am I missing something?
I checked the grid points around, this is not due to bathymetry masking, since the observations with NaN residuals are surrounded by grid points that are not masked.

Divadoxml: change season for thumbnail

Winter is used for the thumbnail URL, but usually it has less data. Would you please change it to Spring or Summer? You can choose based on your experience.

Unable to parse 1991-04-07T12:59:60.000 as Dates.DateTime

Hello,

I'm reading an ODV spreadsheet and get this message:

ArgumentError: unable to parse 1991-04-07T12:59:60.000 as Dates.DateTime

I understand the date 1991-04-07T12:59:60.000 is not totally correct, but should we implement something in DIVAnd to take care of this type of situation, or do the user have to be more restrictive or clear when creating the data files.

Divadoxml: missing email address per originator

Hello,
Even though the email address is no longer displayed in the Sextant HTML, it should exists in the XML.
I quote Erwann:

Concerning the email address, even if we do not display them in the sheet, it is reaaly important to keep these information in the xml because it is a INSPIRE requirement to have at minimum, for each point of contact, an email and an organization.

Wrong WMS link under gmd:MD_BrowseGraphic XML tag using diva2xml

I see an error also in the link inside the test.xml file:
<gmd:graphicOverview xmlns:srv="http://www.isotc211.org/2005/srv">
gmd:MD_BrowseGraphic
gmd:fileName
gco:CharacterStringhttp://ec.oceanbrowser.net/emodnet/Python/web/wms?service=WMS&amp;request=GetMap&amp;version=1.3.0&amp;crs=CRS%3A84&amp;bbox=-42.0%2C24.9%2C-0.099998474%2C48.0&amp;decorated=true&amp;format=image%2Fpng&amp;layers=Atlantic%20Ocean%2FWater_body_chlorophyll-a.4Danl.nc%2AWater_body_chlorophyll-a&amp;styles=vmin%3A2.9031277%2Bvmax%3A7.8576584&amp;transparent=true&amp;height=500&amp;width=800</gco:CharacterString>
</gmd:fileName>
gmd:fileDescription
gco:CharacterStringlarge_thumbnail</gco:CharacterString>
</gmd:fileDescription>
gmd:fileType
gco:CharacterStringpng</gco:CharacterString>
</gmd:fileType>
</gmd:MD_BrowseGraphic>
</gmd:graphicOverview>

partition of the cost function

Is there a function to calculate the costs due to individual constrains, misfit, norm and advection etc. I assume it's buried in s but how to separate these constrains from it?

Incorrect land-sea mask from bathymetry

I'm working on a very small region (Gulf of Trieste) and for the creation of the mask I used the EMODnet Bathymetry product. Based on that bathymetry and using [DIVAnd.load_mask] for the mask creation, I obtained the figure below, on which I also overlay the coastline (using Basemap).

In the northwestern side of the domain I have point seen as sea probably because they are indeed below sea level, seems to be a laguna.

As a solution I was thinking to use the function is_land, available withing Basemap module, which provides the following results:

This looks better but not perfect yet.

Any suggestion (other than editing by hand the bathymetry)?

Failed test

Hello,
Not urgent, but here is one failed test:
curl: (22) The requested URL returned error: 500 Internal Server Error DIVAnd: Error During Test at /root/.julia/dev/DIVAnd/test/runtests.jl:21 Got exception outside of a @test LoadError: failed process: Process(curl -g -L -f -o /tmp/juliahj8cKF https://b2drop.eudat.eu/s/o0vinoQutAC7eb0/download`, ProcessExited(22)) [22]
Stacktrace:
[1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
[2] pipeline_error at ./process.jl:705 [inlined]
[3] #run#504(::Bool, ::Function, ::Cmd) at ./process.jl:663
[4] run at ./process.jl:661 [inlined]
[5] download(::String, ::String) at ./download.jl:27
[6] download(::String) at ./download.jl:45
[7] top-level scope at /root/.julia/dev/DIVAnd/test/test_domain.jl:12
[8] include at ./boot.jl:317 [inlined]
[9] include_relative(::Module, ::String) at ./loading.jl:1044
[10] include(::Module, ::String) at ./sysimg.jl:29
[11] include(::String) at ./client.jl:392
[12] macro expansion at /root/.julia/dev/DIVAnd/test/runtests.jl:84 [inlined]
[13] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
[14] top-level scope at /root/.julia/dev/DIVAnd/test/runtests.jl:22
[15] include at ./boot.jl:317 [inlined]
[16] include_relative(::Module, ::String) at ./loading.jl:1044
[17] include(::Module, ::String) at ./sysimg.jl:29
[18] include(::String) at ./client.jl:392
[19] top-level scope at none:0
[20] eval(::Module, ::Any) at ./boot.jl:319
[21] macro expansion at ./logging.jl:317 [inlined]
[22] exec_options(::Base.JLOptions) at ./client.jl:219
[23] _start() at ./client.jl:425
in expression starting at /root/.julia/dev/DIVAnd/test/test_domain.jl:11
Test Summary: | Pass Error Total
DIVAnd | 240 1 241
`

bathymetry induced extreme values

I'm doing analysis on a 2d lat-depth section. The data is relatively sparse and when adding bottom bathymetry divand introduced some extreme values in some convex regions at the bottom of seafloor where there is actually no data. Smoothing the bathymetry helped but not completed remove these extremes. This doesn't happen if using a flat bottom. Is there a way to control how much influence bathymetry has on the analysis?

optimize correlation length scale for horizontal dimensions

Regarding 2D lat-lon analysis. Two questions:

1. Is it possible to optimize the correlation length scales for both latitude and longitude? I know how to optimize len = (len_lat,len_lon)*scale using bestfactorl given byDIVAnd_cv. But I'd like to have len = (len_lat*scale_lat, len_lon*scale_lon) and optimize both scale_lat and scale_lon.

2. It appears when using DIVAnd_cv the ranges explored are always between 10^-1.5 and 10^1.5 for epsilon2 and 10^-1 and 10^1 for len regardless of nl and ne. So is it possible to expand the search area and what exactly do those two numbers control?

Problem with setting up DIVAnd in Julia 1.0.0 on a linux VM

I've previously made use of the Matlab version of DIVAnd, but after running into memory management issues I've started moving over to Julia. I've installed DIVAnd in Julia 1.0.0 successfully on my Windows 10 laptop to do some test runs, but all my data is on a linux VM (and I expect the full global run will need the 50GB of RAM I've got on teh VM rather than teh 8ish I've got on my laptop).

Anyway, following the install procedure described in the README.me I can successfully pull the source code off github. However, midway through running either using DIVAnd or Pkg.test("DIVAnd") Julia spits out the following error:

ERROR: LoadError: LoadError: UndefVarError: ind2sub not defined
Stacktrace:
 [1] getproperty(::Module, ::Symbol) at ./sysimg.jl:13
 [2] top-level scope at none:0
 [3] include at ./boot.jl:317 [inlined]
 [4] include_relative(::Module, ::String) at ./loading.jl:1038
 [5] include at ./sysimg.jl:29 [inlined]
 [6] include(::String) at /home/ubuntu/.julia/dev/DIVAnd/src/DIVAnd.jl:3
 [7] top-level scope at none:0
 [8] include at ./boot.jl:317 [inlined]
 [9] include_relative(::Module, ::String) at ./loading.jl:1038
 [10] include(::Module, ::String) at ./sysimg.jl:29
 [11] top-level scope at none:2
 [12] eval at ./boot.jl:319 [inlined]
 [13] eval(::Expr) at ./client.jl:389
 [14] top-level scope at ./none:3
in expression starting at /home/ubuntu/.julia/dev/DIVAnd/src/statevector.jl:187
in expression starting at /home/ubuntu/.julia/dev/DIVAnd/src/DIVAnd.jl:27
ERROR: Failed to precompile DIVAnd [efc8151c-67de-5a8f-9a35-d8f54746ae9d] to /home/ubuntu/.julia/compiled/v1.0/DIVAnd/AaVxY.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion at ./logging.jl:313 [inlined]
 [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
 [4] _require(::Base.PkgId) at ./logging.jl:311
 [5] require(::Base.PkgId) at ./loading.jl:852
 [6] macro expansion at ./logging.jl:311 [inlined]
 [7] require(::Module, ::Symbol) at ./loading.jl:834

Now, if I was getting the same error on my windows install I'd just assume it meant some part of the code hadn't been updated now Julia 1.0.0 has dropped support for ind2sub and similar... But it works on my windows install of, supposedly, the same version of Julia, which makes me think there's something else going on.

Note: I installed Julia 1.0.0 to my Windows laptop using the self extracting .exe version, while on the linux VM I compiled Julia 1.0.0 from the source-code.

DIVAnd_residualobs returns residuals different from the "real" ones?

I have a gridded data product. I did DIVAndrun on this data using the same grid as in the original product. However, when I compare the residuals returned by DIVAnd_residualobs and the residuals calculated by differencing the analysis and the original gridded data, I find they are not the same (see the plot. They should be anti-correlated because the signs are opposite. You do see that trend but also a cloud of data, noise?, appears around 0). The residuals are small but clearly not due to rounding errors etc.. In the docs it is said the residuals are calculated by linearly interpolating the gridded analysis to the locations of observations. But since in my case the "data" and "analysis" are at exactly the same locations, I don't see why should these two methods give different results...

Missing `surfextend` documentation

I think we did not describe the option in the doc.
Something like:

• surfextend (false by default): create an additional layer on top of the shallowest layer in order to improve the quality of the results in that layer.

divand Guide

Hi Alex;
Is there any manual (Guide) of the package divand in julia?
Thanks,

fill gaps chlorophyll maps

I want to use the divand package (julia) to reconstruct clouds of chlorophyll maps. I am using the divandrun command but it shows errors.
I'm using a netcdf of 10 images of chlorophyll, I am very lost

Could you help me a little?

SDNObsMetadata returning an empty window

When running the notebook 09-ODV-data-import.ipynb, the line

SDNObsMetadata(obsid[10])

opens a windows but it only contains this text:
"The resource you are looking for has been removed, had its name changed, or is temporarily unavailable."

Maybe related to a change of URL for SDN?

diva2xml: No WMS links are created

Hi,
I've tested with 4 NetCDF files for having Sextant xml, but I see no WMS links.
Would it be possible to verify they have different names per season, maybe by a prefix of season name and ' - ' ?

Thanks,
Menashè

Update emodnet-chemistry.mustache following the attached Sextant xml

The attached example follows the new template.
sextant_final_example.xml.zip
Please note that we always have EMODnet Chemistry as keyword.
Please make sure that the new feature of WMS names (WMSlayername) still functions after modifying the Mustache file.

possible test failure in upcoming Julia version 1.5

A PkgEval run for a PR which changes the generated numbers for randn! indicates that the tests of this package might fail in Julia 1.5 (and on Julia current master). Apologies if this is a false positive.
cf. https://github.com/JuliaCI/NanosoldierReports/blob/7de24e455342298cbef56826b5827f0d7640d2c1/pkgeval/by_hash/b89e35c_vs_098ef24/logs/DIVAnd/1.5.0-DEV-71a4a114c2.log

advection constrain

I read from the paper that the degree of advection constrain can be varied using a parameter but I couldn't find how to specifiy/tune this parameter in the documentation/examples. DIVAndrun accepts a velocity argument but not a parameter for this constrain.

divand_cutter: ERROR: ReadOnlyMemoryError()

Running divand_cutter form divand_climatology_giant.jl produces the following error:

ulia> windowlist,csteps,lmask,alphanormpc = divand.divand_cutter(Lpmnrange,size(mask),moddim,MEMTOFIT)
csteps = [1, 1, 1, 1]
(moddim, MEMTOFIT) = ([0.0, 0.0, 0.0, 0.0], 16)
moddim = [0.0, 0.0, 0.0, 0.0]
MEMTOFIT = 16
problemsize = 196.0
epsilon = 10.142857142857144
stepsize = [134, 134, 2, 12]
((iw1, iw2, isol1, isol2, istore1, istore2), length(windowlist)) = (([1, 1, 1, 1], [162, 162, 6, 12], [1, 1, 1, 1], [148, 148, 4, 12], [1, 1, 1, 1], [148, 148, 4, 12]), 288)
iw = 1
ERROR: ReadOnlyMemoryError()
Stacktrace:
 [1] divand_cutter(::Array{Tuple,1}, ::NTuple{4,Int64}, ::Array{Float64,1}, ::Int64) at /home/ulg/gher/abarth/projects/Julia/divand.jl/src/divand_cutter.jl:187

or crashes Julia completely. It seems that the issue comes from the deepcopy:

windowlist[iw]=deepcopy((iw1,iw2,isol1,isol2,istore1,istore2,))

A simple change to this avoid this issue on the line in questions:

windowlist[iw] = (copy(iw1),copy(iw2),copy(isol1),copy(isol2),copy(istore1),copy(istore2))

But other calls to deepcopy will fail.

Julia 0.6.2 on Linux

Divadoxml: Wrong WMS layers for Observations

An example from the resulted XML file, after importing to Sextant:
http://ec.oceanbrowser.net/emodnet/?server=http://ec.oceanbrowser.net/emodnet/Python/web/wms?SERVICE=WMS&amp;REQUEST=GetCapabilities&amp;VERSION=1.3.0&layers=Baltic%20Sea/Winter%20(December-February)%20-%206-years%20running%20averages/Water_body_silicate.4Danl.nc*obsid

Wrong name and description in gmd:MD_DigitalTransferOptions

Hello,
All three name and description in gmd:MD_DigitalTransferOptions don't follow the emodnet-chemistry.mustache.

Periodic dimension

Hello,
Is it possible to specify periodic dimensions ?

More specifically, I'm thinking of using "Day of the Year" as temporal dimension, ie. to retain seasonal variability but neglect inter annual variability.
To do so we need December 31th (365) to be close a neighbor of Jan 1st (1).

Maybe that is already implemented for longitude ?

Thanks!

Interpolation onto irregular data locations

The ODV docs mention that it can do 3D estimates at arbitrary locations using DIVA gridding results, is there a way to do this using DIVAnd? The problem is quite simple: given a 3D gridded ocean field, estimate the value at certain locations (x,y,z).

ERROR: StackOverflowError:

Hi Alex;
I am running this example file divand_realistic_example.jl And , always, I have the same error.
Similar mistakes I have had running other examples. And I do not have much experience with Julia and I do not know what I can do or if with other versions of julia it works.

Julia Version 0.6.2
Commit d386e40c17 (2017-12-13 18:08 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: AMD Ryzen 5 1600 Six-Core Processor
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Barcelona)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.9.1 (ORCJIT, generic)

Thanks

@time fi = diva(("longitude","latitude","depth","time"),
(lonr,latr,depthr,timer),
(obslon,obslat,obsdepth,obstime),
obsvalue, epsilon2,
(lenx,leny,lenz,lent),
divand.aggregation_monthly;
bathname = bathname
)

WARNING: A::AbstractArray & B::AbstractArray is deprecated, use A .& B instead.

Stacktrace:
[1] depwarn(::String, ::Symbol) at ./deprecated.jl:70
[2] &(::BitArray{1}, ::BitArray{1}) at ./deprecated.jl:57
[3] macro expansion at /home/uganzedo/.julia/v0.6/divand/src/diva.jl:115 [inlined]
[4] (::divand.##728#730{Array{Float64,1},Float64})(::UnitRange{Int64}, ::Int64, ::Int64) at ./distributed/macros.jl:174
[5] run_work_thunk(::Base.Distributed.##158#160{UnitRange{Int64},divand.##728#730{Array{Float64,1},Float64},UnitRange{Int64}}, ::Bool) at ./distributed/process_messages.jl:56
[6] run_work_thunk(::Base.Distributed.RemoteValue, ::Function) at ./distributed/process_messages.jl:65
[7] (::Base.Distributed.##96#98{Base.Distributed.RemoteValue,Base.Distributed.##158#160{UnitRange{Int64},divand.##728#730{Array{Float64,1},Float64},UnitRange{Int64}}})() at ./event.jl:73
while loading no file, in expression starting on line 237
ERROR: StackOverflowError:
collect_to!(::Array{Array{Int64,2},1}, ::Base.Generator{Tuple{Array{Int64,2},Array{Int64,2}},divand.##27#28{DataType}}, ::Int64, ::Int64) at ./array.jl:508
collect(::Base.Generator{Tuple{Array{Int64,2},Array{Int64,2}},divand.##27#28{DataType}}) at ./array.jl:476
promote_array at /home/uganzedo/.julia/v0.6/divand/src/divand.jl:210 [inlined]
len_harmonize(::Tuple{Array{Int64,2},Array{Int64,2}}, ::BitArray{3}) at /home/uganzedo/.julia/v0.6/divand/src/divand.jl:304 (repeats 97 times)
Stacktrace:
[1] wait_ref(::Base.Distributed.RRID, ::Int64) at ./distributed/remotecall.jl:448
[2] call_on_owner(::Function, ::Future, ::Int64, ::Vararg{Int64,N} where N) at ./distributed/remotecall.jl:438
[3] wait(::Future) at ./distributed/remotecall.jl:455
[4] sync_end() at ./task.jl:274
[5] macro expansion at ./task.jl:303 [inlined]
[6] #diva#726(::String, ::String, ::Bool, ::Int64, ::Function, ::NTuple{4,String}, ::Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},Array{Float64,1},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}}, ::Array{Float64,1}, ::Float64, ::Tuple{Int64,Int64,Array{Float64,4},Float64}, ::divand.#aggregation_monthly) at /home/uganzedo/.julia/v0.6/divand/src/diva.jl:112
[7] (::divand.#kw##diva)(::Array{Any,1}, ::divand.#diva, ::NTuple{4,String}, ::Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},Array{Float64,1},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}}, ::Array{Float64,1}, ::Float64, ::Tuple{Int64,Int64,Array{Float64,4},Float64}, ::divand.#aggregation_monthly) at ./:0

DIVAnd.fit issue

Hello,

I am trying to find the optimal correlation length for a 2D layer, i.e. a layer of the global ocean, let's say at 100m. I was doing some tests and used DIVAnd.fit_isotropic taking into account the following inputs: x (the coordinates), v the value, distbin and mincount (I read the doc but distbin is missing in the description of the function). Now, I am trying to use DIVAnd.fit, which takes the same arguments in input but I keep encounting this error:

UndefVarError: Opt not defined

Stacktrace:
[1] #fit#506(::Array{Int64,1}, ::Array{Float64,1}, ::Array{Float64,1}, ::Float64, ::Array{Float64,1}, ::Float64, ::Float64, ::Float64, ::Int64, ::Int64, ::getfield(DIVAnd, Symbol("##508#515")), ::getfield(DIVAnd, Symbol("##509#516")), ::typeof(DIVAnd.fit), ::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Array{Int64,1}, ::Int64) at /home/flo/.julia/packages/DIVAnd/azb67/src/fit.jl:474
[2] fit(::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Array{Int64,1}, ::Int64) at /home/flo/.julia/packages/DIVAnd/azb67/src/fit.jl:422
[3] top-level scope at In[198]:1

I looked at the source code at the line 474 but cannot do anything about it. Do you have any ideas on how to make it work?

Thank you!

mask in diva3d

I was using diva3d to analyze lon-lat-depth data, no time dimension. I created a mask and included it but I got the error

ERROR: expecting a mask of the size (90, 45), but got a mask of the size (90, 45, 33)

In my analysis the size of lon, lat and depth grids are 90, 45 and 33 respectively.
I would assume that this mask should also be 3d, why would it be expecting a 2d mask?

I found out this error was sent by the following codes :

    mask2,pmn,xyi =
        if length(xi) == 4
            DIVAnd.domain(
                bathname,bathisglobal,xi[1],xi[2],xi[3];
                zlevel = zlevel)
        else
            DIVAnd.domain(bathname,bathisglobal,xi[1],xi[2])
        end

    # allow to the user to override the mask
    if mask == nothing
        mask = mask2
    else
        if size(mask) != size(mask2)
            error("expecting a mask of the size $(size(mask2)), " *
                  "but got a mask of the size $(size(mask))")
        end
        # use mask in the following and not mask2
    end

What happens if length(xi) == 4 makes obvious sense since the last dimension is time. What I do not understand is why in

else 
DIVAnd.domain(bathname,bathisglobal,xi[1],xi[2])

only the first two dimensions are used to create the domain even if the domain is 3d?

fit_isotropic arguments

In the doc of that function:

var0,len,distx,covar,fitcovar = fit_isotropic(x,v,distbin,mincount)

the parameters distbin and mincountare not defined or explained, while they may cause differences in the estimated value of the correlation length.

divand test fail

Hi,

I just install Diva 4.7.1 and Julia on a workstation with Windows 8 (64 bit); when I try to test divand:
Pkg.test("divand")

one test did't pass.

See the file attached.
Looking forward for some tips.

Cheers !
George
Julia_Error_Install.txt

Interpolation can fail when using a quite high resolution (compared to NetCDF File)

bathname = "../data/gebco_30sec_8.nc"
#bathname = "../data/gebco_30sec_4.nc"

if !isfile(bathname)
    @info("Download bathymetry")
    download("https://dox.ulg.ac.be/index.php/s/wS6Y8P8NhIF60eG/download", bathname)
    #download("https://dox.ulg.ac.be/index.php/s/RSwm4HPHImdZoQP/download", bathname)
else
    @info("Bathymetry file already downloaded")
end
bathisglobal = true;


dx = dy = 0.005
lonr = 19.45:dx:20.675
latr = 59.5:dy:59.9;

bx,by,b = load_bath(bathname,bathisglobal,lonr,latr)
@show size(b);

BoundsError: attempt to access 11×4 interpolate((19.399999999988694:0.13333333333332575:20.73333333332195,59.533333333340764:0.13333333333333997:59.933333333340784), ::Array{Float64,2}, Gridded(Interpolations.Linear())) with element type Float64 at index [19.45:0.005:20.675, 59.5:0.005:59.9]

Stacktrace:
 [1] throw_boundserror(::Interpolations.GriddedInterpolation{Float64,2,Float64,Interpolations.Gridded{Interpolations.Linear},Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}}, ::Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}) at ./abstractarray.jl:484
 [2] (::Interpolations.GriddedInterpolation{Float64,2,Float64,Interpolations.Gridded{Interpolations.Linear},Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}})(::StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}, ::StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}) at /home/jovyan/.julia/packages/Interpolations/jXoPW/src/gridded/indexing.jl:75
 [3] load_bath(::String, ::Bool, ::StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}, ::StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}) at /home/jovyan/.julia/dev/DIVAnd/src/load_mask.jl:104
 [4] top-level scope at In[27]:1

Keywords should follow the new XML template

Please update emodnet-chemistry.mustache for having P02,P35 and C19 just like in the new template (with geonetwork.thesaurus.external.parameter.NVS).

You may have noticed that now we always have
gmd:keyword
gco:CharacterStringEMODnet Chemistry</gco:CharacterString>
</gmd:keyword>

It's not a mistake.
It is useful for external projects which do harvesting.

Thank you!

Use the same product_id if multiple netCDF files are provided to divadoxml

if multiple netCDF files are provided let's make sure that all netCDF have the same product_id (global netCDF attribute) as the first netCDF file

Divadoxml: empty gmd:URL of WMS and OPeNDAP

Hi,
I get gmd:URL</gmd:URL>

Wrong long name for lat

Found a strange bug (?) in the netCDF output. For whatever reason long name for latitude is replaced with the one for longitude. Here's a part of the ncdump header output:
'variables:

double lon(lon) (no compression);
	lon:units = degrees_east ;
	lon:standard_name = longitude ;
	lon:long_name = longitude ;

double lat(lat) (no compression);
	lat:units = degrees_east ;
	lat:standard_name = longitude ;
	lat:long_name = longitude ;

double depth(depth) (no compression);
	depth:units = meters ;
	depth:positive = down ;
	depth:standard_name = depth ;
	depth:long_name = depth below sea level ;

'

All the other variables are fine. Don't know when it has exactly happened (as I've updated Julia modules for a few times), but files I had produced with the same script on 28 January 2020 had no such problem. I've just updated both DIVAnd (v 2.5.1) and NCDatasets (v 0.10.1) modules but the issue remains.

Upd. I use the diva3d function to produce nc files

divadoxml error LoadError: BoundsError in findbylabel

Hello,
I get the following error upon calling diva2xml (v2.1.1):

Stacktrace:
 [1] findbylabel(::DIVAnd.Vocab.Collection{String}, ::Array{SubString{String},1}) at /home/menashe/.julia/v0.6/DIVAnd/src/Vocab.jl:252
 [2] URLsfromlabels(::String, ::Array{SubString{String},1}) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:331
 [3] #gettemplatevars#814(::String, ::Array{String,1}, ::Bool, ::Function, ::Array{String,1}, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:497
 [4] (::DIVAnd.#kw##gettemplatevars)(::Array{Any,1}, ::DIVAnd.#gettemplatevars, ::Array{String,1}, ::String, ::String, ::String) at ./<missing>:0
 [5] #divadoxml#826(::Bool, ::Dict{String,String}, ::Array{String,1}, ::Function, ::Array{String,1}, ::String, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:733
 [6] (::DIVAnd.#kw##divadoxml)(::Array{Any,1}, ::DIVAnd.#divadoxml, ::Array{String,1}, ::String, ::String, ::String, ::String) at ./<missing>:0
 [7] include_from_node1(::String) at ./loading.jl:576
 [8] include(::String) at ./sysimg.jl:14
 [9] process_options(::Base.JLOptions) at ./client.jl:305
 [10] _start() at ./client.jl:371
while loading /home/menashe/netcdf/Baltic-Sea-Water_body_chlorophyll-a.4Danl/toxml.jl, in expression starting on line 16

toxml.jl:
`using DIVAnd
#download("http://emodnet-chemistry.maris2.nl/download/export.zip","export.zip")
files = [
"Winter (December-February) - 6-years running averages/Water_body_chlorophyll-a.4Danl.nc",
"Spring (March-May) - 6-years running averages/Water_body_chlorophyll-a.4Danl.nc",
"Summer (June-August) - 6-years running averages/Water_body_chlorophyll-a.4Danl.nc",
"Autumn (September-November) - 6-years running averages/Water_body_chlorophyll-a.4Danl.nc"
];

DIVAnd.divadoxml(files,"Water_body_chlorophyll-a","EMODNET-chemistry","/home/menashe/netcdf/export.zip","test.xml";
ignore_errors = false,
additionalvars = Dict(
"title" => "TITLE",
"abstract" => "ABSTRACT",
"product_code" => "Baltic-Water_body_Chlorophyll-a-v2018-ANA",
),
WMSlayername = ["winter","spring","summer","autumn"]
);
`

Clarification needed in documentation about error estimates (cpme, aexerr...)

Okay, I've applied DIVAnd to create a time-varying oxygen atlas (1956-2014, surface to 6000m depth at 0.5deg resolution). But I've run into some problems understanding and interpreting the error fields...

In DIVAnd_cpme and DIVAnd_aexerr the error estimates are described as "the clever poor mans error" and "the almost exact error" respectively, while in DIVAnd_cpme_go the error output is described as "erri: relative error field using the clever poor man's error approach".
So, I want some simple clarification: Do DIVAnd_cpme and DIVAnd_aexerr give the error in units of whichever variable we're gridding or do all three functions return a relative error?

If they are all relative errors there also needs to be some clarity what quantity they're relative to. The now obsolete matlab version specified the error as: "error variance of the analysis field relative to the error variance of the background" (per http://modb.oce.ulg.ac.be/mediawiki/index.php/Divand). However looking at the various SeaData climatologies they appear to treat the relative error as relative to the interpolated value at each grid-point. So, can you clarify if relative errors are relative to the background variance (as per the matlab version, and would explain why the confidence intervals on my oxygen inventory time series look unreasonably small) or the interpolated signal?

Thanks in advance (and sorry for bothering you again).

-Chris Roach

(There's also a question-mark over if these error estimates represent something like the standard deviation or something more like the the standard error or 95% confidence intervals)

(Why aren't I just pulling variances out of s.P? because diag() is horribly slow and in a real-world global case with a non-square grid and bathymetry it's unclear how to map diag(s.P) back to a geographic grid...)

get_originators_from_obsid: BoundsError only with specific netcdf files

Hello,
toxml.jl.zip
I can reproduce this bug with the files mentioned in the attached toxml.ji, but not with the North-East-Atlantic-Ocean-Water_body_chlorophyll-a.4Danl files.
You have the NetCDF files.

$ julia toxml.jl

Info: Loading EDMO information
Info: Lookup obsids
ERROR: LoadError: BoundsError: attempt to access 1-element Array{SubString{String},1} at index [2]
Stacktrace:
[1] #get_originators_from_obsid#801(::Bool, ::Function, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Base.Iterators.Flatten{Array{Array{String,1},1}}) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:278
[2] (::DIVAnd.#kw##get_originators_from_obsid)(::Array{Any,1}, ::DIVAnd.#get_originators_from_obsid, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Base.Iterators.Flatten{Array{Array{String,1},1}}) at ./:0
[3] #getoriginators#800(::Bool, ::Function, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Array{String,1}, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:254
[4] (::DIVAnd.#kw##getoriginators)(::Array{Any,1}, ::DIVAnd.#getoriginators, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Array{String,1}, ::String) at ./:0
[5] #gettemplatevars#804(::String, ::Array{String,1}, ::Bool, ::Function, ::Array{String,1}, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:591
[6] (::DIVAnd.#kw##gettemplatevars)(::Array{Any,1}, ::DIVAnd.#gettemplatevars, ::Array{String,1}, ::String, ::String, ::String) at ./:0
[7] #divadoxml#814(::Bool, ::Dict{String,String}, ::Array{String,1}, ::Function, ::Array{String,1}, ::String, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:724
[8] (::DIVAnd.#kw##divadoxml)(::Array{Any,1}, ::DIVAnd.#divadoxml, ::Array{String,1}, ::String, ::String, ::String, ::String) at ./:0
[9] include_from_node1(::String) at ./loading.jl:576
[10] include(::String) at ./sysimg.jl:14
[11] process_options(::Base.JLOptions) at ./client.jl:305
[12] _start() at ./client.jl:371
while loading /home/menashe/netcdf/North-East-Atlantic-Ocean-Water_body_dissolved_inorganic_nitrogen_DIN.4Danl/toxml.jl, in expression starting on line 9

DIVAnd_qc doesn't work with advection constrain

Some indexing problem with s.yo it would appear, which adds artificial "observations"?

`nomissing` function

In the docstring of nomissing:

a = nomissing(da)

Retun the values of the array da of type Array{Union{T,Missing},N} (potentially containing missing values) as a regular Julia array a of the same element type and checks that no missing values are present.

a = nomissing(da,value)

Retun the values of the array da of type Array{Union{T,Missing},N} as a regular Julia array a by replacing all missing value by value.

1. I'm not sure 'checks that no missing values are present.' is clear. Maybe we should say return an error if the array contains at least one missing value
2. Typo: Retun → Return
3. Typo: arrays contains missing values...
   → array contains missing values...
4. replacing all missing value → values

Also note that the function coalesce is available in Base, we could use it

• remove the missing values (if there are some) or
• turn the array of type Array{Union{T,Missing},N} into a type Array{T,N} if there are no missing value.

the error variance used in divand

The documentation states that

epsilon2: error variance of the observations (normalized by the error variance of the background field). 
epsilon2 can be a scalar (all observations have the same error variance and their errors are decorrelated),
 a vector (all observations can have a different error variance and their errors are decorrelated) or a matrix 
(all observations can have a different error variance and their errors can be correlated). If epsilon2 is a 
scalar, it is thus the inverse of the signal-to-noise ratio.

So what is a reasonable number? And what is the variance of the background field if the background is simply taken as the mean of the observations?
Suppose I have N observations [x1,x2,...,xN] each has an measurement uncertainty (variance) of [e1^2, e2^2,...,eN^2]. The variance of the mean of the observations is s^2 = var([x1,x2,...,xN]), is it reasonable to let epsilon2 = [e1^2, e2^2,...,eN^2]/s^2. I understand the epsilon2 can be optimized but I'd like to know what best a priori setting should be. Thanks!

Divadoxml: local path inside WMS and OPeNDAP URLs

Hello,
WMS and OPeNDAP URLs are wrong since they include the local path, in this example /home/menashe:
Baltic Sea//home/menashe/netcdf/Baltic-Sea-Water_body_silicate/Winter (December-February) - 6-years running averages/Water_body_silicate.4Danl.nc*Water_body_silicate
Also the URL of the thumbnail fails due to the same issue.

elevation_max is -0.0 using diva2xml

Hello,
I see under

<gmd:verticalElement>:
   <gmd:maximumValue>
   <gco:Real>-0.0</gco:Real>
</gmd:maximumValue>

while it should be 0.0

diva3d correlation length fitting using an empty tuple for len

Error message

BoundsError: attempt to access ()
  at index [1]

Stacktrace:
 [1] getindex(::Tuple{}, ::Int64) at ./tuple.jl:24
 [2] (::getfield(DIVAnd, Symbol("#kw##diva3d")))(::NamedTuple{(:fitcorrlen, :bathname, :bathisglobal, :plotres, :ncvarattrib, :ncglobalattrib, :timeorigin),Tuple{Bool,String,Bool,typeof(plotres),OrderedDict{String,String},OrderedDict{String,String},DateTime}}, ::typeof(diva3d), ::Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},Array{Float64,1},TimeSelectorYearListMonthList{Array{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},1},Array{Array{Int64,1},1}}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}}, ::Array{Float64,1}, ::Tuple{}, ::Float64, ::String, ::String) at ./none:0

How to reproduce

I don't have a MWE here but using the notebook 15 of the Diva-Workshop, you can get the error by replacing (lenx, leny, lenz), by (), in the following command:

@time dbinfo = diva3d((lonr,latr,depthr,TS),
              (obslon,obslat,obsdepth,obstime),
              obsval,
              (lenx, leny, lenz),
              epsilon2,
              filename,varname,
              fitcorrlen=true,
              bathname = bathname,
              bathisglobal = bathisglobal,
              plotres = plotres,
              ncvarattrib = ncvarattrib,
              ncglobalattrib = ncglobalattrib,
              timeorigin = timeorigin
       );

Possible solution

Use

sz = (length(lonr),length(latr),length(depthr))
lenx = fill(1.,sz)
leny = fill(1.,sz)
lenz = fill(1.,sz)

Divadoxml: wrong thumbnail link

Hello,
The decoded URL from gmd:MD_BrowseGraphic of the test file you've sent me results in HTTP 502:
http://ec.oceanbrowser.net/emodnet/Python/web/wms:?service=WMS&request=GetMap&version=1.3.0&crs=CRS%3A84&bbox=9.4%2C53.0%2C30.9%2C65.9&decorated=true&format=image%2Fpng&layers=Baltic%20Sea%2FWinter%20%28December-February%29%20-%206-years%20running%20averages%2FWater_body_silicate.4Danl.nc%2AWater_body_silicate_L1&styles=vmin%3A8.960959%2Bvmax%3A314.72522&transparent=true&height=500&width=800

Interpolations upgrade

Interpolations recently released a new, breaking release---see JuliaMath/Interpolations.jl#226 for details.

This package appears to be affected by the bounds-checking change. I recently placed an upper bound for the package manager on your behalf, but if you want to continue following the development of Interpolations you may need to make changes. Sorry for any inconvenience.

Diva2xml: only one WMS and one OPeNDAP URLs

I've created XML for the 4 NetCDF files of Baltic silicate. I would expect 4 OPeNDAP URLs and around 44 WMS URLs, but there is only one of each. The should start from the winter using 'OPeNDAP (winter)' both as name and as description.

Issues with DIVAnd under Julia 1.0.4 and onwards.

Okay, I've been using DIVAnd to create a new global atlas of oxygen concentration. A few months back I migrated form the matlab to julia versions. My script runs fine on Julia 1.0.3 but using Julia v1.0.4, v1.1.0 and v1.1.1 (just tried upgrading in the last few days) it instead crashes with a message MethodError: no method matching setindex_shape_check(::Float64, ::Int64). I've tried this on two different machines (one remote VM and a local compute server) with the same results.

The error (shown below) occurs somewhere in DIVAndrun (currently called as: fij, si = DIVAndrun(maski,(pmi,pni,pti),(xj,yj,tj),(xo,yo,to),resid,(lenx,leny,lent),epsilon; velocity=(ui,vi,0*ui), moddim=mdi) )...

Running several of the tests included in DIVAnd (notably DIVAnd_simple_example_big3D.jl) work fine, which would suggest it is either associated with the moddim input (mostly likely given the error complaining about int64s) or possibly the velocity field.

notably, a number of other packages got broken in the Julia v1.0 to v1.1 update...JuliaLang/julia#30374

ERROR: LoadError: MethodError: no method matching setindex_shape_check(::Float64, ::Int64)
Closest candidates are:
  setindex_shape_check(!Matched::AbstractArray{#s72,1} where #s72, ::Integer) at indices.jl:218
  setindex_shape_check(!Matched::AbstractArray{#s72,1} where #s72, ::Integer, !Matched::Integer) at indices.jl:221
  setindex_shape_check(!Matched::AbstractArray{#s72,2} where #s72, ::Integer, !Matched::Integer) at indices.jl:225
  ...
Stacktrace:
 [1] macro expansion at ./multidimensional.jl:694 [inlined]
 [2] _unsafe_setindex!(::IndexLinear, ::Array{Float64,1}, ::Float64, ::Base.LogicalIndex{Int64,BitArray{1}}) at ./multidimensional.jl:689
 [3] _setindex! at ./multidimensional.jl:684 [inlined]
 [4] setindex! at ./abstractarray.jl:1020 [inlined]
 [5] DIVAnd_obs(::DIVAnd.DIVAnd_struct{Float64,Int64,3,SparseArrays.SparseMatrixCSC{Float64,Int64}}, ::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::LinearAlgebra.Diagonal{Float64,Array{Float64,1}}, ::Array{Float64,2}) at /home/ubuntu/.julia/packages/DIVAnd/2RDHG/src/DIVAnd_obs.jl:44
 [6] #DIVAndrun#265(::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Bool, ::Bool, ::Float64, ::Int64, ::Int64, ::Tuple{}, ::Symbol, ::Array{Int64,1}, ::Array{Float64,2}, ::Array{Any,1}, ::Int64, ::typeof(DIVAnd_pc_none), ::getfield(DIVAnd, Symbol("##267#269")), ::Array{Float64,3}, ::Array{Float64,1}, ::Float64, ::Bool, ::Array{Any,1}, ::Float64, ::Tuple{}, ::Tuple{}, ::Int64, ::Tuple{}, ::Tuple{}, ::typeof(DIVAndrun), ::Type, ::BitArray{3}, ::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Tuple{Float64,Float64,Float64}, ::Float64) at /home/ubuntu/.julia/packages/DIVAnd/2RDHG/src/DIVAndrun.jl:64
 [7] (::getfield(DIVAnd, Symbol("#kw##DIVAndrun")))(::NamedTuple{(:velocity, :moddim),Tuple{Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}},Array{Int64,1}}}, ::typeof(DIVAndrun), ::BitArray{3}, ::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Tuple{Float64,Float64,Float64}, ::Float64) at ./none:0
 [8] top-level scope at ./util.jl:156
 [9] include at ./boot.jl:326 [inlined]
 [10] include_relative(::Module, ::String) at ./loading.jl:1038
 [11] include(::Module, ::String) at ./sysimg.jl:29
 [12] exec_options(::Base.JLOptions) at ./client.jl:267
 [13] _start() at ./client.jl:436

OutOfMemoryError after Loading EDMO information in divadoxml

Hello,
The memory consumption seems problematic. I have input files of total 4.9GB and I get:

Info: Loading EDMO information
ERROR: LoadError: OutOfMemoryError()
Stacktrace:
 [1] convert at ./array.jl:368 [inlined]
 [2] convert at ./array.jl:367 [inlined]
 [3] Array{Union{Char, Missings.Missing},N} where N(::Array{Char,2}) at ./sysimg.jl:77
 [4] getindex(::NCDatasets.CFVariable{Char,Union{Char, Missings.Missing},2}, ::Colon) at /home/menashe/.julia/v0.6/NCDatasets/src/NCDatasets.jl:1097
 [5] loadobsid(::NCDatasets.Dataset, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/load_obs.jl:53
 [6] #Dataset#21(::Array{Any,1}, ::Type{T} where T, ::DIVAnd.##729#730{String}, ::String, ::Vararg{String,N} where N) at /home/menashe/.julia/v0.6/NCDatasets/src/NCDatasets.jl:412
 [7] macro expansion at ./broadcast.jl:155 [inlined]
 [8] macro expansion at ./simdloop.jl:73 [inlined]
 [9] macro expansion at ./broadcast.jl:149 [inlined]
 [10] _broadcast! at ./broadcast.jl:141 [inlined]
 [11] broadcast_t at ./broadcast.jl:270 [inlined]
 [12] broadcast_c at ./broadcast.jl:316 [inlined]
 [13] broadcast at ./broadcast.jl:455 [inlined]
 [14] #getoriginators#810(::Bool, ::Function, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Array{String,1}, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:252
 [15] (::DIVAnd.#kw##getoriginators)(::Array{Any,1}, ::DIVAnd.#getoriginators, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Array{String,1}, ::String) at ./<missing>:0
 [16] #gettemplatevars#814(::String, ::Array{String,1}, ::Bool, ::Function, ::Array{String,1}, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:600
 [17] (::DIVAnd.#kw##gettemplatevars)(::Array{Any,1}, ::DIVAnd.#gettemplatevars, ::Array{String,1}, ::String, ::String, ::String) at ./<missing>:0
 [18] #divadoxml#826(::Bool, ::Dict{String,String}, ::Array{String,1}, ::Function, ::Array{String,1}, ::String, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:733
 [19] (::DIVAnd.#kw##divadoxml)(::Array{Any,1}, ::DIVAnd.#divadoxml, ::Array{String,1}, ::String, ::String, ::String, ::String) at ./<missing>:0
 [20] include_from_node1(::String) at ./loading.jl:576
 [21] include(::String) at ./sysimg.jl:14
 [22] process_options(::Base.JLOptions) at ./client.jl:305
 [23] _start() at ./client.jl:371