gher-uliege / divand.jl Goto Github PK

When running the notebook 09-ODV-data-import.ipynb, the line

SDNObsMetadata(obsid[10])

opens a windows but it only contains this text:
"The resource you are looking for has been removed, had its name changed, or is temporarily unavailable."

Maybe related to a change of URL for SDN?

Hi,
I get gmd:URL</gmd:URL>

Hi,

I just install Diva 4.7.1 and Julia on a workstation with Windows 8 (64 bit); when I try to test divand:
Pkg.test("divand")

one test did't pass.

See the file attached.
Looking forward for some tips.

Cheers !
George
Julia_Error_Install.txt

Hello,
Even though the email address is no longer displayed in the Sextant HTML, it should exists in the XML.
I quote Erwann:

Concerning the email address, even if we do not display them in the sheet, it is reaaly important to keep these information in the xml because it is a INSPIRE requirement to have at minimum, for each point of contact, an email and an organization.

Hello,
The decoded URL from gmd:MD_BrowseGraphic of the test file you've sent me results in HTTP 502:
http://ec.oceanbrowser.net/emodnet/Python/web/wms:?service=WMS&request=GetMap&version=1.3.0&crs=CRS%3A84&bbox=9.4%2C53.0%2C30.9%2C65.9&decorated=true&format=image%2Fpng&layers=Baltic%20Sea%2FWinter%20%28December-February%29%20-%206-years%20running%20averages%2FWater_body_silicate.4Danl.nc%2AWater_body_silicate_L1&styles=vmin%3A8.960959%2Bvmax%3A314.72522&transparent=true&height=500&width=800

Okay, I've been using DIVAnd to create a new global atlas of oxygen concentration. A few months back I migrated form the matlab to julia versions. My script runs fine on Julia 1.0.3 but using Julia v1.0.4, v1.1.0 and v1.1.1 (just tried upgrading in the last few days) it instead crashes with a message MethodError: no method matching setindex_shape_check(::Float64, ::Int64). I've tried this on two different machines (one remote VM and a local compute server) with the same results.

The error (shown below) occurs somewhere in DIVAndrun (currently called as: fij, si = DIVAndrun(maski,(pmi,pni,pti),(xj,yj,tj),(xo,yo,to),resid,(lenx,leny,lent),epsilon; velocity=(ui,vi,0*ui), moddim=mdi) )...

Running several of the tests included in DIVAnd (notably DIVAnd_simple_example_big3D.jl) work fine, which would suggest it is either associated with the moddim input (mostly likely given the error complaining about int64s) or possibly the velocity field.

notably, a number of other packages got broken in the Julia v1.0 to v1.1 update...JuliaLang/julia#30374

ERROR: LoadError: MethodError: no method matching setindex_shape_check(::Float64, ::Int64)
Closest candidates are:
  setindex_shape_check(!Matched::AbstractArray{#s72,1} where #s72, ::Integer) at indices.jl:218
  setindex_shape_check(!Matched::AbstractArray{#s72,1} where #s72, ::Integer, !Matched::Integer) at indices.jl:221
  setindex_shape_check(!Matched::AbstractArray{#s72,2} where #s72, ::Integer, !Matched::Integer) at indices.jl:225
  ...
Stacktrace:
 [1] macro expansion at ./multidimensional.jl:694 [inlined]
 [2] _unsafe_setindex!(::IndexLinear, ::Array{Float64,1}, ::Float64, ::Base.LogicalIndex{Int64,BitArray{1}}) at ./multidimensional.jl:689
 [3] _setindex! at ./multidimensional.jl:684 [inlined]
 [4] setindex! at ./abstractarray.jl:1020 [inlined]
 [5] DIVAnd_obs(::DIVAnd.DIVAnd_struct{Float64,Int64,3,SparseArrays.SparseMatrixCSC{Float64,Int64}}, ::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::LinearAlgebra.Diagonal{Float64,Array{Float64,1}}, ::Array{Float64,2}) at /home/ubuntu/.julia/packages/DIVAnd/2RDHG/src/DIVAnd_obs.jl:44
 [6] #DIVAndrun#265(::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Bool, ::Bool, ::Float64, ::Int64, ::Int64, ::Tuple{}, ::Symbol, ::Array{Int64,1}, ::Array{Float64,2}, ::Array{Any,1}, ::Int64, ::typeof(DIVAnd_pc_none), ::getfield(DIVAnd, Symbol("##267#269")), ::Array{Float64,3}, ::Array{Float64,1}, ::Float64, ::Bool, ::Array{Any,1}, ::Float64, ::Tuple{}, ::Tuple{}, ::Int64, ::Tuple{}, ::Tuple{}, ::typeof(DIVAndrun), ::Type, ::BitArray{3}, ::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Tuple{Float64,Float64,Float64}, ::Float64) at /home/ubuntu/.julia/packages/DIVAnd/2RDHG/src/DIVAndrun.jl:64
 [7] (::getfield(DIVAnd, Symbol("#kw##DIVAndrun")))(::NamedTuple{(:velocity, :moddim),Tuple{Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}},Array{Int64,1}}}, ::typeof(DIVAndrun), ::BitArray{3}, ::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Tuple{Array{Float64,3},Array{Float64,3},Array{Float64,3}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Tuple{Float64,Float64,Float64}, ::Float64) at ./none:0
 [8] top-level scope at ./util.jl:156
 [9] include at ./boot.jl:326 [inlined]
 [10] include_relative(::Module, ::String) at ./loading.jl:1038
 [11] include(::Module, ::String) at ./sysimg.jl:29
 [12] exec_options(::Base.JLOptions) at ./client.jl:267
 [13] _start() at ./client.jl:436

I want to use the divand package (julia) to reconstruct clouds of chlorophyll maps. I am using the divandrun command but it shows errors.
I'm using a netcdf of 10 images of chlorophyll, I am very lost

Could you help me a little?

I've created XML for the 4 NetCDF files of Baltic silicate. I would expect 4 OPeNDAP URLs and around 44 WMS URLs, but there is only one of each. The should start from the winter using 'OPeNDAP (winter)' both as name and as description.

I'm doing analysis on a 2d lat-depth section. The data is relatively sparse and when adding bottom bathymetry divand introduced some extreme values in some convex regions at the bottom of seafloor where there is actually no data. Smoothing the bathymetry helped but not completed remove these extremes. This doesn't happen if using a flat bottom. Is there a way to control how much influence bathymetry has on the analysis?

While trying to figure out the issue #63 I updated and rebuilt DIVAnd. After that I'm getting the following message at the end of the analysis:

[ Info: Creating netCDF file
[ Info: Time step 1 / 1
[ Info: scaled correlation length (min,max) in dimension 1: (200000.0, 200000.0)
[ Info: scaled correlation length (min,max) in dimension 2: (200000.0, 200000.0)
[ Info: scaled correlation length (min,max) in dimension 3: (25.0, 25.0)
[ Info: number of windows: 1
┌ Warning: scalefactore has the value NaN
└ @ DIVAnd C:\Users\ax_li.julia\packages\DIVAnd\gsgE\src\DIVAnd_adaptedeps2.jl:72

The resulting nc file has just metadata, the actual output variable is empty. Also tested on a piece of code which worked correct before the update.

if multiple netCDF files are provided let's make sure that all netCDF have the same product_id (global netCDF attribute) as the first netCDF file

I was using diva3d to analyze lon-lat-depth data, no time dimension. I created a mask and included it but I got the error

ERROR: expecting a mask of the size (90, 45), but got a mask of the size (90, 45, 33)

In my analysis the size of lon, lat and depth grids are 90, 45 and 33 respectively.
I would assume that this mask should also be 3d, why would it be expecting a 2d mask?

I found out this error was sent by the following codes :

    mask2,pmn,xyi =
        if length(xi) == 4
            DIVAnd.domain(
                bathname,bathisglobal,xi[1],xi[2],xi[3];
                zlevel = zlevel)
        else
            DIVAnd.domain(bathname,bathisglobal,xi[1],xi[2])
        end

    # allow to the user to override the mask
    if mask == nothing
        mask = mask2
    else
        if size(mask) != size(mask2)
            error("expecting a mask of the size $(size(mask2)), " *
                  "but got a mask of the size $(size(mask))")
        end
        # use mask in the following and not mask2
    end

What happens if length(xi) == 4 makes obvious sense since the last dimension is time. What I do not understand is why in

else 
DIVAnd.domain(bathname,bathisglobal,xi[1],xi[2])

only the first two dimensions are used to create the domain even if the domain is 3d?

Hello,
I get the following error upon calling diva2xml (v2.1.1):

Stacktrace:
 [1] findbylabel(::DIVAnd.Vocab.Collection{String}, ::Array{SubString{String},1}) at /home/menashe/.julia/v0.6/DIVAnd/src/Vocab.jl:252
 [2] URLsfromlabels(::String, ::Array{SubString{String},1}) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:331
 [3] #gettemplatevars#814(::String, ::Array{String,1}, ::Bool, ::Function, ::Array{String,1}, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:497
 [4] (::DIVAnd.#kw##gettemplatevars)(::Array{Any,1}, ::DIVAnd.#gettemplatevars, ::Array{String,1}, ::String, ::String, ::String) at ./<missing>:0
 [5] #divadoxml#826(::Bool, ::Dict{String,String}, ::Array{String,1}, ::Function, ::Array{String,1}, ::String, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:733
 [6] (::DIVAnd.#kw##divadoxml)(::Array{Any,1}, ::DIVAnd.#divadoxml, ::Array{String,1}, ::String, ::String, ::String, ::String) at ./<missing>:0
 [7] include_from_node1(::String) at ./loading.jl:576
 [8] include(::String) at ./sysimg.jl:14
 [9] process_options(::Base.JLOptions) at ./client.jl:305
 [10] _start() at ./client.jl:371
while loading /home/menashe/netcdf/Baltic-Sea-Water_body_chlorophyll-a.4Danl/toxml.jl, in expression starting on line 16

toxml.jl:
`using DIVAnd
#download("http://emodnet-chemistry.maris2.nl/download/export.zip","export.zip")
files = [
"Winter (December-February) - 6-years running averages/Water_body_chlorophyll-a.4Danl.nc",
"Spring (March-May) - 6-years running averages/Water_body_chlorophyll-a.4Danl.nc",
"Summer (June-August) - 6-years running averages/Water_body_chlorophyll-a.4Danl.nc",
"Autumn (September-November) - 6-years running averages/Water_body_chlorophyll-a.4Danl.nc"
];

DIVAnd.divadoxml(files,"Water_body_chlorophyll-a","EMODNET-chemistry","/home/menashe/netcdf/export.zip","test.xml";
ignore_errors = false,
additionalvars = Dict(
"title" => "TITLE",
"abstract" => "ABSTRACT",
"product_code" => "Baltic-Water_body_Chlorophyll-a-v2018-ANA",
),
WMSlayername = ["winter","spring","summer","autumn"]
);
`

Hello,
Is it possible to specify periodic dimensions ?

More specifically, I'm thinking of using "Day of the Year" as temporal dimension, ie. to retain seasonal variability but neglect inter annual variability.
To do so we need December 31th (365) to be close a neighbor of Jan 1st (1).

Maybe that is already implemented for longitude ?

Thanks!

Running divand_cutter form divand_climatology_giant.jl produces the following error:

ulia> windowlist,csteps,lmask,alphanormpc = divand.divand_cutter(Lpmnrange,size(mask),moddim,MEMTOFIT)
csteps = [1, 1, 1, 1]
(moddim, MEMTOFIT) = ([0.0, 0.0, 0.0, 0.0], 16)
moddim = [0.0, 0.0, 0.0, 0.0]
MEMTOFIT = 16
problemsize = 196.0
epsilon = 10.142857142857144
stepsize = [134, 134, 2, 12]
((iw1, iw2, isol1, isol2, istore1, istore2), length(windowlist)) = (([1, 1, 1, 1], [162, 162, 6, 12], [1, 1, 1, 1], [148, 148, 4, 12], [1, 1, 1, 1], [148, 148, 4, 12]), 288)
iw = 1
ERROR: ReadOnlyMemoryError()
Stacktrace:
 [1] divand_cutter(::Array{Tuple,1}, ::NTuple{4,Int64}, ::Array{Float64,1}, ::Int64) at /home/ulg/gher/abarth/projects/Julia/divand.jl/src/divand_cutter.jl:187

or crashes Julia completely. It seems that the issue comes from the deepcopy:

windowlist[iw]=deepcopy((iw1,iw2,isol1,isol2,istore1,istore2,))

A simple change to this avoid this issue on the line in questions:

windowlist[iw] = (copy(iw1),copy(iw2),copy(isol1),copy(isol2),copy(istore1),copy(istore2))

But other calls to deepcopy will fail.

Julia 0.6.2 on Linux

Winter is used for the thumbnail URL, but usually it has less data. Would you please change it to Spring or Summer? You can choose based on your experience.

Hello,
toxml.jl.zip
I can reproduce this bug with the files mentioned in the attached toxml.ji, but not with the North-East-Atlantic-Ocean-Water_body_chlorophyll-a.4Danl files.
You have the NetCDF files.

$ julia toxml.jl

Info: Loading EDMO information
Info: Lookup obsids
ERROR: LoadError: BoundsError: attempt to access 1-element Array{SubString{String},1} at index [2]
Stacktrace:
[1] #get_originators_from_obsid#801(::Bool, ::Function, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Base.Iterators.Flatten{Array{Array{String,1},1}}) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:278
[2] (::DIVAnd.#kw##get_originators_from_obsid)(::Array{Any,1}, ::DIVAnd.#get_originators_from_obsid, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Base.Iterators.Flatten{Array{Array{String,1},1}}) at ./:0
[3] #getoriginators#800(::Bool, ::Function, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Array{String,1}, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:254
[4] (::DIVAnd.#kw##getoriginators)(::Array{Any,1}, ::DIVAnd.#getoriginators, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Array{String,1}, ::String) at ./:0
[5] #gettemplatevars#804(::String, ::Array{String,1}, ::Bool, ::Function, ::Array{String,1}, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:591
[6] (::DIVAnd.#kw##gettemplatevars)(::Array{Any,1}, ::DIVAnd.#gettemplatevars, ::Array{String,1}, ::String, ::String, ::String) at ./:0
[7] #divadoxml#814(::Bool, ::Dict{String,String}, ::Array{String,1}, ::Function, ::Array{String,1}, ::String, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:724
[8] (::DIVAnd.#kw##divadoxml)(::Array{Any,1}, ::DIVAnd.#divadoxml, ::Array{String,1}, ::String, ::String, ::String, ::String) at ./:0
[9] include_from_node1(::String) at ./loading.jl:576
[10] include(::String) at ./sysimg.jl:14
[11] process_options(::Base.JLOptions) at ./client.jl:305
[12] _start() at ./client.jl:371
while loading /home/menashe/netcdf/North-East-Atlantic-Ocean-Water_body_dissolved_inorganic_nitrogen_DIN.4Danl/toxml.jl, in expression starting on line 9

In my lon-lat-depth analysis using DIVAndrun, observations between the last longitude grid 358 and the first longitude grid 2 were masked out and returned NaN residuals even by setting moddim = [360, 0, 0]. But this does not seem to affect the gridded results as I don't see any obvious discontinuity across 0. Am I missing something?
I checked the grid points around, this is not due to bathymetry masking, since the observations with NaN residuals are surrounded by grid points that are not masked.

The ODV docs mention that it can do 3D estimates at arbitrary locations using DIVA gridding results, is there a way to do this using DIVAnd? The problem is quite simple: given a 3D gridded ocean field, estimate the value at certain locations (x,y,z).

Some indexing problem with s.yo it would appear, which adds artificial "observations"?

Please update emodnet-chemistry.mustache for having P02,P35 and C19 just like in the new template (with geonetwork.thesaurus.external.parameter.NVS).

You may have noticed that now we always have
gmd:keyword
gco:CharacterStringEMODnet Chemistry</gco:CharacterString>
</gmd:keyword>

It's not a mistake.
It is useful for external projects which do harvesting.

Thank you!

Hello,
I see under

<gmd:verticalElement>:
   <gmd:maximumValue>
   <gco:Real>-0.0</gco:Real>
</gmd:maximumValue>

while it should be 0.0

The documentation states that

epsilon2: error variance of the observations (normalized by the error variance of the background field). 
epsilon2 can be a scalar (all observations have the same error variance and their errors are decorrelated),
 a vector (all observations can have a different error variance and their errors are decorrelated) or a matrix 
(all observations can have a different error variance and their errors can be correlated). If epsilon2 is a 
scalar, it is thus the inverse of the signal-to-noise ratio.

So what is a reasonable number? And what is the variance of the background field if the background is simply taken as the mean of the observations?
Suppose I have N observations [x1,x2,...,xN] each has an measurement uncertainty (variance) of [e1^2, e2^2,...,eN^2]. The variance of the mean of the observations is s^2 = var([x1,x2,...,xN]), is it reasonable to let epsilon2 = [e1^2, e2^2,...,eN^2]/s^2. I understand the epsilon2 can be optimized but I'd like to know what best a priori setting should be. Thanks!

In the docstring of nomissing:

a = nomissing(da)

Retun the values of the array da of type Array{Union{T,Missing},N} (potentially containing missing values) as a regular Julia array a of the same element type and checks that no missing values are present.

a = nomissing(da,value)

Retun the values of the array da of type Array{Union{T,Missing},N} as a regular Julia array a by replacing all missing value by value.

1. I'm not sure 'checks that no missing values are present.' is clear. Maybe we should say return an error if the array contains at least one missing value
2. Typo: Retun → Return
3. Typo: arrays contains missing values...
   → array contains missing values...
4. replacing all missing value → values

Also note that the function coalesce is available in Base, we could use it

• remove the missing values (if there are some) or
• turn the array of type Array{Union{T,Missing},N} into a type Array{T,N} if there are no missing value.

The attached example follows the new template.
sextant_final_example.xml.zip
Please note that we always have EMODnet Chemistry as keyword.
Please make sure that the new feature of WMS names (WMSlayername) still functions after modifying the Mustache file.

I have a gridded data product. I did DIVAndrun on this data using the same grid as in the original product. However, when I compare the residuals returned by DIVAnd_residualobs and the residuals calculated by differencing the analysis and the original gridded data, I find they are not the same (see the plot. They should be anti-correlated because the signs are opposite. You do see that trend but also a cloud of data, noise?, appears around 0). The residuals are small but clearly not due to rounding errors etc.. In the docs it is said the residuals are calculated by linearly interpolating the gridded analysis to the locations of observations. But since in my case the "data" and "analysis" are at exactly the same locations, I don't see why should these two methods give different results...

I've previously made use of the Matlab version of DIVAnd, but after running into memory management issues I've started moving over to Julia. I've installed DIVAnd in Julia 1.0.0 successfully on my Windows 10 laptop to do some test runs, but all my data is on a linux VM (and I expect the full global run will need the 50GB of RAM I've got on teh VM rather than teh 8ish I've got on my laptop).

Anyway, following the install procedure described in the README.me I can successfully pull the source code off github. However, midway through running either using DIVAnd or Pkg.test("DIVAnd") Julia spits out the following error:

ERROR: LoadError: LoadError: UndefVarError: ind2sub not defined
Stacktrace:
 [1] getproperty(::Module, ::Symbol) at ./sysimg.jl:13
 [2] top-level scope at none:0
 [3] include at ./boot.jl:317 [inlined]
 [4] include_relative(::Module, ::String) at ./loading.jl:1038
 [5] include at ./sysimg.jl:29 [inlined]
 [6] include(::String) at /home/ubuntu/.julia/dev/DIVAnd/src/DIVAnd.jl:3
 [7] top-level scope at none:0
 [8] include at ./boot.jl:317 [inlined]
 [9] include_relative(::Module, ::String) at ./loading.jl:1038
 [10] include(::Module, ::String) at ./sysimg.jl:29
 [11] top-level scope at none:2
 [12] eval at ./boot.jl:319 [inlined]
 [13] eval(::Expr) at ./client.jl:389
 [14] top-level scope at ./none:3
in expression starting at /home/ubuntu/.julia/dev/DIVAnd/src/statevector.jl:187
in expression starting at /home/ubuntu/.julia/dev/DIVAnd/src/DIVAnd.jl:27
ERROR: Failed to precompile DIVAnd [efc8151c-67de-5a8f-9a35-d8f54746ae9d] to /home/ubuntu/.julia/compiled/v1.0/DIVAnd/AaVxY.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion at ./logging.jl:313 [inlined]
 [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
 [4] _require(::Base.PkgId) at ./logging.jl:311
 [5] require(::Base.PkgId) at ./loading.jl:852
 [6] macro expansion at ./logging.jl:311 [inlined]
 [7] require(::Module, ::Symbol) at ./loading.jl:834

Now, if I was getting the same error on my windows install I'd just assume it meant some part of the code hadn't been updated now Julia 1.0.0 has dropped support for ind2sub and similar... But it works on my windows install of, supposedly, the same version of Julia, which makes me think there's something else going on.

Note: I installed Julia 1.0.0 to my Windows laptop using the self extracting .exe version, while on the linux VM I compiled Julia 1.0.0 from the source-code.

Hello,
The memory consumption seems problematic. I have input files of total 4.9GB and I get:

Info: Loading EDMO information
ERROR: LoadError: OutOfMemoryError()
Stacktrace:
 [1] convert at ./array.jl:368 [inlined]
 [2] convert at ./array.jl:367 [inlined]
 [3] Array{Union{Char, Missings.Missing},N} where N(::Array{Char,2}) at ./sysimg.jl:77
 [4] getindex(::NCDatasets.CFVariable{Char,Union{Char, Missings.Missing},2}, ::Colon) at /home/menashe/.julia/v0.6/NCDatasets/src/NCDatasets.jl:1097
 [5] loadobsid(::NCDatasets.Dataset, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/load_obs.jl:53
 [6] #Dataset#21(::Array{Any,1}, ::Type{T} where T, ::DIVAnd.##729#730{String}, ::String, ::Vararg{String,N} where N) at /home/menashe/.julia/v0.6/NCDatasets/src/NCDatasets.jl:412
 [7] macro expansion at ./broadcast.jl:155 [inlined]
 [8] macro expansion at ./simdloop.jl:73 [inlined]
 [9] macro expansion at ./broadcast.jl:149 [inlined]
 [10] _broadcast! at ./broadcast.jl:141 [inlined]
 [11] broadcast_t at ./broadcast.jl:270 [inlined]
 [12] broadcast_c at ./broadcast.jl:316 [inlined]
 [13] broadcast at ./broadcast.jl:455 [inlined]
 [14] #getoriginators#810(::Bool, ::Function, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Array{String,1}, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:252
 [15] (::DIVAnd.#kw##getoriginators)(::Array{Any,1}, ::DIVAnd.#getoriginators, ::Dict{Tuple{Int64,String},Tuple{Bool,Array{Int64,1}}}, ::Array{String,1}, ::String) at ./<missing>:0
 [16] #gettemplatevars#814(::String, ::Array{String,1}, ::Bool, ::Function, ::Array{String,1}, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:600
 [17] (::DIVAnd.#kw##gettemplatevars)(::Array{Any,1}, ::DIVAnd.#gettemplatevars, ::Array{String,1}, ::String, ::String, ::String) at ./<missing>:0
 [18] #divadoxml#826(::Bool, ::Dict{String,String}, ::Array{String,1}, ::Function, ::Array{String,1}, ::String, ::String, ::String, ::String) at /home/menashe/.julia/v0.6/DIVAnd/src/SDNMetadata.jl:733
 [19] (::DIVAnd.#kw##divadoxml)(::Array{Any,1}, ::DIVAnd.#divadoxml, ::Array{String,1}, ::String, ::String, ::String, ::String) at ./<missing>:0
 [20] include_from_node1(::String) at ./loading.jl:576
 [21] include(::String) at ./sysimg.jl:14
 [22] process_options(::Base.JLOptions) at ./client.jl:305
 [23] _start() at ./client.jl:371

Hello,

I am trying to find the optimal correlation length for a 2D layer, i.e. a layer of the global ocean, let's say at 100m. I was doing some tests and used DIVAnd.fit_isotropic taking into account the following inputs: x (the coordinates), v the value, distbin and mincount (I read the doc but distbin is missing in the description of the function). Now, I am trying to use DIVAnd.fit, which takes the same arguments in input but I keep encounting this error:

UndefVarError: Opt not defined

Stacktrace:
[1] #fit#506(::Array{Int64,1}, ::Array{Float64,1}, ::Array{Float64,1}, ::Float64, ::Array{Float64,1}, ::Float64, ::Float64, ::Float64, ::Int64, ::Int64, ::getfield(DIVAnd, Symbol("##508#515")), ::getfield(DIVAnd, Symbol("##509#516")), ::typeof(DIVAnd.fit), ::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Array{Int64,1}, ::Int64) at /home/flo/.julia/packages/DIVAnd/azb67/src/fit.jl:474
[2] fit(::Tuple{Array{Float64,1},Array{Float64,1}}, ::Array{Float64,1}, ::Array{Int64,1}, ::Int64) at /home/flo/.julia/packages/DIVAnd/azb67/src/fit.jl:422
[3] top-level scope at In[198]:1

I looked at the source code at the line 474 but cannot do anything about it. Do you have any ideas on how to make it work?

Thank you!

Hello,
Not urgent, but here is one failed test:
curl: (22) The requested URL returned error: 500 Internal Server Error DIVAnd: Error During Test at /root/.julia/dev/DIVAnd/test/runtests.jl:21 Got exception outside of a @test LoadError: failed process: Process(curl -g -L -f -o /tmp/juliahj8cKF https://b2drop.eudat.eu/s/o0vinoQutAC7eb0/download`, ProcessExited(22)) [22]
Stacktrace:
[1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
[2] pipeline_error at ./process.jl:705 [inlined]
[3] #run#504(::Bool, ::Function, ::Cmd) at ./process.jl:663
[4] run at ./process.jl:661 [inlined]
[5] download(::String, ::String) at ./download.jl:27
[6] download(::String) at ./download.jl:45
[7] top-level scope at /root/.julia/dev/DIVAnd/test/test_domain.jl:12
[8] include at ./boot.jl:317 [inlined]
[9] include_relative(::Module, ::String) at ./loading.jl:1044
[10] include(::Module, ::String) at ./sysimg.jl:29
[11] include(::String) at ./client.jl:392
[12] macro expansion at /root/.julia/dev/DIVAnd/test/runtests.jl:84 [inlined]
[13] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
[14] top-level scope at /root/.julia/dev/DIVAnd/test/runtests.jl:22
[15] include at ./boot.jl:317 [inlined]
[16] include_relative(::Module, ::String) at ./loading.jl:1044
[17] include(::Module, ::String) at ./sysimg.jl:29
[18] include(::String) at ./client.jl:392
[19] top-level scope at none:0
[20] eval(::Module, ::Any) at ./boot.jl:319
[21] macro expansion at ./logging.jl:317 [inlined]
[22] exec_options(::Base.JLOptions) at ./client.jl:219
[23] _start() at ./client.jl:425
in expression starting at /root/.julia/dev/DIVAnd/test/test_domain.jl:11
Test Summary: | Pass Error Total
DIVAnd | 240 1 241
`

Hi Alex;
Is there any manual (Guide) of the package divand in julia?
Thanks,

A PkgEval run for a PR which changes the generated numbers for randn! indicates that the tests of this package might fail in Julia 1.5 (and on Julia current master). Apologies if this is a false positive.
cf. https://github.com/JuliaCI/NanosoldierReports/blob/7de24e455342298cbef56826b5827f0d7640d2c1/pkgeval/by_hash/b89e35c_vs_098ef24/logs/DIVAnd/1.5.0-DEV-71a4a114c2.log

Interpolations recently released a new, breaking release---see JuliaMath/Interpolations.jl#226 for details.

This package appears to be affected by the bounds-checking change. I recently placed an upper bound for the package manager on your behalf, but if you want to continue following the development of Interpolations you may need to make changes. Sorry for any inconvenience.

An example from the resulted XML file, after importing to Sextant:
http://ec.oceanbrowser.net/emodnet/?server=http://ec.oceanbrowser.net/emodnet/Python/web/wms?SERVICE=WMS&amp;REQUEST=GetCapabilities&amp;VERSION=1.3.0&layers=Baltic%20Sea/Winter%20(December-February)%20-%206-years%20running%20averages/Water_body_silicate.4Danl.nc*obsid

I see an error also in the link inside the test.xml file:
<gmd:graphicOverview xmlns:srv="http://www.isotc211.org/2005/srv">
gmd:MD_BrowseGraphic
gmd:fileName
gco:CharacterStringhttp://ec.oceanbrowser.net/emodnet/Python/web/wms?service=WMS&amp;request=GetMap&amp;version=1.3.0&amp;crs=CRS%3A84&amp;bbox=-42.0%2C24.9%2C-0.099998474%2C48.0&amp;decorated=true&amp;format=image%2Fpng&amp;layers=Atlantic%20Ocean%2FWater_body_chlorophyll-a.4Danl.nc%2AWater_body_chlorophyll-a&amp;styles=vmin%3A2.9031277%2Bvmax%3A7.8576584&amp;transparent=true&amp;height=500&amp;width=800</gco:CharacterString>
</gmd:fileName>
gmd:fileDescription
gco:CharacterStringlarge_thumbnail</gco:CharacterString>
</gmd:fileDescription>
gmd:fileType
gco:CharacterStringpng</gco:CharacterString>
</gmd:fileType>
</gmd:MD_BrowseGraphic>
</gmd:graphicOverview>

I think we did not describe the option in the doc.
Something like:

• surfextend (false by default): create an additional layer on top of the shallowest layer in order to improve the quality of the results in that layer.

Error message

BoundsError: attempt to access ()
  at index [1]

Stacktrace:
 [1] getindex(::Tuple{}, ::Int64) at ./tuple.jl:24
 [2] (::getfield(DIVAnd, Symbol("#kw##diva3d")))(::NamedTuple{(:fitcorrlen, :bathname, :bathisglobal, :plotres, :ncvarattrib, :ncglobalattrib, :timeorigin),Tuple{Bool,String,Bool,typeof(plotres),OrderedDict{String,String},OrderedDict{String,String},DateTime}}, ::typeof(diva3d), ::Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},Array{Float64,1},TimeSelectorYearListMonthList{Array{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},1},Array{Array{Int64,1},1}}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}}, ::Array{Float64,1}, ::Tuple{}, ::Float64, ::String, ::String) at ./none:0

How to reproduce

I don't have a MWE here but using the notebook 15 of the Diva-Workshop, you can get the error by replacing (lenx, leny, lenz), by (), in the following command:

@time dbinfo = diva3d((lonr,latr,depthr,TS),
              (obslon,obslat,obsdepth,obstime),
              obsval,
              (lenx, leny, lenz),
              epsilon2,
              filename,varname,
              fitcorrlen=true,
              bathname = bathname,
              bathisglobal = bathisglobal,
              plotres = plotres,
              ncvarattrib = ncvarattrib,
              ncglobalattrib = ncglobalattrib,
              timeorigin = timeorigin
       );

Possible solution

Use

sz = (length(lonr),length(latr),length(depthr))
lenx = fill(1.,sz)
leny = fill(1.,sz)
lenz = fill(1.,sz)

Hi,
I've tested with 4 NetCDF files for having Sextant xml, but I see no WMS links.
Would it be possible to verify they have different names per season, maybe by a prefix of season name and ' - ' ?

Thanks,
Menashè

Okay, I've applied DIVAnd to create a time-varying oxygen atlas (1956-2014, surface to 6000m depth at 0.5deg resolution). But I've run into some problems understanding and interpreting the error fields...

In DIVAnd_cpme and DIVAnd_aexerr the error estimates are described as "the clever poor mans error" and "the almost exact error" respectively, while in DIVAnd_cpme_go the error output is described as "erri: relative error field using the clever poor man's error approach".
So, I want some simple clarification: Do DIVAnd_cpme and DIVAnd_aexerr give the error in units of whichever variable we're gridding or do all three functions return a relative error?

If they are all relative errors there also needs to be some clarity what quantity they're relative to. The now obsolete matlab version specified the error as: "error variance of the analysis field relative to the error variance of the background" (per http://modb.oce.ulg.ac.be/mediawiki/index.php/Divand). However looking at the various SeaData climatologies they appear to treat the relative error as relative to the interpolated value at each grid-point. So, can you clarify if relative errors are relative to the background variance (as per the matlab version, and would explain why the confidence intervals on my oxygen inventory time series look unreasonably small) or the interpolated signal?

Thanks in advance (and sorry for bothering you again).

-Chris Roach

(There's also a question-mark over if these error estimates represent something like the standard deviation or something more like the the standard error or 95% confidence intervals)

(Why aren't I just pulling variances out of s.P? because diag() is horribly slow and in a real-world global case with a non-square grid and bathymetry it's unclear how to map diag(s.P) back to a geographic grid...)

In the doc of that function:

var0,len,distx,covar,fitcovar = fit_isotropic(x,v,distbin,mincount)

the parameters distbin and mincountare not defined or explained, while they may cause differences in the estimated value of the correlation length.

I read from the paper that the degree of advection constrain can be varied using a parameter but I couldn't find how to specifiy/tune this parameter in the documentation/examples. DIVAndrun accepts a velocity argument but not a parameter for this constrain.

Hello,

I'm reading an ODV spreadsheet and get this message:

ArgumentError: unable to parse 1991-04-07T12:59:60.000 as Dates.DateTime

I understand the date 1991-04-07T12:59:60.000 is not totally correct, but should we implement something in DIVAnd to take care of this type of situation, or do the user have to be more restrictive or clear when creating the data files.

Is there a function to calculate the costs due to individual constrains, misfit, norm and advection etc. I assume it's buried in s but how to separate these constrains from it?

Hello,
WMS and OPeNDAP URLs are wrong since they include the local path, in this example /home/menashe:
Baltic Sea//home/menashe/netcdf/Baltic-Sea-Water_body_silicate/Winter (December-February) - 6-years running averages/Water_body_silicate.4Danl.nc*Water_body_silicate
Also the URL of the thumbnail fails due to the same issue.

Hi Alex;
I am running this example file divand_realistic_example.jl And , always, I have the same error.
Similar mistakes I have had running other examples. And I do not have much experience with Julia and I do not know what I can do or if with other versions of julia it works.

Julia Version 0.6.2
Commit d386e40c17 (2017-12-13 18:08 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: AMD Ryzen 5 1600 Six-Core Processor
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Barcelona)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.9.1 (ORCJIT, generic)

Thanks

@time fi = diva(("longitude","latitude","depth","time"),
(lonr,latr,depthr,timer),
(obslon,obslat,obsdepth,obstime),
obsvalue, epsilon2,
(lenx,leny,lenz,lent),
divand.aggregation_monthly;
bathname = bathname
)

WARNING: A::AbstractArray & B::AbstractArray is deprecated, use A .& B instead.

Stacktrace:
[1] depwarn(::String, ::Symbol) at ./deprecated.jl:70
[2] &(::BitArray{1}, ::BitArray{1}) at ./deprecated.jl:57
[3] macro expansion at /home/uganzedo/.julia/v0.6/divand/src/diva.jl:115 [inlined]
[4] (::divand.##728#730{Array{Float64,1},Float64})(::UnitRange{Int64}, ::Int64, ::Int64) at ./distributed/macros.jl:174
[5] run_work_thunk(::Base.Distributed.##158#160{UnitRange{Int64},divand.##728#730{Array{Float64,1},Float64},UnitRange{Int64}}, ::Bool) at ./distributed/process_messages.jl:56
[6] run_work_thunk(::Base.Distributed.RemoteValue, ::Function) at ./distributed/process_messages.jl:65
[7] (::Base.Distributed.##96#98{Base.Distributed.RemoteValue,Base.Distributed.##158#160{UnitRange{Int64},divand.##728#730{Array{Float64,1},Float64},UnitRange{Int64}}})() at ./event.jl:73
while loading no file, in expression starting on line 237
ERROR: StackOverflowError:
collect_to!(::Array{Array{Int64,2},1}, ::Base.Generator{Tuple{Array{Int64,2},Array{Int64,2}},divand.##27#28{DataType}}, ::Int64, ::Int64) at ./array.jl:508
collect(::Base.Generator{Tuple{Array{Int64,2},Array{Int64,2}},divand.##27#28{DataType}}) at ./array.jl:476
promote_array at /home/uganzedo/.julia/v0.6/divand/src/divand.jl:210 [inlined]
len_harmonize(::Tuple{Array{Int64,2},Array{Int64,2}}, ::BitArray{3}) at /home/uganzedo/.julia/v0.6/divand/src/divand.jl:304 (repeats 97 times)
Stacktrace:
[1] wait_ref(::Base.Distributed.RRID, ::Int64) at ./distributed/remotecall.jl:448
[2] call_on_owner(::Function, ::Future, ::Int64, ::Vararg{Int64,N} where N) at ./distributed/remotecall.jl:438
[3] wait(::Future) at ./distributed/remotecall.jl:455
[4] sync_end() at ./task.jl:274
[5] macro expansion at ./task.jl:303 [inlined]
[6] #diva#726(::String, ::String, ::Bool, ::Int64, ::Function, ::NTuple{4,String}, ::Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},Array{Float64,1},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}}, ::Array{Float64,1}, ::Float64, ::Tuple{Int64,Int64,Array{Float64,4},Float64}, ::divand.#aggregation_monthly) at /home/uganzedo/.julia/v0.6/divand/src/diva.jl:112
[7] (::divand.#kw##diva)(::Array{Any,1}, ::divand.#diva, ::NTuple{4,String}, ::Tuple{StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}},Array{Float64,1},StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}, ::Tuple{Array{Float64,1},Array{Float64,1},Array{Float64,1},Array{DateTime,1}}, ::Array{Float64,1}, ::Float64, ::Tuple{Int64,Int64,Array{Float64,4},Float64}, ::divand.#aggregation_monthly) at ./:0

Found a strange bug (?) in the netCDF output. For whatever reason long name for latitude is replaced with the one for longitude. Here's a part of the ncdump header output:
'variables:

double lon(lon) (no compression);
	lon:units = degrees_east ;
	lon:standard_name = longitude ;
	lon:long_name = longitude ;

double lat(lat) (no compression);
	lat:units = degrees_east ;
	lat:standard_name = longitude ;
	lat:long_name = longitude ;

double depth(depth) (no compression);
	depth:units = meters ;
	depth:positive = down ;
	depth:standard_name = depth ;
	depth:long_name = depth below sea level ;

'

All the other variables are fine. Don't know when it has exactly happened (as I've updated Julia modules for a few times), but files I had produced with the same script on 28 January 2020 had no such problem. I've just updated both DIVAnd (v 2.5.1) and NCDatasets (v 0.10.1) modules but the issue remains.

Upd. I use the diva3d function to produce nc files

Regarding 2D lat-lon analysis. Two questions:

1. Is it possible to optimize the correlation length scales for both latitude and longitude? I know how to optimize len = (len_lat,len_lon)*scale using bestfactorl given byDIVAnd_cv. But I'd like to have len = (len_lat*scale_lat, len_lon*scale_lon) and optimize both scale_lat and scale_lon.

2. It appears when using DIVAnd_cv the ranges explored are always between 10^-1.5 and 10^1.5 for epsilon2 and 10^-1 and 10^1 for len regardless of nl and ne. So is it possible to expand the search area and what exactly do those two numbers control?

I'm working on a very small region (Gulf of Trieste) and for the creation of the mask I used the EMODnet Bathymetry product. Based on that bathymetry and using [DIVAnd.load_mask] for the mask creation, I obtained the figure below, on which I also overlay the coastline (using Basemap).

In the northwestern side of the domain I have point seen as sea probably because they are indeed below sea level, seems to be a laguna.

As a solution I was thinking to use the function is_land, available withing Basemap module, which provides the following results:

This looks better but not perfect yet.

Any suggestion (other than editing by hand the bathymetry)?

Hello,
All three name and description in gmd:MD_DigitalTransferOptions don't follow the emodnet-chemistry.mustache.