circuitscape / omniscape.jl

Functions to compute omnidirectional landscape connectivity using circuit theory and the Omniscape algorithm.

Circuitscape v4

This package is now deprecated, and will no longer be maintained. Please use the latest version: Circuitscape.jl

Circuitscape is a free, open-source program which borrows algorithms from electronic circuit theory to predict patterns of movement, gene flow, and genetic differentiation among plant and animal populations in heterogeneous landscapes. Circuit theory complements least-cost path approaches because it considers effects of all possible pathways across a landscape simultaneously. We are developing Circuitscape for Mac OS X, Windows, and Linux.

Please visit the Circuitscape website for more information.

Authors

Brad McRae, Viral B. Shah, and Tanmay Mohapatra

omniscape.jl's Issues

write_raster is writing empty tifs

Cross posted with reproducible code in yeesian/ArchGDAL.jl/issues/123. It is not consistent. Sometimes things work, and it seems dependent of the environment and which package versions are installed.

Add resistance reclass option

Same as old Circuitscape reclass table option

Travis build fails with parallel and conditional connectivity in windows on Julia 1.3 (pass locally)

All works in Julia 1.2. In 1.3, windows builds in Travis fail due to no output. Example here.

Reduce memory in get_ground

No need to allocate a full-sized array then clip. Waste of memory. Can allocate a square of size 1+radius+buffer and assign value to center pixel. Related to #27.

Switch from @info messages to progress bar to monitor the job's progress

Currently, run_omniscape sends prints to the terminal for every moving window solve, e.g.:

[ Info: Solving target n of N
[ Info: Time taken to solve target n: X seconds

It gets a bit ridiculous when you have a ton of moving window solves to do (for example, a problem I'm running has 138,234 solves)

I think it would be preferable (and easy) to instead use a progress bar from ProgressMeter.jl that reports the % completion along with an ETA.

Add variants of Circuitscape's read_raster and write_raster directly to codebase and retool NoData handling

Need to better handle NoData values. Related to #68.

Use missing for NoData pixels, and just manually reassign missings in resistance and source arrays to -9999 prior to Circuitscape run to make it happy.

Add alternative `run_omniscape` method that accepts arrays and a dict of args

This will improve interoperability with other Julia packages, and allow for a pure Julia analysis pipeline.

Considerations:

• Should arrays be georeferenced, and be of a spatial data type? Or just regular arrays with optional separate arguments for geotransform and wkt. Consider using types from either GeoArrays or GeoData?

Buffer adjustment for sources results in incorrect dims when buffer > 0

Hello all - Can you please advise me on this - I am trying to run a test with Omniscape and it starts normally but then crashes with the following error:

signal (11): Segmentation fault
in expression starting at none:0
in expression starting at none:0
in expression starting at none:0
in expression starting at none:0
in expression starting at none:0

I am running the latest version of Omniscape (downloaded today) with Julia 1.4.2 on a Unix machine

The files I am using are here:
removed by user

run6.ini - is config file
run6-output.txt - output of the run up until crash
PAC6510K.tif - is the raster and assrtd files

Any advice greatly appreciated !

J

vet code after all v0.1.0 features have been added

@ranjanan it would be super helpful to go over the Circuitscape components in the code with you to vet its logic and make sure all Circuitscape inputs are correct.

Reduce memory demand in clip function

Replace the existing clip function with something that only works on the necessary subset of the input array instead of copying and calculating distances for the entire thing. Should help to reduce memory consumption detailed in #27 and in this gist.

Test Omniscape with MKL solver in Circuitscape

Mentioned in Circuitscape/Circuitscape.jl/issues/218. Would like to test in Omniscape using the MKL Circuitscape branch.

add option to mask current flow output to match NoData areas in resistance

Will default to false.

Add 32-bit option

Will save memory for particularly large problems. Need to make sure that this works correctly with Circuitscape's 32-bit option. Granted, 32-bit will not improve things much in CS, because the individual solves are so small.

reduce memory consumption

I'm getting out of memory errors using 1.2 Gb resistance surface on one worker, so I think a closer look needs to be taken at memory consumption. This is on a 32GB RAM machine, so this is no good.

One idea off the bat is to calculate inputs to 32-bit precision, but I think there are likely many more gains to be had by tweaking the code itself.

In theory, the Circuitscape problems themselves should be quite small, <<1M pixels, so most of the consumption is probably happening on the Omniscape side of things.

@ranjanan and @ViralBShah I know you're both busy, but just cc'ing you here in case you have some tips or direction on how to go about addressing this.

Last row/column of targets sometimes not solved.

An indexing error sometimes prevents last row/column of targets from being solved and accumulating current

Update Julia dep to version with deadlock patch once released

The deadlock bug exposed in Windows tests on appveyor (see #13) was fixed by julia/pull/34807. Need to update Julia dep for Omniscape once the patch is backported to either Julia 1.3 or (more likely) 1.4

Add `run_omniscape` method that operates on in-memory arrays

It would be valuable for larger, pure-Julia workflows to have a run_omniscape method that takes a dictionary with Omniscape parameters and in memory arrays or other spatial raster data types for resistance, source strength, and other raster inputs.

Vary block size according to "intactness"

Potential idea for down the line. Small block size is more important in areas with fine-scale heterogeneity, and less so in generally contiguous/homogeneous areas. Allowing block size to vary throughout the landscape may help speed up processing times by prioritizing areas for smaller block sizes.

Issues with runs in parallel

We're trying to run things on more than a single core, from an environment, and have been struggling so far. Here is the script, and we start julia with julia --project (but it doesn't matter):

@info "Activating the environment"
import Pkg; Pkg.activate(".")

@info "Loading Distributed"
using Distributed
addprocs(50; exeflags="--project")

@info "Importing environment on workers"
@everywhere begin
        import Pkg
        @info pwd()
        Pkg.activate(".")
        using Omniscape
end

@info "Run"
run_omniscape("test.ini")

Now, the error message we get is

Starting up Omniscape to use 50 processes in parallel
ERROR: LoadError: On worker 2:
ArgumentError: Package Omniscape not found in current path:
- Run `import Pkg; Pkg.add("Omniscape")` to install the Omniscape package.

I'm guessing there is something we are missing -- the error message is the same in 1.0 to 1.3. Is there an obvious thing we are missing? The issue does not appear if we work from the default environment (but we would rather not install packages in it).

CC'ing @DominiqueCaron & @dlecourstessier

Create default values for Omniscape `.ini` options

Similar to how it's done in Circuitscape, add defaults for Omniscape options so user doesn't need an entry for every argument in their .ini file

Suppress output from Circuitscape; but allow @info from Omniscape

Output is too busy. Ideally we would just have two messages per solve (starting, and finished with time to complete). Right now there are up to 6 messages per solve and in parallel it is impossible to tell which "time to complete" message belongs to which Omniscape solve.

get_targets returns block coordinates outside of array dims.

ERROR: TaskFailedException:
BoundsError: attempt to access 182×335 Array{Float64,2} at index [148:188, 148:188]
Stacktrace:
 [1] throw_boundserror(::Array{Float64,2}, ::Tuple{UnitRange{Int64},UnitRange{Int64}}) at ./abstractarray.jl:537
 [2] checkbounds at ./abstractarray.jl:502 [inlined]
 [3] view at ./subarray.jl:163 [inlined]
 [4] maybeview at ./views.jl:124 [inlined]
 [5] dotview at ./broadcast.jl:1138 [inlined]
 [6] get_source(::Array{Float64,2}, ::Dict{String,Int64}, ::Bool, ::Array{Float64,2}, ::Array{Float64,2}, ::Array{Float64,2}, ::Array{Float64,2}, ::String, ::String, ::Float64, ::Float64, ::Float64, ::Float64; x::Int64, y::Int64, strength::Float64) at /root/.julia/packages/Omniscape/IVo2x/src/functions.jl:161
 [7] solve_target!(::Int64, ::Int64, ::Dict{String,Int64}, ::Array{Float64,2}, ::Array{Float64,2}, ::Array{Float64,2}, ::Dict{String,String}, ::Circuitscape.OutputFlags, ::Bool, ::Bool, ::Bool, ::Array{Float64,2}, ::Array{Float64,2}, ::Array{Float64,2}, ::Array{Float64,2}, ::String, ::String, ::Float64, ::Float64, ::Float64, ::Float64, ::Array{Float64,2}, ::Array{Float64,3}, ::Array{Float64,3}) at /root/.julia/packages/Omniscape/IVo2x/src/functions.jl:445
 [8] macro expansion at /root/.julia/packages/Omniscape/IVo2x/src/run_omniscape.jl:270 [inlined]
 [9] (::Omniscape.var"#45#threadsfor_fun#11"{Dict{String,Int64},Bool,Bool,Bool,Array{Float64,2},String,String,Float64,Float64,Float64,Float64,Dict{String,String},Array{Float64,2},Int64,Circuitscape.OutputFlags,Int64,UnitRange{Int64}})(::Bool) at ./threadingconstructs.jl:61
 [10] (::Omniscape.var"#45#threadsfor_fun#11"{Dict{String,Int64},Bool,Bool,Bool,Array{Float64,2},String,String,Float64,Float64,Float64,Float64,Dict{String,String},Array{Float64,2},Int64,Circuitscape.OutputFlags,Int64,UnitRange{Int64}})() at ./threadingconstructs.jl:28
Stacktrace:
 [1] wait(::Task) at ./task.jl:267
 [2] macro expansion at ./threadingconstructs.jl:69 [inlined]
 [3] run_omniscape(::String) at /root/.julia/packages/Omniscape/IVo2x/src/run_omniscape.jl:265
 [4] top-level scope at none:1

Omniscape settings/conditions that gave rise to this:
block_size = 41
radius = 167
buffer = 0
Source array dims: 18280x14496

So y was < 41 pixels from array edge, which shouldn't happen. This appears to be an issue with get_targets returning blocks that are not fully within the source array. The minimum number of rows (or columns) in a returned source should be radius + 1 + (0.5 * (block_size - 1)). In this case that's equal to 167 + 1 + 20 = 188.

add "file not found" errors for raster inputs

When a file path specified in the .ini does not exist, GDAL throws very cryptic error messages. Add a check to see if the file exists before moving on to reading it in with ArchGDAL.

Review for v0.2.0 release

Need to check code, options, docs, etc. to make sure there are no errors, typos, etc. and make sure v0.2.0 is good to go.

Errored pkg.test() and run_omniscape() returns "BoundsError"

Hi CS and OS community,

I'm attempting to run OS (v 0.4.0) through R (v 4.0.1), however it always returns the same error (see Error 1). So, I then ran a Pkg.test in Julia, which returned a testing error (see Error 2). I then installed a dev version, which passed the package test, but when I attempt to run it on my own data (through R or Julia), it throws Error 1 again. I have almost 300 focal raster grids I would like to analyze, but I've uploaded a single example grid & .ini here.

BTW--because I have no idea what I'm doing, I created my .ini file to be proportional to block_size, radius, and buffer parameters values set in https://github.com/Circuitscape/Omniscape.jl/blob/main/test/input/config4.ini.

Error 1

Error: Error happens in Julia.
BoundsError: attempt to access 29935×29935 Array{Float64,2} at index [-31798:349, -31798:268]
Stacktrace:
 [1] throw_boundserror(::Array{Float64,2}, ::Tuple{UnitRange{Int64},UnitRange{Int64}}) at .\abstractarray.jl:541
 [2] checkbounds at .\abstractarray.jl:506 [inlined]
 [3] view at .\subarray.jl:158 [inlined]
 [4] maybeview at .\views.jl:133 [inlined]
 [5] dotview at .\broadcast.jl:1160 [inlined]
 [6] get_source(::Array{Float64,2}, ::Dict{String,Int64}; x::Int64, y::Int64, strength::Float64) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\functions.jl:99
 [7] calc_correction(::Dict{String,Int64}, ::Dict{String,String}, ::Circuitscape.OutputFlags) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\functions.jl:566
 [8] run_omniscape(::String) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\run_omniscape.jl:135
 [9] top-level scope at none:1
 [10] eval(::Module, ::Any) at .\boot.jl:331
 [11] eval_string(::String) at C:\Users\jbaec

Error 2

julia> Pkg.test("Omniscape")
    Testing Omniscape
Status `C:\Users\jbaecher\AppData\Local\Temp\jl_CJ8m5j\Project.toml`
  [2b7a1792] Circuitscape v5.7.0
  [a38d70fc] Omniscape v0.1.4
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [8dfed614] Test
Status `C:\Users\jbaecher\AppData\Local\Temp\jl_CJ8m5j\Manifest.toml`
  [2169fc97] AlgebraicMultigrid v0.3.0
  [c9ce4bd3] ArchGDAL v0.4.1
  [ec485272] ArnoldiMethod v0.0.4
  [fa961155] CEnum v0.4.1
  [2b7a1792] Circuitscape v5.7.0
  [34da2185] Compat v3.14.0
  [aa819f21] CompatHelper v1.13.6
  [9a962f9c] DataAPI v1.3.0
  [9a8bc11e] DataStreams v0.4.2
  [864edb3b] DataStructures v0.18.2
  [e2ba6199] ExprTools v0.1.1
  [8f5d6c58] EzXML v1.1.0
  [add2ef01] GDAL v1.1.4
  [a7073274] GDAL_jll v3.0.4+2
  [d604d12d] GEOS_jll v3.8.1+0
  [92fee26a] GZip v0.5.1
  [68eda718] GeoFormatTypes v0.3.0
  [cf35fbd7] GeoInterface v0.5.4
  [bc5e4493] GitHub v5.1.7
  [cd3eb016] HTTP v0.8.17
  [d25df0c9] Inflate v0.1.2
  [83e8ac13] IniFile v0.5.0
  [42fd0dbc] IterativeSolvers v0.8.4
  [682c06a0] JSON v0.21.0
  [aacddb02] JpegTurbo_jll v2.0.1+2
  [deac9b47] LibCURL_jll v7.71.1+0
  [29816b5a] LibSSH2_jll v1.9.0+3
  [94ce4f54] Libiconv_jll v1.16.0+6
  [89763e89] Libtiff_jll v4.1.0+1
  [093fc24a] LightGraphs v1.3.0
  [d3a379c0] LittleCMS_jll v2.9.0+0
  [1914dd2f] MacroTools v0.5.5
  [739be429] MbedTLS v1.0.2
  [c8ffd9c3] MbedTLS_jll v2.16.6+1
  [e1d29d7a] Missings v0.4.3
  [78c3b35d] Mocking v0.7.1
  [14a3606d] MozillaCACerts_jll v2020.7.22+0
  [a38d70fc] Omniscape v0.1.4
  [643b3616] OpenJpeg_jll v2.3.1+0
  [bac558e1] OrderedCollections v1.3.0
  [58948b4f] PROJ_jll v7.0.1+0
  [69de0a69] Parsers v1.0.10
  [3cdcf5f2] RecipesBase v1.0.2
  [76ed43ae] SQLite_jll v3.32.3+0
  [699a6c99] SimpleTraits v0.9.3
  [47aef6b3] SimpleWeightedGraphs v1.1.1
  [90137ffa] StaticArrays v0.12.4
  [f269a46b] TimeZones v1.3.2
  [ea10d353] WeakRefStrings v0.6.2
  [02c8fc9c] XML2_jll v2.9.10+2
  [83775a58] Zlib_jll v1.2.11+16
  [3161d3a3] Zstd_jll v1.4.5+1
  [b53b4c65] libpng_jll v1.6.37+5
  [8e850ede] nghttp2_jll v1.40.0+2
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [b77e0a4c] InteractiveUtils
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [44cfe95a] Pkg
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
ERROR: LoadError: MethodError: no method matching Circuitscape.RasterMeta(::Int64, ::Int64, ::Float64, ::Float64, ::Float64, ::Float64, ::Int64)
Closest candidates are:
  Circuitscape.RasterMeta(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) at C:\Users\jbaecher\.julia\packages\Circuitscape\XdJRQ\src\io.jl:20
  Circuitscape.RasterMeta(::Int64, ::Int64, ::Float64, ::Float64, ::Float64, ::Float64, ::Array{Float64,1}, ::String) at C:\Users\jbaecher\.julia\packages\Circuitscape\XdJRQ\src\io.jl:20
Stacktrace:
 [1] calculate_current(::Array{Float64,2}, ::Array{Float64,2}, ::Array{Float64,2}, ::Circuitscape.RasterFlags, ::Dict{String,String}) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\functions.jl:196
 [2] calc_correction(::Dict{String,Int64}, ::Dict{String,String}, ::Circuitscape.OutputFlags) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\functions.jl:585
 [3] run_omniscape(::String) at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\src\run_omniscape.jl:135
 [4] top-level scope at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\test\runtests.jl:3
 [5] include(::String) at .\client.jl:457
 [6] top-level scope at none:6
in expression starting at C:\Users\jbaecher\.julia\packages\Omniscape\YXrXN\test\runtests.jl:3
ERROR: Package Omniscape errored during testing
Stacktrace:
 [1] pkgerror(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Types.jl:52
 [2] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1566
 [3] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:328
 [4] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:315
 [5] #test#61 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:67 [inlined]
 [6] test at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:67 [inlined]
 [7] #test#60 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:66 [inlined]
 [8] test at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:66 [inlined]
 [9] test(::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:65
 [10] test(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:65
 [11] top-level scope at REPL[5]:1

Thanks a ton,

Alex.

Error : LinearAlgebra.LAPACKException(3)

Hi all !

I am currently running Omniscape, but I get this error :

ERROR: LinearAlgebra.LAPACKException(3)

Stacktrace: [1] chklapackerror at /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/lapack.jl:38 [inlined] [2] gesdd!(::Char, ::Array{Float64,2}) at /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/lapack.jl:1602 [3] #svd!#66(::Bool, ::Function, ::Array{Float64,2}) at /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/svd.jl:63 [4] #svd! at ./none:0 [inlined] [5] #svd#67 at /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/svd.jl:106 [inlined] [6] #svd at ./none:0 [inlined] [7] #pinv#31(::Float64, ::Float64, ::Function, ::Array{Float64,2}) at /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/dense.jl:1307 [8] pinv at /builddir/build/BUILD/julia/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/dense.jl:1286 [inlined] [9] Type at /home/dlecourstessier/.julia/packages/AlgebraicMultigrid/eCNmE/src/multilevel.jl:57 [inlined] [10] AlgebraicMultigrid.Pinv(::SparseArrays.SparseMatrixCSC{Float64,Int64}) at /home/dlecourstessier/.julia/packages/AlgebraicMultigrid/eCNmE/src/multilevel.jl:59 [11] smoothed_aggregation(::SparseArrays.SparseMatrixCSC{Float64,Int64}, ::Type{Val{1}}, ::AlgebraicMultigrid.HermitianSymmetry, ::AlgebraicMultigrid.SymmetricStrength{Float64}, ::AlgebraicMultigrid.StandardAggregation, ::AlgebraicMultigrid.JacobiProlongation{Float64}, ::AlgebraicMultigrid.GaussSeidel{AlgebraicMultigrid.SymmetricSweep}, ::AlgebraicMultigrid.GaussSeidel{AlgebraicMultigrid.SymmetricSweep}, ::AlgebraicMultigrid.GaussSeidel{AlgebraicMultigrid.SymmetricSweep}, ::Int64, ::Int64, ::Bool, ::Bool, ::Type) at /home/dlecourstessier/.julia/packages/AlgebraicMultigrid/eCNmE/src/aggregation.jl:54 [12] smoothed_aggregation at /home/dlecourstessier/.julia/packages/AlgebraicMultigrid/eCNmE/src/aggregation.jl:16 [inlined] (repeats 2 times) [13] macro expansion at ./util.jl:213 [inlined] [14] multiple_solver(::Dict{String,String}, ::SparseArrays.SparseMatrixCSC{Float64,Int64}, ::Array{Float64,1}, ::Array{Float64,1}, ::Array{Float64,1}) at /home/dlecourstessier/.julia/packages/Circuitscape/mKAhh/src/raster/advanced.jl:281 [15] calculate_current(::Array{Float64,2}, ::Array{Float64,2}, ::Array{Float64,2}, ::Circuitscape.RasterFlags, ::Dict{String,String}) at /home/dlecourstessier/.julia/packages/Omniscape/3uB6x/src/functions.jl:265 [16] solve_target!(::Int64, ::Int64, ::Dict{String,Int64}, ::Array{Float64,2}, ::Array{Float64,2}, ::Array{Float64,2}, ::Dict{String,String}, ::Circuitscape.OutputFlags, ::Bool, ::Bool, ::Array{Float64,2}, ::Array{Float64,2}, ::Array{Float64,2}) at /home/dlecourstessier/.julia/packages/Omniscape/3uB6x/src/functions.jl:342 [17] run_omniscape(::String) at /home/dlecourstessier/.julia/packages/Omniscape/3uB6x/src/run_omniscape.jl:145 [18] top-level scope at none:0

Does anyone have an idea how I can fix the problem ?

Thanks a lot !

NoData handling is not ideal for condition1, condition2, condition1_future, and condition2_future

Currently, Circuitscape.read_raster overwrites NoData in the rasters with -9999. In (possibly rare) cases, the range of possible values for the condition rasters being fed into Omniscape could include -9999, so setting NoData pixels to -9999 in the array representation would mess things up.

I'm not sure the best way to handle this. One less-than-ideal option comes to mind:

• Impose restrictions on the range of acceptable values for the condition layers so that -9999 is far enough away from the minimum value that it won't interfere with source-target matching.

For now, I'll update the docs to reflect this issue, but I will leave this open until I arrive at a better solution.

EDIT: Duh 🤦 I'll using missing.

Multi-threading deadlock on Windows

Need to identify cause. It is only exposed on Windows in Travis and Appveyor with Julia 1.3, doesn't happen in Julia 1.2. @ranjanan or @ViralBShah any advice would be greatly appreciated!

Here's a post on the Julia discourse I made with links to relevant code blocks in Omniscape.jl

Improve tests

Need to update tests to verify computed results against known results, as in Circuitscape's output_verify.

Further investigate impacts of BLAS_NUM_THREADS on performance

When parallel processing is enabled, having too many BLAS threads (number of logical cores) was causing slowdowns (presumably because there was competition between linear algebra tasks and Omniscape tasks), so I set BLAS_NUM_THREADS to 1 if parallel processing is enabled. 1 might be less than ideal, perhaps setting to number of physical cores would be most efficient, see JuliaLang/julia/issues/33409.

Reduce compute time overhead with parallel processing

Send tasks in batches instead of one at a time?

Overhead per solve is non-trivial (maybe on the order of milliseconds?). With tens or hundreds of thousands of solves, it adds up.

@ranjanan any thoughts on this? I know dividing the number of tasks by the number of workers/threads is not necessarily ideal because then there may be idle workers toward the end of processing. Maybe batches of size n? I can get this implemented if we decide on a strategy, but will probably only bother doing it on the threads branch since that'll be merged into master and released once Julia 1.3 is released.

review arg names for .ini

Solicit some feedback about argument names and settle on final before releasing 0.2.0

Add error throws when required arguments are missing from the .ini

Good thing to tell users whenever they haven't specified enough information to make a minimal omniscape problem.

r_cutoff not applied with resistance_file_is_conductance = true

Here, it is clear that r_cutoff is only applied when resistance_file_is_conductance = false. Need to fix so it is still applied when resistance_file_is_conductance = true, per the docs entry for r_cutoff

Add distance decay option for source strengths

For Circuitscape solves in each moving window, provide the user an option to weight/adjust source strengths by distance to central target pixel using one or more different distance decay functions. This was an option in Omniscape for Python.

This needs to be discussed in the context of circuit theory generally, as well as the ecological assumptions that are made when using this option.

Add option for resistance to be conductance

Keeping in line with Circuitscape option

add .tif support

Add support for reading and writing tif formatted files. Should be relatively straightforward to implement the ideas proposed in Circuitscape/Circuitscape.jl/issues/220

Use geodetic distance instead of number of pixels for radius

Allow radius for moving window to be specified in meters instead of number of pixels when inputs are in GCS WGS 84 (decimal degrees).

Will depend on lower corner coordinates of raster as well as cell size, and use the equation to calculate geodetic distance between two pairs of decimal degrees. Should still be able to do this using a comprehension in Julia.

When using a remote server, ERROR: IOError: stream is closed or unusable

I've been trying to get Omniscape.jl to run for several days and I continually get this error: ERROR: IOError: stream is closed or unusable. It usually happens early in the solving process, but not on the same target. More feedback from the terminal is below.

(Area that I'm analyzing is western WA, 60m pixels; rows=8006, columns=7588. The machine has 20 processors, 640 GB memory).
Thanks for any help! Tom

11 seconds
[ Info: 2020-03-19 07:01:49 : Time taken to solve linear system = 2.82562387 seconds 
[ Info: 2020-03-19 07:01:49 : Time taken to solve linear system = 2.246871727 seconds 
[ Info: 2020-03-19 07:01:49 : Time taken to construct preconditioner = 1.172896151 seconds 
[ Info: 2020-03-19 07:01:49 : Time taken to solve linear system = 2.071580641 seconds 

**ERROR: IOError: stream is closed or unusable** 

Stacktrace: 
[1] (::Base.var"#732#734")(::Task) at .\asyncmap.jl:178 
[2] foreach(::Base.var"#732#734", ::Array{Any,1}) at .\abstractarray.jl:1920 
[3] maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::UnitRange{Int64}) at .\asyncmap.jl:178 
[4] wrap_n_exec_twice(::Channel{Any}, ::Array{Any,1}, ::Distributed.var"#208#211"{WorkerPool}, ::Function, ::UnitRange{Int64}) at .\asyncmap.jl:154 
[5] #async_usemap#717(::Function, ::Nothing, ::typeof(Base.async_usemap), ::Distributed.var"#192#194"{Distributed.var"#192#193#195"{WorkerPool,Omniscape.var"#16#19{Dict{String,Int64},Bool,Array{Float64,2},Dict{String,String},Array{Float64,2},Int64,Circuitscape.OutputFlags}}}, ::UnitRange{Int64}) at .\asyncmap.jl:103 
[6] (::Base.var"#kw##async_usemap")(::NamedTuple{(:ntasks, :batch_size),Tuple{Distributed.var"#208#211"{WorkerPool},Nothing}}, ::typeof(Base.async_usemap), ::Function, ::UnitRange{Int64}) at .\none:0 
[7] #asyncmap#716 at .\asyncmap.jl:81 [inlined] 
[8] #asyncmap at .\none:0 [inlined] 
[9] #pmap#207(::Bool, ::Int64, ::Nothing, ::Array{Any,1}, ::Nothing, ::typeof(pmap), ::Function, ::WorkerPool, ::UnitRange{Int64}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\pmap.jl:126 
[10] pmap(::Function, ::WorkerPool, ::UnitRange{Int64}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\pmap.jl:101 
[11] #pmap#217(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(pmap), ::Function, ::UnitRange{Int64}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\pmap.jl:156 
[12] pmap(::Function, ::UnitRange{Int64}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Distributed\src\pmap.jl:156 
[13] run_omniscape(::String) at C:\Users\tmiewald\.julia\packages\Omniscape\YXrXN\src\run_omniscape.jl:146 
[14] top-level scope at REPL[8]:1 

Add a walkthrough example to Omniscape docs

Provide a section in the documentation with a worked example, describing rationale behind decision points surrounding model parameters, etc., to help provide guidance to users.

Add option to derive source strengths from resistance surface

Original code here:
https://github.com/bmcrae/omniscape/blob/master/OmniScape.py#L30-L32
and here:
https://github.com/bmcrae/omniscape/blob/master/OmniScape.py#L37-L38

Add option to pass connect_four_neighbors_only option to Circuitscape

Hi all, thanks for a great package, enjoying using it. Is it possible to pass and argument to circuitscape to limit connectivity to connect to four neighbours only, rather than the default 8?

i.e in Circuitscape you can set connect_four_neighbors_only = True

Cheers,

Pete

fix issue with NaNs in current flow output

Flow potential calculations sometimes result in 0/0 in calculations, resulting in NaNs. Update to set these areas to 0.

Error in fetch for summing current maps from workers

@ranjanan, here's how to reproduce the error I'm getting with @fetchfrom

using Pkg; Pkg.add(PackageSpec("https://github.com/Circuitscape/Omniscape.jl", rev = "8cb24c1")
using Omniscape; Pkg.test("Omniscape")

Incorporate code for correction rasters for block targets

Need to isolate the Omniscape.py code that performed this. The process is described at the end of the technical report, seen in the screenshot below.

Incorporate climate connectivity into Omniscape

As done by McRae et al. previously, e.g. allow current to only originate from pixels that are/will be climatically similar to the target pixel.

In the future, could possibly extend this to be done for categories as well (e.g. a given biome/habitat type can only be connected to similar habitat types)

Add warning for 0s or negative values in resistance/conductance surface

Will help with troubleshooting and prevent weird Circuitscape errors.

Error: Error building `GDAL`:

Just as a heads up for Windows users, a simple problem and a simple fix:

When first running Pkg.add("Omniscape") I was getting the following error:
...

Building GDAL → C:\Users\jonat\.julia\packages\GDAL\cvf6T\deps\build.log
┌ Error: Error building GDAL:
ERROR: LoadError: LoadError: Your platform ("x86_64-w64-mingw32", parsed as "x86_64-w64-mingw32-gcc8-cxx11") is not supported by this package!

As a result Pkg.test("Omniscape") also failed with the recommendation to:

ERROR: LoadError: GDAL not installed properly, run Pkg.build("GDAL"), restart Julia and try again

This recommendation did not fix the issue but shutting down Julia, restarting Julia and manually adding GDAL has resolved the issue:

Pkg.add("GDAL")

Note I did restart Julia on the second occasion using 'Run as administrator'.
Not sure if that made the difference.
Now running the test package completes ok.
I have a full error log if needs.

For ref - my desktop system:
Windows 10/64GB RAM/Intel i7-8700k/NVIDIA GeForce GTX 1080 Ti

Setting parallel threads on Windows x64 PC

I was wondering if setting the number of threads in the terminal is still the way to get Omniscape (v0.3.0) to work in parallel? I am using a Windows x64 PC with Julia v1.4.2. Everything works great, but Omniscape is only using one worker. Using Circuitscape (v5.6.0) in Julia on the same machine does work in parallel.

Prior to opening Julia, the code I used in the command line terminal:
set JULIA_NUM_THREADS=4

Thanks

GDAL memory error

Not apparent that this is a bug. Stashing here for now.

ERROR: GDALError (CE_Failure, code 2):
	memdataset.cpp, 1541: cannot allocate 1x2119895040 bytes

Stacktrace:
 [1] gdaljl_errorhandler(::GDAL.CPLErr, ::Int32, ::Cstring) at /root/.julia/packages/GDAL/cghbO/src/error.jl:36
 [2] gdalcreate(::Ptr{Nothing}, ::String, ::Int64, ::Int64, ::Int64, ::GDAL.GDALDataType, ::Array{String,1}) at /root/.julia/packages/GDAL/cghbO/src/gdal_h.jl:351
 [3] #unsafe_create#30 at /root/.julia/packages/ArchGDAL/FiLcr/src/dataset.jl:200 [inlined]
 [4] create(::Omniscape.var"#9#10"{String,Array{Float64,1},Array{Float64,2},String}, ::String; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{6,Symbol},NamedTuple{(:driver, :width, :height, :nbands, :dtype, :options),Tuple{ArchGDAL.Driver,Int64,Int64,Int64,DataType,Array{String,1}}}}) at /root/.julia/packages/ArchGDAL/FiLcr/src/context.jl:213
 [5] write_raster(::String, ::Array{Float64,2}, ::String, ::Array{Float64,1}, ::String) at /root/.julia/packages/Omniscape/f13kt/src/io.jl:55
 [6] run_omniscape(::String) at /root/.julia/packages/Omniscape/f13kt/src/run_omniscape.jl:364
 [7] top-level scope at none:1
CPLDestroyMutex: Error = 16 (Device or resource busy)