GithubHelp home page GithubHelp logo

circuitscape / circuitscape.jl Goto Github PK

View Code? Open in Web Editor NEW
125.0 7.0 35.0 2.98 MB

Algorithms from circuit theory to predict connectivity in heterogeneous landscapes

Home Page: https://circuitscape.org

License: MIT License

Julia 100.00%
landscape-ecology julia-language sparse-linear-systems animal-movement landscape-genetics movement-ecology epidemiology forest-fire climate-change circuit-analysis julia

circuitscape.jl's Introduction

Circuitscape

Documentation Chat Build Status
docs gitter Build Status codecov.io

Circuitscape is an open-source program that uses circuit theory to model connectivity in heterogeneous landscapes. Its most common applications include modeling movement and gene flow of plants and animals, as well as identifying areas important for connectivity conservation.

Circuitscape has now been rewritten in Julia for better performance and scalability. This work is based on the Python implemention in Circuitscape.py.

The New Circuitscape - Modern, Fast and Flexible

The new Circuitscape is built entirely in the Julia language, a new programming language for technical computing. Julia is built from the ground up to be fast. As such, this offers a number of advantages over the previous version, and these are detailed below.

Faster and More Scalable

We benchmarked Circuitscape.jl (v0.1.0) with the Python version (v4.0.5) to obtain the following results. We started up Circuitscape with 16 parallel processes, and used benchmark problems from the standard Circuitscape benchmark suite.

These benchmarks were run on a Linux (Ubuntu) server machine with the following specs:

  • Name: Intel(R) Xeon(R) Silver 4114 CPU
  • Clock Speed: 2.20GHz
  • Number of cores: 20
  • RAM: 384 GB

From the benchmark, we see that the new version is upto 4x faster on 16 processes. However, the best performing bar in the chart is Julia-CHOLMOD, which is a new feature introduced.

New Solver Mode - CHOLMOD

Julia-CHOLMOD is a new solver mode used in the new Circuitscape. It performs a cholesky decomposition on the graph constructed, and performs a batched back substitution to compute the voltages. It plugs into the CHOLMOD library, which is part of the SuiteSparse collection of high performance sparse matrix algorithms.

To use the this new mode, include a line in your Circuitscape INI file:

solver = cholmod

The cholesky decomposition is a direct solver method, unlike the algebraic multigrid method used by default in both the old and the new version. The advantage with this new direct method is that it can be much faster than the iterative solution, within a particular problem size.

Word of caution: The cholesky decomposition is not practical to use beyond a certain problem size because of phenomenon called fill-in, which results in loss of sparsity and large memory consumption.

Parallel, everywhere

The old Circuitscape had limited support for parallelism, which worked on Mac and Linux, but didn't work on Windows.

Julia as a programming language is built from the ground up to be parallel, and as a result the new Circuitscape natively supports parallelism on all three platforms.

Single Precision (Experimental)

The new Circuitscape introduces the ability to run problems in single precision as opposed to the standard double precision.

Single precision usually takes much less memory, but trades off against solution accuracy.

Use this new feature by including a line in your config file:

precision = single

Installation

  1. You will need to install Julia on your system first.

  2. Once you start Julia, install Circuitscape by:

julia> using Pkg
julia> Pkg.add("Circuitscape")

If you want the latest development version, you can additionally do:

julia> Pkg.add(PackageSpec(name="Circuitscape", rev="master"))

Check if all the tests are passing by doing the following:

julia> Pkg.test("Circuitscape")

Usage

The current interface to Circuitscape is through the Julia terminal.

julia> using Circuitscape # loads the package into your environment
julia> compute("path/to/config/file.ini")

Contributing

If you have encounter any issues or would like to ask a question, please file a report here. Contributions in the form of pull requests are also welcome!

Notes on INI files

Circuitscape takes as input INI files, which contain paths to the raster map, sources, grounds, and other inputs, as well as flags for each run. If you're using the GUI the INI file will automatically be generated for you and then fed into Circuitscape. But if you're using the Julia prompt, then you must write one yourself. The easiest way to do this is to copy an INI file from the tests and then modify it depending on your problem.

Citation

A preprint is available here: https://proceedings.juliacon.org/papers/10.21105/jcon.00058. You can also use the following BibTeX entry to cite this package:

@article{Anantharaman2020,
  doi = {10.21105/jcon.00058},
  url = {https://doi.org/10.21105/jcon.00058},
  year = {2020},
  publisher = {The Open Journal},
  volume = {1},
  number = {1},
  pages = {58},
  author = {Ranjan Anantharaman and Kimberly Hall and Viral B. Shah and Alan Edelman},
  title = {Circuitscape in Julia: High Performance Connectivity Modelling to Support Conservation Decisions},
  journal = {Proceedings of the JuliaCon Conferences}
}

circuitscape.jl's People

Contributors

andreasnoack avatar dependabot[bot] avatar github-actions[bot] avatar glaroc avatar juliatagbot avatar ranjanan avatar viralbshah avatar vlandau avatar vlucet avatar yeesian avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

circuitscape.jl's Issues

Variable solve times

I'm working with CS.jl in Windows, and have been calling it from R using JuliaCall. I've been running some small scale tests, and have encountered some seemingly odd behavior. When repeatedly solving for 4 points on a small raster (2500 nodes), the task sometimes solves in less than one second:

[2018-06-14T15:45:48 | info | root]: Precision used: Double [2018-06-14T15:45:48 | info | root]: Reading maps [2018-06-14T15:45:48 | info | root]: Resistance/Conductance map has 2500 nodes [2018-06-14T15:45:48 | info | root]: Solver used: AMG accelerated by CG [2018-06-14T15:45:48 | info | root]: Graph has 2500 nodes, 4 focal points and 1 connected components [2018-06-14T15:45:48 | info | root]: Total number of pair solves = 6 [2018-06-14T15:45:48 | info | root]: Triggering resistance calculation shortcut [2018-06-14T15:45:48 | info | root]: Total number of pair solves has been reduced to 3 [2018-06-14T15:45:48 | info | root]: Time taken to construct preconditioner = 0.002524242 seconds [2018-06-14T15:45:48 | info | root]: Time taken to construct local nodemap = 0.000192155 seconds [2018-06-14T15:45:48 | info | root]: Solving pair 1 of 3 [2018-06-14T15:45:48 | info | root]: Time taken to solve linear system = 0.01385796 seconds [2018-06-14T15:45:48 | info | root]: Solving pair 2 of 3 [2018-06-14T15:45:48 | info | root]: Time taken to solve linear system = 0.011062591 seconds [2018-06-14T15:45:48 | info | root]: Solving pair 3 of 3 [2018-06-14T15:45:48 | info | root]: Time taken to solve linear system = 0.011541268 seconds [2018-06-14T15:45:48 | info | root]: Time taken to complete job = 0.048522817

But sometimes minutes to run:
[2018-06-14T15:45:48 | info | root]: Precision used: Double [2018-06-14T15:45:48 | info | root]: Reading maps [2018-06-14T15:45:48 | info | root]: Resistance/Conductance map has 2500 nodes [2018-06-14T15:45:48 | info | root]: Solver used: AMG accelerated by CG [2018-06-14T15:45:48 | info | root]: Graph has 2500 nodes, 4 focal points and 1 connected components [2018-06-14T15:45:48 | info | root]: Total number of pair solves = 6 [2018-06-14T15:45:48 | info | root]: Triggering resistance calculation shortcut [2018-06-14T15:45:48 | info | root]: Total number of pair solves has been reduced to 3 [2018-06-14T15:45:48 | info | root]: Time taken to construct preconditioner = 0.002756596 seconds [2018-06-14T15:45:48 | info | root]: Time taken to construct local nodemap = 0.000189874 seconds [2018-06-14T15:45:48 | info | root]: Solving pair 1 of 3 [2018-06-14T15:46:34 | info | root]: Time taken to solve linear system = 46.273509966 seconds [2018-06-14T15:46:34 | info | root]: Solving pair 2 of 3 [2018-06-14T15:47:21 | info | root]: Time taken to solve linear system = 46.609233071 seconds [2018-06-14T15:47:21 | info | root]: Solving pair 3 of 3 [2018-06-14T15:48:07 | info | root]: Time taken to solve linear system = 46.15073924 seconds [2018-06-14T15:48:07 | info | root]: Time taken to complete job = 139.047756763

The only thing different between these runs is the resistance values in the surface. I've never encountered this behavior with CIRCUITSCAPE, or with other resistance distance methods (e.g., commuteDistance and costDistance from the R package gdistance). Thoughts?

Pkg.status("Circuitscape")
5.2.1 master

julia v. 0.6.3

Thanks!

Run finishes before all pairs are solved in pairwise mode

I have been trying out the new release of Circuitscape in Julia and am having trouble getting the runs to complete. Perhaps I'm missing something?

My graph has 2,013,771 nodes, 200 focal points and 13 connected components (.ini file pasted below). Everything starts just fine on solving the 19,900 pairs, but each time, it stops when it gets to 199 out of 19,900 pairs. There is no error message, only a completion message after the 199th solve:

Solving pair 199 of 19900
[2018-04-18T16:57:49 | info | root]: Time taken to solve linear system = 10.036304987 seconds
[2018-04-18T16:57:50 | info | root]: Time taken to complete job = 2128.810110815

Thank you for all your work to upgrade Circuitscape!

[Options for advanced mode]
ground_file_is_resistances = True
remove_src_or_gnd = keepall
ground_file = (Browse for a ground point file)
use_unit_currents = False
source_file = (Browse for a current source file)
use_direct_grounds = False

[Mask file]
mask_file = (Browse for a raster mask file)
use_mask = False

[Calculation options]
low_memory_mode = False
parallelize = True
solver = cg+amg
print_timings = 1
preemptive_memory_release = False
print_rusages = False
max_parallel = 3

[Short circuit regions (aka polygons)]
polygon_file = (Browse for a short-circuit region file)
use_polygons = False

[Options for one-to-all and all-to-one modes]
use_variable_source_strengths = False
variable_source_file = (Browse for a source strength file)

[Output options]
set_null_currents_to_nodata = False
set_focal_node_currents_to_zero = False
set_null_voltages_to_nodata = False
compress_grids = False
write_cur_maps = 1
write_volt_maps = False
output_file = D:\LCC_runs\wrentit\wrentit_run2.out
write_cum_cur_map_only = True
log_transform_maps = False
write_max_cur_maps = False

[Version]
version = 4.0.5

[Options for reclassification of habitat data]
reclass_file = (Browse for file with reclassification data)
use_reclass_table = False

[Logging Options]
log_level = INFO
log_file = D:\LCC_runs\wrentit\wrentit_run2.log
profiler_log_file = D:\LCC_runs\wrentit\wrentit_run2_rusages.log
screenprint_log = False

[Options for pairwise and one-to-all and all-to-one modes]
included_pairs_file = (Browse for a file with pairs to include or exclude)
use_included_pairs = False
point_file = D:\LCC_runs\wrentit\CS_inputs\wrentit_200_pts_nad83albers_2.asc

[Connection scheme for raster habitat data]
connect_using_avg_resistances = True
connect_four_neighbors_only = False

[Habitat raster or graph]
habitat_map_is_resistances = True
habitat_file = D:\LCC_runs\wrentit\CS_inputs\chfa_res_270_2.asc

[Circuitscape mode]
data_type = raster
scenario = pairwise

OneToAllTests

  • oneToAllVerify1

  • oneToAllVerify2

  • oneToAllVerify3

  • oneToAllVerify4

  • oneToAllVerify5

  • oneToAllVerify6

  • oneToAllVerify7

  • oneToAllVerify8

  • oneToAllVerify9

  • oneToAllVerify10

  • oneToAllVerify11

  • oneToAllVerify12

Parallelism

Write option to use independent solves in parallel

New Benchmark Chart

Need a new benchmark chart now that I'm starting to add parallel cumulative map support by default.

AllToOneTests

  • allToOneVerify1

  • allToOneVerify2

  • allToOneVerify3

  • allToOneVerify4

  • allToOneVerify5

  • allToOneVerify6

  • allToOneVerify7

  • allToOneVerify8

  • allToOneVerify9

  • allToOneVerify10

  • allToOneVerify11

  • allToOneVerify12

Wrong number of pairs & The code is running forever

Hi,
I tried the Circuitscape.jl yesterday. Got some issues:

  1. The code get the total number of pair solves is 1, but in fact it should be 78.
  2. Solving this pair takes forever. The code ran for the whole night and didn't get anything. I think there is not right.

I tried the same data using GUI. Even though it is super slow, It works well. So I guess my habitat and patch data are fine. Then I was wondering did I do something wrong to run the Circuitscape.jl?

Appreciate any help.

AppVeyor

Probably a good time to add an appveyor badge

Output effective resistance files

Hello,
I am having trouble figuring out the output files. For some reason, some of my pairs have not been written in the 3 column output. For example, the output indicates:

1 2 1
1 3 2
1 4 3
1 5 1.2
1 6 1.4
1 7 -1
etc

The -1 indicates a disconnected pair, but what is happening to the first 3 pairs?

When I load the matrix form of the output I actually do get the values for these first 3 pairs. Is there something different in the 3 column output?

Thanks

Robby

CS.jl disconnecting pairs while CS 4.05 is not

Hi,
It seems like CS.jl is setting some pairs to -1, while CS 4.05 is still computing these pairs. I have a total of 268,278 pairs. CS.jl seems to think that 64,270 of these pairs are disconnected. Any ideas as to why CS.jl is doing this?
Thanks
Robby

cs_and julia_output

Hitting inference issue on int32 support

Pkg.checkout("Circuitscape", "int32")
Pkg.checkout("AMG", "int32")
using Circuitscape

leads to:

julia> using Circuitscape
SYSTEM: show(lasterr) caused an error
MethodError(Core.Inference.convert, (AssertionError, "invalid age range update"), 0x0000000000000ac6)
julia> versioninfo()
Julia Version 0.6.2
Commit d386e40c17 (2017-12-13 18:08 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, broadwell)

All raster pairwise tests

  • sgVerify1

  • sgVerify2

  • sgVerify3

  • sgVerify4

  • sgVerify5

  • sgVerify6

  • sgVerify7

  • sgVerify8

  • sgVerify9

  • sgVerify10

  • sgVerify11

  • sgVerify12

  • sgVerify13

  • sgVerify14

  • sgVerify15

PyAMG solver

First use PyAMG solver for everything so that solver variation is not a problem and you can verify correctness

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.