circuitscape / circuitscape.py

Circuitscape uses electronic circuit theory to solve problems in landscape ecology.

Circuitscape v4

This package is now deprecated, and will no longer be maintained. Please use the latest version: Circuitscape.jl

Circuitscape is a free, open-source program which borrows algorithms from electronic circuit theory to predict patterns of movement, gene flow, and genetic differentiation among plant and animal populations in heterogeneous landscapes. Circuit theory complements least-cost path approaches because it considers effects of all possible pathways across a landscape simultaneously. We are developing Circuitscape for Mac OS X, Windows, and Linux.

Please visit the Circuitscape website for more information.

Authors

Brad McRae, Viral B. Shah, and Tanmay Mohapatra

circuitscape.py's Issues

Create a python package for circuitscape.

It would be great to have circuitscape as a separate package, with only the main API module exposed.

The GUI could be a separate package, which depends on the circuitscape base package. This way, the base package won't need the GUI libraries and will be easier to install on clusters, or to be used with ArcGIS, etc.

Add ability to read *.zip files

It would be nice to read *.zip files in addition to *.gz. The former is likely to be more commonly used by our user base.

Write user-friendly node numbers to output files

Currently, node 1 is represented as 1.000000000000000000e+00. If it's possible to have mixed-format output, so that nodes can be integers and other quantities can be floating point, that would be ideal for users. Low priority.

Duplicate test name in verify.py

There were two methods named test_single_ground_all_pairs_resistances_13 in verify.py, one of which I have commented out. The second definition (using sgVerify14.ini) was overriding the prior (using sgVerify13.ini). Results for comparison in baseline_results were for the second definition.

Does this need to be corrected?

Put html documentation on the website

Originally, this issue was: We really should move the documentation from a word file into a markdown file.

Remove all-to-one and one-to-all modes

In the computational code, it may be possible to remove the all-to-one and one-to-all modes. Instead, we can just have the pairwise mode and multiple mode (currently called advanced mode). In the GUI, we can set things up so that multiple mode can be called in all possible combinations of all-to-one, one-to-all, and all-to-all. This will significantly reduce the code complexity and size.

@bmcrae Do you think we can do this?

Need nicer hi-res logo

We need a nicer hi-res logo for use on the site, in the docs, and in the application.

Parallelise pairwise mode when branch currents are computed

In pairwise mode, parallelism can help when branch currents are computed, as every pairwise computation needs to be carried out - as opposed to the mode where only effective resistance is needed, which can be computed with one solve.

Code for solver failure incorrect in some cases

When the solver fails, there should be a -777 entry in the resistance matrix and 3-column resistance files. In one case I ran the value instead was -7.76116397e+02. This can be replicated by running BugTests/Ill-Conditioned/11x11_Mag15_points.txt.gz and 11x11_Mag15_resistances.asc.gz in pairwise mode without creating current or voltage maps.

Save File Dialogs do not work (possibly with wx 2.9)

I just noticed that the save file dialogs are not able to select files and are throwing errors. If it is working elsewhere, it may be specific to PythonCard with wx 2.9.

Add ability to read csv files for network mode

Ideally we would auto-detect and still accommodate tab-delimited inputs, but moving to csv files with .csv extensions as standard for network inputs makes sense.

Is writing compressed current map files faster?

We should experiment to see if writing compressed files (current maps, voltage maps) is faster than writing than uncompressed.

@tanmaykm Also, based on what you were saying about threads and python, perhaps we can use some threading so that the program can move to the next computation while stuff is being written to disk? Can this give some overall speedup even without parallelization?

Remove low-memory mode?

It seems low memory mode can be removed as it does not appear to be helping much.

We can re-enable it / implement differently when specific instances turn up. We could also explore storing amg-hierarchy to disk (if possible).

Write 2- and 3-column resistance, voltage, and current outputs in .csv format

This also seems to make more sense than tab or space-delimited formats.

Add dashes to output when writing to terminal?

In the old version of circuitscape, whenever cs_run was invoked we had it print some extra dashes to the screen using this snippet:

if self.options['screenprint_log'] == True and len(text) > 1 and col == 1: 
            print '    --- ',text,' ---'

The ArcGIS toolboxes look for the dashes when deciding what to print to the ArcGIS dialog during execution. A sample line is:

     ---  Reading maps  ---

Is it possible to add these dashes into the 4.0 code? This is the only remaining compatibility issue I'm aware of.

CSFocalPoints API

Instead of looking at the dimensions of points, perhaps we can pass a flag to figure out if the input is a raster map or a graph.

https://github.com/Circuitscape/Circuitscape/blob/master/src/cs_base.py#L236

Import conflict on Win32

I'm getting close to having a compiled version for win32, but running into a problem that only manifests in the compiled version (the python code runs fine). It seems to result from pyAMG trying to import the io module, which gets confused with our own io.py.

Not sure why this is only happening with the win32 executable. The import conflict can be replicated on both 64- and 32-bit systems by invoking python and importing pyAMG when in thecircuitscape/circuitscape directory.

Allow arbitrary node IDs in network mode

In v3.5, nodes did not need to be sequential. I believe in the current code they need to be ordered from 0 to n-1, where n is the number of nodes. Other numbering schemes, e.g. from 1 to n, cause an index error. We should be able to handle any numbering scheme, as long as nodes are identified by unique integers.

Add advanced mode capability to network mode (arbitrary graph data type)

Would also be nice to implement all-to-one and one-to-all, but overall the pairwise and advanced modes should be priority.

Use threading or other approach to keep GUI from freezing

Right now the long-running solves mean that the GUI freezes up and also that it can get buried under other windows (at least when running on MS Windows). Not sure if threading (e.g. as discussed in link below) can help:
http://www.blog.pythonlibrary.org/2010/05/22/wxpython-and-threads/

This can wait for HTML gui if needed.

We may want to set default screenprint_log = True since messages sent to the console appear more fluid and real-time (and the console window doesn't get buried).

Allow for installation through pip

It would be nice to register circuitscape so that it can be installed with pip install circuitscape.

Experiment with GDAL for IO

GDAL used to be faster than numpy's IO routines at one point. Need to experiment again with GDAL if it is worthwhile to use, or remove the code for calling GDAL otherwise.

Create new log file path when existing path doesn't exist

Right now if I try to load one of the verify problems to the GUI, I'm unable to execute it because it can't find the log file path specified in the config file (e.g., log_file = circuitscape/verify/output/mgNetworkVerify1.log). One solution that could also make manual test runs easier would be to add '../' to the beginning of all paths specified in the verify .ini files. But I assume there is a reason not to do this?

Upload large test problems into a separate repo on github

Once we have ability to read compressed files (#2), we should upload the large test problems into a separate repo.

Command line version should not require wx or PythonCard

This used to work.

julia 05:51:34 {master} ~/Circuitscape$ python bin/csverify.py 
Traceback (most recent call last):
  File "bin/csverify.py", line 5, in <module>
    from circuitscape.verify import cs_verifyall
  File "/home/viral/Circuitscape/circuitscape/__init__.py", line 2, in <module>
    from gui import show_gui
  File "/home/viral/Circuitscape/circuitscape/gui.py", line 19, in <module>
    from PythonCard import dialog, model 
ImportError: No module named PythonCard

Auto detect focal nodes vs focal regions

I think it's OK to remove this from the GUI, since having it there only validates whether the choice matches the input data. I've assigned to myself since the change is really simple.

Make circuitscape API callable

What can we do to make circuitscape API callable from other python code? I can imagine all the inputs being passed in either in-memory, or as filenames, or a dictionary of various things, and similarly, the output being a dictionary of various things.

Exclude time/date/level from default logging

My feeling is most users won't get the distinction between debug/info/warn levels, and also will prefer not to have the time printed for each message (start and end time would be useful however). So I'd favor a default mode where output doesn't include time/date or "INFO/DEBUG" at the beginning of messages, with it turned on only in DEBUG mode (if I'm understanding the purpose of DEBUG mode correctly). That is, the default would look like:

Calling Circuitscape...

Instead of:
12/12/2013 02.04.02.PM:INFO:Calling Circuitscape...

Is there currently a difference between INFO, WARN and ERROR settings? If not we could simplify this choice box.

Parallelize windows code?

Is it possible to implement parallelization in our Windows standalone version? That would be a big leap for the majority of our user base.

Batch mode not working

When batch mode is run with .ini files generated by Circuitscape, the following error is returned:

Traceback (most recent call last):
  File "circuitscape\gui.pyc", line 246, in on_menuFileRunBatch_select
  File "circuitscape\compute.pyc", line 16, in __init__
  File "circuitscape\compute_base.pyc", line 24, in __init__
  File "circuitscape\compute_base.pyc", line 99, in _setup_loggers
  File "circuitscape\compute_base.pyc", line 70, in _create_logger
AttributeError: 'GUI' object has no attribute 'setLevel'

Advanced network mode fails with >1 component

Mario Hofmann's problem appears to result from having multiple components in advanced network mode. When the Mario's input files are run, the following traceback is generated. Brad will commit a verify routine with multiple components.

Graph has 8 nodes and 2 components.
Traceback (most recent call last):
  File "C:\Users\bmcrae\Documents\GitHub\Circuitscape\circuitscape\gui.py", line 527, in on_calcButton_mouseClick
    _voltages, solver_failed = cs.compute()
  File "C:\Users\bmcrae\Documents\GitHub\Circuitscape\circuitscape\profiler.py", line 144, in wrapper
    return func(*args)
  File "C:\Users\bmcrae\Documents\GitHub\Circuitscape\circuitscape\compute.py", line 27, in compute
    result, solver_failed = self.compute_network() # Call module for solving arbitrary graphs (not raster grids)
  File "C:\Users\bmcrae\Documents\GitHub\Circuitscape\circuitscape\profiler.py", line 144, in wrapper
    return func(*args)
  File "C:\Users\bmcrae\Documents\GitHub\Circuitscape\circuitscape\compute.py", line 65, in compute_network
    voltages, current_map, solver_failed = self.advanced_module(g_habitat, out, self.state.source_map, self.state.ground_map)
  File "C:\Users\bmcrae\Documents\GitHub\Circuitscape\circuitscape\profiler.py", line 144, in wrapper
    return func(*args)
  File "C:\Users\bmcrae\Documents\GitHub\Circuitscape\circuitscape\compute.py", line 750, in advanced_module
    out.write_c_map(vc_map_id, node_map=node_map)
  File "C:\Users\bmcrae\Documents\GitHub\Circuitscape\circuitscape\compute_base.py", line 767, in write_c_map
    self._write_store_c_map(name, remove, True, False, voltages, G, node_map, finitegrounds, local_src, local_dst)
  File "C:\Users\bmcrae\Documents\GitHub\Circuitscape\circuitscape\compute_base.py", line 779, in _write_store_c_map
    node_currents_array = Output._append_names_to_node_currents(node_currents, node_map)
  File "C:\Users\bmcrae\Documents\GitHub\Circuitscape\circuitscape\compute_base.py", line 1013, in _append_names_to_node_currents
    output_node_currents[:,0] = node_names[:] # Fixme: fails in network mode when have multiple components
ValueError: could not broadcast input array from shape (2) into shape (8)

One-to-all and all-to-one modes fail when only two focal points exist

This can be replicated with 250k problem in BigTests repo. Has to do with current mapping, and is a bug that has carried over from version 3.5.

Add pythonpath programmatically?

So, one of the hassles of installing on multiple machines (especially Mac, for me, since I'm not very familiar with the file system and hidden files) is setting the pythonpath so that it includes the circuitscape directory. Wouldn't it be possible to just do this in csgui.py at runtime?

Add ability to access network mode from GUI

Right now this can be accomplished by entering "NETWORK" as the polygon file name. Will require re-naming some options (e.g. "Raster habitat map" can become "Raster habitat map or network/graph').

Circuitscape on the cloud

@tanmaykm and I just discussed Circuitscape on the cloud. We are going to try and see what it takes.

Bug in io._guess_file_type (Low priority)

hdr = f.read(10) was causing an error in io._guess_file_type when run after point, ground, or source file name was changed in gui. Code would only execute correctly after cs was restarted. Added try/except code as temporary patch.

Resize images in user guide pdf?

We can easily create narrower margins with pandoc (e.g. use command:

pandoc -V geometry:margin=1in --smart -o circuitscape_4_0.pdf t1.tex

but I'm not sure how to preserve the image sizes. It would be nice to do this, since we get lots of page breaks as currently implemented. I suppose we could redo images and add white space on either side if needed.

Try superlu with incomplete cholesky preconditioning

One possibility to improve memory utilization is to use superlu with incomplete cholesky preconditioning. Scipy seems to have incomplete-LU from superlu, which could allow for a quick trial.

If this works, it could potentially use lesser memory. Trying it casually did not work, and some more effort is required to figure out what's going on.

auto-detect rasters with .txt or .asc extension, and compressed files

Code currently doesn't allow rasters with .txt extension, only .asc. It uses extensions to distinguish between text lists and ascii rastsers. Auto-detecting the difference based on file contents would be better, because .txt is the default extension for rasters exported from ArcGIS.

Ill-conditioned problems

@bmcrae We need to have some ill-conditioned test cases - where the ratio between the largest and smallest values is 1000 or higher. It would be great to have a 1m-illcond and 6m-illcond test case to see how the solvers behave. These should represent the worst case problems people can expect to give to circuitscape and still expect a solution.

Improve include/exclude pairs input file format

Enhancement: Allow option for friendlier list of pairs rather than current matrix format. Code should auto-detect which format is used.

Avoid printing stuff to the screen

We should have all the screen output come to the screen if running from command line, but into the GUI in a console window if running through the GUI.

Delete TODO.md

From the TODO.md file, we should salvage anything that is still applicable and make them into issues. Then, get rid of the file.

Get cs_logo.ico to show up in executable

I just committed new setup scripts (setup-win.py and iss files). All seems to work except cs_logo.ico image doesn't show up.

Send error info to log window

Tracebacks and verification errors are currently written to the terminal window. Is it possible to write these to the log window as well?

GUI redesign?

My original thought was to continue with PythonCard as the GUI, but seeing how it is increasingly difficult to install in the standard ways, I am reconsidering the plan.

@tanmaykm What would be a good way to do the GUI? The server option is nice as it allows the use of a browser, and potentially even deploying the server on the cloud. Do you think we can we explore this possibility with a simple use case?

Remove gc calls, profile memory usage

It is not clear whether the gc calls help in memory utilization. We should remove them altogether, or clearly understand in what situations they are useful.

What we ideally want is a memory profile of the application as it runs, which shows how memory consumption changes over time.

Improve current map performance

This is what I get on the 6m problem. Even though much of the time is spent in solvers, current map creation is currently taking as much time as the solve. It would be good to get to the bottom of where it spends its time. Note that I am using an SSD.

Virals-MacBook-Pro 12:43:49 {master} ~/CIRCUITSCAPE/BigTests/6m$ time python ../../circuitscape/src/csrun.py 6m-1solve.ini 
11/13/2013 12.43.53.PM:INFO:        Reading maps
11/13/2013 12.43.56.PM:INFO:        Processing maps
         3 sec:     load_maps
11/13/2013 12.43.56.PM:DEBUG:Resistance/conductance map has 5250648 nodes
         0 sec:       _construct_node_map
        12 sec:         _construct_g_graph
        14 sec:       _construct_component_map
11/13/2013 12.44.11.PM:DEBUG:          Graph has 5250648 nodes, 2 focal points and 7 components.
         0 sec:         _construct_node_map
        12 sec:         _construct_g_graph
         1 sec:         laplacian
11/13/2013 12.44.26.PM:INFO:          solving focal pair 1 of 1.
multilevel_solver
Number of Levels:     4
Operator Complexity:  1.145
Grid Complexity:      1.118
Coarse Solver:        'splu'
  level   unknowns     nonzeros
    0      5208900     46633606 [87.35%]
    1       564938      6028046 [11.29%]
    2        48399       677925 [ 1.27%]
    3         3285        46111 [ 0.09%]

        25 sec:         create_amg_hierarchy
        21 sec:           solve_linear_system
        21 sec:         single_ground_solver
         0 sec:         _create_voltage_map
        18 sec:         _create_current_maps
        86 sec:       single_ground_all_pair_resistances
       104 sec:     pairwise_module
11/13/2013 12.45.40.PM:DEBUG:       Job took 1 minutes 47 seconds to complete.
       107 sec:   compute_raster
       107 sec: compute
(array([[   0.        ,    1.        ,    2.        ],
       [   1.        ,    0.        ,  104.05655076],
       [   2.        ,  104.05655076,    0.        ]]), False)

real    1m47.936s
user    1m30.988s
sys 0m17.085s

Tests do not work

How do I make the tests work from the toplevel?

Virals-MacBook-Pro 05:05:57 {master} ~/repos/circuitscape$ make test
python bin/csverify.py
======================================================================
ERROR: test_network_multiple_ground_1 (circuitscape.verify.verify_routines.cs_verify)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/viral/repos/circuitscape/circuitscape/verify/verify_routines.py", line 152, in test_network_multiple_ground_1
    test_network_mg(self, 'mgNetworkVerify1')
  File "/Users/viral/repos/circuitscape/circuitscape/verify/verify_routines.py", line 94, in test_network_mg
    _voltages = cs.compute()
  File "/Users/viral/repos/circuitscape/circuitscape/profiler.py", line 127, in wrapper
    return func(*args)
  File "/Users/viral/repos/circuitscape/circuitscape/compute.py", line 34, in compute
    result, solver_failed = self.compute_network() # Call module for solving arbitrary graphs (not raster grids)
  File "/Users/viral/repos/circuitscape/circuitscape/profiler.py", line 127, in wrapper
    return func(*args)
  File "/Users/viral/repos/circuitscape/circuitscape/compute.py", line 45, in compute_network
    focal_nodes = self.read_focal_nodes(self.options.focal_node_file)
  File "/Users/viral/repos/circuitscape/circuitscape/compute.py", line 121, in read_focal_nodes
    focal_nodes = CSIO.load_graph(filename)
  File "/Users/viral/repos/circuitscape/circuitscape/io.py", line 174, in load_graph
    CSIO._check_file_exists(filename)
  File "/Users/viral/repos/circuitscape/circuitscape/io.py", line 24, in _check_file_exists
    raise RuntimeError('File "'  + filename + '" does not exist')
RuntimeError: File "(Browse for file with locations of focal points or areas)" does not exist

----------------------------------------------------------------------
Ran 47 tests in 2.373s

FAILED (errors=1)
Virals-MacBook-Pro 05:06:01 {master} ~/repos/circuitscape$ git reset --hard
HEAD is now at 1a381c7 Update circuitscape_4_0.md
Virals-MacBook-Pro 05:06:04 {master} ~/repos/circuitscape$ make test
python bin/csverify.py
======================================================================
ERROR: test_network_multiple_ground_1 (circuitscape.verify.verify_routines.cs_verify)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/viral/repos/circuitscape/circuitscape/verify/verify_routines.py", line 152, in test_network_multiple_ground_1
    test_network_mg(self, 'mgNetworkVerify1')
  File "/Users/viral/repos/circuitscape/circuitscape/verify/verify_routines.py", line 94, in test_network_mg
    _voltages = cs.compute()
  File "/Users/viral/repos/circuitscape/circuitscape/profiler.py", line 127, in wrapper
    return func(*args)
  File "/Users/viral/repos/circuitscape/circuitscape/compute.py", line 34, in compute
    result, solver_failed = self.compute_network() # Call module for solving arbitrary graphs (not raster grids)
  File "/Users/viral/repos/circuitscape/circuitscape/profiler.py", line 127, in wrapper
    return func(*args)
  File "/Users/viral/repos/circuitscape/circuitscape/compute.py", line 45, in compute_network
    focal_nodes = self.read_focal_nodes(self.options.focal_node_file)
  File "/Users/viral/repos/circuitscape/circuitscape/compute.py", line 121, in read_focal_nodes
    focal_nodes = CSIO.load_graph(filename)
  File "/Users/viral/repos/circuitscape/circuitscape/io.py", line 174, in load_graph
    CSIO._check_file_exists(filename)
  File "/Users/viral/repos/circuitscape/circuitscape/io.py", line 24, in _check_file_exists
    raise RuntimeError('File "'  + filename + '" does not exist')
RuntimeError: File "(Browse for file with locations of focal points or areas)" does not exist

----------------------------------------------------------------------
Ran 47 tests in 2.604s

FAILED (errors=1)

Include cs_run.exe in Windows binaries

If possible, we should have an executable with the same path and function as cs_run.exe in Circuitscape version 3.5. This would allow backward compatibility with Circuitscape for ArcGIS and two components of Linkage Mapper that use Circuitscape.