worldbank / gostnets

Convenience wrapper for networkx analysis using geospatial information, focusing on OSM

GOSTnets: Build, process, and analyze networks

GOSTnets is built on top of geopandas, networkx, osmnx, and rtree.

Installation

From PyPI

pypi support in development

From conda-forge

conda support in development

From Source

1. Clone or download this repository to your local machine. Then, navigate to the root directory of the repository:

   git clone https://github.com/worldbank/GOSTnets.git
   cd GOSTnets

2. Create a virtual environment (optional but recommended):

   python3 -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

3. Install the package in editable mode with dependencies:

   pip install -e .

   The -e flag stands for "editable," meaning changes to the source code will immediately affect the installed package.

Alternative Installations With Optional Dependencies

The following are optional dependencies that can be installed with GOSTnets - which not required to use GOSTnets, but they may be useful for some users, as they enable additional functionality. All of these alternative installs use the same pip install command as above. The examples show how to install these from PyPI, but the same commands can be used to install from source, replacing the package name GOSTnets with a . similar to the example above.

OSM Support (Needed to run functions from load_osm.py)

pip install GOSTnets[osm]

Optimization Support (Needed to run functions from optimization.py)

pip install GOSTnets[opt]

Development Support

pip install GOSTnets[dev]

Usage

Every function contains a docstring which can be brought up in use to check the inputs for various functions. For example:

import GOSTnets as gn
gn.edge_gdf_from_graph?

returns:

Signature: gn.edge_gdf_from_graph(G, crs={'init': 'epsg:4326'}, attr_list=None, geometry_tag='geometry', xCol='x', yCol='y')
#### Function for generating a GeoDataFrame from a networkx Graph object ###
 REQUIRED: a graph object G
 OPTIONAL: crs - projection of format {'init' :'epsg:4326'}. Defaults to
           WGS84. Note: here we are defining the crs of the input geometry -
           we do NOT reproject to this crs. To reproject, consider using
           geopandas' to_crs method on the returned gdf.
           attr_list: list of the keys which you want to be moved over to
           the GeoDataFrame.
           geometry_tag - the key in the data dictionary for each edge which
           contains the geometry info.
           xCol - if no geometry is present in the edge data dictionary, the
           function will try to construct a straight line between the start
           and end nodes, if geometry information is present in their data
           dictionaries.  Pass the Longitude info as 'xCol'.
           yCol - likewise, determining the Latitude tag for the node's data
           dictionary allows us to make a straight line geometry where an
           actual geometry is missing.
 RETURNS: a GeoDataFrame object of the edges in the graph
#-------------------------------------------------------------------------#

These docstrings have been written for every function, and should help new and old users alike with the options and syntax.

License

This licensed under the MIT License. This project is licensed under the MIT License - see the LICENSE file for details.

gostnets's Issues

GOSTnets documentation won't build

tried building and got some errors, in my environment it seemed like Sphinx wasn't expecting Python 3 and couldn't build docs, I don't have time to try to fix now, so making a place marker.

merging Market_Access code from GOST_PublicGoods

It makes sense to merge the Market_Access code from GOST_PublicGoods (https://github.com/worldbank/GOST_PublicGoods/tree/master/Market_Access) to GOSTnets. I would like to gather other peoples' thoughts.

I took a look at the existing code, and here are the potential files to merge and my thoughts:

• Correcting_Missing_Points.ipynb: I'm not sure what this does. I Recommend not merging
• generate_OSRM: This code seems to deal with converting a network to a raster. I recommend not merging, but think about if any code would be useful for InfraSAP
• OSMNX_POIS: This code is useful for fetching OSM POIs via OSMNX, we just need to make sure that the functions are compatible with the latest OSMNX APIs. I Recommend merging.
• MA.py and OD.py: I believe that OD.py is a newer version of MA.py. It is a way for making OSRM calls and creating an OD matrix. I Recommend adding the Mapbox function from MA.py into OD.py, and then merging.

Can we rename the modules in GOSTNets, such as OSMNX_POIs.py --> fetch_pois.py, OD.py --> fetch_od.py ?

Installation issue

Need to specify pip install geopy=1.21.0
Using a new version of geopy (>2) returns this error:

osmids gone for nodes when using edges_and_nodes_csv_to_graph function

Hi gostnets team,

I was wondering how I could get back the osmids for the nodes when I use the (1) edges_and_nodes_csv_to_graph function. When I run the gn.node_gdf_from_graph function to the graph created with (1), instead of having the osmid as node_ID, the index only seem to appear.
Thanks!

Auto linting/styling of code

Add a linter and potentially a styler to automatically lint and style the code as part of the CI process when pull requests are opened and when pushes are made to 'main'.

Finish installation tools

Need to update pip and conda installation options

Add function for calculating traffic

Add a function to calculate the traffic on a road network

Update tutorials

Looks like the tutorials are a bit dated, so far I've found that the networkx.read_gpickle function was retired (https://networkx.org/documentation/stable/release/migration_guide_from_2.x_to_3.0.html#deprecated-code).

Should run through the various notebooks and make the necessary changes for this and other deprecated syntax.

Installation Error: ImportError: DLL load Failed: The specified module could not be found

Received error when assisting a colleague who was using Anaconda on Windows 10 machine. After looking at packages using 'conda list' found out that numpy was installed via conda, and scipy was install via pip. Solution was to uninstall both, and then install scipy with conda (numpy was also installed automatically as a dependency).

Suggesting to look into moving scipy and numpy installaiton from setup.py and instead installing scipy via conda.

Amend missing dependencies

In current installation, dependencies are missing, e.g., scipy and osmnx and it fails.

Create Documentation

Build the documentation on GitHub Pages (should also run and render examples if possible).

Oneway roads need to be taken into account when loading OSM data

The 'oneway' tag doesn't seem to be retained during the load_osm modules initialReadIn method (https://github.com/worldbank/GOSTnets/blob/master/GOSTnets/load_osm.py#L328). The add_missing_reflected_edges function in GOSTnets core can actually take oneway tags into account. Need to look into further...

Update pandana_snap_points function with improved re-projection internals

Would it be possible to update the pandana_snap_points with the more efficient re-projection operations of the pandana_snap_c function?

Snap_points is important for calculating origin-destination direct walking times and the current, slow function single-handedly multiplies the time it takes to generate accessibility analysis.

Including ferries in GOSTNet networks

The GOSTNetsload_osm routine filters solely on highways and thus excludes ferries from routing analysis. This degrades the accuracy of the modeled OD matrix when ferries play an important role in network connectivity.

As ferry routes are recorded as lines (route=ferry) in OSM they are theoretically compatible with GOSTNets, but I'm not sure how devilish the details are. A bonus bit of realism would be assigning wait time costs to ferry start/end nodes as in the asia_railways_v2 implementation.

A useful reference case is from Cox's Bazaar, Bangladesh, where communities across the bay to the north are connected to the town by ferry (see below screenshot). A roads-only network routes origin nodes in this community over a very long road path going first north, then back south. Some offshore island communities to the north are similarly connected to the mainland only by ferries.

Revise LICENSE

Per advise, WB projects should adopt MPL, but we could hold off until further guidance from Legal.

Get package on conda-forge

Create the necessary recipe to get the package hosted on conda-forge

Salting function sucks at creating edges

Our salting function salt_long_lines isn't working properly as noted by @rbanick. I've looked into this and will commit some changes to fix it.

1. Nodes are salted correctly, but long edges are dropped in the resulting graph, which means that "salted" edges are not being created correctly. The part of the script that re-adds salted edges is in a try statement, so errors in the edge creation / data attribution are not being captured.

2. The function is also projecting every edge one by one. Fixing that will make it run a lot quicker.

On a related note, I was testing this with a graph with edges that contained lists [] in their wkt attribute. This is a product of the clean network function. Adding rectify geometry after the second custom_simplify should fix this.

Publish to PyPI

This issue is for migrating and adding Trusted Publisher Management.

Standardize doc-strings

Most functions have doc-strings but they are not consistently formatted. Making them all consistent will improve the quality of the API documentation that will inevitably be auto-generated.

Create docker for graphtools

There is an existing workflow to use graphtools to speed up processing, that documentation needs to be added to the workflow

Create continuous integration workflows

Develop continuous integration workflows to do the following:

• #60
• Build and install the package (#42)
• Run unit and integration tests (could also run examples here as a "test" of sorts) (#67)
• #45
• #53

Improving load_osm module

The load_osm module is used from parsing OSM PBFs and making transportation graphs. It can be slow to run for large graphs.

I timed the load_osm functions and found out that the generateRoadsGDF method takes the most time (about 90% of the time), specifically finding all of the intersections.

At first I thought that the generateRoadsGDF method was not that important and could be skipped. However I ran more tests, and found out that the generateRoadsGDF is indeed necessary.

This is due to how GOSTnets parses the PBF file. The first step of the load_osm process is creating the the OSM_to_network object . During the creation of this object, the OSM PBF file will be parsed and OSM ways will be imported. The ways can not be used directly for a transportation network because they are not always split at intersections. See below for an example:

Therefore running the GOSTNet's generateRoadsGDF method is necessary. See this example for the graph after intersections are generated:

One possible avenue forward is to try to speed up the generateRoadsGDF method. I have not looked deeply into this.

Another avenue is to look at other libraries for inspiration. GOSTnets uses the GDAL OSM driver to parse the PBF and it cannot import the individual nodes that make up the ways. Comparing this with OSMNX, OSMNX uses Overpass queries, and OSMNX can parse both the ways and the nodes. OSMNX creates separate edges between each node pair initially. See ex:

Then OSMNX can optionally simplify where it will merge edges except at intersections. See example after OSMNX simplification:

I also tested Pyrosm, and it will split up the roads like OSMNX does. Pyrosm is able to read from OSM PBF files and save an OSMNX compatible NetworkX graph. The Pyrosm result would still benefit from using the OSMNX simplification method. Additional testing is needed for large graph as it seems like Pyrosm can need a lot of memory in these cases.

Remove the src/ directory?

@g4brielvs wanted to ask about this - why do we nest the GOSTnets directory within src?

Was just noticing locally that my builds from pyproject.toml automatically create a GOSTnets subdirectory within the root repository and that's where the autogenerated _version.py file is placed, which makes me think the pyproject setup is expecting the GOSTnets directory to sit directly beneath the repo root.

Add Country-Specific Road Speeds from OSM

Looking into the methodology behind the friction surface (Malaria Atlas Project, Weiss et al), I found out that they created a table of country, road type, and speed limit based on OSM attributes.

Here is the paragraph describing their road data processing:

Firstly, OSM roads data were extracted in October 2019 and converted into a rasterized surface of road types whereby the fastest road type in each pixel took precedence. In a second processing step, the OSM data were analyzed to determine the median speed limit value for each road type in each country (Supplementary Table 1). These values were subsequently used to attribute road speeds to specific pixels using a lookup table. In cases where a road type was present in a country, but associated speed limit information was unavailable, a global average speed limit for that road type was used instead.
https://www.nature.com/articles/s41591-020-1059-1#MOESM2

I propose that we look into the quality/completeness of this table, and use it to replace default_speeds, or at least have it as an option. The table is available through the paper in the Supplementary Tables 1–3 link.

Thoughts?

@d3netxer @bpstewar

Add optional dependencies to pyproject.toml

Both optional dependencies could be added to the pyproject.toml, depending on #40.

Reevaluate if `__init__.py` imports are necessary

This issue is to reevaluate if the imports on init.py are necessary, as it impacts the loading time.

Using shapefile-generated graphs with GOSTNets?

Hi all,

Is there guidance and/or examples for how to interact graph objects created from shapefiles with GOSTNets?

My current problem is an inability to use shapefile-generated graphs with GOSTNets. The back story is that after preparing some OSM data the recommended way (Tutorials 1 - 3), it became apparent a significant chunk of roads were disconnected in OSM and needed to be manually joined up. Re-creating a networkx graph object from this was straightforward (I got a <networkx.classes.digraph.DiGraph object) but using GOSTNets functions on the graph returns errors.

The most basic function I need is to generate a pickle from the graph, to enable further analysis. Running gn.save(my_new_graph,'my_new_graph','./files/') yields the following error

---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-8-e413f8aac5fb> in <module>
----> 1 gn.save(my_new_graph,'my_new_graph','../files/')

~/git/GOST_PublicGoods/GOSTNets/GOSTNets/GOSTnet.py in save(G, savename, wpath, pickle, edges, nodes)
   1037     """
   1038     if nodes == True:
-> 1039         new_node_gdf = node_gdf_from_graph(G)
   1040         new_node_gdf.to_csv(os.path.join(wpath, '%s_nodes.csv' % savename))
   1041     if edges == True:

~/git/GOST_PublicGoods/GOSTNets/GOSTNets/GOSTnet.py in node_gdf_from_graph(G, crs, attr_list, geometry_tag, xCol, yCol)
    211                 pass
    212 
--> 213         nodes.append(new_column_info)
    214         z += 1
    215 

UnboundLocalError: local variable 'new_column_info' referenced before assignment

This new_column_info error is repeated when running the CleanNetwork function defined in the Tutorial Step 2 notebook.

UnboundLocalError                         Traceback (most recent call last)
<ipython-input-10-76c9fa3ca80f> in <module>
     19     G = my_new_graph # inserting CXB edited graph object instead
     20 
---> 21     G = CleanNetwork(G, wpath, country, UTM, WGS, 0.5, verbose = False)
     22     print('\nend: %s' % time.ctime())
     23     print('\n--- processing complete for: %s ---' % country)

<ipython-input-2-85460a2dea5d> in CleanNetwork(G, wpath, country, UTM, WGS, junctdist, verbose)
     13 
     14     # Squeezes clusters of nodes down to a single node if they are within the snapping tolerance
---> 15     a = gn.simplify_junctions(G, UTM, WGS, junctdist)
     16 
     17     # ensures all streets are two-way

~/git/GOST_PublicGoods/GOSTNets/GOSTNets/GOSTnet.py in simplify_junctions(G, measure_crs, in_crs, thresh)
   1141     G2 = G.copy()
   1142 
-> 1143     gdfnodes = node_gdf_from_graph(G2)
   1144     gdfnodes_proj_buffer = gdfnodes.to_crs(measure_crs)
   1145     gdfnodes_proj_buffer = gdfnodes_proj_buffer.buffer(thresh)

~/git/GOST_PublicGoods/GOSTNets/GOSTNets/GOSTnet.py in node_gdf_from_graph(G, crs, attr_list, geometry_tag, xCol, yCol)
    211                 pass
    212 
--> 213         nodes.append(new_column_info)
    214         z += 1
    215 

UnboundLocalError: local variable 'new_column_info' referenced before assignment

Digging further into the node_gdf_from_graph function by running `edge_gdf = gn.edge_gdf_from_graph(my_new_graph) yields

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-23-2b59cc5ed218> in <module>
----> 1 edge_gdf = gn.edge_gdf_from_graph(cxbr)

~/git/GOST_PublicGoods/GOSTNets/GOSTNets/GOSTnet.py in edge_gdf_from_graph(G, crs, attr_list, geometry_tag, xCol, yCol)
    265             # if it doesn't have a geometry attribute, the edge is a straight
    266             # line from node to node
--> 267             x1 = G.nodes[u][xCol]
    268             y1 = G.nodes[u][yCol]
    269             x2 = G.nodes[v][xCol]

KeyError: 'x'

I am very much new to graph objects so apologies if this is a simple fix. Any suggestions or help would be welcome.

Substitute logging

Throughout the package, print is used for logging. This issue is for substituting it with logging.

Conformance to PEP 621

As part of our ongoing efforts to streamline project configurations and adhere to evolving best practices in the Python ecosystem, we propose the adoption of PEP 621 for declaring project metadata.

OSError: exception: stack overflow (Appears during clean_network)

Received this error when running clean_network on a large graph. In my case I was using it for country-level datasets in Mali and Burkina Faso, running on a Windows 10 machine. My current workaround as suggested by @d3netxer is to skip the cleaning function altogether.

Create Documentation

We need to create documentation - look into ReadTheDocs

Look into preserving bridge and tunnel topology

This was brought up during the workshop, doing a quick search I am not seeing that we are accounting for bridges and tunnels. We probably need to see if can skip creating an intersection if a non-bridge or tunnel intersects a bridge or tunnel, or something else.

proposing new module structure

• for consistency just use lowercase and underscores for library names.
• change name of GOSTnet.py to core.py
• change init.py from:

from GOSTnets.GOSTnet import *

to

from .core import *
from .calculate_od import *
from .optimization import *
etc...

I am working on the Sphinx documentation now and the autodoc functionality. It is spitting out function names like "GOSTnets.GOSTnet.find_graph_avg_speed"; so this way should make the function names look nicer too.

Advanced Snapping

Currently POIs (origins and destinations) are snapped to the graph using pandana snap. This snaps the POI to the nearest node in the graph. This is not ideal when the nodes in the graph are far away, but a part of the line is close. Salting the graph adds more nodes to the graph and increases the likelihood of snapping to a closer nearby node in the graph.

Ideally, you would like to snap the POI to the nearest point on the line in the graph nearest to the POI. This is possible, and described beautifully in this blog post: https://towardsdatascience.com/connecting-pois-to-a-road-network-358a81447944

I have adapted the code for GOSTnets : )

It runs slower than pandana snap, so pandana snap may still be useful to trade accuracy for speed.

Pull request incoming