GithubHelp home page GithubHelp logo

arup-group / genet Goto Github PK

View Code? Open in Web Editor NEW
43.0 9.0 10.0 43.76 MB

Manipulate MATSim networks via a Python API.

License: MIT License

Python 99.98% Dockerfile 0.02%
cml arup city-modelling city-modelling-lab network

genet's Introduction

Network Scenario Generator (GeNet)

DOI

GeNet provides tools to represent and work with a multi-modal transport network with public transport (PT) services. It is based on MATSim's representation of such networks.

The goal of GeNet is to:

  • Provide a formalised in-memory data structure for representing a multi-modal network with a PT service
  • Enable using the data structure for tasks such as generating auxiliary MATSim files e.g. Road Pricing
  • Simplify the process of modifying a network and provide a simple change log to track the differences between the input and output networks.
  • Provide validation methods to check for simple errors such as: whether a Route has more than one Stop or that the underlying graph doesn't have any dead-ends or sources (a place which you can leave but cannot get back to).

The underlying network available to PT services (roads, railways, but also ferry/flight connections) uses a networkx.MultiDiGraph with additional methods for 'links' which are unique in genet.Network (networkx.MultiDiGraph accepts multiple edges between the same from and to node pair; referring to an edge in networkx.MultiDiGraph and genet.Network has the same effects, i.e. the result is a dictionary indexed by the multi edge index). The PT services are represented through the genet.Schedule class which relies on other genet classes: the Schedule relies on a list of genet.Service's, which in turn consists of a list of genet.Route's. Each Route class object has an attribute stops which consists of genet.Stops objects. The Stops carry spatial information for the PT stop.

You can use GeNet's CLI to run pre-baked modifications or checks on networks. You can also write your own python scripts, importing genet as a package, use IPython shell or Jupyter Notebook to load up a network, inspect or change it and save it out to file.

Documentation

For more detailed instructions, see our documentation.

Installation

If you do not plan to make any code changes, you can install GeNet as a Docker image or a Python package.

For more detailed instructions, see our documentation.

As a Docker image

git clone [email protected]:arup-group/genet.git
cd genet
docker build -t "cml-genet" .

As a Python package

To install genet (indexed online as cml-genet), we recommend using the mamba package manager:

git clone [email protected]:arup-group/genet.git
cd genet
mamba create -n genet -c conda-forge -c city-modelling-lab --file requirements/base.txt
mamba activate genet
pip install --no-deps .

Contributing

There are many ways to contribute to genet. Before making contributions to the genet source code, see our contribution guidelines and follow the development install instructions.

If you plan to make changes to the code then please make regular use of the following tools to verify the codebase while you work:

  • pre-commit: run pre-commit install in your command line to load inbuilt checks that will run every time you commit your changes. The checks are: 1. check no large files have been staged, 2. lint python files for major errors, 3. format python files to conform with the pep8 standard. You can also run these checks yourself at any time to ensure staged changes are clean by simple calling pre-commit.
  • pytest - run the unit test suite and check test coverage.

For more information, see our documentation.

Installing a development environment

git clone [email protected]:arup-group/genet.git
cd genet
mamba create -n genet -c conda-forge -c city-modelling-lab --file requirements/base.txt --file requirements/dev.txt
mamba activate genet
pip install --no-deps -e .
ipython kernel install --user --name=genet

Building the documentation

If you are unable to access the online documentation, you can build the documentation locally. First, install a development environment of genet, then deploy the documentation using mike:

mike deploy develop
mike serve

Then you can view the documentation in a browser at http://localhost:8000/.

Credits

This package was created with Cookiecutter and the arup-group/cookiecutter-pypackage project template.

genet's People

Contributors

alex-andrey avatar ana-kop avatar arupwkeu avatar brynpickering avatar dependabot[bot] avatar elizabethc-arup avatar gac55 avatar kasiakoz avatar mfitz avatar pre-commit-ci[bot] avatar syhwawa avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

genet's Issues

feature request - extract_network_id_from_osm_csv warning messages

Hey I was wondering would it be possible to get the following function to trigger a warning message when no matches are found.

road_pricing.extract_network_id_from_osm_csv(
        network, 
        attribute_name, 
        osm_csv_path,
        output_dir
        )

I was able to do a simple check on the output of this function. Based on the number of matches. i think it would be good if there was a warning message if the function failed to find any matches.

osm_df = pd.read_csv(
            config.data_dir+'osm_tolls_with_network_ids.csv'
)

num_matches = len(osm_df.loc[osm_df['network_id'] == True])

if(num_matches == 0):
    print("Failed: 0 matches of osm tolls with network. Double check script is configured correctly!")
else:
    print("Success: Number of matches found = " + str(num_matches)) 

London network and schedule debug

This is a notebook dump of issues/things we tried when debugging the london sim:

Generally the mapping of modes from OSM and GTFS needs to be hyper transparent, suspect this belongs with PUMA though.

Some of the mode encodings are strings. Best to break these up. Also would prefer not to hold other info about links ie artificial, suggest these go in as a different optional attribute.

n.link_attribute_data_under_key('modes').apply(str).value_counts()

['car']                                                              215862
['car', 'bus', 'pt']                                                  79789
['rail']                                                               8162
['car', 'bus']                                                         7240
['artificial', 'bus']                                                  3907
['artificial', 'stopfacilitylink', 'bus']                              2339
['bus']                                                                 732
['artificial', 'subway,metro']                                          616
['rail', 'artificial']                                                  357
['stopfacilitylink', 'artificial', 'subway,metro']                      267
['artificial', 'tram,streetcar,light rail']                             162
['rail', 'stopfacilitylink', 'artificial']                               91
['stopfacilitylink', 'artificial', 'tram,streetcar,light rail']          84
['artificial', 'ferry']                                                  51
['rail', 'subway,metro', 'pt']                                           48
['rail', 'light_rail', 'pt']                                             41
['stopfacilitylink', 'artificial', 'ferry']                              35
['artificial', 'stopfacilitylink', 'bus', 'rail']                        25
['artificial', 'rail', 'bus']                                            14
['rail', 'pt']                                                           12
['rail', 'tram,streetcar,light rail', 'pt']                              12
['pt']                                                                    3
['gondola,suspended cable car', 'artificial']                             2
['gondola,suspended cable car', 'stopfacilitylink', 'artificial']         2

I used the following code to clean the network:

IGNORE = set(['pt','artificial','stopfacilitylink'])
MAP = {
    'car':'car',
    'bus':'bus',
    'rail':'rail',
    'subway':'subway',
    'metro':'subway',
    'tram':'tram',
    'streetcar':'tram',
    'light_rail':'rail',
    'light rail':'rail',
    'ferry':'ferry',
    'gondola':'gondola',
    'suspended cable car':'gondola'
      }
for link_id in list(n.link_id_mapping):
    link_data = n.link_id_mapping[link_id]
    u, v, multi_idx = link_data['from'], link_data['to'], link_data['multi_edge_idx']
    
    data = n.graph[u][v][multi_idx]
    modes = data['modes']
    new_modes = []
    for mode in modes:
        new_modes.extend(mode.split(","))
    new_modes = set(new_modes) - IGNORE
    new_modes = list(set([MAP[m] for m in new_modes]))
    if len(new_modes) == 0:
        n.graph.remove_edge(u, v, multi_idx)
        del n.link_id_mapping[link_id]
    else:
        data['modes'] = new_modes
        nx.set_edge_attributes(n.graph, {(u, v, multi_idx): data})

n.link_attribute_data_under_key('modes').apply(str).value_counts()

['car']               215862
['car', 'bus']         87029
['rail']                8663
['bus']                 6978
['subway']               883
['tram', 'rail']         258
['ferry']                 86
['subway', 'rail']        48
['rail', 'bus']           39
['gondola']                4

Querying schedule modes was a common request, eg:

modes = set()
for service in n.schedule.services.values():
    for route in service.routes.values():
        modes.add(route.mode)

{'bus', 'ferry', 'gondola', 'rail', 'subway', 'tram'}

One of the attempts to clean the network and schedule removed all routes based on the report:

report = n.generate_validation_report()
bad_routes = set()
for line_id, route_dict in report['schedule']['route_level'].items():
    for route_id, debug_dict in route_dict.items():
        if not debug_dict['is_valid_route']:
            bad_routes.add(route_id)

for line_id, route_id in report['routing']['service_routes_with_invalid_network_route']:
    bad_routes.add(route_id)

for service_id in list(n.schedule.services):
    service = n.schedule.services[service_id]
    for route_id in list(service.routes):
        if route_id in bad_routes:
            del n.schedule.services[service_id].routes[route_id]
    #  if not service.routes:
        #  del n.schedule.services[service_id]

We havn't really tested the resulting net and schedule, but it can be found in: /mnt/efs/London/bitsim/early_stopping/london_baseline_1perc_calibration_fred/fred-net

Validation and methods to deal with isolated nodes

Removing links can lead to lingering isolated nodes. Isolated nodes cause an error in MATSim:

java.lang.RuntimeException: For person id=PERSON_ID: getNearestLink returned Null! act=act [type=shop_food][coord=[x=544092.0 | y=713828.0]][linkId=null][startTime=undefined][endTime=15:30:00][duration=undefined][facilityId=null]

Links geojsons by OSM tag in standard outputs

Standard outputs script is supposed to generate geojsons of network links by OSM tag. However, this has not been happening since the structure of link attributes has been changed from

highway_tags = n.link_attribute_data_under_key({'attributes': {'osm:way:highway': 'text'}})

to

highway_tags = n.link_attribute_data_under_key({'attributes': 'osm:way:highway'})

Need to add functionality to ensure that geojsons get made both for the old and the new link attribute structure.

Split links with a new node

Functionality to divide an existing link at a point, using a new node would make it easier to encode new road schemes where a new junction is created.

Filter a network by speed attribute?

This is more a log about matsim debugging than a feature request:

The MATSim multimodal module creates additional networks by filter the input network (for cars) by some maximum speed. Pressumably to prevent people from walking and cycling on motorways for example.

This filtering is achieved with the MATSim multimodal param: cuttoffValueForNonCarModes.

We encountered a connection error with cuttoffValueForNonCarModes set at 25:

java.lang.RuntimeException: No route found from node 21542817 to node 242717999 by mode walk.

This was fixed by increading cuttoffValueForNonCarModes to 200. Pressumably as the networks created then become fully connected. However this means agents with be simulated walking and biking on motorways for example.

It would generally be useful to check for connectiveness of sub networks based on speeds. Hence the need to filter.

Import/export to GMNS

The General Modeling Network Specification is starting to make strides into becoming the standard network exchange format, and being able to import/export networks from that format will likely result in being able to reach all commercial packages as they include GMNS capabilities (as they did for OMX).

https://github.com/zephyr-data-specs/GMNS

Long time to validate network

Latest changes to Schedule object result in a long time for some operations. Validation of the Irish network post simplification took 8 hours
Screenshot 2021-03-01 at 09 28 29

Summary Stats

Methods to print to console and save to file (via standard outputs) summary statistics about the entire network and schedule.

Let's discuss what statistics should be included.

I envision a method summary for both network and schedule; something like this within network:

def summary(self):
   module.summarise_network(self)
   if self.schedule:
       self.schedule.summary()

docker image does not build on M1 mac

$ docker build -t "genet" .

[+] Building 14.3s (6/12)                                                                                                                                                                                                               
 => [internal] load build definition from Dockerfile                                                                                                                                                                               0.0s
 => => transferring dockerfile: 611B                                                                                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                  0.0s
 => => transferring context: 78B                                                                                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/python:3.7-slim-stretch                                                                                                                                                         4.5s
 => CANCELED [internal] load build context                                                                                                                                                                                         9.7s
 => => transferring context: 210.75MB                                                                                                                                                                                              9.7s
 => [1/8] FROM docker.io/library/python:3.7-slim-stretch@sha256:649b13bc3e6a70f7722d5eba4d78afbf4e9cec63193a29148271124840013286                                                                                                   8.3s
 => => resolve docker.io/library/python:3.7-slim-stretch@sha256:649b13bc3e6a70f7722d5eba4d78afbf4e9cec63193a29148271124840013286                                                                                                   0.0s
 => => sha256:649b13bc3e6a70f7722d5eba4d78afbf4e9cec63193a29148271124840013286 1.22kB / 1.22kB                                                                                                                                     0.0s
 => => sha256:1621928b38ecd3937f79cdf10c852ac715ec863b16a80e5de760c06955f316b1 1.37kB / 1.37kB                                                                                                                                     0.0s
 => => sha256:021c6270adc65b3afaa8f328f1b9045e376a81e3a77376a2e0eb7d2370ef03de 8.32kB / 8.32kB                                                                                                                                     0.0s
 => => sha256:9e7a560784c85cb9624bff5b6e479fbb95d5e265873987014f8aec75d509a530 20.39MB / 20.39MB                                                                                                                                   6.4s
 => => sha256:f106dd115c60608b983fe2f2d8a76d91f00d8ceecee4b904961b7b86d75b4ad0 2.22MB / 2.22MB                                                                                                                                     0.9s
 => => sha256:68aee2dccafc08f75ac6d3b7077d617206eded33ce5a4aa260dfe9fbc1a2d70b 9.72MB / 9.72MB                                                                                                                                     3.8s
 => => sha256:99d34799f5133dd79fa5c8813b4ad83ab1ec3c61cec10e596f02e7addd2678e1 241B / 241B                                                                                                                                         1.3s
 => => sha256:16a2d5d45d27528fc5a68912c01840cec7792fa8d8aac16d31ed06e3efb93129 2.63MB / 2.63MB                                                                                                                                     3.3s
 => => extracting sha256:9e7a560784c85cb9624bff5b6e479fbb95d5e265873987014f8aec75d509a530                                                                                                                                          0.9s
 => => extracting sha256:f106dd115c60608b983fe2f2d8a76d91f00d8ceecee4b904961b7b86d75b4ad0                                                                                                                                          0.2s
 => => extracting sha256:68aee2dccafc08f75ac6d3b7077d617206eded33ce5a4aa260dfe9fbc1a2d70b                                                                                                                                          0.4s
 => => extracting sha256:99d34799f5133dd79fa5c8813b4ad83ab1ec3c61cec10e596f02e7addd2678e1                                                                                                                                          0.0s
 => => extracting sha256:16a2d5d45d27528fc5a68912c01840cec7792fa8d8aac16d31ed06e3efb93129                                                                                                                                          0.1s
 => ERROR [2/8] RUN apt-get update && apt-get upgrade -y && apt-get -y install gcc git libspatialindex-dev curl coinor-cbc &&  rm -rf /var/lib/apt/lists/*                                                                         1.4s
------
 > [2/8] RUN apt-get update && apt-get upgrade -y && apt-get -y install gcc git libspatialindex-dev curl coinor-cbc &&  rm -rf /var/lib/apt/lists/*:
#0 0.410 Ign:1 http://deb.debian.org/debian stretch InRelease
#0 0.410 Ign:2 http://security.debian.org/debian-security stretch/updates InRelease
#0 0.423 Ign:3 http://deb.debian.org/debian stretch-updates InRelease
#0 0.424 Ign:4 http://security.debian.org/debian-security stretch/updates Release
#0 0.438 Ign:5 http://deb.debian.org/debian stretch Release
#0 0.443 Ign:6 http://security.debian.org/debian-security stretch/updates/main all Packages
#0 0.451 Ign:7 http://deb.debian.org/debian stretch-updates Release
#0 0.457 Ign:8 http://security.debian.org/debian-security stretch/updates/main arm64 Packages
#0 0.466 Ign:9 http://deb.debian.org/debian stretch/main all Packages
#0 0.472 Ign:6 http://security.debian.org/debian-security stretch/updates/main all Packages
#0 0.478 Ign:10 http://deb.debian.org/debian stretch/main arm64 Packages
#0 0.485 Ign:8 http://security.debian.org/debian-security stretch/updates/main arm64 Packages
#0 0.490 Ign:11 http://deb.debian.org/debian stretch-updates/main all Packages
#0 0.497 Ign:6 http://security.debian.org/debian-security stretch/updates/main all Packages
#0 0.502 Ign:12 http://deb.debian.org/debian stretch-updates/main arm64 Packages
#0 0.508 Ign:8 http://security.debian.org/debian-security stretch/updates/main arm64 Packages
#0 0.514 Ign:9 http://deb.debian.org/debian stretch/main all Packages
#0 0.523 Ign:6 http://security.debian.org/debian-security stretch/updates/main all Packages
#0 0.527 Ign:10 http://deb.debian.org/debian stretch/main arm64 Packages
#0 0.539 Ign:11 http://deb.debian.org/debian stretch-updates/main all Packages
#0 0.539 Ign:8 http://security.debian.org/debian-security stretch/updates/main arm64 Packages
#0 0.554 Ign:6 http://security.debian.org/debian-security stretch/updates/main all Packages
#0 0.554 Ign:12 http://deb.debian.org/debian stretch-updates/main arm64 Packages
#0 0.568 Ign:8 http://security.debian.org/debian-security stretch/updates/main arm64 Packages
#0 0.568 Ign:9 http://deb.debian.org/debian stretch/main all Packages
#0 0.582 Ign:10 http://deb.debian.org/debian stretch/main arm64 Packages
#0 0.582 Ign:6 http://security.debian.org/debian-security stretch/updates/main all Packages
#0 0.597 Ign:11 http://deb.debian.org/debian stretch-updates/main all Packages
#0 0.597 Err:8 http://security.debian.org/debian-security stretch/updates/main arm64 Packages
#0 0.597   404  Not Found
#0 0.614 Ign:12 http://deb.debian.org/debian stretch-updates/main arm64 Packages
#0 0.627 Ign:9 http://deb.debian.org/debian stretch/main all Packages
#0 0.649 Ign:10 http://deb.debian.org/debian stretch/main arm64 Packages
#0 0.661 Ign:11 http://deb.debian.org/debian stretch-updates/main all Packages
#0 0.673 Ign:12 http://deb.debian.org/debian stretch-updates/main arm64 Packages
#0 0.685 Ign:9 http://deb.debian.org/debian stretch/main all Packages
#0 0.698 Ign:10 http://deb.debian.org/debian stretch/main arm64 Packages
#0 0.713 Ign:11 http://deb.debian.org/debian stretch-updates/main all Packages
#0 0.727 Ign:12 http://deb.debian.org/debian stretch-updates/main arm64 Packages
#0 0.748 Ign:9 http://deb.debian.org/debian stretch/main all Packages
#0 0.802 Err:10 http://deb.debian.org/debian stretch/main arm64 Packages
#0 0.802   404  Not Found
#0 0.814 Ign:11 http://deb.debian.org/debian stretch-updates/main all Packages
#0 0.825 Err:12 http://deb.debian.org/debian stretch-updates/main arm64 Packages
#0 0.825   404  Not Found
#0 0.830 Reading package lists...
#0 0.837 W: The repository 'http://security.debian.org/debian-security stretch/updates Release' does not have a Release file.
#0 0.837 W: The repository 'http://deb.debian.org/debian stretch Release' does not have a Release file.
#0 0.837 W: The repository 'http://deb.debian.org/debian stretch-updates Release' does not have a Release file.
#0 0.837 E: Failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/main/binary-arm64/Packages  404  Not Found
#0 0.837 E: Failed to fetch http://deb.debian.org/debian/dists/stretch/main/binary-arm64/Packages  404  Not Found
#0 0.837 E: Failed to fetch http://deb.debian.org/debian/dists/stretch-updates/main/binary-arm64/Packages  404  Not Found
#0 0.837 E: Some index files failed to download. They have been ignored, or old ones used instead.
------
ERROR: failed to solve: executor failed running [/bin/sh -c apt-get update && apt-get upgrade -y && apt-get -y install gcc git libspatialindex-dev curl coinor-cbc &&  rm -rf /var/lib/apt/lists/*]: exit code: 100

Address `xfail` unit test

The unit test test_core_schedule_elements.py::test_splitting_service_edge_case_on_direction_results_in_two_directions is currently marked as xfail, meaning it is executed as normal with the test suite, but expected to fail (which it does). After removing the xfail decorator:

test_core_schedule_elements.py::test_splitting_service_edge_case_on_direction_results_in_two_directions FAILED [100%]
tests/test_core_schedule_elements.py:599 (test_splitting_service_edge_case_on_direction_results_in_two_directions)
[{'1_dir_1'}, {'2_dir_2'}, {'3_dir_1'}, {'4_dir_1'}] != [{'3_dir_1', '4_dir_1', '1_dir_1'}, {'2_dir_2'}]

Expected :[{'3_dir_1', '4_dir_1', '1_dir_1'}, {'2_dir_2'}]
Actual   :[{'1_dir_1'}, {'2_dir_2'}, {'3_dir_1'}, {'4_dir_1'}]
<Click to see difference>

service_edge_case_loopy_and_non_overlapping_graph = <Service instance at 5144736848: with 4 routes>

    def test_splitting_service_edge_case_on_direction_results_in_two_directions(
            service_edge_case_loopy_and_non_overlapping_graph):
        routes, graph_groups = service_edge_case_loopy_and_non_overlapping_graph.split_graph()
>       assert routes == [{'1_dir_1', '3_dir_1', '4_dir_1'}, {'2_dir_2'}]
E       AssertionError: assert [{'1_dir_1'}, {'2_dir_2'}, {'3_dir_1'}, {'4_dir_1'}] == [{'3_dir_1', '4_dir_1', '1_dir_1'}, {'2_dir_2'}]

test_core_schedule_elements.py:603: AssertionError

Assertion failed

We shouldn't leave the test as xfail, we should address why the test is failing, fix whatever needs fixing, and reintroduce the test to the test suite.

Add validation checks for PT stop attributes snapping to other graphs (Intermodal access/egress in matsim)

Hit with a matsim error:

java.lang.NullPointerException: Cannot invoke "org.matsim.api.core.v01.network.Link.getToNode()" because "link" is null
	at ch.sbb.matsim.routing.pt.raptor.SwissRailRaptorRoutingModule.findCoordinate(SwissRailRaptorRoutingModule.java:97) ~[columbus-2.1.0-jar-with-dependencies.jar:2.1.0]
	at ch.sbb.matsim.routing.pt.raptor.SwissRailRaptorRoutingModule.fillWithActivities(SwissRailRaptorRoutingModule.java:73) ~[columbus-2.1.0-jar-with-dependencies.jar:2.1.0]
	at ch.sbb.matsim.routing.pt.raptor.SwissRailRaptorRoutingModule.calcRoute(SwissRailRaptorRoutingModule.java:60) ~[columbus-2.1.0-jar-with-dependencies.jar:2.1.0]
	at org.matsim.core.router.TripRouter.calcRoute(TripRouter.java:183) ~[columbus-2.1.0-jar-with-dependencies.jar:2.1.0]
	at org.matsim.core.router.PlanRouter.run(PlanRouter.java:102) ~[columbus-2.1.0-jar-with-dependencies.jar:2.1.0]
	at org.matsim.core.population.algorithms.PersonPrepareForSim.run(PersonPrepareForSim.java:234) ~[columbus-2.1.0-jar-with-dependencies.jar:2.1.0]
	at org.matsim.core.population.algorithms.ParallelPersonAlgorithmUtils$PersonAlgoThread.run(ParallelPersonAlgorithmUtils.java:146) ~[columbus-2.1.0-jar-with-dependencies.jar:2.1.0]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

Turned out a link was deleted that provided access to a PT stop. Some checks for this should be added to genet.

Set attributes for PT schedule elements in a Schedule

If contained in the Schedule object, the PT schedule element objects: Stop, Route, Service do not exist until requested from the Schedule. A new object is created using data stored in

schedule._graph.nodes
schedule._graph.graph['routes']
schedule._graph.graph['services']

respectively.

Right now we rely on Schedule level methods to change these objects. We should add methods that when an attribute is changed or added e.g. route.mode = 'super_bus' the data in schedule._graph.graph['routes'][route.id] is updated, and a change logged in schedule._graph.graph['change_log'].

Errors should be raised when:

  • attempting to change id of the object
  • attempting to change epsg attribute (projection)
    user should be directed to the dedicated methods for reindexing and reprojections

and warnings for:

  • [Route] changing mode (which affects the modes on the network graph link for the route or could mean that the route needs to be rerouted (depending on the type of mode change))
  • [Route] changing network route (there are genet methods for routing, if user is setting this themselves they should be reminded of these methods and that their route may not be valid)
  • [Route] changing ordered_stops (this will affect the network route which relies on links referenced by stops and their order)
  • [Stop] changing spatial values x or y, since stops are usually snapped to a network link and moving them may mean the snapping is no longer correct

Nb. that schedule._graph is shared with and accessible from lesser objects, e.g. route._graph, if that route if contained within that schedule.

Subset Network method

Right now we only have a method to extract a subgraph of the genet.Network.graph on user-defined conditions subgraph_on_link_conditions. This could and should be extended to a subset_network_on_conditions method which returns another instance of genet.Network. This new object has the graph attribute given using subgraph_on_link_conditions, its' link_id_mapping attribute is subsetted accordingly too. In the first instance it could raise a NotImplemented error if this network has a non-empty Schedule element, until a similar subset_schedule_on_conditions method is developed.

vehicles.xml pretty print

Output xmls are formatted as single strings. In the case of vehicles which is a commonly editted file, it would be useful to have a 'pretty print' option, eg:

        <vehicleType id="Rail">
                <capacity>
                        <seats persons="100"/>
                        <standingRoom persons="0"/>
                </capacity>
                <length meter="200.0"/>
                <width meter="2.8"/>
                <accessTime secondsPerPerson="0.25"/>
                <egressTime secondsPerPerson="0.25"/>
                <doorOperation mode="serial"/>
                <passengerCarEquivalents pce="27.1"/>
        </vehicleType>

Smooth out network 'pinches'

Implement logic to smooth out cases where values on the network such as number of lanes, capacity, speed jumps down and up, e.g.
Screenshot 2022-10-04 at 11 56 48

This has impact on simulation in MATSim introducing bottlenecks for the traffic.

CI build is broken

The open-source GitHub action we are using to send Slack notifications from CI workflows no longer builds, leading not just to missing notifications, but completely failed builds that look like this:

Screenshot 2023-07-18 at 16 25 25

The quick fix is to upgrade the version of the GitHub action we are using in our workflows for Slack notifications.

Replace osmread with a maintained library

the osmread package seems to be no longer maintained and requires pip installing from a specific dev commit in GeNet.

For future maintainability of GeNet, another package that has a bit more weight behind it would be ideal. Even better would be one that has a conda-forge feedstock. For example, pyrosm (although it seems to be having trouble with maintanence too, with a lot of people raising issues and contributing PRs, but the maintainer staying silent).

Even better than that would be to avoid having a dependency for parsing the OSM file. How simple it would be to implement the specific functionality that osmread currently provides @KasiaKoz...?

GeNet produces many warnings

Running the unit test suite produces 453 warnings. Some of these warnings are caused by the same pyproj syntax deprecation described by this recent Puma PR, but there are also others with different origins.

Screen Shot 2021-09-09 at 13 45 31

When I run GeNet in Matesto to simplify a test town network, I see six of these pyproj deprecation warnings:

FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  return _prepare_from_string(" ".join(pjargs))

It would be good to fix some of these - at the very least, we should sort out the pyproj warnings.

links_on_spatial_condition failing for large spatial area

Hey,

I was using the function links_on_spatial_condition with a geojson input and it was failing due to a memory error (see below). The geojson contained a polygon which covered multiple cities. I was able to work around it by using gpd.sjoin(network_gdf,urban_gdf,how='inner', op='intersects'). It might be worth changing the function links_on_spatial_condition to use gpd.sjoin instead of gdf.intersects.

Thanks.

Error:
'Segmentation fault (core dumped)'

Long read times for chunky schedule

Reading in a MATSim schedule of size ~1GB which relates to a non simplified network takes a long time (~5 h O_O)
Looking at memory usage, it looks like a long constant hang, probably when the underlying schedule graph is built. Methods about this may need to be revised.

Support Python 3.11

GeNet does not install in a Python 3.11.0 environment. On an EC2 instance running ubuntu:

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04 LTS
Release:	20.04
Codename:	focal
Collecting osmnx==0.15.0
  Using cached osmnx-0.15.0-py2.py3-none-any.whl (84 kB)
Collecting packaging==20.4
  Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting pandas==1.3.3
  Using cached pandas-1.3.3.tar.gz (4.7 MB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [964 lines of output]
      Ignoring numpy: markers 'python_version == "3.7" and (platform_machine != "arm64" or platform_system != "Darwin") and platform_machine != "aarch64"' don't match your environment
      Ignoring numpy: markers 'python_version == "3.8" and (platform_machine != "arm64" or platform_system != "Darwin") and platform_machine != "aarch64"' don't match your environment
      Ignoring numpy: markers 'python_version == "3.7" and platform_machine == "aarch64"' don't match your environment
      Ignoring numpy: markers 'python_version == "3.8" and platform_machine == "aarch64"' don't match your environment
      Ignoring numpy: markers 'python_version == "3.8" and platform_machine == "arm64" and platform_system == "Darwin"' don't match your environment
      Ignoring numpy: markers 'python_version == "3.9" and platform_machine == "arm64" and platform_system == "Darwin"' don't match your environment

...


            error: Command "gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DNPY_INTERNAL_BUILD=1 -DHAVE_NPY_CONFIG_H=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -DNO_ATLAS_INFO=1 -DHAVE_CBLAS -I/usr/local/include -I/usr/include -I/home/arup/.pyenv/versions/3.11.0/envs/genet-3.11.0/include -Ibuild/src.linux-x86_64-3.11/numpy/core/src/umath -Ibuild/src.linux-x86_64-3.11/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.11/numpy/core/src/common -Inumpy/core/include -Ibuild/src.linux-x86_64-3.11/numpy/core/include/numpy -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/arup/.pyenv/versions/3.11.0/envs/genet-3.11.0/include -I/home/arup/.pyenv/versions/3.11.0/include/python3.11 -Ibuild/src.linux-x86_64-3.11/numpy/core/src/common -Ibuild/src.linux-x86_64-3.11/numpy/core/src/npymath -c build/src.linux-x86_64-3.11/numpy/core/src/multiarray/scalartypes.c -o build/temp.linux-x86_64-3.11/build/src.linux-x86_64-3.11/numpy/core/src/multiarray/scalartypes.o -MMD -MF build/temp.linux-x86_64-3.11/build/src.linux-x86_64-3.11/numpy/core/src/multiarray/scalartypes.o.d -std=c99" failed with exit status 1
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for numpy
      Failed to build numpy
      ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects

      [notice] A new release of pip available: 22.3 -> 22.3.1
      [notice] To update, run: python3.11 -m pip install --upgrade pip
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I can make the pip installation work by upgrading a couple of dependencies in requirements.txt:

$ diff requirements.txt requirements-3.11.txt
44c44
< pandas==1.3.3
---
> pandas==1.5.2
67c67
< pyzmq==19.0.1
---
> pyzmq==24.0.1

However, the unit test suite then fails with a lot of errors:

FAIL Required test coverage of 94.0% not reached. Total coverage: 3.49%
================================================================== short test summary info ===================================================================
ERROR tests/test_auxiliary_files.py - TypeError: 'tuple' object does not support item assignment
ERROR tests/test_auxiliary_files.py - TypeError: 'tuple' object does not support item assignment
ERROR tests/test_core_components_route.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_core_components_route.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_core_components_service.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_core_components_service.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_core_components_stop.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_core_components_stop.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_core_network.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_core_network.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_core_schedule.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_core_schedule.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_core_schedule_elements.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_core_schedule_elements.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_input_gtfs_reader.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_input_gtfs_reader.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_input_matsim_reader.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_input_matsim_reader.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_input_osm_reader.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_input_osm_reader.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_input_osmnx_customised.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_input_osmnx_customised.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_input_read.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_input_read.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_max_stable_set.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_max_stable_set.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_modify_change_log.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_modify_change_log.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_modify_graph.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_modify_graph.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_modify_schedule.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_modify_schedule.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_output_geojson.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_output_geojson.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_output_matsim_xml_writer.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_output_matsim_xml_writer.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_output_sanitiser.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_output_sanitiser.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_road_pricing.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_road_pricing.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_use_schedule.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_use_schedule.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_dict_support.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_dict_support.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_elevation.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_elevation.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_google_directions.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_google_directions.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_graph_operations.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_graph_operations.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_parallel.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_parallel.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_persistence.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_persistence.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_secrets_vault.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_secrets_vault.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_simplification.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_simplification.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_spatial.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_utils_spatial.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_validate_network.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_validate_network.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_validate_schedule.py - RuntimeError: Duplicate Global Set declaration, Any
ERROR tests/test_validate_schedule.py - RuntimeError: Duplicate Global Set declaration, Any
======================================================== 22 passed, 77 warnings, 64 errors in 32.96s =========================================================

An example stack trace:

______________________________________________________ ERROR collecting tests/test_validate_schedule.py ______________________________________________________
tests/test_validate_schedule.py:3: in <module>
    from genet import Schedule, Service, Route, Stop, schedule_elements
genet/__init__.py:1: in <module>
    from genet.core import Network  # noqa: F401
genet/core.py:14: in <module>
    import genet.modify.schedule as modify_schedule
genet/modify/schedule.py:4: in <module>
    from genet.max_stable_set import MaxStableSet
genet/max_stable_set.py:7: in <module>
    from pyomo.environ import *  # noqa: F403
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyomo/environ/__init__.py:103: in <module>
    _import_packages()
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyomo/environ/__init__.py:74: in _import_packages
    _do_import(pname)
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyomo/environ/__init__.py:18: in _do_import
    importlib.import_module(pkg_name)
../.pyenv/versions/3.11.0/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyomo/core/__init__.py:36: in <module>
    import pyomo.core.base._pyomo
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyomo/core/base/__init__.py:20: in <module>
    import pyomo.core.base.boolean_var
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyomo/core/base/boolean_var.py:15: in <module>
    from pyomo.core.base.set import Set, BooleanSet
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyomo/core/base/set.py:4055: in <module>
    DeclareGlobalSet(_AnySet(
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyomo/core/base/set.py:3963: in DeclareGlobalSet
    raise RuntimeError("Duplicate Global Set declaration, %s"
E   RuntimeError: Duplicate Global Set declaration, Any

There are also a fair number of warnings:

====================================================================== warnings summary ======================================================================
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/xdist/plugin.py:227
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/xdist/plugin.py:227
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/xdist/plugin.py:227
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/xdist/plugin.py:227: DeprecationWarning: The --rsyncdir command line argument and rsyncdirs config variable are deprecated.
  The rsync feature will be removed in pytest-xdist 4.0.
    config.issue_config_time_warning(warning, 2)

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/networkx/readwrite/graphml.py:346
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/networkx/readwrite/graphml.py:346
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/networkx/readwrite/graphml.py:346: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    (np.int, "int"), (np.int8, "int"),

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/networkx/readwrite/gexf.py:223
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/networkx/readwrite/gexf.py:223
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/networkx/readwrite/gexf.py:223: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
  Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
    (np.int, "int"), (np.int8, "int"),

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/sklearn/utils/multiclass.py:14
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/sklearn/utils/multiclass.py:14
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/sklearn/utils/multiclass.py:14: DeprecationWarning: Please use `spmatrix` from the `scipy.sparse` namespace, the `scipy.sparse.base` namespace is deprecated.
    from scipy.sparse.base import spmatrix

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/sklearn/utils/optimize.py:18
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/sklearn/utils/optimize.py:18
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/sklearn/utils/optimize.py:18: DeprecationWarning: Please use `line_search_wolfe2` from the `scipy.optimize` namespace, the `scipy.optimize.linesearch` namespace is deprecated.
    from scipy.optimize.linesearch import line_search_wolfe2, line_search_wolfe1

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/sklearn/utils/optimize.py:18
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/sklearn/utils/optimize.py:18
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/sklearn/utils/optimize.py:18: DeprecationWarning: Please use `line_search_wolfe1` from the `scipy.optimize` namespace, the `scipy.optimize.linesearch` namespace is deprecated.
    from scipy.optimize.linesearch import line_search_wolfe2, line_search_wolfe1

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/certifi/core.py:36
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/certifi/core.py:36
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/certifi/core.py:36: DeprecationWarning: path is deprecated. Use files() instead. Refer to https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-from-legacy for migration advice.
    _CACERT_CTX = get_path("certifi", "cacert.pem")

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:17
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:17
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:17: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PANDAS_GE_025 = str(pd.__version__) >= LooseVersion("0.25.0")

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/setuptools/_distutils/version.py:346: 26 warnings
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    other = LooseVersion(other)

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:18
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:18
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:18: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PANDAS_GE_10 = str(pd.__version__) >= LooseVersion("1.0.0")

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:19
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:19
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:19: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PANDAS_GE_11 = str(pd.__version__) >= LooseVersion("1.1.0")

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:20
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:20
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:20: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PANDAS_GE_115 = str(pd.__version__) >= LooseVersion("1.1.5")

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:21
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:21
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:21: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PANDAS_GE_12 = str(pd.__version__) >= LooseVersion("1.2.0")

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:29
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:29
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:29: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    SHAPELY_GE_17 = str(shapely.__version__) >= LooseVersion("1.7.0")

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:30
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:30
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:30: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    SHAPELY_GE_18 = str(shapely.__version__) >= LooseVersion("1.8")

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:31
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:31
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:31: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    SHAPELY_GE_20 = str(shapely.__version__) >= LooseVersion("2.0")

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:212
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:212
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:212
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:212
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/geopandas/_compat.py:212: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PYPROJ_LT_3 = LooseVersion(pyproj.__version__) < LooseVersion("3")

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyparsing.py:108
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyparsing.py:108
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyparsing.py:108: DeprecationWarning: module 'sre_constants' is deprecated
    import sre_constants

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/matplotlib/__init__.py:202: 10 warnings
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/matplotlib/__init__.py:202: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if LooseVersion(module.__version__) < minver:

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyutilib/misc/import_file.py:11
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyutilib/misc/import_file.py:11
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/pyutilib/misc/import_file.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
    import imp

../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/botocore/utils.py:25
../.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/botocore/utils.py:25
  /home/arup/.pyenv/versions/genet-3.11.0/lib/python3.11/site-packages/botocore/utils.py:25: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
    import cgi

Most likely this is a question of upgrading the right dependencies to the correct versions, but this may leave us in a situation where GeNet can no longer be used in a Python 3.7 environment. If we can be backwards compatible, we should.

Elevation tif only accepted in crs=4326

Currently, it is only possible to 'query' the tif elevation file if it is in crs=4326. It would be good to add some clarification on that and maybe a check/ re-projection too

Installing GeNet

From Sarah Hayes
Installing genet

Don’t really understand what this one does. Also when you’d use docker and when you’d use python. And how do you know the installation has worked?

There are some notes about testing but these seem to be for the python version of the install, not sure how to run them with the docker container. I tried running “python -m pytest -vv tests” from the root directory but failed with missing modules (since they’re in the docker container I suppose!)

When I tried the venv way, requirements.txt failed to install rtree=0.9.4 and exited. Then when I tried to run the test scripts, failed with ‘no module named pytest’

Stop attributes data summary is too specific

schedule.summary() method fails here when not all stops have bikeAccessible

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-7-8c7164670021> in <module>
----> 1 n.schedule.summary()

~/genet/genet/schedule_elements.py in summary(self)
   3182             {'attributes': 'bikeAccessible'}).to_dict(orient='index')
   3183         for key in bike_attribs_dict.keys():
-> 3184             bike_accessible_keys.append(bike_attribs_dict[key]['attributes']['bikeAccessible'])
   3185         report['accessibility_tags']['Unique values for bikeAccessible tag'] = set(bike_accessible_keys)
   3186 

KeyError: 'bikeAccessible'

This method should be more general, showing that data there is, rather than assuming these keys are what the user is after.

Automatically generate length for added links

Once we make spatial information for nodes non-optional: #121 we should default to genet calculating straight line (or not straight, if geometry is being added too) distance for links being added, for convenience.

Error when generating routed speeds geodataframe

The error appeared when generating standard outputs for updated Sheffield network using the following command:

n.generate_standard_outputs('/mnt/efs/networks/loyal-tint/fixed_schedule_2/standard_outputs')

The error message:

2022-10-26 11:43:30,925 - Right now routed speeds do not account for services snapping to long network links. Be sure to account for that in your investigations and check the non-routed `pt_speeds`output as well.
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/pandas/core/series.py in __setitem__(self, key, value)
   1061         try:
-> 1062             self._set_with_engine(key, value)
   1063         except (KeyError, ValueError):
​
~/.local/lib/python3.8/site-packages/pandas/core/series.py in _set_with_engine(self, key, value)
   1094         # fails with AttributeError for IntervalIndex
-> 1095         loc = self.index._engine.get_loc(key)
   1096         # error: Argument 1 to "validate_numeric_casting" has incompatible type
​
~/.local/lib/python3.8/site-packages/pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
​
~/.local/lib/python3.8/site-packages/pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
​
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
​
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
​
KeyError: 'geometry'
​
During handling of the above exception, another exception occurred:
​
NotImplementedError                       Traceback (most recent call last)
<ipython-input-47-17011cd6dba8> in <module>
----> 1 n.generate_standard_outputs('/mnt/efs/networks/loyal-tint/fixed_schedule_2/standard_outputs')
​
/mnt/efs/networks/genet-lib/genet/genet/core.py in generate_standard_outputs(self, output_dir, gtfs_day, include_shp_files)
   2137         :return: None
   2138         """
-> 2139         geojson.generate_standard_outputs(self, output_dir, gtfs_day, include_shp_files)
   2140         logging.info('Finished generating standard outputs. Zipping folder.')
   2141         persistence.zip_folder(output_dir)
​
/mnt/efs/networks/genet-lib/genet/genet/output/geojson.py in generate_standard_outputs(n, output_dir, gtfs_day, include_shp_files, schedule_network_factor)
    223     # schedule outputs
    224     if n.schedule:
--> 225         generate_standard_outputs_for_schedule(
    226             n.schedule,
    227             output_dir=os.path.join(output_dir, 'schedule'),
​
/mnt/efs/networks/genet-lib/genet/genet/output/geojson.py in generate_standard_outputs_for_schedule(schedule, output_dir, gtfs_day, include_shp_files, schedule_network_factor, gdf_network_links)
    128     logging.info(f'Generating stop-to-stop speed outputs with network_factor={schedule_network_factor}')
    129     speed_dir = os.path.join(output_dir, 'speed')
--> 130     speeds_gdf = schedule.speed_geodataframe(
    131         network_factor=schedule_network_factor, gdf_network_links=gdf_network_links)
    132     save_geodataframe(
​
/mnt/efs/networks/genet-lib/genet/genet/schedule_elements.py in speed_geodataframe(self, network_factor, gdf_network_links)
    305         df['speed'] = df['distance'] / df['time']
    306         if gdf_network_links is not None:
--> 307             network_distance_df = use_schedule.network_routed_distance_gdf(self, gdf_network_links)
    308             df = gpd.GeoDataFrame(
    309                 df.merge(
​
/mnt/efs/networks/genet-lib/genet/genet/use/schedule.py in network_routed_distance_gdf(schedule, gdf_network_links)
    205     ).assign(route=np.concatenate(routes_df['route'].values), sequence=np.concatenate(routes_df['sequence'].values))
    206     routes_df = routes_df.merge(gdf_network_links[['length', 'geometry']], left_on='route', right_index=True)
--> 207     return routes_df.groupby(['id', 'from_stop', 'to_stop']).apply(combine_route).reset_index(drop=True)
​
~/.local/lib/python3.8/site-packages/pandas/core/groupby/groupby.py in apply(self, func, *args, **kwargs)
   1270         with option_context("mode.chained_assignment", None):
   1271             try:
-> 1272                 result = self._python_apply_general(f, self._selected_obj)
   1273             except TypeError:
   1274                 # gh-20949
​
~/.local/lib/python3.8/site-packages/pandas/core/groupby/groupby.py in _python_apply_general(self, f, data)
   1304             data after applying f
   1305         """
-> 1306         keys, values, mutated = self.grouper.apply(f, data, self.axis)
   1307 
   1308         return self._wrap_applied_output(
​
~/.local/lib/python3.8/site-packages/pandas/core/groupby/ops.py in apply(self, f, data, axis)
    818             # group might be modified
    819             group_axes = group.axes
--> 820             res = f(group)
    821             if not _is_indexed_like(res, group_axes, axis):
    822                 mutated = True
​
/mnt/efs/networks/genet-lib/genet/genet/use/schedule.py in combine_route(group)
    172         length = group['length'].sum()
    173         group = group.iloc[0, :][['id', 'from_stop', 'to_stop']]
--> 174         group['geometry'] = geom
    175         group['network_distance'] = length
    176         return group
​
~/.local/lib/python3.8/site-packages/pandas/core/series.py in __setitem__(self, key, value)
   1068             else:
   1069                 # GH#12862 adding a new key to the Series
-> 1070                 self.loc[key] = value
   1071 
   1072         except TypeError as err:
​
~/.local/lib/python3.8/site-packages/pandas/core/indexing.py in __setitem__(self, key, value)
    721 
    722         iloc = self if self.name == "iloc" else self.obj.iloc
--> 723         iloc._setitem_with_indexer(indexer, value, self.name)
    724 
    725     def _validate_key(self, key, axis: int):
​
~/.local/lib/python3.8/site-packages/pandas/core/indexing.py in _setitem_with_indexer(self, indexer, value, name)
   1722 
   1723             if missing:
-> 1724                 self._setitem_with_indexer_missing(indexer, value)
   1725                 return
   1726 
​
~/.local/lib/python3.8/site-packages/pandas/core/indexing.py in _setitem_with_indexer_missing(self, indexer, value)
   1996 
   1997             # this preserves dtype of the value
-> 1998             new_values = Series([value])._values
   1999             if len(self.obj._values):
   2000                 # GH#22717 handle casting compatibility that np.concatenate
​
~/.local/lib/python3.8/site-packages/pandas/core/series.py in __init__(self, data, index, dtype, name, copy, fastpath)
    437                     data = data.copy()
    438             else:
--> 439                 data = sanitize_array(data, index, dtype, copy)
    440 
    441                 manager = get_option("mode.data_manager")
​
~/.local/lib/python3.8/site-packages/pandas/core/construction.py in sanitize_array(data, index, dtype, copy, raise_cast_failure, allow_2d)
    569             subarr = _try_cast(data, dtype, copy, raise_cast_failure)
    570         else:
--> 571             subarr = maybe_convert_platform(data)
    572             if subarr.dtype == object:
    573                 subarr = cast(np.ndarray, subarr)
​
~/.local/lib/python3.8/site-packages/pandas/core/dtypes/cast.py in maybe_convert_platform(values)
    116 
    117     if isinstance(values, (list, tuple, range)):
--> 118         arr = construct_1d_object_array_from_listlike(values)
    119     else:
    120         # The caller is responsible for ensuring that we have np.ndarray
​
~/.local/lib/python3.8/site-packages/pandas/core/dtypes/cast.py in construct_1d_object_array_from_listlike(values)
   1988     # making a 1D array that contains list-likes is a bit tricky:
   1989     result = np.empty(len(values), dtype="object")
-> 1990     result[:] = values
   1991     return result
   1992 
​
~/.local/lib/python3.8/site-packages/shapely/geometry/base.py in __array_interface__(self)
    852     def __array_interface__(self):
    853         """Provide the Numpy array protocol."""
--> 854         raise NotImplementedError("Multi-part geometries do not themselves "
    855                                   "provide the array interface")
    856 
​
NotImplementedError: Multi-part geometries do not themselves provide the array interface

GTFS reader

GTFS reader is not currently reading or passing information on:

  • service names
  • route names
  • stop names

(human readable names) when creating genet.Schedule objects. These are really useful and should persist to the Schedule object.

Pull all of the apps in the scripts directory into a single command line interface

GeNet has a bunch of useful python applications in the scripts directory to perform tasks such as validating or simplifying a network, or adding elevation and slope data to a network. It would be good to pull all of these different apps into a single, discoverable CLI so that we could issue commands such as genet network simplify, genet network validate or genet network add-link-slopes. The CLI should be discoverable, enabling commands such as genet --help, genet network --help or genet network simplify --help.

need check for np.inf linkspeeds

"inf" link speeds cause matsim to fail, eg:

<link id="pt_150042002011" from="pt_150042002011" to="pt_150042002011" freespeed="inf" capacity="9999.0" permlanes="1.0" oneway="1" modes="bus,artificial,stopfacilitylink" length="1.0"/>

Unit tests failed with error on WSL (Resovled)

After following the commands to install and setup genet, several of the unit tests failed with errors and warnings on the WSL system. This error was resolved when we tried to run it on aws.

==========
Resolved: Was originally executing in both conda and venv environments which may have resulted in clashes, after deactivating the conda envrionment the tests worked fine.

Automatically fix some PT Schedule issues

Fixing of some issues with PT Schedule can be automated:

  • Zero headways - removing duplicated trips happening at the same time
  • Unrealistic speeds - this might be semi-automatic since network routing may be at fault here too and needs to be ruled out before 'fixing' but this would be a method to re-calculate stop arrival and departure offsets

(internal link to project where we used genet to fix these issues)

Copy PT schedule element objects for reuse

Copying an object that exists in a Schedule, changing it and trying to add it again, like so:
image
results in the following error:
image (1)

When the object is copied, it should be 'detached' from the Schedule, access to the graph schedule._graph, which is shared with and accessible from lesser objects, should be severed, a new graph built for that object, that is not connected to the schedule, using the _build_graph method.

Standard outputs generation takes too long

Generating the suite of standard outputs following some modification process (which is typical for a genet script, e.g. the simplification script) is taking too long. The simplification process takes a lot less time in comparison and it is doing a much more complicated job!

On a small Londinium example takes 0.06 min to simplify the network and 0.56 min to generate the standard outputs. This translates to absolute hours for large networks.

KeyError during Network simplification

Network generated from OSM using genet's OSM read method results in a KeyError when attempting to simplify.

2021-04-01 17:54:17,906 - Begin simplifying the graph
2021-04-01 17:54:18,903 - Generating paths to be simplified
2021-04-01 17:54:24,803 - Identified 88788 edge endpoints
2021-04-01 17:54:25,364 - Identified 202870 possible paths
2021-04-01 17:54:25,883 - Processing 202870 paths
2021-04-01 17:54:26,932 - Found 126213 paths to simplify.
2021-04-01 17:54:27,417 - Generated 126213 link ids.
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-24-a4232ca9d4a5> in <module>
----> 1 n.simplify()

~/genet/genet/core.py in simplify(self, no_processes)
    164         if self.is_simplified():
    165             raise RuntimeError('This network has already been simplified. You cannot simplify the graph twice.')
--> 166         simplification.simplify_graph(self, no_processes)
    167         # mark graph as having been simplified
    168         self.graph.graph["simplified"] = True

~/genet/genet/utils/simplification.py in simplify_graph(n, no_processes)
    200 
    201     indexed_paths_to_simplify = dict(zip(n.generate_indices_for_n_edges(len(edges_to_simplify)), edges_to_simplify))
--> 202     indexed_paths_to_simplify = _assemble_path_data(n, indexed_paths_to_simplify)
    203 
    204     nodes_to_remove = set()

~/genet/genet/utils/simplification.py in _assemble_path_data(n, indexed_paths_to_simplify)
     95         return_d[k] = {
     96             'path': path,
---> 97             'link_data': _extract_edge_data(n, path),
     98             'node_data': _extract_node_data(n, path),
     99             'nodes_to_remove': path[1:-1]

~/genet/genet/utils/simplification.py in _extract_edge_data(n, path)
     72         # get edge between these nodes: if multiple edges exist between
     73         # them - smoosh them
---> 74         for multi_idx, edge in n.graph[u][v].items():
     75             for key in edge:
     76                 if key in edge_attributes:

~/genet/venv/lib/python3.7/site-packages/networkx/classes/coreviews.py in __getitem__(self, name)
     79 
     80     def __getitem__(self, name):
---> 81         return AtlasView(self._atlas[name])
     82 
     83     def copy(self):

KeyError: '1859186028'

Losing vehicle ids

veh ids are not maintained from an input schedule. It would be nice to maintain them.

Oneway attribute should be considered an optional flag in network link XML elements

GeNet's network simplification assumes that all MATSim network links have the oneway attribute flag, and throws an error when this is not the case. For example, when simplifying Matesto's test town network:

...
2021-02-12 14:26:37,796 - Processing links for all paths to be simplified
2021-02-12 14:26:37,804 - Adding new simplified links
2021-02-12 14:26:37,862 - Generated 0 link ids.
2021-02-12 14:26:37,873 - Added 6 links
2021-02-12 14:26:37,876 - Simplified graph: 14 to 8 nodes, 26 to 14 edges
2021-02-12 14:26:37,876 - Updating the Schedule
2021-02-12 14:26:37,877 - Updated Stop Link Reference Ids
2021-02-12 14:26:37,878 - Updated Network Routes
2021-02-12 14:26:37,878 - Simplification resulted in 18 links being simplified.
2021-02-12 14:26:37,909 - Writing working-dir/genet/network.xml
Traceback (most recent call last):
  File "simplify_network.py", line 68, in <module>
    n.write_to_matsim(output_dir)
  File "/genet/core.py", line 1248, in write_to_matsim
    matsim_xml_writer.write_matsim_network(output_dir, self)
  File "/genet/outputs_handler/matsim_xml_writer.py", line 84, in write_matsim_network
    link_attributes = prepare_link_attributes(deepcopy(link_attribs))
  File "/genet/outputs_handler/matsim_xml_writer.py", line 58, in prepare_link_attributes
    validate_link_data(link_attributes)
  File "/genet/validate/network_validation.py", line 8, in validate_link_data
    raise AttributeError('Attribute: {} missing from link: {}'.format(necessary_link_attrib, link_attributes))
AttributeError: Attribute: oneway missing from link: {'id': '1-3', 'modes': {'bus', 'cycle', 'car'}, 'permlanes': 1.0, 'capacity': 1000.0, 'freespeed': 10.0, 'to': '3', 'from': '1', 'length': 500.0}
ERROR: Step 'genet-testtown12' failed ('1') !

Where the oneway attribute is present, it should be preserved in the simplified network, but GeNet should not raise an error when it is missing in the source link.

To reproduce the above effect, use the following GeNet script and parameters:

    "simplify_network.py",
    "-n", "matsim12-test-town/network.xml",
    "-s", "matsim12-test-town/transitschedule.xml",
    "-p", "epsg:27700",
    "-od", "working-dir/genet"

MATSim's network DTD describes the oneway attribute thus:

<!ELEMENT link (attributes?)>
<!ATTLIST link id          CDATA #REQUIRED
               from        CDATA #REQUIRED
               to          CDATA #REQUIRED
               length      CDATA #REQUIRED
               freespeed   CDATA #REQUIRED
               capacity    CDATA #REQUIRED
               permlanes   CDATA #REQUIRED
               oneway      CDATA #FIXED    "1"
               origid      CDATA #IMPLIED
               type        CDATA #IMPLIED
               modes       CDATA "car">

Logging message bug

A very minor issue, very similar incorrect logging messages in 2 functions:

  1. logging.info(f'Removed Services with IDs `{service_id}`, and Routes: {route_ids}')

  2. logging.info(f'Removed Routes with IDs {route_ids}, to Services `{service_id}`.')

Should use service_ids not service_id to list all services that have been removed/ from which routes have been removed

(In the second case, might make more sense for the message to say ...belonging to Services {service_ids}

Check attributes for negative values

I have some MATSim warnings for a network that passed GeNet validation:

2021-06-24T18:47:40,300  WARN LinkImpl:120 capacity=-600.0 of link id 7857587829102305401_7857587832100126133 may cause problems
2021-06-24T18:47:40,300  WARN LinkImpl:122  Future occurences of this logging statement are suppressed.
2021-06-24T18:47:40,301  WARN LinkImpl:138 permlanes=-1.0 of link id 7857587829102305401_7857587832100126133 may cause problems

Should we check for negative permlanes and capacities?

Check PT headway values in validation

Add checks for negative or 0 headway values for PT.

(Negative makes no sense, the order trips does not matter and we can always re-order by time)

Make processes requiring new link ID assignment deterministic

Right now processes in GeNet such as:

  • splitting a link
  • simplification

result in links being added, and running the same process twice gives different IDs to different links. The link IDs are currently the smallest integers that are not currently being used in the network - so the method generating IDs should already be giving the same set of IDs (given the exact same set-up and inputs).

The problems I see:

  • when these IDs are distributed to the different links that are being added there is no order to who gets which ID - hence different answer each time
  • we sometimes make small changes, it would be good to have more meaningful indexing that would persist even when input/setup are not entirely the same

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.