shgold / apls

Python code to evaluate the APLS metric

APLS Metric

• Installation Instructions
• Dependencies
• License
• Useage

---

This package evaluates the Average Path Length Similarity (APLS) metric to measure the difference between ground truth and proposal graphs. The metric sums the differences in optimal path lengths between all nodes in the ground truth graph G and the proposal graph G’. This metric was used to score the SpaceNet 3 challenge. For further details, see Blog1 and Blog2.

---

Installation Instructions

pip

pip install apls

---

Dependencies

All dependencies can be found in environment.yml

---

License

See LICENSE.

---

Usage

apls.py compares ground truth and proposal graphs. The actual metric takes up a relatively small portion of this code (functions: single_path_metric, path_sim_metric_, and compute_metric). Much of the code (e.g. cut_linestring, get_closest_edge, insert_point_into_G, insert_control_points, create_graph_midpoints) is concerned with injecting nodes into graphs. We inject nodes (i.e. control points) into the graph at a predetermined distance along edges, and at the location nearest proposal nodes. Injecting nodes is essential to properly compare graphs, though it unfortunately requires quite a bit of code. graphTools.py parses geojson labels into networkx graphs for analysis. Examples for running apls.py are shown below with the attached sample data

# for further details: python apls.py --help

# 1. Compare a ground truth SpaceNet geojson with a submission csv
python apls.py --test_method=gt_json_prop_wkt --output_name=gt_json_prop_wkt \
	--truth_dir=data/gt_json_prop_wkt/ground_truth_randomized \
	--wkt_file=data/gt_json_prop_wkt/proposal/sn3_sample_submission_albu.csv \
	--im_dir=data/images

# 2. Compare a ground truth geojson with a proposal json
python apls.py --test_method=gt_json_prop_json --output_name=gt_json_prop_json \
	--truth_dir=data/gt_json_prop_json/AOI_2_Vegas_Train/spacenetroads \
	--prop_dir=data/gt_json_prop_json/AOI_2_Vegas_Train/osm 
		
# 3. Compare a ground truth geojson with a pickled proposal graph 
python apls.py --test_method=gt_json_prop_pkl --output_name=gt_json_prop_pkl \
	--truth_dir=data/gt_json_prop_pkl/ground_truth_randomized \
	--prop_dir=data/gt_json_prop_pkl/proposal \
	--im_dir=data/images

# 4. Compare a pickled ground truth graph with a pickled proposal graph 
python apls.py --test_method=gt_pkl_prop_pkl --output_name=gt_pkl_prop_pkl \
	--truth_dir=data/gt_pkl_prop_pkl/ground_truth_randomized \
	--prop_dir=data/gt_pkl_prop_pkl/proposal \
	--im_dir=data/images	

Outputs

Running apls.py yields a number of plots in the outputs directory, along with the APLS score

---

SpaceNet Training Masks

Run the create_spacenet_masks.py script to create training masks with spacenet geojsons