lieu is a Python library for deduping venues and addresses around the world using libpostal's international address normalization.
This is a prototype, and various aspects, particularly the locally-sensitive hashing and similarity/dupe-checking algorithms, will be moved into the C layer in libpostal proper after the implementation is tested/settled here.
pip install git+https://github.com/openvenues/lieu
Note: libpostal and its Python binding are required to use this library, setup instructions here.
Inputs are expected to be GeoJSON files. The command-line client works on both standard GeoJSON (wrapped in a FeatureCollection) and line-delimited GeoJSON, but for Spark/EMR the input must be line-delimited GeoJSON so it can be effectively split across machines.
WoF and OSM field names are supported in this project (again: it's a prototype and may change) and are mapped to libpostal's schema.
At present, all venues are required to have the following properties:
- name: the venue/company/professional's name
- street: street name, reasonable in most countries but that assumption may need to be relaxed for e.g. Japan (where "suburb" would be the closest thing to street)
- house_number: needs to be parsed out into its own field. If not, feel free to use libpostal's parser to extract a house number from the street field.
Each record must also have a valid lat/lon (i.e. not Null Island) although it's only used in blocking (grouping similar items to narrow down the number of pairwise checks), so candidate pairs within ~2-3km of each other can still be considered dupes if their names and addresses match. This should work reasonably well for real-world place data where the locations may have been recorded with varying devices and degrees of precision.
The dedupe_geojson command-line tool will be installed in the environment's bin dir and can be used like so:
dedupe_geojson file1.geojson [file2.geojson ...] -o /some/output/dir [--address-only]
It's also possible to dedupe larger/global data sets using Apache Spark and AWS ElasticMapReduce (EMR). Using Spark/EMR should look and feel pretty similar to the command-line script (thanks in large part to the mrjob project from David Marin from Yelp). However, instead of running on your local machine, it spins up a cluster, runs the Spark job, writes the results to S3, shuts down the cluster, and optionally downloads/prints all the results to stdout. There's no need to worry about provisioning the machines or maintaining a standing cluster, and it requires only minimal configuration.
To get started, you'll need to create an Amazon Web Services account and an IAM role that has the permissions required for ElasticMapReduce. Once that's set up, we need to configure the job to use your account:
cd scripts/jobs
cp mrjob.conf.example mrjob.confOpen up mrjob.conf in your favorite text editor. The config is a YAML file and under runners.emr there are comments describing the few required fields (e.g. access key and secret, instance types, number of instances, etc.) and some optional ones (AWS region, spot instance bid price, etc.)
The example config includes a sample of the configuration used for deduping the global SimpleGeo data set (with the number of instances scaled back). The full run used 18 r3.2xlarge machines (num_core_instances=18), an r3.xlarge for the master instance, and the following values for the jobconf section of the config:
| jobconf option | value |
|---|---|
| spark.driver.memory | 16g |
| spark.driver.cores | 3 |
| spark.executor.instances | 36 |
| spark.executor.cores | 4 |
| spark.executor.memory | 30g |
| spark.network.timeout | 900s |
These values should be adjusted depending on the number and type of core instances.
Data should be on S3 as line-delimited GeoJSON files (i.e. not part of a FeatureCollection, just one GeoJSON feature per line) in a bucket that your IAM user can access.
Once the config values are set and the data are on S3, usage is simple:
python dedupe_geojson.py -r emr s3://YOURBUCKET/some/file [more S3 files ...] --output-dir=s3://YOURBUCKET/path/to/output/ --no-output --conf-path=mrjob.confNote: if you want the output streamed back to stdout on the machine running the job (e.g. your local machine), remove the --no-output option.
The output is a per-line JSON response which wraps the original GeoJSON object and references any duplicates. Note that here the original WoF GeoJSON properties have been simplified for readability, indentation has been added, and the addresses from SimpleGeo were parsed with libpostal as a preprocessing step to get the addr:housenumber and addr:street fields (which are not part of the original data set). Here's an example of a duplicate:
{
"is_dupe": true,
"object": {
"geometry": {
"coordinates": [
-122.406645,
37.785415
],
"type": "Point"
},
"properties": {
"addr:full": "870 Market St San Francisco CA 94102",
"addr:housenumber": "870",
"addr:postcode": "94102",
"addr:street": "Market St",
"lieu:guid": "1968d59a119e442fa9c66dc9012be89d",
"name": "Consulate General Of Honduras"
},
"type": "Feature"
},
"possibly_same_as": [
{
"classification": "needs_review",
"explain": {
"name_dupe_threshold": 0.9,
"name_review_threshold": 0.7,
"type": "venue",
"with_unit": false
},
"is_canonical": true,
"object": {
"geometry": {
"coordinates": [
-122.406645,
37.785415
],
"type": "Point"
},
"properties": {
"addr:full": "870 Market St San Francisco CA 94102",
"addr:housenumber": "870",
"addr:postcode": "94102",
"addr:street": "Market St",
"lieu:guid": "d804e17f538b4307a2237dbd7992699c",
"wof:name": "Honduras Consulates",
},
"type": "Feature"
},
"similarity": 0.8511739191000001
}
],
"same_as": [
{
"classification": "likely_dupe",
"explain": {
"name_dupe_threshold": 0.9,
"name_review_threshold": 0.7000000000000001,
"type": "venue",
"with_unit": false
},
"is_canonical": true,
"object": {
"geometry": {
"coordinates": [
-122.406645,
37.785415
],
"type": "Point"
},
"properties": {
"addr:full": "870 Market St San Francisco CA 94102",
"addr:housenumber": "870",
"addr:postcode": "94102",
"addr:street": "Market St",
"lieu:guid": "ec28adce0a134cbfbaacb87e71f4ab34",
"wof:name": "Honduras Consulate General of",
},
"type": "Feature"
},
"similarity": 1.0
}
]
}Note: the property "lieu:guid" is added by the deduping job and should be retained for users who want to keep a canonical index and dedupe files against it regularly. If an incoming record already has a lieu:guid property, it has a higher priority for being considered canonical than an incoming record without said property. This way it's possible to ingest different data sets using a "cleanest-first" policy, so that the more trusted names (i.e. from a human-edited data set like OpenStreetMap) are ingested first and preferred over less-clean data sets where perhaps only the ID needs to be added to the combined record.
In Spark, the output will be split across some number of part-* files on S3 in the directory specified. They can be downloaded and concatenated as needed.
exact_dupe: addresses are not an exact science, so even the term "exact" here means "sharing at least one libpostal expansion in common". As such, "Market Street" and "Market St" would be considered exact matches, as would "Third Avenue" and "3rd Ave", etc. For street name/house number, we require this sort of exact match, but more freedom is allowed in the venue/business name. If both the venue name and the address are exact matches after expansion, they are considered exact dupes.
likely_dupe: a likely dupe may have some minor misspellings, may be missing common words like "Inc" or "Restaurant", and may use different word orders (often the case for professionals such as lawyers e.g. "Michelle Obama" might be written "Obama, Michelle").
needs_review: these entries might be duplicates, and have high similarity,b ut don't quite meet the threshold required for classification as a likely dupe which can be automatically merged. If all of an entry's potential dupes are classified as "needs_review", that entry will not be considered a dupe (is_dupe=False in the response), but it may be prudent to flag the entry for a human to look at. The needs_review entries are stored as a separate list in the response (possibly_same_as) and are sorted in reverse order of their similarity to the candidate object, so the most similar entry will be listed first.
Below were some of the likely dupes extracted during a test-run using WoF/SimpleGeo and a subset of OSM venues in San Francisco (note that all of these also share a house number and street address expansion and have the same geohash or are immediate neighbors):
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent