Comments (17)
You can choose which data you want to import, but yes, if you only import OSM then you'll have a similar amount of addresses as nominatim.
For our geocode.earth cloud service we also import all of openaddresses and then we additionally generate a planet-wide interpolation index which includes TIGER block ranges too.
Hopefully that brings us to pretty much complete coverage in the USA and most of Europe, depending on how willing the governments are in each country to provide open data :)
from docker.
It looks as though the source data has changed with whosonfirst
, we're probably using a slightly older version before the change was made.
I'm going to have to close this issue because its quickly got off-topic and turned in to a general support thread.
As far as I can tell there are no bugs identified in the software and I can't afford to spend any more voluntary hours looking into this for your team.
If you need any additional help setting up a full planet build you can contact us for consultancy
https://geocode.earth/consultancy
from docker.
Hi @moosi did you manage to resolve the issue?
Did the import actually stop with an error or just stopped emitting log lines for a while?
from docker.
Hey @missinglink, the problem is still present.
The import stopped without an error and the pelias cli removed the container, so it assumes the import is finished.
I did some more research on this issue. I used a bigger machine for the import (32 cores, 128 GB RAM) and the import stopped at exactly the same document count (100148808 docs). Today I will download a new .pbf and start another import to make sure my osm file is not corrupted.
I will keep you updated.
from docker.
@missinglink I downloaded the latest osm file (planet-190527.osm.pbf) and checked the md5 sum. The import stopped again after ~100 mio. documents (103504866 docs) so it is safe to say that the .pbf file is not corrupted.
from docker.
100 million documents sounds about right for openstreetmap.
I'm trying to understand what is the problem?
Could it be that it just finished importing or is there something that makes you suspect an error?
from docker.
Okay, so I may got confused with the documentation saying a full planet import should be around 600 mio. documents. When I look at the current pelias dashboard it tells 538.4 mio. addresses are imported. In turn this would mean that ~440 mio. addresses are imported from openaddresses.io. Or does a single document contain more than 1 address?
from docker.
That sounds about right, I think openaddresses has between 400-500 million records worldwide.
A single document only contains one 'thing'.
from docker.
In this case it seems like pelias is only usable for address search when importing osm and openaddresses.io data (e.g. compared to nominatim that only relies on osm data).
Thanks a lot for the input and the fast reply!
from docker.
@missinglink
What was the reason and resolution for this error ? I was trying the same with OSM north-america files and I ran into the same errors. I posted an issue in OSM repo. pelias/openstreetmap#491
Can I safely ignore these "denormalize failed on relation xxxx...." errors , is it suppose to happen or I'm a missing something ?
I can see the the record count increasing by the way in elasticsearch.
Thanks !
from docker.
@missinglink I still think there is a problem with the OSM import. I did some tests on my instance and benchmarked the results using api.geocode.earth . Whenever I hit an address that is provided by openaddresses, both return the correct result. When I hit an address that is based on openstreetmap, the api.geocode.earth API will find the address while my instance often returns a fallback (=whosonfirst result).
OSM addresses found:
- Merianstraße 22 Frankfurt
- Schillerstraße 103 Münster
- Willy-Brandt-Weg 17 Münster
- Eishövel 8 Hamburg
OSM addresses not found:
- Sebastianstraße 53 Bonn
- Meßdorfer Str. 127 Bonn
- Am Kettelerplatz 21 Bonn
- Brucknerstraße 16 München
- Kattowitzer Str. 59 München
- Rosenheimer Str. 16 München
- Josephinenstraße 7 Dresden
- Zwickauer Str. 27 Dresden
- Mohorner Str. 21 Dresden
All full-text addresses are correctly parsed into street, number and city. It seems like for some cities it is working fine, some cites are not imported at all. Also interesting: An address that returns results from openstreetmap and openaddresses using api.geocode.earth will return a fallback on my instance. Thats why I think there is a problem with the OSM import.
Do we have a chance to find out when the last successfull import of api.geocode.earth did take place?
from docker.
Hi @moosi, I just looked at our current config, at time of writing we are running api master-2019-05-28
and an elasticsearch snapshot generated 2019.05.21
.
You mentioned in the issue description that admin-lookup is disabled, is that still the case?
If so, do you get a result for /v1/search?text=Sebastianstraße 53
(omitting the admin info)?
from docker.
There is another full planet build running now which we hope to have available in the next few days.
from docker.
I did the OSM import using "planet-190527.osm.pbf" and an openaddresses download of the 6th of June. Admin lookup is enabled. Here a diff comparison for the search "/v1/search?text=Sebastianstraße 53":
https://jsoncompare.com/#!/diff/id=53fe004f1c0d35ed382ab53858f55c4c/
The address of "Bonn" is missing and the address of "Dinslaken" is differend/missing.
from docker.
It seems that your build is missing the county
and macrocounty
information.
I just checked the current geocode.earth build which is running and that information is still present in our builds:
$ curl -s 'localhost:9200/pelias/address/way%2F284476365?pretty'
{
"_index" : "pelias",
"_type" : "address",
"_id" : "way/284476365",
"_version" : 1,
"found" : true,
"_source" : {
"center_point" : {
"lon" : 8.555305,
"lat" : 51.54399
},
"parent" : {
"continent" : [ "Europe" ],
"country" : [ "Germany" ],
"macrocounty_a" : [ null ],
"country_a" : [ "DEU" ],
"locality_a" : [ null ],
"region_id" : [ "85682513" ],
"county" : [ "Paderborn" ],
"locality" : [ "Büren" ],
"continent_a" : [ null ],
"region_a" : [ "NRW" ],
"macrocounty" : [ "Detmold" ],
"county_id" : [ "102063835" ],
"locality_id" : [ "101810347" ],
"continent_id" : [ "102191581" ],
"region" : [ "Nordrhein-Westfalen" ],
"macrocounty_id" : [ "404227571" ],
"country_id" : [ "85633111" ],
"county_a" : [ "PD" ]
},
"name" : {
"default" : "53 Sebastianstraße"
},
"address_parts" : {
"zip" : "33142",
"number" : "53",
"street" : "Sebastianstraße"
},
"source" : "openstreetmap",
"source_id" : "way/284476365",
"layer" : "address"
}
}
When the openstreetmap
importer starts running there are a few lines output from the pip-service
, is there any information in your logs regarding those two missing layers?
You should see something like this:
info: [wof-pip-service:master] macrocounty worker loaded 371 features in 0.742 seconds
info: [wof-pip-service:master] county worker loaded 40639 features in 42.127 seconds
from docker.
This is the output of the pip-service, so they seem to load correctly:
2019-06-11T07:57:39.362Z - info: [wof-pip-service:master] starting with layers neighbourhood,borough,locality,localadmin,county,macrocounty,macroregion,region,dependency,country,empire,continent,marinearea,ocean
pip-service is now running on port 4200
2019-06-11T07:57:41.336Z - info: [wof-pip-service:master] empire worker loaded 0 features in 0.066 seconds
2019-06-11T07:57:41.926Z - info: [wof-pip-service:master] ocean worker loaded 7 features in 0.404 seconds
2019-06-11T07:57:42.696Z - info: [wof-pip-service:master] dependency worker loaded 32 features in 1.399 seconds
2019-06-11T07:57:42.898Z - info: [wof-pip-service:master] macrocounty worker loaded 23 features in 1.436 seconds
2019-06-11T07:57:43.238Z - info: [wof-pip-service:master] macroregion worker loaded 25 features in 1.77 seconds
2019-06-11T07:57:43.341Z - info: [wof-pip-service:master] marinearea worker loaded 305 features in 1.899 seconds
2019-06-11T07:57:43.354Z - info: [wof-pip-service:master] borough worker loaded 138 features in 1.799 seconds
2019-06-11T07:57:43.624Z - info: [wof-pip-service:master] continent worker loaded 8 features in 2.23 seconds
2019-06-11T07:57:52.812Z - info: [wof-pip-service:master] country worker loaded 199 features in 11.38 seconds
2019-06-11T07:59:21.467Z - info: [wof-pip-service:master] region worker loaded 4268 features in 99.938 seconds
2019-06-11T08:08:39.779Z - info: [wof-pip-service:master] county worker loaded 24845 features in 657.803 seconds
2019-06-11T08:17:43.332Z - info: [wof-pip-service:master] neighbourhood worker loaded 17726 features in 1203.126 seconds
2019-06-11T08:31:41.610Z - info: [wof-pip-service:master] localadmin worker loaded 99206 features in 2039.495 seconds
2019-06-11T08:40:23.448Z - info: [wof-pip-service:master] locality worker loaded 143249 features in 2560.039 seconds
2019-06-11T08:40:23.622Z - info: [wof-pip-service:master] PIP Service Loading Completed!!!
from docker.
I was also thinking about this, and it would definitely be good to add a message to the OSM importer that indicates a successful (or unsuccessful) completion of the import process.
We discussed that in pelias/pelias#255 and it would be nice to use a standard message across all our importers as described there. If anyone wants to take a quick try at adding it, we'd be happy to help them get started with a pull request.
from docker.
Related Issues (20)
- PeliasModelError: invalid regex test, https://yavuzgallery.com should not match /https?:\/\// HOT 2
- Self hosted planet build result deviation HOT 5
- Elasticsearch Container keeps restarting HOT 7
- Can't import pelias docker for Quebec HOT 7
- Planet pelias json config openaddresses HOT 9
- Crash / Abort in Node create_thread on Ubuntu 20.04 HOT 1
- Pelias openaddress import seem not working
- North America openaddresses missing data and incomplete
- Missing Address Layer HOT 3
- network pelias_default was found but has incorrect label com.docker.compose.network set to "" HOT 2
- Duplicate key `csv-importer`
- A network with name pelias_default exists but was not created by compose HOT 4
- Cant start docker folowing documentation commands HOT 6
- update TIGER download to use the 2023 data HOT 2
- Interpolation error HOT 3
- Full Planet Import only takes 300GB in Elastic Search
- How to run Pelias geocoder in docker with my own .pbf file data without external data HOT 5
- Error at import process HOT 2
- Add India Region to Pelias Docker Demo Projects
- Error During Import Process with Pelias for OSM and Other Data Sources HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker.