OSM import stopps without error message about docker

[missinglink]

You can choose which data you want to import, but yes, if you only import OSM then you'll have a similar amount of addresses as nominatim.

For our geocode.earth cloud service we also import all of openaddresses and then we additionally generate a planet-wide interpolation index which includes TIGER block ranges too.

Hopefully that brings us to pretty much complete coverage in the USA and most of Europe, depending on how willing the governments are in each country to provide open data :)

[missinglink]

It looks as though the source data has changed with whosonfirst, we're probably using a slightly older version before the change was made.

I'm going to have to close this issue because its quickly got off-topic and turned in to a general support thread.
As far as I can tell there are no bugs identified in the software and I can't afford to spend any more voluntary hours looking into this for your team.

If you need any additional help setting up a full planet build you can contact us for consultancy
https://geocode.earth/consultancy

[missinglink]

Hi @moosi did you manage to resolve the issue?
Did the import actually stop with an error or just stopped emitting log lines for a while?

[moosi]

Hey @missinglink, the problem is still present.

The import stopped without an error and the pelias cli removed the container, so it assumes the import is finished.

I did some more research on this issue. I used a bigger machine for the import (32 cores, 128 GB RAM) and the import stopped at exactly the same document count (100148808 docs). Today I will download a new .pbf and start another import to make sure my osm file is not corrupted.

I will keep you updated.

[moosi]

@missinglink I downloaded the latest osm file (planet-190527.osm.pbf) and checked the md5 sum. The import stopped again after ~100 mio. documents (103504866 docs) so it is safe to say that the .pbf file is not corrupted.

[missinglink]

100 million documents sounds about right for openstreetmap.

I'm trying to understand what is the problem?
Could it be that it just finished importing or is there something that makes you suspect an error?

[moosi]

Okay, so I may got confused with the documentation saying a full planet import should be around 600 mio. documents. When I look at the current pelias dashboard it tells 538.4 mio. addresses are imported. In turn this would mean that ~440 mio. addresses are imported from openaddresses.io. Or does a single document contain more than 1 address?

[missinglink]

That sounds about right, I think openaddresses has between 400-500 million records worldwide.

A single document only contains one 'thing'.

[moosi]

In this case it seems like pelias is only usable for address search when importing osm and openaddresses.io data (e.g. compared to nominatim that only relies on osm data).

Thanks a lot for the input and the fast reply!

[mohammedayub44]

@missinglink
What was the reason and resolution for this error ? I was trying the same with OSM north-america files and I ran into the same errors. I posted an issue in OSM repo. pelias/openstreetmap#491

Can I safely ignore these "denormalize failed on relation xxxx...." errors , is it suppose to happen or I'm a missing something ?

I can see the the record count increasing by the way in elasticsearch.

Thanks !

[moosi]

@missinglink I still think there is a problem with the OSM import. I did some tests on my instance and benchmarked the results using api.geocode.earth . Whenever I hit an address that is provided by openaddresses, both return the correct result. When I hit an address that is based on openstreetmap, the api.geocode.earth API will find the address while my instance often returns a fallback (=whosonfirst result).

OSM addresses found:

• Merianstraße 22 Frankfurt
• Schillerstraße 103 Münster
• Willy-Brandt-Weg 17 Münster
• Eishövel 8 Hamburg

OSM addresses not found:

• Sebastianstraße 53 Bonn
• Meßdorfer Str. 127 Bonn
• Am Kettelerplatz 21 Bonn
• Brucknerstraße 16 München
• Kattowitzer Str. 59 München
• Rosenheimer Str. 16 München
• Josephinenstraße 7 Dresden
• Zwickauer Str. 27 Dresden
• Mohorner Str. 21 Dresden

All full-text addresses are correctly parsed into street, number and city. It seems like for some cities it is working fine, some cites are not imported at all. Also interesting: An address that returns results from openstreetmap and openaddresses using api.geocode.earth will return a fallback on my instance. Thats why I think there is a problem with the OSM import.

Do we have a chance to find out when the last successfull import of api.geocode.earth did take place?

[missinglink]

Hi @moosi, I just looked at our current config, at time of writing we are running api master-2019-05-28 and an elasticsearch snapshot generated 2019.05.21.

You mentioned in the issue description that admin-lookup is disabled, is that still the case?
If so, do you get a result for /v1/search?text=Sebastianstraße 53 (omitting the admin info)?

[missinglink]

There is another full planet build running now which we hope to have available in the next few days.

[moosi]

I did the OSM import using "planet-190527.osm.pbf" and an openaddresses download of the 6th of June. Admin lookup is enabled. Here a diff comparison for the search "/v1/search?text=Sebastianstraße 53":

https://jsoncompare.com/#!/diff/id=53fe004f1c0d35ed382ab53858f55c4c/

The address of "Bonn" is missing and the address of "Dinslaken" is differend/missing.

[missinglink]

It seems that your build is missing the county and macrocounty information.
I just checked the current geocode.earth build which is running and that information is still present in our builds:

$ curl -s 'localhost:9200/pelias/address/way%2F284476365?pretty'
{
  "_index" : "pelias",
  "_type" : "address",
  "_id" : "way/284476365",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "center_point" : {
      "lon" : 8.555305,
      "lat" : 51.54399
    },
    "parent" : {
      "continent" : [ "Europe" ],
      "country" : [ "Germany" ],
      "macrocounty_a" : [ null ],
      "country_a" : [ "DEU" ],
      "locality_a" : [ null ],
      "region_id" : [ "85682513" ],
      "county" : [ "Paderborn" ],
      "locality" : [ "Büren" ],
      "continent_a" : [ null ],
      "region_a" : [ "NRW" ],
      "macrocounty" : [ "Detmold" ],
      "county_id" : [ "102063835" ],
      "locality_id" : [ "101810347" ],
      "continent_id" : [ "102191581" ],
      "region" : [ "Nordrhein-Westfalen" ],
      "macrocounty_id" : [ "404227571" ],
      "country_id" : [ "85633111" ],
      "county_a" : [ "PD" ]
    },
    "name" : {
      "default" : "53 Sebastianstraße"
    },
    "address_parts" : {
      "zip" : "33142",
      "number" : "53",
      "street" : "Sebastianstraße"
    },
    "source" : "openstreetmap",
    "source_id" : "way/284476365",
    "layer" : "address"
  }
}

When the openstreetmap importer starts running there are a few lines output from the pip-service, is there any information in your logs regarding those two missing layers?

You should see something like this:

info: [wof-pip-service:master] macrocounty worker loaded 371 features in 0.742 seconds
info: [wof-pip-service:master] county worker loaded 40639 features in 42.127 seconds

[moosi]

This is the output of the pip-service, so they seem to load correctly:

2019-06-11T07:57:39.362Z - info: [wof-pip-service:master] starting with layers neighbourhood,borough,locality,localadmin,county,macrocounty,macroregion,region,dependency,country,empire,continent,marinearea,ocean
pip-service is now running on port 4200
2019-06-11T07:57:41.336Z - info: [wof-pip-service:master] empire worker loaded 0 features in 0.066 seconds
2019-06-11T07:57:41.926Z - info: [wof-pip-service:master] ocean worker loaded 7 features in 0.404 seconds
2019-06-11T07:57:42.696Z - info: [wof-pip-service:master] dependency worker loaded 32 features in 1.399 seconds
2019-06-11T07:57:42.898Z - info: [wof-pip-service:master] macrocounty worker loaded 23 features in 1.436 seconds
2019-06-11T07:57:43.238Z - info: [wof-pip-service:master] macroregion worker loaded 25 features in 1.77 seconds
2019-06-11T07:57:43.341Z - info: [wof-pip-service:master] marinearea worker loaded 305 features in 1.899 seconds
2019-06-11T07:57:43.354Z - info: [wof-pip-service:master] borough worker loaded 138 features in 1.799 seconds
2019-06-11T07:57:43.624Z - info: [wof-pip-service:master] continent worker loaded 8 features in 2.23 seconds
2019-06-11T07:57:52.812Z - info: [wof-pip-service:master] country worker loaded 199 features in 11.38 seconds
2019-06-11T07:59:21.467Z - info: [wof-pip-service:master] region worker loaded 4268 features in 99.938 seconds
2019-06-11T08:08:39.779Z - info: [wof-pip-service:master] county worker loaded 24845 features in 657.803 seconds
2019-06-11T08:17:43.332Z - info: [wof-pip-service:master] neighbourhood worker loaded 17726 features in 1203.126 seconds
2019-06-11T08:31:41.610Z - info: [wof-pip-service:master] localadmin worker loaded 99206 features in 2039.495 seconds
2019-06-11T08:40:23.448Z - info: [wof-pip-service:master] locality worker loaded 143249 features in 2560.039 seconds
2019-06-11T08:40:23.622Z - info: [wof-pip-service:master] PIP Service Loading Completed!!!

[orangejulius]

I was also thinking about this, and it would definitely be good to add a message to the OSM importer that indicates a successful (or unsuccessful) completion of the import process.

We discussed that in pelias/pelias#255 and it would be nice to use a standard message across all our importers as described there. If anyone wants to take a quick try at adding it, we'd be happy to help them get started with a pull request.