systopia / de.systopia.osm

CiviCRM OpenStreetmap GeoCoding-Provider

OpenStreetMap CiviCRM GeoCoder

This CiviCRM extension adds the Nominatim OpenStreetMap server as a geocoding option. Just install the extension, and activate via Administer → System Settings → Mapping and Geocoding.

Disclaimer: This CiviCRM extension requests geodata from nominatim.osm.org, a service of OpenStreetMap Foundation. We have been advised that OSM servers might not be suitable for massive data requests, but so far could not get precise information on what this exactly means. Also, we don't know what the consequences of requests too massive for the OSM infrastructure might be. Therefore we recommend that you consider using the throttling option for the "Geocode and Parse Adresses" cronjob if processing data sets containing more than 10.000 addresses.

Find the German README here.

Remark: Be careful when activating the "Geocode and Parse Addresses" scheduled job. It will try to look up all addresses that have not been geocoded yet. That, however, includes all the addresses that couldn't be successfully completed the last time - causing the same failed addresses to be queried again and again.

For most scenarios it should be sufficient to run the job once, since newly entered addresses and address changes should be automatically geocoded.

Links:

• Open-Street-Map-Project
• Open-Street-Map-Server
• API specs

Documentation

• EN: https://docs.civicrm.org/osm/en/latest (automatic publishing)

de.systopia.osm's Issues

makeRequest() method sets wrong parameter variable

See my comment on this commit:
545a65c#r78255978

The function parameter should be $url.

Support Question: High level usage - Can I populate a 'voting district' field for each user based on their address?

Assuming I go into the OpenStreetMaps open source mapper and define voting districts for my local town?

How would this workstream look in CiviCRM?

sync and extend english and german docs

sync english and german docs

internal-reference-5174

Not available for CiviCRM 4.7

Hi,

Please can you update this and release it for CiviCRM 4.7?

Thanks,

John

Smarter Geocoding Cron-Job

The standard CiviCRM geocoding cron-job has a variety of drawbacks:

1. it will always start from contact 0, unless you set a fixed parameter - which is not handy for a cron job
2. it will send all addresses that have not been geocoded, even if they have incomplete information. over. and over. and over.
3. the OSM geocoder will not send incomplete address data, yet the throttle parameter would still apply
4. the throttle parameter should be a setting, currently it's hardcoded to 5s
5. the manual_geo_code attribute is ignored

We propose a replacement cron job with the following features:

1. the job can be configured to skip addresses that have certain attributes missing
2. the job has a configurable throttle
3. the job has a max amount of requests per call
4. it will cycle through the contact list, i.e. it will remember the contact_id where it left off and continue there the next time.
5. the manual_geo_code attribute is respected

Array and string offset access syntax with curly braces is deprecated

Using php 7.4 I'm getting the error message above. This seems to be coming from line 133 in osm.civix.php. I have swapped

if ($entry{0} == '.') {

with

if ($entry[0] == '.') {

Which I think has fixed it

Enhance Cronjob so it can also standardise Addresses

One possible future feature is that the extension will be able standardize addresses when editing contacts. An example is, if you enter the abbreviation "Teststr." in the contact edit mask, a successful Geocoding request returning "Teststraße" will autocorrect your data.

If this feature is implemented, the Cronjob should be enhanced in a way that it can standardize addresses for which a successful request has been made as well. Most likely, this would also imply that one needs additional setting for the cronjob: "standardize": yes/no; maybe also a setting "standardize all contacts": yes/no (for initial standardization).

Truncate coordinates

Unfortunately there's a bug in CiviCRM where geo coordinates longer then 12 characters (as a string) can cause trouble: CRM-17856.

Since this hasn't been fixed yet, I suggest we budge and truncate our coordinates. We can still take it back once it's sorted out in CiviCRM.

Postal Code not working properly with Nominatim

The following seems to be a problem of nominatim, respective the data of OSM. However, there might be a workaround in the OSM extension:

The problem:
Very often, the address is not found, when there is a postal code within the address.
E.g. in the Logfile, I see the following line:

Jul 01 12:46:09 [info] Geocoding failed. No results for: http://nominatim.openstreetmap.org/search?format=json&street=Platz+der+Republik+1&city=Berlin&postalcode=11011&country=Germany&addressdetails=1

I analyzed this on the nominatim UI at http://nominatim.openstreetmap.org and entered

Platz der Republik 1 Berlin 11011

This gave no search result.
Then I left off the postal code:

Platz der Republik 1 Berlin

and voilà, the address was identified.

As a workaround, I commented out the respective lines of code in OpenStreetMapCoding.php:
113 if (CRM_Utils_Array::value('postal_code', $values)) {
114 $params['postalcode'] = $values['postal_code'];
115 }

Now, geocoding seems to work much better, and many addresses that previously weren't geocoded, are working well now.

Any ideas about that?

Handle 429 status

Currently, the service we use (https://nominatim.openstreetmap.org) returns HTTP status code 429 if queried to often, but this isn't properly handled in the current version - the look-up keeps on going. Also, after a while the server seems to block the requests entirely, causing the following error:

May 17 22:58:23  [info] $backTrace = #0 /var/www/sites/all/modules/civicrm/CRM/Core/Error.php(777): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 [internal function](): CRM_Core_Error::exceptionHandler(Object(PEAR_Error))
#2 /var/www/sites/all/modules/civicrm/packages/PEAR.php(931): call_user_func((Array:2), Object(PEAR_Error))
#3 /var/www/sites/all/modules/civicrm/packages/PEAR.php(566): PEAR_Error->PEAR_Error("", 0, 16, (Array:2), NULL)
#4 /var/www/sites/all/modules/civicrm/packages/Net/Socket.php(156): PEAR->raiseError("", 0)
#5 /var/www/sites/all/modules/civicrm/packages/HTTP/Request.php(737): Net_Socket->connect("ssl://nominatim.openstreetmap.org", 443, NULL, NULL, NULL)
#6 /var/www/sites/default/civicrm-extensions/de.systopia.osm/CRM/Utils/Geocode/OpenStreetMapCoding.php(139): HTTP_Request->sendRequest()
#7 /var/www/sites/all/modules/civicrm/CRM/Utils/Address/BatchUpdate.php(183): CRM_Utils_Geocode_OpenStreetMapCoding::format((Array:5), TRUE)
#8 /var/www/sites/all/modules/civicrm/CRM/Utils/Address/BatchUpdate.php(108): CRM_Utils_Address_BatchUpdate->processContacts(Object(CRM_Core_Config), TRUE, FALSE)
#9 /var/www/sites/all/modules/civicrm/api/v3/Job.php(155): CRM_Utils_Address_BatchUpdate->run()
[...]

What needs to be done:

1. Properly detect and process 429 calls - what exactly should happen here needs to be discussed, but at least we should abort the batch update process if there is one.
2. Remove/Reduce Geocoding failed. Address data is incomplete. and Geocoding failed, invalid response code 429 spamming of error log.
3. Update readme file accordingly.

Address standardisation interface

CiviCRM has some "hacks" to include address standardisation, but no proper interface for it. We suggest to draft such an interface and create our services against it. The push this interface into CiviCRM core.

This is particularly interesting wrt ticket #4 and #6

Incompatible with CiviCRM 5.27+ (was: "Error geocoding")

I am getting the following error when saving an address in CiviCRM:

Warning: require_once(HTTP/Request.php): failed to open stream: No such file or directory in require_once() (regel 140 van /home/xxx/domains/xxx.nl/public_html/sites/default/files/civicrm/ext/de.systopia.osm/CRM/Utils/Geocode/OpenStreetMapCoding.php).

What am I missing?

Drupal 7.72
CiviCRM 5.27.1
OSM 1.2.1

Error on address (country) submission

I have an update your details profile which has started to throw a fatal error:

Parse error: syntax error, unexpected 'new' (T_NEW) in /usr/local/php73/lib/php/HTTP/Request.php on line 412

When I remove the address fields it works as expected (I focused in on the address fields as it would save a phone number even when reporting the error). On further testing it was including Country / County that triggered the error.

Through a process of elimination (enabling / disabling extensions) I have narrowed it down to OpenStreetMap Geocoding (de.systopia.osm).

I am using WordPress 5.4.2, CiviCRM 5.27.1, PHP Version 7.3.17. The most significant change was recently upgraded to CiviCRM 5.27.1

Add dummy encoder

Add a dummy encoder that does nothing, b/c the mapping provider is not activated if no coder is. This is for the scenario that you get your geodata from an external source.

Proximity search to a distance of 20 miles from 3042, Victoria, Australia: We could not geocode the destination address.

Well, it used to work with Google but seems it doesn't if we try Advanced search with country+state/province+postal code. Just wondering if it is something OSM - related ( we don't have OSM API key nor we are sure is required ) or it comes from the extension. Please advise. Thanks!

CiviCRM 5.66 with PHP 8.1 - crashes

CiviCRM 5.66 with PHP 8.1 - not sure which one is the problem but when adding a new contact with a valid address I get blank screen with 'website has an error'. If I add a contact then later add an address I get the 'cannot contact the server' message. Unfortunately there are no error messages in the logs. Both problem are 100% solid.

UK address lookups fail because CiviCRM StateProvince field doesn't match the state parameter in Open Street Map

This extension passes along the value of the StateProvince field in CiviCRM as the 'state' parameter when doing address lookups, but this doesn't always match how addresses are registered in Open Street Map. For the UK, the default CiviCRM configuration is to populate the StateProvince field with UK counties, but in Open Street Map, UK states are instead defined as England, Scotland or Wales, and for some reason, this causes address lookups to fail. Often no addresses are found when the county is present in the address, or occasionally a completely wrong address is found.

On my own site, I have patched the extension to ignore the StateProvince field for UK addresses, but we should maybe look for a more sophisticated solution for this issue.

JSON is decoded as object but worked with as array

in CRM_Utils_Geocode_OpenStreetMapCoding::makeRequest() the geocoded API response is retrieved as a JSON string and decoded, but the decoding does not use the associate array option. Later in the code the object is referenced as an array rather than an object, which throws errors.

Geocoding for towns/cities/regions

@lcdservices writes in #12:

FYI -- I ran into this and found that it will not geocode the destination address without a street address. We were trying to trigger a proximity searching using only a city/state/country.
From the nominatim site it does seem to support geocoding with just those values, so I'm not sure why the extension requires the street address.

That's interesting, I didn't know that. The encoder actively suppresses that HERE, but I can't see a reason to lift that - ideally with a configuration option to not change the behaviour for current users. Would give it a try?

ping @Upperholme

Nominatim sends "Network is unreachable (101)"

If you exceed your quota with Nominatim you get a reply giving an appropriate error message. But apparently if you still keep on contacting them excessively, you will eventually be blocked completely.

Unfortunately that causes an error in the extension that ultimately stops the address geocoding and also the currently ongoing address change.

Here's a typical stack trace:

#0 /var/www/sites/all/modules/civicrm/CRM/Core/Error.php(897): CRM_Core_Error::backtrace("backTrace", TRUE)
#1 [internal function](): CRM_Core_Error::exceptionHandler(Object(PEAR_Error))
#2 /var/www/sites/all/modules/civicrm/packages/PEAR.php(931): call_user_func((Array:2), Object(PEAR_Error))
#3 /var/www/sites/all/modules/civicrm/packages/PEAR.php(566): PEAR_Error->PEAR_Error("Network is unreachable", 101, 16, (Array:2), NULL)
#4 /var/www/sites/all/modules/civicrm/packages/Net/Socket.php(156): PEAR->raiseError("Network is unreachable", 101)
#5 /var/www/sites/all/modules/civicrm/packages/HTTP/Request.php(737): Net_Socket->connect("ssl://nominatim.openstreetmap.org", 443, NULL, NULL, NULL)
#6 /var/www/sites/default/civicrm-extensions/de.systopia.osm/CRM/Utils/Geocode/OpenStreetMapCoding.php(140): HTTP_Request->sendRequest()
#7 /var/www/sites/all/modules/civicrm/CRM/Core/BAO/Address.php(397): CRM_Utils_Geocode_OpenStreetMapCoding::format((Array:10))
#8 /var/www/sites/all/modules/civicrm/CRM/Core/BAO/Address.php(152): CRM_Core_BAO_Address::fixAddress((Array:10))
#9 /var/www/sites/all/modules/civicrm/api/v3/Address.php(75): CRM_Core_BAO_Address::add((Array:10), TRUE)
#10 /var/www/sites/all/modules/civicrm/Civi/API/Provider/MagicFunctionProvider.php(89): civicrm_api3_address_create((Array:10))
#11 /var/www/sites/all/modules/civicrm/Civi/API/Kernel.php(96): Civi\API\Provider\MagicFunctionProvider->invoke((Array:9))
#12 /var/www/sites/all/modules/civicrm/api/api.php(23): Civi\API\Kernel->run("Address", "create", (Array:9), NULL)
#13 /var/www/sites/all/modules/civicrm/api/api.php(43): civicrm_api("Address", "create", (Array:9))
[...]

Update civix

Address normalisation Job

Create a cron jon, at least as smart as #2, that will go through existing addresses and see if they can be normalised wrt to spelling, if a unique result can be obtained from the OSM service.