onsdigital / address-index-api

Address Index is an application which resolves addresses

README

What is this repository for?

Address Index is a Play Framework (2.8.8) application which matches addresses.

The system works via large Elasticsearch (7.17.1) indices build primarily from AddressBase Premium data.

The input can be a complete address (from any source), and the system uses advanced data science techniques to determine the most likely matching AddressBase entries with UPRNs (Unique Property Reference Numbers).

Addresses can be matched one at a time or in batches.

Additional functions exist for postcode searching and partial address string matching for typeaheads.

Plans to deploy the application as a service available to all members of the Public Sector Mapping Agreement have been put on hold for the duration of the Census test. The support team are awaiting a decision on the future of this.

How do I get a sneek peak of the API?

Docker images have been created which give a feel for the API. The Docker images include an Elasticsearch 7.17.1 cluster prebuilt with the required Address Index indices. The indices are a subset of AddressBase (the Exeter subset). Another Docker image contains a version of the API that will work with the Elasticsearch indices. We have recently added a third image to deploy the improved Python UI. The docker-compose.yml file also contains a Kibana and Cerebro service to view the cluster. Unfortunately none of the current images are publicly available at the moment, but we hope to remedy this soon. To get started:

1. Run docker-compose up on https://github.com/ONSdigital/address-index-api/blob/develop/docker-compose.yml

2. The cluster status can be viewed with either Cerebro or Kibana:

    Cerebro: http://localhost:1234
    and then http://es:9200
   
    Kibana: http://localhost:5601
   

3. To run API calls open a browser or API testing app, Postman etc.

    http://localhost:9001/
   

   Endpoints can be found in the Swagger definition /openapi/swagger.json. View as HTML with examples here:

   https://github.com/ONSdigital/address-index-api/blob/develop/api-definitions

   The ai-swagger.json can be copied into the Swagger Editor to view it:

   https://editor.swagger.io/

4. The UI points to the local API and is available on

    http://localhost:5000/
   

How do I get set up for development?

1. Required Installations

   • Java 11
   • sbt 1.5.5
   • Scala 2.13.6
   • Elasticsearch 7.17.1
   • An IDE such as IntelliJ is recommended

2. Create Project from GitHub (IntelliJ shown as example)

   • File, New, Project From Version Control, GitHub
   • Git Repository URL - select https://github.com/ONSdigital/address-index-api
   • Parent Directory: any local drive, typically your IdeaProject directory
   • Directory Name: address-index-api or address-index-data
   • Clone

   The references in the build.sbt are used to draw down additional components

3. Run

   • The project consists of an assembly of several subprojects - server, parsers, model and demo-ui

   • The list of sub projects can be seen by running sbt projects from the root of the project.

   • The list contains the project IDs that must be used for all sbt commands which require a Project ID to be supplied, for example:

   • sbt "project address-index-server" run

   • The application.conf of the demo-ui can point to the API on localhost or a deployed copy of the API

   • The application.conf of the server project points to an elastic search endpoint, this can be local or a server

   To run or test the demo-ui and server together on your local machine:

   Open two command windows running sbt as shown above, one for the API and one for UI

   Use run 9001 to have the API on port 9001 and run 9000 to have the UI on 9000 (i.e. from the root of the address-index-api project run the following commands : sbt "project address-index-server" "run 9001" sbt "project address-index-demo-ui" "run 9000" ) If the UI's application.conf is set to look at localhost:9001 for the API it will work.

   Note that when working on the UI you can save changes and it will autodeploy. This doesn't work with the API becuase of the CRFSuite executable. You have to exit out of sbt and rerun.

How do I run unit tests

sbt test

will run them all, or you can select a subproject, or use testOnly feature to restrict what is run.

How do I run performance tests

See Running Performance Tests

Related Repos

Address Index Data - Apache Spark job used to create the Elasticsearch index

Address Index Developers - Flask web site for API users

Address Index UI - New Flask UI to replace current demo-ui

What if I just want to use the API

See API Help and Swagger

address-index-api's Issues

Clarification on how to use the "fallback" option in partial search

As per the API docs, the "fallback" flag when set to true, means that a "Specifies whether a slow fallback query is used in the event of the main query returning no results.". I tested this API with a string in the "input" parameter, which does not have any address matching to it. It returned "addresses": [], in the result, but I don't understand how the "fallback" option comes into play here? Should we specify a different query somewhere, as the query to use on fallback?
Can the README for this API endpoint updated with more clarification?

Compile error on DebugController.scala:20:84

I ran sbt "project address-index-server" run and got this compile ERROR. There are other WARNings as well.

[error] /address-index-api/server/app/uk/gov/ons/addressIndex/server/controllers/DebugController.scala:20:84: not enough arguments for method apply: (i: Int): Char in class StringOps.
[error] Unspecified value parameter i.
[error] override def show(req: SearchRequest): String = SearchBodyBuilderFn(req).string()
[error] ^

My set up is:
Scala code runner version 2.13.6 -- Copyright 2002-2021, LAMP/EPFL and Lightbend, Inc.
sbt version in this project: 1.5.5
sbt script version: 1.9.0
openjdk version "11.0.18" 2023-01-17 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.18.0.10-1.el7_9) (build 11.0.18+10-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.18.0.10-1.el7_9) (build 11.0.18+10-LTS, mixed mode, sharing)

In ./build.sbt ->
Changed Elastic4s to 8.7.0
Changed Elasticsearch rest client version "org.elasticsearch.client" % "elasticsearch-rest-client" % "8.7.0"
My elasticsearch is 8.7.1, but there is no elasticsearch-client for 8.7.1

In ./demo-ui/conf/application.conf I have set loginRequired to false

In ./server/conf/application.conf the uri, uriFullmatch, the password, passwordCen changed to my ES password.

Unable to run docker-compose Apple M1

aj@AJs-MacBook-Pro address-index-api % docker-compose up
Creating network "address-index-api_default" with the default driver
Pulling cerebro (lmenezes/cerebro:)...
latest: Pulling from lmenezes/cerebro
f7ec5a41d630: Pull complete
faf4c47c8c61: Pull complete
810072571faf: Pull complete
395682a2fa05: Pull complete
635a13f1552d: Pull complete
Digest: sha256:95b02e46ec91c00ab33ce21f4d23bf6083d3c900257b9371a56ea218737462e5
Status: Downloaded newer image for lmenezes/cerebro:latest
Pulling kibana (docker.elastic.co/kibana/kibana:7.3.1)...
7.3.1: Pulling from kibana/kibana
48914619bcd3: Already exists
8f2fa4556827: Pull complete
168ccc398ab0: Pull complete
a69be6529845: Pull complete
cf46ff553bf1: Pull complete
5d0a0695dc7a: Pull complete
e034beaf0a5b: Pull complete
e7c5a04610ca: Pull complete
6d397b28159f: Pull complete
182945b7c9d5: Pull complete
Digest: sha256:79d778d55d89d8df99ac4410d2c0eccea65779e5f69d63b1302181f551273d2e
Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.3.1
Pulling address-index-api (thornsj/address-index-api:190420)...
190420: Pulling from thornsj/address-index-api
9a0b0ce99936: Pull complete
db3b6004c61a: Pull complete
f8f075920295: Pull complete
6ef14aff1139: Pull complete
962785d3b7f9: Pull complete
631589572f9b: Pull complete
c55a0c6f4c7b: Pull complete
c038c5c71bb7: Pull complete
c4b9d03d0ee5: Pull complete
Digest: sha256:d63fcac5ffb77de01287479b445d69085d99edcee6a4ef30782ce36598eacee2
Status: Downloaded newer image for thornsj/address-index-api:190420
Creating elasticsearch ... done
Creating address-index-api_cerebro_1           ... done
Creating kibana                                ... done
Creating address-index-api_address-index-api_1 ... done
Attaching to elasticsearch, address-index-api_address-index-api_1, address-index-api_cerebro_1, kibana
elasticsearch        | Error: could not find libjava.so
elasticsearch        | Error: Could not find Java SE Runtime Environment.
elasticsearch exited with code 2
address-index-api_1  | qemu: uncaught target signal 11 (Segmentation fault) - core dumped
cerebro_1            | WARNING: An illegal reflective access operation has occurred
cerebro_1            | WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/cerebro/lib/com.google.inject.guice-4.2.3.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
cerebro_1            | WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
cerebro_1            | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
cerebro_1            | WARNING: All illegal access operations will be denied in a future release
kibana               | {"type":"log","@timestamp":"2021-08-27T07:28:04Z","tags":["info","plugins-system"],"pid":8,"message":"Setting up [1] plugins: [translations]"}
kibana               | {"type":"log","@timestamp":"2021-08-27T07:28:04Z","tags":["info","plugins","translations"],"pid":8,"message":"Setting up plugin"}
kibana               | {"type":"log","@timestamp":"2021-08-27T07:28:04Z","tags":["info","plugins-system"],"pid":8,"message":"Starting [1] plugins: [translations]"}
cerebro_1            | [info] play.api.Play - Application started (Prod) (no global state)
cerebro_1            | [info] p.c.s.AkkaHttpServer - Listening for HTTP on /0.0.0.0:9000

addresses/multiuprn not available in docker based version

In the docker-compose based version I don't see the addresses/multiuprn route available in the /routes

It will be great if you can provide a docker file to quickly bootstrap and play

Clarification on RH API endpoints

What is the difference between the RH API and the regular API.