arnau / addresspack Goto Github PK

Hi,

Love this project. Haven't actually tried a data load yet but I was thinking about the write performance issues.

I see in the docs:

By default, a transaction is commited every 10 files (i.e. every 10,000,000 inserts)

This seems like a large transaction - have you tried experimenting with a much lower commit size, say 500/1000 inserts per transaction?

If that doesn't offer much improvement I did have another idea. I was wondering if splitting up the schema into several separate sqlite3 databases would help - that would enable writing to those databases parallelly.

I believe you can attach up to 10 databases with sqlite3 - so you could then query by attaching the databases together.

Once the data is loaded, a one liner could be provided for the user to execute like so to launch sqlite with everything attached in one go, e.g.:
sqlite3 database1 -cmd "$(cat attach-tables.sql)"
Where attach-tables.sql could be:
ATTACH DATABASE basic_land_property_unit AS blpu; ATTACH DATABASE delivery_point AS dp; ATTACH DATABASE street AS str; -- etc etc
When using sqlite3, I think querying the data should then be fairly transparent to the end user. It helps that, so long as the table names are unique in each schema, it seems you don't have to prepend the table names with the database name.

I'm not sure what performance impact this would have at query time, performing joins across the different databases and so forth.

There is the option to merge multiple databases together after you've written to them - but this would probably be expensive and have similar performance issues to writing to a single db. In fact I imagine you may well end up in a place worse than writing to a single db...

You may be pleased to note that AddressBase Core is now available in GeoPackage, and AddressBase Premium should follow suit.

AddressBase Core has a simplified schema, suitable for many use cases.

GeoPackage is the OGC open standard that builds on SQL Lite.

Yes, I work for OS - thanks Arnau for the pointer / criticism!