Importing data into an existing database / merging databases about libosmscout

[syyyr]

I've actually tried importing more .pbf files and the same time, and it seems that it doesn't work well for me either. Using this command:

~/git/libosmscout/build/Import/Import --typefile ~/git/libosmscout/stylesheets/map.ost --destinationDirectory ~/down/maps ~/down/czech-republic-latest.osm.pbf ~/down/poland-latest.osm.pbf

is giving me this error:

!! Import failed: File '/home/vk/down/maps/bounding.dat' - Cannot read size of file: No such file or directory

I guess this could be solved by somehow merging those two country files first and only then importing them into libosmscout.

[Karry]

Merging databases is not supported. Size of some structures in database files are computed from amount of objects. After merge indexes would need to be re-generated. I even don't want to imagine how complicated it would be...

Importing multiple OSM files is possible - but initial import step require sorted object ids. So, it may be used to import OSM data and generated contour lines (with very high ids). But it is not possible to import two OSM extracts now. It would require some changes in importer. I would rather use some osm tool (osmconvert?) to merge the files first and import the result...

Out of curiosity, what is your usecase? Can you just use multiple databases in your app?

[syyyr]

Out of curiosity, what is your usecase? Can you just use multiple databases in your app?

I have an app, that currently uses QtLocation's MapView to show maps and also draw some points on it. The map shows. This works fine. However, now I need it to also work offline. So, my idea was to download the OSM data, import it into libosmscout, and then use its Qt facilities to show the map. After a while, I might decide that I want more data, so I would download them and then supply them to the PC that needs to show maps offline. I won't have the original files anymore, so I'd like to merge the new db, with the original one. I haven't considered using more DBs at once, because I haven't familiarized myself with the C++ APIs.

That said, I am not sure if this approach makes sense. To be honest, I'm kinda struggling with this.I only really need the OSM data to give me a map and to render it in Qt. I don't know if libosmscout isn't an overkill for this. Maybe I should just prerender the tiles for the locations I want, save them and ship them with my app? I'd think that would be pretty hard on disk space. Also: I'd like the map to be fairly undetailed on the "planet" scale and detailed only in the parts (rectangles) I want.

I also had another idea: if I just want some specific places (small cities) at large detail and the rest of the world in low detail, maybe I could just filter the "planet" data and ship that. I'd say the result size wouldn't be too big and the users wouldn't know that they have data for cities they don't look that. However, I haven't had much success with this approach. I guess I need to look more into how osm data are proccessed.

[Framstag]

Just to be sure, that we have the same wording:

• A database is "unchangeable". There is no way to update a database, especially not on the device. The resources needed for this do not allow such import on the device.
• You can of course always try to "patch" on a binary level by calculating deltas and applying them. This may reduce the amount of data transferred over the need but such process it outside the library. Since libosmscout does much to reduce the database size, I expect huge diffs and a low compression factor. Some year ago I even zipped the database files, to see based on compression size where possible improvements could be.
• You can of course always create multiple (close to disjunct) disjunct databases. So if you divide a country in smaller regions, you can calculate database for each region (but based on the same version of data source) and load missing regions on demand (but they should have the same *.pbf files as source since we rely on the same ids for the same data points in different regions to get routing working).
• Regarding size and detail of database: You can always drop some type of data during import, reducing the size of the database. You can also create reduced style sheets so the database has still the size but the visible data is reduced (for special purposes).
• The "same source version" policy is mainly for location lookup and routing to work reliable cross database. If you are just rendering the constraints may be less.
• Rendering maps is high quality is no fun. So either you get tiles from somewhere (with possible restrictions regrading style etc..), use an "on-device" renderer like libosmscout, try some special solution with data tiles, render on a server (online) or do it yourself.
• My recommendation: Do not do it yourself ;-)

I hope this answers your question. Do not hesitate to ask more or deeper :-)

[syyyr]

Just to be sure, that we have the same wording:

• A database is "unchangeable". There is no way to update a database, especially not on the device. The resources needed for this do not allow such import on the device.
• You can of course always try to "patch" on a binary level by calculating deltas and applying them. This may reduce the amount of data transferred over the need but such process it outside the library. Since libosmscout does much to reduce the database size, I expect huge diffs and a low compression factor. Some year ago I even zipped the database files, to see based on compression size where possible improvements could be.

Acknowledged. Updating the database is not necessarily what I need to do and binary patching the database seems like a hassle.

• You can of course always create multiple (close to disjunct) disjunct databases. So if you divide a country in smaller regions, you can calculate database for each region (but based on the same version of data source) and load missing regions on demand (but they should have the same *.pbf files as source since we rely on the same ids for the same data points in different regions to get routing working).
• Regarding size and detail of database: You can always drop some type of data during import, reducing the size of the database. You can also create reduced style sheets so the database has still the size but the visible data is reduced (for special purposes).

Yeah, I was wondering if maybe this is the way: create just one database with enough data dropped, so that the size is feasible, even if it contains regions, that I don't necessarily want at the time. My problem is that I don't really know what kind of data I need. Right now, I have the "planet" pbf file and am able to filter it via the osmfilter program. However, I don't know what kind of data I need to filter out, so that I can, for example, render this kind of map (this is from osm.org):

I have looked at the "Map Features" OSM wiki page, but I couldn't really figure it out. Everytime I tried filtering something I either got not enough data for a nice map, or too much data and then the size got too big. It might also be that I need some special switch for osmfilter. Like, for example, I don't know if I should be using the --ignore-dependencies switch. At first I thought "probably not" because, of course, they're dependencies, but on second thought, maybe that's the thing that increases the size the most. I guess my question is this: do you have any recommendation on what kind of data I need (or what kind I need to filter) to generate a map similar to the picture I posted?

• The "same source version" policy is mainly for location lookup and routing to work reliable cross database. If you are just rendering the constraints may be less.
• Rendering maps is high quality is no fun. So either you get tiles from somewhere (with possible restrictions regrading style etc..), use an "on-device" renderer like libosmscout, try some special solution with data tiles, render on a server (online) or do it yourself.
• My recommendation: Do not do it yourself ;-)

It's true that I don't need OSM data for any kind of processing, I only really need the map. Which means that I only really need the tiles. However, I do have some restrictions:

• The tiles must be available offline. Which means I either have to render them myself, or download them from somewhere.
• If I'm gonna be rendering them, I still need to figure out the data filtering. Although I guess size won't be that big of an issue. On the other hand, pre-rendered (PNG) tiles might get a little bit larger than raw osm data. I'm not sure.
• If I'm gonna be downloading them, I'll have to find some sort of a service to download them from. I probably won't need very many tiles, but I don't know if a free service will be able to provide them. If not, I'd rather render stuff through libosmscout then.

I hope this answers your question. Do not hesitate to ask more or deeper :-)

Thank you very much!

[Framstag]

Limiting the details in the database:

• You can set in the map.ost file (referenced during import) data type to IGNORE, thus dropping them from the resulting database. You then use the type processor in the libosmscout Import, which defines clearer, simpler types than if you filter on the raw data stream. By supplying different map.ost files you could even generate different versions of database for the same region.
• OK, you libosmscout was designed for the case where you want to render tiles or maps on the device based on a (rather small) local database.
• For filtering data on the device you can further filter data during rendering. The map renderer will only fetch data that is requested in the stylesheet (*.oss file). This way you can locally render special topic maps or use different CSS files for map variants on the same database.
• So there are three ways to filter:
• Creation of a *.osm.pbf file from database using general OSM tools
• Filter during creation of libosmscout database
• Filter during rendering

Creation of above image
-This should be possible with libosmscout but take a long time since you have to load complete *.pbf files to filter 99% of content (everything but country labels) afterwards or you need special *.pbf files that already contain that information. So this has not yet been done.

What you can currently get is

from a shape file with world coastlines with each country loaded as a database on top. In this case there is one sub region of my home country - however with no visible details in this zoom level

One or multipledatabases

• libosmscout is currently not good at having oceans in its database (large areas without any data on it)
• It is thus recommended to have a database for each country (or smaller) and make the renderer render ocean without a database for it.

[Framstag]

@syyyr : What should we do with this ticket? IMHO we have answered all your questions. If you are missing something please make separate issues/feature request etc...If you have further question, please ask.

[syyyr]

Hi, thank you for all the suggestions. I was able to filter the OSM data enough, so that I get a reasonable size for the detail I want. The coastline import helped a lot. So, in the end I'm not going to need to merge libosmscout DBs.

Now, I'm going to try the integrate a mapping widget into my program. I'll open another issue if I run into any problems.

Thanks again!