mpickering / rg-map

Make a map of all georeferenced maps on routegadget

This pipeline generates the static site which has a map of all the orienteering maps in the UK which have been uploaded to routegadget with georeferencing information.

./gen-all <OUTPUT-DIR>

Will generate results/maps.html and output/tiles.

rg-map's Issues

Read directly from vrt when making leaflet

At the moment we read the old image and the old world file to work out where to place the marker. We should just read the vrt?

Related to #3 as once that is implemented, we would have to plot multiple markers per vrt.

Escaping is incorrect for event names

For example, see Tor Hill Woods - Wells, the & is displayed incorrectly.

Automatically set tms=true and maxZoomLevel

This is blocked by a patch to folium.

Build Error

nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-21.05.tar.gz myShell.nix
...
Building test suite 'unit-tests' for funflow-1.5.0..
[1 of 5] Compiling Control.Arrow.Async.Tests ( test/Control/Arrow/Async/Tests.hs, dist/build/unit-tests/unit-tests-tmp/Control/Arrow/Async/Tests.o, dist/build/unit-tests/unit-tests-tmp/Control/Arrow/Async/Tests.dyn_o )
[2 of 5] Compiling Funflow.ContentStore ( test/Funflow/ContentStore.hs, dist/build/unit-tests/unit-tests-tmp/Funflow/ContentStore.o, dist/build/unit-tests/unit-tests-tmp/Funflow/ContentStore.dyn_o )
[3 of 5] Compiling Funflow.SQLiteCoordinator ( test/Funflow/SQLiteCoordinator.hs, dist/build/unit-tests/unit-tests-tmp/Funflow/SQLiteCoordinator.o, dist/build/unit-tests/unit-tests-tmp/Funflow/SQLiteCoordinator.dyn_o )

test/Funflow/SQLiteCoordinator.hs:56:3: error:
    • The constructor ‘Process.OpenExtHandle’ should have 2 arguments, but has been given 3
    • In the pattern: Process.OpenExtHandle h _ _
      In a case alternative:
          Process.OpenExtHandle h _ _ -> return (Just h)
      In the second argument of ‘($)’, namely
        ‘\case
           Process.OpenHandle h -> return (Just h)
           Process.OpenExtHandle h _ _ -> return (Just h)
           Process.ClosedHandle {} -> return Nothing’
   |
56 |   Process.OpenExtHandle h _ _ -> return (Just h)
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^

builder for '/nix/store/c9apzjda73d5bp0ddf4kyjczin2ymk4d-funflow-1.5.0.drv' failed with exit code 1
cannot build derivation '/nix/store/cd6grdjfkzc79gmqws8raw1m61n3yziz-ghc-8.10.4-with-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/cd6grdjfkzc79gmqws8raw1m61n3yziz-ghc-8.10.4-with-packages.drv' failed

So I am trying

funflow = super.haskell.lib.dontCheck (hself.callHackage "funflow" "1.5.0" {});

Run Error

*Main> main
main
[2021-08-04 06:56:58][FFExecutorD.executeLoop][Info][MacBook-Pro][PID 51384][ThreadId 59][funflow-1.5.0-GuwgHTnEu2V6hYgOjVs63C:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:183:9] Initialising connection to coordinator.
[2021-08-04 06:56:59][FFExecutorD.executeLoop.process][Info][MacBook-Pro][PID 51384][ThreadId 59][processId:CreateProcess {cmdspec = RawCommand "perl" ["/Users/dom/rg-map/funflow-example/store/item-8bccabdf0a55dc8a3ac4e73781ca09d8e469a3b1744b6415e432e0ce64e7f53a/get-world-files","/Users/dom/rg-map/funflow-example/store/pending-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/"], cwd = Just "/Users/dom/rg-map/funflow-example/store/pending-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/", env = Just [("NIX_PATH","nixpkgs=https://github.com/mpickering/nixpkgs/archive/547eef61effa37f10c49caebc771c3fceb16f14a.tar.gz")], std_in = Inherit, std_out = UseHandle {handle: /Users/dom/rg-map/funflow-example/store/metadata/hash-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/stdout}, std_err = UseHandle {handle: /Users/dom/rg-map/funflow-example/store/metadata/hash-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/stderr}, close_fds = True, create_group = False, delegate_ctlc = False, detach_console = False, create_new_console = False, new_session = False, child_group = Nothing, child_user = Nothing, use_process_jobs = False}][task:6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0][funflow-1.5.0-GuwgHTnEu2V6hYgOjVs63C:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:127:9] Executing
env: nix-shell: No such file or directory
[2021-08-04 06:56:59][FFExecutorD.executeLoop][Warning][MacBook-Pro][PID 51384][ThreadId 59][task:6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0][funflow-1.5.0-GuwgHTnEu2V6hYgOjVs63C:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:211:19] Task failed
FAILED: External task failed to construct item 'ContentHash "6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0"'. Task info: KnownTask (Failed (ExecutionInfo {_eiExecutor = Executor "MacBook-Pro", _eiElapsed = TimeSpec {sec = 0, nsec = 42243571}}) 127) stdout: Just "/Users/dom/rg-map/funflow-example/store/metadata/hash-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/stdout" stderr: Just "/Users/dom/rg-map/funflow-example/store/metadata/hash-6609deeebc21a5cde43b7eec9353a80f269cc217085449e6da745ec9888a9ab0/stderr" Task: ExternalTask {_etCommand = "perl", _etParams = [Param [ParamPath (IPItem (Item {itemHash = ContentHash "8bccabdf0a55dc8a3ac4e73781ca09d8e469a3b1744b6415e432e0ce64e7f53a"})),ParamText "get-world-files"],Param [ParamOut]], _etEnv = EnvExplicit [("NIX_PATH",Param [ParamText "nixpkgs=",ParamText "https://github.com/mpickering/nixpkgs/archive/547eef61effa37f10c49caebc771c3fceb16f14a.tar.gz"])], _etWriteToStdOut = NoOutputCapture}

Add a buffer around the raster groups

When two maps are close together but not overlapping, artefacts are created at certain zoom levels. If we add a buffer to each map (say 2km) then the number of these artefacts might reduce. This will create bigger groups though which means more blank tiles.

Enforce better discipline in the output directory

For example, the cached images should be in a different directory to the converted images.

Set transparency of the images

Perhaps opacity=0.7 would be best.

Use png rather than jpg

Using png rather than jpg will mean that gdalwarp can set the transparency properly when it rotates images. This currently causes artifacts in the blacks.

Fix the problem of tiling rasters

The problem is that we make tiles for each map individually and then older ones just override earlier ones. We really want to make vrts for maps which overlap which will make much nicer looking tiles.

1. Work out the extent of each image.
2. Group images together which overlap (finding the islands). Make a VRT for each island (note that an island can have just 1 image in it).
3. Make tiles for each of these VRTs

I am unsure how to efficiently achieve point 2 currently.

Investigate difference between tiling levels

At first I used zoom level 16 which produced good results but I wonder if using 12 or 13 would produce
similarly good results.