digitizing points about mapedit

[timelyportfolio]

I tend to agree that repeatMode = TRUE is a more sensible default, or at least in nearly every use case that I have encountered, multiple drawing is the norm.

[mdsumner]

@yeedle I am very happy to do this, but would prefer to start from an example you provide just so it's relevant and not just a dummy example I craft. Are you happy to do that, keen to help.

[tim-salabim]

@yeedle No I haven't. Digitizing off a raster should be no problem at all (as long as the raster has CRS). Georeferencing is something that is way beyond the current state of affairs (and something that is likely beyond the scope of mapedit anyway). Btw, I would take @mdsumner up on his offer if I were you... ;-)

[mdsumner]

Oh ok, that's good - I'll check out the file you list here.

Also, be clear that I do consider it two examples, georeferencing and digitizing are both noble pursuits.

I also want to illustrate georeferencing "from scratch", I know it's possible because I am forced to do so now and then. I usually resort to Manifold which is a desktop GIS, but I'd prefer it to be done in R - note that there are several levels of "georeferencing" though. The first is "it's a solid map already, but we lost the metadata" - this is common when model or observation data get sent to a web page, or in a Word document or something like that. There you only need two points (they provide directly or indirectly the position of the corners and the size of the pixels). It can be in longlat or in some projection, and the georeference is done in that - the "native coordinate system". You often need to transform, when e.g. your two known points (control points) are recorded on the image say as long-lat graticule points, then it's the two points transformed to "native" and that gives the corners and pixel size (it's just a bit of offset-count arithmetic).

More complicated and more like "modelling" is to find known points that are not trivially related to the offset and scale of the image, that's what "rubber sheeting" is and it's a bit of a black art with many tricks, affine/rotate, polynomial fits, triangulation fits and so on. That is not the kind of georeferencing I'm thinking of here. Another kind is full-on image ortho-rectification, which really is modelling and takes into account the geometry of the camera, the motion of the platform and the very detailed topography of which the image is taken of. I'm also not thinking of that kind.

Digitizing after georeferencing is actually pretty trivial, once the image really is a map you are just reading out coordinates - but it's the interactivity, the ability to zoom in and out, make subtle changes while digitizing, the ability to snap to exact locations, use guides and include other layers to really get the drawing right. Also the ability to do the drawing in the coordinate system that makes sense, and what that is is not necessarily obvious.

So, the digitization topic is really about designing the software once we've got everything "on the map". Georeferencing is the workarounds or real full-on work required to get a data set to be a map.

The easy kind of georeferencing is about getting two points and knowing their values in "native projection" and in "native image pixel position". I think it's very important that we practice and develop and discuss a common terminology here, so if any of this makes sense or otherwise please let me know.

I have a few tools in that regard here, and that's where I'll start but they won't be useful without a bit of preparation first:

https://github.com/hypertidy/affinething/blob/master/R/affinething.R

This one sort of used this technique when I thought I wanted to download a lot of weather maps, it worked but has just sat idle since:
https://github.com/mdsumner/bomage/blob/master/R/bomage.R

Here is another one, this was a serious bit of work and taught me a lot - it's the example where you know the native projection (I could infer it because the north-up line is the central longitude, and it turns out, they all use the same central latitude and the same latitude of true scale - it's Polar Stereographic, but several instances of that at different longitudes).

https://github.com/mdsumner/asosi

I think these are fine examples, but they are going to be hard to find anyone but me that cares about them ;)

[tim-salabim]

@mdsumner I have moved this to its own issue as I think this is worth a discussion.

• returning x, y instead of sf is not a problem and straight forward to implement (I am not sure if this is worth implementing though as st_coordinates doesn't seem like much of an effort to achieve this after editing/drawing). Though the notion of drawing vs. digitizing is a discussion worthy of its own.
• in Manifold, how do you get a background map for polar LAEA? Is this a standard feature?
• the multiple add(Circle)Markers at once is something we will investigate
• geo-referencing is something that I would like to postpone until a later version of mapedit (especially after upgrade to leaflet 1.0 - in general having two maps side-by-side with differing crs seems like a major obstacle given the way things are implemented right now) as it seems a little too bold and ambitious for a first iteration, given that the interactive part is only a small part of the process. Yet, this seems like a natural progression so is definitely on our radar.

Thoughts from others more than welcome.

[mdsumner]

You're right, we can always wrap this to have a single function return just longlat, no matter what the map is.

In Manifold I actually manually georeferenced a figure in this case (a two-click control point affine), it's not a standard background. The priority here was me realizing that when I think about these workflows usually it's too distanced from the need, we had the map ready to go but my colleague had no idea how to harvest the points. We could have used project(do.call(cbind(locator())), prj, inv = TRUE) - except that we also really needed to zoom in and out. :) ( I see plainView as a very important middle-ground to the ideal of general projections in leaflet btw. )

I'm happy for georeferencing to be postponed, but I can see I can easily teach others how to do it in R if we have these simpler parts available in mapedit. I've really taken to shiny now too, I finally understand it enough to use it for real work, so hoping to embed mapedit in things more which will help me ideas-wise.

Thanks!

[timelyportfolio]

This feedback is critically important, and I really appreciate the spirit. Here are my thoughts from the geo-illiterate side of things.

• Projections - would it be helpful to have functionality in mapview or mapedit with a selectable list of standard projections so a user can easily toggle?

• Multi-draw points - will look for a plugin, or look into how to modify Leaflet.draw to allow this. I think multi-draw any feature would be very helpful as it is very annoying to press the button after drawing a single feature.

• Embedding in Shiny - check out #21. I would love to know how we might more fully use this new ability.

[mdsumner]

@timelyportfolio you must mean #21? I can't believe how simple it is, I literally copied stuff over and mostly deleted things until it worked!

I had to mapedit:::selectMod/UI but otherwise a breeze! This is great, seriously Shiny just clicked with me this week after so many years. :)

[timelyportfolio]

corrected the wrong number, and I have to fix the imports/exports so it will work out of the box. @mdsumner, glad Shiny clicked :)

[tim-salabim]

Thanks for the lively discussion, guys!

• Projections - they are possible in mapview (set native.crs = TRUE). Obviously with no background map. I don't see a generalised way of providing standard projections as in many instances tile servers for projected tiles only serve a local region. I think this should be up to the user (and taking the shiny modules approach, it should be easy for users to build a 'Antarctic digitizing app' for example). Also, sometimes (more often than not I guess) it is sufficient to load a local raster tile as background and digitize on that (which calls for a more powerful rasterImage implementation).
• Multi-draw is in my view a must-have, just as snap-to-vertices which brings up the point of leaflet 1.x again. My suggestion would be to only include any multi-draw feature if it is really easy and quick to implement. Otherwise I would rather focus on a proper implementation together with snapping on top of leaflet 1.x

[timelyportfolio]

Multi-draw is possible in Leaflet.draw with repeatMode. Here is an example enabling multi-draw for all feature types.

library(mapview)
library(mapedit)
library(leaflet.extras)
library(shiny)

lf <- mapview()@map %>%
  # all types with repeatMode enabled
  addDrawToolbar(
    polylineOptions = drawPolylineOptions(repeatMode = TRUE),
    polygonOptions = drawPolygonOptions(repeatMode = TRUE),
    circleOptions = drawCircleOptions(repeatMode = TRUE),
    rectangleOptions = drawRectangeOptions(repeatMode = TRUE),
    markerOptions = drawMarkerOptions(repeatMode = TRUE),
    editOptions = editToolbarOptions()
  )

editMap(lf)

default TRUE or FALSE

So now the question is what should be the default repeatMode - TRUE or FALSE? I think the UI with repeatMode = TRUE is a little confusing in that a user has to click cancel to stop drawing. If I used for the first time, I would be nervous that cancel would delete my drawings. I wish the button said "stop drawing" or something similar.

[tim-salabim]

can @bhaskarvk help here? I.e. is it possible to rename that button to something less confusing?

[bhaskarvk]

Usually I don't prefer to hack plugin code coz then I end up having to maintain a fork of the hacked plugin, which makes things like upgrading plugin very cumbersome.
But I'll double check if this is something easy to do.

On the issue of projections, I was thinking of having some utility functions in leaflet.extras like leaflet.epsgXYZ(baseTiles=T) which will return a leaflet instance with proper CRS attached and a corresponding base tiles attached (if baseTiles==T). That should help ease constructing leaflet instances for custom CRSes. Thoughts?

[tim-salabim]

re projections, sounds like a convenient idea. I think leaflet.extras is a better place for it than mapview or mapedit

[timelyportfolio]

@bhaskarvk, lines https://github.com/Leaflet/Leaflet.draw/blob/818af15d667dcb86d3cc42ad548b24a7358af3d4/src/Leaflet.draw.js#L84-L90 control the text "Cancel" but do not appear easily customizable, so it seems education of users will be our best approach.

[tim-salabim]

I think repeatMode = TRUE should be the default. It feels a lot more like GIS behaviour that way. I think learning how to finish digitizing is a minor thing compared to the annoyance of having to add one feature at a time.

[bhaskarvk]

Do you guys want that default in leaflet.extra's function, or something you'll take care in mapview?

[timelyportfolio]

I think this better handled in mapedit and would not want to change behavior and defaults in leaflet.extras. I have already made change as I am working on editMap modules. Thanks!

[timelyportfolio]

Does the default repeatMode satisfy the request to the extent of mapedit scope? If so, I will close, and we can work through best ways to handle projections outside of mapedit. Thanks everyone.

[yeedle]

re digitizing features: @tim-salabim did you end up creating an example of how to digitize features off or georeference a raster or pdf? I'm a gis noob so following the entire conversation is a bit of a struggle.

[yeedle]

@mdsumner I'd be more than happy to supply an example that is relevant. A while back I was trying to digitize the poll locations of several towns in Rockland County, NY, USA. The maps (as georeferenced PDFs) are available on Rockland County's GIS portal. Any one of them should serve (I hope) as a good example.

[mdsumner]

@yeedle it looks like this would involve georeferencing and digitizing, the pdf I tried has no extent or CRS. You can digitize in that space with mapedit, but presumably you'd want to georeference this first. For that we need at least the CRS so that's part of what I mean by a real example. I could harvest that somehow but I'd probably end up with a different file to the one you might choose.

I need for it to matter to someone, otherwise it's just me messing around with more incompletely specified map data (the world is full of it!).

This is the file I chose and some information about it. Do you happen to know the map projection in use?

u <- "https://geopower.jws.com/rockland/MapDownload.jsp?id=611072"
download.file(u, "somefile.pdf", mode = "wb")

I think it's baked into the file image, as this is a layout-map-sheet with carto-accoutrements so I'll have a look and report back.

This is what GDAL sees, it's not a very good GeoPDF ... though maybe there's meant to be a .pfw or .xml or similar to go with it.

system("gdalinfo somefile.pdf")
Driver: PDF/Geospatial PDF
Files: somefile.pdf
Size is 5100, 6600
Coordinate System is `'
Metadata:
  CREATION_DATE=D:20170711144020-05
  CREATOR=Esri ArcMap 10.3.0.4322
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 6600.0)
Upper Right ( 5100.0,    0.0)
Lower Right ( 5100.0, 6600.0)
Center      ( 2550.0, 3300.0)
Band 1 Block=5100x1 Type=Byte, ColorInterp=Red
Band 2 Block=5100x1 Type=Byte, ColorInterp=Green
Band 3 Block=5100x1 Type=Byte, ColorInterp=Blue

[yeedle]

You're right about that one, my apologies. I did not realize that some of the maps there are not georeferenced. The second one on the list seems to be georeferenced:

u <- 'https://geopower.jws.com/rockland/MapDownload.jsp?id=203672'
download.file(u, 'somefile.pdf', mode = 'wb')

sf::st_layers('somefile.pdf') 
#> Warning in CPL_get_layers(dsn, options, do_count): GDAL Message 1: organizePolygons() received an unexpected geometry.
#> Either a polygon with interior rings, or a polygon with less than 4 points,
#> or a non-Polygon geometry.  Return arguments as a collection.
#> Driver: PDF 
#> Available layers:
#>                             layer_name geometry_type features fields
#> 1                              Other_4                     87      0
#> 2               New_Data_Frame_Other_3                      1      0
#> 3       New_Data_Frame_Town_Boundaries                     25      0
#> 4             New_Data_Frame_Shoreline                     12      0
#> 5        New_Data_Frame_hudsonRiver_83                    151      0
#> 6  New_Data_Frame_Senatorial_Districts                      2      0
#> 7    New_Data_Frame_Assembly_Districts                     10      0
#> 8            New_Data_Frame_Background                      1      0
#> 9                              Other_2                      6      0
#> 10                        Layers_Other                      3      0
#> 11          Layers_cty_polls_2013_Anno                     65      0
#> 12           Layers_Cty_ED_2012_Anno_2                    273      0
#> 13                    Layers_<Default>                     28      0
#> 14                     Layers_Labels_E                     17      0
#> 15             Layers_Road_Shields_tab                    186      0
#> 16                     Layers_ctybound                   2630      0
#> 17                     Layers_Villages                   1076      0
#> 18                  Layers_Cty_ED_2013                    316      0
#> 19                Layers_major_roads83                   3287      0
#> 20                Layers_lakes_major83                     52      0
#> 21                  Layers_ctyparks_83                     42      0
#> 22                   Layers_st_parks83                    147      0
#> 23               Layers_hudsonRiver_83                    545      0
#> 24                      Layers_legdist                    205      0

sf::st_read('somefile.pdf', layer = 'Layers_cty_polls_2013_Anno')
#> Warning in CPL_read_ogr(dsn, layer, as.character(options), quiet, type, : GDAL Message 1: organizePolygons() received an unexpected geometry.
#> Either a polygon with interior rings, or a polygon with less than 4 points,
#> or a non-Polygon geometry.  Return arguments as a collection.
#> Reading layer `Layers_cty_polls_2013_Anno' from data source `/tmp/RtmpAGX5b9/somefile.pdf' using driver `PDF'
#> Simple feature collection with 65 features and 0 fields
#> geometry type:  POLYGON
#> dimension:      XY
#> bbox:           xmin: 576931.1 ymin: 794726.1 xmax: 658604.3 ymax: 885701.4
#> epsg (SRID):    NA
#> proj4string:    +proj=tmerc +lat_0=38.83333333333334 +lon_0=-74.5 +k=0.9999 +x_0=150000 +y_0=0 +datum=NAD83 +units=us-ft +no_defs

I understand what you're saying about it mattering to someone. At this point I'm not working on that project anymore so this specific example is not really relevant to me as it was then. I do have other maps that I need to digitize but they are unfortunately not georeferenced. I will try to georeference one of them or at least find one with a CRS

[mdsumner]

I was thinking about a brand image that I'd wanted to digitize the nodes of and triangulate, it highlights a few key issues:

https://gist.github.com/mdsumner/449148a78ccea02b19ab1a21bfc5acb3

[mdsumner]

I meant to add: this PDF reports that it has a "rotation" in the transform.

https://geopower.jws.com/rockland/MapDownload.jsp?id=203672'

Basically that's a very rare use of the 5th and 6th values in the GDAL affine "geo-transform" that specify a "shear" for X and Y in terms of the other axis. It's a cheap "rotation" used by some GIS. Raster partly supports it but it's potentially a big complication. You might explore this image in QGIS which will (I think) respect the full transform.

[yeedle]

Thanks @mdsumner, for taking this up. I really appreciate the clear explanations and the helpful links. I'm still going through some of the concepts, but tremendously helpful so far!

[tim-salabim]

Regrading the georeferencing, there is now
https://mrjoh3.shinyapps.io/georefr/
which allows rough georeferencing of images