repair SVG export about cmapplot

[matthewstern]

ggsave() has an internal function plot_dev() which returns various devices with various arg configurations. It's very elegant actually. TL;DR: they use svglite::svglite(), where we use grDevices:svg(). This is likely the issue.

I believe we selected svg() for package and argument consistency (our other vector devices, cairo_pdf and cairo_ps, come from the same package and share a help file). However, because (I think) svg files don't export fonts, but convert letters to paths instead, we should be able to use svglite() without reopening pandora's font box. This should be tested a bit. If it works, the solution will be relatively easy to implement in the save_plot() subfn of finalize_plot().

[nmpeterson]

From the svglite website (emphasis mine):

svglite is a graphics device that produces clean svg output, suitable for use on the web, or hand editing. Compared to the built-in svg(), svglite produces smaller files, and leaves text as is, making it easier to edit the result after creation. It also support multiple nice features such as embedding of web fonts.

This could be really excellent, if it works properly! Seems like this article will be relevant to implementing this with our fonts.

[matthewstern]

Just a reminder that finalize_plot() now invisibly exports the grobTree object that is the final plot, so you can use that to do initial experimentation without messing with the current finalize code.

e.g.:

fp <- finalize_plot(...)

svglite(...)
grid.draw(fp)
dev.off()

[sarahcmap]

Thanks to your tips I just made a few changes in a new branch 'svgfix' and that is working well in Illustrator.

But I'm wondering about the fonts and if we need to make use of the font arguments in svglite that Noel linked to. I tried switching the chart title font to something other than Arial on my mac while not updating the svglite font argument and it exported correctly (with my new specified font). Does anything need to be done here?

[matthewstern]

@sarahcmap I was able to use your script to run an export with Calibri. Cool!

Some notes:

• It works but the text appears wider in the SVG than in other formats (see attached). Not sure what to do about that.
• My quick read of the resources Noel shared is that at least some of the additional argumentation is to help a computer without a font interpret an SVG that calls for that font. So, I'm curious how your Mac responds to this SVG, but I presume you have Calibiri installed on your computer? We may need to do some experimentation with other fonts too.
• Don't forget to add @importFrom svglite svglite and add the pkg to description.

svgtestcalibri.zip

[sarahcmap]

Thanks for checking @matthewstern. I don't have Calibri so the svg is looking pretty funky since it converts to Arial upon opening in Illustrator. I see that all of the font-family attributes in the svg are set to Calibri though so that is getting set properly without any additional args.

That's weird it looks wider for you though...when I tried changing fonts to a non-Arial font I have locally, the svg and pdf were identical, and there was a slight difference compared to the png.

Perhaps it as something to do with the textlength attribute?

Also, it appears that since the textlength gets set for each word, if you edit the text in Illustrator by removing a letter or increasing the font size, it will stay the same width. You can get rid of this by clearing the textlength attribute, or as I just found out with the newest version of svglite there in an argument that allows you to not write the textlength attribute as part of the svg.

I'm curious if turning the textlength off would do anything in your example...If you get a chance, could you try exporting the gTree object with

svglite(filename='', fix_text_size=FALSE)

with svglite version 1.2.3.9000?

+ will do on the import. Thanks!

[nmpeterson]

With svglite, is there any way to embed the fonts in case the user doesn't have them? That would seem to be one benefit of our original method.

[sarahcmap]

With svglite, is there any way to embed the fonts in case the user doesn't have them? That would seem to be one benefit of our original method.

I'm still a little confused on this point. It looks like they recently added a webfonts argument to the package, but I think we need a URL for them to get whitney from or we'd have to put the font file in the cmapplot package? Is that how y'all read this/are we allowed to share the font in this way?

Also when i tried using this argument I got random black fills in some plot areas...but that's a problem for later.

[nmpeterson]

We can't do anything that would allow non-staff to obtain the font files. Font licensing terms are pretty strict.

That said, the CMAP website uses Whitney as a web font, so there may be some way for us to access it (although we would have to be careful not to include any code with access tokens or whatever in the repo...).

If there's no way to embed the fonts, I'm inclined to just stick with the current method and see if there is a particular, fixable issue with the SVG code (because it is stored as HTML, essentially) that is causing an issue with Illustrator. If so, we could include code that "repairs" that HTML after being exported by finalize_plot().

[nmpeterson]

What's really interesting is that the SVG file @matthewstern uploaded does not pass the W3C's validator, but an SVG of the same chart that I just exported using cmapplot 1.0.0 (below) does validate.

test.svg.zip

[sarahcmap]

We can't do anything that would allow non-staff to obtain the font files. Font licensing terms are pretty strict.

That said, the CMAP website uses Whitney as a web font, so there may be some way for us to access it (although we would have to be careful not to include any code with access tokens or whatever in the repo...).

If there's no way to embed the fonts, I'm inclined to just stick with the current method and see if there is a particular, fixable issue with the SVG code (because it is stored as HTML, essentially) that is causing an issue with Illustrator. If so, we could include code that "repairs" that HTML after being exported by finalize_plot().

Mm that's a good point to just look at the svg text and compare it to ones that do work. I'll check that out next week.

What's really interesting is that the SVG file @matthewstern uploaded does not pass the W3C's validator, but an SVG of the same chart that I just exported using cmapplot 1.0.0 (below) does validate.

test.svg.zip

Oh interesting I have never used a validator before. It looked like mostly issues with the id attributes from what I saw when I ran it? Those can be set by the svglite package i believe, but there may be other issues too

[matthewstern]

Let's step back for a second to review the various devices we employ and what our options are (at least as far as I am aware):

• We currently do all our plotting with the core R package grDevices. The package allows for many devices. In-R drawing devices come from this package, and are device dependent: quartz() on mac, windows() on windows, and x11() on Unix/Linux (Source). Vector devices, on the other hand, rely on Cairographics--it is the only option. Raster devices default to type = "windows" but can be set to type = "cairo". On windows machines, this almost certainly uses the "Windows GDI". It's not entirely clear what happens on Mac and Unix, but I would presume quartz() and x11(). We currently use the defaults.

• The Cairo package is also a thing which we do not currently employ. In-R drawing can be implemented with CairoWin or CairoX11, while vector and raster outputs can achieved with CairoPNG(), CairoSVG(), CairoPDF(), etc. All of the above are wrappers of the baseline function Cairo::Cairo(), which potentially make for easy implementation on our end.

• For svg files, the third option is svglite::svglite(), as discussed in this thread. Hadley is an author and a part of R-Lib, so core to R. It's not clear to me whether Cairographics are employed here, although the word "cairo" doesn't show up in the package documentation as far as I can tell.

@nmpeterson @sarahcmap please correct me if I'm wrong here: I think we believe that that grDevices::svg() converts text to shapes while svglite::svglite() saves text as text and then includes encoding that suggests what fonts to use. I'm presuming that svg files don't support font embedding like PDFs do.

@dlcomeaux and I did some testing of various output modes when we were building the second generation of finalize_plot() some months ago. He may have info to add here. My memory is that we were not particularly systematic in our testing. Once we found solutions that seemed to work and were relatively easy to code, we went with them.

We know a lot more now, and I wonder whether it makes sense to check in with our goals here. If we need to adjust things from where they currently stand, is it worth revisitting the various device options holistically before committing to svglite? For example, we could theoretically do all our drawing -- in and out of R -- with the Cairo::Cairo() function. (not suggesting this is a good idea, but it is an option).

Curious for other's thoughts.

EDIT: this may be a very useful guide for us. The author compares outputs from the three svg options (svg, CairoSVG, and svglite) and discusses the differences. It confirms that svglite uses plain text while the others convert to glyphs. And, it contains a section at the end about dealing with webfonts. It looks like embedding fonts is theoretically an option with svglite() via the user_fonts argument but that it is currently not functional?

[matthewstern]

So I'm betting y'all already came across the free range stats blog post cited above and I'm late to the party. I also just found another helpful and recent blog post about Cairo package graphics here, maybe also late to this party? Key takeaways from both combined seem to me to be:

• cairo_pdf() is the best PDF device (great, we're already doing this)
• png() with mode = "cairo_png" is preferred for pngs (we are using this device but not this mode)
• BUT the ragg package might actually be a better implementation for all raster filetypes (would be easy to implement in our structure as all raster files are handled together)
• svglite() is crisper than other svg drivers, but creates a potentially unsolvable font issue if sharing outside of CMAP.

What's the standard use case for svg in finalize_plot()? If it's sharing a file with comms for further editing, svglite() seems like the obvious answer. In this situation, how important is working out the user-fonts situation? If it's for direct posting to the website, it gets grayer, but I don't think that's Comms' intent here.

Unfortunately, the way we've set up filename handling, allowing both an "svg" and "svglite" mode to finalize gets a bit funky.

[nmpeterson]

I agree that the important thing to decide before continuing is what our SVG use case is. If Comms is perfectly happy for PDF to be the standard format we supply to them for editing, then I think our SVGs should be exported with fonts converted to paths/polygons (current cmapplot behavior) so that they can be displayed on websites and rendered correctly for all users in that environment. I wouldn't object to switching to cairoSVG() if it provides some benefit over grDevices::svg().

On the other hand, if Comms would prefer to work with SVG over PDF for any reason, we should see if the svglite format (with references to fonts by name only) works as expected in Illustrator -- specifically the latest version of Illustrator, which is significantly newer than the CS6 versions that are installed on ADOBE1/ADOBE2.

I also would not be opposed to allowing both, with format names like svg-web and svg-comms or something like that. Filename extension could be determined by using stringr::str_split(mode, '-')[[1]][1], and if mode is "svg" then the device could be chosen based on stringr::str_split(mode, '-')[[1]][2]: svglite() if "comms", or cairoSVG()/grDevices::svg() if "web".

Either way, I think we should flesh out the finalize vignette's section on exporting a bit, to go into detail about when it is best to use each format.

[matthewstern]

@nmpeterson @sarahcmap any thoughts on costs/benefits of upgrading to the ragg package for rasters?

• tidyverse blog post celebrating ragg 0.2.0
• ragg appears to be approaching 1.0.0 imminently

[nmpeterson]

Apparently fonts can be embedded in SVG files, but it doesn't appear that any of the available R graphics devices currently support this.

Edit: Oh, and @matthewstern, ragg seems fine. I haven't been involved enough in the coding of the finalize functions to have a strong feeling about it. 1.0.0 was released 23 days ago.

[matthewstern]

FYI svglite has been upgraded to rely on systemfonts and a new blog post on the tidyverse website confirms some of our conclusions about the differences between svg() and svglite(). Worth looking into for the future: https://www.tidyverse.org/blog/2021/02/svglite-2-0-0/

[matthewstern]

Replaced this with #127