Missing fonts in R? about r-base-feedstock

[jdblischak]

@lwasser I tested your solution and confirmed it works. Thanks!

I also did some more experimenting with other font software. I searched the rocker Dockerfiles for font-related packages. They install the following:

• fonts-texgyre
• lmodern
• texlive-fonts-extra
• texlive-fonts-recommended

Installing only fonts-texgyre doesn't fix the problem. Adding lmodern installs a font, but it is strange looking (all caps). Using texlive-fonts-recommended, I was able to get decent results.

[isuruf]

Instead of creating a new recipe, what if we added mscorefonts to the r recipe? It currently installs r-base and r-recommended.

Sounds good to me

[lwasser]

@ocefpaf @jdblischak yay! happy to be able to help find a solution on this one :)
it still it interesting to me that all fonts work with python but not R. but yes we are up and running now so thank you all for the input :)

[dpryan79]

You might use a different base image, such as https://quay.io/repository/dpryan79/mulled_container. For example, your plot will likely look correctly in quay.io/biocontainers/bioconductor-deseq2:1.22.1--r351hf484d3e_0, to select a random bioconda container based on that.

[jdblischak]

I've also ran into this same issue multiple times before. I would really appreciate a solution. I want to be able to setup continuous integration to update a plot every time I push a new commit. However, the plots have empty boxes where the text characters are supposed to be.

Below is a reproducible example using plots in R Markdown. It uses the Docker image rocker/verse to demonstrate that it is possible to create plots from R within a Docker container. In contrast, it uses the Docker image continuumio/miniconda3 to show how the plots are broken. The example rmarkdown-generated HTML files are saved locally in /tmp/r-font/.

mkdir -p /tmp/r-font
echo -e '```{r}\nplot(1:10)\n```\n```{r}\nggplot2::qplot(1:10)\n```\n' > /tmp/r-font/test-verse.Rmd
cp /tmp/r-font/test-verse.Rmd /tmp/r-font/test-conda.Rmd

docker run -it --rm -v /tmp/r-font:/r-font rocker/verse bash
Rscript -e 'rmarkdown::render("r-font/test-verse.Rmd")'
exit

docker run -it --rm -v /tmp/r-font:/r-font continuumio/miniconda3:latest bash
conda config --set always_yes yes
conda config --add channels conda-forge
conda update --all
conda install r-ggplot2 r-rmarkdown
Rscript -e 'rmarkdown::render("r-font/test-conda.Rmd")'
exit

[lwasser]

@jdblischak that's exactly what we are doing. a CI build of lessons that include plots. thank you for providing a reproducible example. hoping this is fixable !!

[lwasser]

hey friends.
using @jdblischak example above i was able to make a bit of progress altho i'm not exactly sure how to implement this in the docker container. i followed this:

https://ubuntuforums.org/showthread.php?t=2409586

running the following (removed sudo as that is not installed in our build)

apt purge ttf-mscorefonts-installer
wget http://ftp.de.debian.org/debian/pool/contrib/m/msttcorefonts/ttf-mscorefonts-installer_3.7_all.deb -P ~/Downloads
apt install ~/Downloads/ttf-mscorefonts-installer_3.7_all.deb
apt-mark hold ttf-mscorefonts-installer

when i rerendered the plots, the fonts were correct. if i wanted to add this to my docker container build (bare with me as i'm really new to all of this linux / docker stuff)... what would be the most efficient way? @ocefpaf has taught me that fewer lines == fewer layers and i'd like to optimize. also when i installed it did ask me to "accept" something so id need to automate that step. i'm guessing we wouldn't need to purge the installer either in this case.

would i need to RUN wget and then RUN apt install with some flag to accept? Many thanks.

[ocefpaf]

The rasterio issue should be solved soon. Thanks for sharing the font installation lines! That is very useful. (I recall a user that package some of those fonts in conda-forge. I need to check if those are a good alternative for the system ones.)

[ocefpaf]

@ocefpaf I tested each one of these packages. Each one worked in that the text rendered on the plot, but of course each one produced slightly different results.

That was a nice surprise! I was not expecting them to work at all 😄

Since mscorefonts is the only one currently on conda-forge, I'd like to discuss the pros and cons of adding this as a dependency for r-base. Plotting is a super common use case for R, but many machines already have some fonts installed. Thus this mainly affects minimal installations such as Docker images.

It is the same problem with the xorg stuff. B/c conda does not have the concept of optional/suggested packages I guess that the best way forward is to not add them but recommend it in our docs.

One alternative would be to create a r-base-core package without it and an r-base that depends on both r-base-core and the rest of the optional packages.

@lwasser looks like that for you Dockerfile you can just add mscorefonts to the list of packages (if you are OK with the fonts results in the plot).

[isuruf]

What's wrong with adding mscorefonts to r?

r is what you get when you install R from their website right? Having three packages r-base, r-core, r would add to confusion

[isuruf]

BTW, is there an r recipe in AnacondaRecipes or was that something created only in conda-forge?

Yes, there is https://anaconda.org/r/r

[jdblischak]

Yep. Take a look on the matplotlib or gdal recipe where we already do that.

@ocefpaf Ah cool. I didn't know about the outputs field. I'd be open to using that to customize the r-base recipe with different optional packages like mscorefonts.

And yeah, the r recipe was copied from defaults. I rarely use it myself, but a lot of the Anaconda documentation recommends using it (or at least it did years ago).

[bgruening]

We just see a few issues raised in Bioconda as well and it would be nice if this could be fixed here before people start hacking around it.

Please take it, I'm not sure about my time the next days. Thanks!

[dbast]

By my understand run_exports only kicks in when a consumer of the fontconfig package is rebuild.

As the dependency chain is r-base -> pango -> fontconfig. A rebuild of pango is required to have the run_exports dependencies for fontconfig added to pango and thus installed via r-base (pango was not rebuild since the fontconfig update).

[jdblischak]

Confirmed! Thanks @bgruening and @dbast! 🎉

Here's the test code:

mkdir -p /tmp/r-font
echo -e '```{r}\nplot(1:10)\n```\n```{r}\nggplot2::qplot(1:10)\n```\n' > /tmp/r-font/test-verse.Rmd
cp /tmp/r-font/test-verse.Rmd /tmp/r-font/test-conda.Rmd

docker run --rm -v /tmp/r-font:/r-font rocker/verse Rscript -e 'rmarkdown::render("r-font/test-verse.Rmd")'
open /tmp/r-font/test-verse.html

docker run --rm -v /tmp/r-font:/r-font continuumio/miniconda3:latest bash -c "
  conda config --set always_yes yes
  conda config --add channels conda-forge
  conda update --all
  conda install r-ggplot2 r-rmarkdown
  Rscript -e 'rmarkdown::render(\"r-font/test-conda.Rmd\")'
"
open /tmp/r-font/test-conda.html

The results aren't identical, but now the conda-generated R graphics have properly rendered fonts.

Result from Rocker container:

Result from conda R in Docker container:

I'm closing this Issue now since the primary problem has been solved: conda R automatically installs font dependencies in a minimal Docker container.

[ocefpaf]

You might use a different base image, such as quay.io/repository/dpryan79/mulled_container. For example, your plot will likely look correctly in quay.io/biocontainers/bioconductor-deseq2:1.22.1--r351hf484d3e_0, to select a random bioconda container based on that.

Good to know @dpryan79. Maybe we can just install the fonts in the current container to avoid pulling more than we really need? I'm not sure what r-base fonts package is expected though.

[lwasser]

you know what is really odd here @ocefpaf ? our python plots are just fine. it's almost as if R can't find the fonts OR it needs different fonts. a colleague of mine sent me this

https://paste.sr.ht/~hrbrmstr/5ebe58a18aaf54f74fbfea5ce493beb2f6b1d8e9#Dockerfile-L96

which actually moves fonts to a different folder. i can ssh into our build to see what is where or look in the docker container but i'm not sure where to start to figure out where R expects fonts to live.
it would be nice if we can just install fonts in this instance given how speedy our build is now.

[mingwandroid]

We'll get to the bottom of this. Do you have reproduction steps though?

[lwasser]

yes! @mingwandroid here is my docker setup: https://github.com/earthlab/r-python-eds-lessons-env/blob/master/Dockerfile

and you can pull the image here: https://hub.docker.com/r/earthlab/r-python-eds-lessons-env
does that help? much appreciated.

[lwasser]

got it!

RUN apt-get update \
    && wget http://ftp.de.debian.org/debian/pool/contrib/m/msttcorefonts/ttf-mscorefonts-installer_3.7_all.deb -P ~/Downloads \
    && apt install ~/Downloads/ttf-mscorefonts-installer_3.7_all.deb -y \
    && apt-mark hold ttf-mscorefonts-installer

things installed but now i am getting lib errors with rasterio so i may head over to that repo to post an issue about it. a few of us have had this error in the past day

Traceback (most recent call last):
  File "import_check.py", line 3, in <module>
    import rasterio
  File "/opt/conda/lib/python3.7/site-packages/rasterio/__init__.py", line 22, in <module>
    from rasterio._base import gdal_version
ImportError: libcfitsio.so.5: cannot open shared object file: No such file or directory

[jdblischak]

I recall a user that package some of those fonts in conda-forge. I need to check if those are a good alternative for the system ones.

@ocefpaf It'd be great if conda-forge could provide some font packages to avoid these plotting issues. Please let me know if you need me to test anything out.

[jdblischak]

The R package systemfonts might be helpful for debugging this issue further if there is a particular font that is missing.

[ocefpaf]

@ocefpaf It'd be great if conda-forge could provide some font packages to avoid these plotting issues. Please let me know if you need me to test anything out.

Here are the current font packages that exists on defaults and conda-forge. I'm not sure they'll work but it is worth testing. We should probably try to fix our texlive package and split the fonts into a separate package that we may re-use in other feedstocks.

font-ttf-dejavu-sans-mono       2.37      h6964260_0  pkgs/main           
font-ttf-inconsolata           2.001      hcb22688_0  pkgs/main           
font-ttf-source-code-pro       2.030      h7457263_0  pkgs/main           
font-ttf-ubuntu                 0.83      h8b1ccd4_0  pkgs/main 
mscorefonts                    0.0.1               1  conda-forge         
mscorefonts                    0.0.1               2  conda-forge 

[jdblischak]

I'm not sure they'll work but it is worth testing.

@ocefpaf I tested each one of these packages. Each one worked in that the text rendered on the plot, but of course each one produced slightly different results.

Since mscorefonts is the only one currently on conda-forge, I'd like to discuss the pros and cons of adding this as a dependency for r-base. Plotting is a super common use case for R, but many machines already have some fonts installed. Thus this mainly affects minimal installations such as Docker images.

[jdblischak]

It is the same problem with the xorg stuff. B/c conda does not have the concept of optional/suggested packages

I agree that this is a limitation.

I guess that the best way forward is to not add them but recommend it in our docs.

Is there a good place in our documentation for this information? A general page on "Tips and Tricks for using R" could be very useful.

One alternative would be to create a r-base-core package without it and an r-base that depends on both r-base-core and the rest of the optional packages.

Instead of creating a new recipe, what if we added mscorefonts to the r recipe? It currently installs r-base and r-recommended. And if a user didn't want mscorefonts installed, they could manually install these instead:

# include mscorefonts
conda install r
# exclude mscorefonts
conda install r-base r-recommended

[ocefpaf]

Instead of creating a new recipe, what if we added mscorefonts to the r recipe ? It currently installs r-base and r-recommended. And if a user didn't want mscorefonts installed, they could manually install these instead:

I was not thinking about a new recipe at all. A split recipe in the same r-base.
However, I'm not sold on r-base and r-recommended. I prefer r-base and r-core b/c that reflects a full working package (not a recommendation) and a slim core (for those who know what they are doing). Also, that reflects what we are already doing with packages like dask and other.

[jdblischak]

I was not thinking about a new recipe at all. A split recipe in the same r-base.

@ocefpaf Could you please elaborate? Do you mean splitting the current r-base recipe into two separate recipes: r-base and r-core? Or is it possible to have a split recipe in the same feedstock? I checked the dask feedstock, and it looked like a standard recipe.

[ocefpaf]

@ocefpaf Could you please elaborate? Do you mean splitting the current r-base recipe into two separate recipes: r-base and r-core?

Nope.

Or is it possible to have a split recipe in the same feedstock?

Yep. Take a look on the matplotlib or gdal recipe where we already do that.

I checked the dask feedstock , and it looked like a standard recipe.

dask is old and has two different different recipes for that, https://github.com/conda-forge/dask-core-feedstock and https://github.com/conda-forge/dask-feedstock, since them we got conda-build with multiple outputs from a single recipe.

[ocefpaf]

What's wrong with adding mscorefonts to r?

Sure. I guess I was not familiar with the r recipe. Hard to find things with a single letter. And I thought that @jdblischak was referencing r-base. (I only install the latter.)

BTW, is there an r recipe in AnacondaRecipes or was that something created only in conda-forge? If it is the latter we are already creating confusion, of the former we are only propagating it 😄

[lwasser]

@lwasser looks like that for you Dockerfile you can just add mscorefonts to the list of packages (if you are OK with the fonts results in the plot).

@ocefpaf yes - mscorefonts work just fine for our needs! thank you again!!

[mingwandroid]

recommended has a special meaning in R. Please don't change it to r-core out anything else!

[jdblischak]

recommended has a special meaning in R. Please don't change it to r-core out anything else!

@mingwandroid Agreed! We are not planning on changing the r-recommended recipe.

The proposal we are disccussing is to have two versions of the r-base recipe: a minimal one, and a more full-featured one (e.g. that also installs font libraries).

[bgruening]

@jdblischak are you planning to work on this?

[jdblischak]

are you planning to work on this?

@bgruening I can if needed, but you are welcome to take it on if you'd like.

[mingwandroid]

Fwiw I use the anaconda-fonts metapackage for the fonts I added to AD. RStudio was the trigger for these via qt.

[bgruening]

Ok, if this does not fly - @mingwandroid are you ok adding the font to the r package?

[bgruening]

@jdblischak would you be ok with adding it to r as well?

[mingwandroid]

Ok, if this does not fly - @mingwandroid are you ok adding the font to the r package?

It is "OK" by me, not perhaps ideal. To explain:

Arguably, at the technical level, these fonts can only get found (given how we're currently building things) via things that use our fontconfig packages or things that use our qt packages so in theory if we were to add a run_exports to those pointing to some agreeably named font metapackage - conda-ecosystem-fonts then fonts will only get installed when they are actually needed.

But I'm fine with this (for now at the very least). Overall, font management is definitely something we should tackle as well as we can in a planned way. I think Spyder may be embedding/vendoring fonts for example, is that true and a preference of Spyder @ccordoba12? or would that be something you'd prefer not to do in conda-forge if we can share fonts more widely? I like noarch data files for these things FWIW, they have always worked well.

[mingwandroid]

So if we have a font disagreement (or we need to add some, I expect ours would be a strict superset), we'd have our own version of conda-ecosystem-fonts that points at anaconda-fonts instead of the one on conda-forge that points at conda-forge-fonts .. or something!

[ccordoba12]

I think Spyder may be embedding/vendoring fonts for example, is that true and a preference of Spyder @ccordoba12?

No, we don't vendor any font at the moment. But it'd be nice to have some fonts installed by default in Anaconda so we can provide the same look and feel for all platforms.

[mingwandroid]

Agreed

[dbast]

The fonts + meta-packages are merged and uploaded.

Now just from R perspective fonts-conda-ecosystem could be added as dependency to:

1. run requirement of r-base or r
2. run_exports of fontconfig
3. as run requirement to individual packages like r-ggplot2

1.+2. are from R perspective almost the same because of the dependency graph (r -> r-base -> pango -> fontconfig), meaning the fonts would end up beeing installed in every conda-forge R environment.

With 3. the fonts only get installed if a real consumer (pkg) depends on it.

In general: 1. does imo not make so much sense, 2. requires a conda-forge wide decision 3. we could do for now (until something regarding 2. is done/decided)...

Any ideas / opinions?

[mingwandroid]

IMHO 1 risks people presuming this will always be the case which makes it inflexible, 2 is technically more correct (and any other package that explicitly knows about these fonts outside of fontconfig conda-forge's x11 libs? - should also get the same, ideally soon), while 3 is very pragmatic.

[bgruening]

This is still an issue for a few uses. @dbast do we want to proceed with 2?

[bgruening]

I hope conda-forge/fontconfig-feedstock#47 solves this issue.

[jdblischak]

@bgruening Thanks for returning to this important issue!

I re-ran my reproducible example from above. Unfortunately the fonts still do not display when they are created inside of a Docker container. Looking at the packages in the environment, the latest build of fontconfig (build 1005) from your recent PR conda-forge/fontconfig-feedstock#47 is downloaded, but it didn't bring fonts-conda-ecosystem with it.

# Run from continuumio/miniconda3:latest
$ conda list font
# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
fontconfig                2.13.1            hba837de_1005    conda-forge

[bgruening]

I don't understand why this is not working :(

@ocefpaf any idea? run_exports are supposed to drag that dependency in, isn't it?

[dbast]

This should work now.

@jdblischak Can you retest? Thanks.