crsh / rmdfiltr

R package that provides a collection of Lua filters that extend the functionality of R Markdown templates.

rmdfiltr

[

rmdfiltr provides a collection of Lua-filters that extend the functionality of R Markdown templates.

Installation

You can install the development version from this GitHub repository with:

# install.packages("remotes")
remotes::install_github("crsh/rmdfiltr")

Example

You can add a filter to any R Markdown template that accepts additional pandoc arguments.

---
title: "Word count test"
output:
  html_document:
    pandoc_args: !expr rmdfiltr::add_wordcount_filter()
---

Of course, you can also use the filters in a custom R Markdown format by adding pandoc arguments with the pre-processor function.

wordcount_html_document = function(...) {
  format <- rmarkdown::html_document(...)
  format$pre_processor <- rmdfiltr::add_wordcount_filter
  format
}

See R Markdown: The Definitive Guide for details on how to create custom formats.

Contributions

Contributions of new filters are welcome. Pleas refer to the contributing guidelines before you start working or open a pull request. Also, please note that the rmdfiltr project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

rmdfiltr's Issues

New filter: Add current date

As of pandoc 2.17, the following is possible, which would make for a useful filter.

function Doc (body, meta, vars)
  vars.date = vars.date or os.date '%B %e, %Y'
  return body, vars
end

replace_ampersands.lua fails if lang: german

Hi @crsh,

when I set lang: "german" in a papaja document, I get the following error:

Error running filter /home/mariusbarth/R/x86_64-pc-linux-gnu-library/4.2/rmdfiltr/replace_ampersands.lua:
...pc-linux-gnu-library/4.2/rmdfiltr/replace_ampersands.lua:13: attempt to index a nil value (field 'c')
stack traceback:
Fehler: pandoc document conversion failed with error 83

Thought this is most likely a problem with rmfiltr, which is why I am posting it here.

Ampersand filter working in `papaja` but not in standard `rmarkdown`

Please briefly describe your problem and what output you expect. If you have a question, please don't use this form. Instead, ask on https://stackoverflow.com/ or https://community.rstudio.com/.

Please include a minimal reproducible example (AKA a reprex). If you've never heard of a reprex before, start by reading https://www.tidyverse.org/help/#reprex.

---

Adding the ampersand filter to an rmarkdown template isn't working, but running a papaja template works.

rmarkdown template YAML:

title: "Untitled"
bibliography: references.bib
citeproc: no
output:
  pdf_document:
    pandoc_args: !expr rmdfiltr::add_replace_ampersands_filter(rmdfiltr::add_citeproc_filter(args = NULL))

references.bib:

@Manual{R-usethis,
  title = {usethis: Automate Package and Project Setup},
  author = {Hadley Wickham and Jennifer Bryan},
  year = {2021},
  note = {R package version 2.0.1},
  url = {https://CRAN.R-project.org/package=usethis},
}

I added [@R-usethis] to the body of the document.

The pandoc call:

/usr/local/bin/pandoc +RTS -K512m -RTS index.utf8.md --to latex --from markdown+autolink_bare_uris+tex_math_single_backslash --output index.tex --lua-filter /Library/Frameworks/R.framework/Versions/4.0/Resources/library/rmarkdown/rmarkdown/lua/pagebreak.lua --lua-filter /Library/Frameworks/R.framework/Versions/4.0/Resources/library/rmarkdown/rmarkdown/lua/latex-div.lua --self-contained --highlight-style tango --pdf-engine pdflatex --variable graphics --filter /usr/local/bin/pandoc-citeproc --lua-filter /Library/Frameworks/R.framework/Versions/4.0/Resources/library/rmdfiltr/replace_ampersands.lua --variable 'geometry:margin=1in'

Output created: index.pdf

The results has (Wickham and Bryan 2021), instead of the expected (Wickham & Bryan 2021). Also happens with html_document. Note that I don't have any issues with the word count filter.

How can I diagnose this further?

Session info

xfun::session_info("rmdfiltr")
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Mojave 10.14.6

Locale: en_AU.UTF-8 / en_AU.UTF-8 / en_AU.UTF-8 / C / en_AU.UTF-8 / en_AU.UTF-8

Package version:
  assertthat_0.2.1  base64enc_0.1.3   digest_0.6.27     evaluate_0.14    
  glue_1.4.2        graphics_4.0.2    grDevices_4.0.2   highr_0.8        
  htmltools_0.5.1.1 jsonlite_1.7.2    knitr_1.31        magrittr_2.0.1   
  markdown_1.1      methods_4.0.2     mime_0.10         rlang_0.4.10     
  rmarkdown_2.7.6   rmdfiltr_0.1.3    stats_4.0.2       stringi_1.5.3    
  stringr_1.4.0     tinytex_0.31.3    tools_4.0.2       utils_4.0.2      
  xfun_0.22         yaml_2.2.1       

Pandoc version: 2.10.1

Quarto support

This isn't a bug report or anything—just a note about how to use {rmdfiltr}'s word count with Quarto.

Quarto doesn't support inline R code in YAML (quarto-dev/quarto-cli#1391 (comment)), so adding !expr rmdfiltr::add_wordcount_filter() to the YAML in a .qmd file doesn't work. Additionally, there's no way to specify pandoc_args, so there's no straightforward way to make citeproc run before the wordcount filter.

Fortunately, this GitHub comment shows that it's possible to make a Lua filter that basically behaves like --citeproc by feeding the whole document to pandoc.utils.citeproc(). That means we can create a little Lua script like citeproc.lua:

-- Lua filter that behaves like `--citeproc`
function Pandoc (doc)
  return pandoc.utils.citeproc(doc)
end

…and then include that as a filter:

format:
  html:
    citeproc: false
    filter:
      - '/path/to/citeproc.lua'
      - '/path/to/wordcount.lua'

This creates a pandoc command that looks something like this, feeding the document to the citeproc "filter" first, then feeding that to the word count script:

pandoc whatever.md --output whatever.html  --lua-filter citeproc.lua --lua-filter wordcount.lua

There's still no way that I've found to dynamically generate /path/to/wordcount.lua like !expr rmdfiltr::add_wordcount_filter(), so users need to download the Lua file themselves and put it somewhere in their project.

The Lua files can also be included in a Quarto format extension like this so there's no need to manually deal with Lua files in the YAML. Instead, users can just specify a wordcount-enabled format:

title: Something
format:
  wordcount-html: default

Support for new languages to the replace_ampersands.lua

Currently, the Lua-filter that replaces ampersands in in-text citations by "and" supports English, Spanish, French, German, and Dutch. Suggestions for additional languages (including the required translation of "and") are welcome.

CRAN release

Hi @crsh,

It would be great if the current dev version of rmdfiltr was available from CRAN. Do you think this will be possible anytime soon?

Best regards,
Marius

replace_ampersands.lua fails if locale is given on command line

replace_ampersands.lua works nicely if the locale is provided in the YAML metadata header, but fails with the error message shown below if the locale is given on pandoc’s command line.

Using the YAML metadata header:

pandoc -s --citeproc -L replace_ampersands.lua  -t plain << EOT

@item1

---
csl: apa.csl
references:
- id: item1
  type: book
  author:
    - family: Doe
      given: Jane
    - family: Roe
      given: Ron
  issued: 2020
  title: The title
  publisher: The Publisher
lang: en-GB
...
EOT

Output:

Doe and Roe (2020)

Doe, J., & Roe, R. (2020). The title. The Publisher.

Using the command line:

pandoc -s --citeproc -M lang=en-GB -L replace_ampersands.lua  -t plain << EOT

@item1

---
csl: apa.csl
references:
- id: item1
  type: book
  author:
    - family: Doe
      given: Jane
    - family: Roe
      given: Ron
  issued: 2020
  title: The title
  publisher: The Publisher
...
EOT

Output:

Error running filter /Users/nb/.local/share/pandoc/filters/replace_ampersands.lua:
...rs/nb/.local/share/pandoc/filters/replace_ampersands.lua:13: attempt to index a nil value (field '?')
stack traceback:
	...rs/nb/.local/share/pandoc/filters/replace_ampersands.lua:13: in function 'get_lang_tag'

Expected output: Same as when using the YAML metadata header – it shouldn’t make a difference by which way the lang field is being set.

Wordcount error (report = "warn")

Hello,

I'm trying to knit a document in R and I'm receiving this error:

Error in rmdfiltr::add_wordcount_filter(args, report = "warn") : 
  unused argument (report = "warn")
Calls: <Anonymous> -> <Anonymous> -> pdf_pre_processor
Execution halted

I'm using papaja but as far as I know I have not changed anything to cause this error.

Thanks for any assistance.

Remove Oxford commas in in-text citations when writing in German

When creating a manuscript in German, the ampersand filter leaves Oxford commas in in-text citations, i.e. "Barth, Stahl, & Haider (2019)" is changed to "Barth, Stahl, und Haider (2019)".
In German, the Oxford comma should be removed.

Character Count in addition to Word Count reported in R Markdown output

The reported Word Count (in the R Markdown tab), e.g.,

9215 words in text body
2884 words in reference section

is extremely useful. Some journals require a character count, however. Running wordcountaddin::text_stats() does of course produce that, but adds data that shouldn't be included (compared to the output reported in the RMarkdown tab). Would it be possible to add to the respective code that produces that word cout output to include the character counts for "text body" and "reference section"?

Either as

9215 words / 56272  characters in text body
2884 words / 2433 characters in reference section

or

9215 words in text body
2884 words in reference section
56272 characters in text body
2433 characters in reference section

or some alternative form.

Does not work with pandoc v2.9.1

I have found that rmdfiltr does not work with the latest pandoc v2.9.1.

Tested rmdfiltr is the latest dev version installed from GitHub.

Please see the following testing environments:

test.Rmd

---
citeproc: no
output:
  pdf_document:
    pandoc_args:
      - !expr rmdfiltr::add_wordcount_filter(rmdfiltr::add_replace_ampersands_filter(rmdfiltr::add_citeproc_filter(args = NULL)))
---

Hello world!

Results

R version 3.6.2 (2019-12-12) -- "Dark and Stormy Night"                                                                                                
     
Copyright (C) 2019 The R Foundation for Statistical Computing                                                                                          
     
Platform: x86_64-w64-mingw32/x64 (64-bit)                                                                                                              
     
                                                                                                                                                       
     
R is free software and comes with ABSOLUTELY NO WARRANTY.                                                                                              
     
You are welcome to redistribute it under certain conditions.                                                                                           
     
Type 'license()' or 'licence()' for distribution details.                                                                                              
     
                                                                                                                                                       
     
R is a collaborative project with many contributors.                                                                                                   
     
Type 'contributors()' for more information and                                                                                                         
     
'citation()' on how to cite R or R packages in publications.                                                                                           
     
                                                                                                                                                       
     
Type 'demo()' for some demos, 'help()' for on-line help, or                                                                                            
     
'help.start()' for an HTML browser interface to help.                                                                                                  
     
Type 'q()' to quit R.                                                                                                                                  
     
                                                                                                                                                       
     
> rmarkdown::render('test.Rmd')                                                                                                                        
     
                                                                                                                                                       
     
                                                                                                                                                       
     
processing file: test.Rmd                                                                                                                              
     
  |......................................................................| 100%                                                                        
     
  ordinary text without R code                                                                                                                         
     
                                                                                                                                                       
     
                                                                                                                                                       
     
output file: test.knit.md                                                                                                                              
     
                                                                                                                                                       
     
"C:/PROGRA~3/CHOCOL~1/bin/pandoc" +RTS -K512m -RTS test.utf8.md --to latex --from markdown+autolink_bare_uris+tex_math_single_backslash --output test.tex
--s
elf-contained --highlight-style tango --pdf-engine pdflatex --variable graphics --lua-filter "C:/Program Files/R/R-3.6.2/library/rmarkdown/rmd/lua/pagebreak.
lua" --lua-filter "C:/Program Files/R/R-3.6.2/library/rmarkdown/rmd/lua/latex-div.lua" "c(\"--filter\", \"C:/PROGRA~3/CHOCOL~1/bin/pandoc-citeproc.exe\",
\"-
-lua-filter\", \"C:/Program Files/R/R-3.6.2/library/rmdfiltr/replace_ampersands.lua\", \"--lua-filter\", \"C:/Program Files/R/R-3.6.2/library/rmdfiltr/wordco
unt.lua\")" --variable "geometry:margin=1in"                                                                                                           
     
pandoc.exe: c("--filter", "C:/PROGRA~3/CHOCOL~1/bin/pandoc-citeproc.exe", "--lua-filter", "C:/Program Files/R/R-3.6.2/library/rmdfiltr/replace_ampersands.lua
", "--lua-filter", "C:/Program Files/R/R-3.6.2/library/rmdfiltr/wordcount.lua"): openBinaryFile: invalid argument (Invalid argument)                   
     
Error: pandoc document conversion failed with error 1                                                                                                  
     
> 

Only on Windows: Error running filter replace_ampersands.lua:13: attempt to index a nil value (field 'c')

Hi @crsh,

this is a problem originally reporterd by @kbading. Only on Windows, using parameter lang: de-DE fails with the most recent versions of R (4.2.2), pandoc (2.19.2) and rmarkdown (2.19). The error message is:

"C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools/pandoc" +RTS -K512m -RTS test.knit.md --to latex --from markdown+autolink_bare_uris+tex_math_single_backslash+raw_attribute --output test.tex --lua-filter "C:\Users\MARIUS~1\AppData\Local\R\WIN-LI~1\4.2\bookdown\RMARKD~1\lua\CUSTOM~1.LUA" --lua-filter "C:\Users\MARIUS~1\AppData\Local\R\WIN-LI~1\4.2\RMARKD~1\RMARKD~1\lua\PAGEBR~1.LUA" --lua-filter "C:\Users\MARIUS~1\AppData\Local\R\WIN-LI~1\4.2\RMARKD~1\RMARKD~1\lua\LATEX-~1.LUA" --embed-resources --standalone --highlight-style tango --pdf-engine pdflatex --include-in-header "C:\Users\MARIUS~1\AppData\Local\R\WIN-LI~1\4.2\papaja\RMARKD~1\TEMPLA~1\apa6\RESOUR~1\APA6_H~1.TEX" --variable graphics --wrap preserve --csl "C:\Users\MARIUS~1\AppData\Local\R\WIN-LI~1\4.2\papaja\rmd\apa6.csl" --citeproc --lua-filter "C:/Users/Marius Barth/AppData/Local/R/win-library/4.2/rmdfiltr/replace_ampersands.lua" --lua-filter "C:/Users/Marius Barth/AppData/Local/R/win-library/4.2/rmdfiltr/wordcount.lua" --lua-filter "C:/Users/Marius Barth/AppData/Local/R/win-library/4.2/papaja/lua/parse_metadata.lua" --variable "documentclass:apa6" --variable "classoption:man" --variable "lang:de-DE" --variable "subparagraph:yes" --variable "block-headings:no" --include-in-header "C:\Users\MARIUS~1\AppData\Local\Temp\RtmpU5bQ5X\includes_12a028fc1da9.tex" 
Error running filter C:/Users/Marius Barth/AppData/Local/R/win-library/4.2/rmdfiltr/replace_ampersands.lua:
.../Local/R/win-library/4.2/rmdfiltr/replace_ampersands.lua:13: attempt to index a nil value (field 'c')
stack traceback:
Fehler: pandoc document conversion failed with error 83
Ausführung angehalten

Here is a reproducible example (finishes on Ubuntu, fails on Windows)
test.zip

Words in document title and captions are not counted

Thank you so much for this amazing package! The count for reference is especially unique useful! Just want to make sure, though, that the current version (0.1.0) still does not count for the title and figure notes, right? As illustrated in the following example, the word count was always 3 no matter how I changed the title or the figure notes. Just in case I didn't set the function properly. Thanks!

---
title: "1 2"
citeproc: no
output: 
  html_document:
    pandoc_args: !expr rmdfiltr::add_wordcount_filter(rmdfiltr::add_citeproc_filter(args = NULL))
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

3 4 5


```{r pressure, echo = FALSE, message = FALSE, warning = FALSE}
# plot(pressure)

library(grid)
library(gridExtra)
library(ggplot2)
g <- ggplot(pressure, aes(x = temperature, y = pressure)) + geom_point()

plot_text <- textGrob("6 7", gp = gpar(fontface = 3, fontsize = 9), x = 0.05, hjust = 0)
# x stands for where the text starts, hjust stands for left, middle, right.

grid.arrange(g, bottom = plot_text)

```

The out put in the "R Markdown" tab is

3 words in text body
0 words in reference section