emilhvitfeldt / smltar

Manuscript of the book "Supervised Machine Learning for Text Analysis in R" by Emil Hvitfeldt and Julia Silge

This is the repository for the book Supervised Machine Learning for Text Analysis in R, by Emil Hvitfeldt and Julia Silge.

Please note that this work is written under a Contributor Code of Conduct and the online version is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. By participating in this project (for example, by submitting an issue with suggestions or edits) you agree to abide by its terms.

smltar's Issues

move stop words after stemming

Make sure to have reasonable ggplot2 text size default

regex appendix

As horrible as it sounds. Should be super short with lots of references of where to find the information

talk about segmentation of Chinese text in tokenization chapter

Good example for a written language without spaces between words

Stemming dumb

stemming
porter stemmer in r - snowballc
explain the algorithm
https://github.com/ayushoriginal/Sentiment-Analysis-Twitter 3.3.1
http://snowball.tartarus.org/algorithms/german/stemmer.html
http://snowballstem.org/
http://www.nltk.org/api/nltk.stem.html?highlight=stemming
create a custom stemming function
regex + stringr. Exclude words etc etc
https://mimno.infosci.cornell.edu/papers/schofield_tacl_2016.pdf
http://eprints.whiterose.ac.uk/1434/1/willettp9_PorterStemmingReview.pdf

http://eprints.whiterose.ac.uk/1434/1/willettp9_PorterStemmingReview.pdf
http://snowball.tartarus.org/texts/introduction.html

What happens with negation

Is the phrase "not uncommon" the same as "common"? (I suspect not)

Fix color output in tibbles

normalize .bib files

Right now we have two .bib files (book.bib and references.bib). Should make more sense to move everything into one for now.

how to install TensorFlow for GH Actions

https://github.com/tidymodels/embed/blob/master/.github/workflows/R-CMD-check.yml

In section 2.4.2 caret in in-line code is being parsed as footnote

In the custom block in section 2.4.2 the in-line code lapply(str_remove_all, pattern = "^[:punct:]+|[:punct:]+$") is being parsed correctly. The caret ^ in the regex is being parsed as a footnote.

ethical issues

• Word embeddings quantify 100 years of gender and ethnic stereotypes
• Introduction to Algorithmic Bias
• What you need to know about Disinformation

Linguistics for NLP practitioners

https://faculty.washington.edu/ebender/100things-sem_prag.html
https://www.cl.cam.ac.uk/teaching/1314/L100/introling.pdf
https://medium.com/@ODSC/linguistics-in-nlp-why-so-complex-4fd1fb9873a3

general addition

phatic expressions

we have 2 `preface`s

Add sentence about assumed modeling experiance

https://text-and-modeling-in-r.netlify.com/preface.html#who-is-this-book-for

It could be a rewording of "machine learning".

Add a link to "if you don't have much experience with predictive modeling"

embedding bias resource list

https://kawine.github.io/blog/nlp/2019/09/23/bias.html
https://www.aclweb.org/anthology/P19-1166/

add typographical conventions section

• italics
• constant width
• bold

etc

tokenization - discussion on tokenizing by spaces

We commonly use spaces as a tokenizing boundary. But this is not always the best choice.

it might behave poorly with double spaces, accidental or otherwise.

• Chinese doesn't use spaces between words
• Vietnamese uses species within words
• German combines multiple words into one

Spaces in words also change over time
today used to be two words "to day"
https://www.etymonline.com/word/today

Set global knitr options

Make sure to have reasonable defaults for things like fig.width, fig.height etc

Using predictive features such as POS tags and dependency parse relations

Wrongly fit a model

Try to fit a model but using wrong pre training (law / fairytale / advertisements / news)

fit a model trained on the wrong language. This should easily work but would give (hopefully) weird results.

write section on constructing custom features

Possibly better option for displaying wall of text in 3.2 and similar?

Instead of creating an image, it might be nicer to have HTML to display the "walls of text" in 3.2, similar to what Bob did here.

Switch to new_session: yes?

https://github.com/EmilHvitfeldt/tidy-nlp-in-R-book/blob/7741776c8ba56596e18991a16d782458676e3e88/_bookdown.yml#L1

If we switch this to yes, then each chapter will knit in a separate session, making each chapter stand alone more cleanly. Right now, the whole book gets pasted together and knit as one. Thoughts?

Make sure packages are referenced first time they are mentioned

Explain tf-idf

move tidytext intro to end of tokenization chapter

fix cross references

Create _chapters folder

stop words removal using non-english languages

first step is to find good languages that displays problems not faced in english.
second step is to find texts for examples in those languages.

Add quick note about change in default theme

https://github.com/EmilHvitfeldt/tidy-nlp-in-R-book/blob/b719bdde30faf4950c85f52a4754fd1cd1a862d9/model_language.Rmd#L10

Lemmatization

Look into expanding lemmatization section. Possibly move it to further reading

Lexicon bias

NRC

white - trust
black - sadness

Add some additional discussion in section 1.2 for Danish

Add a little bit of discussion (similar to existing discussion for English) to section 1.2 for Danish.

Talk about license

License in repo is currently CC0, which does not have any attribution or non-commercial bits in it. As is, this would let somebody else take the book content and sell it themselves. We probably want to change this to something like CC-BY-NC-SA.

make sure words like "document" and "corpus" are explained

fix (PART)

Clarify the difference between characters, strings and character vectors

character vectors is a series of strings which themselves are a series of character symbols

use of the word glyph

Make sure the word glyph is explained before it is used

Add links to "topic this book will not cover" for suggested resources for those subjects

https://smltar.com/preface.html#topics-this-book-will-not-cover

Lots of princesses

The end of https://emilhvitfeldt.github.io/tidy-nlp-in-R-book/stemming.html#compare-some-stemming-options includes a lot of duplicate rows. Is that intentional or an error?

medical lexicals

swap nearest_synonyms() to nearest_neighbors()

The formal definition of synonym in lexical semantics is different then what is commonly used for word embeddings and it might cause confusion.m

global, subject and document stop words

write a short paragraph about how stop words can be roughly into global, subject and document stopwords.

Find source to the creation of the SMART stopword list

It appears that the word list is machine generated and I would like a confirmation on that.

Have "end of chapter" summaries

Limitations of word embeddings when viewed from the lens of lexical semantics

"One word is mapped to only one vector and all the senses of that word are conflated while training embeddings. This is a limitation because the same vector would represent book in I was reading this amazing book and I wanted to book a meeting"

Add a Further Reading section at the end of each chapter

Provide research papers, blog posts or books for the eager reader to delve deeper into that particular topic.

It can also be used for related topics that didn't make the cut in the chapter.

Plots are not rendering

I haven't gotten a chance to dig into what's going, but I think because of where the chapters are stored, knitr is writing the images somewhere that then the website can't find. Images are not rendering on the website, i.e. here:
https://emilhvitfeldt.github.io/tidy-nlp-in-R-book/stemming.html#compare-some-stemming-options

stemming buckets diagram