programming-from-a-to-z / a2z-f18

ITP A2Z Course for Fall 2018

Programming-from-A-to-Z

(or Everything you wanted to know about text but were afraid to ask.)

(or How i learned to stop worrying and love the javascript.)

What can computation add to human communication? Creating computer applications, instead of just using them, will give you a deeper understanding of the essential possibilities of computation. The course focuses on the fundamentals of programming the computer in the context of language and words. We'll explore topics ranging from evaluating text according to its statistical properties to the automated production of text with probabilistic methods to text visualization. Students will learn the JavaScript programming languages and develop projects that can be shared and interacted with online. The end of the semester is spent developing an idea for a final project and implementing it using computer programming.

Week 1 -- Introduction

• Why are we here? What is computational media? What is programming? What is it good for? What kinds of programming languages are there?
• What makes up a web page
  • Structure: HTML
  • Style: CSS
  • Behavior: JavaScript
• Server-side vs. client programming

Week 2 -- Generate text automatically

• basic HTML/CSS
• variables
• simple objects
• Strings

Week 3 -- Make text interactive

• conditionals
• simple functions

Week 4 -- ??????

• more about functions and objects

Week 5 -- Make lots and lots of text

• arrays

Week 6 -- Regular Expressions

Week 7 -- Generating Text - PROJECT 1

• Markov
• Context Free Grammar

Week 8 -- Analyzing Text - PROJECT 2

• concordance
• visualization / drawing

Week 9 -- Bonus Topics

Week 10 -- Propose Final Projects

Week 11 -- One on one speed user testing / feedback

Week 12 -- Final Project Presentations

a2z-f18's Issues

Context Free Grammar Example with JSON has invalid grammar!

This grammar is not valid:
https://github.com/shiffman/A2Z-F18/blob/master/week7-grammar/03_cfg_reader_json/grammars/grammar.json

It uses the same key multiple times:

{
  "S":"NP VP",
  "S":"Interj NP VP",
  "NP":"Det N",
  "NP":"Det N that VP",
  "NP":"Det Adj N",
  "VP":"Vtrans NP",
  "VP":"Vintr",
}

It should use an array for each element instead.

Mastodon notes

The twitter notes should be deprecated and a new page should be made for Mastodon.

https://shiffman.net/a2z/twitter-bots/

word2vec video tutorials

Revamp Twitter Bot Assignment

Given Twitter's recent changes in its API policies and procedures, I would like to look into using other platforms for text / image based bots. Mastodon?

LSTM with ml5.js video tutorials

Projects for text analysis

add Partisan thesauras
http://partisanthesaurus.com/

https://shiffman.net/a2z/text-analysis/

ES6 updates to examples and notes

The a2z notes still point to older examples.
Also the notes should be updated to use ES6 (and ES8 where applicable!)

https://shiffman.net/a2z/

REGEX used to split accented characters

As always, thanks for sharing all this code!

While using it with a spanish text, I found that it didn´t work because of special characters. In concordance.js we have this code splitting the array

// Splitting up the text
        split(text) {
            // Split into array of tokens
            return text.split(/\W+/);
        }

Unfortunately, accented (diacritic) characters are also non-alphanumeric, so word like "selección" and "niño" get chopped into "selecci", "n", "ni", "o" by using that REGEX.

I found a workaround, by using match instead of split

var re = /\S+\s*/g;
tokens = allwords.match(re);

This of course required me to also change the previous code a bit, into

txt = loadStrings('preguntas/todas.txt');
allwords = txt.join("\n");

My proposed solution is not very good either, because after splitting into tokens, I still had to hand clean for many other non-alphanumeric characters, whitespace, line breaks, etc. But that was done while "sanitizing" each word before adding it into keys, and counts. For example

for (var i = 0; i < tokens.length; i++) {
        var word = tokens[i].toLowerCase();

        // Clean some more
         word = word.replace("(", "");
         word = word.replace(")", "");
         word = word.replace(".", "");
         word = word.replace(finBlanco, "");
         word = word.replace(/(\r\n|\n|\t|\r)/gm, "");



        if (!/\d+/.test(word)) {                     // is not a number
            if (sw.indexOf(word) == -1) {            // is not a stop word within a custom sw array
                if (counts[word] === undefined) {    // is a new word
                    counts[word] = 1;
                    keys.push(word);
                } else {
                    counts[word]++;
                }
            }
        }
    }