mohnishkalia / headlinearchiver

Codebase for Headline Archiver project

Client Quickstart

cd client/
npm run start

Cloud Function Quickstart

cd cloudfunc/functions/
npm run deploy

headlinearchiver's Issues

Lots of clutter on the web pages, from both ads and ToC popups

• Utilize xPath navigation to scroll down to a navbar to get full window
  • ex. /html/body/div[5]/div/div/header/div for CNN's navbar
  • Utilize extension loading in puppeteer
    • UBlock Origin
    • Include in puppeteer.launch args
  • puppeteer-extra

App Improvements

• Intro text at top of screen
  • "Comparing CNN and Fox headlines at a specific time, click or tap to expand image"
• Enable firebase analytics right after initialize app
• Infinite scroll with use of last divider element and lib to detect if it is in view
  • Divider in view = get more entries
  • RIO
  • Utilize useEffect with deps user and RIO inView to query more data
• Allow the user to toggle between timeline view and a data grid view
  • Migrate the search and request more button there
  • Toggle Buttons

Allow the user to toggle between timeline view and a data grid view

• Migrate the search and request more button there
  • Toggle Buttons
• Create a context for global state
  • Unified state for shots or separate between table and timeline?

Add min date to pickers, as well as user auth on querying for search view

Enable firebase analytics right after initialize app

Use function metrics to constrain timeouts and memory usage

• Use function metrics

Transfer utility functions to utils.ts

• Change firebase.ts to utils.ts
• Utilize createContext and useContext

Infinite scroll with use of last divider element and lib to detect if it is in view

• Divider in view = get more entries
  • RIO
  • Utilize useEffect with deps user and RIO inView to query more data

Create board to manage issues

Scraping Improvements

• Lots of clutter on the web pages, from both ads and ToC popups
  • Utilize xPath navigation to scroll down to a navbar to get full window
    • ex. /html/body/div[5]/div/div/header/div for CNN's navbar
  • Utilize extension loading in puppeteer
    • UBlock Origin
    • Include in puppeteer.launch args
  • puppeteer-extra
• Include performance metrics on every console log
  • Create function after now that will pretty print the time when the invocation was reached along with a message as a parameter
• Use function metrics to constrain timeouts and memory usage

Add OCR for images

• Allow for a "See Word Cloud" on an image, cloud function or client side thing that parses using tesseract and then renders out a word cloud
• Eventually allow for multiple images or a time range

Include performance metrics on every console log

• Create function after now that will pretty print the time when the invocation was reached along with a message as a parameter

Intro text at top of screen

• "Comparing CNN and Fox headlines at a specific time, click or tap to expand image"