ayushsubedi / datainnews_v2

Data in News project, redone using Github Actions.

Data in NEWS

Data in NEWS portal

How journalists use data in their reporting or in their writing can be viewed as a reflection of the country’s demand for statistics and data. Back in 2020, I developed an indicator model (for The World Bank) to assess the use of data by Nepali news portals based on the methodology proposed by Klein, Galdin, and Mohamedou 2016.

The results from the project was used in the research article published at http://documents1.worldbank.org/curated/en/805261601023506163/pdf/Use-of-Data-in-the-Private-Sector-of-Nepal-The-Current-State-and-Opportunities-in-Finance-Education-and-the-Media.pdf

As of 9th February 2020, the results showed that very few news articles indicate the source of data or mention development indicators. However, articles that discuss data, reports, research, statistics, and related topics do critically engage with these ideas.

Unfortunately, the data collection process was halted because we ran out of our AWS Activate for Startups credits.

TODOS

• create a simple architecture diagram
• curate a list of twitter handles of Nepali newspapers that put URL to their newspapers
• use twint to create datasets (historic)
• check if url is present, url does not 404, and url belongs to the publisher
• use Newspaper to collect important information (author, etc) and complete the dataset
• research dash, swifter and pandarallel
• optimize regular expression functions (if possible this should be performed during scraping)
• create a sqlite database to hold access information (and design schema around other meta) or find and alternative
• add checkpoints (using twint resume or build it manually)

POC

• create a full working version of the product
• research downloading twint as pandas to merge with previous data

Twitter handles

• @kathmandupost
• @thehimalayan
• @NepaliTimes
• @OnlineKhabar_En
• @RepublicaNepal

The Frugal way

• This project leverages on Github Actions for scheduling and scraping purposes.
• This project is hosted on Heroku free tier (free ssl)
• Uses amazing open source projects, primariliy (Twint, Newspaper) etc

datainnews_v2's Issues

create a pandarell poc for newspaper3k

convert url from list type to string type

create a social sharing image

update requirements for poc

complete responsiveness

complete notebook for nepalitime

create a data architecture diagram

perform level1, level2, level3 analysis on nepalitimes

research dash, swifter and pandarallel

clean nepalinews csv

deploy to heroku

check url from the actual news source (some news papers share youtube videos)

create a poc to resume from last analysis

check if url is present, url does not 404, and url belongs to the publisher

change all url to redirect url

optimize regular expression functions (if possible this should be performed during scraping)

experiment with an image background

experiment with different templates for v2

create twint poc on flask context

create a loop for extracting from twint and storing csv

create a basic template to verify addition of new articles

only collect data from 2018

things should work smoothly when empty tweet returned

change website text to update for current implementation

curate a list of twitter handles of Nepali newspapers that put URL to their newspapers

mobile responsive

prepare server for heroku

clean kathmandupost csv

fix inconsistent axis labels

create a basic flask app

add line charts to level1, level2, and level3

Instead of RSS, create a poc to check if twitter is more of a reliable source for scraping

add checkpoints (using twint resume or build it manually)

use Newspaper to collect important information (author, etc) and complete the dataset

complete poc with nepalitimes

add onlinekhabar

use twint to create datasets (between 2018-2021 March)

newspaper not working on kathmandu post

update flask to 2.0 and test async compatibility with twint

create a poc for url extraction with kathmandu post

create a poc line chart

complete historic csv (up until today)

create generic route that can be replicated for any number of newspapers

complete poc with two newpapers

create a financial pitch

create a separate function for url extraction for kathmandupost

make table responsive

move notebooks to a different repo

create a sqlite database to hold access information (and design schema around other meta)

create helper functions and variables