kmcelwee / fortune-100-blm-report

Analysis of Fortune 100 twitter data

Hello! My name's Kevin 👨🏻‍💻

I earned a degree in Music at Princeton in 2018 and soon after discovered programming through data journalism. For two years, I worked for electric utilities, buiding models that increase efficiency on the grid; then for two years, I returned to Princeton as a programmer on staff, writing software for professors and students. I moved to Boston in 2022 and now work at AWS, helping build a cloud-based file system called FSx.

Check out my repos that explore...

⚡ The Energy Sector

Partnered with rural utilities, national labs, and the Department of Energy, I studied how deep learning can be used to predict energy consumption. Increasingly accurate energy predictions can help utilities save a lot of money through a process called "peak shaving." Read about my deep learning research.

After utilities showed more interest in my research, I partnered with Burt County Public Power and NRECA to build a web app that automatically draws in weather data and load data every day to generate a forecast. Check out the product we built.

📈 Data Journalism

• kmcelwee/fortune-100-blm-report: an analysis of the Fortune 100's response on Twitter to the BLM protests in the summer of 2020
• kmcelwee/WhoPaysWriters: an analysis of the website WhoPaysWriters (read the article in the Columbia Journalism Review)
• kmcelwee/scrabble: an analysis of what Scrabble tile values make the game more fair (read the article in Nautilus)
• Read more on my website...

🤖 Twitter bots

• kmcelwee/reddit-says-aww: A bot that posts the top-rated content from the r/aww subreddit to Twitter using GitHub Actions
• kmcelwee/mondrianify-twitter: A wrapper for my mondrianify pipeline.

fortune-100-blm-report's Issues

Make gists and link for each calculation

a la The Correspondent. Span with invisible content attached to word. On click span becomes block that contains content.

Write generate script for weekly fortune 100 tweets overview

Rounding errors found in IDs

Oembed process revealed IDs that had been rounded:

1265004698148495400
1265812305817862100
1266080125805834200

This exists within the dataset in fortune-100-blm-dataset. I believe I manually entered values for Lowe's because of API limits, so that might be what's happening.

• Request oembed for all tweets ending in 00 to double check that it's limited to these tweets.
• Dig into fortune-100-blm-dataset repo and double check scripts.
• Pandas automatically reads the ID column as an integer. Research that this doesn't cause issues.

Adapt scripts to new dataset structure

After stripping out the blm-tweets.csv and changing headers, current make scripts shouldn't work

Create interactive tweet histograms with hover as twitter embed

On the project's index.html, list all companies and their histograms. First just get it done with simple D3, then try to implement hover as twitter embed. Place data files in a docs dir

Improve OEmbed error handling

• Create a file reviewed-oembed-errors.txt that lists IDs reviewed in #14.
• Update collect-twitter-oembeds.py to print out errors that are new.
• Update collect-twitter-oembeds.py to include returned messages with "error" to be included in the "error" field.

Add footnotes of code snippets to each number / statement

This is not supported from GitHub's native markdown, but still important

Include replies to self

Spike on implications of slightly altering reply / thread filtering.

https://twitter.com/pfizer/status/1274002179368652807

Create legend

This isn't straightforward in D3. Not sure how to best move forward

Explore what new questions might be answered by Twitter API v2

Ensure that #blacklivesmatter search is not case-sensitive

Intel tweeted #blacklivesmatter but I may have not looked properly

Try to remove bootstrap

Some tooltip CSS is making things look nice, but it's not necessary.

Highlight by tag

Create an interface that highlights (with large yellow circle?) the tweets with the selected tags

Twitter screenshots useful?

Some of the media within tweets could be useful, and it might be a last-ditch way to get twitter-like tooltips.

Standardize all column names

all lowercase, spaces replaced with underscore

Error viewing ID 1266171968728043520 in oembed

Include it in error list for twitter-oembeds.json, rewrite logic to include. Also research what this used to be.

Try alternating color schemes for each text section

.section:nth-child(odd)

Summarize "BLM" by sector

Add a "Last Updated" item to corporate summaries

Spike options for archiving citations

Request some kind of web archive?

Spike: How to make tweet histograms larger

We have both the static images and the interactive visualizations. For static images, we can place the legend above the histogram instead of to the right. For dynamic images, see how we can increase circle size, reduce margins, and ensure that the graph is taking up the full given width. Also consider widening default text block width.

Address errors in twitter oembeds

found in docs/twitter-oembeds.json with the key "errors"

Add non-twitter-using companies to data viz

Add a brief section about companies without twitter accounts

Add a clear button to data viz interface

Spike implications of "Twitter for Advertisers"

https://twitter.com/CocaCola/status/1268235977614655488

These tweets do not show up in either search or scrape. I found this through a CocaCola reply. There's a lot to go through with this exception

Experiment: place blur on background headers

Include search functionality

https://observablehq.com/@jashkenas/inputs @ autoSelect

Introduce figcaptions

Experiment: Place all notes material after the paragraph

Design some kind of block after each paragraph where we can place images, gists, links, etc.

Import protest data and graph

Summarize full history of #BLM

Make corporate visualizations own id

Edit HTML s/t the ids don't need to be rendered by D3. This may allow the redirect to jump directly to the desired visualization

Create script to search each companies entire history for "BLM", "Black Lives Matter", or "#BlackLivesMatter"

Wrap report better

Container within the report to hold paragraphs and headers? Otherwise it's just explicitly declaring the width of headers and paragraphs and images

Convert IDs to strings, browser can't handle large ints.

IDs are rounded when requested in D3 (or at least in the browser), making them useless (e.g. 1272562122258604032 -> 1272562122258604000). They need to be stored as strings.

Spike on publishing options

Isn't combining markdown with charts what Jupyter was made for? But I want a friendlier medium to any reader... so a proper website? I don't want to invest so much time in that! I could generate a website from Markdown which points to the files generated here. The only concern is interactive graphs, but D3 has certainly been used in GitHub pages elsewhere.

Address oembed error with tweet ID 1280186066469822464

Publish profiles of each quiet corporation

Explore options to organize RJ media into graphic

"Themes of BLM media" might be an interesting section or sidenote. The data has already been collected for the header.

Add "Juneteenth", "BLM", etc. tags

Use Bootstrap tags to insert into visualization on hover.

Write script for "first tweet" histogram

This could be either static or D3. Maybe just static for now.

Shift all right-aligned text to left-align

Add fortune 100 rank to data visualization

Separate report and corporate summaries

Create an index.html with the report and then a all-corporations.html for summaries

Write generate script for number of corps tweeting about RJ

Include max_count in histogram JSOn

This calculation is not straightforward in JS. We can just include it directly in the histogram.json

Slim down oembed JSON

Only need HTML. We can also strip out the script widge js`

Ensure data visualizations are responsive

https://medium.com/@louisemoxy/a-simple-way-to-make-d3-js-charts-svgs-responsive-7afb04bc2e4b

Double check every tweet

After finishing data visualization, run a final pass through all tweets to ensure consistency

Reintroduce hyperlinks and styling