anvaka / sayit

Visualization of related subreddits

visualization of related subreddits

This project builds a graph of related subreddits.

Recommendations are constructed based on Redditors who commented to this subreddit, also commented to...

Play with it here: https://anvaka.github.io/sayit/

The data

I used data from two months worth of comments (August and September of 2018) - which contains ~38 millions user <-> subreddit records.

You can find original data by following this discussion

I computed Jaccard Similarity between subreddits, and then stored results into github pages. Repository is available here. Please let me know if you are curious to learn more about this or anything else - feel free to reach out to me on twitter or via issues in this repository

Note: for very popular subreddits Jaccard Similarity didn't give meaningful results. They all were connected to each other (e.g. /r/aww, /r/pics, /r/funny and so on). I manually collected references to other subreddits from subreddit description where it was available. Where description did not include any recommendation - I looked into actual comments and used most often mentioned subreddits as "related". You can find list of all overrides in the sayit-data repository

Local Build Setup

# install dependencies
npm install

# serve with hot reload at localhost:8080
npm run dev

# build for production with minification
npm run build

# build for production and view the bundle analyzer report
npm run build --report

For a detailed explanation on how things work, check out the guide and docs for vue-loader.

Thanks!

Credit for data goes to Jason Baumgartner also known as u/Stuck_In_the_Matrix.

Huge thanks to Felipe Hoffa for putting the data into BigQuery.

If you like this work and would like to support it - I have a Patreon page, paypal.me, github sponsors.

Thank you!

sayit's Issues

51 vulnerabilities (23 moderate, 28 high)

Very interesting repo. Congrats.
There is a small challenge, npm install returns a warning.
51 vulnerabilities (23 moderate, 28 high)

How to get 'more recent' data?

Was curious how to get 'like the last 2 months' instead of stuff from pre-determined date range?

npm install failed

OS: Ubuntu 18.04
npm version: 3.5.2
error message:

npm WARN deprecated [email protected]: Deprecated. Please use https://github.com/webpack-contrib/mini-css-extract-plugin
npm WARN deprecated [email protected]: out of support
npm WARN deprecated [email protected]: Browserslist 2 could fail on reading Browserslist >3.0 config used in other tools.
npm WARN deprecated [email protected]: core-js@<3.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could
cause a slowdown up to 100x even if nothing is polyfilled. Please, upgrade your dependencies to the actual version of core-js.
npm WARN deprecated [email protected]: Browserslist 2 could fail on reading Browserslist >3.0 config used in other tools.
npm ERR! Linux 5.4.0-42-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install"
npm ERR! node v8.10.0
npm ERR! npm  v3.5.2
npm ERR! code EMISSINGARG

npm ERR! typeerror Error: Missing required argument #1
npm ERR! typeerror     at andLogAndFinish (/usr/share/npm/lib/fetch-package-metadata.js:31:3)
npm ERR! typeerror     at fetchPackageMetadata (/usr/share/npm/lib/fetch-package-metadata.js:51:22)
npm ERR! typeerror     at resolveWithNewModule (/usr/share/npm/lib/install/deps.js:456:12)
npm ERR! typeerror     at /usr/share/npm/lib/install/deps.js:457:7
npm ERR! typeerror     at /usr/share/npm/node_modules/iferr/index.js:13:50
npm ERR! typeerror     at /usr/share/npm/lib/fetch-package-metadata.js:37:12
npm ERR! typeerror     at addRequestedAndFinish (/usr/share/npm/lib/fetch-package-metadata.js:82:5)
npm ERR! typeerror     at returnAndAddMetadata (/usr/share/npm/lib/fetch-package-metadata.js:117:7)
npm ERR! typeerror     at pickVersionFromRegistryDocument (/usr/share/npm/lib/fetch-package-metadata.js:134:20)
npm ERR! typeerror     at /usr/share/npm/node_modules/iferr/index.js:13:50
npm ERR! typeerror This is an error with npm itself. Please report this error at:
npm ERR! typeerror     <http://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /home/m/miscel/sayit/npm-debug.log

thanks.

Some Old subs are not showing up?

hey, i am trying hard to grow Ollywood sub. For some reason the tool is not generating anything or even recognising the sub. could you please check? i am sure low member count is not the reason as i have seen subs with lower count than mine are showing up.

please fix this.

Subreddits with names starting with numbers break edge generation

Lines 75/78 on createRenderer.js break when the subreddit names starts with a number, like 4chan, or 3dprinting.

Really cool tool though. Thanks for making it.

dark mode

dark mode settings would be appreciated.

im using force dark mode on chrome and its impossible to read the white text with white bubbles. Thank you

How can i fixed

i need help please

Edge not drawing when search for r/blender

When search blender the edge are not drawing https://anvaka.github.io/sayit/?query=blender

work for others subs

Could you integrate this with the reddit API, it doesn't work with subreddits created after 2018

it would be cool if the the website was kept up to date, and maybe it could track the change in subreddit relations over time

Reddit CORS block disables the data fetching

"Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://www.reddit.com/r/geologycareers/hot.json. (Reason: CORS request did not succeed)."

One will need to use some JSON-fetching proxy I guess... until they block that server.

How come this differs so much from what "Similar subreddits" on subredditstats.com shows

What it says in the title.

Export

Can you please add an export button, either as JSON or a simple human readable format?
Even just the list of subreddits to copy and paste would be great

Alternative Algorithm based on User Overlap?

Checked https://subredditstats.com/subreddit-user-overlaps

Exclude nsfw subreddits by default

I would imagine, when you fetch the subreddit lists, that they have some sort of NSFW boolean. If this is the case, I think it would be good if the nsfw subreddits were filtered out of results by default, and perhaps enable them with a 'show nsfw' button? Otherwise, great little web app, kudos!

Jaccard Similarity Biased on Bipartite Graphs

You should consider using Standardized Co-incident Ratio (SCR) as a measure of similarity instead of the Jaccard Similarity, as the latter is biased in large bipartite graphs whereas the former is not: https://yongrenshi.weebly.com/uploads/5/7/8/6/57861243/ssr_structuralsimilarity.pdf. Should be a simple switch.

UI improvement suggestions

• It generally does a good job encompassing the full map, but sometimes it does not zoom to the full extent of the created map
• Sometimes the lower tier links get cluttered, might be cool to have a clickable button that would collapse/expand a given connection at a certain level OR space out the lower tier connections more
  
• The preview window does not work in firefox
  
• a "clear" or back button would be nice, although it does a great job of seamlessly transitioning between different searches.
• as a note, i love the constructing layout percentage display, users love that shit and i certainly do lol
• a scroll over effect that highlights the direct connections would be neat

Accessibility feature request: Add toggle to view as Redsim-style list (not graph)

Hi, I really like both this project and redsim, but I've found redsim's simple list interface is much easier to read and navigate from an accessibility standpoint. I see that sayit uses more up-to-date data, though. Would it be possible to add an option, possible a simple toggle, to view sayit's data as a list like redsim shows?

(Alternatively, could you just make redsim use sayit's updated data set?)

Please show the red edges and rectangles for the current query without requiring the user to click on its node.

Also, please don't show a preview of the subreddit when the user clicks on a node.

api?

Do you have an api to get related subreddits?

For example if I type in "Linux" i want an api that returns all related subreddits.

Question: Data used for graph?

I just discovered this project today and did not know the best place to ask a question (let me know if I should put this elsewhere). What data is being put into the network graph? I was reading the README and it says only data from a few years ago but when playing with the web ui I saw posts more resent. Is this now on data being constantly updated? If so is there a place where I can look through the code. Curious to learn more!

Just a thought on network construction methods

First of all, I loved your visualization, ideas etc.

However, I tested your alg. with smaller subs and I got some unrelated answers. I got that you use the method called (people who posted, also posted that) which is fair.

Would you think using subreddit mentions as form of edges can be much accurate?

If you are browsing subreddit A. and that is a small one, but its "relateds" can be subs that are mentioned there. (like citation graphs.)

check r/babysteps for example. thats a manga sub, but the relateds are completely irrelevant, although it could have been r/anime,/manga.

I recommend you to read this https://en.wikipedia.org/wiki/Citation_graph. can be useful. In terms of this wiki, its like building a network of publications where people who published this also published that. can logical, but may not be related.

still I love the idea. :)

Missing new subs?

I noticed some subreddits are not on here, like ChatGPT.

Will this be fixed?

npm found 2 vulnerabilities (1 moderate, 1 high)

thanks for the fantastic tool. tried to run it myself locally, and thought I should share the warnings that latest LTS node + npm showed me:

npm audit text

added 1231 packages from 684 contributors and audited 10704 packages in 11.154s found 2 vulnerabilities (1 moderate, 1 high) run `npm audit fix` to fix them, or `npm audit` for details ➜ sayit git:(master) npm audit

                   === npm audit security report ===

Run npm install --save-dev [email protected] to resolve 1 vulnerability

SEMVER WARNING: Recommended action is a potentially breaking change
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ High │ Missing Origin Validation │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ webpack-dev-server │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ webpack-dev-server [dev] │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ webpack-dev-server │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/725 │
└───────────────┴──────────────────────────────────────────────────────────────┘

Run npm install --save-dev [email protected] to resolve 1 vulnerability

SEMVER WARNING: Recommended action is a potentially breaking change
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate │ Regular Expression Denial of Service │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ mime │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ url-loader [dev] │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ url-loader > mime │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://nodesecurity.io/advisories/535 │
└───────────────┴──────────────────────────────────────────────────────────────┘

found 2 vulnerabilities (1 moderate, 1 high) in 10704 scanned packages
2 vulnerabilities require semver-major dependency updates.