tomlinsonk / site-graph

Visualize the link graph of a website.

Welcome to my GitHub 👋

I write code for my research and sometimes for other things.

My research repositories

• plurality-replicator-dynamics
  📝 Replicating Electoral Success w/ Tanvi Namjoshi, Johan Ugander, and Jon Kleinberg
• irv-moderation
  📝 The Moderating Effect of Instant Runoff Voting @ AAAI 2024 w/ Johan Ugander and Jon Kleinberg
• graph-based-discrete-choice
  📝 Graph-Based Methods for Discrete Choice @ Network Science w/ Austin R. Benson
• irv-ballot-length
  📝 Ballot Length in Instant Runoff Voting @ AAAI 2023 w/ Johan Ugander and Jon Kleinberg
• choice-set-confounding
  📝 Choice Set Confounding in Discrete Choice @ KDD 2021 w/ Johan Ugander and Austin R. Benson
• feature-context-effects
  📝 Learning Interpretable Feature Context Effects in Discrete Choice @ KDD 2021 w/ Austin R. Benson
• choice-set-opt
  📝 Choice Set Optimization Under Discrete Choice Models of Group Decisions @ ICML 2020 w/ Austin R. Benson
• diverging-string-seqs
  📝 Summarizing Diverging String Sequences, with Applications to Chain-Letter Petitions @ CPM 2020 w/ Patty Commins, David Liben-Nowell, and Tina Liu
• cca
  📝 Phase Lengths in the Cyclic Cellular Automaton @ CSC 2019

My project repositories

• site-graph
  🕸️ Makes pretty website link graph visualizations
• kit-emu
  🛠 Emulator for my homebrew KiT computer
• kit-6502-code
  💾 65C02 assembly programs for my homebrew KiT computer
• optimal-optimizers
  📉 Final project for CS 6787: Advanced ML Systems comparing PyTorch optimizers for training discrete choice models
• trashketball
  🏫 Educational tool created for UNC Charlotte Department of Special Education and Child Development
• euler
  🧩 My solutions to Project Euler problems
• mandelbrot
  🧮 Renders the Mandelbrot set
• civ6hof
  🎖️ An online Hall of Fame for Civ 6 (deprecated)
• gravity
  🪐 2D n-body gravity simulation
• flock
  🐦 2D flocking simulation, à la Boids

site-graph's Issues

Do not recommend using --force when --force was specified

p site_graph.py http://127.0.0.1:8080/ --visit-external --force
Warning: not using https. If you really want to use http, run with --force

either I do something wrong way or error is a bit confusing

Consider adding a license

See https://choosealicense.com/ for tl;dr and to confirmation that it is a real problem

Currently no license is mentioned anywhere, what makes this code fully copyrighted. This is a default for any creative work - what includes software.

Please, please add a license. The fact none is listed makes using this software a legal quagmire.

For start - people are not allowed to make pull requests with improvements as it would require distribution of modified code - that is fully copyrighted.

For example currently it is not legal to use, improve or distribute this code. Hopefully this is not the intended effect, given that code was open sourced.

In case that for some reason not releasing it under open license is intentional - mentioning it in the readme also would be a good idea. Though note that given that it is open sourced it will not stop anyone not caring about copyrights.

Popular licences include:

• MIT license that allows people to use code for any purpose
• GNU GPLv3 that intends to allow everything but to block people from distributing closed source versions that build on open source project
• GNU AGPLv3 that in addition requires providing source code when providing software as a service

Feel free to ask me questions :) I know and understand that people often prefer to ignore copyright law, unfortunately it will not make it disappear.

I am posting this as this code fits almost exactly what I need and it would be sad to write my own one from scratch.

Errors with accents or emojis in url?

Urls like https://example.com/tags/🌳arbré are valid in a browser. It would be interpreted like this https://example.com/tags/%F0%9F%8C%B3arbr%C3%A9/

It seems like the script does not do the interpretation work and throws error instead.
Are you aware of this? Is it fixable?

supporting local files

it be OK to make PR adding support for reading local files?

It would be useful for me as I am trying to make/find a decent tool for detecting broken links.

And possible additional feature: detect files that are within folder but are not linked (orphaned html file, not linked from any part of the website and/or unused media files)

(would be happy to make PR if that would be accepted and #3 is resoved)

Add an option to crawl and check also other resources

So not only direct <a> links but also <img src=, favicon, css resources etc.

It would be useful for me in checking for link rot.

(if it is out of scope - no problem!)

function to_numpy_matrix() depreciated in recent networkx version

Hello,

The function to_numpy_matrix() called line 145 (see link below) to produce the adjency matrix is depreciated since networkx 3.0.

If changed to to_numpy_array() it works fine.

https://github.com/tomlinsonk/site-graph/blob/06b0ef38eb58781e20f94bde6cd71009824cdaa3/site_graph.py#L145C1-L145C70

Consider editing repository setings to remove "Packages" section

"Packages No packages published" is displayed right now, fortunately this pointless section can be removed.

Edit repo page config to remove it (cog next to the description).

I am not making a PR as it is defined in proprietary github settings, not in a git repository - and I have no rights to modify repo settings.

Maybe also releases section should be removed?

consider enabling --visit-external by default

At least for me it would be quite confusing to have error checked and shown as broken internal links and the same not happening with external links

At least I was confused. Can it be enabled by default?

consider changing color for error

red nodes are external pages, and yellow nodes are pages with errors

Maybe it would make more sense to use black/red/eyesore red/magenta for errors?

Red for errors is quite typical, maybe color like

would be even better?

(would be happy to make PR if that would be accepted and #3 is resoved)

ModuleNotFoundError: No module named 'bs4'

Hello,
by executing the installation and configuration as described:

git clone https://github.com/tomlinsonk/site-graph.git
cd site-graph
pip3 install -r requirements.txt

everything seems working fine.
Then, trying to crawl the website from inside the site_graph folder

python3 site_graph.py https://marcoXbresciani.codeberg.page

I see this error:

Traceback (most recent call last):
  File "C:\Users\marco.bresciani\Chiavetta\Software\site-graph\site_graph.py", line 2, in <module>
    from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'

I've tried by pip3 install bs4 but no luck.

• Python 3.9.12
• pip 23.2.1 from C:\Python39\lib\site-packages\pip (python 3.9)

joomla

Hello Dear Sir!

I am trying to use you software (its prety cool by the way) but it seems to have a problem with joomla where the links containt article id-s and all subpage is basicly baseurl/index.php...id=654. what can i change or do to make it visualize my entire site including this type of links?