marmust / internet-scanner

see the internet as a physical network of web pages

internet scanner

see the internet as a physical network of web pages

a video explaining everything: https://youtu.be/qkOem3HyR_w

project is fully open source, and you can use the code for any purpose (it will be nice if you credit 8AAFFF and all contributors tho)

To run the build:

1. Download Windows release from the Releases.
2. Unzip the downloaded file to a new folder
3. Run graph renderer.exe
4. You may get an error saying "Windows Protected Your PC". This is normal, click the more options link, then click the Run Anyway button.

The build is only for windows however, you're out of luck if you run any other OS.

To open the Unity Project:

1. Make sure Unity is installed properly on your system (the project is currently being developed in version: 2020.3.35F1).

Please note: If you do not have version 2020.3.35f1 installed, nor see the option to install it in Unity Hub, simply try opening the project in Unity Hub anyways. Unity Hub will prompt you to install the correct version.

2. Download the graph renderer unity project directory from the parent directory: Unity_stuff
3. Open the Unity Launcher.
4. Select: "Import and existing project" (The wording might have changed since the last time I used Unity 5+ years ago, but it should be similar).
5. Navigate to where you put your graph renderer unity project directory and select it. (GitHub will download the folder as a .zip so you'll have to unzip it first.
6. Have fun!

controlls:

• WASD - movement
• CTRL - down
• SPACE - up
• left SHIFT - boost
• LMB - scan
• TAB - open URL prompt

readmy by: Joey451-OG

internet-scanner's Issues

Suggestion: Color "stars" based on hash of (sub)domain name

This would make it easier to see, which stars belong to one page. Maybe make the subdomain influence the hue or saturation value a little...
Or make it selectable, which part contributes to the hash

Add control to hide UI (or add graph-render-only exports)

A use case that I think people might use this for is to see how a website they're a part of connects with others, and then share an image of that with others. Taking screenshots of the graph renderer currently contain the UI, and it would cool if images could be exported without it.

Node Coloring

I have a couple suggestions regarding node coloring and indication. First, it would be nice if unscanned nodes looked different (perhaps their own color?) than scanned nodes, so that you can clearly see what is scanned and what isn't.

Additionally, although mentioned elsewhere I believe, it would be beneficial to assign different colors to clusters of nodes leading back to the same domain. I noticed that a coloring effort has already been made to color nodes that are in the camera's physics range, but I think that coloring would be better suited to differentiate nodes.

Either way, this is my new favorite Unity project and I can't get enough of it, what a gem.

Copying a link

How about adding the ability to copy the link to the site you are currently looking at to your clipboard?
Is there really no such possibility? Maybe i duplicate, but idk.

Resolution...

(I changed Unity project a little bit, before offical "patch", now it works for me.)
I understand that this is stupid, but on my monitor, 1440 by 900 pixels), everything is not displayed correctly)

List controls in Readme

In README.md include a list of controls.

Asset keeps hitchhiking on commits

          idk why this keeps popping up in people's commits, could we add this to the .gitignore?

Originally posted by @Inertia-Squared in #24 (comment)

maker offset while looking at node

The (I assume) marker that shows which node is currently looked at is way off.

I'm using a 2560x1440p monitor in fullscreen.

Brainstorming/Future TODO?

Adding this here for myself, but also if anyone else wants to have a go or pick it up. I have an idea to make this project kind of like an exploration game.

If you've played no man's sky [galaxy map] or starbound [warp drive map], you'll know what I'm going for. Possibly with a game loop similar to those Wikipedia speedruns where a player has to go from one starting link and travel to an end goal link. With maybe the vibes from 'Everything' (the game, not literally everything - I'm not that proficient lol)

I'd love to look into implementing a way of manually traversing the nodes in a gamified manner, where you have to travel by the connections, and you can view the webpages once you arrive, maybe add some travel time and have the player leave notes like suggested in #3 (yes that's what inspired this idea).

Once there's a slightly more solid foundation for the engine I'd love to look into creating a 'game' branch and making this myself.

Just putting it out there for now but will come back to this idea once I'm finished with my fixes branch :)

Idea

Hi ! Nice YT video, here is a funny idea you can add in your project like I've done with this POC project https://github.com/Lecrapouille/IslandedBrowser you can display the graph of your browser bookmarks (ie graph links are keyword or folders). And when you click on one of parent node this will call Firefox opening all sub children at once.

Internet scanner cannot scan area's in an image map.

This is probably something I will onlyn into, since I love messing around with old internet stuff.
This page has an an image map that leads to 3 other pages.

<HTML>
<HEAD><TITLE>3-D Tour of the Solar System</TITLE>
</HEAD>
<BODY BGCOLOR="black" TEXT="red" LINK="black" VLINK="cadetblue">

<MAP NAME="start">
<AREA SHAPE="rect" coords="50,332,179,379"  HREF="HTDOCS/WELCOME.HTM">
<AREA SHAPE="rect" coords="370,330,498,377" HREF="HTDOCS/PLANETS.HTM">
<AREA SHAPE="rect" coords="530,334,651,380" HREF="HTDOCS/GEOLOGY.HTM">
</MAP>
 
<CENTER>
<IMG USEMAP="#start" SRC="HTDOCS/HTIMG/TITLEPG.GIF" WIDTH=700 HEIGHT=400>
</CENTER>

</BODY>
</HTML>

However when I scan it the links don't show up, and I know this isn't because it is on 127.0.0.1 because I replaced the with , and then it did link to the 3 other pages

This is very niche, and I would have loved to make a PR. I sadly know nothing about unity or C.

Linux file ending

That x86_64 is one normal ending for linux.

I don't know Unity but it looks like it will just work ?

links without https:// dont get scanned

graph_renderer_Tod8zt40XG.mp4

should be an easy fix of just adding https:// to links?

don't know where to find

instrucion say to find the version for your computer and send to run a .exe file when the only folder in the build just has a png and a txt file

Some UI shenanigans

My monitor resolution is 2k (2560x1440)

So basically it's offcenter

Nodes get zoomies and orbit base node group (+some clusters unaffected?)

More of a funny bug here, not sure how this could be fixed without a zero-sum physics solve or some unorthodox changes to the physics, but its very fun to watch it go.

Planning to spend a couple hours today looking over all the new changes and help make improvements where I can!

Sequence.01_2.mp4

Use Github Releases for compiled binaries

Rather than having the compiled binaries for each OS in the repo along with the source code, maybe create a github release and add them there?

scan fail due to error 404 not found

what is the reason of this happen in the first place?
ps: i have internet connection while this happen

Update the readme on how to run this.

As the issue states, can you update the readme so it says how to run this code. i would like to try it out but my background is pure python.

Can't scan

i downloaded the windows binary, clicking on a selected node but it doesnt seem to do anything

Ignores scheme-less urls

Relative links in the form of //domain.tld should follow the original site's protocol.
An href="//domain.tld" redirects to https://domain.tld if it's on https://other-domain.tld, but to http://domain.tld if it's on http://other-domain.tld

A lot of ideas :-)

This awesome project really got my imagination going, because I've been thinking about something very similar for a while.
I am particularly excited to take this idea to a level, where using it to find a webpage becomes faster than clicking through a website.
So the practical use of the idea:)
If you like this approach, great, maybe we could even work together on this!:) If not really, or you don't have enough energy & time for it, then please just read it as a ramble:) I don't think that I'd start implementing these on my own, but if you're in, I'm in.

Here's what I think would take the idea closer to practicality:

1. [Search] It would be cool to be able to filter the nodes by a text entry:) It's as simple as filtering the HTML text of the node. The nodes not containing the text could stay there with a much lower opacity or even disappear? One could also navigate to the "next node/website", like when you search in text, and you can "jump to the next found word".
2. [Title] One could see the title of the nodes, maybe the ones being close to you with a higher opacity, and all titles could remain relatively small to avoid madness.
3. [Auto-Discovery] One could automatically let the algorithm discover N "layers" of websites, so that it's ensured, that all websites that are N clicks away, are included in the graph.
4. [Webpage name] The webpage name could appear among / above the nodes, so that you can see, from which website the nodes are coming from. "Coloring by webpage" would also look nice:)
5. [Keywords] If a keyword is frequent among some nodes, it could appear among / above the nodes, just like the Webpage name, so that it becomes intuitive, what the websites are about. E.g. the word "video" on YouTube pages etc.

I have also asked ChatGPT and it pointed out some great things:

1. [User Annotations] Allow users to leave notes or comments on nodes, which could be saved for future reference.
2. [History Tracking] A history feature to keep track of nodes visited, allowing users to navigate back to previously explored sites.
3. [Bookmarking System] Users can bookmark nodes to create a personalized map of frequently accessed or important websites.

Special remark: [Colors & Style] This is just an opportunity space opener remark, but colors & shapes could be used in so many ways:) Using colors for search (word frequency), for history, for bookmarks... And the style of the connection could also be dotted, dashed, and the nodes could have different shapes depending on some criteria... One could set, by which criteria is the map being colored / "shaped"...

At the end, the goal would be to have a better overview and to find things faster...

Like I said, if you like any of this, let's take it to the next level!:)
All the best!

Add anti-aliasing to graph connections

Adding anti aliasing to graph connections would really polish this interesting project. This is a very cool project that has potential! good job on it!

sometimes the "seconds per update" goes negative

[Suggestion] Make the nodes and lines for every domain a different color

Right now all the nodes and lines are white, however, I think it would be easier if every domain had its own (random) color. This would make it easier to differentiate between all the domains that are connected to each other.

(also this is so cool! I have had a similar idea on my mind forever, but I have nowhere near the skills to realize it)

couple of idea

I think there is couple of idea for the internet scanner that could improve it:

1. [threat] the threat setting will give the note color based on the severity of the website [green is savest while red is most dangerous and gray is unavailable]
2. [net traffic] the traffic setting will give the note color based on the severity of website traffic [cyan is least severe while purple is most severe, white being no traffic and gray is unavailable]
3. [region] allow the user to view the website region

suggestion: add a button to scan all unscanned nodes at once

Linux support

Mac

Tested on:

Apple 2021 Macbook Pro
SOC: Apple M1 Pro
RAM: 16GB
OS: Sonoma 14.0

Builds and works fine on my machine, although the physics seem a little weird, it expands really far when i accelerate it. Example video below.

Screen.Recording.2023-11-19.at.4.43.49.PM.mp4

Scanning stuck at 0%.

Even on the most basic sites, scanning doesn't progress at all.

Files shown as nodes

PDF´s and images can be shown as nodes. Is it a feature?
(example link to scan: https://www.gymvr.cz/skola2/)

Importing this to unity

I don't know how to download the file on a MacBook and put it into Unity

wierd physics problem

physics rules:
1: repel nodes in "range" < works well
2: try pulling/pushing connected nodes to "good distance" < doesnt work

(2) explanation:
THIS node - the node from which perspective im writing this
CONNECTED node - the node we are currently pushing/pulling from the perspective of THIS node

(2) should work by first generating a "pull vector3" basically subtract THIS vector3 from CONNECTED vector3
if we just rb.addforce(pull vector3) to CONNECTED node it will fly towards THIS node
if we multiply pull vector3 by a negative value it will fly away.

i will call that value ForceValue (for explanation's sake)

ForceValue will be deicded by a function that takes in the distance between THIS and CONNECTED nodes, and returns:
tanh(distance - DesiredDIstance)

basically if the distance is bigger than DesiredDistance the value will be positive, which will keep the vector on "pull" mode
if the distance is smaller than DesiredDistance then the value will be negative, after multiplying it by the pull vector3, it will become push vector3

alright so whats the problem?
nodes fly towards the -x -y -z quadrant of the world.
i dont know why its happening, but what i do know is that:
if i multiply all that by -1, then the nodes fly towards +x +y +z quadrant.

see NodePhysicsHandler.cs for more info

have a nice day :) thanks

Don't multiply by DeltaTime in the physics update

I don't begive that we should multiply by Delta time in all the physics functions because if each update take a long time, the delta time will be high and every update will result in nodes in the graph getting flung fall off and not being helpful.

Add Release builds

Some hardware (mine) can't handle the Unity Engine. I suggest a cross-platform build to be made.
Please, this project is really interesting.

Remove Nodes

At some point it gets really cluttered, so maybe add a keybind to remove a single node and the option to reset instead of just relaunching

i might have incompatible unity version

im having a bunch of bugs related to the new color system
almost all of them are because of the alpha defaulting to 1
^ its a pretty nice feature but for some reason everything it touches sets the opacity to 0
I can easily fix it by using colors normally (specifying the alpha s 255 every time) but are others having it?
(you can tell if the physics throttle bar isnt showing up, along with some text not being visible)
if its only me then it might be related to my Unity version (im on 2020.3.35f1 personal)
what version are you all using?
thanks in advance :)