blurhy / horizonspider

The spider for ZeroNet search engine Horizon

HorizonSpider - Nodejs

Basic concepts

The spider crawls each site in siteList and submit crawled links to zeronet.

Then zeronet processes them, and downloads those zites.

It continues to traversal siteList and crawls newly downloaded sites to get more links

Usage

Clone this repo and:

yarn

Config

Create .env file and add configs as follows:

ZeroNetPath=D:/ZeroNet
ZeronetDataPath=D:/ZeroNet/Data

Run

yarn crawl

horizonspider's Issues

Data visualization for relationship data of zites

Plan to use d3.js

Site technology detection.Bootstrap,Vue,json.gz etc.

To know the quality of a site.

No module named '__main__.Config'; '__main__' is not a package

HorizonSpider leo$ python3 HorizonSpider.py
Traceback (most recent call last):
File "HorizonSpider.py", line 14, in
from .Config import config
ModuleNotFoundError: No module named 'main.Config'; 'main' is not a package

Manager website for HorizonSpider

• Manage instances of zeronet and crawler
• Log file viewing
• System statistics
• Dashboard site on zeronet
• Auto update trackerlist
• Publish data to Horizon zite

What information do we need to crawl ?

• NLP extracting keywords
• Site info : peers,size,file_count
• Title of each feedFollow object (Blog post title)
• Detecting the technology/web frameworks that the zite uses

Zite type detecting.

Absolute path F:\ZeroNet\

Hi, unsure if that is intentional, though in several files is absolute path (F:\ZeroNet), not relative path.

Crawler for content.json

To know the relationship between .bit domain and zite address.

Crawl content.json for every zite in data folder.

Finally put the data in to data_contentjson.json.gz for Horizon

Data that need to crawl in content.json

• Zite Address
• Zite Domain
• Does the zite has sub content.json.
• modified attr
• title attr
• zeronet_version
• description
• Count of files

error

Cannot connect to admin site TypeError: Cannot read property 'ui_ip' of undefined

Site status data

Execute Page.cmd("siteList",[],r=>console.log(r)) in zerohello
Then you can see

We need:

• content.files/address/cloned_from/description/domain/files/files_optional/includes/modified/title/zeronet_version
• settings.size
• peers

Crawl database.Special crawlers for ZeroBlog/ZeroWiki

Using Site Status Data item,clone from to detect site type,then crawl them.

• ZeroBlog
• ZeroWiki