marksgy / espider

A web spider based on electron

Introduction to Espider

Caution

This is not the ultimate version. There might be some bugs in it. If you encountered some unexpected problems, please help yourself. Uncomment this line in index.js to open the dev tool (the same tool you use in Chrome F12)

What is Espider

A web spider based on electron

How to use it？

1. Insert your target web address.

2. Select your page type

• Page means the entry page of your multi-level spider which includes the action of "turning-page".
• Title means the web link in the Page above that leads you to the next level.
• Content means the bottom-level page where your target damentta lies.
• Search means the kind of web spider that can be used to search a data base and download sth like that (which is still under development)

3. Go into the page to select the content you like

• Click this button
• Select your content by a single click, while Espider will automatically select the same type of the content for you
• If you do it wrong, just right-click the content you have just selected.

4. Close the new window

5. Click the spider button to get the content you want(by default its in your ??folder,and you can configurate it in the settings)

Examples

1. Single page

take MDN for example

Just select the Content type, open a new page, select whatever you want, close the new page,and click the spider button

2. Double-level spider

For example, you want to get all of the APIs from MDN

• Just select the Title type, open a new page, select one of the API name and the rest of them will be labeled automatically.
• Next do what Example 1 did
• You will see a chart appear in the bottom part of the main page.

3. Multi-level spider

For example,The 51job Its a chinese job hunt website

• First, you can select the Page type, which means you need the funtion of turning pages.You slide down to the bottom and click the button indicating pages.
• Second, you can repeat the steps of Example 2 until you reached the content you want.

Questions

1. What's the difference between Page and Title?
   Well,Page is specially modified to cater the page turning function, which will automatically parse the url of the button and form the right url of the next page.

2. Where can I find the output file?
   Just click the "setting" icon on the top-left corner, and you can see the folder you are using(and click the "open folder" button to open it)

espider's Issues

UI