rasan147 / web-leach

Scrap data from websites

Web-leach

Scrap data from websites

Planning to move on a new repo on V8 and make it (both) public

TODO on V7:

• #24

TODO on V8:

• add a reload button on WEBPAGE IMAGES [Right click] that will contact the CLI personally and redownload the file and refresh the image/page /tells to refresh
• add ids to each images & #10
• #14
• #15
• Selectable folder download & update
• NH search engine and multiple links downloader (check Special tools/NH search engine)

TODO on Alpha Version:

• make online data on JSON format #36

web-leach's Issues

[WebPage] Homepage is not using popup manager and broken css

Use popup manager used in web-leach subdirectory pages.

Found error while making manga freak page

When the manga freak donwload folder is deleted and attempted to gen_html.py on it, FileNotFound error occured.

Make a exception message on that

[WebPage] when pop up message is open, clicking transparent bg will close the modal (focus mode)

[CLI Downloader] index thread prints after Ctrl+C

[CLI Downloader] Indexing Issue. Also check if the net is On priodically

[GITHUB] Add a readme.md file with demo video

[CLI] Make all functions files by files separate to lessen pressure

Also make special rippers file based and in a different folder

[CLI] Remove all exec() and use JSON for backend

[CLI Downloader] IPv6 crashes compressor

[CLI Downloader] remove old save data patch on JSON

There was a patch in filetype in precious versions where i had to do exec()
Remove that patch in v0.7

[Logger] Add more logging

[WebPage] Use Hammer.js

https://hammerjs.github.io/

Use this site on focused image and also add a option on window swipe to next chapter

[WebPage] image focus close button z-index low

[WebPage] Create PWA

https://www.freecodecamp.org/news/build-a-pwa-from-scratch-with-html-css-and-javascript/

[CLI] make 'wl-page-2.html' a data/ file and downloadable

make it download from internet to make exe files more flexible
and store it in data/ folder

[CLI Downloader] fresh startup issue

urllib3 import failing

[CLI Downloader] add ?info to show project info summery

[CLI Downloader] failed to load.proj file

[CLI Downloader] add ?del command

Delete the project files and ask if delete the .wlproj and .wllist files

[WebPage] image nav button typo

[WebPage] Public template v3 needs recheck

Compress using some website, comments are not removed, only white spaces are removed.

Also not tested with PC

[WebPage] Popup manager might have memory leak

Every time of initializing, remove contents in the pop up box using removechild()

Coz the way (innerHTML = .....) used here don't remove the event listeners.
Google for more...

[WebPage] brightness overlay issue

The overly is acting weird when scrolling back and forth in android since the nav bar also comes and goes

[WebPage] add ?del command

Delete the project files and ask if delete the .wlproj and .wllist files

[WebPage] add a transparent overlay when sidebar is open and on_click, close sidebar, also remove the overlay

[Logger] AppConfig_ typo in "output codes v6.wltext"

[CLI Downloader] check terminal screen size and show output

Use https://www.geeksforgeeks.org/python-os-get_terminal_size-method/

Add a arg one_line in leach_logger and make what to do in the middle

Also make sure downloader prints in one line by changing progress bar

[CLI Downloader] Too many red alert on offline

[CLI Downloader] auto-fix missing schema in main_link

auto-fix if https:// is not in the link, 1st it will try with https:// then http://

[CLI Downloader] scan for project name after ?command

?command *(project name)

[WebPage] Go to home link broken

Go to home link sends user to ../project name instead of index.html

[CLI Downloader] Inturruption not working

[WebPage] caption missing

Need to increase caption height

Compare old and new sub directory list and keep that data

Use: #14

[CLI Downloader] print_text breaks some times

[CLI Downloader] Add hard cancel by ctrl+C on Connecting to server

[WebPage] closing focused image should move user to the image

After opening a image in focus mode and navigating within, when closed, user should be scrolled to that point where the last image was

[WebPage] while image focus mode is enabled, clicking transparent bg will close the modal (focus mode)

[CLI Downloader] Use JSON for Cached DATA

Cached data probably use exec or eval. use json instead

[CLI] change save format to JSON and depricate <6 save format

To keep low profile and user friendly

[WebPage] Show 🆕 on updated subpages on homepage (and page list)

Also update save file format to compare old and new list

[WebPage] Alert on No Js

When:

1. Alert when js is disabled
2. Also alert when a js plugin fails to load

Process:

1. Show the alert in Html by default. Delete that part when js runs
2. Check if plugin is loaded by function !=='undefined' and show msg

[CLI] make Online server based User login

however an issue: offline login failure

[CLI Downloader] use JSON to change values by name

I used verified and fixed names to update data, but now update them by setattr() and names on JSON for future compatibility

[Logger] Create client side based log viewer.

• Instead of sending html code, send trimmed decompressed code
• Use Js to decode the short hands (on need)
• Use Js to display specific amount of logs at a time (saves cpu and memory ig)

[Logger] Add linked called from "code"

[CLI Downloader] running server verification issue

[CLI Downloader] Ip not showing