Ygg Scraper
| Info: | This is the README file for Ygg Scraper. |
|---|---|
| Author: | Laurent Kislaire <[email protected]> |
| Copyright: | © 2018-2019, Laurent Kislaire. |
| Date: | 2019-03-18 |
| Version: | 1.2.18 |
PURPOSE
Ygg scraper with:
- Shell interface - Any python Cmd2 features can be used: completion, scripts and much more
- RSS feed with torrent using passkey
- Transmission / Rtorrent / Deluge add torrent directly from webapp
- Irc Limnoria interface
- Cloud Flare bypass using cfscrape
- Http and Socks proxy support
DOCUMENTATION
The full documentation is here. This includes links to docker images.
INSTALLATION
Install using a virtualenv:
# Clone and fetch install utils git clone https://github.com/architek/yggscr.git && cd yggscr sudo apt-get install git build-essential python-dev virtualenv # Create virtualenv virtualenv -p python3 .venv .venv/bin/python -m pip install --upgrade pip setuptools wheel # Build in virtualenv source .venv/bin/activate ./setup.py install # Test cd src/yserver yserver yshell # To leave virtualenv deactivate
Note: The user configured in yserver.ini needs read access to the virtualenv directory.
Note: You need at least setuptools 33.1.1.
On Debian Jessie you can use official backports:
echo "deb http://ftp.debian.org/debian/ jessie-backports main contrib non-free" >> /etc/apt/sources.list sudo apt update sudo apt install python3-setuptools -t jessie-backports
Note: If you want the CloudFlare bypass to work, you also need to install the debian package nodejs
USAGE
Directly from the shell
Start shell:
$ yshell Welcome to Ygg Shell. Type help or ? to list commands. > help Documented commands (type help <topic>): ======================================== alias help login print quit shell unalias edit history lscat proxify response shortcuts exclus list_torrents next py search_torrents stats get load ping pyscript set top_day > search_torrents q:cyber c:film s:docu * Cyber guérilla 2.0 (2015) Science&Vie; [VFF] [HDTV] [1080p] x264 [0.93GB] S:26 L:0 | https://yggtorrent.com/torrent/filmvidéo/documentaire/184378-cyber+guérilla+2+0+2015+sciencevie+vff+hdtv+1080p+x264 | None | None * Infrarouge On Nous Ecoute Partie 1 Cyber guerre L Arme Fatale 2015 [1.11GB] S:6 L:0 | https://yggtorrent.com/torrent/filmvidéo/documentaire/22526-infrarouge+on+nous+ecoute+partie+1+cyber+guerre+l+arme+fatale+2015 | None | None > stats EXCEPTION of type 'YggException' occurred with message: 'Not connected' To enable full traceback, run the following command: 'set debug true' > login TheBoss Passw0rdz > stats Ratio:4.19 Down (GB):73.24 Up (GB):306.66
As an IRC bot
Symlink the YBot subdirectory in your supybot plugin directory. Ask the bot for help ;-)
As standalone web server
This server allows searching, downloading torrent file, sending to rtorrent,transmission or deluge client and authenticated RSS.
Fill in your settings in defaults.cfg (at least Hostname, Port to listen to, username and password) and launch the server:
yserver
To access webapp, connect to http://localhost:8333 (or any other config you've set)
Behind apache or nginx using wsgi
The same can be run behind any webserver, here is nginx described:
apt install uwsgi uwsgi-plugin-python3
Create nginx vhost:
upstream _bottle {
server unix:/run/uwsgi/app/yserver/socket;
}
server {
server_name ygg.com;
root /var/www;
listen 80;
listen [::]:80;
location / {
# restrict to 192.168.1.0/24
allow 192.168.1.1/24;
deny all;
uwsgi_read_timeout 20s;
uwsgi_send_timeout 20s;
include uwsgi_params;
uwsgi_pass _bottle;
}
}
Create file /etc/uwsgi/apps-available/yserver.ini:
[uwsgi] plugins = python3 socket = /run/uwsgi/app/yserver/socket virtualenv = /home/user/git/yggscr/.venv chdir = /home/user/git/yggscr/src/yserver file = app.py master = true uid = www-data gid = www-data workers = 2 threads = 2 socket-timeout = 6000000 ;harakiri = 20 ;paste-logger = true ;disable-logging = true debug = true ;reloader = true ;catch-all : set to false to let debugging middleware handle exceptions ;catch-all = false need-app = true vacuum = true ;set-placeholder = ano=true
Create directory for socket for nginx to communicate with uwsgi:
mkdir -p /run/uwsgi/app/yserver chown www-data:www-data /run/uwsgi/app/yserver
Edit yserver.cfg to fit to your need Enable uwsgi app and reload nginx:
cd /etc/uwsgi/apps-enabled ln -s ../apps-available/yserver.ini systemctl restart uwsgi.service systemctl restart nginx
Anonymous Public mode
Note that it's possible to run the webapp without any credentials (see uwsgi 'ano' option). The realtime stats will not be shown and its up to the consumer application to provide the authentication cookie (e.g. the browser itself).
You can have as many instances of the webapp running as you have .ini files. An example can be different configurations (anonymous, user1, user2). Each application has its own configuration and nginx can connect to the correct application through the relevant unix socket.
Example for 2 configurations (internal LAN/external WAN):
http {
[...]
geo $client {
default extra;
192.168.1.1/24 intra;
}
}
upstream _bottle {
server unix:/run/uwsgi/app/yserver/socket;
}
upstream _bottle_ano {
server unix:/run/uwsgi/app/yserver-ano/socket;
}
server {
[...]
location / {
uwsgi_read_timeout 20s;
uwsgi_send_timeout 20s;
include uwsgi_params;
if ( $client = "extra" ) {
uwsgi_pass _bottle_ano;
}
if ( $client = "intra" ) {
uwsgi_pass _bottle;
}
}
}
Graphs
It's easy to generate graphs using this library. The following script plots upload, download and ratio graphs using kibana (use a cron to trigger the cyclic execution):
#!/usr/bin/env python3
from yggscr.ygg import YggBrowser
from time import strftime, localtime
from datetime import datetime
from elasticsearch import Elasticsearch
def get_stats(username, password):
y = YggBrowser()
y.login(username, password)
r = y.get_stats()
t = datetime.utcnow().strftime('%Y/%m/%d %H:%M:%S')
return {'ratio': r['ratio'], 'up': r['up'], 'down': r['down'], 't': t}
def write_index(index, data, doc_type='ratio_torrent', host='localhost', port=9200):
es = Elasticsearch([{'host': host, 'port': port}])
es.index(index=index, doc_type=doc_type, body=data)
print("Wrote data {}".format(data))
data = get_stats('myuser', 'mypassword')
write_index('ygg', data)
NOTES
Because I'm too lazy to do a proper html page, not all options are visible. The webapp is a "passthrough" relay. Any unknown parameter is sent to the server.
The following is an anonymous rss feed about electro music (combining categories):
https://server.example.com/ano/rssearch?category=audio&sub_category=musique&option_genre%3Amultiple[]=1&option_genre%3Amultiple[]=15&option_genre%3Amultiple[]=33&option_genre%3Amultiple[]=34&option_genre%3Amultiple[]=35&option_genre%3Amultiple[]=119&option_genre%3Amultiple[]=124
COPYRIGHT
Copyright (c) 2018-2019, Laurent Kislaire
Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent