johndoee / rippy

Scrape websites using Chrome

Rippy

Rip-it with Rippy

Introduction

Rippy is a downloader designed to scrape websites using a real web browser to find, e.g. video or downloadable files. The targets are website that try to be scrape-resistant and where other downloaders had to give up.

The magic is that Rippy uses a real browser it controls so a lot of the normal anti-bot designs are inefficient, e.g. scrambling javascript. To block Rippy you will have to block browsers. I also enjoy a blocking arms-race, keeps my day bright and fulfilled.

Installation

Currently the only distribution method officially provided is the docker-compose way but all it really requires is Chrome and Python.

wget https://github.com/JohnDoee/rippy-docker/raw/master/docker-compose.yml

You should edit docker-compose.yml. The following values should be changed

• /tmp/media should be changed to where you want rippy to download data, it is in the file twice.
• BASIC_AUTH_PASSWORD should be changed to a unique password
• SECRET_KEY should be changed to something unique
• Optional: Change RIPPY_CONCURRENCY to how many scrape and download threads you want to have.

docker-compose up -d

Usage

Head over to http://ip:51359 and add a job. It should start downloading or prompt you to do something manually.

If the status text says “Waiting” it means you need to open the browser and fill in a captcha or something alike. If you are using the docker-compose setup there should be a button in the upper-right corner of the website to open the browser. It will open a new window with a VNC to the hosted Chromium browser.

New scrapers

Feel free to request a new scraper but there are a few requirements if you want me to implement them: They are scrape resistant, as in, nobody else should be able to download. Check out tools like youtube-dl and JDownloader first. They should not be using an encryption or behind paywall, i.e. I can’t do stuff like netflix (something like that is also not the target at all)

Currently a generic video-site scraper is on the slab as this project is a merge between a reddit post and a generic video-site scraper

Accompanied repositories

Docker-compose file and docker chromium repository

Rippy webinterface

FAQ

Q:	My tab crashed or elements on the website crashed, what should I do?
A:	Close the tab, rippy should notice it shortly and try again.

TODO

• [ ] Add (semi-)generic view player extractor
• [ ] Return (potentially proxied) URL to video instead of downloading

Supported sites

• Avgle

Docker images

Main backend component (this repository)

Webapp and reverse proxy

Chrome accessible via VNC

Logo / icon

frog by habione 404 from the Noun Project

License

MIT

rippy's Issues

How to rip from avgle

Hi:

From the Readme it says "Head over to http://ip:51359 and add a job.". So what "ip" here referring to?

Thanks

Downloading is slow

Nice project! I wonder if we could boost the downloading speed. Maybe using multiple threads or tools like ariac2?

Failed - Failed FFMpeg with returncode 1

Attempted download would initially show Downloading - Finished downloading 0 segments, merging with ffmpeg before changing to Failed - Failed FFMpeg with returncode 1

URL used: https://avgle.com/video/Hi5r7rtMJrH/%E3%82%A4%E3%83%A4%E3%83%A9%E3%81%97%E3%81%84%E3%82%AB%E3%83%A9%E3%83%80-%E3%82%B9%E3%82%BF%E3%82%A4%E3%83%AB-%E6%B9%8A%E8%8E%89%E4%B9%85-%E3%82%AB%E3%83%B3%E3%83%91%E3%83%8B%E3%83%BC%E6%9D%BE%E5%B0%BE-5%E5%B9%B4%E3%81%B6%E3%82%8A%E3%81%AE%E5%86%8D%E4%BC%9A-%E6%84%9F%E6%83%85%E5%89%A5%E3%81%8D%E5%87%BA%E3%81%97%E3%83%AA%E3%82%A2%E3%83%AB%E6%80%A7%E4%BA%A4-ssni-092

Add a retry button

Currently the UI doesn't even allow me to copy the full url. Would love to have a retry button :)

Add a config to disable user_pass authentication

Instead, can we just listen to localhost?