ctrlcctrlv / dchan

Django-powered archive of Q-related chan posts

dChan

Django-powered archive of Q-related chan posts

Installation

First, create a virtualenv and install requirements:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Now start serving Django locally:

python manage.py runserver

Next, load the data:

python manage.py load_chan_data

Process replies (for reply links), 4chan links (for >>links on 4chan threads), search vectors (for efficient full-text search capabilities), and mark Q drops. Each step will take quite a while for large datasets:

python manage.py process_replies
python manage.py process_4chan_links
python manage.py process_search_vectors
python manage.py mark_q_drops

Setting up automatic scraping

For dChan's automatic scrape functions, we also need to configue supervisor (to daemonize celery) and scrapyd.

Install supervisord for your server's distribution, example for Ubuntu:

sudo apt-get install supervisor

Copy the included celery configuration files into supervisor's conf.d directory:

sudo cp supervisor/*.conf /etc/supervisor/conf.d

Scan and update the conf files with supervisorctl:

sudo supervisorctl reread
sudo supervisorctl update

Now download Firefox's geckodriver to enable archive.is scraping. Download the latest geckodriver release from https://github.com/mozilla/geckodriver/releases, then extract the geckodriver file. Make it executable and move it to /usr/local/bin:

chmod +x geckodriver
sudo mv geckodriver /usr/local/bin/