mauri-medina / pricemory

Tracking and display of price history of products from Paraguay

Pricemory

Tracking and display of price history of products from Paraguay

It uses scrapy for scraping shops sites and django for web framework

Tracked shops

• Superseis
• Stock
• Casa Rica
• Nissei
• Tupi
• Los Jardines

Requirements

• Python 3.6+
• PIP
• Django 2.2+
• PostgreSQL 12+

Installation

1. Clone and install project

git clone https://github.com/mauri-medina/pricemory.git
cd pricemory
python3 -m venv env
source env/bin/activate
pip install -r requirements.txt
cp .env.example .env
python manage.py runserver 0.0.0.0:8000

2. Configurations

• the existing values in .env can be used but is recommended to create a new secret key for django.

Usage

Run django

python manage.py runserver 0.0.0.0:8000

There is a database that can be used for testing

Run spider

In crawler directory run:

scrapy crawl spider-name

The spider names are specified in their files, for example to run the spider from super seis:

scrapy crawl superseis_spider

This will save the scrapped data to the database, to save in another formats check scrappy documentation

Run all the spiders

python crawler_runner.py

The script [crawler_runner](crawler/crawler_runner.py) by default will run the spiders sequentially, one spider after another, but other options are available in the script.

The runner output is logged in the file defined in the script

Run django

python manage.py runserver 0.0.0.0:8000

Contributing

Pull requests are welcome.

Any help or feedback is appreciated, especially for the design of the site.

• Fork it!
• Create your feature branch: git checkout -b my-new-feature
• Commit your changes: git commit -am 'Add some feature'
• Push to the branch: git push origin my-new-feature
• Submit a pull request :D

TODO

• General improve of site design
• Add more shops

Contributors / Thanks

• Bruno Ostertag : Database cleanup and sql performance tuning.
• Saúl Zalimben

License

This project is licensed under the terms of the GNU General Public License v3.0