GithubHelp home page GithubHelp logo

deenuu1 / fjob Goto Github PK

View Code? Open in Web Editor NEW
0.0 2.0 0.0 5.72 MB

🔍 Application to search for job advertisements around the world. By scraping multiple job portals, you won't miss anything.

License: MIT License

Python 71.74% Dockerfile 0.26% JavaScript 23.44% HTML 0.19% CSS 4.36%
api celery crawler css django django-rest-framework docker html javascript jobboard jobs jobsearch python react redis rest web-scraping

fjob's Introduction

Contributors Forks Stargazers Issues MIT License LinkedIn


FJob

Application to search for job advertisements around the world. By scraping multiple job portals, you won't miss anything.

Report Bug · Request Feature

homepage

offers1

offers2

profile

contact

About The Project

FJob is a project that will make it easier for you to look for a new job. Every day, scrapers search popular job sites, process the collected data to provide you with new offers that can change your life. In the future, it will be possible for companies to add their own offers that will be specially marked as sponsored to distinguish them from those collected from other websites. The website is translated into two languages - Polish and English. Currently, data on job offers is collected from: nofluffjobs.com, justjoin.it, olx.pl, Pracuj.pl, praca.pl I'm still working on improving the website and adding new scrapers to expand the database of job offers.

Key Features

  1. Scraping and processing data from various websites
  2. Authentication with JWT
  3. Automated scrapers by using Celery, Redis and Django Celery Beat
  4. Reporting broken offers just with a few clicks
  5. Registration, login, password change and account deletion
  6. Contact form

How does scrapers works

Here I used Strategy Pattern to easily add additional scrapers. The first module is GetContentStrategy which is responsible for downloading the content (html) of the page and then saving it to the database.

The second module - Process is responsible for processing data that was saved in a raw state in the first module. Here, details such as the name of the offer, salary, work mode, type of contract, location, skills and much more are extracted. Then the processed data is saved to the database and waits for approval by the administrator in order to display it to users. diagram

List of scrapers

Built With

  • Python
    • Django Rest Framework
    • Django Celery Beat
    • Celery
    • Selenium
    • Gunicorn
  • PostgreSQL (production), SQLite (dev)
  • Docker and Docker-compose
  • React (Javascript + Vite)
  • Redis
  • Nginx

Installation

Development

  1. Clone git repository
git clone https://github.com/DEENUU1/fjob.git
  1. Create dotenv file and add required data
cp .env_example .env
  1. Install all requirements
pip install -r requirements.txt
  1. Run DRF application
python manage.py runserver
  1. Create superuser
python manage.py createsuperuser
  1. Run React application
npm run dev

Production

  1. Clone git repository
git clone https://github.com/DEENUU1/fjob.git
  1. Create dotenv file and add required data
cp .env_example .env
  1. Run docker-compose
docker-compose build
docker-compose up

Tests

To run pytests use this command

pytest

Custom Django commands

  1. Scraper commands
python manage.py olx
python manage.py justjoinit
python manage.py nfj
python manage.py pracujpl
python manage.py pracapl
python manage.py theprotocol

License

See LICENSE.txt for more information.

fjob's People

Contributors

deenuu1 avatar

Watchers

avatar avatar

fjob's Issues

Form params required

  • user cant post only city --> country is also required
  • user cant post only min_salary and max_salary --> query or country are also required
  • user cant post only experience_level --> query or country are also required

Important!

  • Fix tasks.py
  • Finish OfferFilterView for advanced search

Offer model

  • title
  • salary
  • description
  • url
  • street
  • additional_data (json)
  • remote (bool)
  • hybrid (bool)
  • country
  • city
  • date_created
  • date_finished
  • date_downloaded

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.