GithubHelp home page GithubHelp logo

fredgcosta / jarbas Goto Github PK

View Code? Open in Web Editor NEW

This project forked from okfn-brasil/jarbas

0.0 2.0 0.0 1.34 MB

๐ŸŽฉ API for information and suspicions about reimbursements by Brazilian congresspeople

Home Page: https://jarbas.serenatadeamor.org/

License: MIT License

Shell 0.29% JavaScript 0.16% Python 63.59% CSS 0.08% HTML 1.17% Elm 34.71%

jarbas's Introduction

Jarbas โ€” a tool for Serenata de Amor

Build Status Code Climate Coverage Status Updates donate

Jarbas is part of Serenata de Amor โ€” we fight corruption with data science.

Jarbas is in charge of making data from CEAP more accessible. In the near future Jarbas will show what Rosie thinks of each reimbursement made for our congresspeople.

Table of Contents

  1. JSON API endpoints
    1. Reimbursement
    2. Subquota
    3. Applicant
    4. Company
    5. Tapioca Jarbas
  2. Installing
    1. Settings
    2. Using Docker
    3. Local install

JSON API endpoints

Reimbursement

Each Reimbursement object is a reimbursement claimed by a congressperson and identified publicly by its document_id.

Retrieving a specific reimbursement

GET /api/chamber_of_deputies/reimbursement/<document_id>/

Details from a specific reimbursement. If receipt_url wasn't fetched yet, the server won't try to fetch it automatically.

GET /api/chamber_of_deputies/reimbursement/<document_id>/receipt/

URL of the digitalized version of the receipt of this specific reimbursement.

If receipt_url wasn't fetched yet, the server will try to fetch it automatically.

If you append the parameter force (i.e. GET /api/chamber_of_deputies/reimbursement/<document_id>/receipt/?force=1) the server will re-fetch the receipt URL.

Not all receipts are available, so this URL can be null.

Listing reimbursements

GET /api/chamber_of_deputies/reimbursement/

Lists all reimbursements.

Filtering

All these endpoints accepts any combination of the following parameters:

  • applicant_id
  • cnpj_cpf
  • document_id
  • issue_date_start (inclusive)
  • issue_date_end (exclusive)
  • month
  • subquota_id
  • suspicions (boolean, 1 parses to True, 0 to False)
  • has_receipt (boolean, 1 parses to True, 0 to False)
  • year
  • order_by: issue_date (default) or probability (both descending)
  • in_latest_dataset (boolean, 1 parses to True, 0 to False)

For example:

GET /api/chamber_of_deputies/reimbursement/?year=2016&cnpj_cpf=11111111111111&subquota_id=42&order_by=probability

This request will list:

  • all 2016 reimbursements
  • made in the supplier with the CNPJ 11.111.111/1111-11
  • made according to the subquota with the ID 42
  • sorted by the highest probability

Also you can pass more than one value per field (e.g. document_id=111111,222222).

GET /api/chamber_of_deputies/reimbursement/<document_id>/same_day/

Lists all reimbursements of expenses from the same day as document_id.

Subquota

Subqoutas are categories of expenses that can be reimbursed by congresspeople.

Listing subquotas

GET /api/chamber_of_deputies/subquota/

Lists all subquotas names and IDs.

Filtering

Accepts a case-insensitve LIKE filter in as the q URL parameter (e.g. GET /api/chamber_of_deputies/subquota/?q=meal list all applicant that have meal in their names.

Applicant

An applicant is the person (congressperson or theleadership of aparty or government) who claimed the reimbursemement.

List applicants

GET /api/chamber_of_deputies/applicant/

Lists all names of applicants together with their IDs.

Filtering

Accepts a case-insensitve LIKE filter in as the q URL parameter (e.g. GET /api/chamber_of_deputies/applicant/?q=lideranca list all applicant that have lideranca in their names.

Company

A company is a Brazilian company in which congressperson have made expenses and claimed for reimbursement.

Retrieving a specific company

GET /api/company/<cnpj>/

This endpoit gets the info we have for a specific company. The endpoint expects a cnpj (i.e. the CNPJ of a Company object, digits only). It returns 404 if the company is not found.

Tapioca Jarbas

There is also a tapioca-wrapper for the API. The tapioca-jarbas can be installed with pip install tapioca-jarbas and can be used to access the API in any Python script.

Installing

Settings

Copy contrib/.env.sample as .env in the project's root folder and adjust your settings. These are the main variables:

Django settings
Database
Message Broker
  • CELERY_BROKER_URL (string) Celery compatible message broker URL (e.g. amqp://guest:guest@localhost//)
Amazon S3 settings
  • AMAZON_S3_BUCKET (str) Name of the Amazon S3 bucket to look for datasets (e.g. serenata-de-amor-data)
  • AMAZON_S3_REGION (str) Region of the Amazon S3 (e.g. s3-sa-east-1)
  • AMAZON_S3_CEAPTRANSLATION_DATE (str) File name prefix for dataset guide (e.g. 2016-08-08 for 2016-08-08-ceap-datasets.md)
Google settings
  • GOOGLE_ANALYTICS (str) Google Analytics tracking code (e.g. UA-123456-7)
  • GOOGLE_STREET_VIEW_API_KEY (str) Google Street View Image API key
Twitter settings
  • TWITTER_CONSUMER_KEY (str) Twitter API key
  • TWITTER_CONSUMER_SECRET (str) Twitter API secret
  • TWITTER_ACCESS_TOKEN (str) Twitter access token
  • TWITTER_ACCESS_SECRET (str) Twitter access token secret

To get this credentials follow python-twitter instructions.

Using Docker

There are two combinations in terms of With Docker and Docker Compose environments.

  • Develoment: simply running docker-compose โ€ฆ will trigger docker-compose.yml and docker-compose.override.yml with optimun configuration for developing such as:
    • automatic serving static files through Django
    • restarting the Django on Python files changes
    • rebuilding JS from Elm files on save
    • skipping server cache
  • Production: passing a specific configurarion as docker-compose -f docker-compose.yml -f docker-compose.prod.yml โ€ฆ will launch a more robust environment with production in mind, among others:
    • nginx in front of Django
    • server-side cache with memcached
    • manually generate JS after edits on Elm files
    • manually run collectstatic command is static changes
    • manually restarting server on change

That said instructions here keep it simple and runs with the development set up. To swicth always add -f docker-compose.yml -f docker-compose.prod.yml after docker-compose.

Build and start services

$ docker-compose up -d

Create and seed the database with sample data

Creating the database and applying migrations:

$ docker-compose run --rm django migrate

Seeding it with sample data:

$ docker-compose run --rm django reimbursements /mnt/data/reimbursements_sample.xz
$ docker-compose run --rm django companies /mnt/data/companies_sample.xz
$ docker-compose run --rm django suspicions /mnt/data/suspicions_sample.xz
$ docker-compose run --rm django tweets

If you're interesting in having a database full of data you can get the datasets running Rosie. To add a fresh new reimbursements.xz or suspicions.xz brewed by Rosie, or a companies.xz you've got from the toolbox, you just need copy these files to contrib/data and refer to them inside the container from the path /mnt/data/.

Creating search vector

For text search in the dashboard:

$ docker-compose run --rm django searchvector

Acessing Jabas

You can access it at localhost:8000 in development mode or localhost in production mode.

$ docker-compose run --rm django reimbursements path/to/my/fresh_new_reimbursements.xz

To change any of the default environment variables defined in the docker-compose.yml just export it in a local environment variable, so when you run Jarbas it will get them.

Docker Ready?

Not sure? Test it!

$ docker-compose run --rm django check
$ docker-compose run --rm django test

Local install

Requirements

Jarbas requires Python 3.5, Node.js 8, RabbitMQ 3.6, and PostgreSQL 9.6. Once you have pip and npm available install the dependencies:

$ npm install
$ ./node_modules/.bin/elm-package install --yes  # this might not be necessary https://github.com/npm/npm/issues/17316
$ python -m pip install -r requirements-dev.txt
Python's lzma module

In some Linux distros lzma is not installed by default. You can check whether you have it or not with $ python -m lzma. In Debian based systems you can fix that with $ apt-get install liblzma-dev or in macOS with $ brew install xz โ€” but you might have to re-compile your Python.

Setup your environment variables

Basically this means copying contrib/.env.sample as .env in the project's root folder โ€” but there is an entire section on that.

Migrations

Once you're done with requirements, dependencies and settings, create the basic database structure:

$ python manage.py migrate

Load data

To load data you need RabbitMQ running and a Celery worker:

$ celery worker --app jarbas

Now you can load the data from our datasets and get some other data as static files:

$ python manage.py reimbursements <path to reimbursements.xz>
$ python manage.py suspicions <path to suspicions.xz file>
$ python manage.py companies <path to companies.xz>
$ python manage.py tweets
$ python manage.py ceapdatasets

There are sample files to seed yout database inside contrib/data/. You can get full datasets running Rosie or directly with the toolbox.

Creating search vector

For text search in the dashboard:

$ python manage.py searchvector

Generate static files

We generate assets through NodeJS, so run it before Django collecting static files:

$ npm run assets
$ python manage.py collectstatic

Ready?

Not sure? Test it!

$ python manage.py check
$ python manage.py test

Ready!

Run the server with $ python manage.py runserver and load localhost:8000 in your favorite browser.

Using Django Admin

If you would like to access the Django Admin for an alternative view of the reimbursements, you can access it at localhost:8000/admin/ creating an user with:

$ python manage.py createsuperuser

jarbas's People

Contributors

anaschwendler avatar antonioj-mattos avatar brunoarueira avatar caduvieira avatar caiocarrara avatar cassiobotaro avatar cuducos avatar daneoshiga avatar decko avatar dgallinari avatar fcevado avatar feliperuhland avatar giovanisleite avatar gomex avatar guilhermeslucas avatar igorrozani avatar irio avatar jtemporal avatar juhhcarmona avatar lucaslm avatar magnobiet avatar marabesi avatar matheushf avatar paulohp avatar pedrommone avatar petriuslima avatar pyup-bot avatar rogeriochaves avatar sboydd avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.