sreedevk / csv-ingestion-pipeline

A Demo Project Implementing CSV Ingestion Pipelines using the Broadway Framework (Elixir)

GenreMatcher

GenreMatcher is an example implementation of Broadway Pipelines for CSV data ingestion & matching. The sample CSV used here is a movie dataset that was downloaded from kaggle - https://www.kaggle.com/chaitanyahivlekar/large-movie-dataset

The application uses redis streams to communicate between pipelines and processes.

Authors: Sreedev Kodichath, Joseph Giralt

Setup

• Install asdf ASDF Documentation

• Install Language Servers + Compilers + Interpreters

asdf install

Please note that the above command will install the following software on your system:

    - elixir 1.12.2-otp-24
    - erlang 24.0.5
    - postgres 13.2
    - nodejs 8.5.0
    - docker 20.10.8

• Install Elixir + Erlang Dependencies

mix deps.get

• Install Phoenix

mix local.hex                     # will install hex package manager
mix archive.install hex phx_new   # will install phoenix

• Edit .envrc with your local database creds
• Create Database

mix ecto.create

• Run Migrations

mix ecto.migrate