create-speech-to-text-pipeline / pipeline

A tool that can be deployed to process posting and receiving text and audio files from and into a data lake, apply transformation in a distributed manner, and load it into a warehouse in a suitable format to train a speech-to-text model

Speech-to-Text Data Collection

A tool that can be deployed to process posting and receiving text and audio files from and into a data lake, apply transformation in a distributed manner, and load it into a warehouse in a suitable format to train a speech-to-text model.

Data Capture Pipeline

Directory Structure

.
├── airflow
│   ├── dags
│   │   ├── extract_load.py
│   │   └── scripts
│   │       ├── dataloader.py
│   │       ├── db_connection.py
│   │       ├── __init__.py
│   │       └── schema
│   │           └── amharicnews.sql
│   ├── data
│   │   └── AmharicNewsDataset.csv
│   ├── docker-compose.yaml
│   └── logs
│       └── scheduler
│           └── latest -> /opt/airflow/logs/scheduler/2022-10-05
├── backend
│   └── dummy.txt
├── frontend
│   ├── dummy.txt
│   ├── frontend
│   │   ├── package.json
│   │   ├── package-lock.json
│   │   ├── public
│   │   │   ├── favicon.ico
│   │   │   ├── index.html
│   │   │   ├── logo192.png
│   │   │   ├── logo512.png
│   │   │   ├── manifest.json
│   │   │   └── robots.txt
│   │   ├── README.md
│   │   └── src
│   │       ├── App.css
│   │       ├── App.js
│   │       ├── App.test.js
│   │       ├── index.css
│   │       ├── index.js
│   │       ├── logo.svg
│   │       ├── reportWebVitals.js
│   │       └── setupTests.js
│   └── proto.png
├── img
│   ├── logo.png
│   └── pipelineDiagram.png
├── LICENSE
├── logging
│   └── dummy.txt
├── notebook
│   └── Amharic_news_Classification.ipynb
├── README.md
├── requirements.txt
├── screenshots
│   ├── airflowscreenshoot.png
│   └── design diagram.png
└── testing
    ├── dummy.txt
    └── test_dataloading.py

17 directories, 39 files

Run Locally

Clone the project

  git clone https://github.com/create-speech-to-text-pipeline/pipeline

Go to the project directory

  cd pipeline

Install dependencies

  pip3 install -r requirements.txt

Set up pipeline

  python3 setup.py

Screenshots

Authors

• Haylemicheal
• Janerose
• Mohammed
• Nahom
• Patrick
• Wangui
• Yonas

pipeline's Issues

Create a Kafka cluster

• Based on Installing a Kafka Cluster and Creating a Topic - Hands-on Labs | A Cloud Guru, set up a cluster in your assigned AWS machine.

• Your cluster will be responsible for creating a Delta Lake - a bucket in S3 where Spark transformed streaming data from users reading the texts you showed them are stored. (hint You will write a code that can generate an ID for a randomly selected text and its audio equivalent, receives an ID from an API, sends back as json the ID + audio to Kafka like URL

Logging

linked to #11

Backend

prepare API endpoints for kafka - using flask

Planning and design

• Build or simulate a Kafka event source for the text corpus - you should read Breaking News: Everything Is An Event! (Streams, Kafka And You) (florimond.dev)

• Develop an overview of your approach and document it. Explain why this approach and why these tools. Explain how this approach will provide a good data source for the clients’ speech-to-text ML engine. Explain the purpose of each of these tools - should defend it if one asks them why, not simple python code.

README and github actions

Frontend

using react

Code Review

Create a javascript tag

• The tag shall be used in front-end applications to communicate with your Kafka cluster - present a sentence to be read by a user and send back audio and other necessary metadata to your Kafka cluster.

• You should look at the following to understand how an app or a browser captures and sends audio and text events to your kafka cluster
  Using the MediaStream Recording API - Web APIs | MDN (mozilla.org)
  Handling Large Messages with Apache Kafka (CSV, XML, Image, Video, Audio, Files) - Kai Waehner (kai-waehner.de)

Use Spark to transform and load from your Kafka cluster

• Using PySpark, write code that will transform and load the data from the data lake
• By using Kafka as an input source for Spark Structured Streaming and Delta Lake as a storage layer, build a complete streaming data pipeline to consolidate our data - you should read From Kafka to Delta Lake using Apache Spark Structured Streaming (michelin.io)

Testing

Airflow

EDA

Jupyter notebook that illustrate your data exploration with professional plots, readable axes labels, title, and legend; good choice of color