ggarciabas / mlops_aws_punkapi

Machine Learning Engineer - MLOps!

Hi there

🎒 I'm a Ph.D. in computer engineering by UTFPR, Brazil.
🔭 I'm currently with Mercado Livre as Sr. Data Science Engineer.
🌱 I’m currently learning Web development and improving Apache Spark skills.
🌎 I’m looking to collaborate on Open Source applications.
🏍️ I'm a motorcycle instructor and love to ride, especially off-road.
😊 I'm a member of Lord Riders MC and Director of RANCH's project.

mlops_aws_punkapi's Issues

Avaliar e criar variáveis para o modelo.

Possibilidade de criar novas variáveis categóricas de cor da cerveja (SRM e EBC).

Criar função Lambda que consulta o endpoint

Definir consulta para cada 5min.

https://serverlessland.com/patterns/kinesis-lambda-efo

https://serverlessland.com/patterns/kinesis-lambda

https://serverlessland.com/patterns/lambda

Scheduler: https://github.com/aws-samples/serverless-patterns/tree/696fbda8744893f1224e015f4400917e0ed93b95/eventbridge-scheduled-lambda

PutRecord

https://stackoverflow.com/questions/33824904/writing-to-kinesis-stream-using-aws-lambda-function

Criar função Lambda para transformar os dados originais

Resgatar: id, name, abv, ibu, target_fg, target_og, ebc, srm e ph
Salvar como cleaned em formato csv.

https://github.com/aws-samples/serverless-patterns/blob/f28f61363e1dd9bbc2fee8f44728905ad6b04853/lambda-s3/src/app.py

Padrão de resposta para firehouse:
https://github.com/amazon-archives/serverless-app-examples/blob/master/python/kinesis-firehose-process-record-python/lambda_function.py

Criar Firehouse para alimentar o S3 com nome `raw`

Writing to Kinesis Data Firehose Using Kinesis Data Streams

Criar tabela AWS Glue com os dados limpos.

template exemplo

Identifica saída do endpoint : https://api.punkapi.com/v2/beers/random

Endpoint para consumir modelo

Criar Buckets S3 para raw e cleaned.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html

Atualizar função lambda para alimentar o Kinesis Data Stream

Depende de #2

Criar Kinesis Data Stream e atualizar lambda.

Controlling Access to Amazon Kinesis Data Streams Resources Using IAM

AWS::IAM::Role

Ativar scheduler CloudWatch para chamar função lambda a cad 5 minutos.

Criar Firehouse para alimentar função Lambda de transformação.

Retreinar o modelo

Configurar Github Actions para AWS

https://aws.amazon.com/blogs/compute/using-github-actions-to-deploy-serverless-applications/

Treinar modelos de regressão.

Monitorar modelo