aws-big-data-projects Goto Github PK

big-data-solutions

This repository provides Code examples written in Python,Spark-Scala using primarily boto3 SDK API methods and aws cli examples for majority of the AWS Big Data services. There are also nicley written Wiki articles for most of the common issues/challenges faced within BigData world.

bigdata-file-viewer

A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.

bigdata-playground

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

data-analytics-for-mobile-games

Player Unknown's Battlegrounds (PUBG), is a first person shooter game where the goal is to be the last player standing. You are placed on a giant circular map that shrinks as the game goes on, and you must find weapons, armor, and other supplies in order to kill other players / teams and survive.

data-engineering

Construct a modern data stack and orchestration the workflows to create high quality data for analytics and ML applications.

data-pipeline-with-cdk

data-science-ipython-notebooks

Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.

emr-bootstrap-actions

This repository hold the Amazon Elastic MapReduce sample bootstrap actions

emr-serverless-samples

Example code for running Spark and Hive jobs on EMR Serverless.

emr-studio-notebook-examples

This repository contains ready-to-use notebook examples for a wide variety of use cases in Amazon EMR Studio.

ferry

Ferry lets you define, run, and deploy big data applications on AWS, OpenStack, and your local machine using Docker

front-line-concussion-monitoring-system-using-aws-iot-and-serverless-data-lakes

A simple, practical, and affordable system for measuring head trauma within the sports environment, subject to the absence of trained medical personnel made using Amazon Kinesis Data Streams, Kinesis Data Analytics, Kinesis Data Firehose, and AWS Lambda

heartrate-monitoring-using-aws-iot-and-aws-kinesis

you run a script to mimic multiple sensors publishing messages on an IoT MQTT topic, with one message published every second. The events get sent to AWS IoT, where an IoT rule is configured. The IoT rule captures all messages and sends them to Firehose. From there, Firehose writes the messages in batches to objects stored in S3. In S3, you set up a table in Athena and use QuickSight to analyze the IoT data.

hive-emr-on-eks

hopsworks

Hopsworks - Data-Intensive AI platform with a Feature Store

image-caption-generator

In this project, a framework is developed leveraging the capabilities of artificial neural networks to “caption an image based on its significant features”.

iot-and-big-data-application-using-aws-and-apache-kafka

Iot,Big Data Analytics using Apache-kafka,spark and other aws services

iot-data-with-amazon-kinesis

Build a Visualization and Monitoring Dashboard for IoT Data with Amazon Kinesis Analytics and Amazon QuickSight

log-analytics-solution-with-aws

Collect, process, and analyze log data using Amazon Kinesis and Elasticsearch Service

retail-demo-store

AWS Retail Demo Store is a sample retail web application and workshop platform demonstrating how AWS infrastructure and services can be used to build compelling customer experiences for eCommerce, retail, and digital marketing use-cases

run-a-spark-job-within-amazon-emr

Run a Spark job within Amazon EMR

s3uploader-ui

serverless-data-pipeline

Python CDK serverless data pipeline with CI/CD process and Slack notifications.

serverless-data-pipeline-sam

Serverless Data Pipeline powered by Kinesis Firehose, API Gateway, Lambda, S3, and Athena

serverless-etl-cdk

simplify-big-data-analytics-with-amazon-emr-

Simplify Big Data Analytics with Amazon EMR, published by Packt

spark-examples

Spark Examples

spark-scala-examples

This project provides Apache Spark SQL, RDD, DataFrame and Dataset examples in Scala language

sparklearning

A comprehensive Spark guide collated from multiple sources that can be referred to learn more about Spark or as an interview refresher.

streaming-data-solution-for-amazon-kinesis-and-amazon-msk

A solutions that automatically configures the AWS services necessary to easily capture, store, process, and deliver streaming data. This solution helps you solve for real-time streaming use cases like capturing high volume application logs, analyzing clickstream data, continuously delivering to a data lake, and more.