kseniaskirpichnikova / ecbs-5334-data-engineering-3-real-time-data-processing

ECBS 5334 - Data Engineering 3 - Real-Time Data Processing - Zoltan C. Toth

Home | Logs | Amazon Kinsesis | Kinsesis Analytics | Help/Resources.

Welcome

Welcome to Data Engineering 3: Real-Time Data Processing.

• For "official" communication we are using Moodle
• For informal and in-class communication, we are using Slack
• On this page you will find additional courseware, R and Python scripts, which accompany the course
• In case you need help, you can find the instructor's and the TA's contact details at the bottom of this page
• Before you'd reshare/reuse any of these materials keep in mind that Datapao sponsors this course. Datapao owns the copyright of the Spark notebooks (distributed separately) and possibly other materials in this repository. Reach out to Zoltan C. Toth if you have any questions about this.

Syllabus

Data Engineering 3: Real-Time Data Processing

Prerequisites:

Data Analysis 1: Exploration – Business Analytics track;

Data Analysis 2: Finding Patterns with Regressions – Business Analytics track;

Data Engineering 2: Big Data and Cloud Computing

Course description In this course, you will learn which are the underlying concepts of real-time data processing systems. You will get an introduction of different technologies you can employ to solve real-time use-cases with a special focus on Amazon Web Services and Apache Spark

Learning outcomes By the end of the course you will:

• Understand the building blocks of a real-time data infrastructure
• Understand the caveats that arise when you take these systems to production
• You will have hands-on knowledge on how to build a simple, end-to-end real-time pipeline with Amazon Web Services (AWS) and Apache Spark

Readins readings:

Note: Data, presentations, and code for the exercises will be provided.

• Heart Logs: Event Data, Stream Processing, and Data Integration Download

Course schedule and materials for each session This class runs for 2x300 minutes.

Week 1:

• Spark Recap and Aggregations in Spark
• Data Infrastructures overview
• Streaming Basics
• Spark Streaming basics

Week 2: Problems in Production

• Message Brokers
• AWS: Amazon Kinesis and Kinesis SQL
• Handling Late Data
• Exactly once processing

Contact

Instructor	Teaching Assistant
Zoltan C. Toth	Miklos Petridisz
[email protected]	[email protected]
[phone]	[phone]