konnomiya / dataengineering

The Data Engineering subteam of Cornell Data Science

Data Engineering

Who we are:

The CDS Data Engineering subteam exists to provide analysis and processing support to CDS project teams, and to develop institutional knowledge in high throughput computing.

Advisor: Professor Immanuel Trummer
Team Leads: Haram Kim (A&S CS 2020)

Team objectives:

• Improve on existing high throughput computing frameworks
• Develop solutions for data analysis problems in CDS projects
• Provide a reservoir of reference information in data engineering
• Research and publish means of improving existing DE frameworks

Current Projects:

• Spark ML Optimization: Apache Spark's machine learning modules are not as well-studied as those of other platforms. This project seeks to empirically identify optimal settings for Spark's ML modules to best utilize the platform's unique capabilities.
• SkinnerDB Parallelization: This project's objective is to experiment with parallelism in Professor Trummer's recently developed database engine, SkinnerDB. The SkinnerDB uses a machine learning approach to query optimization, in contrast to the heuristic model used by most current database engines, but has not yet been expanded to allow multi-core execution.
• Deterministic Query Approximation: Several recent publications have outlined methods to allow high-speed query approximation with deteministic bounds, but have not yet been applied to a wide range of queries. The objective of this project is to apply several of these techniques to the TPC-H query benchmarks to demonstrate broader applicability.
• GPU Acceleration: The distributed GPU computing deals with the unique task of handling distributed deep learning tasks, which is currently well-optimized for multiple GPUs, but not necessarily across multiple machines. Our goal is to research and optimize current tools in development so that it can be adopted by CDS teams deploying large DL models.

Previous Projects:

• Data streaming: Profiling of real time data streaming through Apache Kafka

• Server monitoring: Real time visualization and monitoring of compute server resource utilization through Cockpit

• File format optimization and profiling: Comparative analysis of a variety of file formats typically used in data science, focusing on CSVs and Apache Parquet

• Spark diagnostics: Deliberate attempts to produce errors while running Apache Spark, both locally and on our servers. Problem specifics and solutions were recorded in case similar issues develop in the future.

Members (SP2018):

• Jo Chuang
• Ethan Cohen
• Audrey Fan
• Katarina Jankov
• Ryan Kannanaikal
• Eashaan Kumar
• Jeong Hyun Lee
• Julia Ng
• Shoheb Ome
• Alanna Perez
• Abhinav Prasad
• Ahad Rizvi
• Deepti Talesra