semyonsinchenko / feature-generation-benchmark

A database-like benchmark of feature generation from time-series data

Semyon Sinchenko

Education

Master degree in solid state physics, Moscow State University, Moscow 2022

Bachelor degree in solid state physics, Moscow Engeneering Physical Institute, Moscow 2016

Skills

• Python
• Java
• Scala
• Data Engineering Stack (Apache Spark, Scala, Databricks, Airflow, Kafka, Ni-Fi, SQL)
• Python ML Stack (NumPy, SciPy, Scikit-Learn, Tensorflow-2, XgBoost, Pandas)

Self-education

• DataCamp specialization "Data Scientist with Python"
• DataCamp specialization "Data Scientist with R"

Expertise

• Data Engineering
• Data Science

Experience

Data Engineer, Raiffeisenbank International AG: 2022-06 - (Now)

Data Scientist / Engineer, Raiffeisenbank Russia: 2018-01 - 2022-06

feature-generation-benchmark's Issues

[GENERATION] Use multiple threads for data generation

The current implementation is single threaded. I see two ways who to improve it:

• make a Rust implementation multi-threading (may tend to non determenistic bahviour of rnd);
• create multiple tasks in python-side on a top level (may tend to memory issues);

[Benchmark] Add Polars implementation

[CI] Run bechmarks on GitHub renners instead of local

Currently all benchmarks are running by myself on the laptop. Could be cool to automate that process.

[CI] Add a release build of data-generator package

It could be cool to provide an optimized and compiled binaries and python wheels for different systems (like Linux, MacOS, Windows) in releases.

[BENCHMARK][SPARK] Implement a pivot-approach for PySpark

[SITE] MKDocs GH actions ignores theme and configurations

It looks like some requirements are missing on the step of generation.