bzantium / adamp

AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights

[2024.06 - Current] LLM Researcher @Kakao

• Developing Decoder based Embedding Models
• Developing Korean LLM on TPUs

[2023.08 - 2024.05] LLM Researcher @Kakaobrain

• Developed Korean Language Foundation Model a.k.a. KoGPT2

[2020.03 - 2023.08] Machine Learning Engineer @SK Telecom & EleutherAI

• Developed Large Language Model for SK Telecom
• Developed Multimodal AI Service at SKT A.
  • AI Eraser (Object Removal with Image Inpainting)
    • role: project manager & implementation of pipeline algorithm (segmentation postprocessing & enhancement in inpainting performance).
• Developed polyglot and oslo project at EleutherAI
  • polyglot: Large Language Models of Well-balanced Competence in Multi-languages
    • role: distributed training of LM with Megatron LM & data crawling, preprocessing and model evaluaton. Published 1.3B, 3.8B, 5.8B, 12.8B polyglot-ko models.
  • oslo: Open Source for Large-scale Optimization
    • role: tensor parallel 1D, 2D, 3D implementation.

Interest

• Foundation Model / NLP / Multimodal AI