GithubHelp home page GithubHelp logo

talhaahmed2000 / summer-school-transformers-2023 Goto Github PK

View Code? Open in Web Editor NEW

This project forked from moritzlaurer/summer-school-transformers-2023

0.0 0.0 0.0 25.99 MB

Course repository for the session "Hands-on Transformers: Fine-Tune your own BERT and GPT" of the Data Science Summer School 2023

Home Page: https://ds3.ai/2023/transformer.html

Jupyter Notebook 100.00%

summer-school-transformers-2023's Introduction

Course: Hands-on Transformers: Fine-Tune your own BERT and GPT

This repository contains the materials for the course "Hands-on Transformers: Fine-Tune your own BERT and GPT" on 22. August 2023 by Moritz Laurer for the Data Science Summer School 2023

The session will be recorded and the recording will be published on YouTube.

Main notebooks used in the course:

1_open_source_toolkit.ipynb: An overview of accessible hardware (Google Colab) and easy-to-use software (Hugging Face Transformers) to use Transformers without a PhD in deep learning.

2_inside_transformers.ipynb: A high-level breakdown of the main components for using Transformers (models & tokenizer).

3_tune_bert.ipynb: An end-to-end training pipeline for training and evaluating a BERT-base model.

4_tune_bert_nli.ipynb: An end-to-end training pipeline for training and evaluating a BERT-NLI model.

5_data_quality_and_cleaning.ipynb: Illustrations of data quality issues in established datasets and a tool (CleanLab) for automatic data cleaning.

6_annotation_interface_argilla.ipynb: A demo of the free data annotation interface Argilla for creating your own training and test data.

7_tune_generative_llm.ipynb: An end-to-end training pipeline for training and evaluating a generative LLM (FLAN-T5).

Course Abstract:

While Transformer models like BERT and GPTs are becoming more popular, there is a persistent misconception that they are very complicated to use. This workshop will demonstrate that this is not the case anymore. There are amazing open-source packages like Hugging Face Transformers that enable anyone with some programming knowledge to use, train and evaluate Transformers.

We will start with an intuitive introduction to transfer learning and discuss its added value for social science use-cases as well as limitations. We will then look at the open-source ecosystem and free hardware options to train Transformers. Building upon a high-level explanation of the main components of Transformers in Hugging Face’s implementation, we will then fine-tune different BERT and GPT models and discuss important aspects of fine-tuning and evaluation.

The code demonstrations will be in Python, but participants without prior knowledge of Python or Transformers are explicitly invited to participate. You will leave the workshop with Jupyter notebooks that enable you to train your own Transformer with your own data for your future research projects.

License:

Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). You are free to copy, change and re-use the code. If you are using larger portions of the codebase, please provide proper attribution. See full license text here.

summer-school-transformers-2023's People

Contributors

moritzlaurer avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.