GithubHelp home page GithubHelp logo

suntzui / bertviz Goto Github PK

View Code? Open in Web Editor NEW

This project forked from jessevig/bertviz

0.0 1.0 0.0 100.01 MB

Tool for visualizing attention in the Transformer model (BERT, GPT-2, Albert, XLNet, RoBERTa, CTRL, etc.)

License: Apache License 2.0

JavaScript 0.10% Python 1.00% Jupyter Notebook 98.90%

bertviz's Introduction

BertViz

BertViz is a tool for visualizing attention in the Transformer model, supporting all models from the transformers library (BERT, GPT-2, XLNet, RoBERTa, XLM, CTRL, etc.). It extends the Tensor2Tensor visualization tool by Llion Jones and the transformers library from HuggingFace.

Blog post:

Paper:

Related blog posts:

Attention-head view

The attention-head view visualizes the attention patterns produced by one or more attention heads in a given transformer layer.

Attention-head view Attention-head view animated

The attention view supports all models from the Transformers library, including:
BERT: [Notebook] [Colab]
GPT-2: [Notebook] [Colab]
XLNet: [Notebook]
RoBERTa: [Notebook]
XLM: [Notebook]
Albert: [Notebook]
DistilBert: [Notebook]
(and others)

Model view

The model view provides a birds-eye view of attention across all of the model’s layers and heads.

Model view

The model view supports all models from the Transformers library, including:
BERT: [Notebook] [Colab]
GPT2: [Notebook] [Colab]
XLNet: [Notebook]
RoBERTa: [Notebook]
XLM: [Notebook]
Albert: [Notebook]
DistilBert: [Notebook]
(and others)

Neuron view

The neuron view visualizes the individual neurons in the query and key vectors and shows how they are used to compute attention.

Neuron view

The neuron view supports the following three models:
BERT: [Notebook] [Colab]
GPT-2 [Notebook] [Colab]
RoBERTa [Notebook]

Execution

Running locally

git clone https://github.com/jessevig/bertviz.git
cd bertviz
jupyter notebook

Click on any of the sample notebooks. Be sure to first install any necessary dependencies from requirements.txt (not required for viewing cached visualizations, described below). Note that the sample notebooks do not cover all Huggingface models, but the code should be similar for those not included.

Viewing cached visualizations

If you wish to view the cached visualization output in a notebook without running the code yourself, run the following commands prior to viewing the notebook:

cd bertviz
jupyter trust <Notebook Name>.ipynb

Running from Colab

Click on any of the Colab links above, and scroll to the bottom of the page. It should be pre-loaded with the visualization.

If you write your own code for executing BertViz in Colab, note that some of the steps are different from those in the Jupyter notebooks (see Colab examples above).

Current limitations

The visualizations works best with shorter sentences and may fail if the input text is very long. The tool is designed such that only one visualization should be included per notebook. If you have issues running the tool in Jupyter Lab, try running with a plain Jupyter notebook.

Authors

Jesse Vig

Citation

When referencing BertViz, please cite this paper.

@inproceedings{vig-2019-multiscale,
    title = "A Multiscale Visualization of Attention in the Transformer Model",
    author = "Vig, Jesse",
    booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
    month = jul,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/P19-3007",
    doi = "10.18653/v1/P19-3007",
    pages = "37--42",
}

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details

Acknowledgments

We thank the authors of the following projects, which are incorporated into this repo:

bertviz's People

Contributors

jessevig avatar pglock avatar

Watchers

James Cloos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.