MaLA-500 is a novel large language model designed to cover an extensive range of 534 languages. This model builds upon LLaMA 2 7B and integrates continued pretraining with vocabulary extension, with an expanded vocabulary size of 260,164, and LoRA low-rank adaptation.
- Continued Pretraining: Enhances the model's ability to adapt to a wide range of languages.
- LoRA Low-Rank Adaptation: LoRA low-rank adaptation refines the model's adaptation capabilities.
- Vocabulary Extension: MaLA-500 boasts an extended vocabulary size of 260,164.
- Multilingual Proficiency: Trained on Glot500-c, covering 534 languages.
Please refer to our paper for more details.
Requirements:
transformers>=4.36.1
peft>=0.6.2
Use the code below to get started with the model.
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-2-7b-hf')
base_model.resize_token_embeddings(260164)
tokenizer = AutoTokenizer.from_pretrained('MaLA-LM/mala-500-10b-v2')
model = PeftModel.from_pretrained(base_model, 'MaLA-LM/mala-500-10b-v2')
Code under the directory of ./tokenization
.
Code under the directory of ./continued_pretraining
.
Customize the run.sh
script for your own clusters or workstation. The script is provided for SLURM-based systems.
You might want to use DeepSpeed. See config examples under ./continued_pretraining/config
.
Code under the directory of ./evaluation
.
@misc{lin2024mala500,
title={MaLA-500: Massive Language Adaptation of Large Language Models},
author={Peiqin Lin and Shaoxiong Ji and Jörg Tiedemann and André F. T. Martins and Hinrich Schütze},
year={2024},
eprint={2401.13303},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
mala-500's People
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.