Comments (5)
it’ll be available on master in a few hours
from transfer-learning-conv-ai.
Hi @williamFalcon ,
sry to ask a question out of this context. But I am stuck with 2 differfent issues.
- how to continue training from the checkpoint. Say, after training for 2 epochs, I tried to load the checkpoint and resume training, but was unable to do so.
- What is the exact CUDA & CuDNN etc. configuration needed to import amp ?
from transfer-learning-conv-ai.
to continue training use load_from_metrics. but that won’t reinstate the training cycle. that’s only supported in the cluster case (see hoc_load hoc_save).
we do have a simpler interface coning in the next few days for it. check oit #27
from transfer-learning-conv-ai.
@williamFalcon, ok I would be waiting for that update. FInetuning the model on custom dataset is taking around 8-9 hours on single V100 GPU. Therefore, we need a method to use the saved checkpoint and resume training from there later on.
from transfer-learning-conv-ai.
@williamFalcon Sounds cool! Could you create a pull request to integrate pytorch-lightning for multi-node multi-GPU fine-tuning? I want to move beyond multi-GPU single-node fine-tuning (which is what this repo currently seems to support) to full-scale distributed fine-tuning with several nodes.
from transfer-learning-conv-ai.
Related Issues (20)
- pip version problems - updated requirements.txt
- Issue when running interact script HOT 1
- Custom vocabulary HOT 1
- GPT2 chat-bot single interaction… Attribute Error: 'NoneType' object has no attribute 'multiprocessing_chunksize' issue:2
- How to use Interaction through a web api call instead CLI
- Train.py not working
- Reason for parameter 'personality_permutations' HOT 1
- Question about "token_type_ids"
- Batchwise padding dataset
- Load model checkpoint
- I just figured it out. You need both "train" and "valid" examples, following the same structure as: HOT 1
- Questions about interact.py
- Is It Possible To Train With Unsupported Language In Spacy
- Questions about segment embedding
- Which Python version to use to install the libraries from requirements.txt
- How should the value of --num_candidates and --personality_permutations be determined?
- the meaning of code
- metrics: HOT 1
- cannot import name 'cached_path' from 'transformers' HOT 3
- How to input personality?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transfer-learning-conv-ai.