Comments (3)
Hi @s4rduk4r - this is a known issue with DeepSpeed 0.14.2, in the meantime you can either build from source or downgrade or wait for the next release. More info was tracked in this thread (and others linked there)
from deepspeed.
Hi, @loadams. Thank you for your reply. I think I'll wait till the next release
from deepspeed.
Thanks, I'll close this issue for now since this is a known issue and resolved already.
from deepspeed.
Related Issues (20)
- nv-nightly CI test failure
- [BUG] The specified pointer resides on host memory and is not registered with any CUDA device. HOT 1
- [BUG] Memory consumption higher when weight layers are clipped after .step() HOT 1
- Use Pipeline Parallelism and get stuck in the mid[BUG]
- RuntimeError: Error(s) in loading state_dict
- [BUG] Does deepspeed work with torch amp autocast?
- [BUG]I found that the parameters of model will be fully transferred to the VRAM of each process. Is this abnormal in my understanding? HOT 5
- [BUG] fp6 canβt load qwen1.5-34b-chat
- [BUG] deepspeed amp seems to convert all input to specific dtype
- Data Loading for DeepSpeed Ulysses and Data Parallelism
- different setting for same (num_gpus * batch_size * grad_accum_steps) output different loss and gradient norm HOT 1
- [BUG] DeepSpeed is loads the whole model to every GPUs instead of partitioning HOT 1
- [BUG] RepeatingLoader may be invalid in the pipe stages neither the fist nor last
- [REQUEST] Supporting custom generation loop (outlines, LMQL, guidance) in DeepSpeedHybridEngine
- [BUG] M1 Mac has an issue with `hostname -I` not being a valid command HOT 2
- [BUG] CUDA OOM error when Hugging Face `ignore_mismatched_sizes` is enabled
- [BUG] Zero3 causes AttributeError: 'NoneType' object has no attribute 'numel' in continual training HOT 3
- [BUG] cannot import name '_get_socket_with_port' from 'torch.distributed.elastic.agent.server.api' HOT 4
- # [REQUEST] Upstream modifications of PaRO
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepspeed.