Comments (6)
Great to hear. Closing the issue, but please reopen (or create a new issue) if you see this behavior return. I will get this merged into the main
branch and it will be part of the next MII release.
from deepspeed-mii.
遇到了同样的问题。
from deepspeed-mii.
@weiqisun I have a fix in #350. If you would like to try that branch before we merge: pip install git+https://github.com/Microsoft/DeepSpeed-MII@mrwyattii/fix-return-error
from deepspeed-mii.
Thanks @mrwyattii! However, I'm still seeing this error message. I confirm I installed the module from your branch since I have the updated _invoke_async
function now in the installed lib file.
from deepspeed-mii.
Hmm, I'm not able to reproduce this with the fix I have in #350. Could you try adding a print statement that shows the contents of response? Please add
print(f"RANK {self.inference_pipeline.local_rank} RESPONSE:", [r.to_msg_dict() for r in responses])
just before the return
statement here:
You will want to modify this file on your local system: /home/dyheal1/mambaforge/envs/test/lib/python3.10/site-packages/mii/grpc_related/modelresponse_server.py
Share the output of that print statement. Thanks!
from deepspeed-mii.
Actually, nvm. With a clean setup from scratch, now the error message is gone! I'm not sure if it was due to a not-properly terminated process. I previously had a server running overnight. I stopped the server this morning before updating the mii module and I still observed the same error message after the update. But then I realized there were two leftover mii processes after I terminated the server by client.terminate_server()
:
dyheal1 291962 0.9 0.0 41352944 474320 pts/8 Sl 11:26 0:10 /home/dyheal1/mambaforge/envs/test/bin/python -m mii.launch.multi_gpu_server --deployment-name mii-endpoint --load-balancer-port 50050 --restful-gateway-port 51080 --restful-gateway-procs 32 --load-balancer --model-config eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAibWV0YS1sbGFtYS9MbGFtYS0yLTdiLWhmIiwgInRva2VuaXplciI6ICJtZXRhLWxsYW1hL0xsYW1hLTItN2ItaGYiLCAidGFzayI6ICJ0ZXh0LWdlbmVyYXRpb24iLCAidGVuc29yX3BhcmFsbGVsIjogMiwgImluZmVyZW5jZV9lbmdpbmVfY29uZmlnIjogeyJ0ZW5zb3JfcGFyYWxsZWwiOiB7InRwX3NpemUiOiAyfSwgInN0YXRlX21hbmFnZXIiOiB7Im1heF90cmFja2VkX3NlcXVlbmNlcyI6IDIwNDgsICJtYXhfcmFnZ2VkX2JhdGNoX3NpemUiOiA3NjgsICJtYXhfcmFnZ2VkX3NlcXVlbmNlX2NvdW50IjogNTEyLCAibWF4X2NvbnRleHQiOiA4MTkyLCAibWVtb3J5X2NvbmZpZyI6IHsibW9kZSI6ICJyZXNlcnZlIiwgInNpemUiOiAxMDAwMDAwMDAwfSwgIm9mZmxvYWQiOiBmYWxzZX19LCAidG9yY2hfZGlzdF9wb3J0IjogMjk1MDAsICJ6bXFfcG9ydF9udW1iZXIiOiAyNTU1NSwgInJlcGxpY2FfbnVtIjogMSwgInJlcGxpY2FfY29uZmlncyI6IFt7Imhvc3RuYW1lIjogImxvY2FsaG9zdCIsICJ0ZW5zb3JfcGFyYWxsZWxfcG9ydHMiOiBbNTAwNTEsIDUwMDUyXSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiZ3B1X2luZGljZXMiOiBbMCwgMV0sICJ6bXFfcG9ydCI6IDI1NTU1fV0sICJtYXhfbGVuZ3RoIjogNDA5NiwgImFsbF9yYW5rX291dHB1dCI6IGZhbHNlLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9
dyheal1 292647 1.1 0.0 41353024 474248 pts/8 Sl 11:30 0:10 /home/dyheal1/mambaforge/envs/test/bin/python -m mii.launch.multi_gpu_server --deployment-name mii-endpoint --load-balancer-port 50050 --restful-gateway-port 51080 --restful-gateway-procs 32 --load-balancer --model-config eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAibWV0YS1sbGFtYS9MbGFtYS0yLTdiLWhmIiwgInRva2VuaXplciI6ICJtZXRhLWxsYW1hL0xsYW1hLTItN2ItaGYiLCAidGFzayI6ICJ0ZXh0LWdlbmVyYXRpb24iLCAidGVuc29yX3BhcmFsbGVsIjogMiwgImluZmVyZW5jZV9lbmdpbmVfY29uZmlnIjogeyJ0ZW5zb3JfcGFyYWxsZWwiOiB7InRwX3NpemUiOiAyfSwgInN0YXRlX21hbmFnZXIiOiB7Im1heF90cmFja2VkX3NlcXVlbmNlcyI6IDIwNDgsICJtYXhfcmFnZ2VkX2JhdGNoX3NpemUiOiA3NjgsICJtYXhfcmFnZ2VkX3NlcXVlbmNlX2NvdW50IjogNTEyLCAibWF4X2NvbnRleHQiOiA4MTkyLCAibWVtb3J5X2NvbmZpZyI6IHsibW9kZSI6ICJyZXNlcnZlIiwgInNpemUiOiAxMDAwMDAwMDAwfSwgIm9mZmxvYWQiOiBmYWxzZX19LCAidG9yY2hfZGlzdF9wb3J0IjogMjk1MDAsICJ6bXFfcG9ydF9udW1iZXIiOiAyNTU1NSwgInJlcGxpY2FfbnVtIjogMSwgInJlcGxpY2FfY29uZmlncyI6IFt7Imhvc3RuYW1lIjogImxvY2FsaG9zdCIsICJ0ZW5zb3JfcGFyYWxsZWxfcG9ydHMiOiBbNTAwNTEsIDUwMDUyXSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiZ3B1X2luZGljZXMiOiBbMCwgMV0sICJ6bXFfcG9ydCI6IDI1NTU1fV0sICJtYXhfbGVuZ3RoIjogNDA5NiwgImFsbF9yYW5rX291dHB1dCI6IGZhbHNlLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9
After manually killing these two processes, I started the server again and the error message is gone. Thanks for the fix!
from deepspeed-mii.
Related Issues (20)
- [FEATURE] Access to logits and final hidden layer HOT 1
- RuntimeError: The server socket has failed to listen on any local network address HOT 1
- Only running one replica even though setting many replicas
- [Problem]errno: 98 - Address already in use
- Performance with vllm HOT 1
- error when using Qwen1.5-32B
- ValueError: Unsupported model type phi3 HOT 1
- BUG in run_batch_processing
- Cannot run Yi-34B-Chat => ValueError: Unsupported q_ratio: 7 HOT 2
- [REQUEST] Mixtral-8x22B support
- [REQUEST] LLAMA-3 support
- Does deepspeed-mii support prefix_allowed_tokens_fn?
- DeepSpeed-MII 能加载量化的int4或者int8的模型吗?
- Tf32 support
- How can I use the same prompt to produce the same text output as vllm
- Support LLava next stronger
- support Qwen
- support Qwen1.5
- support stream
- [BUG] MII Backend Hangs After 9999 Exceptions in `MIIAsyncPipeline.put_request` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepspeed-mii.