Comments (2)
This is due to an imported package pinning the processor affinity, which Ray then inherits when spawning processes.
One solution would be to use psutil to clear the processor affinity on the main thread after import but before spawning Ray threads.
Another would be to assign each worker its own processor affinity after worker creation.
The former would leave you hopping cores, while the latter is not easily solved in the general case of platforms with unknown physical-vs-virtual core counts. The solution I eventually used was to assign each worker a job that assigned it specific cores, based on deployment details.
from aphrodite-engine.
Fixed with #187
We now properly set the affinities at launch.
from aphrodite-engine.
Related Issues (20)
- Initial fetch for `config.json` ignores `--revision`? HOT 3
- Bad generation with GGUF and OpenAI api HOT 1
- [Bug]: openAI endpoint crashing on "no locator available" HOT 1
- [Bug]: Pydantic serializer issue when pinging /v1/models HOT 2
- [Bug]: `ValueError: Out of range float values are not JSON compliant` when requesting logprobs from awq model HOT 1
- [sparsetral and Qwen2idae]: support for mixtral of lora HOT 12
- [Bug]: exl2 is not auto detected HOT 2
- [Usage]: nccl and cupy problem "no cupy" and "NCCL_ERROR_UNHANDLED_CUDA_ERROR" when use TP in wsl HOT 10
- [Bug]: Issue when trying to load a AWQ model with --load-in-4bits for mixtral flavors HOT 3
- Installation fails on NAVI gpu HOT 2
- [Bug]: loading model with int8 kv cache chokes HOT 1
- [Usage]: Question about VRAM requirement and temperature HOT 2
- [Feature]: Support YiForCausalLM HOT 5
- [Misc]: Building docker container requires insane amount of memory HOT 7
- [Bug]: Outlines json guided decoding HOT 2
- [Feature]: BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences HOT 1
- [Bug]: Does --trust-remote-code work? HOT 1
- [Bug]: multi GPU crashes backend HOT 6
- [Bug]: WSL Cuda out of Memory when Trying to Load GGUF Model HOT 8
- [Usage]: load-in-4bit not load after converted, and it seem not use swap well
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aphrodite-engine.