Comments (4)
Thank you.
The OptimizingShortfallEvaluator
needs to work in conjunction with TaskSchedulingService
and as such wouldn't do well implemented outside. However, the shortfall evaluation can provide plugins based extensibility that you could use to achieve your results. For example, if there was a plugin that the evaluator called with the initial list of tasks for which shortfall analysis will be performed. Then, you can filter the list appropriately.
Would that cover your needs?
from fenzo.
I think it would in the sense that we could specify arbitrary pseudo-tasks (e.g. reduce 30 x 10-second tasks to effectively a single long-running pseudo-task) to achieve a desirable (or at least more accurate) scaling result.
Having said that, I still like the simplicity of ShortfallEvaluator
's contract... it provides the things that Fenzo knows about (including TaskSchedulingService
if one should want to do some pseudo-scheduling) and potentially allows an arbitrarily complex/custom algorithm to be performed to end up with the final Map<String, Integer>
result. I think in the longer term we'd want to not just consider the immediate needs (i.e. tasks that could not be scheduled yet), but also factor in historical data so that we potentially adjust the shortfall based on what we expect might happen in the next 15 - 30 minutes (and thus avoid pointless down-scaling, or indeed do pre-emptive up-scaling). Using kludge factors to come up with a list of pseudo-tasks for pseudo-scheduling would likely become unintelligible pretty quickly (even to the author).
Shortfall evaluation is kind of a mini-science in itself, so I think it would be neat to see Fenzo users come up with different approaches for different use cases, some of which could perhaps be fed back into the Fenzo codebase eventually. From that perspective, I still like the thought of being able to implement ShortfallEvaluator
myself and plug that in to the AutoScaler
, but certainly anything that allows some custom logic to be plugged in to shortfall evaluation would be welcome.
from fenzo.
I can see how this can be customized for your example. And likely other examples as well. My initial thought was more on handling the complexity in the "optimizing shortfall analysis" correctly. That requires tight integration with the scheduling service. However, other strategies for shortfall analysis can be achieved without such an integration with the internal scheduler. Your example analyzer can estimate shortfall without requiring a pseudo scheduling run, I think.
If that matches with your thinking, it might be worth the effort to put some detail on how the custom shortfall analyzers can be plugged in.
I agree with you, shortfall evaluation is kind of a mini-science in itself.
from fenzo.
Ideally shortfall evaluation (for us) needs to consider the duration of tasks... so from that perspective one would, for a given set of tasks that could not yet be scheduled, want to pseudo-schedule over a period of time... e.g. if I have some period of time for which scaling up can yield some benefit (which is going to be arbitrarily longer than the time it takes to bring a new slave node into service), then I want to know if I would actually have a shortfall based on the tasks that I expect to run within that period of time. So possibly we'd still want to do pseudo-scheduling, but just as one piece of a more complex equation... i.e. it's just a data point in time, and what we're concerned with is the shape of the graph over a period of time.
My thinking about this is still under-developed, but yes, just being able to specify a custom ShortfallEvaluator is something that would be useful... and it's conceivable that being able to do so with a sub-class of OptimizingShortfallEvaluator might also be useful.
from fenzo.
Related Issues (20)
- contributor agreement. HOT 8
- NamedResourceSetRequest HOT 2
- Resource ranges and "greedy" scheduling HOT 4
- Support v1 Protos (or provide utility to convert) HOT 5
- Support dynamic fitness threshold HOT 3
- Execution failed for task ':fenzo-core:compileJava'.> 无效的源发行版: 1.8 HOT 2
- Update Mesos dependency
- BinPacking with weights HOT 2
- Incorrect handling of reserved resources HOT 2
- Unsafe concurrent access to unknown lease collection in AssignableVMs class
- Pluggable ENI fitness evaluator HOT 1
- After reading Fenzo, I don't understand how to get Framework information on Mesos?
- After reading Fenzo, I don't understand how to get Framework information on Mesos?
- DisableVM does not remember hosts HOT 2
- .travis.yml: The 'sudo' tag is now deprecated in Travis CI
- Dynamic ResAllocs
- CVE-2018-7489 on jackson-databind 2.4.5 HOT 1
- Unused variable
- Unreleased Resource: Streams HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fenzo.