Comments (8)
@manuelstein , @paarijaat , @ruichuan, any thoughts?
from knix.
from knix.
No, I mean for the end users to configure how their workflows could be deployed (i.e., with what kind of container resources). This would be similar to the way AWS Lambda users would pick their configuration with different RAM.
from knix.
The limits of a container would depend on the functions that are deployed in it and the targeted level of concurrency that the container should be able to handle. Based on this, scaling thresholds can be configured. If the concurrency level is low, scaling might start to scale-out early, and response times might suffer during the scale-out. If provided resources allow high concurrency, we accept over-provisioning for better response times.
Should we allow the user to provide an initial resource estimate per function instance?
Could we report/collect the CPU time and memory use after a function invocation has completed?
from knix.
from knix.
from knix.
Yes, that is true. I forgot to consider that the users shouldn't have to reason about the resource usage of their functions.
I think it would be good if we could measure each function instance's CPU and memory usage. It would help to adjust the scaling policy for the kubernetes setup as well as determining the scaling policy for the bare-metal case (which is something we should also work on).
We can start gathering statistics about already deployed functions. When we need to scale out a deployed workflow, we can adjust the resources accordingly for the new scaled instance (e.g., this sandbox had 1CPU and 1GB RAM, and i am now scaling it up for 3rd time in the last 5 minutes, so maybe I'll just give it more CPU and RAM this time). Similarly, if this function is used in another workflow that is being deployed, we can also consider that information.
Then during deployment, we could pick the resources of the sandbox according to some heuristics with regards the functions and states utilized in the workflow. For example, if the workflow contains a 'Parallel' or 'Map' state, we pick a sandbox flavour with more CPU and RAM.
That said, how about we have a few different flavours of sandboxes with different RAM and CPU configurations (e.g., [1 CPU, 1 GB], [1 CPU, 2 GB], [2 CPUs, 4GB]) to choose from? Or we could just have dynamic values in the deployWorkflow procedure.
My original point was that these values are currently fixed and determined at installation time, and not adjustable afterwards during runtime, so we should try to make it more dynamic.
from knix.
See commit f347d3eb986f4a340340fe149e017a220060731e for the management functionality that allows this configuration for kubernetes environments.
Once we have the corresponding changes in the following, we can close this issue.
- SDK
- Cli
- GUI
from knix.
Related Issues (20)
- CentOS support for ansible deployments HOT 2
- Caching of downloaded function dependencies
- Shutting down sandboxes may hang HOT 5
- Upgrading KNIX breaks the datalayer HOT 1
- workflow logs not available due to elasticsearch sharding problem HOT 1
- test asl_Map hangs with maxconcurrency values != 0 HOT 3
- allow user-supplied data be included in the trigger from TriggersFrontend
- bare metal installation fails due to missing dependencies HOT 22
- error when deploy the knix on a host machine HOT 5
- Does it mean I deploy successful? HOT 1
- error when workflow.excute. requests.exceptions.HTTPError: 405 Client Error: Not Allowed for url: http://XXXX:32777 HOT 10
- knix can not support parallel states. HOT 6
- Error when executing java function with maven dependency of Gson HOT 1
- error[E0658]: use of unstable library feature 'renamed_spin_loop' HOT 9
- 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'ansible_ssh_host' HOT 7
- update riak build scripts to use a newer Ubuntu image HOT 3
- Triggers API object update for Java (storage, message queues, timers) HOT 5
- Recovery manager
- KNIX GPU monitoring/accounting capabilities
- Execution log entries get disordered HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from knix.