Comments (4)
This is deliberate. You'll notice how the resources are broken down by executor/task, this behaviour is in place because when the task tracker is idle, we want to kill it and free up a chunk of resource that can be used for other frameworks. The problem is that we can't kill the tracker immediately, since it might have local MAP output data on disk that needs to be served to running reduce tasks on other trackers.
The details of this implementation can be found in #32 and #33.
TL;DR; If you run a big job that has 1000 maps and 200 reducers, without this behaviour the 1000 map slots would sit idle after they'd finished and would not be freed to mesos until the 200 reducers had finished. This is not ideal in a multi-tentant cluster.
It'd be interesting to hear if you have any feedback on the approach, because it was really a first stab in the dark at trying to improve the utilization of mesos when running sparse hadoop jobs.
Note: It's perfectly legal behaviour for all mesos tasks for an executor to finish while the executor stays alive.
from hadoop.
So, I spent a lot of time looking at the failure modes of this code lately.
What we found was that since the executor continued to run, but the task reported finished other task trackers and storm workers would attempt to use the port and then crash due to port bind exceptions.
This essentially crashed storm for long periods, it may be legal but it's not polite. It also had negative impact on hadoop as tasktrackers would fail when reducers needed to be launched. Keep in mind a large map reduce job may take hours to complete so lots of tasks fail (often enough reduce tasks to kill a job).
We did attempt to correct this by creating a blocking blocking version of suicide timer inside the the thread so it would wait for everything to complete before declaring TASK_FINISHED, however as the initial reducers got killed (idle waiting for the shuffle phase) there were no new resources for new task trackers to spawn.
I think the approach has some merit, and I spend some time looking at it but ultimately didn't have the time to work on it. Eventually roled back to 0.0.9.
from hadoop.
What we found was that since the executor continued to run, but the task reported finished other task trackers and storm workers would attempt to use the port and then crash due to port bind exceptions.
Hah, that's a bug here that is trivial to fix, we need to move the port resource to the ExecutorInfo that's defined just above so that the ports don't get offered to other frameworks while it's still alive.
That should solve your issue entirely, assuming Storm also honours the port resource correctly (which i'd expect it does).
from hadoop.
OK, I can try that and report back.
from hadoop.
Related Issues (20)
- Skip extracting hadoop distribution on Task_Tracker creation HOT 1
- Is the offering logging misleading? HOT 3
- Map/Reduce slot allocation is not ideal for small clusters HOT 1
- Launch separate TaskTracker instances for Map and Reduce slots
- Support for the new ContainerInfo (and Mesos<>Docker) HOT 4
- Getting hadoop-mesos to work on older hadoop HOT 2
- Running multiple instances HOT 12
- Framework Authentication HOT 1
- Style Enforcement HOT 6
- Hadoop on Mesos uses only one node? HOT 2
- Maven cannot build package HOT 2
- Deadlock Between MesosScheduler and JobTracker HOT 7
- Can not launch TaskTracker (Error occurred during initialization of VM) HOT 2
- Build failed for mesos with version>=0.27.0 HOT 1
- Can't launched Tasktracker
- The url pseudo distributed operation in configure tab is 404
- Need help in configuring hadoop on mesos cluster HOT 4
- Spark Mesos Docker containerizer cannot run
- Error: Could not find or load main class org.apache.hadoop.mapred.MesosExecutor HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hadoop.