ebay / myriad

Myriad

Project development has moved to: https://github.com/mesos/myriad.

Myriad is a mesos framework designed for scaling YARN clusters on Mesos. Myriad can expand or shrink one or more YARN clusters in response to events as per configured rules and policies.

The name Myriad means, countless or extremely great number. In context of the project, it allows one to expand overall resources managed by Mesos, even when the cluster under mesos management runs other cluster mangaers like YARN.

Videos and SLides

• MesosCon 2014 - Running YARN alongside Mesos (video) (slides)

myriad's Issues

Not able to run Node managers using Myriad??

Hello,

I started Myriad successfully and it is nicely integrated with Mesos as a Framework. But it is showing Node Managers always as a pending task. When I checked the log of Mesos it is offering resource to Myriad but Myriad framework is declining the resources suddenly. I reduced the size of resources for Node Managers in myriad-config-default.yml. But still it is in the same state. I dont have much logs to look into for understanding what is causing the issue. Iam using Mesos 1.0.0 Hadoop 2.7.2 and Myriad executer 0.2.0. Is this a version compatibility issue between Mesos and Myriad?? Any help regarding this issue is really appreciated.

My yarn-root-resourcemanager-mesos.out is as below

Oct 17, 2016 4:11:57 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.myriad.api.ClustersResource as a root resource class Oct 17, 2016 4:11:57 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.myriad.api.ConfigurationResource as a root resource class Oct 17, 2016 4:11:57 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.myriad.api.SchedulerStateResource as a root resource class Oct 17, 2016 4:11:57 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.myriad.api.ControllerResource as a root resource class Oct 17, 2016 4:11:57 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.myriad.api.ArtifactsResource as a root resource class Oct 17, 2016 4:11:57 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider as a provider class Oct 17, 2016 4:11:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' Oct 17, 2016 4:11:58 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider to GuiceManagedComponentProvider with the scope "Singleton" Oct 17, 2016 4:11:59 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.myriad.api.ClustersResource to GuiceManagedComponentProvider with the scope "PerRequest" Oct 17, 2016 4:11:59 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.myriad.api.ConfigurationResource to GuiceManagedComponentProvider with the scope "PerRequest" Oct 17, 2016 4:11:59 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.myriad.api.SchedulerStateResource to GuiceManagedComponentProvider with the scope "PerRequest" Oct 17, 2016 4:11:59 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.myriad.api.ControllerResource to GuiceManagedComponentProvider with the scope "PerRequest" Oct 17, 2016 4:11:59 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.myriad.api.ArtifactsResource to GuiceManagedComponentProvider with the scope "PerRequest" I1017 16:11:59.989440 9720 sched.cpp:226] Version: 1.0.0 I1017 16:11:59.995721 9753 sched.cpp:330] New master detected at [email protected]:5050 I1017 16:11:59.996100 9753 sched.cpp:341] No credentials provided. Attempting to register without authentication I1017 16:11:59.998183 9748 sched.cpp:743] Framework registered with 6215a35e-749e-4f27-bb50-f7c01650da80-0006 Oct 17, 2016 4:12:01 PM com.google.inject.servlet.GuiceFilter setPipeline WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected. Oct 17, 2016 4:12:02 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver as a provider class Oct 17, 2016 4:12:02 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices as a root resource class Oct 17, 2016 4:12:02 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class Oct 17, 2016 4:12:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' Oct 17, 2016 4:12:02 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton" Oct 17, 2016 4:12:02 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" Oct 17, 2016 4:12:03 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices to GuiceManagedComponentProvider with the scope "Singleton"

"sh: 1: hdfs: not found" encountered with "setup-yarn-2.sh" script

I'm seeing the following behavior when trying to run the "setup-yarn-2.sh" script.

Hadoop 2.5.0
Subversion http://svn.apache.org/repos/asf/hadoop/common -r 1616291
Compiled by jenkins on 2014-08-06T17:31Z
Compiled with protoc 2.5.0
From source with checksum 423dcd5a752eddd8e45ead6fd5ff9a24
This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0.jar
sh: 1: hdfs: not found