alexjbush / livy_zeppelin_cdh_csd_parcels Goto Github PK

To use this tool in cm5.7,vi bI modified some thing of bulid.sh and livy_zeppelin_cdh_csd_parcels/livy-csd-src/descriptor/service.sdl .
-------bulid.sh------
-- CM_EXT_BRANCH=cm5-5.12.0
++ CM_EXT_BRANCH=cm5-5.7.0

-- livy_parcel_name="$livy_parcel_folder-el7.parcel"
++ livy_parcel_name="$livy_parcel_folder-el6.parcel"

---------livy_zeppelin_cdh_csd_parcels/livy-csd-src/descriptor/service.sdl---
"cdhVersion": {
-- '''min':''5.11.0"
++ "min": "5.7.0"
}

But, Livy with cloudera manager can not deploy client configuration. and error info as ：

`Version: Cloudera Express 5.7.2 (#17 built by jenkins on 20160722-1347 git: 1ac5976e8ad8f16506c2db236aee83141915c44d)

com.cloudera.cmf.command.CmdExecException:java.lang.NullPointerException
at SeqFlowCmd.java line 120
in com.cloudera.cmf.command.flow.SeqFlowCmd run()
Stack Trace:
SeqFlowCmd.java line 120
in com.cloudera.cmf.command.flow.SeqFlowCmd run()
CmdWorkCommand.java line 94
in com.cloudera.cmf.command.CmdWorkCommand execute()
ServiceHandlerRegistry.java line 778
in com.cloudera.cmf.service.ServiceHandlerRegistry executeCommandHelper()
ServiceHandlerRegistry.java line 738
in com.cloudera.cmf.service.ServiceHandlerRegistry executeCommand()
ServiceHandlerRegistry.java line 733
in com.cloudera.cmf.service.ServiceHandlerRegistry executeCommand()
OperationsManagerImpl.java line 1716
in com.cloudera.server.cmf.components.OperationsManagerImpl executeServiceCmd()
ServiceController.java line 91
in com.cloudera.server.web.cmf.ServiceController serviceDispatch()
line -1
in com.cloudera.server.web.cmf.ServiceController$$FastClassByCGLIB$$696d0ebc invoke()
MethodProxy.java line 191
in net.sf.cglib.proxy.MethodProxy invoke()
Cglib2AopProxy.java line 688
in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
ReflectiveMethodInvocation.java line 150
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
MethodSecurityInterceptor.java line 61
in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
ReflectiveMethodInvocation.java line 172
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
Cglib2AopProxy.java line 621
in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
line -1
in com.cloudera.server.web.cmf.ServiceController$$EnhancerByCGLIB$$4b3745ca serviceDispatch()
NativeMethodAccessorImpl.java line -2
in sun.reflect.NativeMethodAccessorImpl invoke0()
NativeMethodAccessorImpl.java line 62
in sun.reflect.NativeMethodAccessorImpl invoke()
DelegatingMethodAccessorImpl.java line 43
in sun.reflect.DelegatingMethodAccessorImpl invoke()
Method.java line 498
in java.lang.reflect.Method invoke()
HandlerMethodInvoker.java line 176
in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 436
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 424
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
DispatcherServlet.java line 790
in org.springframework.web.servlet.DispatcherServlet doDispatch()
DispatcherServlet.java line 719
in org.springframework.web.servlet.DispatcherServlet doService()
FrameworkServlet.java line 669
in org.springframework.web.servlet.FrameworkServlet processRequest()
FrameworkServlet.java line 585
in org.springframework.web.servlet.FrameworkServlet doPost()
HttpServlet.java line 595
in javax.servlet.http.HttpServlet service()
HttpServlet.java line 668
in javax.servlet.http.HttpServlet service()
ServletHolder.java line 511
in org.mortbay.jetty.servlet.ServletHolder handle()
ServletHandler.java line 1221
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
UserAgentFilter.java line 78
in org.mortbay.servlet.UserAgentFilter doFilter()
GzipFilter.java line 131
in org.mortbay.servlet.GzipFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JAMonServletFilter.java line 48
in com.jamonapi.http.JAMonServletFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JavaMelodyFacade.java line 109
in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
FilterChainProxy.java line 311
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterSecurityInterceptor.java line 116
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
FilterSecurityInterceptor.java line 83
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ExceptionTranslationFilter.java line 113
in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SessionManagementFilter.java line 101
in org.springframework.security.web.session.SessionManagementFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AnonymousAuthenticationFilter.java line 113
in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RememberMeAuthenticationFilter.java line 146
in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextHolderAwareRequestFilter.java line 54
in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RequestCacheAwareFilter.java line 45
in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AbstractAuthenticationProcessingFilter.java line 182
in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
LogoutFilter.java line 105
in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextPersistenceFilter.java line 87
in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ConcurrentSessionFilter.java line 125
in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterChainProxy.java line 173
in org.springframework.security.web.FilterChainProxy doFilter()
DelegatingFilterProxy.java line 237
in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
DelegatingFilterProxy.java line 167
in org.springframework.web.filter.DelegatingFilterProxy doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
CharacterEncodingFilter.java line 88
in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
OncePerRequestFilter.java line 76
in org.springframework.web.filter.OncePerRequestFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
ServletHandler.java line 399
in org.mortbay.jetty.servlet.ServletHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
SessionHandler.java line 182
in org.mortbay.jetty.servlet.SessionHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
ContextHandler.java line 767
in org.mortbay.jetty.handler.ContextHandler handle()
WebAppContext.java line 450
in org.mortbay.jetty.webapp.WebAppContext handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
StatisticsHandler.java line 53
in org.mortbay.jetty.handler.StatisticsHandler handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
Server.java line 326
in org.mortbay.jetty.Server handle()
HttpConnection.java line 542
in org.mortbay.jetty.HttpConnection handleRequest()
HttpConnection.java line 945
in org.mortbay.jetty.HttpConnection$RequestHandler content()
HttpParser.java line 756
in org.mortbay.jetty.HttpParser parseNext()
HttpParser.java line 218
in org.mortbay.jetty.HttpParser parseAvailable()
HttpConnection.java line 404
in org.mortbay.jetty.HttpConnection handle()
SelectChannelEndPoint.java line 410
in org.mortbay.io.nio.SelectChannelEndPoint run()
QueuedThreadPool.java line 582
in org.mortbay.thread.QueuedThreadPool$PoolThread run()
Caused by: java.lang.NullPointerException:
at DynamicClientConfigHandler.java line 67
in com.cloudera.cmf.service.csd.components.DynamicClientConfigHandler getClientConfigMetadata()
Stack Trace:
DynamicClientConfigHandler.java line 67
in com.cloudera.cmf.service.csd.components.DynamicClientConfigHandler getClientConfigMetadata()
AbstractClientConfigHandler.java line 64
in com.cloudera.cmf.service.AbstractClientConfigHandler createClientConfig()
AbstractClientConfigHandler.java line 43
in com.cloudera.cmf.service.AbstractClientConfigHandler createClientConfig()
GenerateClientConfigsCmdWork.java line 189
in com.cloudera.cmf.service.GenerateClientConfigsCmdWork createClientConfig()
GenerateClientConfigsCmdWork.java line 102
in com.cloudera.cmf.service.GenerateClientConfigsCmdWork doWork()
CmdStep.java line 164
in com.cloudera.cmf.command.flow.CmdStep doWork()
SeqCmdWork.java line 98
in com.cloudera.cmf.command.flow.SeqCmdWork doWork()
SeqFlowCmd.java line 118
in com.cloudera.cmf.command.flow.SeqFlowCmd run()
CmdWorkCommand.java line 94
in com.cloudera.cmf.command.CmdWorkCommand execute()
ServiceHandlerRegistry.java line 778
in com.cloudera.cmf.service.ServiceHandlerRegistry executeCommandHelper()
ServiceHandlerRegistry.java line 738
in com.cloudera.cmf.service.ServiceHandlerRegistry executeCommand()
ServiceHandlerRegistry.java line 733
in com.cloudera.cmf.service.ServiceHandlerRegistry executeCommand()
OperationsManagerImpl.java line 1716
in com.cloudera.server.cmf.components.OperationsManagerImpl executeServiceCmd()
ServiceController.java line 91
in com.cloudera.server.web.cmf.ServiceController serviceDispatch()
line -1
in com.cloudera.server.web.cmf.ServiceController$$FastClassByCGLIB$$696d0ebc invoke()
MethodProxy.java line 191
in net.sf.cglib.proxy.MethodProxy invoke()
Cglib2AopProxy.java line 688
in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
ReflectiveMethodInvocation.java line 150
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
MethodSecurityInterceptor.java line 61
in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
ReflectiveMethodInvocation.java line 172
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
Cglib2AopProxy.java line 621
in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
line -1
in com.cloudera.server.web.cmf.ServiceController$$EnhancerByCGLIB$$4b3745ca serviceDispatch()
NativeMethodAccessorImpl.java line -2
in sun.reflect.NativeMethodAccessorImpl invoke0()
NativeMethodAccessorImpl.java line 62
in sun.reflect.NativeMethodAccessorImpl invoke()
DelegatingMethodAccessorImpl.java line 43
in sun.reflect.DelegatingMethodAccessorImpl invoke()
Method.java line 498
in java.lang.reflect.Method invoke()
HandlerMethodInvoker.java line 176
in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 436
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 424
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
DispatcherServlet.java line 790
in org.springframework.web.servlet.DispatcherServlet doDispatch()
DispatcherServlet.java line 719
in org.springframework.web.servlet.DispatcherServlet doService()
FrameworkServlet.java line 669
in org.springframework.web.servlet.FrameworkServlet processRequest()
FrameworkServlet.java line 585
in org.springframework.web.servlet.FrameworkServlet doPost()
HttpServlet.java line 595
in javax.servlet.http.HttpServlet service()
HttpServlet.java line 668
in javax.servlet.http.HttpServlet service()
ServletHolder.java line 511
in org.mortbay.jetty.servlet.ServletHolder handle()
ServletHandler.java line 1221
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
UserAgentFilter.java line 78
in org.mortbay.servlet.UserAgentFilter doFilter()
GzipFilter.java line 131
in org.mortbay.servlet.GzipFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JAMonServletFilter.java line 48
in com.jamonapi.http.JAMonServletFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JavaMelodyFacade.java line 109
in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
FilterChainProxy.java line 311
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterSecurityInterceptor.java line 116
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
FilterSecurityInterceptor.java line 83
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ExceptionTranslationFilter.java line 113
in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SessionManagementFilter.java line 101
in org.springframework.security.web.session.SessionManagementFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AnonymousAuthenticationFilter.java line 113
in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RememberMeAuthenticationFilter.java line 146
in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextHolderAwareRequestFilter.java line 54
in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RequestCacheAwareFilter.java line 45
in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AbstractAuthenticationProcessingFilter.java line 182
in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
LogoutFilter.java line 105
in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextPersistenceFilter.java line 87
in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ConcurrentSessionFilter.java line 125
in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterChainProxy.java line 173
in org.springframework.security.web.FilterChainProxy doFilter()
DelegatingFilterProxy.java line 237
in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
DelegatingFilterProxy.java line 167
in org.springframework.web.filter.DelegatingFilterProxy doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
CharacterEncodingFilter.java line 88
in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
OncePerRequestFilter.java line 76
in org.springframework.web.filter.OncePerRequestFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
ServletHandler.java line 399
in org.mortbay.jetty.servlet.ServletHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
SessionHandler.java line 182
in org.mortbay.jetty.servlet.SessionHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
ContextHandler.java line 767
in org.mortbay.jetty.handler.ContextHandler handle()
WebAppContext.java line 450
in org.mortbay.jetty.webapp.WebAppContext handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
StatisticsHandler.java line 53
in org.mortbay.jetty.handler.StatisticsHandler handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
Server.java line 326
in org.mortbay.jetty.Server handle()
HttpConnection.java line 542
in org.mortbay.jetty.HttpConnection handleRequest()
HttpConnection.java line 945
in org.mortbay.jetty.HttpConnection$RequestHandler content()
HttpParser.java line 756
in org.mortbay.jetty.HttpParser parseNext()
HttpParser.java line 218
in org.mortbay.jetty.HttpParser parseAvailable()
HttpConnection.java line 404
in org.mortbay.jetty.HttpConnection handle()
SelectChannelEndPoint.java line 410
in org.mortbay.io.nio.SelectChannelEndPoint run()
QueuedThreadPool.java line 582
in org.mortbay.thread.QueuedThreadPool$PoolThread run()`

after zeppelin server restart new add interpreters lost.

should test conf is exists first in scripts/control.py

 if run_command("hdfs dfs -test -e %s/interpreter.json" % (os.environ["ZEPPELIN_CONF_FS_DIR"])):
           run_command("hdfs dfs -put -f %s %s" % (completed_filename, os.environ["ZEPPELIN_CONF_FS_DIR"]))

Pyspark Interpreter died in spark2