Feeding huge data (4G) through serving layer gives crash of speed layer: <div clas

After fixing memory issues, i get these crashes in speed layer: <div class="snippe

Ok, I've added more total memory: <a target="_blank" rel="noopener n

KryoException: Buffer overflow. Available: 1, required: 4,about oryxproject/oryx

Comments (19)

stiv-yakovenko commented on July 30, 2024

If you wish I can generate CSV file for this data.

from oryx.

srowen commented on July 30, 2024

Sep 03 09:50:00 htm-psycho-401.zxz.su bash[31144]: Caused by: org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 1, required: 4. To avoid this, increase spark.kryoserializer.buffer.max value.

Is anything on your cluster setting spark.kryoserializer.buffer.max to something? even if that's in MB (a historical default in Spark), that seems small. The default should be 64MB and it's safe to set the max up to about 2047m. You can try setting this value, at least, to make sure.

from oryx.

stiv-yakovenko commented on July 30, 2024

Can you please give a hint how to set this value correctly? As param to spark-submit?

from oryx.

srowen commented on July 30, 2024

If I haven't forgotten the HOCON syntax, it should be sufficient to add something like this to your app config:

batch = {
  ...
  streaming = {
     ...
     config = {
       spark.kryoserializer.buffer.max = "2047m"
     }
  }
}

streaming = {
   ... (same config) ...
}

This will cause it to be sent by spark-submit. You should also look at the logs, because they'll print the exact command and config used. This will help verify that the right params are being set.

from oryx.

stiv-yakovenko commented on July 30, 2024

After fixing memory issues, i get these crashes in speed layer:

Sep 06 11:45:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:00 INFO  BlockManager:54 - Removing RDD 9
Sep 06 11:45:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:00 INFO  KafkaRDD:54 - Removing RDD 8 from persistence list
Sep 06 11:45:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:00 INFO  BlockManager:54 - Removing RDD 8
Sep 06 11:45:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:00 INFO  ReceivedBlockTracker:54 - Deleting batches:
Sep 06 11:45:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:00 INFO  InputInfoTracker:54 - remove old batch metadata: 1536222900000 ms
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:38 INFO  YarnSchedulerBackend$YarnDriverEndpoint:54 - Disabling executor 1.
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:38 INFO  DAGScheduler:54 - Executor lost: 1 (epoch 0)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:38 INFO  BlockManagerMasterEndpoint:54 - Trying to remove executor 1 from BlockManagerMaster.
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:38 ERROR TransportClient:233 - Failed to send RPC 7240739849994500636 to /192.168.147.4:53566: java.nio.channels.ClosedChannelException
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: java.nio.channels.ClosedChannelException
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:38 INFO  BlockManagerMasterEndpoint:54 - Removing block manager BlockManagerId(1, htm-psycho-401.zxz.su, 36859, None)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:38 INFO  BlockManagerMaster:54 - Removed 1 successfully in removeExecutor
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:38 WARN  YarnSchedulerBackend$YarnSchedulerEndpoint:87 - Attempted to get executor loss reason for executor id 1 at RPC address 192.168.147.4:53574, but got no response. Marking as slave lost.
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: java.io.IOException: Failed to send RPC 7240739849994500636 to /192.168.147.4:53566: java.nio.channels.ClosedChannelException
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:987)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:869)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1316)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at java.lang.Thread.run(Thread.java:748)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: Caused by: java.nio.channels.ClosedChannelException
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:38 INFO  DAGScheduler:54 - Shuffle files lost for executor: 1 (epoch 0)
Sep 06 11:45:38 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:45:38 ERROR YarnScheduler:70 - Lost executor 1 on htm-psycho-401.zxz.su: Slave lost
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 ERROR Client:70 - Application application_1536006464455_0008 not found.
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 INFO  Client:54 - Deleted staging directory hdfs://0.0.0.0:9000/user/ec2-user/.sparkStaging/application_1536006464455_0008
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 ERROR YarnClientSchedulerBackend:70 - Yarn application has already exited with state KILLED!
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 INFO  AbstractConnector:318 - Stopped Spark@18324f97{HTTP/1.1,[http/1.1]}{0.0.0.0:4041}
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 INFO  SparkUI:54 - Stopped Spark web UI at http://htm-psycho-401.zxz.su:4041
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 ERROR TransportClient:233 - Failed to send RPC 6480775028423913532 to /192.168.147.4:53566: java.nio.channels.ClosedChannelException
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: java.nio.channels.ClosedChannelException
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint:91 - Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: java.io.IOException: Failed to send RPC 6480775028423913532 to /192.168.147.4:53566: java.nio.channels.ClosedChannelException
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:987)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:869)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1316)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at java.lang.Thread.run(Thread.java:748)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: Caused by: java.nio.channels.ClosedChannelException
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 INFO  SchedulerExtensionServices:54 - Stopping SchedulerExtensionServices
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: (serviceOption=None,
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: services=List(),
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: started=false)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 ERROR Utils:91 - Uncaught exception in thread Yarn application state monitor
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: org.apache.spark.SparkException: Exception thrown in awaitResult:
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:566)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:95)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:155)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:508)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1752)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1924)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.stop(SparkContext.scala:1923)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:112)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: Caused by: java.io.IOException: Failed to send RPC 6480775028423913532 to /192.168.147.4:53566: java.nio.channels.ClosedChannelException
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:987)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:869)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1316)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at java.lang.Thread.run(Thread.java:748)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: Caused by: java.nio.channels.ClosedChannelException
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 INFO  MemoryStore:54 - MemoryStore cleared
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 INFO  BlockManager:54 - BlockManager stopped
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
Sep 06 11:46:13 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:46:13 INFO  SparkContext:54 - Successfully stopped SparkContext
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  JobScheduler:54 - Added jobs for time 1536223800000 ms
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  JobScheduler:54 - Starting job streaming job 1536223800000 ms.0 from job set of time 1536223800000 ms
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  JobScheduler:54 - Finished job streaming job 1536223800000 ms.0 from job set of time 1536223800000 ms
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  UpdateOffsetsFn:60 - Updating offsets: {OryxInput,0=2477636, OryxInput,1=2478832, OryxInput,2=2477745, OryxInput,3=2478777}
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  ZooKeeper:438 - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@589f343b
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  ZkEventThread:65 - Starting ZkClient event thread.
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  JobScheduler:54 - Starting job streaming job 1536223800000 ms.1 from job set of time 1536223800000 ms
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  ZkClient:936 - Waiting for keeper state SyncConnected
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  ClientCnxn:975 - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 ERROR JobScheduler:91 - Error running job streaming job 1536223800000 ms.0
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: java.lang.IllegalStateException: SparkContext has been shutdown
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2019)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2067)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1358)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.take(RDD.scala:1331)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply$mcZ$sp(RDD.scala:1466)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1466)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1466)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.isEmpty(RDD.scala:1465)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.api.java.JavaRDDLike$class.isEmpty(JavaRDDLike.scala:544)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.api.java.AbstractJavaRDDLike.isEmpty(JavaRDDLike.scala:45)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at com.cloudera.oryx.lambda.speed.SpeedLayerUpdate.call(SpeedLayerUpdate.java:53)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at com.cloudera.oryx.lambda.speed.SpeedLayerUpdate.call(SpeedLayerUpdate.java:37)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at scala.util.Try$.apply(Try.scala:192)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at java.lang.Thread.run(Thread.java:748)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  ClientCnxn:852 - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  SpeedLayer:142 - Shutting down model manager
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  SpeedLayer:147 - Shutting down consumer
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  SpeedLayer:152 - Shutting down Spark Streaming; this may take some time
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  ClientCnxn:1235 - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x10160d98a790153, negotiated timeout = 30000
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  ReceiverTracker:54 - ReceiverTracker stopped
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  ZkClient:713 - zookeeper state changed (SyncConnected)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  JobGenerator:54 - Stopping JobGenerator gracefully
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  JobGenerator:54 - Waiting for all received blocks to be consumed for job generation
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  JobGenerator:54 - Waited for all received blocks to be consumed for job generation
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  ZkEventThread:83 - Terminate ZkClient event thread.
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  ZooKeeper:684 - Session: 0x10160d98a790153 closed
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  ClientCnxn:512 - EventThread shut down
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  JobScheduler:54 - Finished job streaming job 1536223800000 ms.1 from job set of time 1536223800000 ms
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  JobScheduler:54 - Total delay: 0.026 s for time 1536223800000 ms (execution: 0.021 s)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 INFO  MapPartitionsRDD:54 - Removing RDD 11 from persistence list
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:50:00 ERROR JobScheduler:91 - Error in job generator
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: java.lang.NullPointerException
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.storage.BlockManagerMaster.removeRdd(BlockManagerMaster.scala:126)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1790)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.unpersist(RDD.scala:217)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$3.apply(DStream.scala:458)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$3.apply(DStream.scala:457)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:108)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:108)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:108)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream.clearMetadata(DStream.scala:457)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$5.apply(DStream.scala:470)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$5.apply(DStream.scala:470)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.immutable.List.foreach(List.scala:381)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream.clearMetadata(DStream.scala:470)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.DStreamGraph$$anonfun$clearMetadata$2.apply(DStreamGraph.scala:134)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.DStreamGraph$$anonfun$clearMetadata$2.apply(DStreamGraph.scala:134)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.DStreamGraph.clearMetadata(DStreamGraph.scala:134)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobGenerator.clearMetadata(JobGenerator.scala:263)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:184)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:89)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:88)
Sep 06 11:50:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  JobScheduler:54 - Added jobs for time 1536224100000 ms
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  JobScheduler:54 - Starting job streaming job 1536224100000 ms.0 from job set of time 1536224100000 ms
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  JobScheduler:54 - Finished job streaming job 1536224100000 ms.0 from job set of time 1536224100000 ms
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  UpdateOffsetsFn:60 - Updating offsets: {OryxInput,0=2477636, OryxInput,1=2478832, OryxInput,2=2477745, OryxInput,3=2478777}
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  JobScheduler:54 - Starting job streaming job 1536224100000 ms.1 from job set of time 1536224100000 ms
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 ERROR JobScheduler:91 - Error running job streaming job 1536224100000 ms.0
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: java.lang.IllegalStateException: SparkContext has been shutdown
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2019)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2067)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1358)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.take(RDD.scala:1331)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply$mcZ$sp(RDD.scala:1466)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1466)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1466)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.isEmpty(RDD.scala:1465)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.api.java.JavaRDDLike$class.isEmpty(JavaRDDLike.scala:544)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.api.java.AbstractJavaRDDLike.isEmpty(JavaRDDLike.scala:45)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at com.cloudera.oryx.lambda.speed.SpeedLayerUpdate.call(SpeedLayerUpdate.java:53)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at com.cloudera.oryx.lambda.speed.SpeedLayerUpdate.call(SpeedLayerUpdate.java:37)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at scala.util.Try$.apply(Try.scala:192)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at java.lang.Thread.run(Thread.java:748)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  ZkEventThread:65 - Starting ZkClient event thread.
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  ZooKeeper:438 - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@496005e2
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  ZkClient:936 - Waiting for keeper state SyncConnected
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  ClientCnxn:975 - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  ClientCnxn:852 - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  ClientCnxn:1235 - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x10160d98a790156, negotiated timeout = 30000
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  ZkClient:713 - zookeeper state changed (SyncConnected)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  ZkEventThread:83 - Terminate ZkClient event thread.
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  ZooKeeper:684 - Session: 0x10160d98a790156 closed
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  ClientCnxn:512 - EventThread shut down
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  JobScheduler:54 - Finished job streaming job 1536224100000 ms.1 from job set of time 1536224100000 ms
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  JobScheduler:54 - Total delay: 0.052 s for time 1536224100000 ms (execution: 0.044 s)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 INFO  MapPartitionsRDD:54 - Removing RDD 13 from persistence list
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 11:55:00 ERROR JobScheduler:91 - Error in job generator
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: java.lang.NullPointerException
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.storage.BlockManagerMaster.removeRdd(BlockManagerMaster.scala:126)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1790)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.unpersist(RDD.scala:217)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$3.apply(DStream.scala:458)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$3.apply(DStream.scala:457)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:108)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:108)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:108)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream.clearMetadata(DStream.scala:457)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$5.apply(DStream.scala:470)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$5.apply(DStream.scala:470)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.immutable.List.foreach(List.scala:381)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream.clearMetadata(DStream.scala:470)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.DStreamGraph$$anonfun$clearMetadata$2.apply(DStreamGraph.scala:134)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.DStreamGraph$$anonfun$clearMetadata$2.apply(DStreamGraph.scala:134)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.DStreamGraph.clearMetadata(DStreamGraph.scala:134)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobGenerator.clearMetadata(JobGenerator.scala:263)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:184)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:89)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:88)
Sep 06 11:55:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  RecurringTimer:54 - Stopped timer for JobGenerator after time 1536224400000
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  JobScheduler:54 - Added jobs for time 1536224400000 ms
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  JobScheduler:54 - Starting job streaming job 1536224400000 ms.0 from job set of time 1536224400000 ms
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  JobGenerator:54 - Stopped generation timer
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  JobScheduler:54 - Finished job streaming job 1536224400000 ms.0 from job set of time 1536224400000 ms
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  UpdateOffsetsFn:60 - Updating offsets: {OryxInput,0=2477636, OryxInput,1=2478832, OryxInput,2=2477745, OryxInput,3=2478777}
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  JobGenerator:54 - Waiting for jobs to be processed and checkpoints to be written
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  JobScheduler:54 - Starting job streaming job 1536224400000 ms.1 from job set of time 1536224400000 ms
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 ERROR JobScheduler:91 - Error running job streaming job 1536224400000 ms.0
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: java.lang.IllegalStateException: SparkContext has been shutdown
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2019)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2067)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1358)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.take(RDD.scala:1331)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply$mcZ$sp(RDD.scala:1466)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1466)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1466)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.isEmpty(RDD.scala:1465)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.api.java.JavaRDDLike$class.isEmpty(JavaRDDLike.scala:544)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.api.java.AbstractJavaRDDLike.isEmpty(JavaRDDLike.scala:45)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at com.cloudera.oryx.lambda.speed.SpeedLayerUpdate.call(SpeedLayerUpdate.java:53)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at com.cloudera.oryx.lambda.speed.SpeedLayerUpdate.call(SpeedLayerUpdate.java:37)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at scala.util.Try$.apply(Try.scala:192)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at java.lang.Thread.run(Thread.java:748)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ZkEventThread:65 - Starting ZkClient event thread.
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 WARN  JobGenerator:66 - Timed out while stopping the job generator (timeout = 300000)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ZooKeeper:438 - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@6e82d48f
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  JobGenerator:54 - Waited for jobs to be processed and checkpoints to be written
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ZkClient:936 - Waiting for keeper state SyncConnected
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  JobGenerator:54 - Stopped JobGenerator
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ClientCnxn:975 - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ClientCnxn:852 - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ClientCnxn:1235 - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x10160d98a79015b, negotiated timeout = 30000
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ZkClient:713 - zookeeper state changed (SyncConnected)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ZkEventThread:83 - Terminate ZkClient event thread.
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ClientCnxn:512 - EventThread shut down
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ZooKeeper:684 - Session: 0x10160d98a79015b closed
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  JobScheduler:54 - Finished job streaming job 1536224400000 ms.1 from job set of time 1536224400000 ms
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  JobScheduler:54 - Total delay: 0.022 s for time 1536224400000 ms (execution: 0.017 s)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  JobScheduler:54 - Stopped JobScheduler
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ContextHandler:910 - Stopped o.s.j.s.ServletContextHandler@5a06904{/streaming,null,UNAVAILABLE,@Spark}
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ContextHandler:910 - Stopped o.s.j.s.ServletContextHandler@286866cb{/streaming/batch,null,UNAVAILABLE,@Spark}
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ContextHandler:910 - Stopped o.s.j.s.ServletContextHandler@14b95942{/static/streaming,null,UNAVAILABLE,@Spark}
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  StreamingContext:54 - StreamingContext stopped successfully
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  SparkContext:54 - SparkContext already stopped.
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: Exception in thread "main" java.lang.IllegalStateException: SparkContext has been shutdown
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2019)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.SparkContext.runJob(SparkContext.scala:2067)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1358)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.take(RDD.scala:1331)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply$mcZ$sp(RDD.scala:1466)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1466)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1466)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.rdd.RDD.isEmpty(RDD.scala:1465)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.api.java.JavaRDDLike$class.isEmpty(JavaRDDLike.scala:544)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.api.java.AbstractJavaRDDLike.isEmpty(JavaRDDLike.scala:45)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at com.cloudera.oryx.lambda.speed.SpeedLayerUpdate.call(SpeedLayerUpdate.java:53)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at com.cloudera.oryx.lambda.speed.SpeedLayerUpdate.call(SpeedLayerUpdate.java:37)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at scala.util.Try$.apply(Try.scala:192)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: at java.lang.Thread.run(Thread.java:748)
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ShutdownHookManager:54 - Shutdown hook called
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-16850b34-eb08-469f-915f-1e97f2942c29
Sep 06 12:00:00 htm-psycho-401.zxz.su bash[29328]: 2018-09-06 12:00:00 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-e33ba689-595b-4e3e-be27-8df08053b020
Sep 06 12:00:00 htm-psycho-401.zxz.su systemd[1]: oryx-speed.service: main process exited, code=exited, status=1/FAILURE
Sep 06 12:00:00 htm-psycho-401.zxz.su systemd[1]: Unit oryx-speed.service entered failed state.
Sep 06 12:00:00 htm-psycho-401.zxz.su systemd[1]: oryx-speed.service failed.

from oryx.

srowen commented on July 30, 2024

That kind of error indicates a problem in the Spark cluster. It suddenly failed to communicate with executors. I would look for other errors, on the executor side and in their logs. Do other Spark jobs work at all?

from oryx.

stiv-yakovenko commented on July 30, 2024

I have restarted all the services and all of them seem to be running correctly:

All the data from the model is gone, I hadn't deleted any files from HDFS:

Now I try to push data into oryx:

And I get this error message:

Sep 07 00:41:00 htm-psycho-401.zxz.su bash[7200]: 2018-09-07 00:41:00 WARN YarnScheduler:66 - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

System was crashing under high load, so I've added more memory to kryo and total memory.

from oryx.

srowen commented on July 30, 2024

I see you have 8 cores and 12GB of RAM. This is pretty small unless you also reduce the resource requirements in your config. YARN is telling you it can't give you the resource you are asking for.

from oryx.

stiv-yakovenko commented on July 30, 2024

Ok, I've added more total memory:

I dont see any problem in batch logs:

Sep 07 01:10:15 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:15 INFO  TaskSetManager:54 - Starting task 28.0 in stage 25.0 (TID 361, htm-psycho-401.zxz.su, executor 1, partition 28, NODE_LOCAL, 7770 bytes)
Sep 07 01:10:15 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:15 INFO  TaskSetManager:54 - Finished task 27.0 in stage 25.0 (TID 360) in 1064 ms on htm-psycho-401.zxz.su (executor 1) (28/32)
Sep 07 01:10:15 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:15 INFO  BlockManagerInfo:54 - Removed taskresult_360 on htm-psycho-401.zxz.su:33540 in memory (size: 8.0 MB, free: 322.2 MB)
Sep 07 01:10:16 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:16 INFO  BlockManagerInfo:54 - Added taskresult_361 in memory on htm-psycho-401.zxz.su:33540 (size: 7.0 MB, free: 315.1 MB)
Sep 07 01:10:16 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:16 INFO  TaskSetManager:54 - Starting task 29.0 in stage 25.0 (TID 362, htm-psycho-401.zxz.su, executor 1, partition 29, NODE_LOCAL, 7770 bytes)
Sep 07 01:10:18 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:18 INFO  BlockManagerInfo:54 - Added taskresult_362 in memory on htm-psycho-401.zxz.su:33540 (size: 6.5 MB, free: 308.6 MB)
Sep 07 01:10:18 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:18 INFO  TaskSetManager:54 - Marking task 29 in stage 25.0 (on htm-psycho-401.zxz.su) as speculatable because it ran more than 1516 ms
Sep 07 01:10:18 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:18 INFO  TaskSetManager:54 - Starting task 30.0 in stage 25.0 (TID 363, htm-psycho-401.zxz.su, executor 1, partition 30, NODE_LOCAL, 7770 bytes)
Sep 07 01:10:18 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:18 INFO  TaskSetManager:54 - Finished task 28.0 in stage 25.0 (TID 361) in 3200 ms on htm-psycho-401.zxz.su (executor 1) (29/32)
Sep 07 01:10:18 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:18 INFO  BlockManagerInfo:54 - Removed taskresult_361 on htm-psycho-401.zxz.su:33540 in memory (size: 7.0 MB, free: 315.7 MB)
Sep 07 01:10:18 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:18 INFO  TaskSetManager:54 - Finished task 29.0 in stage 25.0 (TID 362) in 2392 ms on htm-psycho-401.zxz.su (executor 1) (30/32)
Sep 07 01:10:18 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:18 INFO  BlockManagerInfo:54 - Removed taskresult_362 on htm-psycho-401.zxz.su:33540 in memory (size: 6.5 MB, free: 322.2 MB)
Sep 07 01:10:19 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:19 INFO  BlockManagerInfo:54 - Added taskresult_363 in memory on htm-psycho-401.zxz.su:33540 (size: 7.0 MB, free: 315.2 MB)
Sep 07 01:10:19 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:19 INFO  TaskSetManager:54 - Starting task 31.0 in stage 25.0 (TID 364, htm-psycho-401.zxz.su, executor 1, partition 31, NODE_LOCAL, 7770 bytes)
Sep 07 01:10:19 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:19 INFO  TaskSetManager:54 - Finished task 30.0 in stage 25.0 (TID 363) in 892 ms on htm-psycho-401.zxz.su (executor 1) (31/32)
Sep 07 01:10:19 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:19 INFO  BlockManagerInfo:54 - Removed taskresult_363 on htm-psycho-401.zxz.su:33540 in memory (size: 7.0 MB, free: 322.2 MB)
Sep 07 01:10:20 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:20 INFO  BlockManagerInfo:54 - Added taskresult_364 in memory on htm-psycho-401.zxz.su:33540 (size: 8.9 MB, free: 313.2 MB)
Sep 07 01:10:20 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:20 INFO  TaskSetManager:54 - Finished task 31.0 in stage 25.0 (TID 364) in 1283 ms on htm-psycho-401.zxz.su (executor 1) (32/32)
Sep 07 01:10:20 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:20 INFO  YarnScheduler:54 - Removed TaskSet 25.0, whose tasks have all completed, from pool
Sep 07 01:10:20 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:20 INFO  DAGScheduler:54 - ResultStage 25 (collectAsMap at ALSUpdate.java:187) finished in 32.455 s
Sep 07 01:10:20 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:20 INFO  DAGScheduler:54 - Job 15 finished: collectAsMap at ALSUpdate.java:187, took 32.461504 s
Sep 07 01:10:20 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:20 INFO  BlockManagerInfo:54 - Removed taskresult_364 on htm-psycho-401.zxz.su:33540 in memory (size: 8.9 MB, free: 322.2 MB)
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 489
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 493
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 497
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 490
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 480
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 496
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 486
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 477
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 492
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 481
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  BlockManagerInfo:54 - Removed broadcast_20_piece0 on htm-psycho-401.zxz.su:39017 in memory (size: 4.1 KB, free: 912.3 MB)
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  BlockManagerInfo:54 - Removed broadcast_20_piece0 on htm-psycho-401.zxz.su:33540 in memory (size: 4.1 KB, free: 322.2 MB)
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 482
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 479
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 476
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 498
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 487
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 494
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 491
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 485
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 495
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 483
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 478
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 499
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 475
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 484
Sep 07 01:10:26 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:26 INFO  ContextCleaner:54 - Cleaned accumulator 488
Sep 07 01:10:42 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:42 INFO  ContextCleaner:54 - Cleaned shuffle 2
Sep 07 01:10:42 htm-psycho-401.zxz.su bash[9560]: 2018-09-07 01:10:42 INFO  ContextCleaner:54 - Cleaned shuffle 3

speed layer also has no crashes:

Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  ZkEventThread:65 - Starting ZkClient event thread.
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  ZkClient:936 - Waiting for keeper state SyncConnected
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  ClientCnxn:975 - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  ClientCnxn:852 - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  ClientCnxn:1235 - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x10160d98a7901ef, negotiated timeout = 30000
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  ZkClient:713 - zookeeper state changed (SyncConnected)
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  ZkEventThread:83 - Terminate ZkClient event thread.
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  ClientCnxn:512 - EventThread shut down
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  ZooKeeper:684 - Session: 0x10160d98a7901ef closed
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  JobScheduler:54 - Finished job streaming job 1536271800000 ms.1 from job set of time 1536271800000 ms
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  MapPartitionsRDD:54 - Removing RDD 3 from persistence list
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  JobScheduler:54 - Total delay: 0.196 s for time 1536271800000 ms (execution: 0.188 s)
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  KafkaRDD:54 - Removing RDD 2 from persistence list
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  BlockManager:54 - Removing RDD 3
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  ReceivedBlockTracker:54 - Deleting batches:
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  InputInfoTracker:54 - remove old batch metadata: 1536271200000 ms
Sep 07 01:10:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:10:00 INFO  BlockManager:54 - Removing RDD 2

if I feed 1.9M data.csv again, then I get this crash:

Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at scala.collection.generic.GenTraversableFactory$GenericCanBuildFrom.apply(GenTr
aversableFactory.scala:57)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at scala.collection.generic.GenTraversableFactory$GenericCanBuildFrom.apply(GenTr
aversableFactory.scala:52)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at scala.collection.TraversableLike$class.builder$1(TraversableLike.scala:229)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at scala.collection.AbstractTraversable.map(Traversable.scala:104)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.
scala:348)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoin
t.org$apache$spark$scheduler$cluster$CoarseGrainedSchedulerBackend$DriverEndpoint$$makeOffers(CoarseGrainedSchedulerBackend.scala:2
47)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoin
t$$anonfun$receive$1.applyOrElse(CoarseGrainedSchedulerBackend.scala:136)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:1
17)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:221)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149
)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624
)
Sep 07 01:14:16 htm-psycho-401.zxz.su bash[9560]: at java.lang.Thread.run(Thread.java:748)
Sep 07 01:15:17 htm-psycho-401.zxz.su bash[9560]: Exception in thread "streaming-job-executor-0" java.lang.OutOfMemoryError: GC ove
rhead limit exceeded

how much memory does it need? Or may be this is some older data somewhere in queue caused to crash? Why does it crash?

from oryx.

srowen commented on July 30, 2024

That says your driver is out of memory. It really depends on how big your workloads are, but I'd generally give drivers at least 1GB and executors at least 2GB.

from oryx.

stiv-yakovenko commented on July 30, 2024

I've added more memory:

batch {
   streaming {
     generation-interval-sec = 300
     num-executors = 1
     executor-cores = 1
     executor-memory = "4g"
     driver-memory = "3g"
     config = {
            spark.kryoserializer.buffer.max = "2047m"
     }
   }
}

It doesn't seem to crash, but I still get result 503 after uploading csv file.
batch:

Sep 07 01:45:34 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:34 INFO  TaskSetManager:54 - Finished task 10.0 in stage 21.0 (TID 335) in 3977 ms on htm-psycho-401.zxz.su (executor 1) (11/32)
Sep 07 01:45:34 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:34 INFO  BlockManagerInfo:54 - Removed taskresult_335 on htm-psycho-401.zxz.su:41761 in memory (size: 8.0 MB, free: 175.4 MB)
Sep 07 01:45:34 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:34 INFO  TaskSetManager:54 - Finished task 11.0 in stage 21.0 (TID 336) in 3062 ms on htm-psycho-401.zxz.su (executor 1) (12/32)
Sep 07 01:45:34 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:34 INFO  BlockManagerInfo:54 - Removed taskresult_336 on htm-psycho-401.zxz.su:41761 in memory (size: 6.3 MB, free: 181.7 MB)
Sep 07 01:45:35 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:35 INFO  BlockManagerInfo:54 - Added taskresult_337 in memory on htm-psycho-401.zxz.su:41761 (size: 7.4 MB, free: 174.3 MB)
Sep 07 01:45:35 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:35 INFO  TaskSetManager:54 - Starting task 13.0 in stage 21.0 (TID 338, htm-psycho-401.zxz.su, executor 1, partition 13, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:35 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:35 INFO  TaskSetManager:54 - Finished task 12.0 in stage 21.0 (TID 337) in 966 ms on htm-psycho-401.zxz.su (executor 1) (13/32)
Sep 07 01:45:35 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:35 INFO  BlockManagerInfo:54 - Removed taskresult_337 on htm-psycho-401.zxz.su:41761 in memory (size: 7.4 MB, free: 181.7 MB)
Sep 07 01:45:36 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:36 INFO  BlockManagerInfo:54 - Added taskresult_338 in memory on htm-psycho-401.zxz.su:41761 (size: 7.7 MB, free: 174.0 MB)
Sep 07 01:45:36 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:36 INFO  TaskSetManager:54 - Starting task 14.0 in stage 21.0 (TID 339, htm-psycho-401.zxz.su, executor 1, partition 14, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:36 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:36 INFO  TaskSetManager:54 - Finished task 13.0 in stage 21.0 (TID 338) in 1059 ms on htm-psycho-401.zxz.su (executor 1) (14/32)
Sep 07 01:45:36 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:36 INFO  BlockManagerInfo:54 - Removed taskresult_338 on htm-psycho-401.zxz.su:41761 in memory (size: 7.7 MB, free: 181.7 MB)
Sep 07 01:45:36 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:36 INFO  BlockManagerInfo:54 - Added taskresult_339 in memory on htm-psycho-401.zxz.su:41761 (size: 6.3 MB, free: 175.4 MB)
Sep 07 01:45:36 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:36 INFO  TaskSetManager:54 - Starting task 15.0 in stage 21.0 (TID 340, htm-psycho-401.zxz.su, executor 1, partition 15, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:36 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:36 INFO  TaskSetManager:54 - Finished task 14.0 in stage 21.0 (TID 339) in 823 ms on htm-psycho-401.zxz.su (executor 1) (15/32)
Sep 07 01:45:36 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:36 INFO  BlockManagerInfo:54 - Removed taskresult_339 on htm-psycho-401.zxz.su:41761 in memory (size: 6.3 MB, free: 181.7 MB)
Sep 07 01:45:37 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:37 INFO  BlockManagerInfo:54 - Added taskresult_340 in memory on htm-psycho-401.zxz.su:41761 (size: 8.4 MB, free: 173.2 MB)
Sep 07 01:45:37 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:37 INFO  TaskSetManager:54 - Starting task 16.0 in stage 21.0 (TID 341, htm-psycho-401.zxz.su, executor 1, partition 16, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:38 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:38 INFO  TaskSetManager:54 - Finished task 15.0 in stage 21.0 (TID 340) in 1194 ms on htm-psycho-401.zxz.su (executor 1) (16/32)
Sep 07 01:45:38 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:38 INFO  BlockManagerInfo:54 - Removed taskresult_340 on htm-psycho-401.zxz.su:41761 in memory (size: 8.4 MB, free: 181.7 MB)
Sep 07 01:45:38 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:38 INFO  BlockManagerInfo:54 - Added taskresult_341 in memory on htm-psycho-401.zxz.su:41761 (size: 7.3 MB, free: 174.4 MB)
Sep 07 01:45:38 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:38 INFO  TaskSetManager:54 - Starting task 17.0 in stage 21.0 (TID 342, htm-psycho-401.zxz.su, executor 1, partition 17, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:38 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:38 INFO  TaskSetManager:54 - Finished task 16.0 in stage 21.0 (TID 341) in 951 ms on htm-psycho-401.zxz.su (executor 1) (17/32)
Sep 07 01:45:38 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:38 INFO  BlockManagerInfo:54 - Removed taskresult_341 on htm-psycho-401.zxz.su:41761 in memory (size: 7.3 MB, free: 181.7 MB)
Sep 07 01:45:39 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:39 INFO  BlockManagerInfo:54 - Added taskresult_342 in memory on htm-psycho-401.zxz.su:41761 (size: 6.3 MB, free: 175.4 MB)
Sep 07 01:45:39 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:39 INFO  TaskSetManager:54 - Starting task 18.0 in stage 21.0 (TID 343, htm-psycho-401.zxz.su, executor 1, partition 18, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:39 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:39 INFO  TaskSetManager:54 - Finished task 17.0 in stage 21.0 (TID 342) in 947 ms on htm-psycho-401.zxz.su (executor 1) (18/32)
Sep 07 01:45:39 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:39 INFO  BlockManagerInfo:54 - Removed taskresult_342 on htm-psycho-401.zxz.su:41761 in memory (size: 6.3 MB, free: 181.7 MB)
Sep 07 01:45:40 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:40 INFO  BlockManagerInfo:54 - Added taskresult_343 in memory on htm-psycho-401.zxz.su:41761 (size: 8.0 MB, free: 173.7 MB)
Sep 07 01:45:40 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:40 INFO  TaskSetManager:54 - Starting task 19.0 in stage 21.0 (TID 344, htm-psycho-401.zxz.su, executor 1, partition 19, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:40 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:40 INFO  TaskSetManager:54 - Finished task 18.0 in stage 21.0 (TID 343) in 1131 ms on htm-psycho-401.zxz.su (executor 1) (19/32)
Sep 07 01:45:40 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:40 INFO  BlockManagerInfo:54 - Removed taskresult_343 on htm-psycho-401.zxz.su:41761 in memory (size: 8.0 MB, free: 181.7 MB)
Sep 07 01:45:41 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:41 INFO  BlockManagerInfo:54 - Added taskresult_344 in memory on htm-psycho-401.zxz.su:41761 (size: 7.7 MB, free: 174.0 MB)
Sep 07 01:45:41 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:41 INFO  TaskSetManager:54 - Starting task 20.0 in stage 21.0 (TID 345, htm-psycho-401.zxz.su, executor 1, partition 20, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:41 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:41 INFO  TaskSetManager:54 - Finished task 19.0 in stage 21.0 (TID 344) in 1006 ms on htm-psycho-401.zxz.su (executor 1) (20/32)
Sep 07 01:45:41 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:41 INFO  BlockManagerInfo:54 - Removed taskresult_344 on htm-psycho-401.zxz.su:41761 in memory (size: 7.7 MB, free: 181.7 MB)
Sep 07 01:45:42 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:42 INFO  BlockManagerInfo:54 - Added taskresult_345 in memory on htm-psycho-401.zxz.su:41761 (size: 5.5 MB, free: 176.2 MB)
Sep 07 01:45:42 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:42 INFO  TaskSetManager:54 - Starting task 21.0 in stage 21.0 (TID 346, htm-psycho-401.zxz.su, executor 1, partition 21, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:42 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:42 INFO  TaskSetManager:54 - Finished task 20.0 in stage 21.0 (TID 345) in 752 ms on htm-psycho-401.zxz.su (executor 1) (21/32)
Sep 07 01:45:42 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:42 INFO  BlockManagerInfo:54 - Removed taskresult_345 on htm-psycho-401.zxz.su:41761 in memory (size: 5.5 MB, free: 181.7 MB)
Sep 07 01:45:43 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:43 INFO  BlockManagerInfo:54 - Added taskresult_346 in memory on htm-psycho-401.zxz.su:41761 (size: 9.0 MB, free: 172.7 MB)
Sep 07 01:45:43 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:43 INFO  TaskSetManager:54 - Starting task 22.0 in stage 21.0 (TID 347, htm-psycho-401.zxz.su, executor 1, partition 22, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:43 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:43 INFO  TaskSetManager:54 - Finished task 21.0 in stage 21.0 (TID 346) in 1273 ms on htm-psycho-401.zxz.su (executor 1) (22/32)
Sep 07 01:45:43 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:43 INFO  BlockManagerInfo:54 - Removed taskresult_346 on htm-psycho-401.zxz.su:41761 in memory (size: 9.0 MB, free: 181.7 MB)
Sep 07 01:45:44 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:44 INFO  BlockManagerInfo:54 - Added taskresult_347 in memory on htm-psycho-401.zxz.su:41761 (size: 7.1 MB, free: 174.6 MB)
Sep 07 01:45:44 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:44 INFO  TaskSetManager:54 - Starting task 23.0 in stage 21.0 (TID 348, htm-psycho-401.zxz.su, executor 1, partition 23, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:44 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:44 INFO  TaskSetManager:54 - Finished task 22.0 in stage 21.0 (TID 347) in 927 ms on htm-psycho-401.zxz.su (executor 1) (23/32)
Sep 07 01:45:44 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:44 INFO  BlockManagerInfo:54 - Removed taskresult_347 on htm-psycho-401.zxz.su:41761 in memory (size: 7.1 MB, free: 181.7 MB)
Sep 07 01:45:45 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:45 INFO  BlockManagerInfo:54 - Added taskresult_348 in memory on htm-psycho-401.zxz.su:41761 (size: 8.2 MB, free: 173.4 MB)
Sep 07 01:45:45 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:45 INFO  TaskSetManager:54 - Starting task 24.0 in stage 21.0 (TID 349, htm-psycho-401.zxz.su, executor 1, partition 24, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:45 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:45 INFO  TaskSetManager:54 - Finished task 23.0 in stage 21.0 (TID 348) in 1615 ms on htm-psycho-401.zxz.su (executor 1) (24/32)
Sep 07 01:45:45 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:45 INFO  BlockManagerInfo:54 - Removed taskresult_348 on htm-psycho-401.zxz.su:41761 in memory (size: 8.2 MB, free: 181.7 MB)
Sep 07 01:45:46 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:46 INFO  BlockManagerInfo:54 - Added taskresult_349 in memory on htm-psycho-401.zxz.su:41761 (size: 5.9 MB, free: 175.7 MB)
Sep 07 01:45:46 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:46 INFO  TaskSetManager:54 - Starting task 25.0 in stage 21.0 (TID 350, htm-psycho-401.zxz.su, executor 1, partition 25, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:46 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:46 INFO  TaskSetManager:54 - Finished task 24.0 in stage 21.0 (TID 349) in 943 ms on htm-psycho-401.zxz.su (executor 1) (25/32)
Sep 07 01:45:46 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:46 INFO  BlockManagerInfo:54 - Removed taskresult_349 on htm-psycho-401.zxz.su:41761 in memory (size: 5.9 MB, free: 181.7 MB)
Sep 07 01:45:46 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:46 INFO  BlockManagerInfo:54 - Added taskresult_350 in memory on htm-psycho-401.zxz.su:41761 (size: 6.7 MB, free: 175.0 MB)
Sep 07 01:45:46 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:46 INFO  TaskSetManager:54 - Starting task 26.0 in stage 21.0 (TID 351, htm-psycho-401.zxz.su, executor 1, partition 26, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:47 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:47 INFO  TaskSetManager:54 - Finished task 25.0 in stage 21.0 (TID 350) in 864 ms on htm-psycho-401.zxz.su (executor 1) (26/32)
Sep 07 01:45:47 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:47 INFO  BlockManagerInfo:54 - Removed taskresult_350 on htm-psycho-401.zxz.su:41761 in memory (size: 6.7 MB, free: 181.7 MB)
Sep 07 01:45:48 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:48 INFO  BlockManagerInfo:54 - Added taskresult_351 in memory on htm-psycho-401.zxz.su:41761 (size: 8.4 MB, free: 173.3 MB)
Sep 07 01:45:48 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:48 INFO  TaskSetManager:54 - Starting task 27.0 in stage 21.0 (TID 352, htm-psycho-401.zxz.su, executor 1, partition 27, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:48 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:48 INFO  TaskSetManager:54 - Finished task 26.0 in stage 21.0 (TID 351) in 1196 ms on htm-psycho-401.zxz.su (executor 1) (27/32)
Sep 07 01:45:48 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:48 INFO  BlockManagerInfo:54 - Removed taskresult_351 on htm-psycho-401.zxz.su:41761 in memory (size: 8.4 MB, free: 181.7 MB)
Sep 07 01:45:48 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:48 INFO  BlockManagerInfo:54 - Added taskresult_352 in memory on htm-psycho-401.zxz.su:41761 (size: 7.6 MB, free: 174.0 MB)
Sep 07 01:45:48 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:48 INFO  TaskSetManager:54 - Starting task 28.0 in stage 21.0 (TID 353, htm-psycho-401.zxz.su, executor 1, partition 28, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:49 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:49 INFO  TaskSetManager:54 - Finished task 27.0 in stage 21.0 (TID 352) in 966 ms on htm-psycho-401.zxz.su (executor 1) (28/32)
Sep 07 01:45:49 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:49 INFO  BlockManagerInfo:54 - Removed taskresult_352 on htm-psycho-401.zxz.su:41761 in memory (size: 7.6 MB, free: 181.7 MB)
Sep 07 01:45:49 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:49 INFO  BlockManagerInfo:54 - Added taskresult_353 in memory on htm-psycho-401.zxz.su:41761 (size: 8.0 MB, free: 173.7 MB)
Sep 07 01:45:49 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:49 INFO  TaskSetManager:54 - Starting task 29.0 in stage 21.0 (TID 354, htm-psycho-401.zxz.su, executor 1, partition 29, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:50 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:50 INFO  TaskSetManager:54 - Finished task 28.0 in stage 21.0 (TID 353) in 1042 ms on htm-psycho-401.zxz.su (executor 1) (29/32)
Sep 07 01:45:50 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:50 INFO  BlockManagerInfo:54 - Removed taskresult_353 on htm-psycho-401.zxz.su:41761 in memory (size: 8.0 MB, free: 181.7 MB)
Sep 07 01:45:51 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:51 INFO  BlockManagerInfo:54 - Added taskresult_354 in memory on htm-psycho-401.zxz.su:41761 (size: 9.2 MB, free: 172.5 MB)
Sep 07 01:45:51 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:51 INFO  TaskSetManager:54 - Starting task 30.0 in stage 21.0 (TID 355, htm-psycho-401.zxz.su, executor 1, partition 30, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:51 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:51 INFO  TaskSetManager:54 - Finished task 29.0 in stage 21.0 (TID 354) in 1305 ms on htm-psycho-401.zxz.su (executor 1) (30/32)
Sep 07 01:45:51 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:51 INFO  BlockManagerInfo:54 - Removed taskresult_354 on htm-psycho-401.zxz.su:41761 in memory (size: 9.2 MB, free: 181.7 MB)
Sep 07 01:45:51 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:51 INFO  BlockManagerInfo:54 - Added taskresult_355 in memory on htm-psycho-401.zxz.su:41761 (size: 6.0 MB, free: 175.7 MB)
Sep 07 01:45:51 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:51 INFO  TaskSetManager:54 - Starting task 31.0 in stage 21.0 (TID 356, htm-psycho-401.zxz.su, executor 1, partition 31, NODE_LOCAL, 7770 bytes)
Sep 07 01:45:54 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:54 INFO  TaskSetManager:54 - Marking task 31 in stage 21.0 (on htm-psycho-401.zxz.su) as speculatable because it ran more than 1479 ms
Sep 07 01:45:54 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:54 INFO  BlockManagerInfo:54 - Added taskresult_356 in memory on htm-psycho-401.zxz.su:41761 (size: 6.5 MB, free: 169.2 MB)
Sep 07 01:45:54 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:54 INFO  TaskSetManager:54 - Finished task 30.0 in stage 21.0 (TID 355) in 3564 ms on htm-psycho-401.zxz.su (executor 1) (31/32)
Sep 07 01:45:54 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:54 INFO  BlockManagerInfo:54 - Removed taskresult_355 on htm-psycho-401.zxz.su:41761 in memory (size: 6.0 MB, free: 175.2 MB)
Sep 07 01:45:54 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:54 INFO  TaskSetManager:54 - Finished task 31.0 in stage 21.0 (TID 356) in 2894 ms on htm-psycho-401.zxz.su (executor 1) (32/32)
Sep 07 01:45:54 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:54 INFO  YarnScheduler:54 - Removed TaskSet 21.0, whose tasks have all completed, from pool
Sep 07 01:45:54 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:54 INFO  DAGScheduler:54 - ResultStage 21 (collectAsMap at ALSUpdate.java:187) finished in 33.544 s
Sep 07 01:45:54 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:54 INFO  DAGScheduler:54 - Job 11 finished: collectAsMap at ALSUpdate.java:187, took 33.551173 s
Sep 07 01:45:54 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:45:54 INFO  BlockManagerInfo:54 - Removed taskresult_356 on htm-psycho-401.zxz.su:41761 in memory (size: 6.5 MB, free: 181.7 MB)
Sep 07 01:46:07 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:07 INFO  ALSUpdate:133 - Broadcasting ID-index mappings for 2789778 users, 5998833 items
Sep 07 01:46:07 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:07 INFO  MemoryStore:54 - Block broadcast_17 stored as values in memory (estimated size 393.0 MB, free 1065.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece0 stored as bytes in memory (estimated size 4.0 MB, free 1061.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece0 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1454.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece1 stored as bytes in memory (estimated size 4.0 MB, free 1057.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece1 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1450.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece2 stored as bytes in memory (estimated size 4.0 MB, free 1053.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece2 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1446.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece3 stored as bytes in memory (estimated size 4.0 MB, free 1049.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece3 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1442.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece4 stored as bytes in memory (estimated size 4.0 MB, free 1045.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece4 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1438.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece5 stored as bytes in memory (estimated size 4.0 MB, free 1041.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece5 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1434.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece6 stored as bytes in memory (estimated size 4.0 MB, free 1037.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece6 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1430.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece7 stored as bytes in memory (estimated size 4.0 MB, free 1033.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece7 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1426.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece8 stored as bytes in memory (estimated size 4.0 MB, free 1029.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece8 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1422.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece9 stored as bytes in memory (estimated size 4.0 MB, free 1025.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece9 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1418.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece10 stored as bytes in memory (estimated size 4.0 MB, free 1021.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece10 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1414.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece11 stored as bytes in memory (estimated size 4.0 MB, free 1017.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece11 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1410.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece12 stored as bytes in memory (estimated size 4.0 MB, free 1013.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece12 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1406.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece13 stored as bytes in memory (estimated size 4.0 MB, free 1009.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece13 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1402.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece14 stored as bytes in memory (estimated size 4.0 MB, free 1005.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece14 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1398.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece15 stored as bytes in memory (estimated size 4.0 MB, free 1001.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece15 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1394.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece16 stored as bytes in memory (estimated size 4.0 MB, free 997.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece16 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1390.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece17 stored as bytes in memory (estimated size 4.0 MB, free 993.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece17 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1386.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece18 stored as bytes in memory (estimated size 4.0 MB, free 989.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece18 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1382.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece19 stored as bytes in memory (estimated size 4.0 MB, free 985.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece19 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1378.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece20 stored as bytes in memory (estimated size 4.0 MB, free 981.3 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece20 in memory on htm-psycho-401.zxz.su:42625 (size: 4.0 MB, free: 1374.6 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - Block broadcast_17_piece21 stored as bytes in memory (estimated size 1207.3 KB, free 980.1 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Added broadcast_17_piece21 in memory on htm-psycho-401.zxz.su:42625 (size: 1207.3 KB, free: 1373.4 MB)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  SparkContext:54 - Created broadcast 17 from broadcast at ALSUpdate.java:136
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  MemoryStore:54 - 5 blocks selected for dropping (393.3 MB bytes)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManager:54 - Dropping block broadcast_1 from memory
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManager:54 - Writing block broadcast_1 to disk
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManager:54 - Dropping block broadcast_1_piece0 from memory
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManager:54 - Writing block broadcast_1_piece0 to disk
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Updated broadcast_1_piece0 on disk on htm-psycho-401.zxz.su:42625 (current size: 23.5 KB, original size: 0.0 B)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManager:54 - Dropping block broadcast_16 from memory
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManager:54 - Writing block broadcast_16 to disk
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManager:54 - Dropping block broadcast_16_piece0 from memory
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManager:54 - Writing block broadcast_16_piece0 to disk
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManagerInfo:54 - Updated broadcast_16_piece0 on disk on htm-psycho-401.zxz.su:42625 (current size: 4.1 KB, original size: 0.0 B)
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManager:54 - Dropping block broadcast_17 from memory
Sep 07 01:46:12 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:12 INFO  BlockManager:54 - Writing block broadcast_17 to disk
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 387
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 383
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 393
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 392
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned shuffle 2
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 377
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 396
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 385
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 386
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 384
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 379
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned shuffle 3
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 375
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 378
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 389
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 399
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 394
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 390
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 395
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  ContextCleaner:54 - Cleaned accumulator 397
Sep 07 01:46:19 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:46:19 INFO  BlockManagerInfo:54 - Removed broadcast_16_piece0 on htm-psycho-401.zxz.su:41761 in memory (size: 4.1 KB, free: 181.7 MB)

speed:

Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  BlockManagerInfo:54 - Added broadcast_10_piece0 in memory on htm-psycho-401.zxz.su:42317 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  SparkContext:54 - Created broadcast 10 from broadcast at DAGScheduler.scala:1039
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  DAGScheduler:54 - Submitting 1 missing tasks from ResultStage 10 (MapPartitionsRDD[13] at mapToPair at SpeedLayer.java:97) (first 15 tasks are for partitions Vector(0))
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  YarnScheduler:54 - Adding task set 10.0 with 1 tasks
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  TaskSetManager:54 - Starting task 0.0 in stage 10.0 (TID 18, htm-psycho-401.zxz.su, executor 1, partition 0, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  BlockManagerInfo:54 - Added broadcast_10_piece0 in memory on htm-psycho-401.zxz.su:45221 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  TaskSetManager:54 - Finished task 0.0 in stage 10.0 (TID 18) in 32 ms on htm-psycho-401.zxz.su (executor 1) (1/1)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  YarnScheduler:54 - Removed TaskSet 10.0, whose tasks have all completed, from pool
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  DAGScheduler:54 - ResultStage 10 (isEmpty at SpeedLayerUpdate.java:53) finished in 0.047 s
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  DAGScheduler:54 - Job 10 finished: isEmpty at SpeedLayerUpdate.java:53, took 0.050545 s
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  SparkContext:54 - Starting job: isEmpty at SpeedLayerUpdate.java:53
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  DAGScheduler:54 - Got job 11 (isEmpty at SpeedLayerUpdate.java:53) with 3 output partitions
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  DAGScheduler:54 - Final stage: ResultStage 11 (isEmpty at SpeedLayerUpdate.java:53)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  DAGScheduler:54 - Parents of final stage: List()
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  DAGScheduler:54 - Missing parents: List()
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  DAGScheduler:54 - Submitting ResultStage 11 (MapPartitionsRDD[13] at mapToPair at SpeedLayer.java:97), which has no missing parents
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  MemoryStore:54 - Block broadcast_11 stored as values in memory (estimated size 3.8 KB, free 912.3 MB)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  MemoryStore:54 - Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.4 KB, free 912.3 MB)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  BlockManagerInfo:54 - Added broadcast_11_piece0 in memory on htm-psycho-401.zxz.su:42317 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  SparkContext:54 - Created broadcast 11 from broadcast at DAGScheduler.scala:1039
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  DAGScheduler:54 - Submitting 3 missing tasks from ResultStage 11 (MapPartitionsRDD[13] at mapToPair at SpeedLayer.java:97) (first 15 tasks are for partitions Vector(1, 2, 3))
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  YarnScheduler:54 - Adding task set 11.0 with 3 tasks
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  TaskSetManager:54 - Starting task 0.0 in stage 11.0 (TID 19, htm-psycho-401.zxz.su, executor 1, partition 1, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  BlockManagerInfo:54 - Added broadcast_11_piece0 in memory on htm-psycho-401.zxz.su:45221 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  TaskSetManager:54 - Starting task 1.0 in stage 11.0 (TID 20, htm-psycho-401.zxz.su, executor 1, partition 2, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  TaskSetManager:54 - Finished task 0.0 in stage 11.0 (TID 19) in 26 ms on htm-psycho-401.zxz.su (executor 1) (1/3)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  TaskSetManager:54 - Starting task 2.0 in stage 11.0 (TID 21, htm-psycho-401.zxz.su, executor 1, partition 3, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  TaskSetManager:54 - Finished task 1.0 in stage 11.0 (TID 20) in 12 ms on htm-psycho-401.zxz.su (executor 1) (2/3)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  TaskSetManager:54 - Finished task 2.0 in stage 11.0 (TID 21) in 12 ms on htm-psycho-401.zxz.su (executor 1) (3/3)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  YarnScheduler:54 - Removed TaskSet 11.0, whose tasks have all completed, from pool
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  DAGScheduler:54 - ResultStage 11 (isEmpty at SpeedLayerUpdate.java:53) finished in 0.063 s
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  DAGScheduler:54 - Job 11 finished: isEmpty at SpeedLayerUpdate.java:53, took 0.067594 s
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  JobScheduler:54 - Finished job streaming job 1536273000000 ms.0 from job set of time 1536273000000 ms
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  JobScheduler:54 - Starting job streaming job 1536273000000 ms.1 from job set of time 1536273000000 ms
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  UpdateOffsetsFn:60 - Updating offsets: {OryxInput,0=2576784, OryxInput,1=2579068, OryxInput,2=2577833, OryxInput,3=2579317}
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  ZooKeeper:438 - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@32686a1d
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  ZkEventThread:65 - Starting ZkClient event thread.
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  ZkClient:936 - Waiting for keeper state SyncConnected
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  ClientCnxn:975 - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  ClientCnxn:852 - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  ClientCnxn:1235 - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x10160d98a7901f3, negotiated timeout = 30000
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  ZkClient:713 - zookeeper state changed (SyncConnected)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  ZkEventThread:83 - Terminate ZkClient event thread.
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  ZooKeeper:684 - Session: 0x10160d98a7901f3 closed
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  ClientCnxn:512 - EventThread shut down
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  JobScheduler:54 - Finished job streaming job 1536273000000 ms.1 from job set of time 1536273000000 ms
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  JobScheduler:54 - Total delay: 0.148 s for time 1536273000000 ms (execution: 0.142 s)
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  MapPartitionsRDD:54 - Removing RDD 11 from persistence list
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  BlockManager:54 - Removing RDD 11
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  KafkaRDD:54 - Removing RDD 10 from persistence list
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  BlockManager:54 - Removing RDD 10
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  ReceivedBlockTracker:54 - Deleting batches:
Sep 07 01:30:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:30:00 INFO  InputInfoTracker:54 - remove old batch metadata: 1536272400000 ms
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  JobScheduler:54 - Added jobs for time 1536273300000 ms
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  JobScheduler:54 - Starting job streaming job 1536273300000 ms.0 from job set of time 1536273300000 ms
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  SparkContext:54 - Starting job: isEmpty at SpeedLayerUpdate.java:53
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Got job 12 (isEmpty at SpeedLayerUpdate.java:53) with 1 output partitions
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Final stage: ResultStage 12 (isEmpty at SpeedLayerUpdate.java:53)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Parents of final stage: List()
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Missing parents: List()
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Submitting ResultStage 12 (MapPartitionsRDD[15] at mapToPair at SpeedLayer.java:97), which has no missing parents
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  MemoryStore:54 - Block broadcast_12 stored as values in memory (estimated size 3.8 KB, free 912.3 MB)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  MemoryStore:54 - Block broadcast_12_piece0 stored as bytes in memory (estimated size 2.4 KB, free 912.3 MB)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  BlockManagerInfo:54 - Added broadcast_12_piece0 in memory on htm-psycho-401.zxz.su:42317 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  SparkContext:54 - Created broadcast 12 from broadcast at DAGScheduler.scala:1039
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Submitting 1 missing tasks from ResultStage 12 (MapPartitionsRDD[15] at mapToPair at SpeedLayer.java:97) (first 15 tasks are for partitions Vector(0))
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  YarnScheduler:54 - Adding task set 12.0 with 1 tasks
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  TaskSetManager:54 - Starting task 0.0 in stage 12.0 (TID 22, htm-psycho-401.zxz.su, executor 1, partition 0, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  BlockManagerInfo:54 - Added broadcast_12_piece0 in memory on htm-psycho-401.zxz.su:45221 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  TaskSetManager:54 - Finished task 0.0 in stage 12.0 (TID 22) in 30 ms on htm-psycho-401.zxz.su (executor 1) (1/1)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  YarnScheduler:54 - Removed TaskSet 12.0, whose tasks have all completed, from pool
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - ResultStage 12 (isEmpty at SpeedLayerUpdate.java:53) finished in 0.044 s
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Job 12 finished: isEmpty at SpeedLayerUpdate.java:53, took 0.048293 s
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  SparkContext:54 - Starting job: isEmpty at SpeedLayerUpdate.java:53
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Got job 13 (isEmpty at SpeedLayerUpdate.java:53) with 3 output partitions
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Final stage: ResultStage 13 (isEmpty at SpeedLayerUpdate.java:53)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Parents of final stage: List()
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Missing parents: List()
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Submitting ResultStage 13 (MapPartitionsRDD[15] at mapToPair at SpeedLayer.java:97), which has no missing parents
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  MemoryStore:54 - Block broadcast_13 stored as values in memory (estimated size 3.8 KB, free 912.3 MB)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  MemoryStore:54 - Block broadcast_13_piece0 stored as bytes in memory (estimated size 2.4 KB, free 912.3 MB)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  BlockManagerInfo:54 - Added broadcast_13_piece0 in memory on htm-psycho-401.zxz.su:42317 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  SparkContext:54 - Created broadcast 13 from broadcast at DAGScheduler.scala:1039
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Submitting 3 missing tasks from ResultStage 13 (MapPartitionsRDD[15] at mapToPair at SpeedLayer.java:97) (first 15 tasks are for partitions Vector(1, 2, 3))
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  YarnScheduler:54 - Adding task set 13.0 with 3 tasks
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  TaskSetManager:54 - Starting task 0.0 in stage 13.0 (TID 23, htm-psycho-401.zxz.su, executor 1, partition 1, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  BlockManagerInfo:54 - Added broadcast_13_piece0 in memory on htm-psycho-401.zxz.su:45221 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  TaskSetManager:54 - Starting task 1.0 in stage 13.0 (TID 24, htm-psycho-401.zxz.su, executor 1, partition 2, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  TaskSetManager:54 - Finished task 0.0 in stage 13.0 (TID 23) in 31 ms on htm-psycho-401.zxz.su (executor 1) (1/3)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  TaskSetManager:54 - Starting task 2.0 in stage 13.0 (TID 25, htm-psycho-401.zxz.su, executor 1, partition 3, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  TaskSetManager:54 - Finished task 1.0 in stage 13.0 (TID 24) in 13 ms on htm-psycho-401.zxz.su (executor 1) (2/3)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  TaskSetManager:54 - Finished task 2.0 in stage 13.0 (TID 25) in 16 ms on htm-psycho-401.zxz.su (executor 1) (3/3)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  YarnScheduler:54 - Removed TaskSet 13.0, whose tasks have all completed, from pool
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - ResultStage 13 (isEmpty at SpeedLayerUpdate.java:53) finished in 0.072 s
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  DAGScheduler:54 - Job 13 finished: isEmpty at SpeedLayerUpdate.java:53, took 0.078273 s
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  JobScheduler:54 - Finished job streaming job 1536273300000 ms.0 from job set of time 1536273300000 ms
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  JobScheduler:54 - Starting job streaming job 1536273300000 ms.1 from job set of time 1536273300000 ms
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  UpdateOffsetsFn:60 - Updating offsets: {OryxInput,0=2576784, OryxInput,1=2579068, OryxInput,2=2577833, OryxInput,3=2579317}
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  ZooKeeper:438 - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@18590535
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  ZkEventThread:65 - Starting ZkClient event thread.
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  ZkClient:936 - Waiting for keeper state SyncConnected
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  ClientCnxn:975 - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  ClientCnxn:852 - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  ClientCnxn:1235 - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x10160d98a7901f4, negotiated timeout = 30000
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  ZkClient:713 - zookeeper state changed (SyncConnected)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  ZkEventThread:83 - Terminate ZkClient event thread.
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  ZooKeeper:684 - Session: 0x10160d98a7901f4 closed
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  ClientCnxn:512 - EventThread shut down
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  JobScheduler:54 - Finished job streaming job 1536273300000 ms.1 from job set of time 1536273300000 ms
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  JobScheduler:54 - Total delay: 0.162 s for time 1536273300000 ms (execution: 0.156 s)
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  MapPartitionsRDD:54 - Removing RDD 13 from persistence list
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  BlockManager:54 - Removing RDD 13
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  KafkaRDD:54 - Removing RDD 12 from persistence list
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  ReceivedBlockTracker:54 - Deleting batches:
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  InputInfoTracker:54 - remove old batch metadata: 1536272700000 ms
Sep 07 01:35:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:35:00 INFO  BlockManager:54 - Removing RDD 12
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  JobScheduler:54 - Starting job streaming job 1536273600000 ms.0 from job set of time 1536273600000 ms
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  JobScheduler:54 - Added jobs for time 1536273600000 ms
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  SparkContext:54 - Starting job: isEmpty at SpeedLayerUpdate.java:53
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  DAGScheduler:54 - Got job 14 (isEmpty at SpeedLayerUpdate.java:53) with 1 output partitions
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  DAGScheduler:54 - Final stage: ResultStage 14 (isEmpty at SpeedLayerUpdate.java:53)
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  DAGScheduler:54 - Parents of final stage: List()
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  DAGScheduler:54 - Missing parents: List()
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  DAGScheduler:54 - Submitting ResultStage 14 (MapPartitionsRDD[17] at mapToPair at SpeedLayer.java:97), which has no missing parents
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  MemoryStore:54 - Block broadcast_14 stored as values in memory (estimated size 3.8 KB, free 912.3 MB)
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  MemoryStore:54 - Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.4 KB, free 912.3 MB)
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  BlockManagerInfo:54 - Added broadcast_14_piece0 in memory on htm-psycho-401.zxz.su:42317 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  SparkContext:54 - Created broadcast 14 from broadcast at DAGScheduler.scala:1039
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  DAGScheduler:54 - Submitting 1 missing tasks from ResultStage 14 (MapPartitionsRDD[17] at mapToPair at SpeedLayer.java:97) (first 15 tasks are for partitions Vector(0))
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  YarnScheduler:54 - Adding task set 14.0 with 1 tasks
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  TaskSetManager:54 - Starting task 0.0 in stage 14.0 (TID 26, htm-psycho-401.zxz.su, executor 1, partition 0, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  BlockManagerInfo:54 - Added broadcast_14_piece0 in memory on htm-psycho-401.zxz.su:45221 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  TaskSetManager:54 - Finished task 0.0 in stage 14.0 (TID 26) in 55 ms on htm-psycho-401.zxz.su (executor 1) (1/1)
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  YarnScheduler:54 - Removed TaskSet 14.0, whose tasks have all completed, from pool
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  DAGScheduler:54 - ResultStage 14 (isEmpty at SpeedLayerUpdate.java:53) finished in 0.068 s
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  DAGScheduler:54 - Job 14 finished: isEmpty at SpeedLayerUpdate.java:53, took 0.071840 s
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  JobScheduler:54 - Finished job streaming job 1536273600000 ms.0 from job set of time 1536273600000 ms
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  JobScheduler:54 - Starting job streaming job 1536273600000 ms.1 from job set of time 1536273600000 ms
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  UpdateOffsetsFn:60 - Updating offsets: {OryxInput,0=2626356, OryxInput,1=2629184, OryxInput,2=2627875, OryxInput,3=2629587}
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  ZooKeeper:438 - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@710d987a
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  ZkEventThread:65 - Starting ZkClient event thread.
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  ZkClient:936 - Waiting for keeper state SyncConnected
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  ClientCnxn:975 - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  ClientCnxn:852 - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  ClientCnxn:1235 - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x10160d98a7901f7, negotiated timeout = 30000
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  ZkClient:713 - zookeeper state changed (SyncConnected)
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  ZkEventThread:83 - Terminate ZkClient event thread.
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  ZooKeeper:684 - Session: 0x10160d98a7901f7 closed
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  ClientCnxn:512 - EventThread shut down
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  JobScheduler:54 - Finished job streaming job 1536273600000 ms.1 from job set of time 1536273600000 ms
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  JobScheduler:54 - Total delay: 0.118 s for time 1536273600000 ms (execution: 0.113 s)
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  MapPartitionsRDD:54 - Removing RDD 15 from persistence list
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  BlockManager:54 - Removing RDD 15
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  KafkaRDD:54 - Removing RDD 14 from persistence list
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  BlockManager:54 - Removing RDD 14
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  ReceivedBlockTracker:54 - Deleting batches:
Sep 07 01:40:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:40:00 INFO  InputInfoTracker:54 - remove old batch metadata: 1536273000000 ms
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  JobScheduler:54 - Added jobs for time 1536273900000 ms
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  JobScheduler:54 - Starting job streaming job 1536273900000 ms.0 from job set of time 1536273900000 ms
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  SparkContext:54 - Starting job: isEmpty at SpeedLayerUpdate.java:53
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Got job 15 (isEmpty at SpeedLayerUpdate.java:53) with 1 output partitions
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Final stage: ResultStage 15 (isEmpty at SpeedLayerUpdate.java:53)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Parents of final stage: List()
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Missing parents: List()
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Submitting ResultStage 15 (MapPartitionsRDD[19] at mapToPair at SpeedLayer.java:97), which has no missing parents
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  MemoryStore:54 - Block broadcast_15 stored as values in memory (estimated size 3.8 KB, free 912.3 MB)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  MemoryStore:54 - Block broadcast_15_piece0 stored as bytes in memory (estimated size 2.4 KB, free 912.3 MB)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  BlockManagerInfo:54 - Added broadcast_15_piece0 in memory on htm-psycho-401.zxz.su:42317 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  SparkContext:54 - Created broadcast 15 from broadcast at DAGScheduler.scala:1039
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Submitting 1 missing tasks from ResultStage 15 (MapPartitionsRDD[19] at mapToPair at SpeedLayer.java:97) (first 15 tasks are for partitions Vector(0))
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  YarnScheduler:54 - Adding task set 15.0 with 1 tasks
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  TaskSetManager:54 - Starting task 0.0 in stage 15.0 (TID 27, htm-psycho-401.zxz.su, executor 1, partition 0, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  BlockManagerInfo:54 - Added broadcast_15_piece0 in memory on htm-psycho-401.zxz.su:45221 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  TaskSetManager:54 - Finished task 0.0 in stage 15.0 (TID 27) in 27 ms on htm-psycho-401.zxz.su (executor 1) (1/1)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  YarnScheduler:54 - Removed TaskSet 15.0, whose tasks have all completed, from pool
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - ResultStage 15 (isEmpty at SpeedLayerUpdate.java:53) finished in 0.042 s
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Job 15 finished: isEmpty at SpeedLayerUpdate.java:53, took 0.046513 s
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  SparkContext:54 - Starting job: isEmpty at SpeedLayerUpdate.java:53
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Got job 16 (isEmpty at SpeedLayerUpdate.java:53) with 3 output partitions
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Final stage: ResultStage 16 (isEmpty at SpeedLayerUpdate.java:53)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Parents of final stage: List()
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Missing parents: List()
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Submitting ResultStage 16 (MapPartitionsRDD[19] at mapToPair at SpeedLayer.java:97), which has no missing parents
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  MemoryStore:54 - Block broadcast_16 stored as values in memory (estimated size 3.8 KB, free 912.3 MB)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  MemoryStore:54 - Block broadcast_16_piece0 stored as bytes in memory (estimated size 2.4 KB, free 912.3 MB)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  BlockManagerInfo:54 - Added broadcast_16_piece0 in memory on htm-psycho-401.zxz.su:42317 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  SparkContext:54 - Created broadcast 16 from broadcast at DAGScheduler.scala:1039
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Submitting 3 missing tasks from ResultStage 16 (MapPartitionsRDD[19] at mapToPair at SpeedLayer.java:97) (first 15 tasks are for partitions Vector(1, 2, 3))
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  YarnScheduler:54 - Adding task set 16.0 with 3 tasks
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  TaskSetManager:54 - Starting task 0.0 in stage 16.0 (TID 28, htm-psycho-401.zxz.su, executor 1, partition 1, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  BlockManagerInfo:54 - Added broadcast_16_piece0 in memory on htm-psycho-401.zxz.su:45221 (size: 2.4 KB, free: 912.3 MB)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  TaskSetManager:54 - Starting task 1.0 in stage 16.0 (TID 29, htm-psycho-401.zxz.su, executor 1, partition 2, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  TaskSetManager:54 - Finished task 0.0 in stage 16.0 (TID 28) in 26 ms on htm-psycho-401.zxz.su (executor 1) (1/3)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  TaskSetManager:54 - Starting task 2.0 in stage 16.0 (TID 30, htm-psycho-401.zxz.su, executor 1, partition 3, PROCESS_LOCAL, 7746 bytes)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  TaskSetManager:54 - Finished task 1.0 in stage 16.0 (TID 29) in 15 ms on htm-psycho-401.zxz.su (executor 1) (2/3)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  TaskSetManager:54 - Finished task 2.0 in stage 16.0 (TID 30) in 17 ms on htm-psycho-401.zxz.su (executor 1) (3/3)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  YarnScheduler:54 - Removed TaskSet 16.0, whose tasks have all completed, from pool
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - ResultStage 16 (isEmpty at SpeedLayerUpdate.java:53) finished in 0.069 s
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  DAGScheduler:54 - Job 16 finished: isEmpty at SpeedLayerUpdate.java:53, took 0.073463 s
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  JobScheduler:54 - Finished job streaming job 1536273900000 ms.0 from job set of time 1536273900000 ms
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  JobScheduler:54 - Starting job streaming job 1536273900000 ms.1 from job set of time 1536273900000 ms
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  UpdateOffsetsFn:60 - Updating offsets: {OryxInput,0=2626356, OryxInput,1=2629184, OryxInput,2=2627875, OryxInput,3=2629587}
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  ZooKeeper:438 - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@2796e67b
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  ZkEventThread:65 - Starting ZkClient event thread.
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  ZkClient:936 - Waiting for keeper state SyncConnected
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  ClientCnxn:975 - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  ClientCnxn:852 - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  ClientCnxn:1235 - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x10160d98a7901f8, negotiated timeout = 30000
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  ZkClient:713 - zookeeper state changed (SyncConnected)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  ZkEventThread:83 - Terminate ZkClient event thread.
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  ZooKeeper:684 - Session: 0x10160d98a7901f8 closed
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  ClientCnxn:512 - EventThread shut down
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  JobScheduler:54 - Finished job streaming job 1536273900000 ms.1 from job set of time 1536273900000 ms
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  JobScheduler:54 - Total delay: 0.153 s for time 1536273900000 ms (execution: 0.148 s)
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  MapPartitionsRDD:54 - Removing RDD 17 from persistence list
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  KafkaRDD:54 - Removing RDD 16 from persistence list
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  BlockManager:54 - Removing RDD 17
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  BlockManager:54 - Removing RDD 16
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  ReceivedBlockTracker:54 - Deleting batches:
Sep 07 01:45:00 htm-psycho-401.zxz.su bash[9121]: 2018-09-07 01:45:00 INFO  InputInfoTracker:54 - remove old batch metadata: 1536273300000 ms

serving:

Sep 07 00:57:07 htm-psycho-401.zxz.su java[9123]: 2018-09-07 00:57:07,892 INFO  AppInfoParser:83 Kafka version : 0.11.0.1
Sep 07 00:57:07 htm-psycho-401.zxz.su java[9123]: 2018-09-07 00:57:07,892 INFO  AppInfoParser:84 Kafka commitId : c2a0d5f9b1f45bf5
Sep 07 00:57:08 htm-psycho-401.zxz.su java[9123]: 2018-09-07 00:57:08,075 INFO  AbstractCoordinator:607 Discovered coordinator localhost:9092 (id: 2147483647 rack: null) for group OryxGroup-ServingLayer-157da7e9-f0e9-468e-bb5e-a7293cd7e111.
Sep 07 00:57:08 htm-psycho-401.zxz.su java[9123]: 2018-09-07 00:57:08,078 INFO  ConsumerCoordinator:419 Revoking previously assigned partitions [] for group OryxGroup-ServingLayer-157da7e9-f0e9-468e-bb5e-a7293cd7e111
Sep 07 00:57:08 htm-psycho-401.zxz.su java[9123]: 2018-09-07 00:57:08,078 INFO  AbstractCoordinator:442 (Re-)joining group OryxGroup-ServingLayer-157da7e9-f0e9-468e-bb5e-a7293cd7e111
Sep 07 00:57:08 htm-psycho-401.zxz.su java[9123]: 2018-09-07 00:57:08,100 INFO  AbstractCoordinator:409 Successfully joined group OryxGroup-ServingLayer-157da7e9-f0e9-468e-bb5e-a7293cd7e111 with generation 1
Sep 07 00:57:08 htm-psycho-401.zxz.su java[9123]: 2018-09-07 00:57:08,101 INFO  ConsumerCoordinator:262 Setting newly assigned partitions [OryxUpdate-0] for group OryxGroup-ServingLayer-157da7e9-f0e9-468e-bb5e-a7293cd7e111
Sep 07 00:57:08 htm-psycho-401.zxz.su java[9123]: 2018-09-07 00:57:08,261 INFO  OryxApplication:65 Creating JAX-RS from endpoints in package(s) com.cloudera.oryx.app.serving,com.cloudera.oryx.app.serving.als,com.cloudera.oryx.lambda.serving
Sep 07 00:57:08 htm-psycho-401.zxz.su java[9123]: 2018-09-07 00:57:08,412 INFO  Reflections:229 Reflections took 126 ms to scan 1 urls, producing 17 keys and 106 values
Sep 07 00:57:08 htm-psycho-401.zxz.su java[9123]: 2018-09-07 00:57:08,511 INFO  Reflections:229 Reflections took 68 ms to scan 1 urls, producing 10 keys and 64 values
Sep 07 00:57:08 htm-psycho-401.zxz.su java[9123]: 2018-09-07 00:57:08,565 INFO  Reflections:229 Reflections took 50 ms to scan 1 urls, producing 13 keys and 14 values
Sep 07 00:57:09 htm-psycho-401.zxz.su java[9123]: Sep 07, 2018 12:57:09 AM org.apache.coyote.AbstractProtocol start
Sep 07 00:57:09 htm-psycho-401.zxz.su java[9123]: INFO: Starting ProtocolHandler ["http-nio2-8080"]
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: 2018-09-07 01:00:50,141 INFO  ProducerConfig:223 ProducerConfig values:
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: acks = 1
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: batch.size = 16384
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: bootstrap.servers = [127.0.0.1:9092]
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: buffer.memory = 33554432
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: client.id =
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: compression.type = gzip
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: connections.max.idle.ms = 540000
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: enable.idempotence = false
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: interceptor.classes = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: key.serializer = class org.apache.kafka.common.serialization.StringSerializer
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: linger.ms = 1000
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: max.block.ms = 60000
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: max.in.flight.requests.per.connection = 5
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: max.request.size = 67108864
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: metadata.max.age.ms = 300000
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: metric.reporters = []
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: metrics.num.samples = 2
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: metrics.recording.level = INFO
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: metrics.sample.window.ms = 30000
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: receive.buffer.bytes = 32768
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: reconnect.backoff.max.ms = 1000
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: reconnect.backoff.ms = 50
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: request.timeout.ms = 30000
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: retries = 0
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: retry.backoff.ms = 100
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: sasl.jaas.config = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: sasl.kerberos.kinit.cmd = /usr/bin/kinit
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: sasl.kerberos.min.time.before.relogin = 60000
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: sasl.kerberos.service.name = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: sasl.kerberos.ticket.renew.jitter = 0.05
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: sasl.kerberos.ticket.renew.window.factor = 0.8
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: sasl.mechanism = GSSAPI
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: security.protocol = PLAINTEXT
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: send.buffer.bytes = 131072
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.cipher.suites = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.endpoint.identification.algorithm = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.key.password = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.keymanager.algorithm = SunX509
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.keystore.location = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.keystore.password = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.keystore.type = JKS
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.protocol = TLS
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.provider = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.secure.random.implementation = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.trustmanager.algorithm = PKIX
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.truststore.location = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.truststore.password = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: ssl.truststore.type = JKS
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: transaction.timeout.ms = 60000
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: transactional.id = null
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: value.serializer = class org.apache.kafka.common.serialization.StringSerializer
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: 2018-09-07 01:00:50,156 INFO  AppInfoParser:83 Kafka version : 0.11.0.1
Sep 07 01:00:50 htm-psycho-401.zxz.su java[9123]: 2018-09-07 01:00:50,156 INFO  AppInfoParser:84 Kafka commitId : c2a0d5f9b1f45bf5

why serving layer still gives 503?

from oryx.

srowen commented on July 30, 2024

It means no model has been built yet. You need to make sure the batch layer has built a model successfully and pushed it onto the kafka topic. Here it's set to build every 5 minutes. It may also be unable to build a model from the data; check the logs.

from oryx.

stiv-yakovenko commented on July 30, 2024

You are right, please be so kind to take a look here, this is a tail of batch layer log:

Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 INFO  YarnSchedulerBackend$YarnDriverEndpoint:54 - Disabling executor 1.
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 INFO  DAGScheduler:54 - Executor lost: 1 (epoch 15)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 INFO  BlockManagerMasterEndpoint:54 - Trying to remove executor 1 from BlockManagerMaster.
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 ERROR TransportClient:233 - Failed to send RPC 4632003068203477306 to /192.168.147.4:58358: java.nio.channels.ClosedChannelException
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: java.nio.channels.ClosedChannelException
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_7 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_22 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_31 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_4 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_0 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_8 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_12 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_8 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_1 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_11 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_13 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_5 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_9 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_20 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_2 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_8 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_8 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_7 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_2 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_2 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_11 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  YarnSchedulerBackend$YarnSchedulerEndpoint:87 - Attempted to get executor loss reason for executor id 1 at RPC address 192.168.147.4:58368, but got no response. Marking as slave lost.
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: java.io.IOException: Failed to send RPC 4632003068203477306 to /192.168.147.4:58358: java.nio.channels.ClosedChannelException
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:987)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:869)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1316)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at java.lang.Thread.run(Thread.java:748)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: Caused by: java.nio.channels.ClosedChannelException
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_0 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_11 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_9 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_2 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_3 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_4 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_10 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_3 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_10 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_15 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_30 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_11 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_1 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_26 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_19 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_0 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_7 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_2 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_9 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_12_28 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_1 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_31 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_14 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_13 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_14 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_8 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_7 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_10 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_9 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_1 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_3 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_14 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_6 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_13 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_6 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_5_26 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_6 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_4 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_3 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_12 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_12_30 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_12 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_0 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_15 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_17 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_5_21 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_12_27 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_8 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_6 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_14 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_6 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_6 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_25 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_5_25 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_15 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_14 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_13 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_4 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_11 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_29 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_24 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_0 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_26 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_3 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_5_24 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_23 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_5 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_18 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_6 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_15 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_11 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_7 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_1 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_8 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_5 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_22 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_5 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_14 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_30 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_5 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 ERROR YarnScheduler:70 - Lost executor 1 on htm-psycho-401.zxz.su: Slave lost
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_3 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_5 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_4 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_16 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_10 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_9 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_11 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_6 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_7 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_9 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_20 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_9 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_0 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_12 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_15 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_12 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_7 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_11 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_12_25 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_13 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_61_13 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_0 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_15 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_28 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_2 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_8 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_5_27 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_27 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_13 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_13 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_19 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_14 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_5 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_2 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_3 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_28 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_18 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_2 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_12 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_4 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_10 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_29 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_4 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_3 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_21 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_5_23 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_66_15 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_17 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_1 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_21 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_5 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_10 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_24 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_12 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_4 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_9 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_25 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_67_14 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_23 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_1 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_109_10 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_12_31 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_27 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_0 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  TaskSetManager:66 - Lost task 15.0 in stage 94.0 (TID 778, htm-psycho-401.zxz.su, executor 1): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Slave lost
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_12_29 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_7 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_58_10 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_52_16 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_119_1 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 WARN  BlockManagerMasterEndpoint:66 - No more replicas available for rdd_62_12 !
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 INFO  BlockManagerMasterEndpoint:54 - Removing block manager BlockManagerId(1, htm-psycho-401.zxz.su, 41761, None)
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 INFO  BlockManagerMaster:54 - Removed 1 successfully in removeExecutor
Sep 07 01:58:16 htm-psycho-401.zxz.su bash[10427]: 2018-09-07 01:58:16 INFO  DAGScheduler:54 - Shuffle files lost for executor: 1 (epoch 15)

from oryx.

stiv-yakovenko commented on July 30, 2024

My guess is that huge 4GB data, which I was using for testing previously, is somewhere in kafka or somewhere else, but what can I do to fix that now?

from oryx.

srowen commented on July 30, 2024

The speed layer will ignore previous data, but batch won't. It may be taking a long time to build a model on all that data. You can just delete and recreate the topics.

from oryx.

stiv-yakovenko commented on July 30, 2024

ok, but what does this message mean: No more replicas available ?

from oryx.

srowen commented on July 30, 2024

It means executors died, and enough died that some broadcasted blocks of data were lost and the task failed. It's just a symptom of executors dying. These are really Spark questions rather than related to Oryx. I think it's mostly that you're not giving the cluster enough resources.

from oryx.

stiv-yakovenko commented on July 30, 2024

It looks like entire node has died (unsure which node -- hadoop, yarn, etc...) because too many data and data is stored on /tmp by default

from oryx.

stiv-yakovenko commented on July 30, 2024

Solved the problem, i had not enough space on /tmp, moved to /var, reformatted and now it works! Thank for your support!

from oryx.

KryoException: Buffer overflow. Available: 1, required: 4 about oryx HOT 19 CLOSED

Comments (19)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

Jobs