Due to changes in external dependencies, it's no longer possible to follow the quick start guide successfully.

Mac OS 10.9 Mavericks introduced changes in bundled applications and there are some deeper dependencies in the Maven POM that need to be modified.

It seems source reader got stuck if failing to connect to s socket. We need to define the behavior and not stop mupd8 itself by this failure.

Following can be used as a test server

!/bin/bash

while true ; do
cat ~/git/mupd8/src/test/resources/testapp/T10.data
sleep 2
done | while true ; do
nc -l 1234
done

I have code that does source reader throttling when a hot conductor is backlogged. However it is not enabled, as it hasn't been tested.

A test will require 2 machines/JVMs each reading from a source. On of the JVMs should backlog an a key and the test must make sure that the source reader on the other JVM slows down.

I was able to specify and execute my own data source, map and update tasks. Now I was wondering how to specify the actual number of task instances of each task. I cannot find anything in the config file. Could you please explain this to me?

Please fix when you get chance following exception.

Thanks,
Bhavesh

Here is log:

21 Jan 2015 17:18:23,930 ERROR New I/O server worker #2-1 - HttpServerHandler exception.
java.lang.NullPointerException
at com.walmartlabs.mupd8.AppRuntime$$anonfun$22.apply(AppRuntime.scala:310)
at com.walmartlabs.mupd8.AppRuntime$$anonfun$22.apply(AppRuntime.scala:227)
at com.walmartlabs.mupd8.HttpServerHandler.handleHttpRequest(HttpServer.scala:117)
at com.walmartlabs.mupd8.HttpServerHandler.messageReceived(HttpServer.scala:100)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndFireMessageReceived(ReplayingDecoder.java:527)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:506)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:443)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
21 Jan 2015 17:18:25,102 ERROR New I/O server worker #2-2 - HttpServerHandler exception.
java.lang.NullPointerException
at com.walmartlabs.mupd8.AppRuntime$$anonfun$22.apply(AppRuntime.scala:310)
at com.walmartlabs.mupd8.AppRuntime$$anonfun$22.apply(AppRuntime.scala:227)
at com.walmartlabs.mupd8.HttpServerHandler.handleHttpRequest(HttpServer.scala:117)
at com.walmartlabs.mupd8.HttpServerHandler.messageReceived(HttpServer.scala:100)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndFireMessageReceived(ReplayingDecoder.java:527)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:506)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:443)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

In addition to that, Mupd8 library does throw NP, due to duplicated raws in DB/or previous unclean shutdown (issue with mupd8 code).

10 Feb 2015 19:16:06,627 ERROR MessageServer - MessageServerThread exception.
java.lang.NullPointerException
at com.walmartlabs.mupd8.MessageServer$$anonfun$run$10.apply(MessageServer.scala:132)
at com.walmartlabs.mupd8.MessageServer$$anonfun$run$10.apply(MessageServer.scala:128)
at scala.collection.immutable.Map$Map4.foreach(Map.scala:181)
at com.walmartlabs.mupd8.MessageServer.run(MessageServer.scala:128)
10 Feb 2015 19:16:06,688 ERROR MessageServer - MessageServerThread exception.
java.lang.NullPointerException
at com.walmartlabs.mupd8.MessageServer$$anonfun$run$10.apply(MessageServer.scala:132)
at com.walmartlabs.mupd8.MessageServer$$anonfun$run$10.apply(MessageServer.scala:128)
at scala.collection.immutable.Map$Map4.foreach(Map.scala:181)
at com.walmartlabs.mupd8.MessageServer.run(MessageServer.scala:128)

Thanks,
Bhavesh

mvn archetype:generate
-DinteractiveMode=false
-DarchetypeGroupId=org.apache.maven.archetypes
-DarchetypeArtifactId=maven-archetype-quickstart
-DgroupId=com.walmartlabs.example
-DartifactId=mupd8_demo
-Dpackage=com.walmartlabs.example
-Dversion=1.0-SNAPSHOT

This step gives the following errors

Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:2.2:generate (default-cli) on project mupd8: Unable to add module to the current project as it is not of packaging type 'pom

In my current project, we moved to using 'com.fasterxml'.
Mupd8 still refers to 'org.codehaus' and conflicting.

I made code changes and following up with Ning Zhang for next steps.

Please fix following NPE.

java.lang.NullPointerException
at com.walmartlabs.mupd8.AppRuntime$$anonfun$32.apply(AppRuntime.scala:364)
at com.walmartlabs.mupd8.AppRuntime$$anonfun$32.apply(AppRuntime.scala:364)
at scala.collection.LinearSeqOptimized$class.find(LinearSeqOptimized.scala:100)
at scala.collection.immutable.List.find(List.scala:84)

Thanks,
Bhavesh

If there's an exception thrown in the socket reader initialization (e.g. connection refused), the reader object will be initialized to None. But readLine() assumes the reader object to be properly instantiated at the first initialization, if the reader type is socket (the later re-initialization seems correctly handled by the exponential-wait bit). As a result readLine() produces false-negative for all subsequent hasNext() calls.

A long-term restructure of the SourceReader is still underway, and as it's not coming out soon, I will propose a fix based on current implementation of JONSSource.

The relevant log:

Jun 13, 2013 5:56:02 PM grizzled.slf4j.Logger error
SEVERE: JSONSource: socketReader hit exception
java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
    at java.net.Socket.connect(Socket.java:579)
    at java.net.Socket.connect(Socket.java:528)
    at java.net.Socket.<init>(Socket.java:425)
    at java.net.Socket.<init>(Socket.java:208)
    at com.walmartlabs.mupd8.JSONSource.socketReader(SourceReader.scala:75)
    at com.walmartlabs.mupd8.JSONSource.constructReader(SourceReader.scala:59)
    at com.walmartlabs.mupd8.JSONSource.com$walmartlabs$mupd8$JSONSource$$readLine(SourceReader.scala:119)
    at com.walmartlabs.mupd8.JSONSource$$anonfun$hasNext$1.apply(SourceReader.scala:95)
    at com.walmartlabs.mupd8.JSONSource$$anonfun$hasNext$1.apply(SourceReader.scala:94)
    at scala.Option.orElse(Option.scala:218)
    at com.walmartlabs.mupd8.JSONSource.hasNext(SourceReader.scala:93)
    at com.walmartlabs.mupd8.AppRuntime$SourceThread$1$$anonfun$run$3.apply$mcV$sp(Mupd8Main.scala:1221)
    at scala.util.control.Breaks.breakable(Breaks.scala:39)
    at com.walmartlabs.mupd8.AppRuntime$SourceThread$1.run(Mupd8Main.scala:1218)
    at java.lang.Thread.run(Thread.java:722)

Jun 13, 2013 5:56:02 PM grizzled.slf4j.Logger error
SEVERE: JSONSource: reader readLine failed
java.util.NoSuchElementException: None.get
    at scala.None$.get(Option.scala:274)
    at scala.None$.get(Option.scala:272)
    at com.walmartlabs.mupd8.JSONSource.com$walmartlabs$mupd8$JSONSource$$readLine(SourceReader.scala:108)
    at com.walmartlabs.mupd8.JSONSource$$anonfun$hasNext$1.apply(SourceReader.scala:95)
    at com.walmartlabs.mupd8.JSONSource$$anonfun$hasNext$1.apply(SourceReader.scala:94)
    at scala.Option.orElse(Option.scala:218)
    at com.walmartlabs.mupd8.JSONSource.hasNext(SourceReader.scala:93)
    at com.walmartlabs.mupd8.AppRuntime$SourceThread$1$$anonfun$run$3.apply$mcV$sp(Mupd8Main.scala:1221)
    at scala.util.control.Breaks.breakable(Breaks.scala:39)
    at com.walmartlabs.mupd8.AppRuntime$SourceThread$1.run(Mupd8Main.scala:1218)
    at java.lang.Thread.run(Thread.java:722)

2015-01-02 16:21:31,081 [ERROR] c.w.mupd8.MessageServer [MessageServer] - MessageServerThread exception.
java.lang.NullPointerException: null
at com.walmartlabs.mupd8.MessageServer$$anonfun$run$7.apply(MessageServer.scala:107) ~[classes/:na]
at com.walmartlabs.mupd8.MessageServer$$anonfun$run$7.apply(MessageServer.scala:107) ~[classes/:na]
at grizzled.slf4j.Logger.info(slf4j.scala:128) ~[grizzled-slf4j_2.10-1.0.1.jar:1.0.1]
at grizzled.slf4j.Logging$class.info(slf4j.scala:268) ~[grizzled-slf4j_2.10-1.0.1.jar:1.0.1]
at com.walmartlabs.mupd8.MessageServer.info(MessageServer.scala:44) ~[classes/:na]
at com.walmartlabs.mupd8.MessageServer.run(MessageServer.scala:107) ~[classes/:na]

Hello,
I downloaded mupd8 and did everything in the quickstart tutorial which worked like a charm. Thanks for that. Now I want to continue and execute the example in a cluster environment, but I couldn't find any tutorials for that. So I don't know what I have to change in the config files or in the program itself.
Can anybody help me with this issue?!

I'm running a mupd8 application on 25 nodes for testing purposes at the moment and I recognized that all map instances are scheduled to only one of these 25 nodes. This results in very poor performance, because my map task are very CPU intensive. While most of the other nodes are executing update tasks, some of them aren't executing anything. I cannot find any reasons for that. Also, no exceptions appear in my logging files.

Further details about my application:
-the key of each tuple is selected randomly
-each node executes mupd8 with 4 worker threads

If you need more details, just let me know.

1432eb6

Copy pasting the comment here

This and the previous Cassandra writer change to the background writer thread creates a regression. The old code would throttle writes to maintain a steady speed of writes to the backing database. These changes remove the sleep after every write and put it after all the writes have gone in together. This is a bug that can overload the database. Please revert to the earlier behavior.

Please allow Interrupted signal as kill signal for Source thread. In my opinion, it should.

Thanks,
Bhavesh

21 Jan 2015 16:19:37,399 ERROR SourceReader:request_source:dare-msgq00.sv.walmartlabs.com:9091,dare-msgq01.sv.walmartlabs.com:9091,dare-msgq02.sv.walmartlabs.com:9091:mupd_statsd__metric_events:logmon.heartbeat:32:KafkaSource_HBs - SourceThread: exception during reads. Swallowed to continue next read.
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:63)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
at com.walmart.platform.logevent.common.source.AbstractBaseMupd8KafkaSource.hasNext(AbstractBaseMupd8KafkaSource.java:51)
at com.walmartlabs.mupd8.AppRuntime$SourceThread$1$$anonfun$run$1.apply$mcV$sp(AppRuntime.scala:424)
at scala.util.control.Breaks.breakable(Breaks.scala:37)
at com.walmartlabs.mupd8.AppRuntime$SourceThread$1.run(AppRuntime.scala:418)
at java.lang.Thread.run(Thread.java:744)
21 Jan 2015 16:19:37,401 ERROR SourceReader:request_source:KafkaSource_HBs:14:logmon.heartbeat:mupd_statsd__metric_events - SourceThread: exception during reads. Swallowed to continue next read.

Hello all,

I am trying mupd8 application on a cluster and its working fine with the example given.

What I would like to do, is to have the data source of the application streamed from a different host. The stream of packets contains string type variables. Could someone provide me with example because I tried to edit the configuration file and specify a host that is sending stream of data but it didn't work.

Thanks