Comments (11)
Hi Sami,
I want to confirm your question before I try to find the answer. So same
data source reader already works in node same as mupd8, and now you want to
put this source started on a node other than node running mupd8? If this is
what you want, you can start your data source on the node you like, say
"node1", and replace "mupd8::sources[host]" field in config file with
"node1".
Thanks,
Wei
Wei Sun | 650.556.3886
On Mon, Jul 29, 2013 at 1:44 AM, sami-misa [email protected] wrote:
Hello all,
I am trying mupd8 application on a cluster and its working fine with the
example given.What I would like to do, is to have the data source of the application
streamed from a different host. The stream of packets contains string type
variables. Could someone provide me with example because I tried to edit
the configuration file and specify a host that is sending stream of data
but it didn't work.Thanks
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41
.
from mupd8.
Hi
Thanks for the reply. My problem is that i am not able to change the source of the stream.In the example given in mupd8 , the stream of the application is generated locally from a specified file . But what I wantt to do is to change the source of the stream .I want to have a client application that sends a stream to mupd8 Application. What I have tried to do is to configure the client application to connect using to mupd8 application using sockets and changed the corresponding host and port entries in the application configuration file.Unfortunately it didn't work.
Can you please tell me what I am doing wrong ?
from mupd8.
Here is an example of source config of a socket source. Please let me know
if you have problem understanding it.
Note: "com.walmartlabs.mupd8.JSONSource" only accepts stream each line of
which is a JSON object. If you want to use something other than JSON as
input you need to implement your own class extending Mupd8Source defined
in ./src/main/java/com/walmartlabs/mupd8/application/Mupd8Source.java to
replace JSONSource in config.
"sources" : [
{
"name" : "source1",
"source" : "com.walmartlabs.mupd8.JSONSource",
"host" : "mtest0",
"performer" : "T10Source",
"parameters" : [ "mtest0:1234", "k1" ]
}
]
-Wei
On Tue, Jul 30, 2013 at 1:39 AM, sami-misa [email protected] wrote:
Hi
Thanks for the reply. My problem is that i am not able to change the
source of the stream.In the example given in mupd8 , the stream of the
application is generated locally from a specified file . But what I wantt
to do is to change the source of the stream .I want to have a client
application that sends a stream to mupd8 Application. What I have tried to
do is to configure the client application to connect using to mupd8
application using sockets and changed the corresponding host and port
entries in the application configuration file.Unfortunately it didn't work.
Can you please tell me what I am doing wrong ?—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41#issuecomment-21776917
.
from mupd8.
Hello again,
I have tried the configuration that you have provided unfortunately it doesn't work.
Mupd8 is not giving any error message and says that it is connected to the source stream but actually their is not connection to the source stream as I am not seeing any connection established when I use netstat -vatn .Furthermore the application itself is not running and there is no resultant output.
from mupd8.
Hi,
I am a newbie to mupd8 and I want to configure mupd8 to get an external source stream from a client machine , I tried to configure the IP and port address as given , but that didn't work out for me.
I really need to do it and I can not afford getting the stream from a local file . The packets that my client wants to send contain JSON objects.
Can any one help ?
thanks
from mupd8.
Following is a workable example "sources" array in config file,
{"sources" : [
{
"name" : "source1",
"source" : "com.walmartlabs.mupd8.JSONSource",
"host" : "mtest0",
"performer" : "T10Source",
"parameters" : [ "mtest0:1234", "k1" ]
}
]}
In order to make external socket source work, your data stream first needs
to stream of text line and each line must be a json object. And in
"parameters" array you can put "external host:port" as first parameter and
"key path" to key as second parameter. Also "host" is the hostname or ip
which is going to start this source reader.
If content of your external source is NOT json object at each line, then
you need to implement "Mupd8Source" interface to support your own data
format.
-Wei
On Wed, Aug 7, 2013 at 5:37 AM, mimi-91 [email protected] wrote:
Hi,
I am a newbie to mupd8 and I want to configure mupd8 to get an external
source stream from a client machine , I tried to configure the IP and port
address as given , but that didn't work out for me.I really need to do it and I can not afford getting the stream from a
local file . The packets that my client wants to send contain JSON objects.Can any one help ?
thanks—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41#issuecomment-22247825
.
from mupd8.
Hi,
First of all, thanks for offering this wonderful tool/framework.
My question is: what if I want to extract two or more keys from JSON source stream, e.g. "request:requestUrl" and "response:responseHeaderList:Content-Type", do we support this? and how?
My 2nd question is: for the key, can key be "request:requestHeaderList:Cookie:com.wm.visitor", given the following JSON definition:
{
"request" : {
"protocol" : "HTTP/1.1",
"requestUrl" : "/beacon.gif",
"remotePort" : 5949,
"requestHeaderList" : [
{
"valueList" : [
"SSID=BQA7PxsOAAAAAADIT3ZP13MCHMhPdk8BAAAAAAD03Z1PyE92TwAiAQADrScAAMhPdk8BAA; SSSC=2.G5725851697130140631.1|290.10157; com.wm.visitor=20543076174"
],
"name" : "Cookie"
},
.................
Thanks,
Shenghu
from mupd8.
No, there is no existing way to use object inside array as key now. But you
can implement your own soruce reader to do this. Every source reader needs
to inherit "Mupd8Source" interface.
-Wei
On Wed, Aug 7, 2013 at 10:53 AM, syang0 [email protected] wrote:
Hi,
First of all, thanks for offering this wonderful tool/framework.
My question is: what if I want to extract two or more keys from JSON
source stream, e.g. "request:requestUrl" and
"response:responseHeaderList:Content-Type", do we support this? and how?
My 2nd question is: for the key, can key be
"request:requestHeaderList:Cookie:com.wm.visitor", given the following JSON
definition:{
"request" : {
"protocol" : "HTTP/1.1",
"requestUrl" : "/beacon.gif",
"remotePort" : 5949,
"requestHeaderList" : [
{
"valueList" : [
"SSID=BQA7PxsOAAAAAADIT3ZP13MCHMhPdk8BAAAAAAD03Z1PyE92TwAiAQADrScAAMhPdk8BAA;
SSSC=2.G5725851697130140631.1|290.10157; com.wm.visitor=20543076174"
],
"name" : "Cookie"
},.................
Thanks,
Shenghu—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41#issuecomment-22270171
.
from mupd8.
Thanks Wei.
How about my other question: what if I want to extract two or more keys from JSON source stream, e.g. "request:requestUrl" and "response:responseHeaderList:Content-Type", do we support this? and how?
Regards,
Shenghu
from mupd8.
oops, I missed this question.
Regarding this question, the answer is same. :-) Existing source reader
doesn't support this, but you can add this support into you created source
reader.
-Wei
On Wed, Aug 7, 2013 at 2:40 PM, syang0 [email protected] wrote:
Thanks Wei.
How about my other question: what if I want to extract two or more keys
from JSON source stream, e.g. "request:requestUrl" and
"response:responseHeaderList:Content-Type", do we support this? and how?Regards,
Shenghu—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41#issuecomment-22286438
.
from mupd8.
Hi--just to catch up, I'd add one more remark: Besides trying to overload everything into the source reader, it may be simpler and more powerful to write a simple Mapper to perform the extra steps you desire.
For example, a new Mapper can take its input event, parse out the desired (two in your case above) keys from the JSON, and emit a separate event for each key. As a bonus side effect, the Mapper can be written to emit the events to different streams, so that different recipient Mappers and Updaters can subscribe to each kind of event as desired.
from mupd8.
Related Issues (16)
- Enable source reader throttling on hot conductor backlogs HOT 1
- what happens after source reader fails to connect to a socket?
- Specify the number of task instances HOT 7
- JSONSource: exception in the socket reader initialization results in readLine() false-negative HOT 2
- Writes to the backing database are batched, instead of being throttled HOT 2
- Load is distributed unequally between nodes HOT 8
- Example documentation broken: OS 10.9 and compilation errors HOT 3
- Tutorial example not working HOT 5
- NPE Please FIX it. HOT 2
- NPE in MessageServer Server Thread
- java.lang.InterruptedException Should it kill the source thread or not ?
- NPE Again
- NPE when start-up followed by unclean shutdown HOT 3
- Example for setup in a cluster HOT 7
- Move 'org.codehaus' to 'com.fasterxml'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mupd8.