siddhi-io / siddhi-io-nats

Extension that can be used to communicate (publish/subscribe) with an instance of the nats-streaming server

Siddhi IO NATS

The siddhi-io-nats extension is an extension to Siddhi that receives and publishes events from and to NATS.

For information on Siddhi and it's features refer Siddhi Documentation.

Download

• Versions 2.x and above with group id io.siddhi.extension.* from here.
• Versions 1.x and lower with group id org.wso2.extension.siddhi.* from here.

Latest API Docs

Latest API Docs is 2.0.15.

Features

• nats (Sink)
  

  NATS Sink allows users to subscribe to a Nats or Nats streaming broker and publish messages.

• nats (Source)
  

  NATS Source allows users to subscribe to a NATS broker and receive messages. It has the ability to receive all the message types supported by NATS.

Dependencies

Add following jars when using nats,

• jnats-2.6.5.jar (Add to {SIDDHI_HOME}/jars)
• protobuf-java-3.9.1.jar (Add to {SIDDHI_HOME}/bundles)

Add following jars when using nats streaming

• jnats-2.6.5.jar (Add to {SIDDHI_HOME}/jars)
• java-nats-streaming-2.2.2.jar (Add to {SIDDHI_HOME}/jars)
• protobuf-java-3.9.1.jar (Add to {SIDDHI_HOME}/bundles)

Installation

For installing this extension on various siddhi execution environments refer Siddhi documentation section on adding extensions.

Support and Contribution

• We encourage users to ask questions and get support via StackOverflow, make sure to add the siddhi tag to the issue for better response.

• If you find any issues related to the extension please report them on the issue tracker.

• For production support and other contribution related information refer Siddhi Community documentation.

siddhi-io-nats's Issues

Cannot load sink

Description:
Cannot load NATS sink:

[2024-03-23 11:43:35,302] ERROR {org.wso2.carbon.streaming.integrator.core.internal.StreamProcessorService} - Exception occurred when validating Siddhi App 65a9511f5d9ec417a7fa9a08 io.siddhi.core.exception.SiddhiAppCreationException: Error on '65a9511f5d9ec417a7fa9a08' @ Line: 57. Position: 1, near '@sink(type='nats', 
  destination='...',
  server.urls='...',
  @map(type = 'json', event.grouping.enabled='false')
)'. org/apache/logging/log4j/LogManager
	at io.siddhi.core.util.ExceptionUtil.populateQueryContext(ExceptionUtil.java:58)
	at io.siddhi.core.util.ExceptionUtil.populateQueryContext(ExceptionUtil.java:35)
	at io.siddhi.core.util.parser.helper.DefinitionParserHelper.addEventSink(DefinitionParserHelper.java:588)
	at io.siddhi.core.util.SiddhiAppRuntimeBuilder.defineStream(SiddhiAppRuntimeBuilder.java:119)
	at io.siddhi.core.util.parser.SiddhiAppParser.defineStreamDefinitions(SiddhiAppParser.java:375)
	at io.siddhi.core.util.parser.SiddhiAppParser.parse(SiddhiAppParser.java:231)
	at io.siddhi.core.SiddhiManager.createSiddhiAppRuntime(SiddhiManager.java:87)
	at io.siddhi.core.SiddhiManager.createSiddhiAppRuntime(SiddhiManager.java:97)
	at org.wso2.carbon.streaming.integrator.core.internal.StreamProcessorService.validateAndSave(StreamProcessorService.java:280)
	at org.wso2.carbon.streaming.integrator.core.impl.SiddhiAppsApiServiceImpl.siddhiAppsPut(SiddhiAppsApiServiceImpl.java:136)
	at org.wso2.carbon.streaming.integrator.core.impl.SiddhiAppsApiServiceImpl.siddhiAppsPut(SiddhiAppsApiServiceImpl.java:910)
	at org.wso2.carbon.streaming.integrator.core.api.SiddhiAppsApi.siddhiAppsPut(SiddhiAppsApi.java:104)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.wso2.msf4j.internal.router.HttpMethodInfo.invokeResource(HttpMethodInfo.java:187)
	at org.wso2.msf4j.internal.router.HttpMethodInfo.invoke(HttpMethodInfo.java:143)
	at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.dispatchMethod(MSF4JHttpConnectorListener.java:218)
	at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.lambda$onMessage$58(MSF4JHttpConnectorListener.java:129)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:830)
Caused by: java.lang.NoClassDefFoundError: org/apache/logging/log4j/LogManager
	at io.siddhi.extension.io.nats.sink.nats.NATSCore.<clinit>(NATSCore.java:45)
	at io.siddhi.extension.io.nats.sink.NATSSink.init(NATSSink.java:233)
	at io.siddhi.core.stream.output.sink.Sink.init(Sink.java:111)
	at io.siddhi.core.util.parser.helper.DefinitionParserHelper.addEventSink(DefinitionParserHelper.java:549)
	... 20 more
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.LogManager cannot be found by siddhi-io-nats_2.0.14
	at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:512)
	at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:423)
	at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:415)
	at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:155)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 24 more

Suggested Labels:

Suggested Assignees:

Affected Product Version:

2.0.14

OS, DB, other environment details and versions:

Linux

Steps to reproduce:

Just create a simple sink of type mats

Related Issues:

Support protobuf in source and sink

Description:
Currently, the nats extension for siddhi only supports String.class Line in code, unlike gRPC extension which supports String.class and GeneratedMessageV3.class Line in code. Since nats and nats-streaming both support protobuf messages as payload, this extension throws an exception when trying to send protobuf messages to siddhi via nats, like:

@source(type='nats', destination='channel1',@Map(type='protobuf', class='com.example.InputStream'))
define stream inputStream (timeStamp long, data int)

Exception:

ERROR {io.siddhi.distribution.core.internal.ServiceComponent} - Exception occurred when deploying the Siddhi AppTEST.siddhi io.siddhi.distribution.core.internal.exception.SiddhiAppDeploymentException: io.siddhi.core.exception.SiddhiAppCreationException: Error on 'TEST' @ Line: 6. Position: 110, near '@source(type='nats', destination='channel1',@Map(type='protobuf', class='com.example.InputStream'))'. At stream 'inputStream', source 'nats' produces incompatible '[class java.lang.String, interface java.util.Map]' classes, while it's source mapper 'protobuf' can only consume '[class com.google.protobuf.GeneratedMessageV3]' classes.

Caused by: io.siddhi.core.exception.SiddhiAppCreationException: Error on 'TEST' @ Line: 6. Position: 110, near '@source(type='nats', destination='channel1',@Map(type='protobuf', class='com.example.InputStream'))'. At stream 'inputStream', source 'nats' produces incompatible '[class java.lang.String, interface java.util.Map]' classes, while it's source mapper 'protobuf' can only consume '[class com.google.protobuf.GeneratedMessageV3]' classes.
at io.siddhi.core.util.parser.helper.DefinitionParserHelper.validateSourceMapperCompatibility(DefinitionParserHelper.java:422)
at io.siddhi.core.util.parser.helper.DefinitionParserHelper.addEventSource(DefinitionParserHelper.java:357)
at io.siddhi.core.util.SiddhiAppRuntimeBuilder.defineStream(SiddhiAppRuntimeBuilder.java:114)
at io.siddhi.core.util.parser.SiddhiAppParser.defineStreamDefinitions(SiddhiAppParser.java:373)
at io.siddhi.core.util.parser.SiddhiAppParser.parse(SiddhiAppParser.java:229)
at io.siddhi.core.SiddhiManager.createSiddhiAppRuntime(SiddhiManager.java:66)
at io.siddhi.core.SiddhiManager.createSiddhiAppRuntime(SiddhiManager.java:76)
at io.siddhi.distribution.core.internal.StreamProcessorService.deploySiddhiApp(StreamProcessorService.java:69)
at io.siddhi.distribution.core.internal.StreamProcessorDeployer.deploySiddhiQLFile(StreamProcessorDeployer.java:92)
... 55 more

Suggested Labels:
type/improvement

Suggested Assignees:
@pcnfernando

Affected Product Version:
Siddhi v5.1.0
siddhi-io-nats v 2.0.6 extension

OS, DB, other environment details and versions:
Ubuntu 18.04 LTS
siddhi-runner v0.1.0
JDK 1.8.0-openjdk-amd64

Steps to reproduce:
Run TEST.siddhi using:

TEST.siddhi

@app:name("TEST")
@app:description("Test implementation for Nats and Protobuf in siddhi")
@info(name='Input stream from Nats + Protobuf publisher')
@source(type='nats', destination='channel1',@Map(type='protobuf', class='com.example.InputStream'))
define stream inputStream (timeStamp long, data int)

InputStream.proto

syntax = "proto3";
option java_multiple_files = true;
option java_package = "com.example";
message InputStream {
uint64 timeStamp = 1;
uint32 data = 2;
}

1. Generate Java sources from the InputStream.proto file

protoc InputStream.proto --java_out=./java-sources/

2. Compile Java Sources to Java Classes

javac -cp ".:/home/$USER/siddhi-runner-0.1.0/bin/bootstrap/protobuf-java-3.9.1.jar" ./java-sources/com/example/* -d ./java-classes/

Note: protobuf-java-3.9.1.jar is passed to javac classpath for java source files to be compiled.

3. Create jar file from java-classes

jar cvf com.example.jar -C java-classes .

4. Copy the generated jar file to the siddhi runner classpath, I was not able to load from /jars, so I copied to siddhi-runner-0.1.0/bin/bootstrap/com.example.jar (This is not the recommended way, maybe a separate issue)

5. Run siddhi:

./siddhi-runner-0.1.0/bin/runner.sh -Dapps=/path/to/TEST.siddhi

Will provide more details if needed.

Related Issues:

Passing NATS Streaming Sequence Number within Siddhi Event

Description:
At the moment, Siddhi NATS source uses sequence number to subscribe to a subject from a given number of message sequence. All the messages from the given point of sequence number will be consumed by the NATS source. Siddhi Nats Source uses this support for state persistence/restore.
But, the sequence number is not being passed as an event attribute.

There's a requirement to use the NATS Steaming event's sequence number for Siddhi Stream Joins.
To facilitate this, we can export the sequence number as a transport property so that it can be used within the subsequent processes.

Reference: https://siddhi-io.github.io/siddhi-io-nats/api/2.0.7/#nats-source

It is not possible to use NATSCore because CLUSTER_ID has got a default value

Description:
It is not possible to use NATSCore because CLUSTER_ID has got a default value

Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Support Message Acknowledgements and Redelivery with Siddhi-io-nats

Description:
Once a message has been consumed, if an acknowledgement is not received within the configured timeout interval, NATS Streaming will attempt redelivery of the message.
This timeout interval is specified by the subscription option AckWait.
Need to support configuring this through Siddhi nats source.

Suport Rate limiting/matching within Sink and Source

Description:
Need to support handling mismatch between publisher and subscriber commonly called a "fast producer/slow consumer".

Siddhi support for NATS

Hi,
Currently am using siddhi-runner-5.1.0 in linux environment with NATS-streaming.
The problem is NATS-streaming is adding the huge delay between the message publish almost 3-4 ms and my application processing 991 data points and its adding more over-head.
So can you please provide a support for NATS in siddhi instead of NATS-Streaming.

Durable subscription clientID already registered when server crashes

Description:
Here I tried to use NATS durable subscription. NATS durable subscription takes durable.name and client.id as a kind of unique identification(Ref).

In my scenario, the server that siddhi-runner resides crashes unexpectedly and reconcile it after a while. But in the reconciliation, the siddhi-io-nats has thrown an error like below.

[2019-06-06 05:52:20,096] ERROR {io.siddhi.core.stream.input.source.Source} - Error on 'NatsSource-912f17f9-5d61-41e2-aea4-b35021889a1c-1'. Error while connecting to NATS server at destination: MonitorApp_DevicePowerStream Error while connecting at Source 'nats' at 'DevicePowerStream'. Will retry in '5 sec'. io.siddhi.core.exception.ConnectionUnavailableException: Error while connecting to NATS server at destination: MonitorApp_DevicePowerStream
	at io.siddhi.extension.io.nats.source.NATSSource.connect(NATSSource.java:183)
	at io.siddhi.extension.io.nats.source.NATSSource.connect(NATSSource.java:51)
	at io.siddhi.core.stream.input.source.Source.connectWithRetry(Source.java:159)
	at io.siddhi.core.SiddhiAppRuntime.startSources(SiddhiAppRuntime.java:452)
	at io.siddhi.core.SiddhiAppRuntime.start(SiddhiAppRuntime.java:383)
	at io.siddhi.distribution.core.internal.StreamProcessorService.deploySiddhiApp(StreamProcessorService.java:106)
	at io.siddhi.distribution.core.internal.StreamProcessorDeployer.deploySiddhiQLFile(StreamProcessorDeployer.java:92)
	at io.siddhi.distribution.core.internal.StreamProcessorDeployer.deploy(StreamProcessorDeployer.java:292)
	at org.wso2.carbon.deployment.engine.internal.DeploymentEngine.lambda$deployArtifacts$0(DeploymentEngine.java:291)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.wso2.carbon.deployment.engine.internal.DeploymentEngine.deployArtifacts(DeploymentEngine.java:282)
	at org.wso2.carbon.deployment.engine.internal.RepositoryScanner.sweep(RepositoryScanner.java:112)
	at org.wso2.carbon.deployment.engine.internal.RepositoryScanner.scan(RepositoryScanner.java:68)
	at org.wso2.carbon.deployment.engine.internal.DeploymentEngine.start(DeploymentEngine.java:121)
	at org.wso2.carbon.deployment.engine.internal.DeploymentEngineListenerComponent.onAllRequiredCapabilitiesAvailable(DeploymentEngineListenerComponent.java:216)
	at org.wso2.carbon.kernel.internal.startupresolver.StartupComponentManager.lambda$notifySatisfiableComponents$7(StartupComponentManager.java:266)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.wso2.carbon.kernel.internal.startupresolver.StartupComponentManager.notifySatisfiableComponents(StartupComponentManager.java:252)
	at org.wso2.carbon.kernel.internal.startupresolver.StartupOrderResolver$1.run(StartupOrderResolver.java:204)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
Caused by: java.io.IOException: stan: clientID already registered
	at io.nats.streaming.StreamingConnectionImpl.connect(StreamingConnectionImpl.java:162)
	at io.nats.streaming.StreamingConnectionFactory.createConnection(StreamingConnectionFactory.java:54)
	at io.siddhi.extension.io.nats.source.NATSSource.connect(NATSSource.java:180)
	... 20 more

The reason for this error was when server crashes the NATS durable subscription wasn't clean up properly. This was the issue arised in NATS.

Related Issues:
nats-io/nats-streaming-server#759

Incorrect distributed behavior with Nats

Description:
When we use distributed sink capability with any distribution strategy, Nats sink publishes to the same topic.

Issue with NATS client jars

Description:
The version of jar files put into SiddhiHome/jars have issues.

Affected Product Version:
java-nats-streaming-2.2.2.jar (Add to <SIDDHI_HOME>/jars) , jnats-2.6.5.jar (Add to <SIDDHI_HOME>/jars). When these 2 jars are used it gives an error.

But when using java-nats-streaming-2.2.3.jar , jnats-2.6.0.jar the program works fine.

Stacktrace of the error:
[2019-10-04 19:17:13,820] ERROR {io.siddhi.distribution.editor.core.internal.DebugRuntime} - Siddhi App 'PublishNatsinTextFormat' is in faulty state. io.siddhi.core.exception.SiddhiAppCreationException: Error on 'PublishNatsinTextFormat' @ Line: 51. Position: 21, near '@sink( type = 'nats', cluster.id = 'test-cluster', destination = 'SweetProduction', bootstrap.servers = 'nats://localhost:4222', @map(type = 'json'))'. io/nats/streaming/ConnectionLostHandler at io.siddhi.core.util.ExceptionUtil.populateQueryContext(ExceptionUtil.java:43) at io.siddhi.core.util.parser.helper.DefinitionParserHelper.addEventSink(DefinitionParserHelper.java:588) at io.siddhi.core.util.SiddhiAppRuntimeBuilder.defineStream(SiddhiAppRuntimeBuilder.java:119) at io.siddhi.core.util.parser.SiddhiAppParser.defineStreamDefinitions(SiddhiAppParser.java:374) at io.siddhi.core.util.parser.SiddhiAppParser.parse(SiddhiAppParser.java:230) at io.siddhi.core.SiddhiManager.createSiddhiAppRuntime(SiddhiManager.java:85) at io.siddhi.core.SiddhiManager.createSiddhiAppRuntime(SiddhiManager.java:95) at io.siddhi.distribution.editor.core.internal.DebugRuntime.createRuntime(DebugRuntime.java:202) at io.siddhi.distribution.editor.core.internal.DebugRuntime.<init>(DebugRuntime.java:56) at io.siddhi.distribution.editor.core.internal.DebugProcessorService.start(DebugProcessorService.java:38) at io.siddhi.distribution.editor.core.internal.EditorMicroservice.start(EditorMicroservice.java:759) at io.siddhi.distribution.editor.core.internal.EditorMicroservice.startWithVariables(EditorMicroservice.java:779) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.wso2.msf4j.internal.router.HttpMethodInfo.invokeResource(HttpMethodInfo.java:187) at org.wso2.msf4j.internal.router.HttpMethodInfo.invoke(HttpMethodInfo.java:143) at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.dispatchMethod(MSF4JHttpConnectorListener.java:218) at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.lambda$onMessage$58(MSF4JHttpConnectorListener.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: io/nats/streaming/ConnectionLostHandler at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) at java.lang.Class.getConstructor0(Class.java:3075) at java.lang.Class.newInstance(Class.java:412) at io.siddhi.core.util.SiddhiClassLoader.loadClass(SiddhiClassLoader.java:32) at io.siddhi.core.util.SiddhiClassLoader.loadExtensionImplementation(SiddhiClassLoader.java:48) at io.siddhi.core.util.parser.helper.DefinitionParserHelper.createSink(DefinitionParserHelper.java:839) at io.siddhi.core.util.parser.helper.DefinitionParserHelper.addEventSink(DefinitionParserHelper.java:488) ... 21 more Caused by: java.lang.ClassNotFoundException: io.nats.streaming.ConnectionLostHandler cannot be found by siddhi-io-nats_2.0.6 at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:398) at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:361) at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:353) at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:161) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 29 more