Support protobuf in source and sink about siddhi-io-nats

[BuddhiWathsala]

@KedareshwarChaudhari and @rushilV, the NATS extension and protobuf extension have been released with protobuf communication support. Please test your use cases and let us know about the feature.

You can find out released dependencies from the following releases.

https://github.com/siddhi-io/siddhi-map-protobuf/releases/tag/v1.0.2
https://github.com/siddhi-io/siddhi-io-nats/releases/tag/v2.0.7

[BuddhiWathsala]

I reproduced the issue and currently working on this one. We have to support the protobuf communication directly through the siddhi-io-nats level as you mentioned.

[rushilV]

@BuddhiWathsala thanks, by using siddhi-tooling 5.1.0 those exception and nats source extension not available error are not coming anymore, am using siddhi-io-nats and siddhi-map-protobuf with latest release

[rushilV]

@BuddhiWathsala when can we expect this feature(siddhi-io-nats with protobuf) to complete any specific release date

[BuddhiWathsala]

@rushilV we have identified the issue and we are fixing that. In the worst case, we can release it next week(31/10/19).

If I give more details about this, fix needs to be handled at both siddhi-map-protobuf and siddhi-io-nats. At the moment we are in the progress of improving them. Will update the progress in the thread.

[rushilV]

@BuddhiWathsala Thanks for the update

[kedarac]

@BuddhiWathsala Great Work. Sure will test and update you on the status.

[rushilV]

@BuddhiWathsala Thanks for the support

[rushilV]

@BuddhiWathsala am using "siddhi-tooling-0.1.0" after replacing the siddhi-io-nats and siddhi-map-proobuf with the latest release in lib folder where all the supporting jar files are maintaining am getting an exception also in the code its showing an error "No extension exist for source: Nats"
Attaching the screen shots of the same

[BuddhiWathsala]

First of all, the thing is the Siddhi tooling 0.1.0 is a minor release and I would recommend using the latest v5.1.0 pack if you do not have any specific reason to use that older version.

Second thing is that according to the stack trace, the issue might be that the tooling pack unable to find the siddhi-io-nats JAR. Which means the NATS jar you put might not be recognizable. Did you correctly replace the new JAR from the previous one?. That means you should rename the new one as the previous JAR name and put it into the same <SIDDHI-TOOLING-HOME>/lib directory.

[olsharpe]

Hi @BuddhiWathsala, not sure if I'm missing something but struggling to get this to work using autogen class io.siddhi.extension.map.protobuf.autogenerated.Request

I am running the latest 5.1.2 for both runner and tooling.

• java-nats-streaming-2.2.3
• protobuf-java-3.9.1
• jnats-2.6.5
• siddhi-io-nats-2.0.10
• siddhi-map-protobuf_1.0.4

My test app looks like

@App:name("EMSStream")

@App:description("App consumes events from NATS as a protobuf message of SenML format and inserts the events into EMSStream")
/*
Sample message
 {"n":"temperature1","u":"Cel","v":13.6,"bt":1.5782427e+09},{"n":"batteryLevel","u":"V","v":5},{"n":"signalStrength","u":"%EL","v":72}
*/

@info(name='Input stream from Nats + Protobuf publisher')
@source(type='nats',cluster.id='test-cluster', bootstrap.servers='nats://nats:4222', @map(type='protobuf', class='io.siddhi.extension.map.protobuf.autogenerated.Request'))
define stream InputStream (channel string, subtopic string, publisher string, protocol string, contentType string);

@sink(type='log') 
define stream OutputStream (channel string, publisher string);

@info(name='passThrough')
from InputStream
select *
insert into OutputStream;

Error

[2020-01-05 21:16:09,458] ERROR {io.siddhi.distribution.editor.core.internal.DebugRuntime} - Siddhi App 'EMSStream' is in faulty state. io.siddhi.core.exception.SiddhiAppCreationException: Error on 'EMSStream' @ Line: 11. Position: 171, near '@source(type='nats',cluster.id='test-cluster', bootstrap.servers='nats://nats:4222', @map(type='protobuf', class='io.siddhi.extension.map.protobuf.autogenerated.Request'))'. EMSStream:InputStream: Invalid class name provided in the 'class' parameter, provided class name: io.siddhi.extension.map.protobuf.autogenerated.Request,io.siddhi.extension.map.protobuf.autogenerated.Request cannot be found by siddhi-map-protobuf_1.0.4

Not sure if you have any ideas how I can resolve this

[BuddhiWathsala]

This error indicates that the Siddhi runtime couldn’t find the protobuf JAR that you generated.

@olsharpe, have you put your generated protobuf JAR into <SIDDHI_HOME>/jars directory and restart the server?

[olsharpe]

@BuddhiWathsala aah ok, I forgot to add them to tooling. Seems to work now. Thanks for support

[BuddhiWathsala]

@olsharpe if you need further clarifications please refer to our slack channel here.

[javard]

@BuddhiWathsala using siddhi-tooling-5.1.0, I've faced a similar error but using tcp as a source. :

source 'tcp' produces incompatible '[class [B]' classes, while it's source mapper 'protobuf' can only consume '[class com.google.protobuf.GeneratedMessageV3]' classes