siddhi-io / siddhi-map-avro Goto Github PK
View Code? Open in Web Editor NEWExtension which is used to convert Avro message to/from Siddhi events
Home Page: https://siddhi-io.github.io/siddhi-map-avro/
License: Apache License 2.0
Extension which is used to convert Avro message to/from Siddhi events
Home Page: https://siddhi-io.github.io/siddhi-map-avro/
License: Apache License 2.0
Description:
Switching from design view to source view after changing any element causes the conversion of triple-quotes into single ones in the avro scheme definition.
Affected Product Version:
v1.0.67
Description:
The Avro Map extension is often referenced in conjunction with Siddhi's extnesion for Kafka. Unfortunately none of the available examples for using these together seem to work!
The culprit appears to be that Confluent's support for Avro in Kafka utiilizes an interaction with its Schema Registry to assign a schema ID to each unique schema it finds, and prefixes every message it stores in Kafka with its schema ID in a 5-byte prefix just before every Avro-seraiilized message follows. Those five bytes are not stripped away before this mapper is invoked, and the extra 5 byte header throws the binary decoding off kilter.
It is possible to reproduce the same incorrect decoding by taking a message from an Avro-enabled topic and get the identical wrong result just by retaining the 5-byte header, and then also to cure the problem by removing those five out-of-band header bytes.
Affected Product Version:
2.0.6
OS, DB, other environment details and versions:
MacOS Mojave (10.14.4) running with Kafka 2.3.1's Docker Images on Docker Desktop Community 2.1.0.4
Steps to reproduce:
{"namespace":"sample.avro","type":"record","name":"avro","fields":[{"name":"name","type":"string"},{"name":"amount","type":"double"}]}
sample.json
:{"name":"exnobhbyaitkfiayoyipianxshgkofnmpkef","amount":0.9629494268310558}
version: '3.6'
services:
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
ports:
- 2181:2181
- 2888:2888
- 3888:3888
environment:
- ZOOKEEPER_CLIENT_PORT=2181
- ZOOKEEPER_SERVER_ID=1
- ZOOKEEPER_TICK_TIME=2000
- ZOOKEEPER_SYNC_LIMIT=2
kafka:
image: confluentinc/cp-kafka:5.3.1
ports:
- 9092:9092
links:
- zookeeper
environment:
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_KAFKA_HOST_NAME=kafka
- KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
- KAFKA_BROKER_ID=1
- KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:19092,LISTENER_HOST://0.0.0.0:9092
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:19092,LISTENER_HOST://localhost:9092
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,LISTENER_HOST:PLAINTEXT
- KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT
schema-registry:
image: confluentinc/cp-schema-registry:5.3.1
ports:
- 8081:8081
environment:
- SCHEMA_REGISTRY_HOST_NAME=schema-registry
- SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS=PLAINTEXT://kafka:19092
- SCHEMA_REGISTRY_LISTENERS=http://schema-registry:8081
- SCHEMA_REGISTRY_DEBUG=false
links:
- zookeeper
- kafka
cat sample.json | kafka-avro-console-producer --topic "kafka_result_topic" --broker-list localhost:9092 --property schema.registry.url=http://localhost:8081 --property value.schema=`cat sample.avsc`
@App:name('KafkaSample')
@sink(type='log')
define stream logStream(name string, amount double);
@source(type='kafka', bootstrap.servers='localhost:9092', topic.list='kafka_result_topic', group.id='test', threading.option='single.threaded', is.binary.message='true', @map(type='avro', schema.def = '{"namespace":"sample.avro","type":"record","name":"avro","fields":[{"name":"name","type":"string"},{"name":"amount","type":"double"}]}'
define stream LowProductionAlertStream(name string, amount double);
from LowProductionAlertStream
select *
insert into logStream;
[java] [io.siddhi.core.stream.output.sink.LogSink] : KafkaSample : logStream : Event{timestamp=1573734234725, data=[, 1.410940917531979E224], isExpired=false}
kafka-console-consumer --topic kafka_result_topic --offset 0 --partition 0 --max-messages 1 --bootstrap-server localhost:9092 > onemsg.dat
avro-tools fragtojson --schema-file sample.avsc onemsg.dat
old_sz=$(echo `wc -c onemsg.dat | awk '{print $1}'`)
new_sz = $(echo "$old_sz - 5" | bc)
sh -c "tail -${s}c onemsg.dat" > trunc.dat
avro-tools fragtojson --schema-file sample.avsc onemsg.dat
Observe that this time the 5-byte-shorter message deserialized too our original input was!!
Description:
Following is the stack trace printed at the Siddhi app creation time
[echo] CustomMapping will be overridden if eventDefinition is given.
[echo] If eventDefinition is given, its mandatory to give a file path to the events as well.
[echo] Following mentioned values will be applied if no value is given for optional values.
[echo] Configure
[echo] -bootstrapServerList=localhost:9092(Optional)
[echo] -DtopicName=kafka_sample_topic(Optional)
[echo] -DpartitionNo=(Optional)
[echo] -DsequenceId=(Optional)
[echo] -Dkey=(Optional)
[echo] -DoptionalConfiguration=(Optional)
[echo] -DisBinaryMessage=false(Optional)
[echo] -Dtype=json(Optional) -Ddelay=1000(Optional) -DcustomMapping=false/true(Optional)
[echo] -DeventDefinition='{"customItem":{"address":"{0}","age":{1},"country":"{2}"}}'(Optional)
[echo] -DfilePath=(Optional)
[echo] -DcontinuouslyReadFile=false(optional [if noOfEventsToSend larger than events in file, make it true])
[echo] -DnoOfEventsToSend=5(Optional)
[echo]
[java] [io.siddhi.distribution.sample.kafka.client.KafkaClient] : Initialize kafka producer client.
[java] Exception in thread "main" io.siddhi.core.exception.SiddhiAppCreationException: Error on 'KafkaSink' @ Line: 19. Position: 18, near '@sink(type='kafka',
[java] topic='kafka_avro_test',
[java] bootstrap.servers='localhost:9092',
[java] group.id='group',
[java] is.binary.message='true',
[java] @map(type='avro',schema.def="""
[java] {
[java] "type": "record",
[java] "name": "sweet",
[java] "namespace": "sweet.material",
[java] "fields": [{
[java] "name": "name",
[java] "type": "string"
[java] }, {
[java] "name": "amount",
[java] "type": "double"
[java] }]
[java] }"""))'. feign/FeignException
[java] at io.siddhi.core.util.ExceptionUtil.populateQueryContext(ExceptionUtil.java:43)
[java] at io.siddhi.core.util.parser.helper.DefinitionParserHelper.addEventSink(DefinitionParserHelper.java:588)
[java] at io.siddhi.core.util.SiddhiAppRuntimeBuilder.defineStream(SiddhiAppRuntimeBuilder.java:119)
[java] at io.siddhi.core.util.parser.SiddhiAppParser.defineStreamDefinitions(SiddhiAppParser.java:374)
[java] at io.siddhi.core.util.parser.SiddhiAppParser.parse(SiddhiAppParser.java:230)
[java] at io.siddhi.core.SiddhiManager.createSiddhiAppRuntime(SiddhiManager.java:85)
[java] at io.siddhi.core.SiddhiManager.createSiddhiAppRuntime(SiddhiManager.java:95)
[java] at io.siddhi.distribution.sample.kafka.client.KafkaClient.main(Unknown Source)
[java] Caused by: java.lang.NoClassDefFoundError: feign/FeignException
[java] at java.lang.Class.getDeclaredConstructors0(Native Method)
[java] at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
[java] at java.lang.Class.getConstructor0(Class.java:3075)
[java] at java.lang.Class.newInstance(Class.java:412)
[java] at io.siddhi.core.util.SiddhiClassLoader.loadClass(SiddhiClassLoader.java:32)
[java] at io.siddhi.core.util.SiddhiClassLoader.loadExtensionImplementation(SiddhiClassLoader.java:48)
[java] at io.siddhi.core.util.parser.helper.DefinitionParserHelper.addEventSink(DefinitionParserHelper.java:501)
[java] ... 6 more
[java] Caused by: java.lang.ClassNotFoundException: feign.FeignException
[java] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
[java] at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
[java] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
[java] at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
[java] ... 13 more
[java] Java Result: 1
Suggested Labels:
dependency issue
Affected Product Version:
2.0.2
Description:
I run into problem in my project. I recently implement schema registry to store all my auto generated schema and also use wso2 to read from kafka. But I can only search by schema id. I miss search by name option
Suggested Labels:
There is option in schema registry api to search by name
curl -X GET http://localhost:8081/subjects/<SUBJECT_NAME>/versions/latest
https://docs.confluent.io/platform/current/schema-registry/develop/using.html
Suggested Assignees:
Affected Product Version:
OS, DB, other environment details and versions:
Steps to reproduce:
Related Issues:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.