siddhi-io / siddhi-io-cdc

Extension which consumes CDC events

Siddhi IO CDC

The siddhi-io-cdc extension is an extension to Siddhi that captures change data from databases such as MySQL, MS SQL, PostgreSQL, H2 and Oracle.

For information on Siddhi and it's features refer Siddhi Documentation.

Download

• Versions 3.x and above with group id io.siddhi.extension.* from here.
• Versions 2.x and lower with group id org.wso2.extension.siddhi.* from here.

Latest API Docs

Latest API Docs is 2.0.15.

Features

• cdc (Source)
  

  The CDC source receives events when change events (i.e., INSERT, UPDATE, DELETE) are triggered for a database table. Events are received in the 'key-value' format.
  
  There are two modes you could perform CDC: Listening mode and Polling mode.
  
  In polling mode, the datasource is periodically polled for capturing the changes. The polling period can be configured.
  In polling mode, you can only capture INSERT and UPDATE changes.
  
  On listening mode, the Source will keep listening to the Change Log of the database and notify in case a change has taken place. Here, you are immediately notified about the change, compared to polling mode.
  
  The key values of the map of a CDC change event are as follows.
  
  For 'listening' mode:
      For insert: Keys are specified as columns of the table.
      For delete: Keys are followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.
      For update: Keys are followed followed by the specified table columns. This is achieved via 'before_'. e.g., specifying 'before_X' results in the key being added before the column named 'X'.
  
  For 'polling' mode: Keys are specified as the columns of the table.In order to connect in to the database table for receive CDC events, url, username, password and driverClassName(in polling mode) can be provided in deployment.yaml file under the siddhi namespace as below,

  siddhi:
  extensions:
  -
  extension:
  name: 'cdc'
  namespace: 'source'
  properties:
  url: jdbc:sqlserver://localhost:1433;databaseName=CDC_DATA_STORE
  password: <password>
  username: <>
  driverClassName: com.microsoft.sqlserver.jdbc.SQLServerDriver  

  
  
  Preparations required for working with Oracle Databases in listening mode
  
  Using the extension in Windows, Mac OSX and AIX are pretty straight forward inorder to achieve the required behaviour please follow the steps given below
  
    - Download the compatible version of oracle instantclient for the database version from here and extract
    - Extract and set the environment variable LD_LIBRARY_PATH to the location of instantclient which was exstracted as shown below
    

  export LD_LIBRARY_PATH=<path to the instant client location>
  

  
    - Inside the instantclient folder which was download there are two jars xstreams.jar and ojdbc<version>.jar convert them to OSGi bundles using the tools which were provided in the <distribution>/bin for converting the ojdbc.jar use the tool spi-provider.sh|bat and for the conversion of xstreams.jar use the jni-provider.sh as shown below(Note: this way of converting Xstreams jar is applicable only for Linux environments for other OSs this step is not required and converting it through the jartobundle.sh tool is enough)
    

  ./jni-provider.sh <input-jar> <destination> <comma seperated native library names>
  

  
    once ojdbc and xstreams jars are converted to OSGi copy the generated jars to the <distribution>/lib. Currently siddhi-io-cdc only supports the oracle database distributions 12 and above
  
  *** Configurations for PostgreSQL***
  When using listening mode with PostgreSQL, following properties has to be configured accordingly to create the connection.
  
      slot.name: (default value = debezium) in postgreSQL only one connection can be created from single slot, so to create multiple connection custom slot.name should be provided.
   
      plugin.name: (default value = decoderbufs ) Logical decoding output plugin name which the database is configured with. Other supported values are pgoutput, decoderbufs, wal2json.
  
      table.name: table name should be provided as <schema_name>.<table_name>. As an example, public.customer
  
  
  See parameter: mode for supported databases and change events.

Dependencies

JDBC connector jar should be added to the runtime. Download the JDBC connector jar based on the database type that is being used.

To identify the required JDBC connector jar, please refer to this debezium release documentation

In addition to that, there are some prerequisites that need to be met based on the CDC mode used. Please find them below.

Default mode (Listening mode):

Currently MySQL, PostgreSQL and SQLServer are supported in Listening Mode. To capture the change events, databases have to be configured as shown below.

• MySQL - https://debezium.io/documentation/reference/connectors/mysql.html#setup-the-mysql-server
• PostgreSQL - https://debezium.io/docs/connectors/postgresql/#setting-up-PostgreSQL
• SQLServer - https://debezium.io/docs/connectors/sqlserver/#setting-up-sqlserver

Polling mode:

• Change data capturing table should be have a polling column. Auto Incremental column or Timestamp can be used.

Please see API docs for more details about change data capturing modes.

Installation

For installing this extension on various siddhi execution environments refer Siddhi documentation section on adding extensions.

Running Integration tests in docker containers(Optional)

The CDC functionality are tested with the docker base integration test framework. The test framework initialize a docker container with required configuration before execute the test suit.

Start integration tests

1. Install and run docker

2. To run the integration tests, navigate to the siddhi-io-cdc/ directory and issue the following commands.

   • H2 default:

       mvn clean install
     

   • MySQL 5.7:

        mvn verify -P local-mysql -Dskip.surefire.test=true
     

   • Postgres 9.6:

        mvn verify -P local-postgres -Dskip.surefire.test=true
     

   • MSSQL:

        mvn verify -P local-mssql -Dskip.surefire.test=true
     

   • Oracle 11.2.0.2-xe:

        mvn verify -P local-oracle -Dskip.surefire.test=true
     

Support and Contribution

• We encourage users to ask questions and get support via StackOverflow, make sure to add the siddhi tag to the issue for better response.

• If you find any issues related to the extension please report them on the issue tracker.

• For production support and other contribution related information refer Siddhi Community documentation.

siddhi-io-cdc's Issues

Exception: Class not found: org.wso2.extension.siddhi.io.cdc.source.config.QueryConfiguration when on CDC polling mode

Description:
An

Can't construct a java object for tag:yaml.org,2002:org.wso2.extension.siddhi.io.cdc.source.config.QueryConfiguration; exception=Class not found: org.wso2.extension.siddhi.io.cdc.source.config.QueryConfiguration

error is thrown when a CDC source is defined with polling mode.

Steps to reproduce:
1.Have a siddhi app source defined with CDC Polling mode.

CDC with PostgreSQL only supports decoderbufs logical decoding output plug-in

Description:
Current siddhi-io-cdc 2.0.8 does not support logical decoding output plug-in except decoderbufs since "plugin.name" is not configurable in current implementation.

Affected Product Version:
All the versions upto siddhi-io-cdc 2.0.8

ORA-01843 or ORA-01847 with Oracle 11g

Description:
I configured the cdc in polling mode
@source(type = 'cdc', url = 'jdbc:oracle:thin:@localhost:49161/XE', mode = 'polling', jdbc.driver.name = 'oracle.jdbc.OracleDriver', polling.column = 'dated', polling.interval = '10', username = 'vatrox', password = 'vatrox', table.name = 'LOGS_AUDIT', @map(type = 'keyvalue' ) )

The table fields are

DATED TIMESTAMP(6)
USER_ID VARCHAR2(50 BYTE)
TENANT VARCHAR2(20 BYTE)
LOGLEVEL VARCHAR2(20 BYTE)
MESSAGE VARCHAR2(1000 BYTE)

When I delete all records from the table, when start the process this error appears.
ORA-01847: day of month must be between 1 and last day of month

And when I start the process and some records exists, this is the error.
ORA-01843: not a valid month

Example data...
insert into logs_audit values (to_timestamp('11/04/2018 01:13:00.000000','DD/MM/YYYY HH24:MI:SS.FF'),'jpvadell','-1234','INFO','Trying CDC');

Maybe is something with the NLS_DATE_FORMAT parameter?

Affected Product Version:
Stream Processor 4.4.0 RC1

Steps to reproduce:
Try to use Oracle as DB

Delete events could not be captured on Polling mode

Description:
Delete events could not be captured on Polling mode.

Affected Product Version:
SP 4.4.0

OS, DB, other environment details and versions:
mysql-5.7.21
Mac OS 10.13.1

Steps to reproduce:
Deploy CDCWithPollingMode sample and delete a record from the SweetProductionTable.

No log is generated for the deleted record.

[Documentation] Operation element is optional but marked as mandatory

Description:
$subject

Problem with connector.properties splitting

im not able to pass comma seperated values inside a value of a connector properties
Ex:
connector.properties="mongodb.hosts='server1:27017,server2:27017,server3:27017'"

i wan to acheive this which i cant achieve now wso2 tooling integrated not allowing commas in values
please help me regarding this

Host in the JDBC URL only accepts IP address

Description:
Host in the JDBC URLs for all the databases accept only IP addresses. These need to accept DNS names

Affected Product Version:
v2.0.4

Update the version of Debezium Embedded connector to latest

Description:
The current embedded connector version is 0.10.0.final, update this to a latest stable version along with the rest of the connectors

Test the MSSql support

Description:
We had to bump debezium-connector-sqlserver version since the support has been provided after 0.9.
Need to test the MSSql support.

Related PR:
#29

Unscaled NUMBER datatypes are not decoded properly for Oracle

Description:
When using Oracle database in listening mode, if there are any unscaled NUMBER datatypes they are not getting decoded properly. This causes a casting issue.

Affected Product Version:
v2.0.4

NoClassDefFoundError: org/apache/kafka/common/KafkaException when on CDC listening mode

Description:
java.lang.NoClassDefFoundError: org/apache/kafka/common/KafkaException is thrown when a CDC source is defined with listenning mode.

Steps to reproduce:

1. Have a siddhi app source defined with CDC Listenning mode.

Siddhi CDC MySQL Error with debezium 0.10.0 for listening mode

Description:
I tried CDC listening mode with MySQL with debezium 0.10.0. The MySQL configuration for CDC is done in the database and when starting the siddhi application, it provides this error log.

[2019-10-03 20:07:57,848]  INFO {org.apache.kafka.connect.runtime.WorkerConfig} - Worker configuration property 'internal.key.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[2019-10-03 20:07:57,848]  INFO {org.apache.kafka.connect.runtime.WorkerConfig} - Worker configuration property 'internal.value.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[2019-10-03 20:07:57,927] ERROR {io.siddhi.core.stream.input.source.Source} - Error on 'ParisTMANotifier'. Connection to the database lost. Error while connecting at Source 'cdc' at 'IncidentInputStream'. Will retry in '5 sec'. io.siddhi.core.exception.ConnectionUnavailableException: Connection to the database lost.
at io.siddhi.extension.io.cdc.source.CDCSource.lambda$connect$1(CDCSource.java:424)
at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:899)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: com/mysql/cj/jdbc/Driver
at io.debezium.connector.mysql.MySqlConnectorConfig.<clinit>(MySqlConnectorConfig.java:670)
at io.debezium.connector.mysql.MySqlConnectorTask.getAllConfigurationFields(MySqlConnectorTask.java:446)
at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:38)
at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:793)
... 3 more
Caused by: java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver cannot be found by siddhi-io-cdc_2.0.4.SNAPSHOT
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:448)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:361)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:353)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:161)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more

[2019-10-03 20:08:02,929]  INFO {org.apache.kafka.connect.json.JsonConverterConfig} - JsonConverterConfig values:
converter.type = key
schemas.cache.size = 1000
schemas.enable = true

[2019-10-03 20:08:02,929]  INFO {org.apache.kafka.connect.json.JsonConverterConfig} - JsonConverterConfig values:
converter.type = value
schemas.cache.size = 1000
schemas.enable = false

[2019-10-03 20:08:02,929]  INFO {io.debezium.embedded.EmbeddedEngine$EmbeddedConfig} - EmbeddedConfig values:
access.control.allow.methods =
access.control.allow.origin =
bootstrap.servers = [localhost:9092]
client.dns.lookup = default
config.providers = []
connector.client.config.override.policy = None
header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
key.converter = class org.apache.kafka.connect.json.JsonConverter
listeners = null
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
offset.flush.interval.ms = 60000
offset.flush.timeout.ms = 5000
offset.storage.file.filename =
offset.storage.partitions = null
offset.storage.replication.factor = null
offset.storage.topic =
plugin.path = null
rest.advertised.host.name = null
rest.advertised.listener = null
rest.advertised.port = null
rest.extension.classes = []
rest.host.name = null
rest.port = 8083
ssl.client.auth = none
task.shutdown.graceful.timeout.ms = 5000
value.converter = class org.apache.kafka.connect.json.JsonConverter

[2019-10-03 20:08:02,930]  INFO {org.apache.kafka.connect.runtime.WorkerConfig} - Worker configuration property 'internal.key.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[2019-10-03 20:08:02,930]  INFO {org.apache.kafka.connect.runtime.WorkerConfig} - Worker configuration property 'internal.value.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[2019-10-03 20:08:02,984] ERROR {io.siddhi.core.stream.input.source.Source} - Error on 'ParisTMANotifier'. Connection to the database lost. Error while connecting at Source 'cdc' at 'IncidentInputStream'. Will retry in '5 sec'. io.siddhi.core.exception.ConnectionUnavailableException: Connection to the database lost.
at io.siddhi.extension.io.cdc.source.CDCSource.lambda$connect$1(CDCSource.java:424)
at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:899)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class io.debezium.connector.mysql.MySqlConnectorConfig
at io.debezium.connector.mysql.MySqlConnectorTask.getAllConfigurationFields(MySqlConnectorTask.java:446)
at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:38)
at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:793)
... 3 more

[2019-10-03 20:08:07,988]  INFO {org.apache.kafka.connect.json.JsonConverterConfig} - JsonConverterConfig values:
converter.type = key
schemas.cache.size = 1000
schemas.enable = true

[2019-10-03 20:08:07,988]  INFO {org.apache.kafka.connect.json.JsonConverterConfig} - JsonConverterConfig values:
converter.type = value
schemas.cache.size = 1000
schemas.enable = false

[2019-10-03 20:08:07,989]  INFO {io.debezium.embedded.EmbeddedEngine$EmbeddedConfig} - EmbeddedConfig values:
access.control.allow.methods =
access.control.allow.origin =
bootstrap.servers = [localhost:9092]
client.dns.lookup = default
config.providers = []
connector.client.config.override.policy = None
header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
key.converter = class org.apache.kafka.connect.json.JsonConverter
listeners = null
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
offset.flush.interval.ms = 60000
offset.flush.timeout.ms = 5000
offset.storage.file.filename =
offset.storage.partitions = null
offset.storage.replication.factor = null
offset.storage.topic =
plugin.path = null
rest.advertised.host.name = null
rest.advertised.listener = null
rest.advertised.port = null
rest.extension.classes = []
rest.host.name = null
rest.port = 8083
ssl.client.auth = none
task.shutdown.graceful.timeout.ms = 5000
value.converter = class org.apache.kafka.connect.json.JsonConverter

[2019-10-03 20:08:07,989]  INFO {org.apache.kafka.connect.runtime.WorkerConfig} - Worker configuration property 'internal.key.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[2019-10-03 20:08:07,990]  INFO {org.apache.kafka.connect.runtime.WorkerConfig} - Worker configuration property 'internal.value.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[2019-10-03 20:08:07,990] ERROR {io.siddhi.core.stream.input.source.Source} - Error on 'ParisTMANotifier'. Task EmbeddedEngine{id=ParisTMANotifierIncidentInputStream} rejected from java.util.concurrent.ThreadPoolExecutor@3c2e8849[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2] Error while connecting at Source 'cdc' at 'IncidentInputStream'. java.util.concurrent.RejectedExecutionException: Task EmbeddedEngine{id=ParisTMANotifierIncidentInputStream} rejected from java.util.concurrent.ThreadPoolExecutor@3c2e8849[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
at io.siddhi.extension.io.cdc.source.CDCSource.connect(CDCSource.java:430)
at io.siddhi.extension.io.cdc.source.CDCSource.connect(CDCSource.java:56)
at io.siddhi.core.stream.input.source.Source.connectWithRetry(Source.java:160)
at io.siddhi.core.stream.input.source.Source$1.run(Source.java:185)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Affected Product Version:
v5.1.0-RC1

Missing records when polling records from a table

Description:
Reading Mode: Polling
Database: MS SQL
Stream Integrator reads records from a table while it's being written by another application. Stream Integrator uses CDC with polling mode and timestamp column to identify the changes.
=> Stream Integrator missed some records with the same timestamp.

[L1] CNF with MSSQL

Description:
Following error occurs when connecting to Microsoft SQL server with siddhi-io-cdc-2.0.6 connector. This works fine with siddhi-io-cdc-2.0.5 connector.

io.siddhi.core.exception.ConnectionUnavailableException: Connection to the database lost.
	at io.siddhi.extension.io.cdc.source.CDCSource.lambda$connect$1(CDCSource.java:503)
	at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:899)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: com/microsoft/sqlserver/jdbc/SQLServerDriver
	at io.debezium.connector.sqlserver.SqlServerConnection.<clinit>(SqlServerConnection.java:60)
	at io.debezium.connector.sqlserver.SqlServerConnectorTask.start(SqlServerConnectorTask.java:87)
	at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:49)
	at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:793)
	... 3 more
Caused by: java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerDriver cannot be found by siddhi-io-cdc_2.0.6
	at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:448)
	at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:361)
	at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:353)
	at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:161)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 7 more

Affected Product Version:
2.0.6

OS, DB, other environment details and versions:
JDBC driver: sqljdbc42.jar

Allow slot replication selection for PostgreSQL database

Description:
PostgreSQL requires a unique replication slot for each method: CREATE, UPDATE, DELETE
The siddhi-io-cdc doesn't have the option to change slot.name and create respective replication slot for each method.

slot.name is default to Debezium at the moment.

Suggested Labels:
psql, postgresql, replication

Suggested Assignees:

Affected Product Version:
2.0.9 and earlier

OS, DB, other environment details and versions:
Debian 10.x

Steps to reproduce:
Regular psql connection

Related Issues:

Unable to capture all data changes due to the out of order of the auto incremental column of the database.

Description:

When a multi-threaded client is used to insert data into the database. There can be out of order situations in the auto-increment column. Since Siddhi CDC uses the auto-increment column to keep taracs on new database entries, out of order of the auto-increment column can be cause to miss capturing the database changes.
Suggested Labels:
wso2sp- 4.3.0
Affected Product Version:
wso2sp- 4.3.0

CDC table Regex expression

is there a way to capture mongo dynamic collection in siddhi cdc because in my my system i will have many dynamic collection created for those each time i cant create a siddhi file please help me regarding in this

Thanks

Capture datetime object in mysql database

Description:

Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Add support to capture multiple operations occurs in database through a single CDC source

Description:
add support to capture multiple operations occurs in database through a single CDC source. the Siddhi event which generate in regards to the CDC event now consists with a transport property call operation, which will provide the operation which was captured from the database.

Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Host in Oracle JDBC URL only accepts IP address

Description:
When defining the Oracle URL, only the IP address is accepted as the host. This needs to be modified to accept DNS names as well.

Affected Product Version:
v2.0.4

OS, DB, other environment details and versions:
Oracle

Encountered change event for table whose schema isn't known to this connector

Description:
When I run to use the CDC extension with a database called parisTMA using the following Siddhi query to just log changes in a table.

@source(
    type='cdc',
    url='jdbc:mysql://localhost:3306/parisTMA',
    username='wso2sp',
    password='wso2',
    table.name='IncidentSample',
    operation='insert',
    @map(
        type='keyvalue'
    )
)
define stream IncidentInputStream(incidentId int, incidentName string);

@sink(type='log', prefix='LOGGER')
define stream SampleLog(incidentId int, incidentName string);

@info(name='listen-notification') 
from IncidentInputStream
select incidentId, incidentName
insert into SampleLog;

And it gives the following error.

database lost. Error while connecting at Source 'cdc' at 'IncidentInputStream'. Will retry in '5 sec'. io.siddhi.core.exception.ConnectionUnavailableException: Connection to the database lost.
	at io.siddhi.extension.io.cdc.source.CDCSource.lambda$connect$1(CDCSource.java:424)
	at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:793)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table whose schema isn't known to this connector
	at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:200)
	at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:178)
	at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:452)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1055)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:913)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:559)
	at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:793)
	... 1 more
Caused by: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table whose schema isn't known to this connector
	at io.debezium.connector.mysql.BinlogReader.informAboutUnknownTableIfRequired(BinlogReader.java:700)
	at io.debezium.connector.mysql.BinlogReader.handleUpdateTableMetadata(BinlogReader.java:676)
	at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:436)
	... 5 more

However when I change my database name to paristma it runs properly.

Suggested Labels:
Bug fix

Affected Product Version:
2.0.3

Steps to reproduce:
Using mentioned databases run the above app in Siddhi tooling or Siddhi runner.

Polling - org.postgresql.util.PSQLException: ERROR: operator does not exist

Description:
The polling mode fails with the following exception when using PostgreSQL database:

ERROR {io.siddhi.extension.io.cdc.source.polling.strategies.DefaultPollingStrategy} - Error occurred while processing records in table AM_APPLICATION. {mode=polling, app=xxx, stream=Applications} org.postgresql.util.PSQLException: ERROR: operator does not exist: timestamp without time zone > character varying
  Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.
  Position: 49
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2533)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2268)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:313)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:369)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:159)
	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:109)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
	at io.siddhi.extension.io.cdc.source.polling.strategies.DefaultPollingStrategy.poll(DefaultPollingStrategy.java:98)
	at io.siddhi.extension.io.cdc.source.polling.CDCPoller.run(CDCPoller.java:187)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

The Siddhi application:

@source(type = 'cdc', mode = 'polling', url = 'jdbc:postgresql://xxx/xxx', jdbc.driver.name = 'org.postgresql.Driver', username = 'xxx', password = 'xxx', table.name = 'AM_APPLICATION', polling.column = 'UPDATED_TIME', polling.interval = '600', @map(type = 'keyvalue', fail.on.missing.attribute = 'false'))
define stream Applications(
  application_id int,
  name string,
  subscriber_id int,
  application_tier string,
  callback_url string,
  description string,
  application_status string,
  group_id string,
  created_by string,
  created_time string,
  updated_by string,
  updated_time string,
  uuid string,
  token_type string
);

It works fine with MySQL.

AM_APPLICATION DDL:

CREATE TABLE IF NOT EXISTS AM_APPLICATION (
    APPLICATION_ID INTEGER DEFAULT nextval('am_application_sequence'),
    NAME VARCHAR(100),
    SUBSCRIBER_ID INTEGER,
    APPLICATION_TIER VARCHAR(50) DEFAULT 'Unlimited',
    CALLBACK_URL VARCHAR(512),
    DESCRIPTION VARCHAR(512),
    APPLICATION_STATUS VARCHAR(50) DEFAULT 'APPROVED',
    GROUP_ID VARCHAR(100),
    CREATED_BY VARCHAR(100),
    CREATED_TIME TIMESTAMP,
    UPDATED_BY VARCHAR(100),
    UPDATED_TIME TIMESTAMP,
    UUID VARCHAR(256),
    TOKEN_TYPE VARCHAR(10),
    FOREIGN KEY(SUBSCRIBER_ID) REFERENCES AM_SUBSCRIBER(SUBSCRIBER_ID) ON UPDATE CASCADE ON DELETE RESTRICT,
    PRIMARY KEY(APPLICATION_ID),
    UNIQUE (NAME,SUBSCRIBER_ID),
    UNIQUE (UUID)
);

As a workaround, we tried to use the primary key APPLICATION_ID for the polling.column, but a similar exception on integer operator occurred.

Affected Product Version:
siddhi-io-cdc 2.0.5

OS, DB, other environment details and versions:
WSO2 API-M Analytics 3.1.0
PostgreSQL 9.5.16
postgresql-42.2.12.jar (JDBC library)

CDC Polling mode get errors when source database restart

Description:

A Siddhi app with CDC polling mode get following errors when the source database restart.
These failures caused by the bug in DefaultPollingStrategy class, printEvent(connection) do not verify connection before using it.

---

[2020-11-17 18:48:50,328] ERROR {io.siddhi.extension.io.cdc.source.polling.strategies.DefaultPollingStrategy} - Error occurred while processing records in table SweetProductionTable. {mode=polling, app=CDCWithPollingMode, stream=insertSweetProductionStream} java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:493)
at com.sun.proxy.$Proxy73.prepareStatement(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:315)
at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java)
at io.siddhi.extension.io.cdc.source.polling.strategies.DefaultPollingStrategy.printEvent(DefaultPollingStrategy.java:142)
at io.siddhi.extension.io.cdc.source.polling.strategies.DefaultPollingStrategy.poll(DefaultPollingStrategy.java:86)
at io.siddhi.extension.io.cdc.source.polling.CDCPoller.run(CDCPoller.java:202)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

---

Suggested Labels:

bug
Suggested Assignees:

Affected Product Version:
2.0.6
OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Cron Support for CDCPoller

Description:
Cron Support for CDCPoller

Related Issues:
wso2/streaming-integrator-tooling#86