streamnative / pulsar-io-amqp-1-0

support sink/source for AMQP version 1.0.0

AMQP1_0 connector

The AMQP1_0 connector is a Pulsar IO connector for exchanging data between AMQP 1.0 and Pulsar. It consists of two types of connectors:

• AMQP1_0 source connector

  This connector feeds data from AMQP and writes data to Pulsar topics.

  

• AMQP1_0 sink connector

  This connector pulls data from Pulsar topics and persists data to AMQP.

  

Doc

AMQP1_0 connector version	Apache Pulsar version	Doc
2.7.1.1 2.7.1.2 2.7.1.3	2.7.x	- AMQP1_0 source connector - AMQP1_0 sink connector

Project layout

Below are the sub folders and files of this project and their corresponding descriptions.

├── conf // examples of configuration files of this connector
├── docs // user guides of this connector
├── script // scripts of this connector
├── io-amqp1_0-impl
│   ├── src // source code of this connector
├── tests // integration test
├── src
│   ├── checkstyle // checkstyle configuration files of this connector
│   ├── license // license header for this project. `mvn license:format` can be used for formatting the project with the stored license header in this directory
│   │   └── ALv2
│   ├── spotbugs // spotbugs configuration files of this connector

pulsar-io-amqp-1-0's Issues

Any limitations?

Hi,
Does this IO support all Pulsar features as AMQP, and all AMQP features for Pulsar? Ie. can this be used as a bridge between the protocols without restrictions?

That being Batching + Encryption + Chunking + Dead Letter etc. works transparently between the protocols?

Release AMQP 1.0 connector in streamnative-ci

According to https://github.com/streamnative/streamnative-ci/issues/374, AMQP 1.0 connector will be included in streamnative-ci and appear in sn-pulsar and cloud pulsar images. The release process of AMQP 1.0 connector can refer to the plugin part (e.g. Aop and Kop) in StreamNative Weekly Release Process. There are two things that need to be confirmed:

1. set-pulsar-version.sh and set-project-version.sh
Are there any other scripts (similar in function to set-pulsar-version.sh and set-project-version.sh) to adjust the pulsar version it depends on, and how to use them?

2. Create a project directory under the streamnative-ci/projects/
Please submit a project directory to streamnative-ci/projects. In general, under the submitted project, there should be three shell scripts, i.e. build.sh, get-version.sh and set-version.sh. To add this directory, You can refer to this document and the directories submitted by other projects in streamnative-ci/projects

Please contact me if you have any question :)

[Bug] AMQP passwords displayed in the logging

Describe the bug
Our AMQP passwords are displayed in the logging. This is an security issue for us and therefor undesirable.

To Reproduce
Steps to reproduce the behavior:

1. Start the connector with an configuration

Expected behavior
No password or a redacted password in the logging

Screenshots
The snip from the logging:
2023-02-22T13:54:33,331+0000 [abc/namespace1/amqp-source-0] INFO org.apache.pulsar.functions.instance.JavaInstanceRunnable - Starting Java Instance amqp-source :
Details = tenant: "abc"
namespace: "namespace1"
name: "amqp-source"
className: "org.apache.pulsar.functions.api.utils.IdentityFunction"
autoAck: true
parallelism: 1
source {
className: "org.apache.pulsar.ecosystem.io.amqp.AmqpSource"
configs: "{"protocol":"amqps","password":"XXXXXXXX","port":"443","host":"localhost","queue":"topic::queue","username":"amq-user","onlyTextMessage":true}"
typeClassName: "java.nio.ByteBuffer"
}
sink {
topic: "persistent://abc/namespace1/topic"
typeClassName: "java.nio.ByteBuffer"
forwardSourceMessageProperty: true
}
resources {
cpu: 1.0
ram: 1073741824
disk: 10737418240
}
componentType: SOURCE

Environment (please complete the following information):

• OS: Openshift Kubernetes
• Pulsar version: 2.10.x
• Deployment: cluster
• Connector version: e.g. 2.10.x

Additional context
We now excluded the org.apache.pulsar.functions.instance.JavaInstanceRunnable class from the log4j2 config. Also this configuration is also logged in the pulsar broker. So we had to to exclude some classes there aswell.
I think this can be achieved by annotating the password field in the AmqpBaseConfig class with @ToString.Exclude. Or by overriding the lombok toString method with a custom toString method
@ToString.Exclude
private String password;
I saw the same behavior for the RabbitMQ connector.

Provide a sink/source connector for AMQP-1.0.0

Pulsar supports RabbitMQ sink and source, But RabbitMQ support AMQP_0-9-1, It would be great if Pulsar support AMQP 1.0 connector:
https://pulsar.apache.org/docs/en/io-overview/
https://pulsar.apache.org/docs/en/io-rabbitmq-source/#configuration
https://github.com/apache/pulsar/tree/master/pulsar-io/rabbitmq

And Qpid Support AMQP 1.0.0, we could use it to do the test.

Define of done:

• code ready.
• test added.
• doc added.