Comments (26)
In 4.4.x you don't have the fix. For 4.8 the release should be after summer maybe Sept/Oct. After that we need to release the kamelets and after camel Kafka connector. It takes a while
from camel-kafka-connector.
What version of the connector are you using?
from camel-kafka-connector.
@oscerd Hello, sorry for replying lately, the version destais as: implementation group: 'org.apache.camel.kafkaconnector', name: 'camel-https-kafka-connector', version: '4.0.3'.
from camel-kafka-connector.
The implementation is based on the following Kamelet https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/http-secured-sink.kamelet.yaml and SSLContextParameters is not exposed.
from camel-kafka-connector.
@oscerd Thank you for your replying, our scenario is as below:
- Kafka events will sink to multiple Endpoints URLs by camel-https-kafka-connector.
- Different endpoint URL has different auth Methods some of use "keystore information" for MTLS, some of use "Basic" or "oauth2".
If SSLContextParameters is not exposed, how can we handle MTLS scenario for setting "keystore information"? Do you have any suggestion for that?
from camel-kafka-connector.
We need to create a new Kameelt for that and generating a new connector for that purpose. This requires time.
from camel-kafka-connector.
@oscerd hope can release new connector for SSLContextParameters setting asap.
Currently we cannot directly do the configuration for SSLContextParameters in the file:
https://github.com/apache/camel-kafka-connector/blob/camel-kafka-connector-4.0.x/connectors/camel-https-kafka-connector/src/main/docs/examples/CamelHttpsSinkConnector.properties
or
https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/http-secured-sink.kamelet.yaml
Whether only can create a project like Springboot Project and create bean name like "mySSLContextParameters" of SSLContextParameters according to spring configuration:
"
<camel:sslContextParameters
id="mySSLContextParameters">
<camel:keyManagers
keyPassword="keyPassword">
<camel:keyStore
resource="/users/home/server/keystore.jks"
password="keystorePassword"/>
</camel:keyManagers>
</camel:sslContextParameters>
"
And reference bean mySSLContextParameters by configuration like
" "
As this document explanation:
https://camel.apache.org/components/4.4.x/http-component.html#_setting_up_ssl_for_http_client
I am fresher for Camel Connector, my understanding is correct or not?
from camel-kafka-connector.
You cannot use the approach with bean and SB project in the Kafka connect context
from camel-kafka-connector.
I don't know the ETA for this.
from camel-kafka-connector.
@oscerd One more question, regarding below configuration file
https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/http-secured-sink.kamelet.yaml
1. How it integrate Kafka?
2.Does it support Kafka Connect "Single Message Transform" configuration like below? if it supports could you share some samples for configuration?
"""
transforms=tenantSelection,insertHeaders
transforms.tenantSelection.type=com.services.kafka.connect.transforms.FilterByTenantId
transforms.tenantSelection.include=XXXXXX
transforms.insertHeaders.type=org.apache.kafka.connect.transforms.InsertHeader
transforms.insertHeaders.header=CamelHeader.Content-Type
transforms.insertHeaders.value.literal=application/cloudevents+json
"""
Thanks.
from camel-kafka-connector.
@oscerd One more question, regarding below configuration file https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/http-secured-sink.kamelet.yaml
Does it support Kafka Connect "Single Message Transform" configuration like below? if it supports could you share some samples for configuration?
""" transforms=tenantSelection,insertHeaders
transforms.tenantSelection.type=com.services.kafka.connect.transforms.FilterByTenantId transforms.tenantSelection.include=XXXXXX
transforms.insertHeaders.type=org.apache.kafka.connect.transforms.InsertHeader transforms.insertHeaders.header=CamelHeader.Content-Type transforms.insertHeaders.value.literal=application/cloudevents+json
"""
Thanks.
No, Kamelets are building blocks used to generate the Kafka connectors in this repository, but they are not Kafka connect oriented. If you want to use SMT, you have to extend the connector and add your own SMT.
from camel-kafka-connector.
@oscerd One more question, regarding below configuration file https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/http-secured-sink.kamelet.yaml
Does it support Kafka Connect "Single Message Transform" configuration like below? if it supports could you share some samples for configuration?
""" transforms=tenantSelection,insertHeaders
transforms.tenantSelection.type=com.services.kafka.connect.transforms.FilterByTenantId transforms.tenantSelection.include=XXXXXX
transforms.insertHeaders.type=org.apache.kafka.connect.transforms.InsertHeader transforms.insertHeaders.header=CamelHeader.Content-Type transforms.insertHeaders.value.literal=application/cloudevents+json
"""
Thanks.No, Kamelets are building blocks used to generate the Kafka connectors in this repository, but they are not Kafka connect oriented. If you want to use SMT, you have to extend the connector and add your own SMT.
How does http-secured-sink.kamelet.yaml integrate with Kafka? I cannot find any config related with Kafka? Does it need combine with https://github.com/apache/camel-kamelets/blob/v4.0.3/kamelets/kafka-source.kamelet.yaml ? Could you show some demo for that?
from camel-kafka-connector.
Not http specific, but you can have a look at https://github.com/apache/camel-kamelets-examples/tree/main/jbang/kafka-batch-s3
from camel-kafka-connector.
@oscerd Regarding Kafka sink connector of camel-https-kafka-connector doc reference :https://camel.apache.org/camel-kafka-connector/4.0.x/reference/connectors/camel-https-kafka-sink-connector.html
How does it support oauth2 relevant configuration items something like below http component? Not sure whether it is similar with SSLContextParameters also does not expose to configuration file.
https://camel.apache.org/components/4.4.x/http-component.html#_oauth2_support.
from camel-kafka-connector.
It doesn't expose those parameters. We need a specialized kamelet for that purpose and we need to support camel 4.4.x
from camel-kafka-connector.
@oscerd There is another issue regarding camel-https-kafka-connector,
we know the connector forwards the message (record) value to the HTTP API.
Not sure we can add parameters to have the connector construct a unique HTTP API URL like containing the Kafka record key , topic name and others value that refer to fields from the Kafka record.
For example, we enter http://test:9200/api/messages/${topic}/${key} ?id={id} to have the HTTP API URL contain the topic name, record key and id like "http://test:9200/api/messages/test_topic/key_value?id=3c0aa2b2-ffb8-4855-863e-6edcc8985f67"
If currently we do not support that, whether have plan to implement it?
Thanks
from camel-kafka-connector.
For doing that we need to use toD in the Kamelet instead of to and you'll need to use Camel headers for the purpose. As of today we don't have any plan to support that, first we need to switch to Camel 4.4.x
from camel-kafka-connector.
One more question regarding CamelHttpsSinkConnector configuration.
Could you please help confirm the property "camel.component.https.cookieManagementDisabled" in the file of https://camel.apache.org/camel-kafka-connector/next/reference/connectors/camel-https-kafka-sink-connector.html, whether it is exposed or not?
I do not know how to configure it in the properties file.
Version destais as: implementation group: 'org.apache.camel.kafkaconnector', name: 'camel-https-kafka-connector', version: '4.0.3'.
Thanks
from camel-kafka-connector.
It's not exposed.
from camel-kafka-connector.
@oscerd Now, shall we have some methods or approches to disableCookieManagement?
There is an use case need to disable cookie in the CamelHttpsSinkConnector for HttpClient Object, like below code:
CloseableHttpClient httpClient = HttpClients.custom()
.disableCookieManagement()
.build();
from camel-kafka-connector.
It's not exposed.
@oscerd For the Camel 4.4.x, shall we can expose the "camel.component.https.cookieManagementDisabled" to control the Apache HttpClient can disable the cookie management?
We have the use case that the events are belong to multi-tenants, during events sync to Remote Endpoint, it will cause the issue something like the tenantA’s cookie will send to tenantB’s and lead security issue.
So to avoid this security issue, we want to disableCookieManagement for Apache HttpClient.
Thanks
from camel-kafka-connector.
I created this: apache/camel-kamelets#2042
The target it's 4.7.0, which won't be an LTS, @valdar is working on releasing 4.4.x for Camel-Kafka-connector and the next supported LTS will 4.8.0. So you'll have to wait.
from camel-kafka-connector.
I created this: apache/camel-kamelets#2042
The target it's 4.7.0, which won't be an LTS, @valdar is working on releasing 4.4.x for Camel-Kafka-connector and the next supported LTS will 4.8.0. So you'll have to wait.
@oscerd Cloud you provide the available timeline regarding the version 4.4.x and 4.8.0? we can do the further plan.
Thanks.
from camel-kafka-connector.
@oscerd Whether we not only expose "sslContextParameters","camel.component.https.cookieManagementDisabled", and add another fields as many as we can, something like but not limit "camel.sink.endpoint.authMethodPriority", "camel.sink.endpoint.authPassword", "camel.sink.endpoint.authUsername", "camel.component.https.connectionRequestTimeout","camel.component.https.connectTimeout","camel.component.https.responseTimeout","camel.component.https.soTimeout"
from camel-kafka-connector.
It doesn't expose those parameters. We need a specialized kamelet for that purpose and we need to support camel 4.4.x
Just checked the version 4.4.2 already released, could help confirm that "sslContextParameters" and "oauth2 related parameters" whether exposed or not?
Thanks a lot.
from camel-kafka-connector.
As already said:
In 4.4.x you don't have the fix. For 4.8 the release should be after summer maybe Sept/Oct. After that we need to release the kamelets and after camel Kafka connector. It takes a while
It will be in the next LTS so 4.8.x
from camel-kafka-connector.
Related Issues (20)
- Camel KafkaConnect pods can't start to run HOT 1
- Reading headers from SQS message
- [issue] [salesforce source] sending replayId to re-run old records HOT 1
- ERROR Exchange delivery has failed! Azure Gen2 Data Lake Sink Connector HOT 11
- File name format and File Format in Azure Data Lake Sink Connector HOT 27
- Kafka Events batching HOT 4
- Directory name for Azure Datalake Kafka sink Connector HOT 3
- Azure Camel Kafka Sink Connector - Avro File Format Issue HOT 3
- Security CVE-2022-3509 - com.google.protobuf:protobuf-java > For camel-aws-ddb-sink-kafka-connector-3.21.0 HOT 3
- Block atlassian maven repo so build is faster HOT 3
- Upgrade to Camel 4.4.2 LTS
- Upgrade maven wrapper to 3.9.x
- Upgrade to kafka 3.6.x HOT 1
- Remove jira connectors
- Migrate connectors to kafka 3.6.x plugin discovery HOT 2
- [Feature Request] Add Configuration to Disable Cookies in camel-https-kafka-connector HOT 1
- How to support at least once for CamelKafkasourceSourceConnector
- Make kafka connect connectors compatible with `ServiceLoad` mode of `plugin.discovery`
- kafkaconnector.https.CamelHttpsSinkConnector oauth2 scope is missing HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from camel-kafka-connector.