ERROR Exchange delivery has failed! Azure Gen2 Data Lake Sink Connector about camel-kafka-connector

[oscerd]

This is the error:

org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:583)\n\t... 11 more\nCaused by: com.azure.storage.file.datalake.models.DataLakeStorageException: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.\nIf you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.\nPlease remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.\nStatus code 403, "{"error":{"code":"AuthorizationPermissionMismatch","message":"This request is not authorized to perform this operation using this permission.\nRequestId:63c48fdb-d01f-000d-4c22-6e3603000000\nTime:2024-03-04T10:57:07.2520699Z"}}"\n\tat

It's the clientSecret you're using, try to externalize the property to an external configuration file or use RAW(uqKxxxxxxxxxxxxxxxx6N5eS-lcbw)

[AishD3]

This is the new error , After passing RAW(value of client Secret ID )

"tasks": [
{
"id": 0,
"state": "FAILED",
"worker_id": "10.200.7.206:8083",
"trace": "org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:614)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:336)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:237)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:206)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:257)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:177)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.kafka.connect.errors.ConnectException: Exchange delivery has failed!\n\tat org.apache.camel.kafkaconnector.CamelSinkTask.put(CamelSinkTask.java:210)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:583)\n\t... 11 more\nCaused by: com.microsoft.aad.msal4j.MsalServiceException: AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '328ea5df-d095-4c9f-ba69-79c468792db9'. Trace ID: 3c558c03-90ba-4285-b5f9-d0735d3a5a00 Correlation ID: 8d54f2a5-418a-4770-9917-053fc904885b Timestamp: 2024-03-04 13:24:22Z\n\tat com.microsoft.aad.msal4j.MsalServiceExceptionFactory.fromHttpResponse(MsalServiceExceptionFactory.java:45)\n\tat com.microsoft.aad.msal4j.TokenRequestExecutor.createAuthenticationResultFromOauthHttpResponse(TokenRequestExecutor.java:111)\n\tat com.microsoft.aad.msal4j.TokenRequestExecutor.executeTokenRequest(TokenRequestExecutor.java:34)\n\tat com.microsoft.aad.msal4j.AbstractClientApplicationBase.acquireTokenCommon(AbstractClientApplicationBase.java:129)\n\tat com.microsoft.aad.msal4j.AcquireTokenByAuthorizationGrantSupplier.execute(AcquireTokenByAuthorizationGrantSupplier.java:63)\n\tat com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.acquireTokenByClientCredential(AcquireTokenByClientCredentialSupplier.java:86)\n\tat com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.execute(AcquireTokenByClientCredentialSupplier.java:49)\n\tat com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69)\n\tat com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)\n\t... 1 more\n\tSuppressed: java.lang.Exception: #block terminated with an error\n\t\tat reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)\n\t\tat reactor.core.publisher.Mono.block(Mono.java:1742)\n\t\tat com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:146)\n\t\tat com.azure.storage.file.datalake.DataLakeFileClient.uploadWithResponse(DataLakeFileClient.java:432)\n\t\tat org.apache.camel.component.azure.storage.datalake.client.DataLakeFileClientWrapper.uploadWithResponse(DataLakeFileClientWrapper.java:113)\n\t\tat org.apache.camel.component.azure.storage.datalake.operations.DataLakeFileOperations.upload(DataLakeFileOperations.java:221)\n\t\tat org.apache.camel.component.azure.storage.datalake.DataLakeProducer.process(DataLakeProducer.java:93)\n\t\tat org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:65)\n\t\tat org.apache.camel.processor.SendProcessor.process(SendProcessor.java:172)\n\t\tat org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:475)\n\t\tat org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:196)\n\t\tat org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:164)\n\t\tat org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59)\n\t\tat org.apache.camel.processor.Pipeline.process(Pipeline.java:163)\n\t\tat org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:383)\n\t\tat org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:96)\n\t\tat org.apache.camel.impl.engine.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:212)\n\t\tat org.apache.camel.impl.engine.SharedCamelInternalProcessor$1.process(SharedCamelInternalProcessor.java:109)\n\t\tat org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:81)\n\t\tat org.apache.camel.impl.engine.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:106)\n\t\tat org.apache.camel.support.cache.DefaultProducerCache.send(DefaultProducerCache.java:180)\n\t\tat org.apache.camel.impl.engine.DefaultProducerTemplate.send(DefaultProducerTemplate.java:176)\n\t\tat org.apache.camel.impl.engine.DefaultProducerTemplate.send(DefaultProducerTemplate.java:148)\n\t\tat org.apache.camel.kafkaconnector.CamelSinkTask.put(CamelSinkTask.java:205)\n\t\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:583)\n\t\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:336)\n\t\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:237)\n\t\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:206)\n\t\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)\n\t\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:257)\n\t\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:177)\n\t\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\t\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\t\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\t\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\t\t... 1 more\n"
},

[oscerd]

Are you sure about the credentials?

[AishD3]

Yeah I created a new client secret now and tried with that as well , Is there any other configs I need to check.

[oscerd]

Can you try without raw and placing the secret in an external file?

[AishD3]

With external file also it is not working, From where the client id , tenant id and client secret needs to be picked up . I am picking up from App registrations like below

Is this right

[oscerd]

No, you need to pick up this stuff from Azure Storage Datalake service

[AishD3]

Its working now, App registration and role assignments details can be more clearly documented from where client id , tenant id and clientSecret needs to be picked up . One more last doubt, Is there any config , which will allow to store blob in two different directories when consumed from two different topics.

[AishD3]

@oscerd Do we have any details regarding file name format

[oscerd]

You need to set an header on record: CamelAzureStorageDataLakeFileName, the header name should be "CamelHeader.CamelAzureStorageDataLakeFileName", it will be transformed while running the sync.

[oscerd]

This is resolved. Closing.