avioconsulting / log4j2-sqs-appender Goto Github PK

Hi team!

I have been a error to build project with mvn clean install.

The error is the following:

Failed to execute goal io.fabric8:docker-maven-plugin:0.43.0:start (docker-start) on project log4j2-sqs-appender: Execution docker-start of goal io.fabric8:docker-maven-plugin:0.43.0:start failed: No given, no DOCKER_HOST environment variable, no read/writable '/var/run/docker.sock' or '//./pipe/docker_engine' and no external provider like Docker machine configured -> [Help 1]

Anyone have been this error?

Hey there. I'm not sure what we could be doing wrong; I've gone through every bit of documentation and every walkthrough about this sqs appender I can find, and I still can't get it to work.

Relevant data points:

• We have an existing app that is presently using a JMS appender to send messages to ApacheMQ. This costs us way more than SQS would, hence the desire to switch. (The scalability is nice, too)
• I have added the maven dependency, as described in your docs and elsewhere, and call for version 1.1.0.
• I have configured the appender as described in your docs.
• Regarding SQS / IAM config, I have verified that all of this is setup correctly by successfully sending an SQS message from the command-line on the machine on which the java application is running using the same access key. (Sorry for the run-on sentence there)
• When I run the application and trigger an event which fires off a log message, I get the following in the log:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.amazonaws.util.XpathUtils (file:/var/resultflow/shared/aws-java-sdk-core-1.12.340.jar) to method com.sun.org.apache.xpath.internal.XPathContext.getDTMManager()
WARNING: Please consider reporting this to the maintainers of com.amazonaws.util.XpathUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

I have taken a look at the aws-java-sdk-core maven repo page, and it doesn't appear that there's anything (significantly) newer.

What am I missing?

Thank you for reading and your help!
--Garrett

Reassembling log messages that have been split into multiple SQS messages can be problematic, so can we use S3 to store those large messages?

I see two options:

1. Use Amazon SQS Extended Client Library for Java to store the messages in S3 (either all or just the large ones) and then use the same library to read those messages (See https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html)
2. Write messages larger than 256 KB to S3 and do not send those messages on the SQS queue

I think both options are valid and could be supported so different methods of reading the messages can be supported. For instance, Logstash has an S3 input and an SQS input, but does not use the extended client library for Java, so it cannot read those references from the queue correctly.

Dear Sir,

I am using this log4j2-sqs-appender. we have the requirement to send large message to elastic more than 1MB. When I am using FIFO, getting the following exception.
ERROR 2023-02-07 14:16:08,152 [Log4j2-TF-8-AsyncLoggerConfig-5] [processor: ; event: ] com.avioconsulting.log4j.sqs.SqsManager: Failed to send message to SQS
java.lang.NullPointerException: null
at com.avioconsulting.log4j.sqs.SqsManager.send(SqsManager.java:93) [log4j2-sqs-appender-1.1.0.jar:?]
at com.avioconsulting.log4j.sqs.SqsAppender.append(SqsAppender.java:179) [log4j2-sqs-appender-1.1.0.jar:?]
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161) [log4j-core-2.17.1.jar:2.17.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134) [log4j-core-2.17.1.jar:2.17.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125) [log4j-core-2.17.1.jar:2.17.1]
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89) [log4j-core-2.17.1.jar:2.17.1]
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:542) [log4j-core-2.17.1.jar:2.17.1]
at org.apache.logging.log4j.core.async.AsyncLoggerConfig.callAppenders(AsyncLoggerConfig.java:126) [log4j-core-2.17.1.jar:2.17.1]
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:500) [log4j-core-2.17.1.jar:2.17.1]
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:483) [log4j-core-2.17.1.jar:2.17.1]
at org.apache.logging.log4j.core.async.AsyncLoggerConfig.log(AsyncLoggerConfig.java:120) [log4j-core-2.17.1.jar:2.17.1]
at org.apache.logging.log4j.core.async.AsyncLoggerConfig.logToAsyncLoggerConfigsOnCurrentThread(AsyncLoggerConfig.java:168) [log4j-core-2.17.1.jar:2.17.1]
at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:112) [log4j-core-2.17.1.jar:2.17.1]
at org.apache.logging.log4j.core.async.AsyncLoggerConfigDisruptor$Log4jEventWrapperHandler.onEvent(AsyncLoggerConfigDisruptor.java:98) [log4j-core-2.17.1.jar:2.17.1]
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) [disruptor-3.4.2.jar:?]
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) [disruptor-3.4.2.jar:?]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_352]

Please suggest.

Regards

Hello,

Is it possible to add a parameter to specify a optionnal endpoint.
By default, SQS is only available thru public address but we could create an endpoint in a VPC to access SQS thru it without transit in Internet.

For example, using the boto3 module with Python, it's possible to add an endpoint_url to specify created endpoint and bypass the automatic URL build process.

sqs = session.resource('sqs',endpoint_url='<VPC_AWS_service_endpoint>/')

Best regards

Not an issue yet, just a heavy concern.. We were trying to use a custom log4j2 Kafka appender on CloudHub and the conclusion (confirmed by Mulesoft support) was that it will never work, because the log4j2 is initialized by their application server's classloader, while the Kafka appender is shipped through our worker application, and the system classloader will not delegate to the application classloader. Well, they did not phrase it that technically, but the conclusion was that.

Now I wonder how your custom SQS log4j2 appender could behave otherwise, if it also gets packaged into the application worker's jar file. Won't we meet the exact same classloading issue when deploying to CloudHub? Or how did you get it to work, given how CloudHub works?

Note also that your artifacts are not rolled out to maven central.. Sure I can roll my own releases from source, but you might want to deploy to an official repo as well.