File extension is 'PROCESSING' even after file is processed about kafka-connect-spooldir

[jcustenborder]

You need something that tails the log file and notices when the log has
been rotated. The scenarios I envisioned for this plugin was to handle flat
files that are typically transmitted between companies. This handles
delimited files really well. The processing file extension is to allow
multiple tasks in the same folder.

There are a couple ways we could tackle this.

1.) Set log4j to roll quicker. Maybe every n events or every x minutes.
Then set the pattern for file pickup to .*log.\d+$. I'm on mobile so
you'll need to double check that pattern. Then the current log file will be
left alone and the connector will read the rolled ones.

2.) Use https://github.com/jcustenborder/kafka-connect-syslog and send the
messages in real-time over syslog.

On Tue, Aug 23, 2016, 12:41 PM ajayinfoworks [email protected]
wrote:

Every time I push a file into the directory pointed to by 'input.path', it
gets processed & gets moved to 'finished'. The file is renamed from
'test.log' to 'test.log.PROCESSING'. Why does it say 'PROCESSING' when
processing is done?

Ideally, I am looking for a connector that will watch the 'log' directory
used by log4j. As you know, log4j will start writing lines to a file in a
directory. Once it's time to rotate, it will rename the log file from
'test.log' to 'test.log.1' & then will continue writing to test.log.

This connector works well for the files that are rotated but won't work
for the main 'test.log' file, correct?

I mean, is there any way we can send a message to Kafka topic as soon as a
new line gets written to the main 'test.log' file (and not push the file to
the 'finished.path')?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#2, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAMzgN7vKbkwO7WeFlRJQDQZm_xkRMbwks5qi0zfgaJpZM4JrTvG
.