Offset Management about kafka-connect-spooldir

[patelliandrea]

Another question...
If one file is huge, e.g. millions of lines, and I'm importing it using a batch size of let's say 1000, what happens if the connector dies before the entire was processed?
Seems like that when I'll restart the connector, it'll read the entire file again, starting from the beginning, without skipping the lines that were already sent to kafka.

[jcustenborder]

You're on it. It's been missing offset management for a while. I'm working on a few changes and refactors to cover this and some better schema handling. A new release will be coming soon.

[patelliandrea]

Perfect, thanks. I'm working on it as well... I managed to implement the offset management but I'm struggling to implement the clean up of the folder in a nice way.

[gbehrmann]

Any updates on this subject? I am investigating ways to import from an application that continually appends data to a CSV file and your connector seems like a perfect match (the application will let go of the file when it is renamed, so the automatic renaming to a .PROCESSING file is perfect). However the lack of resuming from the recorded offset is a showstopper for us.

[jcustenborder]

@gbehrmann Missed your comment. I've had this pull out there but haven't received much feedback. #17

[jcustenborder]

This was fixed with #17.