Support watching transient files about logstash-forwarder

[superawesome]

Something like this would be hugely beneficial to us. We have HTTP load balancer log files that are rotated hourly, wherein each new log file has a timestamp in the filename. So monitoring a given list of files won't work well... the list changes every hour. There is a separate file for every domain, and a separate file for every hour... so 10 domains over a span of 24 hours would create a total of 240 files.

Unfortunately for us these logs come from a proprietary tool, so we don't have a lot of flexibility on log rotation... we can't easily do something more like Linux "logrotate" where the filename stays the same.

Even if we hack something together with symlinks (for example, have lumberjack watch a set of symlinks that always point to the current / most-recent actual file), that only covers newly rotated files on existing log sets. It won't help for when new log sets are created (so when we add an 11th domain, lumberjack wouldn't notice even if a suitable symlink popped up).

Better all around (in our case at least) would be for lumberjack to just be able to watch a directory, and automatically pick up new files as they get added.

[jordansissel]

I think this is resolved in newer editions of lsf which watch paths more liberally. Please reopen if this is not the case.