gmethvin / directory-watcher Goto Github PK

According to the javadoc of DirectoryWatcher.watch() (

Block(s) until either the listener stops watching or the DirectoryWatcher is closed.

Actually the methods throws a java.nio.file.ClosedWatchServiceException in case the watcher has been closed.

 java.nio.file.ClosedWatchServiceException
	at io.methvin.watchservice.AbstractWatchService.check(AbstractWatchService.java:94)
	at io.methvin.watchservice.AbstractWatchService.take(AbstractWatchService.java:86)
	at io.methvin.watchservice.MacOSXListeningWatchService.take(MacOSXListeningWatchService.java:38)
	at io.methvin.watcher.DirectoryWatcher.watch(DirectoryWatcher.java:231)
....

Either this should be clarified in the javadoc or closing the watcher should just unblock watch without throwing an exception

Hi @gmethvin, the 0.9.6 artifact is released but it looks like the tag was not pushed. :-)

Using directory-watcher 0.9.9 on macOS Mojave 10.14.6, the following test fails:

Path testFile = Paths.get("target/testFile.txt");
createIfNeeded(testFile);
DirectoryChangeListener mockListener = mock(DirectoryChangeListener.class);
when(mockListener.isWatching()).thenReturn(true);

DirectoryWatcher watcher = DirectoryWatcher.builder()
        .path(testFile.getParent().toAbsolutePath())
        .listener(mockListener)
        .build();
watcher.watchAsync();

try {
    Files.setLastModifiedTime(testFile, FileTime.from(Instant.now()));
    Thread.sleep(100);
}
finally {
    watcher.close();
}

verify(mockListener, atLeastOnce()).onEvent(any());

.. but if you use .path(testFile.getParent()) instead, it passes. Then, hardcode an absolute path and it fails again.

Hi!
This is just a question to investigate if you had the chance to test it within a Docker context.
Currently, I have a Java 8 microservice with a watching thread (using java.nio.file.WatcherService) that listens to changes to a bind-mounted directory. Here is the relevant part inside the docker-compose config:

services:
  filemanager:
    container_name: filemanager
    restart: always
    build:
      context: filemanager
      dockerfile: Dockerfile
    image: filemanager:latest
    ports:
      - ${FILE_MANAGER_PORT}:${FILE_MANAGER_PORT}
    environment:
      - FILE_MANAGER_PORT=${FILE_MANAGER_PORT}
      - MUSIC_DIRECTORY=${MUSIC_CONTAINER_DIRECTORY}
      - MYSQL_HOST=${MYSQL_CONTAINER_NAME}
      - MYSQL_PORT=${MYSQL_LOCAL_PORT}
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - ENV=DEVELOPMENT
    volumes:
      - ${MUSIC_LOCAL_DIRECTORY}:${MUSIC_CONTAINER_DIRECTORY}
    depends_on:
      - mysql

If I run it without Docker or modify files directly inside Docker, the events are correctly dispatched and captured. If instead, I modify files on Ubuntu (MUSIC_LOCAL_DIRECTORY), just the files are modified inside the container but no events get captured.
I'm trying to understand if I need to do something different or if this feature cannot be used in this context (as said in this post for example). Thanks!

Hi

When walking a folder using Files.walkFileTree if a file / folder cannot be accessed an exception is thrown and the whole walking is finished.

It might be good to handle such cases by overriding visitFileFailed and continue instead of breaking

Hi @gmethvin, it's been a few months since the last release and we'd like to use some of the improvements added recently.

Is there anything we can help you with to get a new release?

On OS-X, starting in an empty folder, setting up a watcher:

@ {
  import $ivy.`io.methvin:directory-watcher:0.9.0`
  import io.methvin.watcher.DirectoryWatcher
  val watcher = DirectoryWatcher
        .builder
        .fileHashing(false)
        .path(java.nio.file.Paths.get("."))
        .listener{ event => println("EVENT " + event.path())}
        .build

  watcher.watchAsync()
}

Followed by a recursive symlink:

$ mkdir folder
$ ln -s ../folder folder/link

Causes a bajillion events to fire:

EVENT /Users/lihaoyi/test/folder
EVENT /Users/lihaoyi/test/folder/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link/link/link/link/link/link/link/link/link/link/link/link/link
EVENT /Users/lihaoyi/test/folder/link

I'm not sure if this a property of the underlying OS-X filesystem API, or part of this library. Is there any way to make it not do that?

Not sure how to sugarcoat it.

DirectoryWatcher
    .builder()
    .path(Paths.get("C:\\Users\\myuser\\Downloads\\"))
    .listener(event -> System.out.println("ping"))
    .build();

Doesn't print out "ping" when I create files there. Generic Windows 10, Java 8.

When on Windows, and NTFS partition, I believe we could use NTFS USN journal. Program called Everything have it implemented and it is blazing fast.

There are the open-source implementations of this:
https://sourceforge.net/projects/ntfs-search/
https://github.com/ChrisS85/FastFileSearch

Community, what do you think?
Personally, I'm using Everything every day (:D), thus am thinking it could of some use to this project.

I came across a very nasty issue while working on openhab/openhab-core#3004 and unfortunately I don't have an easy test-case for that (I can't reproduce it locally, I only see it in GitHub CI builds, so I assume it's a timing issue on slower machines).

What we essentially do is to create a WatchService with the following code:

            dirWatcher = DirectoryWatcher.builder().listener(this).path(basePath).build();
            watchThread = new Thread(dirWatcher::watch, name);
            watchThread.start();

We then inject this WatchService into other OSGi services and they get notified about changes in the watched directory. We have integration tests that check if one of these services (FolderObserver) is correctly processing created and changed files:

https://github.com/openhab/openhab-core/blob/51d6b880ba6d40194dd731af80f32fa6ad7e22e7/itests/org.openhab.core.model.core.tests/src/main/java/org/openhab/core/model/core/internal/folder/FolderObserverTest.java

These tests failed quite often in CI builds because we did not receive the created event for files that have been created in the watched directory after the watch service was initialized. Adding Thread.sleep(1000) after the creation of the watch service immediately solved these issues, so I believe it's probably an issue with the initial hashing/initialization, even if I didn't find an obvious reason for thus behavior.

As I said: I'm unable to provide a simple test case for that but maybe someone comes around and has the same issue...

Hi,

There is an issue on Windows10 (but probably it will be also reproducable in other operating systems).
Steps to reproduce:

• start watcher with file hashing
• copy from outside a folder containing some files
• we will get correctly corresponding CREATE and/or MODIFY events
• then delete the copied folder
• we will get one 'DELETE' event (saying that the folder was deleted - but no information about files)
• then copy again the same folder with the same files
• we will get only event regarding the copied folder but no events that the files were also created/modified

This probably has something to do with the "pathHashes"-variable, because when folder is being deleted only the hash of this folder gets deleted but its children remain in the "pathHashes". So if we copy again same files which are inside of some folder, events will be skipped.

If fileHashing is turned off then this use case works.

There should be an option in directory-watcher to follow symlinks and make sure they are handled properly with different configurations.

I've some issues using the library on Windows 7. I don't know if other Systems are also affected.

The problem is that the watcher stops watching after the following scenario:

• Change to watched root folder (/imports)
• Create folder test1
• Change to newly created folder (/imports/test1)
• Create folder test2
• Change to newly created folder (/imports/test1/test2)
• Create folder test3
• Change to newly created folder (/imports/test1/test2/test3)
• Change to your root folder (/imports)
• Try to delete the folder test1
• Windows cannot delete the directory and shows a dialog ("You must be admin ..." or so, I have only the German message)
• Cancel the dialog
• Create a file in the watched root folder (/imports)

In my case the watcher stops watching at this point. I can now create new files or delete existing ones. The watcher will report nothing.

The issue occurs with the following watcher code:

import io.methvin.watcher.DirectoryChangeEvent.EventType._
import io.methvin.watcher.DirectoryWatcher
val watcher = DirectoryWatcher.create(directory.path, { event =>
  event.eventType() match {
    case CREATE   => logger.info(s"${event.path()} got created")
    case MODIFY   => logger.info(s"${event.path()} got modified")
    case DELETE   => logger.info(s"${event.path()} got deleted")
    case OVERFLOW => logger.info(s"${event.path()} got overflow")
  }
})

watcher.watchAsync(context.system.dispatcher)

This is my output:

[info] a.FileImportService - C:\Users\...\files\import\Neuer Ordner got created
[info] a.FileImportService - C:\Users\...\files\import\Neuer Ordner got deleted
[info] a.FileImportService - C:\Users\...\files\import\test1 got created
[info] a.FileImportService - C:\Users\...\files\import\test1\Neuer Ordner got created
[info] a.FileImportService - C:\Users\...\files\import\test1 got modified
[info] a.FileImportService - C:\Users\...\files\import\test1\Neuer Ordner got deleted
[info] a.FileImportService - C:\Users\...\files\import\test1\test2 got created
[info] a.FileImportService - C:\Users\...\files\import\test1\test2\Neuer Ordner got created
[info] a.FileImportService - C:\Users\...\files\import\test1\test2 got modified
[info] a.FileImportService - C:\Users\...\files\import\test1\test2\Neuer Ordner got deleted
[info] a.FileImportService - C:\Users\...\files\import\test1\test2\test3 got created

I do not think it's a windows specific issue, because if I test this with the file monitor from better-files, then I cannot reproduce this issue.

I just tried out your library, as i found the 'pure' java way using the WatchService kind of ugly - nice work to wrap that with this library!

When playing around, I forgot to get my dummy-file's parent directory when I added the path to your DirectoryWatcher's builder. As a result, I got no events when modifying the file.

The WatchService fails with java.nio.file.NotDirectoryException in this case - with your library everything seemed ok and I didn't get any sing of my mistake other than nothing happen...

So my suggestion would be to either fail for non-directory Paths as java does - or, even better, internally just watch the files parent directory and add a filter to get only events for the respective file. That would be really cool... ;-)

DirectoryWatcher.builder()
                .path(dummyPath.getParent())   // <- initially, I forgot getPatent() for my dummy-textfile
                .listener(event -> System.out.println("  \\-> EVENT = " + event))
                .build()
                .watchAsync();

Play Version

2.8.2

API

Scala

Operating System

MacOS Big Sur Developer Beta Seed 1

JDK

JDK 11

Library Dependencies

Expected Behavior

1. sbt run

Actual Behavior

1. sbt run fails to run with below exception

[error] java.lang.UnsatisfiedLinkError: Unable to load library 'Carbon':
[error] dlopen(libCarbon.dylib, 9): image not found
[error] dlopen(libCarbon.dylib, 9): image not found
[error] 	at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:302)
[error] 	at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:455)
[error] 	at com.sun.jna.Library$Handler.<init>(Library.java:192)
[error] 	at com.sun.jna.Native.loadLibrary(Native.java:646)
[error] 	at com.sun.jna.Native.loadLibrary(Native.java:630)
[error] 	at io.methvin.watchservice.jna.CarbonAPI.<clinit>(CarbonAPI.java:20)
[error] 	at io.methvin.watchservice.jna.CFStringRef.toCFString(CFStringRef.java:23)
[error] 	at io.methvin.watchservice.MacOSXListeningWatchService.register(MacOSXListeningWatchService.java:128)
[error] 	at io.methvin.watchservice.WatchablePath.register(WatchablePath.java:50)
[error] 	at io.methvin.watcher.DirectoryWatcher.register(DirectoryWatcher.java:341)
[error] 	at io.methvin.watcher.DirectoryWatcher.registerAll(DirectoryWatcher.java:315)
[error] 	at io.methvin.watcher.DirectoryWatcher.<init>(DirectoryWatcher.java:176)
[error] 	at io.methvin.watcher.DirectoryWatcher$Builder.build(DirectoryWatcher.java:117)
[error] 	at play.dev.filewatch.DefaultFileWatchService.watch(DefaultFileWatchService.scala:38)
[error] 	at play.dev.filewatch.FileWatchService$$anon$1.watch(FileWatchService.scala:87)
[error] 	at play.runsupport.Reloader.<init>(Reloader.scala:443)
[error] 	at play.runsupport.Reloader$.reloader$lzycompute$1(Reloader.scala:283)
[error] 	at play.runsupport.Reloader$.play$runsupport$Reloader$$reloader$1(Reloader.scala:275)
[error] 	at play.runsupport.Reloader$.startDevMode(Reloader.scala:311)
[error] 	at play.sbt.run.PlayRun$.devModeServer$lzycompute$1(PlayRun.scala:98)
[error] 	at play.sbt.run.PlayRun$.devModeServer$1(PlayRun.scala:81)
[error] 	at play.sbt.run.PlayRun$.$anonfun$playRunTask$3(PlayRun.scala:105)
[error] 	at play.sbt.run.PlayRun$.$anonfun$playRunTask$3$adapted(PlayRun.scala:67)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)

Reproducible Test Case

When I copy-paste a file into target dir on Windows, CREATE event is not reported.

This happens because at the moment of hashing the file it still in use (I don't know exactly why), and Files.exists() returns false as per specification (Files.notExists() would also return false in this case).

Proposal: if we get ENTRY_CREATE event for a file, but can't hash it, still report the CREATE event without updating pathHashes.
An additional Files.notExists() check could also be employed to distinguish between "file has already been deleted" and "file exists but couldn't be accessed" situations.

The cause of the problem is because in Better files, File.Monitor.onUnknownEvent() method has removed count parameter in this commit.

I came a situation where notifications are lost (I think because here: https://github.com/gmethvin/directory-watcher/blob/main/core/src/main/java/io/methvin/watcher/DirectoryWatcher.java#L347) the has is not seen as new.

This appears to arise from the manner in which an editor writes the files. I was using neovim.

I don't know if you will fix this - but the issue may be useful to others

neovim has a couple of ways that it writes when handling backups - the filewatcher will fail if backupcopy=no (or "auto" because then it is sometimes "no")

from neovim help

'backupcopy' 'bkc' string (default: "auto")
global or local to buffer |global-local|
When writing a file and a backup is made, this option tells how it's
done. This is a comma-separated list of words.

The main values are:
"yes" make a copy of the file and overwrite the original one
"no" rename the file and write a new one
"auto" one of the previous, what works best

I also saw this swallowing of events also with xed - probably for similar-ish reasons. And not with nano.

There are exceptions in the stack trace for these file changes that might help diagnose:

This is with neovim

Failing case; backupcopy=no; Final ENTRY_MODIFY is swallowed

2023-02-01 18:27:18 DEBUG DirectoryWatcher:301 - ENTRY_DELETE [/home/.../clerk-demo/notebooks/data_science.clj]
2023-02-01 18:27:18 DEBUG DirectoryWatcher:372 - DirectoryWatcher got an exception while watching!
java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: /home/.../clerk-demo/notebooks/data_science.clj/?
        at java.base/sun.nio.fs.UnixPath.encode(UnixPath.java:121)
        at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:68)
        at java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:278)
        at java.base/java.nio.file.Path.of(Path.java:147)
        at java.base/java.nio.file.Paths.get(Paths.java:69)
        at io.methvin.watcher.PathUtils.subMap(PathUtils.java:43)
        at io.methvin.watcher.DirectoryWatcher.runEventLoop(DirectoryWatcher.java:362)
        at io.methvin.watcher.DirectoryWatcher.lambda$watchAsync$1(DirectoryWatcher.java:232)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
        at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1760)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
2023-02-01 18:27:18 DEBUG DirectoryWatcher:301 - ENTRY_CREATE [/home/.../clerk-demo/notebooks/data_science.clj]
2023-02-01 18:27:18 DEBUG DirectoryWatcher:474 - Skipping create event for path [/home/.../data_science.clj]. Path already hashed.
2023-02-01 18:27:18 DEBUG DirectoryWatcher:301 - ENTRY_MODIFY [/home/.../data_science.clj]

Successful case; backupcopy=yes; ENTRY_MODIFY is sent as MODIFY event.

2023-02-01 18:30:19 DEBUG DirectoryWatcher:301 - ENTRY_MODIFY [/home/.../data_science.clj]
2023-02-01 18:30:19 DEBUG DirectoryWatcher:402 - -> MODIFY [/home/.../clerk-demo/notebooks/data_science.clj] (isDirectory: false)

this is on a rather big multiproject sbt build

JNA: Callback io.methvin.watchservice.MacOSXListeningWatchService$MacOSXListeningCallback@4778113f threw the following exception:
java.lang.IllegalStateException: Queue full
	at java.util.AbstractQueue.add(AbstractQueue.java:98)
	at java.util.concurrent.ArrayBlockingQueue.add(ArrayBlockingQueue.java:312)
	at io.methvin.watchservice.AbstractWatchKey.signalEvent(AbstractWatchKey.java:156)
	at io.methvin.watchservice.MacOSXListeningWatchService$MacOSXListeningCallback.invoke(MacOSXListeningWatchService.java:171)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.sun.jna.CallbackReference$DefaultCallbackProxy.invokeCallback(CallbackReference.java:485)
	at com.sun.jna.CallbackReference$DefaultCallbackProxy.callback(CallbackReference.java:515)
	at com.sun.jna.Native.invokeVoid(Native Method)
	at com.sun.jna.Function.invoke(Function.java:374)
	at com.sun.jna.Function.invoke(Function.java:323)
	at com.sun.jna.Library$Handler.invoke(Library.java:236)
	at com.sun.proxy.$Proxy11.CFRunLoopRun(Unknown Source)
	at io.methvin.watchservice.MacOSXListeningWatchService$CFRunLoopThread.run(MacOSXListeningWatchService.java:105)

I have an implementation that uses MacOSXListeningWatchService directly.

There are 2 parts that use it. A runtime and an agent. Does using 2 of these in one process have conflicts. It is appearing as if im no longer receiving events from one use case

I think it would be a nice addition to allow watching a directory non-recursively.

Currently, when registering a Path with the builder, the watcher is either set up with the FILE_TREE modifier or recursively for all descendant directories, depending on system capabilities. I circumvent this by instantiating the watcher with an empty path list, then initializing paths and pathHashes with a single path and hashes for all files, and then calling register(path, false), all using reflection. This does what I want, but needless to say, that's not great practice ;)

Thanks for the great library!

I am trying to run the watcher in Scala. But even though I am calling watcher.start() (which i expect to run forever), the program finishes immediately. What am I doing wrong?

import better.files._
import io.methvin.better.files._
import scala.concurrent.ExecutionContext.Implicits.global

object FileWatcher
{

  def main(args: Array[String]): Unit =
  {
    val myDir = File("/path/to/dir")
    val watcher = new RecursiveFileMonitor(myDir)
    {
      override def onCreate(file: File, count: Int) = println(s"$file got created")

      override def onModify(file: File, count: Int) = println(s"$file got modified $count times")

      override def onDelete(file: File, count: Int) = println(s"$file got deleted")
    }

    watcher.start()
  }

}

We have found that sometimes editing files that are exposed on a samba share results in multiple create/delete events (up to six events in about half a second) instead of a single modify event.

We have now implemented a workaround in our code (openhab/openhab-core#3404) but it would be great if this could be handled here as I believe it'll be encountered by a larger audience.

DirectoryWatcher is trying to calculate the hash of every created/modified file when the event is emitted. Because of it, if the file is locked by the process writting to it, it can happen that some create/modify events are not emitted.

The error happens in PathUtils.hash(Path file) method. If you enable logging in catch clause, you will see that sometimes hash calculation fails with the following exception:

[ForkJoinPool.commonPool-worker-9] ERROR io.methvin.watcher.PathUtils - Exception when calculating hash
java.io.IOException: The process cannot access the file because it is being used by another process
at java.base/java.io.FileInputStream.readBytes(Native Method)
at java.base/java.io.FileInputStream.read(FileInputStream.java:258)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:106)
at com.google.common.io.ByteSource.copyTo(ByteSource.java:246)
at com.google.common.io.ByteSource.hash(ByteSource.java:326)
at io.methvin.watcher.PathUtils.hash(PathUtils.java:49)
at io.methvin.watcher.DirectoryWatcher.notifyCreateEvent(DirectoryWatcher.java:257)
at io.methvin.watcher.DirectoryWatcher.watch(DirectoryWatcher.java:162)
at io.methvin.watcher.DirectoryWatcher.lambda$watchAsync$0(DirectoryWatcher.java:100)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1603)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)

My question is: Is it necessary to calculate the hash of the files at all? Would it be enough to keep only directories in hash map and emit all the events for the files as they happen?

Here's the gist with the test case that reproduces the issue for create event: https://gist.github.com/chromy96/def29f4e9a0430e90f9ae28434fcc0f7

It'd be nice to remove the dependency on Guava so people don't run into version conflicts in other projects that use Guava. We could probably create our own versions of the few utilities we use from Guava (for file hashing for example).

It seems you're calling CFRunLoopStop and CFRunLoopStop, but according to https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/FSEvents_ProgGuide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40005289?language=objc you also need to call FSEventStreamUnscheduleFromRunLoop, FSEventStreamInvalidate, and FSEventStreamRelease. Not sure what the consequence is of not calling those, but I suppose it's best practice to do so. Probably leaking some small amount of memory if we don't.

Hello Greg and community,

This is maybe more of an informal request rather than a real issue. But maybe you are able to provide some info on this nonetheless.

I am currently trying to integrate your directory watcher 0.10.0 into one of our software components. Integration was quite easy but while running our test suite, I've noticed that file deletion events somehow do not get propagated to our application, while at the same time, file creation events and file modification events work without issues. We cannot really explain this behaviour to ourselves.

We run your directory watcher in a separate thread and start it using the .watch() method.

I've built a minimal, single-threaded POC watching a directory. And there, everything seemed to be working normal (all events received, even the deletion events).

I am not sure what could cause those deletion events to not being generated. Do you have any idea? We're running on Java 13 and macOS Catalina 10.15.6 and macOS 11 Big Sur beta.

Any feedback highly appreciated.

Thanks!
André

• Version: 0.5.0
• OS: Windows 10

DirectoryWatcher is watching C:\Users\jun_w\Goobox\ and the folder has New folder and New folder (2), and New folder has New Text Document.txt, i.e.,

Goobox\
   ├ New folder\
   │   └ New Text Document.txt 
   └ New folder (2)\

When I move New folder to New folder (2), I expected ENTRY_CREATE or ENTRY_MODIFY were emitted to New folder and New Text Document.txt but I got only events for New folder.

Here is log entries I got:

DEBUG io.methvin.watcher.DirectoryWatcher - ENTRY_DELETE [C:\Users\jun_w\Goobox\New folder] 
DEBUG io.methvin.watcher.DirectoryWatcher - ENTRY_DELETE [C:\Users\jun_w\Goobox\New folder] 
DEBUG io.methvin.watcher.DirectoryWatcher - ENTRY_CREATE [C:\Users\jun_w\Goobox\New folder (2)\New folder] 
DEBUG io.methvin.watcher.DirectoryWatcher - ENTRY_MODIFY [C:\Users\jun_w\Goobox\New folder (2)] 

This problem didn't happen in macOS 10.13.3.

In our project we subscribe for multiple tracked folders, and when any of them is (re)moved, the watcher closes itself, and that can be tracked in a completable future.

Example watched paths:
/Users/user/parentDir/watchedDir1
/Users/user/parentDir/watchedDir2

However, when we move the parentDir out of the scope (or simply delete it), the watcher stays silent and keeps watching the basically invalid paths. When the tree is recreated, the changes are detected automatically.

This behaviour looks a bit weird as I'd expect at least some callback from the watcher. Why would it watch the invalid path?

Thank you.

I'd like to propose a change to the public API which makes the library more fitting for applications that do not use global mutable loggers.

In my case, I use directory-watcher in Bloop. Our application uses Nailgun under the hood, and its architecture forces us to avoid global mutable loggers (i.e. neither logback nor log4j), and have a strict control over stdout and stderr. As a result, Bloop has its own slf4j-compliant logger which are immutable and local, and so we create them per client.

This architecture is a little bit at odds with the current implementation of the library, which has a static, global logger in DirectoryWatcher. My suggestion is that we remove it and make DirectoryWatcher.create take the logger as a parameter, so that users are in control of the situation.

I took myself the liberty of implementing this at jvican@a1c0e21, and I released it under my own organization so that I can directly depend on it (0.5.2-a1c0e21c) -- so there's no rush to get this merged on my side.

I open this ticket to raise some discussion. If we agree on this change, I'll open up a pull request. I believe my prototype can be massaged to avoid breaking changes downstream by duplicating all the create constructors to take a logger parameter, and those which don't (the current interface) would directly fallback on the previous logger implementation via Logger.getLogger.

Currently, our macOS watch service fires an ENTRY_DELETE event on deletion of the root. The default JDK watchers don't appear to do this.

One solution for consistency is to simply update the implementation to not fire an event for the root.

It might make sense for DirectoryWatcher to fire a special type of event like ROOT_DELETED in this case. An application might want to handle that case specially because the watcher stops watching that directory completely once it's been deleted.

When running inside Wildfly 27.0.1.Final, a call to waitAsync() fails with

12:01:30,947 ERROR [stderr] (default task-2) Caused by: java.lang.NoClassDefFoundError: com/sun/nio/file/ExtendedWatchEventModifier
12:01:30,947 ERROR [stderr] (default task-2) 	at deployment.project-base-flow-cdi-1.0-SNAPSHOT.war//io.methvin.watcher.DirectoryWatcher.register(DirectoryWatcher.java:454)
12:01:30,947 ERROR [stderr] (default task-2) 	at deployment.project-base-flow-cdi-1.0-SNAPSHOT.war//io.methvin.watcher.DirectoryWatcher.registerAll(DirectoryWatcher.java:433)
12:01:30,947 ERROR [stderr] (default task-2) 	at deployment.project-base-flow-cdi-1.0-SNAPSHOT.war//io.methvin.watcher.DirectoryWatcher.registerPaths(DirectoryWatcher.java:282)
12:01:30,947 ERROR [stderr] (default task-2) 	at deployment.project-base-flow-cdi-1.0-SNAPSHOT.war//io.methvin.watcher.DirectoryWatcher.watchAsync(DirectoryWatcher.java:229)
12:01:30,947 ERROR [stderr] (default task-2) 	at deployment.project-base-flow-cdi-1.0-SNAPSHOT.war//io.methvin.watcher.DirectoryWatcher.watchAsync(DirectoryWatcher.java:215)

The problematic code is

    WatchEvent.Modifier[] modifiers =
        useFileTreeModifier
            ? new WatchEvent.Modifier[] {ExtendedWatchEventModifier.FILE_TREE}
            : new WatchEvent.Modifier[] {};

and as it throws a NoClassDefFoundError, the case is not handled by

      } catch (UnsupportedOperationException e) {
        // UnsupportedOperationException should only happen if FILE_TREE is unsupported
        logger.debug("Assuming ExtendedWatchEventModifier.FILE_TREE is not supported", e);
        fileTreeSupported = false;
        // If we failed to use the FILE_TREE modifier, try again without
        registerAll(start, context);
      }

It would make sense to separate the public API from other libraries used internally. This way it's clear to users which API they should be using, and we can minimize breaking changes only to that API.

The following things should be public:

1. A builder for the DirectoryWatcher that provides all configuration functionality. As part of this we could possibly make DirectoryWatcher an interface, but I'm not sure that's really necessary if we make the constructor private. I don't see a use case for alternative implementations.
2. DirectoryChangeEvent. This could possibly be an interface to make it easier to restructure the class hierarchy in the future.
3. The DirectoryChangeListener interface.
4. The FileHasher interface and some built-in implementations—but not the underlying implementation classes.
5. A FileHash interface that defines a minimal API for file hashes and utility methods for creating FileHash instances.
6. The MacOSXListeningWatchService and related classes. We might be able to extract some interfaces to minimize the public API here as well.

The remainder of the classes could either be made package-private or moved to an internal package. I think the latter might be more convenient.

I was having a look at the Carbon file event API and the current implementation of the OSX's watch service. I realized that the events that Carbon notifies us about are at the directory level, and hence in our callback implementation we're iterating on all the files present in that directory at once and finding the ones that were created/modified/deleted.

The main issue with this implementation is performance. It seems we could improve it by enabling events at the file level with the following flag https://developer.apple.com/documentation/coreservices/1455376-fseventstreamcreateflags/kfseventstreamcreateflagfileevents?language=objc. This way, we don't need to list all the recursive files in every event that we receive every time we get a directory event every 500 ms (the default latency).

Hi, we are experimenting with this system because we desperately need something that works across multiple OS's, so let me first say thank you for making this project public.

Everything works great except on shutdown we get a jvm crash. Would you be willing to help us work through this issue? Thanks!

Disconnected from the target VM, address: '127.0.0.1:62174', transport: 'socket'
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fff97e53fb8, pid=51039, tid=0x000000000000db03
#
# JRE version: Java(TM) SE Runtime Environment (8.0_171-b11) (build 1.8.0_171-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.171-b11 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  [CoreFoundation+0xeafb8]  CFMachPortInvalidate+0x58
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# hs_err_pid51039.log
[thread 48131 also had an error]
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

Steps to reproduce:

1. Having following folder structure:

• folder_parent
  • folder_child

2. Copy this structure under folder where DirectoryWatcher is running
3. Remove copied structure
4. Copy again the structure -> now only CREATE Event for folder_parent was created