merofeev / docker-windows-volume-watcher Goto Github PK

View Code? Open in Web Editor NEW

188.0 188.0 30.0 31 KB

A tool to notify Docker contianers about changes in mounts on Windows.

License: MIT License

Python 100.00%

docker-windows-volume-watcher's People

Contributors

Stargazers

Watchers

docker-windows-volume-watcher's Issues

Duplicate notification - a single change triggers 2 notifications to the container.

Hi, all notifications of changes are duplicated, for some reason the container receives double the modification number.
A single file modified change, triggers 2 container notify calls.

docker-volume-watcher *node* -v
INFO:root:Notifier c:\Users/<user>/repository -> app_test_nodejs_run_12:/app created.
INFO:root:Container app_test_nodejs_run_12 has 1 watched directories
INFO:root:Notifying container app_test_nodejs_run_12 about change in /app/x.js
INFO:root:
INFO:root:Notifying container app_test_nodejs_run_12 about change in /app/x.js
INFO:root:

Suggested workaround:

Could you add an option for a slight delay like 100 ms, to prevent docker-volume-watcher from triggering false notifications (when it comes to the same container) ?
e.g. docker-volume-watcher --delay 100

Versions:

pip list

argh                          0.26.2
certifi                       2018.10.15
chardet                       3.0.4
docker                        3.5.1
docker-pycreds                0.3.0
docker-windows-volume-watcher 1.1.0
idna                          2.7
pathtools                     0.1.2
pip                           18.1
pypiwin32                     223
pywin32                       224
PyYAML                        3.13
requests                      2.20.0
setuptools                    39.0.1
six                           1.11.0
urllib3                       1.24
watchdog                      0.9.0
websocket-client              0.53.0

WARNING:root:Bind of container was skipped since it has invalid source path

Hello,

One of my docker container requires access to a directory where it tracks file changes within that directory (using the watchdog library).

I tried using docker-windows-volume-watcher to maintain the watchdog alerts within the docker container, however when I launch it, I get the error "WARNING:root:Bind of container was skipped since it has invalid source path" where the path itself is a standard windows path : C:\path\to\the\directory_containing_the_files

Am I missing something here ?

skip read-only bind volumes

If a bind-mount from the windows host is read-only the attempt to chmod(1) the file will fail; that should be discoverable from the details of the containers, and so let you skip (or at least warn) about those mounts.

Add support for "delete" events (or elaborate more on why not possible)

I know this isn't currently built, however is it possible to propagate notifications on file deletions? I noticed that the documentation pointed out this limitation but I'd like to know if it's due to some issue out of the control of this library or maybe just due to time constraints.

client version 1.35 is too new

When starting the tool, it gives me the error
docker.errors.APIError: 400 Client Error: Bad Request ("client version 1.35 is too new. Maximum supported API version is 1.34")
I'm using docker-toolbox 18.02.0-ce on windows

Full stack trace is

$ docker-volume-watcher.exe
Traceback (most recent call last):
  File "c:\users\hgiesenow\appdata\local\programs\python\python36\lib\site-packages\docker\api\client.py", line 225, in _raise_for_status
    response.raise_for_status()
  File "c:\users\hgiesenow\appdata\local\programs\python\python36\lib\site-packages\requests\models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://192.168.99.100:2376/v1.35/containers/json?limit=-1&all=0&size=0&trunc_cmd=0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\hgiesenow\appdata\local\programs\python\python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\hgiesenow\appdata\local\programs\python\python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\hgiesenow\AppData\Local\Programs\Python\Python36\Scripts\docker-volume-watcher.exe\__main__.py", line 9, in <module>
  File "c:\users\hgiesenow\appdata\local\programs\python\python36\lib\site-packages\docker_volume_watcher\cli.py", line 34, in main
    monitor.find_containers()
  File "c:\users\hgiesenow\appdata\local\programs\python\python36\lib\site-packages\docker_volume_watcher\container_monitor.py", line 67, in find_containers
    for container in self.client.containers.list():
  File "c:\users\hgiesenow\appdata\local\programs\python\python36\lib\site-packages\docker\models\containers.py", line 879, in list
    since=since)
  File "c:\users\hgiesenow\appdata\local\programs\python\python36\lib\site-packages\docker\api\container.py", line 198, in containers
    res = self._result(self._get(u, params=params), True)
  File "c:\users\hgiesenow\appdata\local\programs\python\python36\lib\site-packages\docker\api\client.py", line 231, in _result
    self._raise_for_status(response)
  File "c:\users\hgiesenow\appdata\local\programs\python\python36\lib\site-packages\docker\api\client.py", line 227, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "c:\users\hgiesenow\appdata\local\programs\python\python36\lib\site-packages\docker\errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 400 Client Error: Bad Request ("client version 1.35 is too new. Maximum supported API version is 1.34")

Errors when change files from JetBrains IDE

Hi, thx for your script, it really helps)

But i have some trouble. I think is not your problem, but I can offer a solution.

History

In work i use JetBrains IDE. And it has specific file save algorithm. On file save IDE create tmp file, update it, then move real file to old backup, then replace real with tmp.

In my test tool it looks like:

INFO:root:EVENT created: ./test\index.js___jb_tmp___ - ./test\index.js___jb_tmp___
INFO:root:EVENT modified: ./test\index.js___jb_tmp___ - ./test\index.js___jb_tmp___
INFO:root:EVENT moved: ./test\index.js - ./test\index.js___jb_old___
INFO:root:EVENT moved: ./test\index.js___jb_tmp___ - ./test\index.js
INFO:root:EVENT deleted: ./test\index.js___jb_old___ - ./test\index.js___jb_old___

You can see two suffixes: ___jb_tmp___ and ___jb_old___

Of course i know about ignore_patterns and add following suffixes to this list. But it not filter move event. With ignores patterns ignore_patterns=["*___jb_tmp___", "*___jb_old___"] it looks like:

INFO:root:EVENT moved: ./test\index.js - ./test\index.js___jb_old___
INFO:root:EVENT moved: ./test\index.js___jb_tmp___ - ./test\index.js

Where second path is event.dest_path (what you use as host_path in __change_handler).

Example

If i run your script as:

docker-volume-watcher --verbose -e *.git* *build* *node_modules* *.docker* *.idea* *___jb_tmp___ *___jb_old___ --debounce 0.4

then i get following output (i have 3 containers with one volume):

INFO:root:Notifying container nginx-test about change in /var/www/app/assets/scss
INFO:root:Notifying container php-fpm-test about change in /var/www/app/assets/scss
INFO:root:Notifying container node-test about change in /var/www/app/assets/scss
INFO:root:
INFO:root:Notifying container nginx-test about change in /var/www/app/assets/scss/app.scss___jb_old___
INFO:root:
INFO:root:Notifying container php-fpm-test about change in /var/www/app/assets/scss/app.scss___jb_old___
INFO:root:
INFO:root:Notifying container node-test about change in /var/www/app/assets/scss/app.scss___jb_old___
ERROR:root:Exec run returned non-zero exit code: 1
INFO:root:Notifying container php-fpm-test about change in /var/www/app/assets/scss/app.scss
ERROR:root:Exec run returned non-zero exit code: 1
INFO:root:Notifying container nginx-test about change in /var/www/app/assets/scss/app.scss
ERROR:root:Exec run returned non-zero exit code: 1
INFO:root:Notifying container node-test about change in /var/www/app/assets/scss/app.scss
INFO:root:
INFO:root:
INFO:root:

As you can see, container try to interract with app.scss___jb_old___ that really does not exists at that moment (already moved to app.scss).

Possible solution

You can add replace in processing relative_host_path for remove ___jb_tmp___ and ___jb_old___ substrings from path. In reasult we take 2 events of app.scss that can easly be fixed by debouncer.

What do you think?

I can create PR, but I don't know Python well =)

Error when detecting changes

this is my docker info

Containers: 3 Running: 1 Paused: 0 Stopped: 2 Images: 21 Server Version: 17.12.0-ce Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 89623f28b87a6004d4b785663257362d1658a729 runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 4.9.60-linuxkit-aufs Operating System: Docker for Windows OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 1.934GiB Name: linuxkit-00155d014d01 ID: ATPD:LEOL:LNRR:KEHA:NR6R:AVKN:LQR6:5XF5:W7VT:6HUP:GUS5:ONDU Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): true File Descriptors: 29 Goroutines: 55 System Time: 2018-02-02T01:09:14.2428761Z EventsListeners: 2 Registry: https://index.docker.io/v1/ Labels: Experimental: true Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

I'm running docker-volume-watcher
and when I change some file throws this exception

Exception in thread Thread-1: Traceback (most recent call last): File "c:\users\XXX\appdata\local\programs\python\python36-32\lib\threading.py", line 916, in _bootstrap_inner self.run() File "c:\users\XXX\appdata\local\programs\python\python36-32\lib\site-packages\watchdog\observers\api.py", line 199, in run self.dispatch_events(self.event_queue, self.timeout) File "c:\users\XXX\appdata\local\programs\python\python36-32\lib\site-packages\watchdog\observers\api.py", line 368, in dispatch_events handler.dispatch(event) File "c:\users\XXX\appdata\local\programs\python\python36-32\lib\site-packages\watchdog\events.py", line 454, in dispatch _method_map[event_type](event) File "c:\users\XXX\appdata\local\programs\python\python36-32\lib\site-packages\docker_volume_watcher\container_notifier.py", line 49, in __change_handler self.notify(absolute_path) File "c:\users\XXX\appdata\local\programs\python\python36-32\lib\site-packages\docker_volume_watcher\container_notifier.py", line 66, in notify permissions = permissions.decode('utf-8').strip() AttributeError: 'ExecResult' object has no attribute 'decode'

I'm unable to find anything related to this on docs and previous issues, so any help will be appreciated

thanks in advance

error message constantly printed

pip show docker-windows-volume-watcher
Version: 1.0.8

pip show docker
Version: 3.5.0

docker --version
Docker version 18.06.1-ce, build e68fc7a

when I run:

docker-volume-watcher.exe

this is printed every second:

ERROR:root:Exec run returned non-zero exit code: 1

beside of this everything works as expected (a container is informed when a change is made).

Only "attrib" event is ever sent to inotify

I know this is probably inherent in using "stat" and "chmod" as the propagation technique, but the point stands. It would be nice if events like close_write, modify and so forth could be distinguished.

Events not triggering when watched folder is a Junction or SymLink

I have a folder into which I've junctioned plenty of stuff, but the events only work up until the point I go into the junctioned folder.

Normally, Java won't notice changes in these folders if they are SymLinks, but when using Junctions, the watcher events do trigger. I was hoping junctions would work with this too. :)

Doesn't Work for WSL2

Running this command in Windows subsystem for linux 2 seems to break it.

this is the message i get for my container:


WARNING:root:Bind of container web_1 was skipped since it has invalid source path /run/desktop/mnt/host/d/user/project_dir

Error on when it notices change

Hi, whenever your script finds a change it gives me error. Do you have any suggestions?

printout:

Exception in thread Thread-9:
Traceback (most recent call last):
  File "c:\python\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "c:\python\lib\site-packages\watchdog\observers\api.py", line 199, in run
    self.dispatch_events(self.event_queue, self.timeout)
  File "c:\python\lib\site-packages\watchdog\observers\api.py", line 368, in dispatch_events
    handler.dispatch(event)
  File "c:\python\lib\site-packages\watchdog\events.py", line 454, in dispatch
    _method_map[event_type](event)
  File "c:\python\lib\site-packages\docker_volume_watcher\container_notifier.py", line 49, in __change_handler
    self.notify(absolute_path)
  File "c:\python\lib\site-packages\docker_volume_watcher\container_notifier.py", line 65, in notify
    ['stat', '-c', '%a', absolute_path], privileged=True)
  File "c:\python\lib\site-packages\docker\models\containers.py", line 162, in exec_run
    resp['Id'], detach=detach, tty=tty, stream=stream, socket=socket
  File "c:\python\lib\site-packages\docker\utils\decorators.py", line 34, in wrapper
    return f(self, *args, **kwargs)
  File "c:\python\lib\site-packages\docker\utils\decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "c:\python\lib\site-packages\docker\api\exec_api.py", line 156, in exec_start
    return self._read_from_socket(res, stream, tty)
  File "c:\python\lib\site-packages\docker\api\client.py", line 377, in _read_from_socket
    return six.binary_type().join(gen)
  File "c:\python\lib\site-packages\docker\utils\socket.py", line 73, in frames_iter
    n = next_frame_size(socket)
  File "c:\python\lib\site-packages\docker\utils\socket.py", line 60, in next_frame_size
    data = read_exactly(socket, 8)
  File "c:\python\lib\site-packages\docker\utils\socket.py", line 45, in read_exactly
    next_data = read(socket, n - len(data))
  File "c:\python\lib\site-packages\docker\utils\socket.py", line 32, in read
    return os.read(socket.fileno(), n)
OSError: [Errno 9] Bad file descriptor

Thanks

This is broken with Docker 17.12.0-ce

After upgrading to Docker 17.12.0-ce, I see these warnings and files changes are no longer picked up by docker:

WARNING:root:Bind of container environmentrefreshapp_environmentrefresh_1 was skipped since it has invalid source path /host_mnt/c/Projects/Foundation/EnvironmentRefreshApp/server/Main
WARNING:root:Bind of container environmentrefreshapp_environmentrefresh_1 was skipped since it has invalid source path /host_mnt/c/Projects/Foundation/EnvironmentRefreshApp/server/Controllers
WARNING:root:Bind of container environmentrefreshapp_environmentrefresh_1 was skipped since it has invalid source path /host_mnt/c/Projects/Foundation/EnvironmentRefreshApp/client/src
WARNING:root:Bind of container environmentrefreshapp_environmentrefresh_1 was skipped since it has invalid source path /host_mnt/c/Projects/Foundation/EnvironmentRefreshApp/client/e2e
WARNING:root:Bind of container environmentrefreshapp_environmentrefresh_1 was skipped since it has invalid source path /host_mnt/c/Projects/Foundation/EnvironmentRefreshApp/server/Models
WARNING:root:No mounts match container name pattern * and host directory pattern c:\Projects\Foundation\EnvironmentRefreshApp

Failure when mounted volume is a file

It fails when a volume is mounted to a file. Maybe those bindings should be ignored ?

log std{out,err} when running the stat/chmod commands in the container fails

If running a command inside the container fails then it'd be good to log the output -- that gives a more direct clue about what (if anything) went wrong with the process than just the exit code.

Error on reading permissions when using docker v3.0.0

Issue

I received the following error when watching local file changes while running under docker v3.0.0:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "c:\anwendungen\python36-32\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "c:\anwendungen\python36-32\lib\site-packages\watchdog\observers\api.py", line 199, in run
    self.dispatch_events(self.event_queue, self.timeout)
  File "c:\anwendungen\python36-32\lib\site-packages\watchdog\observers\api.py", line 368, in dispatch_events
    handler.dispatch(event)
  File "c:\anwendungen\python36-32\lib\site-packages\watchdog\events.py", line 454, in dispatch
    _method_map[event_type](event)
  File "c:\anwendungen\python36-32\lib\site-packages\docker_volume_watcher\container_notifier.py", line 49, in __change_handler
    self.notify(absolute_path)
  File "c:\anwendungen\python36-32\lib\site-packages\docker_volume_watcher\container_notifier.py", line 66, in notify
    permissions = permissions.decode('utf-8').strip()
AttributeError: 'ExecResult' object has no attribute 'decode'

Docker version (pip show docker):

Name: docker
Version: 3.0.0
Summary: A Python library for the Docker Engine API.
Home-page: https://github.com/docker/docker-py
Author: Joffrey F
Author-email: [email protected]
License: Apache License 2.0
Location: c:\anwendungen\python36-32\lib\site-packages
Requires: websocket-client, pypiwin32, docker-pycreds, six, requests

Solution

Calling output on permissions solves the issue for docker >=3.0.0.

docker_volume_watcher (https://github.com/merofeev/docker-windows-volume-watcher/blob/master/docker_volume_watcher/container_notifier.py#L66)

...
 permissions = self.container.exec_run(
                ['stat', '-c', '%a', absolute_path], privileged=True)
            permissions = permissions.output.decode('utf-8').strip()
            response = self.container.exec_run(
                ['chmod', permissions, absolute_path], privileged=True)
...

The cause seems to be that the output of exec_run changed to an ExecResult instead of a string with the version bump to 3.0.0: https://github.com/docker/docker-py/blob/05d34ed1fbaa8233a4cf51a0f52b67aef99a9521/docker/models/containers.py#L129.

Ideally, permissions is checked for its type before calling decode on it.

Watcher is triggered but container not notified

I got everything setup and have the watcher running in verbose mode to debug. It correctly recognizes containers and when I update a file in a volume, it's shown in the watcher output. It says it's notifying the container but the file change actions that are expected (hugo server) are never run. Seems like it can read container information fine but has trouble sending information back.