Add support for per-directory Mutagen configuration files about mutagen

[xenoscopic]

@IngCr3at1on Ah, no, I'm mostly just looking at command-line-based scripting for the time being. I think at least a few more releases are going to be necessary before an API becomes totally stable. I'm also not sure a REST API will be the answer since a number of commands require bidirectional streaming RPC (for providing credential information and for streaming state updates). Even the command-line-based scripting might be marked as "experimental" for the first few releases.

[xenoscopic]

Mutagen v0.9.0-beta2 has support for custom TOML configuration files (with the same format as ~/.mutagen.toml), which can be specified with the create command's -c/--configuration-file flag. Settings from a manually specified configuration file will be merged on top of the global configuration file (~/.mutagen.toml). Global configuration an also now be disabled with --no-global-configuration. Command line flags will still take precedence over both.

All of this behavior may become a little bit more automatic in the future (e.g. perhaps a .mutagen.toml file in the current working directory would be used automatically), but I've decided to keep it explicit for the time being until this feature is better-vetted.

I'm going to close this issue, but we can open another to discuss potential automatic loading of these files in the future.

[Toilal]

To extend this feature request, I think it could be interesting to setup a synchronisation session by performing something like mutagen up, mutagen start, mutagen stop, mutagen down command inside the repository directory.

This could create, resume, pause or destroy the session by reading settings inside a project local mutagen.toml lying in the repository.

[session]
alpha = . 
beta = [email protected]:~/projects/my-project

It could be useful to support environment variable substitution, and make some special token available like ${dirname}, ${cwd} ...

export [email protected]

[session]
alpha = ${cwd}
beta = ${MUTAGEN_REMOTE_SSH}:~/projects/${dirname}

It's somehow related to #40

[xenoscopic]

I want to understand what you're asking a little bit better before commenting too much. My understanding is that there are two issues:

1. You want to be able to store a mutagen.toml file in your repository that Mutagen will pick up on when it creates sessions. As @Toilal mentioned, this is somewhat related to issue #40.
2. You want to be able to run an instance of the daemon that is specific to the repository that you're dealing with.

Regarding the first issue, there's not currently a way to do this. The mutagen.toml file in the home directory contains (and will eventually contain more) options that are not session-specific (e.g. options to control the Mutagen daemon), and thus it would need to be adapted to work inside a specific directory. As with #40, the best way to do this for now is just to create a script (called something like mutagen_start.sh) which just invokes mutagen create with the specific options that you'd want to include in your per-repository mutagen.toml. The mutagen create command accepts all of the same options that can be specified in mutagen.toml and will take precedence over options in mutagen.toml, so you can still achieve the same result, i.e. all of your configuration in a single file that lives in the repository.

Regarding the second issue, there aren't really any plans to have per-session daemons at the moment. I think that if the daemon can understand a per-repository mutagen.toml file (or equivalent), then there's not really a need for this. I also plan to do a bit of work to "hide" the daemon from users, e.g. starting it automatically when any Mutagen command is run, so that hopefully it's not front-and-center anymore.

[xenoscopic]

@Toilal Regarding specifying sessions, would this not be something better-suited to a shell script as well? E.g. mutagen_start.sh, mutagen_stop.sh, etc? It would be infinitely more flexible than any file format Mutagen could support, and already supports all of the environment variable substitution that you want.

[kritop]

1. yes basically i have mutagen in a repo (~/dev/dockerized-app-git-repo/bin/mutagen) i want to keep everything contained in the repo to not pollute the developers environment. hence i want also to keep the configuration in the very same repo.
2. i guess this doesn't really matter so much i was just not sure if the configuration file is more relevant for the daemon or the mutagen create command.

[xenoscopic]

@kritop Okay, understood. So regarding (1), I will try to put together a design proposal for this in early November, hopefully also addressing #40, but in the mean time the best workaround would be to just create a shell script that you check into the repository to deliver the configuration directly to the mutagen create command. If you do this nicely, it can almost look like a configuration file, e.g.

#!/bin/sh

mutagen create \
    --option1=value1 \
    --option2=value2 \
    [...other options] \
    --ignore path1 \
    --ignore path2 \
    [...other ignores] \
    . $1

This also gives you the flexibility to substitute environment variables like @Toilal was proposing. Once per-repository configuration files are a reality, it should be relatively easy to convert. If you have a repository that multiple developers work on, I'd be interested in feedback on what such a script ends up looking like for you, because that would inform the design of a configuration file.

Regarding (2), the configuration file is more relevant for the mutagen create command at the moment, but eventually some daemon options will go into it.

It might also be worth adding a --no-global-configuration flag to mutagen create in the mean time that could be added to ensure that ~/.mutagen.toml is ignored if you're using a script or per-repository configuration file.

[kritop]

so basically i could turn this:

[symlink]
mode = "ignore"

[ignore]
default = [
    # System files
    ".DS_Store",
    "._*",
]
vcs = true

into

#!/bin/sh

mutagen create \
    --symlink-mode ignore \
    --ignore-vcs \
    --ignore ".DS_Store" \
    --ignore "._*"  \
    . $1

is that correct?

[xenoscopic]

Yep, that looks correct. Of course, whether or not you want to include the $1 to substitute the remote URL is up to you, perhaps you'd want to structure the script a little differently, and in that example it might be better to quote it as "$1". But the ignore and option parameters are correct.

[Toilal]

Do you think this feature should belong to mutagen core project, or to some kind of companion project ?

The docker team has made this choice with docker and docker-compose.

I just read your previous comments about shell scripts, that's seems to mean some kind of companion project too. (btw, I run mutagen daemon from windows, so shell script is not an option in my case, but maybe I could write a Python project and distribute it as an .exe like docker-compose)

[xenoscopic]

Hey @Toilal, I'm glad you brought this up. Looking forward to v0.9.0 (and Mutagen's future in general), I've been thinking a lot about this topic (and issues #38, #40, #50, #57 by extension). I think that all of these have a related solution (so I'll CC in @matthew-gill, @divoxx, and @justjake).

Essentially, the situation is that people need a way to manage the configuration and lifecycle of Mutagen sessions in coordination with their development workflow. Unfortunately, these workflows vary widely and could include:

• Manual command invocation
• Custom scripts
• Vagrant
• Docker Compose
• Kubernetes
• Command invocation via a text editor plugin
• Some future container orchestration tool that scientists will discover on Mars

I really do think it makes sense to have some sort of automated orchestration mechanism to manage Mutagen sessions, but I think that it might be best for this tool to start life outside of Mutagen, and for Mutagen to provide the hooks and APIs necessary to facilitate this orchestration. This would allow for a lot more flexibility in terms of implementation and interface. It might even make sense to try a couple of different approaches and see what works best (which might be different for different workflows). If a clear winner eventually emerges, we could look at merging it back upstream, but sometimes things can develop faster if they remain decoupled. If someone wants to take up the mantle on this, I'm happy to help from the Mutagen side.

So if we look at what Mutagen would need to provide for an external project to orchestrate sessions, the "obvious" solution would be to have the APIs or scripting interface described in issues #38 and #57. The idea would be that third-party programs could create sessions, track their IDs, and manage their lifecycle. While this is straightforward enough, there are two issues that come to mind:

1. The design of this interface needs to be stable and extensible, and (especially for the list/monitor commands) it's really difficult to get this right, at least right now. Even Mutagen's internal session management APIs are continuing to evolve.
2. Even with these APIs, the orchestration tool using this would have to reimplement a lot of infrastructure (that already exists inside the daemon) for tracking session IDs and managing the associated sessions.

I've been trying to think about how we might be able to take advantage of Mutagen's session tracking infrastructure inside external tools with something less than a full API. One of the ideas I've been playing with is session "tags". A tag would be sort of a session namespace that could be used to reference the sessions within it. The idea would be to allow tools to attach a tag to one or more sessions and then control those sessions in unison using that tag. For example, an orchestration system could do something like:

mutagen create --tag=project1 ./web-build docker://webcontainer/var/www
mutagen create --tag=project1 . docker://apicontainer/path/to/server/root

This would create two sessions, which would still have their own session IDs, but which could also be addressed by tag, e.g.

mutagen flush --tag=project1
mutagen pause --tag=project1
mutagen resume --tag=project1
mutagen terminate --tag=project1

Specifying a tag to the lifecycle management commands would affect all sessions associated with that tag. Sessions could also be associated with multiple tags if a project needed more granular addressing (even down to the individual session level).

The benefit here is that the orchestration tool doesn't need to track individual session IDs and lifecycles - it just needs to generate a unique tag that can be used to control sessions. For custom orchestration scripts, this tag might just be the project name. For more advanced orchestration tools, this might be some randomly generated ID that's then stuck in a .lock file or something. But either way, it allows for granular control without the need to track individual session IDs.

What do you think?

As far as what the orchestration tool might look like, I was thinking that it could function similarly to Docker Compose. Let's say the orchestration tool is called mutagen-choreograph (the exact name is left as an exercise to the developer). You might have a file called mutagen-choreograph.toml that sits inside your project and has a configuration along the lines of:

[sessions]

    [sessions.web-build]
    syncMode = "one-way-safe"
    alpha = "./web-build"
    beta = "docker://webcontainer/var/www"
    ignores = [
        "./web-build/node_modules",
    ]

    [sessions.api]
    syncMode = "one-way-safe"
    alpha = "."
    beta = "docker://apicontainer/path/to/server/root"
    ignores = [
        "*.pyc",
    ]

You might then run mutagen-choreograph create to create/start the sessions. This would generate some random tag and invoke creation (via mutagen create) or each session. You could then have commands like mutagen-choreograph flush/resume/pause/terminate to control their lifecycle.

This is just a rough idea. But the key point is that, in addressing the orchestration problem, it also addresses issues #40 and #43.

If any of you would be interested in experimenting with a tag-based interface for managing sessions, I could potentially get out a v0.9.0-beta1 release in the next 1-2 days which adds experimental support for this. Let me know what you think, and thanks for keeping the discussion alive.

[Toilal]

I really do think it makes sense to have some sort of automated orchestration mechanism to manage Mutagen sessions, but I think that it might be best for this tool to start life outside of Mutagen, and for Mutagen to provide the hooks and APIs necessary to facilitate this orchestration.

Yes, I fully agree ! And it doesn't have to be written in Go too, but it should support all platforms like mutagen.

I've been trying to think about how we might be able to take advantage of Mutagen's session tracking infrastructure inside external tools with something less than a full API. One of the ideas I've been playing with is session "tags". A tag would be sort of a session namespace that could be used to reference the sessions within it. The idea would be to allow tools to attach a tag to one or more sessions and then control those sessions in unison using that tag.

Yes, would be really nice for future features concerning some mutagen-choreograph tool. In the docker world, they introduced Labels that serve this purpose. Maybe we could use the same terminology ? Of course, a single synchronisation session may have one or many label.

As far as what the orchestration tool might look like, I was thinking that it could function similarly to Docker Compose. Let's say the orchestration tool is called mutagen-choreograph (the exact name is left as an exercise to the developer). You might have a file called mutagen-choreograph.toml that sits inside your project and has a configuration along the lines of:

I'm ok with this name, it sounds just like what it's suppose to do :).

This would create two sessions, which would still have their own session IDs, but which could also be addressed by tag, e.g.

mutagen flush --tag=project1
mutagen pause --tag=project1
mutagen resume --tag=project1
mutagen terminate --tag=project1

About this part, I think there should be a way to perform command on all sessions too, and somethink like a filter. The docker system prune command support this kind of syntax which is really explicit, and can support other filter (based on beta url for example, or any other session setting ...)

mutagen pause --all --filter=label=project1
mutagen pause --all --filter=beta=*192.168.1.100*

This label feature may be a good start to init a first version of mutagen-choreograph, but an API will be needed quickly. But anyway this is still a good feature anyway for various mutagen integration.

[xenoscopic]

Yes, I fully agree ! And it doesn't have to be written in Go too, but it should support all platforms like mutagen.

I'd go so far as to say that it shouldn't be written in Go (unless it's your primary development language). Something like JavaScript or Python might be a better fit for faster initial iteration and/or potential integration with your existing toolchains and workflows.

In the docker world, they introduced Labels that serve this purpose. Maybe we could use the same terminology ? Of course, a single synchronisation session may have one or many label.

About this part, I think there should be a way to perform command on all sessions too, and somethink like a filter. The docker system prune command support this kind of syntax which is really explicit, and can support other filter (based on beta url for example, or any other session setting ...)

"Tag" and "label" are essentially synonymous, so I guess I wouldn't mind the "label" terminology if it's more consistent. On the other hand, I wouldn't want people to get confused when talking about synchronization sessions and containers in the same sentence.

Looking at their docs though, the labels they support seem a little more advanced (with key/value pairs). Let me read up on that a bit and see if a similar design makes sense for Mutagen. I can imagine that it might. Thanks for sending these links.

There is currently the -a/--all flag for the flush/pause/resume/terminate commands which can operate on all sessions. I could imagine a globbing/filtering syntax for tags/labels as well.

I'm ok with this name, it sounds just like what it's suppose to do :).

My only concern is that "choreograph" is a long word to type, but I guess there's autocomplete for that. I'll leave the honor of choosing a name to whoever wants to implement such a thing.

This label feature may be a good start to init a first version of mutagen-choreograph, but an API will be needed quickly. But anyway this is still a good feature anyway for various mutagen integration.

I agree, more will be needed, but I'd rather add it as the need arises. The best way to figure out exactly what's needed will probably be through initial and continued development of such a tool.

[IngCr3at1on]

I really like the idea of tags (and personally I prefer tags over labels, if only cause it's shorter) you've outlined here @havoc-io. The issue with 'tags' is it could potentially be confused with git the same as labels and containers (I mean hopefully not but anyway).

Thinking in terms of resource contention (on low resource devices) I would suggest a flag allowing the option to disable (or enable if you prefer) the external (presumably non-gRPC (you could still reuse many of the existing protofiles for this I believe (haven't looked in too much detail yet at said files)?) API.

Oddly enough if I were to write something like choreograph (mutagenisis maybe?) I would use go but it also is my primarily language 😂

[phromo]

If relevant to this design process, I wrote

Usage: mutagen-compose [up | down | reload]

Available at https://gist.github.com/phromo/5dcfe1a0213c96c2d44792eac1c43c61 for my own usecase of having a .mutagen-session.toml in the synced root folder. It currently only handles session-specific ignore but could easily be extended to support all conf.

Sample .mutagen-session.toml:

[roots]
alpha = "~/tmp/mutagen-test"
beta = "[email protected]:~/tmp/mutagen-tmp"

[ignore]
include = ["~/.mutagen/codeignore.toml"]
default = ["*.pyc", ".idea", "*.elc"]

[xenoscopic]

Thinking in terms of resource contention (on low resource devices) I would suggest a flag allowing the option to disable (or enable if you prefer) the external (presumably non-gRPC (you could still reuse many of the existing protofiles for this I believe (haven't looked in too much detail yet at said files)?) API.

@IngCr3at1on Can you elaborate on this? I'm not sure I understand what you mean.

@phromo That's great, thanks for sharing! I'm going to start on the tag implementation described above tomorrow, so hopefully you'll be able to give that a shot and see if it helps clean up any code. I think feedback based on an actual orchestration tool is going to be really helpful in guiding and vetting that design.

[IngCr3at1on]

@IngCr3at1on Can you elaborate on this? I'm not sure I understand what you mean.

Sorry @havoc-io I conflated 2 things into 1...

• the first was just suggesting an option to run the mutagen daemon with the external API disabled
• the second was just me thinking of ways you could build such an API (while maintaing a DRY codebase by reusing your gRPC API for what I assume is meant to be REST); sorry force of habit from building gRPC services and REST APIs as my main job lol.

[xenoscopic]

@IngCr3at1on No problem, thanks for clarifying.

There's really only one daemon API, which is the (internal) gRPC API. There's no external API (gRPC or otherwise) at the moment, though I'm hoping that the internal gRPC API will eventually stabilize to a point where third-party tooling can use it directly. If the API is disabled, all of the session management commands (create/list/monitor/pause/resume/flush/stop) will cease to function. Existing sessions will continue to synchronize, but there won't be any way to control them.

I do agree that certain parts of the Go gRPC implementation are a bit memory-heavy (e.g. there's no message buffer re-use, something that they're still working on AFAIK), but I don't think it's the biggest memory usage culprit in the daemon, so I'm not sure disabling it would help much. I think the bigger issue is just the Go language and runtime in general. The runtime just assumes an environment with a "large" (some might say "modern"... though I wish that weren't the case) amount of resources, probably ~100 MB of memory per process. I've done a bit of work trying to reduce Mutagen's memory usage, but unfortunately it's pretty close to the lowest that it can be (I might be able to shave off another 10%). Even when you do try to optimize for low memory usage in Go, there are still limits to the granularity of your control since the runtime is responsible for releasing memory back to the OS. There may be some gains to be had from upgrading Mutagen's build to Go 1.12, but I don't think it's going to make a huge difference.

This issue probably isn't the place to go into it, but there are some embedded Go projects and other runtime knobs that can be turned which might help facilitate running the daemon in low-memory environments. I'd be willing to experiment with these, but I can't promise any success, because they generally only work in specialized cases. If you'd like, drop me an email with a bit more information about your setup and I'll try to reproduce and work around the resource limits you're encountering.

[IngCr3at1on]

@havoc-io sorry I wasn't entirely clear I wasn't suggesting disabling gRPC itself; I thought from reading part of the issue that you were considering building a secondary REST API over it and I was suggesting we have a way to disable that functionality was all. Sorry again for confusion; as for the other runtime knobs, etc I was thinking about some of those as well but yes definitely not for this issue (or a very high priority at least for me lol).