vcs-python / vcspull

🔄 Synchronize projects via yaml/json manifest. Built using `libvcs`.

$ vcspull ·

Manage and sync multiple git, svn, and mercurial repos via JSON or YAML file. Compare to myrepos, mu-repo. Built on libvcs.

Great if you use the same repos at the same locations across multiple machines or want to clone / update a pattern of repos without having to cd into each one.

• clone / update to the latest repos with $ vcspull
• use filters to specify a location, repo url or pattern in the manifest to clone / update
• supports svn, git, hg version control systems
• automatically checkout fresh repositories
• supports pip-style URL's (RFC3986-based url scheme)

See the documentation, configuration examples, and config generators.

How to

Install

$ pip install --user vcspull

Developmental releases

You can test the unpublished version of vcspull before its released.

• pip:

  $ pip install --user --upgrade --pre vcspull

• pipx:

  $ pipx install --suffix=@next 'vcspull' --pip-args '\--pre' --force

  Then use vcspull@next sync [config]....

Configuration

Add your repos to ~/.vcspull.yaml.

vcspull does not currently scan for repos on your system, but it may in the future

~/code/:
  flask: "git+https://github.com/mitsuhiko/flask.git"
~/study/c:
  awesome: "git+git://git.naquadah.org/awesome.git"
~/study/data-structures-algorithms/c:
  libds: "git+https://github.com/zhemao/libds.git"
  algoxy:
    repo: "git+https://github.com/liuxinyu95/AlgoXY.git"
    remotes:
      tony: "git+ssh://[email protected]/tony/AlgoXY.git"

(see the author's .vcspull.yaml, more configuration)

$HOME/.vcspull.yaml and $XDG_CONFIG_HOME/vcspull/ (~/.config/vcspull) can be used as a declarative manifest to clone you repos consistently across machines. Subsequent syncs of nitialized repos will fetch the latest commits.

Sync your repos

$ vcspull sync

Keep nested VCS repositories updated too, lets say you have a mercurial or svn project with a git dependency:

external_deps.yaml in your project root (any filename will do):

./vendor/:
  sdl2pp: "git+https://github.com/libSDL2pp/libSDL2pp.git"

Clone / update repos via config file:

$ vcspull sync -c external_deps.yaml '*'

See the Quickstart for more.

Pulling specific repos

Have a lot of repos?

you can choose to update only select repos through fnmatch patterns. remember to add the repos to your ~/.vcspull.{json,yaml} first.

The patterns can be filtered by by directory, repo name or vcs url.

Any repo starting with "fla":

$ vcspull sync "fla*"

Any repo with django in the name:

$ vcspull sync "*django*"

Search by vcs + url, since urls are in this format +://:

$ vcspull sync "git+*"

Any git repo with python in the vcspull:

$ vcspull sync "git+*python*

Any git repo with django in the vcs url:

$ vcspull sync "git+*django*"

All repositories in your ~/code directory:

$ vcspull sync "$HOME/code/*"

Donations

Your donations fund development of new features, testing and support. Your money will go directly to maintenance and development of the project. If you are an individual, feel free to give whatever feels right for the value you get out of the project.

See donation options at https://git-pull.com/support.html.

More information

• Python support: >= 3.9, pypy
• VCS supported: git(1), svn(1), hg(1)
• Source: https://github.com/vcs-python/vcspull
• Docs: https://vcspull.git-pull.com
• Changelog: https://vcspull.git-pull.com/history.html
• API: https://vcspull.git-pull.com/api.html
• Issues: https://github.com/vcs-python/vcspull/issues
• Test Coverage: https://codecov.io/gh/vcs-python/vcspull
• pypi: https://pypi.python.org/pypi/vcspull
• Open Hub: https://www.openhub.net/p/vcspull
• License: MIT.

vcspull's Issues

allow specifying custom directory name

By default

~/path/to/dirs:
  project_name:
    repo: 'git+repo_url'

The directory checked out would be ~/path/to/dirs/project_name.

Allow overriding the dir, so project_name could download to ~/path/to/dirs/custom_dir

~/path/to/dirs:
  project_name:
    dir: custom_dir
    repo: 'git+repo_url'

Switch to MIT license

@jcfr Are you ok with licensing your contributions to this project and libvcs as MIT?

The current license for vcspull and libvcs is BSD 3-Clause.

If you agree to licensing your contribution(s) to vcspull and libvcs under MIT, please reply if its ok.

(doing this for another project too tmux-python/libtmux#46 tmux-python/tmuxp#264)

Feature request: command to add new repository

Currently when using this tool one has to edit the config file and then run the vcspull command (maybe with filter on the new repo name) to get the new checkout.

I would propose to add some sort of command where you can do something like vcspull add <repoUrl> which will then generate an entry similar to:

<folder>:
  <repoName>: '<repoUrl>'

in the existing config. would be the folder from which the call is executed Obviously if the already exist it should just add an entry to that block. The should be extractable from the , if not it needs to be passed in the call.

Once this config update was done vcspull should be able to run its normal "checkout/update" logic on that new entry to create it.

That way you can add a new entry in the vcspull config and get the checkout right away.

Any thoughts on this?

Best regards
Segaja

New repo format

To simplify things, lets make the format verbose.

/path/to/parent_dir:
   - name: 'default to vcs default' # optional
   - url: raw_vcs_url # no more ambiguity over how to handle auth # required
   - vcs: 'git' # or 'hg', 'svn' # required
   - prompt_auth: true/false (whether to prompt or skip if not authenticated) # default true
   - rev: afafaf # changeset, revision number or branch, defaults to head
   - username: u # optional
   - password: p # optional
   - externals: True # checkout submodules/svn externals

Contribution script to generate config?

Hey @tony ,

how willing would you be to include some contribution scripts in the repository to help generate an initial config file?

I use vcspull to manage ~70 repos in my work place all under one gitlab group. I wrote a small shell script to scan this group and generate an initial config file out of it.

Initially I thought to just share this internally in my company, but if you are willing we could add this to the repository (after i made it a bit more abstract).

Please let me know your thoughts about this.

Best regards
Segaja

Support for ~/.pullv, filtering by type, glob/fnmatch of repo name, path.

$ pullv

$ pullv -t git
$ pullv -t git hg

$ pullv -r django*

$ pullv -r django hi*

To only update repos in ~/workspace
$ pullv -p ~/workspace/

$ pullv

determine automatically if -p

['./', '/', '~', '../']

determine automatically if -t

['git', 'hg', 'svn']

determine automatically if -r

'[a-z]'

Support for ~ paths.

test module dependency?

Is there a test module that needs to be in place that pip does not import?

$ pip install vcspull
$ vcspull
from test import support
ImportError: No module named 'test'

$ pip install test
$ vcspull
from test import support
ImportError: cannot import name 'support'

Fetch-only git/hg by default, and pull only when no conflict exists

Continue on error

#361

--continue-on-error / etc. (see what other apps use for flags to continue on error)

Semantic versioning

• Note that vcspull python API is internal and not stable
• Add a note to docs 1.x doesn't have a stable API
• Follow it after 2.0 is released

re: #333 (comment)

TypeError: cannot concatenate 'str' and 'list' objects

Hi ,

I have this problem :

$ vcspull
Traceback (most recent call last):
File "/usr/lib64/python2.7/logging/init.py", line 851, in emit
msg = self.format(record)
File "/usr/lib64/python2.7/logging/init.py", line 724, in format
return fmt.format(record)
File "/usr/lib64/python2.7/site-packages/vcspull/log.py", line 76, in format
prefix = self.template(record) % record.dict
File "/usr/lib64/python2.7/site-packages/vcspull/log.py", line 115, in debug_log_template
reset + levelname + asctime + name + module_funcName + lineno + reset
TypeError: cannot concatenate 'str' and 'list' objects
Logged from file cli.py, line 115

Thanks.

Settings file

XDG_USER_CONFIG/vcspull:
  repos/*{json,yaml}
  settings.{yaml,json,toml}

Moving the config files ~/.vcspull/.{json,yaml} -> ~/.config/vcspull/repos/.{json,yaml}

Using relative path for workspace path causes infinite recursion

VERSION: vcspull 0.2.2

Using a relative path instead of an absolute path for the workspace root-dir seems to lead to infinite recursion after checkout of first project.

EXAMPLE: vcspull.yaml (needs: real Git repository)

projects:
  foo: "git+https://github.com/the_user/foo.git"

$ vcspull -c vcspull.cfg co 

NOTE:
It would be nice to use the HERE idiom (the directory where the config-file is located).

Bug when raising exception

vcspull 0.0.10

$ vcspull dotnet
Traceback (most recent call last):
  File "/usr/local/bin/vcspull", line 9, in <module>
    load_entry_point('vcspull==0.0.10', 'console_scripts', 'vcspull')()
  File "/usr/local/lib/python2.7/site-packages/vcspull/cli.py", line 119, in main
    command_load(args)
  File "/usr/local/lib/python2.7/site-packages/vcspull/cli.py", line 144, in command_load
    raise exc.NoConfigsFound(
NameError: global name 'exc' is not defined

Mock tests for svn and git

For #38's additions

simplified git config causes remote upstream rewrite on every run

If I use a simple config entry like:

~/test/cvspull:
  mypackage: 'git+ssh://[email protected]/mygroup/mypackage.git'

I always see the line about Updating remote origin:

Updating remote origin (ssh://[email protected]/mygroup/mypackage.git) with git+ssh://[email protected]/mygroup/mypackage.git

which then in turn causes a retry of the update.

To fix this issue I have to use a config like this:

~/test/cvspull:
  mypackage:
    url: 'git+ssh://[email protected]/mygroup/mypackage.git'
    remotes:
      origin: 'ssh://[email protected]/mygroup/mypackage.git'

This then works, but I would like to have a more compact/simplified config with only one line per repo.

(repogroup and name have been abstracted, as the used gitlab is internal ;) )

Best regards
Segaja

Bump minimum python version

Python 3.9 benefits vcspull and libvcs could use:

• typing.TypedDict as of 3.8 (no need for backport package)
• typing.Protocol
• typing.Literal
• Typing syntax improvements: List[str] -> list[str] (Type Hinting Generics in Standard Collections)

Analysis of distros

• Debian python 3.9

  Bullseye is the codename for Debian 11, released on 2021-08-14.
  https://wiki.debian.org/DebianBullseye
  Python 3, 3.9.1

• RedHat RHEL uses 3.6 by default, has up to 3.9

  You can install Python 3.8 and Python 3.9, including packages built for either version, in parallel with Python 3.6 on the same system, with the exception of the mod_wsgi module. Due to a limitation of the Apache HTTP Server, only one of the python3-mod_wsgi, python38-mod_wsgi, or python39-mod_wsgi packages can be installed on a system.
  source

dead link in documentation

https://github.com/vcs-python/vcspull/blob/master/docs/config-generation.md?plain=1#L112

The link in this line leads me to a "404 Not Found"

Async VCS updating

gevent, threading vcs updates

CLI: Support a list of repo matchstrings

vcspull sqlalchemy clr

Update remotes in existing repos

Hello @tony, it would be very nice if vcspull could update remotes in existing repos. What do you think? For example to add additional remotes to submodules.

`vcspull run-script`

https://github.com/vcs-python/vcspull/issues/new

Perhaps a possibility in the future, also, may be a vcspull run-script <script> command

This could make it possible to interact with vcspull and libvcs based on the environmental context.

vcspull run <script>

This allows running against your environment settings.

System configs: XDG_CONFIG_DIRS/scripts/my_script.py, e.g. /etc/tmuxp/scripts/my_script.py or vcspull run my_script
User configs: You can also have $XDG_CONFIG_DIR/scripts/my_script.py and use vcspull run my_script

Also support XDG_CONFIG_DIRS for global configs on systems

def run(repos, configs, *args, **kwargs):
    # list of repo objects that can synced
    blender = 

Remotes printing to stdout

When pulling repos with remotes:

{u'url':'git+ssh://[email protected]/tony/tmuxp.git', u'remote_name': 'tony'}

Config format

Eliminate shorthands

<folder>:
  <repoName>: '<repoUrl>'

• This would need to be a 2.0.0 thing
• 2.0 could be released today if I wanted to, but we'd end up having versions like Firefox or Chrome does. Eventually we'd end up at vcspull 42.0.1

One config format

• Future-proof: Has to work with all theoretically all vcs and copying systems: git, svn, hg, scp, rsync, http, whatever
• Work immediately out of the box, focus on code clarity:
  • Pass into

    GitRepo(path=pathlib.Path('~/work') / pathlib.Path('libtmux'), **options)
    SVNRepo(path=pathlib.Path('~/work') / pathlib.Path('MySVNProject'), **options)

• Make config file mutations like #333 easier

~/work/:
  libtmux:
    vcs: 'git'
    options:
      origin: 
        url: 'https://github.com/vcs-python/vcspull.git'
  libtmux_mirrored_remotes:
    vcs: 'git'
    options:
      origin: 
        fetch: 'https://github.com/vcs-python/vcspull.git'
        push: 'git+ssh//[email protected]/vcs-python/vcspull.git'
  MySVNProject:
    vcs: 'svn'
    options:
      url: http://unladen-swallow.googlecode.com/svn/trunk/

Recursive VCS's cause race condition

~/study/llvm:
  llvm: 'git+http://llvm.org/git/llvm.git'
~/study/llvm/llvm/tools:
  clang: 'git+http://llvm.org/git/clang.git'
  lldb: 'git+http://llvm.org/git/lldb.git'

Grabbing clang/lldb before llvm will cause a race condition. Git cannot clone into an empty directory, but tools/{clang,lldb} are where they are housed by default.

fatal: destination path '/home/tony/study/llvm/llvm' already exists and is not an empty directory.
|llvm| (git)  Cloning.                                                                             
|llvm| (git)  git clone --progress http://llvm.org/git/llvm.git /home/tony/study/llvm/llvm
fatal: destination path '/home/tony/study/llvm/llvm' already exists and is not an empty directory.
|llvm| (git)  Cloning.                                                                             
|llvm| (git)  git clone --progress http://llvm.org/git/llvm.git /home/tony/study/llvm/llvm
fatal: destination path '/home/tony/study/llvm/llvm' already exists and is not an empty directory.
|llvm| (git)  Cloning.                                                                             
|llvm| (git)  git clone --progress http://llvm.org/git/llvm.git /home/tony/study/llvm/llvm
fatal: destination path '/home/tony/study/llvm/llvm' already exists and is not an empty directory.
|llvm| (git)  Cloning.                                                                             
|llvm| (git)  git clone --progress http://llvm.org/git/llvm.git /home/tony/study/llvm/llvm
fatal: destination path '/home/tony/study/llvm/llvm' already exists and is not an empty directory.
|llvm| (git)  Cloning.       

Potential workaround could be:

git init
git remote add origin <path>
git fetch
git checkout -t origin/master

Dry-run command

It would be nice to have a --dry-run option to see what would happen if I used this configuration. I see two reasons why this would be nice:

• When you're on flaky mobile internet, one may want to edit the configuration and check that it is 'correct' without actually pulling (potentially big) repos
• It would serve as a general syntax check, similar to ansible's "check mode"

Name conflicts

Conflicts

Catch conflicts of same name and:

• different vcs
• different vcs URL

Without this, merging can't be done #231)

Catch same repo, same remote name, different URL

Crashes if repo entered twice

❯ vcspull vcspull tmuxp cihai cli unihan-etl unihan-db libvcs libtmux alagitpull alagitpull
|vcspull| (git)  Updating to 'master'.
|tmuxp| (git)  Updating to 'master'.
|cihai| (git)  Updating to 'master'.
|cli| (git)  Updating to 'master'.
|unihan-etl| (git)  Updating to 'master'.
|unihan-db| (git)  Updating to 'master'.
|libvcs| (git)  Updating to 'master'.
|libtmux| (git)  Updating to 'master'.
|alagitpull| (git)  Updating to 'master'.
Traceback (most recent call last):
  File "/home/x/.local/bin/vcspull", line 11, in <module>
    sys.exit(cli.cli())
  File "/home/x/.local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/x/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/x/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/x/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/vcspull/cli.py", line 107, in update
    list(map(update_repo, found_repos))
  File "/home/x/.local/lib/python3.6/site-packages/vcspull/cli.py", line 120, in update_repo
    repo_dict['pip_url'] = repo_dict.pop('url')
KeyError: 'url'

❯ vcspull --version
vcspull 1.1.0

~
❯ python --version
Python 2.7.15rc1

vcspull cd

cd to repository by name

Support git+ssh://user@host:relative-path/repo.git syntax

It would be great if vcspull supported relative paths for git+ssh remotes by using the colon character as a separator between host and path.

StringIO and console_to_str breaks on python 3

Because 'str' can't be decoded.

shell_command_after is without effect

After digging into the code, i've found a reference to a shell_command_after params but it seems never used. Do I miss something?

CLI Refactor

https://github.com/tony/vcspull/tree/click

Consider allowing starting with/, ./, ~ or any string with a / within to be treated as pulling by file path.

$ vcspull ./.config
$ vcspull ./.config/*
$ vcspull ~/*

If starting with http, git, file, try to find match with vcs url

$ vcspull https://github.com/tony/vcspull.git

(done) Default, match repo name

$ vcspull vim

(done) Pull multiple repos by vcs name delimited by space

vcspull neovim vim

vcspull search

search repos

using the same directory doesn't merge repos

~/study/c++/:
  uMario:
    repo: "git+https://github.com/jakowskidev/uMario_Jakowski"
    remotes:
      tony: "[email protected]:tony/uMario_Jakowski.git"

~/study/c++/:
  sdl2-playproject: "git+ssh://[email protected]/tony/sdl2-playproject.git"
  aseprite:
    repo: 'git+https://github.com/aseprite/aseprite.git'
    remotes:
      tony: 'git+ssh://[email protected]/tony/aseprite.git'

won't merge repos. fix

error when synchronizing repository outside master branch

❯ vcspull django
fatal: ref HEAD is not a symbolic ref
Traceback (most recent call last):
  File "/Users/me/Library/Python/2.7/bin/vcspull", line 11, in <module>
    sys.exit(cli.cli())
  File "/Users/me/Library/Python/2.7/lib/python/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/me/Library/Python/2.7/lib/python/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/me/Library/Python/2.7/lib/python/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/me/Library/Python/2.7/lib/python/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/me/Library/Python/2.7/lib/python/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/me/Library/Python/2.7/lib/python/site-packages/vcspull/cli.py", line 107, in update
    list(map(update_repo, found_repos))
  File "/Users/me/Library/Python/2.7/lib/python/site-packages/vcspull/cli.py", line 125, in update_repo
    r.update_repo()
  File "/Users/me/Library/Python/2.7/lib/python/site-packages/libvcs/git.py", line 157, in update_repo
    symref = self.run(['symbolic-ref', '--short', 'HEAD'])
  File "/Users/me/Library/Python/2.7/lib/python/site-packages/libvcs/base.py", line 105, in run
    cwd=cwd
  File "/Users/me/Library/Python/2.7/lib/python/site-packages/libvcs/util.py", line 178, in run
    raise exc.CommandError(output=output, returncode=code, cmd=cmd)
libvcs.exc.CommandError: Command failed with code 128: git symbolic-ref --short HEAD

~/study/django/django tags/2.0

Scan for vcs repos and save, ignore feature

Allow scanning for repos with a find/ag.

$ vcspull scan [path]  # prompt to merge into config (creating backup first) or create new

Should allow ignore dirs/patterns. Be fast, leverage https://github.com/ggreer/the_silver_searcher if in PATH.

Make directories recursively

Repos starting with `git` don't sync

~/study/rust/:
gitui: 'git+https://github.com/extrawurst/gitui.git'

vcspull gitui noop

Feature idea: merge remotes

Assuming two repos with the same name, same URL, merge remotes

This way work/mycompany.yaml's remote repo will merge with the generic one at vcspull.yaml

`tmpdir` (`py.path.local`) -> `tmp_path` (`pathlib.Path`)

Move instances of tmpdir aka py.path.local to tmp_path pathlib.Path from stdlib

companion: vcs-python/libvcs#304

Logger error

|volo| (git)  Already up-to-date. Already up-to-date.
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/local/lib/python2.7/dist-packages/pullv/log.py", line 60, in format
    prefix = self.template(record) % record.__dict__
ValueError: unsupported format character ')' (0x29) at index 529
Logged from file git.py, line 168
|bower| (git)  Updated
        Updating 747cbc1..d24da6

Deprecate python 2.x

Deprecate support for python 2.x

syncing svn repositories leads to error

command output:

╰─ vcspull -c ./vcspull.yaml
Traceback (most recent call last):
  File "/home/aschleifer/.local/bin/vcspull", line 8, in <module>
    sys.exit(cli.cli())
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/aschleifer/.local/lib/python3.10/site-packages/vcspull/cli.py", line 111, in update
    list(map(update_repo, found_repos))
  File "/home/aschleifer/.local/lib/python3.10/site-packages/vcspull/cli.py", line 147, in update_repo
    current_remote = r.remote(config_remote_name)
AttributeError: 'SubversionRepo' object has no attribute 'remote'

config:

/home/aschleifer/tmp/vcspull:
  config: 'svn+https://<svn_host>/svn/<repo_path>/trunk/config'

It does make the actual new svn checkout, but the error shows up.

On a side note I also noticed that for git repos you see the output of the git command when it is executed, but for svn you see nothing, which makes it difficult to see the progress when you pull big/slow repositories.

Initial checkout fails in latest version (1.3.1)

|django-rq-dashboard| (git)  Adding remote origin <git+https://github.com/brutasse/django-rq-dashboard.git>
fatal: remote origin already exists.
Traceback (most recent call last):
  File "/home/t/.local/bin/vcspull", line 8, in <module>
    sys.exit(cli.cli())
  File "/home/t/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/t/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/t/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/t/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/t/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/t/.local/lib/python3.7/site-packages/vcspull/cli.py", line 117, in update
    list(map(update_repo, found_repos))
  File "/home/t/.local/lib/python3.7/site-packages/vcspull/cli.py", line 159, in update_repo
    r.update_repo()  # Creates repo if not exists and fetches
  File "/home/t/.local/lib/python3.7/site-packages/libvcs/git.py", line 139, in update_repo
    self.obtain()
  File "/home/t/.local/lib/python3.7/site-packages/libvcs/git.py", line 127, in obtain
    self.remote_set(name=r['remote_name'], url=r['url'])
  File "/home/t/.local/lib/python3.7/site-packages/libvcs/git.py", line 331, in remote_set
    self.run(['remote', 'add', name, url])
  File "/home/t/.local/lib/python3.7/site-packages/libvcs/base.py", line 108, in run
    cwd=cwd,
  File "/home/t/.local/lib/python3.7/site-packages/libvcs/util.py", line 178, in run
    raise exc.CommandError(output=output, returncode=code, cmd=cmd)
libvcs.exc.CommandError: Command failed with code 128: git remote add origin https://github.com/brutasse/django-rq-dashboard.git

Remote git repositories aren't cleaning vcs protocol from url's

git+https://github.com/i3/i3.git isn't chopping off the git+ when adding remotes.

No URL found for submodules breaks the pull

Hello there,
While trying to use vcspull I noticed that it breaks and stops when a submodule is miss configured or does not exist.
Maybe would be better to validate the configuration and if it passes continue with the checkout and if not, warn the user and continue to checkout the next repo.

Architecture / untangling

The hackable repo manager, built on libvcs. With great defaults, and presets if those don't work

Config Loaders and backends

Finds configs (based on template loaders in django) and processes them them

• Loader: Searches
• Backend: Parser, e.g. yaml, json, toml, ini, python

CONFIG_LOADERS = [
    {
        "BACKEND": "vcspull.configs.loaders.json.JSONLoader",
        "XDG_DIRS": True,
        # optionally
        "DIRS": [
            "/custom/lookup/path",
        ],
    },
    {"BACKEND": "vcspull.configs.loaders.json.YAMLLoader", "XDG_DIRS": True},
    {"BACKEND": "vcspull.configs.loaders.json.TOMLLoader", "XDG_DIRS": True},
    {"BACKEND": "vcspull.configs.loaders.json.PythonLoader", "XDG_DIRS": True},
]

Config processors

(inspired by context processors / middleware):

helpers that expand config, e.g.

repo: 'git+https://github.com/vcs-python/vcspull'

repo:
  remotes:
    origin:
      push: 'git+https://github.com/vcs-python/vcspull'
      pull: 'git+https://github.com/vcs-python/vcspull'

CONFIG_PROCESSORS = [
    # before:
    #   repo: 'git+https://github.com/vcs-python/vcspull'
    # after:
    #   repo:
    #     remotes:
    #        origin:
    #          push: 'git+https://github.com/vcs-python/vcspull'
    #          pull: 'git+https://github.com/vcs-python/vcspull'
    "vcspull.configs.processors.simple_repo",
    # before:
    #  repo:
    #    origin: 'git+https://github.com/vcs-python/vcspull'
    # after:
    #  repo:
    #     origin:
    #      push: 'git+https://github.com/vcs-python/vcspull'
    #      pull: 'git+https://github.com/vcs-python/vcspull'
    "vcspull.configs.processors.simple_remote",
]

URL Parsers / adapters

e.g. pip, npm/yarn, normal vcs URLs

Merge / conflict resolutions

Function to handle merge behavior when same repo name exists. `

'MERGE_BEHAVIOR: 'vcspull.config.merge.default'

default = deep_extend

def overwrite(a, b, config_a, config_b):
    return b

def extend(a, b, config_a, config_b):
    a.extend(b)

def deep_extend(a, b, config_a, config_b):
    # deep merge code

above: hopefully if we pass config_a and config_b, if at all, they'd be immutable

see:https://github.com/TehShrike/deepmerge#custommerge

VCS backends

VCS_BACKENDS = {
    'git': 'libvcs.git.Git',
    'svn': 'libvcs.svn.Svn'
    'hg': 'libvcs.hg.Mercurial'
}

Allows people to override with custom behavior, esp. overriding sync()

Initialization

vcspull settings:

{
  "extends": "vcspull.settings.default",
  "CONFIG_LOADERS": [],
  "CONFIG_PROCESSORS": []
}

Consider a json wrapper for jsonc

Strip removing trailing commas and comments

VCSPull(settings='str_or_dict', init=True)

vp = VCSPull(settings='vcspull.config.default')
vp = VCSPull(settings='~/.config/vcspull/settings.py')
vp = VCSPull(settings='~/.config/vcspull/settings.json')

Without initializing / finding configs:

vp = VCSPull(settings='~/.config/vcspull/settings.json', init=False)

vp.settings
vp.init()  # Finds configs via loaders, parses via backends, builds index of repos

command to list all repos to ease creating backup ignores

use case:

when creating a general system backup, i feed it with a file tilling it what folders to ignore (like steam games, caches
i would like to extend that file with the names of the vcs repos, because those i "backup" with the actual dvcs