Supporting multiple git repositories? about spring-cloud-config

[warmuuh]

also the whole branch-stuff becomes a bit dirty because if you have e.g. a branch for app1, version X and a branch for app2, version Y, the Y-branch should not contain any artefacts from app1, but master then contains all artefacts? also the branch-naming should then be explicit (i.e. with prefix) to differentiate.

[brianlaframboise]

I second this. Our current configuration solution involves one git repository per application so that a) each application's configuration is isolated from all others and b) each git repo can have its own set of permissions (administered in GitHub). It would be great if Spring Cloud Config Server could support a similar setup.

@warmuuh As for the isolation, a possible workaround would be to create the initial config repo with only a basic commit (ex: a README). Then create a branch per-application and commit application-specific config files in the relevant branch.

As an upside, this would isolate application files from each other, preventing an accidental "search, replace all, commit, push" workflow from affecting multiple apps.

However, creating speculative configuration would require creating a branch-from-a-branch (which may get confusing or be error prone for some users). Also, this doesn't solve the git credential problem of all users being able to push changes to any app (even if the configs are isolated in various branches).

[dsyer]

I don't think it would be terribly difficult to implement a git-repository-per-application version of EnvironmentRepository. If anyone wants to contribute some code, please feel free (see the guidelines on contributing in the readme).

[iceycake]

I will give it a shot since this is a deal breaker for our internal project.

[brianlaframboise]

@dsyer it might be a little more complicated than that. Currently the encrypt.key value is global to the Spring Cloud Config Server. Supposing @iceycake implements such an EnvironmentRepository, the Config Server would still allow any client to decrypt any other project's encrypted values.

I know this behaviour is the current stated design of the Config Server. However, assuming we want the Config Server to aggregate multiple git repositories so that they can be independently managed and audited, it seems fitting to extend that capability to the encryption of the properties inside that git repository.

The solution here might be (as the documentation alludes) to embed the Config Server into a Spring Boot app with Spring Security enabled with custom/advanced authorization rules such that various clients can only access the configurations to which they are entitled.

Alternatively, instead of security the reading of the properties (given that they may already be in a relatively open git repo), the extra layer of authorization could apply to the /decrypt endpoint, though that API would have to be changed to somehow be aware of the project for which it is decrypting the value.

[dsyer]

The decrypt endpoint is only used (if at all) for local properties in a client. The config server does decryption before shipping configuration, so it could in principle be modified to use a different key per repository (or per app). There would still be some challenges with key management.

Unless someone has a bright idea about the key management, wouldn't it be quite useful to stick with a global encryption service, since no client app devs actually need to or can do the encryption? All secrets are encrypted the same way, but if this change was implemented, they can live in different git repositories for the convenience of developers.

On a related note, I had thought that hashing with something like the application name is probably a useful enhancement (like Travis CI does with its secrets), so that the same secret value in different applications has a different cipher. (That's a completely different feature, so if you like that idea or think it's worth discussing please open another issue.)

[iceycake]

As for our use case, I believe we can live with a global key on the config server because all of the repositories are under the same organization. I understand there is a need of different keys in different use case but my implementation will try to stick with a single key.

[brianlaframboise]

To be clear I don't yet actually have that use case either. I could live with a shared encryption key for now as well.

I just assumed that if @iceycake needed properties in separate repos then they should be encrypted with different keys for maximum isolation.

[iceycake]

I finally able to allocate the upcoming few days to work on this issue. Currently, the endpoints look like something like

   /{name}/{profiles}/{label}

I am wondering if it is acceptable to do something like:

   /{name}/{profiles}/{label}?repo=team1

where we have the repo name defines in the properties like:

   spring:
      cloud:
         config:
            server:
               git:
                  - name: team1
                    uri: http://gitserver/team1/config-repo
                  - name: team2
                    uri: http://gitserver/team2/config-repo

By doing this, we can backward compatible with the current implementation.

[dsyer]

I think it's better to hide the repo IDs from the client. You would need to know (and configure) which apps belong to which repo, but isn't it better to do that in a static way? E.g.

spring:
  cloud:
     config:
        server:
           git:
              repos:
                 team1: 
                    uri: http://gitserver/team1/config-repo
                    apps: foo,bar*
                 team2: 
                    uri: http://gitserver/team2/config-repo
                    apps: spam
                 default: 
                    uri: http://gitserver/default/config-repo
                    apps: *

The wildcard in the fallback repo can probably be optionally omitted (and spring.cloud.config.server.git.uri could be used as a default for the fallback, so existing apps would continue to work).

[iceycake]

In our use case, the config server deployment is developer self serving. As a result, we will not know what app associates to what team/repo. In addition, app name collision can happen with the number of development teams we have. That's why associating the app name and repo id is a bad idea.

[dsyer]

I suspect it might be better in that case to concatenate the repo name with the app name?

[iceycake]

This would work.

   http://configserver/{app-name}/{profiles}/{label}

spring:
  cloud:
     config:
        server:
           git:
              repos:
                 team1: 
                    uri: http://gitserver/team1/config-repo
                    app-name: team1-*
                 team2: 
                    uri: http://gitserver/team2/config-repo
                    app-name: team2-*
                 default: 
                    uri: http://gitserver/default/config-repo
                    app-name: *

[dsyer]

Cool, so all we need is a PatternMatchUtils to filter the repository list before we finish the find? That probably means we can implement this feature as a very thin wrapper around the existing JGitEnvironmentRepository.