rancher / fleet

Deploy workloads from Git to large fleets of Kubernetes clusters

Introduction

Fleet is GitOps at scale. Fleet is designed to manage multiple clusters. It's also lightweight enough that it works great for a single cluster too, but it really shines when you get to a large scale. By large scale we mean either a lot of clusters, a lot of deployments, or a lot of teams in a single organization.

Fleet can manage deployments from git of raw Kubernetes YAML, Helm charts, or Kustomize or any combination of the three. Regardless of the source all resources are dynamically turned into Helm charts and Helm is used as the engine to deploy everything in the cluster. This gives a high degree of control, consistency, and auditability. Fleet focuses not only on the ability to scale, but to give one a high degree of control and visibility to exactly what is installed on the cluster.

Quick Start

For more information, have a look at Fleet's documentation.

Install

Get helm if you don't have it. Helm 3 is just a CLI and won't do bad insecure things to your cluster.

For instance, using Homebrew:

brew install helm

Install the Fleet Helm charts (there's two because we separate out CRDs for ultimate flexibility.)

helm -n cattle-fleet-system install --create-namespace --wait \
    fleet-crd https://github.com/rancher/fleet/releases/download/v0.9.3/fleet-crd-0.9.3.tgz
helm -n cattle-fleet-system install --create-namespace --wait \
    fleet https://github.com/rancher/fleet/releases/download/v0.9.3/fleet-0.9.3.tgz

Add a Git Repo to watch

Change spec.repo to your git repo of choice. Kubernetes manifest files that should be deployed should be in /manifests in your repo.

cat > example.yaml << "EOF"
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: sample
  # This namespace is special and auto-wired to deploy to the local cluster
  namespace: fleet-local
spec:
  # Everything from this repo will be run in this cluster. You trust me right?
  repo: "https://github.com/rancher/fleet-examples"
  paths:
  - simple
EOF

kubectl apply -f example.yaml

Get Status

Get status of what Fleet is doing:

kubectl -n fleet-local get fleet

You should see something like this get created in your cluster.

kubectl get deploy frontend

NAME       READY   UP-TO-DATE   AVAILABLE   AGE
frontend   3/3     3            3           116m

Enjoy and read the docs.

License

For developer and maintainer documentation, see DEVELOPING.md.

fleet's Issues

Support --insecure-skip-tls-verify

Support --insecure-skip-tls-verify so the generated agent-token's encoded kubeconfig has insecure-skip-tls-verify: true specified.

fleet install agent-token --insecure-skip-tls-verify > token

Allowing folks to manage clusters without messing with certificates, certSANs, and CAs. My particular use case is internal only developer environments, not production hardened clusters.

Awesome project by the way.

Private Git Repo Usage - Fails

All attempts to use a private github repo following the specific syntax either git@github:username/repo.git (ssh) and https://github.com/username/repo.git (https) fail during the inital authentication stage. Secrets are properly listed in namespace.

Verified that the https auth is correct and ran the highlighted git ls-remote command manually.

Additionally when testing the ssh auth, the same error occurs as in #93 (comment).

Again verified ssh is working properly by executing the git ls-remote command manually. If I understand the usage the ssh git url should be correct. If not please let me know.

Matching a ClusterGroup by name from a GitRepo definition is not selecting the group and thus not deploying the bundle

No bundles are send to the clusters if I try matching with clusterGroup but clusterSelector works.

ClusterGroup

kind: ClusterGroup
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: production-group
  namespace: clusters
spec:
  selector:
    matchLabels:
      env: prod

k get clustergroups.fleet.cattle.io -n clusters
NAME               CLUSTERS-READY   BUNDLES-READY   STATUS
production-group   3/3              3/3

GitRepo

kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: simple
  namespace: clusters
spec:
  repo: "https://github.com/rancher/fleet-examples"
  bundleDirs:
  - simple
  targets:
  - name: prod
    clusterSelector: null
    clusterGroupSelector: null
    clusterGroup: production-group

k get gitrepos.fleet.cattle.io -n clusters
NAME     REPO                                        COMMIT                                     BUNDLEDEPLOYMENTS-READY   STATUS
simple   https://github.com/rancher/fleet-examples   14b81f64b5d33ca0783ceec4f7b8b6f9f41251a0   0/0

Applied GitRepo spec:

spec:
  bundleDirs:
  - simple
  repo: https://github.com/rancher/fleet-examples
  targets:
  - clusterGroup: production-group
    name: prod
status:
  commit: 14b81f64b5d33ca0783ceec4f7b8b6f9f41251a0
  conditions:
  - lastUpdateTime: "2020-09-02T21:47:17Z"
    status: "True"
    type: Ready
  - lastUpdateTime: "2020-09-02T19:32:11Z"
    status: "False"
    type: Reconciling
  - lastUpdateTime: "2020-09-02T16:44:23Z"
    status: "False"
    type: Stalled
  - lastUpdateTime: "2020-09-02T19:32:11Z"
    status: "True"
    type: Synced
  - lastUpdateTime: "2020-09-02T21:47:17Z"
    status: "True"
    type: Accepted
  display:
    readyBundleDeployments: 0/0
  observedGeneration: 5
  summary:
    desiredReady: 0
    ready: 0

Based on doc: https://fleet.rancher.io/gitrepo-targets/

Cluster Registration: Agent Initiated Registration
Kubernetes: v1.18.8+k3s1
Rancher HA install: master-head (09/02/2020) ef11fe3

Fleet - Clusters show 0 node ready after upgrading from v2.4.8 to v2.5-head

What kind of request is this (question/bug/enhancement/feature request):
bug

Steps to reproduce (least amount of steps as possible):

• run Rancher:v2.4.8 HA
• add some custom clusters
• upgrade Rancher to v2.5-head 65f3525
• go the dashboard -> continues delivery -> clusters

Result:

• some clusters show 0 nodes available

• force upgrade the cluster can solve the problem

Other details that may be helpful:

All clusters show the right number of nodes after a while.

N:N mappings between bundles and clusters

Hi,

I love the idea of that project! It's going to be great, I believe :)

I think that we should consider extending/changing a relation between bundles and clusters.
As for now I understand that a bundle defines 1:N relation clusters - inside a bundle I can define overlays for a cluster or group of clusters. This is fine assuming that you have multiple clusters under one tenant.

In a scenario when I manage multiple clusters for multiple customers, however, it's not enough as I don't want to set tenant-specific settings inside bundle definition.

To show the case let's assume I have some "golden configuration" for Grafana Helm chart.
I create a bundle, specify some common values for helm chart, then specify some overlays (like replicaCount: 1 for dev/stage and replicaCount: 2 for prod envs). And so far it works, but...
For each tenant I need to set separate settings for ingress (like hostname or TLS settings).
In the current version (correct me if I'm wrong) I need to specify that as another overlay for each tenant.
I think this is bad, because it tightly couples tenant-specific config with the bundle settings, so it's impossible e.g. to opensource that bundle or share it between tenants, because that way I'll disclosure some tenant specific information (or in the worst case - a list of my customers).

I think we should introduce some new CRD that will allow to:

1. Specify cluster/cluster group selector to select the cluster
2. Specify bundle(s) (e.g. by using labels) that should be applied to clusters from #1
3. Specify some additional overlays (like custom helm values or other manifests (e.g. sealed secrets)) for that matched clusters and bundles

By implementing that, effectively we'll allow:

1. N:N relationships between clusters and bundles, so:
2. Bundles can become shareable, even publicly
3. One will be able to apply the same set of "general" bundles to multiple tenants
4. For each tenant there will be a possibility to set specific values

I'm new to the project and have close to zero experience with that, so correct me if I'm wrong at any point, please :)

I'm very interested what do you think about my idea :)

Best regards

Łukasz Tomaszkiewicz

Progressive rollout

Did you consider adding something that would progressively rollout changes to bundled clusters?

Failed to deploy apps into clusters in Rancher single-install setup

the clusters are provisioned by Rancher

> k -n clusters get clusters -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels}{"\n"}{end}'
cluster-dc8270631e5b	map[env:dev fleet.cattle.io/cluster:cluster-dc8270631e5b]
cluster-ddff1b5b592d	map[env:test fleet.cattle.io/cluster:cluster-ddff1b5b592d]
cluster-fc9e80ee366b	map[env:prod fleet.cattle.io/cluster:cluster-fc9e80ee366b]

use the following yaml to deploy a GitRepo CRD (source https://fleet.rancher.io/gitrepo-targets/)

kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: myrepo
  namespace: clusters
spec:
  repo: http://github.com/rancher/fleet-examples
  bundleDirs:
  - simple

  # Targets are evaluated in order and the first one to match is used. If
  # no targets match then the evaluated cluster will not be deployed to.
  targets:
  # The name of target. If not specified a default name of the format "target000"
  # will be used
  - name: prod
    # A selector used to match clusters.  The structure is the standard
    # metav1.LabelSelector format. If clusterGroupSelector or clusterGroup is specified,
    # clusterSelector will be used only to further refine the selection after
    # clusterGroupSelector and clusterGroup is evaluated.
    clusterSelector:
      matchLabels:
        env: prod
    # A selector used to match cluster groups.
    clusterGroupSelector:
      matchLabels:
        region: us-east
    # A specific clusterGroup by name that will be selected
    clusterGroup: group1

Results:

> k -n clusters get all
NAME                     READY   STATUS   RESTARTS   AGE
pod/myrepo-c6af9-8lhfb   0/2     Error    0          2m24s
pod/myrepo-c6af9-cb7sk   0/2     Error    0          7m25s
pod/myrepo-c6af9-gx5jq   0/2     Error    0          7m43s
pod/myrepo-c6af9-ktlfw   0/2     Error    0          5m4s
pod/myrepo-c6af9-ltvd6   0/2     Error    0          7m5s
pod/myrepo-c6af9-n5h99   0/2     Error    0          6m25s
pod/myrepo-c6af9-wzf7m   0/2     Error    0          7m35s

NAME                     COMPLETIONS   DURATION   AGE
job.batch/myrepo-c6af9   0/1           7m44s      7m44s

log of the container step-git-source

{"level":"error","ts":1599093912.062322,"caller":"git/git.go:41","msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin --update-head-ok --force warning:]: exit status 128\nwarning: redirecting to https://github.com/rancher/fleet-examples/\nfatal: couldn't find remote ref warning\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:41\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:116\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:53\nruntime.main\n\truntime/proc.go:203"} {"level":"fatal","ts":1599093912.0624912,"caller":"git-init/main.go:54","msg":"Error fetching git repository: failed to fetch [warning:]: exit status 128","stacktrace":"main.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:54\nruntime.main\n\truntime/proc.go:203"}

---

Update:
It turns out this bug happens only in the Rancher single-install setup

Agent cluster is not appearing in the Fleet section but is listed in the Cluster explorer

Clusters (besides local) are shown properly in the Cluster Explorer:

https://<rancher_server>/dashboard/c/c-xxxxx/explorer/fleet.cattle.io.cluster

This Agent cluster is not shown in the Fleet "explorer"

* Check the URL is different in the fleet explorer

https://<rancher_server>/dashboard/c/c-xxxxx/fleet/fleet.cattle.io.cluster

Rancher master-c08b0bb17fee40c76468020b352e2b146756aa4b-head Single install

• 2 DO clusters One Manager and One Agent

kubectl -n clusters get clusters.fleet.cattle.io
NAME                   BUNDLES-READY   NODES-READY   SAMPLE-NODE                  LAST-SEEN              STATUS
cluster-cd5078be17d0   1/1             1/1           izaac-linode-single2-all-1   2020-09-12T02:02:48Z

bundleDirs should be documented as a string array, it's currently a string in the docs.

kubectl apply -f example.yaml
The GitRepo "sample" is invalid: spec.bundleDirs: Invalid value: "string": spec.bundleDirs in body must be of type array: "string"

fleet apply -o - prints control characters

I tried to integrate fleet with ArgoCD to get GitOps for bundles, but got the error:

rpc error: code = Unknown desc = Failed to unmarshal manifest: error converting YAML to JSON: yaml: control characters are not allowed

Steps to reproduce:

1. Create sample bundle:

name: prometheus-node-exporter
chart: https://kubernetes-charts.storage.googleapis.com/prometheus-node-exporter-1.9.1.tgz
defaultNamespace: monitoring

targets:
- clusterSelector: {}
  values:
    resources:
      limits:
        cpu: 50m
        memory: 50Mi
      requests:
        cpu: 10m
        memory: 30Mi

    image:
      tag: v1.0.0-rc.0

    extraArgs:
    - --collector.thermal_zone

2. Run:

fleet apply -o - | less

3. Result:

ESC[0AapiVersion: fleet.cattle.io/v1alpha1
kind: Bundle
metadata:
  name: prometheus-node-exporter
  namespace: default
spec:

Workaround:

If you need to use fleet in argocd, you can define plugin as follows:

- name: fleet
  generate:
  command: ["sh", "-c"]
  args:
  - |
    /usr/local/bin/fleet apply -o - | tr -d '\000-\011\013\014\016-\037' | sed -e 's/\[0A//g'

Documentation and/or support for deleting clusters, cluster groups, and bundles.

I've not come across any examples or or other doc on how to cleanup registered bundles, clusters, or cluster groups in the case of renames or delete of the items. If it exists please point me to it. Thank you.

CIS v2 scan on a hardened cluster fails when fleet is deployed

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible):

• On a cluster - CIS 1.5 hardened - Deploy CIS v2, OPA, Logging, monitoring and istio charts

• The CIS 1.5 hardened cluster scan when run, shows 2 tests - 5.1.5 and 5.3.2 as failures.

• following namespaces break test 5.1.5

kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace
"cattle-gatekeeper-system"
"cattle-logging-system"
"cattle-monitoring-system"
"dashboard-shells"
"default"
"fleet-system"
"istio-system"
"kube-system"

Expected Result:
The charts should comply with hardening steps

Other details that may be helpful:

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): master-head - commit id: 8fd81a9b4
• Installation option (single install/HA): Single node

fleet-agent cannot find secret in local cluster in Rancher single-install setup

run rancher:master-0f691dc70f86bbda3d6563af11779300a6191584-head in the single-install mode

The following line floods the log of the pod fleet-agent-7dfdfd5846-xjw96

time="2020-09-15T00:18:30Z" level=info msg="Waiting for secret fleet-clusters-system/c-09ea1d541bf704218ec6fc9ab2d60c0392543af636c1c3a90793946522685 for request-2vz49: secrets \"c-09ea1d541bf704218ec6fc9ab2d60c0392543af636c1c3a90793946522685\" not found"

gz#14319

docs error

README.md --> Quick Start --> 2.Install Fleet Manager on Kubernetes cluster. The fleet CLI will use your current kubectl config to access the cluster. --> fleet install controller | kubectl apply -f -

In the latest version there is no longer an controller command, replaced by a manager.

$ fleet install -h
Generate manifests for installing server and agent

Usage:
  flt install [command]

Available Commands:
  agent-config Generate cluster specific agent config
  agent-token  Generate cluster group token and render manifest to register clusters into a specific cluster group
  manager      Generate deployment manifest to run the fleet controller
  simulator    Generate manifest to install a cluster simulator

Flags:
  -h, --help   help for install

Global Flags:
  -k, --kubeconfig string   kubeconfig for authentication
  -n, --namespace string    namespace (default "default")

Use "flt install [command] --help" for more information about a command.

Credentials management

Hey, I'm very interested in using rancher fleet for our cluster apps but I have one concern.

We do not want to store sensitive data in clear text inside our repo and we currently we rely hashicorp vault integration to pull the data when our push based deploys runs.

Is there a way to somehow inject sensitive data at deploy time in fleet so we don't have sensitive data in clear text in our repos ?

Thanks a lot

Fleet Manager / Controller on GKE - connection issues

Hi,

Strange issue, I kinda got this working but for the past couple days I've been getting connection issues.

My setup:

GKE (google managed kubernetes), public cluster, master authorized network list contains public IPs of various IOT kubernetes clusters and myself.
kubectl configured and can use kubectl cli commands.

But, things like curl'ing fails, like in the guide for multi-cluster-install:

curl -fLk https://<master ip>:6443/version

results in Operation timed out

This appears to affect registration of clusters as I also see connection timeout errors.

kubectl get nodes
NAME                                      STATUS   ROLES    AGE   VERSION
gke-cluster-1-node-pool-1-294d904c-8sd0   Ready    <none>   36m   v1.15.12-gke.20
gke-cluster-1-node-pool-1-294d904c-b6df   Ready    <none>   36m   v1.15.12-gke.20
gke-cluster-1-node-pool-1-294d904c-bh9d   Ready    <none>   36m   v1.15.12-gke.20

curl -fLk ${API_SERVER_URL}/version
curl: (7) Failed to connect to <...master ip...> port 6443: Operation timed out

Have tried destroying and rebuilding the GKE cluster (fleet manager) in various zones with same result, not sure what's going on.

I understand this is more an issue with Google, just wondering if anyone has got this working on GKE, any advice is appreciated.

Thanks
Yusuf

Scalability of fleet

Thanks for open sourcing the project.

Scaling the number of Kubernetes cluster under management is the main focus of fleet; however, it's not clear how fleet solves the problem with current architecture. From what I know by briefly scanning the code base, fleet is using Kubernetes standard API constructs like informer, listers, via client-go. If we compare Cattle'd Cluster with Node, then it sounds like saying "Kubernetes supports millions of Nodes" (ignoring external resources like ingress for now), which I think, is not a well-studied scalability issue for now.

Is my understanding correct? I'm happy to know more details :)

Fleet agents dont pick up newly created CRDs

Fleet agent controllers dont pick up newly created CRD's applied by other bundles.

For example I deployed a bundle containing OPA gatekeeper constraint templates across using fleet.

Subsequent bundles to deploy constraints using these new Constraint Template CRD's dont get recognized by the agent until the agent is restarted.

Possibly related to this: kubernetes-sigs/controller-runtime#321

Bundle is applied across cluster but still manager/controller still shows 4/8

IssueType - Bug

Steps to reproduce:
Create 4 k3s clusters
create directory openfaas with file bundle.yaml as below:

name: openfaas
chart: https://openfaas.github.io/faas-netes/openfaas-5.6.3.tgz
defaultNamespace: openfaas
overlays:
- name: commonoptions
  values:
    image:
      imagePullPolicy: Always
targets:
- clusterSelector: {}

create a folder manifests in openfaas with 2 files secrets.yaml and namespaces.yaml as below:

apiVersion: v1
data:
  basic-auth-password: ZTUzNzY4ODY5ZGEyMWRkZDAwYTk4OWEyN2FjYTA4YWJhMWUyMWI3MQ==
  basic-auth-user: YWRtaW4=
kind: Secret
metadata:
  name: basic-auth
  namespace: openfaas
type: Opaque

apiVersion: v1
kind: Namespace
metadata:
  name: openfaas
  labels:
    role: openfaas-system
    access: openfaas-system
    istio-injection: enabled
---
apiVersion: v1
kind: Namespace
metadata:
  name: openfaas-fn
  labels:
    istio-injection: enabled
    role: openfaas-fn

• do a fleet apply ./openfaas/

Now the applications get deployed to all the clusters and we can access as well but the manager shows :

[root@f1 ~]# kubectl get fleet
NAME                                 CLUSTERS-READY   CLUSTERS-DESIRED   STATUS
bundle.fleet.cattle.io/fleet-agent   4                4                  
bundle.fleet.cattle.io/openfaas      0                4                  Modified: 4 (default-default-group/cluster-51b5ba5d-6aeb-4ed7-9cf2-33cdaa267eaa... )
NAME                                   CLUSTER-COUNT   BUNDLES-READY   BUNDLES-DESIRED   STATUS
clustergroup.fleet.cattle.io/default   4               4               8                 Modified: 4 (openfaas... )

kubectl get bundledeployment -A -o yaml patch: '{"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"gateway"},{"name":"faas-netes"}],"containers":[{"imagePullPolicy":null,"name":"faas-netes"}]}}}}'

Cluster selector doesn't work on fleet.yml (Does work on git repo)

• get 1 manager cluster, 2 agent clusters

> k get cluster -A --show-labels
NAMESPACE     NAME                   BUNDLES-READY   NODES-READY   SAMPLE-NODE      LAST-SEEN              STATUS   LABELS
clusters      cluster-4ab647a24107   1/1             1/1           jiaqi-agent2     2020-09-03T20:57:28Z            env=dev,fleet.cattle.io/cluster=cluster-4ab647a24107
clusters      cluster-78f22966ef28   1/1             1/1           jiaqi-agent1     2020-09-03T20:54:47Z            env=test,fleet.cattle.io/cluster=cluster-78f22966ef28
fleet-local   local                  1/1             1/1           jiaqi-manager1   2020-09-03T20:52:55Z            name=local,objectset.rio.cattle.io/hash=67f475f626e5473772f6084a291170a9de5942d1

• deploy the following gitrepo CR https://github.com/rancher/fleet-examples/tree/master/helm

kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: helm
  namespace: fleet-local
spec:
  repo: https://github.com/rancher/fleet-examples/
  bundleDirs:
  - helm

• we expect the following fleet.yaml file in the repo takes effect

namespace: fleet-helm-example
targets:
- name: dev
  values:
    replication: false
  clusterSelector:
    matchLabels:
      env: dev

- name: test
  values:
    replicas: 3
  clusterSelector:
    matchLabels:
      env: test

- name: prod
  values:
    serviceType: LoadBalancer
    replicas: 3
  clusterSelector:
    matchLabels:
      env: prod

Results:

• the chart is deployed into the local cluster

> k get gitrepo -A
NAMESPACE     NAME   REPO                                         COMMIT                                     BUNDLEDEPLOYMENTS-READY   STATUS
fleet-local   helm   https://github.com/rancher/fleet-examples/   14b81f64b5d33ca0783ceec4f7b8b6f9f41251a0   1/1

> k get pods -n fleet-helm-example
NAME                            READY   STATUS    RESTARTS   AGE
frontend-6d4776d49-7hxqq        1/1     Running   0          4m59s
redis-master-7d557b94bb-qvkh4   1/1     Running   0          4m59s
redis-slave-5f777c9d45-v592l    1/1     Running   0          4m59s
redis-slave-5f777c9d45-vkgbk    1/1     Running   0          4m59s

Expected:

• the apps should deploy to the clusters with the target label.

---

Update:
I realized the namespace for the gitrepo CR and clusters CR are not the same, which is why the app goes to the local cluster.
But after I correct he namespace in the gitrepo CR, the app deployed to no cluster.

> k get gitrepo -n clusters
NAME   REPO                                         COMMIT                                     BUNDLEDEPLOYMENTS-READY   STATUS
helm   https://github.com/rancher/fleet-examples/   14b81f64b5d33ca0783ceec4f7b8b6f9f41251a0   0/0

k get bundle -n clusters
NAME          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-agent   2/2
helm-helm     0/0

Agent Registration - not able to connect to manager cluster behind Rancher API proxy

Steps:

• run Rancher:master-head
• provision 2 clusters
• cluster-1 runs the Fleet manager: https://fleet.rancher.io/multi-cluster-install/
• cluster-2 runs the Fleet agent: https://fleet.rancher.io/agent-initiated/

Result:

• see the following errors in the logs of the agent pods in cluster-2
  time="2020-09-02T22:28:06Z" level=error msg="Failed to register agent: looking up secret fleet-system/fleet-agent-bootstrap: the server has asked for the client to provide credentials (post clusterregistrations.meta.k8s.io)"

More info:
It turns out I cannot use the Rancher’s Endpoint for the cluster, instead I need to use the authorized cluster endpoint and its corresponding CA cert:

https://<fqdn>/k8s/clusters/c-vx96p - NO
https://<ip>:6443 - YES

Manager initiated registration fails on AKS

When trying to use manager initiated registration with a AKS cluster provisioning fails with the following message:

failed to testing kubeconfig: failed to test connection to https://example.hcp.northeurope.azmk8s.io:443: the server has asked for the client to provide credentials

This is possibly caused by AKS requiring authentication for any request to the API and client.Discovery().ServerVersion() at modules/cli/agentmanifest/agent.go#L100 not performing authentication when checking the version.

Here's an example of the configuration I used:

apiVersion: fleet.cattle.io/v1alpha1
kind: Cluster
metadata:
  name: example
  namespace: fleet-clusters
  labels:
    demo: "true"
    env: dev
spec:
  kubeConfigSecret: example-kubeconfig
---
apiVersion: v1
kind: Secret
metadata:
  name: example-kubeconfig
  namespace: fleet-clusters
data:
  apiServerURL: aHR0cHM6Ly9leGFtcGxlLmhjcC5ub3J0aGV1cm9wZS5hem1rOHMuaW86NDQz
  apiServerCA: bmls
  value: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBibWxzCiAgICBzZXJ2ZXI6IGh0dHBzOi8vZXhhbXBsZS5oY3Aubm9ydGhldXJvcGUuYXptazhzLmlvOjQ0MwogIG5hbWU6IGV4YW1wbGUKY29udGV4dHM6Ci0gY29udGV4dDoKICAgIGNsdXN0ZXI6IGV4YW1wbGUKICAgIHVzZXI6IGNsdXN0ZXJBZG1pbl9leGFtcGxlX2V4YW1wbGUKICBuYW1lOiBleGFtcGxlLWFkbWluCmN1cnJlbnQtY29udGV4dDogZXhhbXBsZS1hZG1pbgpraW5kOiBDb25maWcKcHJlZmVyZW5jZXM6IHt9CnVzZXJzOgotIG5hbWU6IGNsdXN0ZXJBZG1pbl9leGFtcGxlX2V4YW1wbGUKICB1c2VyOgogICAgY2xpZW50LWNlcnRpZmljYXRlLWRhdGE6IGJtbHMKICAgIGNsaWVudC1rZXktZGF0YTogYm1scwogICAgdG9rZW46IDAxMjM0NTY3ODlhYmNkZWYK

Private Git Repo Usage - Fails Auth

All attempts to use a private github repo following the specific syntax either git@github:username/repo.git (ssh) and https://github.com/username/repo.git (https) fail during the inital authentication stage. Secrets are properly listed in namespace.

Verified that the https auth is correct and ran the highlighted git ls-remote command manually.

Additionally when testing the ssh auth, the same error occurs as in #93 (comment).

Again verified ssh is working properly by executing the git ls-remote command manually. If I understand the usage the ssh git url should be correct. If not please let me know.

Unify "paused" property

Hi, I've noticed that for cluster CRD as well as for bundle CRD there's a paused property, but for ClusterGroup the same property is called pause. I suggest unifying it by renaming ClusterGroup property to paused.

127.0.0.1 invalid host

I'm testing fleet (version 0.2.0) using kind and I ran into this kind of error:

$ fleet install agent-token > token
Error: invalid host 127.0.0.1, use --server-url to set a proper server URL
Usage:
  flt install agent-token [flags]

Flags:
  -c, --ca-file string            File containing optional CA cert for fleet controller cluster
  -g, --group string              Cluster group to generate config for (default "default")
  -h, --help                      help for agent-token
      --no-ca                     Use no custom CA for a fleet controller that is signed by a well known CA with a proper CN.
      --server-url string         The full URL to the fleet controller cluster
      --system-namespace string   System namespace of the controller (default "fleet-system")
      --token-only                Generate only the token
  -t, --ttl string                How long the generated registration token is valid, 0 means forever (default "1440m")

Global Flags:
  -k, --kubeconfig string   kubeconfig for authentication
  -n, --namespace string    namespace (default "default")

FATA[0000] invalid host 127.0.0.1, use --server-url to set a proper server URL 

I got the same error when using --server-url with localhost

Auth for private Helm registries

There was nothing in the documentation in regards to using authentication for a helm chart registry.

helm:
    secret: secretName

Can we add the ability to either add a secret or username/password for private registries

gz#14091
gz#14979

Respect current context in Kubeconfig

The fleet cli defaults to the default namespace rather than whatever your current context default is. It would be really nice if it could use context

Clusters - ErrNoAgent status in Dashboard

Clusters are added properly to the Fleet and I can deploy bundles.

NAME                   BUNDLES-READY   NODES-READY   SAMPLE-NODE   LAST-SEEN              STATUS   LABELS
cluster-99e84a6ed8ff   2/2             2/2           izaac-k3s2    2020-09-02T20:55:58Z            env=prod,fleet.cattle.io/cluster=cluster-99e84a6ed8ff
cluster-e6fe23a74821   2/2             2/2           izaac-k3s3    2020-09-02T20:52:06Z            env=prod,fleet.cattle.io/cluster=cluster-e6fe23a74821

But the Dashboard shows them with a ErrNoAgent status.

Cluster Registration: Agent Initiated Registration
Kubernetes: v1.18.8+k3s1
Rancher HA install: master-head (09/02/2020) ef11fe3

Excerpt API response: GET /v1/fleet.cattle.io.clusters

"status": {
            "agent": {
                "lastSeen": "2020-09-02T20:50:58Z",
                "namespace": "fleet-system",
                "readyNodeNames": ["izaac-k3s2", "izaac-k3s2-worker"],
                "readyNodes": 2
            },
            "conditions": [{
                "lastUpdateTime": "2020-09-02T19:32:46Z",
                "status": "True",
                "type": "Ready"
            }, {
                "lastUpdateTime": "2020-09-02T20:50:58Z",
                "status": "True",
                "type": "Processed"
            }],
            "display": {
                "readyBundles": "2/2",
                "readyNodes": "2/2",
                "sampleNode": "izaac-k3s2",
                "state": "ErrNoAgent"
            },
            "namespace": "cluster-clusters-cluster-99e84a6ed8ff",
            "summary": {
                "desiredReady": 2,
                "ready": 2
            }
        }

Ability to configure Rancher via Fleet

There should be a CRD and resource schema that can be realized in the Fleet cluster that describes a Rancher master. Not the Kubernetes cluster that Rancher runs on -- the Rancher instance itself. Basically any Rancher configuration setting that doesn't affect a managed cluster directly, should be configurable via Fleet.

• Setup Ingress to allow both FQDN and short-name access to Rancher
• AzureAD Authentication
• Enabling/Disabling node drivers
• Set the default user role
• Add new custom global roles
• Add local users (service accounts) and assign roles
• Add new custom cluster and project role templates
• Add custom pod security policy templates
• Everything on the global "configuration" page

This isn't a comprehensive list, and also isn't exclusive -- we would get value from having just some of this implemented, as long as there's a path forward for more to get implemented over time.

fleet does not work on kind cluster

I created a kind cluster and want to use it as manager cluster, and when generate token on manager cluster, it failed as follows:

root@gyliu-c11:~/fleet# fleet install agent-token > token
Error: invalid host 127.0.0.1, use --server-url to set a proper server URL
Usage:
  flt install agent-token [flags]

Flags:
  -c, --ca-file string            File containing optional CA cert for fleet management server
  -g, --group string              Cluster group to generate config for (default "default")
  -h, --help                      help for agent-token
      --no-ca
      --server-url string         The full URL to the fleet management server
      --system-namespace string   System namespace of the manager (default "fleet-system")
  -t, --ttl string                How long the generated registration token is valid, 0 means forever (default "1440m")

Global Flags:
  -k, --kubeconfig string   kubeconfig for authentication
  -n, --namespace string    namespace (default "default")

FATA[0001] invalid host 127.0.0.1, use --server-url to set a proper server URL

Ability to configure node OS (sysctl/cronjobs/etc) via Fleet

It should be possible to create Fleet resources that describe configuration settings that need to be made on the host. For example, sysctl settings or cronjobs or Docker configuration or package management.

It would be best to use an existing open-source tool for this, possibly Puppet, but maybe something like Ansible or Chef even CloudInit. The Fleet resources would contain snippets or modules of this policy, and the controller would basically just run Puppet or Ansible or Chef or whatever in a privileged container. This leaves it open-ended so that we can make as many (or as few) configuration changes as we like on the host itself, ensuring that it doesn't get in the way of our existing configuration management tooling.

Change CLI to fleet and not flt

The feedback, which seems valid, is that it's better if it's a simple word that can be easily typed the same way you say it. Basically avoid saying, "Hey type fleet apply, that's F, L, T space apply."

Add additional build flags to reduce the size of the binary

I tested this out locally, and I was able to reduce the size of the fleet binary by about 14M by adding the -s -w flags to a build on Darwin.

I wasn't certain about how your scripts were building and publishing the binaries, so I didn't make a PR. Looks like -s is added for Linux, but the darwin builds don't get it, and neither gets -w

Add --agent-image flag to the agent-token command

thanks for the fl(ee)t ⛵

For the rendering the agent register manifest, currently there's no flag to specify the agent image.
If a different registry for the agent-cluster should be used, the renders manifest must be changed afterwards.

Clarity on gitRepo with private git repository

Hi,

I've successfully added the public rancher gitRepo - https://github.com/rancher/fleet-examples and deployed the multi-cluster/kustomizer example.

I've forked the repo, created a deployment ssh key, add the private ssh key as a secret:

kubectl -n $(CLUSTERS_NS) create secret generic example-repo-key --from-file=ssh-privatekey=./deploykeys/examples --type=kubernetes.io/ssh-auth

I've created another gitRepo resource:

kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  # Any name can be used here
  name: examples-repo-private
  namespace: clusters
spec:
  repo: [email protected]:mumblez/fleet-examples/
  branch: master

  # For a private registry you must supply a clientSecretName. A default
  # secret can be set at the namespace level using the BundleRestriction
  # type. Secrets must be of the type "kubernetes.io/ssh-auth" or
  # "kubernetes.io/basic-auth". The secret is assumed to be in the
  # same namespace as the GitRepo
  #
  clientSecretName: example-repo-key

  paths:
  - multi-cluster/kustomize

  targets:
  - name: dev
    clusterSelector:
      matchLabels:
        env: dev

16:57:55 $ kubectl -n clusters get gitrepo
NAME                    REPO                                         COMMIT                                     BUNDLEDEPLOYMENTS-READY   STATUS
examples-repo-private   [email protected]:mumblez/fleet-examples/                                                  0/0
examples-repo-public    https://github.com/rancher/fleet-examples/   541764c8c332f2b9a1262b8ef0d9fdc557d01159   1/1

17:06:33 $ kubectl -n clusters get bundle
NAME                                           BUNDLEDEPLOYMENTS-READY   STATUS
examples-repo-public-multi-cluster-kustomize   1/1
fleet-agent                                    1/1

Not sure what I'm doing wrong but it doesn't appear the private repo is working, the key is definately added as a secret the it's in the same namespace, I'm guessing the key for the secret should be something other than ssh-privatekey ?

Is fleet compatible with Managed Clusters?

When I hear the term "Managed Kubernetes" I think of hosted Kubernetes solutions like AWS, GCP and Digital Ocean.

Since fleet uses the term "manage" 11 times in the readme and in the blog post it's not obvious to me if this works with "Managed" clusters or not.

From the install instructions, it appears that it wouldn't matter if you had control over the Kubernetes Master servers and all fleet is really doing is taking care of what's deployed on the cluster.

I really like the word fleet as it conveys really well the spirit of this project. I'm less convinced by the term "manage" as it already has a strong presence in the kubernetes community.

I would also think about renaming the fleet manager to fleet controller or fleet master.

I like what fleet is doing. I don't write much go but would love to contribute from time to time. Do you have a slack or gitter?

Ability to configure Calico via Fleet

We should be able to create and manage Calico policy using the Calico-native spec; Fleet should then run calicoctl to import that configuration into the target cluster(s).

Implementation of this feature in Fleet would most likely allow us to eliminate Aqua from our environment and switch to using Calico exclusively for our cluster-wide firewall policy.

I'm aware that calicoctl creates Kubernetes-native resources like CRDs when it runs, and that in theory these could be managed directly using Fleet's baseline resource management functionality. But the value here is that Calico has a documented schema for its network policies, and we would much rather manage Calico policies in their native schema rather than a "translated" version. This also helps to make us more immune to changes between Calico versions.

53 character limit in bundle names

There seems to be a 53 character limit in the names of bundles. It would be ideal if this could be greater, or if we could define what is used in the name. It seems to be repo name + path + fleet.yaml paths

Fleet agent does not deploy on a windows cluster

On v2.5-head - commit id: ce572b5f

• Deploy a windows cluster.
• Noticed fleet agent does not deploy.
• Error ImagePullBackOff: Back-off pulling image "rancher/fleet-agent:v0.3.0-beta6"

Release CLI is using DEV version

I'm using the 0.2.0 release binary: https://github.com/rancher/fleet/releases/download/v0.2.0/fleet_0.2.0_darwin_amd64.tar.gz

But the version seems to be dev:

$ fleet --version
flt version dev (HEAD)

This results in using dev tagged container images as a default.

Doc - Example yaml for defining targets has a typo

kubectl apply -f targets.yaml

https://fleet.rancher.io/gitrepo-targets/#defining-targets

{"level":"error","ts":1599065407.1952014,"caller":"git/git.go:41","msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin --update-head-ok --force warning:]: exit status 128\nwarning: redirecting to https://github.com/rancher/fleet-examples/\nfatal: couldn't find remote ref warning\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:41\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:116\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:53\nruntime.main\n\truntime/proc.go:203"}
{"level":"fatal","ts":1599065407.1954148,"caller":"git-init/main.go:54","msg":"Error fetching git repository: failed to fetch [warning:]: exit status 128","stacktrace":"main.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:54\nruntime.main\n\truntime/proc.go:203"}

I was able to remedy this issue by setting repo this way:

   ...
    repo: "https://github.com/rancher/fleet-examples"
   ...

It seems github repo doesn't support insecure http ?

Using clusterGroupSelector in GitRepo is not deploying the bundles to the clusters alpha9

rancher/fleet-agent:v0.3.0-alpha9
Rancher version: master-head (09/16/2020) HA k8s 59936f7533147e7c6fb0965244a82ae487b4845d

k get bundles -n fleet-default
NAME                                                          BUNDLEDEPLOYMENTS-READY   STATUS
fleet-agent                                                   1/1
simple-multi-cluster-helm                                     0/0
simple-simple                                                 0/0
test                                                          0/0
test-multi-cluster-helm                                       0/0
test-multi-cluster-helm-external                              0/0
test-multi-cluster-helm-kustomize                             0/0
test-multi-cluster-kustomize                                  0/0
test-multi-cluster-manifests                                  0/0

Keeping this same configuration but using clusterGroup or clusterSelector instead of clusterGroupSelector* works.
This also previously worked in alpha8 using bundleDirs

---
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  creationTimestamp: "2020-09-16T19:28:44Z"
  generation: 8
  name: simple
  namespace: fleet-default
  resourceVersion: "67604"
  selfLink: /apis/fleet.cattle.io/v1alpha1/namespaces/fleet-default/gitrepos/simple
  uid: e9e9b67c-244d-4479-ac69-e3e1b195eb08
spec:
  paths:
  - multi-cluster/helm
  repo: https://github.com/rancher/fleet-examples/
  targets:
  - clusterGroupSelector:
      matchLabels:
        group: group1
    name: group1
---
apiVersion: fleet.cattle.io/v1alpha1
kind: Cluster
metadata:
  annotations:
    objectset.rio.cattle.io/applied: '{"metadata":{"name":"c-wzlpj","namespace":"fleet-default","creationTimestamp":null,"labels":{"management.cattle.io/cluster-display-name":"test-83927","management.cattle.io/cluster-name":"c-wzlpj","objectset.rio.cattle.io/hash":"f4b0c1abd92d94af2ec1824c855e4bd41c0adc25"},"annotations":{"objectset.rio.cattle.io/id":"fleet-cluster","objectset.rio.cattle.io/owner-gvk":"management.cattle.io/v3,
      Kind=Cluster","objectset.rio.cattle.io/owner-name":"c-wzlpj","objectset.rio.cattle.io/owner-namespace":""},"ownerReferences":[{"apiVersion":"management.cattle.io/v3","kind":"Cluster","name":"c-wzlpj","uid":"17cfa8f9-fc73-4508-95e8-dbbbe1d1c136","controller":false,"blockOwnerDeletion":false}]},"spec":{"kubeConfigSecret":"c-wzlpj-kubeconfig"},"status":{"summary":{"ready":0,"desiredReady":0},"display":{},"agent":{"lastSeen":null}}}'
    objectset.rio.cattle.io/id: fleet-cluster
    objectset.rio.cattle.io/owner-gvk: management.cattle.io/v3, Kind=Cluster
    objectset.rio.cattle.io/owner-name: c-wzlpj
    objectset.rio.cattle.io/owner-namespace: ""
  creationTimestamp: "2020-09-16T19:18:43Z"
  generation: 2
  labels:
    env: test
    group: testgroup1
    management.cattle.io/cluster-display-name: test-83927
    management.cattle.io/cluster-name: c-wzlpj
    objectset.rio.cattle.io/hash: f4b0c1abd92d94af2ec1824c855e4bd41c0adc25
    manager: fleetcontroller
    operation: Update
    time: "2020-09-16T22:23:58Z"
  name: c-wzlpj
  namespace: fleet-default
  ownerReferences:
  - apiVersion: management.cattle.io/v3
    blockOwnerDeletion: false
    controller: false
    kind: Cluster
    name: c-wzlpj
    uid: 17cfa8f9-fc73-4508-95e8-dbbbe1d1c136
  resourceVersion: "67267"
  selfLink: /apis/fleet.cattle.io/v1alpha1/namespaces/fleet-default/clusters/c-wzlpj
  uid: 71b2967d-6d17-4fdd-8a7d-72be344cd183
spec:
  clientID: 7j5l52zpq5h696q7cp2nbcjt6rjj4g9k8zphrdmbqk6f7mprgdxgg5
  kubeConfigSecret: c-wzlpj-kubeconfig
---
apiVersion: fleet.cattle.io/v1alpha1
kind: ClusterGroup
labels:
  group: group1
metadata:
  creationTimestamp: "2020-09-16T19:27:46Z"
  generation: 3
  name: group1
  namespace: fleet-default
  resourceVersion: "59841"
  selfLink: /apis/fleet.cattle.io/v1alpha1/namespaces/fleet-default/clustergroups/group1
  uid: 65ea4ad1-3589-41f6-bdf8-ff0c8edb49bf
spec:
  selector:
    matchLabels:
      group: testgroup1

Single Cluster Quickstart Not Working

Tried provisioning a single cluster fleet via quickstart directions on a fresh single node k3s.

root@ubuntu-focal:~# helm -n fleet-system install --create-namespace --wait \
>     fleet-crd https://github.com/rancher/fleet/releases/download/v0.3.0-alpha6/fleet-crd-0.3.0-alpha6.tgz
NAME: fleet-crd
LAST DEPLOYED: Thu Sep  3 20:11:26 2020
NAMESPACE: fleet-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
root@ubuntu-focal:~# helm -n fleet-system install --create-namespace --wait \
>     fleet https://github.com/rancher/fleet/releases/download/v0.3.0-alpha6/fleet-0.3.0-alpha6.tgz
NAME: fleet
LAST DEPLOYED: Thu Sep  3 20:11:35 2020
NAMESPACE: fleet-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
root@ubuntu-focal:~# cat > example.yaml << "EOF"
> apiVersion: fleet.cattle.io/v1alpha1
> kind: GitRepo
> metadata:
>   name: sample
>   # This namespace is special and auto-wired to deploy to the local cluster
>   namespace: fleet-local
> spec:
>   # Everything from this repo will be ran in this cluster. You trust me right?
>   repo: "https://github.com/rancher/fleet-examples"
>   bundleDirs:
>   - simple
> EOF
root@ubuntu-focal:~#
root@ubuntu-focal:~# kubectl apply -f example.yaml
gitrepo.fleet.cattle.io/sample created
root@ubuntu-focal:~# kubectl -n fleet-local get fleet
NAME                                   CLUSTERS-READY   BUNDLES-READY   STATUS
clustergroup.fleet.cattle.io/default   0/1 (local)      0/2             NotApplied(2) [Bundle fleet-agent: ]

NAME                             REPO                                        COMMIT                                     BUNDLEDEPLOYMENTS-READY   STATUS
gitrepo.fleet.cattle.io/sample   https://github.com/rancher/fleet-examples   14b81f64b5d33ca0783ceec4f7b8b6f9f41251a0   0/1                       NotApplied(1) [Bundle sample-simple: ]

Provide mechanism to identify clusters

Right now clusters can really only be identified by labels. Additionally labels can only be set on create by the cluster, and modified later in the fleet manager. This makes identifying clusters that don't have useful labels quite difficult.

Clusters should report some high level basic information in order to identify them.

[Question] Comparison with kubefed

Interesting project. We are currently using kubefed (https://github.com/kubernetes-sigs/kubefed) in our project together with k3s to deploy services to our fleet of (edge) clusters. I'm wondering how this compares?

Kustomize is not applied if base is another github repo

It seems, that fleet way of kustomize only works with base/overlay yaml files, present in the repo.
So if kustomization.yaml just provides github repo base, it fails.

Error:
Bundle gitlab-dev-gcp-minio-operator: error while running post render on files: accumulating resources: accumulateFile "accumulating resources from 'github.com/minio/operator?ref=v3.0.25': 'github.com/minio/operator?ref=v3.0.25' doesn't exist", accumulateDirector: "couldn't make target for path '/tmp/kustomize-674907914/repo': unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization' in directory '/tmp/kustomize-674907914/repo'"

See file contents below:

kustomization.yaml

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - github.com/minio/operator?ref=v3.0.25
namespace: minio-operator

gitrepo.yaml

---
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: repo
  namespace: fleet-local
spec:
  repo: some-repo
  branch: master
  clientSecretName: some-creds
  paths:
  - path/to/kustomization.yaml

Failed to deploy any app in Rancher master-head

Steps:

• Running the latest rancher:master-head in HA mode
• Provision clusters
• the clusters are registered to Fleet automatically
• add the label env=dev to cluster-1
• deeply the following gitRepo CRD

kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: myrepo
  namespace: fleet-default
spec:
  repo: http://github.com/rancher/fleet-examples
  bundleDirs:
  - simple
  targets:
  - name: dev
    clusterSelector:
      matchLabels:
        env: dev

Results:
the following error shows in the bundle's pod

{"level":"error","ts":1600111382.5115998,"caller":"git/git.go:41","msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin --update-head-ok --force warning:]: exit status 128\nwarning: redirecting to https://github.com/rancher/fleet-examples/\nfatal: couldn't find remote ref warning\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:41\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:116\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:53\nruntime.main\n\truntime/proc.go:203"}
{"level":"fatal","ts":1600111382.511774,"caller":"git-init/main.go:54","msg":"Error fetching git repository: failed to fetch [warning:]: exit status 128","stacktrace":"main.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:54\nruntime.main\n\truntime/proc.go:203"}

fleet-examples are not properly deployed after following the quickstart guide

cat kustomize.yaml

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: kustomize
  namespace: fleet-local
spec:
  repo: "https://github.com/rancher/fleet-examples"
  bundleDirs:
    - kustomize

kubectl get deploy kustomize
Error from server (NotFound): deployments.apps "kustomize" not found

kubectl -n fleet-local get fleet
NAME                                REPO                                         COMMIT                                     BUNDLEDEPLOYMENTS-READY   STATUS
gitrepo.fleet.cattle.io/kustomize   https://github.com/rancher/fleet-examples    3614aa139d8bee616fc6849d54d8d836aedfd700   0/1                       NotApplied(1) [Bundle kustomize-kustomize: ]
gitrepo.fleet.cattle.io/simple      https://github.com/rancher/fleet-examples/   3614aa139d8bee616fc6849d54d8d836aedfd700   0/1                       NotApplied(1) [Bundle simple-simple: ]

NAME                                   CLUSTERS-READY   BUNDLES-READY   STATUS
clustergroup.fleet.cattle.io/default   0/1 (local)      0/4             NotApplied(4) [Bundle fleet-agent: ]

Fleet token generation command not working

Steps to reproduce-

• have a plain k3s cluster
• Install fleet
• make it as a manager by running
  fleet install manager | kubectl apply -f -
• run fleet install agent-token > token
  above command results in error Error: invalid host 127.0.0.1, use --server-url to set a proper server URL
  if you check the kubeconfig or kubectl config view the API is at https://127.0.0.1:6443 but still fleet refuses to generate token

Fleet cannot recognize the Monitoring v2 chart

I tried to install the monitoring v2 chart using Fleet, but it looks like either Fleet cannot recognize the chart correctly or the chart does not meet Fleet's requirements for git repo.

>  cat app-monitoring.yaml
kind: GitRepo
apiVersion: fleet.cattle.io/v1alpha1
metadata:
  name: monitoring
  namespace: clusters
spec:
  repo: https://github.com/rancher/charts
  branch: dev-v2.5
  bundleDirs:
  - packages/rancher-monitoring
  targets:
  - name: prod
    clusterSelector:
      matchLabels:
        env: dev

> k -n clusters get cluster --show-labels
NAME                   BUNDLES-READY   NODES-READY   SAMPLE-NODE    LAST-SEEN              STATUS   LABELS
cluster-4ab647a24107   1/1             3/3           jiaqi-agent2   2020-09-04T17:12:28Z            env=dev,fleet.cattle.io/cluster=cluster-4ab647a24107
cluster-78f22966ef28   1/1             3/3           jiaqi-agent1   2020-09-04T17:14:47Z            env=test,fleet.cattle.io/cluster=cluster-78f22966ef28

> k -n clusters get gitrepo
NAME         REPO                                COMMIT                                     BUNDLEDEPLOYMENTS-READY   STATUS
monitoring   https://github.com/rancher/charts   4e030b0ca7ae349ea3b28c4593b00d142da2b80e   0/0

> k -n clusters logs monitoring-c2e0d-d5j7z fleet
time="2020-09-04T20:28:25Z" level=warning msg="packages/rancher-monitoring: no resources found to deploy"
time="2020-09-04T20:28:25Z" level=fatal msg="no fleet.yaml or bundle.yaml found at the following paths: [packages/rancher-monitoring]"

> k -n clusters logs monitoring-c2e0d-d5j7z step-git-source
{"level":"info","ts":1599251304.983305,"caller":"git/git.go:136","msg":"Successfully cloned https://github.com/rancher/charts @ 53dc624ee65e0c91c6e735690c07c5f91fcbe9ab (grafted, HEAD) in path /workspace/source"}
{"level":"info","ts":1599251305.0369422,"caller":"git/git.go:177","msg":"Successfully initialized and updated submodules in path /workspace/source"}