What would you like to be added:
Once the csi-operator is released, a new csi-operator image needs to be integrated into TKE's release package pipeline.
Why is this needed:
Prerequisites for publication.

Describe the bug

when creating a new cluster, specify the target node’s username and password, if input non-root username and password, the installation will get error like below:

Note: User ubuntu is a valid user for the node, and it has sudo permission

Environment

• OS: Ubuntu 16.04.1 LTS
• kernel: 4.4.0-130-generic x86_64

tke-installer-v0.15.0 doesn't contain local-tcr and kubernetes images, so I need to download these images manually.

installer log:
v0.1: digest: sha256:cc153f2ed523952e85a73fafaff87d05fb14fb7e068ef330fb671274c4ed5998 size: 734
2019/12/04 08:08:38 upload registry.tke.com/library/configmap-reload:v0.1 to registry success[46/46]
2019/12/04 08:08:38 3.Push images to registry.tke.com/library [Success] [2.350667s]

docker images:
# docker images | grep kube
tkestack/kube-state-metrics v1.6.0 87970984231a 6 months ago 230MB
registry.tke.com/library/kube-state-metrics v1.6.0 87970984231a 6 months ago 230MB
# docker images | grep registry.tke.com | wc -l
46
# docker images | grep registry.tke.com | grep kube
registry.tke.com/library/kube-state-metrics v1.6.0 87970984231a 6 months ago 230MB

Describe the bug
Just use the tke-installer-x86_64-10fd8a9.run version to deploy tkestack, but the installation hang at ‘Install tke-monitor-api doing’
Check the pod status, it is pending on Insufficient CPU.
Hope the installer will check the node’s cpu&mem capacity before installing the master node

[root@ruyingzhe8 ~]# kubectl get pods -n tke
NAME                                       READY   STATUS    RESTARTS   AGE
influxdb-0                                 1/1     Running   0          100s
tke-auth-586bd7d8f5-slwf2                  1/1     Running   0          2m53s
tke-auth-586bd7d8f5-ssnvx                  1/1     Running   0          2m53s
tke-business-api-68bd76b66f-sjs8t          1/1     Running   0          2m3s
tke-business-api-68bd76b66f-zm9vv          1/1     Running   0          2m3s
tke-business-controller-547b5bf744-w9b2l   1/1     Running   0          105s
tke-business-controller-547b5bf744-zzkmk   1/1     Running   0          105s
tke-monitor-api-5664f5c876-qjtpq           1/1     Running   0          95s
tke-monitor-api-5664f5c876-zhrjb           0/1     Pending   0          95s
tke-platform-api-6dd5b47bd7-42jr5          1/1     Running   0          2m38s
tke-platform-api-6dd5b47bd7-qz4f2          1/1     Running   0          2m38s
tke-platform-controller-7b5755f47-mp6bs    1/1     Running   0          2m23s
tke-platform-controller-7b5755f47-wpbgf    1/1     Running   0          2m23s
tke-registry-api-74bdb659b9-5kp6p          1/1     Running   0          2m18s
[root@ruyingzhe8 ~]# 
[root@ruyingzhe8 ~]# 
[root@ruyingzhe8 ~]# kubectl describe pod -n tke tke-monitor-api-5664f5c876-zhrjb
Name:               tke-monitor-api-5664f5c876-zhrjb
Namespace:          tke
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             app=tke-monitor-api
                    pod-template-hash=5664f5c876
Annotations:        <none>
Status:             Pending
IP:                 
Controlled By:      ReplicaSet/tke-monitor-api-5664f5c876
Containers:
  tke-monitor-api:
    Image:      registry.tke.com/library/tke-monitor-api:10fd8a9
    Port:       9455/TCP
    Host Port:  0/TCP
    Args:
      -C=/app/conf/tke-monitor-api.toml
    Limits:
      cpu:     500m
      memory:  1Gi
    Requests:
      cpu:        250m
      memory:     256Mi
    Liveness:     http-get https://:9455/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:    http-get https://:9455/healthz/ping delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /app/certs from certs-volume (rw)
      /app/conf from tke-monitor-api-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-s2vkn (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  certs-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      certs
    Optional:  false
  tke-monitor-api-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      tke-monitor-api
    Optional:  false
  default-token-s2vkn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-s2vkn
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  35s (x4 over 116s)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.

Environment

• OS: CentOS Linux release 7.6.1810 (Core)
• kernel: 3.10.0-957.21.3.el7.x86_64
• docker-ce: 18.09.9

安装好集群后，如何再添加node节点

Describe the bug

I just run the tke-installer:v0.15.0 image and to install the first new master node for global cls.
After I click the accept button and wait 5min, the console look like the picture attached.

Login to the master node, there are few images on the node, just run:

[root@VM_252_13_centos docker]# docker images
REPOSITORY                         TAG                 IMAGE ID            CREATED             SIZE
tkestack/kube-apiserver            v1.14.6             0e422c9884cf        3 months ago        209MB
tkestack/kube-controller-manager   v1.14.6             4bb274b1f2c3        3 months ago        157MB
tkestack/kube-scheduler            v1.14.6             d27987bc993e        3 months ago        81.6MB
tkestack/etcd                      v3.3.12             28c771d7cfbf        9 months ago        40.6MB
tkestack/pause                     3.1                 da86e6ba6ca1        23 months ago       742kB
[root@VM_252_13_centos ~]# docker pull registry.tke.com/library/tke-auth:v0.15.0
Error response from daemon: Get https://registry.tke.com/v2/: dial tcp 10.0.252.2:443: connect: connection refused
[root@VM_252_13_centos ~]# cat /etc/docker/daemon.json 
{
  "debug": false,
  "insecure-registries": [
    "docker.io"
  ],

registry.tke.com is the default registry configuration I set in the console.

Environment

• OS: CentOS Linux release 7.6.1810 (Core)
• kernel: 3.10.0-957.21.3.el7.x86_64
• docker-ce: 18.09.9

installer log:
Loaded image: tkestack/log-collector:v1.1.0
Loaded image: tkestack/nvidia-device-plugin:1.0.0-beta4
Loaded image: tkestack/tke-notify-api:v0.15.0
Loaded image: tkestack/tke-platform-controller:v0.15.0
Loaded image: tkestack/tke-registry-api:v0.15.0
2019/12/03 09:22:31 1.Load images [Success] [57.028089s]
2019/12/03 09:22:31 2.Setup local registry doing
f4383bf275822cf6eb76df15e591be92b94cae557016c6741c37b3a86d05b0d4
2019/12/03 09:22:32 2.Setup local registry [Failed] [0.404418s] error open hosts: no such file or directory

cmd/tke-installer/app/installer/installer.go
line 1148: localHosts := hosts.LocalHosts{Host: server, File: "hosts"}
If specify ' File:"hosts" ', localHost.File would be 'hosts' rather than '/etc/hosts'

What happened: install cmd error

• error
  brew service start etcd

• true
  brew services start etcd

Describe the bug
There are some duplicated policies in ‘access management’->’policy management’, the policy’s name and description are same but the id is different.

Environment

• OS: CentOS Linux release 7.6.1810 (Core)
• kernel: 3.10.0-957.21.3.el7.x86_64
• docker-ce: 18.09.9

What happened:
Use TKE local registry, then push a large image (> 2GB):
The push refers to repository
87b1b5a611f8: Layer already exists
68dd2761fd81: Pushing [==================================================>] 2.086GB/2.086GB
received unexpected HTTP status: 504 Gateway Timeout

Environment:

• TKE version: v1.0.0
• Global or business cluster: global
• Kubernetes version (use kubectl version):
• Install addons:
• Others:

What would you like to be added:
Once the tapp is released, a new tapp image needs to be integrated into tke's release package pipeline. At the same time, you need to specify the cluster's tapp resource forwarding in the platform api to ensure that the forwarded path matches the CRD API group and version of the tapp.

Why is this needed:
Prerequisites for publication.

What would you like to be added:
Once the galaxy is released, a new galaxy image needs to be integrated into TKE's release package pipeline.

Why is this needed:
Prerequisites for publication.

What would you like to be added:
Once the LBCF is released, a new lb-controlling-framework image needs to be integrated into TKE's release package pipeline. At the same time, you need to specify the cluster's:

• LBCFBackendRecord
• LBCFDriver
• LBCFBackendGroup
• LBCFLoadBalancer

resources forwarding in the platform api to ensure that the forwarded path matches the CRD API group and version of the LBCF.

Why is this needed:
Prerequisites for publication.

Create a business cluster with the config 2048 services per cluster on console, finally get only 256 services per cluster. And the --service-cluster-ip-range is 192.168.255.0/24, actually only support 256 services.

Environment:

• TKE version: v1.0.0
• Global or business cluster: business cluster
• Kubernetes version (use kubectl version):
• Install addons:
• Others:

What would you like to be added:
Once the gpu-manager is released, a new gpu-manager image needs to be integrated into TKE's release package pipeline.

Why is this needed:
Prerequisites for publication.

What would you like to be added:
Support different scenarios or cluster scale installation
Why is this needed:
Reduce resource constraints

What happened:
2019/12/12 05:01:53 upload registry.tke.com/library/configmap-reload:v0.1 to registry success[48/48]
2019/12/12 05:01:53 3.Push images to registry.tke.com/library [Success] [274.427330s]
2019/12/12 05:01:53 4.Generate certificates for TKE components doing
2019/12/12 05:01:54 4.Generate certificates for TKE components [Success] [0.921570s]
2019/12/12 05:01:54 5.Prepare front proxy certificates doing
2019/12/12 05:01:55 5.Prepare front proxy certificates [Failed] [0.644614s] error create file error:/etc/kubernetes/admin.crt:file does not exist

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• TKE version:
• Global or business cluster:
• Kubernetes version (use kubectl version):
• Install addons:
• Others:

root@VM-64-34-ubuntu:/home/ubuntu# uname -a
Linux VM-64-34-ubuntu 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@VM-64-34-ubuntu:/home/ubuntu# cat tke-installer-x86_64-v1.0.0.run.sha256
dfb68e96c5831a3cda5725ed2f5ad10136bf6bd244afac095b71dc9a32682e82 tke-installer-x86_64-v1.0.0.run

Could you please add Docker Root Dir, registry(already configure registry.tke.com, but don't work in docker daemon.json) and http_proxy configuration of the Global cluster on installer web page? Otherwise, after docker installed, its root dir is /var/lib/docker and its registry is docker.io and my network cannot connect to docker.io,

docker info

Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.09.9
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.107-1-tlinux2-0046
Operating System: Tencent tlinux 2.2 (Final)
OSType: linux
Architecture: x86_64
CPUs: 80
Total Memory: 125.3GiB
Name: TENCENT64.site
ID: AVWD:B3RY:H7V6:2L3D:VUZO:PJ6M:E4B3:SEC2:5DQR:SRLJ:UM75:7KSN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://docker.io/
Live Restore Enabled: true
Product License: Community Engine

Describe the bug

When using the make release command to build the installer, errors occurred like below.

===========> Pushing tke-platform-api c72b21f image to tkestack
The push refers to repository [docker.io/tkestack/tke-platform-api]
675b40083459: Preparing
83b390cc87eb: Preparing
6132d92de499: Preparing
77cae8ab23bf: Preparing
denied: requested access to the resource is denied
make[2]: *** [image.push.tke-platform-api] Error 1
make[2]: Leaving directory `/root/gopath/src/tkestack.io/tke'
make[1]: *** [push] Error 2
make[1]: Leaving directory `/root/gopath/src/tkestack.io/tke'
make: *** [release] Error 2

Since I don’t have the permission of tkestack repo in dockerhub, I can’t build the tke-installer image for now.

Hope tkestack support releasing image repos other than tkestack/*, user can customize their own repository

Environment

• OS: Linux VM_149_11_centos 3.10.107-1-tlinux2_kvm_guest-0049 # 1 SMP Tue Jul 30 23:46:29 CST 2019 x86_64 x86_64 x86_64 GNU/Linux
• golang: go version go1.12.4 linux/amd64

What would you like to be added:

Why is this needed:

Hello

What would you like to be added:
Could you plz upload your monitor.json and notify.json for running locally?

And Can you give me your contact information? Like wechat group

Why is this needed:

What would you like to be added:
Support installer to listen on all network interface
Why is this needed:
In the user multi-NIC scenario, it is impossible to determine the specific network topology of the user

What would you like to be added:

Why is this needed:

What happened:
After installation, the registry address in global master node hosts is refer to the installer node

Environment:

• TKE version: v1.0.0
• Global or business cluster: global
• Kubernetes version (use kubectl version):
• Install addons:
• Others:

What would you like to be added:

Why is this needed:

What would you like to be added:

• Reduce response time to request deployment events.
• Requesting deployment events only returns events relate to the exact deployment.

Why is this needed:

• Requesting deployment events timeout with too many deployments with same labels.

Describe the bug
Just follow the guide in “organization resource”->”registry management”->”your registory”->”image upload guide”, the tag and pull commands runs into error:
the guide under “organization resource”->”access certificate” has the same problem.

# sudo docker tag nginx:latest https://default.registry.tke.com/myreg/nginx:latest
Error parsing reference: "https://default.registry.tke.com/myreg/nginx:latest" is not a valid repository/tag: invalid reference format

Environment

• OS: CentOS Linux release 7.6.1810 (Core)
• kernel: 3.10.0-957.21.3.el7.x86_64
• docker-ce: 18.09.9

Describe the bug
Just use the tke-installer-x86_64-10fd8a9.run version to deploy tkestack, and the installation complete successfully.
But cannot access the console link, raise HTTP 502 ERROR.
Just check the pod status, found the the-gateway pod is in creating status.
The installer should check all of the components status, before sending the install task success message.

Environment

• OS: CentOS Linux release 7.6.1810 (Core)
• kernel: 3.10.0-957.21.3.el7.x86_64
• docker-ce: 18.09.9

incremental compilation
when I use make release to build the installer image, it will cost duplicative effort to generate the images.tar.gz, even the file already exist.

incremental installation
each time when I start a new installation, tke-installer always load the image.tar.gz, no matter if i have done this before.

root     16070 16052  0 15:48 ?        00:00:00 /app/bin/tke-installer
root     16187 16070  3 15:49 ?        00:00:00 docker load -i images.tar.gz
root     16197  5766 12 15:49 ?        00:00:01 docker-untar / /var/lib/docker/tmp/docker-import-404992003

What would you like to be added:
Add more check at container network configuration

Why is this needed:
The IP address ranges of node network and container network cannot conflict, it will cause unexpected network problem.

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• TKE version:
• Global or business cluster:
• Kubernetes version (use kubectl version):
• Install addons:
• Others:

What happened:
installer log:
2019/12/04 13:09:47 OnInitialize.EnsureKubeadmInitUploadConfigPhase [Failed] [0.113197s] reason: FailedProcess message: exec "kubeadm init phase upload-config all --config=/etc/kubernetes/kubeadm-config.yaml" failed:exit 1:stderr error execution phase upload-config/kubeadm: error uploading the kubeadm ClusterConfiguration: unable to create ConfigMap: Post https://10.19.0.152:6443/api/v1/namespaces/kube-system/configmaps: Forbidden :error %!s() retry: 10

How to reproduce it (as minimally and precisely as possible):
install Global to one node

Anything else we need to know?:
~/.kube/config apiserver address is 127.0.0.1:6443, and kubectl get cs is ok:
# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
But if I changed the apiserver address to node ip 10.19.0.152, then it returned Forbidden:
# kubectl get cs
Unable to connect to the server: Forbidden
So, I think the certificate is error.

Environment:

• TKE version: v0.15.0
• Global or business cluster: Global
• Kubernetes version (use kubectl version):
• Install addons:
• Others:

Describe the bug
Create an user in ‘access management’->’user management’, open a new browser and use the new created one to login,
Then it jump into an error page, endless, again and again, even close the page and reopen.

Environment

• OS: CentOS Linux release 7.6.1810 (Core)
• kernel: 3.10.0-957.21.3.el7.x86_64
• docker-ce: 18.09.9

What would you like to be added:
Once the volume-decorator is released, a new volume-decorator image needs to be integrated into TKE's release package pipeline.

Why is this needed:
Prerequisites for publication.

# kubectl create -f cluster2.yaml

error: SchemaError(io.tkestack.tke.pkg.monitor.services.rest.Response.data): Unknown primitive type: "interface{}"

cluster2.yaml
apiVersion: platform.tkestack.io/v1
kind: Cluster
metadata:
generateName: cls
spec:
apiServerExtraArgs:
service-node-port-range: 1-65535
displayName: cluster2
clusterCIDR: 192.168.0.0/16
dnsDomain: cluster.local
features:
ipvs: false
finalizers:
- cluster
networkDevice: eth1
properties:
maxClusterServiceNum: 2048
maxNodePodNum: 256
tenantID: default
type: Baremetal
version: 1.14.6
machines:
- ip: xxx
port: xxx
username: root
password: xxx

Environment:

• TKE version: v1.0.0
• Global or business cluster: Global
• Kubernetes version (use kubectl version):
• Install addons:
• Others:

now I want to know some knowledge about TKE. Have wechat group ?

What happened:
I'm trying to import tkeclientset as my development library.

What you expected to happen:

go mod vendor -v without error

How to reproduce it (as minimally and precisely as possible):
go.mod and main.go running go mod vendor -v

module github.com/John-Lin/client

go 1.13

replace tkestack.io/tke => github.com/tkestack/tke v1.0.1

require tkestack.io/tke v1.0.1

package main

import (
    tkeclientset "tkestack.io/tke/api/client/clientset/versioned"
)

func main() {
    restConfig := &rest.Config{
        Host:        "https://127.0.0.1:9443",
        BearerToken: "token",
        Timeout:     time.Second * 5,
        TLSClientConfig: rest.TLSClientConfig{
            Insecure: true,
        },
    }

    client, err := tkeclientset.NewForConfig(restConfig)
    if err != nil {
        fmt.Printf("%v", err)
    }
}

Anything else we need to know?:

Environment:

• TKE version: v1.0.1
• Global or business cluster: None
• Kubernetes version (use kubectl version): None
• Install addons: None

What would you like to be added:
When to install TKEStack, if If set no-HA mode, the tke-installer should block user to add multi master node

Why is this needed:
If set no-HA mode, there is no VIP assigned to the apiservers, other components will connect multi apiservers address in random, that will involve risks.

go mod vendor

go: github.com/coreos/[email protected] requires
        github.com/prometheus/[email protected]+incompatible: reading https://goproxy.io/github.com/prometheus/prometheus/@v/v2.9.2+incompatible.mod: 410 Gone

狗日的