PTAL: Please Take A Look
LGTM: Look Good To Me
SGTM: Sound Good To Me
iff: if and only if
WIP: Work In Progress
AFAIK: As Far As I Know
FYI: For Your Information
IIRC: If I Remember Correctly

Phenomenons

Can not SSH connect the machine through one user, but it is OK for root or other user

# ssh [email protected]
shell request failed on channel 0

# tail -f 10 /var/log/secure
......
Feb  4 09:55:06 10.10.10.1 sshd[6137]: Accepted password for apps from 10.10.10.2 port 33216 ssh2
Feb  4 09:55:06 10.10.10.1 sshd[6137]: pam_unix(sshd:session): session opened for user apps by (uid=0)
Feb  4 09:55:06 10.10.10.1 sshd[6627]: error: do_exec_pty: fork: Resource temporarily unavailable
Feb  4 09:55:06 10.10.10.1 sshd[6137]: pam_unix(sshd:session): session closed for user apps

Can not switch to one user from root by su - xxx, but it is OK for other user

# su - xxx
Last login: Tue Apr 26 09:48:44 CST 2016 from application03 on pts/0
su: failed to execute /bin/bash: Resource temporarily unavailable

Analyses

The root cause is that the resource for the user is insufficient as there are too many processes created by this user. When the the user login through su - xxx or SSH, a new process will be created by the user. If the new process can not be created as the user's resource is insufficient, the login will be failed.

Check the process of the user xxx:

$ ps -ef | grep ^xxx|wc -l
1246

Solution

[[email protected] ~]# vi /etc/security/limits.d/20-nproc.conf
[[email protected] ~]# cat /etc/security/limits.d/20-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     65535 (#Increase the limit from 1024 to 65535)
root       soft    nproc     unlimited

Fail to run Klar in Docker according to doc Dockerized version.

Error log:

# docker run -it --rm klar
panic: standard_init_linux.go:178: exec user process caused "no such file or directory" [recovered]
	panic: standard_init_linux.go:178: exec user process caused "no such file or directory"

goroutine 1 [running, locked to thread]:
panic(0x6f2340, 0xc420138540)
	/usr/lib/golang/src/runtime/panic.go:500 +0x1a1
github.com/urfave/cli.HandleAction.func1(0xc420097748)
	/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x247
panic(0x6f2340, 0xc420138540)
	/usr/lib/golang/src/runtime/panic.go:458 +0x243
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc420097198, 0xc4200260c8, 0xc420097238)
	/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x18f
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc420057590, 0xaab9c0, 0xc420138540)
	/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353
main.glob..func8(0xc42009aa00, 0x0, 0x0)
	/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/main_unix.go:26 +0x66
reflect.Value.call(0x6dd0c0, 0x768ec8, 0x13, 0x73b509, 0x4, 0xc420097708, 0x1, 0x1, 0x4d17a8, 0x731360, ...)
	/usr/lib/golang/src/reflect/value.go:434 +0x5c8
reflect.Value.Call(0x6dd0c0, 0x768ec8, 0x13, 0xc420097708, 0x1, 0x1, 0xac1700, 0xc4200976e8, 0x4da786)
	/usr/lib/golang/src/reflect/value.go:302 +0xa4
github.com/urfave/cli.HandleAction(0x6dd0c0, 0x768ec8, 0xc42009aa00, 0x0, 0x0)
	/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x1e0
github.com/urfave/cli.Command.Run(0x73b6d5, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74cc8a, 0x51, 0x0, ...)
	/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xc3b
github.com/urfave/cli.(*App).Run(0xc4200bc000, 0xc42000c120, 0x2, 0x2, 0x0, 0x0)
	/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x611
main.main()
	/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/main.go:137 +0xbd6

Docker Version:

# docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-28.git1398f24.el7.centos.x86_64
 Go version:      go1.7.4
 Git commit:      1398f24/1.12.6
 Built:           Fri May 26 17:28:18 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-28.git1398f24.el7.centos.x86_64
 Go version:      go1.7.4
 Git commit:      1398f24/1.12.6
 Built:           Fri May 26 17:28:18 2017
 OS/Arch:         linux/amd64

Investigation:
Replace base image scratch with centos:7, there will be no problem.

References

• Best Practices for Designing a Pragmatic RESTful API
• Microsoft REST API Guidelines
• Restful API Design References
• RESTful最佳实践
• RESTful API设计最佳实践
• RESTful API 设计最佳实践

Problem

ENTRYPOINT [ "echo", "$HOME" ] will not do variable substitution on $HOME.

Solutions

exec form does not invoke a command shell. There are 2 solutions:

• Use shell form: ENTRYPOINT echo $HOME
• execute a shell directly in exec form: ENTRYPOINT [ "sh", "-c", "echo $HOME" ]

References

• Dockerfile Entrypoint Reference

Error log

error writing event (write tcp kubernetes.master.com:54753: connection reset by peer)
got unexpected response error (etcdserver: request timed out)

Version must match the regular expression: ^\s*v?([0-9]+(?:\.[0-9]+)*)(.*)*$, which has 2 parts: ${numbers}-${extra}.

Two types of version spec:

• Semantic:
  • numbers must has 3 components, like v0.1.0
  • each part can not start with 0
  • extra part must match the regular expression: ^(?:-([0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*))?(?:\+([0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*))?\s*$
• Generic:
  • numbers at least has 2 components

References

• 语义化版本 2.0.0
• K8s Version Util

Test for MongoDB

There is not in-memory MongoDB for test. Mgo which is the MongoDB driver for Go, also has to run its unit test on a local MongoDB server.

References

• RESTful Go: An API Server with 90%+ Test Coverage in 260 Lines of Code
• Unit Testing mGo
• GoMock: GoMock is a mocking framework for the Go programming language.
• Mocking Mongo in Golang

Configuration for log driver : --log-driver --log-opt json-file max-size=2m --log-opt max-file=5

The rotated json log files are deleted by Docker, but the fd of these files are still kept by Docker, so their disk size can not be released otherwise restart the Docker.

The number of kept fd is very large, if it larger then the config --default-ulimit nofile=131072 of Docker, the Docker can not work normally.

# ls -li "/proc/`cat /var/run/docker.pid`/fd" | grep '\(deleted\)' | wc -l
11405
# ls -li "/proc/`cat /var/run/docker.pid`/fd" | grep '\(deleted\)' | grep 48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931 | wc -l
4882
# ls -li "/proc/`cat /var/run/docker.pid`/fd" | grep '\(deleted\)' | grep 48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931 | head -n 3
1250037181 lr-x------ 1 root root 64 Oct 21 07:00 10000 -> /var/lib/docker/containers/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931-json.log.4 (deleted)
1250037182 lr-x------ 1 root root 64 Oct 21 07:00 10001 -> /var/lib/docker/containers/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931-json.log.4 (deleted)
1250037183 lr-x------ 1 root root 64 Oct 21 07:00 10002 -> /var/lib/docker/containers/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931-json.log.4 (deleted)

Illustration : Total number of kept fd is 11405, number of kept fd for one container is 4882.

When there are about 1K images on the Docker host, I found two problems:

• Wait about 30 seconds to start push image after run docker push command
• Containers can not be removed immediately, and Docker daemon has errors:

# docker rm -f c44ec3f542f0
Failed to remove container (c44ec3f542f0): Error response from daemon: Could not kill running container, cannot remove - [2] Container does not exist: container destroyed

Summary of key features in each Docker release.

v1.13.0

• Default storage driver has switch to overlay2 or overlay(if the kernel supports) from devicemapper
• Not compatible with plugins installed with Docker 1.12
• Configurable timeout (in seconds) to stop containers gracefully before daemon exit
• Deprecate MAINTAINER in Dockerfile (Not really deprecated, just begin the process)
• Add support for live reloading insecure registry in configuration
• Add --update-max-failure-ratio, --update-monitor and --rollback flags to docker service update
• Support docker plugin & docker stack
• Support docker stack deploy from a Compose file PR 27998
• Add new docker system command with df and prune subcommands for system resource management
• Restructure CLI commands(New docker image and docker container commands)

Known Issues

• Docker lost all containers on crash #30564
• dockerd died unexpectedly crashreport #30433

v1.12.0

• New HEALTHCHECK Dockerfile instruction to support user-defined healthchecks
• Add support for comments in .dockerignore
• Keep containers running when daemon shuts down
• Add support for reloading daemon configuration through systemd

Benchmark Results

v1.11.0

• With Docker 1.11, a Linux docker installation is now made of 4 binaries (docker, docker-containerd, docker-containerd-shim and docker-runc).
• The First Runtime Built On Containerd And Based On OCI Technology
• Fix many bugs for networking

Benchmark Results

Reference

时隔半年，Docker 再发重大版本 1.13！
What’s New in Docker 1.13

Tools

• Godep: Dependency tool for go
• Dep: Official experimental Go dependency management tool
• Govendor: Go vendor tool that works with the standard vendor file

References

• Package Management Tools
• A Survey on Build Tools of Golang and Java

Background

Want to migrate the Docker Registry from the old one(R1) to the new one(R2). R1 already has many old images, R2 is empty but will be the only registry after R1 deprecated in the future.
All new images will be pushed into R2. When pull images, directly pull them from R2 if found in R2, otherwise pull from R1.
There is a Nginx(Harbor) in front of them, and distributes the image requests to Docker Registry. The requests are firstly directed to R2. If the image is not found in R2, there will be an 404 error, then Nginx needs to redirect this request to R1.

Example

nginx.conf

worker_processes  1;
error_log  logs/error.log;

pid        E:/nginx-1.6.3/logs/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    sendfile        on;
    keepalive_timeout  65;
         
    server {
        listen       80;
        server_name  localhost;
                   
        # Docker registry migration
        error_page  404 =301 https://github.com$request_uri;
    }
}

Test
The test url http://localhost/abcde will be directed to https://github.com/abcde. The $request_uri in nginx.conf makes the request uri kept when directed.

References

• nginx 重定向 404

References

• java加密解密研究8、MAC算法家族

公司内部可以ping通外部网站，但是浏览器上网必须使用代理才能。因此，这个代理只是针对http，https等部分协议做了限制，而对其他协议是没有限制的，例如：ssh。

$ ping github.com

正在 Ping github.com [192.30.255.112] 具有 32 字节的数据:
来自 192.30.255.112 的回复: 字节=32 时间=201ms TTL=49
来自 192.30.255.112 的回复: 字节=32 时间=200ms TTL=49
来自 192.30.255.112 的回复: 字节=32 时间=198ms TTL=49
来自 192.30.255.112 的回复: 字节=32 时间=197ms TTL=49

192.30.255.112 的 Ping 统计信息:
    数据包: 已发送 = 4，已接收 = 4，丢失 = 0 (0% 丢失)，
往返行程的估计时间(以毫秒为单位):
    最短 = 197ms，最长 = 201ms，平均 = 199ms

# go get默认使用的git protocol是HTTPS
$ go get github.com/Microsoft/go-winio
# cd .; git clone https://github.com/Microsoft/go-winio E:\gocode\src\github.com\Microsoft\go-winio
Cloning into 'E:\gocode\src\github.com\Microsoft\go-winio'...
fatal: unable to access 'https://github.com/Microsoft/go-winio/': Failed to connect to github.com port 443: Timed out
package github.com/Microsoft/go-winio: exit status 128

# 手动使用SSH的git protocol可以正常git clone
$ git clone [email protected]:coreos/clair.git /e/gocode/src/github.com/coreos/clair
Cloning into 'E:/gocode/src/github.com/coreos/clair'...
remote: Counting objects: 6647, done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 6647 (delta 1), reused 2 (delta 1), pack-reused 6631
Receiving objects: 100% (6647/6647), 14.08 MiB | 377.00 KiB/s, done.
Resolving deltas: 100% (2725/2725), done.

https://kccnceu18.sched.com/

Categories

• Keyboard Shortcuts
• Closing issues using keywords
• CODEOWNERS(Example)

References

• Github Help
• 12 件可以用 GitHub 完成的很酷的事情
• GitHub 漫游指南

Problem
Using Docker-in-Docker for your CI or testing environment? Think twice.

Solutions
Share volumes /var/run/docker.sock and /etc/sysconfig/docker between Docker Host and its containers.
If the program in the containers needs to use Docker, it can call the APIs of the Docker Clients, such as go-dockerclient.
If need to run Docker commands such as build, run, push or pull, the volume /usr/bin/docker needs to be shared in addition.

Products

• Kublr: Comprehensive Enterprise-Ready Container Orchestration
• Aspen Mesh: Aspen Mesh makes it simple to monitor, manage and control microservices.

When design REST APIs, two points should be considered:

• There are enough informations for client, include code, message, data and so on.
• The response format for ALL APIs is unified.

The response from ALL APIs with the same format, but with different fields (empty will be omitted in some case) and data type/struct. So we need to correctly parse the data from the response. Here is an example implemented in Java.

1. Get the patch

# wget https://github.com/kubernetes/kubernetes/pull/44199.patch

https://github.com/kubernetes/kubernetes/pull/44199.patch will be redirected to https://patch-diff.githubusercontent.com/raw/kubernetes/kubernetes/pull/44199.patch

2. Apply the patch

Category

• How to rsync
• Install software by rpm
• Uninstall softwares without its dependency

Reference

• Linux Command Reference
• Hong Kong Linux User Group
• perf Examples

In Docker Daemon, there is a LayerUploadManager and LayerDownloadManager for pushing and pulling ALL layers. The push and pull speed are constricted by maxUploadConcurrency and maxDownloadConcurrency.

In Docker 1.10, the two limitation are NOT configurable as they are const in daemon.go :

const (
	// maxDownloadConcurrency is the maximum number of downloads that
	// may take place at a time for each pull.
	maxDownloadConcurrency = 3
	// maxUploadConcurrency is the maximum number of uploads that
	// may take place at a time for each push.
	maxUploadConcurrency = 5
)

In Docker 1.12, maxUploadConcurrency and maxDownloadConcurrency are configurable in docker daemon command through this PR 22445.

Solutions

1. Git clone upstream repo
2. Change the URL of remote origin to your forked repo address in .git/config
3. Add the upstream repo into remote
4. Fetch both the upstream and your forked repos

References

• How to Rebase a Pull Request

You can use the empty image scratch as the base image to build the minimal Docker image for your applications. scratch is a special image with truly 0B size.

But you should take care when use it for your Go applications, because the minimal application image maybe can not run. You can refer to optiopay/klar#51 for details.

Analysis

We compiled our app, but it still is dynamically linked to the libraries it needs to run (i.e., all the C libraries it binds to). Unfortunately, scratch is empty, so there are no libraries and no loadpath for it to look in.

So there will be error when run the minimal application image:

standard_init_linux.go:178: exec user process caused "no such file or directory"

Solution

Statically compile our app with all libraries built in:

CGO_ENABLED=0 go build -a -installsuffix cgo -o main .

References

• Building Minimal Docker Containers for Go Applications
• Command cgo

Requirements for log collection:

• No Invasive: No need to modify applications
• Low Cost: No need to run log collector in EVERY pod
• Friendly: No need to customize K8s

Conclusions

• The performance of Docker v1.13 is much better than v1.10.3.
• The CPU spike is because docker processes request much faster than before, the benchmark sends more request in the same period. Explanation from the author

Experiment

Environment
Physical Machine: 8CPUs, 32GB memory
OS: CentOS Linux release 7.2.1511 (Core)
Storage Driver: Device Mapper

Benchmark
Docker Micro Benchmark: Docker performance benchmark from Google, used by Kubernetes to test the performance of its depended Docker.

1. Benchmark list/inspect with varying number of containers

Using benchmark.sh -c

• Container Number
  

• Latency

  • Docker 1.10.3
    
  • Docker 1.13
    

• CPU Usage

  • Docker 1.10.3
    
  • Docker 1.13
    

2. Benchmark list/inspect with varying operation intervals

Using benchmark.sh -i

• ps [all=true] Latency

  • Docker 1.10.3
    
  • Docker 1.13
    

• ps [all=false] Latency

  • Docker 1.10.3
    
  • Docker 1.13
    

• inspect Latency

  • Docker 1.10.3
    
  • Docker 1.13
    

• CPU Usage

  • Docker 1.10.3
    
  • Docker 1.13
    

3. Benchmark list/inspect with varying number of goroutines

Using benchmark.sh -r

• ps [all=true] Latency

  • Docker 1.10.3
    
  • Docker 1.13
    

• inspect Latency

  • Docker 1.10.3
    
  • Docker 1.13
    

• CPU Usage

  • Docker 1.10.3
    
  • Docker 1.13
    

Targets

• Capacity
• Bottleneck
• Latency

Benchmarks

• Kubemark
• PerfKit Benchmarker

Reference

• CI Kubernetes Kubemark Results
• Evaluating Container Platforms at Scale
• Measuring Node Performance
• Scheduler Performance Test
• Kubernetes Performance Measurements and Roadmap(中文版)
• Updates to Performance and Scalability in Kubernetes 1.3 -- 2,000 node 60,000 pod clusters(中文版)
• 1000 nodes and beyond: updates to Kubernetes performance and scalability in 1.2
• Improving Kubernetes Scheduler Performance (中文版)
• Scale and Performance Testing of Kubernetes
• Chaos Monkey

Phenomenon

When the memory limit in k8s container spec is less than 4MB, the pod will can not be created and will be always in ContainerCreating.
You will easily meet this problem when use m as the memory unit, because m is just 1/1000 byte, which is too small. So m can be used for memory, but it is not recommended.

Analysis

You can refer to Docker Doc, or verify by the command:

# docker run -m 128 nginx:latest
/usr/bin/docker-current: Error response from daemon: Minimum memory limit allowed is 4MB.

I think this limitation is not set by Docker, but the requirement of Linux kernel.
When the memory limit is less than 4MB, the created container will be immediately stopped by oom-killer.

Tools to support swagger for Go

• go-restful: package for building REST-style Web Services using Google Go.
• go-swagger: Swagger 2.0 implementation for go.
• goa: Design-based APIs and microservices in Go.
• swagger-codegen: Swagger Code Generator.

Background

Gitlab has update to v11.0, which does not support Gitlab API V3 any more. We want to support both Gitlab API V3 and V4 at the same time, but the Gitlab client go-gitlab does not support. So need to import both version of this library.

Reference

• Diamond dependency problem in go dep
• Go and Package Versioning
• How do I import a specific version of a package using go get?
  

Architecture

Implementation

• Load balancer for apiserver
  • Nginx:
  • HAproxy:
  • LVS: High performance in large scale cluster.
• Controller manager & scheduler: Stateful application, only one instance is running at once, the others are standby.
  • leader election options: Default value of --leader-elect is false in v1.2.0, is true in v1.5.3+. There are some other options about leader election, such as --leader-elect-lease-duration, --leader-elect-renew-deadline, --leader-elect-retry-period.

References

• Building High-Availability Clusters
• Setting up High-Availability Kubernetes Masters on GCE
• Highly Available Kubernetes Clusters on GCE
• Kubernetes 1.6.0 高可靠集群部署
• Kubernetes Master High Availability 高级实践

Softwares

• Chrome
• Chrome Extension: Markdown Viewer

Usage

Local Files

1. Navigate to chrome://extensions
2. Make sure that the Allow access to file URLs checkbox is checked for the Markdown Viewer extension
3. Open the local file through the file address like: file:///Users/robin/gocode/src/github.com/supereagle/goline/doc/api.md

Remote Files

1. Click on the Markdown Viewer icon and select Options
2. Add the origin that you want enabled for the Markdown Viewer extension

Reference

• Markdown Viewer / Chrome Extension

Choices

• go-dockerclient
• Official client (Used by Openshift S2I)

Both of them implement the Docker Engine API. But go-dockerclient is much EASIER to use as it composes the response of Docker engine APIs.

Take container logs as an example

• go-dockerclient

type LogsOptions struct {
	Context           context.Context
	Container         string        `qs:"-"`
	OutputStream      io.Writer     `qs:"-"`
	ErrorStream       io.Writer     `qs:"-"`
	InactivityTimeout time.Duration `qs:"-"`
	Tail              string

	Since      int64
	Follow     bool
	Stdout     bool
	Stderr     bool
	Timestamps bool

	// Use raw terminal? Usually true when the container contains a TTY.
	RawTerminal bool `qs:"-"`
}

func (c *Client) Logs(opts LogsOptions) error {
	if opts.Container == "" {
		return &NoSuchContainer{ID: opts.Container}
	}
	if opts.Tail == "" {
		opts.Tail = "all"
	}
	path := "/containers/" + opts.Container + "/logs?" + queryString(opts)
	return c.stream("GET", path, streamOptions{
		setRawTerminal:    opts.RawTerminal,
		stdout:            opts.OutputStream,
		stderr:            opts.ErrorStream,
		inactivityTimeout: opts.InactivityTimeout,
		context:           opts.Context,
	})
}

• Official client
  https://github.com/moby/moby/blob/v17.05.0-ce/client/container_logs.go

There is an error when create pod:

Error from server: error when creating "nginx.yaml": Pod "nginx" is forbidden: no API token found for service account default/default, retry after the token is automatically created and added to the service account

nginx.yml for pod:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:latest
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "1"

Envs:

• Kubernetes: v1.5.2
• Docker: v1.12.6

The Common Network Interface (CNI) is used by Kubernetes for its network, and is also adopted by Contiv. Want to apply Contiv network in Kubernetes.

Reference

• Contiv Networking with Kubernetes

Harbor currently does not support a HA deployment. However, Harbor HA is necessary in most production system.

References:

• Harbor HA feature design proposal/discussion
• 用 Harbor 和 Kubernetes 构建高可用企业级镜像仓库

Deploy MongoDB v3.6.4 in k8s, its entrypoint does not work, so the init DB scripts in /docker-entrypoint-initdb.d/ does not run.

Error Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mgo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mgo
  template:
    metadata:
      labels:
        app: mgo
    spec:
      containers:
      - name: mgo
        command:
        - mongod
        - "--replSet"
        - rs0
        - "--bind_ip"
        - 0.0.0.0
        - "--smallfiles"
        - "--noprealloc"
        image: mongo:3.6.4
        env:
        - name: MONGO_INITDB_ROOT_USERNAME
          value: admin
        ports:
        - containerPort: 27017

Reason

K8s uses spec.template.spec.containers[0].command for the command run by the container, and use args for arguments passed to the command.

Reference

• https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/

Summary of key features in each Kubernetes release.

v1.5.0

v1.4.0

When Jenkins master has too much jobs, it will be very slow. In order to know what's wrong with Jenkins, we can enable JMX of Jenkins, and use Java VisualVM to monitor Jenkins threads. A deadlock is found out by this way.

1. Enable JMX of Jenkins
   Add the config of JMX in /etc/sysconfig/jenkins: JENKINS_JAVA_OPTIONS="-Djava.rmi.server.hostname=jenkins.master.com -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
2. Restart Jenkins master
3. Remotely monitor Jenkins threads by Java VisualVM

References

• Monitor Jenkins Java Threads Usage
• 使用 VisualVM 进行性能分析及调优

The advantages of Jenkins is its rich plugins, however it will cost much effort to maintain these plugins. What CI/CD PaaS needs is more than jenkins.

References

• Hyperloop，让发布简洁高效
• 打造云上代码交付链，CodePipeline实践分享
• 基于Docker的CI/CD流水线实践

在家里访问Github网页以及git clone Github项目，速度非常慢，经常网页打不开，代码clone不下来。家里网络是20M联通独享宽带，浏览器使用翻墙代理，访问Google都非常快。ping域名github.com也能解析到正确的IP上，只是经常会丢包。

$ ping github.com

正在 Ping github.com [192.30.253.113] 具有 32 字节的数据:
来自 192.30.253.113 的回复: 字节=32 时间=286ms TTL=51
请求超时。
请求超时。
来自 192.30.253.113 的回复: 字节=32 时间=290ms TTL=51

192.30.253.113 的 Ping 统计信息:
    数据包: 已发送 = 4，已接收 = 2，丢失 = 2 (50% 丢失)，
往返行程的估计时间(以毫秒为单位):
    最短 = 286ms，最长 = 290ms，平均 = 288ms

On Docker host, we can run docker exec -it ${CONTAINER_ID/CONTAINER_NAME} ${COMMAND} to login and exec command in containers.

On Kubernetes master, we can run kubectl exec -it ${POD_NAME} ${COMMAND} to login and exec command in the container of the pod. If there are multiple containers of the pod, can run the command in specified container.

Both of these two ways are inconvenient and unsecured, as they need to login the Docker host or Kubernetes master, and have the permission to run some other dangerous commands.

A better way is to login and run command in container through websocket.

Reference

• LAIN 平台远程进入容器功能设计与实现
• GoTTY: Share your terminal as a web application
• GoTTY Client: Terminal client for GoTTY
• 使用 Tornado 实现 Docker 容器的 exec

Category

• Object中的fields必须是exported，才能被JSON/YAML Marshal/Unmarshal

EBooks

• Build Web Application With Golang
• Go Programming Blueprints

References

• Going Go Programming
• Training for Golang
• Go开发者
• Working with Go
• 50 Shades of Go: Traps, Gotchas, and Common Mistakes for New Golang Devs

Problem

The directory is added, but the content of the directory is not added. Any changes of the content in the directory will not be listed by git status, and these changes will not be added by git add.

Solution

 git rm --cached directory
 git add directory

Reference

• Git: fatal: Pathspec is in submodule

Download and install KDiff3.

Installation of KDiff3

Installation on Mac

1. Download KDiff3
2. Install as app(drag and drop the kdiff3 into your Applications)

Configuration for merge tool of git

Configuration on Windows

git config --global --add merge.tool kdiff3
git config --global --add mergetool.kdiff3.path "C:/Program Files/KDiff3/kdiff3.exe"
git config --global --add mergetool.kdiff3.trustExitCode false

git config --global --add diff.guitool kdiff3
git config --global --add difftool.kdiff3.path "C:/Program Files/KDiff3/kdiff3.exe"
git config --global --add difftool.kdiff3.trustExitCode false

Configuration on Mac

git config --global --add merge.tool kdiff3
git config --global mergetool.kdiff3.cmd '/Applications/kdiff3.app/Contents/MacOS/kdiff3 $BASE $LOCAL $REMOTE -o $MERGED'

Background

Unix agents are launched via SSH, they can completely controlled by Jenkins master. If config their availability as Keep this agent online as much as possible, they can automatically connect to Jenkins master after restart.
But it is different for Windows agents, they are usually launched via java web start through JNLP protocol. This requires the Windows agents to actively open a JNLP file to establish a TCP connection to the Jenkins master. After Windows agents restart, they are offline and needed to be manually launched via java web start, which is not automatical enough.

Solution

Create the automatical connection script agent.bat in startup folder through Win -> All Programs -> Startup, this folder is always like C:\Users\robin.yue\AppData\Roaming\Microsoft\Windows\Start Menu\Programs.
The content of the script agent.bat:

start /b java -jar D:\jenkins\slave.jar -jnlpUrl http://jenkins-master.vclound.com:8080/computer/test-slave/slave-agent.jnlp -secret 0439590ddfb022a84fde1a1ceb55953300db165ce0fb13ec6b5de908e365cc3b

There are two ways to launch Windows agents:

• Launch agent from browser
• Run from agent command line

The script agent.bat uses the second way, and is added into the startup to automatically run this command after restart.

Category

• Container Engines
• Container Orchestration
• Container Images
• CI
• Messages
• Monitor
• Storage
• Log
• Languages
• Documentation
• Others

Container Engine

• Docker: An application container engine(v1.6.0 -> v1.9.2 -> v1.10.3 -> v1.13.2)

Container Orchestration

• Kubernetes: Production-Grade Container Scheduling and Management(v1.1.4 -> v1.2.4 -> v1.5.3)
• Mesos + Marathon
• Swarm: A Docker-native clustering system

Container Images

• Docker Registry: The registry to pack, ship, store, and deliver image
• Harbor: An enterprise-class container registry server
• Clair: An open source project for the static analysis of vulnerabilities in appc and docker containers
• CIS Docker 1.13.0 Benchmark: Provides prescriptive guidance for establishing a secure configuration posture for Docker container

CI

• Jenkins: Most popular automation tool for your development workflow(v1.599 -> v2.7.2)
• Gitlab CI

Messages

• NATS: The cloud native messaging system
• Kafka: A distributed streaming platform

Monitor

• Prometheus: A systems and service monitoring system
• Zabbix: The ultimate enterprise-level software designed for real-time monitoring of millions of metrics collected from tens of thousands of servers, virtual machines and network devices

Storage

• ETCD: Distributed reliable key-value store
  • etcdtool
• MongoDB
  • Studio 3T(Renamed from MongoChef)
• MySQL
  • Navicat for MySQL

Log

Languages

• Golang: v1.4.2 -> v1.5.2 -> v1.7.4
  • Go Tools: Various tools to help you happy hacking!
  • Gogland: Commercial IDE by JetBrains aimed at providing an ergonomic environment for Go development
  • LiteIDE: A simple, open source, cross-platform Go IDE
  • Godep: Dependency tool for go
  • Ginkgo: A BDD Testing Framework for Go
  • Gomega: A matcher/assertion library preferred for Ginkgo
  • go-torch: Stochastic flame graph profiler for Go programs
  • gofuzz: A library for populating go objects with random values
• Java
• Shell

Documentation

• Swagger: A framework to design, build, document, and consume RESTful APIs
  • Go Swagger: Swagger 2.0 implementation for go
• Markdown

Others

• ProcessOn: A on-line tool to design and share graph
• Xshell: A powerful terminal emulator that supports SSH, SFTP, TELNET, RLOGIN and SERIAL
• Postman:
• Font Awesome: The iconic font and CSS toolkit
• Balsamiq Mockups: A powerful software to fastly drawn your prototype
• Lucidchart: A diagram maker
• Xmind: The Most Popular Mind Mapping Tool on the Planet
• Apowersoft: A powerful soft for screen and audio recording
• Codepile: Easily Share Piles of Code
• Axure RP: Design Tool for Rapid Prototyping
• asciiflow: Infinite ASCII diagrams, save to Google Drive, resize, freeform draw, and export straight to text/html
• LICEcap: simple animated screen captures
• Short URL: Free tool to shorten a URL or reduce a link making it easy to remember. (Google URL Shortener has been turning down)
• Draw:

Knowledge Points

Reference

• SpringBoot Learning

In a platform constructed by microservices, services communicate through REST APIs. If use Swagger to describe the REST APIs for each service, there will be too many Swagger documentations. It is better to manage and display them in one Swagger UI.

Reference

Question: Best way to aggregate multiple Swagger endpoints in a single HTML page
Load multiple resources to swagger-ui - work by @ponelat