supereagle / experiences Goto Github PK
View Code? Open in Web Editor NEWSummary of practical experience in work.
Summary of practical experience in work.
PTAL: Please Take A Look
LGTM: Look Good To Me
SGTM: Sound Good To Me
iff: if and only if
WIP: Work In Progress
AFAIK: As Far As I Know
FYI: For Your Information
IIRC: If I Remember Correctly
# ssh [email protected]
shell request failed on channel 0
# tail -f 10 /var/log/secure
......
Feb 4 09:55:06 10.10.10.1 sshd[6137]: Accepted password for apps from 10.10.10.2 port 33216 ssh2
Feb 4 09:55:06 10.10.10.1 sshd[6137]: pam_unix(sshd:session): session opened for user apps by (uid=0)
Feb 4 09:55:06 10.10.10.1 sshd[6627]: error: do_exec_pty: fork: Resource temporarily unavailable
Feb 4 09:55:06 10.10.10.1 sshd[6137]: pam_unix(sshd:session): session closed for user apps
su - xxx
, but it is OK for other user# su - xxx
Last login: Tue Apr 26 09:48:44 CST 2016 from application03 on pts/0
su: failed to execute /bin/bash: Resource temporarily unavailable
The root cause is that the resource for the user is insufficient as there are too many processes created by this user. When the the user login through su - xxx
or SSH, a new process will be created by the user. If the new process can not be created as the user's resource is insufficient, the login will be failed.
Check the process of the user xxx
:
$ ps -ef | grep ^xxx|wc -l
1246
[[email protected] ~]# vi /etc/security/limits.d/20-nproc.conf
[[email protected] ~]# cat /etc/security/limits.d/20-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
* soft nproc 65535 (#Increase the limit from 1024 to 65535)
root soft nproc unlimited
Fail to run Klar in Docker according to doc Dockerized version.
Error log:
# docker run -it --rm klar
panic: standard_init_linux.go:178: exec user process caused "no such file or directory" [recovered]
panic: standard_init_linux.go:178: exec user process caused "no such file or directory"
goroutine 1 [running, locked to thread]:
panic(0x6f2340, 0xc420138540)
/usr/lib/golang/src/runtime/panic.go:500 +0x1a1
github.com/urfave/cli.HandleAction.func1(0xc420097748)
/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x247
panic(0x6f2340, 0xc420138540)
/usr/lib/golang/src/runtime/panic.go:458 +0x243
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc420097198, 0xc4200260c8, 0xc420097238)
/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x18f
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc420057590, 0xaab9c0, 0xc420138540)
/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353
main.glob..func8(0xc42009aa00, 0x0, 0x0)
/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/main_unix.go:26 +0x66
reflect.Value.call(0x6dd0c0, 0x768ec8, 0x13, 0x73b509, 0x4, 0xc420097708, 0x1, 0x1, 0x4d17a8, 0x731360, ...)
/usr/lib/golang/src/reflect/value.go:434 +0x5c8
reflect.Value.Call(0x6dd0c0, 0x768ec8, 0x13, 0xc420097708, 0x1, 0x1, 0xac1700, 0xc4200976e8, 0x4da786)
/usr/lib/golang/src/reflect/value.go:302 +0xa4
github.com/urfave/cli.HandleAction(0x6dd0c0, 0x768ec8, 0xc42009aa00, 0x0, 0x0)
/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x1e0
github.com/urfave/cli.Command.Run(0x73b6d5, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74cc8a, 0x51, 0x0, ...)
/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xc3b
github.com/urfave/cli.(*App).Run(0xc4200bc000, 0xc42000c120, 0x2, 0x2, 0x0, 0x0)
/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x611
main.main()
/builddir/build/BUILD/docker-1398f249013601ab999d286910664d70fd1329a2/runc-f5721697226d42d9efeea37fa3a7eb1d208fa1a2/main.go:137 +0xbd6
Docker Version:
# docker version
Client:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-28.git1398f24.el7.centos.x86_64
Go version: go1.7.4
Git commit: 1398f24/1.12.6
Built: Fri May 26 17:28:18 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-28.git1398f24.el7.centos.x86_64
Go version: go1.7.4
Git commit: 1398f24/1.12.6
Built: Fri May 26 17:28:18 2017
OS/Arch: linux/amd64
Investigation:
Replace base image scratch
with centos:7
, there will be no problem.
ENTRYPOINT [ "echo", "$HOME" ]
will not do variable substitution on $HOME.
exec form does not invoke a command shell. There are 2 solutions:
ENTRYPOINT echo $HOME
ENTRYPOINT [ "sh", "-c", "echo $HOME" ]
Error log
error writing event (write tcp kubernetes.master.com:54753: connection reset by peer)
got unexpected response error (etcdserver: request timed out)
Version must match the regular expression: ^\s*v?([0-9]+(?:\.[0-9]+)*)(.*)*$
, which has 2 parts: ${numbers}-${extra}.
Two types of version spec:
^(?:-([0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*))?(?:\+([0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*))?\s*$
There is not in-memory MongoDB for test. Mgo which is the MongoDB driver for Go, also has to run its unit test on a local MongoDB server.
Configuration for log driver : --log-driver --log-opt json-file max-size=2m --log-opt max-file=5
The rotated json log files are deleted by Docker, but the fd
of these files are still kept by Docker, so their disk size can not be released otherwise restart the Docker.
The number of kept fd
is very large, if it larger then the config --default-ulimit nofile=131072
of Docker, the Docker can not work normally.
# ls -li "/proc/`cat /var/run/docker.pid`/fd" | grep '\(deleted\)' | wc -l
11405
# ls -li "/proc/`cat /var/run/docker.pid`/fd" | grep '\(deleted\)' | grep 48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931 | wc -l
4882
# ls -li "/proc/`cat /var/run/docker.pid`/fd" | grep '\(deleted\)' | grep 48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931 | head -n 3
1250037181 lr-x------ 1 root root 64 Oct 21 07:00 10000 -> /var/lib/docker/containers/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931-json.log.4 (deleted)
1250037182 lr-x------ 1 root root 64 Oct 21 07:00 10001 -> /var/lib/docker/containers/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931-json.log.4 (deleted)
1250037183 lr-x------ 1 root root 64 Oct 21 07:00 10002 -> /var/lib/docker/containers/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931/48d47b3988780a08ffc8a6dd1bd7252ff2e2fbed31092e1014eacc5f9569e931-json.log.4 (deleted)
Illustration : Total number of kept fd
is 11405, number of kept fd
for one container is 4882.
When there are about 1K images on the Docker host, I found two problems:
docker push
command# docker rm -f c44ec3f542f0
Failed to remove container (c44ec3f542f0): Error response from daemon: Could not kill running container, cannot remove - [2] Container does not exist: container destroyed
Summary of key features in each Docker release.
overlay2
or overlay
(if the kernel supports) from devicemapper
--update-max-failure-ratio
, --update-monitor
and --rollback
flags to docker service updatedocker plugin
& docker stack
docker stack deploy
from a Compose file PR 27998docker system
command with df
and prune
subcommands for system resource managementdocker image
and docker container
commands)Want to migrate the Docker Registry from the old one(R1) to the new one(R2). R1 already has many old images, R2 is empty but will be the only registry after R1 deprecated in the future.
All new images will be pushed into R2. When pull images, directly pull them from R2 if found in R2, otherwise pull from R1.
There is a Nginx(Harbor) in front of them, and distributes the image requests to Docker Registry. The requests are firstly directed to R2. If the image is not found in R2, there will be an 404 error, then Nginx needs to redirect this request to R1.
nginx.conf
worker_processes 1;
error_log logs/error.log;
pid E:/nginx-1.6.3/logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name localhost;
# Docker registry migration
error_page 404 =301 https://github.com$request_uri;
}
}
Test
The test url http://localhost/abcde
will be directed to https://github.com/abcde
. The $request_uri
in nginx.conf makes the request uri kept when directed.
公司内部可以ping通外部网站,但是浏览器上网必须使用代理才能。因此,这个代理只是针对http,https等部分协议做了限制,而对其他协议是没有限制的,例如:ssh。
$ ping github.com
正在 Ping github.com [192.30.255.112] 具有 32 字节的数据:
来自 192.30.255.112 的回复: 字节=32 时间=201ms TTL=49
来自 192.30.255.112 的回复: 字节=32 时间=200ms TTL=49
来自 192.30.255.112 的回复: 字节=32 时间=198ms TTL=49
来自 192.30.255.112 的回复: 字节=32 时间=197ms TTL=49
192.30.255.112 的 Ping 统计信息:
数据包: 已发送 = 4,已接收 = 4,丢失 = 0 (0% 丢失),
往返行程的估计时间(以毫秒为单位):
最短 = 197ms,最长 = 201ms,平均 = 199ms
# go get默认使用的git protocol是HTTPS
$ go get github.com/Microsoft/go-winio
# cd .; git clone https://github.com/Microsoft/go-winio E:\gocode\src\github.com\Microsoft\go-winio
Cloning into 'E:\gocode\src\github.com\Microsoft\go-winio'...
fatal: unable to access 'https://github.com/Microsoft/go-winio/': Failed to connect to github.com port 443: Timed out
package github.com/Microsoft/go-winio: exit status 128
# 手动使用SSH的git protocol可以正常git clone
$ git clone [email protected]:coreos/clair.git /e/gocode/src/github.com/coreos/clair
Cloning into 'E:/gocode/src/github.com/coreos/clair'...
remote: Counting objects: 6647, done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 6647 (delta 1), reused 2 (delta 1), pack-reused 6631
Receiving objects: 100% (6647/6647), 14.08 MiB | 377.00 KiB/s, done.
Resolving deltas: 100% (2725/2725), done.
Problem
Using Docker-in-Docker for your CI or testing environment? Think twice.
Solutions
Share volumes /var/run/docker.sock
and /etc/sysconfig/docker
between Docker Host and its containers.
If the program in the containers needs to use Docker, it can call the APIs of the Docker Clients, such as go-dockerclient.
If need to run Docker commands such as build, run, push or pull, the volume /usr/bin/docker
needs to be shared in addition.
When design REST APIs, two points should be considered:
The response from ALL APIs with the same format, but with different fields (empty will be omitted in some case) and data type/struct. So we need to correctly parse the data from the response. Here is an example implemented in Java.
# wget https://github.com/kubernetes/kubernetes/pull/44199.patch
https://github.com/kubernetes/kubernetes/pull/44199.patch will be redirected to https://patch-diff.githubusercontent.com/raw/kubernetes/kubernetes/pull/44199.patch
In Docker Daemon, there is a LayerUploadManager
and LayerDownloadManager
for pushing and pulling ALL layers. The push and pull speed are constricted by maxUploadConcurrency
and maxDownloadConcurrency
.
In Docker 1.10, the two limitation are NOT configurable as they are const in daemon.go :
const (
// maxDownloadConcurrency is the maximum number of downloads that
// may take place at a time for each pull.
maxDownloadConcurrency = 3
// maxUploadConcurrency is the maximum number of uploads that
// may take place at a time for each push.
maxUploadConcurrency = 5
)
In Docker 1.12, maxUploadConcurrency
and maxDownloadConcurrency
are configurable in docker daemon
command through this PR 22445.
origin
to your forked repo address in .git/config
You can use the empty image scratch
as the base image to build the minimal Docker image for your applications. scratch
is a special image with truly 0B
size.
But you should take care when use it for your Go applications, because the minimal application image maybe can not run. You can refer to optiopay/klar#51 for details.
We compiled our app, but it still is dynamically linked to the libraries it needs to run (i.e., all the C libraries it binds to). Unfortunately, scratch is empty, so there are no libraries and no loadpath for it to look in.
So there will be error when run the minimal application image:
standard_init_linux.go:178: exec user process caused "no such file or directory"
Statically compile our app with all libraries built in:
CGO_ENABLED=0 go build -a -installsuffix cgo -o main .
Requirements for log collection:
EVERY
podVariables | Metrics | V1.10.3(Baseline) | V1.13 |
Container number | Latency | 100% | 25% |
CPU | 100% | 105% | |
Operation intervals | Inspect latency | 100% | 80% |
List alive latency | 100% | 50% | |
List all latency | 100% | 30% | |
CPU | 100% | 500% | |
Goroutine number | Inspect latency | 100% | 50% |
List all latency | 100% | 10% | |
CPU | 100% | 300% |
Environment
Physical Machine: 8CPUs, 32GB memory
OS: CentOS Linux release 7.2.1511 (Core)
Storage Driver: Device Mapper
Benchmark
Docker Micro Benchmark: Docker performance benchmark from Google, used by Kubernetes to test the performance of its depended Docker.
Using benchmark.sh -c
Using benchmark.sh -i
ps
[all=true] Latency
ps
[all=false] Latency
inspect
Latency
CPU Usage
Using benchmark.sh -r
When the memory limit in k8s container spec is less than 4MB
, the pod will can not be created and will be always in ContainerCreating
.
You will easily meet this problem when use m
as the memory unit, because m
is just 1/1000 byte, which is too small. So m
can be used for memory, but it is not recommended.
You can refer to Docker Doc, or verify by the command:
# docker run -m 128 nginx:latest
/usr/bin/docker-current: Error response from daemon: Minimum memory limit allowed is 4MB.
I think this limitation is not set by Docker, but the requirement of Linux kernel.
When the memory limit is less than 4MB
, the created container will be immediately stopped by oom-killer.
Gitlab has update to v11.0, which does not support Gitlab API V3 any more. We want to support both Gitlab API V3 and V4 at the same time, but the Gitlab client go-gitlab does not support. So need to import both version of this library.
--leader-elect
is false
in v1.2.0, is true
in v1.5.3+. There are some other options about leader election, such as --leader-elect-lease-duration
, --leader-elect-renew-deadline
, --leader-elect-retry-period
.chrome://extensions
Allow access to file URLs
checkbox is checked for the Markdown Viewer extensionOptions
Both of them implement the Docker Engine API. But go-dockerclient is much EASIER to use as it composes the response of Docker engine APIs.
type LogsOptions struct {
Context context.Context
Container string `qs:"-"`
OutputStream io.Writer `qs:"-"`
ErrorStream io.Writer `qs:"-"`
InactivityTimeout time.Duration `qs:"-"`
Tail string
Since int64
Follow bool
Stdout bool
Stderr bool
Timestamps bool
// Use raw terminal? Usually true when the container contains a TTY.
RawTerminal bool `qs:"-"`
}
func (c *Client) Logs(opts LogsOptions) error {
if opts.Container == "" {
return &NoSuchContainer{ID: opts.Container}
}
if opts.Tail == "" {
opts.Tail = "all"
}
path := "/containers/" + opts.Container + "/logs?" + queryString(opts)
return c.stream("GET", path, streamOptions{
setRawTerminal: opts.RawTerminal,
stdout: opts.OutputStream,
stderr: opts.ErrorStream,
inactivityTimeout: opts.InactivityTimeout,
context: opts.Context,
})
}
There is an error when create pod:
Error from server: error when creating "nginx.yaml": Pod "nginx" is forbidden: no API token found for service account default/default, retry after the token is automatically created and added to the service account
nginx.yml for pod:
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx:latest
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "1"
Envs:
The Common Network Interface (CNI) is used by Kubernetes for its network, and is also adopted by Contiv. Want to apply Contiv network in Kubernetes.
Reference
Harbor currently does not support a HA deployment. However, Harbor HA is necessary in most production system.
References:
Deploy MongoDB v3.6.4 in k8s, its entrypoint does not work, so the init DB scripts in /docker-entrypoint-initdb.d/
does not run.
Error Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mgo
spec:
replicas: 1
selector:
matchLabels:
app: mgo
template:
metadata:
labels:
app: mgo
spec:
containers:
- name: mgo
command:
- mongod
- "--replSet"
- rs0
- "--bind_ip"
- 0.0.0.0
- "--smallfiles"
- "--noprealloc"
image: mongo:3.6.4
env:
- name: MONGO_INITDB_ROOT_USERNAME
value: admin
ports:
- containerPort: 27017
K8s uses spec.template.spec.containers[0].command
for the command run by the container, and use args
for arguments passed to the command.
When Jenkins master has too much jobs, it will be very slow. In order to know what's wrong with Jenkins, we can enable JMX of Jenkins, and use Java VisualVM to monitor Jenkins threads. A deadlock is found out by this way.
/etc/sysconfig/jenkins
: JENKINS_JAVA_OPTIONS="-Djava.rmi.server.hostname=jenkins.master.com -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"The advantages of Jenkins is its rich plugins, however it will cost much effort to maintain these plugins. What CI/CD PaaS needs is more than jenkins.
在家里访问Github网页以及git clone Github项目,速度非常慢,经常网页打不开,代码clone不下来。家里网络是20M联通独享宽带,浏览器使用翻墙代理,访问Google都非常快。ping域名github.com
也能解析到正确的IP上,只是经常会丢包。
$ ping github.com
正在 Ping github.com [192.30.253.113] 具有 32 字节的数据:
来自 192.30.253.113 的回复: 字节=32 时间=286ms TTL=51
请求超时。
请求超时。
来自 192.30.253.113 的回复: 字节=32 时间=290ms TTL=51
192.30.253.113 的 Ping 统计信息:
数据包: 已发送 = 4,已接收 = 2,丢失 = 2 (50% 丢失),
往返行程的估计时间(以毫秒为单位):
最短 = 286ms,最长 = 290ms,平均 = 288ms
On Docker host, we can run docker exec -it ${CONTAINER_ID/CONTAINER_NAME} ${COMMAND}
to login and exec command in containers.
On Kubernetes master, we can run kubectl exec -it ${POD_NAME} ${COMMAND}
to login and exec command in the container of the pod. If there are multiple containers of the pod, can run the command in specified container.
Both of these two ways are inconvenient and unsecured, as they need to login the Docker host or Kubernetes master, and have the permission to run some other dangerous commands.
A better way is to login and run command in container through websocket.
The directory is added, but the content of the directory is not added. Any changes of the content in the directory will not be listed by git status
, and these changes will not be added by git add
.
git rm --cached directory
git add directory
Download and install KDiff3.
git config --global --add merge.tool kdiff3
git config --global --add mergetool.kdiff3.path "C:/Program Files/KDiff3/kdiff3.exe"
git config --global --add mergetool.kdiff3.trustExitCode false
git config --global --add diff.guitool kdiff3
git config --global --add difftool.kdiff3.path "C:/Program Files/KDiff3/kdiff3.exe"
git config --global --add difftool.kdiff3.trustExitCode false
git config --global --add merge.tool kdiff3
git config --global mergetool.kdiff3.cmd '/Applications/kdiff3.app/Contents/MacOS/kdiff3 $BASE $LOCAL $REMOTE -o $MERGED'
Unix agents are launched via SSH, they can completely controlled by Jenkins master. If config their availability as Keep this agent online as much as possible
, they can automatically connect to Jenkins master after restart.
But it is different for Windows agents, they are usually launched via java web start through JNLP protocol. This requires the Windows agents to actively open a JNLP file to establish a TCP connection to the Jenkins master. After Windows agents restart, they are offline and needed to be manually launched via java web start, which is not automatical enough.
Create the automatical connection script agent.bat
in startup folder through Win -> All Programs -> Startup
, this folder is always like C:\Users\robin.yue\AppData\Roaming\Microsoft\Windows\Start Menu\Programs
.
The content of the script agent.bat
:
start /b java -jar D:\jenkins\slave.jar -jnlpUrl http://jenkins-master.vclound.com:8080/computer/test-slave/slave-agent.jnlp -secret 0439590ddfb022a84fde1a1ceb55953300db165ce0fb13ec6b5de908e365cc3b
There are two ways to launch Windows agents:
The script agent.bat
uses the second way, and is added into the startup to automatically run this command after restart.
In a platform constructed by microservices, services communicate through REST APIs. If use Swagger to describe the REST APIs for each service, there will be too many Swagger documentations. It is better to manage and display them in one Swagger UI.
Question: Best way to aggregate multiple Swagger endpoints in a single HTML page
Load multiple resources to swagger-ui - work by @ponelat
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.