caicloud / cyclone Goto Github PK

Powerful workflow engine and end-to-end pipeline solutions implemented with native Kubernetes resources. https://cyclone.dev

License: Apache License 2.0

Go 66.51% Shell 2.57% Makefile 0.84% Dockerfile 0.40% JavaScript 28.56% HTML 0.13% Less 0.80% Mustache 0.19%

continuous-integration continuous-delivery ci-cd kubernetes workflow dag ai-pipeline

cyclone's People

Contributors

Stargazers

Watchers

Forkers

superxi911-zz gaocegege perhapszzy ddysher nikawang markthink gitter-badger cmingxu qiuxiaolian gyliu513 no2key yyljlyy tommywu23 haozhixin keontang scorpiocph gooops chengcanmm77 learn-docker axcoto supereagle xiaomuqiao xi911 huzhengchuan zofuthan ninefive louisqiao my3157 chengly54 zjj2wry loky555 dockernuts gophersgang timopek caitong93 ive4 jiangzhonghui itsky365 oneoaas-golib icehofman lubaoyilang nuaays khanchan yuejack bbbmj nanbo99 xieydd wotheai heliangcheng anchoret2678 quxiaofeng liangxugang koalacxr h3llowor1d aleishus prprs red7hj zhujian7 dfzhizi robbbit vizn charygao forging2012 mickylee ipaste wzqnls aryanugroho zjfqxa nustee liwandaniel yanghongkjxy lorgine-li kz33 akinswin th002 ihuangyaoshi jonessen chenrui2014 liuyihuicaicloud cgrigor caicloud-bot db12130 chenyu85 developers23 taozongwang sunjiakang clark1230 daimazai cd1989 sandysuehe tskdsb caosm ma2331550908 huxiangtao h0-lab skymysky woerwin awesomegolang yuanfeng0905 caicaihe

cyclone's Issues

Some mistakes in setup guide docs

There are some mistakes in setup docs:

scripts/build-image.sh not exist use scripts/build-server-image.sh instead
The name of clair container created by docker-compose is cyclone_clair_1 not clair_clair.(docker-compose version 1.9.0)

Use k8s Job to execute CI/CD workflow

Migrate images from cargo.caicloud.io/caicloud to cargo.caicloud.io/circle

We should use the circle, not caicloud to store images.

Add list response metadata and method

Successful collection GET operation MUST return metadata with 'total' and 'items' array, i.e.

{
"metadata": {
"total": 100,
"itemsLength": 10
},
"items": [ {}, {}, {} ]
}

Collection GET (A.K.A. List APIs)

SHOULD support pagination query: GET /api/v0/items?start=&queryName2=

start: (number, optional) - query start index, default 0.
limit: (number, optional) - specify the number of records, default +Inf, return all.

Support custom plugin

Let Cyclone to be a open source platform.

Now, Cyclone can only rolling update deployment for k8s. It need to support create new deployment. Firstly, we can read deployment config from files (maybe k8s config map) to create new deployment. Secondly, we need to config new deployment from web.

Add pipeline record REST APIs

Need to provide the REST APIs to operate the pipeline record.

change the logic of operation during the integration stage

As we discussed on 11/15th, we need to modify the operation durning the integration stage. Cyclone uses the image which waits to be published to run a container, and execute integration test in this container. Now, Cyclone needs to run this container as a service, and execute integration test in another container whose image specialed in the integration section of yaml file.

Delete "Output" field in event struct and modify worker code

Describe：
Log file operations should only exist in the worker part, so the "Output" field does not need to exist in the event struct , and modify the worker code.

Add pipeline data structure and CRUD

Pipeline is used to describe the config and workflow of CI/CD.

Send time for every workflow steps

Cyclone need to count sending time for every workflow steps.

[Bug report] Online API document is unavailable

What happened

Online API document is unavailable

What you expected to happen

Online API document is available

Reproduce steps (as minimally and precisely as possible)

Visit online API document in this page

caicloud-yml-reference_zh-CN.md 中的 deploy 单词写错了

https://github.com/caicloud/cyclone/blob/master/docs/caicloud-yml-reference_zh-CN.md#depoly

fixes API doc

Our API doc uses Swagger UI with go-restful, and now it needs to input the http://<host>:7099/apidocs.json into the input box, we should optimize the API doc by removing the input step 😉

Error while the quota is not enough

When I run e2e-k8s.sh, the command-line interface shows me that "All system worker are busy, wait for 10 seconds" and then it gets failed.

Use project to manage the CICD workflows

The project is used to manager a group of related CICD workflows, just like Jenkins folder to manage pipelines. It can be a real project, which has several or many applications or micro services.

Todo Lists:

Support project CRUD
Support project REST API

sync documentation with code about network config.

Now Cyclone's service discovery in integration step has changed, we should rewrite related sections in the documentation.

ref #74 0e1839a

Send heart beat fail after cyclone server is restarted

Describe:
When i restarted the cyclone server, the worker can't send heart beat and log info to server through websocket.

rebuild & build special commit

Cyclone need to rebuild or build special commit .

Create caicloud.yml from web

Let Cyclone more simple.

Workflow

We need to redefine out union build function. Not only union CD, but also union CI.

remove human work when update godep

Now cyclone's godep has some mannual work, it is not encouraged. So we should update godep and make sure that there is no human intervention when update the denpendency.

function doc

We need to add function doc to introduce what Cyclone can do.

Add project REST APIs

Need to provide the REST APIs to operate the project.

add "HOWTO test the code(unit test and e2e test)"

We need a document for anyone wanting to either write code which directly accesses the Cyclone API, or to contribute directly to the Cyclone project. It should include:

API doc(Just move the API doc to this document)
The process of developing and contributing code to the Cyclone project. (Same as https://github.com/kubernetes/kubernetes/blob/master/docs/devel/README.md)
HOWTO setup dev environment
HOWTO test the code(unit test and e2e test)

cache docker images in worker container.

Now we pull the images every time we run the integration step. Maybe we could mount a volume to /var/lib/docker, or something else.

Let CI/CD workflow be more free

Now, the CI/CD workflow is prebuild -> build -> integration -> post build -> deploy. It's Inflexible. We need to let user can compose the workflow freely.

cyclone integrates cyclone

We could use cyclone to test and deploy cyclone, just like other CI tools.

support unit test cases
support e2e test cases

e2e test error when using golang 1.7

ref https://travis-ci.org/caicloud/cyclone/builds/176273667

It seems that there is an error raised by circle-worker:

json: cannot unmarshal string into Go value of type api.VersionLogCreateResponse

------------------------------
• Failure [14.006 seconds]
Yaml
/home/travis/gopath/src/github.com/caicloud/cyclone/tests/yaml/yaml_test.go:270
  with right version information
  /home/travis/gopath/src/github.com/caicloud/cyclone/tests/yaml/yaml_test.go:269
    with the integration operation
    /home/travis/gopath/src/github.com/caicloud/cyclone/tests/yaml/yaml_test.go:219
      should be able to get version via HTTP GET method. [It]
      /home/travis/gopath/src/github.com/caicloud/cyclone/tests/yaml/yaml_test.go:213
      Expected
          <api.VersionStatus>: failed
      to equal
          <api.VersionStatus>: healthy
      /home/travis/gopath/src/github.com/caicloud/cyclone/tests/yaml/yaml_test.go:212
------------------------------

add more unit test cases

Now code coverage of cyclone is a little low 😕 , so we need more test cases.

[FEATURE REQUES] Add issue template & pull request template

Environment

[Include version, operating system name, and other relevant details]
xxx

What happened

What you expected to happen

Reproduce steps (as minimally and precisely as possible)

[As minimally and precisely as possible]
xxx

Anything else

Web UI for Cyclone

Add pipeline record data structure and CRUD

Pipeline record is the running record of the pipeline.

worker node docker version verification

Now we check the docker version in worker node to make sure that the version is same as WORKER_NODE_DOCKER_VERSION defined in circle server.

I think we should research the changelog of docker and reduce the claim about docker version :) Maybe above 1.10 is OK.

add test coverage analysis

Now cyclone has some unit test cases but we don't have a tool to get the test coverage analysis.

There are some tools, such as gocover.io and goveralls.

gocover.io is ease of use but doesn't support multi-package test now(ref https://github.com/vieux/gocover.io/issues/15), so maybe we could use goveralls instead.

etcd start failed

ERROR: for cyclone_etcd_1 Cannot start service etcd: driver failed programming external connectivity on endpoint cyclone_etcd_1 (24b767714249941982970cfd2e43141ff31cdcdef92b44c73545fc4c6120a8f9): Error starting userland proxy: listen tcp 0.0.0.0:2380: bind: address already in use

ERROR: for etcd Cannot start service etcd: driver failed programming external connectivity on endpoint cyclone_etcd_1 (24b767714249941982970cfd2e43141ff31cdcdef92b44c73545fc4c6120a8f9): Error starting userland proxy: listen tcp 0.0.0.0:2380: bind: address already in use
ERROR: Encountered errors while bringing up the project

Convert pipeline config to service

Cyclone only supports to running service, as a workaround to run pipeline, need to convert it to service.

add badge support

We should support badge in cyclone, just like travis.

not websocket protocol

Environment

[Include version, operating system name, and other relevant details]
OS：mac
Docker：
Client:

Version:      17.03.1-ce
API version:  1.27
Go version:   go1.7.5
Git commit:   c6d412e
Built:        Tue Mar 28 00:40:02 2017
OS/Arch:      darwin/amd64

Server:
Version:      17.03.1-ce
API version:  1.27 (minimum version 1.12)
Go version:   go1.7.5
Git commit:   c6d412e
Built:        Fri Mar 24 00:00:50 2017
OS/Arch:      linux/amd64
Experimental: true

What happened

部署后，访问localhost:8000
报错：not websocket protocol

Add pipeline REST APIs

Need to provide the REST APIs to operate the pipeline.

lost log in mongo

Describe:
The worker container was shut down because of timeout or be cancelled, this cause that mongo do not have the log about CI/CD.

Solution：
Server after the closure of the worker container put the log in kafka into the mongo

update document about docker image

Now the docker image is not found at cargo.caicloud.io, we should fix it immediately.

Due: 11.9

./scripts/build-image.sh failed

./scripts/build-image.sh
+++++ Start building cyclone server
can't load package: package github.com/caicloud/cyclone: open /go/src/github.com/caicloud/cyclone: permission denied

allow multi cyclone config files in one repo

Now we only support config file named "caicloud.yml", we need to support customized file name.

The job has done, waiting for a PR.

add scripts about node regsitration

Now there is no script to register nodes easily, there is a scripts/tools/api.py but it is complicated, we need to refactor it, or rewrite it in bash.

[BUG REPORT] Test template

Environment

[Include version, operating system name, and other relevant details]
xxx

What happened

What you expected to happen

Reproduce steps (as minimally and precisely as possible)

[As minimally and precisely as possible]
xxx

Anything else

Code optimization by go report

optimize code by goreport

unknown error caused by websocket package

INFO[0182] watch unfinshed events set: "{\"event_id\":\"e8986325-ec7e-440b-873c-d4f1329e2d69\",\"operation\":\"create-version\",\"service\":{\"_id\":\"230773ca-5515-4e69-9975-8ed157defd65\",\"user_id\":\"fake-user-id\",\"name\":\"test-service\",\"username\":\"gaocegege\",\"description\":\"This is a test-service\",\"version_fails\":[\"36526ffd-d07f-4f18-bf3d-d79ef09e929b\"],\"repository\":{\"url\":\"https://github.com/gaocegege/cyclone\",\"vcs\":\"git\",\"status\":\"healthy\"},\"jconfig\":{},\"profile\":{\"profiles\":[{\"email\":\"[email protected]\",\"cellphone\":\"useless\"}],\"setting\":\"sendwhenfinished\"},\"last_createtime\":\"2016-11-15T10:04:50.449+08:00\",\"last_versionname\":\"v0.1.3\"},\"version\":{\"_id\":\"e8986325-ec7e-440b-873c-d4f1329e2d69\",\"service_id\":\"230773ca-5515-4e69-9975-8ed157defd65\",\"name\":\"v0.1.0\",\"description\":\"just for test\",\"create_time\":\"2016-11-15T10:05:47.711198846+08:00\",\"url\":\"https://github.com/gaocegege/cyclone\",\"status\":\"pending\",\"yaml_deploy_status\":\"norun\",\"operation\":\"integration\",\"operator\":\"api\",\"build_resource\":{\"memory\":5.36870912e+08,\"cpu\":512}},\"project\":{\"create_time\":\"0001-01-01T00:00:00Z\"},\"project_version\":{\"create_time\":\"0001-01-01T00:00:00Z\"},\"data\":{\"Token\":\"\",\"service-name\":\"test-service\",\"username\":\"gaocegege\",\"version-name\":\"v0.1.0\"},\"worker_info\":{\"docker_host\":\"0.0.0.0:2375\",\"container_id\":\"46f49a6d4d92267413637e3a4fd2b90201a5132f960e3c237c887b03da976935\",\"due_time\":\"2016-11-15T12:05:49.293378316+08:00\",\"used_resource\":{}},\"status\":\"running\"}"
  _file_=manager.go:103
INFO[0192] check token: e8986325-ec7e-440b-873c-d4f1329e2d69  _file_=event.go:169
INFO[0192] receive heart_beat packet                     _file_=application.go:113
ERRO[0192] packet data unmarshal recover: %sruntime error: invalid memory address or nil pointer dereference  _file_=application.go:56
ERRO[0192] set tcp [::1]:8000: use of closed network connection  _file_=server.go:56
INFO[0231] check token: e8986325-ec7e-440b-873c-d4f1329e2d69  _file_=event.go:169
INFO[0231] set etcd: {"event_id":"e8986325-ec7e-440b-873c-d4f1329e2d69","operation":"create-version","service":{"_id":"230773ca-5515-4e69-9975-8ed157defd65","user_id":"fake-user-id","name":"test-service","username":"gaocegege","description":"This is a test-service","version_fails":["36526ffd-d07f-4f18-bf3d-d79ef09e929b"],"repository":{"url":"https://github.com/gaocegege/cyclone","vcs":"git","status":"healthy"},"jconfig":{},"profile":{"profiles":[{"email":"[email protected]","cellphone":"useless"}],"setting":"sendwhenfinished"},"last_createtime":"2016-11-15T10:04:50.449+08:00","last_versionname":"v0.1.3"},"version":{"_id":"e8986325-ec7e-440b-873c-d4f1329e2d69","service_id":"230773ca-5515-4e69-9975-8ed157defd65","name":"v0.1.0","description":"just for test","commit":"a179d6c8446e32cd43165e426204b2005d96cefa","create_time":"2016-11-15T10:05:47.711198846+08:00","url":"https://github.com/gaocegege/cyclone","status":"pending","yaml_deploy_status":"norun","operation":"integration","operator":"api","build_resource":{"memory":5.36870912e+08,"cpu":512}},"project":{"create_time":"0001-01-01T00:00:00Z"},"project_version":{"create_time":"0001-01-01T00:00:00Z"},"data":{"Token":"","service-name":"test-service","username":"gaocegege","version-name":"v0.1.0"},"worker_info":{"docker_host":"0.0.0.0:2375","container_id":"46f49a6d4d92267413637e3a4fd2b90201a5132f960e3c237c887b03da976935","due_time":"2016-11-15T12:05:49.293378316+08:00","used_resource":{}},"status":"fail","error_msg":"container meets error"}  _file_=event.go:152
INFO[0231] watch unfinshed events set: "{\"event_id\":\"e8986325-ec7e-440b-873c-d4f1329e2d69\",\"operation\":\"create-version\",\"service\":{\"_id\":\"230773ca-5515-4e69-9975-8ed157defd65\",\"user_id\":\"fake-user-id\",\"name\":\"test-service\",\"username\":\"gaocegege\",\"description\":\"This is a test-service\",\"version_fails\":[\"36526ffd-d07f-4f18-bf3d-d79ef09e929b\"],\"repository\":{\"url\":\"https://github.com/gaocegege/cyclone\",\"vcs\":\"git\",\"status\":\"healthy\"},\"jconfig\":{},\"profile\":{\"profiles\":[{\"email\":\"[email protected]\",\"cellphone\":\"useless\"}],\"setting\":\"sendwhenfinished\"},\"last_createtime\":\"2016-11-15T10:04:50.449+08:00\",\"last_versionname\":\"v0.1.3\"},\"version\":{\"_id\":\"e8986325-ec7e-440b-873c-d4f1329e2d69\",\"service_id\":\"230773ca-5515-4e69-9975-8ed157defd65\",\"name\":\"v0.1.0\",\"description\":\"just for test\",\"commit\":\"a179d6c8446e32cd43165e426204b2005d96cefa\",\"create_time\":\"2016-11-15T10:05:47.711198846+08:00\",\"url\":\"https://github.com/gaocegege/cyclone\",\"status\":\"pending\",\"yaml_deploy_status\":\"norun\",\"operation\":\"integration\",\"operator\":\"api\",\"build_resource\":{\"memory\":5.36870912e+08,\"cpu\":512}},\"project\":{\"create_time\":\"0001-01-01T00:00:00Z\"},\"project_version\":{\"create_time\":\"0001-01-01T00:00:00Z\"},\"data\":{\"Token\":\"\",\"service-name\":\"test-service\",\"username\":\"gaocegege\",\"version-name\":\"v0.1.0\"},\"worker_info\":{\"docker_host\":\"0.0.0.0:2375\",\"container_id\":\"46f49a6d4d92267413637e3a4fd2b90201a5132f960e3c237c887b03da976935\",\"due_time\":\"2016-11-15T12:05:49.293378316+08:00\",\"used_resource\":{}},\"status\":\"fail\",\"error_msg\":\"container meets error\"}"
  _file_=manager.go:103
INFO[0231] Afer release, the userid fake-user-id's left resource  memory 4294967296.000000 cpu  4096.000000  _file_=resource.go:146
INFO[0231] create version post hook                      _file_=event.go:167
ERRO[0231] Unable to get remote according coderepository  _file_=event.go:193 user_id=fake-user-id
INFO[0231] Env variant SUCCESSTEMPLATE found, using env value: ./notify/provider/success.html  _file_=osutil.go:44
INFO[0231] Env variant ERRORTEMPLATE found, using env value: ./notify/provider/error.html  _file_=osutil.go:44
INFO[0237] Env variant SUCCESSTEMPLATE found, using env value: ./notify/provider/success.html  _file_=osutil.go:44
INFO[0237] Env variant ERRORTEMPLATE found, using env value: ./notify/provider/error.html  _file_=osutil.go:44
INFO[0243] watch finshed events delete: "{Key: /events/unfinished/e8986325-ec7e-440b-873c-d4f1329e2d69, CreatedIndex: 18, ModifiedIndex: 18, TTL: 0}"
  _file_=manager.go:122

I'm focusing on #21 , and meets this error. I don't know the cause now. But it doesn't impact the CI and CD step, just about log.

FYI

Clean the code relating about the old `project`

project was used as a union building in last year. Now, we use this concept to group pipelines. We need to clean the code relating about the old project

caicloud / cyclone Goto Github PK

cyclone's People

Contributors

Stargazers

Watchers

Forkers

cyclone's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs