alibaba / derrick

🐳A tool to help you containerize application in seconds

Derrick

Homepage: alibaba.github.io/derrick

Overview

Derrick is a tool to help you containerize application in seconds. Derrick focuses on developer workflow in local development environment. Derrick will inspect your workspace first, then generate definitions and templates to boost your journey to run apps as containerized services. This includes generating Dockerfile for your app, k8s definitions to deploy it, Helm Chart or Kustomize folders for multi-environment setup, Terraform templates to bootstrap your infrastructure resources, CI/CD pipelines to build/test/deploy the whole thing continously. You can use Derrick to set up your DevOps processes in cloud-native way.

Using Derrick is very simple:

1. derrick gen to automatically inspect the workspace and generate the Dockerfile for your application, or
2. derrick list to show all available riggings and pick one via derrick gen -r <rigging> to generate the Dockerfile.
3. Use your favorite text editor to modify the Dockerfile or other manifests before building and shipping the containers.
4. Integrate into your workflow and have fun.

Documentation

• Installation
• Getting started

Language Support

Here is the list of supported languages and frameworks. If you want to support more languages and frameworks, please submit an issue.

• Java
• PHP
• Python
• Golang
• NodeJS

License

This software is released under the Apache 2.0 license.

derrick's Issues

Installation fails due to conflicting idna version

Hi, users are unable to run derrick due to dependency conflict with idna package.

As shown in the following full dependency graph of derrick, derrick requires idna>=2.6，while requests==2.20.0 requires idna>=2.5,<2.8.

According to pip’s “first found wins” installation strategy, idna==2.8 is the actually installed version.
However, idna==2.8 does not satisfy idna>=2.5,<2.8.

Dependency tree---------

derrick-master
| +-chardet(version range:==3.0.4)
| +-docopt(version range:>=0.6.2)
| +-idna(version range:>=2.6)
| +-jinja2(version range:>=2.10.1)
| | +-MarkupSafe(version range:>=0.23)
| +-MarkupSafe(version range:>=1.0)
| +-nose(version range:>=1.3)
| +-pkginfo(version range:>=1.4.1)
| +-prompt-toolkit(version range:==1.0.15)
| | +-six(version range:>=1.9.0)
| | +-wcwidth(version range:*)
| +-pychalk(version range:==1.1.0)
| | +-six(version range:*)
| +-Pygments(version range:>=2.2.0)
| | +-coverage(version range:*)
| | +-nose(version range:*)
| | +-pyflakes(version range:*)
| | +-pylint(version range:*)
| | +-tox(version range:*)
| +-requests(version range:==2.20.0)
| | +-certifi(version range:>=2017.4.17)
| | +-chardet(version range:<3.1.0,>=3.0.2)
| | +-idna(version range:>=2.5,<2.8)
| | +-urllib3(version range:>=1.21.1,<1.25)
| +-requests-toolbelt(version range:==0.8.0)
| | +-requests(version range:>=2.0.1,<3.0.0)
| | | +-certifi(version range:>=2017.4.17)
| | | +-chardet(version range:<3.1.0,>=3.0.2)
| | | +-idna(version range:>=2.5,<2.8)
| | | +-urllib3(version range:>=1.21.1,<1.25)
| +-setuptools-git(version range:==1.2)
| +-simplejson(version range:>=3.11.1)
| +-six(version range:>=1.11.0)
| +-tqdm(version range:>=4.9.12)
| +-urllib3(version range:>=1.23)
| +-wcwidth(version range:>=0.1.7)
| +-whaaaaat(version range:>=0.5.2)
| | +-prompt-toolkit(version range:>=1.0.14)
| | | +-six(version range:>=1.9.0)
| | | +-wcwidth(version range:*)
| | +-pygments(version range:>=2.2.0)
| | | +-coverage(version range:*)
| | | +-nose(version range:*)
| | | +-pyflakes(version range:*)
| | | +-pylint(version range:*)
| | | +-tox(version range:*)
| | +-regex(version range:>=2016.11.21)

Thanks for your help.
Best,
Neolith

Laravel Dock Support

Add kubernetes resource yaml support

Add kubernetes resource yaml support

Release public custom Rigging

Release public custom Rigging to pip and complete docs

Add JVM flags best practices for containerization

Add jenkinsx workflow demo

FYI.

Optimize Java Image Build Cache

When building Java image, we should:

1. copy pom.xml and download dependencies first
2. copy the rest of the project and build jar output then

In this way we can optimize the caching of deps.

Additionally, we need to be careful that maven:3.5.0-jdk-8-alpine will treat /root/.m2 as a volume:

We should use -Dmaven.repo.local to specify maven local cache dir like below:

# First stage: complete build environment
FROM maven:3.5.0-jdk-8-alpine AS builder

# To resolve dependencies in a safe way (no re-download when the source code changes)
ADD ./pom.xml pom.xml
RUN  mvn install -Dmaven.repo.local=./.m2

ADD ./src src/
# package jar
RUN mvn -Dmaven.repo.local=./.m2 install -Dmaven.test.skip=true

在go语言上运行失败

• 本地go语言版本
  
• derrick 版本

0.1.2

运行命令

derrick init

运行出现问题
Failed to detect your application's platform.Maybe you can upgrade Derrick to get more platforms supported.

文档给的go语言示例程序也有这样的问题

Clean the .DS_Store file

-_-
Have a nice day!

Generate Deployment object

install error

in Mac 10.13.3,

under the default python, it can't install derrick, the error message is:

Found existing installation: six 1.4.1
DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 778, in install
requirement.uninstall(auto_confirm=True)
File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 754, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip/utils/init.py", line 267, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-aVZQJj-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

please help

REGISTRY_ENDPOINT and IMAGE_WITH_TAG could not setting separtely in Spring Cloud project.

1.derrick init
2.hint me "? What's your image repo"
3.input the repo(I wonder it means the registry), so it would be registry.x.y
4.in the derrick_conf, I found the ""image_with_tag": "registry.x.y"":(
5.on the other hand, if I input the image_with_tag, for example, helloworld
6.the REGISTRY_ENDPOINT in Jenkinsfile would be "helloworld":(

So I wonder REGISTRY_ENDPOINT and IMAGE_WITH_TAG should be setting separtely to fix the bug.

增加下这个环境变量配置 JAVA_TOOL_OPTIONS.

如果用户跑 derrick 的 环境里有这个 环境变量，也一起传递到生产配置中去

init complains chalk has no attribute 'format_cyan'

(derrick) demo $ derrick init
Detecting Java version ...
? What's your image repo demo
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/init.py", line 859, in emit
msg = self.format(record)
File "/Users/user/.virtualenvs/derrick/lib/python2.7/site-packages/derrick/core/logger.py", line 21, in format
return fmt.format(record)
File "/Users/user/.virtualenvs/derrick/lib/python2.7/site-packages/derrick/core/logger.py", line 34, in format
_chalk = get_chalk_color(level)
File "/Users/user/.virtualenvs/derrick/lib/python2.7/site-packages/derrick/core/logger.py", line 52, in get_chalk_color
_chalk = chalk.format_cyan
AttributeError: 'module' object has no attribute 'format_cyan'
Logged from file logger.py, line 79

构建 Kustomize 多环境目录

Provide best practice in Dockerfile to handle SIGTERM

Reference: https://hynek.me/articles/docker-signals/

在生成中可以选择是否接入我们ahas/mse/arms，如果需要，自动生成对应的annotation

https://help.aliyun.com/document_detail/125726.html?spm=5176.11065259.1996646101.searchclickresult.4a4d753eL7FQGy

https://help.aliyun.com/document_detail/193575.html?spm=a2c4g.11174283.6.911.f1632ceejRCWIj

https://help.aliyun.com/document_detail/184033.html?spm=a2c4g.11186623.6.1021.20ea253cSiqFyB

https://help.aliyun.com/document_detail/170446.html

https://cs.console.aliyun.com/?spm=5176.12818093.favorites.dcsk.4cae16d0UkiCwZ#/k8s/catalog/detail/incubator_ack-ahas-sentinel-pilot

https://cs.console.aliyun.com/?spm=5176.12818093.favorites.dcsk.4cae16d0UkiCwZ#/k8s/catalog/detail/incubator_ack-arms-pilot

https://cs.console.aliyun.com/?spm=5176.12818093.favorites.dcsk.4cae16d0UkiCwZ#/k8s/catalog/detail/incubator_ack-mse-pilot

Add cpu and memory request and limit

support helm charts

JenkinsX support

设置liveness和readiness

构建构建 kubernetes deployment manifests 目录

Use local .m2 dir to optimize `mvn install` caching

Add Rigging Management

Add Rigging Management (list,install,upgrade,uninstall)

Release Home page

Release Home page

php support php-fpm + nginx

The maven:3-jdk-8 image has the Error: Could not find or load main class org.apache.maven.surefire.booter.ForkedBooter

https://stackoverflow.com/questions/53010200/maven-surefire-could-not-find-forkedbooter-class

Add Rigging Automatic generation

Add Rigging Automatic generation

service and deployment has blank space in name which makes deployment fail

As title, generated service and deployment name will contain invalid characters when derrick project directory name has blank space. Suggested fix should be using normalized name instead.

兼容问题

该工具只有在prompt-toolkit==1.0.15环境才能使用，这和jupyter-console``ipython是冲突的。
强烈建议保持对最新版本工具链的兼容，或者构建二进制包。

构建 helm 多环境目录

Add spark operator support

Failed to load rigging or commands in disk,because of cannot import name Token

At the first run, i got the following error
Failed to load rigging or commands in disk,because of cannot import name Token
what to do about this?
Derrick is installed under sudo.

Release custom command

Release custom command (explain explain current Dockerfile to help developer learn Dockerfile)

python support uwsgi mode

Your should run `derrick config` first

This should be documented in README or docs instead of executing derrick init first.

➜  /Users/zhouzhengxi/Downloads/sshtunnel git:(master) ✗ derrick init --debug
Your should run `derrick config` first
➜  /Users/zhouzhengxi/Downloads/sshtunnel git:(master) ✗ derrick --version
0.0.24

When reinit the project, derrick should load the original input as the default value

Derrick直接构建 helm/多环境/ kustomize 目录结构然后yaml放到对应的子目录里

下载安装ringging

根据文档中的提示使用pip来安装相应的ringging包下载不了

安装报错这个如何解决

Collecting certifi>=2017.4.17
Using cached certifi-2019.11.28-py2.py3-none-any.whl (156 kB)
ERROR: requests 2.20.0 has requirement idna<2.8,>=2.5, but you'll have idna 2.8 which is incompatible.
ERROR: requests 2.20.0 has requirement urllib3<1.25,>=1.21.1, but you'll have urllib3 1.25.8 which is incompatible.
Installing collected packages: regex, whaaaaat, pychalk, simplejson, setuptools-git, idna, nose, certifi, chardet, urllib3, requests, requests-toolbelt, pkginfo, tqdm, python-derrick
Running setup.py install for regex ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ru3YeY/regex/setup.py'"'"'; file='"'"'/tmp/pip-install-ru3YeY/regex/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-zdC9ru/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python2.7/regex
cwd: /tmp/pip-install-ru3YeY/regex/
Complete output (20 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/regex
copying regex_2/init.py -> build/lib.linux-x86_64-2.7/regex
copying regex_2/regex.py -> build/lib.linux-x86_64-2.7/regex
copying regex_2/_regex_core.py -> build/lib.linux-x86_64-2.7/regex
copying regex_2/test_regex.py -> build/lib.linux-x86_64-2.7/regex
running build_ext
building 'regex._regex' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/regex_2
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-5Z483E/python2.7-2.7.17=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c regex_2/_regex.c -o build/temp.linux-x86_64-2.7/regex_2/_regex.o
regex_2/_regex.c:48:10: fatal error: Python.h: No such file or directory
#include "Python.h"
^~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ru3YeY/regex/setup.py'"'"'; file='"'"'/tmp/pip-install-ru3YeY/regex/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-zdC9ru/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python2.7/regex Check the logs for full command output.

[proposal] remove docker-compose yaml as default.

We will deprecate docker-compose in the next version of derrick.

Base image change to use openjdk

Automatically parse artifactId and version from pom.xml

Alpine base image cannot receive SIGTERM in k8s pod

Incompatible with prompt_toolkit version 2.0

The package whaaaaat is incompatible with prompt_toolkit version 2.0. So for new users or one who just upgraded prompt_toolkit would incounter error when using derrick.

Failed to load rigging or commands in disk,because of cannot import name Token

I already post the issue to that whaaaaat.

Temporary fix:

pip uninstall prompt-toolkit
pip install -Iv prompt_toolkit==1.0.15

May consider other packages before whaaaaat fix that.

java 9 failed to detect version number in python3

python2.7 works.

Reason that Python 3 fails is:

Regex search in JavaVersionDetector.get_most_relative_version() will report 'TypeError: cannot use a string pattern on a bytes-like object' error.

matches = re.search(regex, version)

springmvc custom rigging support

RT

Add Golang Support

Golang Rigging Support

pip3 install

mac 10.15.7

lishuang@newuser derrick % sudo pip3 install python-derrick
Requirement already satisfied: python-derrick in /Library/Python/3.7/site-packages (0.1.2)
Requirement already satisfied: whaaaaat==0.5.2 in /Library/Python/3.7/site-packages (from python-derrick) (0.5.2)
Requirement already satisfied: pychalk==1.1.0 in /Library/Python/3.7/site-packages (from python-derrick) (1.1.0)
Requirement already satisfied: docopt==0.6.2 in /Library/Python/3.7/site-packages (from python-derrick) (0.6.2)
Requirement already satisfied: simplejson==3.13.2 in /Library/Python/3.7/site-packages (from python-derrick) (3.13.2)
Requirement already satisfied: prompt-toolkit==1.0.15 in /Library/Python/3.7/site-packages (from python-derrick) (1.0.15)
Requirement already satisfied: Jinja2==2.10 in /Library/Python/3.7/site-packages (from python-derrick) (2.10)
Requirement already satisfied: chardet==3.0.4 in /Library/Python/3.7/site-packages (from python-derrick) (3.0.4)
Requirement already satisfied: MarkupSafe>=0.23 in /Library/Python/3.7/site-packages (from Jinja2==2.10->python-derrick) (1.1.1)
Requirement already satisfied: six>=1.9.0 in /Library/Python/3.7/site-packages (from prompt-toolkit==1.0.15->python-derrick) (1.15.0)
Requirement already satisfied: wcwidth in /Library/Python/3.7/site-packages (from prompt-toolkit==1.0.15->python-derrick) (0.2.5)
Requirement already satisfied: Pygments>=2.2.0 in /Library/Python/3.7/site-packages (from whaaaaat==0.5.2->python-derrick) (2.8.1)
Requirement already satisfied: regex>=2016.11.21 in /Library/Python/3.7/site-packages (from whaaaaat==0.5.2->python-derrick) (2021.3.17)

没有derrick命令

lishuang@newuser derrick % de
debinhex.pl debinhex5.28.pl decryptdir delv dev_mkdb
debinhex5.18.pl declare defaults desdp

咋用的

安装不成功，是什么原因？

执行 derrick -h

输出：
Failed to load rigging or commands in disk,because of No module named python_platform
Usage:

Derrick feature list from the community

基础配置

• 支持 push 到自定义镜像仓库
  docker hub, push 到 acr? (默认)

• 去掉 docker-compose.yml

• 用户自定义template: 优先级低，咱不实现

语言检测和编译

• 可通过cli 参数制定镜像
  比如：derrick init --image=xxx/yyy:latest

• Base image 可定义
  使用认证过的 base image，允许用户指定 base image

• 支持 Java
  Java maven 优先，可配置 jdk，cpu-limits
  spring cloud: https://medium.com/@TimvanBaarsen/first-look-at-cloud-native-buildpacks-support-in-spring-boot-2-3-milestone-1-ece8e72ed93f

• 支持 PHP
  PHP: 优先级中

• 不同环境生成一套构建模版
  支持 dev/staging/production，需要与 derrick up 部署到 Kubernetes 联动

• 非内置的构建 template，开放模版给用户
  heroku framework 文档+模版

镜像构建和部署

• 生成 Helm Chart
  docker/kubernetes-deployment.yml --> helm chart build
  基于 base chart template 构建 https://gimlet.io/onechart/getting-started/

其他

• 用户界面 cli 为主，暂不考虑 API