GithubHelp home page GithubHelp logo

tinkoff / nginx-builder Goto Github PK

View Code? Open in Web Editor NEW
154.0 23.0 30.0 57 KB

A tool to build deb or rpm package of required Nginx version from the source code, with the ability to connect third-party modules. Nginx parameters are set in the yaml configuration file.

License: Apache License 2.0

Python 97.78% Shell 2.22%
nginx nginx-server nginx-module ci-cd build build-automation build-tool builder debian centos

nginx-builder's Issues

OS_VERSION = distro.lsb_release_attr('release')[0] IndexError: string index out of range

Hello.
Try run python3

[root@grafana-apatsev Nginx-builder]# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)

Log

[root@grafana-apatsev Nginx-builder]# yum install -y python3-pip
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.reconn.ru
 * extras: mirror.reconn.ru
 * updates: mirror.reconn.ru
Resolving Dependencies
--> Running transaction check
---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed
--> Processing Dependency: python(abi) = 3.6 for package: python3-pip-9.0.3-5.el7.noarch
--> Processing Dependency: python3-setuptools for package: python3-pip-9.0.3-5.el7.noarch
--> Processing Dependency: /usr/bin/python3 for package: python3-pip-9.0.3-5.el7.noarch
--> Running transaction check
---> Package python3.x86_64 0:3.6.8-10.el7 will be installed
--> Processing Dependency: python3-libs(x86-64) = 3.6.8-10.el7 for package: python3-3.6.8-10.el7.x86_64
--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python3-3.6.8-10.el7.x86_64
---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed
--> Running transaction check
---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================================
 Package                                   Arch                          Version                               Repository                   Size
=================================================================================================================================================
Installing:
 python3-pip                               noarch                        9.0.3-5.el7                           base                        1.8 M
Installing for dependencies:
 python3                                   x86_64                        3.6.8-10.el7                          base                         69 k
 python3-libs                              x86_64                        3.6.8-10.el7                          base                        7.0 M
 python3-setuptools                        noarch                        39.2.0-10.el7                         base                        629 k

Transaction Summary
=================================================================================================================================================
Install  1 Package (+3 Dependent packages)

Total download size: 9.4 M
Installed size: 48 M
Downloading packages:
(1/4): python3-pip-9.0.3-5.el7.noarch.rpm                                                                                 | 1.8 MB  00:00:00     
(2/4): python3-3.6.8-10.el7.x86_64.rpm                                                                                    |  69 kB  00:00:00     
(3/4): python3-setuptools-39.2.0-10.el7.noarch.rpm                                                                        | 629 kB  00:00:00     
(4/4): python3-libs-3.6.8-10.el7.x86_64.rpm                                                                               | 7.0 MB  00:00:05     
-------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                            1.6 MB/s | 9.4 MB  00:00:05     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : python3-setuptools-39.2.0-10.el7.noarch                                                                                       1/4 
  Installing : python3-pip-9.0.3-5.el7.noarch                                                                                                2/4 
  Installing : python3-3.6.8-10.el7.x86_64                                                                                                   3/4 
  Installing : python3-libs-3.6.8-10.el7.x86_64                                                                                              4/4 
  Verifying  : python3-libs-3.6.8-10.el7.x86_64                                                                                              1/4 
  Verifying  : python3-setuptools-39.2.0-10.el7.noarch                                                                                       2/4 
  Verifying  : python3-3.6.8-10.el7.x86_64                                                                                                   3/4 
  Verifying  : python3-pip-9.0.3-5.el7.noarch                                                                                                4/4 

Installed:
  python3-pip.noarch 0:9.0.3-5.el7                                                                                                               

Dependency Installed:
  python3.x86_64 0:3.6.8-10.el7             python3-libs.x86_64 0:3.6.8-10.el7             python3-setuptools.noarch 0:39.2.0-10.el7            

Complete!
[root@grafana-apatsev Nginx-builder]# pip3 install -r requirements.txt
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting PyYAML==4.2b1 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/bd/da/0a49c1a31c60634b93fd1376b3b7966c4f81f2da8263f389cad5b6bbd6e8/PyYAML-4.2b1.tar.gz (263kB)
    100% |████████████████████████████████| 266kB 1.7MB/s 
Collecting requests==2.20.0 (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/f1/ca/10332a30cb25b627192b4ea272c351bce3ca1091e541245cccbace6051d8/requests-2.20.0-py2.py3-none-any.whl (60kB)
    100% |████████████████████████████████| 61kB 9.7MB/s 
Collecting GitPython==2.1.9 (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/88/9c/b462dddb492204417f88d538b0931e87631f2a98afe89842929f4ed9ca5b/GitPython-2.1.9-py2.py3-none-any.whl (447kB)
    100% |████████████████████████████████| 450kB 3.0MB/s 
Collecting distro==1.3.0 (from -r requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl
Collecting packaging==19.0 (from -r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/91/32/58bc30e646e55eab8b21abf89e353f59c0cc02c417e42929f4a9546e1b1d/packaging-19.0-py2.py3-none-any.whl
Collecting beautifulsoup4==4.7.1 (from -r requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/1d/5d/3260694a59df0ec52f8b4883f5d23b130bc237602a1411fa670eae12351e/beautifulsoup4-4.7.1-py3-none-any.whl (94kB)
    100% |████████████████████████████████| 102kB 11.5MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests==2.20.0->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 8.7MB/s 
Collecting certifi>=2017.4.17 (from requests==2.20.0->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl (154kB)
    100% |████████████████████████████████| 163kB 2.1MB/s 
Collecting idna<2.8,>=2.5 (from requests==2.20.0->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 10.3MB/s 
Collecting urllib3<1.25,>=1.21.1 (from requests==2.20.0->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/01/11/525b02e4acc0c747de8b6ccdab376331597c569c42ea66ab0a1dbd36eca2/urllib3-1.24.3-py2.py3-none-any.whl (118kB)
    100% |████████████████████████████████| 122kB 9.7MB/s 
Collecting gitdb2>=2.0.0 (from GitPython==2.1.9->-r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/03/6c/99296f89bad2ef85626e1df9f677acbee8885bb043ad82ad3ed4746d2325/gitdb2-2.0.6-py2.py3-none-any.whl (63kB)
    100% |████████████████████████████████| 71kB 10.8MB/s 
Collecting six (from packaging==19.0->-r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting pyparsing>=2.0.2 (from packaging==19.0->-r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/11/fa/0160cd525c62d7abd076a070ff02b2b94de589f1a9789774f17d7c54058e/pyparsing-2.4.2-py2.py3-none-any.whl (65kB)
    100% |████████████████████████████████| 71kB 10.8MB/s 
Collecting soupsieve>=1.2 (from beautifulsoup4==4.7.1->-r requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/5d/42/d821581cf568e9b7dfc5b415aa61952b0f5e3dede4f3cbd650e3a1082992/soupsieve-1.9.4-py2.py3-none-any.whl
Collecting smmap2>=2.0.0 (from gitdb2>=2.0.0->GitPython==2.1.9->-r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/55/d2/866d45e3a121ee15a1dc013824d58072fd5c7799c9c34d01378eb262ca8f/smmap2-2.0.5-py2.py3-none-any.whl
Installing collected packages: PyYAML, chardet, certifi, idna, urllib3, requests, smmap2, gitdb2, GitPython, distro, six, pyparsing, packaging, soupsieve, beautifulsoup4
  Running setup.py install for PyYAML ... done
Successfully installed GitPython-2.1.9 PyYAML-4.2b1 beautifulsoup4-4.7.1 certifi-2019.9.11 chardet-3.0.4 distro-1.3.0 gitdb2-2.0.6 idna-2.7 packaging-19.0 pyparsing-2.4.2 requests-2.20.0 six-1.12.0 smmap2-2.0.5 soupsieve-1.9.4 urllib3-1.24.3
[root@grafana-apatsev Nginx-builder]# ./main.py build -f config.yaml 
Traceback (most recent call last):
  File "./main.py", line 6, in <module>
    from src import downloader
  File "/root/Nginx-builder/src/downloader.py", line 3, in <module>
    from src import config
  File "/root/Nginx-builder/src/config.py", line 9, in <module>
    OS_VERSION = distro.lsb_release_attr('release')[0]
IndexError: string index out of range
[root@grafana-apatsev Nginx-builder]# mc

[root@grafana-apatsev Nginx-builder]# ./main.py build -f config.yaml -r 1
Traceback (most recent call last):
  File "./main.py", line 6, in <module>
    from src import downloader
  File "/root/Nginx-builder/src/downloader.py", line 3, in <module>
    from src import config
  File "/root/Nginx-builder/src/config.py", line 9, in <module>
    OS_VERSION = distro.lsb_release_attr('release')[0]
IndexError: string index out of range

builder - ERROR - [Errno 2] No such file or directory

While build, i got error:

.................
ix -I /root/rpmbuild/SOURCES/modules/ngx_devel_kit/objs -I objs/addon/ndk -I objs \
        -o objs/ngx_modules.o \
        objs/ngx_modules.c
sed -e "s|%%PREFIX%%|/etc/nginx|" \
        -e "s|%%PID_PATH%%|/var/run/nginx.pid|" \
        -e "s|%%CONF_PATH%%|/etc/nginx/nginx.conf|" \
        -e "s|%%ERROR_LOG_PATH%%|/var/log/nginx/error.log|" \
        < man/nginx.8 > objs/nginx.8
make[1]: Leaving directory '/root/rpmbuild/BUILD/nginx-1.19.3'


RPM build errors:

builder - ERROR - [Errno 2] No such file or directory: '/root/rpmbuild/RPMS/x86_64'

command:
./main.py build -f config.yaml

Build inside docker:
docker run --rm -it -v $(pwd):/nginx-builder:rw tinkoffcreditsystems/nginx-builder:centos-latest /bin/bash
OS: centos
cat config.yaml

nginx_version: 1.19.3
output_package: rpm
modules:
  - module:
      name: nginx-auth-ldap
      git_url: https://github.com/kvspb/nginx-auth-ldap.git
      git_branch: master
      dependencies:
        - openldap-devel
#  - module:
#      name: nginx-module-vts
#      git_url: https://github.com/vozlt/nginx-module-vts.git
#      git_branch: master
  - module:
      name: ngx_http_substitutions_filter_module
      git_url: https://github.com/yaoweibin/ngx_http_substitutions_filter_module.git
  - module:
      name: headers-more-nginx-module
      web_url: https://github.com/openresty/headers-more-nginx-module/archive/v0.261.zip
  - module:
      name: nginx-module-vts
      git_url: https://github.com/vozlt/nginx-module-vts.git
      git_tag: v0.1.18
  - module:
      name: ngx_devel_kit
      git_url: https://github.com/simplresty/ngx_devel_kit.git
      git_tag: v0.3.0
  - module:
      name: ngx_cache_purge
      git_url: https://github.com/FRiCKLE/ngx_cache_purge.git
  - module:
      name: ngx_http_dyups_module
      git_url: https://github.com/yzprofile/ngx_http_dyups_module.git
  - module:
      name: http_gzip_static_module
      type: embedded
  - module:
      name: http_v2_module
      type: embedded
  - module:
      name: http_ssl_module
      type: embedded
  - module:
      name: http_addition_module
      type: embedded
  - module:
      name: http_auth_request_module
      type: embedded
  - module:
      name: http_gunzip_module
      type: embedded
  - module:
      name: http_gzip_static_module
      type: embedded
  - module:
      name: http_realip_module
      type: embedded
  - module:
      name: http_secure_link_module
      type: embedded
  - module:
      name: http_slice_module
      type: embedded
  - module:
      name: http_stub_status_module
      type: embedded
  - module:
      name: http_sub_module
      type: embedded
  - module:
      name: stream_realip_module
      type: embedded
  - module:
      name: stream_ssl_preread_module
      type: embedded
configure_params:
  - "--with-openssl-opt='enable-tls1_3'"

Used by http_dav_module instead of nginx-dav-ext-module

Hello. I try build nginx with nginx-dav-ext-module.

config.yaml

nginx_version: 1.14.0
output_package: rpm
modules:
  - module:
      name: http_dav_module
      git_url: https://github.com/arut/nginx-dav-ext-module.git
      git_tag: v3.0.0
      type: embedded
      dependencies:
        - libxml2-devel
        - libxslt-devel

Build log:
https://gist.github.com/patsevanton/f767374cff5c9d6181ad0727e82490c5

nginx -V

nginx version: nginx/1.14.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) 
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/usr/lib/nginx/modules --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-debug --user=nginx --group=nginx --with-pcre-jit --with-compat --with-file-aio --with-threads --with-stream --with-cc-opt= --with-ld-opt= --add-module=/root/rpmbuild/SOURCES/modules/**http_dav_module** --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

nginx config /etc/nginx/conf.d/webdav.conf

server {
    listen 80;
    server_name _;
    location /upload {
        client_max_body_size 1g;
        root /usr/share/nginx/html/1;
        dav_methods PUT DELETE MKCOL COPY MOVE;
}

nginx -t

nginx: [emerg] unknown directive "dav_methods" in /etc/nginx/conf.d/webdav.conf:8
nginx: configuration file /etc/nginx/nginx.conf test failed

Пытается выполнить CFLAGS/LDFLAGS в spec-е

Describe the bug
Почему-то в спек попадают заведомо нерабочие команды CFLAGS и LDFLAGS.

To Reproduce
Собрать контейнер из Dockerfile.centos, запустить ./main.py build -f example.config.yaml
В выводе видим, что CFLAGS/LDFLAGS подставляются в спек как команды, в результате:

+ umask 022
+ cd /root/rpmbuild/BUILD
+ cd nginx-1.17.0
++ CFLAGS
/var/tmp/rpm-tmp.51HkTl: line 29: CFLAGS: command not found
++ LDFLAGS
/var/tmp/rpm-tmp.51HkTl: line 29: LDFLAGS: command not found
+++ pcre-config --cflags
++ echo -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
+ ./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/usr/lib/nginx/modules --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-debug --user=nginx --group=nginx --with-http_gzip_static_module --with-http_v2_module --with-http_ssl_module --with-pcre-jit --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_gunzip_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_stub_status_module --with-http_sub_module --with-stream --with-stream_realip_module --with-stream_ssl_preread_module --with-cc-opt= --with-ld-opt= --add-module=/root/rpmbuild/SOURCES/modules/nginx-auth-ldap --add-module=/root/rpmbuild/SOURCES/modules/ngx_http_substitutions_filter_module --add-module=/root/rpmbuild/SOURCES/modules/headers-more-nginx-module-0.261 --add-module=/root/rpmbuild/SOURCES/modules/nginx-module-vts --add-module=/root/rpmbuild/SOURCES/modules/ngx_devel_kit --add-module=/root/rpmbuild/SOURCES/modules/ngx_cache_purge --add-module=/root/rpmbuild/SOURCES/modules/ngx_http_dyups_module '--with-cc-opt=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' '--with-ld-opt=-Wl,-z,relro -Wl,-z,now -pie' --with-debug
+ make -j4

Expected behavior
Ожидалось что CFLAGS/LDFLAGS будут выставлены как переменные среды либо будут заданы в аргументах make. Чтобы они что-то делали. Либо не заданы вообще.

Desktop (please complete the following information):
В докер-контейнере.

неверный формат архива tar.gz

При попытке сборки ловлю сообщение:

~/Nginx-builder$ ./main.py build -f banana-config.yaml
builder - INFO - Parse yaml file: banana-config.yaml
builder - INFO - /usr/src/nginx dir already exist
builder - INFO - Download scripts for build deb package
builder - INFO - Downloading nginx src...
builder - INFO - --> http://nginx.org/download/nginx-1.16.tar.gz
builder - INFO - Downloading 3d-party modules...
builder - INFO - /usr/src/nginx/modules dir already exist
builder - INFO - Module nginx_cookie_flag will download by branch
builder - INFO - -- Done: nginx_cookie_flag
builder - INFO - Downloading dependencies
builder - INFO - Building .deb package
builder - ERROR - Archive format nginx-1.16.tar.gz is not valid
builder - ERROR - 'NoneType' object has no attribute 'rstrip'

nginx: [emerg] unknown directive "dav_methods"

Config

server {
    listen 80;
    server_name _;
    location /upload {
        client_max_body_size 1g;
        root /usr/share/nginx/html/1;
        dav_methods PUT DELETE MKCOL COPY MOVE;
}

Status nginx

-- Unit nginx.service has begun starting up.
nginx[15509]: nginx: [emerg] unknown directive "dav_methods" in /etc/nginx/conf.d/webdav.conf:13
systemd[1]: nginx.service: control process exited, code=exited status=1
systemd[1]: Failed to start nginx - high performance web server.
-- Subject: Unit nginx.service has failed

config.yaml

nginx_version: 1.17.1
output_package: rpm
modules:
  - module:
      name: http_dav_module
      git_url: https://github.com/arut/nginx-dav-ext-module.git
      git_tag: v3.0.0
      dependencies:
        - libxml2-devel
        - libxslt-devel

    # static module
    $ ./configure --with-http_dav_module --add-module=/path/to/nginx-dav-ext-module

Log

builder - INFO - Module dir /usr/src/nginx/modules/http_dav_module
builder - INFO - -- Done: http_dav_module
builder - INFO - Download dependencies
builder - INFO - Running command: yum install --assumeyes libxslt-devel libxml2-devel
builder - INFO - Loaded plugins: fastestmirror, fusioninventory-agent
Loading mirror speeds from cached hostfile
Package libxslt-devel-1.1.28-5.el7.x86_64 already installed and latest version
Package libxml2-devel-2.9.1-6.el7_2.3.x86_64 already installed and latest version
Nothing to do

builder - INFO - Building .rpm package
builder - INFO - Running command: rpm --eval '%{_topdir}'
builder - INFO - /root/rpmbuild

builder - INFO - Running command: rpmbuild -bb nginx.spec
+ umask 022
+ cd /root/rpmbuild/BUILD
+ cd /root/rpmbuild/BUILD
+ rm -rf nginx-1.17.1
+ /usr/bin/tar -xf -
+ /usr/bin/gzip -dc /root/rpmbuild/SOURCES/nginx-1.17.1.tar.gz
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd nginx-1.17.1
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ cp /root/rpmbuild/SOURCES/nginx.init.in .
+ sed -e 's|%DEFAULTSTART%|2 3 4 5|g' -e 's|%DEFAULTSTOP%|0 1 6|g' -e 's|%PROVIDES%|nginx|g'
+ sed -e 's|%DEFAULTSTART%||g' -e 's|%DEFAULTSTOP%|0 1 2 3 4 5 6|g' -e 's|%PROVIDES%|nginx-debug|g'
+ exit 0
+ umask 022
+ cd /root/rpmbuild/BUILD
+ cd nginx-1.17.1
+++ pcre-config --cflags
++ echo -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
+ ./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/usr/lib/nginx/modules --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-debug --user=nginx --group=nginx --with-http_gzip_static_module --with-http_v2_module --with-http_ssl_module --with-pcre-jit --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_gunzip_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_stub_status_module --with-http_sub_module --with-stream --with-stream_realip_module --with-stream_ssl_preread_module --with-cc-opt= --with-ld-opt= --add-module=/root/rpmbuild/SOURCES/modules/http_dav_module '--with-cc-opt=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' '--with-ld-opt=-Wl,-z,relro -Wl,-z,now -pie' --with-debug

добавить опцию "зачистки поляны"

Последующие сборки спотыкаются об оставшиеся от предыдущих артефакты (исходники, пакеты).
1 Хочется иметь возможность все это зачищать (например ./main.py clear -f config.yaml).
2 Хочется иметь возможность "не спотыкаться" об имеющиеся исходники и перезаписывать пакет.

не добавляется --with для embedded для модуля, у которого есть флаг git_url

config.yaml

nginx_version: 1.14.0
output_package: rpm
modules:
  - module:
      name: http_dav_module
      git_url: https://github.com/arut/nginx-dav-ext-module.git
      git_tag: v3.0.0
      type: embedded
      dependencies:
        - libxml2-devel
        - libxslt-devel

Log

builder - INFO - Downloading 3d-party modules...
builder - INFO - Module http_dav_module will download by tag
builder - INFO - -- Done: http_dav_module

++ echo -O2 ... --with-cc-opt= --with-ld-opt= --add-module=/root/rpmbuild/SOURCES/modules/http_dav_module ...

ModuleNotFoundError: No module named 'gitdb.utils.compat'

Hello! Try use Nginx-builder.

Version

cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)

Install and run

yum install -y epel-release rpmdevtools mc git 
yum install -y gcc make automake yum-utils redhat-lsb-core openssl-devel zlib-devel pcre-devel epel-release
yum install -y python36 python36-devel python36-pip
git  clone https://github.com/TinkoffCreditSystems/Nginx-builder.git
cd Nginx-builder/
pip3.6 install -r requirements.txt --no-cache-dir
./main.py build -f example.config.yaml -r 1

Error:

./main.py build -f example.config.yaml 
Traceback (most recent call last):
  File "./main.py", line 6, in <module>
    from src import downloader
  File "/root/Nginx-builder/src/downloader.py", line 5, in <module>
    import git
  File "/usr/local/lib/python3.6/site-packages/git/__init__.py", line 38, in <module>
    from git.exc import *                       # @NoMove @IgnorePep8
  File "/usr/local/lib/python3.6/site-packages/git/exc.py", line 9, in <module>
    from git.compat import UnicodeMixin, safe_decode, string_types
  File "/usr/local/lib/python3.6/site-packages/git/compat.py", line 16, in <module>
    from gitdb.utils.compat import (
ModuleNotFoundError: No module named 'gitdb.utils.compat'

Can not build when using 'local_url'

Describe the bug
Can not build when using 'local_url'
If this archive is on a web (s3) and using 'web_url' it builds ok.

To Reproduce
config example:
nginx_version: 1.14.0
output_package: deb
modules:

  • module:
    name: ngx_pagespeed
    local_url: /root/ngx_pagespeed/incubator-pagespeed-ngx-1.13.35.2-stable.tar.gz
    dependencies:
    - debhelper
    - dh-systemd
    - quilt
    - libssl-dev
    - libpcre3-dev
    - zlib1g-dev
    - uuid-dev

Screenshots
builder - INFO - Parse yaml file: config.yaml
builder - INFO - Download scripts for build deb package
builder - INFO - Downloading nginx src...
builder - INFO - --> http://nginx.org/download/nginx-1.14.0.tar.gz
builder - INFO - Downloading 3d-party modules...
builder - ERROR - 'str' object is not callable

Desktop (please complete the following information):

  • OS: Ubuntu 18.04

Question: Additional dependencies for build

What can I do if module require some library unpacked in module dir?
For example pagespeed require psol (I need download psol-${version}-x64.tar.gz and unpack it to pagespeed module folder)

Can I build modules as dynamic and pack it to separate rpm / deb packages?

git error: already exists and is not an empty directory.

After second run inside docker container i got error:

builder - ERROR - Cmd('git') failed due to: exit code(128)
  cmdline: git clone --branch=master -v https://github.com/yaoweibin/ngx_http_substitutions_filter_module.git /usr/src/nginx/modules/ngx_http_substitutions_filter_module
  stderr: 'fatal: destination path '/usr/src/nginx/modules/ngx_http_substitutions_filter_module' already exists and is not an empty directory.
'

config.yaml

nginx_version: 1.19.3
output_package: rpm
modules:
  - module:
      name: nginx-auth-ldap
      git_url: https://github.com/kvspb/nginx-auth-ldap.git
      git_branch: master
      dependencies:
        - openldap-devel
  - module:
      name: nginx-module-vts
      git_url: https://github.com/vozlt/nginx-module-vts.git
      git_branch: master
  - module:
      name: ngx_http_substitutions_filter_module
      git_url: https://github.com/yaoweibin/ngx_http_substitutions_filter_module.git
  - module:
      name: headers-more-nginx-module
      web_url: https://github.com/openresty/headers-more-nginx-module/archive/v0.261.zip
  - module:
      name: nginx-module-vts
      git_url: https://github.com/vozlt/nginx-module-vts.git
      git_tag: v0.1.18
  - module:
      name: ngx_devel_kit
      git_url: https://github.com/simplresty/ngx_devel_kit.git
      git_tag: v0.3.0
  - module:
      name: ngx_cache_purge
      git_url: https://github.com/FRiCKLE/ngx_cache_purge.git
  - module:
      name: ngx_http_dyups_module
      git_url: https://github.com/yzprofile/ngx_http_dyups_module.git
  - module:
      name: http_gzip_static_module
      type: embedded
  - module:
      name: http_v2_module
      type: embedded
  - module:
      name: http_ssl_module
      type: embedded
  - module:
      name: http_addition_module
      type: embedded
  - module:
      name: http_auth_request_module
      type: embedded
  - module:
      name: http_gunzip_module
      type: embedded
  - module:
      name: http_gzip_static_module
      type: embedded
  - module:
      name: http_realip_module
      type: embedded
  - module:
      name: http_secure_link_module
      type: embedded
  - module:
      name: http_slice_module
      type: embedded
  - module:
      name: http_stub_status_module
      type: embedded
  - module:
      name: http_sub_module
      type: embedded
  - module:
      name: stream_realip_module
      type: embedded
  - module:
      name: stream_ssl_preread_module
      type: embedded
configure_params:
  - "--with-openssl-opt='enable-tls1_3'"

Run inside docker:
./main.py build -f config.yaml -r r01

OS: centos

Cannot find nginx source rpm(SRPM) with version 1.16.0 in url http://nginx.org/packages/mainline/centos/7/SRPMS

Пытаюсь собрать 16 версию nginx

cat config.yaml 
nginx_version: 1.16.0
output_package: rpm
modules:
  - module:
      name: nginx-dav-ext-module
      git_url: https://github.com/arut/nginx-dav-ext-module.git
      git_tag: v3.0.0

./main.py build -f config.yaml 
builder - INFO - Parse yaml file: config.yaml
builder - INFO - Download scripts for build rpm package
builder - INFO - Running command: rpmdev-setuptree
builder - INFO - 
builder - INFO - Downloading nginx src...
builder - ERROR - Cannot find nginx source rpm(SRPM) with version 1.16.0 in url http://nginx.org/packages/mainline/centos/7/SRPMS

Может добавить использование https://nginx.org/download/nginx-1.16.0.tar.gz ?
Например, код взять из https://nginx.org/download/nginx-1.16.0.tar.gz, а остальные файлы с с других SRPM ( http://nginx.org/packages/mainline/centos/7/SRPMS/nginx-1.15.9-1.el7_4.ngx.src.rpm )?

Support apply patch

Hello!
Thanks for Nginx-builder!
Please add support apply patch
For example:

nginx_version: 1.14.1
output_package: deb
modules:
  - module:
      name: nginx_upstream_check_module
      git_url: https://github.com/yaoweibin/nginx_upstream_check_module.git
      patch: https://github.com/yaoweibin/nginx_upstream_check_module/blob/master/check_1.14.0+.patch

stderr: 'fatal: destination path '/usr/src/nginx/modules/ngx_http_substitutions_filter_module' already exists and is not an empty directory. 

lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.5.1804 (Core) 
Release:	7.5.1804
Codename:	Core

When run second time get error:

./main.py build -f example.config.yaml 
builder - INFO - Parse yaml file: example.config.yaml
builder - INFO - /usr/src/nginx dir already exist
builder - INFO - Download scripts for build rpm package
builder - INFO - Running command: rpmdev-setuptree
builder - INFO - 
builder - INFO - Downloading nginx src...
builder - INFO - Running command: rpm --upgrade --verbose --hash http://nginx.org/packages/mainline/centos/7/SRPMS/nginx-1.13.9-1.el7_4.ngx.src.rpm
warning: /var/tmp/rpm-tmp.LZDJZS: Header V4 RSA/SHA1 Signature, key ID 7bd9bf62: NOKEY
warning: user builder does not exist - using root
warning: group builder does not exist - using root
builder - INFO - Retrieving http://nginx.org/packages/mainline/centos/7/SRPMS/nginx-1.13.9-1.el7_4.ngx.src.rpm
Updating / installing...
nginx-1:1.13.9-1.el7_4.ngx            ########################################

builder - INFO - Downloading 3d-party modules...
builder - INFO - /usr/src/nginx/modules dir already exist
builder - INFO - Module nginx-auth-ldap will download by branch
builder - ERROR - Cmd('git') failed due to: exit code(128)
  cmdline: git clone --branch=master -v https://github.com/kvspb/nginx-auth-ldap.git /usr/src/nginx/modules/nginx-auth-ldap
  stderr: 'fatal: destination path '/usr/src/nginx/modules/nginx-auth-ldap' already exists and is not an empty directory.
'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.