GithubHelp home page GithubHelp logo

tinkoff / nginx-builder Goto Github PK

View Code? Open in Web Editor NEW
154.0 23.0 30.0 57 KB

A tool to build deb or rpm package of required Nginx version from the source code, with the ability to connect third-party modules. Nginx parameters are set in the yaml configuration file.

License: Apache License 2.0

Python 97.78% Shell 2.22%
nginx nginx-server nginx-module ci-cd build build-automation build-tool builder debian centos

nginx-builder's Introduction

Nginx-builder

Build Status

Description/Описание

ENG

Tool for building deb or rpm package [Nginx] (http://nginx.org/) of the required version from the source code, with the ability to connect third-party modules. Nginx parameters are set in the configuration file of yaml format.

RU

Инструмент для сборки deb или rpm пакета Nginx требуемой версии из исходных кодов, с возможностью подключения сторонних модулей. Параметры Nginx задаются в конфигурационном файле формата yaml.

Docker

ENG

Now we are at the Docker hub: https://cloud.docker.com/repository/docker/tinkoffcreditsystems/nginx-builder

RU

Теперь мы есть и на Docker hub: https://cloud.docker.com/repository/docker/tinkoffcreditsystems/nginx-builder

Execution options/Параметры запуска

ENG

You can start the assembler both directly on the host machine and in the docker container, for example

RU

Запускать сборщик можно, как непосредственно на хост машине, так и в docker контейнере, например

Example run/Пример запуска в docker образе "ubuntu-latest" или "centos-latest

docker run --rm -it -v $(pwd):/nginx-builder:rw tinkoffcreditsystems/nginx-builder:centos-latest /bin/bash
docker run --rm -it -v $(pwd):/nginx-builder:rw tinkoffcreditsystems/nginx-builder:ubuntu-latest /bin/bash

Конфигурация

ENG

The main configuration file is in yaml format. Description of parameters:

---
nginx_version: the necessary version of nginx
output_package: type of output package deb or rpm
modules:
  - module:
      name: The name of the module. If not specified, taken from the last part of the URL
      git_url: git file URL
      git_tag: The name of the tag. (Optional)
      git_branch: The name of the branch. (Optional). If neither tag nor branch is specified, the master branch is taken by default
      patch: Relative path to patch for source code. (Optional)
      dependencies: 
        - list of dependencies for building the module (Optional)
    module:
      name: The name of the module. If not specified, taken from the last part of the URL
      web_url: Link to the archive with the module source code
    module:
      name: The name of the module. If not specified, taken from the last part of the URL
      local_url: Path to the module source code archive
    module:
      name: The name of the embedded module
      type: embedded  
configure_params: list of custom build parameters

The configuration file with advanced settings is located in src/config.py. In most cases it does not need to be changed.

RU

Основной конфигурационный файл в yaml формате. Описание параметров:

---
nginx_version: необходимая версия nginx
output_package: тип выходного пакета deb или rpm
modules:
  - module:
      name: Название модуля. Если не указано, берется из последней части URL
      git_url: URL git файла
      git_tag: название тэга. (Не обязательно)
      git_branch: название ветки. (Не обязательно). Если не указан ни tag, ни branch по умолчанию берется master ветка
      patch: Относительный путь к патчу для исходного кода. (Не обязательно).
      dependencies: 
        - список зависимостей для сборки модуля (Не обязательно)
    module:
      name: Название модуля. Если не указано, берется из последней части URL
      web_url: Ссылка на архив с исходным кодом модуля
    module:
      name: Название модуля. Если не указано, берется из последней части URL
      local_url: Путь к архиву с исходным кодом модуля
    module:
      name: Название модуля встроенного модуля
      type: embedded
configure_params: список дополнительных параметров сборки

Конфигурационный файл с расширенными настройками расположен в src/config.py. В большинстве случаев менять его не нужно.

Manual script execution/Ручной запуск скрипта

Requirements/Требования

  • python >= 3.5

ENG

You will also need packages to compile Nginx. Their list can be seen in the Dockerfile.

pip3 install -r requirements.txt
./main.py build -f [config_file].yaml -r [revision_number]
  • revision number optional parameter, used to version assemblies

RU

Также потребуются пакеты для компиляции Nginx. Их перечень можно увидеть в Dockerfile

pip3 install -r requirements.txt
./main.py build -f [конфиг_файл].yaml -r [номер_ревизии]
  • номер ревизии опциональный параметр, служит для версионирования сборок

nginx-builder's People

Contributors

dependabot[bot] avatar frodox avatar iskrant avatar javdet avatar msaf1980 avatar patsevanton avatar therb1 avatar vadikgo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nginx-builder's Issues

Пытается выполнить CFLAGS/LDFLAGS в spec-е

Describe the bug
Почему-то в спек попадают заведомо нерабочие команды CFLAGS и LDFLAGS.

To Reproduce
Собрать контейнер из Dockerfile.centos, запустить ./main.py build -f example.config.yaml
В выводе видим, что CFLAGS/LDFLAGS подставляются в спек как команды, в результате:

+ umask 022
+ cd /root/rpmbuild/BUILD
+ cd nginx-1.17.0
++ CFLAGS
/var/tmp/rpm-tmp.51HkTl: line 29: CFLAGS: command not found
++ LDFLAGS
/var/tmp/rpm-tmp.51HkTl: line 29: LDFLAGS: command not found
+++ pcre-config --cflags
++ echo -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
+ ./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/usr/lib/nginx/modules --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-debug --user=nginx --group=nginx --with-http_gzip_static_module --with-http_v2_module --with-http_ssl_module --with-pcre-jit --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_gunzip_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_stub_status_module --with-http_sub_module --with-stream --with-stream_realip_module --with-stream_ssl_preread_module --with-cc-opt= --with-ld-opt= --add-module=/root/rpmbuild/SOURCES/modules/nginx-auth-ldap --add-module=/root/rpmbuild/SOURCES/modules/ngx_http_substitutions_filter_module --add-module=/root/rpmbuild/SOURCES/modules/headers-more-nginx-module-0.261 --add-module=/root/rpmbuild/SOURCES/modules/nginx-module-vts --add-module=/root/rpmbuild/SOURCES/modules/ngx_devel_kit --add-module=/root/rpmbuild/SOURCES/modules/ngx_cache_purge --add-module=/root/rpmbuild/SOURCES/modules/ngx_http_dyups_module '--with-cc-opt=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' '--with-ld-opt=-Wl,-z,relro -Wl,-z,now -pie' --with-debug
+ make -j4

Expected behavior
Ожидалось что CFLAGS/LDFLAGS будут выставлены как переменные среды либо будут заданы в аргументах make. Чтобы они что-то делали. Либо не заданы вообще.

Desktop (please complete the following information):
В докер-контейнере.

ModuleNotFoundError: No module named 'gitdb.utils.compat'

Hello! Try use Nginx-builder.

Version

cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)

Install and run

yum install -y epel-release rpmdevtools mc git 
yum install -y gcc make automake yum-utils redhat-lsb-core openssl-devel zlib-devel pcre-devel epel-release
yum install -y python36 python36-devel python36-pip
git  clone https://github.com/TinkoffCreditSystems/Nginx-builder.git
cd Nginx-builder/
pip3.6 install -r requirements.txt --no-cache-dir
./main.py build -f example.config.yaml -r 1

Error:

./main.py build -f example.config.yaml 
Traceback (most recent call last):
  File "./main.py", line 6, in <module>
    from src import downloader
  File "/root/Nginx-builder/src/downloader.py", line 5, in <module>
    import git
  File "/usr/local/lib/python3.6/site-packages/git/__init__.py", line 38, in <module>
    from git.exc import *                       # @NoMove @IgnorePep8
  File "/usr/local/lib/python3.6/site-packages/git/exc.py", line 9, in <module>
    from git.compat import UnicodeMixin, safe_decode, string_types
  File "/usr/local/lib/python3.6/site-packages/git/compat.py", line 16, in <module>
    from gitdb.utils.compat import (
ModuleNotFoundError: No module named 'gitdb.utils.compat'

не добавляется --with для embedded для модуля, у которого есть флаг git_url

config.yaml

nginx_version: 1.14.0
output_package: rpm
modules:
  - module:
      name: http_dav_module
      git_url: https://github.com/arut/nginx-dav-ext-module.git
      git_tag: v3.0.0
      type: embedded
      dependencies:
        - libxml2-devel
        - libxslt-devel

Log

builder - INFO - Downloading 3d-party modules...
builder - INFO - Module http_dav_module will download by tag
builder - INFO - -- Done: http_dav_module

++ echo -O2 ... --with-cc-opt= --with-ld-opt= --add-module=/root/rpmbuild/SOURCES/modules/http_dav_module ...

OS_VERSION = distro.lsb_release_attr('release')[0] IndexError: string index out of range

Hello.
Try run python3

[root@grafana-apatsev Nginx-builder]# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 

Log

[root@grafana-apatsev Nginx-builder]# yum install -y python3-pip
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.reconn.ru
 * extras: mirror.reconn.ru
 * updates: mirror.reconn.ru
Resolving Dependencies
--> Running transaction check
---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed
--> Processing Dependency: python(abi) = 3.6 for package: python3-pip-9.0.3-5.el7.noarch
--> Processing Dependency: python3-setuptools for package: python3-pip-9.0.3-5.el7.noarch
--> Processing Dependency: /usr/bin/python3 for package: python3-pip-9.0.3-5.el7.noarch
--> Running transaction check
---> Package python3.x86_64 0:3.6.8-10.el7 will be installed
--> Processing Dependency: python3-libs(x86-64) = 3.6.8-10.el7 for package: python3-3.6.8-10.el7.x86_64
--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python3-3.6.8-10.el7.x86_64
---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed
--> Running transaction check
---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================================
 Package                                   Arch                          Version                               Repository                   Size
=================================================================================================================================================
Installing:
 python3-pip                               noarch                        9.0.3-5.el7                           base                        1.8 M
Installing for dependencies:
 python3                                   x86_64                        3.6.8-10.el7                          base                         69 k
 python3-libs                              x86_64                        3.6.8-10.el7                          base                        7.0 M
 python3-setuptools                        noarch                        39.2.0-10.el7                         base                        629 k

Transaction Summary
=================================================================================================================================================
Install  1 Package (+3 Dependent packages)

Total download size: 9.4 M
Installed size: 48 M
Downloading packages:
(1/4): python3-pip-9.0.3-5.el7.noarch.rpm                                                                                 | 1.8 MB  00:00:00     
(2/4): python3-3.6.8-10.el7.x86_64.rpm                                                                                    |  69 kB  00:00:00     
(3/4): python3-setuptools-39.2.0-10.el7.noarch.rpm                                                                        | 629 kB  00:00:00     
(4/4): python3-libs-3.6.8-10.el7.x86_64.rpm                                                                               | 7.0 MB  00:00:05     
-------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                            1.6 MB/s | 9.4 MB  00:00:05     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : python3-setuptools-39.2.0-10.el7.noarch                                                                                       1/4 
  Installing : python3-pip-9.0.3-5.el7.noarch                                                                                                2/4 
  Installing : python3-3.6.8-10.el7.x86_64                                                                                                   3/4 
  Installing : python3-libs-3.6.8-10.el7.x86_64                                                                                              4/4 
  Verifying  : python3-libs-3.6.8-10.el7.x86_64                                                                                              1/4 
  Verifying  : python3-setuptools-39.2.0-10.el7.noarch                                                                                       2/4 
  Verifying  : python3-3.6.8-10.el7.x86_64                                                                                                   3/4 
  Verifying  : python3-pip-9.0.3-5.el7.noarch                                                                                                4/4 

Installed:
  python3-pip.noarch 0:9.0.3-5.el7                                                                                                               

Dependency Installed:
  python3.x86_64 0:3.6.8-10.el7             python3-libs.x86_64 0:3.6.8-10.el7             python3-setuptools.noarch 0:39.2.0-10.el7            

Complete!
[root@grafana-apatsev Nginx-builder]# pip3 install -r requirements.txt
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting PyYAML==4.2b1 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/bd/da/0a49c1a31c60634b93fd1376b3b7966c4f81f2da8263f389cad5b6bbd6e8/PyYAML-4.2b1.tar.gz (263kB)
    100% |████████████████████████████████| 266kB 1.7MB/s 
Collecting requests==2.20.0 (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/f1/ca/10332a30cb25b627192b4ea272c351bce3ca1091e541245cccbace6051d8/requests-2.20.0-py2.py3-none-any.whl (60kB)
    100% |████████████████████████████████| 61kB 9.7MB/s 
Collecting GitPython==2.1.9 (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/88/9c/b462dddb492204417f88d538b0931e87631f2a98afe89842929f4ed9ca5b/GitPython-2.1.9-py2.py3-none-any.whl (447kB)
    100% |████████████████████████████████| 450kB 3.0MB/s 
Collecting distro==1.3.0 (from -r requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl
Collecting packaging==19.0 (from -r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/91/32/58bc30e646e55eab8b21abf89e353f59c0cc02c417e42929f4a9546e1b1d/packaging-19.0-py2.py3-none-any.whl
Collecting beautifulsoup4==4.7.1 (from -r requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/1d/5d/3260694a59df0ec52f8b4883f5d23b130bc237602a1411fa670eae12351e/beautifulsoup4-4.7.1-py3-none-any.whl (94kB)
    100% |████████████████████████████████| 102kB 11.5MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests==2.20.0->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 8.7MB/s 
Collecting certifi>=2017.4.17 (from requests==2.20.0->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl (154kB)
    100% |████████████████████████████████| 163kB 2.1MB/s 
Collecting idna<2.8,>=2.5 (from requests==2.20.0->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 10.3MB/s 
Collecting urllib3<1.25,>=1.21.1 (from requests==2.20.0->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/01/11/525b02e4acc0c747de8b6ccdab376331597c569c42ea66ab0a1dbd36eca2/urllib3-1.24.3-py2.py3-none-any.whl (118kB)
    100% |████████████████████████████████| 122kB 9.7MB/s 
Collecting gitdb2>=2.0.0 (from GitPython==2.1.9->-r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/03/6c/99296f89bad2ef85626e1df9f677acbee8885bb043ad82ad3ed4746d2325/gitdb2-2.0.6-py2.py3-none-any.whl (63kB)
    100% |████████████████████████████████| 71kB 10.8MB/s 
Collecting six (from packaging==19.0->-r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting pyparsing>=2.0.2 (from packaging==19.0->-r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/11/fa/0160cd525c62d7abd076a070ff02b2b94de589f1a9789774f17d7c54058e/pyparsing-2.4.2-py2.py3-none-any.whl (65kB)
    100% |████████████████████████████████| 71kB 10.8MB/s 
Collecting soupsieve>=1.2 (from beautifulsoup4==4.7.1->-r requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/5d/42/d821581cf568e9b7dfc5b415aa61952b0f5e3dede4f3cbd650e3a1082992/soupsieve-1.9.4-py2.py3-none-any.whl
Collecting smmap2>=2.0.0 (from gitdb2>=2.0.0->GitPython==2.1.9->-r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/55/d2/866d45e3a121ee15a1dc013824d58072fd5c7799c9c34d01378eb262ca8f/smmap2-2.0.5-py2.py3-none-any.whl
Installing collected packages: PyYAML, chardet, certifi, idna, urllib3, requests, smmap2, gitdb2, GitPython, distro, six, pyparsing, packaging, soupsieve, beautifulsoup4
  Running setup.py install for PyYAML ... done
Successfully installed GitPython-2.1.9 PyYAML-4.2b1 beautifulsoup4-4.7.1 certifi-2019.9.11 chardet-3.0.4 distro-1.3.0 gitdb2-2.0.6 idna-2.7 packaging-19.0 pyparsing-2.4.2 requests-2.20.0 six-1.12.0 smmap2-2.0.5 soupsieve-1.9.4 urllib3-1.24.3
[root@grafana-apatsev Nginx-builder]# ./main.py build -f config.yaml 
Traceback (most recent call last):
  File "./main.py", line 6, in <module>
    from src import downloader
  File "/root/Nginx-builder/src/downloader.py", line 3, in <module>
    from src import config
  File "/root/Nginx-builder/src/config.py", line 9, in <module>
    OS_VERSION = distro.lsb_release_attr('release')[0]
IndexError: string index out of range
[root@grafana-apatsev Nginx-builder]# mc

[root@grafana-apatsev Nginx-builder]# ./main.py build -f config.yaml -r 1
Traceback (most recent call last):
  File "./main.py", line 6, in <module>
    from src import downloader
  File "/root/Nginx-builder/src/downloader.py", line 3, in <module>
    from src import config
  File "/root/Nginx-builder/src/config.py", line 9, in <module>
    OS_VERSION = distro.lsb_release_attr('release')[0]
IndexError: string index out of range

stderr: 'fatal: destination path '/usr/src/nginx/modules/ngx_http_substitutions_filter_module' already exists and is not an empty directory.

lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.5.1804 (Core) 
Release:	7.5.1804
Codename:	Core

When run second time get error:

./main.py build -f example.config.yaml 
builder - INFO - Parse yaml file: example.config.yaml
builder - INFO - /usr/src/nginx dir already exist
builder - INFO - Download scripts for build rpm package
builder - INFO - Running command: rpmdev-setuptree
builder - INFO - 
builder - INFO - Downloading nginx src...
builder - INFO - Running command: rpm --upgrade --verbose --hash http://nginx.org/packages/mainline/centos/7/SRPMS/nginx-1.13.9-1.el7_4.ngx.src.rpm
warning: /var/tmp/rpm-tmp.LZDJZS: Header V4 RSA/SHA1 Signature, key ID 7bd9bf62: NOKEY
warning: user builder does not exist - using root
warning: group builder does not exist - using root
builder - INFO - Retrieving http://nginx.org/packages/mainline/centos/7/SRPMS/nginx-1.13.9-1.el7_4.ngx.src.rpm
Updating / installing...
nginx-1:1.13.9-1.el7_4.ngx            ########################################

builder - INFO - Downloading 3d-party modules...
builder - INFO - /usr/src/nginx/modules dir already exist
builder - INFO - Module nginx-auth-ldap will download by branch
builder - ERROR - Cmd('git') failed due to: exit code(128)
  cmdline: git clone --branch=master -v https://github.com/kvspb/nginx-auth-ldap.git /usr/src/nginx/modules/nginx-auth-ldap
  stderr: 'fatal: destination path '/usr/src/nginx/modules/nginx-auth-ldap' already exists and is not an empty directory.
'

Question: Additional dependencies for build

What can I do if module require some library unpacked in module dir?
For example pagespeed require psol (I need download psol-${version}-x64.tar.gz and unpack it to pagespeed module folder)

Can I build modules as dynamic and pack it to separate rpm / deb packages?

builder - ERROR - [Errno 2] No such file or directory

While build, i got error:

.................
ix -I /root/rpmbuild/SOURCES/modules/ngx_devel_kit/objs -I objs/addon/ndk -I objs \
        -o objs/ngx_modules.o \
        objs/ngx_modules.c
sed -e "s|%%PREFIX%%|/etc/nginx|" \
        -e "s|%%PID_PATH%%|/var/run/nginx.pid|" \
        -e "s|%%CONF_PATH%%|/etc/nginx/nginx.conf|" \
        -e "s|%%ERROR_LOG_PATH%%|/var/log/nginx/error.log|" \
        < man/nginx.8 > objs/nginx.8
make[1]: Leaving directory '/root/rpmbuild/BUILD/nginx-1.19.3'


RPM build errors:

builder - ERROR - [Errno 2] No such file or directory: '/root/rpmbuild/RPMS/x86_64'

command:
./main.py build -f config.yaml

Build inside docker:
docker run --rm -it -v $(pwd):/nginx-builder:rw tinkoffcreditsystems/nginx-builder:centos-latest /bin/bash
OS: centos
cat config.yaml

nginx_version: 1.19.3
output_package: rpm
modules:
  - module:
      name: nginx-auth-ldap
      git_url: https://github.com/kvspb/nginx-auth-ldap.git
      git_branch: master
      dependencies:
        - openldap-devel
#  - module:
#      name: nginx-module-vts
#      git_url: https://github.com/vozlt/nginx-module-vts.git
#      git_branch: master
  - module:
      name: ngx_http_substitutions_filter_module
      git_url: https://github.com/yaoweibin/ngx_http_substitutions_filter_module.git
  - module:
      name: headers-more-nginx-module
      web_url: https://github.com/openresty/headers-more-nginx-module/archive/v0.261.zip
  - module:
      name: nginx-module-vts
      git_url: https://github.com/vozlt/nginx-module-vts.git
      git_tag: v0.1.18
  - module:
      name: ngx_devel_kit
      git_url: https://github.com/simplresty/ngx_devel_kit.git
      git_tag: v0.3.0
  - module:
      name: ngx_cache_purge
      git_url: https://github.com/FRiCKLE/ngx_cache_purge.git
  - module:
      name: ngx_http_dyups_module
      git_url: https://github.com/yzprofile/ngx_http_dyups_module.git
  - module:
      name: http_gzip_static_module
      type: embedded
  - module:
      name: http_v2_module
      type: embedded
  - module:
      name: http_ssl_module
      type: embedded
  - module:
      name: http_addition_module
      type: embedded
  - module:
      name: http_auth_request_module
      type: embedded
  - module:
      name: http_gunzip_module
      type: embedded
  - module:
      name: http_gzip_static_module
      type: embedded
  - module:
      name: http_realip_module
      type: embedded
  - module:
      name: http_secure_link_module
      type: embedded
  - module:
      name: http_slice_module
      type: embedded
  - module:
      name: http_stub_status_module
      type: embedded
  - module:
      name: http_sub_module
      type: embedded
  - module:
      name: stream_realip_module
      type: embedded
  - module:
      name: stream_ssl_preread_module
      type: embedded
configure_params:
  - "--with-openssl-opt='enable-tls1_3'"

Support apply patch

Hello!
Thanks for Nginx-builder!
Please add support apply patch
For example:

nginx_version: 1.14.1
output_package: deb
modules:
  - module:
      name: nginx_upstream_check_module
      git_url: https://github.com/yaoweibin/nginx_upstream_check_module.git
      patch: https://github.com/yaoweibin/nginx_upstream_check_module/blob/master/check_1.14.0+.patch

Used by http_dav_module instead of nginx-dav-ext-module

Hello. I try build nginx with nginx-dav-ext-module.

config.yaml

nginx_version: 1.14.0
output_package: rpm
modules:
  - module:
      name: http_dav_module
      git_url: https://github.com/arut/nginx-dav-ext-module.git
      git_tag: v3.0.0
      type: embedded
      dependencies:
        - libxml2-devel
        - libxslt-devel

Build log:
https://gist.github.com/patsevanton/f767374cff5c9d6181ad0727e82490c5

nginx -V

nginx version: nginx/1.14.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) 
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/usr/lib/nginx/modules --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-debug --user=nginx --group=nginx --with-pcre-jit --with-compat --with-file-aio --with-threads --with-stream --with-cc-opt= --with-ld-opt= --add-module=/root/rpmbuild/SOURCES/modules/**http_dav_module** --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

nginx config /etc/nginx/conf.d/webdav.conf

server {
    listen 80;
    server_name _;
    location /upload {
        client_max_body_size 1g;
        root /usr/share/nginx/html/1;
        dav_methods PUT DELETE MKCOL COPY MOVE;
}

nginx -t

nginx: [emerg] unknown directive "dav_methods" in /etc/nginx/conf.d/webdav.conf:8
nginx: configuration file /etc/nginx/nginx.conf test failed

добавить опцию "зачистки поляны"

Последующие сборки спотыкаются об оставшиеся от предыдущих артефакты (исходники, пакеты).
1 Хочется иметь возможность все это зачищать (например ./main.py clear -f config.yaml).
2 Хочется иметь возможность "не спотыкаться" об имеющиеся исходники и перезаписывать пакет.

Can not build when using 'local_url'

Describe the bug
Can not build when using 'local_url'
If this archive is on a web (s3) and using 'web_url' it builds ok.

To Reproduce
config example:
nginx_version: 1.14.0
output_package: deb
modules:

  • module:
    name: ngx_pagespeed
    local_url: /root/ngx_pagespeed/incubator-pagespeed-ngx-1.13.35.2-stable.tar.gz
    dependencies:
    - debhelper
    - dh-systemd
    - quilt
    - libssl-dev
    - libpcre3-dev
    - zlib1g-dev
    - uuid-dev

Screenshots
builder - INFO - Parse yaml file: config.yaml
builder - INFO - Download scripts for build deb package
builder - INFO - Downloading nginx src...
builder - INFO - --> http://nginx.org/download/nginx-1.14.0.tar.gz
builder - INFO - Downloading 3d-party modules...
builder - ERROR - 'str' object is not callable

Desktop (please complete the following information):

  • OS: Ubuntu 18.04

nginx: [emerg] unknown directive "dav_methods"

Config

server {
    listen 80;
    server_name _;
    location /upload {
        client_max_body_size 1g;
        root /usr/share/nginx/html/1;
        dav_methods PUT DELETE MKCOL COPY MOVE;
}

Status nginx

-- Unit nginx.service has begun starting up.
nginx[15509]: nginx: [emerg] unknown directive "dav_methods" in /etc/nginx/conf.d/webdav.conf:13
systemd[1]: nginx.service: control process exited, code=exited status=1
systemd[1]: Failed to start nginx - high performance web server.
-- Subject: Unit nginx.service has failed

config.yaml

nginx_version: 1.17.1
output_package: rpm
modules:
  - module:
      name: http_dav_module
      git_url: https://github.com/arut/nginx-dav-ext-module.git
      git_tag: v3.0.0
      dependencies:
        - libxml2-devel
        - libxslt-devel
    # static module
    $ ./configure --with-http_dav_module --add-module=/path/to/nginx-dav-ext-module

Log

builder - INFO - Module dir /usr/src/nginx/modules/http_dav_module
builder - INFO - -- Done: http_dav_module
builder - INFO - Download dependencies
builder - INFO - Running command: yum install --assumeyes libxslt-devel libxml2-devel
builder - INFO - Loaded plugins: fastestmirror, fusioninventory-agent
Loading mirror speeds from cached hostfile
Package libxslt-devel-1.1.28-5.el7.x86_64 already installed and latest version
Package libxml2-devel-2.9.1-6.el7_2.3.x86_64 already installed and latest version
Nothing to do

builder - INFO - Building .rpm package
builder - INFO - Running command: rpm --eval '%{_topdir}'
builder - INFO - /root/rpmbuild

builder - INFO - Running command: rpmbuild -bb nginx.spec
+ umask 022
+ cd /root/rpmbuild/BUILD
+ cd /root/rpmbuild/BUILD
+ rm -rf nginx-1.17.1
+ /usr/bin/tar -xf -
+ /usr/bin/gzip -dc /root/rpmbuild/SOURCES/nginx-1.17.1.tar.gz
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd nginx-1.17.1
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ cp /root/rpmbuild/SOURCES/nginx.init.in .
+ sed -e 's|%DEFAULTSTART%|2 3 4 5|g' -e 's|%DEFAULTSTOP%|0 1 6|g' -e 's|%PROVIDES%|nginx|g'
+ sed -e 's|%DEFAULTSTART%||g' -e 's|%DEFAULTSTOP%|0 1 2 3 4 5 6|g' -e 's|%PROVIDES%|nginx-debug|g'
+ exit 0
+ umask 022
+ cd /root/rpmbuild/BUILD
+ cd nginx-1.17.1
+++ pcre-config --cflags
++ echo -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
+ ./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --modules-path=/usr/lib/nginx/modules --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-debug --user=nginx --group=nginx --with-http_gzip_static_module --with-http_v2_module --with-http_ssl_module --with-pcre-jit --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_gunzip_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_stub_status_module --with-http_sub_module --with-stream --with-stream_realip_module --with-stream_ssl_preread_module --with-cc-opt= --with-ld-opt= --add-module=/root/rpmbuild/SOURCES/modules/http_dav_module '--with-cc-opt=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' '--with-ld-opt=-Wl,-z,relro -Wl,-z,now -pie' --with-debug

git error: already exists and is not an empty directory.

After second run inside docker container i got error:

builder - ERROR - Cmd('git') failed due to: exit code(128)
  cmdline: git clone --branch=master -v https://github.com/yaoweibin/ngx_http_substitutions_filter_module.git /usr/src/nginx/modules/ngx_http_substitutions_filter_module
  stderr: 'fatal: destination path '/usr/src/nginx/modules/ngx_http_substitutions_filter_module' already exists and is not an empty directory.
'

config.yaml

nginx_version: 1.19.3
output_package: rpm
modules:
  - module:
      name: nginx-auth-ldap
      git_url: https://github.com/kvspb/nginx-auth-ldap.git
      git_branch: master
      dependencies:
        - openldap-devel
  - module:
      name: nginx-module-vts
      git_url: https://github.com/vozlt/nginx-module-vts.git
      git_branch: master
  - module:
      name: ngx_http_substitutions_filter_module
      git_url: https://github.com/yaoweibin/ngx_http_substitutions_filter_module.git
  - module:
      name: headers-more-nginx-module
      web_url: https://github.com/openresty/headers-more-nginx-module/archive/v0.261.zip
  - module:
      name: nginx-module-vts
      git_url: https://github.com/vozlt/nginx-module-vts.git
      git_tag: v0.1.18
  - module:
      name: ngx_devel_kit
      git_url: https://github.com/simplresty/ngx_devel_kit.git
      git_tag: v0.3.0
  - module:
      name: ngx_cache_purge
      git_url: https://github.com/FRiCKLE/ngx_cache_purge.git
  - module:
      name: ngx_http_dyups_module
      git_url: https://github.com/yzprofile/ngx_http_dyups_module.git
  - module:
      name: http_gzip_static_module
      type: embedded
  - module:
      name: http_v2_module
      type: embedded
  - module:
      name: http_ssl_module
      type: embedded
  - module:
      name: http_addition_module
      type: embedded
  - module:
      name: http_auth_request_module
      type: embedded
  - module:
      name: http_gunzip_module
      type: embedded
  - module:
      name: http_gzip_static_module
      type: embedded
  - module:
      name: http_realip_module
      type: embedded
  - module:
      name: http_secure_link_module
      type: embedded
  - module:
      name: http_slice_module
      type: embedded
  - module:
      name: http_stub_status_module
      type: embedded
  - module:
      name: http_sub_module
      type: embedded
  - module:
      name: stream_realip_module
      type: embedded
  - module:
      name: stream_ssl_preread_module
      type: embedded
configure_params:
  - "--with-openssl-opt='enable-tls1_3'"

Run inside docker:
./main.py build -f config.yaml -r r01

OS: centos

Cannot find nginx source rpm(SRPM) with version 1.16.0 in url http://nginx.org/packages/mainline/centos/7/SRPMS

Пытаюсь собрать 16 версию nginx

cat config.yaml 
nginx_version: 1.16.0
output_package: rpm
modules:
  - module:
      name: nginx-dav-ext-module
      git_url: https://github.com/arut/nginx-dav-ext-module.git
      git_tag: v3.0.0
./main.py build -f config.yaml 
builder - INFO - Parse yaml file: config.yaml
builder - INFO - Download scripts for build rpm package
builder - INFO - Running command: rpmdev-setuptree
builder - INFO - 
builder - INFO - Downloading nginx src...
builder - ERROR - Cannot find nginx source rpm(SRPM) with version 1.16.0 in url http://nginx.org/packages/mainline/centos/7/SRPMS

Может добавить использование https://nginx.org/download/nginx-1.16.0.tar.gz ?
Например, код взять из https://nginx.org/download/nginx-1.16.0.tar.gz, а остальные файлы с с других SRPM ( http://nginx.org/packages/mainline/centos/7/SRPMS/nginx-1.15.9-1.el7_4.ngx.src.rpm )?

неверный формат архива tar.gz

При попытке сборки ловлю сообщение:

~/Nginx-builder$ ./main.py build -f banana-config.yaml
builder - INFO - Parse yaml file: banana-config.yaml
builder - INFO - /usr/src/nginx dir already exist
builder - INFO - Download scripts for build deb package
builder - INFO - Downloading nginx src...
builder - INFO - --> http://nginx.org/download/nginx-1.16.tar.gz
builder - INFO - Downloading 3d-party modules...
builder - INFO - /usr/src/nginx/modules dir already exist
builder - INFO - Module nginx_cookie_flag will download by branch
builder - INFO - -- Done: nginx_cookie_flag
builder - INFO - Downloading dependencies
builder - INFO - Building .deb package
builder - ERROR - Archive format nginx-1.16.tar.gz is not valid
builder - ERROR - 'NoneType' object has no attribute 'rstrip'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.