GithubHelp home page GithubHelp logo

container-common-scripts's Introduction

Common build helpers for sclorg containers

This repository is aimed to be added as git submodule into particular containers' source repositories. By default, the path to submodule should be named 'common'.

Usage

This section explains the usage of the shared scripts in this repository when it is used as a submodule in a container source repository.

Once you have the repository set as a submodule include common.mk into your root Makefile in order to access the default rules used to call shared scripts.

Default rules:

make or make build This rule will build an image without tagging it with any tags after it is built. If the builder is podman, then the image would also be squashed by default during the build. make build will also expect a README.md so that it can transform it into a man page that gets added to the image so make sure it is available and that you have the go-md2man tool installed on your host.

make tag Use this rule if you want to tag an image after it is built. It will be tagged with two tags - name:latest and name:version. Depends on build

make test or make check This rule will run the testsuite scripts contained in the container source repositories. It expects the test to be available at $gitroot/$version/test/run Depends on tag as some tests might need to have the images tagged (s2i).

make test-openshift Similar to make test but runs testsuite for Openshift 3, expected to be found at $gitroot/$version/test/run-openshift

make test-openshift-4 Similar to make test but runs testsuite for Openshift 4, expected to be found at $gitroot/$version/test/run-openshift-remote-cluster

make betka Runs script betka.sh that generates sources for the dist-git repo. Only Fedora, RHEL7 and RHEL8 are supported. For source generation into Fedora or RHEL dist-git repositories, some parameters are mandatory. DOCKER_IMAGE parameter for Fedora case is quay.io/rhscl/cwt-generator, for RHEL world please ask [email protected], [email protected], or [email protected].

E.g. command for the source generation into Fedora dist-repo https://src.fedoraproject.org/container/nodejs into main branch is: make betka TARGET=fedora VERSIONS=12

The sources are not generated directly into dist-git repository, but into created results directory.

make clean Runs scripts that clean-up the working dir. Depends on the clean-images rule by default and additional clean rules can be provided through the clean-hook variable.

make clean-images Best-effort to remove the last set of images that have been built using the scripts.

make shellcheck Check the shell syntax of the files specified by $SHELLCHECK_FILES variable. See SHELLCHECK_FILES variable description below for more info (default is .). The files matching this specification are then filtered, to not show results twice for symlinks. Only files with a suffix .sh or shell shebang are scanned with the shellcheck utility. See run-shellcheck.sh in this repo for more detailed info. Once the shell syntax issues are fixed, CI that runs make shellcheck for each PR can be turned on by putting .travis.yml file into the root of the image's repository. .travis.yml for its content.

There are additional variables that you can use that the default rules are prepared to work with:

VERSIONS Names of the directories in which the Dockerfiles are contained. Needs to be defined in your Dockerfile for the scripts to know which versions to build.

OS OS version you want to build the images for. Currently the scripts are able to build for centos (default), centos6, c8s, c9s, rhel7, rhel8, rhel9, and fedora.

SKIP_SQUASH When set to 1 the build script will skip the squash phase of the build.

CUSTOM_REPO Set this variable to the path to your local .repo files you want to have available inside the image while building. Useful for building rhel-based images on an unsubscribed box. Be aware that you cannot write to any .repo files used this way inside the image as they will be mounted into the image as read-only.

DOCKER_BUILD_CONTEXT Use this variable in case you want to have a different context for your builds. By default the context of the build is the versioned directory the Dockerfiles are contained in.

SHELLCHECK_FILES One or more files or directories to be scanned by the shellcheck, default is ., which means a whole repository directory. If a directory is provided then all of its content is scanned as well.

CT_OCP4_TEST Set to true if you want to test container in OpenShift 4 environment.

CVP Set to true if you want to test container in Container Validation Pipeline environment.

clean-hook Append Makefile rules to this variable to make sure additional cleaning actions are run when make clean is called.

Files affecting behaviour

The directory with Dockerfiles can optionally include some specific files that affect how scripts in this repo work:

.exclude-<OS> If this file exists, the tooling will not run the build and tests for the specific Dockerfile. For example, if .exclude-rhel7 file exists, the Dockerfile.rhel7 will not be expected in the same directory, build and tests will be skipped. Content of the file is not important at this point.

.devel-repo-<OS> This file is useful if we need to work with RPMs that are not available publically yet. Content of the file is not important at this point. If such a file exists in the repository, then the building scripts will take a look at a correspondent variable, e.g. DEVEL_REPO_rhel8, and will use the repository file defined by that variable. That means that definition of the DEVEL_REPO_rhel8 variable is a responsibility of the test/CI environment.

.build-args-<OS> Some images require some specific build options, let them be set in a file specific for a Dockerfile. This is useful for example for setting capabilities for micro images, that install RPMs into an alternative directory and podman versions of 4.4.0+ do not set such capabilities by default.

Regression tests

make check Runs the tests of few images that use this set of scripts. If the tests of those images pass, this repo is considered to be working.

make shellcheck Check the shell syntax of all *.sh files tracked by the git in this repository.

Dependencies for testsuite:

  • /usr/bin/docker (either docker or podman + podman-docker)
  • git
  • go-md2man
  • make
  • source-to-image
  • shellcheck

container-common-scripts's People

Contributors

bparees avatar caruccio avatar csrwng avatar dhodovsk avatar ewolinetz avatar fila43 avatar frenzymadness avatar gabemontero avatar hhorak avatar jamacku avatar jbornemann avatar jhadvig avatar mcyprian avatar mfojtik avatar michal-josef-spacek avatar mnagy avatar mohammedzee1000 avatar phracek avatar pkubatrh avatar praiskup avatar pvalena avatar rhcarvalho avatar ryanj avatar soltysh avatar stevekuznetsov avatar tomastomecek avatar torsava avatar trepik avatar yrro avatar zmiklank avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

container-common-scripts's Issues

Distgen does different stuff when called several times

This is quite suspicious, that calling distgen few times in a row results in different output every time. I smell timestamps on files might be the cause here.

(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  DG     9.6/cccp.yml
  DG     9.6/root/usr/share/container-scripts/postgresql/README.md
  DG     9.6/root/usr/share/container-scripts/postgresql/common.sh
  DG     9.6/root/usr/bin/run-postgresql-slave
  DG     9.6/root/usr/share/container-scripts/postgresql/openshift-custom-recovery.conf.template
  DG     10/cccp.yml
  DG     10/root/usr/share/container-scripts/postgresql/README.md
  DG     10/root/usr/share/container-scripts/postgresql/common.sh
  DG     10/root/usr/bin/run-postgresql-slave
  DG     10/root/usr/share/container-scripts/postgresql/openshift-custom-recovery.conf.template
  DG     11/cccp.yml
  DG     11/root/usr/share/container-scripts/postgresql/README.md
  DG     11/root/usr/share/container-scripts/postgresql/common.sh
  DG     11/root/usr/bin/run-postgresql-slave
  DG     11/root/usr/share/container-scripts/postgresql/openshift-custom-recovery.conf.template
  CP     9.6/root/usr/libexec/fix-permissions
  CP     9.6/content_sets.yml
  CP     9.6/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql-replication.conf.template
  CP     9.6/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql.conf.template
  CP     9.6/root/usr/share/container-scripts/postgresql/scl_enable
  CP     9.6/root/usr/bin/run-postgresql
  CP     9.6/root/usr/bin/run-postgresql-master
  CP     9.6/root/usr/bin/container-entrypoint
  CP     9.6/root/usr/bin/usage
  CP     9.6/root/usr/libexec/check-container
  CP     9.6/root/usr/share/container-scripts/postgresql/start/set_passwords.sh
  CP     9.6/s2i/bin/assemble
  CP     9.6/s2i/bin/usage
  CP     10/root/usr/libexec/fix-permissions
  CP     10/content_sets.yml
  CP     10/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql-replication.conf.template
  CP     10/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql.conf.template
  CP     10/root/usr/share/container-scripts/postgresql/scl_enable
  CP     10/root/usr/bin/run-postgresql
  CP     10/root/usr/bin/run-postgresql-master
  CP     10/root/usr/bin/container-entrypoint
  CP     10/root/usr/bin/usage
  CP     10/root/usr/libexec/check-container
  CP     10/root/usr/share/container-scripts/postgresql/start/set_passwords.sh
  CP     10/s2i/bin/assemble
  CP     10/s2i/bin/usage
  CP     11/root/usr/libexec/fix-permissions
  CP     11/content_sets.yml
  CP     11/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql-replication.conf.template
  CP     11/root/usr/share/container-scripts/postgresql/openshift-custom-postgresql.conf.template
  CP     11/root/usr/share/container-scripts/postgresql/scl_enable
  CP     11/root/usr/bin/run-postgresql
  CP     11/root/usr/bin/run-postgresql-master
  CP     11/root/usr/bin/container-entrypoint
  CP     11/root/usr/bin/usage
  CP     11/root/usr/libexec/check-container
  CP     11/root/usr/share/container-scripts/postgresql/start/set_passwords.sh
  CP     11/s2i/bin/assemble
  CP     11/s2i/bin/usage
  LN     9.6/README.md
  LN     9.6/test
  LN     9.6/s2i/bin/run
  LN     10/README.md
  LN     10/test
  LN     10/s2i/bin/run
  LN     11/README.md
  LN     11/test
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk
(python36-tools) [root@cont-ci-rhel7-00343358-20200526-161859 postgresql-container]# make generate
MANIFEST_FILE="manifest.sh" \
VERSIONS="9.6 10 11 12" \
DG="/bin/dg" /usr/bin/env bash common/generate.sh
  LN     9.6/s2i/bin/run
  LN     10/s2i/bin/run
  LN     11/s2i/bin/run
  LN     12/s2i/bin/run
rm auto_targets.mk

Symlink loop

I'm using Vagrant for testing and it rejects to sync common scripts to the virtual machine (using rsync) because there is an infinite loop of symlinks: common/tests/failures/check contains common which is a symlink back to ../../.. which causes this loop:

common/tests/failures/check/common/tests/failures/check/common/tests/failures/check/common/tests/failures/check/common…

Is this something we need to have here?

`latest` directory in container git repositories?

It is really ugly that we loose the git history all the time with new versions of our containers, speaking in examples - consider PostgreSQL container:

  1. working with 9.2 directory, we iterated a lot (git blame and git history gives us useful info
  2. but then we needed to work on 9.4, so we copied and pasted whole 9.2 directory into 9.4, which means that the whole 9.4 directory lost all the useful commit history
  3. moving to 9.5 made this even worse, and upcoming 9.6 won't help obviously

Well, e.g. in Fedora dist-git, we have master branch which guarantees that all the necessary info for git blame persists. It is fair also to keep the "authorship" of contributions.

Perhaps we should have e.g. latest directory now (instead of 9.6) and make the 9.6 be a symbolic link to latest?

test_from_dockerfile is broken

I've broken test_from_dockerfile by the implementation of the branch name support in b3d5a0a

It seems that the systems our CI is running on are too old and the readarray command there doesn't provide -d option. From my testing, the option is supported in bash version 5 but not in version 4.

I'll try to fix it asap. Please do not update in the meantime.

Unable to generate files for nondefault OS versions

I am looking into the new generate/generate-all makefile rules and I realized there is a DG_CONF and conf value hardcoded in common.mk and generate.sh file as well. It is not even possible to change it using some environment variable. These scripts support only Fedora 27, s2i-python-container, postgresql-container and maybe some other projects contains only Fedora 26 Dockerfile, which means these files can't be properly generated using generate rule from common scripts.

RHEL7 build issue: installing docker_py=1.10.6 fails

Seems like this is result of some work-around from 5fe4cbb commit. But explicitly installing v1.10.6 fails for me with:

$ easy_install -q --user docker_py==1.10.6 docker-squash==1.0.5
error: Setup script exited with error in docker-py setup command: Invalid environment marker: python_version < "3.5"

OTOH, easy_install -q --user docker-squash==1.0.5 works just fine. Do we still need explicit version of docker_py?

issue with make syntax error

##Description

When running make, I get an syntax error in common/build.sh

Versions

Using OSX 10.13.2
Tried both ZSH & Bash
GNU bash, version 4.4.12(1)-release (x86_64-apple-darwin17.0.0)
GNU Make 4.2.1 Built for x86_64-apple-darwin17.3.0

$ make build TARGET=rhel7 VERSIONS=7.1

VERSION="7.1" SKIP_SQUASH=0 UPDATE_BASE= OS=rhel7 CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="5.5" common/build.sh
common/build.sh: line 55: syntax error near unexpected token `{stdout_fd}'
make[1]: *** [common/common.mk:53: 7.1] Error 2
make[1]: Leaving directory '/Users/bobby/code/s2i-php-container'
make: *** [common/common.mk:42: build-serial] Error 2

$ make check

VERSION="5.6" SKIP_SQUASH=1 UPDATE_BASE= OS=centos7 CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="5.5" common/build.sh
common/build.sh: line 55: syntax error near unexpected token `{stdout_fd}'
make[1]: *** [common/common.mk:53: 5.6] Error 2
make[1]: Leaving directory '/Users/bobby/code/s2i-php-container'
make: *** [common/common.mk:42: build-serial] Error 2

Hmm

I've tried every syntax checker there is, and it all looks fine - as it probably is given others are using it. But don't know why this is happening?

parse_output ()
{
  local command=$1 filter=$2 var=$3 stream=$4
  local raw_output= rc=0
  {
      raw_output=$(_parse_output_inner)
  } {stdout_fd}>&1 {stderr_fd}>&2
  rc=$?
  eval "$var=\$raw_output"
  (exit $rc)
}

Any clues? :/

Implement $(clean-hook) variable

So anyone can hook some custom target into 'make clean', like:

include common/common.mk 
clean-hook += my_clean

(naming by automake)

Avoid the `Dockerfile.rhel7` file, and use just `Dockerfile` if distgen is used

There are ugly hacks in the distgen logic related to *.rhel7 lsuffix (e.g. #75 fights with that), but there's no easy way to drop the old layout because it is required by:

  • non distgen-enabled images
  • common script scripts
  • rebuild helper
  • CI (probably)

So we would have to either (a) move all containers towards the new layout, or (b) support two layouts. IMO, (b) is less intrusive (using generators in source repo should stay opt-in) and I would prefer that. But anyways, take this issue as a tracker and place to discuss issues.

Additional note to #75, it would be much more natural if we generated (distgen layout) just:

postgresql-container/
└── output/
    ├── Dockerfile
    └── root/

Or, on demand (e.g. to file the generated branch):

postgresql-container/
├── 10
│   ├── centos-7-x86_64
│   │   ├── Dockerfile
│   │   └── root
│   └── rhel-7-x86_64
│       ├── Dockerfile
│       └── root/
...

`make test-openshift` shouldn't configure contributor's box

.. and simply expect that OpenShift is installed on the box, or fail (a bit related to #40).

It would be really suspicious to require our end-users/contributors to run the openshift test-scripts "locally" ATM, since the tasks require root account. We could have some script for user's convenience (doing the oc installation), but the images in CI should be as much prepared as possible, same as user's boxes -- and we should only provide the login info into the openshift instance.

A general test for npm/nodejs

We should have a simple test that would be called for images where we expect nodejs and npm to be available. Something like npm install react, npm --version, ...

provide a way to build RHEL images on Fedora box

.. and also on not subscribed RHEL boxes (so far, docker build for RHEL images works
only on successfully subscribed RHEL7 box).

Per @hhorak report, it is possible to (optionally) provide -v /host.repo:/etc/yum.repos.d/host.repo option for docker build . command.

Add tests for this repo

By changing this repo we should not directly influence other images, since they contain specific references to this repo and updating those references should be tested on each-image basis. It might still make sense to run builds of other images (not necessarily tests for those, since this repo contains mostly build-time only things) -- just to see how much harm a particular PR can cause in the future, which should help creating the PR without requiring to change code in depended repositories.

Add versioning to Fedora image names

Up until now we did not have an issue with different versions for the same Fedora release as the version was tied to the release (we could not have had eg. both nodejs 10 and 12 on Fedora XY).
This has however changed with modularity and we need to update the image tags to reflect the possibility.

Recently hit in: sclorg/s2i-nodejs-container#246

Easier debugging ideas

Brain dump of some ideas how to make the debugging easier:

  • print docker/podman version at the beginning of the test (plus some other related packages)
  • do some performance check at the beginning (how quickly network works by pulling some small image and installing a package; how quickly a container starts) -- seeing bigger numbers might indicate that the issues are caused by infra performance
  • add possibility to not dispose a machine immediately -- some message like [keep-alive: 2h] in the github comment could make the machine be kept a little longer after the test is done (see https://plugins.jenkins.io/ghprb/ for how to work with the comments)
  • similarly as the point above [debug] could turn on the debug (more verbose) output for the tests

Fix CVP pipeline issues

During the testing in CVP this issue was caught:

Testing in CVP environment. No need to login to OpenShift cluster. This is already done by CVP pipeline.
In project cvp-5c79a57dd70cc51dd4c37825-jlgfj (CVP sandbox) (cvp-5c79a57dd70cc51dd4c37825-jlgfj) on server https://172.27.0.1:443

You have no services, deployment configs, or build configs.
Run 'oc new-app' to create an application.
Testing in CVP environment. No need to create OpenShift project. This is done by CVP pipeline
Testing image nodejs-14 in CVP pipeline.
error: only a partial match was found for "nodejs-14:14": "cvp-5c79a57dd70cc51dd4c37825-jlgfj/nodejs-14:latest"

The argument "nodejs-14:14" only partially matched the following Docker image, OpenShift image stream, or template:

* Image stream "nodejs-14" (tag "latest") in project "cvp-5c79a57dd70cc51dd4c37825-jlgfj"
  Use --image-stream="cvp-5c79a57dd70cc51dd4c37825-jlgfj/nodejs-14:latest" to specify this image or template

Waiting for nodejs-14-testing pod becoming ready .................................................................................................... FAIL
NAME                                       IMAGE REPOSITORY                                                                                TAGS      UPDATED
imagestream.image.openshift.io/nodejs-14   image-registry.openshift-image-registry.svc:5000/cvp-5c79a57dd70cc51dd4c37825-jlgfj/nodejs-14   latest    5 minutes ago
In project cvp-5c79a57dd70cc51dd4c37825-jlgfj (CVP sandbox) (cvp-5c79a57dd70cc51dd4c37825-jlgfj) on server https://172.27.0.1:443

You have no services, deployment configs, or build configs.
Run 'oc new-app' to create an application.
Build config bc/ does not exist for some reason.
Import probably failed.
OpenShift tests for nodejs-14 failed.

There are two possibilities:

  1. In case of CVP, we have to use tag latest.
  2. During the import into OpenShift 4, we can tag it to nodejs-14:14.
    The taging can be done in file: https://github.com/sclorg/container-common-scripts/blob/master/test-openshift.yaml
    especially after this ansible command:
    https://github.com/sclorg/container-common-scripts/blob/master/test-openshift.yaml#L53

In my opinion, the first option is better.

WDYT?

Pull parent images for build from quay.io

We can see errors during build like this:

https://ci.centos.org/job/SCLo-container-container-common-scripts-test/202/console

22:24:55 + cd sources
22:24:55 + make check
22:24:55   RUN TEST 'path_foreach'
22:24:55   RUN TEST 'random_string'
22:24:55   RUN TEST 'test_npm'
22:24:55 + . test-lib.sh
22:24:55 ++ EXPECTED_EXIT_CODE=0
22:24:55 + NPM_REGISTRY=
22:24:55 ++ ct_build_s2i_npm_variables
22:24:55 ++ npm_variables=
22:24:55 ++ '[' -n '' ']'
22:24:55 ++ echo ''
22:24:55 + output=
22:24:55 + test x == x
22:24:55 + ca_file=/etc/pki/ca-trust/source/anchors/RH-IT-Root-CA.crt
22:24:55 + NPM_REGISTRY=https://foobar.registry.org
22:24:55 + '[' -f /etc/pki/ca-trust/source/anchors/RH-IT-Root-CA.crt ']'
22:24:55   RUN TEST 'image_availability'
22:25:50 quay.io/centos7/ruby-24-centos7 could not be downloaded via 'docker'
22:25:50 image_availability test completed successfully.
22:25:50   RUN TEST 'public_image_name'
22:25:50 public_image_name test completed successfully.
22:25:50 cd tests/failures/check && make tag && ! make check && make clean
22:25:50 make[1]: Entering directory `/root/sources/tests/failures/check'
22:25:50 make[2]: Entering directory `/root/sources/tests/failures/check'
22:25:50 mkdir -p v0/root
22:25:50 go-md2man -in "v0/README.md" -out "v0/root/help.1"
22:25:50 chmod a+r "v0/root/help.1"
22:25:50 VERSION="v0" SKIP_SQUASH=1 UPDATE_BASE= OS=centos7 CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="" CUSTOM_REPO="" REGISTRY=""quay.io/"" /usr/bin/env bash common/build.sh
22:25:50 -> Version v0: building image from 'Dockerfile' ...
22:25:50 -> Pulling image centos/s2i-core-centos7 before building image from Dockerfile.
22:25:50 Using default tag: latest
22:25:50 Trying to pull repository docker.io/centos/s2i-core-centos7 ... 
22:25:51 toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
22:25:51 Pulling image centos/s2i-core-centos7 failed.
22:25:51 Let's wait 5 seconds and try again.
22:25:56 Using default tag: latest
22:25:56 Trying to pull repository docker.io/centos/s2i-core-centos7 ... 
22:25:56 toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

squash.py not working on RHEL8

The squash.py script fails on RHEL8 as it is using docker_squash but we are using podman on RHEL8.

Not sure if we even need to squash the images during the tests in this repository, so disabling the squashing phase would be an easy way to work around this issue.

re-run docker build in case of HTTP 500 failure

Sometimes, we got an infrastructure error like this:

https://ci.centos.org/job/SCLo-container-python-rh-fedora/281/console

16:44:11 VERSION="3.7" SKIP_SQUASH=1 UPDATE_BASE=1 OS=fedora CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="" CUSTOM_REPO="" /usr/bin/env bash common/build.sh
16:44:11 -> Version 3.7: building image from 'Dockerfile.fedora' ...
16:44:11 Sending build context to Docker daemon 94.72 kB

16:44:11 Step 1/14 : FROM registry.fedoraproject.org/f31/s2i-base:latest
16:44:11 Trying to pull repository registry.fedoraproject.org/f31/s2i-base ... 
16:44:12 latest: Pulling from registry.fedoraproject.org/f31/s2i-base
16:44:12 82422234ceca: Pulling fs layer
16:44:12 64f48977383f: Pulling fs layer
16:44:12 c8ba9d4c9a9a: Pulling fs layer
16:44:12 error pulling image configuration: received unexpected HTTP status: 503 Service Unavailable
16:44:12 make[1]: Leaving directory `/root/sources'
16:44:12 make[1]: *** [3.7] Error 1
16:44:12 make: *** [build-serial] Error 2
16:44:12 Build step 'Execute shell' marked build as failure
16:44:12 [PostBuildScript] - Executing post build scripts.
16:44:12 [SCLo-container-python-rh-fedora] $ /bin/bash /tmp/jenkins2251871663693525052.sh
16:44:12 Warning: Permanently added 'n37.pufty.ci.centos.org,172.19.3.101' (ECDSA) to the list of known hosts.
16:44:12 [PostBuildScript] - Executing post build scripts.

This can be solved in this code: https://github.com/sclorg/container-common-scripts/blob/master/build.sh#L111
I would say, try it two times, not more.

avoid the 'sleep N' race conditions

E.g. #65, #62. Basically, OpenShift isn't providing enough API for avoiding such hacks, but that should at least be discussed with openshift upstream so we know how to do proper scripting correctly. Reading the openshift testing library (edit: i mean test-lib-openshift.sh), I more and more think that the library should be more than just testing library, but rather convenient library for daily work with openshift...

Find Umask issue soon

With umask set to 077, the image build and test might fail. While we might not fix this properly, it would be good to find out that the umask is the cause soon by having explicit check for umask during:

  • make build
  • a new function in test-lib.sh to be called in every test/run

Image does not get tagged after build

I'm not sure if this can be caused by mine setup. It started to behave like this only recently.

$ basename "`pwd`"
s2i-ruby-container

I have pulled most recent common scripts(a8d2885) and I'm on subscribed RHEL7 VM.

$ PATH="$PATH:~/.local/bin" make build TARGET=rhel7 VERSIONS=2.4 TEST_MODE=yes
 . . .
2017-09-20 16:19:59,739 root         INFO     New squashed image ID is c92ecb5b50cb3337dd7256d28823bc2a134571cdd575546ec0d40dd8ab5cf60a
2017-09-20 16:20:17,347 root         INFO     Done
ruby 2.4 => c92ecb5b50cb3337dd7256d28823bc2a134571cdd575546ec0d40dd8ab5cf60a
make[1]: Leaving directory `/home/vagrant/Work/RH/centos/containers/s2i-ruby-container'

No tagged image is present, but it's found by ID (found in output above).

$ docker images -a | grep -E '(s2i|ruby|c92ecb5b50cb)'
<none>                                                      <none>              c92ecb5b50cb        About an hour ago   1.05 GB
registry.access.redhat.com/rhscl/s2i-base-rhel7             1                   d46c583c0e87        3 weeks ago         459.8 MB

Is this expected behavior?

Support centos6 builds

IIUC In order to maintain compatibility of c++ software with centos6, it is necessary to build it on centos6.

I've got patches for creating devtoolset images for centos6 but they require having support for that on this project.

Would it be possible to add this support so that adding a Dockerfile.centos6 file creates centos6 versions?

At your disposal for any suggestions/critiques/ideas

Show size of the image during test

It might be included in one of the general functions that is used in all containers (or supposed to be used that way). For example the ct_show_results.

Start and stop OpenShift 3 properly

The RHSCL CI is blocked when OpenShift 3.X does not start up.

We have to investigate how to start OpenShift 3. properly so tests are not blocked.

See.

Try PULL image more then one if service response HTTP 505

If the docker registry returns HTTP 503, I would prefer to run docker pull once more.
Sometimes, a second run works properly.

What do you think? @pkubatrh @hhorak @praiskup.

See the error:

make[1]: Entering directory `/tmp/daily_scl_tests/s2i-nodejs-container'
mkdir -p 10/root
go-md2man -in "10/README.md" -out "10/root/help.1"
chmod a+r "10/root/help.1"
VERSION="10" SKIP_SQUASH=1 UPDATE_BASE= OS=fedora CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="" CUSTOM_REPO="" /usr/bin/env bash common/build.sh
-> Version 10: building image from 'Dockerfile.fedora' ...
Sending build context to Docker daemon 50.69 kB

Step 1/13 : FROM registry.fedoraproject.org/f29/s2i-base:latest
Trying to pull repository registry.fedoraproject.org/f29/s2i-base ... 
latest: Pulling from registry.fedoraproject.org/f29/s2i-base
7692efc5f81c: Pulling fs layer
aaf5ad2e1aa3: Pulling fs layer
05ba4d604a56: Pulling fs layer
error pulling image configuration: received unexpected HTTP status: 503 Service Temporarily Unavailable
make[1]: *** [10] Error 1
make[1]: Leaving directory `/tmp/daily_scl_tests/s2i-nodejs-container'
make: *** [build-serial] Error 2

Remove temporary hidden files before build

Hidden files .image-id and .image-id.raw are not removed after build.

So in scenario:

  1. make test TARGET=centos7
  2. make test TARgET=fedora

Fedora is not supported for all versions. But the second point uses the hidden files from previous build and for some versions it tests centos based images.

Also for some images these hidden files are in .gitignore, so user can't be even aware that some old files needs to be cleaned.

Bug in test openshift

Tests in openshift are failing in python container with the latest version of common module.

16:44:12 Uploading image centos/python-27-centos7:2.7 as python-27-centos7:2.7:2.7
16:44:17 Error response from daemon: Get http://172.30.1.1:5000/v1/users/: dial tcp 172.30.1.1:5000: connect: connection refused
16:44:22 Login Succeeded
16:44:22 Error parsing reference: "172.30.1.1:5000/sclorg-test-31971/python-27-centos7:2.7:2.7" is not a valid repository/tag

See the image name with two tags python-27-centos7:2.7:2.7. The full log is here.

The commit 24a8d12 seems suspicious to me. Is it possible that the IMAGE_NAME variable already contains the tag and the ct_os_test_s2i_app_func function adds another one in image_tagged?

local image_name=${1}
local app=${2}
local context_dir=${3}
local check_command=${4}
local oc_args=${5:-}
local image_name_no_namespace=${image_name##*/}
local service_name="${image_name_no_namespace}-testing"
local image_tagged="${image_name_no_namespace}:${VERSION}"
local namespace

I've checkouted the submodule in sclorg/s2i-python-container#394 from 57d2c55 back to e65a5f5 and we'll see. The non-os tests there already failed but for a different reason.

Move common parts of the tests into a common Bash script

Many tests share similar or same parts, like:

  • cleaning the containers after running the test
  • running the container while storing the ID into a cidfile
  • checking the man help.1 file
  • ...

So we should eventually have at least set of functions shared in more containers to make the tests maintenance easier.

Testing upstream containers in OpenShift 4 environment

This issue is going to be used as a tracker for testing SCL containers in OpenShift 4 environment.
Please reference all PRs in container-common-scripts into this issue.

Tests are run by PR comment [test-openshift-4].

What should be done:

PR:

Proof of Concept: sclorg/s2i-php-container#286

reproducible_builds: ensure that `umask` doesn't affect the container build

  • git doesn't track file permissions, thus
  • git clone initiates the permissions according the user's umask value
  • docker build just copies the files as-is into container
  • especial problems are with root/usr directory created by ADD root / command, this affects the whole image (/usr directory might be unreadable)

In RPMs' specfiles, we explicitly set the permissions in %install phase (or by %attr in %files), though there's no such way in Dockerfile.

Missing Dockerfile should not make tests fail

Sometimes it happens that we have only RHEL packages available and CentOS are not yet build, but we want to submit the PR with new version. Or we might only have Fedora variant available, but no CentOS or RHEL Dockerfiles. So, the testing (building) script should work the way that the testing would not be run (and test whould not fail) in cases the particular Dockerfile variant is missing (e.g. only Dockerfile.rhel7 is available and not Dockerfile for CentOS).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.