When DC cannot be deployed it wait forever about osio-pipeline

[hrishin]

error logs

[Pipeline] stage
[Pipeline] { (Processing Template)
[Pipeline] sh
[workspace] Running shell script
+ set -u
+ set -e
+ for i in 'hshinde-{stage,run}'
+ oc delete all --all -n hshinde-stage
deploymentconfig "nodejs-health-check-osio-master" deleted
imagestream "nodejs-health-check-osio-master" deleted
imagestream "runtime-osio-master" deleted
route "nodejs-health-check-osio-master" deleted
pod "nodejs-health-check-osio-master-2-2zt6r" deleted
service "nodejs-health-check-osio-master" deleted
+ for i in 'hshinde-{stage,run}'
+ oc delete all --all -n hshinde-run
deploymentconfig "nodejs-health-check-osio-master" deleted
imagestream "nodejs-health-check-osio-master" deleted
imagestream "runtime-osio-master" deleted
route "nodejs-health-check-osio-master" deleted
pod "nodejs-health-check-osio-master-2-sp99x" deleted
service "nodejs-health-check-osio-master" deleted
+ for i in hshinde 'hshinde-{stage,run}'
+ oc apply -f- -n hshinde
+ oc process -f .openshiftio/application.yaml SOURCE_REPOSITORY_REF=1f4e5af SUFFIX_NAME=-osio-master SOURCE_REPOSITORY_URL=https://github.com/hrishin/nodejs-health-check.git

[hrishin]

Since most of this work is done by shelling out OC command. Hence Jenkins throws memory allocation issues during fork -> exec.

[hrishin]

For further testing, increased memory limit of Jenkins from 512Mi to 1 Gi. After running 20+ builds I've never seen that memory issue again.

Resource usage stats of Jenkins pod.

[hrishin]

Memory snapshot of Jenkins container. It clearly appears the invoking sh command may be creating zombie processes.

[chmouel]

We need to handle those failure gracefully in new. Pipeline, maybe with another timeout around the oc command?

…

On Wed, Jul 25, 2018, 18:28 Hrishikesh Shinde ***@***.***> wrote: Memory snapshot of Jenkins container. It clearly appears the invoking sh command may be creating zombie processes. [image: image] <https://user-images.githubusercontent.com/28615674/43198307-d286310c-902b-11e8-8164-bf5141adc0bd.png> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGCpHiq9UsJhid88-h0lKHHCwzLeMqlks5uKFZdgaJpZM4VVbyM> .

[hrishin]

@chmouel this can be handle gracefully by executing a job in new slave agent?

[chmouel]

why do we need a slave agent? why not just a timeout { }

[hrishin]

yes @chmouel lets put timeout events for such cases.
However running jobs in slave pods with enough memory would prevent such kind problems, right?

[chmouel]

yeah right, but let's try to make sure we make this works with the current limiters of what we have on OSIO, we can't really assume we can request more memory/pods/cpu in the osio tenant,

[hrishin]

This is fixed now.