Very bad performance when run inside pod about aws-codebuild-jenkins-plugin

[caiohasouza]

Someone ?

[leoherran-aws]

Sorry to hear you're seeing poor performance in this use case. I will research the root cause, but a possible workaround could be the solution to this issue, which is to allow users to specify the path of a zip file to upload for the build. This way you could zip the source yourself in a step before the build. This feature is still pending, but I will update this issue when further progress is made.

[Petezah]

I've seen this also, but didn't realize it was the zip operation that was taking so long. I've seen this on ECS (EC2 launch type) slaves, and Fargate slaves now. Also, the operation maxes out the CPU on the Jenkins master at the same time, making all jobs, even ones running on master, take twice as long. And the job on the slave takes over 30 minutes to start (usually takes 10-20 seconds on master).

@XDexter were you able to workaround the issue by zipping the source yourself?

[Petezah]

Just an update to my comment above: as a test, I tried the following script prior to executing codebuild:

zip dir: '', glob: '', zipFile: 'src.zip'
stash includes: 'src.zip', name: 'src'
cleanWs deleteDirs: true
unstash 'src'

(stash/unstash is there so that I can delete the entire workspace prior to codebuild uploading my single file)
It took 5 seconds to zip and stash the file, and 5 seconds to clean the workspace and unstash. After this, it took about 30 seconds to kick off the codebuild project. So it definitely seems to be the zipping code in this plugin that is causing the slowdown.

[caiohasouza]

Hi @Petezah

I got it, my temporary workaround was ran the awscodebuild command on master node (and no in slave). Yes, i did the same test (zip manually on slave container and all works normally).

Regards

[leoherran-aws]

Until the feature I mentioned is implemented, a workaround could be to zip and upload your source to S3 manually, then use that source to run the build on CodeBuild. Not ideal, but we plan to make this use case easier in the future.

[Petezah]

I believe the issue is that when a slave is running this step, the Master is trying to zip files that exist on the Slave--ie. the zip file is being built on the Master while files are being read off of the slave. This is potentially really slow over IPC. The solution would be to run the whole process on the Slave, and send the whole zip file to the master if necessary.

I'm seeing if I can plumb up a solution, hopefully.

[Petezah]

No, sorry, I misspoke; the files being zipped are on the Slave--they are streamed over IPC to the Master and zipped, and streamed back over IPC into the zip file, which also exists on the Slave. So the zipping action occurs on the Master process, but the files all exist on the Slave. The reason the Zip utility step, mentioned above, is faster is because the whole thing happens inside a MasterToSlave callable.

[Petezah]

I have tested this PR on a local instance + one of my remote slaves, and it fixes the issue for me.

[caiohasouza]

@Petezah It's looks good! @leobaran-aws It's possible merge this PR to master? To be possible i install on my Jenkins?

[leoherran-aws]

Fixed by 69244e3, which will be released in version 0.28 this week.

[caiohasouza]

Thank you!