test-origin-image: merge PR with master? about vagrant-openshift

[mfojtik]

@danmcp is @openshift-bot merging with master when we run tests in origin? my opinion is that you don't want to merge with master as you want to test the state of your tree. if we merge against master, the test might be inconsistent (it passes in jenkins, but fails in your branch and vice versa).

[rhcarvalho]

It wouldn't be "Continuous Integration" if we don't actually integrate the code, no?

[mfojtik]

@rhcarvalho ok, I think i'm fine merging it to master prior to test execution. what will happen when the merge fail?

[bparees]

I believe the way this works is that the branch has already been rebased on top of master at this point.

So this is already happening. I think when the merge/rebase fails you get the "needs rebase" message and then it probably just tests your branch directly.

[bparees]

@rhcarvalho @mfojtik https://github.com/openshift/test-pull-requests

[mfojtik]

@bparees the job @rhcarvalho is referring to is managed by the Jenkins GH pull tester plugin...

[rhcarvalho]

@rhcarvalho @mfojtik https://github.com/openshift/test-pull-requests

@bparees I believe the code I'm referring to is unrelated to that (I think I didn't even knew it existed, thanks for the link).

The Jenkins job, for example for MongoDB, is configured to run:

...
vagrant test-origin-image --image mongodb --source openshift/$JOB_NAME --ref $ghprbActualCommit
...

Then it goes as I pointed in the issue description, we clone the image repo, fetch all PR refs and checkout to the commit of the PR.

If a PR by itself works, but after integrating with the existing code (iow, after merge) the changes do not work, we're letting it slip.

[bparees]

yeah i didn't recognize this was about the image tests, my bad.

you're right that the image tests just test the branch. Where the rubber hits the road for those is in the push images job which tests everything again before publishing an image. So we're still protected from pushing out a bad image, but it is not ideal to merge a broken PR and then leave the push job in a broken state until we revert the commit. That's never happened that i'm aware of, so this isn't a very high priority in my mind, but it's true that the current situation isn't ideal.