Many members of the development team report frustration with the workflow experience related to integration testing. The key pain point is broken code being merged to dev prior to the automated run of integration tests. In short: unit tests pass and allow the merge, then integration tests fail after broken code is in dev.
Proposed new system
Order of testing in CI related to PR activity
Open PR cues automated unit tests in CI
Passed unit tests allow merge
Merge of PR cues automated integration tests in CI
RISK if integration tests fail, the broken code is already in dev
Please add your thoughts, experiences, and recommendations to this section. Select edit to add content.
We need to make sure that integration tests are run prior to merging to dev. This process could be semi-automatic and we would just need to make sure that the last (and not every) commit of a PR is tested. Once a PR has been reviewed and the reviewer has verified that no one is trying to exploit the vulnerabilities of our CI infrastructure, they could start the integration tests by some simple means, e.g. by commenting on github or by adding a label to the issue. A green integration tests report for a PR would be a prerequisite for merging
I understand the risk, but don't deem it significant or worth mitigating, even without additional precautions. I believe having the i-tests run only for the members of https://github.com/rchain-drone/ would be more than enough.
Why do we need the privileged mode for docker containers anyway? To package the current code into a docker container? How I see it, the container should be done in a way that allows us to "upgrade" it with newer code by simply mounting a volume with the updated code. We don't have to build or upload any docker artifacts in the PR build. Doesn't that invalidate the 'docker.sock' issue?
Running i-tests on every push to a PR seems wasteful (e.g. I do many pushes per PR, often within minutes). So maybe it should be done on demand. The 'demand' should also automatically include 'ok, I want to merge this'.
I'd like to see both unit and integration tests run via ci before a PR merge. Reasons we didn't do this in the past related to ability of Travis alone and later Travis + GitLab to support this. I think Drone can support this now.