Post mortem for CORE-997: False positives in GitLab CI

Timeline

• Sun., Aug. 12 - PMs successfully ran RNode from dev branch
• Mon., Aug. 13
  • 2:30pm MDT - PMs unable to connect to dev bootstrap.  CORE-997 - Getting issue details... STATUS  . Conversation in comments
  • 4:30pm MDT
    • Pawel investigates functionality of https://github.com/rchain/rchain/pull/1267 in local test suite. Finds that it fails tests.  
    • Pawel and Kelly investigate integration tests that passed in GitLab CI. Discovered https://gitlab.com/rchain/rchain/pipelines/27806610 failed silently, appearing to pass. The commit prior passed correctly. Every commit after https://gitlab.com/rchain/rchain/pipelines/27806610 shows the same silent file in the logs of the p-2-p network test.
  • 5:00pm MDT Pawel works to fix issue in https://github.com/rchain/rchain/pull/1267. Fix is in https://github.com/rchain/rchain/pull/1285/

Analysis of what went wrong and recommendations

• https://github.com/rchain/rchain/pull/1267 included a mix of both process and test updates A best practice is to separate these changes into two PRs.
• https://github.com/rchain/rchain/pull/1267 was not tested locally A best practice for PR that make changes to networking is to test them locally before committing. When tested locally after the fact, changes in this PR proved to be breaking.
• Integration tests appeared to pass while failing in logs and not generating any error messages via notifications. Jeremy Busk, any recommendations for mitigating this in the future?