We’ve been discussing testing infrastructure a lot in the past few months and this topic aims to collect that discussion.
Currently we have the following setup:
- D61/TS runs an internal bitbucket and bamboo server that run:
- pull request checks (style, compile, licenses, etc)
- continuous integration test (either on the master branch of a specific repo, or, more commonly on repo collections/manifests, e.g. sel4bench, sel4test, camkes, verification). If any of these change, the entire collection is tested. Sometimes they are out of sync (simultaneous PRs to multiple repos) and break because of that, and sometimes something actually breaks that was not caught in a pull request check
- continuous release to GitHub: if a subset of the CI tests passes, the master branches of the corresponding set of repos is automatically published to GitHub.
- point releases (“actual” seL4 releases) automatically tag the corresponding repos and create manifests, all after the corresponding checks are passing
- verification has a “testboard” check that can check any arbitrary verification manifest as long as bamboo can pull from the corresponding repo (can be on GitHub or bitbucket)
- the systems has a slightly more ad-hoc, but faster setup where you can run a specific manifest manually (is this a correct description?)
This kind of works, but contributing via GitHub is painful:
- if you make a PR on GitHub, you are not seeing the actual current state of the master branch
- you have no visibility of the current CI status (are hardware tests passing/failing on master etc)
- this leads to more perception of slowness than there actually is, and it can be quite frustrating to think everything is ready for merge, but nothing is happening for days and weeks, because some other repo somewhere is breaking things or some hardware is flakey and the GitHub deployment didn’t work for some reason, but nobody noticed.
So, the proposal is to get rid of the bitbucket step and make GitHub the source of truth for all seL4 repos. This is in line with the handover to the foundation, who should be running the main copy of the repos in any case.
What do we need to do to make this happen and to make it a nice and efficient experience for contributors and developers?