Contributor Minutes on 2022-05-22
- Lily-Ayta (Lily)
- Mingcong Bai (Bai)
- Tianhao Chai (Chai)
- Mag Mell (Saki)
- Zixing Liu (Liu)
Autobuild3 Test Integration
- CambeR: Let's take a quick read at the current Google Docs: RFC draft.
Any questions and points for discussion so far? Let's leave the details for
- CambeR: My basic thought with this RFC is to offer a Autobuild3 switch
to enable/disable integrated tests, contained within the
- CambeR: My basic thought with this RFC is to offer a Autobuild3 switch to enable/disable integrated tests, contained within the
- Bai: Should the tests be enabled by default? I'm thinking that, to minimise
developer impact, maybe we should keep tests off by default for now.
- CambeR: Yes.
- Liu: I think we should allow multiple test manifest files
- Bai: Agreed, I have an alternative design (which I should outline later),
where test manifest follows a
define-like format, which in turn defines a few test template scripts or allows a corresponding custom test script.
- Bai: Agreed, I have an alternative design (which I should outline later), where test manifest follows a
- CambeR: Should we allow unprevileged tests? Some tests may require a non-root
- Liu: How should we do this? Maybe we could look at other projects, such as Google's Bazel.
- Saki: Or we could just create a normal user for tests in BuildKit?
- Bai: I think it would be better to just let Autobuild3 handle user management - creating a temporary one-time user and destroying said user after test sequence. I say this because not everyone would use Autobuild3 with BuildKit.
- Saki: Okay... Or let systemd handle it?
- Liu: Let's leave the details for later.
- Liu: I want to highlight an issue, however. Autobuild3 has poor handling for
syntax errors in scripts, where it doesn't return an error code when it
encounters them. Bash's
trap()function wouldn't catch them either.
- CambeR: Speaking of error codes, we might need to create other error codes to denote test statuses.
- Bai: Well, again, this is probably a detail for later. Not to mention that this wouldn't be a test-specific issue.
- Liu: It might also be worth considering what these "tests" entails - unit
tests or integrated test for other functions of a package? And should build
abort on test error, or just report success/warning/failures all in one go?
- CambeR: Yes on reporting all results in one go.
- Liu: Having mentioned Autobuild3's known design issue above, I think it might also be worth including a special return code for Autobuild3's syntax error detection failures.
- Liu: What is with
acbs-build --with-check? I think we should at the very least notify developers when tests are available but not enabled.
- Liu: You know what. I think we should enable tests when they are available, there would be no point to this otherwise - developers will never test the packages if they don't have to - what is the point?
- CambeR: But Saki said that it would be a waste of time.
- Liu: Again, if we don't enable this, I don't see anyone would use this function.
- Bai: I second Liu. I also think that it's necessary for a
reasonable quality standard.
- Saki: But it simply takes too long...
- Liu: Again, I stand by my argument.
- Saki: Perhaps you are right.
- Liu: In that case, I think we should change that switch to
- Lily: Though I think we also need to consider slower devices, or devices
that simply couldn't complete certain tests.
- Liu: Well, all Autobuild3 variables are architecture-aware, so you could enable or disable tests on demand.
- Bai: Also that Retro architectures are actually built on modern, high-performance hardware, so this shouldn't be too much of a concern.
- Bai: So to recap. (1) We need to make sure that tests are opt-out, not opt-in
and (2) test opt-out should only be granted on the basis of expected failure,
not time constraint.
- Liu: Or when packages contain build-time tests that are run by default.
- (Liu and Bai debates on whether GCC contains default build-time unit tests - inconclusive, but irrelevant [and long] argument.)
- Liu: How do we calculate the costs for certain test inclusions? Some tests
might only offer marginal gains in terms of test coverage but would take
an unreasonable amount of time.
- Saki: Also, what happens when we have emergency fixes? Especially those
without version changes.
- Liu: But regression tests would be necessary nonetheless.
- Bai: Yeah, if a fix breaks another thing, it kind of defeats the purpose, no?
- Liu: Yeah, remember that recent Microsoft fiasco with their emergency hotfix breaking VPNs?
- Saki: Also, what happens when we have emergency fixes? Especially those without version changes.
- CambeR: When should the checks be enforced? Would we need to test packages
twice in the topic procedure?
- Liu: Tests could be flaky so yes.
- Bai: Liu, I think she means if we should test packages both during topic upload and merges into Stable.
- Liu: I think it's a simple matter of adhering to the topic guidelines, so
- Bai: Seconded.
- Saki: But now I am again doubting if it would be worth it... If we already
tested pre-merge, why should we test again?
- Liu: The fact that we are merging into Stable is exactly why we should test again then.
- Bai: And that you wouldn't be able to detect test issues if you don't test during the topic prodecures, no?
- Saki: But what if tests take longer than build time?
- Bai: It's an irrelevant detail, it's a matter of necessity.
- CambeR: Also, some upstreams provide unit tests, so we could also just use
- Liu: But some of them have incomplete unit test coverage.
- Bai: Here, let me propose something. Let's push for a gradual test integration
scheme, where we use an honour system - when tests are included in
autobuild/tests, they must be run and mandates success or documented expected failures. At any rate, this would be better than not having tests at all.
- Liu: Yes, and we should allow for certain architectural- and package-type-specific exceptions.
- Lily: Should we also push for CI on the Pull Requests level, with a bot to
handle scheduling and reporting like Rust's Homu?
We could open local instances to test code functions and enforce at the
community repo level (no test pass no upload).
- Liu: This would be very difficult to implement.
- Bai: I also think this would be over-engineering our demands.
- Saki: We should also perhaps allow for tests to be run after package upload.
- Liu: That just defeats the purpose, doesn't it?
- CambeR: It might also be worth it to include a .deb metadata field to
indicate whether a package has been tested.
- Liu: Well, who is then going to manage and observe this metadata? dpkg/apt ignores extra metadata entries.
- CambeR: Then how should the users know?
- Liu: I don't think it would be visible anyways.
- Liu: So, to recap. No need to change ACBS/Ciel, Autobuild3 test integration is to be implemented on an opt-out basis.
Other Distro Review (5)
- Bai: The distro to review this week was voted to be Ubuntu MATE.
- Review starts, detailed discussion omitted
Note: See the notes here for the review summary.
- CambeR: Continue revising RFC.