Contributor Minutes on 2022-05-22

Attendees

Lily-Ayta (Lily)
Mingcong Bai (Bai)
Tianhao Chai (Chai)
CambeR
Mag Mell (Saki)
Zixing Liu (Liu)

Agenda

Autobuild3 Test Integration

CambeR: Let's take a quick read at the current Google Docs: RFC draft. Any questions and points for discussion so far? Let's leave the details for later revisions.
- CambeR: My basic thought with this RFC is to offer a Autobuild3 switch to enable/disable integrated tests, contained within the autobuild/tests path.
Bai: Should the tests be enabled by default? I'm thinking that, to minimise developer impact, maybe we should keep tests off by default for now.
- CambeR: Yes.
Liu: I think we should allow multiple test manifest files in autobuild/tests.
- Bai: Agreed, I have an alternative design (which I should outline later), where test manifest follows a define-like format, which in turn defines a few test template scripts or allows a corresponding custom test script.
CambeR: Should we allow unprevileged tests? Some tests may require a non-root environment.
- Liu: How should we do this? Maybe we could look at other projects, such as Google's Bazel.
- Saki: Or we could just create a normal user for tests in BuildKit?
  - Bai: I think it would be better to just let Autobuild3 handle user management - creating a temporary one-time user and destroying said user after test sequence. I say this because not everyone would use Autobuild3 with BuildKit.
  - Saki: Okay... Or let systemd handle it?
  - Liu: Let's leave the details for later.
Liu: I want to highlight an issue, however. Autobuild3 has poor handling for syntax errors in scripts, where it doesn't return an error code when it encounters them. Bash's trap() function wouldn't catch them either.
- CambeR: Speaking of error codes, we might need to create other error codes to denote test statuses.
- Bai: Well, again, this is probably a detail for later. Not to mention that this wouldn't be a test-specific issue.
Liu: It might also be worth considering what these "tests" entails - unit tests or integrated test for other functions of a package? And should build abort on test error, or just report success/warning/failures all in one go?
- CambeR: Yes on reporting all results in one go.
- Liu: Having mentioned Autobuild3's known design issue above, I think it might also be worth including a special return code for Autobuild3's syntax error detection failures.
Liu: What is with acbs-build --with-check? I think we should at the very least notify developers when tests are available but not enabled.
- Liu: You know what. I think we should enable tests when they are available, there would be no point to this otherwise - developers will never test the packages if they don't have to - what is the point?
- CambeR: But Saki said that it would be a waste of time.
  - Liu: Again, if we don't enable this, I don't see anyone would use this function.
- Bai: I second Liu. I also think that it's necessary for a reasonable quality standard.
  - Saki: But it simply takes too long...
  - Liu: Again, I stand by my argument.
  - Saki: Perhaps you are right.
- Liu: In that case, I think we should change that switch to acbs-build --no-check instead.
- Lily: Though I think we also need to consider slower devices, or devices that simply couldn't complete certain tests.
  - Liu: Well, all Autobuild3 variables are architecture-aware, so you could enable or disable tests on demand.
  - Bai: Also that Retro architectures are actually built on modern, high-performance hardware, so this shouldn't be too much of a concern.
Bai: So to recap. (1) We need to make sure that tests are opt-out, not opt-in and (2) test opt-out should only be granted on the basis of expected failure, not time constraint.
- Liu: Or when packages contain build-time tests that are run by default.
- (Liu and Bai debates on whether GCC contains default build-time unit tests - inconclusive, but irrelevant [and long] argument.)
Liu: How do we calculate the costs for certain test inclusions? Some tests might only offer marginal gains in terms of test coverage but would take an unreasonable amount of time.
- Saki: Also, what happens when we have emergency fixes? Especially those without version changes.
  - Liu: But regression tests would be necessary nonetheless.
  - Bai: Yeah, if a fix breaks another thing, it kind of defeats the purpose, no?
  - Liu: Yeah, remember that recent Microsoft fiasco with their emergency hotfix breaking VPNs?
CambeR: When should the checks be enforced? Would we need to test packages twice in the topic procedure?
- Liu: Tests could be flaky so yes.
- Bai: Liu, I think she means if we should test packages both during topic upload and merges into Stable.
- Liu: I think it's a simple matter of adhering to the topic guidelines, so test twice.
  - Bai: Seconded.
- Saki: But now I am again doubting if it would be worth it... If we already tested pre-merge, why should we test again?
  - Liu: The fact that we are merging into Stable is exactly why we should test again then.
  - Bai: And that you wouldn't be able to detect test issues if you don't test during the topic prodecures, no?
- Saki: But what if tests take longer than build time?
  - Bai: It's an irrelevant detail, it's a matter of necessity.
CambeR: Also, some upstreams provide unit tests, so we could also just use those directly.
- Liu: But some of them have incomplete unit test coverage.
Bai: Here, let me propose something. Let's push for a gradual test integration scheme, where we use an honour system - when tests are included in autobuild/tests, they must be run and mandates success or documented expected failures. At any rate, this would be better than not having tests at all.
- Liu: Yes, and we should allow for certain architectural- and package-type-specific exceptions.
Lily: Should we also push for CI on the Pull Requests level, with a bot to handle scheduling and reporting like Rust's Homu? We could open local instances to test code functions and enforce at the community repo level (no test pass no upload).
- Liu: This would be very difficult to implement.
- Bai: I also think this would be over-engineering our demands.
Saki: We should also perhaps allow for tests to be run after package upload.
- Liu: That just defeats the purpose, doesn't it?
- CambeR: It might also be worth it to include a .deb metadata field to indicate whether a package has been tested.
  - Liu: Well, who is then going to manage and observe this metadata? dpkg/apt ignores extra metadata entries.
  - CambeR: Then how should the users know?
  - Liu: I don't think it would be visible anyways.
Liu: So, to recap. No need to change ACBS/Ciel, Autobuild3 test integration is to be implemented on an opt-out basis.

Other Distro Review (5)

Bai: The distro to review this week was voted to be Ubuntu MATE.
Review starts, detailed discussion omitted

Note: See the notes here for the review summary.

Actionables

CambeR: Continue revising RFC.