AOSC Wiki / Developer / Infrastructure / .
Also available in: 简体中文

Packages Site

The description of the packages site and the FAQ

The Packages Site§

The packages site is introduced in 2017, to show various information about packages in AOSC OS. It's based on previous work in paklist.

Usage§

Human§

Machine§

Project framework§

This website is composed of several projects mainly written in Python. Some of them can be used independently for different purposes.

abbs-meta§

abbs-meta is used to extract information from abbs tree and its Git repo. First, it use reposync.py to convert Git to Fossil repo. We use Fossil because the Git API is hard to use, calling external programs and writing temp files are slow, and Fossil repo database is easy to use directly. Then, we updated the abbs.db database for each commit, to preserve history. This operation modifies package_* tables. Syncing this from scratch may cost 10 hours. bashvar.py can parse most spec files. It calls bash when there are complex string operations. There are tools to add checksums (addchksum.{py,sh}) and to increase REL (increaserel.py).

packages-site§

packages-site is the website backend. dpkgrepo.py is used to record the DPKG packages information. This operation modifies dpkg_* tables. The package source definition is in dpkgrepo.py. The update process is shown in update.sh. The website updates once an hour. There is one hour of proxy/CDN/browser cache, so the delay may be 2 hours. The website is served in main.py. We use Bottle framework and Jinja2 template system. A lot of scary SQL is used to generate the reports. Version comparison is done using an extension for SQLite, which must be built first using make.

piss§

piss, "Projects Information Storage System", is a software version checker. For each package in abbs.db, it first guesses the upstream type using the URL, then do at most two requests to get the version/tag/file list. A lot of heuristics are used to get the latest version from the list. Then it stores the information in piss.db. If all guesses are wrong, it uses https://release-monitoring.org/. The database is updated every four hours.

Deploying§

The recommended resource requirement is at least 1GiB of free space (current usage: 450MiB), at least 512MiB of total RAM (actually won't exceed 100M). Software requirement is Python 3.5+ or PyPy3.5, recent SQLite3 with FTS5 support, Fossil 2.6+, bash, Git. We currently use uWSGI and nginx to host the website. See https://github.com/AOSC-Dev/packages-site#deploy for simple instructions. Set up systemd timers for update.sh and piss.

PostgreSQL Hot-Standby Configuration§

If a hot-standby instance is required, you can set up like this:

Please note that if you perform this operation while the primary server is running, you need to perform the following steps as fast as you can to avoid de-sync.

Plans§

FAQ§

Q: The site is down§

A: Please first check https://aoscpkgs.gumble.pw/, which is the mirror site. Then please report back which site is useable to site admin @gumblex in the Telegram groups.

Q: The site is not syncing§

A: Please first check https://packages.aosc.io/aosc-os-abbs/timeline to see if it's a git-to-fossil syncing problem, then check https://packages.aosc.io/updates to see it's a really syncing problem. If git-to-fossil syncing is not working, the cause is usually a force-push or maybe a rebase. If it's not that case, then there is something unusual in the repo and/or the program. Please report your findings to the site admin.

Q: Some packages are not updated§

A: Assuming the site is correctly syncing (see above), your updated packages may be in the 'bugfix' branch. Currently the site doesn't show any info about this branch, but it's well recorded in the database. The related work is halted now until the new workflow introduced AOSCC 2018 is in effect. Please send suggestions on how to show multiple branches on the UI, in the Telegram group or on the mailing list.

Q: Upstream version is wrong§

A: PISS uses a lot of heuristics, and resorts to Anitya if we can't figure out. Therefore, there are two sources of errors. If the error is on our side, we can improve the heuristics, or give it more information. Currently the only information we can use is package name, current version, url, source type (tarball/git/hg).

Q: Why no (reverse) dependency graph§

A: Too big to draw. There are also many cyclic dependencies.