The Packages Site§

The packages site is introduced in 2017, to show various information about packages in AOSC OS. It's based on previous work in paklist.

Usage§

Human§

• "Latest Source Updates" are most recent updates in git.
• In "Repositories and Trees", "repositories" are DPKG sources and "trees" are abbs repos. "Ghost" means the packages exist in the DPKG source but not in the abbs repo. "Lagging" means there is no binary package of the updated version in abbs repo. "Missing" means the packages exist in the abbs repo but not in the DPKG source. The links in "Source Tree" column point to GitHub pages. "Outdated" means the packages have updated upstream versions. Click on the numbers and you will get the respective list.
• The Changelog in the package page is generated with git commits.
• Enter your raw SQL query here: https://packages.aosc.io/query/. This is read-only and have some limits. Useful for generating custom reports.

Machine§

• Add ?type=json to (almost) every endpoints, or send the X-Requested-With: XMLHttpRequest HTTP header, then you will get an json response. Use page=? for each page (see the "page" object), or page=all to avoid paging.
• API version: https://packages.aosc.io/api_version
• All package list: https://packages.aosc.io/list.json
• SQLite3 database download: https://packages.aosc.io/data/abbs.db, and piss.db.
• List of deb packages that can be removed: https://packages.aosc.io/cleanmirror/(repo), where (repo) is such as "amd64", "amd64/testing".

Project framework§

This website is composed of several projects mainly written in Python. Some of them can be used independently for different purposes.

abbs-meta§

abbs-meta is used to extract information from abbs tree and its Git repo. First, it use reposync.py to convert Git to Fossil repo. We use Fossil because the Git API is hard to use, calling external programs and writing temp files are slow, and Fossil repo database is easy to use directly. Then, we updated the abbs.db database for each commit, to preserve history. This operation modifies package_* tables. Syncing this from scratch may cost 10 hours. bashvar.py can parse most spec files. It calls bash when there are complex string operations. There are tools to add checksums (addchksum.{py,sh}) and to increase REL (increaserel.py).

packages-site§

packages-site is the website backend. dpkgrepo.py is used to record the DPKG packages information. This operation modifies dpkg_* tables. The package source definition is in dpkgrepo.py. The update process is shown in update.sh. The website updates once an hour. There is one hour of proxy/CDN/browser cache, so the delay may be 2 hours. The website is served in main.py. We use Bottle framework and Jinja2 template system. A lot of scary SQL is used to generate the reports. Version comparison is done using an extension for SQLite, which must be built first using make.

piss§

piss, "Projects Information Storage System", is a software version checker. For each package in abbs.db, it first guesses the upstream type using the URL, then do at most two requests to get the version/tag/file list. A lot of heuristics are used to get the latest version from the list. Then it stores the information in piss.db. If all guesses are wrong, it uses https://release-monitoring.org/. The database is updated every four hours.

Closely related§

• https://github.com/gumblex/bottle-sqlite, patched Bottle SQLite plugin with custom function/collation/aggregate/extension support.
• https://github.com/gumblex/htmllisting-parser, parses various directory listings generated by nginx/apache/other web servers.
• https://github.com/gumblex/fossilpy, Fossil repo read-only support in Python.

Deploying§

The recommended resource requirement is at least 1GiB of free space (current usage: 450MiB), at least 512MiB of total RAM (actually won't exceed 100M). Software requirement is Python 3.5+ or PyPy3.5, recent SQLite3 with FTS5 support, Fossil 2.6+, bash, Git. We currently use uWSGI and nginx to host the website. See https://github.com/AOSC-Dev/packages-site#deploy for simple instructions. Set up systemd timers for update.sh and piss.

PostgreSQL Hot-Standby Configuration§

If a hot-standby instance is required, you can set up like this:

Please note that if you perform this operation while the primary server is running, you need to perform the following steps as fast as you can to avoid de-sync.

• Transfer cold backup files from primary to secondary
  • Create a new user with replication permissions, this user needs to have a password assigned
  • Backup the database on the primary server: pg_basebackup -F t -D /path/to/backup/path -U postgres -P
  • Transfer backup files in /path/to/backup/path to the secondary server
  • On the secondary server: tar -C /var/lib/postgresql/data/ -xvf </path/to/backup/path/>/base.tar && tar -C /tmp/backups/ -xvf </path/to/backup/path>/pg_wal.tar, then open the postgresql.conf or postgresql.auto.conf file and append: primary_conninfo = 'host=<hostname> port=<port> user=<user_with_replication_perms> password=<password>' and restore_command = 'cp /tmp/backups/%f %p'
• Promote the secondary warm backup to hot-standby
  • On the secondary server: touch /var/lib/postgresql/data/standby.signal and then start postgresql server
  • Wait until you see "consistent recovery state reached at " line appears in the log file
• Add a local read-only user
  • On the secondary server: stop postgresql server, then remove /var/lib/postgresql/data/standby.signal, and restart postgresql server
  • Add a new user to the secondary server using createuser. Or you can re-use the replication user's name and assign a password like this in the psql shell: ALTER USER <name> PASSWORD '<password>';
  • Then touch /var/lib/postgresql/data/standby.signal and then start postgresql server

Plans§

• Add file lists.
• Add so dep/provides.
• Search file lists.
• Show the newer version of (staging, bugfix). (waiting for branch change)
• Add package "issues" page.

FAQ§

Q: The site is down§

A: Please first check https://aoscpkgs.gumble.pw/, which is the mirror site. Then please report back which site is useable to site admin @gumblex in the Telegram groups.

Q: The site is not syncing§

A: Please first check https://packages.aosc.io/aosc-os-abbs/timeline to see if it's a git-to-fossil syncing problem, then check https://packages.aosc.io/updates to see it's a really syncing problem. If git-to-fossil syncing is not working, the cause is usually a force-push or maybe a rebase. If it's not that case, then there is something unusual in the repo and/or the program. Please report your findings to the site admin.

Q: Some packages are not updated§

A: Assuming the site is correctly syncing (see above), your updated packages may be in the 'bugfix' branch. Currently the site doesn't show any info about this branch, but it's well recorded in the database. The related work is halted now until the new workflow introduced AOSCC 2018 is in effect. Please send suggestions on how to show multiple branches on the UI, in the Telegram group or on the mailing list.

Q: Upstream version is wrong§

A: PISS uses a lot of heuristics, and resorts to Anitya if we can't figure out. Therefore, there are two sources of errors. If the error is on our side, we can improve the heuristics, or give it more information. Currently the only information we can use is package name, current version, url, source type (tarball/git/hg).

Q: Why no (reverse) dependency graph§

A: Too big to draw. There are also many cyclic dependencies.