Packages Site

The description of the packages site and the FAQ

The Packages Site

The packages site is introduced in last year, to show various information about packages in AOSC OS. It's based on previous work in paklist.

Project framework

This website is composed of several projects mainly written in Python. Some of them can be used independently for different purposes.

abbs-meta

abbs-meta is used to extract information from abbs tree and its Git repo. First, it use reposync.py to convert Git to Fossil repo. We use Fossil because the Git API is hard to use, calling external programs and writing temp files are slow, and Fossil repo database is easy to use directly. Then, we updated the abbs.db database for each commit, to preserve history. This operation modifies package_* tables. Syncing this from scratch may cost 10 hours. bashvar.py can parse most spec files. It calls bash when there are complex string operations. There are tools to add checksums (addchksum.{py,sh}) and to increase REL (increaserel.py).

packages-site

packages-site is the website backend. dpkgrepo.py is used to record the DPKG packages information. This operation modifies dpkg_* tables. The package source definition is in dpkgrepo.py. The update process is shown in update.sh. The website updates once an hour. There is one hour of proxy/CDN/browser cache, so the delay may be 2 hours.
The website is served in main.py. We use Bottle framework and Jinja2 template system. A lot of scary SQL is used to generate the reports. Version comparison is done using custom collation function in SQLite.

piss

piss, "Projects Information Storage System", is a software version checker. For each package in abbs.db, it first guesses the upstream type using the URL, then do at most two requests to get the version/tag/file list. A lot of heuristics are used to get the latest version from the list. Then it stores the information in piss.db. If all guesses are wrong, it uses https://release-monitoring.org/. The database is updated every four hours.

Deploying

The recommended resource requirement is at least 1GiB of free space (current usage: 450MiB), at least 512MiB of total RAM (actually won't exceed 100M). Software requirement is Python 3.5+ or PyPy3.5, recent SQLite3 with FTS5 support, Fossil 2.6+, bash, Git.
We currently use uWSGI and nginx to host the website. See https://github.com/AOSC-Dev/packages-site#deploy for simple instructions. Set up systemd timers for update.sh and piss.

Usage

Human

  • "Latest Source Updates" are most recent updates in git.
  • In "Repositories and Trees", "repositories" are DPKG sources and "trees" are abbs repos. "Ghost" means the packages exist in the DPKG source but not in the abbs repo. "Lagging" means there is no binary package of the updated version in abbs repo. "Missing" means the packages exist in the abbs repo but not in the DPKG source. The links in "Source Tree" column point to GitHub pages. "Outdated" means the packages have updated upstream versions. Click on the numbers and you will get the respective list.
  • The Changelog in the package page is generated with git commits.
  • Enter your raw SQL query here: https://packages.aosc.io/query/. This is read-only and have some limits. Useful for generating custom reports.

Machine

Plans

  • Add file lists.
  • Add so dep/provides.
  • Search file lists.
  • Show the newer version of (staging, bugfix). (waiting for branch change)
  • Add package "issues" page.

FAQ

Q: The site is down

A: Please first check https://aoscpkgs.gumble.pw/, which is the mirror site. Then please report back which site is useable to site admin @gumblex in the Telegram groups.

Q: The site is not syncing

A: Please first check https://packages.aosc.io/aosc-os-abbs/timeline to see if it's a git-to-fossil syncing problem, then check https://packages.aosc.io/updates to see it's a really syncing problem. If git-to-fossil syncing is not working, the cause is usually a force-push or maybe a rebase. If it's not that case, then there is something unusual in the repo and/or the program. Please report your findings to the site admin.

Q: Some packages are not updated

A: Assuming the site is correctly syncing (see above), your updated packages may be in the 'bugfix' branch. Currently the site doesn't show any info about this branch, but it's well recorded in the database. The related work is halted now until the new workflow introduced AOSCC 2018 is in effect. Please send suggestions on how to show multiple branches on the UI, in the Telegram group or on the mailing list.

Q: Upstream version is wrong

A: PISS uses a lot of heuristics, and resorts to Anitya if we can't figure out. Therefore, there are two sources of errors. If the error is on our side, we can improve the heuristics, or give it more information. Currently the only information we can use is package name, current version, url, source type (tarball/git/hg).

Q: Why no (reverse) dependency graph

A: Too big to draw. There are also many cyclic dependencies.