<!-- TITLE: Packages Site --> <!-- SUBTITLE: The description of the packages site and the FAQ --> # The Packages Site The [packages site](https://packages.aosc.io/) is introduced in 2017, to show various information about packages in AOSC OS. It's based on previous work in [paklist](https://github.com/AOSC-Dev/paklist). ## Usage ### Human - "Latest Source Updates" are most recent updates in git. - In "Repositories and Trees", "repositories" are DPKG sources and "trees" are abbs repos. "Ghost" means the packages exist in the DPKG source but not in the abbs repo. "Lagging" means there is no binary package of the updated version in abbs repo. "Missing" means the packages exist in the abbs repo but not in the DPKG source. The links in "Source Tree" column point to GitHub pages. "Outdated" means the packages have updated upstream versions. Click on the numbers and you will get the respective list. - The Changelog in the package page is generated with git commits. - Enter your raw SQL query here: https://packages.aosc.io/query/. This is read-only and have some limits. Useful for generating custom reports. ### Machine - Add `?type=json` to (almost) every endpoints, or send the `X-Requested-With: XMLHttpRequest` HTTP header, then you will get an json response. Use `page=?` for each page (see the "page" object), or `page=all` to avoid paging. - API version: https://packages.aosc.io/api_version - All package list: https://packages.aosc.io/list.json - SQLite3 database download: https://packages.aosc.io/data/abbs.db, and `piss.db`. - List of deb packages that can be removed: https://packages.aosc.io/cleanmirror/(repo), where (repo) is such as "amd64", "amd64/testing". ## Project framework This website is composed of several projects mainly written in Python. Some of them can be used independently for different purposes. ### abbs-meta [abbs-meta](https://github.com/AOSC-Dev/abbs-meta) is used to extract information from abbs tree and its Git repo. First, it use `reposync.py` to convert Git to Fossil repo. We use Fossil because the Git API is hard to use, calling external programs and writing temp files are slow, and Fossil repo database is easy to use directly. Then, we updated the `abbs.db` database for each commit, to preserve history. This operation modifies `package_*` tables. Syncing this from scratch may cost 10 hours. `bashvar.py` can parse most spec files. It calls bash when there are complex string operations. There are tools to add checksums (`addchksum.{py,sh}`) and to increase REL (`increaserel.py`). ### packages-site [packages-site](https://github.com/AOSC-Dev/packages-site) is the website backend. `dpkgrepo.py` is used to record the DPKG packages information. This operation modifies `dpkg_*` tables. The package source definition is in `dpkgrepo.py`. The update process is shown in `update.sh`. The website updates once an hour. There is one hour of proxy/CDN/browser cache, so the delay may be 2 hours. The website is served in `main.py`. We use [Bottle](https://bottlepy.org/) framework and Jinja2 template system. A lot of scary SQL is used to generate the reports. Version comparison is done using an extension for SQLite, which must be built first using `make`. ### piss [piss](https://github.com/AOSC-Dev/piss), "Projects Information Storage System", is a software version checker. For each package in abbs.db, it first guesses the upstream type using the URL, then do at most two requests to get the version/tag/file list. A lot of heuristics are used to get the latest version from the list. Then it stores the information in piss.db. If all guesses are wrong, it uses https://release-monitoring.org/. The database is updated every four hours. ### Closely related - https://github.com/gumblex/bottle-sqlite, patched Bottle SQLite plugin with custom function/collation/aggregate/extension support. - https://github.com/gumblex/htmllisting-parser, parses various directory listings generated by nginx/apache/other web servers. - https://github.com/gumblex/fossilpy, Fossil repo read-only support in Python. ## Deploying The recommended resource requirement is at least 1GiB of free space (current usage: 450MiB), at least 512MiB of total RAM (actually won't exceed 100M). Software requirement is Python 3.5+ or PyPy3.5, recent SQLite3 with FTS5 support, Fossil 2.6+, bash, Git. We currently use uWSGI and nginx to host the website. See https://github.com/AOSC-Dev/packages-site#deploy for simple instructions. Set up systemd timers for update.sh and piss. ## Plans - Add file lists. - Add so dep/provides. - Search file lists. - Show the newer version of (staging, bugfix). (waiting for branch change) - Add package "issues" page. # FAQ ## Q: The site is down A: Please first check https://aoscpkgs.gumble.pw/, which is the mirror site. Then please report back which site is useable to site admin @gumblex in the Telegram groups. ## Q: The site is not syncing A: Please first check https://packages.aosc.io/aosc-os-abbs/timeline to see if it's a git-to-fossil syncing problem, then check https://packages.aosc.io/updates to see it's a really syncing problem. If git-to-fossil syncing is not working, the cause is usually a force-push or maybe a rebase. If it's not that case, then there is something unusual in the repo and/or the program. Please report your findings to the site admin. ## Q: Some packages are not updated A: Assuming the site is correctly syncing (see above), your updated packages may be in the 'bugfix' branch. Currently the site doesn't show any info about this branch, but it's well recorded in the database. The related work is halted now until the new workflow introduced AOSCC 2018 is in effect. Please send suggestions on how to show multiple branches on the UI, in the Telegram group or on the mailing list. ## Q: Upstream version is wrong A: PISS uses a lot of heuristics, and resorts to Anitya if we can't figure out. Therefore, there are two sources of errors. If the error is on our side, we can improve the heuristics, or give it more information. Currently the only information we can use is package name, current version, url, source type (tarball/git/hg). ## Q: Why no (reverse) dependency graph A: Too big to draw. There are also many cyclic dependencies.