p-vector

Information about the deb repo manager p-vector

Introduction

P-vector manages AOSC OS's deb repository, generates index files and analyzes problems in packaging.

Usage

p-vector config.yaml (scan|release [--force]|sync|analyze [full]|reset [pv|sync])

  • scan: scan the directories containing packages and record them in the database.
  • release: generate the Packages, InRelease and Contents files for apt.
  • sync: download SQLite databases from packages.aosc.io and load them into the PostgreSQL database.
  • analyze: analyze problems in packaging.
  • reset: drop tables.

Database

p-vector uses a PostgreSQL database. We recommend using the latest version, even though 9.6 is still usable. Specify the connection string in db_pgconn in the config file.

Tables

p-vector native tables

These tables contain information necessary to generate indices for apt and to analyze problems.

  • pv_repos: lists information about repos
    • name*: primary key
    • realname: name without branch
    • path: path of deb repo
    • testing: 0-2, the level of stablility
    • branch: stable, testing, explosive
    • component: main, bsp, opt
    • architecture: all, amd64, ...
  • pv_packages: metadata of every .deb file
    • package*
    • version*
    • repo*
    • architecture
    • filename
    • size
    • sha256
    • mtime: modification time of the deb file
    • debtime: the modification time of control file
    • section: as listed in deb file
    • installed_size: KiB, listed in deb file
    • maintainer
    • description
    • _vercomp: comparable version for sorting
  • pv_package_duplicate: deb files with same package, version and repo
    • (same as above, primary key is filename)
  • pv_package_dependencies
    • package*
    • version*
    • repo*
    • relationship*: Depends, Breaks, ...
    • value: as listed in deb (parsed in v_dpkg_dependencies)
  • pv_package_sodep: dynamic library dependency of deb file
    • package
    • version
    • repo
    • depends: 0 provides, 1 depends
    • name: libfoo.so
    • ver: .1.2.3
  • pv_package_files: files in deb packages
    • package
    • version
    • repo
    • path: without leading /
    • name
    • size
    • ftype: file types, eg. reg, dir, lnk, sock, chr, blk, fifo
    • perm: permission stored as integer
    • uid
    • gid
    • uname
    • gname
  • pv_package_issues: current package issues
    • id*
    • package
    • version
    • repo
    • errno: issue code
    • level: -1 critical, 0 error, 1 warning
    • filename
    • ctime: creation time
    • mtime: modification time (same package and file, different detail)
    • atime: verification time
    • detail: json field
  • pv_issues_stats: package issue statistics log
    • repo
    • errno
    • cnt: issue count
    • total: package count
    • updated
  • pv_dbsync: packages site database sync status
    • name: database filename
    • etag
    • updated

tables from packages site

These tables are copied from the packages site.

abbs.db

  • trees
  • tree_branches
  • packages
  • package_dependencies
  • package_duplicate
  • package_spec
  • package_versions
  • dpkg_repo_stats

*-marks.db

  • repo_branches
  • repo_committers
  • repo_marks
  • repo_package_basherr
  • repo_package_rel

piss.db

  • package_upstream
  • upstream_status
  • anitya_link
  • anitya_projects

Views

These views come from abbs.db of the packages site.

  • v_packages: latest packages in source trees
  • v_package_upstream: package upstream version and url

Materialized Views

These views show more information about packages and their relationships, but are expensive to compute.

  • v_packages_new: packages with the largest versions
    • (same as pv_packages)
  • v_dpkg_dependencies: parsed dpkg dependencies
    • package
    • version
    • repo
    • relationship
    • nr: number of dependency. If dependency specified contains | (or), then nr is same
    • deppkg: depended package
    • deparch: archtecture of depended package
    • relop: version relationship
    • depver: version requirement
    • depvercomp: version requirement for comparison
  • v_so_breaks: when updating package, which dep_packages will be broken?
    • package: the package that is depended on
    • repo
    • soname
    • sover
    • sodepver: the sover that other packages require
    • dep_package: the reverse dependency
    • dep_repo
    • dep_version

Analyze

Here we list the actual implementation of issue detection.

  • 101: find entries in table repo_package_basherr where package name is null
  • 102: find entries in table repo_package_basherr where package name is not null (package name is defined in defines)
  • 103: regex ^[a-z0-9][a-z0-9+.-]*$. The debian policy specifies that package name should be at least 2 characters, but we set an exception here for r.
  • 301: debtime is null. If we can't open the deb file, we won't be able to get mtime of control.
  • 302: size < 10M and size < one third of median package size
  • 303: pool/(lib)<initial letter>/<package name>_<version>_<architecture>.deb
  • 311: regex ^.+ <.+@.+>$
  • 321: packages doesn't contain files in /usr/local or ^(bin|boot|etc|lib|opt|run|sbin|srv|usr|var)/?.*
  • 322: executable zero-size files, whose names are not NEWS, ChangeLog, INSTALL, TODO, COPYING, AUTHORS, README, README.md, README.txt, empty, placeholder, placeholder.txt, .*, __init__.p*
  • 323: uid>999 OR gid>999
  • 324: non-executable file in /bin, /sbin, /usr/bin; or non-executable directory
  • 401: find packages whose build dependencies are not found in source
  • 402: find entries in table package_duplicate
  • 411: find packages whose dependencies (Depends, Pre-Depends, Recommends, Suggests) not found in related deb repos
  • 412: find entries in table pv_package_duplicate
  • 421: find packages p1 and p2, where p1 and p2 have same regular files, p1 != p2, p1.arch=p2.arch or p2.arch=all, p1.repo_component=p2.repo_component, p1.testing <= p2.testing; also excludes Breaks, Replaces, Conflicts listed in deb package
  • 431: find package p2 that provides the library that package p1 needs. If no version matches, then find the same so-name, such that p2 is the most popular provider. Return fuzzy matches and empty matches.
  • 432: recursively find package p2 that provides the library that package p1 needs. Return library providers that not found in PKGDEP. Don't report missing dependencies in aosc-os-core. If p1 dep p2 dep p3 and p1 sodep p3, but p1 doesn't declare PKGDEP on p3, this situation is NOT reported. If p1 dep/sodep p2 dep p3, and p1 sodep p3, but p1 doesn't declare PKGDEP on p3, this situation is reported.