#!/usr/bin/python3 -I
"""walnut-games-helper install|remove|status|remove-data|install-repo-data ID, import-data ID PKG, adopt
(walnut-games package): the one root program the Walnut Games app and walnut-data run, through sudo
(/etc/sudoers.d/walnut-games: group sudo, no password, this path only).

Design
- The caller names a game by its catalog id, nothing else. The id must match ^[a-z0-9][a-z0-9-]{0,63}$
  and be in the packaged, root-owned /usr/share/walnut-games/catalog.json; the packages come from
  that file and each name must match ^[a-z0-9][a-z0-9+.-]*[a-z0-9]$. No package name, option or path
  comes from argv or the environment (python3 -I; every command gets a fixed PATH and environment).
  import-data's PKG is the one exception, and only as a choice: it must be one of the game's catalog
  data_setup packages (required or optional).
  Packages that carry licensed game data are refused whatever the catalog says: the catalog never
  lists them. The first lock is checked before every install: the candidate's apt Section must not
  be local/* or non-free/* (the licensed suite; game-data-packager marks non-redistributable data
  local/), and a Section that cannot be read refuses too. The second lock is the name: licensed or
  nonfree as a word of the package name.
- One action at a time: every action but status holds an flock on /run/walnut-games/lock (a root-owned
  directory: /run/lock is writable by everyone, where a user's file of that name would block the helper
  or be followed) for its whole run; a second one is refused at once ("busy", exit 2). The holder
  writes /run/walnut-games/state ("PID ACTION ID"); status takes no lock and reports "importing" while
  the lock's holder (/proc/locks) is that state's pid importing this game's data.
- install: apt-get update when the package lists are older than a day, then apt-get install with
  --no-remove (an install never takes a package away), the configuration files kept
  (--force-confold), noninteractive, waiting up to 60 s for another apt's lock. apt's status
  descriptor is read on a pipe of our own (sudo closes every descriptor above 2) and relayed.
- install of a game with catalog former_packages (older names of its packages: Debian's sauerbraten,
  which sauerbraten-walnut Conflicts/Replaces) one of which is installed: apt-get -s install first, every
  Remv must be one of those names (else refused, nothing changed), then the install without --no-remove
  when the simulation removes one. Any other install keeps --no-remove.
- remove of a game sharing a package with another catalog game that is installed (all its packages,
  and its data when it needs some: data_setup present_any): that package stays (an engine: dsda-doom
  for Freedoom, Doom and Heretic); what goes is the game's other packages and its installed
  data_setup packages (required, optional, present_any), and nothing else may (no metapackage).
- remove: apt-get -s remove first. It may remove the game's own packages and the old game metapackages
  (ps3-games, ps3-games-free: no longer built since 2026-10-28, but a console keeps the last version it
  had, which depends on every game); anything else in the simulation refuses the removal, before a
  single package is touched. When a metapackage is in the removal, the other installed packages it
  depends on are marked manual first, so removing one game never leaves the rest of the games waiting
  for an autoremove. This program never runs autoremove.
- adopt (no ID): every installed package of every catalog game (its packages and its data_setup
  required, optional and present_any names), and every installed dependency of an old metapackage
  still installed (OLD_METAS: the drafts and licensed data the catalog doesn't list, and
  ps3-games-integration), that apt has marked automatic is marked manual (apt-mark manual, those
  names only). The metapackages installed the games as their dependencies, so they are automatic:
  once a metapackage goes, apt-get autoremove would take every game with it.
  Walnut Games runs it at each start (sudo -n, failures ignored); the automatic marks are read from
  /var/lib/apt/extended_states, so when nothing is left to adopt it runs no apt-mark and changes
  nothing.
- import-data ID PKG: the player's own game files, as the data part of the deb game-data-packager
  made for them (walnut-data runs it as the player: `dpkg-deb --fsys-tarfile X.deb | sudo -n
  walnut-games-helper import-data ID PKG`), checked against the root-owned
  /usr/share/walnut-games/data-manifest.json (tools/gamedb/gdp-manifest.py: per package its files'
  paths, sizes and checksums, which are required, its version and its same-game Conflicts/Replaces).
  Refused while a game runs (ps3-game-mode-watch.service active). The tar stream on stdin is read
  member by member (tarfile 'r|', never extracted as such): regular files and directories, and the
  symlinks the manifest lists (gdp's own: Quake III's pk3s under /usr/lib/quake3, the exact path and
  target; each target relative and resolving under usr/, no file or link under a link path; made by
  this program after every file is written, all of them required); names relative, normalized,
  without '..', each once; the stream must end with tar's end-of-archive block (tarfile alone takes a
  stream cut between two members, or inside a header, for a whole one); a file must be a manifest path with one of its
  sizes (checked before its data is read) and, hashed while it is copied, one of its checksums; the
  files game-data-packager generates itself (usr/share/doc/PKG/copyright and changelog*,
  usr/share/lintian/, and for the Doom family usr/share/applications, pixmaps and icons) are read
  past and dropped (at most 64 MB of them); anything else refuses. Files are written by this program
  (O_CREAT|O_EXCL|O_NOFOLLOW, mode 0644 whatever the tar says) into a mkdtemp under
  /var/lib/walnut-games/import (0700, swept at the start, removed in finally), within the manifest's
  byte total; every required file must be there. The package's metadata is this program's own:
  DEBIAN/control (Package PKG, Version the manifest's, which is game-data-packager's: exempt from
  +ps3.N, Architecture all, Section local/games, a fixed description, Conflicts/Replaces only the
  manifest's), its doc/copyright, no maintainer scripts; dpkg-deb --root-owner-group -Znone -b. Then
  apt-get -s install --reinstall: the only Inst must be PKG and every Remv a manifest Conflicts/
  Replaces package (quake-registered replaces quake-shareware), or nothing is installed; then the
  install, with --no-remove unless the simulation removes such a package. Free space is checked
  in two steps: before the stream, room for the staged files (the manifest's total T) in the staging
  file system and, on another file system, T on /usr; once the stream has ended (walnut-data's own
  deb, whose space comes back only then, no longer counts), room for dpkg-deb's temporary data
  member and the deb (2T) beside the staged files, which go once the deb is made. Checking 3T up
  front, while walnut-data's deb still held T, refused a Quake III import (484 MB) that fitted. The package
  stays manually installed (apt-get install of a file): autoremove leaves it.
  Music packages (the manifest's tracks: quake-music, quake2-music) are the weaker case: CD audio ripped
  with cdparanoia or encoded from a BIN/CUE differs from copy to copy, so there is no checksum. A track
  must be one of the manifest's exact paths (usr/share/games/GAME/.../music/NN.ogg, checked against
  TRACK_RE when the manifest is read), each once, at most its max_size bytes (64 MB), and an Ogg Vorbis
  stream (its first page: OggS, version 0, beginning of stream, then the 01 'vorbis' identification
  header); every listed track must be there. What it holds beyond that is not checked: root writes a
  bounded file of the player's choice under the game's music directory, mode 0644, owned by root, read
  only by the game's Vorbis decoder. Free space for tracks is checked track by track from the stream's
  sizes (room: the file, dpkg-deb's temporary member and the deb), and /usr once the deb is made.
- remove-data ID: the game's installed data packages (data_setup required, optional and present_any:
  its data and [[licensed]] names), nothing else (apt-get -s remove first, as remove).
- install-repo-data ID: the first of the game's data_setup present_any names that the configured apt
  sources have (the licensed suite's builds of the same data: rct2-data), from apt, which checks the
  suite's signature; the licensed-Section lock is waived for exactly these names. apt-get -s first:
  every Inst must be one of those names, nothing is removed (--no-remove).
- SIGHUP and SIGPIPE are ignored: closing the app must not stop dpkg halfway. apt and dpkg get
  SIGPIPE back at its default (subprocess restore_signals); SIGHUP stays ignored for them.
- Output, one item per line, flushed: "plan STEP..." (the steps this action takes, once at its start:
  lists, download, install, remove; check, deb, apt for import-data), "step STEP text" (a step begins),
  "progress N text" (N percent of the current step), "count DONE TOTAL" (bytes of the current step,
  when known: the download, from apt's "Need to get" and its percentages, and import-data's checked
  copy), "log text", "error text", "status text". Walnut Games draws one bar over the steps from
  these (walnut_games.Progress). When stdout is gone the program keeps working and says nothing.
- Exit codes: 0 done, 2 refused (a message says why, nothing was changed; "busy" too), 1 apt, dpkg or
  dpkg-deb failed.
- stdlib only. The functions take the catalog path and a command runner so that
  packages/walnut/apps/walnut-games/tests/test_walnut_games.py runs them on a PC with a fake apt-get.
"""
import fcntl
import hashlib
import json
import os
import posixpath
import re
import selectors
import shutil
import signal
import subprocess
import sys
import tarfile
import tempfile
import time

CATALOG = "/usr/share/walnut-games/catalog.json"
LISTS = "/var/lib/apt/lists"
APT_GET, APT_MARK, DPKG_QUERY = "/usr/bin/apt-get", "/usr/bin/apt-mark", "/usr/bin/dpkg-query"
APT_CACHE = "/usr/bin/apt-cache"
DPKG_DEB, SYSTEMCTL = "/usr/bin/dpkg-deb", "/usr/bin/systemctl"
MANIFEST = "/usr/share/walnut-games/data-manifest.json"
RUN_DIR = "/run/walnut-games"            # the lock and the state file
IMPORT_DIR = "/var/lib/walnut-games/import"
USR = "/usr"
GAME_MODE_UNIT = "ps3-game-mode-watch.service"   # active while a game runs in game mode (ps3-game-mode on)
TRACK_EXPECT = 8 << 20     # a music track's Ogg Vorbis file at oggenc's default quality (~6 minutes)
SKIP_CAP = 64 << 20        # bytes of game-data-packager's own generated files read past in one import
SPACE_MARGIN = 64 << 20    # free space kept beyond the import's needs
VERSION_RE = re.compile(r"[0-9][A-Za-z0-9.+~-]{0,63}\Z")
ID_RE = re.compile(r"[a-z0-9][a-z0-9-]{0,63}\Z")
PKG_RE = re.compile(r"[a-z0-9][a-z0-9+.-]*[a-z0-9]\Z")
# the old metapackages that depended on the whole suite (built no more since 2026-10-28; consoles keep
# the last version they had): their removal along with a game is allowed (and prepared: their
# dependencies are marked manual first)
METAS = ("ps3-games", "ps3-games-free")
# adopt: the old metapackages whose dependencies it keeps (ps3-games-licensed: the licensed suite's, the
# same age), and apt's automatic marks
OLD_METAS = METAS + ("ps3-games-licensed",)
EXTENDED_STATES = "/var/lib/apt/extended_states"
LICENSED_RE = re.compile(r"(^|-)(licensed|nonfree)(-|$)")
LICENSED_SECTIONS = ("local/", "non-free/")
ENV = {"PATH": "/usr/sbin:/usr/bin:/sbin:/bin", "LC_ALL": "C", "LANGUAGE": "C", "DEBIAN_FRONTEND": "noninteractive",
       "APT_LISTCHANGES_FRONTEND": "none", "NEEDRESTART_MODE": "a", "HOME": "/root"}
APT_OPTS = ["-y", "-o", "Dpkg::Options::=--force-confold", "-o", "DPkg::Lock::Timeout=60",
            "-o", "APT::Get::Assume-Yes=true"]
STALE_SECONDS = 24 * 3600


class Refused(Exception):
    """a request the helper will not carry out (exit code 2)"""


class AptFailed(Exception):
    """apt-get or apt-mark failed (exit code 1)"""


CURRENT = {"step": None}     # the step said last ("step STEP text"): apt_stream goes on in it


def say(kind, text=""):
    if kind == "step":
        CURRENT["step"] = text.split(" ", 1)[0]
    try:
        sys.stdout.write((kind + " " + text).rstrip() + "\n")
        sys.stdout.flush()
    except (OSError, ValueError):   # the app is gone: carry on, dpkg must finish
        pass


# ---- the catalog and the names ----------------------------------------------------------------

def load_catalog(path=CATALOG, check_owner=True):
    """{id: entry}; with check_owner the file must be root's and not writable by anyone else"""
    st = os.stat(path)
    if check_owner and (st.st_uid != 0 or st.st_mode & 0o022):
        raise Refused("%s is not owned by root or is writable by others" % path)
    with open(path, encoding="utf-8") as f:
        data = json.load(f)
    return {g["id"]: g for g in data["games"] if isinstance(g.get("id"), str)}


def game_packages(game_id, games):
    """the validated package names of a game; Refused for a bad id, an unknown game, a bad name"""
    if not isinstance(game_id, str) or not ID_RE.match(game_id):
        raise Refused("not a game id: %r" % (game_id[:80] if isinstance(game_id, str) else type(game_id).__name__))
    entry = games.get(game_id)
    if entry is None:
        raise Refused("unknown game: %s" % game_id)
    pkgs = entry.get("packages")
    if not isinstance(pkgs, list) or not pkgs:
        raise Refused("%s lists no packages" % game_id)
    for p in pkgs:
        if not isinstance(p, str) or not PKG_RE.match(p):
            raise Refused("%s: bad package name %r" % (game_id, p))
        if LICENSED_RE.search(p):
            raise Refused("%s: %s carries licensed data and is not installed from here" % (game_id, p))
    return list(dict.fromkeys(pkgs))


# ---- running commands -------------------------------------------------------------------------

def run(cmd):
    """(returncode, stdout) of cmd with the fixed environment, stdin closed, stderr merged in"""
    p = subprocess.run(cmd, env=ENV, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                       text=True, errors="replace")
    return p.returncode, p.stdout


def installed(pkgs, runner=run):
    """the subset of pkgs that dpkg has installed (status ii, or an unpacked/half-configured state)"""
    rc, out = runner([DPKG_QUERY, "-W", "-f", "${Package}\t${db:Status-Abbrev}\n"] + list(pkgs))
    have = set()
    for line in out.splitlines():
        name, _, st = line.partition("\t")
        # the status letter: i installed, H/U/F/W/t part-way (still there); n, c are not installed
        if name in pkgs and st[1:2] and st[1:2] in "iHUFWt":
            have.add(name)
    return have


def lists_age(lists=LISTS):
    """seconds since the newest package list was fetched; None when there are none"""
    try:
        times = [os.stat(os.path.join(lists, n)).st_mtime for n in os.listdir(lists) if n.endswith("Packages")
                 or n.endswith("Packages.xz") or n.endswith("_Release")]
    except OSError:
        return None
    return time.time() - max(times) if times else None


def apt_steps(args):
    """the steps (for dlstatus, for pmstatus) an apt-get run reports in: update downloads the lists;
    install downloads and installs (a .deb of import-data: installs only, as "apt"); remove removes"""
    if args[:1] == ["update"]:
        return "lists", None
    if args[:1] == ["remove"]:
        return None, "remove"
    if any(a.endswith(".deb") for a in args):
        return None, "apt"
    return "download", "install"


STEP_TEXT = {"lists": "updating the package lists", "download": "downloading", "install": "installing",
             "remove": "removing", "apt": "installing the package"}


def apt_stream(args):
    """run apt-get ARGS with its status on our own pipe, relaying 'step', 'progress', 'count' and 'log'
    lines (apt_steps: which step its download and its unpacking count for); the return code. A line of
    apt's output that is an error stays in the returned tail."""
    r, w = os.pipe()
    cmd = [APT_GET] + APT_OPTS + ["-o", "APT::Status-Fd=%d" % w] + args
    p = subprocess.Popen(cmd, env=ENV, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                         pass_fds=(w,), text=False)
    os.close(w)
    sel = selectors.DefaultSelector()
    sel.register(r, selectors.EVENT_READ, "status")
    sel.register(p.stdout, selectors.EVENT_READ, "out")
    bufs = {"status": b"", "out": b""}
    tail = []
    steps = dict(zip(("dl", "pm"), apt_steps(args)))
    state = {"step": CURRENT["step"], "need": None}
    while sel.get_map():
        for key, _ in sel.select():
            fd = key.fileobj
            chunk = os.read(fd if isinstance(fd, int) else fd.fileno(), 4096)
            if not chunk:
                sel.unregister(fd)
                continue
            kind = key.data
            bufs[kind] += chunk
            while b"\n" in bufs[kind]:
                line, bufs[kind] = bufs[kind].split(b"\n", 1)
                text = line.decode("utf-8", "replace").rstrip()
                if kind == "status":
                    for msg in status_says(status_line(text), steps, state):
                        say(*msg)
                elif text:
                    n = need_bytes(text)
                    if n is not None:
                        state["need"] = n
                    say("log", text)
                    tail.append(text)
                    del tail[:-30]
    os.close(r)
    p.stdout.close()
    return p.wait(), tail


def status_line(line):
    """apt's APT::Status-Fd line as (kind, percent, text): "dl" (dlstatus: downloading), "pm" (pmstatus:
    unpacking, configuring, removing), "log" (pmerror, pmconffile: percent None); None for the rest"""
    parts = line.split(":", 3)
    if len(parts) < 4:
        return None
    kind, _, pct, text = parts
    if kind in ("pmerror", "pmconffile"):
        return ("log", None, "%s %s" % (kind, text.strip()))
    if kind not in ("dlstatus", "pmstatus"):
        return None
    try:
        pct = float(pct)
    except ValueError:
        return None
    return (kind[:2], max(0.0, min(100.0, pct)), text.strip())


def status_says(st, steps, state):
    """the lines one status_line result makes: a "step" line when apt moves into the step it counts for
    (steps: {"dl": step, "pm": step}), its "progress" in that step, and a "count" of the download's bytes
    once apt has said how much it needs (state["need"]); state["step"] is the step it is in"""
    if st is None:
        return []
    kind, pct, text = st
    if kind == "log":
        return [("log", text)]
    out, step = [], steps.get(kind)
    if step and state.get("step") != step:
        state["step"] = step
        out.append(("step", "%s %s" % (step, STEP_TEXT.get(step, step))))
    out.append(("progress", "%d %s" % (pct, text or ("downloading" if kind == "dl" else "installing"))))
    if kind == "dl" and state.get("need"):
        out.append(("count", "%d %d" % (state["need"] * pct / 100, state["need"])))
    return out


NEED_RE = re.compile(r"^Need to get ([0-9][0-9.,]*) ?([kMG]?B)(?:/([0-9][0-9.,]*) ?([kMG]?B))? of archives")
UNITS = {"B": 1, "kB": 1000, "MB": 1000 ** 2, "GB": 1000 ** 3}


def need_bytes(line):
    """the bytes apt-get still has to download, from its "Need to get 45.2 MB of archives." (or "12.0
    MB/45.2 MB", part of it in the cache: the first); None for any other line"""
    m = NEED_RE.match(line)
    if not m:
        return None
    try:
        return int(float(m.group(1).replace(",", "")) * UNITS[m.group(2)])
    except (KeyError, ValueError):
        return None


# ---- install and remove -----------------------------------------------------------------------

def check_sections(pkgs, runner=run):
    """the first lock: Refused when a package's candidate Section is local/* or non-free/*, or when
    a Section cannot be read (fail closed)"""
    rc, out = runner([APT_CACHE, "show", "--no-all-versions"] + list(pkgs))
    if rc != 0:
        raise Refused("cannot read the package sections (apt-cache failed, %d): nothing was changed" % rc)
    sections = {}
    for stanza in re.split(r"\n\s*\n", out):
        fields = {}
        for line in stanza.splitlines():
            k, sep, v = line.partition(":")
            if sep and not line[:1].isspace():
                fields.setdefault(k, v.strip())
        if "Package" in fields:
            sections.setdefault(fields["Package"], []).append(fields.get("Section", ""))
    for p in pkgs:
        secs = sections.get(p)
        if not secs or not all(secs):
            raise Refused("cannot read the section of %s: nothing was changed" % p)
        for sec in secs:
            if sec.startswith(LICENSED_SECTIONS):
                raise Refused("%s is in section %s: it carries licensed data and is not installed from here"
                              % (p, sec))


def do_install(game_id, games, stream=apt_stream, age=lists_age, runner=run):
    pkgs = game_packages(game_id, games)
    a = age()
    stale = a is None or a > STALE_SECONDS
    say("plan", ("lists " if stale else "") + "download install")
    if stale:
        say("step", "lists updating the package lists")
        rc, tail = stream(["update"])
        if rc != 0:
            # an old index may still hold the package; say so and try
            say("log", "apt-get update failed (%d): installing from the lists as they are" % rc)
    check_sections(pkgs, runner)
    former = former_packages(game_id, games, pkgs)
    gone = []
    if former and installed(former, runner):
        # a renamed package (sauerbraten-walnut Conflicts/Replaces sauerbraten): apt removes the old name,
        # which --no-remove refuses; the simulation may remove that game's former names and nothing else
        rc, sim = runner([APT_GET, "-s", "-o", "Debug::NoLocking=1", "install"] + pkgs)
        if rc != 0:
            say("error", "apt-get -s install failed: " + (sim.strip().splitlines() or ["?"])[-1])
            raise AptFailed()
        gone = removal_set(sim)
        extra = [p for p in gone if p not in former]
        if extra:
            raise Refused("installing %s would also remove %s: nothing was changed"
                          % (game_id, ", ".join(extra[:8]) + (" ..." if len(extra) > 8 else "")))
        if gone:
            say("log", "%s takes the place of %s" % (" ".join(pkgs), " ".join(gone)))
    say("step", "download downloading " + " ".join(pkgs))
    rc, tail = stream(["install"] + ([] if gone else ["--no-remove"]) + pkgs)
    if rc != 0:
        say("error", "apt-get install failed (%d)%s" % (rc, ": " + tail[-1] if tail else ""))
        raise AptFailed()
    say("progress", "100 installed")


def former_packages(game_id, games, pkgs):
    """the game's catalog former_packages (build.toml: older names of its packages, sauerbraten for
    sauerbraten-walnut), validated, without its current packages; [] when it has none"""
    v = games[game_id].get("former_packages") or []
    if not isinstance(v, list):
        raise Refused("%s: bad former_packages" % game_id)
    for p in v:
        if not isinstance(p, str) or not PKG_RE.match(p):
            raise Refused("%s: bad former package name %r" % (game_id, p))
    return [p for p in dict.fromkeys(v) if p not in pkgs]


def removal_set(sim_output):
    """the package names apt-get -s lists as 'Remv'"""
    return [m.group(1) for line in sim_output.splitlines() if (m := re.match(r"Remv (\S+)", line))]


def depends_of(pkg, runner=run):
    """the package names in pkg's Depends (every alternative), versions dropped"""
    rc, out = runner([DPKG_QUERY, "-W", "-f", "${Depends}", pkg])
    names = []
    for part in out.split(","):
        for alt in part.split("|"):
            m = re.match(r"\s*([a-z0-9][a-z0-9+.-]*)", alt)
            if m and PKG_RE.match(m.group(1)):
                names.append(m.group(1))
    return list(dict.fromkeys(names))


def data_packages(game_id, games):
    """the validated names in a game's catalog data_setup: required, optional and present_any (the
    licensed suite's builds of the same data); [] for a game without one"""
    ds = games.get(game_id, {}).get("data_setup")
    if not isinstance(ds, dict):
        return []
    names = []
    for k in ("required", "optional", "present_any"):
        v = ds.get(k) or []
        if not isinstance(v, list):
            raise Refused("%s: bad data_setup %s" % (game_id, k))
        for p in v:
            if not isinstance(p, str) or not PKG_RE.match(p):
                raise Refused("%s: bad data package name %r" % (game_id, p))
            names.append(p)
    return list(dict.fromkeys(names))


def in_use(other, games, runner=run):
    """another catalog game counts as installed: all its packages are, and, when it has data it can't
    run without (data_setup present_any, else required), one of those too"""
    g = games[other]
    pkgs = [p for p in g.get("packages") or [] if isinstance(p, str) and PKG_RE.match(p)]
    if not pkgs or len(installed(pkgs, runner)) != len(pkgs):
        return False
    ds = g.get("data_setup") if isinstance(g.get("data_setup"), dict) else {}
    need = [p for p in (ds.get("present_any") or ds.get("required") or []) if isinstance(p, str) and PKG_RE.match(p)]
    return not need or bool(installed(need, runner))


def shared_with(game_id, pkgs, games, runner=run):
    """{package: the other installed game listing it} for this game's packages another installed
    catalog game lists too (an engine several games run: dsda-doom for Freedoom, Doom, Heretic)"""
    out = {}
    for other in sorted(games):
        if other == game_id:
            continue
        common = [p for p in pkgs if p not in out and p in (games[other].get("packages") or [])]
        if common and in_use(other, games, runner):
            out.update({p: other for p in common})
    return out


def do_remove(game_id, games, runner=run, stream=apt_stream):
    pkgs = game_packages(game_id, games)
    data = data_packages(game_id, games)
    have = [p for p in pkgs if p in installed(pkgs, runner)]
    shared = shared_with(game_id, have, games, runner) if have else {}
    if shared:
        # the engine stays for the other game: only what is this game's alone goes, and its data
        say("log", "kept for %s: %s" % (", ".join(sorted(set(shared.values()))), " ".join(sorted(shared))))
        pkgs = [p for p in pkgs if p not in shared] + [p for p in data if p not in pkgs]
        have = [p for p in pkgs if p in installed(pkgs, runner)]
    if not have:
        say("progress", "100 not installed" if not shared else "100 nothing of this game's own to remove")
        return
    say("plan", "remove")
    say("step", "remove removing " + " ".join(have))
    rc, sim = runner([APT_GET, "-s", "-o", "Debug::NoLocking=1", "remove"] + have)
    if rc != 0:
        say("error", "apt-get -s remove failed: " + (sim.strip().splitlines() or ["?"])[-1])
        raise AptFailed()
    gone = removal_set(sim)
    extra = [p for p in gone if p not in pkgs and (shared or p not in METAS)]
    if extra:
        raise Refused("removing %s would also remove %s: nothing was changed"
                      % (game_id, ", ".join(extra[:8]) + (" ..." if len(extra) > 8 else "")))
    metas = [p for p in gone if p in METAS]
    if metas:
        keep = []
        for m in metas:
            keep += [d for d in depends_of(m, runner) if d not in pkgs]
        keep = [d for d in dict.fromkeys(keep) if d in installed(keep, runner)]
        if keep:
            say("progress", "5 keeping the rest of the games suite")
            rc, out = runner([APT_MARK, "manual"] + keep)
            if rc != 0:
                say("error", "apt-mark manual failed: " + (out.strip().splitlines() or ["?"])[-1])
                raise AptFailed()
    say("progress", "10 removing " + " ".join(have))
    rc, tail = stream(["remove"] + have)
    if rc != 0:
        say("error", "apt-get remove failed (%d)%s" % (rc, ": " + tail[-1] if tail else ""))
        raise AptFailed()
    say("progress", "100 removed")


def auto_marked(path=EXTENDED_STATES):
    """the package names apt has marked automatic (Auto-Installed: 1 in extended_states); a missing
    file is none"""
    try:
        with open(path, encoding="utf-8", errors="replace") as f:
            text = f.read()
    except FileNotFoundError:
        return set()
    auto = set()
    for stanza in re.split(r"\n\s*\n", text):
        name = re.search(r"(?m)^Package:\s*(\S+)", stanza)
        if name and re.search(r"(?m)^Auto-Installed:\s*1\s*$", stanza):
            auto.add(name.group(1))
    return auto


def adoptable(games):
    """every valid package name of every catalog game: its packages and its data_setup names"""
    names = []
    for gid in sorted(games):
        g = games[gid]
        pkgs = g.get("packages") if isinstance(g.get("packages"), list) else []
        try:
            data = data_packages(gid, games)
        except Refused:
            data = []
        names += [p for p in pkgs + data if isinstance(p, str) and PKG_RE.match(p)]
    return list(dict.fromkeys(names))


def do_adopt(games, runner=run, states=EXTENDED_STATES):
    """mark the installed, automatic packages of the catalog's games and of the old metapackages still
    installed manual (apt-mark manual)"""
    auto = auto_marked(states)
    names = adoptable(games)
    for m in sorted(installed(OLD_METAS, runner)):
        names += [d for d in depends_of(m, runner) if d not in names]
    cand = [p for p in names if p in auto]
    have = installed(cand, runner) if cand else set()
    todo = [p for p in cand if p in have]
    if not todo:
        say("progress", "100 nothing to adopt")
        return
    say("progress", "10 marking manual: " + " ".join(todo))
    rc, out = runner([APT_MARK, "manual"] + todo)
    if rc != 0:
        say("error", "apt-mark manual failed: " + (out.strip().splitlines() or ["?"])[-1])
        raise AptFailed()
    say("progress", "100 adopted %d" % len(todo))


def do_status(game_id, games, runner=run):
    pkgs = game_packages(game_id, games)
    have = installed(pkgs, runner)
    say("status", "installed" if len(have) == len(pkgs) else "partial" if have else "not-installed")


# ---- one action at a time ---------------------------------------------------------------------

class Busy(Refused):
    """another action holds the lock (exit code 2)"""


def run_dir(check_owner=True):
    """RUN_DIR, made root's 0755 when missing; Refused when it is a link, not a directory, or (with
    check_owner) not root's or writable by others"""
    try:
        os.mkdir(RUN_DIR, 0o755)
    except FileExistsError:
        pass
    st = os.lstat(RUN_DIR)
    if not stat_isdir(st.st_mode) or (check_owner and (st.st_uid != 0 or st.st_mode & 0o022)):
        raise Refused("%s is not a directory of root's" % RUN_DIR)
    return RUN_DIR


def stat_isdir(mode):
    return (mode & 0o170000) == 0o040000


def take_lock(action, game_id, check_owner=True):
    """the lock's descriptor, the state file written; Busy when another action holds it"""
    d = run_dir(check_owner)
    fd = os.open(os.path.join(d, "lock"), os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW | os.O_CLOEXEC, 0o600)
    try:
        fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        os.close(fd)
        held = lock_state(check_owner)
        raise Busy("busy: another walnut-games-helper action is running%s"
                   % (" (%s %s)" % (held[0], held[1]) if held else ""))
    tmp = os.path.join(d, "state.new")
    try:
        os.unlink(tmp)
    except FileNotFoundError:
        pass
    with os.fdopen(os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o644), "w") as f:
        f.write("%d %s %s\n" % (os.getpid(), action, game_id))
    os.replace(tmp, os.path.join(d, "state"))
    return fd


def release_lock(fd):
    try:
        os.unlink(os.path.join(RUN_DIR, "state"))
    except OSError:
        pass
    os.close(fd)


def lock_holder(path, locks="/proc/locks"):
    """the pid holding an flock on path (by its inode, /proc/locks), or None"""
    try:
        ino = os.lstat(path).st_ino
        with open(locks) as f:
            lines = f.read().splitlines()
    except OSError:
        return None
    for line in lines:
        parts = line.split()
        # "1: FLOCK  ADVISORY  WRITE 1234 00:1a:5678 0 EOF" (a blocked waiter's line has "->")
        if len(parts) >= 6 and parts[1] == "FLOCK" and parts[2] != "->" and parts[5].rsplit(":", 1)[-1] == str(ino):
            try:
                return int(parts[4])
            except ValueError:
                return None
    return None


def lock_state(check_owner=True):
    """(action, game id) of the running action, from the state file when its pid holds the lock; or None"""
    try:
        d = run_dir(check_owner)
        with open(os.path.join(d, "state")) as f:
            pid, action, gid = f.read().split()[:3]
        if lock_holder(os.path.join(d, "lock")) == int(pid):
            return action, gid
    except (OSError, ValueError, Refused):
        pass
    return None


# ---- the player's own data --------------------------------------------------------------------

def load_manifest(path=None, check_owner=True):
    """{package: entry} of the root-owned data manifest (tools/gamedb/gdp-manifest.py)"""
    path = path or MANIFEST
    st = os.stat(path)
    if check_owner and (st.st_uid != 0 or st.st_mode & 0o022):
        raise Refused("%s is not owned by root or is writable by others" % path)
    with open(path, encoding="utf-8") as f:
        data = json.load(f)
    return data["packages"], str(data.get("gdp_version", "?"))


def data_setup_of(game_id, games):
    """the game's catalog data_setup (validated id); Refused for an unknown game or one without one"""
    if not isinstance(game_id, str) or not ID_RE.match(game_id):
        raise Refused("not a game id: %r" % (game_id[:80] if isinstance(game_id, str) else type(game_id).__name__))
    if game_id not in games:
        raise Refused("unknown game: %s" % game_id)
    ds = games[game_id].get("data_setup")
    if not isinstance(ds, dict):
        raise Refused("%s has no data to set up" % game_id)
    return ds


def importable(game_id, pkg, games):
    """PKG when it is one of the game's data_setup required or optional packages, else Refused"""
    ds = data_setup_of(game_id, games)
    if not isinstance(pkg, str) or not PKG_RE.match(pkg):
        raise Refused("not a package name: %r" % (pkg[:80] if isinstance(pkg, str) else type(pkg).__name__))
    allowed = [p for k in ("required", "optional") for p in (ds.get(k) or []) if isinstance(p, str)]
    if pkg not in allowed:
        raise Refused("%s is not a data package of %s" % (pkg, game_id))
    return pkg


def check_entry(pkg, entry):
    """the manifest entry of pkg, its names checked (a broken manifest refuses, it never writes odd fields)"""
    if not isinstance(entry, dict) or not isinstance(entry.get("files"), dict) or \
            not (entry["files"] or entry.get("tracks")):
        raise Refused("the data manifest has no files for %s" % pkg)
    if "tracks" in entry:
        check_tracks(pkg, entry)
    if not VERSION_RE.match(str(entry.get("version", ""))) or not ID_RE.match(str(entry.get("game", ""))):
        raise Refused("the data manifest's version or game of %s is not usable" % pkg)
    if entry.get("edition") not in EDITIONS:
        raise Refused("the data manifest's edition of %s is not one of %s" % (pkg, ", ".join(EDITIONS)))
    for k in ("conflicts", "replaces"):
        if not all(isinstance(p, str) and PKG_RE.match(p) and p != pkg for p in entry.get(k) or []):
            raise Refused("the data manifest's %s of %s are not package names" % (k, pkg))
    check_symlinks(pkg, entry)
    for path, alts in entry["files"].items():
        if member_path(path) != path or not isinstance(alts, list) or not alts or \
                not all(isinstance(a, dict) and isinstance(a.get("size"), int) and a["size"] >= 0
                        and any(k in a for k in HASHES)
                        and all(isinstance(a[k], str) and re.match(r"[0-9a-fA-F]{32,64}\Z", a[k]) for k in HASHES if k in a)
                        for a in alts):
            raise Refused("the data manifest's entry %r of %s is not usable" % (path[:80], pkg))
    if not all(p in entry["files"] for p in entry.get("required") or []):
        raise Refused("the data manifest's required files of %s are not among its files" % pkg)
    return entry


LINKS_MAX = 256


def check_symlinks(pkg, entry):
    """a package's symlinks entry, {path: target}: the links game-data-packager puts in its deb (Quake III's
    /usr/lib/quake3/base/baseq3/pak0.pk3 -> ../../../../share/games/quake3-data/baseq3/pak0.pk3). Each
    path a plain relative name that is none of its files or tracks and has no file, track or other link
    under it (nothing is ever written through a link); each target relative, at most 255 bytes, no NUL,
    resolving from the link's directory to a path under usr/."""
    links = entry.get("symlinks") or {}
    tracks = (entry.get("tracks") or {}).get("paths") or [] if isinstance(entry.get("tracks"), dict) else []
    if not isinstance(links, dict) or len(links) > LINKS_MAX:
        raise Refused("the data manifest's symlinks of %s are not usable" % pkg)
    others = set(entry["files"]) | set(tracks)
    every = others | set(links)
    for path, target in links.items():
        ok = isinstance(path, str) and member_path(path) == path and path and path not in others and \
            isinstance(target, str) and 0 < len(target.encode()) <= 255 and not target.startswith("/") and \
            "\0" not in target and \
            posixpath.normpath(posixpath.join(posixpath.dirname(path), target)).startswith("usr/") and \
            not any(p.startswith(path + "/") for p in every)
        if not ok:
            raise Refused("the data manifest's symlink %r of %s is not usable" % (str(path)[:80], pkg))


def check_tracks(pkg, entry):
    """a music package's tracks entry: exact paths, each a numbered .ogg file directly in a music/ directory
    under /usr/share/games/GAME, not one of its checksummed files; max_size a positive int up to
    TRACK_MAX_LIMIT"""
    t = entry["tracks"]
    paths = t.get("paths") if isinstance(t, dict) else None
    size = t.get("max_size") if isinstance(t, dict) else None
    if not isinstance(paths, list) or not paths or len(paths) > 99 or len(set(paths)) != len(paths) or \
            not isinstance(size, int) or isinstance(size, bool) or not 0 < size <= TRACK_MAX_LIMIT or \
            not all(isinstance(p, str) and TRACK_RE.match(p) and member_path(p) == p and p not in entry["files"]
                    for p in paths):
        raise Refused("the data manifest's tracks of %s are not usable" % pkg)


# a music track's installed path: usr/share/games/GAME/[DIR/]music/[NAME]NN.ogg
TRACK_RE = re.compile(r"usr/share/games/[a-z0-9][a-z0-9-]*/(?:[A-Za-z0-9_-]+/)?music/[A-Za-z_-]{0,16}[0-9]{2}\.ogg\Z")
TRACK_MAX_LIMIT = 256 << 20


def ogg_vorbis_start(head):
    """whether head (a file's first bytes) starts an Ogg Vorbis stream: an Ogg page (capture pattern OggS,
    version 0) that begins the stream (header type BOS), whose first packet is Vorbis's identification
    header (01 'vorbis')"""
    if len(head) < 28 or head[:4] != b"OggS" or head[4] != 0 or not head[5] & 0x02:
        return False
    nseg = head[26]
    start = 27 + nseg
    return head[start:start + 7] == b"\x01vorbis"


HASHES = ("sha256", "sha1", "md5")
EDITIONS = ("shareware", "demo", "full", "expansion", "music", "remaster")
RELEASE_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9._+-]{0,79}\Z")


def member_path(name):
    """a tar member's name as a relative path ('./usr/x' -> 'usr/x', the top directory ''), or None when
    it is absolute, has '..', '.' or empty parts, or a NUL"""
    n = name
    while n.startswith("./"):
        n = n[2:]
    if n in ("", ".", "./"):
        return ""
    n = n[:-1] if n.endswith("/") else n
    if not n or n.startswith("/") or "\0" in n or any(p in ("", ".", "..") for p in n.split("/")):
        return None
    return n if posixpath.normpath(n) == n else None


def generated(path, pkg, entry):
    """a file game-data-packager writes itself into every build (its changelog and copyright, lintian
    overrides) or, for the Doom family, its menu entries and icons: dropped from the stream"""
    doc = "usr/share/doc/%s/" % pkg
    if path.startswith(doc):
        rest = path[len(doc):]
        if "/" not in rest and (rest == "copyright" or rest.startswith("changelog")):
            return True
    if path.startswith("usr/share/lintian/"):
        return True
    return bool(entry.get("launchers")) and path.startswith(
        ("usr/share/applications/", "usr/share/pixmaps/", "usr/share/icons/"))


def game_running(runner=run):
    """a game runs in game mode (its watcher unit is active); False without systemctl (no game mode)"""
    try:
        rc, out = runner([SYSTEMCTL, "is-active", GAME_MODE_UNIT])
    except FileNotFoundError:
        return False
    return out.strip() == "active"


def free_bytes(path):
    st = os.statvfs(path)
    return st.f_bavail * st.f_frsize, os.stat(path).st_dev


def check_space(total, stage_dir, usr=None, streamed=False):
    """Refused unless there is room for what comes next. Before the stream: the staged files (total) in
    the staging file system and, on another file system, /usr for the installed files (on one file
    system the staged files go once the deb is made, before the install). streamed: the files are
    staged, walnut-data's deb that held their space too is gone; dpkg-deb's temporary data member and
    the deb need twice the total more."""
    usr = usr or USR
    s_free, s_dev = free_bytes(stage_dir)
    u_free, u_dev = free_bytes(usr)
    need_s = (2 if streamed else 1) * total + SPACE_MARGIN
    need_u = 0 if streamed or s_dev == u_dev else total + SPACE_MARGIN
    for path, free, need in ((stage_dir, s_free, need_s), (usr, u_free, need_u)):
        if need and free < need:
            raise Refused("not enough space: %s has %d MB free, the import needs %d MB"
                          % (path, free >> 20, (need + (1 << 20) - 1) >> 20))


def import_dir(check_owner=True):
    """IMPORT_DIR (root's, 0700), every staging directory a crashed import left swept away"""
    os.makedirs(os.path.dirname(IMPORT_DIR), 0o755, exist_ok=True)
    try:
        os.mkdir(IMPORT_DIR, 0o700)
    except FileExistsError:
        pass
    st = os.lstat(IMPORT_DIR)
    if not stat_isdir(st.st_mode) or (check_owner and st.st_uid != 0) or st.st_mode & 0o077:
        raise Refused("%s is not a private directory of root's" % IMPORT_DIR)
    for n in os.listdir(IMPORT_DIR):   # we hold the lock: nothing in here is in use
        p = os.path.join(IMPORT_DIR, n)
        if os.path.isdir(p) and not os.path.islink(p):
            shutil.rmtree(p, ignore_errors=True)
        else:
            os.unlink(p)
    return IMPORT_DIR


def read_stream(src, pkg, entry, root, cap):
    """the files of the tar stream src written under root, checked against the manifest entry;
    {path: the release name of the alternative it matched, or None}. Refused on anything else."""
    files = entry["files"]
    tracks = set((entry.get("tracks") or {}).get("paths") or [])
    tracks_max = (entry.get("tracks") or {}).get("max_size", 0)
    links = entry.get("symlinks") or {}
    done, total, skipped = {}, 0, 0
    # what the copy is expected to come to, for the count lines: each file at its largest size, a music
    # track at its expected size
    want = sum(max(a["size"] for a in alts) for alts in files.values()) + len(tracks) * TRACK_EXPECT
    src = StreamTail(src)
    try:
        tf = tarfile.open(fileobj=src, mode="r|")
        for m in tf:
            path = member_path(m.name)
            if path is None:
                raise Refused("not a plain relative name in the data: %r" % m.name[:120])
            if m.type == tarfile.DIRTYPE:
                continue           # this program makes the directories its files need
            if m.type == tarfile.SYMTYPE and path in links:
                if m.linkname != links[path] or path in done:
                    raise Refused("%s in the data is not the link game-data-packager makes" % path[:120])
                done[path] = None  # made after every file is written (none is under a link)
                continue
            if m.type not in (tarfile.REGTYPE, tarfile.AREGTYPE):
                raise Refused("%s in the data is not a regular file (a link or a device)" % path[:120])
            if path in done or path == "":
                raise Refused("%s is in the data twice" % path[:120])
            if path in tracks:
                if m.size > tracks_max:
                    raise Refused("%s is larger than a music track may be (%d bytes)" % (path[:120], m.size))
                total += m.size
                if total > cap:
                    raise Refused("the data is larger than %s's files" % pkg)
                room(root, m.size, total - m.size)
                say("progress", "%d checking %s" % (min(99, 100 * len(done) // max(len(files) + len(tracks), 1)), path))
                copy_ogg(tf.extractfile(m), m.size, root, path)
                done[path] = None
                continue
            if path not in files:
                if generated(path, pkg, entry):
                    skipped += m.size
                    if skipped > SKIP_CAP:
                        raise Refused("too much besides the game's files in the data")
                    done[path] = None
                    continue
                raise Refused("%s is not a file of %s" % (path[:120], pkg))
            alts = [a for a in files[path] if a["size"] == m.size]
            if not alts:
                raise Refused("%s has the wrong size (%d bytes)" % (path[:120], m.size))
            total += m.size
            if total > cap:
                raise Refused("the data is larger than %s's files" % pkg)
            say("progress", "%d checking %s" % (min(99, 100 * total // max(want, 1)), path))
            say("count", "%d %d" % (total, want))
            done[path] = copy_checked(tf.extractfile(m), m.size, alts, root, path)
        if not src.zero_block_at(tf.offset):
            # tarfile ends quietly at a stream cut between two members or inside a header
            raise Refused("the data stream is cut short (it stops without tar's end-of-archive block)")
    except tarfile.TarError as e:
        raise Refused("the data stream is broken or cut short (%s)" % e)
    missing = [p for p in list(entry.get("required") or []) + sorted(tracks) + sorted(links) if p not in done]
    if missing:
        raise Refused("%d of %s's files are missing, e.g. %s" % (len(missing), pkg, missing[0]))
    for path in sorted(links):
        dest = os.path.join(root, path)
        os.makedirs(os.path.dirname(dest), 0o755, exist_ok=True)
        try:
            os.symlink(links[path], dest)
        except FileExistsError:
            raise Refused("%s is in the data twice" % path[:120])
    return {p: r for p, r in done.items() if p in files}


class StreamTail:
    """a file object read through, its last bytes kept: tarfile in stream mode ends without an error when
    the stream stops at a member's end or inside a header (dpkg-deb --fsys-tarfile cut short), so the
    end-of-archive block (512 zero bytes) is looked for where tarfile read its last header"""
    KEEP = 4 * tarfile.RECORDSIZE

    def __init__(self, f):
        self.f, self.total, self.kept, self.chunks = f, 0, 0, []

    def read(self, n=-1):
        b = self.f.read(n)
        if b:
            self.total += len(b)
            self.kept += len(b)
            self.chunks.append(b)
            while len(self.chunks) > 1 and self.kept - len(self.chunks[0]) >= self.KEEP:
                self.kept -= len(self.chunks.pop(0))
        return b

    def zero_block_at(self, offset):
        start = offset - (self.total - self.kept)
        tail = b"".join(self.chunks)
        return start >= 0 and len(tail) >= start + tarfile.BLOCKSIZE and \
            tail[start:start + tarfile.BLOCKSIZE] == bytes(tarfile.BLOCKSIZE)


def room(root, size, written):
    """Refused unless the staging file system still has room for a file of size bytes after the written
    ones, then for dpkg-deb's temporary data member and the deb of them all (a music package's size is
    known only from its stream)"""
    free, _ = free_bytes(root)
    if free < 3 * size + 2 * written + SPACE_MARGIN:
        raise Refused("not enough space: %s has %d MB free, the music needs about %d MB more"
                      % (root, free >> 20, (3 * size + 2 * written + SPACE_MARGIN) >> 20))


def copy_ogg(f, size, root, path):
    """a music track (no checksum: ripped and encoded audio differs from copy to copy) written like
    copy_checked; Refused unless it is an Ogg Vorbis stream"""
    first = []

    def check(chunk):
        if not first:
            first.append(True)
            if not ogg_vorbis_start(chunk[:1024]):
                raise Refused("%s is not an Ogg Vorbis file" % path[:120])

    copy_file(f, size, root, path, check)
    if not first:
        raise Refused("%s is empty" % path[:120])


def copy_file(f, size, root, path, on_chunk):
    """f's size bytes written to root/path (a new file, mode 0644), each chunk given to on_chunk first;
    Refused when it is cut short (the file stays: the staging directory goes as a whole)"""
    dest = os.path.join(root, path)
    os.makedirs(os.path.dirname(dest), 0o755, exist_ok=True)
    fd = os.open(dest, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW | os.O_CLOEXEC, 0o644)
    got = 0
    try:
        os.fchmod(fd, 0o644)
        while True:
            chunk = f.read(1 << 20)
            if not chunk:
                break
            got += len(chunk)
            if got > size:
                break
            on_chunk(chunk)
            view = memoryview(chunk)
            while view:
                view = view[os.write(fd, view):]
    finally:
        os.close(fd)
    if got != size:
        raise Refused("%s is cut short in the data" % path[:120])


def copy_checked(f, size, alts, root, path):
    """f's size bytes written to root/path (a new file, mode 0644), hashed on the way; the release name
    of the alternative whose checksum it has. Refused for none."""
    algos = sorted({k for a in alts for k in HASHES if k in a})
    hs = {k: hashlib.new(k) for k in algos}

    def update(chunk):
        for h in hs.values():
            h.update(chunk)

    copy_file(f, size, root, path, update)
    digest = {k: h.hexdigest() for k, h in hs.items()}
    for a in alts:
        if all(digest[k] == a[k].lower() for k in HASHES if k in a):
            rel = a.get("release")
            return rel if isinstance(rel, str) and RELEASE_RE.match(rel) else None
    raise Refused("%s is not a copy game-data-packager knows (its checksum differs)" % path[:120])


def release_label(found, entry):
    """X-Walnut-Release: the release names of the alternatives the required files matched ("cd",
    "1.9ud"; several joined with +), or None when gdp names none"""
    names = sorted({r for p, r in found.items() if r and p in (entry.get("required") or [])})
    label = "+".join(names)
    return label if names and RELEASE_RE.match(label) else None


def control_text(pkg, entry, gdp_version, release, installed_kb):
    lines = ["Package: " + pkg, "Version: " + entry["version"], "Architecture: all", "Section: local/games",
             "Priority: optional", "Maintainer: Walnut <walnut@localhost>", "Installed-Size: %d" % installed_kb]
    if entry.get("conflicts"):
        lines.append("Conflicts: " + ", ".join(entry["conflicts"]))
    if entry.get("replaces"):
        lines.append("Replaces: " + ", ".join(entry["replaces"]))
    lines.append("X-Walnut-Edition: " + entry["edition"])
    if release:
        lines.append("X-Walnut-Release: " + release)
    lines += ["Description: game data for %s, from the player's own copy" % entry["game"],
              " Built on this console by Walnut Games (walnut-games-helper import-data) from the",
              " player's own files, each checked against game-data-packager %s's checksums." % gdp_version,
              " Not redistributable."]
    return "\n".join(lines) + "\n"


COPYRIGHT = """This package was built on this console by Walnut Games (walnut-data import, then
walnut-games-helper import-data) from the player's own copy of the game: game-data-packager {gdp}'s
package {pkg}, every file checked against its checksums.

The game data is the copyrighted work of its publishers. It comes from the player's own copy and is
not redistributable: do not share this package or its files.
"""


def sim_sets(out):
    """(the names apt-get -s installs, the names it removes)"""
    inst = [m.group(1) for line in out.splitlines() if (m := re.match(r"Inst (\S+)", line))]
    return inst, removal_set(out)


def do_import_data(game_id, pkg, games, src, manifest, gdp_version, runner=run, stream=apt_stream,
                   check_owner=True):
    importable(game_id, pkg, games)
    if pkg not in manifest:
        raise Refused("the data manifest has no %s" % pkg)
    entry = check_entry(pkg, manifest[pkg])
    if game_running(runner):
        raise Refused("a game is running: close it first")
    os.umask(0o022)
    base = import_dir(check_owner)
    tr = entry.get("tracks") or {}
    files_cap = sum(max(a["size"] for a in alts) for alts in entry["files"].values())
    cap = files_cap + len(tr.get("paths") or []) * tr.get("max_size", 0)
    # a music package's tracks are checked for room one by one as they come (room): counted here at
    # their expected size, not their bound
    check_space(files_cap + len(tr.get("paths") or []) * TRACK_EXPECT, base)
    stage = tempfile.mkdtemp(prefix="import-", dir=base)
    try:
        root = os.path.join(stage, "root")
        os.mkdir(root, 0o755)
        say("plan", "check deb apt")
        say("step", "check checking the files of " + pkg)
        found = read_stream(src, pkg, entry, root, cap)
        doc = os.path.join(root, "usr/share/doc", pkg)
        os.makedirs(doc, 0o755, exist_ok=True)
        with os.fdopen(os.open(os.path.join(doc, "copyright"), os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW,
                               0o644), "w") as f:
            f.write(COPYRIGHT.format(gdp=gdp_version, pkg=pkg))
        kb = 0
        for d, dirs, names in os.walk(root):
            kb += len(dirs) + sum((os.lstat(os.path.join(d, n)).st_size + 1023) // 1024 for n in names)
        check_space(kb * 1024, base, streamed=True)
        os.mkdir(os.path.join(root, "DEBIAN"), 0o755)
        release = release_label(found, entry)
        with os.fdopen(os.open(os.path.join(root, "DEBIAN/control"), os.O_WRONLY | os.O_CREAT | os.O_EXCL
                               | os.O_NOFOLLOW, 0o644), "w") as f:
            f.write(control_text(pkg, entry, gdp_version, release, kb))
        deb = os.path.join(stage, "%s.deb" % pkg)
        say("step", "deb making the package " + pkg)
        # dpkg-deb writes the data member to a temporary file in TMPDIR first (/tmp is a 55 MB tmpfs
        # on the PS3: RCT2's 550 MB failed with ENOSPC there), so it goes next to the deb
        ENV["TMPDIR"] = stage
        try:
            rc, out = runner([DPKG_DEB, "--root-owner-group", "-Znone", "-b", root, deb])
        finally:
            ENV.pop("TMPDIR", None)
        if rc != 0:
            say("error", "dpkg-deb failed: " + (out.strip().splitlines() or ["?"])[-1])
            raise AptFailed()
        shutil.rmtree(root)                 # the deb has it: room for the installed copy
        if tr.get("paths") and free_bytes(USR)[0] < kb * 1024 + SPACE_MARGIN:
            raise Refused("not enough space: %s has %d MB free, the music needs %d MB"
                          % (USR, free_bytes(USR)[0] >> 20, (kb * 1024 + SPACE_MARGIN) >> 20))
        rc, sim = runner([APT_GET, "-s", "-o", "Debug::NoLocking=1", "install", "--no-install-recommends",
                          "--reinstall", deb])
        if rc != 0:
            say("error", "apt-get -s install failed: " + (sim.strip().splitlines() or ["?"])[-1])
            raise AptFailed()
        inst, gone = sim_sets(sim)
        allowed = set(entry.get("conflicts") or []) | set(entry.get("replaces") or [])
        if inst != [pkg] or [p for p in gone if p not in allowed]:
            extra = [p for p in inst if p != pkg] + [p for p in gone if p not in allowed]
            raise Refused("installing %s would also change %s: nothing was changed"
                          % (pkg, ", ".join(extra[:8]) or "nothing (apt would not install it)"))
        if gone:
            say("log", "%s takes the place of %s" % (pkg, ", ".join(gone)))
        say("step", "apt installing " + pkg)
        rc, tail = stream(["install", "--no-install-recommends", "--reinstall"] + ([] if gone else ["--no-remove"])
                          + [deb])
        if rc != 0:
            say("error", "apt-get install failed (%d)%s" % (rc, ": " + tail[-1] if tail else ""))
            raise AptFailed()
        say("progress", "100 installed " + pkg + (" (%s, %s)" % (entry["edition"], release) if release else
                                                   " (%s)" % entry["edition"]))
    finally:
        shutil.rmtree(stage, ignore_errors=True)


def do_remove_data(game_id, games, runner=run, stream=apt_stream):
    data_setup_of(game_id, games)
    if game_running(runner):
        raise Refused("a game is running: close it first")
    names = data_packages(game_id, games)
    have = [p for p in names if p in installed(names, runner)]
    if not have:
        say("progress", "100 no data of %s installed" % game_id)
        return
    say("plan", "remove")
    say("step", "remove removing " + " ".join(have))
    rc, sim = runner([APT_GET, "-s", "-o", "Debug::NoLocking=1", "remove"] + have)
    if rc != 0:
        say("error", "apt-get -s remove failed: " + (sim.strip().splitlines() or ["?"])[-1])
        raise AptFailed()
    extra = [p for p in removal_set(sim) if p not in names]
    if extra:
        raise Refused("removing %s's data would also remove %s: nothing was changed"
                      % (game_id, ", ".join(extra[:8]) + (" ..." if len(extra) > 8 else "")))
    say("progress", "10 removing " + " ".join(have))
    rc, tail = stream(["remove"] + have)
    if rc != 0:
        say("error", "apt-get remove failed (%d)%s" % (rc, ": " + tail[-1] if tail else ""))
        raise AptFailed()
    say("progress", "100 removed")


def candidates(names, runner=run):
    """the names among names that the apt sources have a candidate for (apt-cache policy)"""
    rc, out = runner([APT_CACHE, "policy"] + list(names))
    have, cur = set(), None
    for line in out.splitlines():
        m = re.match(r"(\S+):\s*$", line)
        if m:
            cur = m.group(1)
        elif cur and (m := re.match(r"\s+Candidate:\s*(\S+)", line)) and m.group(1) != "(none)":
            have.add(cur)
    return [n for n in names if n in have]


def do_install_repo_data(game_id, games, stream=apt_stream, age=lists_age, runner=run):
    ds = data_setup_of(game_id, games)
    names = data_packages(game_id, games)
    any_of = [p for p in ds.get("present_any") or [] if p in names]
    if installed(any_of, runner):
        say("progress", "100 the data of %s is installed" % game_id)
        return
    a = age()
    stale = a is None or a > STALE_SECONDS
    say("plan", ("lists " if stale else "") + "download install")
    if stale:
        say("step", "lists updating the package lists")
        rc, tail = stream(["update"])
        if rc != 0:
            say("log", "apt-get update failed (%d): looking in the lists as they are" % rc)
    have = candidates(any_of, runner)
    if not have:
        raise Refused("your package sources have none of %s" % ", ".join(any_of))
    pkg = have[0]
    rc, sim = runner([APT_GET, "-s", "-o", "Debug::NoLocking=1", "install", "--no-remove", "--no-install-recommends", pkg])
    if rc != 0:
        say("error", "apt-get -s install failed: " + (sim.strip().splitlines() or ["?"])[-1])
        raise AptFailed()
    inst, gone = sim_sets(sim)
    extra = [p for p in inst if p not in names] + gone
    if pkg not in inst or extra:
        raise Refused("installing %s would also change %s: nothing was changed"
                      % (pkg, ", ".join(extra[:8]) or "nothing (apt would not install it)"))
    say("step", "download downloading " + pkg)
    # the licensed-Section lock (check_sections) is waived: these are the game's data names, from apt
    rc, tail = stream(["install", "--no-remove", "--no-install-recommends", pkg])
    if rc != 0:
        say("error", "apt-get install failed (%d)%s" % (rc, ": " + tail[-1] if tail else ""))
        raise AptFailed()
    say("progress", "100 installed " + pkg)


# ---- main -------------------------------------------------------------------------------------

ACTIONS = {"install": 3, "remove": 3, "status": 3, "remove-data": 3, "install-repo-data": 3, "import-data": 4,
           "adopt": 2}


def main(argv, catalog=CATALOG, check_owner=True, need_root=True, stdin=None, manifest=None):
    if len(argv) < 2 or ACTIONS.get(argv[1]) != len(argv):
        print("usage: walnut-games-helper install|remove|status|remove-data|install-repo-data ID\n"
              "       walnut-games-helper import-data ID PKG   (the data's tar stream on stdin)\n"
              "       walnut-games-helper adopt", file=sys.stderr)
        return 2
    signal.signal(signal.SIGHUP, signal.SIG_IGN)
    signal.signal(signal.SIGPIPE, signal.SIG_IGN)
    action, game_id = argv[1], (argv[2] if len(argv) > 2 else "-")
    lock = None
    try:
        if need_root and os.geteuid() != 0:
            raise Refused("must run as root (through sudo)")
        if action == "adopt":
            games = load_catalog(catalog, check_owner)
            lock = take_lock(action, game_id, check_owner)
            do_adopt(games)
            return 0
        if not ID_RE.match(game_id):
            raise Refused("not a game id: %r" % game_id[:80])
        if action == "import-data" and not PKG_RE.match(argv[3]):
            raise Refused("not a package name: %r" % argv[3][:80])
        games = load_catalog(catalog, check_owner)
        if action == "status":
            if lock_state(check_owner) == ("import-data", game_id):
                say("status", "importing")
            else:
                do_status(game_id, games)
            return 0
        if action in ("install", "remove"):   # the request checked before the lock: a bad one never waits
            game_packages(game_id, games)
        elif action == "import-data":
            importable(game_id, argv[3], games)
        else:
            data_setup_of(game_id, games)
        lock = take_lock(action, game_id, check_owner)
        if action == "import-data":
            man, gdp_version = load_manifest(manifest, check_owner)
            do_import_data(game_id, argv[3], games, stdin if stdin is not None else sys.stdin.buffer, man,
                           gdp_version, check_owner=check_owner)
        else:
            {"install": do_install, "remove": do_remove, "remove-data": do_remove_data,
             "install-repo-data": do_install_repo_data}[action](game_id, games)
    except Refused as e:
        say("error", str(e))
        print("walnut-games-helper: " + str(e), file=sys.stderr)
        return 2
    except AptFailed:
        return 1
    except (OSError, ValueError, KeyError) as e:   # a missing or broken catalog, apt-get not found
        say("error", "%s: %s" % (type(e).__name__, e))
        return 1
    finally:
        if lock is not None:
            release_lock(lock)
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv))
