aboutsummaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorSoccera1 <[email protected]>2026-07-13 18:33:53 +1000
committerSoccera1 <[email protected]>2026-07-13 18:33:53 +1000
commit876afc0c11d00bdcb50343ccd2e408810e6c9dff (patch)
tree297f7ec6832485c46626a40baca91b0e9088b9d1 /docs
parent6e2513d28da68283e795e2a04a7d68cb3af1cb7a (diff)
downloadbnu-876afc0c11d00bdcb50343ccd2e408810e6c9dff.tar.gz
bnu-876afc0c11d00bdcb50343ccd2e408810e6c9dff.zip
Re-write docs
Diffstat (limited to 'docs')
-rw-r--r--docs/gnu-test-boundaries.md89
-rw-r--r--docs/qemu-test-matrix.md155
-rw-r--r--docs/testing.md114
3 files changed, 299 insertions, 59 deletions
diff --git a/docs/gnu-test-boundaries.md b/docs/gnu-test-boundaries.md
index ebac2ef..29ef1ca 100644
--- a/docs/gnu-test-boundaries.md
+++ b/docs/gnu-test-boundaries.md
@@ -1,77 +1,48 @@
-# GNU coreutils 9.11 test boundaries
+# GNU test boundaries
## Scope
-BNU tracks the 733 `.sh` and `.pl` command-test scripts in GNU coreutils
-9.11's standard `TESTS = $(all_tests) $(factor_tests)` inventory. Individual
-scripts can contain multiple assertions. GNU's C-internal `gnulib-tests` suite
-is outside this JavaScript command-compatibility scope.
+BNU tracks the 733 `.sh` and `.pl` scripts in GNU coreutils 9.11's standard
+command-test inventory. A script can contain multiple assertions. GNU's
+C-internal `gnulib-tests` are outside this scope.
-The upstream `all_root_tests` variable contains 41 names that are already part
-of the 733; they are routed to a root guest rather than counted twice. The
-runner derives both sets from the extracted upstream `Makefile.in`, currently
-yielding 692 non-root tests and 41 root tests.
+The inventory contains 692 non-root tests and 41 tests GNU marks as root-only.
+The QEMU matrix routes those groups, plus security-module cases, to suitable
+guests.
-The accepted result is 727 tests with a `PASS` record on at least one suitable
-platform and the following six Bun runtime or target-ABI boundaries. A normal
-BNU JavaScript command cannot begin in any of these six environments.
+The accepted matrix result is 727 passing tests and the six boundaries below.
+In each case, the environment prevents a normal BNU command from beginning; it
+is not a skipped command behavior.
-## Accepted boundaries
+## Boundaries
-| Upstream test | Boundary before BNU can execute |
+| Upstream test | Reason BNU cannot execute |
|---|---|
-| `tests/df/no-mtab-status-masked-proc.sh` | The test replaces or masks `/proc`. Bun aborts during runtime startup because required procfs facilities are unavailable. |
-| `tests/id/gnu-zero-uids.sh` | The behavior is specific to GNU/Hurd. The official Gentoo amd64 Hurd guest supplies that ABI, but Bun is a Linux ELF executable requesting Linux's loader and glibc ABI, so it cannot start on Hurd. |
-| `tests/nproc/nproc-quota-systemd.sh` | The test starts the command in an exact `SCHED_DEADLINE` systemd scope. Bun traps during startup in that scope; ordinary and other scheduler/cgroup quota cases run. |
-| `tests/sort/sort-continue.sh` | The test closes file descriptors below Bun's runtime startup requirement. Bun aborts while initializing its event loop, before the `sort` JavaScript runs. |
-| `tests/sort/sort-merge-fdlimit.sh` | The imposed descriptor limit is below Bun's runtime startup floor, so the runtime aborts before dispatching BNU. |
-| `tests/stat/stat-mount.sh` | The test hides or replaces `/proc`. Bun aborts during startup before the `stat` JavaScript can execute. |
+| `tests/df/no-mtab-status-masked-proc.sh` | The test masks or replaces `/proc`. Bun aborts during startup because required procfs facilities are unavailable. |
+| `tests/id/gnu-zero-uids.sh` | The behavior requires the GNU/Hurd ABI. Bun supplies no compatible GNU/Hurd runtime. |
+| `tests/nproc/nproc-quota-systemd.sh` | The test starts the command in an exact `SCHED_DEADLINE` systemd scope. Bun traps during startup in that scope. |
+| `tests/sort/sort-continue.sh` | The test closes file descriptors below Bun's runtime startup requirement. Bun aborts while initializing its event loop. |
+| `tests/sort/sort-merge-fdlimit.sh` | The imposed file-descriptor limit is below Bun's runtime startup floor. |
+| `tests/stat/stat-mount.sh` | The test masks or replaces `/proc`. Bun aborts before the `stat` implementation can run. |
-These entries are not allowances for arbitrary skips. The QEMU matrix embeds
-this exact sorted allowlist and fails when any other inventory member lacks a
-`PASS` record. If a future Bun version makes one of these tests pass, the
-matrix reports that the documented boundary is stale so it can be removed.
+## Enforcement
-## Resolved former skips
+The allowlist is exact. The matrix fails if any other inventory member lacks a
+passing record. It also reports an allowlisted test that unexpectedly passes so
+the stale boundary can be removed.
-The following cases formerly depended on unavailable host state or GNU C
-debugger hooks and now have executable coverage:
+The standard, SELinux MLS, and SMACK guests supply coverage for Linux
+platform-specific cases. The GNU/Hurd guest verifies the target environment for
+the ABI boundary; it cannot run BNU because the available Bun executable targets
+Linux.
-| Tests | Resolution |
-|---|---|
-| `tests/df/no-mtab-status.sh`, `tests/df/skip-duplicates.sh`, `tests/df/skip-rootfs.sh` | Deterministic JavaScript mount-table fixtures exercise the same parser cases without obsolete or synthetic host mounts. |
-| `tests/du/bigtime.sh` | A JavaScript timestamp fixture supplies the upstream out-of-range timestamp independently of the host filesystem range. |
-| `tests/rm/r-root.sh` | The adapted fixture intercepts the same removal boundary and validates abbreviated `--no-preserve-root` rejection without GNU `remove.c` symbols. |
-| `tests/tail/inotify-race.sh`, `tests/tail/inotify-race2.sh` | A synchronous JavaScript hook replaces the GNU `tail.c` debugger breakpoint and deterministically reproduces both races. |
-| `tests/chcon/chcon.sh`, `tests/ls/selinux.sh`, `tests/id/setgid.sh` | These run in the Gentoo amd64 SELinux guest with the MLS policy active. |
-| `tests/id/smack.sh`, `tests/mkdir/smack-root.sh`, `tests/mkdir/smack-no-root.sh` | These run in the Gentoo amd64 SMACK guest; the no-root case runs as `nobody` on the XFS guest root. BNU selects the active LSM and reads or writes `security.SMACK64`. |
-
-Terminal-dependent tests receive a controlling pseudo-terminal. Expensive and
-very-expensive paths, including generated factor tests `t00` through `t40`, are
-enabled by the matrix. Root, SELinux, SMACK, and Hurd cases are routed to their
-dedicated Gentoo guests.
+## Reproducing the result
-## Reproduction and evidence
-
-Run the bounded unit suite and complete platform matrix with:
+Run the local suite and provision or reuse the complete matrix:
```sh
scripts/run-all-tests.sh --setup
```
-The process is sequential and KVM-only. It defaults to 1536 MiB per guest,
-uses one compiler job for guest kernel/package builds, and applies an aggregate
-RSS ceiling to every GNU test process tree. See the QEMU section in the main
-[README](../README.md#portable-qemu-test-matrix) for host prerequisites and
-image setup.
-
-Each run writes its inventory, per-platform logs, allowed boundaries, passed
-set, unresolved set, and unexpected-unresolved set beneath:
-
-```text
-/var/tmp/bnu-qemu/results/TIMESTAMP/
-```
-
-That directory is generated evidence and is intentionally not stored in Git.
-The durable source-controlled contract is this document plus the executable
-reconciliation in `scripts/run-gnu-qemu-matrix.sh`.
+See [Testing](testing.md) for the host-side harness and
+[QEMU test matrix](qemu-test-matrix.md) for guest setup and result files.
diff --git a/docs/qemu-test-matrix.md b/docs/qemu-test-matrix.md
new file mode 100644
index 0000000..0632ba0
--- /dev/null
+++ b/docs/qemu-test-matrix.md
@@ -0,0 +1,155 @@
+# QEMU test matrix
+
+The QEMU matrix runs the GNU coreutils 9.11 command-test inventory in isolated
+Gentoo environments. It exists for cases that cannot be validated reliably on
+an ordinary development host, including root-only, SELinux, and SMACK tests.
+
+This setup is intended for maintainers. It downloads and provisions several
+guest images and requires KVM.
+
+## Environments
+
+| Guest | Purpose |
+|---|---|
+| Standard Gentoo Linux | Non-root command tests |
+| Gentoo Linux with SELinux MLS | Root-only and SELinux tests |
+| Gentoo Linux with SMACK | SMACK-specific tests |
+| Gentoo GNU/Hurd preview | Verification of the unsupported target-ABI boundary |
+
+The runner starts one VM at a time. It defaults to 1536 MiB RAM and two vCPUs
+per guest.
+
+## Host requirements
+
+The host must provide:
+
+- Bash and Bun;
+- readable and writable `/dev/kvm`;
+- `qemu-img` and `qemu-system-x86_64`;
+- OVMF firmware;
+- OpenSSH client tools;
+- `curl` and GnuPG;
+- `mkfs.ext4`;
+- either `xorriso` or `genisoimage`;
+- `awk`, `sort`, `comm`, `sha512sum`, `tar`, `xz`, and `timeout`.
+
+Package names vary by distribution. Check the actual executables and firmware
+paths without downloading or booting anything:
+
+```sh
+scripts/qemu/check-host.sh
+scripts/setup-qemu-environments.sh --dry-run
+scripts/run-gnu-qemu-matrix.sh --dry-run
+```
+
+There is no software-emulation fallback. The scripts require KVM and use
+`-cpu host`.
+
+## Storage and downloads
+
+State is stored under `/var/tmp/bnu-qemu` by default. Set `BNU_QEMU_DIR` to
+use another writable location:
+
+```sh
+BNU_QEMU_DIR=/path/to/bnu-qemu scripts/setup-qemu-environments.sh --dry-run
+```
+
+Allow about 30 GiB of free space for a complete first setup. Images are sparse,
+but downloads, package data, kernel sources, modules, and guest overlays consume
+real space. The setup checks available space before it starts.
+
+Downloaded Gentoo artifacts are authenticated before use. Automated release
+artifacts are checked with Gentoo's service keys. The Hurd preview is checked
+against its pinned signing-key fingerprint and published SHA-512 digest.
+
+Temporary extraction and seed-building data use `${TMPDIR:-/tmp}`.
+
+## Provisioning
+
+Create or resume every environment:
+
+```sh
+scripts/setup-qemu-environments.sh
+```
+
+Provision a single environment with `--only`:
+
+```sh
+scripts/setup-qemu-environments.sh --only standard
+scripts/setup-qemu-environments.sh --only selinux
+scripts/setup-qemu-environments.sh --only smack
+scripts/setup-qemu-environments.sh --only hurd
+```
+
+Provisioning is resumable. Ready markers prevent completed guests from being
+rebuilt on later runs.
+
+The SELinux guest uses the MLS policy in enforcing mode. Bun requires executable
+anonymous memory for its FFI compiler, so provisioning enables the policy's
+standard `allow_execmem` boolean. The SMACK guest uses a separately built
+kernel and a distinct overlay.
+
+## Running the matrix
+
+Run the local suite followed by the complete matrix:
+
+```sh
+scripts/run-all-tests.sh --setup
+```
+
+Run only the GNU matrix:
+
+```sh
+scripts/run-gnu-qemu-matrix.sh --setup
+```
+
+Omit `--setup` after the guests have been provisioned. Each run rebuilds the
+payload from the current checkout and attaches it read-only to the guests.
+
+The matrix reconciles all 733 test names. It fails if any test lacks a passing
+record unless that test is one of the six documented Bun runtime or ABI
+boundaries. It also reports a boundary that unexpectedly starts passing so the
+allowlist can be updated.
+
+Results are written to:
+
+```text
+$BNU_QEMU_DIR/results/TIMESTAMP/
+```
+
+The directory contains the inventory, per-guest logs, passing and unresolved
+sets, the accepted boundary list, and any unexpected unresolved tests.
+
+## Configuration
+
+The main environment variables are:
+
+| Variable | Default | Purpose |
+|---|---:|---|
+| `BNU_QEMU_DIR` | `/var/tmp/bnu-qemu` | Persistent state and results |
+| `BNU_QEMU_MEMORY` | `1536M` | Memory assigned to each VM |
+| `BNU_QEMU_CPUS` | `2` | vCPUs assigned to each VM |
+| `BNU_QEMU_SSH_PORT` | `22222` | Host port used for guest SSH |
+| `BNU_QEMU_TIMEOUT` | `900` | Guest SSH readiness timeout in seconds |
+| `BNU_QEMU_MIN_FREE_GIB` | phase-dependent | Override the free-space guard |
+| `BNU_OVMF_CODE` | auto-detected | Explicit OVMF code image |
+| `BNU_OVMF_VARS` | auto-detected | Explicit OVMF variable template |
+
+Heavy guest builds are configured to use one compiler job. Change VM sizing
+only when the host has enough resources for the test limits and guest overhead.
+
+## Smaller root smoke test
+
+A separate initramfs runner checks privileged ownership changes, special-file
+creation, and `cp -p` metadata preservation without provisioning the full
+matrix:
+
+```sh
+BNU_QEMU_KERNEL=/boot/vmlinuz-... \
+BNU_QEMU_INITRAMFS=/boot/initramfs-....img \
+ scripts/run-qemu-root-smoke.sh
+```
+
+It requires a matching host kernel and initramfs, KVM,
+`qemu-system-x86_64`, `zstd`, and `cpio`. It uses `lsinitrd` when
+available and otherwise attempts direct decompression.
diff --git a/docs/testing.md b/docs/testing.md
new file mode 100644
index 0000000..0f6b89e
--- /dev/null
+++ b/docs/testing.md
@@ -0,0 +1,114 @@
+# Testing BNU
+
+BNU has three test levels:
+
+1. the local Bun suite;
+2. selected or complete GNU command tests on the current host;
+3. the complete cross-environment GNU test matrix in KVM guests.
+
+Most changes need the local suite and relevant GNU tests. The VM matrix is for
+release-level or platform-specific validation.
+
+## Local suite
+
+```sh
+bun run test
+```
+
+Each test runs in a fresh process. The runner defaults to a 1 GiB aggregate
+process-tree RSS limit, a 3 GiB per-process address-space limit, and one
+`factor` worker.
+
+The defaults can be changed for a controlled test run:
+
+```sh
+BNU_TEST_RSS_LIMIT=2GiB \
+BNU_TEST_MEMORY_LIMIT=4GiB \
+BNU_TEST_TIMEOUT=600000 \
+BNU_FACTOR_WORKERS=2 \
+ bun run test
+```
+
+## GNU command tests
+
+The repository includes `coreutils-9.11.tar.xz` and a harness that extracts
+and adapts its command tests to invoke BNU.
+
+With no paths, the harness runs a maintained default selection:
+
+```sh
+bun run test:gnu
+```
+
+To select tests:
+
+```sh
+bun run test:gnu -- --list
+bun run test:gnu -- tests/misc/echo.sh
+bun run test:gnu -- tests/misc/echo.sh tests/misc/printenv.sh
+bun run test:gnu -- --all --list
+bun run test:gnu -- --all
+```
+
+By default, failed GNU tests are reported but do not make the harness return a
+failure status. Use `--strict` in CI or whenever the exit status matters:
+
+```sh
+bun run test:gnu -- --strict tests/misc/echo.sh
+```
+
+### Harness options
+
+| Option | Effect |
+|---|---|
+| `--all` | Select all 733 tests in GNU's standard command-test inventory. |
+| `--list` | Print the selected test names without running them. |
+| `--strict` | Return a non-zero status if any selected test fails. |
+| `--root-tests` | Restrict the selection to tests GNU marks as root-only. |
+| `--nonroot-tests` | Exclude tests GNU marks as root-only. |
+| `--expensive` | Set `RUN_EXPENSIVE_TESTS=yes`. |
+| `--very-expensive` | Enable both expensive and very-expensive GNU cases. |
+| `--tty` | Give every selected test a controlling pseudo-terminal. |
+| `--keep` | Preserve the extracted source and inner temporary directories. |
+| `--timeout VALUE` | Set the per-test timeout; the default is 300 seconds. |
+| `--max-output SIZE` | Set the retained output limit; the default is 1 MiB. |
+| `--rss-limit SIZE` | Set the process-tree RSS limit; the default is 1536 MiB. |
+| `--memory-limit SIZE` | Set the per-process address-space limit; the default is 3 GiB. |
+| `--tarball PATH` | Use a different GNU coreutils source tarball. |
+
+Known terminal-dependent tests receive a pseudo-terminal automatically.
+Root-only tests still require a suitable privileged environment. The complete
+matrix routes them to a dedicated guest.
+
+Examples:
+
+```sh
+bun run test:gnu -- --strict --timeout 10s tests/misc/printenv.sh
+bun run test:gnu -- --expensive tests/tail/big-4gb.sh
+bun run test:gnu -- --strict --very-expensive tests/factor/t26.sh
+```
+
+GNU's `gnulib-tests` exercise internal C portability code and are not part of
+this harness.
+
+## Complete KVM matrix
+
+The complete entry point runs the local suite, then the GNU inventory in its
+required guest environments:
+
+```sh
+scripts/run-all-tests.sh --setup
+```
+
+Later runs can reuse the provisioned guests:
+
+```sh
+scripts/run-all-tests.sh
+```
+
+Use `scripts/run-gnu-qemu-matrix.sh` when the local suite is not needed. The
+matrix accepts only the boundaries listed in
+[GNU test boundaries](gnu-test-boundaries.md).
+
+Provisioning, host requirements, storage, and result locations are documented
+in [QEMU test matrix](qemu-test-matrix.md).