diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/gnu-test-boundaries.md | 89 | ||||
| -rw-r--r-- | docs/qemu-test-matrix.md | 155 | ||||
| -rw-r--r-- | docs/testing.md | 114 |
3 files changed, 299 insertions, 59 deletions
diff --git a/docs/gnu-test-boundaries.md b/docs/gnu-test-boundaries.md index ebac2ef..29ef1ca 100644 --- a/docs/gnu-test-boundaries.md +++ b/docs/gnu-test-boundaries.md @@ -1,77 +1,48 @@ -# GNU coreutils 9.11 test boundaries +# GNU test boundaries ## Scope -BNU tracks the 733 `.sh` and `.pl` command-test scripts in GNU coreutils -9.11's standard `TESTS = $(all_tests) $(factor_tests)` inventory. Individual -scripts can contain multiple assertions. GNU's C-internal `gnulib-tests` suite -is outside this JavaScript command-compatibility scope. +BNU tracks the 733 `.sh` and `.pl` scripts in GNU coreutils 9.11's standard +command-test inventory. A script can contain multiple assertions. GNU's +C-internal `gnulib-tests` are outside this scope. -The upstream `all_root_tests` variable contains 41 names that are already part -of the 733; they are routed to a root guest rather than counted twice. The -runner derives both sets from the extracted upstream `Makefile.in`, currently -yielding 692 non-root tests and 41 root tests. +The inventory contains 692 non-root tests and 41 tests GNU marks as root-only. +The QEMU matrix routes those groups, plus security-module cases, to suitable +guests. -The accepted result is 727 tests with a `PASS` record on at least one suitable -platform and the following six Bun runtime or target-ABI boundaries. A normal -BNU JavaScript command cannot begin in any of these six environments. +The accepted matrix result is 727 passing tests and the six boundaries below. +In each case, the environment prevents a normal BNU command from beginning; it +is not a skipped command behavior. -## Accepted boundaries +## Boundaries -| Upstream test | Boundary before BNU can execute | +| Upstream test | Reason BNU cannot execute | |---|---| -| `tests/df/no-mtab-status-masked-proc.sh` | The test replaces or masks `/proc`. Bun aborts during runtime startup because required procfs facilities are unavailable. | -| `tests/id/gnu-zero-uids.sh` | The behavior is specific to GNU/Hurd. The official Gentoo amd64 Hurd guest supplies that ABI, but Bun is a Linux ELF executable requesting Linux's loader and glibc ABI, so it cannot start on Hurd. | -| `tests/nproc/nproc-quota-systemd.sh` | The test starts the command in an exact `SCHED_DEADLINE` systemd scope. Bun traps during startup in that scope; ordinary and other scheduler/cgroup quota cases run. | -| `tests/sort/sort-continue.sh` | The test closes file descriptors below Bun's runtime startup requirement. Bun aborts while initializing its event loop, before the `sort` JavaScript runs. | -| `tests/sort/sort-merge-fdlimit.sh` | The imposed descriptor limit is below Bun's runtime startup floor, so the runtime aborts before dispatching BNU. | -| `tests/stat/stat-mount.sh` | The test hides or replaces `/proc`. Bun aborts during startup before the `stat` JavaScript can execute. | +| `tests/df/no-mtab-status-masked-proc.sh` | The test masks or replaces `/proc`. Bun aborts during startup because required procfs facilities are unavailable. | +| `tests/id/gnu-zero-uids.sh` | The behavior requires the GNU/Hurd ABI. Bun supplies no compatible GNU/Hurd runtime. | +| `tests/nproc/nproc-quota-systemd.sh` | The test starts the command in an exact `SCHED_DEADLINE` systemd scope. Bun traps during startup in that scope. | +| `tests/sort/sort-continue.sh` | The test closes file descriptors below Bun's runtime startup requirement. Bun aborts while initializing its event loop. | +| `tests/sort/sort-merge-fdlimit.sh` | The imposed file-descriptor limit is below Bun's runtime startup floor. | +| `tests/stat/stat-mount.sh` | The test masks or replaces `/proc`. Bun aborts before the `stat` implementation can run. | -These entries are not allowances for arbitrary skips. The QEMU matrix embeds -this exact sorted allowlist and fails when any other inventory member lacks a -`PASS` record. If a future Bun version makes one of these tests pass, the -matrix reports that the documented boundary is stale so it can be removed. +## Enforcement -## Resolved former skips +The allowlist is exact. The matrix fails if any other inventory member lacks a +passing record. It also reports an allowlisted test that unexpectedly passes so +the stale boundary can be removed. -The following cases formerly depended on unavailable host state or GNU C -debugger hooks and now have executable coverage: +The standard, SELinux MLS, and SMACK guests supply coverage for Linux +platform-specific cases. The GNU/Hurd guest verifies the target environment for +the ABI boundary; it cannot run BNU because the available Bun executable targets +Linux. -| Tests | Resolution | -|---|---| -| `tests/df/no-mtab-status.sh`, `tests/df/skip-duplicates.sh`, `tests/df/skip-rootfs.sh` | Deterministic JavaScript mount-table fixtures exercise the same parser cases without obsolete or synthetic host mounts. | -| `tests/du/bigtime.sh` | A JavaScript timestamp fixture supplies the upstream out-of-range timestamp independently of the host filesystem range. | -| `tests/rm/r-root.sh` | The adapted fixture intercepts the same removal boundary and validates abbreviated `--no-preserve-root` rejection without GNU `remove.c` symbols. | -| `tests/tail/inotify-race.sh`, `tests/tail/inotify-race2.sh` | A synchronous JavaScript hook replaces the GNU `tail.c` debugger breakpoint and deterministically reproduces both races. | -| `tests/chcon/chcon.sh`, `tests/ls/selinux.sh`, `tests/id/setgid.sh` | These run in the Gentoo amd64 SELinux guest with the MLS policy active. | -| `tests/id/smack.sh`, `tests/mkdir/smack-root.sh`, `tests/mkdir/smack-no-root.sh` | These run in the Gentoo amd64 SMACK guest; the no-root case runs as `nobody` on the XFS guest root. BNU selects the active LSM and reads or writes `security.SMACK64`. | - -Terminal-dependent tests receive a controlling pseudo-terminal. Expensive and -very-expensive paths, including generated factor tests `t00` through `t40`, are -enabled by the matrix. Root, SELinux, SMACK, and Hurd cases are routed to their -dedicated Gentoo guests. +## Reproducing the result -## Reproduction and evidence - -Run the bounded unit suite and complete platform matrix with: +Run the local suite and provision or reuse the complete matrix: ```sh scripts/run-all-tests.sh --setup ``` -The process is sequential and KVM-only. It defaults to 1536 MiB per guest, -uses one compiler job for guest kernel/package builds, and applies an aggregate -RSS ceiling to every GNU test process tree. See the QEMU section in the main -[README](../README.md#portable-qemu-test-matrix) for host prerequisites and -image setup. - -Each run writes its inventory, per-platform logs, allowed boundaries, passed -set, unresolved set, and unexpected-unresolved set beneath: - -```text -/var/tmp/bnu-qemu/results/TIMESTAMP/ -``` - -That directory is generated evidence and is intentionally not stored in Git. -The durable source-controlled contract is this document plus the executable -reconciliation in `scripts/run-gnu-qemu-matrix.sh`. +See [Testing](testing.md) for the host-side harness and +[QEMU test matrix](qemu-test-matrix.md) for guest setup and result files. diff --git a/docs/qemu-test-matrix.md b/docs/qemu-test-matrix.md new file mode 100644 index 0000000..0632ba0 --- /dev/null +++ b/docs/qemu-test-matrix.md @@ -0,0 +1,155 @@ +# QEMU test matrix + +The QEMU matrix runs the GNU coreutils 9.11 command-test inventory in isolated +Gentoo environments. It exists for cases that cannot be validated reliably on +an ordinary development host, including root-only, SELinux, and SMACK tests. + +This setup is intended for maintainers. It downloads and provisions several +guest images and requires KVM. + +## Environments + +| Guest | Purpose | +|---|---| +| Standard Gentoo Linux | Non-root command tests | +| Gentoo Linux with SELinux MLS | Root-only and SELinux tests | +| Gentoo Linux with SMACK | SMACK-specific tests | +| Gentoo GNU/Hurd preview | Verification of the unsupported target-ABI boundary | + +The runner starts one VM at a time. It defaults to 1536 MiB RAM and two vCPUs +per guest. + +## Host requirements + +The host must provide: + +- Bash and Bun; +- readable and writable `/dev/kvm`; +- `qemu-img` and `qemu-system-x86_64`; +- OVMF firmware; +- OpenSSH client tools; +- `curl` and GnuPG; +- `mkfs.ext4`; +- either `xorriso` or `genisoimage`; +- `awk`, `sort`, `comm`, `sha512sum`, `tar`, `xz`, and `timeout`. + +Package names vary by distribution. Check the actual executables and firmware +paths without downloading or booting anything: + +```sh +scripts/qemu/check-host.sh +scripts/setup-qemu-environments.sh --dry-run +scripts/run-gnu-qemu-matrix.sh --dry-run +``` + +There is no software-emulation fallback. The scripts require KVM and use +`-cpu host`. + +## Storage and downloads + +State is stored under `/var/tmp/bnu-qemu` by default. Set `BNU_QEMU_DIR` to +use another writable location: + +```sh +BNU_QEMU_DIR=/path/to/bnu-qemu scripts/setup-qemu-environments.sh --dry-run +``` + +Allow about 30 GiB of free space for a complete first setup. Images are sparse, +but downloads, package data, kernel sources, modules, and guest overlays consume +real space. The setup checks available space before it starts. + +Downloaded Gentoo artifacts are authenticated before use. Automated release +artifacts are checked with Gentoo's service keys. The Hurd preview is checked +against its pinned signing-key fingerprint and published SHA-512 digest. + +Temporary extraction and seed-building data use `${TMPDIR:-/tmp}`. + +## Provisioning + +Create or resume every environment: + +```sh +scripts/setup-qemu-environments.sh +``` + +Provision a single environment with `--only`: + +```sh +scripts/setup-qemu-environments.sh --only standard +scripts/setup-qemu-environments.sh --only selinux +scripts/setup-qemu-environments.sh --only smack +scripts/setup-qemu-environments.sh --only hurd +``` + +Provisioning is resumable. Ready markers prevent completed guests from being +rebuilt on later runs. + +The SELinux guest uses the MLS policy in enforcing mode. Bun requires executable +anonymous memory for its FFI compiler, so provisioning enables the policy's +standard `allow_execmem` boolean. The SMACK guest uses a separately built +kernel and a distinct overlay. + +## Running the matrix + +Run the local suite followed by the complete matrix: + +```sh +scripts/run-all-tests.sh --setup +``` + +Run only the GNU matrix: + +```sh +scripts/run-gnu-qemu-matrix.sh --setup +``` + +Omit `--setup` after the guests have been provisioned. Each run rebuilds the +payload from the current checkout and attaches it read-only to the guests. + +The matrix reconciles all 733 test names. It fails if any test lacks a passing +record unless that test is one of the six documented Bun runtime or ABI +boundaries. It also reports a boundary that unexpectedly starts passing so the +allowlist can be updated. + +Results are written to: + +```text +$BNU_QEMU_DIR/results/TIMESTAMP/ +``` + +The directory contains the inventory, per-guest logs, passing and unresolved +sets, the accepted boundary list, and any unexpected unresolved tests. + +## Configuration + +The main environment variables are: + +| Variable | Default | Purpose | +|---|---:|---| +| `BNU_QEMU_DIR` | `/var/tmp/bnu-qemu` | Persistent state and results | +| `BNU_QEMU_MEMORY` | `1536M` | Memory assigned to each VM | +| `BNU_QEMU_CPUS` | `2` | vCPUs assigned to each VM | +| `BNU_QEMU_SSH_PORT` | `22222` | Host port used for guest SSH | +| `BNU_QEMU_TIMEOUT` | `900` | Guest SSH readiness timeout in seconds | +| `BNU_QEMU_MIN_FREE_GIB` | phase-dependent | Override the free-space guard | +| `BNU_OVMF_CODE` | auto-detected | Explicit OVMF code image | +| `BNU_OVMF_VARS` | auto-detected | Explicit OVMF variable template | + +Heavy guest builds are configured to use one compiler job. Change VM sizing +only when the host has enough resources for the test limits and guest overhead. + +## Smaller root smoke test + +A separate initramfs runner checks privileged ownership changes, special-file +creation, and `cp -p` metadata preservation without provisioning the full +matrix: + +```sh +BNU_QEMU_KERNEL=/boot/vmlinuz-... \ +BNU_QEMU_INITRAMFS=/boot/initramfs-....img \ + scripts/run-qemu-root-smoke.sh +``` + +It requires a matching host kernel and initramfs, KVM, +`qemu-system-x86_64`, `zstd`, and `cpio`. It uses `lsinitrd` when +available and otherwise attempts direct decompression. diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..0f6b89e --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,114 @@ +# Testing BNU + +BNU has three test levels: + +1. the local Bun suite; +2. selected or complete GNU command tests on the current host; +3. the complete cross-environment GNU test matrix in KVM guests. + +Most changes need the local suite and relevant GNU tests. The VM matrix is for +release-level or platform-specific validation. + +## Local suite + +```sh +bun run test +``` + +Each test runs in a fresh process. The runner defaults to a 1 GiB aggregate +process-tree RSS limit, a 3 GiB per-process address-space limit, and one +`factor` worker. + +The defaults can be changed for a controlled test run: + +```sh +BNU_TEST_RSS_LIMIT=2GiB \ +BNU_TEST_MEMORY_LIMIT=4GiB \ +BNU_TEST_TIMEOUT=600000 \ +BNU_FACTOR_WORKERS=2 \ + bun run test +``` + +## GNU command tests + +The repository includes `coreutils-9.11.tar.xz` and a harness that extracts +and adapts its command tests to invoke BNU. + +With no paths, the harness runs a maintained default selection: + +```sh +bun run test:gnu +``` + +To select tests: + +```sh +bun run test:gnu -- --list +bun run test:gnu -- tests/misc/echo.sh +bun run test:gnu -- tests/misc/echo.sh tests/misc/printenv.sh +bun run test:gnu -- --all --list +bun run test:gnu -- --all +``` + +By default, failed GNU tests are reported but do not make the harness return a +failure status. Use `--strict` in CI or whenever the exit status matters: + +```sh +bun run test:gnu -- --strict tests/misc/echo.sh +``` + +### Harness options + +| Option | Effect | +|---|---| +| `--all` | Select all 733 tests in GNU's standard command-test inventory. | +| `--list` | Print the selected test names without running them. | +| `--strict` | Return a non-zero status if any selected test fails. | +| `--root-tests` | Restrict the selection to tests GNU marks as root-only. | +| `--nonroot-tests` | Exclude tests GNU marks as root-only. | +| `--expensive` | Set `RUN_EXPENSIVE_TESTS=yes`. | +| `--very-expensive` | Enable both expensive and very-expensive GNU cases. | +| `--tty` | Give every selected test a controlling pseudo-terminal. | +| `--keep` | Preserve the extracted source and inner temporary directories. | +| `--timeout VALUE` | Set the per-test timeout; the default is 300 seconds. | +| `--max-output SIZE` | Set the retained output limit; the default is 1 MiB. | +| `--rss-limit SIZE` | Set the process-tree RSS limit; the default is 1536 MiB. | +| `--memory-limit SIZE` | Set the per-process address-space limit; the default is 3 GiB. | +| `--tarball PATH` | Use a different GNU coreutils source tarball. | + +Known terminal-dependent tests receive a pseudo-terminal automatically. +Root-only tests still require a suitable privileged environment. The complete +matrix routes them to a dedicated guest. + +Examples: + +```sh +bun run test:gnu -- --strict --timeout 10s tests/misc/printenv.sh +bun run test:gnu -- --expensive tests/tail/big-4gb.sh +bun run test:gnu -- --strict --very-expensive tests/factor/t26.sh +``` + +GNU's `gnulib-tests` exercise internal C portability code and are not part of +this harness. + +## Complete KVM matrix + +The complete entry point runs the local suite, then the GNU inventory in its +required guest environments: + +```sh +scripts/run-all-tests.sh --setup +``` + +Later runs can reuse the provisioned guests: + +```sh +scripts/run-all-tests.sh +``` + +Use `scripts/run-gnu-qemu-matrix.sh` when the local suite is not needed. The +matrix accepts only the boundaries listed in +[GNU test boundaries](gnu-test-boundaries.md). + +Provisioning, host requirements, storage, and result locations are documented +in [QEMU test matrix](qemu-test-matrix.md). |
