diff options
| author | Soccera1 <[email protected]> | 2026-07-13 18:33:53 +1000 |
|---|---|---|
| committer | Soccera1 <[email protected]> | 2026-07-13 18:33:53 +1000 |
| commit | 876afc0c11d00bdcb50343ccd2e408810e6c9dff (patch) | |
| tree | 297f7ec6832485c46626a40baca91b0e9088b9d1 | |
| parent | 6e2513d28da68283e795e2a04a7d68cb3af1cb7a (diff) | |
| download | bnu-876afc0c11d00bdcb50343ccd2e408810e6c9dff.tar.gz bnu-876afc0c11d00bdcb50343ccd2e408810e6c9dff.zip | |
Re-write docs
| -rw-r--r-- | README.md | 241 | ||||
| -rw-r--r-- | docs/gnu-test-boundaries.md | 89 | ||||
| -rw-r--r-- | docs/qemu-test-matrix.md | 155 | ||||
| -rw-r--r-- | docs/testing.md | 114 |
4 files changed, 359 insertions, 240 deletions
@@ -1,218 +1,97 @@ -# bnu +# BNU -A Bun JavaScript rewrite of GNU coreutils, currently implemented as a multi-call CLI. +BNU is an independent implementation of GNU coreutils for +[Bun](https://bun.sh/). It runs as a multi-call command: the first argument +selects the utility. ```sh bun ./bin/bnu.js echo hello bun ./bin/bnu.js sort file.txt +bun ./bin/bnu.js cp source destination ``` -The normal local test suite runs each test in a fresh process with a 1 GiB -aggregate process-tree RSS ceiling and a 3 GiB per-process address-space -ceiling. `factor` also defaults to one worker; controlled audits can opt in to -bounded parallelism with `BNU_FACTOR_WORKERS`. +Linux is the primary tested platform. BNU is not built from GNU coreutils +source and does not provide its internal C interfaces. -```sh -bun run test -``` - -There is also an optional harness for running GNU coreutils' command-level -upstream tests from the bundled tarball against the Bun wrappers. With no test -arguments it runs a small curated subset; pass upstream test paths for targeted -runs, or use `--all` to -select the 733 `.sh`/`.pl` entries in GNU's standard `TESTS` list. The 41 -root-test names are members of that same list and are rerun with root privileges -rather than counted as additional tests. GNU's separate `gnulib-tests` validate -its internal C portability library and are outside this JavaScript command -compatibility scope. By default compatibility failures are -reported without making the command fail; add `--strict` to return a non-zero -status when any GNU test fails. Each GNU test has a default 300 second timeout -to account for the Bun-backed wrappers' startup cost in broad all-command -upstream sweeps; override it with -`--timeout`. Pass `--expensive` to set `RUN_EXPENSIVE_TESTS=yes`, or -`--very-expensive` to set both `RUN_EXPENSIVE_TESTS=yes` and -`RUN_VERY_EXPENSIVE_TESTS=yes` for upstream tests that are disabled by default. -Known terminal-dependent upstream tests are automatically run in their own -controlling pseudo-terminal. Pass `--tty` to force that mode for every selected -test. Pass `--keep` to preserve both the extracted source tree and GNU tests' -inner temporary directories for syscall-log and failure-artifact inspection. -The harness enforces a 1536 MiB aggregate RSS ceiling for each test process -tree and a 3 GiB per-process address-space ceiling by default; override these -with `--rss-limit` and `--memory-limit` when a controlled stress audit requires -different bounds. - -```sh -bun run test:gnu --list -bun run test:gnu -bun run test:gnu --all --list -bun run test:gnu tests/misc/echo.sh -bun run test:gnu --strict tests/misc/echo.sh -bun run test:gnu --timeout 10s tests/misc/printenv.sh -bun run test:gnu --rss-limit 1GiB --memory-limit 3GiB tests/misc/echo.sh -bun run test:gnu tests/stty/stty-invalid.sh -bun run test:gnu --expensive tests/tail/big-4gb.sh -bun run test:gnu --very-expensive tests/misc/sort.sh -# A representative 96-bit contiguous-range factor stress case. -bun run test:gnu --strict --very-expensive tests/factor/t26.sh -``` - -## Portable QEMU test matrix +## Requirements -The complete upstream command-test inventory can run without depending on the -host distribution or host root access. The matrix uses official, authenticated -Gentoo amd64 artifacts to create four separate environments beneath -`/var/tmp/bnu-qemu`: +Running BNU from this repository requires Bun. Some commands also rely on +operating-system facilities such as Linux extended attributes, ACLs, inotify, +or security modules. -- a standard Gentoo guest for the 692 non-root inventory members; -- a stage4 built from Gentoo's hardened SELinux systemd stage3, with the MLS - policy active in enforcing mode, for the 41 root tests and SELinux cases; -- a distinct qcow2 SMACK overlay with a custom kernel built using one compiler - job and a 2 GiB guest swap file; -- the official Gentoo amd64 GNU/Hurd preview and a separate BNU matrix overlay. - -The all-tests entry point first runs the local bounded unit suite, then creates -or resumes the guests, builds an ext4 payload from the current checkout, runs -one VM at a time, and reconciles all 733 upstream test names: +No dependency installation is needed for the basic CLI: ```sh -scripts/run-all-tests.sh --setup -# Subsequent runs reuse the guests and refresh only the payload: -scripts/run-all-tests.sh +bun ./bin/bnu.js --help +bun ./bin/bnu.js COMMAND --help ``` -Use `scripts/run-gnu-qemu-matrix.sh` directly when only the upstream matrix is -wanted. +## Command wrappers -The host can be any Linux distribution. It needs Bash, Bun, KVM access, QEMU -for x86_64, OVMF, OpenSSH clients, curl, GnuPG, e2fsprogs (`mkfs.ext4`), xorriso -or genisoimage, and ordinary POSIX archive/checksum tools. Package names differ -between distributions, so the scripts check executable names rather than -assuming a package manager. `/dev/kvm` must be readable and writable by the -calling user; there is intentionally no TCG fallback. A quick dependency and -action check that downloads or boots nothing is available with: +The CLI can generate command-name wrappers for use in a separate directory: ```sh -scripts/qemu/check-host.sh -scripts/setup-qemu-environments.sh --dry-run -scripts/run-gnu-qemu-matrix.sh --dry-run +bun run link-commands -- ./dist/bin +./dist/bin/echo hello +./dist/bin/wc README.md ``` -Persistent downloads, base images, isolated overlays, readiness markers, and -timestamped results stay under `BNU_QEMU_DIR` (default `/var/tmp/bnu-qemu`). -Set that variable to another user-writable directory if necessary. Temporary -seed staging and GNU extraction trees use `${TMPDIR:-/tmp}` and may be wiped -between runs. The setup is phase-resumable; completed image markers prevent a -costly rebuild. Individual environments can be prepared with -`scripts/setup-qemu-environments.sh --only standard|selinux|smack|hurd`. +Add that directory to `PATH` only when you intentionally want BNU commands to +take precedence over the system coreutils. -Every VM is KVM-only, defaults to 1536 MiB RAM and two vCPUs, and heavy work is -strictly sequential. Guest package/kernel builds use `-j1`; each GNU test gets -a 1 GiB aggregate process-tree RSS ceiling and a 3 GiB per-process virtual -memory ceiling. Override VM sizing only when needed with `BNU_QEMU_MEMORY` and -`BNU_QEMU_CPUS`. Official artifacts are cached only after their Gentoo OpenPGP -signature is verified using Gentoo's published service-key bundle; the Hurd -image's developer signature is pinned to its full Gentoo signing-key -fingerprint and also gets its published SHA-512 check. +The top-level `--help` output lists the available commands. Use +`COMMAND --help` for a command's accepted options. -Allow roughly 30 GiB of free space for a first complete setup: qcow2 images are -sparse, but Portage downloads, the SELinux root, kernel sources, modules, and -the one-job SMACK build still consume real blocks. The setup checks free space -before downloading or booting. `BNU_QEMU_MIN_FREE_GIB` can override that guard -when the state directory is on a specially provisioned thin or compressed -filesystem. +## Compatibility -The final reconciliation fails if any inventory member lacks a PASS on its -appropriate platform, except for the six documented Bun startup/ABI boundaries -in [`docs/gnu-test-boundaries.md`](docs/gnu-test-boundaries.md). -Logs are written to `/var/tmp/bnu-qemu/results/TIMESTAMP` by default. +The compatibility target is the observable command-line behavior of GNU +coreutils 9.11. The upstream package contains 733 command-test scripts; BNU has +passing test results for 727 of them. The other six cannot start the Bun runtime +in the environment constructed by the test. -For a smaller Linux root-only smoke check, the repository also retains an -initramfs runner. It assembles the current Bun executable and source tree into -a matching zstd-compressed initramfs, then checks privileged ownership changes, -special-file creation, and `cp -p` metadata preservation as guest root: +That result is useful test coverage, not a claim of complete equivalence. +Individual scripts contain many cases, and the inventory does not cover every +option combination, locale, filesystem, kernel, or failure mode. GNU's internal +`gnulib-tests` are also outside BNU's scope. -```sh -BNU_QEMU_KERNEL=/boot/vmlinuz-... \ -BNU_QEMU_INITRAMFS=/boot/initramfs-....img \ - scripts/run-qemu-root-smoke.sh -``` +Host capabilities affect some results: + +- ACL, extended-attribute, security-context, reflink, sparse-file, device, and + notification behavior depends on kernel and filesystem support. +- Ownership changes, `chroot`, clock changes, device creation, and + security-label operations may require elevated privileges. +- Locale, account databases, terminal configuration, and mount state can + change output or diagnostics. -The smoke runner requires `qemu-system-x86_64`, `zstd`, `cpio`, KVM access, and -a kernel/initramfs pair that can boot on the host CPU. It uses `lsinitrd` when -available to unpack concatenated or compressed input images, with direct -decompressor fallbacks. Compression is single-threaded and it uses KVM with -`-cpu host`; `BNU_QEMU_MEMORY` overrides its default 1536 MiB guest allocation. +See [GNU test boundaries](docs/gnu-test-boundaries.md) for the six runtime/ABI +cases. -SELinux guests must select SELinux in the kernel's active LSM list, not merely -compile it in. On kernels whose `CONFIG_LSM` default omits SELinux, boot with -`lsm=selinux` (in addition to `security=selinux selinux=1`); otherwise -libselinux process attributes and thread-local creation contexts can fail with -`EINVAL` even though `sestatus` reports an enabled policy. Bun's FFI compiler -also requires executable anonymous memory in enforcing mode; the dedicated -test guest enables the policy's standard `allow_execmem` boolean rather than -carrying an ad-hoc local allow rule. +## Development -To create command-name wrappers such as `cat`, `wc`, and `sort`: +Run the local test suite with: ```sh -bun run link-commands -- ./dist/bin -./dist/bin/echo hello +bun run test ``` -## Implemented Commands +Run the default selection of upstream GNU tests with: -`[`, `arch`, `b2sum`, `base32`, `base64`, `basename`, `basenc`, `cat`, `chcon`, -`chgrp`, `chmod`, `chown`, `chroot`, `cksum`, `comm`, `coreutils`, `cp`, -`csplit`, `cut`, `date`, `dd`, `df`, `dir`, `dircolors`, `dirname`, `du`, -`echo`, `env`, `expand`, `expr`, `factor`, `false`, `fmt`, `fold`, `ginstall`, -`groups`, `head`, `hostid`, `hostname`, `id`, `install`, `join`, `kill`, `link`, -`ln`, `logname`, `ls`, `md5sum`, `mkdir`, `mkfifo`, `mknod`, `mktemp`, `mv`, -`nice`, `nl`, `nohup`, `nproc`, `numfmt`, `od`, `paste`, `pathchk`, `pinky`, -`pr`, `printenv`, `printf`, `ptx`, `pwd`, `readlink`, `realpath`, `rm`, -`rmdir`, `runcon`, `seq`, `sha1sum`, `sha224sum`, `sha256sum`, `sha384sum`, -`sha512sum`, `shred`, `shuf`, `sleep`, `sm3sum`, `sort`, `split`, `stat`, -`stdbuf`, `stty`, `sum`, `sync`, `tac`, `tail`, `tee`, `test`, `timeout`, -`touch`, `tr`, `true`, `truncate`, `tsort`, `tty`, `uname`, `unexpand`, `uniq`, -`unlink`, `uptime`, `users`, `vdir`, `wc`, `who`, `whoami`, `yes`. - -The GNU coreutils 9.11 command-test inventory is reconciled against the current -source: 727 of its 733 scripts have passing host or QEMU evidence. The remaining -six are confirmed Bun startup or target-ABI boundaries: two tests remove the -procfs facilities Bun needs to start, two start below Bun's required file- -descriptor floor, one traps during Bun startup in a `SCHED_DEADLINE` systemd -scope, and one requires a GNU/Hurd ABI for which no Bun runtime exists. This -includes root-only reruns and activated terminal, SELinux MLS, SMACK, expensive, -and very-expensive paths. The six test names, exact rationale, and evidence are -in [`docs/gnu-test-boundaries.md`](docs/gnu-test-boundaries.md). - -This 733-script scope is not every assertion in the package: individual scripts -contain multiple cases, and it excludes GNU's C-internal `gnulib-tests`. -Passing it does not imply that every GNU extension or platform combination is -implemented; BNU remains an independent JavaScript implementation rather than -GNU's native C code. +```sh +bun run test:gnu +``` -## Compatibility Notes +The upstream harness reports failures without failing the command unless +`--strict` is supplied. Full test options, resource limits, and the KVM test +matrix are documented in [Testing](docs/testing.md). -BNU targets command-line compatibility with GNU coreutils 9.11 for the -commands listed above. It is an independent JavaScript implementation, so -compatibility applies to observable command behavior rather than GNU internals -or binary interfaces. Passing the upstream command-test inventory does not -guarantee that every option combination, platform, locale, or filesystem -behaves identically. +## Repository layout -Some behavior depends on the host: +- `bin/bnu.js` — multi-call entry point +- `src/coreutils.js` — command implementations +- `tests/` — local Bun tests +- `scripts/` — test harnesses, wrapper generation, and VM tooling +- `docs/` — testing and compatibility documentation -- ACLs, extended attributes, security contexts, reflinks, sparse files, inotify, - device nodes, and similar features require corresponding operating-system and - filesystem support. -- Ownership changes, `chroot`, clock changes, device creation, and security-label - operations may require elevated privileges. -- Sorting, character handling, user and group lookup, terminal behavior, and - diagnostics can vary with the active locale and system configuration. -- Six upstream tests cannot start BNU because of Bun runtime or target-ABI - constraints. They are documented in - [`docs/gnu-test-boundaries.md`](docs/gnu-test-boundaries.md). +## License -Use `bnu COMMAND --help` for the options exposed by a command. The local and -upstream test suites provide the most precise record of currently verified -behavior. +See [LICENSE](LICENSE). diff --git a/docs/gnu-test-boundaries.md b/docs/gnu-test-boundaries.md index ebac2ef..29ef1ca 100644 --- a/docs/gnu-test-boundaries.md +++ b/docs/gnu-test-boundaries.md @@ -1,77 +1,48 @@ -# GNU coreutils 9.11 test boundaries +# GNU test boundaries ## Scope -BNU tracks the 733 `.sh` and `.pl` command-test scripts in GNU coreutils -9.11's standard `TESTS = $(all_tests) $(factor_tests)` inventory. Individual -scripts can contain multiple assertions. GNU's C-internal `gnulib-tests` suite -is outside this JavaScript command-compatibility scope. +BNU tracks the 733 `.sh` and `.pl` scripts in GNU coreutils 9.11's standard +command-test inventory. A script can contain multiple assertions. GNU's +C-internal `gnulib-tests` are outside this scope. -The upstream `all_root_tests` variable contains 41 names that are already part -of the 733; they are routed to a root guest rather than counted twice. The -runner derives both sets from the extracted upstream `Makefile.in`, currently -yielding 692 non-root tests and 41 root tests. +The inventory contains 692 non-root tests and 41 tests GNU marks as root-only. +The QEMU matrix routes those groups, plus security-module cases, to suitable +guests. -The accepted result is 727 tests with a `PASS` record on at least one suitable -platform and the following six Bun runtime or target-ABI boundaries. A normal -BNU JavaScript command cannot begin in any of these six environments. +The accepted matrix result is 727 passing tests and the six boundaries below. +In each case, the environment prevents a normal BNU command from beginning; it +is not a skipped command behavior. -## Accepted boundaries +## Boundaries -| Upstream test | Boundary before BNU can execute | +| Upstream test | Reason BNU cannot execute | |---|---| -| `tests/df/no-mtab-status-masked-proc.sh` | The test replaces or masks `/proc`. Bun aborts during runtime startup because required procfs facilities are unavailable. | -| `tests/id/gnu-zero-uids.sh` | The behavior is specific to GNU/Hurd. The official Gentoo amd64 Hurd guest supplies that ABI, but Bun is a Linux ELF executable requesting Linux's loader and glibc ABI, so it cannot start on Hurd. | -| `tests/nproc/nproc-quota-systemd.sh` | The test starts the command in an exact `SCHED_DEADLINE` systemd scope. Bun traps during startup in that scope; ordinary and other scheduler/cgroup quota cases run. | -| `tests/sort/sort-continue.sh` | The test closes file descriptors below Bun's runtime startup requirement. Bun aborts while initializing its event loop, before the `sort` JavaScript runs. | -| `tests/sort/sort-merge-fdlimit.sh` | The imposed descriptor limit is below Bun's runtime startup floor, so the runtime aborts before dispatching BNU. | -| `tests/stat/stat-mount.sh` | The test hides or replaces `/proc`. Bun aborts during startup before the `stat` JavaScript can execute. | +| `tests/df/no-mtab-status-masked-proc.sh` | The test masks or replaces `/proc`. Bun aborts during startup because required procfs facilities are unavailable. | +| `tests/id/gnu-zero-uids.sh` | The behavior requires the GNU/Hurd ABI. Bun supplies no compatible GNU/Hurd runtime. | +| `tests/nproc/nproc-quota-systemd.sh` | The test starts the command in an exact `SCHED_DEADLINE` systemd scope. Bun traps during startup in that scope. | +| `tests/sort/sort-continue.sh` | The test closes file descriptors below Bun's runtime startup requirement. Bun aborts while initializing its event loop. | +| `tests/sort/sort-merge-fdlimit.sh` | The imposed file-descriptor limit is below Bun's runtime startup floor. | +| `tests/stat/stat-mount.sh` | The test masks or replaces `/proc`. Bun aborts before the `stat` implementation can run. | -These entries are not allowances for arbitrary skips. The QEMU matrix embeds -this exact sorted allowlist and fails when any other inventory member lacks a -`PASS` record. If a future Bun version makes one of these tests pass, the -matrix reports that the documented boundary is stale so it can be removed. +## Enforcement -## Resolved former skips +The allowlist is exact. The matrix fails if any other inventory member lacks a +passing record. It also reports an allowlisted test that unexpectedly passes so +the stale boundary can be removed. -The following cases formerly depended on unavailable host state or GNU C -debugger hooks and now have executable coverage: +The standard, SELinux MLS, and SMACK guests supply coverage for Linux +platform-specific cases. The GNU/Hurd guest verifies the target environment for +the ABI boundary; it cannot run BNU because the available Bun executable targets +Linux. -| Tests | Resolution | -|---|---| -| `tests/df/no-mtab-status.sh`, `tests/df/skip-duplicates.sh`, `tests/df/skip-rootfs.sh` | Deterministic JavaScript mount-table fixtures exercise the same parser cases without obsolete or synthetic host mounts. | -| `tests/du/bigtime.sh` | A JavaScript timestamp fixture supplies the upstream out-of-range timestamp independently of the host filesystem range. | -| `tests/rm/r-root.sh` | The adapted fixture intercepts the same removal boundary and validates abbreviated `--no-preserve-root` rejection without GNU `remove.c` symbols. | -| `tests/tail/inotify-race.sh`, `tests/tail/inotify-race2.sh` | A synchronous JavaScript hook replaces the GNU `tail.c` debugger breakpoint and deterministically reproduces both races. | -| `tests/chcon/chcon.sh`, `tests/ls/selinux.sh`, `tests/id/setgid.sh` | These run in the Gentoo amd64 SELinux guest with the MLS policy active. | -| `tests/id/smack.sh`, `tests/mkdir/smack-root.sh`, `tests/mkdir/smack-no-root.sh` | These run in the Gentoo amd64 SMACK guest; the no-root case runs as `nobody` on the XFS guest root. BNU selects the active LSM and reads or writes `security.SMACK64`. | - -Terminal-dependent tests receive a controlling pseudo-terminal. Expensive and -very-expensive paths, including generated factor tests `t00` through `t40`, are -enabled by the matrix. Root, SELinux, SMACK, and Hurd cases are routed to their -dedicated Gentoo guests. +## Reproducing the result -## Reproduction and evidence - -Run the bounded unit suite and complete platform matrix with: +Run the local suite and provision or reuse the complete matrix: ```sh scripts/run-all-tests.sh --setup ``` -The process is sequential and KVM-only. It defaults to 1536 MiB per guest, -uses one compiler job for guest kernel/package builds, and applies an aggregate -RSS ceiling to every GNU test process tree. See the QEMU section in the main -[README](../README.md#portable-qemu-test-matrix) for host prerequisites and -image setup. - -Each run writes its inventory, per-platform logs, allowed boundaries, passed -set, unresolved set, and unexpected-unresolved set beneath: - -```text -/var/tmp/bnu-qemu/results/TIMESTAMP/ -``` - -That directory is generated evidence and is intentionally not stored in Git. -The durable source-controlled contract is this document plus the executable -reconciliation in `scripts/run-gnu-qemu-matrix.sh`. +See [Testing](testing.md) for the host-side harness and +[QEMU test matrix](qemu-test-matrix.md) for guest setup and result files. diff --git a/docs/qemu-test-matrix.md b/docs/qemu-test-matrix.md new file mode 100644 index 0000000..0632ba0 --- /dev/null +++ b/docs/qemu-test-matrix.md @@ -0,0 +1,155 @@ +# QEMU test matrix + +The QEMU matrix runs the GNU coreutils 9.11 command-test inventory in isolated +Gentoo environments. It exists for cases that cannot be validated reliably on +an ordinary development host, including root-only, SELinux, and SMACK tests. + +This setup is intended for maintainers. It downloads and provisions several +guest images and requires KVM. + +## Environments + +| Guest | Purpose | +|---|---| +| Standard Gentoo Linux | Non-root command tests | +| Gentoo Linux with SELinux MLS | Root-only and SELinux tests | +| Gentoo Linux with SMACK | SMACK-specific tests | +| Gentoo GNU/Hurd preview | Verification of the unsupported target-ABI boundary | + +The runner starts one VM at a time. It defaults to 1536 MiB RAM and two vCPUs +per guest. + +## Host requirements + +The host must provide: + +- Bash and Bun; +- readable and writable `/dev/kvm`; +- `qemu-img` and `qemu-system-x86_64`; +- OVMF firmware; +- OpenSSH client tools; +- `curl` and GnuPG; +- `mkfs.ext4`; +- either `xorriso` or `genisoimage`; +- `awk`, `sort`, `comm`, `sha512sum`, `tar`, `xz`, and `timeout`. + +Package names vary by distribution. Check the actual executables and firmware +paths without downloading or booting anything: + +```sh +scripts/qemu/check-host.sh +scripts/setup-qemu-environments.sh --dry-run +scripts/run-gnu-qemu-matrix.sh --dry-run +``` + +There is no software-emulation fallback. The scripts require KVM and use +`-cpu host`. + +## Storage and downloads + +State is stored under `/var/tmp/bnu-qemu` by default. Set `BNU_QEMU_DIR` to +use another writable location: + +```sh +BNU_QEMU_DIR=/path/to/bnu-qemu scripts/setup-qemu-environments.sh --dry-run +``` + +Allow about 30 GiB of free space for a complete first setup. Images are sparse, +but downloads, package data, kernel sources, modules, and guest overlays consume +real space. The setup checks available space before it starts. + +Downloaded Gentoo artifacts are authenticated before use. Automated release +artifacts are checked with Gentoo's service keys. The Hurd preview is checked +against its pinned signing-key fingerprint and published SHA-512 digest. + +Temporary extraction and seed-building data use `${TMPDIR:-/tmp}`. + +## Provisioning + +Create or resume every environment: + +```sh +scripts/setup-qemu-environments.sh +``` + +Provision a single environment with `--only`: + +```sh +scripts/setup-qemu-environments.sh --only standard +scripts/setup-qemu-environments.sh --only selinux +scripts/setup-qemu-environments.sh --only smack +scripts/setup-qemu-environments.sh --only hurd +``` + +Provisioning is resumable. Ready markers prevent completed guests from being +rebuilt on later runs. + +The SELinux guest uses the MLS policy in enforcing mode. Bun requires executable +anonymous memory for its FFI compiler, so provisioning enables the policy's +standard `allow_execmem` boolean. The SMACK guest uses a separately built +kernel and a distinct overlay. + +## Running the matrix + +Run the local suite followed by the complete matrix: + +```sh +scripts/run-all-tests.sh --setup +``` + +Run only the GNU matrix: + +```sh +scripts/run-gnu-qemu-matrix.sh --setup +``` + +Omit `--setup` after the guests have been provisioned. Each run rebuilds the +payload from the current checkout and attaches it read-only to the guests. + +The matrix reconciles all 733 test names. It fails if any test lacks a passing +record unless that test is one of the six documented Bun runtime or ABI +boundaries. It also reports a boundary that unexpectedly starts passing so the +allowlist can be updated. + +Results are written to: + +```text +$BNU_QEMU_DIR/results/TIMESTAMP/ +``` + +The directory contains the inventory, per-guest logs, passing and unresolved +sets, the accepted boundary list, and any unexpected unresolved tests. + +## Configuration + +The main environment variables are: + +| Variable | Default | Purpose | +|---|---:|---| +| `BNU_QEMU_DIR` | `/var/tmp/bnu-qemu` | Persistent state and results | +| `BNU_QEMU_MEMORY` | `1536M` | Memory assigned to each VM | +| `BNU_QEMU_CPUS` | `2` | vCPUs assigned to each VM | +| `BNU_QEMU_SSH_PORT` | `22222` | Host port used for guest SSH | +| `BNU_QEMU_TIMEOUT` | `900` | Guest SSH readiness timeout in seconds | +| `BNU_QEMU_MIN_FREE_GIB` | phase-dependent | Override the free-space guard | +| `BNU_OVMF_CODE` | auto-detected | Explicit OVMF code image | +| `BNU_OVMF_VARS` | auto-detected | Explicit OVMF variable template | + +Heavy guest builds are configured to use one compiler job. Change VM sizing +only when the host has enough resources for the test limits and guest overhead. + +## Smaller root smoke test + +A separate initramfs runner checks privileged ownership changes, special-file +creation, and `cp -p` metadata preservation without provisioning the full +matrix: + +```sh +BNU_QEMU_KERNEL=/boot/vmlinuz-... \ +BNU_QEMU_INITRAMFS=/boot/initramfs-....img \ + scripts/run-qemu-root-smoke.sh +``` + +It requires a matching host kernel and initramfs, KVM, +`qemu-system-x86_64`, `zstd`, and `cpio`. It uses `lsinitrd` when +available and otherwise attempts direct decompression. diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..0f6b89e --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,114 @@ +# Testing BNU + +BNU has three test levels: + +1. the local Bun suite; +2. selected or complete GNU command tests on the current host; +3. the complete cross-environment GNU test matrix in KVM guests. + +Most changes need the local suite and relevant GNU tests. The VM matrix is for +release-level or platform-specific validation. + +## Local suite + +```sh +bun run test +``` + +Each test runs in a fresh process. The runner defaults to a 1 GiB aggregate +process-tree RSS limit, a 3 GiB per-process address-space limit, and one +`factor` worker. + +The defaults can be changed for a controlled test run: + +```sh +BNU_TEST_RSS_LIMIT=2GiB \ +BNU_TEST_MEMORY_LIMIT=4GiB \ +BNU_TEST_TIMEOUT=600000 \ +BNU_FACTOR_WORKERS=2 \ + bun run test +``` + +## GNU command tests + +The repository includes `coreutils-9.11.tar.xz` and a harness that extracts +and adapts its command tests to invoke BNU. + +With no paths, the harness runs a maintained default selection: + +```sh +bun run test:gnu +``` + +To select tests: + +```sh +bun run test:gnu -- --list +bun run test:gnu -- tests/misc/echo.sh +bun run test:gnu -- tests/misc/echo.sh tests/misc/printenv.sh +bun run test:gnu -- --all --list +bun run test:gnu -- --all +``` + +By default, failed GNU tests are reported but do not make the harness return a +failure status. Use `--strict` in CI or whenever the exit status matters: + +```sh +bun run test:gnu -- --strict tests/misc/echo.sh +``` + +### Harness options + +| Option | Effect | +|---|---| +| `--all` | Select all 733 tests in GNU's standard command-test inventory. | +| `--list` | Print the selected test names without running them. | +| `--strict` | Return a non-zero status if any selected test fails. | +| `--root-tests` | Restrict the selection to tests GNU marks as root-only. | +| `--nonroot-tests` | Exclude tests GNU marks as root-only. | +| `--expensive` | Set `RUN_EXPENSIVE_TESTS=yes`. | +| `--very-expensive` | Enable both expensive and very-expensive GNU cases. | +| `--tty` | Give every selected test a controlling pseudo-terminal. | +| `--keep` | Preserve the extracted source and inner temporary directories. | +| `--timeout VALUE` | Set the per-test timeout; the default is 300 seconds. | +| `--max-output SIZE` | Set the retained output limit; the default is 1 MiB. | +| `--rss-limit SIZE` | Set the process-tree RSS limit; the default is 1536 MiB. | +| `--memory-limit SIZE` | Set the per-process address-space limit; the default is 3 GiB. | +| `--tarball PATH` | Use a different GNU coreutils source tarball. | + +Known terminal-dependent tests receive a pseudo-terminal automatically. +Root-only tests still require a suitable privileged environment. The complete +matrix routes them to a dedicated guest. + +Examples: + +```sh +bun run test:gnu -- --strict --timeout 10s tests/misc/printenv.sh +bun run test:gnu -- --expensive tests/tail/big-4gb.sh +bun run test:gnu -- --strict --very-expensive tests/factor/t26.sh +``` + +GNU's `gnulib-tests` exercise internal C portability code and are not part of +this harness. + +## Complete KVM matrix + +The complete entry point runs the local suite, then the GNU inventory in its +required guest environments: + +```sh +scripts/run-all-tests.sh --setup +``` + +Later runs can reuse the provisioned guests: + +```sh +scripts/run-all-tests.sh +``` + +Use `scripts/run-gnu-qemu-matrix.sh` when the local suite is not needed. The +matrix accepts only the boundaries listed in +[GNU test boundaries](gnu-test-boundaries.md). + +Provisioning, host requirements, storage, and result locations are documented +in [QEMU test matrix](qemu-test-matrix.md). |
