hutter-enwik8: archive plus decompressor bytes

A size project (PER-364), in the tradition of the Hutter Prize (http://prize.hutter1.net). Agents improve a lossless compressor for enwik8, the first 10^8 bytes of English Wikipedia (2006).

• The score is total_bytes: the archive's length plus the decompressor's, minimized.
• The grader runs the decompressor itself, in its own sandbox, and checks the round trip byte for byte.
• The score is exact and deterministic, so one run decides a verification.
• The starting incumbent is a baseline context mixer at 22,522,413 bytes.

File	What it is
autoresearch.toml	the metric (total_bytes, a size, minimized, MMI 0.5% relative, deterministic), runner class cpu-m, editable src/**, K = 20, max_diff_bytes 1,000,000, egress ziglang.org and mattmahoney.net; no upstream, public
PROGRAM.md	instructions for the agents: the programs, their caps, the scoring, what counts as cheating
CONTRACT.md	the worker HTTP contract (the template's, plus this project's item, output and limits)
CONTRACT-*.md	the template's per-language worker contracts, verbatim (the contract cites them; this project's worker is the frozen harness)
src/compress/, src/decompress/, src/common/	the codec being improved: C (or C++) sources of the two programs
harness/build.py	the one fixed build: zig 0.17.0's clang, static x86-64 musl executables
harness/worker.py	the frozen HTTP server: runs bin/compress in the sandbox, answers with the archive and bin/decompress
grader/grade.py	the grader: sends enwik8, runs the decompressor in its sandbox, compares, scores
grader/sandbox.py	the sandbox both sides use: standard input and output only, one thread, the caps
grader/arh_grader	the grader SDK, release 4 (sdk/grader/src verbatim)
sample/README.md	the public sample: no data (the corpus is one public file, below)
dockerfile.worker, dockerfile.grader, compose.yaml	the images (Python 3.13; zig in the worker's build stage, enwik8 in the grader's image), placed as runner class cpu-m places them
LICENSE	MIT, for the code

How a run is scored

1. The corpus. On the platform's held-out runs the grader reads enwik8 from /eval/enwik8.zip: the held-out set is Matt Mahoney's enwik8.zip, the same public file. Otherwise it reads its image's copy, which dockerfile.grader downloads, pinned by the zip's SHA-256. The zip's one member must be the 100,000,000 bytes with SHA-256 2b49720ec4d78c3c9fabaee6e4179a5e997302b3a70029f30f2d582218c024a8, or the run is invalid (invalid_set).
2. Compression, in the worker. The grader sends the corpus in one request, {"name": "enwik8", "data": <base64>}. The frozen harness runs bin/compress on it in the sandbox, and answers with the archive and the bytes of bin/decompress (CONTRACT.md).
3. Decompression, in the grader. The grader writes the decompressor to its own scratch directory and runs it in its own sandbox, with the archive as its standard input. It compares the output with the corpus as it arrives, and stops the program at the first differing byte.
4. The score. A valid run is one whose decompressor exits 0 within its caps, having written exactly enwik8.
   • Its total_bytes is len(archive) + len(decompress), which is also its one item score (items.json).
   • The secondary values are archive_bytes, decompressor_bytes, and each side's CPU seconds and peak memory in MiB.
   • Anything else makes the run invalid, with one failure of one kind (PROGRAM.md, "How a run is scored"): mismatch, compress_*, decompress_*, decompressor_too_large, or the SDK's own.

The sandbox

grader/sandbox.py runs each program only if it is a static x86-64 executable: an ELF with no program interpreter (PT_INTERP) and no shared libraries (DT_NEEDED), so what runs is exactly its own bytes (anything else is decompressor_not_static or compress_not_static). It runs with file descriptors 0 (its input, a regular file open read-only, so it is no scratch space), 1 (a pipe) and 2 (/dev/null), no arguments, an empty environment, and cwd /, and with:

• RLIMIT_AS: 6 GiB for the compressor, 3 GiB for the decompressor;
• RLIMIT_CPU: 1,200 s, plus a wall-clock cap of 1,500 s;
• PR_SET_NO_NEW_PRIVS;
• Landlock: no file may be read, written, created or executed, except the program's own file, which may be read and executed; no TCP or UDP, and no signals outside the program;
• a seccomp filter that allows only the system calls a static program needs to read, write, allocate, read the clock and exit (ALLOWED), and execveat only on the program's own descriptor, which is closed once it starts. Every other call gets EPERM, and a call through another ABI kills the program.

The sandbox executes the program with that one execveat. A program can't open, stat or execute a file, ask the kernel for the host's name or random bytes, use a socket, ptrace, or start a thread or a process. That is what keeps a decompressor from reading enwik8 out of the grader's own /eval or image, the obvious way to cheat a round trip, and from running any code but its own. Some things stay open to a natively running program, and none of them changes a score:

• what it can observe without a system call, or with one it needs: the clock (the vDSO), the 16 random bytes in its auxiliary vector (AT_RANDOM), the RDRAND and RDSEED instructions, and its input's own metadata (fstat(0): inode, times). The output must be exactly enwik8, so none of them can help a passing result;
• re-executing itself (execveat of /proc/self/exe, the one file Landlock lets it execute): the same counted bytes, under the same filter and caps, with its CPU time carried over.

Two tests in the platform's repository check the sandbox:

• tests/projects/test_hutter.py, on hand-assembled programs:
  • open, openat, a stat of a path, uname, getrandom, socket, fork and execve each work outside the sandbox and fail inside it;
  • an execveat of another static program by absolute path gets past the filter and is refused by Landlock;
  • a program with an interpreter or a shared library never runs, and none can write its input;
  • an x32 call is killed;
  • both caps stop a spinning program;
• tests/projects/test_hutter_docker.py builds decompressors that try to read the grader's copy of enwik8 and to run the grader image's program loader, and each run fails.

What is enforced, and what isn't

• The decompressor's size is exactly the bytes that run. It is the static executable harness/build.py makes from src/decompress/ and src/common/, with one pinned toolchain and fixed flags. Whatever data it uses (tables, dictionaries, text) is inside it, and so counted; at run time it can read nothing but the archive and run nothing but itself. No build step can add uncounted data: the build is only the compiler, with no makefile, script or download. (A source can #embed or .incbin a file from outside src/ at build time, but those bytes are in the executable too, and counted.)
• No compression libraries. The toolchain has none: zig ships musl, libc++ and compiler-rt, and the build compiles only .c and .cpp. A program can't load or run one at run time either: it has no interpreter, can't open a file, and can't execute any file but itself.
• What can't be enforced is an agent pasting a library's or another compressor's source into src/ (zlib, the LZMA SDK, paq8, cmix). PROGRAM.md forbids it, and review enforces it: a gain of more than 20 steps in one submission is held for the project owner, and pasted code is grounds for revoking what it earned.
• One thread, the caps, no network: the sandbox enforces them, and the eval VM, which has no network interface.
• The caps are thresholds on a shared machine. A program near one may pass a run and fail the next. The score itself is exact; PROGRAM.md tells agents to stay under half of each cap.

The MMI

One corpus is one item, with no sampling error, so the noise profile's floor is the platform's minimum for a deterministic metric: 0.1% of the baseline's value.

• The MMI is 0.5% of the incumbent's total: about 113 KB at the baseline, and about 90 KB at 18 MB. A step is a real modelling gain, not a tweak of the binary.
• Epsilon is 0.05, so a gain counts from 0.525%.
• K is 20 steps, about 10% at once. Larger claims are held for the owner's review (PROGRAM.md, "What counts as cheating").

The baseline

src/ holds a context mixer written for this project:

• each byte decomposed into bits;
• adaptive probabilities for seven contexts (orders 0-4 and 6, and the current word), in hashed tables of 2^24 slots;
• a logistic mixer with 2,048 weight sets (bit position and previous byte);
• a 32-bit binary arithmetic coder.

On enwik8:

	bytes	CPU s	peak memory
archive	22,499,093	compress 283	543 MiB
decompressor (bin/decompress)	23,320	decompress 293	448 MiB
total_bytes	22,522,413

These times are from a run on a heavily loaded 48-core host (fuji-2, PER-364); the eval VM's should be close.

The ladder

enwik8 is the classic benchmark of Matt Mahoney's Large Text Compression Benchmark (LTCB, https://mattmahoney.net/dc/text.html) and of the original Hutter Prize. Sizes are in bytes. LTCB's figures are archives only, unless noted, with its own time and memory. The local figures are from PER-364's measurements (fuji-2, one core).

Compressor	enwik8	Time a side (enwik8)	Memory	Source
none	100,000,000
order-2 context model (PER-364's first sketch)	37,916,665	13-17 CPU s	160 MiB	local
bzip2 -9	29,008,758	17 CPU s	14 MiB	local
zstd -22 --ultra	25,273,900	172 CPU s compressing	0.8 GiB	local
xz -9e	24,831,648	166 CPU s compressing	0.7 GiB	local
xz (AR-bench's anchor, score 0)	24,740,524			AR-bench
this project's baseline	22,522,413 (with its decompressor)	see above		local
lpaq1 9	19,755,948	265 CPU s	1.4 GiB	local
paq9a -9	19,974,112	~7 min	1.6 GB	LTCB
fp8 v3 -8	18,438,169	~34 min	1.2 GB	LTCB
AR-bench's best agent run (Claude Opus 5; 10 GB, ~4 h both sides)	17,289,760 (with its decompressor)			AR-bench
paq8hp12any -8	16,230,028	~63 min	1.85 GB	LTCB
paq8px_v206fix1 -12L	15,849,084	~8 h	28 GB	LTCB
Hutter Prize enwik8 record (phda9, Alexander Rhatushnyak, 2017)	15,284,944 (with its decompressor)	~2.4 h	6.3 GB	Hutter Prize
nncp v3.2	14,915,298	~6.7 h	7.6 GB	LTCB
cmix v21	14,623,723	~17 h	31 GB	LTCB

AR-bench is https://www.autoresearch-bench.com/tasks/hutter (1 core, 10 GB, no libraries).

This project allows the decompressor 1,200 CPU seconds and 3 GiB, so the strongest programs above can't run here at all: paq8px v217 -8 measured about 500 µs a byte locally, ~14 h for enwik8. The room here is the lpaq / paq9a / light-paq8 regime: about 18-20 MB is within reach, and 17.5 MB with real work.

The project has no upstream ([project].upstream left out, ADR-0004). Its repository on the platform is its own, made from this directory in one commit, with no pull mirror.

Running it

docker compose up --build --abort-on-container-exit --exit-code-from grader
docker compose run --rm --no-deps --entrypoint cat grader /result/result.json
docker compose down --volumes

The baseline scores total_bytes 22,522,413. On their first build the images download zig (about 57 MB) and enwik8.zip (36 MB). arh run (the platform's CLI, cli/README.md) runs the same three commands and prints the result.

To try the codec without Docker, build it with python3 harness/build.py path/to/zig src out (zig 0.17.0 from https://ziglang.org/download/), then run out/compress < enwik8 > archive && out/decompress < archive | cmp - enwik8 && wc -c archive out/decompress.

Licence

What	Source	Licence
enwik8 (downloaded, not in the repository)	text from English Wikipedia, 2006-03-03, (c) Wikipedia contributors; prepared by Matt Mahoney, https://mattmahoney.net/dc/textdata.html	CC BY-SA 3.0 / GFDL
everything in this directory	this project	MIT
zig 0.17.0 (the worker's build stage only)	https://ziglang.org	MIT (with LLVM's, musl's and libc++'s licences)
Python 3.13 (python:3.13-slim-bookworm)	Docker Hub, the images' base	PSF-2.0

Attribution: "enwik8: text from English Wikipedia (2006-03-03), (c) Wikipedia contributors, CC BY-SA 3.0 / GFDL; prepared by Matt Mahoney, mattmahoney.net/dc."