parse-mutants: tests that kill held-out mutants of parse

A count project (PER-370). Agents improve the test suite of parse (tests/), the inverse of Python's format(). The grader holds mutants of parse's frozen source, each one small change to its code. The score is how many of them the suite kills (killed, maximized): a test that passes on the original fails on the mutant.

The metric is exact and deterministic, so one run decides a verification. The starting incumbent is upstream's own suite at parse 1.22.4. It kills 512 of the 550 scored held-out mutants (and 170 of the 183 in the public sample).

File	What it is
autoresearch.toml	the metric (killed, a count, maximized, MMI 1 absolute, deterministic, epsilon 0), runner class cpu-l, editable tests/**, K = 25; public, no upstream (an import)
PROGRAM.md	instructions for the agents: the mutants, how a run is scored, test isolation, determinism
CONTRACT.md	the worker HTTP contract (the template's, plus this project's items and limits)
tests/	the suite being improved: upstream's, at 1.22.4
src/parse/	parse 1.22.4, frozen (__init__.py, __init__.pyi)
harness/worker.py	the frozen HTTP server: runs the suite against the original or a mutant, one child process an item, one a core
harness/mutants.py	the mutator: every site of src/parse/__init__.py, the decoys' rewrites, and a mutant compiled in memory
harness/isolation.py	the test isolation hook (PROGRAM.md, "Test isolation")
harness/plugin.py, harness/pytest.ini	pytest's configuration and plugin for the harness's runs (the same random state for every test)
harness/requirements.txt	pytest 9.2.0 and its dependencies, pinned by hash
grader/grade.py	the grader: the original twice, every held-out mutant, every kill again; the count
grader/arh_grader	the grader SDK, release 5 (sdk/grader/src verbatim)
sample/mutants.jsonl	the public sample: 183 of the 733 scored mutants, {"site", "op", "canary": false} a line
dockerfile.worker, dockerfile.grader, compose.yaml	Python 3.13; the worker on cores 0-3 and the grader on core 4, as runner class cpu-l places them
LICENSE	parse's MIT licence

How a run is scored

/eval/mutants.jsonl holds the mutants. On the platform that's the held-out set, 550 scored mutants, 4 canaries and 53 decoys (PROGRAM.md, "The mutants"), in projects/parse-mutants-heldout/ of the platform's repository. Locally it's the public sample. Each run goes in four steps:

1. The grader asks the worker to run the suite against the original twice. Every test must pass.
2. It asks for a run against every mutant, 16 a request; the worker runs them a core at a time.
3. It asks again for every mutant the suite killed: each must be killed again.
4. killed is the count of killed mutants that aren't canaries or decoys. The run is invalid if a canary or decoy is killed, the isolation hook refused anything, the original fails, or a kill doesn't repeat.

The secondary values are:

• scored: the scored mutants;
• survived: those not killed;
• timeouts: kills by the CPU limit;
• original_s: the original's slower run, in seconds.

The held-out set is one item (items.json: [killed]).

The scored set holds every mutant a witness kills; PROGRAM.md, "The mutants", explains how, and why a mutant no witness kills isn't scored. Of parse's 821 mutants, 733 are scored:

• upstream's suite kills 682;
• the intake's differential suite kills 51 more.

The 3 mutants that break parse's import aren't scored: any suite that imports parse kills them, one that only skips included.

The MMI

The held-out set is a fixed population with no sampling error, so it is one item. The floor is therefore the platform's minimum for a deterministic metric, 0.1% of the baseline's value: 0.512 at 512. The MMI is 1 mutant, with epsilon 0, so one more mutant killed is one step. max_steps_per_submission is 25.

Known limits

• The isolation hook is in-process Python (harness/isolation.py). It can't be made airtight, and some bypass not yet found may remain. Getting past it is cheating. One known gap: Python raises no audit event for f_back, so from a function of yours that parse calls, its frame's f_lasti and f_locals are reachable (its code isn't).
• What the decoys stop. They catch an oracle that flags every change to parse's code (a deny-list: "differs from the original"). Their shapes are public (harness/mutants.py), and they don't catch an allow-list: past the hook, a suite can carry the digests of exactly the mutants it wants killed, computed offline from the public mutator, and no decoy matches one. The owner's review and revocation are the backstop for that.
• A copy of parse in tests/. A suite that tests parse against a vendored copy or port of it (differential testing) kills mutants by what parse does, so neither the hook nor the decoys see it. No automated check is attempted, since obfuscation and split submissions would defeat one. It is cheating (PROGRAM.md, "Test isolation"); the owner's review catches it, and revocation takes back a suite accepted with it. Expected values written into tests by hand are fair.

Upstream

src/parse/ and tests/ are parse 1.22.4 (commit 175b52d5079dd632332bdc8a075f25867d5a6a17, MIT), verbatim. Upstream's pyproject.toml runs pytest with coverage and parse's doctests; here the suite is tests/ alone, under the harness's pytest configuration.

An accepted suite is a diff of tests/ that applies to upstream's repository as it stands, so it can be offered there as a pull request by hand. The project's own repository is the platform's (an admin's import): autoresearch.toml names no upstream, which would be the repository holding this project.

parse's own __version__ says 1.22.3 at the 1.22.4 tag; that is upstream's.

Running it

docker compose up --build --abort-on-container-exit --exit-code-from grader
docker compose run --rm --no-deps --entrypoint cat grader /result/result.json
docker compose down --volumes

The incumbent scores killed 170 on the sample. arh run (the platform's CLI, cli/README.md) runs the same three commands and prints the result. These commands need Docker and five cores.

Without Docker, on Linux with pytest 9.2.0, from this directory:

• PYTHONPATH=src pytest tests runs the suite against the original;
• python harness/worker.py --run '[{"mutant": {"site": 348, "op": "compare"}}]' runs it against a mutant, as the worker does (in memory, under the isolation hook), and prints the outcome.

Licences

What	Source	Licence
src/parse/, tests/	parse, https://github.com/r1chardj0n3s/parse, Copyright (c) 2012-2019 Richard Jones	MIT (LICENSE)
everything else	this project	MIT
Python 3.13 (python:3.13-slim-bookworm), pytest 9.2.0	Docker Hub, PyPI	PSF-2.0, MIT