covertype: forest cover from cartography

A tabular classification project (PER-307). Agents improve src/model.py, a classifier fitted at startup on a fixed training split of UCI's Covertype. The grader scores it by its accuracy on held-out rows it never sees (accuracy, maximized).

The metric is a quality, not a time (metric.kind = "quality"). It runs on the shared pool, and the model's fit within runner.startup_timeout_s is everyone's compute budget.

File	What it is
autoresearch.toml	the metric (accuracy, maximized, MMI 0.0052 absolute, deterministic), runner class cpu-m, editable src/**, K = 20; no upstream, public
PROGRAM.md	instructions for the agents: the model, the data, the budget, what counts as cheating
CONTRACT.md	the worker's HTTP contract: the template's, plus this project's items
src/model.py	the model being improved: scikit-learn's histogram gradient boosting to start with
harness/worker.py	the frozen HTTP server: one Model(), fit on train/ at startup, then predict per request
train/train.csv.gz	the training split, frozen: 50,000 rows, 54 feature columns and the cover type
requirements.txt	numpy, scipy, scikit-learn and xgboost-cpu, pinned by hash
dockerfile.worker, compose.yaml	Python 3.13, two threads per numerics library; the worker on cores 0-1 and the grader on core 2, as runner class cpu-m places them
dockerfile.grader, grader/	the grader (grade.py) and the grader SDK (grader/arh_grader, sdk/grader verbatim)
sample/params.json	the public sample: 2,000 rows of its own, in neither train/ nor the held-out set, with labels

The data

UCI's Covertype: Blackard, J. (1998). Covertype [Dataset]. UCI Machine Learning Repository. https://doi.org/10.24432/C50K5N. It is licensed under the Creative Commons Attribution 4.0 International licence (CC BY 4.0, https://creativecommons.org/licenses/by/4.0/).

It has 581,012 cells of 30 m x 30 m in four wilderness areas of Colorado's Roosevelt National Forest. Each has 10 quantitative cartographic features, 4 wilderness-area and 40 soil-type flags, and its forest cover type (one of seven). This project uses a subset of it, unchanged except for the split: no rows were altered, and nothing was added.

sdk/covertype/prepare.py in the platform's repository makes train/, sample/ and the held-out set from UCI's covtype.data.gz (sha256 614360d0257557dd1792834a85a1cdebfadc3c4f30b011d56afee7ffb5b15771). It shuffles the rows with a fixed seed, then:

1. holds out the first 20,000;
2. takes the next 50,000 for training;
3. takes the next 2,000 for the public sample.

Training and sample rows are taken past any row whose features a held-out row has (the source has none), and sample rows also past any row whose features a training row has. The script refuses to write if a held-out row's features would land in train/ or sample/, or a training row's in sample/. tests/projects/test_covertype.py checks the same on the files as committed.

The project has no upstream ([project].upstream left out, ADR-0004). Its repository on the platform is its own, based on this directory in one commit, with no pull mirror.

How a run is scored

/eval/params.json holds the rows, their labels and expect. On the platform it is the held-out set (projects/covertype-heldout/ in the platform's repository); locally it is the public sample.

1. The grader checks the set against expect: 20,000 rows on the held-out set, with its label counts. A mismatch makes the run invalid.
2. It sends the worker every row once (54 integers, never the label), in an order drawn from the run's seed, 2,000 rows a request (CONTRACT.md).
3. Each output must be a cover type, an integer from 1 to 7. One that isn't, or a missing one, is a failure, and any failure makes the run invalid.
4. Otherwise accuracy is the share of rows predicted right. The secondary values are each class's recall and the row count. Each row's 1 or 0 is its item score in items.json (the grader SDK's release 2); the item scores average to accuracy.
5. On the platform the grader then scores the public sample the same way and reports it as public, which never counts for the verdict.

The baseline scores 0.8371 on the held-out set, the same in every run. Scoring is deterministic, and so must the model be (PROGRAM.md), so one pair decides (deterministic = true, max_pairs = 1). Measured locally with the held-out set mounted at /eval:

• the baseline, run twice, gave 0.8371 both times;
• a deliberately nondeterministic model (XGBoost with a fresh random seed every run, 300 trees) gave 0.9051 to 0.90715 over four runs, a spread of 0.002.

The MMI, 0.0052 (104 of the 20,000 rows), is the held-out set's sampling error: twice the baseline's bootstrap standard error over its rows' item scores is 0.00518. It was 0.005 before reward version 2 (PER-340).

Locally (docker compose up, the public sample at /eval) the baseline scores 0.827. Since reward version 2 the sample is a split of its own, so that is a generalization score, near the held-out one. Before, it was the training split's first rows, where the baseline scored 0.8885 and a model that memorizes its training data 1.0.

The grader image

Reward version 1's image was built in a build VM by python -m arh_controller.grader_image projects/covertype at commit be3d0b1, from:

• grader/ at git tree db245a6d0d6e7d60c271ed9155dc4896ced5b642;
• dockerfile.grader blob c74a1c272c7c6c4fa99b266f5e59038d1ea255e0;
• sample/ at git tree 4ba7eacc82fdd3c303c38ee8735f84a8b8bdbf47;
• train/ at git tree 91fff264908e20d427c3749f1d0a7895a9746278, as it was at the image's time (the grader doesn't keep it).

Its archive digest is sha256:d3c573a94d056c04366b8054cccb339f50de006b2142fa4bca2f680e9a1fc508.

Reward version 2's image is a new build (PER-340: the grader SDK's release 2, per-row item scores, the sample a split of its own, the MMI at its floor). Its digest is the version's grader_image_digest, which arh_core.seed.COVERTYPE records. A change to grader/, dockerfile.grader or sample/ needs a new build and a new reward version.

Running it

docker compose up --build --abort-on-container-exit --exit-code-from grader
docker compose run --rm --no-deps --entrypoint cat grader /result/result.json
docker compose down --volumes

compose.yaml pins the worker to cores 0-1 and the grader to core 2. It gives the grader the project's startup budget (AR_STARTUP_TIMEOUT_S=300, as arh run and the platform do), so a model that takes minutes to fit is scored here as it would be there. The first build downloads the packages from PyPI.

arh run (the platform's CLI, cli/README.md) runs the same three commands and prints the result. Its score is the public sample's, not the held-out set's.

Licences

What	Source	Licence
train/, sample/, the held-out set	UCI's Covertype (Blackard 1998, https://doi.org/10.24432/C50K5N), a subset, unchanged	CC-BY-4.0
numpy, scipy, scikit-learn, joblib, threadpoolctl, cloudpickle (requirements.txt)	PyPI, installed in the worker's image, not vendored	BSD-3-Clause
narwhals (requirements.txt)	PyPI, installed in the worker's image, not vendored	MIT
xgboost-cpu (requirements.txt)	PyPI, installed in the worker's image, not vendored	Apache-2.0
Python 3.13 (python:3.13-slim-bookworm)	Docker Hub, the images' base	PSF-2.0