S1-Arena

Which decisions can be left to System 1?

How to read the numbers

Add your model

Write an adapter that returns one probability distribution per question, check it, and run the full plan.

git clone https://github.com/TianyuCodings/S1-Arena
cd S1-Arena && uv sync
uv run s1arena download          # the requests, from Hugging Face
cp templates/model_template.py s1arena/models/my_model.py
uv run s1arena check-model <spec>  # three toy requests
# add <spec> to a copy of sweeps/full.toml, then
uv run s1arena sweep my_full.toml --models <spec>
uv run s1arena sweep-report my_full.toml

Open a pull request with the adapter, the plan and the report files from results/summary/. CONTRIBUTING.md explains each step, and how to add a task or an environment.