Methodology
How every benchmark is designed, executed, and evaluated.
How Tests Are Executed
Every benchmark follows a standardized execution pipeline:
- Design — Define a specific, measurable question with a documented scoring rubric
- Prepare — Write standardized prompts and prepare test materials
- Execute — Run each model via direct API access with documented parameters
- Evaluate — Score outputs using automated tools and/or human review
- Publish — Release full methodology, prompts, scores, and downloadable artifacts
All API calls are made to official provider endpoints. We do not use proxies, caching layers, or third-party routing that could alter model behavior.
Prompt Standardization
Every model in a benchmark receives identical prompts. We do not tune prompts per model. If a standardized prompt disadvantages a specific model, that is a valid experimental result.
See our Methodology Updates for the full prompt standardization policy.
Temperature & Parameters
| Task Type | Temperature | Rationale |
|---|---|---|
| Coding, math, extraction | 0 | Deterministic, reproducible outputs |
| Technical writing | 0.3 | Minor variation acceptable |
| Creative writing | 0.7 | Variation expected and documented |
All parameters (max tokens, top_p, seed) are documented on each benchmark page.
Context
We document the full context provided to each model, including token counts. If a model cannot fit the required context window, it is excluded from that benchmark with a documented reason. We never silently truncate context.
Evaluation
Evaluation methods vary by benchmark type:
- Automated — Test suites, constraint validators, citation verifiers
- Human review — Blind scoring by domain experts using published rubrics
- Hybrid — Automated pre-screening followed by human review of edge cases
Scoring
Each benchmark publishes its own rubric with explicit weights. Scores are on a 0–100 scale. The leaderboard aggregates scores across benchmarks — see Leaderboard for aggregation methodology.
Human Review
When human review is involved:
- Reviewers are blind to which model produced which output
- Minimum 2 reviewers per benchmark; disagreements resolved by a third
- Reviewers have verified domain expertise
- Individual reviewer scores are not published; aggregate scores are
Methodology Updates
Rubric revisions, prompt policy changes, and other methodological updates live in Methodology Updates. We keep them separate from benchmark results so the site stays focused on experiments and reproducibility.