Skip to content
Los Doritos Benchmarks
Los Doritos Benchmarks — Home
Open menu
Benchmarks Leaderboard Methodology Reports About Search

Methodology Updates

Versioned notes for methodology changes. This replaces the old blog structure.

May 20, 2025

Prompt Standardization Guidelines

Fair benchmarks require fair prompts. This document describes how we standardize prompts across all experiments.

Principles

  1. Same prompt, same parameters — Every model in a benchmark receives identical prompts and documented parameters.
  2. No model-specific tuning — We do not optimize prompts per model. If a model performs poorly with a standardized prompt, that is a valid result.
  3. Prompts are published — Every benchmark page includes the exact prompts used, with no redactions except for copyrighted test material, which is described but not reproduced.

Temperature Policy

  • Default: temperature 0 for deterministic tasks such as coding, math, and extraction.
  • temperature 0.3 for creative writing benchmarks where some variation is expected.
  • Temperature is always documented in the benchmark parameters section.

Context Window Usage

We document how much of the context window is used and whether truncation occurred. Models that cannot fit the required context are excluded from that benchmark with a documented reason.

June 1, 2025

Introducing Our Benchmark Scoring Rubric v1.0

We are publishing the exact scoring rubric used across all Los Doritos benchmarks. Transparency in evaluation is as important as transparency in execution.

Weighted Criteria

  1. Outcome-based metrics dominate — Correctness typically carries 50–70% of the total weight.
  2. Process metrics are secondary — Explanation quality, diff size, and formatting typically carry 10–30%.
  3. Efficiency metrics are reported but rarely weighted — Cost and latency are always recorded but only weighted when the benchmark explicitly tests efficiency.

Human Review Protocol

  • Reviewers are blind to which model produced which output.
  • Minimum of 2 reviewers per benchmark.
  • Disagreements are resolved by a third reviewer.
  • Reviewer identities are not published; credentials are verified internally.

Versioning

This is rubric version 1.0. Future changes will be logged on this page with clear dates and changelog-style notes. Historical benchmarks will not be retroactively rescored unless a methodology error is discovered.