Skip to content
Los Doritos Benchmarks
Los Doritos Benchmarks — Home
Open menu
Benchmarks Leaderboard Methodology Reports About Search
Reasoning hard Featured

Multi-Step Logic Puzzle — River Crossing with Constraints

Five models solve a classic reasoning puzzle with added constraints. Scored on logical validity, step completeness, and constraint satisfaction.

Published May 28, 2025

Goal

Evaluate structured reasoning and constraint-tracking ability across frontier models without tool use.

Models Compared

  • o1 OpenAI 2024-12-17
  • Claude 3.5 Sonnet Anthropic 20241022
  • Gemini 2.0 Pro Google gemini-2.0-pro-exp-02-05
  • DeepSeek R1 DeepSeek deepseek-reasoner
  • Mistral Large Mistral mistral-large-2411

Exact Prompts

system system 
Solve the puzzle step by step. Verify each step against all constraints before proceeding.
user user 
Three missionaries and three cannibals must cross a river using a boat that holds at most two people.
The boat cannot cross empty. At no point may cannibals outnumber missionaries on either bank.

ADDITIONAL CONSTRAINTS:
1. Missionary A and Cannibal B cannot be alone together on any bank.
2. The boat must return with at least one person (no solo equipment trips).
3. Exactly 11 crossings are required (not 11 trips — 11 one-way crossings).

Provide the complete sequence of crossings.

Parameters

Temperature
0
Max Tokens
8192
Top P
1
Seed
7

Context

No external tools. Models must show step-by-step reasoning. Puzzle has 4 agents, 3 constraints, and 12 decision points.

Evaluation

Method
Automated constraint validator + expert review
Rubric
Valid solution (50%), optimal steps (20%), constraint compliance (20%), explanation clarity (10%)
Human Review
Yes (3 reviewers, blind)

Detailed Analysis

Evaluation Notes

The base river-crossing puzzle is well-represented in training data. The added constraints (especially constraint 3 requiring exactly 11 crossings) differentiate models that pattern-match from those that reason through state spaces.

All reasoning traces are preserved in the downloadable JSON artifact.

Strengths & Weaknesses

o1

  • Complete valid solution in exactly 11 crossings
  • Explicit constraint checking at each step

DeepSeek R1

  • Excellent cost efficiency
  • Detailed chain-of-thought

Claude 3.5 Sonnet

  • Violated constraint 1 on step 7
  • Required 13 crossings instead of 11

Mistral Large

  • Invalid state on crossing 4
  • Did not acknowledge constraint 3

Verdict

o1 delivers the most reliable reasoning on constrained puzzles. DeepSeek R1 offers remarkable value. Non-reasoning models struggle with the added constraints.