Python Bug Fix — Null Reference in Async Handler
Four frontier models attempt to fix a subtle async/await bug in a FastAPI endpoint. Evaluated on correctness, test pass rate, and minimal diff size.
Published June 12, 2025 · Updated June 14, 2025
Goal
Determine which models can reliably fix production Python bugs without introducing regressions or over-engineering the solution.
Models Compared
- GPT-4o OpenAI 2024-11-20
- Claude 3.5 Sonnet Anthropic 20241022
- Gemini 2.0 Flash Google gemini-2.0-flash-001
- Llama 3.3 70B Meta 3.3-70B-Instruct
Exact Prompts
You are a senior Python engineer. Fix the bug with the smallest possible change.
Do not refactor unrelated code. Return only the corrected file.
The following test fails intermittently under load:
```
FAILED tests/test_user_handler.py::test_concurrent_user_fetch
AssertionError: Expected 10 users, got 7
RuntimeError: Task got Future attached to a different loop
```
File: handlers/user.py
[Full file content provided — 186 lines]
Fix the bug. Explain your reasoning briefly, then provide the corrected code.
Parameters
- Temperature
- 0
- Max Tokens
- 4096
- Top P
- 1
- Seed
- 42
Context
A FastAPI application with 847 lines across 12 files. The bug manifests only under concurrent load when the database connection pool is exhausted. Unit tests pass in isolation but fail in integration. Models receive the full `handlers/user.py` file (186 lines) and the failing test output.
Outputs
GPT-4o
Identified missing `await` on `db.fetch_all()` inside the async generator. Applied 3-line fix.
Claude 3.5 Sonnet
Root cause: event loop mismatch from cached connection. Fixed with proper async context manager.
Gemini 2.0 Flash
Suggested refactoring entire handler to sync — incorrect approach for async framework.
Llama 3.3 70B
Fixed the await issue but introduced a race condition in the connection pool.
Evaluation
- Method
- Automated test suite (42 tests) + human code review
- Rubric
- Pass all tests (60%), minimal diff (20%), no new warnings (10%), explanation quality (10%)
- Human Review
- Yes (2 reviewers, blind)
Detailed Analysis
Reasoning Analysis
All models correctly identified the file and the failing test. The divergence occurred in how they diagnosed the RuntimeError: Task got Future attached to a different loop error.
Claude 3.5 Sonnet traced the issue to a cached database connection created outside the current event loop — a subtle bug that only manifests under concurrent load. GPT-4o found a missing await which fixed the symptom but not the underlying connection lifecycle issue.
Execution Environment
- Runtime: Python 3.12.4, FastAPI 0.111.0, pytest 8.2.0
- Hardware: AWS c6i.xlarge (4 vCPU, 8 GB RAM)
- API Access: Direct provider APIs, no proxy or caching layer
- Timestamp: 2025-06-12T14:30:00Z — 2025-06-12T16:45:00Z UTC
Strengths & Weaknesses
Claude 3.5 Sonnet
- Correct root cause identification
- Cleanest diff (4 lines changed)
- Added defensive connection check
GPT-4o
- Fast and accurate
- Good explanation of async pitfalls
Gemini 2.0 Flash
- Suggested sync refactor in async context
- Did not run mental model of concurrency
Llama 3.3 70B
- Fixed primary bug but introduced race condition
- Over-commented the solution
Verdict
Claude 3.5 Sonnet wins on accuracy and code quality. GPT-4o is a close second with better latency. Gemini 2.0 Flash failed to understand the async context.