Research expert
Research Synthesis — Climate Policy Papers (2020–2024)
Models synthesize findings from 8 provided research papers on carbon pricing. Scored on citation accuracy, synthesis quality, and hallucination rate.
Published May 15, 2025
Goal
Test whether models can accurately synthesize provided sources without hallucinating citations or misrepresenting findings.
Models Compared
- GPT-4o OpenAI 2024-11-20
- Claude 3.5 Sonnet Anthropic 20241022
- Gemini 2.0 Pro Google gemini-2.0-pro-exp-02-05
Exact Prompts
system system
Synthesize the provided research papers. Only cite papers that are in the provided set.
Flag any areas where papers disagree. Do not add external knowledge.
user user
Write a 2,000-word research synthesis answering:
1. What is the range of estimated carbon price elasticity across studies?
2. Which policy designs show the strongest emission reductions?
3. Where do the papers disagree, and why?
Cite each claim with (Author, Year). Only use the 8 provided papers.
Parameters
- Temperature
- 0
- Max Tokens
- 8192
- Top P
- 1
Context
Eight peer-reviewed papers (provided as full text, ~45,000 tokens total) on carbon pricing effectiveness across OECD countries 2020–2024. Models must cite specific papers by author and year.
Evaluation
- Method
- Citation verification script + domain expert review
- Rubric
- Citation accuracy (40%), synthesis quality (25%), disagreement identification (20%), no hallucination (15%)
- Human Review
- Yes (2 reviewers, blind)
Detailed Analysis
Hallucination Tracking
We ran an automated citation verifier against all (Author, Year) pairs in each output.
Strengths & Weaknesses
Claude 3.5 Sonnet
- Zero hallucinated citations
- Identified 3 areas of disagreement
- Accurate elasticity range (0.3–0.7)
GPT-4o
- One citation to paper not in provided set
- Missed disagreement on developing economies
Gemini 2.0 Pro
- Two incorrect author-year pairs
- Oversimplified policy design section
Verdict
Claude 3.5 Sonnet is the most trustworthy for research synthesis with provided sources. Citation verification is essential — even top models hallucinate references.