What they built.
The GPT Researcher contributors investigated a specific research-system decision: which retrieved passages should reach the writer. Their benchmark records source material, reuses it across filter configurations, and publishes the resulting measurements instead of changing every part of the workflow at once.
This is stronger evidence than a runnable code snippet because the authors report an executed comparison and publish metrics. It still has a clear boundary: the full generated reports are not committed, and the small experiment is not an independent assessment of overall research quality.
An executed benchmark with inspectable scores and methods, not hypothetical healthcare code. Full generated reports are not public in this repository.
Inspect the evaluation artifactsHow the tools work together
Holding the retrieved pages and writer model constant makes differences in filtering easier to interpret. Known-answer questions and model-based judgments offer different signals. Neither replaces human source review, and a higher relevance score alone does not establish that every final conclusion is accurate.
Workflow at a glance
Record shared inputs
Collect pages for 20 SimpleQA questions and eight open-ended research questions.
Replay report generation
Change the context filter while keeping the recorded pages and writer model fixed.
Compare results
Score known answers, relevant passages and blinded report preferences; publish per-run metrics.
What the evidence supports.
The authors report 19/20 SimpleQA answers for Jev and keyword filtering, versus 18/20 for embeddings. The small task set and same model family for writing/judging limit generalization.
GPT Researcher context-filter benchmark SourceHow you could adapt the idea
Adapt the evaluation pattern before adopting its preferred configuration. Assemble questions that resemble your work, retain the same inputs for comparisons, and inspect errors as well as averages. Record actual costs and keep sensitive documents out of public result files. These are editorial suggestions for a new evaluation.
Effort, cost and limitations
- Effort
- A technical evaluation using recorded inputs, replay scripts and model-based judging; it was not rerun for this directory.
- Cost
- Published tracking is approximately $40-$41 for writing/filtering across configurations, plus an estimated $8-$15 judging. These are historical benchmark totals, not a production quote.
Only 28 questions, one writer model and research_report; detailed/deep modes were not evaluated.
Actual report text is not committed; this qualifies as a builder evaluation demonstration, not an inspectable final-report collection.
Go to the original work.
Published method, results and caveatsBenchmark code and scoresExplicit artifact availabilitySource-backed editorial account; PrisAIm did not run or independently benchmark this implementation.