We tested 4 different LLMs in a real-world RAG application, while preserving everything else: Gemini 3.5 Flash‑Lite at low thinking level (currently used), Gemini at minimal, GPT‑6 Luna at low thinking, Claude Haiku 5.5 at low, and Mercury 3 at low. GPT‑6 Luna gave the best answers, while lowering LLM cost by 84%: $0.60 vs. $3.75 per 1,000 answers. Claude Haiku also gave better answers than Gemini and reduced the cost by 70%, but it was the slowest. Mercury 3 and Gemini at minimal thinking cut the cost with no clear change in answer quality, but had other issues which Luna did not.
155 questions × 3 runs × 5 LLM setups · four Stoic texts · local, Apple M4 Pro · graded by two AI models
Published October 28, 2026
As evidenced by the recent rebuild of our Stoic Q&A app, less is often more. A simpler architectural structure and RAG tech stack resulted in much more accurate answers, although the LLM was switched as well: from Gemini 2.5 Flash to Gemini 3.5 Flash‑Lite. Median response time fell from 12.6 s to 4.6 s, and LLM cost from $7.23 to $3.72 per 1,000 answers. Although this study cannot establish which changes, and exactly how much, contributed to the app’s overall better performance, simpler architecture and tech stack may be the way to go. It’s also worth experimenting with newer - and often cheaper - LLMs to find the best balance between cost and performance.
140 questions × 3 runs × 2 versions · four Stoic texts · local, Apple M4 Pro · graded by a human and two AI models
Published October 26, 2026 · updated October 28, 2026
GPT‑6.1 Sol cost 76% less than GPT‑6 Astra to grade the same 900 question pairs, and took about the same time: 62.7 s per review vs. 61.0 s. Both matched Claude Opus 5.5’s factual verdicts 98.1% of the time. On pass/fail judgment, Astra agreed with Opus 94.4% vs. Sol’s 93.6%.
180 question sets × 2 judges · 900 question pairs · Claude Opus 5.5 as reference
Published October 5, 2026 · updated October 30, 2026
Gemini 3.1 Flash‑Lite wrote complete, usable question sets for 90% of 300 childcare providers, grounded in official state inspection records processed by DaycareFacts.com. Gemma 4 E4B with thinking on, run locally on a Mac Mini (M4 Pro, 24 GB RAM): 73%, and with thinking off: 65%. Gemini was also substantially faster, with 1.81 s per set vs. 7.45 s and 4.64 s, respectively.
300 childcare providers × 3 setups · Florida, California and Texas inspection records · graded by three AI models
Published October 3, 2026
Average response time fell from 9.1 s to 1.6 s, and LLM cost from $4.03 to $0.46 per 1,000 questions. The difference in answer accuracy was too small to call either way.
100 questions × 3 runs × 5 setups · 2026 F‑150 owner’s manual · local, Apple M4 Pro · graded by two AI models
Published September 25, 2026
$1,281,235 ad spend · 64,009 leads · $19.97 cost per lead · 40.04% conversion rate · $9,167,785 in sales · 716% return on ad spend
2016–2024 · 5 metro areas, East Coast · Google Ads
Published September 25, 2026
GPT‑6 Astra finished first in all 26 grading batches, averaging 4.1 minutes per 30 answers vs. GPT‑5.6 Sol’s 7.3. They agreed on 95% of verdicts, but Sol graded the answers more strictly.
717 answers × 2 judges · 26 batches · extra-high vs. high reasoning
Published September 25, 2026
$912,316 ad spend · 37,837 leads · $24.07 cost per lead · 40.88% conversion rate · $6,323,556 in sales · 693% return on ad spend
2016–2024 · Tampa and Orlando, FL · Google Ads
Published September 25, 2026