We tested 4 different LLMs in a real-world RAG application, while preserving everything else: Gemini 3.5 Flash‑Lite at low thinking level (currently used), Gemini at minimal, GPT‑6 Luna at low thinking, Claude Haiku 5.5 at low, and Mercury 3 at low. GPT‑6 Luna gave the best answers, while lowering LLM cost by 84%: $0.60 vs. $3.75 per 1,000 answers. Claude Haiku also gave better answers than Gemini and reduced the cost by 70%, but it was the slowest. Mercury 3 and Gemini at minimal thinking cut the cost with no clear change in answer quality, but had other issues which Luna did not.
Published October 28, 2026
As evidenced by the recent rebuild of our Stoic Q&A app, less is often more. A simpler architectural structure and RAG tech stack resulted in much more accurate answers, although the LLM was switched as well: from Gemini 2.5 Flash to Gemini 3.5 Flash‑Lite. Median response time fell from 12.6 s to 4.6 s, and LLM cost from $7.23 to $3.72 per 1,000 answers. Although this study cannot establish which changes, and exactly how much, contributed to the app’s overall better performance, simpler architecture and tech stack may be the way to go. It’s also worth experimenting with newer - and often cheaper - LLMs to find the best balance between cost and performance.
Published October 26, 2026 · updated October 28, 2026
GPT‑6.1 Sol cost 76% less than GPT‑6 Astra to grade the same 900 question pairs, and took about the same time: 62.7 s per review vs. 61.0 s. Both matched Claude Opus 5.5’s factual verdicts 98.1% of the time. On pass/fail judgment, Astra agreed with Opus 94.4% vs. Sol’s 93.6%.
Published October 5, 2026