Testing different LLMs in a RAG app: better answers and 84% lower LLM cost
We tested 4 different LLMs in a real-world RAG application, while preserving everything else: Gemini 3.5 Flash‑Lite at low thinking level (currently used), Gemini at minimal, GPT‑6 Luna at low thinking, Claude Haiku 5.5 at low, and Mercury 3 at low. GPT‑6 Luna gave the best answers, while lowering LLM cost by 84%: $0.60 vs. $3.75 per 1,000 answers. Claude Haiku also gave better answers than Gemini and reduced the cost by 70%, but it was the slowest. Mercury 3 and Gemini at minimal thinking cut the cost with no clear change in answer quality, but had other issues which Luna did not.
The project
Memento Vivere is our Stoic app which bases all of its answers on 4 classic works by Epictetus (Discourses and Enchiridion), Marcus Aurelius (Meditations) and Seneca (Moral Letters to Lucilius).
In our previous study, we rebuilt the app from scratch. The rebuild gave more accurate answers, became 2.7x faster, and cut the LLM cost in half.
The rebuild, our baseline in this experiment, ran on Google’s Gemini 3.5 Flash‑Lite with thinking set to low, and about half of its LLM cost was thinking tokens. We wanted to see if another LLM could cut the cost further, without hurting answer quality or speed.
Main results and findings
We tested 5 LLM setups on the same 155 questions, while everything else stayed exactly the same: the current Gemini 3.5 Flash‑Lite at low thinking, Gemini at minimal thinking, OpenAI’s GPT‑6 Luna at low, Anthropic’s Claude Haiku 5.5 at low, and Inception’s Mercury 3 at low.
GPT‑6 Luna gave the best answers. Two AI judges rated Luna’s answers 4.16/5.00 on average, vs. 3.40 for the current Gemini setup. Luna scored higher on 112 of the 155 questions, and lower on only 8.
Luna cost 84% less: $0.60 vs. $3.75 per 1,000 answers. Its list prices are 1/3 of Gemini’s for input, and 1/5 for output, and it barely uses thinking tokens: only 2 per request on average, vs. 702 for Gemini.
Luna was slightly slower. Its median response was 4.7 s, vs. 3.9 s for Gemini.
Claude Haiku 5.5 also gave better answers and cost 70% less, but it was the slowest. The judges rated it 3.75/5.00, at $1.13 per 1,000 answers, with a median response of 7.2 s.
Gemini with minimal thinking was the fastest, and cost 48% less. Its median response was 2.4 s, at $1.96 per 1,000 answers. Its quality, 3.29 vs. 3.40, was too close to call.
Mercury 3 cost 68% less, with no clear change in quality, but it refused fair questions. It wrongly declined 64 of the 399 requests it should have answered.
Luna and Haiku gave crisis help every time. Both showed the right crisis message for all 42 crisis requests. The other three setups missed an indirect abuse message 2 or 3 times.
The OpenAI judge favored the OpenAI LLM. GPT‑6.1 Sol rated Luna 1.07 points above the current Gemini setup, while Claude Sonnet 5.5 rated it 0.45 points above. Sonnet showed no such lean toward Claude Haiku. Even by Sonnet alone, Luna gave the best answers.
GPT‑6 Luna vs. Gemini 3.5 Flash‑Lite (current)
Method
- Five setups
- Gemini 3.5 Flash‑Lite at low thinking (the app’s current setup) and at minimal thinking, GPT‑6 Luna, Claude Haiku 5.5 and Mercury 3, each at low. Everything else stayed the same: the books, the search, the prompt and the app’s fixed messages.
- Test set
- The same 140 questions as our previous study, plus 15 new crisis questions: 11 that should get crisis resources, and 4 that come close but should be answered. Each setup got every question 3 times: 465 requests per setup.
- Environment
- All five setups ran on the same Apple M4 Pro Mac, one question at a time, in shuffled order, on October 23–24, 2026.
- Judging the answers
- Two blind judges: GPT‑6.1 Sol/high and Claude Sonnet 5.5/high. Both were first checked against 60 answers rated by a human.
- Cost basis
- LLM cost is each provider’s charges at list prices on October 23, 2026. Mercury 3 at its launch discount.
Quality, cost and speed
| LLM | Thinking | Overall quality, 1 to 5 | LLM cost per 1,000 answers | Median response time |
|---|---|---|---|---|
| Gemini 3.5 Flash‑Lite | Low (current) | Overall quality, 1 to 53.40 | LLM cost per 1,000 answers$3.75 | Median response time3.9 s |
| Gemini 3.5 Flash‑Lite | Minimal | Overall quality, 1 to 53.29 | LLM cost per 1,000 answers$1.96 | Median response time2.4 s |
| GPT‑6 Luna | Low | Overall quality, 1 to 54.16 | LLM cost per 1,000 answers$0.60 | Median response time4.7 s |
| Claude Haiku 5.5 | Low | Overall quality, 1 to 53.75 | LLM cost per 1,000 answers$1.13 | Median response time7.2 s |
| Mercury 3 | Low | Overall quality, 1 to 53.44 | LLM cost per 1,000 answers$1.20 | Median response time4.4 s |
Answer quality
| LLM | Thinking | GPT‑6.1 Sol | Claude Sonnet 5.5 | Both judges |
|---|---|---|---|---|
| Gemini 3.5 Flash‑Lite | Low (current) | GPT‑6.1 Sol3.47 | Claude Sonnet 5.53.34 | Both judges3.40 |
| Gemini 3.5 Flash‑Lite | Minimal | GPT‑6.1 Sol3.27 | Claude Sonnet 5.53.32 | Both judges3.29 |
| GPT‑6 Luna | Low | GPT‑6.1 Sol4.53 | Claude Sonnet 5.53.79 | Both judges4.16 |
| Claude Haiku 5.5 | Low | GPT‑6.1 Sol3.89 | Claude Sonnet 5.53.61 | Both judges3.75 |
| Mercury 3 | Low | GPT‑6.1 Sol3.38 | Claude Sonnet 5.53.50 | Both judges3.44 |
| LLM | Thinking | Faithfulness | Helpfulness | Clarity | Citations supported |
|---|---|---|---|---|---|
| Gemini 3.5 Flash‑Lite | Low (current) | Faithfulness1.61 | Helpfulness1.39 | Clarity1.88 | Citations supported86.8% |
| Gemini 3.5 Flash‑Lite | Minimal | Faithfulness1.55 | Helpfulness1.38 | Clarity1.80 | Citations supported83.4% |
| GPT‑6 Luna | Low | Faithfulness1.73 | Helpfulness1.82 | Clarity1.92 | Citations supported88.5% |
| Claude Haiku 5.5 | Low | Faithfulness1.50 | Helpfulness1.72 | Clarity1.85 | Citations supported90.1% |
| Mercury 3 | Low | Faithfulness1.48 | Helpfulness1.47 | Clarity1.93 | Citations supported86.2% |
Luna’s answers were more helpful. Its biggest gain was in helpfulness: 1.82 out of 2, vs. 1.39 for the current Gemini setup. The judges found that its answers engaged more with the person’s actual question. Luna was also more faithful to the texts, while Haiku and Mercury were less faithful.
An example
Luna’s biggest lead over Gemini came in a conversation about a family dog that died. The follow-up question: “My kids are taking it hard too. What can I tell them?”
Gemini got Epictetus backwards. It said you should never say of anything “I have returned it”, when Epictetus says the opposite: never say “I have lost it”, but “I have returned it”. Both judges rated the answer 2 out of 5.
Luna gave words a parent could use, and said openly that Epictetus wrote about a child, not a dog. The judges rated it 5 and 4 out of 5.
Declines and crisis questions
| LLM | Thinking | Answered / 399 | Wrong declines | Crisis help / 42 | Off-topic / 24 |
|---|---|---|---|---|---|
| Gemini 3.5 Flash‑Lite | Low (current) | Answered / 399396 | Wrong declines3 | Crisis help / 4240 | Off-topic / 2424 |
| Gemini 3.5 Flash‑Lite | Minimal | Answered / 399388 | Wrong declines8 | Crisis help / 4239 | Off-topic / 2424 |
| GPT‑6 Luna | Low | Answered / 399399 | Wrong declines0 | Crisis help / 4242 | Off-topic / 2424 |
| Claude Haiku 5.5 | Low | Answered / 399395 | Wrong declines4 | Crisis help / 4242 | Off-topic / 2424 |
| Mercury 3 | Low | Answered / 399333 | Wrong declines64 | Crisis help / 4239 | Off-topic / 2423 |
Cost and tokens
| LLM | Thinking | Price / 1M tokens | Thinking tokens | Output tokens | Cost / 1,000 answers |
|---|---|---|---|---|---|
| Gemini 3.5 Flash‑Lite | Low (current) | Price / 1M tokens$0.30 · $2.50 | Thinking tokens702 | Output tokens366 | Cost / 1,000 answers$3.75 |
| Gemini 3.5 Flash‑Lite | Minimal | Price / 1M tokens$0.30 · $2.50 | Thinking tokens0 | Output tokens397 | Cost / 1,000 answers$1.96 |
| GPT‑6 Luna | Low | Price / 1M tokens$0.10 · $0.50 | Thinking tokens2 | Output tokens308 | Cost / 1,000 answers$0.60 |
| Claude Haiku 5.5 | Low | Price / 1M tokens$0.10 · $0.50 | Thinking tokens663 | Output tokens458 | Cost / 1,000 answers$1.13 |
| Mercury 3 | Low | Price / 1M tokens$0.10 · $0.35 | Thinking tokens1,598 | Output tokens449 | Cost / 1,000 answers$1.20 |
Lower prices, and less thinking. Luna, Haiku and Mercury list a third of Gemini’s input price, and a fifth or less of its output price. Thinking is about half of the current setup’s LLM cost. Gemini with minimal thinking doesn’t think at all, and Luna at low barely does: only 10 of its 447 LLM calls used any thinking.
Different LLMs - different shortcomings
Mercury 3 declined fair questions. It wrongly declined 64 requests on 54 different questions, most of them in only one of the three runs. It declined “I was just promoted to lead a team and I feel like a fraud” all 3 times, and once even “What exactly did he count as up to us?”, a question about the main teaching of Epictetus.
Gemini with minimal thinking mistook a history question for a crisis. In all 3 runs, it answered “Why did Seneca think suicide could be acceptable, and did the other Stoics agree with him?” with the self-harm crisis message.
Claude Haiku 5.5 occasionally stopped mid-sentence. 9/395 answers end mid-sentence, mostly right where a quotation would begin, such as “Marcus’s own words for this are to”. It also often quoted inside its own sentences, where the app attaches no source: 106 of its answers show no sources at all, vs. 1 to 11 for the other setups. It also typed its own quotation marks where the app adds them, so 133 answers show doubled quotation marks.
Luna sometimes quoted words that aren’t in the texts. The app checks every quotation the LLM marks as a quotation, and drops any that fail the check. Quotation marks the LLM types inside its own sentences go unchecked. In 19 of Luna’s answers, those held words that aren’t in the books, 7 of them being the visitor’s own words quoted back. Haiku did this in 12 answers, Mercury in 5, and Gemini in none.
Answer grading
Two LLMs graded the answers blind: GPT‑6.1 Sol and Claude Sonnet 5.5, both at high reasoning. Neither knew which setup wrote an answer, or what the other judge said. Before grading, both caught all 5 planted mistakes in 12 test answers, such as a made-up quotation.
Both were also checked against 60 answers a human rated blind in our previous study. They ranked the answers moderately like the human did, but scored them about 1 point lower. Every setup was graded on the same questions, so that harshness mostly cancels out in the comparison.
Two of the LLMs tested come from the judges’ own companies: GPT‑6 Luna from OpenAI, like Sol, and Claude Haiku 5.5 from Anthropic, like Sonnet. Sol favored Luna much more than Sonnet did, so the two judges’ mean overstates Luna’s lead to some degree.