RAG optimization: reduced response time and LLM cost by 80%+
Average response time fell from 9.1 s to 1.6 s, and LLM cost from $4.03 to $0.46 per 1,000 questions. The difference in answer accuracy was too small to call either way.
Main results and findings
The LLM switch made the biggest impact. Generation time fell from 5.96 s to 0.92 s, and the switch accounted for 81% of the cost saving. Gemini 3 Flash Preview produced about 860 hidden reasoning tokens per question on average, compared to Flash‑Lite’s zero. The visible text was about 70 tokens for both. Each model ran on its default settings, so we can’t tell how much of the gain came from the model itself, and how much from skipping the reasoning step.
Removing the reranker saved 0.7 s per request, and the number of correct answers went from 206 to 211. It shows that the reranker added time without improving the answers.
Smaller chunks negatively affected answer accuracy, while saving little in cost. The prompt shrank by more than a third and LLM cost went down by 7%, but correct answers fell from 211 to 198. Cutting the context from five chunks to four made the system decline answerable questions more often, 16 times vs. 9.
The current setup never timed out. Every timeout in the study hit the older preview model: 27 requests failed after about 2 minutes, and 19 more were saved by a retry. The cause was not established.
The current setup was faster on every question. Its slowest response, 2.64 s, was faster than the original setup’s fastest, 2.96 s.
Why
AskHeyBoss answers staff questions from a company’s own documents. It splits them into short passages (chunks), finds the few that best match a question, and has an AI model answer from those alone: retrieval-augmented generation, or RAG. When the passages don’t support an answer, it is built to say so instead of guessing.
In early 2026, after quick speed and cost tests, we made four changes to the AskHeyBoss app’s RAG system to answer user questions faster and cheaper. The current setup reflects those changes.
Those tests timed the answers, counted how many came back, and recorded LLM cost, but none checked their accuracy. This experiment rebuilt the original setup and applied the four changes one at a time, measuring every step against the same 100 questions.
The test document is Ford’s 2026 F‑150 owner’s manual: 736 pages of procedures, conditions and model-specific detail, standing in for a large, complex company document.
Original setup (A) vs. current setup (C)
Method
- Original setup (A)
- A reranker (a second model that re-sorts the passages found), ms-marco-MiniLM-L‑6-v2; chunks of 400 words, overlap 80; 5 chunks of context; answer model Gemini 3 Flash Preview. Hybrid vector and keyword search, with gemini-embedding-001 embeddings.
- Changes tested
- Four changes, one at a time: B1 · the reranker removed; B2 · chunks cut from 400 to 180 words, overlap from 80 to 20; B3 · context cut from 5 chunks to 4; C · the answer model switched from Gemini 3 Flash Preview to Gemini 3.1 Flash‑Lite.
- Test set
- Ford’s 2026 F‑150 owner’s manual, 736 pages. 100 questions — 80 the manual answers and 20 it does not — each asked 3 times per setup: 1,500 requests.
- How it was run
- AskHeyBoss’s real query endpoint on a local Apple M4 Pro, calling the Gemini API. One request at a time, in shuffled order, on September 23, 2026. Embedding model, hybrid search settings, prompt and temperature held constant.
- How answers were judged
- Blind, by two AI models, GPT‑6 Astra and GPT‑5.6 Sol, against the manual. No human grading.
- Cost basis
- LLM cost is Gemini charges to answer a user question. Loading the manual, hosting and hardware are not counted.
Five setups, one change at a time
Each setup kept the changes from the earlier setup, and added one new to test.
| Setup | Answer model | Average | Median | Correct / 240 | Cost / 1,000 |
|---|---|---|---|---|---|
| A · original setup | Gemini 3 Flash Preview | Average9.09 s | Median5.23 s | Correct / 240206 | Cost / 1,000$4.03 |
| B1 · reranker removed | Gemini 3 Flash Preview | Average8.23 s | Median4.56 s | Correct / 240211 | Cost / 1,000$4.02 |
| B2 · smaller chunks | Gemini 3 Flash Preview | Average8.72 s | Median4.90 s | Correct / 240198 | Cost / 1,000$3.74 |
| B3 · 4 context chunks, not 5 | Gemini 3 Flash Preview | Average11.26 s | Median4.72 s | Correct / 240194 | Cost / 1,000$3.36 |
| C · Flash‑Lite, current setup | Gemini 3.1 Flash‑Lite | Average1.60 s | Median1.57 s | Correct / 240201 | Cost / 1,000$0.46 |
Answer accuracy
| Outcome | Question type | A · original | C · current |
|---|---|---|---|
| Correct | Answerable (240) | A · original206 | C · current201 |
| Partly right | Answerable | A · original28 | C · current24 |
| Declined, though the manual has it | Answerable | A · original3 | C · current15 |
| Timed out | Answerable | A · original3 | C · current0 |
| Correctly declined | Not in the manual (60) | A · original49 | C · current57 |
| Answered anyway | Not in the manual | A · original8 | C · current3 |
| Timed out | Not in the manual | A · original3 | C · current0 |
Counting both kinds of correct — a right answer, or a correct “the manual doesn’t cover this” — the original setup was right 85% of the time and the current one 86%.
The current setup declines more readily: right more often when the manual has no answer, 57 of 60 vs. 49, and wrong more often when it does, 15 declines vs. 3. When it did answer, 88.2% of its answers were fully correct, vs. 85.1%. Neither that gain nor the drop from 206 to 201 correct answers is big enough to call; both are within the noise of a 100-question test (95% interval for the accuracy change: 3.6 percentage points worse to 9.6 points better).
How the answers were graded
The questions. GPT‑6 Astra (extra-high reasoning) drafted 150 candidate questions from the manual. GPT‑5.6 Sol (high reasoning) checked every reference answer against it, and searched the whole manual to confirm the unanswerable questions are in fact unanswerable. The final 100 were locked before any setup ran.
Blind grading. Both LLMs judged every answer against its reference answer and the passages the system retrieved. Neither was told which setup produced it, how fast it was, what it cost, or what the other judge said. Both first passed 13 planted test answers, including deliberately wrong facts.
Agreement. Both judging LLMs agreed on 95% of verdicts. If they scored an answer differently, a fresh, blind GPT‑6 Astra review made the final call.
Ford’s manual contradicts itself. In its explanation of how long to drive after inflating the tires before the low-pressure light goes off, PDF page 590 says “up to two minutes”, and PDF pages 594–595 say “at least two minutes”. GPT‑6 Astra caught it, and a separate GPT‑5.6 Sol check confirmed it; answers to that question were regraded with both pages in view, which moved both setups equally.
The limits. No human checked the grading. Both judges are OpenAI models, so they may share blind spots. They were not identical either: graded by Sol alone, setup C gets 192 answers right vs. setup A’s 204. When it did answer, 84.2% of its answers were fully correct, vs. 83.6% for setup A. Read our study on how we used AI for judging, and what we learned in the process.
What we’ll test next
The same current setup with the original, larger chunks, to see whether the answers lost to smaller chunks come back, and what the bigger prompt adds to response time and LLM cost. Not yet run.