RAG optimization: reduced response time and LLM cost by 80%+
Average response time fell from 9.1 s to 1.6 s, and LLM cost from $4.03 to $0.46 per 1,000 questions. The difference in answer accuracy was too small to call either way.
Main results and findings
The model switch made the biggest impact. Gemini 3 Flash Preview produced about 850 hidden reasoning tokens per answer on average, alongside a reply of about 70, compared to Flash-Lite’s zero. Generation time fell from 5.96 s to 0.92 s, and Flash-Lite’s tokens cost half as much. The switch accounts for about 81% of the cost saving. Each model ran on its default settings, so we can’t tell how much of the gain came from the model itself and how much from skipping the reasoning step.
Removing the reranker saved 0.7 s per request at no measurable cost to answers: correct answers went from 206 to 211.
Smaller chunks cost answers and saved little. The prompt shrank by more than a third and LLM cost by 7%, yet correct answers fell from 211 to 198. Cutting the context from five chunks to four then made the system decline answerable questions more often, 16 times against 9.
The current setup never timed out. Every timeout in the study hit the older preview model: 27 requests failed after about 2 minutes, and 19 more were saved by a retry. The cause was not established. The current setup’s slowest response, 2.64 s, was faster than the original setup’s fastest, 2.96 s.
Why
AskHeyBoss answers staff questions from a company’s own documents. It splits them into short passages (chunks), finds the few that best match a question, and has an AI model answer from those alone: retrieval-augmented generation, or RAG. When the passages don’t support an answer, it is built to say so instead of guessing.
In early 2026, after quick speed and cost tests, we made four changes to the AskHeyBoss app’s RAG system to answer user questions faster and cheaper. The current setup reflects those changes.
The original tests timed the answers, counted how many came back, and recorded LLM cost, but none checked their accuracy. This experiment rebuilt the original setup and applied the four changes one at a time, measuring every step against the same 100 questions.
The test document is Ford’s 2026 F‑150 owner’s manual: 736 pages of procedures, conditions and model-specific detail, standing in for a large, complex company document.
Original setup (A) vs. current setup (C)
Method
- Original setup (A)
- A reranker (a second model that re-sorts the passages found), ms-marco-MiniLM-L‑6-v2; chunks of 400 words, overlap 80; 5 chunks of context; answer model Gemini 3 Flash Preview. Hybrid vector and keyword search, with gemini-embedding-001 embeddings.
- Changes tested
- Four changes, one at a time: B1 · the reranker removed; B2 · chunks cut from 400 to 180 words, overlap from 80 to 20; B3 · context cut from 5 chunks to 4; C · the answer model switched from Gemini 3 Flash Preview to Gemini 3.1 Flash-Lite.
- Test set
- Ford’s 2026 F‑150 owner’s manual, 736 pages. 100 questions — 80 the manual answers and 20 it does not — each asked 3 times per setup: 1,500 requests.
- How it was run
- AskHeyBoss’s real query endpoint on a local Apple M4 Pro, calling the Gemini API. One request at a time, in shuffled order, on September 23, 2026. Embedding model, hybrid search settings, prompt and temperature held constant.
- How answers were judged
- Blind, by two AI models, GPT‑6 Astra and GPT‑5.6 Sol, against the manual. No human grading.
- Cost basis
- LLM cost is Gemini charges to answer a user question. Loading the manual, hosting and hardware are not counted.
Five setups, one change at a time
Each setup kept the changes from the earlier setup, and added one new to test.
| Setup | Answer model | Average | Median | Correct / 240 | Cost / 1,000 |
|---|---|---|---|---|---|
| A · original setup | Gemini 3 Flash Preview | Average9.09 s | Median5.23 s | Correct / 240206 | Cost / 1,000$4.03 |
| B1 · reranker removed | Gemini 3 Flash Preview | Average8.23 s | Median4.56 s | Correct / 240211 | Cost / 1,000$4.02 |
| B2 · smaller chunks | Gemini 3 Flash Preview | Average8.72 s | Median4.90 s | Correct / 240198 | Cost / 1,000$3.74 |
| B3 · 4 context chunks, not 5 | Gemini 3 Flash Preview | Average11.26 s | Median4.72 s | Correct / 240194 | Cost / 1,000$3.36 |
| C · Flash-Lite, current setup | Gemini 3.1 Flash-Lite | Average1.60 s | Median1.57 s | Correct / 240201 | Cost / 1,000$0.46 |
Answer accuracy
| Outcome | Question type | A · original | C · current |
|---|---|---|---|
| Correct | Answerable (240) | A · original206 | C · current201 |
| Partly right | Answerable | A · original28 | C · current24 |
| Declined, though the manual has it | Answerable | A · original3 | C · current15 |
| Timed out | Answerable | A · original3 | C · current0 |
| Correctly declined | Not in the manual (60) | A · original49 | C · current57 |
| Answered anyway | Not in the manual | A · original8 | C · current3 |
| Timed out | Not in the manual | A · original3 | C · current0 |
Counting both kinds of correct — a right answer, or a correct “the manual doesn’t cover this” — the original setup was right 85% of the time and the current one 86%.
The current setup declines more readily: right more often when the manual has no answer, 57 of 60 against 49, and wrong more often when it does, 15 declines against 3. When it did answer, 88.2% of its answers were fully correct, against 85.1%. Neither that gain nor the drop from 206 to 201 correct answers is big enough to call; both are within the noise of a 100-question test (95% interval for the accuracy change: 3.6 points worse to 9.6 points better).
How the answers were graded
The questions. GPT‑6 Astra (extra-high reasoning) drafted 150 candidate questions from the manual. GPT‑5.6 Sol (high reasoning) checked every reference answer against it, and searched the whole manual to confirm the unanswerable questions really are. The final 100 were frozen before any setup ran.
Blind grading. Both judged every answer against its reference answer and the passages the system retrieved. Neither was told which setup produced it, how fast it was, what it cost or what the other judge said. Both first passed 13 planted test answers, including deliberately wrong facts.
Agreement. They agreed on 95% of verdicts. Wherever they scored an answer differently, a fresh, blind Astra review made the final call.
Ford’s manual contradicts itself. On how long to drive after inflating the tires before the low-pressure light goes off, PDF page 590 says “up to two minutes” and PDF pages 594–595 say “at least two minutes”. Astra caught it and a separate Sol check confirmed it; answers to that question were regraded with both pages in view, which moved both setups equally.
The limits. No human checked the grading, and both judges are OpenAI models, so they may share blind spots. They were not identical either: graded by Sol alone, C gets 192 answers right to A’s 204, and 84.2% of its answers against A’s 83.6%. Read our study on how we used AI for judging, and what we learned in the process.
What we’ll test next
The same current setup with the original, larger chunks, to see whether the answers lost to smaller chunks come back, and what the bigger prompt adds to response time and LLM cost. Not yet run.