Case study

Grading RAG answers: GPT‑6 Astra vs. GPT‑5.6 Sol

GPT‑6 Astra finished first in all 26 grading batches, averaging 4.1 minutes per 30 answers vs. GPT‑5.6 Sol’s 7.3. They agreed on 95% of verdicts, but Sol graded the answers more strictly.

Marty Paukstys Published September 25, 2026

Main results and findings

Use two AI judges, not one. Both LLMs provided different grading results. Graded by GPT‑6 Astra alone, the RAG experiment’s current setup got 6 fewer of its 240 answers right than the original setup. Graded by GPT‑5.6 Sol alone, it got 12 fewer correct answers.

A second judge also shows where the grading is borderline (in this study, 35 of 717 answers).

GPT‑6 Astra took 44% less time than GPT‑5.6 Sol to grade the same answers, even though it ran at extra-high reasoning vs. Sol’s high.

Why

This study is a byproduct of the RAG experiment done on September 23, 2026.

The RAG experiment tested four changes to the AskHeyBoss app’s RAG system, one at a time, measuring response time, LLM cost and answer accuracy. Each of its five setups, from the original to the current one, answered the same 100 questions three times. The test document was Ford’s 2026 F‑150 owner’s manual, 736 pages long.

The experiment had all of its 1,500 requests graded twice, by two AI judges working blind. GPT‑6 Astra kept returning its batches first, even though it ran at the higher reasoning setting, where a model thinks longer before it answers. The session records already held how long every batch took, so this comparison needed no new model calls.

GPT‑6 Astra vs. GPT‑5.6 Sol

4.1 min

Astra’s average per 30 answers

Sol 7.3 min, 44% less time

26 of 26

Batches Astra finished first

Its slowest full batch beat Sol’s fastest

Astra per answer, all 26 batches · Sol 14.9 s 8.4 s

Astra median batch · Sol 6.8 min 4.0 min

Verdicts the two agreed on 95%

Planted test answers each judge got right 13 of 13

Method

Two AI judges: GPT‑6 Astra at extra-high reasoning and GPT‑5.6 Sol at high, each run as a Codex agent on an existing ChatGPT subscription. Astra also ran the RAG experiment itself.
What they graded: The RAG experiment’s 1,500 requests. Merging identical answers left 717, graded in 26 batches: 23 of 30 answers, then 16, 10 and 1. Each batch went to both judges, usually about 15 seconds apart, with the same answers, grading rules, reference answers and retrieved passages. Both graded blind: neither knew which setup produced an answer or what the other judge said.
How time was measured: Each judge’s own start-to-finish time on its batch: reading, checking the manual, grading and writing up.
When: September 23, 2026, during the RAG experiment.

Grading time and results

On the same batch, Sol took 1.2 to 2.6 times as long as Astra, and 3.1 times in the one batch where it got a clarification partway through.

Table 1. Duration per batch of 30 answers

Judge	Reasoning	Average	Median	Fastest	Slowest
GPT‑6 Astra	Extra high	Average247.5 s	Median239.8 s	Fastest192.0 s	Slowest329.1 s
GPT‑5.6 Sol	High	Average440.2 s	Median408.6 s	Fastest341.7 s	Slowest648.4 s

23 full batches per judge. Astra also finished first on the last three, of 16, 10 and 1 answers. Times cover the whole job, not just the model’s reply. Sol’s slowest, 648.4 s, is the one batch where it got a clarification partway through. In the 8 full batches graded while no other batch was running, Astra still took 39% less time and finished first in all 8. Download CSV

Table 2. Correct answers out of 240

Setup	Answer model	Astra alone	Sol alone	Final
A · original setup	Gemini 3 Flash Preview	Astra alone207	Sol alone204	Final206
B1 · reranker removed	Gemini 3 Flash Preview	Astra alone210	Sol alone209	Final211
B2 · smaller chunks	Gemini 3 Flash Preview	Astra alone198	Sol alone190	Final198
B3 · 4 context chunks, not 5	Gemini 3 Flash Preview	Astra alone193	Sol alone189	Final194
C · Flash‑Lite, current setup	Gemini 3.1 Flash‑Lite	Astra alone201	Sol alone192	Final201

The RAG experiment’s five setups, each answering its 80 answerable questions 3 times. Final is the count the RAG study published, after a fresh Astra review settled every disagreement. That review sided with Astra in 30 of the 35 disagreements, so Final is not independent of Astra. Answers to the tire-pressure question are counted as regraded (see Other checks). Download CSV

The two judges agreed on the verdict for 95% of the 717 answers (Cohen’s kappa 0.89, an agreement score that allows for chance, where 1 is perfect). They disagreed on 35 answers. Sol gave the harsher grade 24 times vs. Astra’s 11. Most disagreements, 23 of 35, were only about completeness (whether an answer carried every required fact and condition).

Other checks

Astra noticed a contradiction in Ford’s manual itself. Explaining how long to drive after inflating the tires before the low-pressure light goes off, the manual says both “up to two minutes” and “at least two minutes”. Astra flagged it on all 8 answers to that question vs. Sol on 1. A fresh Sol, asked to check the manual after Astra’s flag, confirmed the conflict. Answers to that question were then regraded with both pages in view.

Sol checked Astra’s questions. Astra drafted 150 test questions and 12 for a trial run. Sol reviewed all 162, rejected none and edited 44, mostly to drop facts the reference answer required but the question never asked for. The final 100 questions were chosen from the 150.

Both passed the planted tests. Before grading began, each judge graded 13 planted test answers with known verdicts, from correct answers to deliberately wrong facts, and passed all of them. On the planted wrong facts, though, GPT‑6 Astra was the stricter of the two.