🔍 Read the full analysis: What GPT-6 Astra Found When It Checked Claude’s Model Changes on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
GPT-6 Astra reviewed a Claude Opus 5.5 experiment that edited the weights of the local model Ornith 1.5, finding reproducible problems in the test rig and weight conversion. After fixes, all 12 of the author’s tests passed, but four of six provenance checks in the second review still failed; the benchmark’s legal score remained zero.
OpenAI’s GPT-6 Astra, running in Codex, reviewed a Claude Opus 5.5 experiment that edited the weights of a local model and found faults that could have made some results misleading. The review did not run a competing model-improvement test; it checked the experiment’s code and measurements, and its findings matter because the headline benchmark result—10 tasks answered rather than four—came with a legal score of zero.
The experiment, led by a Claude Opus 5.5 session, attempted to improve the local model Ornith 1.5 for legal tasks. It used a directional ablation inspired by a post from MIT’s Computer Science and Artificial Intelligence Laboratory, weakening a model direction identified as a “loop direction” without additional training. The source reports that on 10 published Lexassay tasks, the edited model answered all 10, compared with four for the unedited model. Its legal score remained zero.
GPT-6 Astra was assigned to review the implementation, not change it. It ran separate checks and reproducible examples while Claude retained responsibility for the experiment, benchmark runs and blind grading by three Claude agents. The reviewer said it did not inspect the hidden benchmark prompts, topics, answers or reference solutions. The runs and related controls are published on Lexassay, according to the source article.
The first review identified four faults: a resumed run could reuse results from a different model or sampling setup; an unknown task ID could produce a successful run with no evaluations; a comma conflict broke the real command line despite passing parser unit tests; and converting weights changed values even when the edit strength was zero. The Claude session changed the implementation in response, including rejecting unknown IDs, changing the sampling-setting syntax and storing edited tensors in higher-precision bf16 rather than re-quantising them to the compact 6-bit format.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Independent Checks Changed the Results
The review shows how passing unit tests can miss failures in the actual command used to run an experiment. In this case, the review’s examples exposed paths that could leave old answers in place, mark an empty evaluation as successful or prevent a planned control from running. Those issues could affect whether a benchmark is reproducible and whether its results can be attributed to the intended model change.
It also challenged what the measurements could support. The source says the reviewer’s second round found that all 12 author tests passed, while four of six reviewer provenance checks failed. A separate conversion measurement found that representing one real weight matrix in a different number format moved it by 0.51% and changed 915,360 of 1,048,576 values. That is a measurement of the conversion, not a measure of legal performance or a general error rate.
The corrected interpretation is narrower than a claim that the model became better at law. The model answered more benchmark tasks, but its legal score stayed at zero. The review also prompted corrections to descriptions of exact identity at zero edit strength, the average number of wrong citations per answer, the scope of the parallel Codex work and how strongly one control supported a causal claim.
Products Worth Considering
Trezor Safe 7 Crypto Hardware Wallet with Bluetooth and Color Touchscreen
Ledger Flex Crypto Wallet with 2.8-Inch E Ink Touchscreen
Ledger Flex + Folio Crypto Hardware Wallet
As an affiliate, we earn on qualifying purchases.
How Claude and GPT-6 Astra Split the Work
The two AI sessions began work on the experiment on November 4. After the participants realized they were working on the same project, they split the roles. Claude kept the plan, test rig, model edit, benchmark runs and grading. GPT-6 Astra was asked to “review and validate the other session’s implementation,” according to its report.
The separation was intended to make the review independent of the main implementation. GPT-6 Astra wrote its own check programs, while Claude made the fixes. The reviewer was instructed not to alter the main test rig, not to rely on green tests alone and not to inspect the hidden tasks. The source says early preparatory files from the GPT-6 Astra session were set aside after the roles were decided, and no second edited model resulted from them.
The project’s updated account, dated November 9, 2026, says its figures now refer to the 10 published Lexassay tasks and that the runs are publicly available there. Its author describes the work as an AI experiment and review, not legal advice.
Limits of the Review and Benchmark
The published account describes a second review round in which all 12 author tests passed but four of six reviewer provenance checks failed. It does not provide enough detail in the supplied material to resolve each of those four checks or establish what each failure means for the final benchmark results. The source also says the reviewer did not rerun every test during its final documentation step.
The review was not a fourth grader, and its conversion measurement was made on a CPU rather than the production GPU path. The account does not establish whether conversion behavior would match on that GPU path. Nor does a 10-task published benchmark, with a legal score of zero, establish broader legal capability. The source presents the change in task-answer count, but the benchmark result should not be read as evidence that the model achieved legal competence.
Published Runs and Remaining Checks
The source says the experiment’s weight-edited run, edit control and model-card sampling are public on Lexassay. Readers can compare the published materials, but the account does not name a further review date, an expanded benchmark or a planned production-GPU validation. Those steps should not be assumed to be scheduled.
The immediate open issue is how the remaining provenance-check failures relate to the corrected implementation and whether additional independent runs reproduce the published results. Until more evidence is available, the reported 10-task outcome should be treated as a result from this specific experiment, alongside the stated limitations—not as a general finding about legal-model performance.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and measurements for a Claude Opus 5.5-led experiment that edited the weights of Ornith 1.5. It did not run a competing model-improvement experiment.
What did the experiment report on the 10 tasks?
The source says the edited model answered 10 of 10 published tasks, compared with four for the unedited model. It also says the legal score remained zero.
What faults did the review find?
It found that resumed runs could reuse old results, unknown task IDs could yield a successful empty run, comma-separated settings broke the real command line despite passing unit tests, and weight conversion changed values at zero edit strength.
Does the result show that the model became good at law?
No. The reported legal score was zero. The increase in answered tasks does not by itself establish legal capability or broader performance.
Were all of the reviewer’s concerns resolved?
The source reports that all 12 author tests passed in the second review round, while four of six reviewer provenance checks failed. It also says the reviewer did not rerun every test in its final documentation step, so the supplied account does not establish that every concern was resolved.
Source: ThorstenMeyerAI.com
