🔍 Read the full analysis: Could Claude Edit A Local Model? GPT-6 Astra Checked Our Home Experiment on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A Claude Opus 5.5 session helped edit the weights of a local AI model, Ornith 1.5 35B-A3B, to reduce looping. In the reported comparison, completed answers rose from 4 of 15 to 12 of 15, but the legal score remained zero, and the unedited model completed 13 of 15 with recommended sampling settings. OpenAI’s GPT-6 Astra reviewed the experiment and identified faults; the result is a limited home test, not evidence that the edit improves legal ability.
A home experiment reported November 5 found that a targeted edit to a local AI model helped it finish more answers, but did not improve its score on a legal benchmark. The test used code written in a Claude Opus 5.5 session to modify Ornith 1.5 35B-A3B; OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, results and claims and identified faults in the work.
In the reported comparison, the edited model completed 12 of 15 answers, up from 4 of 15 under the same setup before the edit. Its legal score remained zero. The experiment’s author said the 12 edited answers contained 62 wrong or invented citations, an average of 5.2 per answer. Across 75 checked criteria, the model met seven, compared with three before editing.
The edit targeted a direction associated with looping in the model’s internal signals. The Claude session wrote code to run the model, measure that direction and remove it at half strength from the output weights of 27 of 40 layers. The test involved no additional training, no new legal data and no AI directly reading the model weights as text. The author describes the approach as directional ablation, a known technique, rather than a reproduction of the MIT researchers’ weight-space programming method.
A comparison using the model card’s recommended sampling settings produced a different result: the unedited model completed 13 of 15 answers. That makes the edit’s practical advantage uncertain. The author also disclosed that Claude carried out the edit and Claude agents performed blind grading. GPT-6 Astra served as a reviewer from a different company, but the source describes one experiment and one independent review, not competing replications.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, 15 tasks, graded blind by three Claude agents
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Completed Answers, Same Legal Score
The experiment illustrates a distinction that matters for anyone evaluating AI systems: finishing an answer is not the same as answering correctly. Reducing repetition may make a model more usable in some tasks, but this test did not show a gain in legal knowledge or reliable citation. The unchanged zero score and reported citation errors are a warning against treating smoother output as evidence of accuracy.
It also shows why comparisons need to control for ordinary generation settings. The unedited model completed 13 of 15 answers with the recommended sampling settings, compared with 12 after the weight edit in the reported comparison. That does not establish that the edit made performance worse across settings; it does mean the reported completion gain cannot be read as a clear practical improvement over a well-configured original model.
For developers experimenting with local models, the result is a small, specific demonstration of a possible intervention and its limits. It does not establish that the method generalizes to other models, benchmarks or legal questions. Because legal answers can affect consequential decisions, the zero score and citation problems make clear that the tested model should not be treated as a source of legal advice.
Popular right now
KitchenAid 5.5-Qt Bowl-Lift Stand Mixer, Porcelain White (KSM55SXXX)
Bose QuietComfort Wireless Noise Cancelling Headphones, Black
Ninja Foodi DZ201 8-Quart DualZone 6-in-1 Air Fryer
As an affiliate, we earn on qualifying purchases.
From MIT Weight Editing to a Home Test
The experiment followed a report about work by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL. Their proposed approach, called weight-space programming, has a language model inspect another network’s weights and edit them directly, without a training loop. In a reported Qwen example, the researchers attributed repetitive output to a direction in the model’s internal signals; after editing, the model’s maths score rose from 7.9% to 48.1%, according to the source article.
The home test asked a narrower question: could reducing looping help a local model on a legal benchmark? The benchmark, Lexassay, uses 15 questions on German, US and EU law, without internet access or tools, and checks citations against primary sources. The source says five questions are hidden to limit tuning. Ornith was selected because it looped in prior runs; other tested local models either lacked the same looping problem or were unsuitable for this experiment. The source reports that Ornith had previously scored 1 out of 100 on the benchmark.
The methods were different. The MIT account describes an AI reasoning about weights directly, while the home test used a model run to record internal signals and compute a direction for removal. The source also notes prior research using single-direction edits, including work on refusal behavior. These points provide context for the technique, but do not independently verify either experiment’s results.
“The edit repairs finishing, not legal knowledge.”
— The experiment’s author
Limits of One Home Experiment
The source describes one run per setting, a small development set and a loop detector that changed during the experiment. It also says a change in the local setup alone shifted the original model from 10 completed answers to four. These limitations make it difficult to isolate the effect of the weight edit or determine how stable the result would be across repeated runs.
The grading has further limits: Claude carried out the edit, Claude agents performed the blind grading, and the source says three graders came from the same model family. GPT-6 Astra’s review found real faults, but the provided account does not detail every fault, say how each was resolved or report a separate replication. The home test also used a different method from the MIT work, so it does not independently confirm that work’s specific procedure or results.
It remains unclear whether the edit would improve completion reliably on other prompts, models or machines, or whether a revised comparison with fixed settings would show an advantage over the original model. The source does not provide enough information to establish that the edited model’s answers are legally dependable; the reported benchmark score was zero.
Replication and Benchmark Updates
The immediate next step for evaluating the claim would be repeated runs with the same model, hardware, token limits and sampling settings, comparing edited and unedited versions. Reporting the loop-detection method, grader disagreements and how GPT-6 Astra’s identified faults were handled would help readers assess the results. Tests on questions held back from development would also be needed before drawing conclusions about broader performance.
The source says Lexassay’s public site still shows only the original Ornith result, not the new runs described here. It does not announce a date for updating the benchmark or provide a scheduled follow-up experiment. Until further results are available, the reported finding is limited to this test: the edit was technically applied, more answers finished in one comparison, and the legal score stayed at zero.
Key Questions
Did Claude directly edit the model’s weights?
Not by reasoning about the weights as text. According to the source, a Claude Opus 5.5 session wrote code that ran Ornith, measured a loop-associated direction and removed it at half strength from selected weights.
Did the edit improve Ornith’s legal performance?
The reported legal score remained zero. The edited model completed more answers than the unedited model in one comparison, but the source reports 62 wrong or invented citations among the 12 completed answers.
Why does the article mention GPT-6 Astra?
OpenAI’s GPT-6 Astra, running in Codex, reviewed the experiment’s code, numbers and claims. The source says it found real faults. It was an independent review, not a second experiment or replication.
Was the edited model better than the original with recommended settings?
Not by the completion count reported: the unedited model completed 13 of 15 answers using the model card’s recommended sampling settings, while the edited model completed 12 of 15 in the fair comparison described. The source cautions that settings and experimental limits complicate comparisons.
Can the result be used as legal advice?
No. This was a limited AI experiment, and the model’s legal benchmark score was zero. The results do not establish that its answers or citations are accurate, and the source explicitly says the article is not legal advice.
Source: ThorstenMeyerAI.com
