Read the full analysis: Home Experiment: Editing AI Weights With Claude And GPT-6 Astra on ThorstenMeyerAI.com
TL;DR
A home experiment used Claude-generated code to remove a looping direction from the weights of local model Ornith 1.5 35B-A3B. On the same setup, the edited model completed all 10 published Lexassay tasks, compared with 4 before the edit, but scored zero on legal accuracy. The result is limited: it was one run per setting, and the unchanged model completed 9 of 10 tasks when run with the sampling settings recommended in its model card.
A home experiment reported November 5 found that an edit to the weights of local AI model Ornith 1.5 35B-A3B helped it finish more legal benchmark tasks, but did not improve its legal accuracy. On the same experimental setup, the edited model answered 10 of 10 published tasks, up from 4 of 10 before editing, while its legal score remained at zero.
The experiment was inspired by work from four MIT CSAIL researchers on editing a model’s weights to address a behavior such as repetitive looping. The home test used a related but different method: a Claude Opus 5.5 session wrote code that ran Ornith, measured an internal direction associated with looping, and removed that direction at half strength from the output weights of 27 of the model’s 40 layers. The researchers did not have an AI read and edit the weights directly, and they did not train the model or add legal data.
In the reported comparison using the same setup, the unchanged model completed 4 of 10 tasks; the edited version completed all 10. That improvement did not translate into reliable legal answers. The edited responses met only 7 of 50 checked criteria, up from 3, and included 55 wrong or invented citations. The legal score was zero both before and after the edit.
The report says OpenAI’s GPT-6 Astra, running in Codex, reviewed the code, figures and claims, and identified faults. It was an independent review, not a second experiment. The account also discloses that Claude carried out the edit and Claude agents handled blind grading, making the external review relevant to how readers assess the results.
Ornith 1.5 35B-A3B · local
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark promptsFind the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequencesRun the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before itWeaken it in the weights: W ← W − α·r·(rᵀW) on the MLP output matrices of all 256 experts and the shared expertBake an edited copy; the original files stay untouched and the edited weights stay internalBenchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
Unchanged modelreference0.46Full strength, α 1rejected: broke controls0.03Half strength, α 0.5chosen, runtime test0.22Half strength, baked copythe benchmarked file0.14Middle layers 12–26 onlyworse0.70One direction, all layers + attentionmuch worse0.92–0.97Conversion only, no editcontrol, quick test0.55
The cost of the edit
Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrongHalf strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese onceBefore the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
A Fix for Loops, Not Legal Knowledge
The result illustrates a distinction that matters for anyone evaluating AI systems: finishing a response is not the same as answering correctly. A model that gets stuck in repetitive reasoning may be made more usable by changing its behavior, yet still lack the knowledge or reliability needed for a specialist task. Here, the edit improved completion, but the benchmark’s citation checks continued to show serious problems.
That distinction is especially relevant in law, where a fluent answer can be misleading if its authorities are fabricated or inaccurate. The experiment’s authors say the edited answers contained 55 wrong or invented citations. The result is not evidence that weight editing improves legal capability; rather, it suggests this particular edit addressed one failure mode while leaving the central accuracy problem in place.
The comparison also shows why experimental controls matter. With the model-card-recommended sampling settings and no weight edit, Ornith completed 9 of 10 tasks on its original setup. The reported 4-to-10 comparison uses the same setup, but the separate 9-of-10 run indicates that settings alone can affect whether the model finishes. The edit’s apparent benefit should not be read as a general performance gain.
The MIT Method and Lexassay Test
The home test followed a report by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab of MIT CSAIL. Their work describes “weight-space programming”: using a general-purpose model to inspect and change another model’s weights without a conventional training loop. In a Qwen example, the researchers attributed looping to a direction in the model’s internal signals. The source account says their edit affected 0.016% of the weights and improved the model’s mathematics score from 7.9% to 48.1%.
The home experiment was not a replication of that method. Instead, it used directional ablation, a technique that records a model’s internal signals during runs, calculates a direction and projects it out of selected weights. The test targeted Ornith because the report says it showed a looping pattern in the benchmark; other local models either failed without looping or could not be tested under comparable machine and token-budget conditions.
The evaluation used Lexassay, a legal benchmark covering German, U.S. and European Union law. It asks 15 questions without internet access or tools, and checks cited statutes and court decisions against primary sources. The November 9 update says the figures in the report refer to the 10 published tasks; five additional tasks remain hidden. The runs, answers, grades and citation checks are publicly available through Lexassay, according to the source.
“The edit repairs finishing, not legal knowledge.”
— Thorsten Meyer, describing the home experiment
Limits of the One-Run Comparison
The experiment reports one run per setting, uses a small development set, and changed its loop detector during the work. Those limits make it difficult to establish how consistently the edit affects completion or whether the measured change would hold across different prompts, settings or models. The report also says three graders came from the same model family, despite the separate GPT-6 Astra review.
The setups are not fully comparable: the unchanged model completed 8 of 10 tasks in a published run but only 4 of 10 on the setup used for the same-setup comparison. It also reached 9 of 10 when run with the sampling settings recommended in its model card. The source does not establish how much of the completion difference came from the weight edit rather than other setup choices.
It remains unclear whether the method would produce similar results on other models, on the five hidden Lexassay tasks, or in repeated trials. The findings do not establish that the edited model is suitable for legal use; the reported zero score and citation errors argue against treating it as reliable legal assistance.
Hidden Tasks and Repeat Trials
The published Lexassay runs provide a record others can inspect, including answers, grades and citation checks. A stronger test would repeat the comparison across multiple runs, keep sampling and token limits consistent, and evaluate the hidden tasks without tuning the edit against them. The source does not announce a schedule for those tests.
Until broader results are available, the finding should be treated as a limited home experiment: a weight edit was associated with better completion on one setup, but did not improve the reported legal score. The article states that its experiment is not legal advice.
Key Questions
What changed in the model?
Code written in a Claude Opus 5.5 session measured a direction associated with looping and removed it at half strength from the output weights in 27 of Ornith’s 40 layers. The experiment did not train the model or add legal data.
Did the edit make the model better at law?
Not according to the reported legal score. It remained at zero, although the edited model completed more tasks and met 7 of 50 checked criteria, compared with 3 before editing.
Why did task completion rise from 4 of 10 to 10 of 10?
The report links the change to removing a direction associated with repetitive looping. But it describes one run per setting, and a separate unchanged-model run completed 9 of 10 when using the sampling settings recommended in its model card. The results do not isolate the edit’s effect across repeated trials.
Was this the same method used by the MIT researchers?
No. The home test used directional ablation: it ran the model, measured internal signals and altered selected weights. The MIT work described in the source had GPT-6 Astra reason about another model’s weights directly.
Can the edited model be relied on for legal advice?
The report does not support that conclusion. The legal score stayed at zero, and the edited answers included 55 wrong or invented citations. The source explicitly says the experiment is not legal advice.
Source: ThorstenMeyerAI.com