🔍 Read the full analysis: Why GPT-6 Astra Changed Qwen’s Weights To Tackle JSON Tab Escaping on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
An OpenAI GPT-6 Astra agent made a rank-one edit to one output-projection matrix in Qwen3-4B-Instruct-2507, targeting errors that copied raw tabs into JSON strings. On a frozen test, exact answers rose from 49 of 120 to 70, with 21 cases fixed and none lost. The synthetic result is narrow, table-row extraction remained at zero of 30, and no independent review has been completed.
OpenAI’s GPT-6 Astra, running as a Codex agent, changed one matrix in the open Qwen3-4B-Instruct-2507 model and raised its exact score on a narrow JSON tab-escaping test from 49 to 70 of 120 cases. The reported gain was 21 additional correct cases, with no cases that had been correct becoming wrong; the result applies to synthetic tests, not a broad evaluation of the model.
The experiment targeted a specific formatting failure: copying short text containing tabs and other special characters into a JSON string without changing the original text. JSON requires a real tab inside a string to be represented by the escape sequence \\t; the output counts as correct only if parsing the JSON reproduces the source text exactly. A valid JSON object that loses, alters or mishandles a character still fails the test.
According to the experiment account, Astra inspected the model’s weights and applied a calculated rank-one update to the output projection of the final multilayer perceptron layer. The work did not use gradient training, and the other 397 stored tensors remained bit-identical to the original. The final tests were frozen before the edited model’s result was measured, the account says.
The figures reported after an update on November 9 refer to 10 published Lexassay tasks, with runs available at lexassay.com. On the specific frozen test described here, the original model answered 49 of 120 cases exactly and the edited version answered 70. The report also says the original model managed 16 of 32 screening cases, while a more explicit prompt scored the same 16 of 32. These are separate test results and should not be treated as a general measure of Qwen’s capabilities.
Infographic 01 · The measured result
Where the edit helped, and where it did not
Qwen3-4B-Instruct-2507, original versus one edited matrix, on synthetic texts that must survive the trip into JSON without losing a character.
Final test · 120 cases by group
- Texts with tabsthe target failureOriginal19/30Edit30/30
- Line breaksalready perfectOriginal30/30Edit30/30
- Mixed special characterstabs, quotes, backslashesOriginal0/30Edit10/30
- Second table rowstill unsolvedOriginal0/30Edit0/30
Development, validation and controls
- Developmentused to fit the edit · not independentOriginal24/40Edit40/40
- Validationnew cases · 9 wins, 0 lossesOriginal10/40Edit19/40
- Random directioncontrol on validation · same strengthOriginal10/40Edit10/40
- Plain-text tabscontrol · real tabs must surviveOriginal10/10Edit10/10
Exact answer = after normal JSON parsing, the text field matches the source text character for character. Same prompt, temperature 0, seed and output limit for both models; saved and reloaded edit, no hooks, no grammar constraints, no repair. Final test, validation and controls frozen before any independent output of the edited model. Fifty of 120 final answers are still wrong.
Sources: our experiment records, 5 Nov 2026 (Lexassay research, internal): experiment 3 result report and final result by GPT-6 Astra working as a Codex agent (OpenAI). Synthetic test cases, not published.
Powered by Thorsten Meyer AI
A Narrow Fix for Structured Data
The result addresses a small but practical source of errors in structured data exchange. Software commonly passes text through JSON in APIs, logs and tool outputs. If a model inserts an unescaped tab, drops it or changes it to spaces, a downstream system may reject the data or receive text that no longer matches the source. Exact character preservation matters in workflows where formatting is part of the content.
The experiment also tests a different approach to model improvement: changing a small part of a model’s stored weights rather than relying only on a revised prompt or conventional retraining. The reported 21-case gain is evidence that a focused edit can improve performance on a known failure mode in this test. It does not show that the method will generalize to other tasks, models or real-world data. The remaining errors and unchanged table-extraction score put a clear limit on what this result establishes.
Popular right now
Shark ZU102 Rotator Pet Upright Vacuum with PowerFins HairPro & Odor Neutralizer Technology, Charcoal
Dyson V15 Detect Cordless Vacuum
Samsung 43-Inch M70H Series 4K Mini LED Smart TV (2026)
As an affiliate, we earn on qualifying purchases.
The Earlier Qwen Test
This was the third experiment in a week of work on editing model weights. In the preceding Qwen experiment, GPT-6 Astra attempted to improve extraction of five fields, such as email addresses and due dates, from synthetic business messages. The model’s two versions each scored 97 of 200 on the frozen final test. The report says both also produced 47 invalid answers, most often by placing literal tab characters in JSON strings instead of escaping them.
That earlier failure supplied the clue for the new experiment: rather than search broadly for weaknesses, Astra focused on the recurring tab-escaping error. The source account also describes a separate earlier edit by Claude to a local model called Ornith; that work is not part of this Qwen result. Claude did not participate in the third experiment, according to the account, and helped draft the article about it afterward.
The experiment follows discussion of “weight-space programming,” a research direction in which an AI system inspects and changes another neural network’s parameters. Here, the reported intervention was limited to one matrix. The small scale of the change and the fixed test set make the result interpretable as a targeted trial, but not as evidence of a general-purpose model-editing capability.
“Codex’s own experiment”
— Experiment account
How Far the Result Holds
The test cases were synthetic, and the reported improvement covers one narrowly defined task. The edited model still gave incorrect answers in 50 of 120 cases. In a separate table-row extraction test, its score remained zero of 30, according to the source account. The report does not establish whether the gains carry over to real documents, other JSON structures, different prompts or unrelated tasks.
The source says the experiment has not yet been checked by a second AI, despite an earlier project review in which GPT-6 Astra identified problems in another experiment. The article does not provide an independent replication, a detailed breakdown of the 21 corrected cases, or evidence about performance across multiple runs. It is also not clear from the supplied material how the reported result varies with different model versions or decoding settings.
Independent Checks and Broader Tests
The immediate next step is an independent review of the experiment and its records; the source account says that check has not yet happened. The runs are reported as public on lexassay.com, giving other reviewers material to examine, though the supplied information does not specify a review date or a planned replication schedule.
Further testing would need to establish whether the edit preserves its gains on new, non-synthetic examples and whether it affects other tasks. Until those checks are available, the finding is best treated as a limited result on a frozen test: a targeted weight change improved one measured error pattern, while substantial failures and questions about generalization remain.
Key Questions
What did GPT-6 Astra change in Qwen?
According to the experiment account, it applied a calculated rank-one update to the output projection matrix in Qwen3-4B-Instruct-2507’s final MLP layer. The other 397 stored tensors were reported unchanged.
What does JSON tab escaping mean?
A tab inside a JSON string must be represented with the escape sequence \\t, rather than inserted as a raw tab. The test checked whether parsing the JSON returned the exact original text.
How much did the score improve?
On the reported frozen test, exact answers increased from 49 of 120 to 70 of 120. The account says 21 cases were fixed and none were broken; the result is specific to this test.
Did the edit improve Qwen across other tasks?
The supplied results do not show broad improvement. Table-row extraction remained at 0 of 30, and the source reports 50 incorrect answers out of 120 on the final JSON-escaping test.
Has another reviewer verified the experiment?
Not yet, according to the source article. It says the experiment has not had an independent second-AI review, although the runs are reported as public on lexassay.com.
Source: ThorstenMeyerAI.com
