What Matters When Picking AI For A News Pipeline? I Tested 71 Setups

🔍 Read the full analysis: What Matters When Picking AI For A News Pipeline? I Tested 71 Setups on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get smart everyday buys delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says a test of 71 AI model configurations found that GPT-6 Astra produced the highest-scoring news articles, while the much cheaper GPT-6 Luna was the model he chose to run in production. The October 13 report from ThorstenMeyerAI.com also describes removing two fallback models after they produced invented quotes or articles his pipeline could not use.

Meyer tested models on the same news-writing task used by his production pipeline, using real sources and his production prompt. Each configuration generated an English and a German article. The test covered API and subscription models, free-tier options, and open-weight models running on his hardware. Two AI judges scored the work on source fidelity, fabrication, clarity about uncertainty, news value, structure, polish and, for German, language quality.

In the standard test, Meyer reports that GPT-6 Astra scored 4.00 out of 5 in both languages and received a 5.00 source-fidelity score. In a further run of 10 articles, mostly based on short sources, Astra averaged 3.93 across 12 articles. Its reported price was about $0.12 per article, compared with about $0.0012 for GPT-6 Luna. Meyer said Astra took around 50 seconds per article; Luna took about 17 seconds.

Luna scored 3.67 in English and 3.68 in German in the standard results. In the robustness run it averaged 3.67, with source fidelity of 4.71 out of 5, and every article it wrote was valid under the pipeline’s output checks. Meyer reports a recent production cost of about $0.0013 per article, with a median generation time of 17 seconds. He said Astra was too expensive for his news workflow and that Luna was selected despite Astra’s higher writing score.

Why Production Cost Changed the Choice

The results illustrate a practical distinction for newsrooms and publishers using AI: the highest score is not automatically the best production choice. A model’s cost and speed matter when articles are generated at volume, but they have to be weighed against reliability and factual discipline. In Meyer’s test, Astra scored higher, while Luna’s lower reported cost and consistent valid output made it his operational choice.

The failure cases also point to a risk that a single headline model score can miss. Meyer says GPT-4.1 nano, used as a backup, scored 1.62 in English and 1.68 in German, with a source-fidelity score of 1.00. Judges flagged quotes attributed to an industry analyst and an anonymous company spokesperson that were not in the source. The local model gpt-oss-120b produced zero valid articles in four attempts, according to Meyer, and generated named quotes not supported by the material. A failover model can therefore affect published output even if the main model performs well.

How the News Pipeline Was Tested

The test ran from September 8 through October 13, 2026, and used two source texts: a longer product filing of 6,340 characters and a short political news item of 619 characters. Meyer said most of his news starts with short sources, making the shorter item a more representative test of his normal workload. The strongest configurations were also assessed in a robustness run using 10 additional real sources, mostly short ones.

Two judges, GPT-6.1 Sol with high reasoning effort and Claude Opus 5.5, rated articles from 1 to 5. Meyer says the report includes 95% bootstrap intervals and counts an article only if it produces valid output after the same repair step used in production. He set a rule that differences below about 0.3 points in the two-article standard test should be treated as noise, and said the larger robustness run was a more reliable basis for ranking.

The report also cautions against assuming that performance transfers between tasks. Meyer said Claude Opus 5.5, which he uses for software development and describes as a leading coding model on a separate benchmark, scored 3.46 in English and 3.39 in German as a news writer. Claude Sonnet 5.5 scored 3.67 and 3.61, respectively, but Meyer estimated its API-equivalent cost at about $0.036 per article, around 30 times Luna’s reported cost.

“For a high-volume task, the right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, ThorstenMeyerAI.com

Limits of the Model Comparison

The reported results are Meyer’s own benchmark, not an independent evaluation of the models across publishers, beats or languages. The standard test used two sources, and although a later run added 10 articles, its sample remains limited. The article does not provide enough information to establish how results would change with different prompts, editorial standards, source types or model versions.

The scores also depend on two AI judges, including Claude Opus 5.5, which was itself among the tested writing models. Meyer reports the judges’ criteria and the use of bootstrap intervals, but the published material does not establish whether human editors independently reviewed the full set of outputs. Actual costs and response times may also vary by provider, usage level and configuration.

Meyer says the two weak fallback models were removed from the news route on October 7. The source text ends while describing the replacement routing, so it does not fully specify the current fallback chain. It also does not give a longer-term audit of Luna’s performance in published production articles.

Production Use and Further Testing

Meyer says GPT-6 Luna remains the production choice, with reasoning effort set to “none”. He reported that the route had been changed after the October 7 review, when the two fallback models were removed. His results suggest the next practical test is whether Luna continues to meet the quality and validity bar across a larger, ongoing sample of real articles.

The report does not announce a date for another benchmark or a planned model change. Meyer’s approach points to retesting when models or prices change, while checking not only average scores but also source fidelity, fabricated material, output validity, cost and latency. Readers should treat the published figures as results from this particular pipeline and test period, rather than a general ranking for all news organizations.

Key Questions

Which model scored highest in the test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. Meyer reports an average of 3.93 across 12 articles in the robustness run.

Why did Meyer choose GPT-6 Luna for production?

Meyer said Luna’s reported cost was about $0.0012 per article, roughly 100 times below Astra’s $0.12, while its robustness-run score was 3.67 and all its articles were valid for the pipeline.

Did higher reasoning effort improve the articles?

Meyer tested six Luna effort settings and said differences were within the test’s noise. The maximum setting cost about 12 times as much and took about 12 times as long as “none,” without a reported quality gain.

What problems did the fallback models produce?

Meyer says GPT-4.1 nano produced unsupported quotes flagged by the judges. The local gpt-oss-120b model generated no valid articles in four attempts and included invented named quotes, so both were removed from the news route.

Does the test establish which AI is best for every newsroom?

No. It measured 71 configurations on Meyer’s prompt, sources, languages and production checks. The findings may not carry over to other editorial workflows, source types or model versions.

Source: ThorstenMeyerAI.com