I Put My News Pipeline Through 71 AI Model Configurations

🔍 Read the full analysis: I Put My News Pipeline Through 71 AI Model Configurations on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get hardware and tech essentials delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he tested 71 AI model configurations on real news-writing tasks from September 8 to October 13, finding that GPT-6 Astra scored highest while GPT-6 Luna was the model he chose for production because it cost far less. His report also says two backup models were removed from his news pipeline after testing exposed fabricated quotations or output the system could not use.

The test used Meyer’s production prompt and two real news sources: a 6,340-character product filing and a 619-character political news item. Each configuration produced an English and a German article. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, scored the work on source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with German-language quality included in the German edition.

GPT-6 Astra scored 4.00 out of 5 in both languages in the standard test and achieved 3.93 across 12 articles in a follow-up robustness run. Meyer reported perfect source-fidelity scores in the standard test. It cost about $0.12 per article and took roughly 50 seconds. GPT-6 Luna scored 3.67 in English and 3.68 in German in the standard test; in the robustness run it scored 3.67, with source fidelity of 4.71. Meyer reported a production cost of about $0.0013 per article and a median completion time of 17 seconds.

The report says the pipeline’s former backup models performed poorly. GPT-4.1 nano scored 1.62 in English and 1.68 in German, with a source-fidelity score of 1.00; the judges flagged quotations Meyer said were invented. A locally run gpt-oss-120b produced zero valid articles in four attempts, according to the report. Meyer says both models were removed from the news route on October 7.

Cost and Reliability in News Writing

The results highlight a practical trade-off for publishers using AI to produce high volumes of routine articles: the model with the best test score may not be the one that makes sense to run on every story. Meyer reports that Astra scored higher, but cost about 100 times as much per article as Luna. That gap makes the production decision depend not just on writing quality, but on whether the additional quality is worth the extra expense for a given workflow.

The reported fallback failures point to a separate operational risk. A pipeline may rely on secondary models when its preferred service is unavailable, but Meyer says the backups in this test either introduced unsupported quotations or failed the production format. For newsrooms, those outcomes underline why testing the entire route—including fallback behavior—matters. The findings are one publisher’s benchmark, not evidence that the same models will perform identically on other prompts, sources or editorial standards.

How Meyer Tested the Models

Meyer describes the work as a test of his own production task rather than a general ranking of model intelligence. The source set contained one long document and one short item; he says the short source better reflected much of the material his pipeline handles. Each model wrote for both an English-language edition and a German-language edition aimed at readers in Germany, Austria and Switzerland.

The report says an article counted only if it produced valid output after the same repair step used in production. Meyer also cautioned that the initial test contained only two source articles and treated score differences under about 0.3 points as noise. He presented the follow-up run, which tested stronger configurations on 10 additional real sources, mostly short ones, as a more reliable basis for comparing models. The reported results include 95% bootstrap intervals.

In a separate comparison, Meyer found that additional reasoning effort did not produce a dependable quality gain for GPT-6 Luna. The highest effort setting cost about 12 times as much and took about 12 times as long as the no-effort setting, while the score differences fell within the report’s uncertainty. He says the production pipeline therefore keeps the no-effort setting.

“Public leaderboards tell you which model is smartest. They don’t tell you which model should write your articles.”

— Thorsten Meyer, in his October 13 report

Limits of the Reported Benchmark

The results come from one publisher’s pipeline, prompt and source set, and do not establish how these models would perform in other newsrooms or on a broader range of reporting tasks. The standard test used only two sources, and Meyer himself says small score differences should be treated as noise. The robustness run covered more stories but was still limited to 10 additional sources.

The report relies on two AI judges, one of which, Claude Opus 5.5, was also among the evaluated models. The published material does not independently verify the scores, cost estimates or judges’ assessments. It also does not provide enough detail here to establish how often the reported failures would occur across a larger volume of production. The source excerpt ends while describing the revised fallback route, so the complete current failover configuration is not clear.

The Pipeline After October 7

Meyer says GPT-6 Luna is now the production model and that the two underperforming backups were removed on October 7. The report does not specify a future date for another benchmark or identify a replacement fallback in the material provided. The next useful evidence would be performance over a larger set of live articles, including whether Luna maintains its reported quality and whether the revised route can handle service interruptions without introducing unsupported details.

Key Questions

Which model scored highest in the test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It scored 3.93 across 12 articles in the follow-up robustness run, according to Meyer.

Why did Meyer choose GPT-6 Luna for production?

Meyer says Luna scored 3.67 in the robustness run and produced valid articles in the test, while costing about $0.0013 per article in production. He reported Astra at about $0.12 per article, roughly 100 times as much.

How broad was the test?

The test covered 71 model configurations, two source articles, and English and German editions. A follow-up robustness run tested stronger configurations on 10 more real sources. Meyer says the small initial source set limits how confidently close scores can be compared.

What happened with the fallback models?

Meyer says GPT-4.1 nano generated quotations the judges identified as invented, while locally run gpt-oss-120b produced no valid articles in four attempts. He reports removing both models from the news route on October 7.

Did more reasoning effort improve the articles?

In Meyer’s test of six GPT-6 Luna effort settings, he found no reliable quality gain. He reported that the maximum setting cost and took about 12 times as much as the no-effort setting, so the production pipeline remained on no effort.

Source: ThorstenMeyerAI.com