Who Published This Headline? Editorial Fingerprinting in French News Under Topic Control

Amr Sobhy

Workshop on Web as Corpus (WaC-13) @ EMNLP 2026·Peer-reviewed

Who published this headline?

A model reads one French news headline — topic held fixed, every name masked, from a year it was never trained on — and names the publisher at nearly six times chance. Most of what it is reading turns out to be typography.

By Amr Sobhy

Poster presentation

Identifying the outlet is the easy part. The harder question is what that signal is made of, and whether any of it is framing. Balancing categories is not enough: an outlet’s headlines differ because of how it writes and because of what it chooses to cover, and category control cannot separate the two. Holding the exact reported event constant can. When it does, three of four pre-specified differences fall to near zero.

Year: 2026
Venue: WaC-13 @ EMNLP 2026; Poster presentation
Research area: Computational journalism, Stylometry, French news media, NLP

Read the paper · on publicationData and code · planned release

170,160

headlines, balanced across ten outlets and three categories

5.8×

chance, on an entity-masked headline from an unseen year

44%

of the character model’s margin is presentation: casing, punctuation, typography

1 of 4

pre-specified framing contrasts survive matched-event control

Two questions, not one

Whether a model can tell which outlet wrote a headline is one question. Whether the difference it finds is framing is another, and the second does not follow from the first.

An outlet’s headlines differ from its rivals’ for two reasons at once: how it writes, and what it chooses to cover. Balancing the corpus by category removes the crudest version of the second — a business paper can no longer be identified by the sheer volume of its finance coverage — but an agenda survives category control. Within Politique, outlets still cover different stories on different days.

So the study climbs a ladder of increasingly strict control: balance the categories, mask the named entities, strip the surface form, and finally hold the exact reported event constant by comparing only headlines about the same story. Each rung removes one explanation. What is left at the top is the part that can be read as framing.

INT

POL

ÉCO

Le Monde

Le Figaro

Les Échos

Libération

La Croix

Le Parisien

Franceinfo

BFMTV

TF1 INFO

Ouest-France

Thirty cells, 5,672 headlines each. Identical by construction — that is the point.

01 / 08

Ten outlets, three hard-news categories, every cell the same size.

Outlet and topic are entangled: a paper that covers more finance looks distinctive when a model is really just reading the word « Bourse ». Balancing removes the crude version of that. Every outlet×category cell holds 5,672 headlines — 170,160 in all, drawn from an existing collection of the outlets’ online editions between 2021 and 2025.

Ten titles, anchored on ACPM’s daily rankings: six national dailies, three broadcast newsrooms, and Ouest-France for regional reach.

INT

POL

ÉCO

Le Monde

Le Figaro

Les Échos

Libération

La Croix

Le Parisien

Franceinfo

BFMTV

TF1 INFO

Ouest-France

Thirty cells, 5,672 headlines each. Identical by construction — that is the point.

02 / 08

A ten-way guess is right one time in ten.

Chance accuracy is 0.100. Always predicting the most frequent outlet scores 0.094 on the temporal split.

Every number that follows is measured against that floor. Models train on 2021–2024 and are tested on 2025 — a year none of them has seen.

0.100

chance accuracy

0.094 · majority class, temporal split

Ten-way task. Models train on 2021–2024 and are tested on 2025.

03 / 08

Three frontier models, handed the ten outlet names, land below a bag of words.

Prompted zero-shot with the outlet list and a raw 2025 headline, three open-weight LLMs recover the publisher at 0.19 to 0.21 — roughly twice chance, and under a word-level logistic regression trained on the corpus at 0.326.

Trained models climb from there, and the largest single gain is the step from words to characters. A fine-tuned CamemBERT reaches 0.576 on entity-masked text.

Zero-shot LLMs

0.19–0.21

Word-LR

0.326

Char-SVM

0.496

XLM-R

0.561

CamemBERT

0.576

chance 0.100

Entity-masked, temporal split. The LLM rung is a range across three open-weight models, not a point estimate.

04 / 08

It is not reading the masthead.

The obvious explanation is self-reference: outlets name themselves, and the model learns the name. Masking every outlet self-mention should then cost accuracy.

It does not. CamemBERT scores 0.615 on raw text and 0.620 with self-references masked. If anything the masked condition is marginally ahead, and the direction is not stable across seeds.

0.615

raw

0.620

self-references masked

Raw text, not entity-masked — so both sit above the 0.576 headline figure. Masking every self-mention does not cost accuracy.

05 / 08

Lowercase it, delete the punctuation, and two-fifths of the signal goes with it.

The character model is the one that comes apart in stages. On raw text it reaches 0.496. Lowercasing and deleting every punctuation mark, digit and space collapses it to 0.324.

Masking every named entity on top of that lowers it to 0.270. A word-level model on the same stripped text tracks the character model to within 0.002 at every stage, so what survives is genuinely lexical rather than a sub-word artifact.

chance

0.496

raw

0.324

no case, punctuation, digits

0.270

… and every entity masked

Char-SVM, temporal split. Stages are cumulative and applied in this order.

06 / 08

Forty-four percent presentation, fourteen percent names, forty-three percent word choice.

Splitting the character model’s above-chance margin three ways: roughly 44% is presentation — typography, casing, numerals; about 14% is which entities get named; about 43% is a lexical residual that survives removing both.

The shares are ablation-defined and order-dependent, not an additive causal partition, and they are specific to the interpretable character model. The transformer cannot be stripped in stages, though its entity component loses a comparable share.

44%

presentation — typography, casing, numerals

14%

named entities

43%

lexical residual surviving both

Share of the char-SVM’s above-chance margin; shares are rounded and do not sum to 100. Ablation-defined and order-dependent.

07 / 08

The residual is not tone, and it is not only beat.

An interpretable tone family — sentiment, stance, hedging — recovers the outlet at 0.13, barely above chance, and adds almost nothing to twenty surface features. Whatever the residual is, it is not reducible to tone at the single-headline level.

Beat vocabulary is the better candidate: « évêques » for La Croix, « cryptos » for Les Échos, « commune » for Ouest-France. Stripping the top outlet-distinctive content words closes under 40% of the residual’s margin and leaves a floor at 2.34× chance. The curve flattens rather than decaying.

The vertical axis does not start at zero — the whole decay sits inside a 0.09 band. Tone features alone reach 0.13.

08 / 08

Same event, same window: three of four differences fall to near zero.

Category control cannot separate how an outlet writes from what it covers. Holding the exact reported event constant can. Clustering near-duplicate, same-window headlines across outlets yields 626 events spanning 3,006 headlines and all ten titles.

Within those clusters both pre-specified attribution contrasts collapse, and so does valence. One contrast survives correction: BFMTV makes institutional and corporate actors the grammatical subject more often than the panel average.

626 events, 3,006 headlines, all ten outlets. The interval is shown only for the confirmed contrast, where a standard error is published.

What counts as the same event

The whole deflation rests on this clustering step, so it carries its own validation.

Events are recovered as cross-outlet clusters of near-duplicate, same-window headlines, at a cosine cutoff of 0.70 and a minimum of four distinct outlets per cluster. The cutoff is not a claim that this similarity equals same-event identity; it is a threshold chosen to keep precision high at a known cost in recall.

On 400 sampled pairs the rule finds 184 positives at a precision of 0.944, with a sample recall of 0.370. A second rater agreed on same-event identity at κ = 0.82 with 92% raw agreement, and a 30-cluster audit checked purity directly.

Recall over the full 478,110-title pool is not estimable without exhaustive same-event gold, and none exists. The clusters are therefore a high-precision sample of the events these outlets covered together, not a census of them.

The one difference that survives

Within the same event, BFMTV assigns an agentive subject more often than the average outlet.

The estimate is β = +0.056 with a one-sided p of 0.002 and a Bonferroni-corrected p of 0.008. It holds after removing syndicated near-duplicates, under fixed-model leave-one-outlet re-scoring, and across a 626-event jackknife.

The excess is concentrated in institutional and corporate agents — ministries, companies, states. Restricting the subject to an animate human referent leaves too few within-event instances to resolve the contrast, so the result is institutional-actor framing and not a claim about how individuals are described.

The agency extractor agrees with human coders at κ = 0.63, with precision 0.89 and recall of about 0.66 against the pooled 240-headline gold set. Under nondifferential under-detection that would attenuate the estimate rather than inflate it, but differential extraction error remains a limitation. The observed effect is slightly below the nominal 0.060 minimum detectable effect at 80% power, with attained power of about 0.73 at the Bonferroni threshold.

How to misread this

Five sentences this study does not support.

The model detects media bias.

It detects house style, and most of that is typography.

44% of the character model’s margin is casing, punctuation and numerals. Nothing in the attribution task measures accuracy, slant or good faith.

BFMTV is biased toward institutions.

In headlines about the same event, BFMTV makes institutional and corporate actors the grammatical subject more often than the panel average.

It is a measured difference in sentence construction against nine other outlets. It concerns ministries, companies and states, not individuals, and it says nothing about intent.

Outlets do not differ in how they frame stories.

Three of four pre-specified contrasts did not survive event control at this sample size.

626 events power the design for moderate within-event effects. A null bounds an effect; it does not establish its absence.

This is everything these ten outlets published.

It is the articles available in an existing collection of their online editions.

No exhaustive crawl is claimed. L’Humanité was excluded for a collection shortfall against the per-category floor, not an editorial judgement.

Six times chance means a person could do the same.

A model trained on this corpus does, on a balanced ten-way task.

The number is not a claim about human readers, and it does not transfer to an unbalanced real-world mix of headlines.

This is a headline study. It does not read article bodies, it does not know whether a headline matches the piece beneath it, and it cannot see the stories an outlet chose not to cover.

What is released

Article identifiers, content hashes, extraction and sampling code, derived features, per-headline category labels, the framing codebook, gold annotations, and the corpus-rebuild manifest.

Raw headline text is not released, for copyright reasons. The manifest supports rebuilding the balanced benchmark from public URLs and checking each record against the original hashes.

Released result tables are verifiable, but the 478,110-title full-pool input for the matched-event analysis is not shipped, so that analysis cannot be independently reconstructed from the package alone.

Questions

Is this a media bias score?

No. It measures whether a publisher is identifiable from the form of a headline, and what that identifiability is made of. Nothing in it tracks accuracy, and nothing tracks a left–right position. An outlet can be highly identifiable and entirely accurate.

Why headlines rather than whole articles?

The headline is the unit most readers actually see, and it is the hardest case: if a publisher fingerprint survives in fifteen words, it is unlikely to be an artifact of document length. It is also the narrower claim — nothing here describes the articles beneath.

Why these ten outlets?

The selection criterion is anchored on ACPM, France’s press-measurement body: a generalist national title with continuous 2021–2025 hard-news coverage and enough volume per category. Six national dailies, three broadcast newsrooms and one regional daily meet it. L’Humanité was excluded for a collection shortfall, not an editorial judgement.

Does the model still work on newer text?

The reported numbers are already out-of-year: training runs on 2021–2024 and testing on 2025, a year the models never saw. Drift beyond that window is untested.

Can the corpus be rebuilt?

Identifiers and content hashes are released so each record can be re-fetched and checked. Raw text is not, for copyright reasons, and the full article pool behind the matched-event analysis is not shipped.

Citation

@inproceedings{sobhy2026editorial,
  title     = {Who Published This Headline? Editorial Fingerprinting in French
               News Under Topic Control},
  author    = {Sobhy, Amr},
  booktitle = {Proceedings of the 13th Web as Corpus Workshop (WaC-13) at EMNLP 2026},
  year      = {2026},
  note      = {Accepted as a poster. To appear.},
}

To appear in the proceedings of WaC-13 at EMNLP 2026.

This work was supported by

Contact

For research questions, collaborations, or media inquiries.

Amr Sobhy

Keep reading

More from the lab

See all publications →