Language
A cross-publisher French news classification benchmark from Le French News Lab
By Amr Sobhy
FrenchNews-7 is a cross-publisher benchmark for French news editorial desk classification. It introduces a harmonized seven-class taxonomy derived from publisher practice, an 87,637-article corpus spanning 13 publishers, a manifest-only dataset release, and a released CamemBERT model designed for transfer across outlets rather than single-site classification alone.
87,637
labeled articles
13
French news publishers
7
shared desk categories
72.3%
labels read directly from publisher URL slugs
1
public model on Hugging Face
1
manifest-only dataset on Hugging Face
87,637 French news articles collected from 13 publishers. Each color represents one of seven editorial desk categories — Politique, Société, International, Sport, Économie, Culture & Loisirs, and Sciences & Technologies.
The goal: a benchmark large and diverse enough to train and evaluate a desk classifier that works across different newsrooms, not just one.
Every article in the corpus is assigned to exactly one of seven editorial desk categories. Each block below represents one category, sized in proportion to the number of articles it contains. Société, International, and Culture & Loisirs together make up nearly two-thirds of the corpus, while Sciences & Technologies is the smallest at under 5%.
The taxonomy was built through cross-publisher analysis rather than copying any single outlet's own section structure, which makes the benchmark usable across different newsrooms.
72.2% of articles were labeled by reading the category directly from the article's URL — French news URLs typically contain a path segment like /sport/ or /politique/. This required no model, just deterministic rules applied to publisher URL patterns.
The remaining 27.8% had ambiguous or missing URL signals and were labeled by an LLM. A quality study on 300 of these articles used two LLMs, an adjudicator, and a blinded human annotator with no knowledge of the URL taxonomy; the blinded human independently matched 84% of the assigned labels (Cohen's κ = 0.806), confirming the labels carry reliable signal.
URL-slug labels: deterministic, no model. LLM labels: quality-audited on 300 articles; blinded-human label agreement κ = 0.806.
Four models were tested on the full article text (headline + body). CamemBERT-base achieves a macro-F1 of 0.847 and accuracy of 0.860 — the best result in the full-text setting, and the model we have released publicly on Hugging Face.
On headline-only input, CamemBERTav2-base leads narrowly (0.797 vs. 0.794). In both settings, French-specific pretrained models outperform mBERT, the multilingual baseline.
The real test of any classifier is whether it works on publishers it was never trained on. We held out 2,100 articles from four publishers excluded from training and measured how well the model's categories held up.
Six of the seven categories achieve a recall of 0.81 or above on these unseen outlets — solid generalization for a model that has never seen those newsrooms.
Économie is the exception — what counts as business content varies more across publishers than any other category.
Can a large general-purpose model skip the training data entirely and still classify French news? We tested GPT-OSS-120B in zero-shot mode — given only a category list and the article text, no examples, no fine-tuning.
Our fine-tuned CamemBERT-base outperforms it by +4.6 percentage points in macro-recall: 0.799 vs. 0.753. A 110M-parameter model trained on domain-specific labeled data beats a 120 billion-parameter model given nothing but instructions — fine-tuning on the right data still wins.
Fine-tuned 110M-param model beats 120B zero-shot by +4.6 pp in macro-recall (0.799 vs. 0.753).
FrenchNews-7 draws from 13 outlets spanning national dailies, regional papers, news magazines, and digital-native sites — a deliberate mix of editorial traditions, audience profiles, and writing styles.
That diversity is the point. A model that only works on one outlet is not a benchmark — it is an in-house tool. FrenchNews-7 is designed to measure whether desk classification transfers across the French press as a whole.
News organizations do not always classify stories in the same way. One publisher may place a story under politics, while another may treat a similar story as society or economy. That makes it difficult to compare editorial coverage across outlets or build reliable French-language classification tools.
FrenchNews-7 was created to make that comparison possible. It provides a shared seven-category desk framework across 13 French publishers and a benchmark explicitly designed to test cross-publisher transfer rather than only in-domain performance.
FrenchNews-7 harmonizes publisher-specific editorial sections into a shared 7-label taxonomy.
Domestic social affairs, human interest, health, education, crime, and civil society.
Arts, books, cinema, music, television, leisure, and lifestyle coverage.
Foreign affairs, geopolitics, and major events outside France.
French politics, elections, institutions, government, and legislative affairs.
Sports reporting across disciplines and competitions.
Business, finance, markets, macroeconomics, and corporate news.
Scientific research, technology, digital innovation, and environment-science topics.
FrenchNews-7 starts from an empirical analysis of publisher URL structures across 13 French outlets. That analysis reveals seven editorial families shared across publishers, which become the benchmark's harmonized label space.
Annotation follows a two-bucket pipeline: 72.2% of labels are assigned deterministically from publisher URL slugs, while the remaining 27.8% of ambiguous cases are labeled by an LLM and verified through a quality audit on a 300-article sample.
In practice
This hybrid design preserves publisher-native supervision where it is reliable, while still producing a shared taxonomy that can be inferred from article text when publisher metadata is ambiguous or not comparable.
FrenchNews-7 is accompanied by paper, model, and dataset resources intended to support reproducible research use.
The paper explains the benchmark design, taxonomy, labeling pipeline, evaluation setup, and main results.
A public CamemBERT-base classifier is available for seven-way French news editorial desk classification.
The dataset release is manifest-only: it includes metadata, labels, taxonomy files, and reconstruction assets, but does not redistribute article text.
Questions, collaborations, and media inquiries can be directed to Le French News Lab.
The main results highlight both benchmark performance and cross-publisher transfer.
French news editorial desk classification can transfer beyond a single publisher: on four held-out outlets, six of seven categories reach recall at or above 0.810.
Using headline + body text consistently outperforms headline-only classification, with CamemBERT-base gaining 5.3 macro-F1 points over its headline-only variant.
Models built for French perform better than broader multilingual baselines on this task.
Cross-publisher transfer is uneven: Économie remains the hardest category, while the other six categories generalize strongly.
On the main test split, CamemBERT-base with headline + body reaches macro-F1 0.847 and accuracy 0.860. The released fine-tuned model outperforms GPT-OSS-120B zero-shot by +4.6 macro-recall points (0.799 vs. 0.753).
Technical details below
FrenchNews-7 is a 7-way French news editorial desk-classification benchmark built from 87,637 labeled articles across 13 publishers. The public release includes a fine-tuned CamemBERT-base model plus a manifest-only dataset designed for reproducibility without redistributing article text.
| Total articles | 87,637 |
| Publishers | 13 |
| Labels | 7 |
| Train split | 61,345 |
| Validation split | 13,146 |
| Test split | 13,146 |
| Public release | Fine-tuned model + manifest dataset resources |
| Reference model | CamemBERT-base FrenchNews-7 |
| Main benchmark result | macro-F1 0.847 on test split |
The benchmark uses 74 deterministic slug-mapping rules for Bucket A articles (72.2%) and an LLM-based labeling step for Bucket B articles (27.8%) whose URLs carry no unambiguous category signal. A quality audit on 300 Bucket B articles using a blinded human annotator found 84% label agreement (Cohen's κ = 0.806), supporting the reliability of the LLM-labeled subset.
The paper reports both standard test-split performance and cross-publisher generalization on 2,100 articles from four held-out outlets. Six of seven categories reach recall at or above 0.810, while Économie remains the weakest boundary.
FrenchNews-7 is intended to support research and benchmarking without redistributing copyrighted publisher content.
The public release includes repository-authored materials such as labels, taxonomy definitions, metadata packaging, and reconstruction utilities. It does not include raw article text.
Any downstream fetching, storage, processing, or redistribution of source material remains the responsibility of the user and should comply with applicable law, publisher terms, robots directives, and institutional policy.
If you use FrenchNews-7 or the associated classifier, please cite the current working-paper version.
@misc{sobhy2026frenchnews7,
title = {FrenchNews-7: Benchmarking Cross-Publisher French News Editorial Desk Classification},
author = {Amr Sobhy},
year = {2026},
note = {Working paper, version April 2026},
}The citation will be updated with an arXiv identifier and final publication details when they are available.
For research collaborations, media questions, or institutional inquiries, contact Le French News Lab.
Le French News Lab
