How the ranking is built, and what it is measured against
A reference page: the evidence the model reads and what each piece is measurably worth, the checks the reported gain had to survive, and the post-mortem on the benchmark version this one replaced.
What the model reads
Every ranking traces back to real, dated, publicly-sourced facts. A gradient-boosted model (XGBoost) is trained on six feature groups, each grounded in a specific public source:
Scientific assessment scrutiny
How many EFSA toxicological assessments exist, over what span of years. Sustained scientific attention is itself a signal, whatever each assessment concluded.
Links to flagged substances
Shared hazard classifications and metabolic degradation links to other substances already flagged as concerning.
Sales and usage trends
Tonnage sold over time, trend direction, and volatility. A substance quietly losing market share behaves differently from one still expanding.
Independent literature signal
How a substance's share of hazard-flavoured scientific literature (Europe PMC) compares to the rest of the field in the same year. This is the one signal here that sits upstream of the regulatory process itself.
Hazard classification history
How many severe hazard codes a substance carries under EU CLP: carcinogenicity, aquatic toxicity, reproductive toxicity, and how recently a classification was added.
Classification intentions
Whether ECHA has recorded an intention to propose a harmonised classification, and when. A declared intention is the earliest formal sign that a dossier is moving.
What each one is worth
Average precision added when that group alone is given to the model on top of approval age. Two groups come out slightly negative. They stay in, because dropping a feature after seeing its sign is how a benchmark stops measuring anything.
EFSA assessments and ECHA classification intentions read the regulator's own pipeline, so a result resting on them would be reporting regulatory intent rather than anticipating it. Measured as blocks, those two contribute 0.085 and the four that sit outside the pipeline contribute 0.067. The result does not depend on reading the regulator.
The checks behind the headline
The reported gain is 0.102 for approval age alone against 0.242 with the evidence added. Why the question had to be reformulated to get there is on the landing page. These are the checks that number had to survive.
- Seed spread
- the gain is +0.140 against ±0.014 across seeds
- Feature lag
- reading every feature three years old decays it to +0.025 rather than collapsing it
- Permutation
- shuffling whole substance histories puts it at p = 0.024
- Forward split
- fitted on 2019, its top 50 holds 11 real withdrawals against the baseline's 4
And what limits it
Measured by the same kind of check rather than estimated.
- Interactions
- a linear model recovers about 20% of the gain, so it does not live in any single feature
- Calibration
- raw scores are overconfident and need calibrating before they can be read as probabilities
- One era
- most events fall in the 2017 to 2021 renewal wave, and a model fitted before it does not transfer into it
- Not age-free
- approval age is about 47% recoverable from the evidence, so the evidence-only arm carries some of the calendar
The first benchmark, and what its lead times measure
The Hazium Early Warning Benchmark fixes ten historical EU pesticide bans, real regulatory actions, not hypothetical cases. It then works through every annual cutoff from 2009, using only evidence dated before each one, and records the earliest year each substance entered the riskiest twenty of roughly 5,900.
Read the lead times as a window, not a warning
Chlorpyrifos is marked 132 months ahead of its 2020 ban. That is January 2009, the first cutoff tested, so the number says it was already ranked high when the window opened rather than that the model saw it coming. Seven of the ten were in the top twenty at or within two years of that first cutoff, and ranking on approval age alone reproduces the same dates.
The pattern across the whole chart is one thing: early on substances that were already old and heavily assessed in 2009, four years behind the regulator on imidacloprid, level on dimethoate, and absent on epoxiconazole. A benchmark whose headline metric is bounded by its own start date cannot separate foresight from seniority, which is why the question was reformulated and the survival result is what this project now reports.
What the chart still shows honestly is the comparison against the regulator's own first public concern, a dated milestone the model never reads. The literature signal is a model input, so it is excluded from that comparison.
Click a substance for its use and full rank history.
The landmarks it misses
Case by case: on chlorpyrifos, its methyl sister, thiacloprid and mancozeb the substance sat among the riskiest roughly a decade before EFSA's first public concern, though all four were long-standing approvals by 2009. On the neonicotinoids it was early relative to the 2013 EU restriction, with national bans already emerging. On dimethoate it moved level with the regulator, on imidacloprid it flagged four years late, and epoxiconazole it never flagged. Where a substance had a real public controversy the chart marks that too, and most landmarks had no public profile at all at the point they entered the band.
What a permutation test can and cannot rule out
A ranking model with 25 positives in 5,933 substances can look good by accident. So the labels were permuted and the whole thing refitted, fifty times. Real average precision is 0.230 against a shuffled maximum of 0.013, p = 0.020. The result is not noise.
That answers one question: can the model beat chance. It cannot answer whether something simpler beats the model. No permutation test would surface approval age, because only running the baseline does.
Both versions stay published, each correct for the question it asked, in one citable dataset: HEWB v1.4 and v2 on HuggingFace, CC-BY-4.0.
HEWB v1.4. Flag dates come from the frozen benchmark run under strict pre-cutoff evidence discipline; out-of-fold scores are averaged over repeated cross-validation, so the ranks hold steady across resampling. Regulatory milestone dates are hand-verified against the enacting act or EFSA output.