Hazium
I wanted to know whether public data could flag a dangerous pesticide before anyone knew to look. What I built predicts something else. This is how that happened, and what it cost.
- dated facts
- 41,917
- public sources
- 5
- tests
- 415
- licence
- AGPL-3.0
Python 3.12, XGBoost, scikit-learn, SHAP, Pydantic. Site in Next.js.
Picking a target I could score
There is no list of substances that turned out to be dangerous. That is the thing you are trying to find out. There is a list of substances the EU withdrew: dated, public, unambiguous.
So I predicted withdrawals. It is the obvious move, I made it early, and then I stopped looking at it. Everything below follows from that.
The graph
Five public EU and Swedish sources, no shared identifier, no shared schema. I resolved them into one graph: 41,917 facts, each carrying the date it became public.
The dating is the part that took the time. A model scored at a 2015 cutoff sees what was public in 2015 and nothing else, so it cannot quietly score itself on the future.
Over that graph I trained gradient-boosted trees (XGBoost) on six dated feature groups: EFSA assessment history, hazard classifications under CLP, ECHA classification intentions, sales trajectory, independent literature signal, and graph links to substances already flagged. Scores are out of fold, folds grouped by substance.