herg-uncertainty
A calibrated hERG cardiotoxicity classifier
Predicting whether a compound blocks the hERG cardiac ion channel is a standard early drug-safety screen. The centerpiece of this project isn't raw accuracy — it's a selective-prediction system: calibrated uncertainty plus an explicit applicability-domain check, so the model degrades gracefully outside its training domain instead of failing silently.
The bigger goal behind this: a positive AMES or carcinogenicity call today routinely triggers real in-vivo rodent testing (see Phases 9 and 10 below). Better computational safety screening — models that know when to trust themselves and when not to — is one of the concrete, unglamorous levers for reducing how much of that testing drug development actually needs, in line with where FDA New Approach Methodology guidance is heading. This project doesn't replace any of that. But it's aimed at the kind of problem that, pushed on long enough by enough people, eventually could.
To be specific about what "pushed on long enough" actually means: eliminating animal testing isn't one breakthrough, it's a slow stack of smaller ones — better in-vitro assays, organ-on-chip models, and computational predictions regulators and pharma teams can actually act on. This project is aimed at that last piece, and only a sliver of it. A model that's right 85% of the time isn't enough on its own to skip a rodent study; a model that can also flag the compounds it's notconfident about is closer to something a toxicologist could build a real workflow around — reserving traditional testing for the cases that actually need it instead of running it on everything by default. That's a long way from where this project is today. But it's the direction the applicability-domain work here is pointed, one piece among many that would all have to hold up together.
Every number below is real, pulled directly from the project's results — including the ones that didn't work. Charts are interactive: hover any point or bar for exact values.
Phase 0–1
Baseline: can a plain classifier do this at all?
Starting point: Random Forest and XGBoost on Morgan fingerprints (2,048 bits) plus ~200 RDKit physicochemical descriptors, trained on TDC's 648-compound hERG set with two different train/test splits. Scaffold split is the harder, more realistic test — it holds out entire chemical scaffolds rather than random compounds, so the model can't just memorize close analogs.
AUROC (0.83–0.85) lands within the published literature band, with no sign of the >0.95 red flag that usually means data leakage. The visible weak point is specificity(0.46–0.51) — these models catch blockers well but aren't confident at clearing non-blockers, which sets up exactly why an uncertainty layer matters.
Phase 2 / 6 / 9 / 10
Does accuracy actually drop outside the training domain?
This is the project's central claim, tested five separate times across different datasets and endpoints: split every test compound into "in-domain" or "out-of-domain" using a k-NN Tanimoto (structural similarity) applicability-domain check, then compare AUROC on each half separately.
The gap is real and consistent in direction on 4 of 5 configurations — the exception, TDC's random split, has only 6 out-of-domain compounds and shouldn't be trusted at that sample size. The cleanest validation is the genuine prospective set (Phase 6): 3,896 compounds confirmed to have zero overlap — exact or near-duplicate — with anything the model trained on. The AD gap replicates there too, on real external data, not just an internal test split.
Phase 6 — internal test vs. genuine prospective set (n=3,896)
Overall accuracy genuinely drops on truly novel compounds (0.876 → 0.797) — the internal numbers were somewhat optimistic even without any leakage. But the in-domain/out-of-domain gap holds up on data that was never touched during dataset curation, at real sample size — the closest thing to genuine external evidence this project can produce.
On the ChEMBL set specifically, the conformal-prediction coverage came in at 88.8% against a 90% nominal target, with a Venn-ABERS calibration error (ECE) of 0.080— the model's confidence scores are honest, not just its bare predictions.
Phase 3
Does a graph neural net beat classical ML here?
Built a GIN (Graph Isomorphism Network) from scratch in PyTorch Geometric and ran a learning curve against Random Forest — same training subsets (500 to 8,379 compounds), same held-out test set, 3 random-subsample repeats each.
Hypothesis confirmed: RF beats the GIN at every single training size, and the gap doesn't close as data grows — it holds fairly steady around 0.05–0.06 AUROC from 500 compounds up to the full 8,379. This matches prior published findings (Siramshetty 2020, Jiang 2021, Arab 2024): classical descriptor-based ML matches or beats graph neural nets at hERG's current data scale. Reported as confirmation, not a novel result.
Phase 4
Calibrating uncertainty by domain, not just by class
Instead of calibrating conformal prediction by class label alone, bucket the test set into 3 bins by applicability-domain distance and calibrate each bin separately. A single train/test split made this look like a clean win — so it was re-run across 10 random splits before trusting the result.
The 10-seed check tells a more honest story than the first run did: AD-conditioning costs about 2 percentage points of coverage in the smaller bin 1 (87.3% → 85.1%), but gains more than that back in bin 2 — the bulk of the test set (89.9% → 92.1%). Overall: 89.4% → 91.1% coverage against a 90% target. A real, reproducible improvement where most of the data lives, at a real, smaller, reproducible cost elsewhere — shown with both sides, not just the flattering number.
Phase 5 / 5.5 / 5.6
Chasing a signal for which predictions are wrong — mostly a null result
Three different attempts to answer "can we predict which specific compounds the model will get wrong," beyond just flagging out-of-domain compounds in aggregate.
Meta-AUROC — does each signal separate right from wrong predictions?
A dedicated failure-predictor classifier (trained directly on "was the base model wrong") landed mid-pack — it did notbeat the hand-crafted signals it was meant to replace. All four signals sit in a narrow, modest band (0.52–0.60): converging evidence that there may not be a strong, learnable "which prediction is wrong" signal in 2D structural features alone for this problem.
An activity-cliff detector (flagging compounds whose nearest training neighbors disagree with each other) was rebuilt once after a literature review, with a more principled significance test — catch rate went from 43% to 15%, but lift over chance stayed essentially flat (1.29x → 1.27x) — ruling out "implemented sloppily" without finding a stronger signal underneath.
Phase 7
Extending to two more cardiac ion channels
The same pipeline, unmodified, applied to Nav1.5 and Cav1.2 — two more channels relevant to cardiac drug safety. Both baselines score higher than hERG's, but on much smaller test sets (170 and 81 compounds), so read these as single-run snapshots, not yet robustness-checked the way hERG's Phase 4 result was.
Cav1.2's 801-compound training set produces zero out-of-domain test compounds — its bar is simply missing above, not zero — mirroring the same small-dataset problem TDC's original hERG set had. An overnight audit also found real near-duplicate leakage in Nav1.5 (11.2% of test compounds); tested directly, removing it didn't change the results.
Phase 8
A composite risk score, checked against a hard problem
Combined all three channels into one proxy risk score and checked it against 28 CiPA reference drugs with published clinical risk categories — a real external check, in theory.
The real finding here is structural, not numerical: 21 of 28 "famous" reference drugs turned out to already be in the training data. Only 5 were genuinely held out — far too few to trust either number above, which is exactly why Phase 6's prospective validation mattered: famous drugs are close to useless for externally validating models trained on public bioactivity databases.
Phase 9
Does the applicability-domain claim generalize past cardiotox?
First test on a structurally unrelated endpoint: AMES bacterial mutagenicity (7,278 compounds) — a positive result routinely triggers real in-vivo genotoxicity follow-up testing, so this isn't just a new benchmark to run.
The gap replicates (0.855 → 0.694, a 0.16 AUROC drop) — smaller than hERG's but the same direction, on a completely different toxicity mechanism. One reversal worth flagging plainly: across every cardiac-channel config, Venn-ABERS uncertainty beat AD distance as a risk-ranking signal; on AMES, AD distance ranks better. Reported as observed, not smoothed into the prior framing.
Phase 10
Carcinogenicity: the weakest, most honestly mixed result
TDC's 280-compound carcinogenicity set was too small to even compute an out-of-domain AUROC. Rather than stop there, a 1,294-compound companion dataset was built from scratch from the Carcinogenic Potency Database — a real data-engineering job (CAS numbers, not SMILES; per-experiment votes aggregated into one call per chemical).
On the larger set, the in/out-of-domain gap replicates but is the weakest in the project (0.736 → 0.701). More notably, the 10-seed robustness check on AD-conditioned conformal prediction — the technique that worked on hERG and AMES — did not help here (88.2% → 88.1%, essentially identical). The clearest properly-checked case in the project of this technique genuinely not helping on a real endpoint — a useful counterweight to the cleaner wins elsewhere.