Why this evaluation matters before you stake synthesis budget on it
When a scoring tool returns a ranked shortlist, it is making an implicit claim: the compounds at the top are more worth synthesizing than the ones at the bottom. That claim can be right or wrong in ways that cost you bench weeks. A model trained on a narrow chemical series, evaluated on a test set that overlaps its training data, and returning confidence intervals computed from held-out validation alone will look credible on paper. On your actual compound set, it may perform no better than a random draw.
The questions below are not exhaustive. They are the ones we found ourselves wishing we had asked earlier, before trusting scores we should not have trusted.
What training data was used, and how was it partitioned?
A molecular scoring model is only as useful as its training data is representative of your chemistry. The first question to ask any vendor is: what is the source of the binding affinity labels in your training set? Public repositories like ChEMBL and BindingDB contain millions of activity values across thousands of protein targets, but coverage is highly uneven. Kinases, proteases, and GPCRs are densely represented. Orphan transporters, allosteric sites on poorly-characterised enzymes, and covalent binders are sparse.
If your target sits in a poorly-covered area, a model trained primarily on dense-coverage targets will generalise to your chemistry only to the extent that the underlying molecular features transfer. That may be quite limited.
The partitioning question is equally important. A model whose test set was split randomly from its training set will show inflated performance metrics. The compounds in the test set will be close neighbours (in fingerprint space) of compounds in the training set, so the model is not demonstrating generalisation, it is demonstrating memorisation. Scaffold-split validation, where entire scaffolds are held out during training, gives a more honest picture of how the model will behave on genuinely novel chemistry. Ask: did you use random or scaffold-based partitioning in your published benchmarks? If the answer is random, discount the reported AUC accordingly.
What does the benchmark set actually test?
A vendor may cite a high enrichment factor on a standard benchmark like DUD-E or MUV. These are useful but have known limitations. DUD-E decoys are generated to match property distributions of actives, which means the model may be learning to distinguish activity by physicochemical properties rather than binding geometry. DUD-E performance is a floor, not a ceiling.
More relevant to your decision is whether the vendor has benchmark results on a target class similar to yours, and whether that benchmark used known actives that were not in the training data at the time the benchmark was run. A retrospective run on a target where public activity data appeared after the training cutoff is the closest thing to a prospective test you can ask for without running the experiment yourself.
We are not saying DUD-E performance is meaningless. We are saying it should not be the only evidence you evaluate before trusting a scoring tool on a novel target.
What does the confidence band actually represent?
This is the question that most often reveals the difference between vendors who understand their uncertainty quantification and those who do not. A confidence band on a predicted binding score could mean several different things.
It could be a Gaussian process posterior variance, which has a principled probabilistic interpretation but depends on kernel choice and assumes the query compound is interpolated from training data. It could be a Monte Carlo dropout estimate, which approximates Bayesian uncertainty but is sensitive to dropout rate and architecture. It could be an ensemble standard deviation across independently trained models, which captures model variability but not data noise. Or it could simply be the RMSE of the model on a validation set, reported as a fixed-width band that carries no per-compound meaning at all.
Ask the vendor to show you what happens to the confidence band width as you move to compounds increasingly distant (by Tanimoto similarity) from the training set. If the band does not widen as novelty increases, the uncertainty estimate is not tracking the right thing. A well-calibrated uncertainty estimate should be wider for out-of-distribution queries than for in-distribution ones.
Running your own retrospective on known actives
The most informative test you can run before trusting a scoring tool on your program is a retrospective enrichment analysis using your own known actives and known inactives. If your program has progressed to the point where you have IC50 data on at least 30-50 compounds covering a reasonable potency range, you can use this data directly.
Submit the full set as a blinded batch (strip the activity labels), receive the ranked output, and compute enrichment at various fractions of the list. An enrichment factor of 3-5x at the top 10% of the ranked list is a reasonable minimum for a tool to be useful as a prioritisation filter. Below that, the ordering adds less value than selecting by physicochemical properties alone.
If you do not yet have sufficient experimental data for a retrospective, ask whether the vendor can run one using a public target with similar binding mode to yours and share the results. A vendor confident in their model's performance should be willing to do this.
Consistency across chemotype boundaries
One subtle failure mode in scoring tools is inconsistent rank ordering across chemotype boundaries. Within a congeneric series, a model can learn relative SAR very well because the training data provides dense coverage of that scaffold. Across a scaffold hop, the model may assign systematically different absolute score levels to two chemotypes even when the binding geometries are comparable.
This matters if you are comparing scores across a diverse compound set. A compound from scaffold A scoring 7.2 kcal/mol may not be directly comparable to a compound from scaffold B scoring 7.4 kcal/mol if the model was trained on different densities of data for each scaffold family. The ranked shortlist within each chemotype is often more reliable than cross-chemotype comparisons. Ask whether the vendor's scoring is calibrated across chemotype boundaries, and how. A model that scores one scaffold class consistently higher than another due to training data imbalance will mislead multi-chemotype prioritisation decisions.
Check how the model handles ADMET property predictions alongside binding
If the tool produces integrated ADMET scores alongside binding affinity, ask the same provenance questions: what training data for each property, how was it partitioned, and what is the benchmark performance for each individual property rather than an aggregated composite. Composite ADMET scores blend properties with very different model confidence levels, which can mask specific weaknesses. A tool might have excellent hERG inhibition prediction and weak CYP induction prediction; a composite score obscures that difference.
Separate property flags, rather than a single composite tox number, make it easier to identify which part of the ADMET prediction is grounded in strong training data and which part deserves more caution. The evaluation framework you apply to binding prediction applies to each tox property independently.
The evaluation is ongoing, not a one-time check
Even a tool that passes all of the above checks at the start of your program will need ongoing validation as your experimental data accumulates. Each new batch of IC50 results is an opportunity to check whether the model's rank ordering was predictive. If you see systematic discordance between predicted rank and measured potency within a scaffold series, that is a signal worth investigating: has the model saturated its useful range for this target, or is there a feature of your chemistry it is not capturing well?
The goal is not to find a perfectly reliable scoring tool. The goal is to understand, continuously, where the tool is reliable and where it is not, so you can weight its output appropriately when making synthesis decisions. That understanding develops through use and calibration, not from a single upfront evaluation.