Synthical logo
Your space
Activity icon
Favorites icon
Account icon
From chemRxiv
University of Cambridge

On The Difficulty of Validating Molecular Generative Models Realistically: A Case Study on Public and Proprietary Data

While a multitude of deep generative models have recently emerged there exists no best practice for their practically relevant validation. On the one hand, novel de novo-generated molecules cannot be refuted by retrospective validation (so that this type of validation is biased); but on the other hand prospective validation is expensive and then often biased by the human selection process. In this case study, we frame retrospective validation as the ability to mimic human drug design, by answering the following question: Can a generative model trained on early-stage project compounds generate middle/late-stage compounds de novo? To this end, we used experimental data that contains the elapsed time of a synthetic expansion following hit identification from five public (where the time series was pre-processed to better reflect realistic synthetic expansions) and six in-house project datasets, and used REINVENT as a widely adopted RNN-based generative model. After splitting the dataset and training REINVENT on early-stage compounds, we found that rediscovery of middle/late-stage compounds was much higher in public projects (at 1.60%, 0.64%, and 0.21% of the top 100, 500, and 5,000 scored generated compounds) than in in-house projects (where the values were 0.00%, 0.03%, and 0.04%, respectively). Similarly, average single nearest neighbour similarity between early- and middle/late-stage compounds in public projects was higher between active compounds than inactive compounds; however, for in-house projects the converse was true, which makes rediscovery (if so desired) more difficult. We hence show that the generative model recovers very few middle/late-stage compounds from real-world drug discovery projects, highlighting the fundamental difference between purely algorithmic design and drug discovery as a real-world process. Evaluating de novo compound design approaches appears, based on the current study, difficult or even impossible to do retrospectively. "Scientific Contribution" This contribution hence illustrates aspects of evaluating the performance of generative models in a real-world setting which have not been extensively described previously and which hopefully contribute to their further future development.
Upvote icon
Published on November 20, 2023
Copy BibTeX
Cross iconSummary
There is no AI-powered summary yet, because we do not have a budget to generate summaries for all articles.
1. Buy subscription
We will thank you for helping thousands of people to save their time at the top of the generated summary.
If you buy our subscription, you will be able to summarize multiple articles.
Pay $8
≈10 summaries
Pay $32
≈60 summaries
2. Share on socials
If this article gets to top-5 in trends, we'll summarize it for free.
Copy link