Minimizing late-stage failure in drug development with transformer models: Enhancing drug screening and pharmacokinetic predictions

Drug discovery is a multifaceted and iterative process aimed at identifying novel therapeutic compounds capable of modulating disease pathways. It typically begins with target identification, where key biological molecules—such as proteins, enzymes, or receptors involved in disease mechanisms—are pinpointed as potential intervention points. Following identification of target, the focus shifts to initial candidate selection, which involves screening large chemical libraries to identify compounds that interact effectively through mechanisms such as inhibition or activation [1], [2]. Promising compounds are refined through medicinal chemistry to enhance their potency, selectivity, and drug-like properties. This process is iterative, relying on continuous feedback from biochemical, cellular, and in vivo assays to optimize pharmacokinetic (PK) and pharmacodynamic (PD) profiles. Among these steps one key component is testing molecules for their ADME-T (absorption, distribution, metabolism, excretion and toxicity) profiles. Testing candidates for these properties is vital since approximately 40 % of compounds fail during the ADME-T testing stage due to unfavorable pharmacokinetic and toxicity profiles [3], [4], [5], which substantially increases the cost and extends the timelines of drug development.

ADME-T testing evaluates how a drug is absorbed, distributed, metabolized, and excreted and its potential toxicity. These factors are crucial for determining whether a compound can be safely and effectively used as a drug, as they influence its bioavailability, therapeutic efficacy, and safety profile. Poor performance in these areas can lead to a drug candidate's failure in later stages of development. For example, while the estimated median cost of research and development of drugs is 319.3 million USD, after accounting for the cost of the failed trials, it surges to a median of 1.14 billion USD [6]. This underscores the importance of integrating ADME-T testing immediately after target identification and initial candidate selection. This ensures that compounds with poor pharmacokinetic or toxicity profiles are eliminated early in the drug discovery process before advancing to more resource-intensive stages like lead optimization and clinical trials.

Given these high stakes, the increasing role of data-driven models in drug discovery is reshaping how we approach therapeutic compound identification and development [6], [7], [8], [9], [10]. Techniques such as decision trees (DTs) have been used to aid in the clear classification of compounds with potential toxicity or therapeutic benefits [10]. Artificial neural networks (ANNs) are adept at identifying properties like lipophilicity [11] and aqueous solubility [12], [13], [14]. Methods like support vector machines (SVMs), k-nearest neighbors (kNN), and random forests (RFs) have been effectively applied to predict ADME-T properties [15], [16], [17]. These techniques utilize molecular descriptors to represent compounds, predicting how these compounds will interact in biological systems. While these methods have facilitated advancements in property prediction and candidate screening, they are often constrained by their reliance on hand-crafted molecular descriptors and feature engineering [15]. This feature dependency can limit scalability and flexibility, primarily when predicting multiple drug-related properties. In a data-driven context, bypassing feature engineering and predicting diverse properties with a unified model is crucial to streamlining drug discovery workflows.

To address these limitations, our model leverages transformer-based architectures [16], which use self-attention mechanisms to generate molecular embeddings directly from SMILES sequences [17]. Transformers have been widely used in the chemical domain, including for process modeling and control. For example, attention mechanisms have been applied in time-series transformers (TSTs) to capture long- and short-term dependencies in chemical processes and predict the next system state, enabling real-time process optimization and control [18]. Additionally, they have been used to enhance system-to-system transferability and optimize batch crystallization [19], [20]. More prominently, transformers have been employed for molecular property prediction, where their ability to capture intricate molecular relationships has proven essential like in predicting nonlinear properties in ionic liquids, polymeric and protein-based systems by generating context aware chemical embeddings [21], [22], [23]. These embeddings capture intricate molecular features and interactions without needing explicit feature engineering, offering a significant advantage over conventional ML models that depend on predefined molecular descriptors. By their nature, transformers capture long-range dependencies and relationships in molecular sequences, making them more flexible and scalable in multi-task property prediction than traditional methods. The generalizability of transformers allows for seamless application to diverse prediction tasks, from toxicity profiling to absorption and distribution characteristics, through a unified framework that simplifies the prediction pipeline. Building on this, we have enhanced the transformer architecture by integrating linear attention mechanisms, which improve computational efficiency, particularly when handling large molecular sequences. Additionally, the use of Rotary Positional Embedding (RoPE) allows the model to capture spatial relationships between atoms, further enriching the quality of the embeddings. Our transformer-based methodology addresses a critical challenge in drug discovery by minimizing late-stage failures caused by unfavorable pharmacokinetic and toxicity profiles. By incorporating ADME-T predictions early in the process, the approach ensures that only molecules with optimal properties progress to subsequent stages, significantly reducing the risk of costly failures. The framework's modular design allows for easy adaptation to specific drug discovery needs, enabling the addition or removal of properties and adjustment of thresholds based on therapeutic focus or domain requirements. This adaptability ensures the model's relevance across diverse contexts, supporting efficient evaluation of pharmacological properties.

As shown in Fig. 1 during the stage 1 screening, the embeddings generated by the transformer model are passed through a feed-forward network to predict drug-like properties. These predictions are based on Lipinski's Rule of 5, along with the verification of biological activity against the target. In the stage 2 screening, the same transformer-generated embeddings used to predict key ADME-T properties, including solubility, BBB penetration, enzyme inhibition, and toxicity profiles by passing the embeddings through simple feed-forward networks. This approach ensures that the transformer architecture efficiently generates embeddings, which are then used to carry out the necessary property predictions through a feed-forward network. The prediction of a wide array of properties is possible since the transformer's embeddings capture intricate molecular features and relationships. Finally, in stage 3 screening, the transformer predicts the IC50 values, ensuring that only the most potent and promising candidates proceed to experimental testing. We showcased a case study on screening molecules for HIV Integrase 1, demonstrating the real-world application of the developed, integrated approach for more effective drug candidate selection.