Toxicity Assessment in Preclinical Histopathology via Class-Aware Mahalanobis Distance for Known and Novel Anomalies

2026-02-02

AI Summary

This study addresses the challenges in preclinical drug toxicity assessment posed by limited expert resources and the scarcity of rare pathological samples. The authors propose an AI-driven anomaly detection framework that leverages whole-slide images to segment healthy and known pathological regions in rodent liver tissue while effectively identifying rare anomalies absent from the training data. The method integrates the DINOv2 vision transformer with LoRA fine-tuning to achieve high-precision tissue segmentation and introduces a novel class-aware Mahalanobis distance combined with class-specific thresholding to distinguish between known pathologies and unknown anomalies. Evaluated on mouse liver data, the approach demonstrates exceptional performance, misclassifying only 0.16% of lesions as healthy and 0.35% of healthy tissue as pathological, thereby significantly enhancing generalization and detection accuracy.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Drug-induced toxicity remains a leading cause of failure in preclinical development and early clinical trials. Detecting adverse effects at an early stage is critical to reduce attrition and accelerate the development of safe medicines. Histopathological evaluation remains the gold standard for toxicity assessment, but it relies heavily on expert pathologists, creating a bottleneck for large-scale screening. To address this challenge, we introduce an AI-based anomaly detection framework for histopathological whole-slide images (WSIs) in rodent livers from toxicology studies. The system identifies healthy tissue and known pathologies (anomalies) for which training data is available. In addition, it can detect rare pathologies without training data as out-of-distribution (OOD) findings. We generate a novel dataset of pixelwise annotations of healthy tissue and known pathologies and use this data to fine-tune a pre-trained Vision Transformer (DINOv2) via Low-Rank Adaptation (LoRA) in order to do tissue segmentation. Finally, we extract features for OOD detection using the Mahalanobis distance. To better account for class-dependent variability in histological data, we propose the use of class-specific thresholds. We optimize the thresholds using the mean of the false negative and false positive rates, resulting in only 0.16\% of pathological tissue classified as healthy and 0.35\% of healthy tissue classified as pathological. Applied to mouse liver WSIs with known toxicological findings, the framework accurately detects anomalies, including rare OOD morphologies. This work demonstrates the potential of AI-driven histopathology to support preclinical workflows, reduce late-stage failures, and improve efficiency in drug development.

Problem

• toxicity assessment
• histopathology
• anomaly detection
• out-of-distribution
• preclinical drug development

Innovation

• Class-Aware Mahalanobis Distance
• Out-of-Distribution Detection
• Vision Transformer
• Low-Rank Adaptation
• Histopathology