What an Autonomous Agent Discovers About Molecular Transformer Design: Does It Transfer?

2026-03-30

AI Summary

Whether domain-specific Transformer architectures are truly necessary for molecular sequence modeling remains underexplored. This work proposes an autonomous-agent-based neural architecture search framework, conducting 3,106 experiments (on a single GPU) across SMILES, protein, and English text datasets to jointly optimize architectures and hyperparameters. The study reveals that while each domain exhibits distinct architectural preferences, all discovered innovations transfer across domains with less than 1% performance degradation, suggesting that observed differences stem from path dependence in the search process rather than inherent domain requirements. Furthermore, hyperparameter tuning dominates performance gains in SMILES tasks, whereas architectural changes account for 81% of the improvement in natural language tasks. The authors open-source their decision framework and toolkit to facilitate efficient reuse within the molecular modeling community.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Deep learning models for drug-like molecules and proteins overwhelmingly reuse transformer architectures designed for natural language, yet whether molecular sequences benefit from different designs has not been systematically tested. We deploy autonomous architecture search via an agent across three sequence types (SMILES, protein, and English text as control), running 3,106 experiments on a single GPU. For SMILES, architecture search is counterproductive: tuning learning rates and schedules alone outperforms the full search (p = 0.001). For natural language, architecture changes drive 81% of improvement (p = 0.009). Proteins fall between the two. Surprisingly, although the agent discovers distinct architectures per domain (p = 0.004), every innovation transfers across all three domains with <1% degradation, indicating that the differences reflect search-path dependence rather than fundamental biological requirements. We release a decision framework and open-source toolkit for molecular modeling teams to choose between autonomous architecture search and simple hyperparameter tuning.

Problem

• molecular transformer
• architecture search
• SMILES
• protein sequences
• transferability

Innovation

• autonomous architecture search
• molecular transformers
• cross-domain transfer
• SMILES representation
• hyperparameter tuning