Selected work

Representative Papers

Sample-efficient Integration of New Modalities into Large Language Models

Sep 04, 2025

Integrating low-resource novel modalities—such as satellite/astronomical images, IMU signals, and molecular data—into large language models (LLMs) remains challenging due to prohibitive data and computational requirements. Method: We propose a hypernetwork-based few-shot modality adaptation framework. It employs a shared projector to unify heterogeneous modality embeddings into a common latent space and introduces a modality-agnostic hypernetwork that generates task-specific adapters for arbitrary-dimensional novel modalities using only 32 labeled samples. To enhance generalization, we incorporate isometric transformations during training to increase representation diversity. Contribution/Results: Our method achieves comparable performance to full modality fine-tuning on multimodal benchmarks while requiring only 1/64 the training data. It significantly lowers the barrier to modality expansion and enables zero-shot extensibility—allowing seamless integration of previously unseen modalities without retraining the base LLM.

0 citationsRead paper

Should We Still Pretrain Encoders with Masked Language Modeling?

Jul 01, 2025

This work investigates the relative merits of masked language modeling (MLM) versus causal language modeling (CLM) for text representation pretraining in the large-model era. Method: We conduct a large-scale, controlled empirical study across 30 models spanning 210M to 1B parameters, involving over 15,000 fine-tuning evaluations. We further propose a novel two-stage “CLM→MLM” training strategy and assess low-cost adaptation of existing CLM models to MLM objectives. Contribution/Results: We find that MLM consistently achieves superior downstream performance, while CLM exhibits higher data efficiency and fine-tuning stability. The two-stage strategy attains optimal performance under fixed compute budgets, and CLM-to-MLM adaptation significantly reduces training cost. Crucially, MLM’s advantage stems not from confounding factors such as scale or dataset bias, but from the intrinsic suitability of its objective for representation learning; CLM’s value lies in training efficiency and robustness—indicating complementarity, not mutual exclusivity, between the two paradigms.

0 citationsRead paper

Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs

Jun 20, 2025

This study addresses the fundamental trade-off between task specialization (e.g., machine translation) and general-purpose capabilities (e.g., dialogue, reasoning, instruction following) in multilingual large language models. We propose a Pareto-optimal multi-stage training paradigm: continued pretraining → supervised fine-tuning → preference optimization → verifiable-reward reinforcement learning, integrated with multi-task data generation and rigorous filtering. We develop a family of multilingual models at three scales—2B, 9B, and 72B parameters—and introduce IF-MT, the first dedicated benchmark for instruction-following machine translation evaluation. Experimental results demonstrate that our 2B and 9B models outperform Llama 3.3-70B; the 72B model achieves state-of-the-art performance on high-resource language translation, Multilingual Arena Hard, and IF-MT—marking the first instance where translation specialization and broad general capabilities are simultaneously and synergistically enhanced.

0 citationsRead paper

Instituto de Telecomunicac{c}~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning

Jun 20, 2025

This study investigates the feasibility of building a unified speech-to-text model under strict copyright constraints (exclusively CC-BY–licensed data) and model size limitations (<2B parameters). We propose a two-stage training paradigm: first, modality-aligned pretraining to jointly optimize a continuous speech encoder and a lightweight language decoder in an end-to-end manner; second, instruction fine-tuning augmented with controllable synthetic data to enhance generalization. To our knowledge, this is the first work achieving direct alignment between small language models and continuous speech representations on the IWSLT short-track benchmark. Our approach achieves state-of-the-art performance across three tasks—automatic speech recognition, speech translation, and spoken question answering—demonstrating that high-quality data curation and architecture co-design significantly improve cross-task generalization in compact models. The method thus strikes a balanced trade-off among computational efficiency, task performance, and licensing compliance.

0 citationsRead paper

Long-Context Generalization with Sparse Attention

Jun 19, 2025

To address inaccurate pattern localization and degraded generalization in long sequences, this work identifies that conventional Softmax attention—enforcing dense, normalized attention distributions—is susceptible to interference from irrelevant tokens, leading to attention dispersion and representation collapse. We propose ASEntmax, a sparse attention mechanism built upon α-entmax: it introduces a learnable, adaptive temperature parameter enabling dynamic interpolation between sparsity and density; additionally, we design a novel positional encoding scheme to enhance long-range modeling. Seamlessly integrated into standard Transformer architectures, ASEntmax achieves significant improvements over Softmax, Sparsemax, and fixed-temperature α-entmax baselines across multiple long-context generalization tasks. Notably, it is the first method to enable end-to-end adaptive optimization of sparse attention parameters, combining theoretical rigor with strong empirical performance.

0 citationsRead paper

Recent publications

Latest Papers

Sample-efficient Integration of New Modalities into Large Language Models

Sep 04, 2025

Integrating low-resource novel modalities—such as satellite/astronomical images, IMU signals, and molecular data—into large language models (LLMs) remains challenging due to prohibitive data and computational requirements. Method: We propose a hypernetwork-based few-shot modality adaptation framework. It employs a shared projector to unify heterogeneous modality embeddings into a common latent space and introduces a modality-agnostic hypernetwork that generates task-specific adapters for arbitrary-dimensional novel modalities using only 32 labeled samples. To enhance generalization, we incorporate isometric transformations during training to increase representation diversity. Contribution/Results: Our method achieves comparable performance to full modality fine-tuning on multimodal benchmarks while requiring only 1/64 the training data. It significantly lowers the barrier to modality expansion and enables zero-shot extensibility—allowing seamless integration of previously unseen modalities without retraining the base LLM.

0 citationsRead paper

Should We Still Pretrain Encoders with Masked Language Modeling?

Jul 01, 2025

This work investigates the relative merits of masked language modeling (MLM) versus causal language modeling (CLM) for text representation pretraining in the large-model era. Method: We conduct a large-scale, controlled empirical study across 30 models spanning 210M to 1B parameters, involving over 15,000 fine-tuning evaluations. We further propose a novel two-stage “CLM→MLM” training strategy and assess low-cost adaptation of existing CLM models to MLM objectives. Contribution/Results: We find that MLM consistently achieves superior downstream performance, while CLM exhibits higher data efficiency and fine-tuning stability. The two-stage strategy attains optimal performance under fixed compute budgets, and CLM-to-MLM adaptation significantly reduces training cost. Crucially, MLM’s advantage stems not from confounding factors such as scale or dataset bias, but from the intrinsic suitability of its objective for representation learning; CLM’s value lies in training efficiency and robustness—indicating complementarity, not mutual exclusivity, between the two paradigms.

0 citationsRead paper

Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs

Jun 20, 2025

This study addresses the fundamental trade-off between task specialization (e.g., machine translation) and general-purpose capabilities (e.g., dialogue, reasoning, instruction following) in multilingual large language models. We propose a Pareto-optimal multi-stage training paradigm: continued pretraining → supervised fine-tuning → preference optimization → verifiable-reward reinforcement learning, integrated with multi-task data generation and rigorous filtering. We develop a family of multilingual models at three scales—2B, 9B, and 72B parameters—and introduce IF-MT, the first dedicated benchmark for instruction-following machine translation evaluation. Experimental results demonstrate that our 2B and 9B models outperform Llama 3.3-70B; the 72B model achieves state-of-the-art performance on high-resource language translation, Multilingual Arena Hard, and IF-MT—marking the first instance where translation specialization and broad general capabilities are simultaneously and synergistically enhanced.

0 citationsRead paper

Instituto de Telecomunicac{c}~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning

Jun 20, 2025

This study investigates the feasibility of building a unified speech-to-text model under strict copyright constraints (exclusively CC-BY–licensed data) and model size limitations (<2B parameters). We propose a two-stage training paradigm: first, modality-aligned pretraining to jointly optimize a continuous speech encoder and a lightweight language decoder in an end-to-end manner; second, instruction fine-tuning augmented with controllable synthetic data to enhance generalization. To our knowledge, this is the first work achieving direct alignment between small language models and continuous speech representations on the IWSLT short-track benchmark. Our approach achieves state-of-the-art performance across three tasks—automatic speech recognition, speech translation, and spoken question answering—demonstrating that high-quality data curation and architecture co-design significantly improve cross-task generalization in compact models. The method thus strikes a balanced trade-off among computational efficiency, task performance, and licensing compliance.

0 citationsRead paper

Long-Context Generalization with Sparse Attention

Jun 19, 2025

To address inaccurate pattern localization and degraded generalization in long sequences, this work identifies that conventional Softmax attention—enforcing dense, normalized attention distributions—is susceptible to interference from irrelevant tokens, leading to attention dispersion and representation collapse. We propose ASEntmax, a sparse attention mechanism built upon α-entmax: it introduces a learnable, adaptive temperature parameter enabling dynamic interpolation between sparsity and density; additionally, we design a novel positional encoding scheme to enhance long-range modeling. Seamlessly integrated into standard Transformer architectures, ASEntmax achieves significant improvements over Softmax, Sparsemax, and fixed-temperature α-entmax baselines across multiple long-context generalization tasks. Notably, it is the first method to enable end-to-end adaptive optimization of sparse attention parameters, combining theoretical rigor with strong empirical performance.

0 citationsRead paper