GUIDE: Generative Utility Inference and Decision Engine
为解决AI对齐中用户偏好测量难题，GUIDE通过结合贝叶斯自适应采样和符号表示学习的方法，在对话中高效推断用户偏好。
0 citationsRead paper
Selected work
Representative Papers
Transformers Learn Faster with Semantic Focus
This study investigates the impact of sparse attention mechanisms on the learnability and generalization of Transformers, moving beyond conventional computational efficiency considerations. We analyze two classes of sparsity patterns: input-dependent (semantic focusing) and input-agnostic. Our methodology integrates learning dynamics modeling, softmax stability analysis, Lipschitz characterization of the loss, and theoretical derivation of convergence and generalization error bounds. We establish, for the first time, the intrinsic mechanism by which semantic focusing accelerates training and quantitatively link it to attention convergence and generalization performance. Empirically, input-dependent sparse attention significantly improves convergence speed and generalization, whereas input-agnostic sparsity yields no such benefit. Theoretically, we derive sufficient conditions under which semantic focusing provably enhances optimization and generalization. Our work provides a novel analytical framework for designing and understanding sparse attention mechanisms in Transformer architectures.
0 citationsRead paper
Recent publications
Latest Papers
GUIDE: Generative Utility Inference and Decision Engine
为解决AI对齐中用户偏好测量难题，GUIDE通过结合贝叶斯自适应采样和符号表示学习的方法，在对话中高效推断用户偏好。
0 citationsRead paper
Transformers Learn Faster with Semantic Focus
This study investigates the impact of sparse attention mechanisms on the learnability and generalization of Transformers, moving beyond conventional computational efficiency considerations. We analyze two classes of sparsity patterns: input-dependent (semantic focusing) and input-agnostic. Our methodology integrates learning dynamics modeling, softmax stability analysis, Lipschitz characterization of the loss, and theoretical derivation of convergence and generalization error bounds. We establish, for the first time, the intrinsic mechanism by which semantic focusing accelerates training and quantitatively link it to attention convergence and generalization performance. Empirically, input-dependent sparse attention significantly improves convergence speed and generalization, whereas input-agnostic sparsity yields no such benefit. Theoretically, we derive sufficient conditions under which semantic focusing provably enhances optimization and generalization. Our work provides a novel analytical framework for designing and understanding sparse attention mechanisms in Transformer architectures.
0 citationsRead paper