Legislating World-Model-Based Planning with Legal Reasoning
本文针对机器人法律规范控制问题，提出了一种结合可废止道义逻辑(DDL)和学习世界模型的法律规划框架，以在行动前进行干预。
0 citationsRead paper
Selected work
Representative Papers
RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention
This work addresses the challenge of deploying extremely large Mixture-of-Experts (MoE) language models (26–120B parameters) on consumer-grade hardware, where excessive memory demands from model weights, KV caches, and expert sublayers hinder practical inference. To overcome this, the authors propose a three-axis compression framework comprising architecture-aware mixed-precision quantization (2/4/8-bit), an LRU-driven expert offloading mechanism, and IsoQuant—a novel KV cache compression method that combines Walsh–Hadamard transforms with SO(4) rotations to enable efficient low-rank isotropic quantization. IsoQuant further incorporates end-to-end fused GPU kernels for direct 3-bit tensor processing. Experiments demonstrate that the approach enables Gemma 4-26B-A4B and Qwen3-30B-A3B to run on 16GB GPUs and Nemotron-H 120B on 32GB GPUs, achieving inference speeds of 9–19 tokens per second, perplexity degradation no worse than +0.0012, and perfect 100% retrieval accuracy over 32K-context evaluations.
0 citationsRead paper
Automated Background Swapping for Robustness against Spurious Backgrounds
Deep neural networks often rely on spurious correlations between background features and class labels, which degrades their generalization performance. To address this issue, this work proposes AutoBackSwap, a method that leverages an auxiliary network trained with only partial annotations from a few hundred images to disentangle foreground and background components. By inpainting novel backgrounds and recombining them with original foregrounds, AutoBackSwap automatically constructs augmented samples that break these spurious correlations. Notably, the approach does not require any out-of-distribution data and consistently enhances model robustness across multiple classification tasks involving confounding backgrounds, outperforming existing methods in both accuracy and reliability.
0 citationsRead paper
Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies
This work addresses the common observation that large language models fine-tuned via evolutionary strategies (ES) often exhibit performance degradation on previously learned tasks—a phenomenon frequently misattributed to irreversible catastrophic forgetting. The study demonstrates that this decline is in fact a recoverable form of performance drift, not unique to ES. To mitigate this issue, the authors propose Anchored Weight Decay (AWD), a lightweight regularization method that stabilizes performance on old tasks by constraining the optimization trajectory to remain proximate to the initial model in parameter space. Experimental results show that AWD substantially alleviates forgetting without compromising performance on the target task, achieving stability comparable to that of large ES populations at minimal computational overhead, thereby affirming the viability of ES for continual learning.
0 citationsRead paper
Efficient Pre-Training of LLMs through Truncated SVD Layers
This work addresses the high computational cost of large language model (LLM) pretraining due to massive parameter counts, a challenge exacerbated by the inability of existing low-rank methods to simultaneously support dynamic rank selection and strict weight orthogonality. The authors propose the TSVD framework, which maintains a low-rank structure with orthogonality constraints throughout pretraining. It employs a spectral energy–based heuristic to adaptively determine the optimal rank and introduces a caching mechanism to efficiently preserve orthogonality. TSVD is the first method to concurrently enable dynamic low-rank adaptation and rigorous orthogonal constraints during training. Experiments demonstrate that it matches or exceeds full-parameter baseline performance across multiple model scales while substantially reducing computational overhead, establishing a new paradigm for efficient and scalable LLM pretraining.
0 citationsRead paper
Recent publications
Latest Papers
Legislating World-Model-Based Planning with Legal Reasoning
本文针对机器人法律规范控制问题，提出了一种结合可废止道义逻辑(DDL)和学习世界模型的法律规划框架，以在行动前进行干预。
0 citationsRead paper
RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention
This work addresses the challenge of deploying extremely large Mixture-of-Experts (MoE) language models (26–120B parameters) on consumer-grade hardware, where excessive memory demands from model weights, KV caches, and expert sublayers hinder practical inference. To overcome this, the authors propose a three-axis compression framework comprising architecture-aware mixed-precision quantization (2/4/8-bit), an LRU-driven expert offloading mechanism, and IsoQuant—a novel KV cache compression method that combines Walsh–Hadamard transforms with SO(4) rotations to enable efficient low-rank isotropic quantization. IsoQuant further incorporates end-to-end fused GPU kernels for direct 3-bit tensor processing. Experiments demonstrate that the approach enables Gemma 4-26B-A4B and Qwen3-30B-A3B to run on 16GB GPUs and Nemotron-H 120B on 32GB GPUs, achieving inference speeds of 9–19 tokens per second, perplexity degradation no worse than +0.0012, and perfect 100% retrieval accuracy over 32K-context evaluations.
0 citationsRead paper
Automated Background Swapping for Robustness against Spurious Backgrounds
Deep neural networks often rely on spurious correlations between background features and class labels, which degrades their generalization performance. To address this issue, this work proposes AutoBackSwap, a method that leverages an auxiliary network trained with only partial annotations from a few hundred images to disentangle foreground and background components. By inpainting novel backgrounds and recombining them with original foregrounds, AutoBackSwap automatically constructs augmented samples that break these spurious correlations. Notably, the approach does not require any out-of-distribution data and consistently enhances model robustness across multiple classification tasks involving confounding backgrounds, outperforming existing methods in both accuracy and reliability.
0 citationsRead paper
Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies
This work addresses the common observation that large language models fine-tuned via evolutionary strategies (ES) often exhibit performance degradation on previously learned tasks—a phenomenon frequently misattributed to irreversible catastrophic forgetting. The study demonstrates that this decline is in fact a recoverable form of performance drift, not unique to ES. To mitigate this issue, the authors propose Anchored Weight Decay (AWD), a lightweight regularization method that stabilizes performance on old tasks by constraining the optimization trajectory to remain proximate to the initial model in parameter space. Experimental results show that AWD substantially alleviates forgetting without compromising performance on the target task, achieving stability comparable to that of large ES populations at minimal computational overhead, thereby affirming the viability of ES for continual learning.
0 citationsRead paper
Efficient Pre-Training of LLMs through Truncated SVD Layers
This work addresses the high computational cost of large language model (LLM) pretraining due to massive parameter counts, a challenge exacerbated by the inability of existing low-rank methods to simultaneously support dynamic rank selection and strict weight orthogonality. The authors propose the TSVD framework, which maintains a low-rank structure with orthogonality constraints throughout pretraining. It employs a spectral energy–based heuristic to adaptively determine the optimal rank and introduces a caching mechanism to efficiently preserve orthogonality. TSVD is the first method to concurrently enable dynamic low-rank adaptation and rigorous orthogonal constraints during training. Experiments demonstrate that it matches or exceeds full-parameter baseline performance across multiple model scales while substantially reducing computational overhead, establishing a new paradigm for efficient and scalable LLM pretraining.
0 citationsRead paper