minimum description length

Neural networks such as Transformers lack theoretically grounded measures of model complexity, hindering principled model selection and compression. Method: Grounded in the Minimum Description Length (MDL) principle and Kolmogorov complexity, we establish the first asymptotically optimal description length objective for Transformers and construct the first MDL framework with computational universality guarantees. We propose a differentiable, optimization-friendly variational objective using an adaptive Gaussian mixture prior to approximate MDL. Contribution/Results: This work introduces the first theoretically sound, Transformer-specific MDL-based complexity measure. Empirical evaluation confirms that the proposed objective favors low-complexity models with strong generalization performance. However, it also exposes a critical practical limitation: standard optimizers struggle to converge from random initialization. Overall, our framework provides a novel information-theoretic foundation for model selection and compression in deep learning, bridging theoretical guarantees with practical neural architecture design.

Classical Minimum Description Length (MDL) theory relies on exact optimization, yet practical models can only approximately optimize the objective, leading to a gap between theory and practice. This work presents the first systematic study of predictive reliability under additive approximation errors and regularization in approximate MDL. We introduce a balanced MDL objective and combine additive slack analysis, affinity-telescoping arguments, and a likelihood-ratio-based stopping-time technique. We prove that when the regularization strength λ ≥ 1, the cumulative prediction error remains bounded; conversely, if λ < 1, overfitting is inevitable and multiplicative approximation becomes infeasible. Our results demonstrate that classical MDL is robust to any fixed additive optimization error, and this condition is theoretically tight.

This work proposes a novel approach that actively embeds the Minimum Description Length (MDL) principle into the optimization process of deep learning to jointly enhance model simplicity and generalization. By constructing a cognitive manifold driven by coupled Ricci flows, the method introduces a geometry-driven MDL Drive mechanism that dynamically compresses internal representations during training, balancing fidelity and complexity. Theoretical analysis establishes that this mechanism guarantees a monotonically decreasing description length, undergoes a finite number of topological phase transitions, and exhibits universal critical behavior. With a per-iteration computational complexity of O(N log N), the algorithm automatically simplifies model architecture, improves generalization, and demonstrates numerical stability alongside exponential convergence in empirical evaluations.

This work addresses the lack of systematic methodologies in model optimization, which often relies on heuristic choices and struggles to accommodate diverse deployment constraints. It formalizes model compression and acceleration as a constraint-aware multi-objective engineering decision problem, establishing a unified and actionable framework grounded in five key dimensions: data availability, latency, memory footprint, accuracy tolerance, and retraining budget. By integrating techniques such as quantization, pruning, knowledge distillation, parameter-efficient fine-tuning (PEFT), and inference optimization, the study proposes tailored optimization pipelines for four representative industrial scenarios, delivering a reproducible and quantifiable guide for technology selection.

The academic positioning of low-code development relative to classical model-driven development remains ambiguous, and the relationship between their respective research communities lacks systematic clarification. Method: This paper conducts the first meta-scientific study, integrating bibliometric analysis, author-venue-topic network modeling, and cross-community comparative analysis to quantitatively characterize the low-code community’s scale, disciplinary diversity, publication venue distribution, and scholarly output characteristics—and to systematically compare them with those of the classical model-driven development community. Contribution/Results: We find that the low-code community exhibits strong interdisciplinarity and conference-centric publication patterns, and has significantly diverged from traditional modeling communities. These findings provide empirical grounding for conceptualizing low-code as an independent research trajectory, reveal opportunities for disciplinary integration, and identify critical interfaces for collaborative innovation—thereby informing the reconfiguration and convergence of the broader modeling research community.

This work investigates the implicit structural representation capability of large language models (LLMs) for combinatorial optimization problems and its utility for downstream decision-making tasks—specifically solver selection. Methodologically, we systematically analyze how hidden layers of LLMs encode problem structure across four benchmark optimization problems and three instance encodings, employing both direct prompting and neuron probing techniques. Our key contributions are threefold: First, we provide the first empirical evidence that intermediate LLM layers capture optimization problem structures highly aligned with classical hand-crafted features. Second, these implicit representations achieve solver recommendation accuracy on par with traditional feature-engineering approaches at the instance level. Third, the representations exhibit strong cross-problem generalization robustness. Collectively, these findings offer new insights into the internal mechanisms of LLMs in symbolic reasoning tasks and advance their trustworthy deployment in operations research and optimization.

This work proposes the notion of “representational complexity” grounded in algorithmic information theory to quantify the redundant information inherent in describing dynamical systems through structures and rules, and establishes a lower bound on their minimal description length. By reframing the choice of network representation as a trade-off between informational cost and mechanistic transparency, the study reveals—for the first time—the informational equivalence between graphs and hypergraphs under constrained modeling paradigms, along with the conditions under which one representation is preferred over the other. Integrating Kolmogorov complexity theory with computable description-length estimation techniques, this research delineates the applicability boundaries of distinct network formalisms under scientific modeling constraints and provides a practical framework for estimating representational complexity.

This study investigates whether large language models (LLMs) underperform in generating domain-specific languages—such as AMPL for algebraic modeling—compared to general-purpose programming languages like Python, specifically within mathematical optimization contexts. To address this, the authors propose EXEOS, a method that leverages LLMs to translate natural language descriptions into either AMPL or Python code, augmented with a solver-feedback-driven iterative refinement mechanism to enhance executability and correctness. The first systematic comparison of its kind demonstrates that, across public benchmarks and real-world Kinaxis supply chain cases, LLM-generated AMPL code matches or even surpasses Python in quality. These findings affirm the competitiveness of domain-specific languages in specialized optimization tasks and highlight the critical role of solver-in-the-loop iterative refinement in improving the generation of formal specifications.

Conventional static core-set selection fails to adapt to the heterogeneous requirements across different training stages. Method: This paper proposes a dynamic multi-objective adaptive core-set selection framework that dynamically switches sampling strategies according to training progression—emphasizing class balance in early stages, feature diversity in mid-stages, and prediction uncertainty in late stages—thereby enabling the first training-process-aware, multi-objective co-optimization. Contribution/Results: We theoretically establish a (1−1/e)-approximation guarantee. By integrating submodular optimization, active learning, and representation analysis, our method achieves O(n log n) computational efficiency. Empirically, it attains full-dataset accuracy on multiple benchmarks while significantly reducing memory overhead. Moreover, it is the first work to quantitatively characterize the dynamic evolution of data utility throughout training.

This study addresses the limitations of traditional Design Structure Matrix (DSM) modularization approaches, which rely solely on graph-based optimization and lack engineering semantic context, often failing to align with practical design requirements. The authors propose a novel DSM modularization paradigm integrating large language models (LLMs), leveraging prompt engineering and iterative refinement to embed system-level semantic information directly into the partitioning process—achieving high-quality results without custom optimization code. Central to this work is the "semantic alignment hypothesis," which elucidates how improper incorporation of domain knowledge can degrade performance. Through systematic experiments across five representative engineering cases using three mainstream LLMs, the method demonstrates convergence to reference-quality modularization within 30 iterations, offering a reproducible and practical pathway for LLM-driven engineering design optimization.