Selected work

Representative Papers

Joint Optimization for Greedy Longest-match Tokenization

Jul 25, 2026

This work addresses the suboptimality of traditional subword segmentation methods like Byte Pair Encoding (BPE), which rely on heuristic rules and ignore the greedy longest-match decoding strategy used during inference, leading to inefficient compression. The paper formulates vocabulary learning as an integer programming problem that jointly optimizes token selection and segmentation decisions, introducing a greedy consistency constraint to align training and deployment tokenization. To enable scalable optimization, the authors propose a linear programming relaxation approach that facilitates end-to-end training tailored for greedy decoding. Experiments show that, with 32K and 64K vocabularies, their method reduces token counts by up to 0.78% compared to BPE, closing 89.6%–99.4% of the gap to the theoretical optimal compression, while achieving solutions within only 0.008%–0.176% of the LP lower bound.

0 citationsRead paper

POTATR: A Lightweight Image-to-Graph Model for Page-Level Table Extraction

Jun 08, 2026

Existing lightweight table extraction methods struggle to simultaneously achieve high accuracy, computational efficiency, and contextual awareness. This work proposes POTATR, a lightweight image-to-graph model built upon an extended Table Transformer architecture, which for the first time attains state-of-the-art performance with only 29 million parameters—surpassing large multimodal models. POTATR incorporates page-level contextual awareness, spatial bounding box localization, and supports external OCR integration as well as cross-page table merging. Evaluated on the PubTables-v2 single-page benchmark, it achieves a GriTS_Con score of 0.964 while offering over 130× faster inference and approximately 300× lower computational cost compared to leading alternatives.

0 citationsRead paper

Tokenization with Split Trees

May 21, 2026

This work addresses limitations in traditional subword tokenization methods regarding compression efficiency and contextual utilization. It proposes a novel segmentation approach based on binary partition trees, formulating tokenization as a recursive search problem over such trees. By precomputing byte n-gram frequencies to construct a full binary tree and leveraging integer programming—along with its linear programming relaxation—the method efficiently derives a near-optimal vocabulary tailored for compression-rate optimization through recursive inference. Evaluated on English text, the approach reduces token counts by over 11% compared to baselines like BPE, yielding substantial gains in Rényi efficiency. In training 1.5B-parameter language models, it achieves CORE score improvements of 2.6%–7.6% and outperforms competitors on 13 out of 22 benchmark tasks.

0 citationsRead paper

Tokenisation via Convex Relaxations

May 21, 2026

This work addresses a fundamental limitation of existing subword tokenization algorithms—such as BPE and Unigram—which rely on greedy strategies that optimize only local objectives and thus struggle to approach global optimality. The paper presents the first formalization of tokenizer construction as a linear programming problem and introduces ConvexTok, a novel algorithm based on convex optimization. By leveraging convex relaxation and optimization techniques, ConvexTok enables globally optimal vocabulary selection. The method provably approximates the theoretical optimum with less than 1% error under typical vocabulary sizes, consistently outperforms baseline tokenizers in segmentation quality and language model bits-per-byte (BpB), and yields measurable improvements in downstream task performance.

0 citationsRead paper

Not How Many, But Which: Parameter Placement in Low-Rank Adaptation

May 12, 2026

This study investigates how to select trainable parameters under a fixed parameter budget in Low-Rank Adaptation (LoRA) to maximize model performance. It reveals that the effectiveness of parameter selection strategies is highly dependent on the training paradigm: under supervised fine-tuning, random and gradient-guided selection perform comparably, whereas only gradient-guided selection yields gains in GRPO-based reinforcement learning fine-tuning. Building on this insight, the authors propose an efficient scoring method that leverages gradient structure to rapidly identify critical parameters. This approach requires less than 0.5% of the full training cost and completes selection within ten seconds, consistently improving performance across models ranging from 1.5B to 8B parameters. The identified critical parameters are predominantly concentrated in the value (V), output (O), and down-projection matrices.

0 citationsRead paper

Recent publications

Latest Papers

Joint Optimization for Greedy Longest-match Tokenization

Jul 25, 2026

This work addresses the suboptimality of traditional subword segmentation methods like Byte Pair Encoding (BPE), which rely on heuristic rules and ignore the greedy longest-match decoding strategy used during inference, leading to inefficient compression. The paper formulates vocabulary learning as an integer programming problem that jointly optimizes token selection and segmentation decisions, introducing a greedy consistency constraint to align training and deployment tokenization. To enable scalable optimization, the authors propose a linear programming relaxation approach that facilitates end-to-end training tailored for greedy decoding. Experiments show that, with 32K and 64K vocabularies, their method reduces token counts by up to 0.78% compared to BPE, closing 89.6%–99.4% of the gap to the theoretical optimal compression, while achieving solutions within only 0.008%–0.176% of the LP lower bound.

0 citationsRead paper

POTATR: A Lightweight Image-to-Graph Model for Page-Level Table Extraction

Jun 08, 2026

Existing lightweight table extraction methods struggle to simultaneously achieve high accuracy, computational efficiency, and contextual awareness. This work proposes POTATR, a lightweight image-to-graph model built upon an extended Table Transformer architecture, which for the first time attains state-of-the-art performance with only 29 million parameters—surpassing large multimodal models. POTATR incorporates page-level contextual awareness, spatial bounding box localization, and supports external OCR integration as well as cross-page table merging. Evaluated on the PubTables-v2 single-page benchmark, it achieves a GriTS_Con score of 0.964 while offering over 130× faster inference and approximately 300× lower computational cost compared to leading alternatives.

0 citationsRead paper

Tokenization with Split Trees

May 21, 2026

This work addresses limitations in traditional subword tokenization methods regarding compression efficiency and contextual utilization. It proposes a novel segmentation approach based on binary partition trees, formulating tokenization as a recursive search problem over such trees. By precomputing byte n-gram frequencies to construct a full binary tree and leveraging integer programming—along with its linear programming relaxation—the method efficiently derives a near-optimal vocabulary tailored for compression-rate optimization through recursive inference. Evaluated on English text, the approach reduces token counts by over 11% compared to baselines like BPE, yielding substantial gains in Rényi efficiency. In training 1.5B-parameter language models, it achieves CORE score improvements of 2.6%–7.6% and outperforms competitors on 13 out of 22 benchmark tasks.

0 citationsRead paper

Tokenisation via Convex Relaxations

May 21, 2026

This work addresses a fundamental limitation of existing subword tokenization algorithms—such as BPE and Unigram—which rely on greedy strategies that optimize only local objectives and thus struggle to approach global optimality. The paper presents the first formalization of tokenizer construction as a linear programming problem and introduces ConvexTok, a novel algorithm based on convex optimization. By leveraging convex relaxation and optimization techniques, ConvexTok enables globally optimal vocabulary selection. The method provably approximates the theoretical optimum with less than 1% error under typical vocabulary sizes, consistently outperforms baseline tokenizers in segmentation quality and language model bits-per-byte (BpB), and yields measurable improvements in downstream task performance.

0 citationsRead paper

Not How Many, But Which: Parameter Placement in Low-Rank Adaptation

May 12, 2026

This study investigates how to select trainable parameters under a fixed parameter budget in Low-Rank Adaptation (LoRA) to maximize model performance. It reveals that the effectiveness of parameter selection strategies is highly dependent on the training paradigm: under supervised fine-tuning, random and gradient-guided selection perform comparably, whereas only gradient-guided selection yields gains in GRPO-based reinforcement learning fine-tuning. Building on this insight, the authors propose an efficient scoring method that leverages gradient structure to rapidly identify critical parameters. This approach requires less than 0.5% of the full training cost and completes selection within ten seconds, consistently improving performance across models ranging from 1.5B to 8B parameters. The identified critical parameters are predominantly concentrated in the value (V), output (O), and down-projection matrices.

0 citationsRead paper