programming languages

This study systematically reviews 35 papers on LLM-assisted translation of natural language requirements into formal specifications (e.g., Dafny, C, Java), supporting VERIFAI’s goals of requirement traceability and formal verification. We propose the first application paradigm taxonomy for this task, identifying three core capabilities—syntactic translation, constraint completion, and error detection—and common bottlenecks including low accuracy and weak logical consistency. Methodologically, we integrate Elicit AI–assisted literature retrieval, cross-database validation, manual curation, and thematic coding analysis. Our contributions include identifying key research directions: enhancing model interpretability, establishing accuracy assurance mechanisms, and enabling domain adaptation. These insights provide both theoretical foundations and practical pathways for LLM-driven formal requirements engineering. (149 words)

This paper addresses the risk of large language models (LLMs) generating toxic content—such as discriminatory or aggressive text—in software engineering (SE) contexts. It presents the first systematic survey of toxicity detection and mitigation methods specifically for SE. The authors propose a domain-specific toxicity assessment framework comprising: (1) SE-oriented corpus preprocessing; (2) multi-dimensional human annotation; (3) a hybrid detection model integrating prompt engineering and fine-tuning; and (4) an LLM-driven rewriting technique for toxicity mitigation. Ablation experiments demonstrate that LLM-based rewriting reduces average text toxicity by 37.2%. The study identifies critical gaps in existing approaches—particularly in modeling SE-specific contextual semantics and enabling fine-grained toxicity attribution—and provides both theoretical foundations and practical guidelines for integrating responsible AI practices into software development workflows.

This study addresses the lack of empirical investigation into systematic coding-style discrepancies between large language models (LLMs) and human developers. We first propose a taxonomy of coding-style inconsistency and conduct multi-model comparative experiments across three dimensions: readability, conciseness, and robustness. Our methodology integrates expert annotation, quantitative style assessment, and code-quality metric design. Key contributions include: (1) the first principled taxonomy of coding-style inconsistency; (2) empirical identification of prevalent LLM-specific stylistic deviations—including redundant comments, excessive defensive programming, and non-idiomatic syntax; and (3) experimental validation that such inconsistencies significantly impair code maintainability and team collaboration efficiency, alongside actionable mitigation strategies. The work provides both theoretical foundations and practical guidelines for enhancing the engineering viability of LLM-generated code.

This work addresses the challenges of IDE development posed by the rapid evolution of smart contract languages such as Move by presenting a high-performance IDE support system built atop the Move compiler and adhering to the Language Server Protocol (LSP). Through deep integration with existing language toolchains and the application of incremental parsing and optimized semantic analysis techniques, the system efficiently delivers rich IDE features even as the language undergoes continuous iteration. Deployed successfully within the Sui platform’s Move ecosystem, it significantly enhances developer experience and yields a reusable, evolution-aware IDE construction strategy applicable to other emerging programming language ecosystems.

Existing LLM code evaluation methodologies rely excessively on synthetic benchmarks and neglect real-world development practices. Method: We propose the first developer-empirical evaluation framework, featuring a unified, automated platform for multi-model integration and execution validation, complemented by a mixed-methods survey across 60 software practitioners from 11 countries. Contribution/Results: Our systematic assessment quantifies LLM-generated code across functionality, syntactic correctness, and engineering practicality. We identify critical capability gaps—including API invocation, contextual modeling, and error recovery—and establish empirically grounded mappings between model capabilities and engineering deployability. The findings provide actionable, evidence-based guidance for LLM selection, prompt engineering optimization, and IDE/toolchain integration in industrial settings.

This work investigates the persistent value of syntax-aware representations in billion-parameter large language models (LLMs) for code generation. While syntactic errors have markedly declined in ultra-large LLMs, the utility of explicit syntactic information remains questionable. To address this, we propose GrammarCoder—a family of models that explicitly integrate programming language grammar via context-free grammar (CFG)-guided decoding, syntax-constrained token prediction, and a Transformer-adapted syntax embedding enhancement module. Our study provides the first empirical evidence that syntactic information continues to significantly improve semantic discrimination—not merely syntactic correctness—in ultra-large LLMs, effectively mitigating semantic errors induced by minor code perturbations. On HumanEval+ and MBPP+, GrammarCoder achieves substantial accuracy gains; syntax error rates approach zero, and semantic error rates decrease by 12.7%.

Existing LLM code-generation evaluations rely heavily on controlled benchmarks (e.g., HumanEval), which poorly reflect real-world development practices. Method: This paper presents the first large-scale empirical study of code generated by ChatGPT and GitHub Copilot on GitHub, integrating repository crawling, language identification, commit-history tracing, complexity measurement, and pattern mining to characterize distribution, evolution, and maintenance properties. Contributions/Results: (1) Quantifies low real-world adoption: LLM-generated code constitutes <1.2% of total codebase volume on average; only 3–8% of such code undergoes modification for bug fixes; and generation is heavily skewed toward Python, Java, and TypeScript. (2) Reveals that associated projects tend to be small-scale, actively evolving, yet critically deficient in documentation. (3) Bridges the gap between controlled benchmarking and engineering practice, establishing an empirical foundation for assessing LLM code trustworthiness and guiding tool optimization.

Existing log generation methods are predominantly evaluated in monolingual settings, leaving their cross-lingual effectiveness unclear. This work constructs a multilingual logging benchmark comprising 150,000 instances across five programming languages and presents the first systematic evaluation of three state-of-the-art approaches and five large language models in cross-lingual log generation. The study reveals significant disparities in logging difficulty across languages, attributed to language-specific logging insertion patterns and idioms, underscoring the need for tailored multilingual logging strategies. Experimental results demonstrate that UniLog achieves the strongest overall performance, with JavaScript proving more amenable to log generation than Python, which poses greater challenges. Moreover, merely scaling model size or training data yields limited gains in multilingual logging efficacy.

Existing benchmarks primarily assess language models on localized programming tasks, failing to capture their capability to construct complete software systems from scratch. This work introduces ProgramBench, the first end-to-end evaluation framework grounded in behavioral equivalence, which requires agents to autonomously design and implement full codebases based solely on program specifications and documentation, with correctness verified through behavioral test suites. The benchmark encompasses 200 real-world software tasks—including CLI tools, FFmpeg, and SQLite—supports unconstrained, open-ended code generation, and incorporates agent-driven fuzz testing to automatically produce behavioral test cases. Evaluation across nine prominent language models reveals that none can fully solve any task; the best-performing model passes 95% of tests on only 3% of tasks and tends to generate single-file implementations structurally divergent from human-written code.

This work addresses a critical limitation of large language models (LLMs) in generating executable code that is compatible with users’ specific software environments, as existing evaluations typically assume isolated or default settings. The paper formally defines and investigates the task of Environment-Aware Code Generation (EACG) for the first time, proposing a tri-axis adaptation strategy that jointly aligns model behavior along data, parameter, and cache dimensions. To support comprehensive evaluation, the authors introduce VersiBCB, the first benchmark enabling multi-package dependency resolution, execution validation, and analysis of deprecation issues. Experimental results demonstrate that the proposed approach substantially improves the compatibility and executability of generated code in real-world heterogeneous environments, revealing fundamental gaps in current LLMs’ capacity for environment awareness.

This work addresses the lack of correctness guarantees in existing text-to-code generation methods, which still rely heavily on manual inspection and thus hinder development efficiency. To bridge this gap, the paper introduces verifiable formal assertions into the code generation process for the first time: a large language model simultaneously produces C code and candidate assertions, which are then jointly verified using a bounded model checker. This approach provides partial correctness guarantees and enhances code interpretability. Experiments across 18 programming tasks demonstrate that the method efficiently generates code accompanied by verifiable assertions. Furthermore, a user study involving over 400 participants confirms that these assertions significantly improve developers’ understanding of the generated code.