ai copilots

Prior work lacks a systematic understanding of how input parameters—such as prompt design, temperature, number of candidate solutions, and context—affect code generation in language models, hindering their reliable deployment. Method: This paper conducts the first controlled experiments on GitHub Copilot and OpenAI Codex, establishing a reproducible parameter perturbation framework grounded in HumanEval and LeetCode benchmarks. Contribution/Results: We empirically uncover strong, nonlinear couplings among temperature, prompt formulation, and candidate count—demonstrating that optimizing any single parameter in isolation is ineffective and that joint parameter tuning is essential. This challenges conventional manual hyperparameter tuning and provides both theoretical grounding and empirical evidence for automated parameter optimization. Experimental results show substantial correctness improvements under coordinated tuning; however, optimal configurations exhibit high sensitivity to parameter changes, underscoring the necessity of systematic, holistic parameter control.

This study investigates the impact of highly autonomous AI coding agents (e.g., OpenHands) versus conventional programming assistants (e.g., GitHub Copilot) on developer workflows, efficiency, and experience. Method: Employing a human factors experimental design, we conduct the first empirical comparison of these two paradigms in realistic development settings, integrating behavioral telemetry, task performance metrics, and subjective user feedback for multidimensional analysis. Results: Coding agents significantly reduce user interaction overhead and enable end-to-end task automation, thereby expanding the scope of tasks delegable to AI. However, they introduce adoption barriers—including intent opacity and behavioral unpredictability. Contribution: We empirically demonstrate that increased automation precipitates a fundamental shift in human–AI collaboration paradigms and derive evidence-based interaction design principles specifically for highly autonomous AI systems.

Automating in-app assistants powered by large language models (LLMs) in feature-rich software poses challenges in balancing automation with user control, learnability, and task adaptability. Method: This paper comparatively evaluates fully automated (AutoCopilot) versus semi-automated, guidance-driven (GuidedCopilot) paradigms, introducing a task- and interface-state-aware enhancement mechanism—comprising contextual preview snippets and adaptive instruction generation—and implementing a dual-paradigm prototype integrating visual step-by-step guidance with dynamic instruction execution. Contribution/Results: A user study (N=20) provides the first empirical evidence that GuidedCopilot significantly outperforms AutoCopilot in exploratory and creative tasks, achieving superior outcomes across perceived control, practical utility, and learnability. A follow-up design exploration (N=10) confirms that the proposed enhancements further improve interaction depth and contextual adaptability.

Deploying cloud-native large language models (LLMs) in Apple’s sandboxed iOS/macOS environments is challenging due to strict security constraints, network latency, and bandwidth limitations. Method: This paper proposes a local-cloud collaborative multi-model programming framework. Its core innovations include (1) a lightweight, sandbox-aware RAG-driven context-aware prompting mechanism that dynamically constructs minimal, semantically relevant contexts via on-device vector retrieval, drastically reducing data transmission overhead; and (2) an Xcode plugin-based architecture integrated with adaptive multi-model scheduling to enable low-latency, high-security remote LLM invocation. Results: Experiments demonstrate significant improvements over baselines: +19.3% CodeBLEU score for code generation quality and +42 points in Net Promoter Score (NPS) for developer experience. The framework has been deployed in production as “Copilot for Xcode,” establishing a reusable technical paradigm for AI-powered programming tools on resource- and security-constrained platforms.

This study addresses the limitation of large language models (LLMs) in software engineering—namely, their suboptimal response quality due to insufficient project-specific context (e.g., objectives, architecture, and collaboration norms). It presents the first systematic empirical investigation of machine-readable contextual instructions authored by developers in open-source projects, exemplified by Cursor rules. Using a novel taxonomy comprising five thematic categories (e.g., Conventions, Guidelines), the authors conduct qualitative coding and cross-repository comparison across 401 open-source repositories containing such rules. The analysis uncovers organizational patterns and cross-project and cross-language variation in contextual instruction usage. As a key contribution, the study proposes the first structured taxonomy of developer-authored contextual instructions in open-source settings. This taxonomy provides foundational design principles and empirical grounding for developing context-aware AI programming tools.

Current AI coding assistants (e.g., GitHub Copilot) rely on static context, failing to adapt to developers’ real-time cognitive states—leading to overly generic prompts and excessive code refactoring. This work introduces the first dynamic prompt optimization method grounded in real-time eye-tracking feedback: it integrates gaze metrics—including fixation trajectories and pupil dilation—to construct an attention-aware closed-loop interaction system that delivers personalized code suggestions and demand-driven refactoring. Our core contribution lies in incorporating fine-grained cognitive state modeling into prompt engineering, thereby avoiding one-size-fits-all interventions. Empirical evaluation demonstrates that our approach significantly improves code comprehension accuracy (+23.6%), reduces comprehension time (−31.4%), and enhances subjective readability ratings (p < 0.01), outperforming both standard Copilot and predefined-strategy baselines.

This study investigates the diverse needs and challenges that arise from cognitive differences among developers when interacting with AI-powered programming assistants. Employing a mixed-methods approach, the research integrates think-aloud protocols and quantitative analyses from 27 professional developers and students to systematically uncover how cognitive diversity influences human-AI collaborative programming. The work identifies five distinct interaction patterns and ten core user requirements, and proposes a conceptual model that elucidates how problem-solving styles and prior experience shape interaction behaviors. This framework offers both theoretical grounding and practical guidance for the design, research, and deployment of programming assistants that account for individual cognitive differences.

This study investigates behavioral differences and influencing factors in how engineering students utilize GitHub Copilot during authentic open-source contributions. Through an embedded course-based empirical design, the research systematically analyzes students’ usage patterns and subjective evaluations of Copilot’s multimodal features—including chat interaction, code generation, comment-driven suggestions, and repository-aware recommendations—by integrating survey responses with real task data. Findings indicate that the chat and code generation functionalities are the most preferred, and that gender, programming proficiency, and familiarity with AI significantly affect both usage frequency and perceived usefulness. This work provides the first empirical evidence of differential adoption behaviors among engineering students engaging with AI-powered programming assistants in real-world open-source contexts, offering a foundational understanding of human–AI collaboration in engineering education.

This study addresses context engineering for AI agents in open-source projects: specifically, how to structure project-specific knowledge—such as architecture, interfaces, and coding conventions—to improve code generation quality. We conduct the first systematic empirical analysis of 466 open-source AI configuration files (e.g., AGENTS.md), augmented with version-history tracking to characterize their evolution. Our findings reveal that current configurations lack standardized structure, exhibit high syntactic and semantic heterogeneity, and evolve via incremental expansion and collaborative maintenance. Our contributions are threefold: (1) a characterization of structural diversity and evolutionary patterns in real-world AI configuration files; (2) identification of key design dimensions governing contextual effectiveness; and (3) a novel paradigm that links structural optimization of context to generation quality improvement—providing empirical grounding and actionable design guidelines for standardizing AI-ready open-source infrastructure.

Current prompt engineering practices in AI programming assistants lack systematicity and struggle to effectively integrate requirements engineering principles, resulting in a gap between user intent and code implementation. This work introduces a requirements engineering perspective into prompt design, proposing a conceptual “prompt triplet” model that treats prompts as lightweight, evolvable artifacts integrating functional and quality requirements, general solution strategies, and concrete implementation details. Through conceptual modeling, analysis of real-world prompt corpora, dataset construction, and controlled experiments, the study provides preliminary validation of the three proposed components and formulates four testable hypotheses concerning prompt evolution, user variability, requirements validation, and code quality. These contributions lay an empirical foundation for advancing prompt engineering toward a more disciplined, requirements-driven paradigm.