Human or LLM? A Comparative Study on Accessible Code Generation Capability

2025-03-20

AI Summary

Large language models (LLMs) are increasingly used for web development, yet their generated code’s accessibility compliance—particularly regarding color contrast, alternative text, and ARIA attributes—remains poorly understood and unquantified relative to human-authored code. Method: We systematically evaluate GPT-4o and Qwen2.5-Coder across accessibility dimensions and propose FeedA11y, a feedback-driven ReAct framework integrating multi-turn self-critique, zero/few-shot prompting, and axe-core–based automated validation to overcome limitations of static prompting. Contribution/Results: Our evaluation reveals that while LLMs surpass human performance on foundational accessibility (e.g., GPT-4o achieves 91.4% color contrast compliance), they lag significantly on semantic aspects like ARIA conformance. FeedA11y improves ARIA compliance by 37.2% and exceeds human benchmarks across all core accessibility metrics. This work is the first to empirically characterize the hierarchical accessibility capabilities of LLMs and demonstrate closed-loop, automation-augmented optimization for accessible code generation.

Technology Category

• Humans and AI: AI for Accessibility
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Web accessibility is essential for inclusive digital experiences, yet the accessibility of LLM-generated code remains underexplored. This paper presents an empirical study comparing the accessibility of web code generated by GPT-4o and Qwen2.5-Coder-32B-Instruct-AWQ against human-written code. Results show that LLMs often produce more accessible code, especially for basic features like color contrast and alternative text, but struggle with complex issues such as ARIA attributes. We also assess advanced prompting strategies (Zero-Shot, Few-Shot, Self-Criticism), finding they offer some gains but are limited. To address these gaps, we introduce FeedA11y, a feedback-driven ReAct-based approach that significantly outperforms other methods in improving accessibility. Our work highlights the promise of LLMs for accessible code generation and emphasizes the need for feedback-based techniques to address persistent challenges.

Problem

• Compare accessibility of LLM-generated vs human-written web code.
• Evaluate advanced prompting strategies for improving code accessibility.
• Introduce FeedA11y, a feedback-driven approach to enhance accessibility.

Innovation

• Empirical comparison of LLM and human code accessibility
• Advanced prompting strategies for accessibility improvements
• FeedA11y: Feedback-driven ReAct-based accessibility enhancement