ICX360: In-Context eXplainability 360 Toolkit

2025-11-14

AI Summary

To address the limited interpretability of large language model (LLM) outputs, this paper introduces ICX360—the first open-source Python explanation toolkit that unifies black-box (perturbation-based analysis) and white-box (gradient computation and context-sensitive attribution) paradigms. Methodologically, ICX360 supports multi-granularity attribution across high-stakes scenarios including retrieval-augmented generation (RAG), natural language generation (NLG), and jailbreaking attacks. Its key contribution lies in the first unified framework enabling synergistic modeling and plug-and-play adaptation of both explanation paradigms. The toolkit provides a quick-start guide, comprehensive tutorials, and modular APIs. Open-sourced on GitHub, ICX360 has been empirically validated on multiple representative LLM tasks, demonstrating significant improvements in decision-path transparency and user controllability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLMs) have become ubiquitous in everyday life and are entering higher-stakes applications ranging from summarizing meeting transcripts to answering doctors'questions. As was the case with earlier predictive models, it is crucial that we develop tools for explaining the output of LLMs, be it a summary, list, response to a question, etc. With these needs in mind, we introduce In-Context Explainability 360 (ICX360), an open-source Python toolkit for explaining LLMs with a focus on the user-provided context (or prompts in general) that are fed to the LLMs. ICX360 contains implementations for three recent tools that explain LLMs using both black-box and white-box methods (via perturbations and gradients respectively). The toolkit, available at https://github.com/IBM/ICX360, contains quick-start guidance materials as well as detailed tutorials covering use cases such as retrieval augmented generation, natural language generation, and jailbreaking.

Problem

• Developing tools to explain LLM outputs like summaries and responses
• Providing open-source toolkit for explaining LLMs using context and prompts
• Implementing black-box and white-box explanation methods for LLMs

Innovation

• Open-source toolkit for explaining LLM outputs
• Combines black-box and white-box explanation methods
• Focuses on user-provided context and prompts