POS-ISP: Pipeline Optimization at the Sequence Level for Task-aware ISP

2026-04-08

AI Summary

This work addresses key challenges in task-aware image signal processing (ISP) optimization—namely, training-inference inconsistency, unstable training dynamics, and high computational overhead. To overcome these limitations, the authors formulate ISP pipeline optimization as a global sequence prediction problem and introduce a sequence-level reinforcement learning framework. This framework enables end-to-end prediction of both the optimal module ordering and their corresponding parameters in a single forward pass, directly optimizing for downstream task performance as the reward signal. By eliminating intermediate supervision and avoiding redundant module execution, the proposed method achieves significantly improved task accuracy while reducing computational cost, thereby enabling more stable and efficient ISP design.

Abstract

Recent work has explored optimizing image signal processing (ISP) pipelines for various tasks by composing predefined modules and adapting them to task-specific objectives. However, jointly optimizing module sequences and parameters remains challenging. Existing approaches rely on neural architecture search (NAS) or step-wise reinforcement learning (RL), but NAS suffers from a training-inference mismatch, while step-wise RL leads to unstable training and high computational overhead due to stage-wise decision-making. We propose POS-ISP, a sequence-level RL framework that formulates modular ISP optimization as a global sequence prediction problem. Our method predicts the entire module sequence and its parameters in a single forward pass and optimizes the pipeline using a terminal task reward, eliminating the need for intermediate supervision and redundant executions. Experiments across multiple downstream tasks show that POS-ISP improves task performance while reducing computational cost, highlighting sequence-level optimization as a stable and efficient paradigm for task-aware ISP. The project page is available at https://w1jyun.github.io/POS-ISP

Problem

• ISP optimization
• module sequence
• task-aware
• pipeline optimization
• sequence-level

Innovation

• sequence-level optimization
• task-aware ISP
• reinforcement learning
• modular pipeline
• global sequence prediction