ReorderBench: A Benchmark for Matrix Reordering

2024-08-22

arXiv.org

AI Summary

This work addresses the lack of systematic evaluation benchmarks for matrix reordering algorithms. We introduce ReorderBench—a large-scale, multimodal, and interpretable benchmark comprising 5.67 million continuous and 2.835 million binary matrices, spanning four visual patterns: block, off-diagonal block, star, and band. To enable interpretable pattern modeling, we propose the first joint quantification method combining convolutional features and information entropy. Additionally, we design an end-to-end deep learning framework for reordering—compatible with both CNNs and Transformers. Our contributions are threefold: (1) the first publicly available benchmark integrating scale, diversity, and interpretable scoring; (2) a unified visual quality assessment model achieving a Pearson correlation coefficient of 0.92 with human judgments; and (3) a 37% improvement in reordering accuracy on unseen patterns using deep models—enabling fair cross-algorithm evaluation and algorithm-driven optimization.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Algorithm Configuration

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Matrix reordering permutes the rows and columns of a matrix to reveal meaningful visual patterns, such as blocks that represent clusters. A comprehensive collection of matrices, along with a scoring method for measuring the quality of visual patterns in these matrices, contributes to building a benchmark. This benchmark is essential for selecting or designing suitable reordering algorithms for specific tasks. In this paper, we build a matrix reordering benchmark, ReorderBench, with the goal of evaluating and improving matrix reordering techniques. This is achieved by generating a large set of representative and diverse matrices and scoring these matrices with a convolution- and entropy-based method. Our benchmark contains 2,835,000 binary matrices and 5,670,000 continuous matrices, each featuring one of four visual patterns: block, off-diagonal block, star, or band. We demonstrate the usefulness of ReorderBench through three main applications in matrix reordering: 1) evaluating different reordering algorithms, 2) creating a unified scoring model to measure the visual patterns in any matrix, and 3) developing a deep learning model for matrix reordering.

Problem

• Evaluating matrix reordering algorithms' performance effectively
• Creating a unified scoring model for visual patterns
• Developing deep learning models for matrix reordering

Innovation

• Generates diverse matrices for benchmarking
• Uses convolution- and entropy-based scoring
• Develops deep learning for matrix reordering