SUBstrayt

SUBstrayt is a modular open-source library designed to perform fast substring search and text retrieval over large collections. It aims to be embeddable in other software and to support multi-language text through Unicode, with an emphasis on predictable latency, scalable indexing, and a flexible query interface.

The architecture of SUBstrayt centers on an indexing layer that blends suffix-based structures—such as suffix arrays and suffix trees—with inverted indexing. This combination enables efficient substring queries across extensive datasets. The system typically separates indexing, query planning, and result execution, allowing for different backends and optimizations to be swapped in as needed.

Query processing in SUBstrayt supports exact substring matches, pattern-based queries with wildcards, and approximate matching using distance thresholds. It can process streaming input and provides ranking of results by relevance or other criteria. The query planner selects appropriate index structures and execution strategies to optimize for locality and throughput, adapting to the size and characteristics of the corpus.

Use cases for SUBstrayt span digital archives, code search, log analysis, large-scale literature collections, and even biological sequence search where substring patterns are informative. While multiple implementations and bindings may exist in the ecosystem, the design emphasizes a language-agnostic core with pluggable backends and bindings for common programming environments.

In discussion and documentation, SUBstrayt is presented as a conceptual design for efficient substring search rather than a single, official product. Related concepts include substring search, suffix arrays, suffix trees, and text indexing.