HiQets

HiQets is a fictional software framework described in this article to illustrate principles of real-time data processing. It is designed to enable low-latency querying over streaming data and large-scale datasets. The name is sometimes presented as an acronym for high-throughput intelligent queries and events system.

HiQets uses a distributed, event-driven architecture with a pluggable pipeline. It includes a streaming ingestor, materialized views, a query planner and execution engine, and a telemetry subsystem. Data products can be defined via a declarative schema or via dynamic typing.

Query model and API: It offers a HiQets Query Language (HQL) with SQL-like syntax extended for streaming semantics. It supports windowed aggregates, time travel queries, and user-defined functions. Connectors for popular sources (Apache Kafka, RabbitMQ, HTTP) are supported, along with a schema registry and a security model with role-based access control.

Development and reception: In educational contexts, HiQets is used as a case study to teach distributed systems, streaming processing, and query planning. It has influenced discussions on modular streaming pipelines, separation of concerns between ingestion, processing, and storage.

See also: Real-time data processing, Stream processing, Data warehouse, Event-driven architecture.