Real-Time Stream Processor

Open Source

A high-performance stream processing engine written in Rust that guarantees exactly-once semantics while maintaining sub-millisecond latency.

Source Code

Problem

Existing solutions either sacrificed correctness for speed or required complex infrastructure. I wanted something simple, fast, and correct.

Architecture

Event-driven architecture with a custom WAL (Write-Ahead Log) implementation. Uses Kafka for ingestion and Redis for state management.

Challenges

Implementing exactly-once semantics in a distributed setting without a heavy coordinator. Solved using idempotent consumers and deterministic processing.

Lessons Learned

Rust's ownership model forces you to think about concurrency correctly from the start. This prevented an entire class of bugs I would have had in other languages.

Tech Stack

RustApache KafkaRedisgRPCTokioProtocol Buffers

Details

Timeline2024

StatusOpen Source