Queue Entropy Analysis

Concurrent producer–consumer pipeline with backpressure, validated using Shannon entropy over SPY ticks.
Tech Stack: C++17 Concurrent Systems Market Microstructure Producer–Consumer Backpressure
Source Code

Summary

I built a concurrent producer–consumer pipeline for high‑frequency market data, centered on a dual‑mutex optimized queue with atomic size tracking and batch pops. The system feeds a Shannon entropy calculator over discretized trader actions (BUY/HOLD/SELL) to prove end‑to‑end correctness and stability under realistic load.

On a realistic SPY simulation (base ~$695.42, ±0.02% random walk), the pipeline reported 1.12164 bits (medium entropy) with no backpressure and zero dropped ticks in the test run. This project demonstrates systems engineering for low‑latency pipelines and a behavioral measurement framework, not a trading strategy.

Impact / Why It Matters

  • Decoupled ingestion & analysis – Heavy entropy updates no longer block tick ingestion, reducing the risk of missed events during volatile periods.
  • Stable under bursty load – Backpressure at 90% capacity prevents memory bloat and keeps the pipeline controllable when producers outpace consumers.
  • Lower contention – Dual‑mutex queue reduces single‑lock bottlenecks and deadlock risk compared to naive single‑lock designs.
  • Research‑ready foundation – Thread‑safe, low‑allocation hot path for future HFT benchmarking and real order‑flow integration.

What I Built

  • OptimizedQueue (dual‑mutex) – Separate head/tail locks, atomic size counter, batch pops, and 90% capacity backpressure.
  • Concurrent Market Pipeline – Producers never wait for entropy math; consumers pull batches and update a sliding entropy window.
  • Thread‑Safety Primitives – thread_local producer state, mutex‑protected snapshots returned by value, atomic metrics for latency/throughput.
  • Shannon Entropy Engine – Base‑2 entropy over 3 action states; range 0.0–1.585 bits, unit‑tested against known distributions.
  • Adaptive Sliding Window – Dynamically scales 50–500 observations based on entropy change rate to balance noise filtering and responsiveness.

Results & Evidence

Correctness & stability:

  • Unit + simulation + pipeline + queue edge tests: 23/23 passed.
  • Edge cases: empty windows, single‑action windows, uniform distributions.
  • Theoretical maximum entropy (~1.585 bits) confirmed.

Live SPY pipeline run:

1
2
3
4
5

SPY Live: $692.42 (0.01%)
Entropy: 1.12164 bits (Medium regime)
Queue size: 0
Processed events: 8
Backpressure: Not triggered

Performance note:

  • Repo includes a short synthetic micro‑benchmark (e.g., ~5M packets/sec on some machines).
  • This is not a sustained throughput claim; real performance needs dedicated benchmarking on target hardware.

Tech & Methods

  • Language: C++17 (+ pthread)
  • Queue: Dual‑mutex OptimizedQueue + standard ConcurrentQueue
  • Concurrency:
    • thread_local producer state
    • Mutex‑protected snapshots returned by value
    • Atomic metrics with lock‑free telemetry
  • Entropy: ( H = -\sum_i P(x_i)\log_2 P(x_i) ), 3‑state system (BUY/HOLD/SELL)
  • Data: Simulated live SPY feed (±0.02% random walk)
  • Window: Adaptive 50–500 events, driven by entropy change rate

Limits & Next Steps

Validated:

  • Correct queue behavior (push/pop, batch ops, backpressure)
  • Thread‑safety under concurrent producers/consumers
  • End‑to‑end pipeline stability with entropy computation

Not (yet) validated:

  • Sustained throughput / latency on real exchange feeds
  • Whether lock‑free queues are necessary for target hardware
  • Long‑horizon entropy–volatility correlation on real order flow

Next steps:

  • Real exchange order‑flow integration
  • Extended benchmarks on HFT‑class hardware
  • Time‑series analysis of entropy vs. realized volatility
concurrent-queue market-pipeline c++-systems hft-infrastructure backpressure