Summary
Shannon Entropy Market Analysis – Phase 2 applies information theory to live intraday data (Feb 2026), quantifying trader‑behavior complexity via Shannon entropy (0.0–1.585 bits for 3 action types: buy/hold/sell).
Key observation: In this dataset, lower entropy tends to coincide with higher short‑term volatility; one stress episode at 0.599 bits preceded 4.999+ volatility spikes. The rolling correlation is r = −0.193, indicating a weak negative association in this sample.
This is a behavioral measurement framework validated end‑to‑end on real market data, not a trading strategy.
Impact / Why It Matters
- New lens on market stress – Entropy provides a structural, information‑theoretic view of trader behavior beyond raw price/volume.
- Regime detection – Distinct entropy/volatility profiles for stress, normal volatile, and quiet periods in this sample.
- End‑to‑end pipeline – From live data to 3‑panel visualizations, showing that the framework can run on real intraday feeds.
- Foundation for ML – Clean features (entropy, regime labels) for future semi‑supervised / unsupervised models and RL regime classifiers.
What I Built
- Live pipeline – Data ingestion, processing, and 3‑panel visualization (correlation, time‑series, summary stats).
- Entropy engine – Theoretical maximum 1.585 bits (3 trader actions) confirmed; observed range ~0.599 (stress) to ~1.50 (normal).
- Rolling‑window analysis – 100‑period entropy vs 10‑period volatility (price std dev).
- Regime signatures – Distinct entropy/volatility profiles for stress, normal volatile, and quiet periods in this sample.
- Visualization – 3‑panel PNG outputs (scatter, entropy/volatility series, stats).
Results & Evidence
Live pipeline (Feb 2026):
- Records processed: 100% success
- Entropy range: 0.000–1.50 bits
- Volatility range: 0.000–191.424
- Correlation: ρ = −0.193 (negative, weak)
- Stress episode: 0.599 bits → 4.999+ volatility
Mathematical checks:
- Theoretical maximum entropy confirmed (1.585 bits for 3 equiprobable actions).
- Edge cases validated: no variance and uniform distributions.
- 60 market windows: stress / normal / volatile regimes distinguishable by entropy/volatility profiles.
Tech & Methods
- Language: C++17 core + Python visualization
- Data: Live SPY CSV (c, h, l, o, dp, t)
- Entropy: Shannon ( H = -\sum p_i \log_2 p_i ), 100‑period rolling
- Volatility: Price standard deviation, 10‑period rolling
- Visualization: 3‑panel PNG (scatter, entropy/volatility series, stats)
- Correlation: Pearson ( \rho = -0.193 )
Limits & Next Steps
Validated:
- Live pipeline operational
- Entropy/volatility regime signatures identified in‑sample
- Weak negative correlation (r = −0.193) observed
Not (yet) validated:
- Out‑of‑sample robustness across instruments and regimes
- Predictive performance for crash detection in production
- Integration with real exchange order‑flow and low‑latency execution
Next steps:
- Broader datasets (multiple symbols, longer horizons)
- Real exchange order‑flow integration for finer‑grained entropy
- Semi‑supervised / unsupervised models + RL for regime classification
Links
- Repo: https://github.com/Devjosef/Shannon-Entropy
- Dataset: Kaggle – “Shannon Entropy Market Simulation (60 windows)”
- Notebook: Kaggle – “Notebook for Shannon Dataset”