StreamLens/README.md

StreamLens - Ethernet Traffic Analyzer

Advanced network traffic analyzer for pcap files and live streams with specialized protocol dissection for aviation and industrial networks. Features sigma-based outlier identification, real-time statistical analysis, and both TUI and modern GUI interfaces with interactive signal visualization.

Quick Start

# Install dependencies  
pip install scapy numpy matplotlib

# For GUI mode (optional but recommended):
pip install PySide6

# For macOS users - install tkinter support for TUI visualization:
brew install python-tk@3.13

# Launch modern GUI with interactive plots
python streamlens.py --gui --pcap file.pcap

# GUI mode only (then open file via File menu)
python streamlens.py --gui

# Analyze pcap file with modern TUI (Flow Analysis, Packet Decoder, Statistical Analysis views)
python streamlens.py --pcap file.pcap

# Use classic TUI interface instead of modern (preserves original layout)
python streamlens.py --pcap file.pcap --classic

# Live capture with real-time statistics
python streamlens.py --live --interface eth0

# Console output with outlier reporting  
python streamlens.py --pcap file.pcap --no-tui

# Generate comprehensive outlier report
python streamlens.py --pcap file.pcap --report

# Get pcap file information
python streamlens.py --pcap file.pcap --info

# Adjust outlier threshold (default: 3.0 sigma)
python streamlens.py --pcap file.pcap --outlier-threshold 2.0

# With BPF filter for live capture
python streamlens.py --live --filter "port 319 or port 320"

Features

🖥️ Modern Dark-Themed GUI Interface with Optimized Layout

• Professional Dark Theme: Modern color palette with #1e1e1e backgrounds and optimized contrast
• Content-Fitted Columns: Headers automatically resize to fit content, not wider than necessary
• Full-Width Utilization: Grid view uses entire screen width with prioritized wide signal plots
• Optimized Row Height: 25% taller rows (30px) for better visual balance and plot visibility
• Wide Embedded Plots: 8x2.5 figure size with minimal horizontal margins for maximum signal detail
• Intelligent Column Sizing: Auto-resizes to content with smart minimums and plot column priority
• Professional Qt Interface: Cross-platform GUI built with PySide6 with native look and feel
• Embedded Signal Plots: Chapter 10 signal plots rendered directly in the flow table cells
• Synchronous Plot Rendering: Plots appear immediately when table loads, no background threads
• Chapter 10 Flow Highlighting: Flows with Chapter 10 data are highlighted in modern blue and bold
• Smart Signal Caching: Avoids repeated processing of the same flow's signal data
• Flow Detail Panel: Dockable bottom panel with dark theme styling
• Background PCAP Loading: Progress bar with non-blocking file processing
• Outlier Threshold Control: Real-time adjustment of sigma-based outlier detection
• Threading Safety: Main-thread plot creation eliminates Qt threading violations
• No Floating Windows: All plots stay embedded in the grid interface

🖥️ Modern TUI Interface (Default) with Three Focused Views

• 1: Flow Analysis View: Enhanced multi-column flow overview with protocol hierarchy
  • Source | Proto | Destination | Extended | Frame Type | Metrics layout
  • Transport protocols (TCP, UDP, ICMP, IGMP) clearly separated from extended protocols
  • Extended protocol column for specialized protocols (CH10, PTP, IENA, NTP)
  • Frame type column showing most common frame type per flow (CH10-Data, TMATS, PTP Sync)
  • Left-aligned text columns with IP:port format for precise endpoint identification
  • Performance rankings by packet count, outliers, and enhanced decoder availability
• 2: Packet Decoder View: Deep protocol inspection and field extraction
  • Three-panel layout: Enhanced Flows | Frame Analysis | Field Inspector
  • Real-time decoded field display with tree-view navigation
  • Tab-based interface switching with comprehensive field value inspection
• 3: Statistical Analysis View: Timing analysis, outliers, and quality metrics
  • Four analysis modes: Overview, Outlier Analysis, Quality Metrics, Timing Analysis
  • Performance ranking with health metrics and network consistency indicators
  • Detailed outlier breakdown with sigma deviation calculations
• Modern Navigation: 1/2/3 view switching with context-sensitive help and status bars
• Enhanced Protocol Support: Specialized views for Chapter 10, PTP, IENA with quality indicators
• Cross-View Communication: Selected flows persist across view switches for comprehensive analysis

📊 Classic TUI Interface (--classic flag) with Professional Table Layout

• Optimized Three-Panel Layout: Flows list (70% width), flow details (30% width), optional timeline (bottom)
• Professional Table Formatting: Right-aligned numeric columns (#Frames, Bytes, ΔT Avg) with proper spacing
• Comprehensive Flow Display: Shows Src:Port, Dst:Port, Transport Protocol, Traffic Classification, and Encoding
• Transport Layer Analysis: Displays TCP, UDP, ICMP, IGMP protocols with port information
• Traffic Classification: Identifies Unicast, Multicast, and Broadcast traffic patterns
• Hierarchical Frame Types: Expandable tree view showing packet type breakdowns with aligned sub-rows
• Magnitude Indicators: Consistent byte formatting (1.2M, 428K, 1234B) with right alignment
• Sigma-Based Flow Sorting: Flows automatically sorted by largest outlier sigma deviation
• Real-time Navigation: Arrow keys to navigate between flows with instant detail updates
• Smart Protocol Detection: Prioritizes specialized protocols (Chapter 10, PTP, IENA) over generic ones
• Visual Timeline: ASCII timeline showing frame timing deviations with outlier highlighting
• Live Statistics: Real-time running averages and outlier detection during capture

Core Analysis Engine

• Flow-based Analysis: Groups packets by source-destination IP pairs with timing statistics
• Configurable Outlier Detection: Adjustable sigma threshold (default: 3.0σ)
• Multi-layer Protocol Analysis: Ethernet, IP, UDP, TCP with specialized dissectors
• Real-time Statistical Updates: Running statistics for live capture mode
• High Jitter Flow Identification: Coefficient of variation analysis

Specialized Protocol Dissectors

• Chapter 10 (IRIG 106-17): Complete packet dissection including data types, timestamps, and payload analysis
• PTP (IEEE 1588-2019): Precision Time Protocol message parsing with sync, delay, and announce messages
• IENA (Airbus): Industrial Ethernet Network Architecture with P/D/N/M/Q message types

📊 Chapter 10 Signal Visualization with Dark Theme Integration

• Wide Embedded GUI Plots: Chapter 10 flows display matplotlib plots directly in flow table with 8x2.5 sizing
• Dark Theme Plot Integration: Plots use #1e1e1e backgrounds with white text and modern #0078d4 signal colors
• Optimized Plot Margins: Minimal horizontal margins (8% left, 98% right) for maximum signal visualization area
• TUI Signal Plots: Press v in the TUI to generate signal files (threading-safe)
• Signal Consolidation: Automatically combines multiple packets from the same channel into continuous signals
• TMATS Integration: Automatically extracts channel metadata from TMATS frames for proper signal scaling
• Multi-channel Support: Displays multiple channels with proper engineering units and scaling
• Threading Safety: GUI uses main-thread plot creation, TUI saves plots to files to avoid segfaults
• No Floating Windows: All GUI plots stay embedded in the table interface
• Both Modes: Works for both PCAP analysis and live capture
• Enhanced Visual Quality: 150px plot height with professional styling and grid overlays

Protocol Detection & Fallbacks

• Automatic protocol identification based on port numbers and packet structure
• Fallback to common protocols: HTTP, HTTPS, SSH, DNS, DHCP, NTP, SNMP, IGMP, ICMP
• Multicast detection for aviation/industrial networks
• Enhanced error handling and validation

Installation

# Clone or download the project
cd streamlens

# Install dependencies  
pip install scapy numpy matplotlib PySide6

# Run the analyzer
python streamlens.py --help

Key Features Highlights

🎯 Sigma-Based Flow Prioritization

Flows are automatically sorted by their largest outlier sigma deviation, putting the most problematic flows at the top of the list for immediate attention.

📊 Real-time Statistics

Live capture mode provides running averages and outlier detection as packets arrive, with TUI updates every 500ms.

🔍 Configurable Analysis

Adjust outlier detection sensitivity with --outlier-threshold (default: 3.0σ) to fine-tune analysis for your specific network conditions.

📈 Comprehensive Reporting

Generate detailed outlier reports with --report flag showing frame-by-frame sigma deviations and timing analysis.

GUI Usage

Main Interface

• Menu Bar: File operations (Open PCAP, Monitor NIC), View controls, Help system
• Toolbar: File operations and outlier threshold adjustment
• Central Flow Table: Full-width table with file info, flow data, and integrated signal plots
• Flow Detail Panel: Dockable bottom panel showing comprehensive flow information
• Status Bar: Loading progress and operation feedback

Workflow

GUI Mode (Recommended)

1. Launch GUI with PCAP: python streamlens.py --gui --pcap file.pcap
2. Immediate Analysis: Flow table displays instantly with all flow data and wide embedded plots
3. Optimized Display: Content-fitted columns, 25% taller rows, and full-width utilization
4. Wide Plot Visualization: Chapter 10 flows show detailed signal plots with minimal margins
5. Browse Flows: View flows in the dark-themed table (Chapter 10 flows highlighted in modern blue)
6. Analyze Details: Select flows to view detailed information in the dark-themed bottom panel
7. Adjust Threshold: Use toolbar spinner to change outlier detection sensitivity

Modern TUI Mode (Default)

1. Launch Modern TUI: python streamlens.py --pcap file.pcap
2. Flow Analysis View (1): Visual flow overview with protocol detection and performance ranking
3. Packet Decoder View (2): Deep packet inspection with three-panel layout for field analysis
4. Statistical Analysis View (3): Comprehensive timing analysis and outlier detection
5. View Navigation: Use 1/2/3 to switch between analysis perspectives
6. Context-Sensitive Help: Press H for detailed help overlay with all controls
7. Enhanced Protocol Analysis: Specialized displays for Chapter 10, PTP, IENA protocols

Classic TUI Mode (--classic flag)

1. Launch Classic TUI: python streamlens.py --pcap file.pcap --classic
2. Professional Table View: Right-aligned numeric columns with transport protocol and classification
3. Navigate Flows: Use ↑↓ to browse flows sorted by sigma deviation
4. Expand Details: Use → to show frame type breakdowns with hierarchical display
5. Signal Visualization: Press 'v' on Chapter 10 flows to generate signal plot files
6. Timeline Analysis: Press 't' to toggle timing visualization panel
7. Live Monitoring: Real-time statistics updates during network capture

TUI Controls

Modern TUI Controls (Default)

• 1: Switch to Flow Analysis View (enhanced multi-column layout)
• 2: Switch to Packet Decoder View (three-panel inspection)
• 3: Switch to Statistical Analysis View (timing and quality analysis)
• H: Toggle comprehensive help overlay
• ↑↓: Navigate items in current view
• Enter: Select flow/packet for detailed analysis
• Tab: Switch panels (when available)
• V: Visualize signals (Flow Analysis)
• D: Deep decode selected flow
• E: Export decoded data
• R: Refresh statistics
• O: Show outlier details
• Q: Quit application

Classic TUI Controls (--classic flag)

• ↑↓: Navigate between flows and frame types in main view
• →: Expand flow to show frame type breakdowns
• ←: Collapse flow details
• v: Visualize Chapter 10 signals for selected flow (saves plot files)
• t: Toggle timeline panel on/off
• d: Switch to frame dissection view
• m or ESC: Return to main view
• q: Quit application

Timeline Visualization

The bottom panel displays a visual timeline of the selected flow's timing behavior:

• Horizontal axis: Progression through packet sequence
• Vertical axis: Deviation from average inter-arrival time (centered on average)
• Characters: · = normal timing, •/○ = moderate deviation, █/▄ = outliers
• Scale: Automatically adjusts to show full range of deviations
• Info bar: Shows total frames, deviation range, and outlier count

Project Structure

streamlens/
├── streamlens.py                    # Main entry point
├── analyzer/                        # Core analysis package
│   ├── main.py                     # CLI argument handling and main logic
│   ├── analysis/                   # Analysis engine
│   │   ├── core.py                # Main analyzer class
│   │   ├── flow_manager.py        # Flow tracking and management
│   │   └── statistics.py          # Statistical analysis and outlier detection
│   ├── models/                     # Data structures
│   │   ├── flow_stats.py          # Flow and frame type statistics
│   │   └── analysis_results.py    # Analysis result containers
│   ├── protocols/                  # Protocol dissectors
│   │   ├── base.py                # Base dissector interface
│   │   ├── chapter10.py           # IRIG106 telemetry protocol
│   │   ├── ptp.py                 # IEEE 1588 Precision Time Protocol
│   │   ├── iena.py                # Airbus IENA protocol
│   │   └── standard.py            # Standard protocol detection
│   ├── gui/                        # Modern GUI Interface with Docking Panels
│   │   ├── __init__.py            # GUI package initialization
│   │   ├── main_window.py         # PySide6 main window with docking system
│   │   └── dock_panels.py         # Dockable panel implementations (flow list, plots, details)
│   ├── tui/                        # Text User Interface
│   │   ├── interface.py           # Classic TUI controller
│   │   ├── modern_interface.py    # Modern TUI with three-view interface
│   │   ├── navigation.py          # Navigation handling
│   │   ├── modern_views/          # Modern TUI view controllers
│   │   │   ├── flow_analysis.py   # Flow Analysis View (F1)
│   │   │   ├── packet_decoder.py  # Packet Decoder View (F2)
│   │   │   └── statistical_analysis.py # Statistical Analysis View (F3)
│   │   └── panels/                # Classic TUI panel components
│   │       ├── flow_list.py       # Flow list panel
│   │       ├── detail_panel.py    # Flow details panel
│   │       └── timeline.py        # Timeline visualization panel
│   └── utils/                      # Utility modules
│       ├── pcap_loader.py         # PCAP file handling
│       ├── live_capture.py        # Live network capture
│       └── signal_visualizer.py   # Chapter 10 signal visualization (thread-safe)
└── *.pcapng                        # Sample capture files