StreamLens/ai.comprehensive_replay.md

StreamLens - Comprehensive Development Guide

Project Overview

Build a sophisticated Python-based network traffic analysis tool called "StreamLens" that analyzes both PCAP files and live network streams. The tool specializes in telemetry and avionics protocols with advanced statistical timing analysis, outlier detection, and sigma-based flow prioritization. Features a modern modular architecture with both a text-based user interface (TUI) and a professional PySide6 GUI with interactive matplotlib signal visualization.

Core Requirements

Primary Functionality

• PCAP Analysis: Load and analyze packet capture files with file validation and info extraction
• Live Capture: Real-time network traffic monitoring with threading support
• Flow Analysis: Group packets by source-destination IP pairs with comprehensive statistics
• Advanced Statistical Analysis: Configurable sigma thresholds, running averages, coefficient of variation
• Sigma-Based Outlier Detection: Identify packets with timing deviations using configurable thresholds (default 3σ)
• Flow Prioritization: Automatically sort flows by largest sigma deviation for efficient analysis
• Interactive TUI: Three-panel interface with real-time updates and navigation
• Modern GUI Interface: Professional PySide6-based GUI with embedded matplotlib plots
• Modular Architecture: Clean separation of concerns with analyzers, models, protocols, TUI, GUI, and utilities

Advanced Features

• Specialized Protocol Support: Chapter 10 (IRIG106), PTP (IEEE 1588), IENA (Airbus)
• Frame Type Classification: Hierarchical breakdown by protocol and frame types with per-type statistics
• Timeline Visualization: ASCII-based timing deviation charts with dynamic scaling
• Real-time Statistics: Running averages and outlier detection during live capture
• Comprehensive Reporting: Detailed outlier reports with sigma deviation calculations
• High Jitter Detection: Coefficient of variation analysis for identifying problematic flows
• Configurable Analysis: Adjustable outlier thresholds and analysis parameters
• Chapter 10 Signal Visualization: Real-time matplotlib-based signal plotting with TMATS integration
• Interactive Signal Analysis: Press 'v' in TUI to generate signal files, or view embedded plots in GUI table
• Threading-Safe Visualization: Main-thread plot creation for GUI, file output for TUI to prevent segmentation faults
• Embedded Plot Integration: Chapter 10 flows display signal plots directly in the flow table cells
• Cross-Platform GUI: PySide6-based interface with file dialogs, progress bars, and embedded matplotlib widgets

Architecture Overview

Core Components

1. Data Structures (dataclasses)

@dataclass
class FrameTypeStats:
    frame_type: str
    count: int
    total_bytes: int
    timestamps: List[float]
    frame_numbers: List[int]
    inter_arrival_times: List[float]
    avg_inter_arrival: float
    std_inter_arrival: float
    outlier_frames: List[int]
    outlier_details: List[Tuple[int, float]]

@dataclass
class FlowStats:
    src_ip: str
    dst_ip: str
    frame_count: int
    timestamps: List[float]
    frame_numbers: List[int]
    inter_arrival_times: List[float]
    avg_inter_arrival: float
    std_inter_arrival: float
    outlier_frames: List[int]
    outlier_details: List[Tuple[int, float]]
    total_bytes: int
    protocols: Set[str]
    detected_protocol_types: Set[str]
    frame_types: Dict[str, FrameTypeStats]

2. Main Analysis Engine (EthernetAnalyzer)

• Packet Processing: Single-packet and batch processing capabilities
• Flow Management: Track unique source-destination pairs
• Protocol Detection: Multi-layer protocol identification
• Statistical Calculation: Mean, standard deviation, outlier detection using 2σ threshold
• Live Capture: Threading support for real-time analysis

3. Enhanced Frame Dissection System

Base Dissector Interface

class EnhancedFrameDissector:
    def dissect_frame(self, packet: Packet, frame_num: int) -> Dict[str, Any]

Specialized Protocol Dissectors

Chapter 10 Dissector (Chapter10Dissector):

• IRIG106-17 telemetry standard support
• Sync pattern detection (0xEB25)
• Header parsing: channel ID, data types, timing counters
• Data format support: Analog, PCM, Ethernet Format 0
• Container format handling (embedded Ch10 in other protocols)

PTP Dissector (PTPDissector):

• IEEE 1588-2019 Precision Time Protocol
• Message types: Sync, Delay_Req, Announce, etc.
• Timestamp extraction and correction field parsing
• Grandmaster clock quality analysis

IENA Dissector (IENADissector):

• Airbus Improved Ethernet Network Architecture
• Packet types: P-type, D-type, N-type, M-type, Q-type
• Parameter ID extraction and delay analysis
• Variable-length message parsing

4. Chapter 10 Packet System (chapter10_packet.py)

Core Packet Class

class Chapter10Packet:
    def __init__(self, packet, original_frame_num: Optional[int] = None)
    def _parse_ch10_header(self) -> Optional[Dict]
    def get_data_payload(self) -> Optional[bytes]
    def decode_data(self, tmats_scaling_dict, tmats_content) -> Optional[DecodedData]

Data Decoders

• AnalogDecoder: Multi-channel analog data with scaling
• PCMDecoder: Pulse Code Modulation format
• AnalogTimeFormatDecoder: Time-synchronized analog data with TMATS integration

TMATS Integration

• Scaling parameter extraction from TMATS content
• Channel configuration parsing
• Engineering unit conversion
• Gain/offset/full-scale calculations

5. Text User Interface (TUIInterface)

Three-Panel Layout

┌─ FLOWS (Left 60%) ──┐│ DETAILS (Right 40%) ─┐
│ IP flows + frame    ││ Flow info + table    │
│ type breakdowns     ││ Frame type details   │
├────────────────────── ────────────────────────┤
│ TIMING VISUALIZATION (Bottom 30%)            │
│ ASCII timeline with deviation plotting       │
└───────────────────────────────────────────────┘

Key Interface Features

• Flow List: Hierarchical display of flows and frame types
• Detail Panel:
  • Flow summary (packets, bytes, protocols)
  • Frame type table: Type | #Pkts | Bytes | Avg ΔT | 2σ Out
  • Timing statistics and threshold calculations
• Timeline Panel: ASCII deviation chart with scaling
• Navigation: Arrow keys, view switching (main/dissection)

Technical Implementation Details

Statistical Analysis Algorithm

def calculate_statistics(self):
    for flow in self.flows.values():
        # Calculate inter-arrival times
        flow.avg_inter_arrival = statistics.mean(flow.inter_arrival_times)
        flow.std_inter_arrival = statistics.stdev(flow.inter_arrival_times)
        
        # Detect outliers (>2σ threshold)
        threshold = flow.avg_inter_arrival + (2 * flow.std_inter_arrival)
        for i, inter_time in enumerate(flow.inter_arrival_times):
            if inter_time > threshold:
                frame_number = flow.frame_numbers[i + 1]
                flow.outlier_frames.append(frame_number)
                flow.outlier_details.append((frame_number, inter_time))

Protocol Classification Hierarchy

1. Specialized Protocols: Chapter 10, PTP, IENA (highest priority)
2. Port-based Detection: DNS (53), HTTP (80), NTP (123), etc.
3. Transport Layer: TCP, UDP, ICMP, IGMP
4. Fallback: Generic classification

Frame Type Classification Logic

def _classify_frame_type(self, packet, dissection):
    layers = dissection.get('layers', {})
    
    # Chapter 10 takes precedence
    if 'chapter10' in layers:
        if self._is_tmats_frame(packet, layers['chapter10']):
            return 'TMATS'
        else:
            return 'CH10-Data'
    
    # PTP message types
    if 'ptp' in layers:
        msg_type = layers['ptp'].get('message_type_name', 'Unknown')
        return f'PTP-{msg_type}'
    
    # IENA packet types  
    if 'iena' in layers:
        packet_type = layers['iena'].get('packet_type_name', 'Unknown')
        return f'IENA-{packet_type}'
    
    # Fallback to transport/application protocols
    return self._classify_standard_protocols(packet)

Timeline Visualization Algorithm

• Deviation Calculation: deviation = inter_arrival_time - average_inter_arrival
• Scaling: Vertical position based on deviation magnitude
• Character Selection:
  • · for normal timing (within 10% of average)
  • •/○ for moderate deviations (within 50%)
  • █/▄ for significant outliers
• Time Mapping: Horizontal position maps to actual timestamps

Dependencies and Requirements

Core Libraries

• scapy: Packet capture and parsing (pip install scapy)
• numpy: Numerical computations (pip install numpy)
• matplotlib: Signal visualization and plotting (pip install matplotlib)
• PySide6: Modern Qt-based GUI framework (pip install PySide6)
• tkinter: GUI backend for matplotlib (usually included with Python)
• curses: Terminal UI framework (built-in on Unix systems)
• statistics: Statistical calculations (built-in)
• struct: Binary data parsing (built-in)
• threading: Live capture support (built-in)

Modern Modular File Structure

streamlens/
├── streamlens.py                    # Main entry point
├── analyzer/                        # Core analysis package
│   ├── __init__.py                  # Package initialization
│   ├── main.py                     # CLI handling and main application logic
│   ├── analysis/                   # Analysis engine modules
│   │   ├── __init__.py            # Analysis package init
│   │   ├── core.py                # EthernetAnalyzer main class
│   │   ├── flow_manager.py        # FlowManager for packet processing
│   │   └── statistics.py          # StatisticsEngine with sigma calculations
│   ├── models/                     # Data structures and models
│   │   ├── __init__.py            # Models package init
│   │   ├── flow_stats.py          # FlowStats and FrameTypeStats dataclasses
│   │   └── analysis_results.py    # AnalysisResult containers
│   ├── protocols/                  # Protocol dissector system
│   │   ├── __init__.py            # Protocols package init
│   │   ├── base.py                # ProtocolDissector base classes
│   │   ├── chapter10.py           # Chapter10Dissector (IRIG106)
│   │   ├── ptp.py                 # PTPDissector (IEEE 1588)
│   │   ├── iena.py                # IENADissector (Airbus)
│   │   └── standard.py            # StandardProtocolDissector
│   ├── gui/                        # Modern GUI Interface system with Embedded Plots
│   │   ├── __init__.py            # GUI package init
│   │   ├── main_window.py         # StreamLensMainWindow with PySide6 and docking system
│   │   └── dock_panels.py         # Dockable panel implementations (flow list, plots, details)
│   ├── tui/                        # Text User Interface system
│   │   ├── __init__.py            # TUI package init
│   │   ├── interface.py           # TUIInterface main controller
│   │   ├── navigation.py          # NavigationHandler for input handling
│   │   └── panels/                # UI panel components
│   │       ├── __init__.py        # Panels package init
│   │       ├── flow_list.py       # FlowListPanel for flow display
│   │       ├── detail_panel.py    # DetailPanel for flow details
│   │       └── timeline.py        # TimelinePanel for visualization
│   └── utils/                      # Utility modules
│       ├── __init__.py            # Utils package init
│       ├── pcap_loader.py         # PCAPLoader for file handling
│       ├── live_capture.py        # LiveCapture for network monitoring
│       └── signal_visualizer.py   # Chapter 10 signal visualization (thread-safe)
├── *.pcapng                        # Sample capture files for testing
├── README.md                       # User guide and quick start
└── ai.comprehensive_replay.md      # This comprehensive development guide

Command Line Interface

# Launch modern GUI with interactive plots (RECOMMENDED)
python streamlens.py --gui --pcap file.pcap

# GUI mode only (then open file via File menu)
python streamlens.py --gui

# Analyze PCAP file with TUI (flows sorted by largest sigma outliers)
python streamlens.py --pcap file.pcap

# Console output mode with sigma deviation display
python streamlens.py --pcap file.pcap --no-tui

# Generate comprehensive outlier report
python streamlens.py --pcap file.pcap --report

# Get PCAP file information only
python streamlens.py --pcap file.pcap --info

# Live capture with real-time statistics
python streamlens.py --live --interface eth0

# Configure outlier threshold (default: 3.0 sigma)
python streamlens.py --pcap file.pcap --outlier-threshold 2.0

# With BPF filtering for targeted capture
python streamlens.py --live --filter "port 319 or port 320"

Key Algorithms and Techniques

1. Inter-arrival Time Calculation

• Track timestamps for each packet in a flow
• Calculate time differences between consecutive packets
• Handle both flow-level and frame-type-level statistics

2. Advanced Outlier Detection and Flow Prioritization

• Use configurable sigma rule: outliers are packets with inter-arrival times > (mean + N×std_dev)
• Default threshold: 3σ (configurable via --outlier-threshold parameter)
• Calculate maximum sigma deviation for each flow across all outliers
• Store both frame numbers and actual time deltas for detailed analysis
• Apply at both flow and frame-type granularity
• Key Innovation: Sort flows by largest sigma deviation to prioritize most problematic flows

def get_max_sigma_deviation(self, flow: FlowStats) -> float:
    """Get the maximum sigma deviation for any outlier in this flow"""
    max_sigma = 0.0
    
    # Check flow-level outliers
    if flow.outlier_details and flow.std_inter_arrival > 0:
        for frame_num, inter_arrival_time in flow.outlier_details:
            sigma_deviation = (inter_arrival_time - flow.avg_inter_arrival) / flow.std_inter_arrival
            max_sigma = max(max_sigma, sigma_deviation)
    
    # Check frame-type-level outliers
    for frame_type, ft_stats in flow.frame_types.items():
        if ft_stats.outlier_details and ft_stats.std_inter_arrival > 0:
            for frame_num, inter_arrival_time in ft_stats.outlier_details:
                sigma_deviation = (inter_arrival_time - ft_stats.avg_inter_arrival) / ft_stats.std_inter_arrival
                max_sigma = max(max_sigma, sigma_deviation)
    
    return max_sigma

3. Protocol Dissection Pipeline

• Layer-by-layer parsing starting with Ethernet/IP/UDP
• Specialized protocol detection based on ports, patterns, payload analysis
• Error handling and graceful degradation for malformed packets

4. TMATS Parameter Parsing

• Line-by-line parsing of TMATS configuration data
• Key-value pair extraction with hierarchical channel mapping
• Parameter validation and default value assignment

5. Live Capture Architecture with Real-time Statistics

• Threaded packet capture using scapy.sniff()
• Real-time running statistical updates during capture
• Thread-safe data structures and stop conditions
• Running averages calculated incrementally for performance
• Live outlier detection with immediate flagging
• TUI updates every 0.5 seconds during live capture

6. Chapter 10 Signal Visualization System

• TMATS Parser: Extracts channel metadata from Telemetry Attributes Transfer Standard frames
• Signal Decoders: Support for analog and PCM format data with proper scaling
• Matplotlib Integration: External plotting windows with interactive capabilities
• Real-time Visualization: Works for both PCAP analysis and live capture modes
• Multi-channel Display: Simultaneous plotting of multiple signal channels with engineering units

class SignalVisualizer:
    def visualize_flow_signals(self, flow: FlowStats, packets: List[Packet]) -> None:
        # Extract TMATS metadata for channel configurations
        tmats_metadata = self._extract_tmats_from_flow(packets)
        
        # Decode signal data using Chapter 10 decoders
        signal_data = self._extract_signals_from_flow(packets, tmats_metadata)
        
        # Launch matplotlib window in background thread
        self._create_signal_window(flow_key, signal_data, flow)

7. PySide6 GUI Architecture with Embedded Plot Integration

• Professional Qt Interface: Cross-platform GUI built with PySide6 for native look and feel
• Embedded Matplotlib Integration: Signal plots rendered directly in table cells using FigureCanvas
• Main-Thread Plot Creation: All matplotlib widgets created in GUI main thread for Qt safety
• Background PCAP Processing: Threading only for data loading with progress bar and non-blocking UI
• Flow List Widget: Sortable table with sigma deviations, protocols, and embedded signal plots
• Synchronous Plot Rendering: Plots appear immediately when table populates, no background threads
• Threading Safety: Eliminated Qt threading violations by removing background plot generation
• No Floating Windows: All plots stay embedded in the table interface

class StreamLensMainWindow(QMainWindow):
    def __init__(self):
        # Create main interface with docking panels
        self.flow_list_dock = FlowListDockWidget(self)  # Flow table with embedded plots
        self.flow_detail_dock = FlowDetailDockWidget(self)  # Detail panel
        
    def on_pcap_loaded(self, analyzer):
        # Populate flows table with embedded plots (synchronous)
        self.flow_list_dock.populate_flows_table()  # Creates plots in main thread
        
class FlowListDockWidget(QWidget):
    def _create_integrated_plot_widget(self, flow, flow_key):
        # Create embedded matplotlib widget in main thread
        figure = Figure(figsize=(6, 2))
        canvas = FigureCanvas(figure)  # Qt widget created in main thread
        self._populate_integrated_plot(figure, canvas, flow, flow_key)
        return plot_widget

8. Embedded Plot Architecture (Recent Enhancement)

• Qt Threading Compliance: All matplotlib widgets created in main GUI thread
• Synchronous Plot Rendering: Plots appear immediately when table loads, no async threads
• FigureCanvas Integration: Matplotlib FigureCanvas widgets embedded directly in table cells
• No Floating Windows: Complete elimination of popup matplotlib windows
• Signal Caching: Processed signal data cached to avoid repeated extraction
• Main Thread Safety: Removed PlotGenerationThread to prevent Qt threading violations

def _create_integrated_plot_widget(self, flow: 'FlowStats', flow_key: str) -> QWidget:
    """Create matplotlib widget embedded in table cell"""
    plot_widget = QWidget()
    layout = QVBoxLayout()
    
    # Create figure and canvas in main thread
    figure = Figure(figsize=(6, 2))
    canvas = FigureCanvas(figure)  # Qt widget - must be in main thread
    canvas.setMaximumHeight(120)
    
    layout.addWidget(canvas)
    plot_widget.setLayout(layout)
    
    # Populate plot synchronously
    self._populate_integrated_plot(figure, canvas, flow, flow_key)
    return plot_widget

9. Modular Architecture Design

• Separation of Concerns: Clean boundaries between analysis, UI, protocols, and utilities
• Package Structure: Logical grouping of related functionality
• Dependency Injection: Components receive dependencies through constructors
• Interface-based Design: Protocol dissectors implement common interfaces
• Error Handling: Graceful degradation and comprehensive error reporting

Extensibility Points

Adding New Protocol Dissectors

1. Implement dissector class with dissect(packet) method
2. Return DissectionResult with protocol type and parsed fields
3. Register in EnhancedFrameDissector.dissectors dictionary

Custom Frame Type Classification

• Override _classify_frame_type() method
• Add custom logic based on dissection results
• Return descriptive frame type strings

Statistical Analysis Extensions

• Extend FrameTypeStats and FlowStats dataclasses
• Add custom statistical calculations in calculate_statistics()
• Implement additional outlier detection algorithms

Performance Considerations

• Memory Management: Use generators for large PCAP files
• Threading: Separate capture and analysis threads for live mode
• Statistical Efficiency: Incremental statistics calculation
• TUI Optimization: Efficient screen drawing with curses error handling

Testing and Validation

The project includes comprehensive test suites:

• Protocol Dissection Tests: Validate parsing accuracy
• Statistical Analysis Tests: Verify timing calculations
• TUI Layout Tests: Interface rendering validation
• Integration Tests: End-to-end workflow verification

Current Implementation Status (2025-01-26 - Latest Update)

✅ Fully Implemented Features

• Core Analysis Engine: Complete with flow tracking, statistical analysis, and outlier detection
• TUI Interface: Three-panel layout with navigation, timeline visualization, and protocol dissection
• GUI Interface: Professional PySide6 interface with docking panels and embedded plots
• Protocol Dissectors: Chapter 10, PTP, IENA, and standard protocol support
• Signal Visualization: Both TUI (file output) and GUI (embedded plots) working
• PCAP Loading: Background threading with progress bars
• Live Capture: Real-time analysis with network interface monitoring
• Statistical Reporting: Comprehensive outlier analysis and sigma-based flow prioritization
• Threading Safety: Proper Qt integration eliminates segmentation faults

🔧 Recent Improvements (Latest Session)

Phase 1: Embedded Plot Foundation

• Embedded Plot Integration: Removed floating windows, plots now embedded in flow table
• Qt Threading Compliance: Fixed threading violations by moving plot creation to main thread
• Synchronous Rendering: Plots appear immediately when table loads
• Matplotlib Configuration: Proper backend setup prevents unwanted popup windows
• Signal Visualizer Protection: GUI mode blocks floating window creation

Phase 2: Grid Refactoring

• Expanded Data Columns: Increased from 6 to 10 columns with comprehensive flow data
• User-Resizable Columns: All columns adjustable via QHeaderView.Interactive
• Dense Data Display: Added Max σ, Avg ΔT, Std ΔT, Outliers, and Protocols columns

Phase 3: Modern Dark Theme Implementation

• Professional Color Palette: Modern #1e1e1e backgrounds with optimized contrast ratios
• Comprehensive Dark Styling: Applied to tables, headers, scrollbars, menus, and toolbars
• Plot Theme Integration: Matplotlib plots styled with dark backgrounds and modern #0078d4 colors
• Color-Coded Data: Sigma values and outliers highlighted with modern red/amber colors

Phase 4: Layout Optimization (Final)

• Full-Width Utilization: Grid view now uses entire screen width effectively
• Prioritized Wide Plots: Increased from 6x2 to 8x2.5 figure size with 600px minimum column width
• 25% Taller Rows: Increased from 24px to 30px for better visual balance
• Content-Fitted Headers: Columns auto-resize to fit content, not unnecessarily wide
• Optimized Plot Margins: Reduced horizontal margins from 15%/95% to 8%/98% for maximum signal area

🚀 Recommended Usage

# Primary recommended workflow - Professional GUI with embedded plots
python streamlens.py --gui --pcap file.pcap

# Enhanced terminal-based analysis with professional table layout
python streamlens.py --pcap file.pcap

# Live network monitoring with real-time TUI updates
python streamlens.py --live --interface eth0

# Comprehensive reporting with sigma-based analysis
python streamlens.py --pcap file.pcap --report

📁 Project Structure Status

All major components implemented and working:

• ✅ analyzer/analysis/ - Core analysis engine
• ✅ analyzer/models/ - Data structures
• ✅ analyzer/protocols/ - Protocol dissectors
• ✅ analyzer/gui/ - Modern GUI with embedded plots
• ✅ analyzer/tui/ - Text-based interface
• ✅ analyzer/utils/ - Utilities and signal visualization

🎯 Key Technical Achievements

1. Sigma-Based Flow Prioritization: Automatically sorts flows by timing outliers
2. Optimized Embedded Matplotlib Integration: Wide plots (8x2.5) with minimal margins in Qt table cells
3. Modern Dark Theme UI: Professional #1e1e1e color scheme with optimal contrast and modern styling
4. Content-Adaptive Layout: Intelligent column sizing with full-width utilization and 25% taller rows
5. Multi-Protocol Support: Specialized dissectors for aviation/industrial protocols
6. Real-Time Analysis: Live capture with running statistics
7. Threading-Safe Architecture: Qt-compliant plot creation eliminates segmentation faults
8. Cross-Platform Professional GUI: PySide6-based interface with native look and feel

📊 Current GUI Specifications

• Table Layout: 10 columns with content-fitted headers and user resizing
• Row Height: 30px (25% increase) for improved visual balance
• Plot Dimensions: 8x2.5 matplotlib figures with 150px height and minimal margins
• Color Scheme: Modern dark theme (#1e1e1e) with #0078d4 accents and proper contrast
• Width Utilization: Full screen width with plot column priority (600px minimum + stretching)

🏆 Final Implementation Quality

The StreamLens GUI now represents a professional-grade network analysis interface with:

• Pixel-perfect dark theme styling across all components
• Optimized screen real estate usage with intelligent column sizing
• Wide, detailed signal visualizations embedded directly in the flow table
• Threading-safe architecture preventing crashes and UI blocking
• Modern user experience with instant plot rendering and responsive interactions

This comprehensive description captures the full scope and technical depth of the StreamLens Ethernet Traffic Analyzer, enabling recreation of this sophisticated telemetry analysis tool with its current state-of-the-art GUI implementation.