StreamLens/README.md

# StreamLens - Ethernet Traffic Analyzer

Advanced network traffic analyzer for pcap files and live streams with specialized protocol dissection for aviation and industrial networks. Features sigma-based outlier identification, real-time statistical analysis, and both TUI and modern GUI interfaces with interactive signal visualization.

## Quick Start

```bash
# Install dependencies  
pip install scapy numpy matplotlib

# For GUI mode (optional but recommended):
pip install PySide6

# For macOS users - install tkinter support for TUI visualization:
brew install python-tk@3.13

# Launch modern GUI with interactive plots
python streamlens.py --gui --pcap file.pcap

# GUI mode only (then open file via File menu)
python streamlens.py --gui

# Analyze pcap file with TUI (flows sorted by largest sigma outliers)
python streamlens.py --pcap file.pcap

# Live capture with real-time statistics
python streamlens.py --live --interface eth0

# Console output with outlier reporting  
python streamlens.py --pcap file.pcap --no-tui

# Generate comprehensive outlier report
python streamlens.py --pcap file.pcap --report

# Get pcap file information
python streamlens.py --pcap file.pcap --info

# Adjust outlier threshold (default: 3.0 sigma)
python streamlens.py --pcap file.pcap --outlier-threshold 2.0

# With BPF filter for live capture
python streamlens.py --live --filter "port 319 or port 320"
```

## Features

### 🖥️ Modern GUI Interface (New!)
- **Professional Qt Interface**: Cross-platform GUI built with PySide6
- **Interactive Flow List**: Sortable table showing flows with sigma deviations, protocols, and frame types
- **Automatic Plot Rendering**: Click any flow to instantly view signal plots (no button needed)
- **Embedded Matplotlib Plots**: Interactive signal visualization with zoom, pan, and navigation toolbar
- **Background PCAP Loading**: Progress bar with non-blocking file processing
- **File Management**: Open PCAP files via dialog or command line
- **Smart Status Feedback**: Color-coded status messages for different flow types and states
- **Threading Safety**: Proper Qt threading eliminates segmentation faults

### Enhanced TUI Interface
- **Three-Panel Layout**: Flows list (top-left), flow details (top-right), timing visualization (bottom)
- **Sigma-Based Flow Sorting**: Flows automatically sorted by largest outlier sigma deviation
- **Real-time Navigation**: Arrow keys to navigate between flows with instant detail updates
- **Protocol-aware Display**: Shows detected protocols in flow list and details
- **Smart Protocol Detection**: Prioritizes specialized protocols (Chapter 10, PTP, IENA) over generic ones
- **Detailed Outlier Analysis**: Individual rows showing frame numbers and exact time deltas for outlier packets
- **Visual Timeline**: ASCII timeline showing frame timing deviations with outlier highlighting
- **Live Statistics**: Real-time running averages and outlier detection during capture

### Core Analysis Engine
- **Flow-based Analysis**: Groups packets by source-destination IP pairs with timing statistics
- **Configurable Outlier Detection**: Adjustable sigma threshold (default: 3.0σ)
- **Multi-layer Protocol Analysis**: Ethernet, IP, UDP, TCP with specialized dissectors
- **Real-time Statistical Updates**: Running statistics for live capture mode
- **High Jitter Flow Identification**: Coefficient of variation analysis

### Specialized Protocol Dissectors
- **Chapter 10 (IRIG 106-17)**: Complete packet dissection including data types, timestamps, and payload analysis
- **PTP (IEEE 1588-2019)**: Precision Time Protocol message parsing with sync, delay, and announce messages  
- **IENA (Airbus)**: Industrial Ethernet Network Architecture with P/D/N/M/Q message types

### 📊 Chapter 10 Signal Visualization
- **Interactive GUI Plots**: Select any flow to automatically view embedded matplotlib plots
- **TUI Signal Plots**: Press `v` in the TUI to generate signal files (threading-safe)
- **Signal Consolidation**: Automatically combines multiple packets from the same channel into continuous signals
- **TMATS Integration**: Automatically extracts channel metadata from TMATS frames for proper signal scaling
- **Multi-channel Support**: Displays multiple channels with proper engineering units and scaling
- **Threading Safety**: GUI uses proper Qt integration, TUI saves plots to files to avoid segfaults
- **Both Modes**: Works for both PCAP analysis and live capture
- **Matplotlib Features**: Full zoom, pan, save, and navigation capabilities

### Protocol Detection & Fallbacks
- Automatic protocol identification based on port numbers and packet structure
- Fallback to common protocols: HTTP, HTTPS, SSH, DNS, DHCP, NTP, SNMP, IGMP, ICMP
- Multicast detection for aviation/industrial networks
- Enhanced error handling and validation

## Installation

```bash
# Clone or download the project
cd streamlens

# Install dependencies  
pip install scapy numpy matplotlib PySide6

# Run the analyzer
python streamlens.py --help
```

## Key Features Highlights

### 🎯 Sigma-Based Flow Prioritization
Flows are automatically sorted by their largest outlier sigma deviation, putting the most problematic flows at the top of the list for immediate attention.

### 📊 Real-time Statistics
Live capture mode provides running averages and outlier detection as packets arrive, with TUI updates every 500ms.

### 🔍 Configurable Analysis
Adjust outlier detection sensitivity with `--outlier-threshold` (default: 3.0σ) to fine-tune analysis for your specific network conditions.

### 📈 Comprehensive Reporting
Generate detailed outlier reports with `--report` flag showing frame-by-frame sigma deviations and timing analysis.

## GUI Usage

### Main Interface
- **Left Panel**: File information and flow list sorted by sigma deviation
- **Right Panel**: Interactive matplotlib plot area with navigation toolbar
- **Status Bar**: Loading progress and operation feedback

### Workflow
1. **Launch GUI**: `python streamlens.py --gui`
2. **Open PCAP**: File → Open PCAP... or use command line `--pcap` flag
3. **Select Flow**: Click on any flow in the table to automatically view signal plots
4. **Interact**: Use matplotlib toolbar to zoom, pan, save plots
5. **Navigate**: Click different flows to instantly see their signal visualizations

## TUI Controls

- **↑↓**: Navigate between flows in main view
- **v**: Visualize Chapter 10 signals for selected flow (saves plot files)
- **t**: Toggle timeline panel on/off
- **d**: Switch to frame dissection view  
- **m** or **ESC**: Return to main view
- **q**: Quit application

## Timeline Visualization

The bottom panel displays a visual timeline of the selected flow's timing behavior:

- **Horizontal axis**: Progression through packet sequence
- **Vertical axis**: Deviation from average inter-arrival time (centered on average)
- **Characters**: `·` = normal timing, `•`/`○` = moderate deviation, `█`/`▄` = outliers
- **Scale**: Automatically adjusts to show full range of deviations
- **Info bar**: Shows total frames, deviation range, and outlier count

## Project Structure

```
streamlens/
├── streamlens.py                    # Main entry point
├── analyzer/                        # Core analysis package
│   ├── main.py                     # CLI argument handling and main logic
│   ├── analysis/                   # Analysis engine
│   │   ├── core.py                # Main analyzer class
│   │   ├── flow_manager.py        # Flow tracking and management
│   │   └── statistics.py          # Statistical analysis and outlier detection
│   ├── models/                     # Data structures
│   │   ├── flow_stats.py          # Flow and frame type statistics
│   │   └── analysis_results.py    # Analysis result containers
│   ├── protocols/                  # Protocol dissectors
│   │   ├── base.py                # Base dissector interface
│   │   ├── chapter10.py           # IRIG106 telemetry protocol
│   │   ├── ptp.py                 # IEEE 1588 Precision Time Protocol
│   │   ├── iena.py                # Airbus IENA protocol
│   │   └── standard.py            # Standard protocol detection
│   ├── gui/                        # Modern GUI Interface (NEW!)
│   │   ├── __init__.py            # GUI package initialization
│   │   └── main_window.py         # PySide6 main window with matplotlib integration
│   ├── tui/                        # Text User Interface
│   │   ├── interface.py           # Main TUI controller
│   │   ├── navigation.py          # Navigation handling
│   │   └── panels/                # UI panel components
│   │       ├── flow_list.py       # Flow list panel
│   │       ├── detail_panel.py    # Flow details panel
│   │       └── timeline.py        # Timeline visualization panel
│   └── utils/                      # Utility modules
│       ├── pcap_loader.py         # PCAP file handling
│       ├── live_capture.py        # Live network capture
│       └── signal_visualizer.py   # Chapter 10 signal visualization (thread-safe)
└── *.pcapng                        # Sample capture files
```