208 lines
11 KiB
Markdown
208 lines
11 KiB
Markdown
# StreamLens - Ethernet Traffic Analyzer
|
||
|
||
Advanced network traffic analyzer for pcap files and live streams with specialized protocol dissection for aviation and industrial networks. Features sigma-based outlier identification, real-time statistical analysis, and both TUI and modern GUI interfaces with interactive signal visualization.
|
||
|
||
## Quick Start
|
||
|
||
```bash
|
||
# Install dependencies
|
||
pip install scapy numpy matplotlib
|
||
|
||
# For GUI mode (optional but recommended):
|
||
pip install PySide6
|
||
|
||
# For macOS users - install tkinter support for TUI visualization:
|
||
brew install python-tk@3.13
|
||
|
||
# Launch modern GUI with interactive plots
|
||
python streamlens.py --gui --pcap file.pcap
|
||
|
||
# GUI mode only (then open file via File menu)
|
||
python streamlens.py --gui
|
||
|
||
# Analyze pcap file with TUI (flows sorted by largest sigma outliers)
|
||
python streamlens.py --pcap file.pcap
|
||
|
||
# Live capture with real-time statistics
|
||
python streamlens.py --live --interface eth0
|
||
|
||
# Console output with outlier reporting
|
||
python streamlens.py --pcap file.pcap --no-tui
|
||
|
||
# Generate comprehensive outlier report
|
||
python streamlens.py --pcap file.pcap --report
|
||
|
||
# Get pcap file information
|
||
python streamlens.py --pcap file.pcap --info
|
||
|
||
# Adjust outlier threshold (default: 3.0 sigma)
|
||
python streamlens.py --pcap file.pcap --outlier-threshold 2.0
|
||
|
||
# With BPF filter for live capture
|
||
python streamlens.py --live --filter "port 319 or port 320"
|
||
```
|
||
|
||
## Features
|
||
|
||
### 🖥️ Modern Dark-Themed GUI Interface with Optimized Layout
|
||
- **Professional Dark Theme**: Modern color palette with #1e1e1e backgrounds and optimized contrast
|
||
- **Content-Fitted Columns**: Headers automatically resize to fit content, not wider than necessary
|
||
- **Full-Width Utilization**: Grid view uses entire screen width with prioritized wide signal plots
|
||
- **Optimized Row Height**: 25% taller rows (30px) for better visual balance and plot visibility
|
||
- **Wide Embedded Plots**: 8x2.5 figure size with minimal horizontal margins for maximum signal detail
|
||
- **Intelligent Column Sizing**: Auto-resizes to content with smart minimums and plot column priority
|
||
- **Professional Qt Interface**: Cross-platform GUI built with PySide6 with native look and feel
|
||
- **Embedded Signal Plots**: Chapter 10 signal plots rendered directly in the flow table cells
|
||
- **Synchronous Plot Rendering**: Plots appear immediately when table loads, no background threads
|
||
- **Chapter 10 Flow Highlighting**: Flows with Chapter 10 data are highlighted in modern blue and bold
|
||
- **Smart Signal Caching**: Avoids repeated processing of the same flow's signal data
|
||
- **Flow Detail Panel**: Dockable bottom panel with dark theme styling
|
||
- **Background PCAP Loading**: Progress bar with non-blocking file processing
|
||
- **Outlier Threshold Control**: Real-time adjustment of sigma-based outlier detection
|
||
- **Threading Safety**: Main-thread plot creation eliminates Qt threading violations
|
||
- **No Floating Windows**: All plots stay embedded in the grid interface
|
||
|
||
### Enhanced TUI Interface
|
||
- **Three-Panel Layout**: Flows list (top-left), flow details (top-right), timing visualization (bottom)
|
||
- **Sigma-Based Flow Sorting**: Flows automatically sorted by largest outlier sigma deviation
|
||
- **Real-time Navigation**: Arrow keys to navigate between flows with instant detail updates
|
||
- **Protocol-aware Display**: Shows detected protocols in flow list and details
|
||
- **Smart Protocol Detection**: Prioritizes specialized protocols (Chapter 10, PTP, IENA) over generic ones
|
||
- **Detailed Outlier Analysis**: Individual rows showing frame numbers and exact time deltas for outlier packets
|
||
- **Visual Timeline**: ASCII timeline showing frame timing deviations with outlier highlighting
|
||
- **Live Statistics**: Real-time running averages and outlier detection during capture
|
||
|
||
### Core Analysis Engine
|
||
- **Flow-based Analysis**: Groups packets by source-destination IP pairs with timing statistics
|
||
- **Configurable Outlier Detection**: Adjustable sigma threshold (default: 3.0σ)
|
||
- **Multi-layer Protocol Analysis**: Ethernet, IP, UDP, TCP with specialized dissectors
|
||
- **Real-time Statistical Updates**: Running statistics for live capture mode
|
||
- **High Jitter Flow Identification**: Coefficient of variation analysis
|
||
|
||
### Specialized Protocol Dissectors
|
||
- **Chapter 10 (IRIG 106-17)**: Complete packet dissection including data types, timestamps, and payload analysis
|
||
- **PTP (IEEE 1588-2019)**: Precision Time Protocol message parsing with sync, delay, and announce messages
|
||
- **IENA (Airbus)**: Industrial Ethernet Network Architecture with P/D/N/M/Q message types
|
||
|
||
### 📊 Chapter 10 Signal Visualization with Dark Theme Integration
|
||
- **Wide Embedded GUI Plots**: Chapter 10 flows display matplotlib plots directly in flow table with 8x2.5 sizing
|
||
- **Dark Theme Plot Integration**: Plots use #1e1e1e backgrounds with white text and modern #0078d4 signal colors
|
||
- **Optimized Plot Margins**: Minimal horizontal margins (8% left, 98% right) for maximum signal visualization area
|
||
- **TUI Signal Plots**: Press `v` in the TUI to generate signal files (threading-safe)
|
||
- **Signal Consolidation**: Automatically combines multiple packets from the same channel into continuous signals
|
||
- **TMATS Integration**: Automatically extracts channel metadata from TMATS frames for proper signal scaling
|
||
- **Multi-channel Support**: Displays multiple channels with proper engineering units and scaling
|
||
- **Threading Safety**: GUI uses main-thread plot creation, TUI saves plots to files to avoid segfaults
|
||
- **No Floating Windows**: All GUI plots stay embedded in the table interface
|
||
- **Both Modes**: Works for both PCAP analysis and live capture
|
||
- **Enhanced Visual Quality**: 150px plot height with professional styling and grid overlays
|
||
|
||
### Protocol Detection & Fallbacks
|
||
- Automatic protocol identification based on port numbers and packet structure
|
||
- Fallback to common protocols: HTTP, HTTPS, SSH, DNS, DHCP, NTP, SNMP, IGMP, ICMP
|
||
- Multicast detection for aviation/industrial networks
|
||
- Enhanced error handling and validation
|
||
|
||
## Installation
|
||
|
||
```bash
|
||
# Clone or download the project
|
||
cd streamlens
|
||
|
||
# Install dependencies
|
||
pip install scapy numpy matplotlib PySide6
|
||
|
||
# Run the analyzer
|
||
python streamlens.py --help
|
||
```
|
||
|
||
## Key Features Highlights
|
||
|
||
### 🎯 Sigma-Based Flow Prioritization
|
||
Flows are automatically sorted by their largest outlier sigma deviation, putting the most problematic flows at the top of the list for immediate attention.
|
||
|
||
### 📊 Real-time Statistics
|
||
Live capture mode provides running averages and outlier detection as packets arrive, with TUI updates every 500ms.
|
||
|
||
### 🔍 Configurable Analysis
|
||
Adjust outlier detection sensitivity with `--outlier-threshold` (default: 3.0σ) to fine-tune analysis for your specific network conditions.
|
||
|
||
### 📈 Comprehensive Reporting
|
||
Generate detailed outlier reports with `--report` flag showing frame-by-frame sigma deviations and timing analysis.
|
||
|
||
## GUI Usage
|
||
|
||
### Main Interface
|
||
- **Menu Bar**: File operations (Open PCAP, Monitor NIC), View controls, Help system
|
||
- **Toolbar**: File operations and outlier threshold adjustment
|
||
- **Central Flow Table**: Full-width table with file info, flow data, and integrated signal plots
|
||
- **Flow Detail Panel**: Dockable bottom panel showing comprehensive flow information
|
||
- **Status Bar**: Loading progress and operation feedback
|
||
|
||
### Workflow
|
||
1. **Launch GUI with PCAP**: `python streamlens.py --gui --pcap file.pcap` (recommended)
|
||
2. **Alternative Launch**: `python streamlens.py --gui`, then File → Open PCAP...
|
||
3. **Immediate Analysis**: Flow table displays instantly with all flow data and wide embedded plots
|
||
4. **Optimized Display**: Content-fitted columns, 25% taller rows, and full-width utilization
|
||
5. **Wide Plot Visualization**: Chapter 10 flows show detailed signal plots with minimal margins
|
||
6. **Browse Flows**: View flows in the dark-themed table (Chapter 10 flows highlighted in modern blue)
|
||
7. **Analyze Details**: Select flows to view detailed information in the dark-themed bottom panel
|
||
8. **Adjust Threshold**: Use toolbar spinner to change outlier detection sensitivity
|
||
9. **Multi-Flow Comparison**: Compare signals across different flows in the same optimized view
|
||
|
||
## TUI Controls
|
||
|
||
- **↑↓**: Navigate between flows in main view
|
||
- **v**: Visualize Chapter 10 signals for selected flow (saves plot files)
|
||
- **t**: Toggle timeline panel on/off
|
||
- **d**: Switch to frame dissection view
|
||
- **m** or **ESC**: Return to main view
|
||
- **q**: Quit application
|
||
|
||
## Timeline Visualization
|
||
|
||
The bottom panel displays a visual timeline of the selected flow's timing behavior:
|
||
|
||
- **Horizontal axis**: Progression through packet sequence
|
||
- **Vertical axis**: Deviation from average inter-arrival time (centered on average)
|
||
- **Characters**: `·` = normal timing, `•`/`○` = moderate deviation, `█`/`▄` = outliers
|
||
- **Scale**: Automatically adjusts to show full range of deviations
|
||
- **Info bar**: Shows total frames, deviation range, and outlier count
|
||
|
||
## Project Structure
|
||
|
||
```
|
||
streamlens/
|
||
├── streamlens.py # Main entry point
|
||
├── analyzer/ # Core analysis package
|
||
│ ├── main.py # CLI argument handling and main logic
|
||
│ ├── analysis/ # Analysis engine
|
||
│ │ ├── core.py # Main analyzer class
|
||
│ │ ├── flow_manager.py # Flow tracking and management
|
||
│ │ └── statistics.py # Statistical analysis and outlier detection
|
||
│ ├── models/ # Data structures
|
||
│ │ ├── flow_stats.py # Flow and frame type statistics
|
||
│ │ └── analysis_results.py # Analysis result containers
|
||
│ ├── protocols/ # Protocol dissectors
|
||
│ │ ├── base.py # Base dissector interface
|
||
│ │ ├── chapter10.py # IRIG106 telemetry protocol
|
||
│ │ ├── ptp.py # IEEE 1588 Precision Time Protocol
|
||
│ │ ├── iena.py # Airbus IENA protocol
|
||
│ │ └── standard.py # Standard protocol detection
|
||
│ ├── gui/ # Modern GUI Interface with Docking Panels
|
||
│ │ ├── __init__.py # GUI package initialization
|
||
│ │ ├── main_window.py # PySide6 main window with docking system
|
||
│ │ └── dock_panels.py # Dockable panel implementations (flow list, plots, details)
|
||
│ ├── tui/ # Text User Interface
|
||
│ │ ├── interface.py # Main TUI controller
|
||
│ │ ├── navigation.py # Navigation handling
|
||
│ │ └── panels/ # UI panel components
|
||
│ │ ├── flow_list.py # Flow list panel
|
||
│ │ ├── detail_panel.py # Flow details panel
|
||
│ │ └── timeline.py # Timeline visualization panel
|
||
│ └── utils/ # Utility modules
|
||
│ ├── pcap_loader.py # PCAP file handling
|
||
│ ├── live_capture.py # Live network capture
|
||
│ └── signal_visualizer.py # Chapter 10 signal visualization (thread-safe)
|
||
└── *.pcapng # Sample capture files
|
||
``` |