This commit is contained in:
2025-07-25 21:45:07 -04:00
parent f75c757b12
commit d77dd386f3
19 changed files with 2131 additions and 38 deletions

View File

@@ -2,7 +2,7 @@
## Project Overview
Build a sophisticated Python-based network traffic analysis tool called "StreamLens" that analyzes both PCAP files and live network streams. The tool specializes in telemetry and avionics protocols with advanced statistical timing analysis, outlier detection, and sigma-based flow prioritization. Features a modern modular architecture with a text-based user interface (TUI) for interactive analysis.
Build a sophisticated Python-based network traffic analysis tool called "StreamLens" that analyzes both PCAP files and live network streams. The tool specializes in telemetry and avionics protocols with advanced statistical timing analysis, outlier detection, and sigma-based flow prioritization. Features a modern modular architecture with both a text-based user interface (TUI) and a professional PySide6 GUI with interactive matplotlib signal visualization.
## Core Requirements
@@ -14,7 +14,8 @@ Build a sophisticated Python-based network traffic analysis tool called "StreamL
- **Sigma-Based Outlier Detection**: Identify packets with timing deviations using configurable thresholds (default 3σ)
- **Flow Prioritization**: Automatically sort flows by largest sigma deviation for efficient analysis
- **Interactive TUI**: Three-panel interface with real-time updates and navigation
- **Modular Architecture**: Clean separation of concerns with analyzers, models, protocols, TUI, and utilities
- **Modern GUI Interface**: Professional PySide6-based GUI with embedded matplotlib plots
- **Modular Architecture**: Clean separation of concerns with analyzers, models, protocols, TUI, GUI, and utilities
### Advanced Features
- **Specialized Protocol Support**: Chapter 10 (IRIG106), PTP (IEEE 1588), IENA (Airbus)
@@ -24,6 +25,10 @@ Build a sophisticated Python-based network traffic analysis tool called "StreamL
- **Comprehensive Reporting**: Detailed outlier reports with sigma deviation calculations
- **High Jitter Detection**: Coefficient of variation analysis for identifying problematic flows
- **Configurable Analysis**: Adjustable outlier thresholds and analysis parameters
- **Chapter 10 Signal Visualization**: Real-time matplotlib-based signal plotting with TMATS integration
- **Interactive Signal Analysis**: Press 'v' in TUI to generate signal files, or use GUI for embedded interactive plots
- **Threading-Safe Visualization**: Proper Qt integration for GUI, file output for TUI to prevent segmentation faults
- **Cross-Platform GUI**: PySide6-based interface with file dialogs, progress bars, and embedded matplotlib widgets
## Architecture Overview
@@ -207,6 +212,9 @@ def _classify_frame_type(self, packet, dissection):
### Core Libraries
- **scapy**: Packet capture and parsing (`pip install scapy`)
- **numpy**: Numerical computations (`pip install numpy`)
- **matplotlib**: Signal visualization and plotting (`pip install matplotlib`)
- **PySide6**: Modern Qt-based GUI framework (`pip install PySide6`)
- **tkinter**: GUI backend for matplotlib (usually included with Python)
- **curses**: Terminal UI framework (built-in on Unix systems)
- **statistics**: Statistical calculations (built-in)
- **struct**: Binary data parsing (built-in)
@@ -215,7 +223,7 @@ def _classify_frame_type(self, packet, dissection):
### Modern Modular File Structure
```
streamlens/
├── ethernet_analyzer_modular.py # Main entry point
├── streamlens.py # Main entry point
├── analyzer/ # Core analysis package
│ ├── __init__.py # Package initialization
│ ├── main.py # CLI handling and main application logic
@@ -235,6 +243,9 @@ streamlens/
│ │ ├── ptp.py # PTPDissector (IEEE 1588)
│ │ ├── iena.py # IENADissector (Airbus)
│ │ └── standard.py # StandardProtocolDissector
│ ├── gui/ # Modern GUI Interface system (NEW!)
│ │ ├── __init__.py # GUI package init
│ │ └── main_window.py # StreamLensMainWindow with PySide6 and matplotlib
│ ├── tui/ # Text User Interface system
│ │ ├── __init__.py # TUI package init
│ │ ├── interface.py # TUIInterface main controller
@@ -247,7 +258,8 @@ streamlens/
│ └── utils/ # Utility modules
│ ├── __init__.py # Utils package init
│ ├── pcap_loader.py # PCAPLoader for file handling
── live_capture.py # LiveCapture for network monitoring
── live_capture.py # LiveCapture for network monitoring
│ └── signal_visualizer.py # Chapter 10 signal visualization (thread-safe)
├── *.pcapng # Sample capture files for testing
├── README.md # User guide and quick start
└── ai.comprehensive_replay.md # This comprehensive development guide
@@ -255,26 +267,32 @@ streamlens/
## Command Line Interface
```bash
# Launch modern GUI with interactive plots (RECOMMENDED)
python streamlens.py --gui --pcap file.pcap
# GUI mode only (then open file via File menu)
python streamlens.py --gui
# Analyze PCAP file with TUI (flows sorted by largest sigma outliers)
python ethernet_analyzer_modular.py --pcap file.pcap
python streamlens.py --pcap file.pcap
# Console output mode with sigma deviation display
python ethernet_analyzer_modular.py --pcap file.pcap --no-tui
python streamlens.py --pcap file.pcap --no-tui
# Generate comprehensive outlier report
python ethernet_analyzer_modular.py --pcap file.pcap --report
python streamlens.py --pcap file.pcap --report
# Get PCAP file information only
python ethernet_analyzer_modular.py --pcap file.pcap --info
python streamlens.py --pcap file.pcap --info
# Live capture with real-time statistics
python ethernet_analyzer_modular.py --live --interface eth0
python streamlens.py --live --interface eth0
# Configure outlier threshold (default: 3.0 sigma)
python ethernet_analyzer_modular.py --pcap file.pcap --outlier-threshold 2.0
python streamlens.py --pcap file.pcap --outlier-threshold 2.0
# With BPF filtering for targeted capture
python ethernet_analyzer_modular.py --live --filter "port 319 or port 320"
python streamlens.py --live --filter "port 319 or port 320"
```
## Key Algorithms and Techniques
@@ -331,7 +349,54 @@ def get_max_sigma_deviation(self, flow: FlowStats) -> float:
- Live outlier detection with immediate flagging
- TUI updates every 0.5 seconds during live capture
### 6. Modular Architecture Design
### 6. Chapter 10 Signal Visualization System
- **TMATS Parser**: Extracts channel metadata from Telemetry Attributes Transfer Standard frames
- **Signal Decoders**: Support for analog and PCM format data with proper scaling
- **Matplotlib Integration**: External plotting windows with interactive capabilities
- **Real-time Visualization**: Works for both PCAP analysis and live capture modes
- **Multi-channel Display**: Simultaneous plotting of multiple signal channels with engineering units
```python
class SignalVisualizer:
def visualize_flow_signals(self, flow: FlowStats, packets: List[Packet]) -> None:
# Extract TMATS metadata for channel configurations
tmats_metadata = self._extract_tmats_from_flow(packets)
# Decode signal data using Chapter 10 decoders
signal_data = self._extract_signals_from_flow(packets, tmats_metadata)
# Launch matplotlib window in background thread
self._create_signal_window(flow_key, signal_data, flow)
```
### 7. PySide6 GUI Architecture with Threading Safety
- **Professional Qt Interface**: Cross-platform GUI built with PySide6 for native look and feel
- **Embedded Matplotlib Integration**: Interactive plots with zoom, pan, and navigation toolbar
- **Background Processing**: Threading for PCAP loading with progress bar and non-blocking UI
- **Flow List Widget**: Sortable table with sigma deviations, protocols, and frame types
- **Signal Visualization**: Click-to-visualize Chapter 10 flows with embedded matplotlib widgets
- **Threading Safety**: Proper Qt integration prevents matplotlib segmentation faults
```python
class StreamLensMainWindow(QMainWindow):
def __init__(self):
# Create main interface with flow list and plot area
self.flows_table = QTableWidget() # Sortable flow list
self.plot_widget = PlotWidget() # Embedded matplotlib
def load_pcap_file(self, file_path: str):
# Background loading with progress bar
self.loading_thread = PCAPLoadThread(file_path)
self.loading_thread.progress_updated.connect(self.progress_bar.setValue)
self.loading_thread.loading_finished.connect(self.on_pcap_loaded)
def visualize_selected_flow(self):
# Interactive signal visualization
signal_data = signal_visualizer._extract_signals_from_flow(packets, tmats)
self.plot_widget.plot_flow_signals(flow, signal_data, flow_key)
```
### 8. Modular Architecture Design
- **Separation of Concerns**: Clean boundaries between analysis, UI, protocols, and utilities
- **Package Structure**: Logical grouping of related functionality
- **Dependency Injection**: Components receive dependencies through constructors