working well, good docs. TUI.
This commit is contained in:
138
README.md
Normal file
138
README.md
Normal file
@@ -0,0 +1,138 @@
|
||||
# StreamLens - Ethernet Traffic Analyzer
|
||||
|
||||
Advanced TUI-based network traffic analyzer for pcap files and live streams with specialized protocol dissection for aviation and industrial networks. Features sigma-based outlier identification and real-time statistical analysis.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install scapy numpy
|
||||
|
||||
# Analyze pcap file with TUI (flows sorted by largest sigma outliers)
|
||||
python ethernet_analyzer_modular.py --pcap file.pcap
|
||||
|
||||
# Live capture with real-time statistics
|
||||
python ethernet_analyzer_modular.py --live --interface eth0
|
||||
|
||||
# Console output with outlier reporting
|
||||
python ethernet_analyzer_modular.py --pcap file.pcap --no-tui
|
||||
|
||||
# Generate comprehensive outlier report
|
||||
python ethernet_analyzer_modular.py --pcap file.pcap --report
|
||||
|
||||
# Get pcap file information
|
||||
python ethernet_analyzer_modular.py --pcap file.pcap --info
|
||||
|
||||
# Adjust outlier threshold (default: 3.0 sigma)
|
||||
python ethernet_analyzer_modular.py --pcap file.pcap --outlier-threshold 2.0
|
||||
|
||||
# With BPF filter for live capture
|
||||
python ethernet_analyzer_modular.py --live --filter "port 319 or port 320"
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
### Enhanced TUI Interface
|
||||
- **Three-Panel Layout**: Flows list (top-left), flow details (top-right), timing visualization (bottom)
|
||||
- **Sigma-Based Flow Sorting**: Flows automatically sorted by largest outlier sigma deviation
|
||||
- **Real-time Navigation**: Arrow keys to navigate between flows with instant detail updates
|
||||
- **Protocol-aware Display**: Shows detected protocols in flow list and details
|
||||
- **Smart Protocol Detection**: Prioritizes specialized protocols (Chapter 10, PTP, IENA) over generic ones
|
||||
- **Detailed Outlier Analysis**: Individual rows showing frame numbers and exact time deltas for outlier packets
|
||||
- **Visual Timeline**: ASCII timeline showing frame timing deviations with outlier highlighting
|
||||
- **Live Statistics**: Real-time running averages and outlier detection during capture
|
||||
|
||||
### Core Analysis Engine
|
||||
- **Flow-based Analysis**: Groups packets by source-destination IP pairs with timing statistics
|
||||
- **Configurable Outlier Detection**: Adjustable sigma threshold (default: 3.0σ)
|
||||
- **Multi-layer Protocol Analysis**: Ethernet, IP, UDP, TCP with specialized dissectors
|
||||
- **Real-time Statistical Updates**: Running statistics for live capture mode
|
||||
- **High Jitter Flow Identification**: Coefficient of variation analysis
|
||||
|
||||
### Specialized Protocol Dissectors
|
||||
- **Chapter 10 (IRIG 106-17)**: Complete packet dissection including data types, timestamps, and payload analysis
|
||||
- **PTP (IEEE 1588-2019)**: Precision Time Protocol message parsing with sync, delay, and announce messages
|
||||
- **IENA (Airbus)**: Industrial Ethernet Network Architecture with P/D/N/M/Q message types
|
||||
|
||||
### Protocol Detection & Fallbacks
|
||||
- Automatic protocol identification based on port numbers and packet structure
|
||||
- Fallback to common protocols: HTTP, HTTPS, SSH, DNS, DHCP, NTP, SNMP, IGMP, ICMP
|
||||
- Multicast detection for aviation/industrial networks
|
||||
- Enhanced error handling and validation
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Clone or download the project
|
||||
cd streamlens
|
||||
|
||||
# Install dependencies
|
||||
pip install scapy numpy
|
||||
|
||||
# Run the analyzer
|
||||
python ethernet_analyzer_modular.py --help
|
||||
```
|
||||
|
||||
## Key Features Highlights
|
||||
|
||||
### 🎯 Sigma-Based Flow Prioritization
|
||||
Flows are automatically sorted by their largest outlier sigma deviation, putting the most problematic flows at the top of the list for immediate attention.
|
||||
|
||||
### 📊 Real-time Statistics
|
||||
Live capture mode provides running averages and outlier detection as packets arrive, with TUI updates every 500ms.
|
||||
|
||||
### 🔍 Configurable Analysis
|
||||
Adjust outlier detection sensitivity with `--outlier-threshold` (default: 3.0σ) to fine-tune analysis for your specific network conditions.
|
||||
|
||||
### 📈 Comprehensive Reporting
|
||||
Generate detailed outlier reports with `--report` flag showing frame-by-frame sigma deviations and timing analysis.
|
||||
|
||||
## TUI Controls
|
||||
|
||||
- **↑↓**: Navigate between flows in main view
|
||||
- **d**: Switch to frame dissection view
|
||||
- **m** or **ESC**: Return to main view
|
||||
- **q**: Quit application
|
||||
|
||||
## Timeline Visualization
|
||||
|
||||
The bottom panel displays a visual timeline of the selected flow's timing behavior:
|
||||
|
||||
- **Horizontal axis**: Progression through packet sequence
|
||||
- **Vertical axis**: Deviation from average inter-arrival time (centered on average)
|
||||
- **Characters**: `·` = normal timing, `•`/`○` = moderate deviation, `█`/`▄` = outliers
|
||||
- **Scale**: Automatically adjusts to show full range of deviations
|
||||
- **Info bar**: Shows total frames, deviation range, and outlier count
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
streamlens/
|
||||
├── ethernet_analyzer_modular.py # Main entry point
|
||||
├── analyzer/ # Core analysis package
|
||||
│ ├── main.py # CLI argument handling and main logic
|
||||
│ ├── analysis/ # Analysis engine
|
||||
│ │ ├── core.py # Main analyzer class
|
||||
│ │ ├── flow_manager.py # Flow tracking and management
|
||||
│ │ └── statistics.py # Statistical analysis and outlier detection
|
||||
│ ├── models/ # Data structures
|
||||
│ │ ├── flow_stats.py # Flow and frame type statistics
|
||||
│ │ └── analysis_results.py # Analysis result containers
|
||||
│ ├── protocols/ # Protocol dissectors
|
||||
│ │ ├── base.py # Base dissector interface
|
||||
│ │ ├── chapter10.py # IRIG106 telemetry protocol
|
||||
│ │ ├── ptp.py # IEEE 1588 Precision Time Protocol
|
||||
│ │ ├── iena.py # Airbus IENA protocol
|
||||
│ │ └── standard.py # Standard protocol detection
|
||||
│ ├── tui/ # Text User Interface
|
||||
│ │ ├── interface.py # Main TUI controller
|
||||
│ │ ├── navigation.py # Navigation handling
|
||||
│ │ └── panels/ # UI panel components
|
||||
│ │ ├── flow_list.py # Flow list panel
|
||||
│ │ ├── detail_panel.py # Flow details panel
|
||||
│ │ └── timeline.py # Timeline visualization panel
|
||||
│ └── utils/ # Utility modules
|
||||
│ ├── pcap_loader.py # PCAP file handling
|
||||
│ └── live_capture.py # Live network capture
|
||||
└── *.pcapng # Sample capture files
|
||||
```
|
||||
373
ai.comprehensive_replay.md
Normal file
373
ai.comprehensive_replay.md
Normal file
@@ -0,0 +1,373 @@
|
||||
# StreamLens - Comprehensive Development Guide
|
||||
|
||||
## Project Overview
|
||||
|
||||
Build a sophisticated Python-based network traffic analysis tool called "StreamLens" that analyzes both PCAP files and live network streams. The tool specializes in telemetry and avionics protocols with advanced statistical timing analysis, outlier detection, and sigma-based flow prioritization. Features a modern modular architecture with a text-based user interface (TUI) for interactive analysis.
|
||||
|
||||
## Core Requirements
|
||||
|
||||
### Primary Functionality
|
||||
- **PCAP Analysis**: Load and analyze packet capture files with file validation and info extraction
|
||||
- **Live Capture**: Real-time network traffic monitoring with threading support
|
||||
- **Flow Analysis**: Group packets by source-destination IP pairs with comprehensive statistics
|
||||
- **Advanced Statistical Analysis**: Configurable sigma thresholds, running averages, coefficient of variation
|
||||
- **Sigma-Based Outlier Detection**: Identify packets with timing deviations using configurable thresholds (default 3σ)
|
||||
- **Flow Prioritization**: Automatically sort flows by largest sigma deviation for efficient analysis
|
||||
- **Interactive TUI**: Three-panel interface with real-time updates and navigation
|
||||
- **Modular Architecture**: Clean separation of concerns with analyzers, models, protocols, TUI, and utilities
|
||||
|
||||
### Advanced Features
|
||||
- **Specialized Protocol Support**: Chapter 10 (IRIG106), PTP (IEEE 1588), IENA (Airbus)
|
||||
- **Frame Type Classification**: Hierarchical breakdown by protocol and frame types with per-type statistics
|
||||
- **Timeline Visualization**: ASCII-based timing deviation charts with dynamic scaling
|
||||
- **Real-time Statistics**: Running averages and outlier detection during live capture
|
||||
- **Comprehensive Reporting**: Detailed outlier reports with sigma deviation calculations
|
||||
- **High Jitter Detection**: Coefficient of variation analysis for identifying problematic flows
|
||||
- **Configurable Analysis**: Adjustable outlier thresholds and analysis parameters
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
### Core Components
|
||||
|
||||
#### 1. Data Structures (`dataclasses`)
|
||||
```python
|
||||
@dataclass
|
||||
class FrameTypeStats:
|
||||
frame_type: str
|
||||
count: int
|
||||
total_bytes: int
|
||||
timestamps: List[float]
|
||||
frame_numbers: List[int]
|
||||
inter_arrival_times: List[float]
|
||||
avg_inter_arrival: float
|
||||
std_inter_arrival: float
|
||||
outlier_frames: List[int]
|
||||
outlier_details: List[Tuple[int, float]]
|
||||
|
||||
@dataclass
|
||||
class FlowStats:
|
||||
src_ip: str
|
||||
dst_ip: str
|
||||
frame_count: int
|
||||
timestamps: List[float]
|
||||
frame_numbers: List[int]
|
||||
inter_arrival_times: List[float]
|
||||
avg_inter_arrival: float
|
||||
std_inter_arrival: float
|
||||
outlier_frames: List[int]
|
||||
outlier_details: List[Tuple[int, float]]
|
||||
total_bytes: int
|
||||
protocols: Set[str]
|
||||
detected_protocol_types: Set[str]
|
||||
frame_types: Dict[str, FrameTypeStats]
|
||||
```
|
||||
|
||||
#### 2. Main Analysis Engine (`EthernetAnalyzer`)
|
||||
- **Packet Processing**: Single-packet and batch processing capabilities
|
||||
- **Flow Management**: Track unique source-destination pairs
|
||||
- **Protocol Detection**: Multi-layer protocol identification
|
||||
- **Statistical Calculation**: Mean, standard deviation, outlier detection using 2σ threshold
|
||||
- **Live Capture**: Threading support for real-time analysis
|
||||
|
||||
#### 3. Enhanced Frame Dissection System
|
||||
|
||||
##### Base Dissector Interface
|
||||
```python
|
||||
class EnhancedFrameDissector:
|
||||
def dissect_frame(self, packet: Packet, frame_num: int) -> Dict[str, Any]
|
||||
```
|
||||
|
||||
##### Specialized Protocol Dissectors
|
||||
|
||||
**Chapter 10 Dissector** (`Chapter10Dissector`):
|
||||
- IRIG106-17 telemetry standard support
|
||||
- Sync pattern detection (0xEB25)
|
||||
- Header parsing: channel ID, data types, timing counters
|
||||
- Data format support: Analog, PCM, Ethernet Format 0
|
||||
- Container format handling (embedded Ch10 in other protocols)
|
||||
|
||||
**PTP Dissector** (`PTPDissector`):
|
||||
- IEEE 1588-2019 Precision Time Protocol
|
||||
- Message types: Sync, Delay_Req, Announce, etc.
|
||||
- Timestamp extraction and correction field parsing
|
||||
- Grandmaster clock quality analysis
|
||||
|
||||
**IENA Dissector** (`IENADissector`):
|
||||
- Airbus Improved Ethernet Network Architecture
|
||||
- Packet types: P-type, D-type, N-type, M-type, Q-type
|
||||
- Parameter ID extraction and delay analysis
|
||||
- Variable-length message parsing
|
||||
|
||||
#### 4. Chapter 10 Packet System (`chapter10_packet.py`)
|
||||
|
||||
##### Core Packet Class
|
||||
```python
|
||||
class Chapter10Packet:
|
||||
def __init__(self, packet, original_frame_num: Optional[int] = None)
|
||||
def _parse_ch10_header(self) -> Optional[Dict]
|
||||
def get_data_payload(self) -> Optional[bytes]
|
||||
def decode_data(self, tmats_scaling_dict, tmats_content) -> Optional[DecodedData]
|
||||
```
|
||||
|
||||
##### Data Decoders
|
||||
- **AnalogDecoder**: Multi-channel analog data with scaling
|
||||
- **PCMDecoder**: Pulse Code Modulation format
|
||||
- **AnalogTimeFormatDecoder**: Time-synchronized analog data with TMATS integration
|
||||
|
||||
##### TMATS Integration
|
||||
- Scaling parameter extraction from TMATS content
|
||||
- Channel configuration parsing
|
||||
- Engineering unit conversion
|
||||
- Gain/offset/full-scale calculations
|
||||
|
||||
#### 5. Text User Interface (`TUIInterface`)
|
||||
|
||||
##### Three-Panel Layout
|
||||
```
|
||||
┌─ FLOWS (Left 60%) ──┐│ DETAILS (Right 40%) ─┐
|
||||
│ IP flows + frame ││ Flow info + table │
|
||||
│ type breakdowns ││ Frame type details │
|
||||
├────────────────────── ────────────────────────┤
|
||||
│ TIMING VISUALIZATION (Bottom 30%) │
|
||||
│ ASCII timeline with deviation plotting │
|
||||
└───────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
##### Key Interface Features
|
||||
- **Flow List**: Hierarchical display of flows and frame types
|
||||
- **Detail Panel**:
|
||||
- Flow summary (packets, bytes, protocols)
|
||||
- Frame type table: `Type | #Pkts | Bytes | Avg ΔT | 2σ Out`
|
||||
- Timing statistics and threshold calculations
|
||||
- **Timeline Panel**: ASCII deviation chart with scaling
|
||||
- **Navigation**: Arrow keys, view switching (main/dissection)
|
||||
|
||||
## Technical Implementation Details
|
||||
|
||||
### Statistical Analysis Algorithm
|
||||
```python
|
||||
def calculate_statistics(self):
|
||||
for flow in self.flows.values():
|
||||
# Calculate inter-arrival times
|
||||
flow.avg_inter_arrival = statistics.mean(flow.inter_arrival_times)
|
||||
flow.std_inter_arrival = statistics.stdev(flow.inter_arrival_times)
|
||||
|
||||
# Detect outliers (>2σ threshold)
|
||||
threshold = flow.avg_inter_arrival + (2 * flow.std_inter_arrival)
|
||||
for i, inter_time in enumerate(flow.inter_arrival_times):
|
||||
if inter_time > threshold:
|
||||
frame_number = flow.frame_numbers[i + 1]
|
||||
flow.outlier_frames.append(frame_number)
|
||||
flow.outlier_details.append((frame_number, inter_time))
|
||||
```
|
||||
|
||||
### Protocol Classification Hierarchy
|
||||
1. **Specialized Protocols**: Chapter 10, PTP, IENA (highest priority)
|
||||
2. **Port-based Detection**: DNS (53), HTTP (80), NTP (123), etc.
|
||||
3. **Transport Layer**: TCP, UDP, ICMP, IGMP
|
||||
4. **Fallback**: Generic classification
|
||||
|
||||
### Frame Type Classification Logic
|
||||
```python
|
||||
def _classify_frame_type(self, packet, dissection):
|
||||
layers = dissection.get('layers', {})
|
||||
|
||||
# Chapter 10 takes precedence
|
||||
if 'chapter10' in layers:
|
||||
if self._is_tmats_frame(packet, layers['chapter10']):
|
||||
return 'TMATS'
|
||||
else:
|
||||
return 'CH10-Data'
|
||||
|
||||
# PTP message types
|
||||
if 'ptp' in layers:
|
||||
msg_type = layers['ptp'].get('message_type_name', 'Unknown')
|
||||
return f'PTP-{msg_type}'
|
||||
|
||||
# IENA packet types
|
||||
if 'iena' in layers:
|
||||
packet_type = layers['iena'].get('packet_type_name', 'Unknown')
|
||||
return f'IENA-{packet_type}'
|
||||
|
||||
# Fallback to transport/application protocols
|
||||
return self._classify_standard_protocols(packet)
|
||||
```
|
||||
|
||||
### Timeline Visualization Algorithm
|
||||
- **Deviation Calculation**: `deviation = inter_arrival_time - average_inter_arrival`
|
||||
- **Scaling**: Vertical position based on deviation magnitude
|
||||
- **Character Selection**:
|
||||
- `·` for normal timing (within 10% of average)
|
||||
- `•`/`○` for moderate deviations (within 50%)
|
||||
- `█`/`▄` for significant outliers
|
||||
- **Time Mapping**: Horizontal position maps to actual timestamps
|
||||
|
||||
## Dependencies and Requirements
|
||||
|
||||
### Core Libraries
|
||||
- **scapy**: Packet capture and parsing (`pip install scapy`)
|
||||
- **numpy**: Numerical computations (`pip install numpy`)
|
||||
- **curses**: Terminal UI framework (built-in on Unix systems)
|
||||
- **statistics**: Statistical calculations (built-in)
|
||||
- **struct**: Binary data parsing (built-in)
|
||||
- **threading**: Live capture support (built-in)
|
||||
|
||||
### Modern Modular File Structure
|
||||
```
|
||||
streamlens/
|
||||
├── ethernet_analyzer_modular.py # Main entry point
|
||||
├── analyzer/ # Core analysis package
|
||||
│ ├── __init__.py # Package initialization
|
||||
│ ├── main.py # CLI handling and main application logic
|
||||
│ ├── analysis/ # Analysis engine modules
|
||||
│ │ ├── __init__.py # Analysis package init
|
||||
│ │ ├── core.py # EthernetAnalyzer main class
|
||||
│ │ ├── flow_manager.py # FlowManager for packet processing
|
||||
│ │ └── statistics.py # StatisticsEngine with sigma calculations
|
||||
│ ├── models/ # Data structures and models
|
||||
│ │ ├── __init__.py # Models package init
|
||||
│ │ ├── flow_stats.py # FlowStats and FrameTypeStats dataclasses
|
||||
│ │ └── analysis_results.py # AnalysisResult containers
|
||||
│ ├── protocols/ # Protocol dissector system
|
||||
│ │ ├── __init__.py # Protocols package init
|
||||
│ │ ├── base.py # ProtocolDissector base classes
|
||||
│ │ ├── chapter10.py # Chapter10Dissector (IRIG106)
|
||||
│ │ ├── ptp.py # PTPDissector (IEEE 1588)
|
||||
│ │ ├── iena.py # IENADissector (Airbus)
|
||||
│ │ └── standard.py # StandardProtocolDissector
|
||||
│ ├── tui/ # Text User Interface system
|
||||
│ │ ├── __init__.py # TUI package init
|
||||
│ │ ├── interface.py # TUIInterface main controller
|
||||
│ │ ├── navigation.py # NavigationHandler for input handling
|
||||
│ │ └── panels/ # UI panel components
|
||||
│ │ ├── __init__.py # Panels package init
|
||||
│ │ ├── flow_list.py # FlowListPanel for flow display
|
||||
│ │ ├── detail_panel.py # DetailPanel for flow details
|
||||
│ │ └── timeline.py # TimelinePanel for visualization
|
||||
│ └── utils/ # Utility modules
|
||||
│ ├── __init__.py # Utils package init
|
||||
│ ├── pcap_loader.py # PCAPLoader for file handling
|
||||
│ └── live_capture.py # LiveCapture for network monitoring
|
||||
├── *.pcapng # Sample capture files for testing
|
||||
├── README.md # User guide and quick start
|
||||
└── ai.comprehensive_replay.md # This comprehensive development guide
|
||||
```
|
||||
|
||||
## Command Line Interface
|
||||
```bash
|
||||
# Analyze PCAP file with TUI (flows sorted by largest sigma outliers)
|
||||
python ethernet_analyzer_modular.py --pcap file.pcap
|
||||
|
||||
# Console output mode with sigma deviation display
|
||||
python ethernet_analyzer_modular.py --pcap file.pcap --no-tui
|
||||
|
||||
# Generate comprehensive outlier report
|
||||
python ethernet_analyzer_modular.py --pcap file.pcap --report
|
||||
|
||||
# Get PCAP file information only
|
||||
python ethernet_analyzer_modular.py --pcap file.pcap --info
|
||||
|
||||
# Live capture with real-time statistics
|
||||
python ethernet_analyzer_modular.py --live --interface eth0
|
||||
|
||||
# Configure outlier threshold (default: 3.0 sigma)
|
||||
python ethernet_analyzer_modular.py --pcap file.pcap --outlier-threshold 2.0
|
||||
|
||||
# With BPF filtering for targeted capture
|
||||
python ethernet_analyzer_modular.py --live --filter "port 319 or port 320"
|
||||
```
|
||||
|
||||
## Key Algorithms and Techniques
|
||||
|
||||
### 1. Inter-arrival Time Calculation
|
||||
- Track timestamps for each packet in a flow
|
||||
- Calculate time differences between consecutive packets
|
||||
- Handle both flow-level and frame-type-level statistics
|
||||
|
||||
### 2. Advanced Outlier Detection and Flow Prioritization
|
||||
- Use configurable sigma rule: outliers are packets with inter-arrival times > (mean + N×std_dev)
|
||||
- Default threshold: 3σ (configurable via --outlier-threshold parameter)
|
||||
- Calculate maximum sigma deviation for each flow across all outliers
|
||||
- Store both frame numbers and actual time deltas for detailed analysis
|
||||
- Apply at both flow and frame-type granularity
|
||||
- **Key Innovation**: Sort flows by largest sigma deviation to prioritize most problematic flows
|
||||
|
||||
```python
|
||||
def get_max_sigma_deviation(self, flow: FlowStats) -> float:
|
||||
"""Get the maximum sigma deviation for any outlier in this flow"""
|
||||
max_sigma = 0.0
|
||||
|
||||
# Check flow-level outliers
|
||||
if flow.outlier_details and flow.std_inter_arrival > 0:
|
||||
for frame_num, inter_arrival_time in flow.outlier_details:
|
||||
sigma_deviation = (inter_arrival_time - flow.avg_inter_arrival) / flow.std_inter_arrival
|
||||
max_sigma = max(max_sigma, sigma_deviation)
|
||||
|
||||
# Check frame-type-level outliers
|
||||
for frame_type, ft_stats in flow.frame_types.items():
|
||||
if ft_stats.outlier_details and ft_stats.std_inter_arrival > 0:
|
||||
for frame_num, inter_arrival_time in ft_stats.outlier_details:
|
||||
sigma_deviation = (inter_arrival_time - ft_stats.avg_inter_arrival) / ft_stats.std_inter_arrival
|
||||
max_sigma = max(max_sigma, sigma_deviation)
|
||||
|
||||
return max_sigma
|
||||
```
|
||||
|
||||
### 3. Protocol Dissection Pipeline
|
||||
- Layer-by-layer parsing starting with Ethernet/IP/UDP
|
||||
- Specialized protocol detection based on ports, patterns, payload analysis
|
||||
- Error handling and graceful degradation for malformed packets
|
||||
|
||||
### 4. TMATS Parameter Parsing
|
||||
- Line-by-line parsing of TMATS configuration data
|
||||
- Key-value pair extraction with hierarchical channel mapping
|
||||
- Parameter validation and default value assignment
|
||||
|
||||
### 5. Live Capture Architecture with Real-time Statistics
|
||||
- Threaded packet capture using scapy.sniff()
|
||||
- Real-time running statistical updates during capture
|
||||
- Thread-safe data structures and stop conditions
|
||||
- Running averages calculated incrementally for performance
|
||||
- Live outlier detection with immediate flagging
|
||||
- TUI updates every 0.5 seconds during live capture
|
||||
|
||||
### 6. Modular Architecture Design
|
||||
- **Separation of Concerns**: Clean boundaries between analysis, UI, protocols, and utilities
|
||||
- **Package Structure**: Logical grouping of related functionality
|
||||
- **Dependency Injection**: Components receive dependencies through constructors
|
||||
- **Interface-based Design**: Protocol dissectors implement common interfaces
|
||||
- **Error Handling**: Graceful degradation and comprehensive error reporting
|
||||
|
||||
## Extensibility Points
|
||||
|
||||
### Adding New Protocol Dissectors
|
||||
1. Implement dissector class with `dissect(packet)` method
|
||||
2. Return `DissectionResult` with protocol type and parsed fields
|
||||
3. Register in `EnhancedFrameDissector.dissectors` dictionary
|
||||
|
||||
### Custom Frame Type Classification
|
||||
- Override `_classify_frame_type()` method
|
||||
- Add custom logic based on dissection results
|
||||
- Return descriptive frame type strings
|
||||
|
||||
### Statistical Analysis Extensions
|
||||
- Extend `FrameTypeStats` and `FlowStats` dataclasses
|
||||
- Add custom statistical calculations in `calculate_statistics()`
|
||||
- Implement additional outlier detection algorithms
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- **Memory Management**: Use generators for large PCAP files
|
||||
- **Threading**: Separate capture and analysis threads for live mode
|
||||
- **Statistical Efficiency**: Incremental statistics calculation
|
||||
- **TUI Optimization**: Efficient screen drawing with curses error handling
|
||||
|
||||
## Testing and Validation
|
||||
|
||||
The project includes comprehensive test suites:
|
||||
- **Protocol Dissection Tests**: Validate parsing accuracy
|
||||
- **Statistical Analysis Tests**: Verify timing calculations
|
||||
- **TUI Layout Tests**: Interface rendering validation
|
||||
- **Integration Tests**: End-to-end workflow verification
|
||||
|
||||
This comprehensive description captures the full scope and technical depth of the Ethernet Traffic Analyzer, enabling recreation of this sophisticated telemetry analysis tool.
|
||||
@@ -237,4 +237,23 @@ class StatisticsEngine:
|
||||
'tracked_flows': total_flows,
|
||||
'total_outliers': total_outliers,
|
||||
'update_frequency': 'per_packet'
|
||||
}
|
||||
}
|
||||
|
||||
def get_max_sigma_deviation(self, flow: FlowStats) -> float:
|
||||
"""Get the maximum sigma deviation for any outlier in this flow"""
|
||||
max_sigma = 0.0
|
||||
|
||||
# Check flow-level outliers
|
||||
if flow.outlier_details and flow.std_inter_arrival > 0:
|
||||
for frame_num, inter_arrival_time in flow.outlier_details:
|
||||
sigma_deviation = (inter_arrival_time - flow.avg_inter_arrival) / flow.std_inter_arrival
|
||||
max_sigma = max(max_sigma, sigma_deviation)
|
||||
|
||||
# Check frame-type-level outliers
|
||||
for frame_type, ft_stats in flow.frame_types.items():
|
||||
if ft_stats.outlier_details and ft_stats.std_inter_arrival > 0:
|
||||
for frame_num, inter_arrival_time in ft_stats.outlier_details:
|
||||
sigma_deviation = (inter_arrival_time - ft_stats.avg_inter_arrival) / ft_stats.std_inter_arrival
|
||||
max_sigma = max(max_sigma, sigma_deviation)
|
||||
|
||||
return max_sigma
|
||||
@@ -157,12 +157,17 @@ def print_console_results(analyzer: EthernetAnalyzer):
|
||||
print(f"Update Frequency: {rt_stats.get('update_frequency', 'N/A')}")
|
||||
|
||||
print(f"\n=== FLOW STATISTICS ===")
|
||||
flows_sorted = sorted(summary['flows'].values(), key=lambda x: x.frame_count, reverse=True)
|
||||
flows_sorted = sorted(summary['flows'].values(), key=lambda x: (
|
||||
analyzer.statistics_engine.get_max_sigma_deviation(x),
|
||||
x.frame_count
|
||||
), reverse=True)
|
||||
|
||||
for flow in flows_sorted:
|
||||
max_sigma = analyzer.statistics_engine.get_max_sigma_deviation(flow)
|
||||
print(f"\nFlow: {flow.src_ip} -> {flow.dst_ip}")
|
||||
print(f" Packets: {flow.frame_count}")
|
||||
print(f" Total Bytes: {flow.total_bytes:,}")
|
||||
print(f" Max Sigma Deviation: {max_sigma:.2f}σ")
|
||||
print(f" Protocols: {', '.join(flow.protocols)}")
|
||||
if flow.detected_protocol_types:
|
||||
print(f" Enhanced Protocols: {', '.join(flow.detected_protocol_types)}")
|
||||
@@ -217,15 +222,20 @@ def generate_outlier_report(analyzer: EthernetAnalyzer, threshold_sigma: float):
|
||||
print("DETAILED FLOW ANALYSIS")
|
||||
print("=" * 80)
|
||||
|
||||
flows_sorted = sorted(summary['flows'].values(), key=lambda x: x.frame_count, reverse=True)
|
||||
flows_sorted = sorted(summary['flows'].values(), key=lambda x: (
|
||||
analyzer.statistics_engine.get_max_sigma_deviation(x),
|
||||
x.frame_count
|
||||
), reverse=True)
|
||||
|
||||
for flow_idx, flow in enumerate(flows_sorted, 1):
|
||||
max_sigma = analyzer.statistics_engine.get_max_sigma_deviation(flow)
|
||||
print(f"\n[FLOW {flow_idx}] {flow.src_ip} -> {flow.dst_ip}")
|
||||
print("-" * 60)
|
||||
|
||||
# Flow summary
|
||||
print(f"Total Packets: {flow.frame_count:,}")
|
||||
print(f"Total Bytes: {flow.total_bytes:,}")
|
||||
print(f"Max Sigma Deviation: {max_sigma:.2f}σ")
|
||||
print(f"Protocols: {', '.join(flow.protocols)}")
|
||||
if flow.detected_protocol_types:
|
||||
print(f"Enhanced Protocols: {', '.join(flow.detected_protocol_types)}")
|
||||
|
||||
@@ -176,7 +176,11 @@ class TUIInterface:
|
||||
stdscr.addstr(status_y, 0, status[:width-1], curses.A_REVERSE)
|
||||
|
||||
def _get_flows_list(self):
|
||||
"""Get sorted list of flows"""
|
||||
"""Get sorted list of flows - prioritize by largest sigma outlier"""
|
||||
flows_list = list(self.analyzer.flows.values())
|
||||
flows_list.sort(key=lambda x: x.frame_count, reverse=True)
|
||||
# Sort by maximum sigma deviation first, then by frame count as secondary criterion
|
||||
flows_list.sort(key=lambda x: (
|
||||
self.analyzer.statistics_engine.get_max_sigma_deviation(x),
|
||||
x.frame_count
|
||||
), reverse=True)
|
||||
return flows_list
|
||||
Reference in New Issue
Block a user