StreamLens/UPDATE_RATE_SUMMARY.md

StreamLens Update Rate Optimization Summary

Problem

The TUI was updating too frequently during PCAP parsing, causing:

• High CPU usage during parsing
• Choppy/stuttering interface
• Poor responsiveness due to excessive updates

Solution

Slowed down all update rates across the parsing pipeline and TUI layers.

Changes Made

1. Background Analyzer Updates (background_analyzer.py)

Flow Update Batch Size:

• Before: Every 10 packets (very frequent)
• After: Every 100 packets (10x slower)
• Impact: Reduces flow update callbacks from UI threads

Progress Monitor Updates:

• Before: Every 0.5 seconds
• After: Every 2.0 seconds (4x slower)
• Impact: Progress bar and parsing stats update less frequently

Monitor Thread Sleep:

• Before: 0.1 second sleep between checks
• After: 0.5 second sleep between checks (5x slower)
• Impact: Reduces background thread CPU usage

2. TUI Update Timers (app_v2.py)

Metrics Timer:

• Before: Every 2.0 seconds (0.5 Hz)
• After: Every 5.0 seconds (0.2 Hz)
• Impact: Flow counts, packet rates, outlier counts update less frequently

Flow Timer:

• Before: Every 5.0 seconds (0.2 Hz)
• After: Every 10.0 seconds (0.1 Hz)
• Impact: Flow table data refreshes less frequently

Performance Benefits

CPU Usage Reduction

• Flow callbacks: 10x reduction (100 packets vs 10 packets)
• Progress updates: 4x reduction (2.0s vs 0.5s intervals)
• Monitor overhead: 5x reduction (0.5s vs 0.1s sleep)
• TUI metrics: 2.5x reduction (5.0s vs 2.0s intervals)
• TUI flows: 2x reduction (10.0s vs 5.0s intervals)

User Experience Improvements

• ✅ Smoother parsing - Less frequent UI interruptions
• ✅ Lower CPU usage - More resources for actual packet processing
• ✅ Stable interface - Buttons and data don't flicker/jump
• ✅ Better responsiveness - UI isn't blocked by constant updates
• ✅ Maintained functionality - All features still work, just update slower

Update Timeline During Parsing

Fast File (< 1000 packets)

• Progress updates every 2 seconds
• Flow updates every 100 packets (could be ~1-5 times total)
• TUI refreshes every 5-10 seconds

Medium File (1000-10000 packets)

• Progress updates every 2 seconds
• Flow updates every 100 packets (~10-100 times total)
• TUI refreshes every 5-10 seconds

Large File (> 10000 packets)

• Progress updates every 2 seconds
• Flow updates every 100 packets (100+ times total)
• TUI refreshes every 5-10 seconds

Technical Details

Thread Safety

• All update rate changes maintain thread safety
• Background parsing still uses proper locks and queues
• UI updates still use call_from_thread() for thread-safe UI updates

Button Visibility Fix (Bonus)

• Buttons now pre-created at TUI initialization instead of dynamic creation
• Eliminates button creation delays during parsing
• Frame type buttons visible immediately: [1. Overview] [2. CH10-Data] [3. UDP] etc.

Backward Compatibility

• All existing functionality preserved
• Same keyboard shortcuts (1-9,0 for filtering)
• Same data accuracy and completeness
• Same analysis features and reports

Testing

Successfully tested with:

• ✅ Button creation at initialization
• ✅ Slower update rate configuration
• ✅ Real PCAP file parsing (1 PTPGM.pcapng)
• ✅ TUI responsiveness and button visibility
• ✅ Parsing completion and final results

Conclusion

The PCAP parsing is now much smoother with significantly reduced CPU usage while maintaining all functionality. Users will experience:

• 10x fewer flow update interruptions
• 4x fewer progress update interruptions
• 5x less monitor thread overhead
• 2-3x fewer TUI refresh interruptions

The interface remains fully functional but is now much more pleasant to use during parsing operations.