cluster4npu/cluster4npu_ui/TODO.md

Cluster4NPU Pipeline TODO

Current Status

✅ Pipeline Core: Multi-stage pipeline with device auto-detection working
✅ Hardware Integration: Kneron NPU dongles connecting and initializing successfully
✅ Auto-resize Preprocessing: Model input shape detection and automatic preprocessing implemented
❌ Data Input Sources: Missing camera and file input implementations
❌ Result Persistence: No result saving or output mechanisms
❌ End-to-End Workflow: Gaps between UI configuration and core pipeline execution

---

Priority 1: Essential Components for Complete Inference Workflow

1. Data Source Implementation

Status: 🔴 Critical Missing Components
Location: Need to create new classes in core/functions/ or extend existing ones

1.1 Camera Input Source

• File: core/functions/camera_source.py (new)
• Class: CameraSource
• Purpose: Wrapper around cv2.VideoCapture for camera input
• Integration: Connect to InferencePipeline.put_data()
• Features:
  • Multiple camera index support
  • Resolution and FPS configuration
  • Format conversion (BGR → model input format)
  • Error handling for camera disconnection

1.2 Video File Input Source

• File: core/functions/video_source.py (new)
• Class: VideoFileSource
• Purpose: Process video files frame by frame
• Integration: Feed frames to InferencePipeline
• Features:
  • Support common video formats (MP4, AVI, MOV)
  • Frame rate control and seeking
  • Batch processing capabilities
  • Progress tracking

1.3 Image File Input Source

• File: core/functions/image_source.py (new)
• Class: ImageFileSource
• Purpose: Process single images or image directories
• Integration: Single-shot inference through pipeline
• Features:
  • Support common image formats (JPG, PNG, BMP)
  • Batch directory processing
  • Image validation and error handling

1.4 RTSP/HTTP Stream Source

• File: core/functions/stream_source.py (new)
• Class: RTSPSource, HTTPStreamSource
• Purpose: Process live video streams
• Integration: Real-time streaming to pipeline
• Features:
  • Stream connection management
  • Reconnection on failure
  • Buffer management and frame dropping

2. Result Persistence System

Status: 🔴 Critical Missing Components
Location: core/functions/result_handler.py (new)

2.1 Result Serialization

• Class: ResultSerializer
• Purpose: Convert inference results to standard formats
• Features:
  • JSON export with timestamps
  • CSV export for analytics
  • Binary format for performance
  • Configurable fields and formatting

2.2 File Output Manager

• Class: FileOutputManager
• Purpose: Handle result file writing and organization
• Features:
  • Timestamped file naming
  • Directory organization by date/pipeline
  • File rotation and cleanup
  • Output format configuration

2.3 Real-time Result Streaming

• Class: ResultStreamer
• Purpose: Stream results to external systems
• Features:
  • WebSocket result broadcasting
  • REST API endpoints
  • Message queue integration (Redis, RabbitMQ)
  • Custom callback system

3. Input/Output Integration Bridge

Status: 🔴 Critical Missing Components
Location: core/functions/pipeline_manager.py (new)

3.1 Pipeline Configuration Manager

• Class: PipelineConfigManager
• Purpose: Convert UI configurations to executable pipelines
• Integration: Bridge between UI and core pipeline
• Features:
  • Parse UI node configurations
  • Instantiate appropriate data sources
  • Configure result handlers
  • Manage pipeline lifecycle

3.2 Unified Workflow Orchestrator

• Class: WorkflowOrchestrator
• Purpose: Coordinate complete data flow from input to output
• Features:
  • Input source management
  • Pipeline execution control
  • Result handling and persistence
  • Error recovery and logging

---

Priority 2: Enhanced Preprocessing and Auto-resize

4. Enhanced Preprocessing System

Status: 🟡 Partially Implemented
Location: core/functions/Multidongle.py (existing) + new preprocessing modules

4.1 Current Auto-resize Implementation

• Location: Multidongle.py:354-371 (preprocess_frame method)
• Features: ✅ Already implemented
  • Automatic model input shape detection
  • Dynamic resizing based on model requirements
  • Format conversion (BGR565, RGB8888, YUYV, RAW8)
  • Aspect ratio handling

4.2 Enhanced Preprocessing Pipeline

• File: core/functions/preprocessor.py (new)
• Class: AdvancedPreprocessor
• Purpose: Extended preprocessing capabilities
• Features:
  • Smart cropping: Maintain aspect ratio with intelligent cropping
  • Normalization: Configurable pixel value normalization
  • Augmentation: Real-time data augmentation for training
  • Multi-model support: Different preprocessing for different models
  • Caching: Preprocessed frame caching for performance

4.3 Model-Aware Preprocessing

• Enhancement: Extend existing Multidongle class
• Location: core/functions/Multidongle.py:188-199 (model_input_shape detection)
• Features:
  • Dynamic preprocessing: Adjust preprocessing based on model metadata
  • Model-specific optimization: Tailored preprocessing for different model types
  • Preprocessing profiles: Saved preprocessing configurations per model

---

Priority 3: UI Integration and User Experience

5. Dashboard Integration

Status: 🟡 Partially Implemented
Location: ui/windows/dashboard.py (existing)

5.1 Real-time Pipeline Monitoring

• Enhancement: Extend existing Dashboard class
• Features:
  • Live inference statistics
  • Real-time result visualization
  • Performance metrics dashboard
  • Error monitoring and alerts

5.2 Input Source Configuration

• Integration: Connect UI input nodes to actual data sources
• Features:
  • Camera selection and preview
  • File browser integration
  • Stream URL validation
  • Input source testing

6. Result Visualization

Status: 🔴 Not Implemented
Location: ui/widgets/result_viewer.py (new)

6.1 Result Display Widget

• Class: ResultViewer
• Purpose: Display inference results in UI
• Features:
  • Real-time result streaming
  • Result history and filtering
  • Export capabilities
  • Customizable display formats

---

Priority 4: Advanced Features and Optimization

7. Performance Optimization

Status: 🟡 Basic Implementation
Location: Multiple files

7.1 Memory Management

• Enhancement: Optimize existing queue systems
• Files: InferencePipeline.py, Multidongle.py
• Features:
  • Smart queue sizing based on available memory
  • Frame dropping under load
  • Memory leak detection and prevention
  • Garbage collection optimization

7.2 Multi-device Load Balancing

• Enhancement: Extend existing multi-dongle support
• Location: core/functions/Multidongle.py (existing auto-detection)
• Features:
  • Intelligent device allocation
  • Load balancing across devices
  • Device health monitoring
  • Automatic failover

8. Error Handling and Recovery

Status: 🟡 Basic Implementation
Location: Throughout codebase

8.1 Comprehensive Error Recovery

• Enhancement: Extend existing error handling
• Features:
  • Automatic device reconnection
  • Pipeline restart on critical errors
  • Input source recovery
  • Result persistence on failure

---

Implementation Roadmap

Phase 1: Core Data Flow (Weeks 1-2)

1. ✅ Complete: Pipeline deployment and device initialization
2. 🔄 In Progress: Auto-resize preprocessing (mostly implemented)
3. Next: Implement basic camera input source
4. Next: Add simple result file output
5. Next: Create basic pipeline manager

Phase 2: Complete Workflow (Weeks 3-4)

1. Add video file input support
2. Implement comprehensive result persistence
3. Create UI integration bridge
4. Add real-time monitoring

Phase 3: Advanced Features (Weeks 5-6)

1. Enhanced preprocessing pipeline
2. Performance optimization
3. Advanced error handling
4. Result visualization

Phase 4: Production Features (Weeks 7-8)

1. Multi-device load balancing
2. Advanced stream input support
3. Analytics and reporting
4. Configuration management

---

Key Code Locations for Current Auto-resize Implementation

Model Input Shape Detection

• File: core/functions/Multidongle.py
• Lines: 188-199 (model_input_shape property)
• Status: ✅ Working - detects model input dimensions from NEF files

Automatic Preprocessing

• File: core/functions/Multidongle.py
• Lines: 354-371 (preprocess_frame method)
• Status: ✅ Working - auto-resizes based on model input shape
• Features: Format conversion, aspect ratio handling

Pipeline Data Processing

• File: core/functions/InferencePipeline.py
• Lines: 165-240 (_process_data method)
• Status: ✅ Working - integrates preprocessing with inference
• Features: Inter-stage processing, result accumulation

Format Conversion

• File: core/functions/Multidongle.py
• Lines: 382-396 (_convert_format method)
• Status: ✅ Working - supports BGR565, RGB8888, YUYV, RAW8

---

Notes for Development

1. Auto-resize is already implemented ✅ - The system automatically detects model input shape and resizes accordingly
2. Priority should be on input sources - Camera and file input are the critical missing pieces
3. Result persistence is essential - Current system only provides callbacks, need file output
4. UI integration gap - UI configuration doesn't connect to core pipeline execution
5. Performance is good - Multi-threading and device management are solid foundations

The core pipeline and preprocessing are working well - the focus should be on completing the input/output ecosystem around the existing robust inference engine.