masonhuang/cluster4npu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
cluster4npu/cluster4npu_ui/TODO.md

10 KiB
Raw Blame History

Cluster4NPU Pipeline TODO

Current Status

✅ Pipeline Core: Multi-stage pipeline with device auto-detection working
✅ Hardware Integration: Kneron NPU dongles connecting and initializing successfully
✅ Auto-resize Preprocessing: Model input shape detection and automatic preprocessing implemented
✅ Data Input Sources: Camera and video file inputs implemented
✅ Result Persistence: Result saving to file implemented
✅ End-to-End Workflow: UI configuration now connects to core pipeline execution
✅ Bug Fixes: Addressed file path and data processing issues ✅ Real-time Viewer: Implemented a live view for real-time inference visualization

Priority 1: Essential Components for Complete Inference Workflow

1. Data Source Implementation

Status: 🔴 Critical Missing Components
Location: Need to create new classes in core/functions/ or extend existing ones

1.1 Camera Input Source

  • File: core/functions/camera_source.py (new)
  • Class: CameraSource
  • Purpose: Wrapper around cv2.VideoCapture for camera input
  • Integration: Connect to InferencePipeline.put_data()
  • Features:
    • Multiple camera index support
    • Resolution and FPS configuration
    • Format conversion (BGR → model input format)
    • Error handling for camera disconnection

1.2 Video File Input Source

  • File: core/functions/video_source.py (new)
  • Class: VideoFileSource
  • Purpose: Process video files frame by frame
  • Integration: Feed frames to InferencePipeline
  • Features:
    • Support common video formats (MP4, AVI, MOV)
    • Frame rate control and seeking
    • Batch processing capabilities
    • Progress tracking

1.3 Image File Input Source

  • File: core/functions/image_source.py (new)
  • Class: ImageFileSource
  • Purpose: Process single images or image directories
  • Integration: Single-shot inference through pipeline
  • Features:
    • Support common image formats (JPG, PNG, BMP)
    • Batch directory processing
    • Image validation and error handling

1.4 RTSP/HTTP Stream Source

  • File: core/functions/stream_source.py (new)
  • Class: RTSPSource, HTTPStreamSource
  • Purpose: Process live video streams
  • Integration: Real-time streaming to pipeline
  • Features:
    • Stream connection management
    • Reconnection on failure
    • Buffer management and frame dropping

2. Result Persistence System

Status: 🔴 Critical Missing Components
Location: core/functions/result_handler.py (new)

2.1 Result Serialization

  • Class: ResultSerializer
  • Purpose: Convert inference results to standard formats
  • Features:
    • JSON export with timestamps
    • CSV export for analytics
    • Binary format for performance
    • Configurable fields and formatting

2.2 File Output Manager

  • Class: FileOutputManager
  • Purpose: Handle result file writing and organization
  • Features:
    • Timestamped file naming
    • Directory organization by date/pipeline
    • File rotation and cleanup
    • Output format configuration

2.3 Real-time Result Streaming

  • Class: ResultStreamer
  • Purpose: Stream results to external systems
  • Features:
    • WebSocket result broadcasting
    • REST API endpoints
    • Message queue integration (Redis, RabbitMQ)
    • Custom callback system

3. Input/Output Integration Bridge

Status: 🔴 Critical Missing Components
Location: core/functions/pipeline_manager.py (new)

3.1 Pipeline Configuration Manager

  • Class: PipelineConfigManager
  • Purpose: Convert UI configurations to executable pipelines
  • Integration: Bridge between UI and core pipeline
  • Features:
    • Parse UI node configurations
    • Instantiate appropriate data sources
    • Configure result handlers
    • Manage pipeline lifecycle

3.2 Unified Workflow Orchestrator

  • Class: WorkflowOrchestrator
  • Purpose: Coordinate complete data flow from input to output
  • Features:
    • Input source management
    • Pipeline execution control
    • Result handling and persistence
    • Error recovery and logging

Priority 2: Enhanced Preprocessing and Auto-resize

4. Enhanced Preprocessing System

Status: 🟡 Partially Implemented
Location: core/functions/Multidongle.py (existing) + new preprocessing modules

4.1 Current Auto-resize Implementation

  • Location: Multidongle.py:354-371 (preprocess_frame method)
  • Features: ✅ Already implemented
    • Automatic model input shape detection
    • Dynamic resizing based on model requirements
    • Format conversion (BGR565, RGB8888, YUYV, RAW8)
    • Aspect ratio handling

4.2 Enhanced Preprocessing Pipeline

  • File: core/functions/preprocessor.py (new)
  • Class: AdvancedPreprocessor
  • Purpose: Extended preprocessing capabilities
  • Features:
    • Smart cropping: Maintain aspect ratio with intelligent cropping
    • Normalization: Configurable pixel value normalization
    • Augmentation: Real-time data augmentation for training
    • Multi-model support: Different preprocessing for different models
    • Caching: Preprocessed frame caching for performance

4.3 Model-Aware Preprocessing

  • Enhancement: Extend existing Multidongle class
  • Location: core/functions/Multidongle.py:188-199 (model_input_shape detection)
  • Features:
    • Dynamic preprocessing: Adjust preprocessing based on model metadata
    • Model-specific optimization: Tailored preprocessing for different model types
    • Preprocessing profiles: Saved preprocessing configurations per model

Priority 3: UI Integration and User Experience

5. Dashboard Integration

Status: 🟡 Partially Implemented
Location: ui/windows/dashboard.py (existing)

5.1 Real-time Pipeline Monitoring

  • Enhancement: Extend existing Dashboard class
  • Features:
    • Live inference statistics
    • Real-time result visualization
    • Performance metrics dashboard
    • Error monitoring and alerts

5.2 Input Source Configuration

  • Integration: Connect UI input nodes to actual data sources
  • Features:
    • Camera selection and preview
    • File browser integration
    • Stream URL validation
    • Input source testing

6. Result Visualization

Status: 🔴 Not Implemented
Location: ui/widgets/result_viewer.py (new)

6.1 Result Display Widget

  • Class: ResultViewer
  • Purpose: Display inference results in UI
  • Features:
    • Real-time result streaming
    • Result history and filtering
    • Export capabilities
    • Customizable display formats

Priority 4: Advanced Features and Optimization

7. Performance Optimization

Status: 🟡 Basic Implementation
Location: Multiple files

7.1 Memory Management

  • Enhancement: Optimize existing queue systems
  • Files: InferencePipeline.py, Multidongle.py
  • Features:
    • Smart queue sizing based on available memory
    • Frame dropping under load
    • Memory leak detection and prevention
    • Garbage collection optimization

7.2 Multi-device Load Balancing

  • Enhancement: Extend existing multi-dongle support
  • Location: core/functions/Multidongle.py (existing auto-detection)
  • Features:
    • Intelligent device allocation
    • Load balancing across devices
    • Device health monitoring
    • Automatic failover

8. Error Handling and Recovery

Status: 🟡 Basic Implementation
Location: Throughout codebase

8.1 Comprehensive Error Recovery

  • Enhancement: Extend existing error handling
  • Features:
    • Automatic device reconnection
    • Pipeline restart on critical errors
    • Input source recovery
    • Result persistence on failure

Implementation Roadmap

Phase 1: Core Data Flow (Weeks 1-2)

  1. ✅ Complete: Pipeline deployment and device initialization
  2. 🔄 In Progress: Auto-resize preprocessing (mostly implemented)
  3. Next: Implement basic camera input source
  4. Next: Add simple result file output
  5. Next: Create basic pipeline manager

Phase 2: Complete Workflow (Weeks 3-4)

  1. Add video file input support
  2. Implement comprehensive result persistence
  3. Create UI integration bridge
  4. Add real-time monitoring

Phase 3: Advanced Features (Weeks 5-6)

  1. Enhanced preprocessing pipeline
  2. Performance optimization
  3. Advanced error handling
  4. Result visualization

Phase 4: Production Features (Weeks 7-8)

  1. Multi-device load balancing
  2. Advanced stream input support
  3. Analytics and reporting
  4. Configuration management

Key Code Locations for Current Auto-resize Implementation

Model Input Shape Detection

  • File: core/functions/Multidongle.py
  • Lines: 188-199 (model_input_shape property)
  • Status: ✅ Working - detects model input dimensions from NEF files

Automatic Preprocessing

  • File: core/functions/Multidongle.py
  • Lines: 354-371 (preprocess_frame method)
  • Status: ✅ Working - auto-resizes based on model input shape
  • Features: Format conversion, aspect ratio handling

Pipeline Data Processing

  • File: core/functions/InferencePipeline.py
  • Lines: 165-240 (_process_data method)
  • Status: ✅ Working - integrates preprocessing with inference
  • Features: Inter-stage processing, result accumulation

Format Conversion

  • File: core/functions/Multidongle.py
  • Lines: 382-396 (_convert_format method)
  • Status: ✅ Working - supports BGR565, RGB8888, YUYV, RAW8

Notes for Development

  1. Auto-resize is already implemented ✅ - The system automatically detects model input shape and resizes accordingly
  2. Priority should be on input sources - Camera and file input are the critical missing pieces
  3. Result persistence is essential - Current system only provides callbacks, need file output
  4. UI integration gap - UI configuration doesn't connect to core pipeline execution
  5. Performance is good - Multi-threading and device management are solid foundations

The core pipeline and preprocessing are working well - the focus should be on completing the input/output ecosystem around the existing robust inference engine.