kneron_model_converter

warrenchen/kneron_model_converter

Fork 0

Commit Graph

Author	SHA1	Message	Date
jim800121chen	0cae1b1130	feat(worker): 啟用 staging 真實轉檔（取代 stub）+ 修 ref_images S3 path staging worker 原本跑 WORKER_MODE=stub（假轉檔、產佔位 NEF），visionA 因此反映拿到的是 stub。本次讓真實轉檔上線並打通 onnx→bie→nef 完整鏈路。新增： - services/workers/Dockerfile.real：真實版 worker image，COPY repo 自帶 ktc/ + toolchain/prebuild（2GB），設 USE_PREBUILD/LD_LIBRARY_PATH/PYTHONPATH/KTC_DISABLE_MP/WORKER_MODE=real；排除 libs/dynasty+libs/compiler 省 3.2G - .dockerignore：控制 build context（排除被 USE_PREBUILD 取代的舊系統 binaries） - docker-compose.real.yml：override 三 worker 改用 registry image，保留 base stub build 供一鍵 rollback 修正（e2e 抓到的兩個問題）： - jobService.js: ref_images S3 key 由 jobs/{id}/ref_images/ 改為 jobs/{id}/input/ref_images/，對齊系統主約定（local.js/legacy.js/bie worker consumer.py），修好 bie 量化抓不到 dataset 的 blocker。加 contract guard test 鎖死寫入 prefix == worker 讀取 prefix - security.md: 同步 ref_images S3 key 路徑描述驗證（staging e2e）： - platform 530（與模型相符）下 onnx→bie→nef 全綠，產真 NEF 636KB（合法 FlatBuffer、非 STUB），/result 可拉、結果上 MinIO - 已知約定：呼叫端 platform 必須與模型量化 platform 一致（配錯會在 bie 量化階段失敗） scheduler 全套件 667 tests pass。 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-22 12:43:24 +08:00
jim800121chen	d8a9517c9d	feat(task-scheduler): Phase 0.8b — API key auth + /result endpoint Auth pillar 從 OAuth 2.0 resource server 改成 pre-shared API key （visionA ↔ converter 1:1 internal trust）。新增 GET /api/v1/jobs/:id/result streaming endpoint 給 visionA backend 中轉 NEF 下載。 Phase A（auth 切換）： - 新增 apiKeyMiddleware（constant-time compare、tokenFingerprint、4 audit events） - 砍 OAuth middleware + JWKS（保留 oauthClient 供 promote → FAA 使用） - 4 個 endpoint 換掛 requireApiKey - 加 TRUST_PROXY env + Express trust proxy 設定（forensic source_ip） Phase B（/result endpoint）： - streaming NEF download with 5min timeout + concurrent cap 10 - Two-tier rate limit（burst 5/10s + sustained 20/min） - Bandwidth quota（1 GB/hr + 6 GB/24hr）by token_fingerprint - Range header silently ignored + Accept-Ranges: none - filename quote-escape + RFC 5987 fallback + sanitize - 8 個 /result audit events（forensic 完整）設計演進記錄：docs/TODO-visionA-integration-v2.md（5/2 OAuth → 5/16 API key → 5/16 download via converter；對應 visionA repo ADR-015/016） Tests: 597 → 666 (+69)、29 suites all pass Security: APPROVE WITH CONDITIONS（單 instance 部署、6 新 env、24hr 監控） npm audit: 3 vuln → 0（transitive AWS SDK xml chain） Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 22:47:28 +08:00
jim800121chen	cff9236699	docs: migrate Autoflow shared documents to docs/autoflow/ Move PRD, design specs, architecture docs, and TDD from .autoflow/ (personal/per-branch layer) to docs/autoflow/ (shared layer that goes into git) per the new Autoflow workspace layout. Files moved: - 02-prd/PRD.md - 03-design/design-review.md - 03-design/user-flow-cross-system.md - 04-architecture/TDD.md - 04-architecture/design-doc.md - 04-architecture/security.md The originals were never tracked, so git mv reduced to a filesystem rename with no history to preserve. .autoflow/ remains for personal notes (progress.md, review reports, testing logs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-01 10:59:21 +08:00

Author

SHA1

Message

Date

jim800121chen

0cae1b1130

feat(worker): 啟用 staging 真實轉檔（取代 stub）+ 修 ref_images S3 path

staging worker 原本跑 WORKER_MODE=stub（假轉檔、產佔位 NEF），visionA 因此反映拿到的是 stub。本次讓真實轉檔上線並打通 onnx→bie→nef 完整鏈路。

新增：
- services/workers/Dockerfile.real：真實版 worker image，COPY repo 自帶 ktc/ + toolchain/prebuild（2GB），設 USE_PREBUILD/LD_LIBRARY_PATH/PYTHONPATH/KTC_DISABLE_MP/WORKER_MODE=real；排除 libs/dynasty+libs/compiler 省 3.2G
- .dockerignore：控制 build context（排除被 USE_PREBUILD 取代的舊系統 binaries）
- docker-compose.real.yml：override 三 worker 改用 registry image，保留 base stub build 供一鍵 rollback

修正（e2e 抓到的兩個問題）：
- jobService.js: ref_images S3 key 由 jobs/{id}/ref_images/ 改為 jobs/{id}/input/ref_images/，對齊系統主約定（local.js/legacy.js/bie worker consumer.py），修好 bie 量化抓不到 dataset 的 blocker。加 contract guard test 鎖死寫入 prefix == worker 讀取 prefix
- security.md: 同步 ref_images S3 key 路徑描述

驗證（staging e2e）：
- platform 530（與模型相符）下 onnx→bie→nef 全綠，產真 NEF 636KB（合法 FlatBuffer、非 STUB），/result 可拉、結果上 MinIO
- 已知約定：呼叫端 platform 必須與模型量化 platform 一致（配錯會在 bie 量化階段失敗）

scheduler 全套件 667 tests pass。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-22 12:43:24 +08:00

jim800121chen

d8a9517c9d

feat(task-scheduler): Phase 0.8b — API key auth + /result endpoint

Auth pillar 從 OAuth 2.0 resource server 改成 pre-shared API key
（visionA ↔ converter 1:1 internal trust）。新增 GET /api/v1/jobs/:id/result
streaming endpoint 給 visionA backend 中轉 NEF 下載。

Phase A（auth 切換）：
- 新增 apiKeyMiddleware（constant-time compare、tokenFingerprint、4 audit events）
- 砍 OAuth middleware + JWKS（保留 oauthClient 供 promote → FAA 使用）
- 4 個 endpoint 換掛 requireApiKey
- 加 TRUST_PROXY env + Express trust proxy 設定（forensic source_ip）

Phase B（/result endpoint）：
- streaming NEF download with 5min timeout + concurrent cap 10
- Two-tier rate limit（burst 5/10s + sustained 20/min）
- Bandwidth quota（1 GB/hr + 6 GB/24hr）by token_fingerprint
- Range header silently ignored + Accept-Ranges: none
- filename quote-escape + RFC 5987 fallback + sanitize
- 8 個 /result audit events（forensic 完整）

設計演進記錄：docs/TODO-visionA-integration-v2.md（5/2 OAuth → 5/16 API key
→ 5/16 download via converter；對應 visionA repo ADR-015/016）

Tests: 597 → 666 (+69)、29 suites all pass
Security: APPROVE WITH CONDITIONS（單 instance 部署、6 新 env、24hr 監控）
npm audit: 3 vuln → 0（transitive AWS SDK xml chain）

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-17 22:47:28 +08:00

jim800121chen

cff9236699

docs: migrate Autoflow shared documents to docs/autoflow/

Move PRD, design specs, architecture docs, and TDD from .autoflow/
(personal/per-branch layer) to docs/autoflow/ (shared layer that
goes into git) per the new Autoflow workspace layout.

Files moved:
- 02-prd/PRD.md
- 03-design/design-review.md
- 03-design/user-flow-cross-system.md
- 04-architecture/TDD.md
- 04-architecture/design-doc.md
- 04-architecture/security.md

The originals were never tracked, so git mv reduced to a filesystem
rename with no history to preserve. .autoflow/ remains for personal
notes (progress.md, review reports, testing logs).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-01 10:59:21 +08:00

3 Commits