warrenchen/kneron_model_converter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
29 Commits 1 Branch 0 Tags
Commit Graph

29 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
jim800121
407fe10a8c Merge pull request 'feat(worker): onnx 階段自動移除尾端 Softmax + pre-check regex 收窄' (#6) from feat/softmax-tail-removal into main 2026-07-05 21:55:39 +00:00
jim800121chen
1c53253e6e feat(worker): onnx 階段自動移除尾端 Softmax + pre-check regex 收窄 
問題：同一顆含 Softmax 的 tflite，正式站能編出 KL520 真 .nef，
本專案 worker 在 nef 階段撞 UnimplementedFeature [Softmax] exit 6。
根因：runtime ktc 的 eliminate_tail 是 no-op（閹割版、只印警告），
尾端 Softmax 未被移除；正式站產物實測為 logits 輸出（Softmax 已砍）。

修法（staging 實測驗證、與正式站產物權重段逐 byte 相同）：
- onnx/core.py：新增 remove_tail_softmax()，onnx2onnx_flow 後對所有
  platform 移除 terminal Softmax（cut_nodes、不用 cut_types 避免誤砍
  中間層下游）、多 output rewire 逐項驗證 fail-loud、onnx.checker、
  removed_tail_softmax metadata
- precheck.py：R3 誤擋收窄——刪 hw_not_support_col 欄位名誤命中、
  op capture 改必須、抽不到具體 op 名改放行+warning（真 fail-open）、
  刪自由文字猜 op fallback、加噪音 token 過濾
- tests：+16（29 passed；terminal/no-op/多 output/fail-open 全覆蓋）
- docs：TDD §12、design-doc ADR-012、PRD 輸出語意變更（機率→logits）

行為變更：所有 platform NEF 輸出改 logits（與正式站一致）、
分類後處理需呼叫端自行補 softmax。

Review：兩輪（0C/3M→0C/0M）；W3 docker integration 6/6 PASS
（520+Softmax e2e 產真 .nef 799,844 bytes、720 回歸、precheck 放行）。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-06 05:54:13 +08:00
jim800121
25ce7f87a8 Merge pull request 'chore(deploy): worker tag → real-precheck-20260703' (#5) from chore/compose-precheck-tag into main 2026-07-02 19:03:13 +00:00
jim800121chen
bc58bb3f20 chore(deploy): worker image tag 更新為 real-precheck-20260703 
staging 三 worker 已 pull + 重啟成含 platform pre-check 的 image
（digest sha256:522726ff...）。同步 compose image tag，舊 tag
real-tflite-20260702 保留註解供 rollback。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-03 03:02:59 +08:00
jim800121
cf4426bd6c Merge pull request 'feat(worker): platform/operator pre-check（不支援 op 早期失敗）' (#4) from feat/platform-precheck into main 2026-07-02 18:51:26 +00:00
jim800121chen
5a9c1b0140 feat(worker): 加 platform/operator pre-check（不支援 op 早期快速失敗） 
轉檔時若 op 在目標 platform 不支援（如 Softmax@520），原本會白跑完 onnx+bie
兩階段、最後在 nef 的 batch_compile 拋難懂的 C++ backtrace（undefined CPU op）。
本次在 onnx 階段（拿到 out.onnx 後、進 bie 前）用 Kneron IP Evaluator
（ktc ModelConfig.evaluate / ip_evaluator mode 0 會跑 compiler frontend）
早期偵測不支援 op，快速失敗並回清楚訊息。

- 新增 services/backends/precheck.py：run_precheck 掃 evaluate 的不支援訊號
  （HardwareNotSupport / UnimplementedFeature / undefined CPU op / EmptyNode）
  抽出 op 名、raise UnsupportedOperatorError；錯誤走既有 _push_done(fail) 回報
- services/workers/onnx/core.py：enable_precheck default-on；pre-check 與既有
  evaluate 共用同一次 evaluate() 呼叫（避免跑兩次 compiler frontend）
- error 訊息：「platform 520 不支援 operator Softmax，請改用支援的 platform
  （如 720/730）或將該 operator 移至 host 端後處理。」
- fail-open：只有明確訊號才擋、evaluate 環境問題無訊號則放行（不誤擋能跑的模型）
- 17 tests pass（含多 op / dedup）

follow-up（部署後、需 docker toolchain 環境）：integration smoke 真觸發
Softmax@520 確認 onnx 階段就擋；依實測 frontend 耗時評估是否收窄 default-on 粒度。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-03 02:51:07 +08:00
jim800121
57ffc619f9 Merge pull request 'chore(deploy): worker image tag 更新為 real-tflite（含 tensorflow）' (#3) from chore/compose-tflite-tag into main 2026-07-02 07:46:20 +00:00
jim800121chen
7e8ced1d90 chore(deploy): worker image tag 更新為 real-tflite-20260702（含 tensorflow） 
staging 三個 worker 已 pull + 重啟成加了 tensorflow 的 tflite-capable image
（digest sha256:29bc3f60...）。同步 docker-compose.real.yml 的 image tag，
避免下次從 git 部署用到舊的 real-20260622（無 TF）。舊 tag 保留在註解供 rollback。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-02 15:46:01 +08:00
jim800121
9dfd0e4d77 Merge pull request 'fix(worker): real image 加 tensorflow 支援 tflite' (#2) from fix/tflite-tensorflow into main 2026-07-02 07:32:53 +00:00
jim800121chen
21f863ae1e fix(worker): real image 加 tensorflow 支援 tflite 輸入格式 
staging real worker 轉 tflite 失敗：tflite→onnx 需要 import tensorflow，但
Dockerfile.real 當初為省 image 大小沒裝 TF（只驗過 onnx/bie/nef 輸入）。

改 services/workers/Dockerfile.real：
- 加 tensorflow-cpu==2.13.1（Python 3.9 相容 + numpy<2 相容 + ktc TF compat
  patch 預期範圍的交集；-cpu 省 ~1.5GB CUDA）
- 加 igraph==1.0.0（tree_structure.py 硬 import、pin 對齊本檔版本鎖哲學）
- 補 typing-extensions>=4.8.0（TF 會把它降版打爆 fastapi import）
- numpy<2 精確化為 numpy==1.23.5（落在原 <2 範圍 + TF 接受窗口、語意子集非破鎖）

本機驗證：image 4.36GB→5.71GB；tflite→onnx 通（產 188 節點合法 ONNX）；既有
BIE→NEF 回歸產出同一顆真 NEF（621,956 bytes）、prebuild ldd 全 resolved。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-02 15:31:59 +08:00
jim800121
a440600807 Merge pull request 'feat: 啟用 staging 真實轉檔 + 修 ref_images/promote OAuth bug' (#1) from feat/real-conversion-worker into main 2026-06-24 20:47:36 +00:00
jim800121chen
f3f0cf1c40 fix(auth): promote 換 token 改用 client_secret_post（修 MC 401 invalid_client） 
promote 流程的 OAuth client 去 Member Center（OpenIddict）換 service token 時
用 HTTP Basic Auth 送 client 憑證，但 MC 只接受 client_secret_post（憑證放 form
body），導致 MC 回 401 invalid_client → promote 拿不到 token 去 FAA PUT → 對
visionA 回 500。根因由 visionA 端實測定位（交接檔 converter-promote-oauth-handoff.md）。

改 apps/task-scheduler/src/auth/oauthClient.js：
- client_id/client_secret 從 Authorization Basic header 移進 form body（既有 body
  上新增、保留 grant_type/scope/audience）
- 移除 Authorization header 與 buildBasicAuthHeader() + _internals export
- 檔頭 design 註解更新為 client_secret_post，避免被改回 Basic

安全約束維持：client_secret 僅進必要 body、絕不進任何 log（既有 "secret never in
log" 測試跑全路徑 grep 驗證、現變成 body 帶 secret 情境的鎖）。

test：oauthClient.test.js 改斷言鎖「headers 無 Authorization + body 含 client_id/
client_secret」防回退。scheduler 全套件 666 pass。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-24 23:07:10 +08:00
jim800121chen
0cae1b1130 feat(worker): 啟用 staging 真實轉檔（取代 stub）+ 修 ref_images S3 path 
staging worker 原本跑 WORKER_MODE=stub（假轉檔、產佔位 NEF），visionA 因此反映拿到的是 stub。本次讓真實轉檔上線並打通 onnx→bie→nef 完整鏈路。

新增：
- services/workers/Dockerfile.real：真實版 worker image，COPY repo 自帶 ktc/ + toolchain/prebuild（2GB），設 USE_PREBUILD/LD_LIBRARY_PATH/PYTHONPATH/KTC_DISABLE_MP/WORKER_MODE=real；排除 libs/dynasty+libs/compiler 省 3.2G
- .dockerignore：控制 build context（排除被 USE_PREBUILD 取代的舊系統 binaries）
- docker-compose.real.yml：override 三 worker 改用 registry image，保留 base stub build 供一鍵 rollback

修正（e2e 抓到的兩個問題）：
- jobService.js: ref_images S3 key 由 jobs/{id}/ref_images/ 改為 jobs/{id}/input/ref_images/，對齊系統主約定（local.js/legacy.js/bie worker consumer.py），修好 bie 量化抓不到 dataset 的 blocker。加 contract guard test 鎖死寫入 prefix == worker 讀取 prefix
- security.md: 同步 ref_images S3 key 路徑描述

驗證（staging e2e）：
- platform 530（與模型相符）下 onnx→bie→nef 全綠，產真 NEF 636KB（合法 FlatBuffer、非 STUB），/result 可拉、結果上 MinIO
- 已知約定：呼叫端 platform 必須與模型量化 platform 一致（配錯會在 bie 量化階段失敗）

scheduler 全套件 667 tests pass。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-22 12:43:24 +08:00
jim800121chen
cbd1b9db28 fix(task-scheduler): Bug #10 — convention path fallback（visionA promote/result 拿不到 NEF） 
visionA e2e 撞到：promote / result endpoint 在 status=COMPLETED 仍拿不到
NEF（409 source_not_available / 404 result_not_found）。

根因：worker (services/workers/consumer.py:118) 把 NEF/BIE/ONNX 上傳到
固定 convention path `jobs/{job_id}/out.{output_name}`、但 scheduler 端
advanceJob (jobService.js:246) 沒接收 worker done event 的 output path、
所以 job.output.{source}_path 永遠 null、讀取端拿不到。

修法 A（讀取端 fallback、最低風險）：
- promote.js getJobOutputKey() + result.js extractNefObjectKey() 在
  status=COMPLETED + jobId 有效 + source ∈ {onnx,bie,nef} 時、反推
  convention path
- 不改 worker / 不改 advanceJob / 不改 redis schema
- fallback 放最後、保留 result_object_keys / output.{source}_path 兩種
  顯式設定優先級

Phase 2 backlog（待補完）：
- 補完 worker → scheduler done event 寫 output path
- advanceJob 接收 output path 並寫進 redis
- 清掉本批 fallback dead branch + promote 409 source_not_available
  dead branch（fallback 後 valid source 永遠拿得到 key）

Tests: 666/666 pass（無回歸）
Reviewer: ✅ 通過、guard 嚴格、對齊 worker convention、無 path traversal 風險

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 15:55:22 +08:00
jim800121chen
b8457ddb95 fix(task-scheduler): Bug #8 — terminal release 保留 job record（visionA poll 404） 
visionA e2e 撞到：job COMPLETED 後 < 1s poll GET /api/v1/jobs/:id 回 404。
根因：jobService terminal release（COMPLETED / FAILED）path 用了
release_active_job.lua、該 Lua 給 enqueue rollback 用、會 DEL job:{id}
+ SREM user:jobs Set、post-completion API 拿不到 record。

修法 A（不改舊 Lua、新增專用 Lua）：
- 新增 src/redis/luaScripts/release_lock_only.lua — 只 DEL active_job lock、
  保留 job:{id} record 與 user:{}:jobs Set、給正常 terminal 用
- 新增 releaseActiveLockOnly() JS wrapper（同 releaseActiveJob 的 API surface +
  atomic guard）
- jobService.js terminal release path 改用 releaseActiveLockOnly
- enqueue 失敗 rollback path 仍用舊 releaseActiveJob（語意正確、該情境 job
  尚未 schedule 完成、清乾淨才對）
- t9 unit / integration test 用 destructuring alias rename 避免改 27 個
  assertion

post-completion API 路徑（visionA poll / Phase B /result / promote）都需要
job record 仍在 — 本修法解此契約。

Tests: 666/666 pass（無回歸）
Reviewer: ✅ 通過、設計嚴謹（atomic guard、rollback vs terminal 語意分離）

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 14:02:54 +08:00
jim800121chen
3b7aa4c79a fix(task-scheduler): server.js 漏傳 minio dep 給 jobService（visionA e2e 撞到） 
visionA 跑 Phase 0.8b e2e 時 POST /api/v1/jobs 回 502 storage_unavailable。
根因：server.js 建立 jobService 時沒把 minio facade 傳進去、
jobService.js 走 `deps.minio || null` fallback、writeInputToMinIO()
因為 minio=null throw「minio dep is required」、API 回 502。

修法：傳 minio facade 進 createJobService deps。
legacy CRUD 介面（不依賴 minio）行為不變—minio 是 optional dep。

Tests: 666/666 pass（無回歸）
Reviewer: ✅ 通過、correctness 軸無 Critical/Major

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 12:12:55 +08:00
jim800121chen
aeaecb8c06 fix(compose): Phase 0.8b deploy blocker — env 透傳 + 命名規格 
d8a9517 commit 漏改 docker-compose.yml：scheduler service environment block
沒透傳 Phase 0.8b 新 env、即使 stage .env 設了 container 也讀不到、
deploy 後 CONVERTER_API_KEY undefined 會啟動 503 reject all requests。

docker-compose.yml：
- 新增 10 個 Phase 0.8b env 透傳（CONVERTER_API_KEY 無 default fail-secure、
  其他用 ${VAR:-default} fail-soft）
- 砍 9 個已廢 OAuth resource-server env（MEMBER_CENTER_ISSUER / JWKS_URL /
  AUDIENCE / CONVERTER_TENANT_ID / SCOPE_* / JWKS_* / JWT_*）
- 保留 8 個 promote → FAA 用 env（MEMBER_CENTER_TOKEN_URL /
  KNERON_CONVERTER_CLIENT_ID/SECRET / FILE_ACCESS_AGENT_* /
  OAUTH_TOKEN_* / PROMOTE_TIMEOUT_MS）

docs/autoflow/04-architecture/api/api-result.md §16：
- 新增 Env Naming Reference Table（30 個 canonical env names）
- 拍板 source code 為 single source of truth、env.example 對齊
- 確認 /result 8 個 env + 其他 22 個的命名規格
- 留歷史記錄：Orchestrator 之前用過想像中縮寫名（_MAX / _HOURLY_QUOTA /
  RESULT_CONCURRENT_STREAM_MAX）造成命名混亂、§16 為未來 prompt 引用標準

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 01:01:59 +08:00
jim800121chen
d8a9517c9d feat(task-scheduler): Phase 0.8b — API key auth + /result endpoint 
Auth pillar 從 OAuth 2.0 resource server 改成 pre-shared API key
（visionA ↔ converter 1:1 internal trust）。新增 GET /api/v1/jobs/:id/result
streaming endpoint 給 visionA backend 中轉 NEF 下載。

Phase A（auth 切換）：
- 新增 apiKeyMiddleware（constant-time compare、tokenFingerprint、4 audit events）
- 砍 OAuth middleware + JWKS（保留 oauthClient 供 promote → FAA 使用）
- 4 個 endpoint 換掛 requireApiKey
- 加 TRUST_PROXY env + Express trust proxy 設定（forensic source_ip）

Phase B（/result endpoint）：
- streaming NEF download with 5min timeout + concurrent cap 10
- Two-tier rate limit（burst 5/10s + sustained 20/min）
- Bandwidth quota（1 GB/hr + 6 GB/24hr）by token_fingerprint
- Range header silently ignored + Accept-Ranges: none
- filename quote-escape + RFC 5987 fallback + sanitize
- 8 個 /result audit events（forensic 完整）

設計演進記錄：docs/TODO-visionA-integration-v2.md（5/2 OAuth → 5/16 API key
→ 5/16 download via converter；對應 visionA repo ADR-015/016）

Tests: 597 → 666 (+69)、29 suites all pass
Security: APPROVE WITH CONDITIONS（單 instance 部署、6 新 env、24hr 監控）
npm audit: 3 vuln → 0（transitive AWS SDK xml chain）

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 22:47:28 +08:00
jim800121chen
cff9236699 docs: migrate Autoflow shared documents to docs/autoflow/ 
Move PRD, design specs, architecture docs, and TDD from .autoflow/
(personal/per-branch layer) to docs/autoflow/ (shared layer that
goes into git) per the new Autoflow workspace layout.

Files moved:
- 02-prd/PRD.md
- 03-design/design-review.md
- 03-design/user-flow-cross-system.md
- 04-architecture/TDD.md
- 04-architecture/design-doc.md
- 04-architecture/security.md

The originals were never tracked, so git mv reduced to a filesystem
rename with no history to preserve. .autoflow/ remains for personal
notes (progress.md, review reports, testing logs).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-01 10:59:21 +08:00
jim800121chen
7404ca9bc8 docs: align Design.md with Phase 1 architecture 
Restructure section 2 from component overview to tech selection table
covering Scheduler, Worker, Queue, Job State, Artifact Store, Web UI.
Document why Redis Stream for the queue (language-neutral, consumer
groups, single Redis instance, Crash Reset alignment) and detail
worker horizontal scaling via consumer groups.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-01 10:55:14 +08:00
jim800121chen
4d381c0b50 feat(task-scheduler): Phase 1 — modularize server + add OAuth/JWKS + /api/v1/* routes 
Refactor server.js (647 → 99 lines) into 30+ modules under src/:
- auth/: JWKS validation, JWT middleware, OAuth client_credentials
- routes/v1/: jobs (POST/GET/:id) + promote with input validation
- routes/legacy.js: existing /jobs multipart path (backward compatible)
- services/: jobService, healthService, sseService, statusMapper,
  doneListener
- middleware/: requestId, errorHandler, perClientRateLimit,
  uploadConcurrency, upload (multer + storage)
- redis/: Lua scripts for atomic claim/release_active_job
- storage/: local + minio adapters; fileAccessAgent/: PUT promote client
- config.js: env var validation with fail-fast

Phase 1 features (T1–T11):
- T1 Auth middleware + JWKS (Member Center OAuth2 resource server)
- T2 OAuth client (Member Center client_credentials, Basic auth)
- T3 /api/v1/* router skeleton
- T4 server.js refactor (legacy endpoints fully preserved, real-Redis
  regression verified — existing worker consumer group untouched)
- T5 POST /api/v1/jobs (multipart, OWASP-audited, 2 Critical / 6 Major
  fixed; Risk-A/B documented as accepted)
- T6 GET /api/v1/jobs + GET /:id (cursor pagination, ETag, IDOR-safe)
- T7 POST /jobs/:id/promote (FAA PUT with own service token, 300s
  timeout, fail-fast on missing FAA URL)
- T8 /health upgrade (healthy/degraded/unhealthy + 30s background cache)
- T9 stage_timings (release_active_job in terminal states)
- T10 env + Docker integration (MULTIPART_* + concurrency limiter)
- T11 README (498 lines) + OpenAPI 3.0 spec (1588 lines)

Tests: 630 pass across 29 suites. Updated Dockerfile + .dockerignore +
docker-compose.yml env passthrough (no hardcoded secrets, fail-fast on
missing required vars).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-01 10:55:05 +08:00
jim800121chen
548f53ccbf chore: untrack .env and ignore Autoflow personal layer 
- Stop tracking .env (was committed in 31f61b5 with plaintext MinIO
  secrets; existing history cleanup is tracked as Risk-A in Phase 1
  backlog and will be addressed before launch)
- Ignore .autoflow/ entirely (per-branch personal notes layer)
- Ignore two local-only reference assets: drawio architecture export
  and toolchain manual PDF

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-01 10:54:45 +08:00
jim800121chen
75e3a9b2cc Add worker dependencies documentation with binary mapping per stage 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-03 00:09:14 +08:00
jim800121chen
efa67d59a4 Add web frontend, MinIO storage, monitoring, and docker-compose deployment 
- Frontend: rewrite Home.vue to match backend POST /jobs API (remove single-stage options)
- Frontend: add Monitor page (/monitor) for queue and job monitoring
- Frontend: add job history with localStorage tracking (per-browser)
- Frontend: fix Nginx proxy rewrite (/api -> /) and add 500MB upload limit
- Backend: add MinIO storage support (STORAGE_BACKEND=minio) alongside local mode
- Backend: add GET /queues/stats API for queue monitoring
- Backend: fix download handler for MinIO (buffer mode for Node 18 compat)
- Workers: add S3/MinIO download/upload in consumer.py with isolated temp dirs
- Workers: add s3_storage.py helper with lifecycle rule (7-day TTL)
- Docker: add docker-compose.yml with all services (web, scheduler, redis, workers)
- Docker: ports mapped to 9500 (web) and 9501 (scheduler)
- Config: add .env to .gitignore to protect secrets

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-01 15:04:09 +08:00
warrenchen
fdebf4db5d Refactor workers to use backend interfaces for quantization, compilation, and evaluation; add optional flags for simulation in request schemas and update documentation accordingly. 2026-02-06 08:24:08 +00:00
warrenchen
bc98456d74 Add initial draft of Toolchain Flow Modularization Notes 2026-02-05 17:08:17 +00:00
warrenchen
e93a1a5996 Complete tests for workers, with source file format onnx and tflite 2026-01-29 03:04:41 +00:00
warrenchen
0000f19d5e Refactor requirements.txt to update and organize dependencies 2026-01-28 06:17:12 +00:00
warrenchen
31f61b5122 Initail project 2026-01-28 06:16:04 +00:00