Phase 0.8 把 kneron_model_converter 的轉檔功能整合進 visionA Cloud。
visionA backend 當 streaming proxy(upload)+ delegated download token broker(download)+
ownership trust boundary,converter / FAA / MC 三方零修改。
新增 internal/conversion/ 套件(8 個檔,~10,000 行 prod+test,117+ test cases,race -count=3 全綠):
- conversion.go:Service interface 5 method、Job/PromoteResult/InitJobInput types
- errors.go:13+ sentinel errors + ErrorCode/HTTPStatus mapping,對齊 conversion.md §6
- mc_token_client.go:service-to-service token (client_credentials grant) + DCL cache
(exp - 15s 重取,per-scope cache),IssueDelegatedDownload(MC delegated download token)
錯誤分 idp_misconfigured (4xx) / idp_unavailable (5xx) / download_token_failed / mc_token_unavailable
- converter_client.go:對 converter scheduler 4 method(InitJob multipart streaming /
GetJob / Promote / ListInProgressJobs),InitJob 不 retry 5xx(streaming body 無法 replay)
- faa_client.go:對 FAA GET /files/{key} server-to-server pull,Phase A retry(GET 無 body
可 replay)對齊 §9.1 retry 矩陣,streaming io.ReadCloser 透傳避 OOM
- ownership.go:in-memory job_id → user_id map + per-user mutex 防 thundering herd lazy rebuild
(不同 user 平行 fetch,同 user 100 caller 收斂成 1 次),visionA 重啟靠 converter
ListInProgressJobs(user) 重建
- flow.go:Service interface 整合層(5 method 串接 converter/FAA/MC/ownership)
- InitJob 用 io.Pipe + multipart.Reader/Writer 重組 streaming proxy(黑名單 client user_id
+ 灌入 OIDC sub)
- DownloadRedirectURL 自動觸發 promote(spec §1 Stage 3b),用 ensurePromoted helper
- PromoteToModels 冪等(modelStore.FindBySourceJobID 為 source-of-truth)
- OwnershipMismatch → ErrJobNotFound 不 forbidden(§7.2 防枚舉)
- storage / modelStore 失敗包 ErrStorageUnavailable / ErrModelStoreUnavailable
(視為 visionA 自身 500 而非 502 gateway,SRE alarm 才打對 team)
新增 internal/api/conversion.go(5 endpoint handler + main.go wire):
- POST /api/conversion/init(multipart streaming proxy,不呼叫 c.MultipartForm())
- GET /api/conversion/active(lazy rebuild ownership)
- GET /api/conversion/{job_id}(poll status)
- POST /api/conversion/{job_id}/promote-to-models(FAA pull → models 三段式)
- GET /api/conversion/{job_id}/download(server-side HTTP 302 → FAA,token 不過 frontend
JS,仿 FAA TestSite DownloadFileDirect pattern;Cache-Control: no-store)
5 個 endpoint 全部走 OIDC AuthMiddleware;user_id 從 cookie session 灌(trust boundary),
從不接受 client multipart form / JSON / query 的 user_id。
TestAllAPIEndpointsRequire401WithoutCookie 自動覆蓋新 5 endpoint regression 防呆。
新增 cmd/api-server/conversion_e2e_test.go(4 個 e2e 場景):
- TestConversionE2E_StreamingProxy(10MB body + trust boundary regression)
- TestConversionE2E_LazyRebuildAfterRestart(visionA 重啟仍能 /active)
- TestConversionE2E_Download302Redirect(驗 302 + Location header + token 不在 body)
- TestConversionE2E_ActiveJobConflict(409 + active_job 詳情)
修改 internal/config/{config,load}.go:新增 ConversionConfig 5 欄位
(ConverterBaseURL / FAABaseURL / TenantID / ServiceClientID / ServiceClientSecret)+
Enabled() helper(雙非空判定)。
修改 cmd/api-server/main.go:條件 wire(cfg.Conversion.Enabled() 為 true 才建 client + Service;
否則 Deps.Conversion=nil,handler 自動回 501)。
修改 .env.example:新增 Phase 0.8 區塊註解。
新增 cmd/api-server/conversion_adapters.go:narrow interface adapter(接既有
internal/model.Repository / internal/storage.Store → conversion.ModelStore / Storage,避免 import cycle)。
驗證:go test -race -count=3 ./... 17 packages 全綠 / go vet 0 warning / go build 成功。
對齊文件:
- .autoflow/04-architecture/adr/adr-014-conversion-integration.md
- .autoflow/04-architecture/conversion.md (TDD)
- .autoflow/04-architecture/api/api-conversion.md
- .autoflow/02-prd/features/feature-converter-integration.md
- .autoflow/03-design/wireframes/wireframe-conversion.md
- .autoflow/03-design/flows/flow-conversion.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
315 lines
12 KiB
Go
315 lines
12 KiB
Go
// Command api-server 是 visionA-backend 的對前端 REST + WebSocket 伺服器。
|
||
//
|
||
// 雛形雙 binary 架構(Q1 裁決):
|
||
// - api-server **無狀態**:所有 session 狀態都在 remote-proxy 那邊
|
||
// - 透過 ProxyClientStore + Forwarder 走 internal HTTP 跟 remote-proxy 溝通
|
||
//
|
||
// 對應文件:
|
||
// - `.autoflow/04-architecture/TDD.md` §2.4(雙 binary 部署)/ §10(前端資料流)
|
||
// - `.autoflow/04-architecture/api/api-spec.md`(前端用的 REST API)
|
||
// - `.autoflow/04-architecture/api/api-internal.md`(api-server ↔ remote-proxy)
|
||
// - `.autoflow/04-architecture/tunnel.md` §5
|
||
package main
|
||
|
||
import (
|
||
"context"
|
||
"errors"
|
||
"net"
|
||
"net/http"
|
||
"os"
|
||
"os/signal"
|
||
"strconv"
|
||
"syscall"
|
||
"time"
|
||
|
||
"github.com/gin-gonic/gin"
|
||
|
||
"visiona-backend/internal/api"
|
||
"visiona-backend/internal/auth"
|
||
"visiona-backend/internal/config"
|
||
"visiona-backend/internal/conversion"
|
||
"visiona-backend/internal/converter"
|
||
"visiona-backend/internal/device"
|
||
"visiona-backend/internal/logger"
|
||
"visiona-backend/internal/model"
|
||
"visiona-backend/internal/oidc"
|
||
"visiona-backend/internal/session"
|
||
"visiona-backend/internal/storage"
|
||
"visiona-backend/internal/usersession"
|
||
)
|
||
|
||
// defaultSigningSecret 與 config/load.go 保持一致 — 用於啟動警告。
|
||
const defaultSigningSecret = "dev-signing-secret-do-not-use-in-prod"
|
||
|
||
// shutdownTimeout 是收到 SIGINT/SIGTERM 後等待進行中請求完成的最長時間。
|
||
const shutdownTimeout = 10 * time.Second
|
||
|
||
// sessionCleanupInterval 是 OIDC user session store 的後台清理頻率。
|
||
// 設 5 分鐘是 dev / prod 都合理的值:足夠頻繁讓 idle session 不久留,
|
||
// 又不會過度消耗 CPU。
|
||
const sessionCleanupInterval = 5 * time.Minute
|
||
|
||
func main() {
|
||
cfg := config.Load()
|
||
log := logger.New(cfg.Logger.Level).With("service", "api-server")
|
||
|
||
// Validate config(特別是 OIDC 啟用時的必填欄位)。
|
||
if err := cfg.Validate(); err != nil {
|
||
log.Error("invalid config", "error", err)
|
||
os.Exit(1)
|
||
}
|
||
|
||
// 啟動警告:signing secret 為預設值(同 remote-proxy 行為)。
|
||
// 此 secret 同時給 storage presigned URL 與(未來)pairing token hash 用。
|
||
if cfg.Auth.SigningSecret == defaultSigningSecret {
|
||
log.Warn("signing secret 仍為預設 dev 值(storage/pairing 共用)",
|
||
"action", "請在生產環境設定環境變數 VISIONA_STORAGE_SIGNING_SECRET",
|
||
"affects", "storage presigned URL, pairing token hash (phase 1)")
|
||
}
|
||
|
||
// ===== Storage =====
|
||
// 用 LocalFS(Phase 0 雛形);signing secret 共用同一份。
|
||
storageStore, err := storage.NewLocalFSStore(cfg.Storage.RootDir, cfg.Storage.BaseURL, cfg.Auth.SigningSecret)
|
||
if err != nil {
|
||
log.Error("failed to init storage", "error", err)
|
||
os.Exit(1)
|
||
}
|
||
log.Info("storage initialized",
|
||
"backend", cfg.Storage.Backend,
|
||
"root", cfg.Storage.RootDir,
|
||
"base_url", cfg.Storage.BaseURL)
|
||
|
||
// ===== Pairing / Session Token(OIDC 之外的雛形 token store) =====
|
||
pairingStore := auth.NewInMemoryPairingStore()
|
||
sessionTokenStore := auth.NewInMemorySessionTokenStore()
|
||
|
||
// ===== OIDC + User Session(OB5:唯一認證路徑) =====
|
||
// cfg.Validate() 已確保所有必填欄位存在,這裡可以放心 wire。
|
||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||
oidcProvider, err := oidc.NewProvider(ctx, oidc.ProviderConfig{
|
||
IssuerURL: cfg.OIDC.IssuerURL,
|
||
ClientID: cfg.OIDC.ClientID,
|
||
ClientSecret: cfg.OIDC.ClientSecret,
|
||
RedirectURL: cfg.OIDC.RedirectURL,
|
||
})
|
||
cancel()
|
||
if err != nil {
|
||
log.Error("failed to init OIDC provider",
|
||
"error", err,
|
||
"issuer", cfg.OIDC.IssuerURL,
|
||
"hint", "確認 IdP discovery (.well-known/openid-configuration) 可達")
|
||
os.Exit(1)
|
||
}
|
||
|
||
userSessionStore := usersession.NewInMemoryStore()
|
||
userSessionMgr := usersession.NewManager(userSessionStore, usersession.CookieConfig{
|
||
Name: cfg.UserSession.CookieName,
|
||
Domain: cfg.UserSession.CookieDomain,
|
||
Path: "/",
|
||
Secure: cfg.UserSession.CookieSecure,
|
||
HTTPOnly: true,
|
||
SameSite: http.SameSiteLaxMode,
|
||
MaxAge: int(cfg.UserSession.AbsoluteTTL.Seconds()),
|
||
SigningKey: []byte(cfg.UserSession.Secret),
|
||
})
|
||
log.Info("OIDC initialized",
|
||
"issuer", cfg.OIDC.IssuerURL,
|
||
"client_id", cfg.OIDC.ClientID,
|
||
"redirect_url", cfg.OIDC.RedirectURL,
|
||
"frontend_url", cfg.OIDC.PostLoginURL,
|
||
"cookie_secure", cfg.UserSession.CookieSecure,
|
||
"absolute_ttl", cfg.UserSession.AbsoluteTTL,
|
||
"idle_ttl", cfg.UserSession.IdleTTL,
|
||
)
|
||
|
||
// ===== Session(api-server 端透過 ProxyClient 走 internal HTTP) =====
|
||
proxyClient := session.NewHTTPProxyClient(cfg.Session.ProxyInternalURL, log)
|
||
forwarder := session.NewForwarder(cfg.Session.ProxyInternalURL, log)
|
||
sessionStore := session.NewProxyClientStore(proxyClient, forwarder)
|
||
log.Info("session store initialized",
|
||
"backend", "proxy-client",
|
||
"proxy_internal_url", cfg.Session.ProxyInternalURL)
|
||
|
||
// ===== Repositories(in-memory,雛形) =====
|
||
deviceRepo := device.NewInMemoryRepository()
|
||
modelRepo := model.NewInMemoryRepository()
|
||
|
||
// ===== Converter(stub,Phase 2 才實作) =====
|
||
converterClient := converter.NewStubClient()
|
||
|
||
// ===== Phase 0.8 Conversion(轉檔功能整合) =====
|
||
// 對齊 .autoflow/04-architecture/conversion.md。
|
||
//
|
||
// 啟用條件:cfg.Conversion.Enabled() — ConverterBaseURL + FAABaseURL 都非空。
|
||
// 啟用時必須有 ServiceClientID/Secret(client_credentials grant 必要)。
|
||
// 不啟用時 deps.Conversion 為 nil,5 個 endpoint 自動回 501(registerConversionRoutes 處理)。
|
||
var conversionService conversion.Service
|
||
if cfg.Conversion.Enabled() {
|
||
// service token 機制依賴 ServiceClientID/Secret — 沒設就 fatal,避免半設定狀態
|
||
if cfg.OIDC.ServiceClientID == "" || cfg.OIDC.ServiceClientSecret == "" {
|
||
log.Error("conversion enabled but service client credentials missing",
|
||
"hint", "set VISIONA_OIDC_SERVICE_CLIENT_ID + VISIONA_OIDC_SERVICE_CLIENT_SECRET, or unset CONVERTER/FAA base URL to disable")
|
||
os.Exit(1)
|
||
}
|
||
|
||
mcTokenClient := conversion.NewMCTokenClient(conversion.MCTokenClientOpts{
|
||
Issuer: cfg.OIDC.IssuerURL,
|
||
ClientID: cfg.OIDC.ServiceClientID,
|
||
ClientSecret: cfg.OIDC.ServiceClientSecret,
|
||
Logger: log,
|
||
})
|
||
converterAPIClient := conversion.NewConverterClient(conversion.ConverterClientOpts{
|
||
BaseURL: cfg.Conversion.ConverterBaseURL,
|
||
Tokens: mcTokenClient,
|
||
Logger: log,
|
||
})
|
||
faaAPIClient := conversion.NewFAAClient(conversion.FAAClientOpts{
|
||
BaseURL: cfg.Conversion.FAABaseURL,
|
||
Tokens: mcTokenClient,
|
||
Logger: log,
|
||
})
|
||
ownership := conversion.NewOwnership(converterAPIClient, log)
|
||
|
||
// narrow adapter(避免 conversion 直接 import internal/model / internal/storage)
|
||
modelStoreAdapter := newConversionModelStoreAdapter(modelRepo)
|
||
storageAdapter := newConversionStorageAdapter(storageStore)
|
||
|
||
var convErr error
|
||
conversionService, convErr = conversion.NewService(conversion.FlowOpts{
|
||
Converter: converterAPIClient,
|
||
FAA: faaAPIClient,
|
||
MCToken: mcTokenClient,
|
||
Ownership: ownership,
|
||
ModelStore: modelStoreAdapter,
|
||
Storage: storageAdapter,
|
||
TenantID: cfg.Conversion.TenantID,
|
||
FAABaseURL: cfg.Conversion.FAABaseURL,
|
||
DelegatedTTLSeconds: cfg.Conversion.DelegatedTTLSeconds,
|
||
Logger: log,
|
||
})
|
||
if convErr != nil {
|
||
log.Error("failed to init conversion service", "error", convErr)
|
||
os.Exit(1)
|
||
}
|
||
log.Info("conversion service initialized",
|
||
"converter_base_url", cfg.Conversion.ConverterBaseURL,
|
||
"faa_base_url", cfg.Conversion.FAABaseURL,
|
||
"tenant_id", cfg.Conversion.TenantID,
|
||
"delegated_ttl_sec", cfg.Conversion.DelegatedTTLSeconds)
|
||
} else {
|
||
log.Info("conversion service disabled (set VISIONA_CONVERTER_BASE_URL + VISIONA_FAA_BASE_URL to enable)")
|
||
}
|
||
|
||
// ===== Seed demo data(可選) =====
|
||
if cfg.Server.SeedDemoData {
|
||
if err := seedDemoData(deviceRepo, modelRepo, pairingStore, cfg.Auth.StaticUserID, log); err != nil {
|
||
log.Warn("seed demo data failed", "error", err)
|
||
}
|
||
}
|
||
|
||
// ===== API Router =====
|
||
gin.SetMode(gin.ReleaseMode)
|
||
// Phase 0.7 security fix C1:StaticUserID 不再注入 Deps(見 .autoflow/05-implementation/review/phase-0.7-security-audit.md)
|
||
// dev seed 仍直接讀 cfg.Auth.StaticUserID;stage/prod 不影響(VISIONA_SEED_DEMO_DATA=false)。
|
||
router := api.NewRouter(api.Deps{
|
||
Logger: log,
|
||
PairingStore: pairingStore,
|
||
SessionTokenStore: sessionTokenStore,
|
||
SessionStore: sessionStore,
|
||
Forwarder: forwarder,
|
||
DeviceRepo: deviceRepo,
|
||
ModelRepo: modelRepo,
|
||
Storage: storageStore,
|
||
Converter: converterClient,
|
||
Conversion: conversionService, // Phase 0.8(nil 時 /api/conversion/* 回 501)
|
||
MaxUploadSizeMB: cfg.Model.MaxSizeMB,
|
||
CORSAllowedOrigins: cfg.CORS.AllowedOrigins,
|
||
RelayPublicURL: cfg.Server.RelayPublicURL,
|
||
|
||
// OIDC(OB5:唯一認證路徑)
|
||
OIDCProvider: oidcProvider,
|
||
SessionManager: userSessionMgr,
|
||
OIDCPostLoginURL: cfg.OIDC.PostLoginURL,
|
||
})
|
||
|
||
addr := net.JoinHostPort(cfg.Server.Host, strconv.Itoa(cfg.Server.Port))
|
||
srv := &http.Server{
|
||
Addr: addr,
|
||
Handler: router,
|
||
ReadHeaderTimeout: 10 * time.Second, // 防 slow-loris(對齊 security.md)
|
||
}
|
||
|
||
// ===== User session cleanup goroutine =====
|
||
cleanupCtx, cleanupCancel := context.WithCancel(context.Background())
|
||
defer cleanupCancel()
|
||
go runUserSessionCleanup(cleanupCtx, userSessionStore, cfg.UserSession.IdleTTL, cfg.UserSession.AbsoluteTTL, log)
|
||
|
||
// ===== 啟動 server =====
|
||
errCh := make(chan error, 1)
|
||
go func() {
|
||
log.Info("api-server listening",
|
||
"addr", addr,
|
||
"proxy_internal_url", cfg.Session.ProxyInternalURL,
|
||
"seed_demo_data", cfg.Server.SeedDemoData,
|
||
"oidc_issuer", cfg.OIDC.IssuerURL,
|
||
)
|
||
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||
errCh <- err
|
||
}
|
||
}()
|
||
|
||
// 等 signal 或錯誤
|
||
quit := make(chan os.Signal, 1)
|
||
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
|
||
select {
|
||
case <-quit:
|
||
log.Info("shutdown signal received")
|
||
case err := <-errCh:
|
||
log.Error("api-server error, shutting down", "error", err)
|
||
}
|
||
|
||
// Graceful shutdown
|
||
shutdownCtx, cancel := context.WithTimeout(context.Background(), shutdownTimeout)
|
||
defer cancel()
|
||
if err := srv.Shutdown(shutdownCtx); err != nil {
|
||
log.Warn("api-server shutdown error", "error", err)
|
||
}
|
||
cleanupCancel() // 停掉 user session cleanup goroutine
|
||
log.Info("api-server stopped")
|
||
}
|
||
|
||
// runUserSessionCleanup 是 OIDC user session store 的 background cleanup 迴圈。
|
||
//
|
||
// 每 sessionCleanupInterval 跑一次 store.CleanupExpired,把 idle / absolute timeout
|
||
// 的 session 清掉。失敗只 log 不 panic(cleanup 不應拖垮主 process)。
|
||
//
|
||
// ctx 取消(process shutdown)即退出。
|
||
func runUserSessionCleanup(ctx context.Context, store usersession.Store, idleTTL, absTTL time.Duration, log loggerLike) {
|
||
ticker := time.NewTicker(sessionCleanupInterval)
|
||
defer ticker.Stop()
|
||
for {
|
||
select {
|
||
case <-ctx.Done():
|
||
return
|
||
case <-ticker.C:
|
||
cctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||
removed, err := store.CleanupExpired(cctx, idleTTL, absTTL)
|
||
cancel()
|
||
if err != nil {
|
||
log.Warn("user session cleanup failed", "error", err)
|
||
continue
|
||
}
|
||
if removed > 0 {
|
||
log.Info("user session cleanup", "removed", removed)
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
// loggerLike 是 runUserSessionCleanup 需要的最小 logger 介面,避免直接綁 *slog.Logger
|
||
// 而能在 test 中 stub。
|
||
type loggerLike interface {
|
||
Info(msg string, args ...any)
|
||
Warn(msg string, args ...any)
|
||
}
|