Phase 0.8 把 kneron_model_converter 的轉檔功能整合進 visionA Cloud。
visionA backend 當 streaming proxy(upload)+ delegated download token broker(download)+
ownership trust boundary,converter / FAA / MC 三方零修改。
新增 internal/conversion/ 套件(8 個檔,~10,000 行 prod+test,117+ test cases,race -count=3 全綠):
- conversion.go:Service interface 5 method、Job/PromoteResult/InitJobInput types
- errors.go:13+ sentinel errors + ErrorCode/HTTPStatus mapping,對齊 conversion.md §6
- mc_token_client.go:service-to-service token (client_credentials grant) + DCL cache
(exp - 15s 重取,per-scope cache),IssueDelegatedDownload(MC delegated download token)
錯誤分 idp_misconfigured (4xx) / idp_unavailable (5xx) / download_token_failed / mc_token_unavailable
- converter_client.go:對 converter scheduler 4 method(InitJob multipart streaming /
GetJob / Promote / ListInProgressJobs),InitJob 不 retry 5xx(streaming body 無法 replay)
- faa_client.go:對 FAA GET /files/{key} server-to-server pull,Phase A retry(GET 無 body
可 replay)對齊 §9.1 retry 矩陣,streaming io.ReadCloser 透傳避 OOM
- ownership.go:in-memory job_id → user_id map + per-user mutex 防 thundering herd lazy rebuild
(不同 user 平行 fetch,同 user 100 caller 收斂成 1 次),visionA 重啟靠 converter
ListInProgressJobs(user) 重建
- flow.go:Service interface 整合層(5 method 串接 converter/FAA/MC/ownership)
- InitJob 用 io.Pipe + multipart.Reader/Writer 重組 streaming proxy(黑名單 client user_id
+ 灌入 OIDC sub)
- DownloadRedirectURL 自動觸發 promote(spec §1 Stage 3b),用 ensurePromoted helper
- PromoteToModels 冪等(modelStore.FindBySourceJobID 為 source-of-truth)
- OwnershipMismatch → ErrJobNotFound 不 forbidden(§7.2 防枚舉)
- storage / modelStore 失敗包 ErrStorageUnavailable / ErrModelStoreUnavailable
(視為 visionA 自身 500 而非 502 gateway,SRE alarm 才打對 team)
新增 internal/api/conversion.go(5 endpoint handler + main.go wire):
- POST /api/conversion/init(multipart streaming proxy,不呼叫 c.MultipartForm())
- GET /api/conversion/active(lazy rebuild ownership)
- GET /api/conversion/{job_id}(poll status)
- POST /api/conversion/{job_id}/promote-to-models(FAA pull → models 三段式)
- GET /api/conversion/{job_id}/download(server-side HTTP 302 → FAA,token 不過 frontend
JS,仿 FAA TestSite DownloadFileDirect pattern;Cache-Control: no-store)
5 個 endpoint 全部走 OIDC AuthMiddleware;user_id 從 cookie session 灌(trust boundary),
從不接受 client multipart form / JSON / query 的 user_id。
TestAllAPIEndpointsRequire401WithoutCookie 自動覆蓋新 5 endpoint regression 防呆。
新增 cmd/api-server/conversion_e2e_test.go(4 個 e2e 場景):
- TestConversionE2E_StreamingProxy(10MB body + trust boundary regression)
- TestConversionE2E_LazyRebuildAfterRestart(visionA 重啟仍能 /active)
- TestConversionE2E_Download302Redirect(驗 302 + Location header + token 不在 body)
- TestConversionE2E_ActiveJobConflict(409 + active_job 詳情)
修改 internal/config/{config,load}.go:新增 ConversionConfig 5 欄位
(ConverterBaseURL / FAABaseURL / TenantID / ServiceClientID / ServiceClientSecret)+
Enabled() helper(雙非空判定)。
修改 cmd/api-server/main.go:條件 wire(cfg.Conversion.Enabled() 為 true 才建 client + Service;
否則 Deps.Conversion=nil,handler 自動回 501)。
修改 .env.example:新增 Phase 0.8 區塊註解。
新增 cmd/api-server/conversion_adapters.go:narrow interface adapter(接既有
internal/model.Repository / internal/storage.Store → conversion.ModelStore / Storage,避免 import cycle)。
驗證:go test -race -count=3 ./... 17 packages 全綠 / go vet 0 warning / go build 成功。
對齊文件:
- .autoflow/04-architecture/adr/adr-014-conversion-integration.md
- .autoflow/04-architecture/conversion.md (TDD)
- .autoflow/04-architecture/api/api-conversion.md
- .autoflow/02-prd/features/feature-converter-integration.md
- .autoflow/03-design/wireframes/wireframe-conversion.md
- .autoflow/03-design/flows/flow-conversion.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
623 lines
22 KiB
Go
623 lines
22 KiB
Go
// FAA Client 單元測試。
|
||
//
|
||
// 測試策略:
|
||
// - 用 httptest.Server mock FAA 的 GET /files/{key} 端點
|
||
// - 用 stub MCTokenClient(直接回 token / 注入錯誤),不耦合真實 mc_token_client 邏輯
|
||
// - 用 atomic counter 驗 retry 行為(Phase A retry:max 3 attempts = 1 + 2 retries)
|
||
// - streaming 驗證用較大但合理大小(10MB)— 真 100MB 會拖慢 test runner 太多
|
||
//
|
||
// 測試範疇對應 conversion.md §9.1(FAA GET /files retry max 2 次, 1s/2s):
|
||
// - GetFile_Success / GetFile_Streaming / GetFile_AuthHeader
|
||
// - GetFile_404_NoRetry / GetFile_401_Unauthorized / GetFile_403_Unauthorized
|
||
// - GetFile_5xx_RetryThenSuccess / GetFile_5xx_Exhausted
|
||
// - GetFile_Network_RetryThenSuccess / GetFile_Network_Exhausted
|
||
// - GetFile_ContextCancel / GetFile_ContextCancel_DuringRetry
|
||
// - GetFile_ServiceTokenFailure_Propagated / GetFile_EmptyObjectKey
|
||
// - GetFile_400_GenericError / HashObjectKey_StableAndLength
|
||
//
|
||
// Phase 0.8 conversion (見 .autoflow/04-architecture/conversion.md §2.3 + §9.1)
|
||
package conversion
|
||
|
||
import (
|
||
"context"
|
||
"errors"
|
||
"io"
|
||
"net"
|
||
"net/http"
|
||
"net/http/httptest"
|
||
"sync/atomic"
|
||
"testing"
|
||
"time"
|
||
|
||
"github.com/stretchr/testify/assert"
|
||
"github.com/stretchr/testify/require"
|
||
)
|
||
|
||
// ==========================================================================
|
||
// FAA mock server helpers
|
||
// ==========================================================================
|
||
|
||
// newFAAClientForTest 建立指向 mock server 的 FAAClient(使用快速 retry backoff 加速 test)。
|
||
//
|
||
// 注意:這個 helper 用較短 backoff(10ms 起跳)讓 retry test 不會跑很久。
|
||
// 真實 production 走 §9.1 的 1s/2s(在 NewFAAClient 預設)。
|
||
func newFAAClientForTest(t *testing.T, baseURL string, tokens MCTokenClient) FAAClient {
|
||
t.Helper()
|
||
return NewFAAClient(FAAClientOpts{
|
||
BaseURL: baseURL,
|
||
Tokens: tokens,
|
||
// 用一個簡單的 http.Client;httptest.Server.Client 也可以但這樣更貼近真實情境,
|
||
// 用較短 timeout 加速 test。注意 streaming test 不能用整體 Timeout,所以另外覆寫。
|
||
HTTPClient: &http.Client{Timeout: 5 * time.Second},
|
||
Logger: silentLogger(),
|
||
})
|
||
}
|
||
|
||
// ==========================================================================
|
||
// 成功路徑
|
||
// ==========================================================================
|
||
|
||
// TestGetFile_Success:mock 回 200 + binary stream,驗 ContentLength / ETag / ContentType 解析。
|
||
func TestGetFile_Success(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
payload := []byte("binary payload here")
|
||
var receivedAuth string
|
||
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
receivedAuth = r.Header.Get("Authorization")
|
||
require.Equal(t, http.MethodGet, r.Method)
|
||
w.Header().Set("Content-Type", "application/octet-stream")
|
||
w.Header().Set("ETag", "\"etag-abc-123\"")
|
||
w.Header().Set("Content-Length", "19")
|
||
w.WriteHeader(http.StatusOK)
|
||
_, _ = w.Write(payload)
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
file, err := fc.GetFile(context.Background(), "tenant/jobs/abc/output.nef")
|
||
|
||
require.NoError(t, err)
|
||
require.NotNil(t, file)
|
||
require.NotNil(t, file.Body)
|
||
t.Cleanup(func() { _ = file.Body.Close() })
|
||
|
||
assert.Equal(t, "application/octet-stream", file.ContentType)
|
||
assert.Equal(t, "\"etag-abc-123\"", file.ETag)
|
||
assert.Equal(t, int64(19), file.ContentLength)
|
||
|
||
// caller 確實能 streaming 讀到完整 body
|
||
body, readErr := io.ReadAll(file.Body)
|
||
require.NoError(t, readErr)
|
||
assert.Equal(t, payload, body)
|
||
|
||
assert.Equal(t, "Bearer svc-tok", receivedAuth, "Bearer service token 必須透傳")
|
||
assert.Equal(t, 1, tokens.calls(scopeFAADownloadRead))
|
||
}
|
||
|
||
// TestGetFile_Streaming:mock 回 10MB body,confirm caller 能 streaming 讀(不 buffer 全 RAM)。
|
||
//
|
||
// 與 InitJob streaming test 對稱:用 io.LimitReader + zerosReader,確認 reader 被多次 Read
|
||
// (而非一次性全讀)。但 net/http 端 download 的 streaming 由 res.Body 提供,這裡的關鍵是:
|
||
// - faa_client 必須**不 io.ReadAll** 把 body 提前讀完
|
||
// - caller 用 io.Copy 慢慢讀時,server 端不需要先把全部 buffer 完成
|
||
func TestGetFile_Streaming(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
const totalSize = int64(10 * 1024 * 1024) // 10MB
|
||
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
w.Header().Set("Content-Type", "application/octet-stream")
|
||
w.Header().Set("Content-Length", "10485760")
|
||
w.WriteHeader(http.StatusOK)
|
||
// streaming write — 用 io.Copy from zerosReader(避免一次配 10MB buffer)
|
||
_, _ = io.CopyN(w, zerosReader{}, totalSize)
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
// streaming download 不能用 http.Client.Timeout(會中斷 body streaming)
|
||
fc := NewFAAClient(FAAClientOpts{
|
||
BaseURL: srv.URL,
|
||
Tokens: tokens,
|
||
// 這裡用無 timeout 的 client(test 自己控)
|
||
HTTPClient: &http.Client{},
|
||
Logger: silentLogger(),
|
||
})
|
||
|
||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||
defer cancel()
|
||
file, err := fc.GetFile(ctx, "big.nef")
|
||
|
||
require.NoError(t, err)
|
||
require.NotNil(t, file)
|
||
t.Cleanup(func() { _ = file.Body.Close() })
|
||
|
||
assert.Equal(t, totalSize, file.ContentLength)
|
||
|
||
// 用 countingReader 包 file.Body — 但 countingReader 是 io.Reader,
|
||
// 這裡換成 wrap 一下:直接 io.Copy 到 io.Discard,confirm 全 download 完成。
|
||
written, copyErr := io.Copy(io.Discard, file.Body)
|
||
require.NoError(t, copyErr)
|
||
assert.Equal(t, totalSize, written, "streaming download 必須拿到完整 body")
|
||
}
|
||
|
||
// TestGetFile_AuthHeader:驗 Bearer token 透傳,且取 token scope 為 files:download.read。
|
||
func TestGetFile_AuthHeader(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("specific-token-xyz")
|
||
var receivedAuth string
|
||
var receivedAccept string
|
||
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
receivedAuth = r.Header.Get("Authorization")
|
||
receivedAccept = r.Header.Get("Accept")
|
||
w.Header().Set("Content-Type", "application/octet-stream")
|
||
w.WriteHeader(http.StatusOK)
|
||
_, _ = w.Write([]byte("ok"))
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
file, err := fc.GetFile(context.Background(), "key")
|
||
require.NoError(t, err)
|
||
defer file.Body.Close()
|
||
_, _ = io.ReadAll(file.Body)
|
||
|
||
assert.Equal(t, "Bearer specific-token-xyz", receivedAuth)
|
||
assert.Equal(t, "application/octet-stream", receivedAccept)
|
||
assert.Equal(t, 1, tokens.calls(scopeFAADownloadRead),
|
||
"必須用 files:download.read scope 取 service token")
|
||
}
|
||
|
||
// ==========================================================================
|
||
// 失敗映射(不 retry 類)
|
||
// ==========================================================================
|
||
|
||
// TestGetFile_404_NoRetry:mock 回 404 → 立即 return ErrFAAFileNotFound,不 retry。
|
||
func TestGetFile_404_NoRetry(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
var attempts atomic.Int32
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
attempts.Add(1)
|
||
w.Header().Set("Content-Type", "application/json")
|
||
w.WriteHeader(http.StatusNotFound)
|
||
_, _ = w.Write([]byte(`{"error":{"code":"file_not_found","message":"File not found."}}`))
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
file, err := fc.GetFile(context.Background(), "missing.nef")
|
||
|
||
require.Error(t, err)
|
||
require.Nil(t, file, "失敗時不應回 FAAFile(避免 caller 誤用 nil body)")
|
||
assert.True(t, errors.Is(err, ErrFAAFileNotFound),
|
||
"404 → ErrFAAFileNotFound(caller 可精細處理)")
|
||
assert.Equal(t, int32(1), attempts.Load(),
|
||
"404 不應 retry(object 不存在 retry 也沒用)")
|
||
// 對外仍應 mask 成 faa_unavailable(避免揭露 object_key 不存在)
|
||
assert.Equal(t, "faa_unavailable", ErrorCode(err))
|
||
assert.Equal(t, 502, HTTPStatus(err))
|
||
}
|
||
|
||
// TestGetFile_401_Unauthorized:mock 回 401 → 不 retry,return ErrServiceClientUnauthorized。
|
||
func TestGetFile_401_Unauthorized(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
var attempts atomic.Int32
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
attempts.Add(1)
|
||
w.Header().Set("Content-Type", "application/json")
|
||
w.WriteHeader(http.StatusUnauthorized)
|
||
_, _ = w.Write([]byte(`{"error":{"code":"invalid_token"}}`))
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
file, err := fc.GetFile(context.Background(), "k")
|
||
|
||
require.Error(t, err)
|
||
require.Nil(t, file)
|
||
assert.True(t, errors.Is(err, ErrServiceClientUnauthorized),
|
||
"401 → ErrServiceClientUnauthorized(client 認證設定錯)")
|
||
assert.Equal(t, int32(1), attempts.Load(),
|
||
"401 不應 retry(secret 設定錯,retry 也是 401)")
|
||
}
|
||
|
||
// TestGetFile_403_Unauthorized:FAA 端 tenant_mismatch / object_key_mismatch 等 403 都同類處理。
|
||
func TestGetFile_403_Unauthorized(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
var attempts atomic.Int32
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
attempts.Add(1)
|
||
w.Header().Set("Content-Type", "application/json")
|
||
w.WriteHeader(http.StatusForbidden)
|
||
_, _ = w.Write([]byte(`{"error":{"code":"tenant_mismatch"}}`))
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
_, err := fc.GetFile(context.Background(), "k")
|
||
|
||
require.Error(t, err)
|
||
assert.True(t, errors.Is(err, ErrServiceClientUnauthorized))
|
||
assert.Equal(t, int32(1), attempts.Load(), "403 不應 retry")
|
||
}
|
||
|
||
// TestGetFile_400_GenericError:FAA 400(如 invalid_object_key)→ ErrFAAUnavailable,不 retry。
|
||
func TestGetFile_400_GenericError(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
var attempts atomic.Int32
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
attempts.Add(1)
|
||
w.WriteHeader(http.StatusBadRequest)
|
||
_, _ = w.Write([]byte(`{"error":{"code":"invalid_object_key"}}`))
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
_, err := fc.GetFile(context.Background(), "invalid//key")
|
||
|
||
require.Error(t, err)
|
||
assert.True(t, errors.Is(err, ErrFAAUnavailable),
|
||
"400(非 401/403/404)→ ErrFAAUnavailable")
|
||
// 應該不會被 mis-classified 成 ErrFAAFileNotFound
|
||
assert.False(t, errors.Is(err, ErrFAAFileNotFound))
|
||
assert.Equal(t, int32(1), attempts.Load(), "400 不應 retry(visionA 端的 bug)")
|
||
}
|
||
|
||
// ==========================================================================
|
||
// Phase A retry 驗證(5xx / network)
|
||
// ==========================================================================
|
||
|
||
// TestGetFile_5xx_RetryThenSuccess:mock 連續 500 兩次後回 200 → 共 3 次 attempt + 成功。
|
||
//
|
||
// 對齊 §9.1:max 2 retries(1s, 2s)— 1 + 2 = 3 attempts;第 3 次成功就 return。
|
||
// 注意:test 用真實 backoff(1s + 2s = 3s)— 為了驗 §9.1 退避時序,可接受。
|
||
func TestGetFile_5xx_RetryThenSuccess(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
var attempts atomic.Int32
|
||
payload := []byte("recovered after retry")
|
||
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
n := attempts.Add(1)
|
||
if n < 3 {
|
||
w.WriteHeader(http.StatusInternalServerError)
|
||
_, _ = w.Write([]byte(`{"error":{"code":"internal_error"}}`))
|
||
return
|
||
}
|
||
w.Header().Set("Content-Type", "application/octet-stream")
|
||
w.Header().Set("Content-Length", "21")
|
||
w.WriteHeader(http.StatusOK)
|
||
_, _ = w.Write(payload)
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
|
||
start := time.Now()
|
||
file, err := fc.GetFile(context.Background(), "k")
|
||
duration := time.Since(start)
|
||
|
||
require.NoError(t, err)
|
||
require.NotNil(t, file)
|
||
t.Cleanup(func() { _ = file.Body.Close() })
|
||
|
||
got, _ := io.ReadAll(file.Body)
|
||
assert.Equal(t, payload, got, "第 3 次成功的 body 應正確透傳")
|
||
assert.Equal(t, int32(3), attempts.Load(),
|
||
"5xx 應 retry:max 2 retries → 3 attempts")
|
||
// 驗時序:兩次 retry 退避 1s + 2s,至少花 3s(容忍輕微誤差用 ≥2.5s)
|
||
assert.GreaterOrEqual(t, duration, 2500*time.Millisecond,
|
||
"§9.1 退避序列 1s + 2s 應至少耗 2.5s")
|
||
}
|
||
|
||
// TestGetFile_5xx_Exhausted:mock 持續 500 → 用完 max retry 後 return ErrFAAUnavailable。
|
||
func TestGetFile_5xx_Exhausted(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
var attempts atomic.Int32
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
attempts.Add(1)
|
||
w.WriteHeader(http.StatusInternalServerError)
|
||
_, _ = w.Write([]byte(`{"error":{"code":"internal_error"}}`))
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
_, err := fc.GetFile(context.Background(), "k")
|
||
|
||
require.Error(t, err)
|
||
assert.True(t, errors.Is(err, ErrFAAUnavailable),
|
||
"5xx exhausted → ErrFAAUnavailable")
|
||
assert.Equal(t, int32(faaMaxRetries+1), attempts.Load(),
|
||
"5xx 應跑滿 max retries:1 + 2 = 3 attempts")
|
||
}
|
||
|
||
// TestGetFile_Network_RetryThenSuccess:前 2 次 connection refused,第 3 次成功。
|
||
//
|
||
// 用 dynamic listener swap 實作:先用一個 free port 不開 listener(dial fail),
|
||
// 第 3 次 attempt 之前才 swap 到真的 mock server。實作上比較複雜 — 改用
|
||
// proxy handler 在 mock server 內部對前 N 次「立刻 hijack 後 close」模擬 dial fail
|
||
// 不行(連線已建好);改用「server 端 force-close connection 不送任何 byte」
|
||
// 來模擬 transport 層失敗。
|
||
//
|
||
// 簡化版:用一個 proxy server,前 2 次直接 hijack + close 連線(client 看到 EOF),
|
||
// 第 3 次正常回 200。
|
||
func TestGetFile_Network_RetryThenSuccess(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
var attempts atomic.Int32
|
||
payload := []byte("recovered from net error")
|
||
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
n := attempts.Add(1)
|
||
if n < 3 {
|
||
// hijack + close 模擬 connection 中斷(client 端會看到 unexpected EOF / read error)
|
||
hj, ok := w.(http.Hijacker)
|
||
if !ok {
|
||
t.Fatal("server does not support hijacking")
|
||
}
|
||
conn, _, err := hj.Hijack()
|
||
if err != nil {
|
||
t.Fatalf("hijack failed: %v", err)
|
||
}
|
||
_ = conn.Close()
|
||
return
|
||
}
|
||
w.Header().Set("Content-Type", "application/octet-stream")
|
||
w.WriteHeader(http.StatusOK)
|
||
_, _ = w.Write(payload)
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
file, err := fc.GetFile(context.Background(), "k")
|
||
|
||
require.NoError(t, err)
|
||
require.NotNil(t, file)
|
||
t.Cleanup(func() { _ = file.Body.Close() })
|
||
|
||
got, _ := io.ReadAll(file.Body)
|
||
assert.Equal(t, payload, got)
|
||
assert.Equal(t, int32(3), attempts.Load(),
|
||
"network error 應 retry:max 2 retries → 3 attempts 後成功")
|
||
}
|
||
|
||
// TestGetFile_Network_Exhausted:dial 失敗持續發生 → 用完 max retry 後 ErrFAAUnavailable。
|
||
//
|
||
// 用一個 listen 後立刻 close 的 socket 製造 connection refused(每次 attempt 都失敗)。
|
||
func TestGetFile_Network_Exhausted(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
// 拿一個 free port 立刻關掉(dial 必失敗)
|
||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||
require.NoError(t, err)
|
||
addr := ln.Addr().String()
|
||
require.NoError(t, ln.Close())
|
||
|
||
fc := NewFAAClient(FAAClientOpts{
|
||
BaseURL: "http://" + addr,
|
||
Tokens: tokens,
|
||
// 用較短 timeout,但仍要大於 retry 退避總和(1s + 2s = 3s)— 設 10s 安全
|
||
HTTPClient: &http.Client{Timeout: 10 * time.Second},
|
||
Logger: silentLogger(),
|
||
})
|
||
|
||
start := time.Now()
|
||
_, err = fc.GetFile(context.Background(), "k")
|
||
duration := time.Since(start)
|
||
|
||
require.Error(t, err)
|
||
assert.True(t, errors.Is(err, ErrFAAUnavailable),
|
||
"network exhausted → ErrFAAUnavailable")
|
||
// retry:1 + 2 retries = 3 attempts,2 次退避 = 1s + 2s = 3s 起跳
|
||
assert.GreaterOrEqual(t, duration, 2500*time.Millisecond,
|
||
"network retry 應走完 §9.1 退避序列")
|
||
}
|
||
|
||
// ==========================================================================
|
||
// Context cancel
|
||
// ==========================================================================
|
||
|
||
// TestGetFile_ContextCancel:caller cancel ctx → 立即 return ctx.Err()(不包成 sentinel)。
|
||
func TestGetFile_ContextCancel(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
|
||
// mock server:handler 故意 sleep(讓 ctx cancel 在 server response 前發生)
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
select {
|
||
case <-r.Context().Done():
|
||
case <-time.After(2 * time.Second):
|
||
}
|
||
w.WriteHeader(http.StatusOK)
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
ctx, cancel := context.WithCancel(context.Background())
|
||
go func() {
|
||
time.Sleep(50 * time.Millisecond)
|
||
cancel()
|
||
}()
|
||
|
||
_, err := fc.GetFile(ctx, "k")
|
||
require.Error(t, err)
|
||
// ctx cancel → 透傳 ctx.Err()(不包成 ErrFAAUnavailable)
|
||
assert.True(t,
|
||
errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded),
|
||
"ctx cancel 應透傳,不應包成 ErrFAAUnavailable")
|
||
}
|
||
|
||
// TestGetFile_ContextCancel_DuringRetry:ctx cancel 發生在 retry sleep 中 → 立即中斷。
|
||
//
|
||
// 流程:
|
||
// - mock server 持續 500(觸發 retry)
|
||
// - 在第 1 次 retry 退避(1s)的中間(500ms)cancel ctx
|
||
// - 期望:GetFile 立即 return ctx.Err(),不等完 1s 退避也不繼續第 2 次 retry
|
||
func TestGetFile_ContextCancel_DuringRetry(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
var attempts atomic.Int32
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
attempts.Add(1)
|
||
w.WriteHeader(http.StatusInternalServerError)
|
||
_, _ = w.Write([]byte(`{"error":{"code":"internal_error"}}`))
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
|
||
ctx, cancel := context.WithCancel(context.Background())
|
||
go func() {
|
||
// 等第 1 次 attempt 跑完 + 進 retry sleep 後再 cancel
|
||
// 第 1 次 attempt 約 < 100ms;第 1 次 retry 退避 1s,在 500ms cancel
|
||
time.Sleep(500 * time.Millisecond)
|
||
cancel()
|
||
}()
|
||
|
||
start := time.Now()
|
||
_, err := fc.GetFile(ctx, "k")
|
||
duration := time.Since(start)
|
||
|
||
require.Error(t, err)
|
||
assert.True(t,
|
||
errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded),
|
||
"retry sleep 中 cancel → 透傳 ctx.Err()")
|
||
// 應在 cancel 後立即中斷(< 1s 整體時間)— 不該等完 1s 退避或進入第 2 次 retry
|
||
assert.Less(t, duration, 900*time.Millisecond,
|
||
"ctx cancel 應立即中斷 retry sleep(不等完退避)")
|
||
// attempts 應為 1(第 1 次 attempt 後進 retry sleep 就被 cancel)
|
||
assert.Equal(t, int32(1), attempts.Load(),
|
||
"cancel 後不應再嘗試第 2 次 attempt")
|
||
}
|
||
|
||
// ==========================================================================
|
||
// Token 失敗透傳
|
||
// ==========================================================================
|
||
|
||
// TestGetFile_ServiceTokenFailure_Propagated:MCTokenClient 失敗 → 透傳原 sentinel。
|
||
//
|
||
// 對應 mc_token_client.go 的 ErrIDPMisconfigured / ErrServiceClientUnauthorized / ErrIDPUnavailable,
|
||
// 不應被 faa_client 升級成 ErrFAAUnavailable(會丟失 i18n 區分 idp_misconfig vs idp_down vs faa_down)。
|
||
func TestGetFile_ServiceTokenFailure_Propagated(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
cases := []struct {
|
||
name string
|
||
tokenErr error
|
||
}{
|
||
{"idp_misconfigured", ErrIDPMisconfigured},
|
||
{"service_client_unauthorized", ErrServiceClientUnauthorized},
|
||
{"idp_unavailable", ErrIDPUnavailable},
|
||
}
|
||
|
||
for _, tc := range cases {
|
||
tc := tc
|
||
t.Run(tc.name, func(t *testing.T) {
|
||
t.Parallel()
|
||
tokens := newStubTokenClient("")
|
||
tokens.setError(tc.tokenErr)
|
||
|
||
// server 不應被打(token 取不到就 fail)
|
||
var serverHit atomic.Int32
|
||
mux := http.NewServeMux()
|
||
mux.HandleFunc("/files/", func(w http.ResponseWriter, r *http.Request) {
|
||
serverHit.Add(1)
|
||
w.WriteHeader(http.StatusOK)
|
||
})
|
||
srv := httptest.NewServer(mux)
|
||
t.Cleanup(srv.Close)
|
||
|
||
fc := newFAAClientForTest(t, srv.URL, tokens)
|
||
_, err := fc.GetFile(context.Background(), "k")
|
||
|
||
require.Error(t, err)
|
||
assert.True(t, errors.Is(err, tc.tokenErr),
|
||
"token 錯誤應透傳;不應包成 ErrFAAUnavailable")
|
||
assert.Equal(t, int32(0), serverHit.Load(),
|
||
"token 取不到時不應打 FAA")
|
||
})
|
||
}
|
||
}
|
||
|
||
// ==========================================================================
|
||
// 額外:empty object_key validation
|
||
// ==========================================================================
|
||
|
||
// TestGetFile_EmptyObjectKey:保護性 validation — 空字串 object_key 應立即 fail。
|
||
func TestGetFile_EmptyObjectKey(t *testing.T) {
|
||
t.Parallel()
|
||
|
||
tokens := newStubTokenClient("svc-tok")
|
||
fc := NewFAAClient(FAAClientOpts{
|
||
BaseURL: "http://invalid",
|
||
Tokens: tokens,
|
||
Logger: silentLogger(),
|
||
})
|
||
|
||
_, err := fc.GetFile(context.Background(), "")
|
||
require.Error(t, err)
|
||
// 不需走網路就應該 fail(token 沒被呼叫)
|
||
assert.Equal(t, 0, tokens.calls(scopeFAADownloadRead),
|
||
"empty object_key 應立即 fail,不該打 token endpoint")
|
||
}
|
||
|
||
// ==========================================================================
|
||
// hashObjectKey unit test(log 用 hash 函式的穩定性)
|
||
// ==========================================================================
|
||
|
||
// TestHashObjectKey_StableAndLength:同 input 應產生同 output;長度固定 16。
|
||
func TestHashObjectKey_StableAndLength(t *testing.T) {
|
||
t.Parallel()
|
||
h1 := hashObjectKey("tenant/jobs/abc/output.nef")
|
||
h2 := hashObjectKey("tenant/jobs/abc/output.nef")
|
||
h3 := hashObjectKey("tenant/jobs/xyz/output.nef")
|
||
|
||
assert.Equal(t, h1, h2, "同 object_key 應產生同 hash(log 可追蹤同一 request)")
|
||
assert.NotEqual(t, h1, h3, "不同 object_key hash 應不同")
|
||
assert.Len(t, h1, objectKeyHashLen, "hash 長度固定")
|
||
}
|