补充BgFilter全进程与真实链路冒烟验证
新增32/40/48并发、五类故障与harness自测。 新增真实OSS和BgFilter契约冒烟并严格清理临时对象。 补充本地运行方式、凭据边界与staging验收范围。
This commit is contained in:
@@ -464,6 +464,23 @@ BgFilter 成功二进制不是一份新的业务资产：
|
||||
- 生产运行期巡检同时检查 `genarrative-bgfilter-worker.service` 为 active 且 `127.0.0.1:8083/readyz` 成功，不能只依赖 systemd 自动重启。
|
||||
- `npm run dev` 启动独立 BgFilter 子进程并使用解析后的第五个端口；`all` 角色不内嵌 listener，单模块入口、watch、状态文件和退出清理没有遗留进程或硬编码端口。
|
||||
|
||||
本地全进程调度门禁使用已构建的 `api-server` binary 和 loopback mock provider，不读取真实 OSS / BgFilter 密钥，也不访问真实外部服务。`load-smoke` 固定验证 `R = 32 / 40 / 48、N = 4、Q = 128`；`fault-smoke` 用独立 worker / mock 生命周期验证 `Q` 满快速拒绝、queue deadline、`503 → 200` 顺序重试、两次 `503` 后 provider exhausted，以及 provider 成功响应 body 中途 reset 后第二次 attempt 串行成功。默认读取 `server-rs/target/debug/api-server(.exe)`；在 WSL 或自定义 target 目录运行时，通过 `GENARRATIVE_BGFILTER_SMOKE_BINARY` 指定 binary：
|
||||
|
||||
```bash
|
||||
cargo build -p api-server --manifest-path server-rs/Cargo.toml
|
||||
npm run bgfilter-worker:smoke-test
|
||||
npm run bgfilter-worker:load-smoke
|
||||
npm run bgfilter-worker:fault-smoke
|
||||
```
|
||||
|
||||
真实 OSS + BgFilter 契约冒烟不属于默认门禁，会访问真实服务并产生调用成本。执行前必须在当前进程环境中以不回显方式注入与 loopback worker 相同的一次性 `GENARRATIVE_BGFILTER_INTERNAL_TOKEN`，不得把 token 写进命令行、仓库 env 文件或日志；OSS / BgFilter 配置只放本地私密环境。分别设置 `GENARRATIVE_BGFILTER_SMOKE_MODE=flat` 与 `complex` 后运行：
|
||||
|
||||
```bash
|
||||
cargo run -p platform-oss --example bgfilter_worker_live_smoke --manifest-path server-rs/Cargo.toml
|
||||
```
|
||||
|
||||
该 harness 固定使用 `generated-character-drafts/bgfilter-smoke/<requestId>/source.png`：PUT 前必须先确认 HEAD=404，随后验证私有上传、无鉴权 401 精确错误契约、真实 `200 image/png`，最后要求 DELETE 2xx 且 HEAD=404。正常失败也会尝试清理；若进程崩溃或被强杀，必须按输出的 object key 人工复核。它验证真实 OSS / BgFilter 边界，不替代完整父流程验收；父 flat 的 fallback 本地证据仍由真实阿里云 `segment_smoke` 与父路由单测组合提供，`mock worker 失败 → 父 flat → 真实阿里云`、完整动画写回及计费 / lease / 退款链留在 staging 验收。
|
||||
|
||||
实现后按范围运行：
|
||||
|
||||
```bash
|
||||
|
||||
@@ -608,6 +608,10 @@ npm run container:down
|
||||
`npm run container:config` 默认只做 quiet 校验，避免把本地 env 中的 token 展开到终端；确需排查完整 compose 时再传 `-- --print`。
|
||||
隔离验证 worker 队列和 API-only 更新时使用 `npm run container:worker-smoke -- smoke`。该命令不复用 `deploy/container/api-server.env`，会在 `deploy/container/worker-smoke/` 生成本机专用 env 与端口 state，并使用 unsupported job 验证 worker claim / fail 回写，不需要真实外部生成密钥；本机 crates.io 网络不稳时使用 `--local-binary`，由容器内 Cargo 复用本机 Cargo 缓存构建，并把产物放进 Debian bookworm smoke runtime。
|
||||
|
||||
独立 BgFilter worker 的本机全进程验证先运行 `cargo build -p api-server --manifest-path server-rs/Cargo.toml`，再依次运行 `npm run bgfilter-worker:smoke-test`、`npm run bgfilter-worker:load-smoke` 和 `npm run bgfilter-worker:fault-smoke`。三条命令只使用动态 loopback 端口、假 OSS 签名配置和本地 mock provider；不会读取仓库 `.env*` 或请求真实 BgFilter / OSS。自定义或 WSL binary 通过 `GENARRATIVE_BGFILTER_SMOKE_BINARY` 指定。当前 fault 范围包含 overload、queue deadline、两类 HTTP 状态顺序重试结果，以及 provider 成功响应 body 中途 reset 后第二次 attempt 串行成功；慢读、大响应、父侧客户端断连与 SIGTERM 排空另行验证。
|
||||
|
||||
需要复核真实 OSS + BgFilter 契约时，先启动只监听 loopback 的 worker，并在 worker 与 smoke 的当前进程环境中以不回显方式注入同一个一次性 `GENARRATIVE_BGFILTER_INTERNAL_TOKEN`；token 不得写入命令参数、仓库 env 文件或日志。OSS / BgFilter 凭据继续只放本地私密环境。分别设置 `GENARRATIVE_BGFILTER_SMOKE_MODE=flat` 和 `complex`，运行 `cargo run -p platform-oss --example bgfilter_worker_live_smoke --manifest-path server-rs/Cargo.toml`。该命令会访问真实服务并产生调用成本；成功标准是 PUT 前 HEAD=404、私有上传成功、无鉴权请求返回精确 401 JSON、带鉴权请求返回 `200 image/png`、DELETE 2xx 且最终 HEAD=404。对象只写入 `generated-character-drafts/bgfilter-smoke/<requestId>/source.png`；正常失败会继续清理，进程崩溃或被强杀时需按输出 object key 人工复核。此 smoke 不经过用户 job、计费或父 flat fallback；完整 `mock worker 失败 → 父 flat → 真实阿里云` 和动画写回链在 staging 验收。
|
||||
|
||||
OpenTelemetry 现阶段默认开启 OTLP traces / metrics / logs，但本地日志与 Nginx 文件日志仍保留：
|
||||
|
||||
- 生产与容器 `api-server` env 模板默认 `GENARRATIVE_OTEL_ENABLED=true`；压测、排障或短期要关闭 OTLP 时，必须显式设置 `GENARRATIVE_OTEL_ENABLED=false`。
|
||||
|
||||
@@ -8,6 +8,9 @@
|
||||
"dev:spacetime": "node scripts/dev.mjs spacetime",
|
||||
"dev:api-server": "node scripts/dev.mjs api-server",
|
||||
"dev:bgfilter-worker": "node scripts/dev.mjs bgfilter-worker",
|
||||
"bgfilter-worker:load-smoke": "node scripts/bgfilter-worker-load-smoke.mjs",
|
||||
"bgfilter-worker:fault-smoke": "node scripts/bgfilter-worker-load-smoke.mjs fault",
|
||||
"bgfilter-worker:smoke-test": "node --test scripts/bgfilter-worker-load-smoke.test.mjs",
|
||||
"dev:web": "node scripts/dev.mjs web",
|
||||
"dev:admin-web": "node scripts/dev.mjs admin-web",
|
||||
"server-manager:panel": "cargo run -p server-manager-panel --manifest-path server-rs/Cargo.toml",
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,190 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import http from 'node:http';
|
||||
import { afterEach, describe, test } from 'node:test';
|
||||
|
||||
import {
|
||||
buildIsolatedWorkerEnv,
|
||||
createProviderGate,
|
||||
createProviderSequenceBehavior,
|
||||
SMOKE_PNG_BYTES,
|
||||
startMockBgfilterProvider,
|
||||
} from './bgfilter-worker-load-smoke.mjs';
|
||||
|
||||
const providers = [];
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(providers.splice(0).map((provider) => provider.close()));
|
||||
});
|
||||
|
||||
describe('bgfilter worker smoke harness', () => {
|
||||
test('worker 环境不继承真实服务密钥并固定使用假 OSS 配置', () => {
|
||||
const env = buildIsolatedWorkerEnv({
|
||||
processEnv: {
|
||||
ALIYUN_OSS_ACCESS_KEY_SECRET: 'real-oss-secret',
|
||||
GENARRATIVE_BGFILTER_INTERNAL_TOKEN: 'real-internal-token',
|
||||
GENARRATIVE_EDITOR_BGFILTER_TOKEN: 'real-provider-token',
|
||||
PATH: '/safe/bin',
|
||||
VECTOR_ENGINE_API_KEY: 'real-vector-secret',
|
||||
},
|
||||
providerBaseUrl: 'http://127.0.0.1:19001',
|
||||
tempRoot: '/tmp/bgfilter-load-smoke-test',
|
||||
token: 'ephemeral-test-token',
|
||||
workerPort: 19002,
|
||||
});
|
||||
|
||||
assert.equal(env.PATH, '/safe/bin');
|
||||
assert.equal(
|
||||
env.GENARRATIVE_BGFILTER_INTERNAL_TOKEN,
|
||||
'ephemeral-test-token',
|
||||
);
|
||||
assert.equal(env.ALIYUN_OSS_ENDPOINT, 'oss-cn-shanghai.invalid');
|
||||
assert.notEqual(env.ALIYUN_OSS_ACCESS_KEY_SECRET, 'real-oss-secret');
|
||||
assert.equal(env.GENARRATIVE_EDITOR_BGFILTER_TOKEN, undefined);
|
||||
assert.equal(env.VECTOR_ENGINE_API_KEY, undefined);
|
||||
assert.ok(!Object.values(env).includes('real-internal-token'));
|
||||
assert.ok(!Object.values(env).includes('real-provider-token'));
|
||||
assert.ok(!Object.values(env).includes('real-vector-secret'));
|
||||
});
|
||||
|
||||
test('loopback mock 完整读取 multipart 后记录并发并返回合法 PNG 字节', async () => {
|
||||
const provider = await startMockBgfilterProvider({ delayMs: 25 });
|
||||
providers.push(provider);
|
||||
const request = multipartFixture();
|
||||
|
||||
const responses = await Promise.all([
|
||||
postMultipart(provider.baseUrl, request),
|
||||
postMultipart(provider.baseUrl, request),
|
||||
]);
|
||||
|
||||
for (const response of responses) {
|
||||
assert.equal(response.statusCode, 200);
|
||||
assert.equal(response.contentType, 'image/png');
|
||||
assert.ok(response.body.equals(SMOKE_PNG_BYTES));
|
||||
}
|
||||
const stats = provider.snapshot();
|
||||
assert.equal(stats.active, 0);
|
||||
assert.equal(stats.peak, 2);
|
||||
assert.equal(stats.requests, 2);
|
||||
assert.deepEqual(stats.violations, []);
|
||||
assert.equal(stats.timeline.filter((event) => event.event === 'start').length, 2);
|
||||
assert.equal(stats.timeline.filter((event) => event.event === 'finish').length, 2);
|
||||
});
|
||||
|
||||
test('provider gate 与 sequence behavior 生成无重叠 timeline', async () => {
|
||||
const gate = createProviderGate();
|
||||
const provider = await startMockBgfilterProvider({
|
||||
behavior: createProviderSequenceBehavior([503, 200]),
|
||||
delayMs: 5,
|
||||
gate,
|
||||
});
|
||||
providers.push(provider);
|
||||
const request = multipartFixture();
|
||||
let firstSettled = false;
|
||||
const first = postMultipart(provider.baseUrl, request).finally(() => {
|
||||
firstSettled = true;
|
||||
});
|
||||
|
||||
await provider.waitFor((stats) => stats.active === 1, { timeoutMs: 1_000 });
|
||||
await new Promise((resolve) => setTimeout(resolve, 20));
|
||||
assert.equal(firstSettled, false);
|
||||
gate.release();
|
||||
assert.equal((await first).statusCode, 503);
|
||||
assert.equal((await postMultipart(provider.baseUrl, request)).statusCode, 200);
|
||||
|
||||
const stats = provider.snapshot();
|
||||
assert.equal(stats.peak, 1);
|
||||
assert.deepEqual(
|
||||
stats.timeline.map((event) => [
|
||||
event.attempt,
|
||||
event.event,
|
||||
event.statusCode ?? null,
|
||||
]),
|
||||
[
|
||||
[1, 'start', null],
|
||||
[1, 'finish', 503],
|
||||
[2, 'start', null],
|
||||
[2, 'finish', 200],
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test('provider 可在成功响应 body 中途 reset 并记录完成类型', async () => {
|
||||
const provider = await startMockBgfilterProvider({
|
||||
behavior: createProviderSequenceBehavior([
|
||||
{ resetMidBody: true, statusCode: 200 },
|
||||
]),
|
||||
delayMs: 5,
|
||||
});
|
||||
providers.push(provider);
|
||||
|
||||
await assert.rejects(postMultipart(provider.baseUrl, multipartFixture()));
|
||||
|
||||
const stats = provider.snapshot();
|
||||
assert.equal(stats.active, 0);
|
||||
assert.equal(stats.peak, 1);
|
||||
assert.equal(stats.requests, 1);
|
||||
assert.deepEqual(stats.violations, []);
|
||||
assert.equal(stats.timeline[1]?.completion, 'mid_body_reset');
|
||||
});
|
||||
});
|
||||
|
||||
function multipartFixture() {
|
||||
const boundary = 'bgfilter-load-smoke-boundary';
|
||||
const body = Buffer.from(
|
||||
[
|
||||
`--${boundary}`,
|
||||
'Content-Disposition: form-data; name="image_url"',
|
||||
'',
|
||||
'https://example.invalid/source.png',
|
||||
`--${boundary}`,
|
||||
'Content-Disposition: form-data; name="seg_model"',
|
||||
'',
|
||||
'birefnet',
|
||||
`--${boundary}`,
|
||||
'Content-Disposition: form-data; name="background_mode"',
|
||||
'',
|
||||
'complex',
|
||||
`--${boundary}`,
|
||||
'Content-Disposition: form-data; name="cross_check"',
|
||||
'',
|
||||
'off',
|
||||
`--${boundary}--`,
|
||||
'',
|
||||
].join('\r\n'),
|
||||
'utf8',
|
||||
);
|
||||
return { body, boundary };
|
||||
}
|
||||
|
||||
function postMultipart(baseUrl, { body, boundary }) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const request = http.request(
|
||||
`${baseUrl}/remove-background`,
|
||||
{
|
||||
agent: false,
|
||||
headers: {
|
||||
'Content-Length': String(body.length),
|
||||
'Content-Type': `multipart/form-data; boundary=${boundary}`,
|
||||
},
|
||||
method: 'POST',
|
||||
},
|
||||
(response) => {
|
||||
const chunks = [];
|
||||
response.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
|
||||
response.once('aborted', () => {
|
||||
reject(new Error('mock provider 响应在 body 中途中止'));
|
||||
});
|
||||
response.once('error', reject);
|
||||
response.once('end', () => {
|
||||
resolve({
|
||||
body: Buffer.concat(chunks),
|
||||
contentType: String(response.headers['content-type'] ?? ''),
|
||||
statusCode: response.statusCode ?? 0,
|
||||
});
|
||||
});
|
||||
},
|
||||
);
|
||||
request.once('error', reject);
|
||||
request.end(body);
|
||||
});
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user