Merge remote-tracking branch 'origin/master' into hermes/hermes-1e775b03
Some checks failed
CI / verify (pull_request) Has been cancelled

# Conflicts:
#	docs/technical/README.md
#	src/components/custom-world-home/CustomWorldCreationHub.tsx
#	src/components/custom-world-home/creationWorkShelf.ts
#	src/components/platform-entry/PlatformEntryFlowShellImpl.tsx
#	src/components/rpg-entry/RpgEntryFlowShell.agent.interaction.test.tsx
This commit is contained in:
2026-05-12 15:02:47 +08:00
141 changed files with 13407 additions and 2277 deletions

205
scripts/loadtest/README.md Normal file
View File

@@ -0,0 +1,205 @@
# Genarrative 作品列表 K6 压测
本目录用于对“作品列表/公开广场”读接口做本地压测。数据源来自私有 SpacetimeDB migration但提取脚本只输出作品 profile 白名单表并对用户、作者、作品号、asset id 等标识做稳定映射。
## 文件
- `extract-works-list-data.mjs`:从 migration JSON 提取作品列表压测数据;本地输出也会脱敏路由 ID因此默认用于列表接口压测详情接口需先把同一份脱敏数据导入目标环境。
- `k6-works-list.js`K6 压测脚本。
- `data/spacetime-migration-7.local.json`:本地私有原始数据副本,已被 `.gitignore` 忽略,不要提交。
- `data/works-list.local.json`:本地脱敏压测数据,已被 `.gitignore` 忽略,不要提交。
- `data/works-list.sample.json`:可提交的少量脱敏样例。
## 数据边界
允许导入的表:
- `puzzle_work_profile`
- `custom_world_profile`
- `match3d_work_profile`
- `square_hole_work_profile`
- `big_fish_work_profile`
- `visual_novel_work_profile`
明确不导入:
- 账号/认证:`user_account``auth_identity``refresh_session``auth_store_snapshot`
- 钱包/邀请:`profile_wallet_ledger``profile_redeem_*``profile_invite_*`
- 游玩历史/埋点/存档:`public_work_play_daily_stat``profile_played_world``puzzle_runtime_run``profile_save_archive``runtime_snapshot`
- AI 任务过程:`ai_task``ai_task_stage``ai_text_chunk`
- asset 二进制:`asset_object``asset_entity_binding`
提取脚本会移除 `source_session_id` / `source_agent_session_id` 等会话派生字段;这些字段不属于作品列表卡片压测必要字段。
## 重新提取数据
从仓库根目录执行:
```bash
npm run loadtest:extract-works -- \
--input scripts/loadtest/data/spacetime-migration-7.local.json \
--output scripts/loadtest/data/works-list.local.json \
--sample-output scripts/loadtest/data/works-list.sample.json
```
也可以直接执行:
```bash
node scripts/loadtest/extract-works-list-data.mjs \
--input scripts/loadtest/data/spacetime-migration-7.local.json \
--output scripts/loadtest/data/works-list.local.json \
--sample-output scripts/loadtest/data/works-list.sample.json
```
当前 local 全量提取结果:
- `puzzle_work_profile`: 80
- `custom_world_profile`: 1
- `match3d_work_profile`: 0
- `normalizedWorks`: 81
当前可提交 sample 结果:
- `puzzle_work_profile`: 3
- `custom_world_profile`: 1
- `match3d_work_profile`: 0
- `normalizedWorks`: 4
## 真实接口
已从 `server-rs/crates/api-server/src/app.rs` 确认的读接口:
公开接口,无需 Bearer token
- `GET /api/runtime/puzzle/gallery`
- `GET /api/runtime/puzzle/gallery/{profile_id}`
- `GET /api/runtime/custom-world-gallery`
- `GET /api/runtime/custom-world-gallery/{owner_user_id}/{profile_id}`
- `GET /api/runtime/custom-world-gallery/by-code/{code}`
需要 Bearer token 的个人作品列表接口:
- `GET /api/runtime/puzzle/works`
- `GET /api/runtime/puzzle/works/{profile_id}`
- `GET /api/runtime/custom-world/works`
K6 脚本默认只跑公开列表接口;传入 `AUTH_TOKEN` 后会额外跑需要登录态的个人作品列表接口。当前真实列表 handler 未暴露分页/排序 query 参数,因此脚本不追加 `limit/offset`;若后续接口增加分页参数,再在 K6 中补随机分页。
详情接口默认不压测,因为本地数据中的 `profile_id` / `owner_user_id` 已脱敏,直接请求未导入脱敏数据的目标服务会 404。只有在目标环境已导入同一份脱敏数据或改用真实 ID 本地文件时,才设置 `DETAIL_RATIO` 大于 0详情请求不把 404 视为成功。
## 启动服务
按项目约定启动本地 dev 栈:
```bash
npm run dev
```
注意端口可能漂移。以启动日志中的实际 api-server 端口为准,然后传给 K6。
注意K6 的 `open()` 会按 `k6-works-list.js` 所在目录解析相对路径,因此 `WORKS_DATA` 应写成 `data/works-list.local.json`,不要写成 `scripts/loadtest/data/works-list.local.json`
Bash / Git Bash
```bash
BASE_URL=http://127.0.0.1:<actual-api-port> WORKS_DATA=data/works-list.local.json npm run loadtest:k6:works -- --summary-trend-stats="avg,min,med,p(90),p(95),p(99),max"
```
PowerShell
```powershell
$env:BASE_URL="http://127.0.0.1:<actual-api-port>"
$env:WORKS_DATA="data/works-list.local.json"
npm run loadtest:k6:works -- --summary-trend-stats="avg,min,med,p(90),p(95),p(99),max"
```
## Smoke
```bash
BASE_URL=http://127.0.0.1:8787 \
WORKS_DATA=data/works-list.local.json \
SCENARIO=smoke \
DETAIL_RATIO=0 \
npm run loadtest:k6:works
```
默认1 VU / 30s。
## Baseline
```bash
BASE_URL=http://127.0.0.1:8787 \
WORKS_DATA=data/works-list.local.json \
SCENARIO=baseline \
VUS=10 \
DURATION=3m \
DETAIL_RATIO=0 \
npm run loadtest:k6:works
```
默认阈值:
- `http_req_failed < 1%`
- `http_req_duration p95 < 800ms`
- `http_req_duration p99 < 1500ms`
- `works_list_shape_error_rate < 1%`
## Spike
```bash
BASE_URL=http://127.0.0.1:8787 \
WORKS_DATA=data/works-list.local.json \
SCENARIO=spike \
START_RPS=5 \
PEAK_RPS=100 \
HOLD=2m \
DETAIL_RATIO=0 \
npm run loadtest:k6:works
```
默认阈值:
- `http_req_failed < 5%`
- `http_req_duration p95 < 2000ms`
- `works_list_shape_error_rate < 5%`
## 带登录态压测个人作品列表
先通过本地登录或接口获取 access token然后传入
```bash
BASE_URL=http://127.0.0.1:8787 \
AUTH_TOKEN='<access-token>' \
SCENARIO=smoke \
DETAIL_RATIO=0 \
npm run loadtest:k6:works
```
不要把 token 写入仓库文件、README 或 shell history 中可共享的位置。
## 详情接口压测
仅当目标环境存在 `WORKS_DATA` 中的同一批 `profileId/ownerUserId` 时启用:
```bash
BASE_URL=http://127.0.0.1:8787 \
WORKS_DATA=data/works-list.local.json \
SCENARIO=smoke \
DETAIL_RATIO=0.35 \
npm run loadtest:k6:works
```
如果详情请求返回 404说明压测数据 ID 未导入目标环境或目标服务数据不一致,应先修正数据源,不要把 404 当成功。
## 排障
- 如果公开 gallery 返回 `creation_entry_disabled` 或 503检查本地 creation entry 配置是否禁用了对应入口。
- 如果个人作品列表返回 401确认 `AUTH_TOKEN` 是当前 api-server 可识别的 access token。
- 如果详情全部 404确认是否已向目标环境导入与 `WORKS_DATA` 一致的数据。
## 验证命令
```bash
npx vitest run scripts/loadtest/extract-works-list-data.test.ts
npx eslint scripts/loadtest/extract-works-list-data.mjs scripts/loadtest/extract-works-list-data.test.ts scripts/loadtest/k6-works-list.js
```

View File

@@ -0,0 +1,214 @@
{
"source": "spacetime-migration-7.local.json",
"generatedAt": "2026-05-11T13:09:51.569Z",
"counts": {
"puzzle_work_profile": 3,
"custom_world_profile": 1,
"match3d_work_profile": 0
},
"tables": {
"puzzle_work_profile": [
{
"profile_id": "profile-001",
"work_id": "work-001",
"owner_user_id": "user-001",
"author_display_name": "author-001",
"cover_asset_id": "asset-001",
"cover_image_src": "/generated-puzzle-assets/puzzle-session-f38101d7277040fcb6fbc41fea8b714a/puzzle-session-f38101d7277040fcb6fbc41fea8b714a-candidate-2/asset-1777649330373133/image.png",
"work_title": "化学家",
"level_name": "文学家",
"summary": "几个文学家正站在山上面对着瀑布侃侃而谈",
"work_description": "一个穿着白大褂的化学家正在做酷炫的化学实验,背景是化学实验室",
"levels_json": "[{\"level_id\":\"puzzle-level-1777649242577-7\",\"level_name\":\"文学家\",\"picture_description\":\"几个文学家正站在山上面对着瀑布侃侃而谈\",\"candidates\":[{\"candidate_id\":\"puzzle-session-f38101d7277040fcb6fbc41fea8b714a-candidate-2\",\"image_src\":\"/generated-puzzle-assets/puzzle-session-f38101d7277040fcb6fbc41fea8b714a/puzzle-session-f38101d7277040fcb6fbc41fea8b714a-candidate-2/asset-1777649330373133/image.png\",\"asset_id\":\"asset-1777649330373133\",\"prompt\":\"几个文学家正站在山上面对着瀑布侃侃而谈\",\"actual_prompt\":\"请生成一张高清插画。画面主体:几个文学家正站在山上面对着瀑布侃侃而谈。画面…",
"anchor_pack_json": "{\"theme_promise\":{\"key\":\"themePromise\",\"label\":\"题材承诺\",\"value\":\"化学家\",\"status\":\"Locked\"},\"visual_subject\":{\"key\":\"visualSubject\",\"label\":\"画面主体\",\"value\":\"一个穿着白大褂的化学家正在做酷炫的化学实验,背景是化学实验室\",\"status\":\"Locked\"},\"visual_mood\":{\"key\":\"visualMood\",\"label\":\"视觉气质\",\"value\":\"清晰、适合拼图切块\",\"status\":\"Inferred\"},\"composition_hooks\":{\"key\":\"compositionHooks\",\"label\":\"拼图记忆点\",\"value\":\"主体轮廓、色块分区、局部细节\",\"status\":\"Inferred\"},\"tags_and_forbidden\":{\"key\":\"tagsAndForbidden\",\"label\":\"标签与禁忌\",\"value\":\"化学家、拼图、插画;禁止标题字\",\"status\":\"I…",
"theme_tags_json": "[\"化学家\",\"拼图\",\"插画\",\"禁止标题字\"]",
"publication_status": {
"Published": []
},
"play_count": 1,
"like_count": 0,
"remix_count": 1,
"updated_at": {
"__timestamp_micros_since_unix_epoch__": 1777703338322544
},
"created_at": {
"__timestamp_micros_since_unix_epoch__": 1777648804043558
},
"published_at": {
"__timestamp_micros_since_unix_epoch__": 1777649364112270
}
},
{
"profile_id": "profile-002",
"work_id": "work-002",
"owner_user_id": "user-002",
"author_display_name": "author-002",
"work_title": "我不知道",
"level_name": "",
"summary": "你猜我是谁",
"work_description": "你猜我是谁",
"levels_json": "[{\"level_id\":\"puzzle-level-1\",\"level_name\":\"\",\"picture_description\":\"真不知道\",\"candidates\":[],\"selected_candidate_id\":null,\"cover_image_src\":null,\"cover_asset_id\":null,\"generation_status\":\"idle\"}]",
"anchor_pack_json": "{\"theme_promise\":{\"key\":\"themePromise\",\"label\":\"题材承诺\",\"value\":\"我不知道\",\"status\":\"Locked\"},\"visual_subject\":{\"key\":\"visualSubject\",\"label\":\"画面主体\",\"value\":\"真不知道\",\"status\":\"Locked\"},\"visual_mood\":{\"key\":\"visualMood\",\"label\":\"视觉气质\",\"value\":\"清晰、适合拼图切块\",\"status\":\"Inferred\"},\"composition_hooks\":{\"key\":\"compositionHooks\",\"label\":\"拼图记忆点\",\"value\":\"主体轮廓、色块分区、局部细节\",\"status\":\"Inferred\"},\"tags_and_forbidden\":{\"key\":\"tagsAndForbidden\",\"label\":\"标签与禁忌\",\"value\":\"我不知道、拼图、插画;禁止标题字\",\"status\":\"Inferred\"}}",
"theme_tags_json": "[\"我不知道\"]",
"publication_status": {
"Draft": []
},
"play_count": 0,
"like_count": 0,
"remix_count": 0,
"updated_at": {
"__timestamp_micros_since_unix_epoch__": 1777619351714201
},
"created_at": {
"__timestamp_micros_since_unix_epoch__": 1777619336673245
}
},
{
"profile_id": "profile-003",
"work_id": "work-003",
"owner_user_id": "user-003",
"author_display_name": "author-002",
"work_title": "",
"level_name": "",
"summary": "",
"work_description": "",
"levels_json": "[{\"level_id\":\"puzzle-level-1\",\"level_name\":\"\",\"picture_description\":\"\",\"candidates\":[],\"selected_candidate_id\":null,\"cover_image_src\":null,\"cover_asset_id\":null,\"generation_status\":\"idle\"}]",
"anchor_pack_json": "{\"theme_promise\":{\"key\":\"themePromise\",\"label\":\"题材承诺\",\"value\":\"\",\"status\":\"Missing\"},\"visual_subject\":{\"key\":\"visualSubject\",\"label\":\"画面主体\",\"value\":\"\",\"status\":\"Missing\"},\"visual_mood\":{\"key\":\"visualMood\",\"label\":\"视觉气质\",\"value\":\"\",\"status\":\"Missing\"},\"composition_hooks\":{\"key\":\"compositionHooks\",\"label\":\"拼图记忆点\",\"value\":\"\",\"status\":\"Missing\"},\"tags_and_forbidden\":{\"key\":\"tagsAndForbidden\",\"label\":\"标签与禁忌\",\"value\":\"\",\"status\":\"Missing\"}}",
"theme_tags_json": "[\"拼图\",\"插画\",\"清晰构图\"]",
"publication_status": {
"Draft": []
},
"play_count": 0,
"like_count": 0,
"remix_count": 0,
"updated_at": {
"__timestamp_micros_since_unix_epoch__": 1777622285252380
},
"created_at": {
"__timestamp_micros_since_unix_epoch__": 1777622285252380
}
}
],
"custom_world_profile": [
{
"profile_id": "profile-081",
"owner_user_id": "user-002",
"author_display_name": "author-012",
"author_public_user_code": "author-code-001",
"world_name": "青春飞扬校园",
"summary_text": "在现代校园中,玩家摆脱内卷,追求真实成长",
"subtitle": "反内卷的自由学习之旅",
"profile_payload_json": "{\"anchorContent\":null,\"anchorPack\":null,\"attributeSchema\":{\"generatedFrom\":{\"conflictCore\":\"与传统教育模式的冲突\",\"settingSummary\":\"在现代校园中,玩家摆脱内卷,追求真实成长\",\"tone\":\"积极向上,充满活力与创新\",\"worldName\":\"青春飞扬校园\",\"worldType\":\"CUSTOM\"},\"id\":\"schema:rpg-agent:1e15b44d:v1\",\"schemaVersion\":1,\"slots\":[{\"name\":\"知识储备\",\"slotId\":\"axis_a\"},{\"name\":\"创新思维\",\"slotId\":\"axis_b\"},{\"name\":\"社交能力\",\"slotId\":\"axis_c\"},{\"name\":\"抗压能力\",\"slotId\":\"axis_d\"},{\"name\":\"自我认知\",\"slotId\":\"axis_e\"},{\"name\":\"团队协作\",\"slotId\":\"axis_f\"}],\"worldId\":\"custom:青春飞扬校…",
"publication_status": {
"Draft": []
},
"play_count": 0,
"like_count": 0,
"remix_count": 0,
"updated_at": {
"__timestamp_micros_since_unix_epoch__": 1777532006629209
},
"created_at": {
"__timestamp_micros_since_unix_epoch__": 1777531745887256
}
}
],
"match3d_work_profile": []
},
"profileIds": {
"puzzle": [
"profile-001",
"profile-002",
"profile-003"
],
"customWorld": [
"profile-081"
],
"match3d": [],
"squareHole": [],
"bigFish": [],
"visualNovel": []
},
"workIds": {
"puzzle": [
"work-001",
"work-002",
"work-003"
],
"customWorld": [],
"match3d": [],
"squareHole": [],
"bigFish": [],
"visualNovel": []
},
"normalizedWorks": [
{
"type": "puzzle",
"workId": "work-001",
"profileId": "profile-001",
"ownerUserId": "user-001",
"title": "化学家",
"subtitle": "几个文学家正站在山上面对着瀑布侃侃而谈",
"publicationStatus": {
"Published": []
},
"playCount": 1,
"likeCount": 0,
"remixCount": 1,
"coverImageSrc": "/generated-puzzle-assets/puzzle-session-f38101d7277040fcb6fbc41fea8b714a/puzzle-session-f38101d7277040fcb6fbc41fea8b714a-candidate-2/asset-1777649330373133/image.png",
"updatedAt": {
"__timestamp_micros_since_unix_epoch__": 1777703338322544
}
},
{
"type": "puzzle",
"workId": "work-002",
"profileId": "profile-002",
"ownerUserId": "user-002",
"title": "我不知道",
"subtitle": "你猜我是谁",
"publicationStatus": {
"Draft": []
},
"playCount": 0,
"likeCount": 0,
"remixCount": 0,
"updatedAt": {
"__timestamp_micros_since_unix_epoch__": 1777619351714201
}
},
{
"type": "puzzle",
"workId": "work-003",
"profileId": "profile-003",
"ownerUserId": "user-003",
"title": "",
"subtitle": "",
"publicationStatus": {
"Draft": []
},
"playCount": 0,
"likeCount": 0,
"remixCount": 0,
"updatedAt": {
"__timestamp_micros_since_unix_epoch__": 1777622285252380
}
},
{
"type": "customWorld",
"profileId": "profile-081",
"ownerUserId": "user-002",
"title": "青春飞扬校园",
"subtitle": "反内卷的自由学习之旅",
"publicationStatus": {
"Draft": []
},
"playCount": 0,
"likeCount": 0,
"remixCount": 0,
"updatedAt": {
"__timestamp_micros_since_unix_epoch__": 1777532006629209
}
}
]
}

View File

@@ -0,0 +1,370 @@
#!/usr/bin/env node
import { readFile, writeFile } from 'node:fs/promises';
import { basename } from 'node:path';
import { fileURLToPath } from 'node:url';
const ALLOWED_TABLES = new Set([
'puzzle_work_profile',
'custom_world_profile',
'match3d_work_profile',
'square_hole_work_profile',
'big_fish_work_profile',
'visual_novel_work_profile',
]);
const WORK_TABLE_TYPES = {
puzzle_work_profile: 'puzzle',
custom_world_profile: 'customWorld',
match3d_work_profile: 'match3d',
square_hole_work_profile: 'squareHole',
big_fish_work_profile: 'bigFish',
visual_novel_work_profile: 'visualNovel',
};
const TABLE_OUTPUT_ORDER = [
'puzzle_work_profile',
'custom_world_profile',
'match3d_work_profile',
'square_hole_work_profile',
'big_fish_work_profile',
'visual_novel_work_profile',
];
const WORK_TYPES = ['puzzle', 'customWorld', 'match3d', 'squareHole', 'bigFish', 'visualNovel'];
const SHORT_TEXT_LIMIT = 120;
const LONG_TEXT_LIMIT = 500;
const SENSITIVE_PATTERN = /(token|secret|password|passwd|phone|wallet|credential|authorization|auth[_-]?key|api[_-]?key)/giu;
class StableMapper {
constructor(prefix) {
this.prefix = prefix;
this.values = new Map();
}
map(value) {
if (value === undefined || value === null || value === '') return value;
const key = String(value);
if (!this.values.has(key)) {
this.values.set(
key,
`${this.prefix}-${String(this.values.size + 1).padStart(3, '0')}`,
);
}
return this.values.get(key);
}
}
function createContext() {
return {
user: new StableMapper('user'),
session: new StableMapper('session'),
author: new StableMapper('author'),
authorCode: new StableMapper('author-code'),
publicWorkCode: new StableMapper('public-work-code'),
coverAsset: new StableMapper('asset'),
work: new StableMapper('work'),
profile: new StableMapper('profile'),
};
}
function createWorkTypeBuckets() {
return Object.fromEntries(WORK_TYPES.map((type) => [type, []]));
}
function unwrapSpacetimeOption(value) {
if (
value &&
typeof value === 'object' &&
!Array.isArray(value) &&
Object.keys(value).length === 1
) {
if (Object.prototype.hasOwnProperty.call(value, 'some')) return value.some;
if (Object.prototype.hasOwnProperty.call(value, 'none')) return undefined;
}
return value;
}
function truncateText(value, limit) {
if (value === undefined || value === null) return value;
const text = String(value).replace(/\s+/g, ' ').trim();
if (text.length <= limit) return text;
return `${text.slice(0, limit)}`;
}
function redactSensitiveText(value) {
if (value === undefined || value === null) return value;
return String(value).replace(SENSITIVE_PATTERN, '[redacted]');
}
function sanitizeCoverImageSrc(value) {
const unwrapped = unwrapSpacetimeOption(value);
if (unwrapped === undefined || unwrapped === null || unwrapped === '') return unwrapped;
const text = String(unwrapped);
if (text.startsWith('data:image/')) return '[redacted-data-image]';
let withoutQuery = text.split('?')[0].split('#')[0];
if (withoutQuery.length > 180) withoutQuery = `${withoutQuery.slice(0, 180)}`;
return withoutQuery;
}
function sanitizeLargeJson(value) {
const unwrapped = unwrapSpacetimeOption(value);
if (unwrapped === undefined || unwrapped === null) return unwrapped;
if (typeof unwrapped === 'string') {
return truncateText(redactSensitiveText(unwrapped), LONG_TEXT_LIMIT);
}
try {
return truncateText(redactSensitiveText(JSON.stringify(unwrapped)), LONG_TEXT_LIMIT);
} catch {
return truncateText(redactSensitiveText(String(unwrapped)), LONG_TEXT_LIMIT);
}
}
function firstDefined(row, keys) {
for (const key of keys) {
if (row[key] !== undefined && row[key] !== null) return row[key];
}
return undefined;
}
function sanitizeShortField(row, sanitized, key) {
if (row[key] !== undefined) {
sanitized[key] = truncateText(unwrapSpacetimeOption(row[key]), SHORT_TEXT_LIMIT);
}
}
function sanitizeWorkRow(row, ctx) {
const sanitized = {};
const profileId = unwrapSpacetimeOption(firstDefined(row, ['profile_id', 'profileId']));
const workId = unwrapSpacetimeOption(firstDefined(row, ['work_id', 'workId']));
if (profileId !== undefined) sanitized.profile_id = ctx.profile.map(profileId);
if (workId !== undefined) sanitized.work_id = ctx.work.map(workId);
if (row.owner_user_id !== undefined) {
sanitized.owner_user_id = ctx.user.map(unwrapSpacetimeOption(row.owner_user_id));
}
if (row.user_id !== undefined) sanitized.user_id = ctx.user.map(unwrapSpacetimeOption(row.user_id));
if (row.author_display_name !== undefined) {
sanitized.author_display_name = ctx.author.map(unwrapSpacetimeOption(row.author_display_name));
}
if (row.public_work_code !== undefined) {
sanitized.public_work_code = ctx.publicWorkCode.map(unwrapSpacetimeOption(row.public_work_code));
}
if (row.author_public_user_code !== undefined) {
sanitized.author_public_user_code = ctx.authorCode.map(
unwrapSpacetimeOption(row.author_public_user_code),
);
}
if (row.cover_asset_id !== undefined) {
sanitized.cover_asset_id = ctx.coverAsset.map(unwrapSpacetimeOption(row.cover_asset_id));
}
if (row.cover_image_src !== undefined) sanitized.cover_image_src = sanitizeCoverImageSrc(row.cover_image_src);
for (const key of [
'title',
'work_title',
'level_name',
'world_name',
'summary',
'summary_text',
'description',
'work_description',
'subtitle',
]) {
sanitizeShortField(row, sanitized, key);
}
for (const key of ['levels_json', 'profile_payload_json', 'anchor_pack_json', 'theme_tags_json']) {
if (row[key] !== undefined) sanitized[key] = sanitizeLargeJson(row[key]);
}
const passthroughKeys = [
'publication_status',
'publicationStatus',
'play_count',
'playCount',
'like_count',
'likeCount',
'remix_count',
'remixCount',
'updated_at',
'created_at',
'published_at',
'visibility',
'status',
'category',
'tags',
];
for (const key of passthroughKeys) {
if (row[key] !== undefined) sanitized[key] = unwrapSpacetimeOption(row[key]);
}
return sanitized;
}
function normalizeWork(tableName, row) {
const type = WORK_TABLE_TYPES[tableName];
return {
type,
workId: row.work_id,
profileId: row.profile_id,
ownerUserId: row.owner_user_id,
publicWorkCode: row.public_work_code,
title: row.title ?? row.work_title ?? row.level_name ?? row.world_name,
subtitle: row.subtitle ?? row.summary_text ?? row.summary ?? row.work_description ?? row.description,
publicationStatus: row.publicationStatus ?? row.publication_status ?? row.status,
playCount: row.playCount ?? row.play_count ?? 0,
likeCount: row.likeCount ?? row.like_count ?? 0,
remixCount: row.remixCount ?? row.remix_count ?? 0,
coverImageSrc: row.cover_image_src,
updatedAt: row.updated_at,
};
}
function toRowsByTable(input) {
const tables = Array.isArray(input?.tables) ? input.tables : [];
const result = new Map();
for (const table of tables) {
if (!ALLOWED_TABLES.has(table?.name)) continue;
result.set(table.name, Array.isArray(table.rows) ? table.rows : []);
}
return result;
}
export function extractWorksListData(input, options = {}) {
const ctx = createContext();
const rowsByTable = toRowsByTable(input);
const outputTables = {};
const counts = {};
const profileIds = createWorkTypeBuckets();
const workIds = createWorkTypeBuckets();
const normalizedWorks = [];
for (const tableName of TABLE_OUTPUT_ORDER) {
const sourceRows = rowsByTable.get(tableName);
if (!sourceRows) continue;
const sanitizedRows = sourceRows.map((row) => sanitizeWorkRow(row, ctx));
outputTables[tableName] = sanitizedRows;
counts[tableName] = sanitizedRows.length;
const type = WORK_TABLE_TYPES[tableName];
if (type) {
for (const row of sanitizedRows) {
if (row.profile_id) profileIds[type].push(row.profile_id);
if (row.work_id) workIds[type].push(row.work_id);
normalizedWorks.push(normalizeWork(tableName, row));
}
}
}
return {
source: options.source ?? 'unknown',
generatedAt: options.generatedAt ?? new Date().toISOString(),
counts,
tables: outputTables,
profileIds,
workIds,
normalizedWorks,
};
}
function createSampleOutput(output, maxRowsPerTable = 3) {
const tables = {};
const counts = {};
const allowedWorkIds = new Set();
const allowedProfileIds = new Set();
for (const [tableName, rows] of Object.entries(output.tables)) {
tables[tableName] = rows.slice(0, maxRowsPerTable);
counts[tableName] = tables[tableName].length;
const type = WORK_TABLE_TYPES[tableName];
if (type) {
for (const row of tables[tableName]) {
if (row.work_id) allowedWorkIds.add(row.work_id);
if (row.profile_id) allowedProfileIds.add(row.profile_id);
}
}
}
const profileIds = Object.fromEntries(
Object.entries(output.profileIds).map(([type, ids]) => [
type,
ids.filter((id) => allowedProfileIds.has(id)).slice(0, maxRowsPerTable),
]),
);
const workIds = Object.fromEntries(
Object.entries(output.workIds).map(([type, ids]) => [
type,
ids.filter((id) => allowedWorkIds.has(id)).slice(0, maxRowsPerTable),
]),
);
const normalizedWorks = output.normalizedWorks
.filter((work) => allowedWorkIds.has(work.workId) || allowedProfileIds.has(work.profileId))
.slice(0, maxRowsPerTable * 6);
return {
...output,
counts,
tables,
profileIds,
workIds,
normalizedWorks,
};
}
function parseArgs(argv) {
const args = {};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === '--input' || arg === '--output' || arg === '--sample-output') {
const value = argv[index + 1];
if (!value || value.startsWith('--')) throw new Error(`${arg} requires a value`);
args[arg.slice(2)] = value;
index += 1;
} else if (arg === '--help' || arg === '-h') {
args.help = true;
} else {
throw new Error(`Unknown argument: ${arg}`);
}
}
return args;
}
function usage() {
return 'Usage: node scripts/loadtest/extract-works-list-data.mjs --input <migration.json> --output <works-list.local.json> [--sample-output <works-list.sample.json>]';
}
export async function runCli(argv = process.argv.slice(2)) {
const args = parseArgs(argv);
if (args.help) {
console.log(usage());
return;
}
if (!args.input) throw new Error('Missing required --input. ' + usage());
if (!args.output) throw new Error('Missing required --output. ' + usage());
const raw = await readFile(args.input, 'utf8');
const migration = JSON.parse(raw);
const output = extractWorksListData(migration, { source: basename(args.input) });
await writeFile(args.output, `${JSON.stringify(output, null, 2)}\n`, 'utf8');
if (args['sample-output']) {
const sample = createSampleOutput(output);
await writeFile(args['sample-output'], `${JSON.stringify(sample, null, 2)}\n`, 'utf8');
}
console.log(
`works-list extracted: source=${output.source}, tables=${Object.keys(output.tables).length}, normalizedWorks=${output.normalizedWorks.length}`,
);
for (const [tableName, count] of Object.entries(output.counts)) {
console.log(` ${tableName}: ${count}`);
}
}
const isDirectRun = process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1];
if (isDirectRun) {
runCli().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exitCode = 1;
});
}

View File

@@ -0,0 +1,247 @@
import { execFile } from 'node:child_process';
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { promisify } from 'node:util';
import { describe, expect, it } from 'vitest';
import { extractWorksListData } from './extract-works-list-data.mjs';
const execFileAsync = promisify(execFile);
const scriptPath = fileURLToPath(new URL('./extract-works-list-data.mjs', import.meta.url));
const fixtureMigration = {
schema_version: 7,
tables: [
{
name: 'puzzle_work_profile',
rows: [
{
profile_id: 'profile-real-aaa',
work_id: 'work-real-aaa',
owner_user_id: 'owner-secret-123',
author_display_name: 'Alice Secret',
author_public_user_code: 'author-code-secret',
public_work_code: 'public-code-secret',
title: '超长标题'.repeat(20),
summary: 'summary '.repeat(80),
description: 'description '.repeat(120),
publication_status: 'published',
play_count: 42,
like_count: 7,
cover_asset_id: { some: 'asset-secret-cover' },
cover_image_src: { some: 'https://cdn.example.test/cover.png?token=***&sig=abc' },
levels_json: JSON.stringify({ secret: 'level-token-value', data: 'x'.repeat(2000) }),
theme_tags_json: JSON.stringify(['化学家', '实验室']),
remix_count: 2,
updated_at: '2026-05-01T00:00:00Z',
},
{
profile_id: 'profile-real-bbb',
work_id: 'work-real-bbb',
owner_user_id: 'owner-secret-123',
author_display_name: 'Alice Secret',
publication_status: 'draft',
play_count: 3,
},
],
},
{
name: 'custom_world_profile',
rows: [
{
profile_id: 'world-profile-secret',
work_id: 'world-work-secret',
owner_user_id: 'world-owner-secret',
title: '世界作品',
profile_payload_json: '{"large":"' + 'y'.repeat(2000) + '"}',
},
],
},
{
name: 'public_work_play_daily_stat',
rows: [
{
source_type: 'puzzle',
profile_id: 'profile-real-aaa',
owner_user_id: 'owner-secret-123',
user_id: 'player-secret-456',
source_session_id: 'session-secret-789',
played_day: '2026-05-01',
play_count: 12,
updated_at: '2026-05-02T00:00:00Z',
},
],
},
{
name: 'user_account',
rows: [
{
user_id: 'owner-secret-123',
phone: '+8613800138000',
auth_token: 'auth-token-secret',
wallet_balance: 999,
},
],
},
{
name: 'refresh_session',
rows: [{ token: 'refresh-token-secret', source_session_id: 'session-secret-789' }],
},
{
name: 'profile_wallet_ledger',
rows: [{ wallet_id: 'wallet-secret', amount: 100 }],
},
],
};
async function withTempDir(fn) {
const dir = await mkdtemp(path.join(tmpdir(), 'works-list-test-'));
try {
return await fn(dir);
} finally {
await rm(dir, { recursive: true, force: true });
}
}
describe('extractWorksListData', () => {
it('只保留作品 profile 白名单表,禁用的行为/敏感表不会出现在输出 JSON 字符串中', () => {
const output = extractWorksListData(fixtureMigration, { source: 'fixture.local.json' });
const serialized = JSON.stringify(output);
expect(Object.keys(output.tables).sort()).toEqual([
'custom_world_profile',
'puzzle_work_profile',
]);
expect(serialized).not.toContain('public_work_play_daily_stat');
expect(serialized).not.toContain('user_account');
expect(serialized).not.toContain('refresh_session');
expect(serialized).not.toContain('profile_wallet_ledger');
expect(serialized).not.toContain('+8613800138000');
expect(serialized).not.toContain('auth-token-secret');
expect(serialized).not.toContain('wallet-secret');
});
it('不会输出 owner/user/session/auth/token/phone/wallet 等敏感原值owner 稳定映射', () => {
const output = extractWorksListData(fixtureMigration, { source: 'fixture.local.json' });
const serialized = JSON.stringify(output);
for (const secret of [
'owner-secret-123',
'player-secret-456',
'session-secret-789',
'Alice Secret',
'author-code-secret',
'public-code-secret',
'asset-secret-cover',
'SECRET_TOKEN',
]) {
expect(serialized).not.toContain(secret);
}
expect(output.tables.puzzle_work_profile[0].owner_user_id).toBe('user-001');
expect(output.tables.puzzle_work_profile[1].owner_user_id).toBe('user-001');
expect(output.tables.puzzle_work_profile[0].author_display_name).toBe('author-001');
expect(serialized).not.toContain('level-token-value');
});
it('puzzle 数据生成 profileIds/workIds 和 normalizedWorks并保留列表展示字段', () => {
const output = extractWorksListData(fixtureMigration, { source: 'fixture.local.json' });
expect(output.source).toBe('fixture.local.json');
expect(output.generatedAt).toEqual(expect.any(String));
expect(output.counts.puzzle_work_profile).toBe(2);
expect(output.profileIds.puzzle).toEqual(['profile-001', 'profile-002']);
expect(output.workIds.puzzle).toEqual(['work-001', 'work-002']);
expect(output.normalizedWorks).toEqual(
expect.arrayContaining([
expect.objectContaining({
type: 'puzzle',
workId: 'work-001',
profileId: 'profile-001',
publicationStatus: 'published',
playCount: 42,
title: expect.any(String),
remixCount: 2,
}),
]),
);
expect(output.tables.puzzle_work_profile[0].cover_image_src).toBe('https://cdn.example.test/cover.png');
expect(output.tables.puzzle_work_profile[0].theme_tags_json).toBe('["化学家","实验室"]');
});
it('data image、URL token 和绝对输入路径不会泄露到输出', async () => {
await withTempDir(async (dir) => {
const input = path.join(dir, 'migration.local.json');
const output = path.join(dir, 'works-list.local.json');
await writeFile(
input,
JSON.stringify({
tables: [
{
name: 'puzzle_work_profile',
rows: [
{
profile_id: 'profile-real',
work_id: 'work-real',
cover_image_src: { some: 'data:image/png;base64,SECRET_IMAGE_BYTES' },
levels_json: JSON.stringify({ token: 'SECRET_TOKEN_VALUE', title: 'safe' }),
},
],
},
],
}),
'utf8',
);
await execFileAsync(process.execPath, [scriptPath, '--input', input, '--output', output]);
const extracted = JSON.parse(await readFile(output, 'utf8'));
const serialized = JSON.stringify(extracted);
expect(extracted.source).toBe('migration.local.json');
expect(serialized).not.toContain(dir);
expect(serialized).not.toContain('SECRET_IMAGE_BYTES');
expect(serialized).not.toContain('SECRET_TOKEN_VALUE');
expect(extracted.tables.puzzle_work_profile[0].cover_image_src).toBe('[redacted-data-image]');
});
});
it('sample-output 只输出少量脱敏样例', async () => {
await withTempDir(async (dir) => {
const input = path.join(dir, 'migration.local.json');
const output = path.join(dir, 'works-list.local.json');
const sampleOutput = path.join(dir, 'works-list.sample.json');
const manyRows = Array.from({ length: 5 }, (_, index) => ({
profile_id: `profile-real-${index}`,
work_id: `work-real-${index}`,
owner_user_id: `owner-secret-${index}`,
title: `作品 ${index}`,
publication_status: 'published',
play_count: index,
}));
await writeFile(
input,
JSON.stringify({ tables: [{ name: 'puzzle_work_profile', rows: manyRows }] }),
'utf8',
);
await execFileAsync(process.execPath, [scriptPath, '--input', input, '--output', output, '--sample-output', sampleOutput]);
const sample = JSON.parse(await readFile(sampleOutput, 'utf8'));
const serialized = JSON.stringify(sample);
expect(sample.tables.puzzle_work_profile).toHaveLength(3);
expect(sample.normalizedWorks).toHaveLength(3);
expect(serialized).not.toContain('owner-secret-0');
expect(serialized).not.toContain('work-real-0');
});
});
it('CLI 参数缺失时退出非 0 并输出清晰错误', async () => {
await expect(execFileAsync(process.execPath, [scriptPath, '--input', 'missing.json'])).rejects.toMatchObject({
code: 1,
stderr: expect.stringContaining('--output'),
});
});
});

View File

@@ -0,0 +1,229 @@
/* global __ENV */
import { check, sleep } from 'k6';
import { SharedArray } from 'k6/data';
import http from 'k6/http';
import { Rate, Trend } from 'k6/metrics';
// k6 resolves open() paths relative to this script file, not the shell cwd.
const DEFAULT_WORKS_DATA = 'data/works-list.local.json';
const WORKS_DATA = __ENV.WORKS_DATA || DEFAULT_WORKS_DATA;
const BASE_URL = (__ENV.BASE_URL || 'http://127.0.0.1:8787').replace(/\/+$/u, '');
const AUTH_TOKEN = __ENV.AUTH_TOKEN || '';
const SCENARIO = __ENV.SCENARIO || 'smoke';
const REQUEST_TIMEOUT = __ENV.REQUEST_TIMEOUT || '30s';
const SLEEP_MIN_SECONDS = Number(__ENV.SLEEP_MIN_SECONDS || '0.5');
const SLEEP_MAX_SECONDS = Number(__ENV.SLEEP_MAX_SECONDS || '2');
const DETAIL_RATIO = Number(__ENV.DETAIL_RATIO || '0');
const worksListShapeErrorRate = new Rate('works_list_shape_error_rate');
const worksDetailShapeErrorRate = new Rate('works_detail_shape_error_rate');
const worksListDuration = new Trend('works_list_duration');
const worksDetailDuration = new Trend('works_detail_duration');
const data = new SharedArray('works-list-data', () => [JSON.parse(open(WORKS_DATA))])[0];
const normalizedWorks = Array.isArray(data.normalizedWorks) ? data.normalizedWorks : [];
const scenarioOptions = {
smoke: {
scenarios: {
smoke: {
executor: 'constant-vus',
vus: Number(__ENV.VUS || 1),
duration: __ENV.DURATION || '30s',
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(95)<800'],
works_list_shape_error_rate: ['rate<0.01'],
},
},
baseline: {
scenarios: {
baseline: {
executor: 'constant-vus',
vus: Number(__ENV.VUS || 10),
duration: __ENV.DURATION || '3m',
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(95)<800', 'p(99)<1500'],
works_list_shape_error_rate: ['rate<0.01'],
},
},
spike: {
scenarios: {
spike: {
executor: 'ramping-arrival-rate',
preAllocatedVUs: Number(__ENV.PREALLOCATED_VUS || 50),
maxVUs: Number(__ENV.MAX_VUS || 200),
timeUnit: '1s',
stages: [
{ target: Number(__ENV.START_RPS || 5), duration: __ENV.RAMP_UP || '30s' },
{ target: Number(__ENV.PEAK_RPS || 100), duration: __ENV.HOLD || '2m' },
{ target: Number(__ENV.END_RPS || 5), duration: __ENV.RAMP_DOWN || '30s' },
],
},
},
thresholds: {
http_req_failed: ['rate<0.05'],
http_req_duration: ['p(95)<2000'],
works_list_shape_error_rate: ['rate<0.05'],
},
},
};
export const options = scenarioOptions[SCENARIO] || scenarioOptions.smoke;
const PUBLIC_ENDPOINTS = [
{
name: 'puzzle_gallery_list',
method: 'GET',
path: '/api/runtime/puzzle/gallery',
expectCollectionKeys: ['items', 'works', 'entries'],
},
{
name: 'custom_world_gallery_list',
method: 'GET',
path: '/api/runtime/custom-world-gallery',
expectCollectionKeys: ['entries', 'items', 'works'],
},
];
const AUTH_ENDPOINTS = [
{
name: 'puzzle_works_list',
method: 'GET',
path: '/api/runtime/puzzle/works',
expectCollectionKeys: ['items', 'works'],
},
{
name: 'custom_world_works_list',
method: 'GET',
path: '/api/runtime/custom-world/works',
expectCollectionKeys: ['items', 'entries', 'works'],
},
];
function requestParams(endpointName) {
const headers = { 'x-genarrative-response-envelope': 'v1' };
if (AUTH_TOKEN) headers.Authorization = `Bearer ${AUTH_TOKEN}`;
return {
headers,
timeout: REQUEST_TIMEOUT,
tags: { endpoint: endpointName },
};
}
function buildUrl(path) {
return `${BASE_URL}${path}`;
}
function parseJson(response) {
try {
return response.json();
} catch (_) {
return null;
}
}
function unwrapPayload(json) {
if (!json || typeof json !== 'object') return null;
if (json.data && typeof json.data === 'object') return json.data;
return json;
}
function hasCollection(payload, keys) {
return keys.some((key) => Array.isArray(payload?.[key]));
}
function firstCollection(payload, keys) {
for (const key of keys) {
if (Array.isArray(payload?.[key])) return payload[key];
}
return [];
}
function hasListItemShape(payload, keys) {
const collection = firstCollection(payload, keys);
if (collection.length === 0) return true;
const item = collection[0];
const hasId = Boolean(
item?.profileId || item?.profile_id || item?.workId || item?.work_id || item?.publicWorkCode,
);
const hasTitle = Boolean(
item?.title || item?.workTitle || item?.work_title || item?.levelName || item?.worldName,
);
return hasId && hasTitle;
}
function randomItem(items) {
if (!items.length) return null;
return items[Math.floor(Math.random() * items.length)];
}
function listEndpoints() {
return AUTH_TOKEN ? PUBLIC_ENDPOINTS.concat(AUTH_ENDPOINTS) : PUBLIC_ENDPOINTS;
}
function detailEndpointFor(work) {
if (!work || !work.profileId) return null;
if (work.type === 'puzzle') {
return {
name: 'puzzle_gallery_detail',
path: `/api/runtime/puzzle/gallery/${encodeURIComponent(work.profileId)}`,
expectKeys: ['item', 'work', 'entry'],
};
}
if (work.type === 'customWorld' && work.profileId && work.ownerUserId) {
return {
name: 'custom_world_gallery_detail',
path: `/api/runtime/custom-world-gallery/${encodeURIComponent(work.ownerUserId)}/${encodeURIComponent(work.profileId)}`,
expectKeys: ['entry', 'item', 'work'],
};
}
return null;
}
function performListRequest(endpoint) {
const url = buildUrl(endpoint.path);
const response = http.request(endpoint.method, url, null, requestParams(endpoint.name));
worksListDuration.add(response.timings.duration, { endpoint: endpoint.name });
const json = parseJson(response);
const payload = unwrapPayload(json);
const ok = check(response, {
[`${endpoint.name} status is 200`]: (res) => res.status === 200,
[`${endpoint.name} returns json object`]: () => Boolean(payload),
[`${endpoint.name} has collection`]: () => hasCollection(payload, endpoint.expectCollectionKeys),
[`${endpoint.name} list item shape`]: () => hasListItemShape(payload, endpoint.expectCollectionKeys),
});
worksListShapeErrorRate.add(!ok, { endpoint: endpoint.name });
}
function performDetailRequest() {
const endpoint = detailEndpointFor(randomItem(normalizedWorks));
if (!endpoint) return;
const response = http.get(buildUrl(endpoint.path), requestParams(endpoint.name));
worksDetailDuration.add(response.timings.duration, { endpoint: endpoint.name });
const json = parseJson(response);
const payload = unwrapPayload(json);
const ok = check(response, {
[`${endpoint.name} status is 200`]: (res) => res.status === 200,
[`${endpoint.name} has detail payload`]: () => endpoint.expectKeys.some((key) => payload?.[key]),
});
worksDetailShapeErrorRate.add(!ok, { endpoint: endpoint.name });
}
export default function () {
for (const endpoint of listEndpoints()) {
performListRequest(endpoint);
}
if (normalizedWorks.length && DETAIL_RATIO > 0 && Math.random() < DETAIL_RATIO) {
performDetailRequest();
}
const jitter = SLEEP_MIN_SECONDS + Math.random() * Math.max(0, SLEEP_MAX_SECONDS - SLEEP_MIN_SECONDS);
sleep(jitter);
}