Merge branch 'codex/container-simulate'

# Conflicts: # .hermes/shared-memory/decision-log.md # server-rs/crates/api-server/src/puzzle.rs # server-rs/crates/spacetime-client/src/mapper.rs
2026-05-19 10:07:45 +08:00
parent a0979138b1 f557bc3f06
commit c9b3468b29
40 changed files with 2795 additions and 165 deletions
--- a/deploy/container/README.md
+++ b/deploy/container/README.md
@@ -6,20 +6,28 @@

 ```text
 Docker Compose
+├─ spacetimedb  :3101，独立数据卷，供 api-server 连接
 ├─ nginx        :80 -> api-server:8082，负责静态站点、/admin/、/api/ 反代、upstream timing log、连接限制
-├─ api-server   :8082，Linux release 构建，连接外部 SpacetimeDB
+├─ api-server   :8082，Linux release 构建，连接 compose 内 SpacetimeDB
 ├─ otelcol      :4317/4318，debug exporter，接收 traces / metrics / logs
 └─ k6           profile=loadtest 时临时启动，在 compose 网络内压 nginx
 ```

+当前容器模拟参数按 `genarrative-release` 服务器采样值收口为 2 vCPU / 2 GiB RAM / 4096 soft nofile / 768 worker_connections，并已在 compose 里落实到 `spacetimedb cpus=1.0 mem_limit=896m`、`api-server cpus=2.0 mem_limit=1g`、`nginx cpus=0.5 mem_limit=128m`、`otelcol cpus=0.25 mem_limit=128m`、`k6 cpus=1.0 mem_limit=512m`。SpacetimeDB 同时设置 `--page_pool_max_size=402653184`，给 reducer、订阅与运行时保留更多非 page pool 内存。
+容器 `api-server` 默认 `GENARRATIVE_API_WORKER_THREADS=4`，用于让 Tokio 在 2 vCPU 配额内有更多 I/O 调度 worker；该值不会突破 compose 里的 `cpus=2.0` CPU 上限。
+Collector 镜像使用 `otel/opentelemetry-collector-contrib:0.151.0`。
+生产服务器若启用 Collector，则由 `deploy/systemd/otelcol-contrib.service` 和 `deploy/otelcol/genarrative-debug.yaml` 托管，不走容器镜像。
+
 默认 host 端口：

+- `http://127.0.0.1:13101`：容器 SpacetimeDB。
 - `http://127.0.0.1:18080`：容器 Nginx。
 - `127.0.0.1:4317` / `127.0.0.1:4318`：容器 Collector OTLP gRPC / HTTP。

 如端口冲突，可设置：

 ```powershell
+$env:GENARRATIVE_CONTAINER_SPACETIME_PORT="13102"
 $env:GENARRATIVE_CONTAINER_HTTP_PORT="18081"
 $env:GENARRATIVE_CONTAINER_OTLP_HTTP_PORT="14318"
 $env:GENARRATIVE_CONTAINER_OTLP_GRPC_PORT="14317"
@@ -33,21 +41,29 @@ npm run container:init

 该命令会从 `deploy/container/api-server.env.example` 生成本地 `deploy/container/api-server.env`。真实 token、库名和外部服务密钥只写本地 env 文件，不提交 Git。

-Docker Desktop 下默认通过 `host.docker.internal:3101` 连接宿主机上 `npm run dev` 启动的 SpacetimeDB：
+Docker Desktop 下默认通过 `http://spacetimedb:3101` 连接 compose 内 SpacetimeDB；宿主机只负责用 CLI 发布模块：

 ```env
-GENARRATIVE_SPACETIME_SERVER_URL=http://host.docker.internal:3101
+GENARRATIVE_SPACETIME_SERVER_URL=http://spacetimedb:3101
 GENARRATIVE_SPACETIME_DATABASE=genarrative-loadtest
 GENARRATIVE_SPACETIME_TOKEN=
 ```

-Linux Docker Engine 如果不能解析 `host.docker.internal`，Compose 已配置 `host-gateway`；仍不通时把 `GENARRATIVE_SPACETIME_SERVER_URL` 改成宿主机网关 IP 或同网络内的 SpacetimeDB 地址。
+宿主机发布模块时，先用 CLI 向 `http://127.0.0.1:13101` 发布到 `genarrative-loadtest`，再启动 `npm run container:up`。
+
+Linux Docker Engine 若要从宿主机 CLI 连到容器内服务，直接用 `http://127.0.0.1:13101`；容器内部服务之间统一走 `http://spacetimedb:3101`。
+
+## 构建工具链
+
+`api-server` 容器镜像只构建 Linux release API 二进制，不构建 `spacetime-module`。当前 `api-server -> spacetime-client -> spacetimedb-sdk 2.2.0` 依赖链要求 Rust 1.93，因此 `deploy/container/api-server.Dockerfile` 的 Rust builder 固定为 `rust:1.93-bookworm`。如果本机 Docker Hub 拉取失败，可以先在本机准备同名本地 builder 镜像，但不要把临时 bootstrap 容器或私有 registry 凭据写入仓库。

 ## 启动与验证

 ```bash
 npm run container:config
 npm run container:build
+npm run container:up -- spacetimedb
+spacetime publish genarrative-loadtest --server http://127.0.0.1:13101 --module-path server-rs/crates/spacetime-module --yes --build-options="--debug"
 npm run container:up
 npm run container:ps
 curl -sS http://127.0.0.1:18080/api/runtime/puzzle/gallery
@@ -103,7 +119,30 @@ $env:DETAIL_RATIO="0"
 npm run container:k6
 ```

-如果要压 1000 HTTP req/s，把 `PEAK_RPS` 调到 `500`；如果要压 5000 HTTP req/s，把 `PEAK_RPS` 调到 `2500`，并同时提高 `PREALLOCATED_VUS` / `MAX_VUS`，观察是否先被带宽、Nginx `limit_conn` 或 api-server 背压限制。
+容器内 `api-server` 资源上限与 Nginx 连接模型已经按 `genarrative-release` 的 2C / 2G / `nofile=4096` / `worker_connections=768` 收口；如果你要改成别的机器，就先重新采样再改这里。
+
+SpacetimeDB 容器默认只提供运行时，不自动发布模块。首次启动或清理 `spacetime-data` 卷后，先只启动 `spacetimedb` 服务，再发布模块：
+
+```bash
+npm run container:up -- spacetimedb
+spacetime publish genarrative-loadtest --server http://127.0.0.1:13101 --module-path server-rs/crates/spacetime-module --yes --build-options="--debug"
+```
+
+发布完成后再执行 `npm run container:up` 和 `npm run container:k6`。如果 `deploy/container/api-server.env` 里的 `GENARRATIVE_SPACETIME_DATABASE` 改成了别的库名，发布命令里的库名也要同步修改。
+
+如果要压 1000 HTTP req/s，把 `PEAK_RPS` 调到 `500`；如果要压 5000 HTTP req/s，把 `PEAK_RPS` 调到 `2500`，并同时提高 `PREALLOCATED_VUS` / `MAX_VUS`，观察是否先被带宽、Nginx `limit_conn` / `limit_req` 或 api-server 分组背压限制。当前容器 Nginx 对公开 gallery list 使用 `genarrative_gallery_rps`，公开详情和普通 API 使用 `genarrative_api_rps`，后台 API 使用 `genarrative_admin_rps`；api-server 侧对应 `GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS`、`GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS` 和 `GENARRATIVE_API_ADMIN_MAX_CONCURRENT_REQUESTS`。
+
+2026-05-19 的 2C / 2G 容器压测结论：公开 gallery list 的 `limit_conn=320`、`limit_req rate=5000r/s burst=4096` 与 `GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=320` 是当前发布口径。用宿主机 k6 打 `http://127.0.0.1:18080`，`PEAK_RPS=2500` 等价于约 5000 HTTP req/s 的两接口组合压测；连续 10 轮不重启 SpacetimeDB 的平均实际吞吐约 `4219 HTTP req/s`，总计 `1,897,357` 个 200、`212,542` 个 429、`0` 个 5xx，200 请求平均 `p95=123ms`、`p99=234ms`。该档会让 SpacetimeDB 内存从约 `366MiB` 累积到约 `885MiB / 896MiB`，下游内存先到危险区。当前不要为了降低“剩余 CPU”继续抬公开列表并发；下一步应减少成功列表请求后的 SpacetimeDB tracking 写入或优化下游连接 / 订阅状态，而不是放大入口并发。
+
+### 内存采样
+
+排查 API 容器内存时，优先对比压测前后的 `/proc/$pid/smaps_rollup` 和 cgroup 当前/峰值，不把 Windows 任务管理器总占用当成单进程结论：
+
+```bash
+docker exec genarrative-container-loadtest-api-server-1 sh -c 'pid=$(pidof api-server); grep VmRSS /proc/$pid/status; grep RssAnon /proc/$pid/status; cat /proc/$pid/smaps_rollup | grep Anonymous; echo cgroup_current=$(cat /sys/fs/cgroup/memory.current); echo cgroup_peak=$(cat /sys/fs/cgroup/memory.peak)'
+```
+
+`/healthz` 也能复现的内存尖峰应先按连接层、service clone 或 allocator 高水位排查，不要直接归因到 SpacetimeDB procedure、作品列表 cache 或业务 DTO。2026-05-18 验证：`AppState` 改为 `Arc<AppStateInner>` 浅拷贝后，容器内直连 `api-server:8082/healthz` 的 500 HTTP req/s、`PREALLOCATED_VUS=100`、30 秒压测完成 `15001` 次请求，`http_req_failed=0`、`dropped_iterations=0`，API 进程 RSS 从约 18 MiB 升至约 52 MiB，cgroup 峰值约 47 MiB，未再出现 1 GiB 级尖峰。

 ## OTLP

@@ -123,6 +162,18 @@ npm run container:logs -- otelcol

 Collector 日志会输出 traces / metrics / logs。接 Rider、Jaeger、Tempo、Prometheus、Grafana 或托管平台时，另建独立 Collector 配置，不直接改生产 systemd 或 Nginx 模板。

+容器内需要临时转发到 Grafana Cloud 时，切换 Collector 配置并从当前 shell 传入 Grafana Cloud 凭据；真实 token 不写入仓库文件：
+
+```powershell
+$env:GENARRATIVE_CONTAINER_OTELCOL_CONFIG="./otelcol.grafana.yaml"
+$env:GRAFANA_CLOUD_OTLP_ENDPOINT="https://..."
+$env:GRAFANA_CLOUD_BASIC_AUTH_HEADER="Basic ..."
+npm run container:up
+npm run container:logs -- otelcol
+```
+
+`deploy/container/otelcol.grafana.yaml` 会同时保留本地 debug exporter，并通过 `otlphttp/grafana` 把 traces / metrics / logs 发到 Grafana Cloud。
+
 ## 隔离边界

 - 不改生产 systemd 单元。
--- a/deploy/container/api-server.Dockerfile
+++ b/deploy/container/api-server.Dockerfile
@@ -1,4 +1,4 @@
-FROM rust:1.88-bookworm AS rust-builder
+FROM rust:1.93-bookworm AS rust-builder
 WORKDIR /workspace

 COPY server-rs ./server-rs
@@ -15,7 +15,7 @@ RUN apt-get update && \

 COPY --from=rust-builder /tmp/api-server /usr/local/bin/api-server

-RUN mkdir -p /var/lib/genarrative/auth && \
+RUN mkdir -p /var/lib/genarrative/auth /var/lib/genarrative/tracking-outbox && \
    chown -R genarrative:genarrative /srv/genarrative /var/lib/genarrative

 USER genarrative
@@ -24,7 +24,8 @@ EXPOSE 8082
 ENV GENARRATIVE_ENV=container \
    GENARRATIVE_API_HOST=0.0.0.0 \
    GENARRATIVE_API_PORT=8082 \
-    GENARRATIVE_AUTH_STORE_PATH=/var/lib/genarrative/auth/auth-store.json
+    GENARRATIVE_AUTH_STORE_PATH=/var/lib/genarrative/auth/auth-store.json \
+    GENARRATIVE_TRACKING_OUTBOX_DIR=/var/lib/genarrative/tracking-outbox

 CMD ["api-server"]

@@ -36,6 +37,7 @@ COPY apps/admin-web/package.json ./apps/admin-web/package.json
 RUN npm ci

 COPY index.html metadata.json tsconfig.json vite.config.ts ./
+COPY scripts/vite-cli.mjs scripts/admin-web-build.mjs ./scripts/
 COPY src ./src
 COPY public ./public
 COPY media ./media
--- a/deploy/container/api-server.env.example
+++ b/deploy/container/api-server.env.example
@@ -9,8 +9,16 @@ GENARRATIVE_API_LOG=info,tower_http=info
 GENARRATIVE_API_LISTEN_BACKLOG=1024
 GENARRATIVE_API_WORKER_THREADS=4
 GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512
+GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=320
+GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS=64
+GENARRATIVE_API_ADMIN_MAX_CONCURRENT_REQUESTS=16
+GENARRATIVE_TRACKING_OUTBOX_ENABLED=true
+GENARRATIVE_TRACKING_OUTBOX_DIR=/var/lib/genarrative/tracking-outbox
+GENARRATIVE_TRACKING_OUTBOX_BATCH_SIZE=500
+GENARRATIVE_TRACKING_OUTBOX_FLUSH_INTERVAL_MS=1000
+GENARRATIVE_TRACKING_OUTBOX_MAX_BYTES=268435456

-GENARRATIVE_OTEL_ENABLED=false
+GENARRATIVE_OTEL_ENABLED=true
 OTEL_SERVICE_NAME=genarrative-api
 OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4318
 OTEL_RESOURCE_ATTRIBUTES=deployment.environment=container,service.namespace=genarrative
@@ -21,9 +29,8 @@ GENARRATIVE_JWT_SECRET=CHANGE_ME_FOR_CONTAINER
 AUTH_REFRESH_COOKIE_SECURE=false
 GENARRATIVE_AUTH_STORE_PATH=/var/lib/genarrative/auth/auth-store.json

-# Docker Desktop 下连接宿主机 npm run dev 启动的 SpacetimeDB。
-# Linux Docker Engine 可改成宿主机网关 IP，或在 compose 里接入同一网络内的 SpacetimeDB。
-GENARRATIVE_SPACETIME_SERVER_URL=http://host.docker.internal:3101
+# 默认连接 compose 内部 SpacetimeDB；宿主机发布模块使用 127.0.0.1:13101。
+GENARRATIVE_SPACETIME_SERVER_URL=http://spacetimedb:3101
 GENARRATIVE_SPACETIME_DATABASE=genarrative-loadtest
 GENARRATIVE_SPACETIME_TOKEN=
 GENARRATIVE_SPACETIME_POOL_SIZE=8
--- a/deploy/container/docker-compose.loadtest.yml
+++ b/deploy/container/docker-compose.loadtest.yml
@@ -1,11 +1,48 @@
 name: genarrative-container-loadtest

 services:
+  spacetimedb:
+    image: clockworklabs/spacetime:v2.2.0
+    user: root
+    command:
+      [
+        "start",
+        "--listen-addr",
+        "0.0.0.0:3101",
+        "--data-dir",
+        "/var/lib/spacetimedb",
+        "--page_pool_max_size",
+        "402653184",
+        "--non-interactive",
+      ]
+    cpus: "1.0"
+    mem_limit: 896m
+    ports:
+      - "${GENARRATIVE_CONTAINER_SPACETIME_PORT:-13101}:3101"
+    volumes:
+      - spacetime-data:/var/lib/spacetimedb
+    ulimits:
+      nofile:
+        soft: 4096
+        hard: 4096
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          "spacetime server ping http://127.0.0.1:3101 >/dev/null 2>&1",
+        ]
+      interval: 10s
+      timeout: 5s
+      retries: 12
+      start_period: 20s
+
  api-server:
    build:
      context: ../..
      dockerfile: deploy/container/api-server.Dockerfile
      target: api-runtime
+    cpus: "2.0"
+    mem_limit: 1g
    env_file:
      - ./api-server.env
    environment:
@@ -16,7 +53,14 @@ services:
      - "host.docker.internal:host-gateway"
    volumes:
      - api-auth-store:/var/lib/genarrative/auth
+      - api-tracking-outbox:/var/lib/genarrative/tracking-outbox
+    ulimits:
+      nofile:
+        soft: 4096
+        hard: 4096
    depends_on:
+      spacetimedb:
+        condition: service_healthy
      otelcol:
        condition: service_started
    healthcheck:
@@ -31,15 +75,23 @@ services:
      context: ../..
      dockerfile: deploy/container/api-server.Dockerfile
      target: nginx-runtime
+    cpus: "0.5"
+    mem_limit: 128m
    depends_on:
      api-server:
        condition: service_healthy
+      spacetimedb:
+        condition: service_healthy
    ports:
      - "${GENARRATIVE_CONTAINER_HTTP_PORT:-18080}:80"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - nginx-logs:/var/log/nginx
+    ulimits:
+      nofile:
+        soft: 4096
+        hard: 4096
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1/api/runtime/puzzle/gallery"]
      interval: 10s
@@ -48,10 +100,16 @@ services:
      start_period: 20s

  otelcol:
-    image: otel/opentelemetry-collector-contrib:0.125.0
+    image: otel/opentelemetry-collector-contrib:0.151.0
    command: ["--config=/etc/otelcol/config.yaml"]
+    cpus: "0.25"
+    mem_limit: 128m
+    environment:
+      GRAFANA_CLOUD_OTLP_ENDPOINT: ${GRAFANA_CLOUD_OTLP_ENDPOINT:-}
+      GRAFANA_CLOUD_BASIC_AUTH_HEADER: ${GRAFANA_CLOUD_BASIC_AUTH_HEADER:-}
+      HOSTNAME: ${HOSTNAME:-genarrative-container-loadtest}
    volumes:
-      - ./otelcol.yaml:/etc/otelcol/config.yaml:ro
+      - ${GENARRATIVE_CONTAINER_OTELCOL_CONFIG:-./otelcol.yaml}:/etc/otelcol/config.yaml:ro
    ports:
      - "${GENARRATIVE_CONTAINER_OTLP_GRPC_PORT:-4317}:4317"
      - "${GENARRATIVE_CONTAINER_OTLP_HTTP_PORT:-4318}:4318"
@@ -59,6 +117,8 @@ services:
  k6:
    image: grafana/k6:0.52.0
    profiles: ["loadtest"]
+    cpus: "1.0"
+    mem_limit: 512m
    depends_on:
      nginx:
        condition: service_healthy
@@ -81,5 +141,7 @@ services:
    command: ["run", "k6-works-list.js"]

 volumes:
+  spacetime-data:
  api-auth-store:
+  api-tracking-outbox:
  nginx-logs:
--- a/deploy/container/nginx.conf
+++ b/deploy/container/nginx.conf
@@ -1,7 +1,7 @@
 worker_processes auto;

 events {
-    worker_connections 4096;
+    worker_connections 768;
 }

 http {
@@ -21,6 +21,9 @@ http {
    }

    limit_conn_zone $binary_remote_addr zone=genarrative_api_conn:10m;
+    limit_req_zone $binary_remote_addr zone=genarrative_gallery_rps:10m rate=5000r/s;
+    limit_req_zone $binary_remote_addr zone=genarrative_api_rps:10m rate=300r/s;
+    limit_req_zone $binary_remote_addr zone=genarrative_admin_rps:10m rate=30r/s;

    sendfile on;
    keepalive_timeout 65;
@@ -48,6 +51,8 @@ http {
        error_log /var/log/nginx/genarrative.error.log warn;
        limit_conn_status 429;
        limit_conn_log_level warn;
+        limit_req_status 429;
+        limit_req_log_level warn;

        root /srv/genarrative/web;
        index index.html;
@@ -55,6 +60,7 @@ http {
        location ^~ /admin/api/ {
            default_type application/json;
            limit_conn genarrative_api_conn 64;
+            limit_req zone=genarrative_admin_rps burst=16 nodelay;

            proxy_pass http://genarrative_api/admin/api/;
            proxy_http_version 1.1;
@@ -82,9 +88,90 @@ http {
            try_files $uri =404;
        }

+        location = /api/runtime/puzzle/gallery {
+            default_type application/json;
+            limit_conn genarrative_api_conn 320;
+            limit_req zone=genarrative_gallery_rps burst=4096 nodelay;
+
+            proxy_pass http://genarrative_api;
+            proxy_http_version 1.1;
+            proxy_buffering off;
+            proxy_read_timeout 3600s;
+            proxy_send_timeout 3600s;
+            add_header X-Accel-Buffering no always;
+            proxy_set_header Connection "";
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header X-Forwarded-Host $host;
+            proxy_set_header X-Request-Id $request_id;
+        }
+
+        location = /api/runtime/custom-world-gallery {
+            default_type application/json;
+            limit_conn genarrative_api_conn 320;
+            limit_req zone=genarrative_gallery_rps burst=4096 nodelay;
+
+            proxy_pass http://genarrative_api;
+            proxy_http_version 1.1;
+            proxy_buffering off;
+            proxy_read_timeout 3600s;
+            proxy_send_timeout 3600s;
+            add_header X-Accel-Buffering no always;
+            proxy_set_header Connection "";
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header X-Forwarded-Host $host;
+            proxy_set_header X-Request-Id $request_id;
+        }
+
+        location ~ ^/api/runtime/puzzle/gallery/[^/]+$ {
+            default_type application/json;
+            limit_conn genarrative_api_conn 32;
+            limit_req zone=genarrative_api_rps burst=32 nodelay;
+
+            proxy_pass http://genarrative_api;
+            proxy_http_version 1.1;
+            proxy_buffering off;
+            proxy_read_timeout 3600s;
+            proxy_send_timeout 3600s;
+            add_header X-Accel-Buffering no always;
+            proxy_set_header Connection "";
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header X-Forwarded-Host $host;
+            proxy_set_header X-Request-Id $request_id;
+        }
+
+        location ~ ^/api/runtime/custom-world-gallery/[^/]+/[^/]+$ {
+            default_type application/json;
+            limit_conn genarrative_api_conn 32;
+            limit_req zone=genarrative_api_rps burst=32 nodelay;
+
+            proxy_pass http://genarrative_api;
+            proxy_http_version 1.1;
+            proxy_buffering off;
+            proxy_read_timeout 3600s;
+            proxy_send_timeout 3600s;
+            add_header X-Accel-Buffering no always;
+            proxy_set_header Connection "";
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header X-Forwarded-Host $host;
+            proxy_set_header X-Request-Id $request_id;
+        }
+
        location ~ ^/api(?:/|$) {
            default_type application/json;
            limit_conn genarrative_api_conn 64;
+            limit_req zone=genarrative_api_rps burst=64 nodelay;

            proxy_pass http://genarrative_api;
            proxy_http_version 1.1;
@@ -106,7 +193,7 @@ http {
        }

        location ~ ^/v1/database/[^/]+/subscribe$ {
-            proxy_pass http://host.docker.internal:3101;
+            proxy_pass http://spacetimedb:3101;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
@@ -115,7 +202,7 @@ http {
        }

        location ^~ /v1/identity {
-            proxy_pass http://host.docker.internal:3101;
+            proxy_pass http://spacetimedb:3101;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
--- a/deploy/container/otelcol.grafana.yaml
+++ b/deploy/container/otelcol.grafana.yaml
@@ -0,0 +1,36 @@
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+
+processors:
+  batch:
+    timeout: 5s
+    send_batch_size: 512
+    send_batch_max_size: 1024
+
+exporters:
+  debug:
+    verbosity: basic
+  otlp_http/grafana:
+    endpoint: ${env:GRAFANA_CLOUD_OTLP_ENDPOINT}
+    headers:
+      Authorization: ${env:GRAFANA_CLOUD_BASIC_AUTH_HEADER}
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug, otlp_http/grafana]
+    metrics:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug, otlp_http/grafana]
+    logs:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug, otlp_http/grafana]
--- a/deploy/env/api-server.env.example
+++ b/deploy/env/api-server.env.example
@@ -8,7 +8,15 @@ GENARRATIVE_API_LOG=info,tower_http=info
 GENARRATIVE_API_LISTEN_BACKLOG=1024
 GENARRATIVE_API_WORKER_THREADS=4
 GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512
-GENARRATIVE_OTEL_ENABLED=false
+GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=320
+GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS=64
+GENARRATIVE_API_ADMIN_MAX_CONCURRENT_REQUESTS=16
+GENARRATIVE_TRACKING_OUTBOX_ENABLED=true
+GENARRATIVE_TRACKING_OUTBOX_DIR=/var/lib/genarrative/tracking-outbox
+GENARRATIVE_TRACKING_OUTBOX_BATCH_SIZE=500
+GENARRATIVE_TRACKING_OUTBOX_FLUSH_INTERVAL_MS=1000
+GENARRATIVE_TRACKING_OUTBOX_MAX_BYTES=268435456
+GENARRATIVE_OTEL_ENABLED=true
 OTEL_SERVICE_NAME=genarrative-api
 OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
 OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.namespace=genarrative
--- a/deploy/nginx/genarrative-dev-http.conf
+++ b/deploy/nginx/genarrative-dev-http.conf
@@ -14,6 +14,9 @@ upstream genarrative_api {
 }

 limit_conn_zone $binary_remote_addr zone=genarrative_api_conn:10m;
+limit_req_zone $binary_remote_addr zone=genarrative_gallery_rps:10m rate=5000r/s;
+limit_req_zone $binary_remote_addr zone=genarrative_api_rps:10m rate=300r/s;
+limit_req_zone $binary_remote_addr zone=genarrative_admin_rps:10m rate=30r/s;

 server {
    listen 80;
@@ -22,6 +25,8 @@ server {
    error_log /var/log/nginx/genarrative.error.log warn;
    limit_conn_status 429;
    limit_conn_log_level warn;
+    limit_req_status 429;
+    limit_req_log_level warn;

    gzip on;
    gzip_vary on;
@@ -48,6 +53,7 @@ server {
    location ^~ /admin/api/ {
        default_type application/json;
        limit_conn genarrative_api_conn 64;
+        limit_req zone=genarrative_admin_rps burst=16 nodelay;

        if ($genarrative_maintenance) {
            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
@@ -85,10 +91,107 @@ server {
        try_files $uri =404;
    }

+    location = /api/runtime/puzzle/gallery {
+        default_type application/json;
+        limit_conn genarrative_api_conn 320;
+        limit_req zone=genarrative_gallery_rps burst=4096 nodelay;
+
+        if ($genarrative_maintenance) {
+            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
+        }
+
+        proxy_pass http://genarrative_api;
+        proxy_http_version 1.1;
+        proxy_buffering off;
+        proxy_read_timeout 3600s;
+        proxy_send_timeout 3600s;
+        add_header X-Accel-Buffering no always;
+        proxy_set_header Connection "";
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $host;
+        proxy_set_header X-Request-Id $request_id;
+    }
+
+    location = /api/runtime/custom-world-gallery {
+        default_type application/json;
+        limit_conn genarrative_api_conn 320;
+        limit_req zone=genarrative_gallery_rps burst=4096 nodelay;
+
+        if ($genarrative_maintenance) {
+            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
+        }
+
+        proxy_pass http://genarrative_api;
+        proxy_http_version 1.1;
+        proxy_buffering off;
+        proxy_read_timeout 3600s;
+        proxy_send_timeout 3600s;
+        add_header X-Accel-Buffering no always;
+        proxy_set_header Connection "";
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $host;
+        proxy_set_header X-Request-Id $request_id;
+    }
+
+    location ~ ^/api/runtime/puzzle/gallery/[^/]+$ {
+        default_type application/json;
+        limit_conn genarrative_api_conn 32;
+        limit_req zone=genarrative_api_rps burst=32 nodelay;
+
+        if ($genarrative_maintenance) {
+            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
+        }
+
+        proxy_pass http://genarrative_api;
+        proxy_http_version 1.1;
+        proxy_buffering off;
+        proxy_read_timeout 3600s;
+        proxy_send_timeout 3600s;
+        add_header X-Accel-Buffering no always;
+        proxy_set_header Connection "";
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $host;
+        proxy_set_header X-Request-Id $request_id;
+    }
+
+    location ~ ^/api/runtime/custom-world-gallery/[^/]+/[^/]+$ {
+        default_type application/json;
+        limit_conn genarrative_api_conn 32;
+        limit_req zone=genarrative_api_rps burst=32 nodelay;
+
+        if ($genarrative_maintenance) {
+            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
+        }
+
+        proxy_pass http://genarrative_api;
+        proxy_http_version 1.1;
+        proxy_buffering off;
+        proxy_read_timeout 3600s;
+        proxy_send_timeout 3600s;
+        add_header X-Accel-Buffering no always;
+        proxy_set_header Connection "";
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $host;
+        proxy_set_header X-Request-Id $request_id;
+    }
+
    # 临时兼容主站仍在使用的 /api/* HTTP facade；前端完成 SpacetimeDB SDK 迁移后删除。
    location ~ ^/api(?:/|$) {
        default_type application/json;
        limit_conn genarrative_api_conn 64;
+        limit_req zone=genarrative_api_rps burst=64 nodelay;

        if ($genarrative_maintenance) {
            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
--- a/deploy/nginx/genarrative.conf
+++ b/deploy/nginx/genarrative.conf
@@ -12,6 +12,9 @@ upstream genarrative_api {
 }

 limit_conn_zone $binary_remote_addr zone=genarrative_api_conn:10m;
+limit_req_zone $binary_remote_addr zone=genarrative_gallery_rps:10m rate=5000r/s;
+limit_req_zone $binary_remote_addr zone=genarrative_api_rps:10m rate=300r/s;
+limit_req_zone $binary_remote_addr zone=genarrative_admin_rps:10m rate=30r/s;

 server {
    listen 80;
@@ -20,6 +23,8 @@ server {
    error_log /var/log/nginx/genarrative.error.log warn;
    limit_conn_status 429;
    limit_conn_log_level warn;
+    limit_req_status 429;
+    limit_req_log_level warn;

    location /.well-known/acme-challenge/ {
        root /var/www/html;
@@ -35,6 +40,10 @@ server {
    server_name genarrative.example.com;
    access_log /var/log/nginx/genarrative.access.log genarrative_upstream;
    error_log /var/log/nginx/genarrative.error.log warn;
+    limit_conn_status 429;
+    limit_conn_log_level warn;
+    limit_req_status 429;
+    limit_req_log_level warn;

    gzip on;
    gzip_vary on;
@@ -64,6 +73,7 @@ server {
    location ^~ /admin/api/ {
        default_type application/json;
        limit_conn genarrative_api_conn 64;
+        limit_req zone=genarrative_admin_rps burst=16 nodelay;

        if ($genarrative_maintenance) {
            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
@@ -101,10 +111,107 @@ server {
        try_files $uri =404;
    }

+    location = /api/runtime/puzzle/gallery {
+        default_type application/json;
+        limit_conn genarrative_api_conn 320;
+        limit_req zone=genarrative_gallery_rps burst=4096 nodelay;
+
+        if ($genarrative_maintenance) {
+            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
+        }
+
+        proxy_pass http://genarrative_api;
+        proxy_http_version 1.1;
+        proxy_buffering off;
+        proxy_read_timeout 3600s;
+        proxy_send_timeout 3600s;
+        add_header X-Accel-Buffering no always;
+        proxy_set_header Connection "";
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $host;
+        proxy_set_header X-Request-Id $request_id;
+    }
+
+    location = /api/runtime/custom-world-gallery {
+        default_type application/json;
+        limit_conn genarrative_api_conn 320;
+        limit_req zone=genarrative_gallery_rps burst=4096 nodelay;
+
+        if ($genarrative_maintenance) {
+            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
+        }
+
+        proxy_pass http://genarrative_api;
+        proxy_http_version 1.1;
+        proxy_buffering off;
+        proxy_read_timeout 3600s;
+        proxy_send_timeout 3600s;
+        add_header X-Accel-Buffering no always;
+        proxy_set_header Connection "";
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $host;
+        proxy_set_header X-Request-Id $request_id;
+    }
+
+    location ~ ^/api/runtime/puzzle/gallery/[^/]+$ {
+        default_type application/json;
+        limit_conn genarrative_api_conn 32;
+        limit_req zone=genarrative_api_rps burst=32 nodelay;
+
+        if ($genarrative_maintenance) {
+            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
+        }
+
+        proxy_pass http://genarrative_api;
+        proxy_http_version 1.1;
+        proxy_buffering off;
+        proxy_read_timeout 3600s;
+        proxy_send_timeout 3600s;
+        add_header X-Accel-Buffering no always;
+        proxy_set_header Connection "";
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $host;
+        proxy_set_header X-Request-Id $request_id;
+    }
+
+    location ~ ^/api/runtime/custom-world-gallery/[^/]+/[^/]+$ {
+        default_type application/json;
+        limit_conn genarrative_api_conn 32;
+        limit_req zone=genarrative_api_rps burst=32 nodelay;
+
+        if ($genarrative_maintenance) {
+            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
+        }
+
+        proxy_pass http://genarrative_api;
+        proxy_http_version 1.1;
+        proxy_buffering off;
+        proxy_read_timeout 3600s;
+        proxy_send_timeout 3600s;
+        add_header X-Accel-Buffering no always;
+        proxy_set_header Connection "";
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host $host;
+        proxy_set_header X-Request-Id $request_id;
+    }
+
    # 临时兼容主站仍在使用的 /api/* HTTP facade；前端完成 SpacetimeDB SDK 迁移后删除。
    location ~ ^/api(?:/|$) {
        default_type application/json;
        limit_conn genarrative_api_conn 64;
+        limit_req zone=genarrative_api_rps burst=64 nodelay;

        if ($genarrative_maintenance) {
            return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
--- a/deploy/otelcol/genarrative-debug.yaml
+++ b/deploy/otelcol/genarrative-debug.yaml
@@ -0,0 +1,23 @@
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 127.0.0.1:4317
+      http:
+        endpoint: 127.0.0.1:4318
+
+exporters:
+  debug:
+    verbosity: normal
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      exporters: [debug]
+    metrics:
+      receivers: [otlp]
+      exporters: [debug]
+    logs:
+      receivers: [otlp]
+      exporters: [debug]
--- a/deploy/systemd/otelcol-contrib.service
+++ b/deploy/systemd/otelcol-contrib.service
@@ -0,0 +1,22 @@
+[Unit]
+Description=Genarrative OpenTelemetry Collector Contrib
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=otelcol
+Group=otelcol
+WorkingDirectory=/etc/otelcol
+ExecStart=/usr/local/bin/otelcol-contrib --config=/etc/otelcol/genarrative-debug.yaml
+Restart=always
+RestartSec=5
+LimitNOFILE=65535
+
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=full
+ReadWritePaths=/etc/otelcol /var/log/genarrative
+
+[Install]
+WantedBy=multi-user.target