feat(deploy): prepare offline provision tools and container loadtest

2026-05-18 16:58:48 +08:00
parent 4f6c97ae92
commit 3eb292b403
12 changed files with 443 additions and 61 deletions
--- a/deploy/container/README.md
+++ b/deploy/container/README.md
@@ -6,20 +6,27 @@

 ```text
 Docker Compose
+├─ spacetimedb  :3101，独立数据卷，供 api-server 连接
 ├─ nginx        :80 -> api-server:8082，负责静态站点、/admin/、/api/ 反代、upstream timing log、连接限制
-├─ api-server   :8082，Linux release 构建，连接外部 SpacetimeDB
+├─ api-server   :8082，Linux release 构建，连接 compose 内 SpacetimeDB
 ├─ otelcol      :4317/4318，debug exporter，接收 traces / metrics / logs
 └─ k6           profile=loadtest 时临时启动，在 compose 网络内压 nginx
 ```

+当前容器模拟参数按 `genarrative-release` 服务器采样值收口为 2 vCPU / 2 GiB RAM / 4096 soft nofile / 768 worker_connections，并已在 compose 里落实到 `spacetimedb cpus=1.0 mem_limit=768m`、`api-server cpus=2.0 mem_limit=1g`、`nginx cpus=0.25 mem_limit=128m`、`otelcol cpus=0.25 mem_limit=128m`、`k6 cpus=0.5 mem_limit=512m`。
+Collector 镜像使用 `otel/opentelemetry-collector-contrib:0.151.0`。
+生产服务器若启用 Collector，则由 `deploy/systemd/otelcol-contrib.service` 和 `deploy/otelcol/genarrative-debug.yaml` 托管，不走容器镜像。
+
 默认 host 端口：

+- `http://127.0.0.1:13101`：容器 SpacetimeDB。
 - `http://127.0.0.1:18080`：容器 Nginx。
 - `127.0.0.1:4317` / `127.0.0.1:4318`：容器 Collector OTLP gRPC / HTTP。

 如端口冲突，可设置：

 ```powershell
+$env:GENARRATIVE_CONTAINER_SPACETIME_PORT="13102"
 $env:GENARRATIVE_CONTAINER_HTTP_PORT="18081"
 $env:GENARRATIVE_CONTAINER_OTLP_HTTP_PORT="14318"
 $env:GENARRATIVE_CONTAINER_OTLP_GRPC_PORT="14317"
@@ -33,21 +40,25 @@ npm run container:init

 该命令会从 `deploy/container/api-server.env.example` 生成本地 `deploy/container/api-server.env`。真实 token、库名和外部服务密钥只写本地 env 文件，不提交 Git。

-Docker Desktop 下默认通过 `host.docker.internal:3101` 连接宿主机上 `npm run dev` 启动的 SpacetimeDB：
+Docker Desktop 下默认通过 `http://spacetimedb:3101` 连接 compose 内 SpacetimeDB；宿主机只负责用 CLI 发布模块：

 ```env
-GENARRATIVE_SPACETIME_SERVER_URL=http://host.docker.internal:3101
+GENARRATIVE_SPACETIME_SERVER_URL=http://spacetimedb:3101
 GENARRATIVE_SPACETIME_DATABASE=genarrative-loadtest
 GENARRATIVE_SPACETIME_TOKEN=
 ```

-Linux Docker Engine 如果不能解析 `host.docker.internal`，Compose 已配置 `host-gateway`；仍不通时把 `GENARRATIVE_SPACETIME_SERVER_URL` 改成宿主机网关 IP 或同网络内的 SpacetimeDB 地址。
+宿主机发布模块时，先用 CLI 向 `http://127.0.0.1:13101` 发布到 `genarrative-loadtest`，再启动 `npm run container:up`。
+
+Linux Docker Engine 若要从宿主机 CLI 连到容器内服务，直接用 `http://127.0.0.1:13101`；容器内部服务之间统一走 `http://spacetimedb:3101`。

 ## 启动与验证

 ```bash
 npm run container:config
 npm run container:build
+npm run container:up -- spacetimedb
+spacetime publish genarrative-loadtest --server http://127.0.0.1:13101 --module-path server-rs/crates/spacetime-module --yes --build-options="--debug"
 npm run container:up
 npm run container:ps
 curl -sS http://127.0.0.1:18080/api/runtime/puzzle/gallery
@@ -103,6 +114,17 @@ $env:DETAIL_RATIO="0"
 npm run container:k6
 ```

+容器内 `api-server` 资源上限与 Nginx 连接模型已经按 `genarrative-release` 的 2C / 2G / `nofile=4096` / `worker_connections=768` 收口；如果你要改成别的机器，就先重新采样再改这里。
+
+SpacetimeDB 容器默认只提供运行时，不自动发布模块。首次启动或清理 `spacetime-data` 卷后，先只启动 `spacetimedb` 服务，再发布模块：
+
+```bash
+npm run container:up -- spacetimedb
+spacetime publish genarrative-loadtest --server http://127.0.0.1:13101 --module-path server-rs/crates/spacetime-module --yes --build-options="--debug"
+```
+
+发布完成后再执行 `npm run container:up` 和 `npm run container:k6`。如果 `deploy/container/api-server.env` 里的 `GENARRATIVE_SPACETIME_DATABASE` 改成了别的库名，发布命令里的库名也要同步修改。
+
 如果要压 1000 HTTP req/s，把 `PEAK_RPS` 调到 `500`；如果要压 5000 HTTP req/s，把 `PEAK_RPS` 调到 `2500`，并同时提高 `PREALLOCATED_VUS` / `MAX_VUS`，观察是否先被带宽、Nginx `limit_conn` 或 api-server 背压限制。

 ## OTLP
--- a/deploy/container/api-server.env.example
+++ b/deploy/container/api-server.env.example
@@ -7,7 +7,7 @@ GENARRATIVE_API_HOST=0.0.0.0
 GENARRATIVE_API_PORT=8082
 GENARRATIVE_API_LOG=info,tower_http=info
 GENARRATIVE_API_LISTEN_BACKLOG=1024
-GENARRATIVE_API_WORKER_THREADS=4
+GENARRATIVE_API_WORKER_THREADS=2
 GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512

 GENARRATIVE_OTEL_ENABLED=false
@@ -21,9 +21,8 @@ GENARRATIVE_JWT_SECRET=CHANGE_ME_FOR_CONTAINER
 AUTH_REFRESH_COOKIE_SECURE=false
 GENARRATIVE_AUTH_STORE_PATH=/var/lib/genarrative/auth/auth-store.json

-# Docker Desktop 下连接宿主机 npm run dev 启动的 SpacetimeDB。
-# Linux Docker Engine 可改成宿主机网关 IP，或在 compose 里接入同一网络内的 SpacetimeDB。
-GENARRATIVE_SPACETIME_SERVER_URL=http://host.docker.internal:3101
+# 默认连接 compose 内部 SpacetimeDB；宿主机发布模块使用 127.0.0.1:13101。
+GENARRATIVE_SPACETIME_SERVER_URL=http://spacetimedb:3101
 GENARRATIVE_SPACETIME_DATABASE=genarrative-loadtest
 GENARRATIVE_SPACETIME_TOKEN=
 GENARRATIVE_SPACETIME_POOL_SIZE=8
--- a/deploy/container/docker-compose.loadtest.yml
+++ b/deploy/container/docker-compose.loadtest.yml
@@ -1,11 +1,47 @@
 name: genarrative-container-loadtest

 services:
+  spacetimedb:
+    image: clockworklabs/spacetime:v2.2.0
+    command:
+      [
+        "start",
+        "--listen-addr",
+        "0.0.0.0:3101",
+        "--data-dir",
+        "/var/lib/spacetimedb",
+        "--page_pool_max_size",
+        "536870912",
+        "--non-interactive",
+      ]
+    cpus: "1.0"
+    mem_limit: 768m
+    ports:
+      - "${GENARRATIVE_CONTAINER_SPACETIME_PORT:-13101}:3101"
+    volumes:
+      - spacetime-data:/var/lib/spacetimedb
+    ulimits:
+      nofile:
+        soft: 4096
+        hard: 4096
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          "spacetime server ping http://127.0.0.1:3101 >/dev/null 2>&1",
+        ]
+      interval: 10s
+      timeout: 5s
+      retries: 12
+      start_period: 20s
+
  api-server:
    build:
      context: ../..
      dockerfile: deploy/container/api-server.Dockerfile
      target: api-runtime
+    cpus: "2.0"
+    mem_limit: 1g
    env_file:
      - ./api-server.env
    environment:
@@ -16,7 +52,13 @@ services:
      - "host.docker.internal:host-gateway"
    volumes:
      - api-auth-store:/var/lib/genarrative/auth
+    ulimits:
+      nofile:
+        soft: 4096
+        hard: 4096
    depends_on:
+      spacetimedb:
+        condition: service_healthy
      otelcol:
        condition: service_started
    healthcheck:
@@ -31,15 +73,23 @@ services:
      context: ../..
      dockerfile: deploy/container/api-server.Dockerfile
      target: nginx-runtime
+    cpus: "0.25"
+    mem_limit: 128m
    depends_on:
      api-server:
        condition: service_healthy
+      spacetimedb:
+        condition: service_healthy
    ports:
      - "${GENARRATIVE_CONTAINER_HTTP_PORT:-18080}:80"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - nginx-logs:/var/log/nginx
+    ulimits:
+      nofile:
+        soft: 4096
+        hard: 4096
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1/api/runtime/puzzle/gallery"]
      interval: 10s
@@ -48,8 +98,10 @@ services:
      start_period: 20s

  otelcol:
-    image: otel/opentelemetry-collector-contrib:0.125.0
+    image: otel/opentelemetry-collector-contrib:0.151.0
    command: ["--config=/etc/otelcol/config.yaml"]
+    cpus: "0.25"
+    mem_limit: 128m
    volumes:
      - ./otelcol.yaml:/etc/otelcol/config.yaml:ro
    ports:
@@ -59,6 +111,8 @@ services:
  k6:
    image: grafana/k6:0.52.0
    profiles: ["loadtest"]
+    cpus: "0.5"
+    mem_limit: 512m
    depends_on:
      nginx:
        condition: service_healthy
@@ -81,5 +135,6 @@ services:
    command: ["run", "k6-works-list.js"]

 volumes:
+  spacetime-data:
  api-auth-store:
  nginx-logs:
--- a/deploy/container/nginx.conf
+++ b/deploy/container/nginx.conf
@@ -1,7 +1,7 @@
 worker_processes auto;

 events {
-    worker_connections 4096;
+    worker_connections 768;
 }

 http {
@@ -106,7 +106,7 @@ http {
        }

        location ~ ^/v1/database/[^/]+/subscribe$ {
-            proxy_pass http://host.docker.internal:3101;
+            proxy_pass http://spacetimedb:3101;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
@@ -115,7 +115,7 @@ http {
        }

        location ^~ /v1/identity {
-            proxy_pass http://host.docker.internal:3101;
+            proxy_pass http://spacetimedb:3101;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
--- a/deploy/otelcol/genarrative-debug.yaml
+++ b/deploy/otelcol/genarrative-debug.yaml
@@ -0,0 +1,23 @@
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 127.0.0.1:4317
+      http:
+        endpoint: 127.0.0.1:4318
+
+exporters:
+  debug:
+    verbosity: normal
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      exporters: [debug]
+    metrics:
+      receivers: [otlp]
+      exporters: [debug]
+    logs:
+      receivers: [otlp]
+      exporters: [debug]
--- a/deploy/systemd/otelcol-contrib.service
+++ b/deploy/systemd/otelcol-contrib.service
@@ -0,0 +1,22 @@
+[Unit]
+Description=Genarrative OpenTelemetry Collector Contrib
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=otelcol
+Group=otelcol
+WorkingDirectory=/etc/otelcol
+ExecStart=/usr/local/bin/otelcol-contrib --config=/etc/otelcol/genarrative-debug.yaml
+Restart=always
+RestartSec=5
+LimitNOFILE=65535
+
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=full
+ReadWritePaths=/etc/otelcol /var/log/genarrative
+
+[Install]
+WantedBy=multi-user.target