diff --git a/.hermes/shared-memory/decision-log.md b/.hermes/shared-memory/decision-log.md index 6ddc9f83..30602617 100644 --- a/.hermes/shared-memory/decision-log.md +++ b/.hermes/shared-memory/decision-log.md @@ -322,6 +322,7 @@ ## 2026-05-18 Windows Jenkins PowerShell 统一改为显式 powershell.exe 启动 +- 后续更新：该决策仅适用于历史 Windows Jenkins 节点；当前 `Genarrative-Stdb-Module-Build` 已改为 Linux agent，实际执行路径不再依赖该口径。 - 背景：`Genarrative-Stdb-Module-Build` 在 Windows Jenkins 本地环境里调用裸 `powershell` step 时触发 `CreateProcess error=5, 拒绝访问`，而 `powershell.exe` 本体与 workspace ACL 都正常。 - 决策：Windows Jenkins 上凡是需要执行 PowerShell 逻辑的流水线，优先通过 `bat` 显式调用 `%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -File ...`，不要再依赖 Jenkins `powershell` step 的隐式启动器。 - 追加决策：`Genarrative-Stdb-Module-Build` 的 Checkout 逻辑应复用 Jenkins GitSCM 已完成的工作区状态。`COMMIT_HASH` 为空或已与当前 `HEAD` 一致时，不再额外执行 `git clean` / `git checkout`；只有需要切到指定且不同的 commit 时才补 fetch、校验和切换，避免在 Windows workspace 里二次清理触发权限拒绝。 @@ -366,6 +367,7 @@ ## 2026-05-19 生产 provision 改为 Windows 下载包后由目标机本地安装 +- 后续更新：该口径已被 `2026-06-01 生产 Jenkins 流水线统一改为 Linux 优先并先查 localhost` 取代；当前 `Genarrative-Server-Provision` 不再走 Windows 下载阶段，而是在 Linux build 节点直接准备 `provision-tools/`。 - 背景：当前 `development` provision 目标实际就是 Linux agent `genarrative-build-01`，之前把 `Prepare Provision Tools` 放在 `linux && genarrative-build` 会让目标机自己连 GitHub 和 `install.spacetimedb.com`，违背“Windows 本机先下载再传到目标机”的运维要求。 - 决策：`Genarrative-Server-Provision` 拆成 Windows 下载阶段和 Linux 目标机安装阶段。Windows 节点的 `Download Provision Tool Archives` 只下载 `spacetime-x86_64-unknown-linux-gnu.tar.gz` 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz`，通过 `stash/unstash` 传到目标 Linux 节点；目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，只消费已下载件生成 `provision-tools/`，缺包直接失败，不回退外网下载。 - 追加决策：Server-Provision 的 Windows helper 不再对 Jenkins `writeFile` 刚写出的 `.ps1` 做原地 UTF-8 BOM 重写，而是由显式 `powershell.exe` 按 UTF-8 读入脚本文本，并用 `ScriptBlock::Create(...)` 在内存中执行；这样既保留中文脚本内容，又避免同一个 workspace 脚本被立即重写时触发 `拒绝访问`。 @@ -1013,12 +1015,21 @@ ## 2026-05-19 server provision 下载件固定由 Windows 节点断点续传 +- 后续更新：该口径已被 `2026-06-01 生产 Jenkins 流水线统一改为 Linux 优先并先查 localhost` 取代；当前不再维护 Windows 下载阶段和 `.download` 断点续传 helper。 - 背景：`SpacetimeDB` 和 `otelcol-contrib` release 资产在 Linux 目标机直接下载很慢；改到 Windows Jenkins 节点下载后，GitHub 大文件仍可能出现 `curl: (18)` 响应体截断。 - 决策：`Genarrative-Server-Provision` 的 `Download Provision Tool Archives` 阶段继续只在 Windows 节点下载，再通过 `stash/unstash` 交给目标 Linux agent；下载前查 GitHub release asset `digest`，本地最终文件 SHA256 命中即跳过，`.download` 临时文件用于 `curl -C -` 断点续传，完整返回但 digest 不匹配才清理重下。 - 影响范围：`jenkins/Jenkinsfile.production-server-provision`、目标机 `scripts/prepare-server-provision-tools.sh` 的本地下载件消费路径、生产 provision 运维排障。 - 验证方式：Windows 下载日志应出现 digest 查询、已存在校验跳过或 `curl 断点续传`；Linux 目标机阶段只使用 `provision-tool-downloads/` 中的 tarball，不访问 GitHub 下载地址。 - 关联文档：`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 +## 2026-06-01 生产 Jenkins 流水线统一改为 Linux 优先并先查 localhost + +- 背景：生产流水线长期混用 Windows、Linux 和公网 Git 入口，导致构建 / 发布 / provision 的 checkout 口径分叉；同时 `Genarrative-Server-Provision` 还残留过 Windows 下载 helper，和当前 Linux 构建 / 发布部署路径不一致。 +- 决策：生产 Jenkins 流水线统一把执行节点收口到 Linux label，`Pipeline script from SCM` 仍保留公网域名，但所有生产流水线首次 `GitSCM checkout` 先尝试 `http://127.0.0.1:3000/GenarrativeAI/Genarrative.git`，失败后再回退到 `https://git.genarrative.world/GenarrativeAI/Genarrative.git`；`Genarrative-Stdb-Module-Build`、`Genarrative-Server-Provision`、`Genarrative-Notify-Email` 也都切到 Linux 节点。`Genarrative-Server-Provision` 的工具准备不再依赖 Windows helper，而是在 Linux build 节点直接生成 `provision-tools/` 后交给后续 Linux 发布阶段。 +- 影响范围：`jenkins/Jenkinsfile.production-*`、`scripts/jenkins-checkout-source.sh`、`scripts/prepare-server-provision-tools.sh`、生产运维文档。 +- 验证方式：扫描 Jenkinsfile 时应看到 `linux && genarrative-*` 节点和 localhost-first checkout 口径；`Genarrative-Server-Provision` 日志不再出现 Windows 相关 helper 输出，工具准备阶段应直接生成 `provision-tools/`。 +- 关联文档：`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 + ## 个人任务与埋点首版边界冻结 - 背景：“我的”Tab、任务、奖励、钱包和埋点涉及用户、运营、分析多条链路，需要避免范围泛化。 diff --git a/.hermes/shared-memory/pitfalls.md b/.hermes/shared-memory/pitfalls.md index 90d4b5db..e6af2a55 100644 --- a/.hermes/shared-memory/pitfalls.md +++ b/.hermes/shared-memory/pitfalls.md @@ -307,6 +307,7 @@ ## Windows provision 下载截断要断点续传而不是回退目标机下载 +- 当前状态：已废弃。2026-06-01 起生产 Jenkins 流水线统一切到 Linux agent，`Genarrative-Server-Provision` 不再维护 Windows 下载阶段。 - 现象：`Genarrative-Server-Provision` 在 `Download Provision Tool Archives` 阶段出现 `curl: (18) end of response ... bytes missing`，常见于 `otelcol-contrib_0.151.0_linux_amd64.tar.gz` 等 GitHub release 大文件。 - 原因：这是 Windows Jenkins 节点到 GitHub 的响应体被截断；若每轮都删除 `.download` 临时文件，就会丢掉已下载部分，下一次又从头开始。 - 处理：Windows 下载函数保留 `${Output}.download`，`curl` 失败时下一轮使用 `-C -` 断点续传；最终只以 GitHub release asset 的 SHA256 `digest` 作为放行条件，完整返回但 digest 不匹配才删除临时文件重新下载。不要把 SpacetimeDB 或 `otelcol-contrib` 下载挪回 Linux 目标机。 @@ -1162,12 +1163,12 @@ - 验证：运行 `git ls-files --stage scripts/prepare-server-provision-tools.sh`，确认 mode 为 `100755`；重新跑 `Genarrative-Server-Provision` 时应进入工具下载/打包日志，而不是停在 `Permission denied`。 - 关联：`jenkins/Jenkinsfile.production-server-provision`、`scripts/prepare-server-provision-tools.sh`、`scripts/jenkins-checkout-source.sh`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 -## Server-Provision 下载阶段不要放回 genarrative-build-01 +## Server-Provision 工具准备只在 Linux build 节点做一次 -- 现象：`Genarrative-Server-Provision` 日志里 `Prepare Provision Tools` 显示 `Running on genarrative-build-01 in /root/...`，随后在该节点上下载 GitHub release 或 `install.spacetimedb.com` 失败。 -- 原因：`genarrative-build-01` 在当前 provision 流程里是 Linux 目标发布机/目标 agent，不是用户本地 Windows 下载环境；把下载阶段放在 `linux && genarrative-build` 等于让目标机自己外连。 -- 处理：下载必须发生在 Jenkins `windows` 节点的 `Download Provision Tool Archives` 阶段，先下载 SpacetimeDB Linux release tarball 和 `otelcol-contrib` Linux amd64 包，再 `stash/unstash` 到目标 Linux 节点。目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，缺少下载件直接失败，不回退联网下载。 -- 验证：Jenkins 日志应先出现 `Running on ... windows` 和 `[prepare-provision-downloads] 下载 ...`，目标节点只出现 `[prepare-provision-tools] 使用已下载的 ...`；如果目标节点出现 `下载 otelcol-contrib:` 或 `下载 SpacetimeDB 官方安装器脚本:`，说明又回退到错误路径。 +- 现象：`Genarrative-Server-Provision` 在后续目标发布节点重复执行 `scripts/prepare-server-provision-tools.sh`，或日志里出现目标节点继续访问 GitHub release / `install.spacetimedb.com`。 +- 原因：当前流水线已经改成 Linux build 节点一次性准备 `provision-tools/` 并 stash 给目标发布阶段；如果目标发布阶段又重新准备工具包，就会重复下载并把目标节点暴露到外网依赖。 +- 处理：只允许 `Prepare Provision Tools` 阶段在 `linux && genarrative-build` 节点生成 `provision-tools/`；后续 `Provision Server` 阶段只 `unstash 'server-provision-tools'` 并安装其中的 `spacetime` 与 `otelcol-contrib`，不要再运行 `prepare-server-provision-tools.sh`。 +- 验证：Jenkins 日志应先在 Linux build 节点出现 `[prepare-provision-tools] 工具包已准备`，后续目标发布节点只出现安装 / systemd / Nginx provision 日志；目标节点不应出现 `下载 otelcol-contrib:` 或 `下载 SpacetimeDB 官方安装器脚本:`。 - 关联：`jenkins/Jenkinsfile.production-server-provision`、`scripts/prepare-server-provision-tools.sh`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 ## 个人任务 scope 不得扩成 work/site/module @@ -1486,6 +1487,7 @@ ## Windows Jenkins `powershell` step 在 Stdb module 构建里曾触发 CreateProcess error=5 +- 当前状态：已废弃。`Genarrative-Stdb-Module-Build` 已切到 Linux agent，不再执行 Windows PowerShell 流程。 - 现象：`Genarrative-Stdb-Module-Build` 在 Windows Jenkins 节点上报 `java.io.IOException: Cannot run program "powershell" (in directory "C:\\Users\\DSK\\.jenkins-local\\workspace\\Genarrative-Stdb-Module-Build"): CreateProcess error=5, 拒绝访问。`；日志里能看到 `durable-task` 已写出 `powershellWrapper.ps1`，但在真正启动裸 `powershell` 子进程时失败。 - 原因：Jenkins durable-task 的 `powershell` step 依赖一个隐式命令解析/启动路径，在这台 Windows 本地 Jenkins 环境里会被拒绝。`powershell.exe` 本体和 workspace ACL 都是正常的，问题出在 Jenkins step 的启动方式，而不是 PowerShell 脚本内容。修复后若日志能打印 `[jenkins-powershell] exe:`，但随后仅报 `拒绝访问` / `script returned exit code 5`，通常已经不是 PowerShell 启动失败，而是 Checkout 脚本内部命令在 Windows workspace 里触发权限拒绝。若 `.jenkins-*.ps1` 里中文 `throw '[stdb-build] ...'` 报 `MissingArrayIndexExpression`，则是 Windows PowerShell 5.1 用 `-File` 解析无 BOM UTF-8 脚本时按本地 ANSI 误解码。 - 处理：把 `jenkins/Jenkinsfile.production-stdb-module-build` 的 `Checkout` 和 `Build Stdb Module` 两处 `powershell` step 收口成 `runWindowsPowerShell(...)` helper，先用 `writeFile` 写出临时 `.ps1`，再用显式 `powershell.exe` 把脚本重写成 UTF-8 with BOM，最后通过 `%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -File ...` 执行。这个 helper 写在 Groovy GString 里时，PowerShell 的 `$path` / `$text` / `$true` 必须写成 `\$path` / `\$text` / `\$true`，否则 Jenkinsfile 会在 Groovy 编译阶段报 `unexpected token: true`。Checkout 阶段优先复用 Jenkins GitSCM 已完成的工作区结果；`COMMIT_HASH` 为空或已经等于当前 `HEAD` 时不再重复 `git fetch` / `git checkout` / `git clean`，只有确实要切到另一个指定 commit 时才补 fetch、归属校验和 checkout。 diff --git a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md index 4e958977..2c601658 100644 --- a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md +++ b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md @@ -234,14 +234,14 @@ Jenkins 按 web / api / Spacetime module / build / deploy / publish 拆分 `Genarrative-Web-Build` 的主站构建失败若出现 Rollup 报错 `"xxx" is not exported by "src/services/publicWorkCode.ts"`，优先按前端公开作品号工具缺失处理，而不是排查 Jenkins 节点环境。修复时要让 `publicWorkCode.ts` 的 `buildPublicWorkCode` 与 `isSamePublicWorkCode` 成对导出，并补 `src/services/publicWorkCode.test.ts` 覆盖对应玩法前缀；随后用 `npm run build:production-release -- --component web --name <临时名>` 复现 Jenkins web 构建路径。 -生产 Jenkins 的 `Pipeline script from SCM` 由 Windows controller 读取 Jenkinsfile，SCM URL 继续使用 `https://git.genarrative.world/GenarrativeAI/Genarrative.git`。运行在 `linux && genarrative-build` 构建机上的 `Genarrative-Full-Build-And-Deploy` 源码解析阶段和 `Genarrative-Web-Build` checkout 阶段，优先使用 `http://127.0.0.1:3000/GenarrativeAI/Genarrative.git`，失败后回退到 `https://git.genarrative.world/GenarrativeAI/Genarrative.git`；两层 checkout 都必须保留单分支 refspec、`shallow=true`、`depth=1`、`noTags=true` 与 `honorRefspec=true`，后续二次源码确认继续走 `scripts/jenkins-checkout-source.sh`。 +生产 Jenkins 的 `Pipeline script from SCM` 仍由 Jenkins controller 读取 Jenkinsfile，SCM URL 继续使用 `https://git.genarrative.world/GenarrativeAI/Genarrative.git`。现在所有生产流水线 job 的首次 checkout 都先走 `http://127.0.0.1:3000/GenarrativeAI/Genarrative.git`，失败后回退到 `https://git.genarrative.world/GenarrativeAI/Genarrative.git`；两层 checkout 都必须保留单分支 refspec、`shallow=true`、`depth=1`、`noTags=true` 与 `honorRefspec=true`，后续二次源码确认继续走 `scripts/jenkins-checkout-source.sh`。 `Genarrative-Stdb-Module-Publish` 在 `Pipeline script from SCM` 阶段如果一开始就报 `No such DSL method 'pipeline'`，优先检查 `jenkins/Jenkinsfile.production-stdb-module-publish` 是否带 UTF-8 BOM。Jenkins Declarative Pipeline 的首个 token 必须是纯 `pipeline`；仓库中的 Jenkinsfile 应保存为 UTF-8 without BOM，只有临时写给 Windows PowerShell 5.1 `-File` 执行的 `.ps1` 才需要按对应 helper 转成带 BOM。验证时可检查文件前三字节不再是 `EF BB BF`，并运行 `validateDeclarativePipeline` 或重放该流水线。 -`Genarrative-Stdb-Module-Build` 或 SpacetimeDB module 构建失败若出现 Rust `E0425 cannot find function migrate_*`，优先排查 `server-rs/crates/spacetime-module/src/runtime/creation_entry_config.rs` 等同文件内默认种子迁移 helper 是否在分支合并时只保留了调用、漏掉了函数定义。修复时不要直接删除迁移调用；应恢复只纠偏历史默认种子且不覆盖后台手动配置的 helper，并用 `cargo check -p spacetime-module --manifest-path server-rs/Cargo.toml` 复现 Jenkins module 编译路径。 +`Genarrative-Stdb-Module-Build` 或 SpacetimeDB module 构建失败若出现 Rust `E0425 cannot find function migrate_*`，优先排查 `server-rs/crates/spacetime-module/src/runtime/creation_entry_config.rs` 等同文件内默认种子迁移 helper 是否在分支合并时只保留了调用、漏掉了函数定义。`Genarrative-Stdb-Module-Build` 现在运行在 `linux && genarrative-build` 节点上，Checkout 与 Build 都走 bash + cargo + sccache，不再依赖 Windows PowerShell 或 Git Bash。修复时不要直接删除迁移调用；应恢复只纠偏历史默认种子且不覆盖后台手动配置的 helper，并用 `cargo check -p spacetime-module --manifest-path server-rs/Cargo.toml` 复现 Jenkins module 编译路径。 -Windows Stdb module 构建流水线运行在 Jenkins `windows` 节点上。该流水线需要执行 PowerShell 逻辑时，统一通过 `bat` 显式调用 `%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe`，不要直接使用 Jenkins `powershell` step；本地 Jenkins durable-task 曾在 `Genarrative-Stdb-Module-Build` workspace 中启动裸 `powershell` 时触发 `CreateProcess error=5, 拒绝访问`。临时 `.ps1` 由 Jenkins `writeFile` 写出后要先转成 UTF-8 with BOM 再交给 Windows PowerShell 5.1 `-File` 解析，避免中文错误消息在无 BOM UTF-8 下被当成本地 ANSI 误解码。Checkout 阶段要优先复用 Jenkins GitSCM 已经完成的结果：`COMMIT_HASH` 为空或与当前 `HEAD` 一致时，不要再额外 `git clean` / `git checkout`，只在确实需要切到别的指定 commit 时才补 fetch、校验和切换。排查时先看对应 build log、`@tmp/durable-*` 下的 `powershellWrapper.ps1`，以及日志中的 `[jenkins-powershell] user/exe`。 +`Genarrative-Server-Provision` 现在也运行在 `linux && genarrative-build` / `linux && genarrative-release-deploy` 节点上，`Prepare Provision Tools` 会在 Linux build 节点直接准备 SpacetimeDB 与 `otelcol-contrib` 交付件，再 stash 给后续发布阶段；旧 Windows 下载 helper 已退役。`Genarrative-Stdb-Module-Build`、`Genarrative-Server-Provision` 和 `Genarrative-Notify-Email` 都不再需要单独的 Windows 节点口径。 生产环境变量模板：`deploy/env/api-server.env.example`。真实密钥只放服务器，不提交 Git，不写入文档示例。 @@ -252,9 +252,8 @@ Windows Stdb module 构建流水线运行在 Jenkins `windows` 节点上。该 - `api-server` 生产模板默认 `GENARRATIVE_API_LISTEN_BACKLOG=1024`、`GENARRATIVE_API_WORKER_THREADS=4`；本地未设置 worker threads 时继续使用 Tokio 默认值。 - `GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512` 开启应用内 HTTP 并发背压；`GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=320`、`GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS=64`、`GENARRATIVE_API_ADMIN_MAX_CONCURRENT_REQUESTS=16` 分别限制公开列表、公开详情和后台 API 热路径。超过许可时直接返回 `429 Too Many Requests` 和 `Retry-After: 1`，`/healthz` 不受该限制。这些值不是 RPS 限速；如果压测中 429 上升但内存和 p95 收敛，说明背压正在保护进程。直连 `api-server` 的极高 RPS 压测若出现 `connection refused`，通常已经打到 TCP 监听 / accept 层，应同时检查 backlog、Nginx upstream keepalive 和前置限流。 - `genarrative-api.service` 设置 `LimitNOFILE=65535`、`TasksMax=2048`；上线后用 `systemctl show genarrative-api.service -p LimitNOFILE -p TasksMax` 和 `cat /proc/$(pidof api-server)/limits` 核对。 -- Server provision 不在目标机联网下载 SpacetimeDB 或 `otelcol-contrib`。`Genarrative-Server-Provision` 先在 Windows Jenkins 节点执行 `Download Provision Tool Archives`，把 `spacetime-x86_64-unknown-linux-gnu.tar.gz` 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz` 先下载到工作区，再通过 `stash/unstash` 带到 `genarrative-build-01`；Windows 下载前会先查 GitHub release asset 的 `digest` 字段做 SHA256 校验，已有本地文件且 digest 一致就直接复用，不再重复下载。目标 Linux 节点上的 `scripts/prepare-server-provision-tools.sh` 只消费这些本地下载件生成 `provision-tools/`，再交给 `scripts/jenkins-server-provision.sh` 安装 `/stdb/spacetime`、`/stdb/bin/current/*` 和 `/usr/local/bin/otelcol-contrib`。Windows 侧的 `runWindowsPowerShell(...)` 现在会先 `writeFile` 生成 UTF-8 `.ps1`，再直接把脚本文本读入内存并通过 `ScriptBlock::Create(...)` 执行，避免对同一个 workspace 脚本做原地 BOM 重写。排查时除了看下载日志，还要看 `[jenkins-powershell] workspace:`、`[jenkins-powershell] script:` 和 `[jenkins-powershell] loaded bytes:`。注意 `scripts/jenkins-checkout-source.sh` 会执行 `git reset --hard` / `git clean`，因此被直接执行的新增脚本必须以 Git `100755` 模式提交，或在二次 checkout 之后再补 `chmod +x`。 -- Windows 下载阶段如果出现 `curl: (18)` 或响应体截断，流水线会保留同名 `.download` 临时文件并用 `curl -C -` 断点续传；只有完整返回但 SHA256 digest 仍不匹配时才删除临时文件后重新下载。目标 Linux 节点仍只接收 `stash/unstash` 带过去的本地下载件，不回退外网下载。 -- Windows 下载阶段如果走代理，在 `Genarrative-Server-Provision` 参数 `PROVISION_DOWNLOAD_PROXY` 填写 Windows Jenkins 节点可访问的 HTTP 代理，例如 `http://127.0.0.1:7890`；不要填写目标 release 机器视角的 `127.0.0.1`，除非代理确实运行在该 Windows 节点本机。Linux 目标机阶段会强制要求使用本地下载件，缺少文件直接失败，不再回退到外网下载。 +- Server provision 不再通过 Windows helper 下载。`Genarrative-Server-Provision` 的 `Prepare Provision Tools` 在 Linux build 节点直接准备 `spacetime-x86_64-unknown-linux-gnu.tar.gz` 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz`，再 stash `provision-tools/` 给后续发布阶段；如果 build 节点需要代理，在 `PROVISION_DOWNLOAD_PROXY` 配置 Linux 侧可访问的 HTTP 代理。后续 Linux 目标节点只消费 `provision-tools/`，不再回退到外网下载。 +- `Genarrative-Stdb-Module-Build`、`Genarrative-Web-Build`、`Genarrative-Api-Build`、`Genarrative-*Deploy`、`Genarrative-Database-Import/Export`、`Genarrative-Full-Build-And-Deploy` 和 `Genarrative-Notify-Email` 的生产流水线现都以 Linux agent 为主，统一走 `http://127.0.0.1:3000/GenarrativeAI/Genarrative.git` 优先、`https://git.genarrative.world/GenarrativeAI/Genarrative.git` 备用的 checkout 口径。 - `otelcol-contrib.service` 作为可选系统服务加入 provision，默认监听 `127.0.0.1:4317/4318` 并使用 `deploy/otelcol/genarrative-debug.yaml`。api-server 是否发送 OTLP 仍由 `GENARRATIVE_OTEL_ENABLED` 控制，服务 unit 见 `deploy/systemd/otelcol-contrib.service`。 - Nginx `/api/` 与 `/admin/api/` 通过 `genarrative_api` upstream 代理到 `127.0.0.1:8082`，upstream keepalive 为 64；`limit_conn` 负责连接 / 并发保护，`limit_req` 负责入口 RPS 快拒绝。当前模板把公开 gallery list 单独放到 `genarrative_gallery_rps`，默认 `rate=5000r/s`、`burst=4096`、`limit_conn=320`；公开详情和普通 API 放到 `genarrative_api_rps`，后台 API 放到 `genarrative_admin_rps`。通用 `/api` location 设置 `client_max_body_size 64m` 是反代兜底，防止拼图入口页 / 新增关卡本地参考图 Data URL 或旧兼容请求在到达 `api-server` 前被默认 1 MiB 上限拦截；拼图本地参考图前后端统一限制 6MB，历史图片仍提交 `referenceImageAssetObjectId(s)`。若线上出现 `413 Request Entity Too Large` 且 access log 中 `request_time=0.000`、`upstream_status=-`，说明请求在 Nginx 层被拦截，先用 `nginx -T | grep client_max_body_size` 检查 release 模板是否已渲染并 reload，同时检查前端是否超出 6MB 或错误提交了未压缩大图。`limit_conn_status 429` 和 `limit_req_status 429` 必须在 HTTP 与 HTTPS server 中同时生效；若线上压测看到 `limiting connections by zone "genarrative_api_conn"` 却返回 503，优先检查 `nginx -T` 里 HTTPS server 是否缺少这些状态码，以及 `/api/runtime/puzzle/gallery` 是否误落到通用 `location ~ ^/api` 的 `limit_conn=64`。压测时看 `/var/log/nginx/genarrative.access.log` 中的 `request_time`、`upstream_connect_time`、`upstream_header_time`、`upstream_response_time`、`upstream_status`、`request_id`。 - 作品列表 K6 脚本一次 iteration 默认请求两个公开接口，因此约 50 HTTP req/s 的目标命令使用 `SCENARIO=spike START_RPS=5 PEAK_RPS=25 HOLD=60s END_RPS=5 DETAIL_RATIO=0 npm run loadtest:k6:works`。 @@ -272,7 +271,7 @@ npm run container:k6 npm run container:down ``` -容器方案默认暴露 `http://127.0.0.1:18080`，`api-server` 在容器内监听 `0.0.0.0:8082`，Nginx 通过 `api-server:8082` upstream 反代 `/api/` 和 `/admin/api/`。SpacetimeDB 也纳入 compose，容器内由 `spacetimedb:3101` 提供服务，宿主机通过 `http://127.0.0.1:13101` 进行模块发布；Collector 镜像使用 `otel/opentelemetry-collector-contrib:0.151.0`。生产 provision 侧则通过 Windows Jenkins 下载件在目标 Linux 节点生成 `provision-tools/otelcol-contrib`，再安装本机 `otelcol-contrib.service`，真实库名、token 和外部服务密钥只写本地 `deploy/container/api-server.env`，不提交 Git。完整拓扑、端口、k6 参数和 OTLP debug exporter 使用方法见 `deploy/container/README.md`。 +容器方案默认暴露 `http://127.0.0.1:18080`，`api-server` 在容器内监听 `0.0.0.0:8082`，Nginx 通过 `api-server:8082` upstream 反代 `/api/` 和 `/admin/api/`。SpacetimeDB 也纳入 compose，容器内由 `spacetimedb:3101` 提供服务，宿主机通过 `http://127.0.0.1:13101` 进行模块发布；Collector 镜像使用 `otel/opentelemetry-collector-contrib:0.151.0`。生产 provision 侧现在由 Linux build 节点直接准备 `provision-tools/otelcol-contrib`，再交给后续 Linux 发布阶段安装本机 `otelcol-contrib.service`，真实库名、token 和外部服务密钥只写本地 `deploy/container/api-server.env`，不提交 Git。完整拓扑、端口、k6 参数和 OTLP debug exporter 使用方法见 `deploy/container/README.md`。 `npm run container:config` 默认只做 quiet 校验，避免把本地 env 中的 token 展开到终端；确需排查完整 compose 时再传 `-- --print`。 OpenTelemetry 现阶段默认开启 OTLP traces / metrics / logs，但本地日志与 Nginx 文件日志仍保留： diff --git a/jenkins/Jenkinsfile.production-api-build b/jenkins/Jenkinsfile.production-api-build index f278b3bd..43294be0 100644 --- a/jenkins/Jenkinsfile.production-api-build +++ b/jenkins/Jenkinsfile.production-api-build @@ -10,7 +10,8 @@ pipeline { } environment { - GIT_REMOTE_URL = 'https://git.genarrative.world/GenarrativeAI/Genarrative.git' + GIT_REMOTE_URL = 'http://127.0.0.1:3000/GenarrativeAI/Genarrative.git' + GIT_REMOTE_FALLBACK_URL = 'https://git.genarrative.world/GenarrativeAI/Genarrative.git' CARGO_HOME = '/home/dsk/.cache/genarrative-jenkins/api-server/cargo-home' CARGO_TARGET_DIR = '/home/dsk/.cache/genarrative-jenkins/api-server/cargo-target/prod-release' CARGO_INCREMENTAL = '0' @@ -33,23 +34,36 @@ pipeline { stages { stage('Checkout') { steps { - checkout([ - $class: 'GitSCM', - branches: [[name: "*/${params.SOURCE_BRANCH}"]], - doGenerateSubmoduleConfigurations: false, - extensions: [ - [$class: 'CleanBeforeCheckout'], - [$class: 'CloneOption', shallow: true, depth: 1, noTags: true, timeout: 30, honorRefspec: true], - ], - userRemoteConfigs: [[url: "${GIT_REMOTE_URL}", refspec: "+refs/heads/${params.SOURCE_BRANCH}:refs/remotes/origin/${params.SOURCE_BRANCH}"]], - ]) + script { + def checkoutFromRemote = { String remoteUrl -> + checkout([ + $class: 'GitSCM', + branches: [[name: "*/${params.SOURCE_BRANCH}"]], + doGenerateSubmoduleConfigurations: false, + extensions: [ + [$class: 'CleanBeforeCheckout'], + [$class: 'CloneOption', shallow: true, depth: 1, noTags: true, timeout: 30, honorRefspec: true], + ], + userRemoteConfigs: [[url: remoteUrl, refspec: "+refs/heads/${params.SOURCE_BRANCH}:refs/remotes/origin/${params.SOURCE_BRANCH}"]], + ]) + } + try { + checkoutFromRemote(env.GIT_REMOTE_URL) + env.EFFECTIVE_GIT_REMOTE_URL = env.GIT_REMOTE_URL + } catch (error) { + echo "Git 主地址拉取失败: ${env.GIT_REMOTE_URL}，改用备用地址: ${env.GIT_REMOTE_FALLBACK_URL}" + checkoutFromRemote(env.GIT_REMOTE_FALLBACK_URL) + env.EFFECTIVE_GIT_REMOTE_URL = env.GIT_REMOTE_FALLBACK_URL + } + } sh ''' bash -lc ' set -euo pipefail chmod +x scripts/jenkins-checkout-source.sh SOURCE_BRANCH="${SOURCE_BRANCH:-master}" \ COMMIT_HASH="${COMMIT_HASH:-}" \ - GIT_REMOTE_URL="${GIT_REMOTE_URL}" \ + GIT_REMOTE_URL="${EFFECTIVE_GIT_REMOTE_URL:-${GIT_REMOTE_URL}}" \ + GIT_REMOTE_FALLBACK_URL="${GIT_REMOTE_FALLBACK_URL:-}" \ SOURCE_COMMIT_FILE=".jenkins-source-commit" \ scripts/jenkins-checkout-source.sh ' diff --git a/jenkins/Jenkinsfile.production-notify-email b/jenkins/Jenkinsfile.production-notify-email index 61f5a969..38936efe 100644 --- a/jenkins/Jenkinsfile.production-notify-email +++ b/jenkins/Jenkinsfile.production-notify-email @@ -1,5 +1,7 @@ pipeline { - agent none + agent { + label 'linux && genarrative-build' + } options { disableConcurrentBuilds() diff --git a/jenkins/Jenkinsfile.production-server-provision b/jenkins/Jenkinsfile.production-server-provision index ffa5e3b0..25d7b229 100644 --- a/jenkins/Jenkinsfile.production-server-provision +++ b/jenkins/Jenkinsfile.production-server-provision @@ -1,27 +1,3 @@ -def runWindowsPowerShell(String scriptName, String scriptBody) { - def scriptPath = ".jenkins-${scriptName}.ps1" - writeFile file: scriptPath, text: scriptBody, encoding: 'UTF-8' - bat label: "PowerShell ${scriptName}", script: """ -@echo off -setlocal -set "GENARRATIVE_POWERSHELL=%SystemRoot%\\System32\\WindowsPowerShell\\v1.0\\powershell.exe" -if not exist "%GENARRATIVE_POWERSHELL%" ( - echo [jenkins-powershell] powershell.exe not found: %GENARRATIVE_POWERSHELL% - exit /b 1 -) -echo [jenkins-powershell] user: -whoami -echo [jenkins-powershell] workspace: %CD% -echo [jenkins-powershell] exe: %GENARRATIVE_POWERSHELL% -if not exist "%CD%\\${scriptPath}" ( - echo [jenkins-powershell] script not found: %CD%\\${scriptPath} - exit /b 1 -) -"%GENARRATIVE_POWERSHELL%" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -Command "try { \$path = Join-Path (Get-Location).ProviderPath '${scriptPath}'; Write-Host '[jenkins-powershell] script:' \$path; \$text = [System.IO.File]::ReadAllText(\$path, [System.Text.Encoding]::UTF8); Write-Host '[jenkins-powershell] loaded bytes:' ([System.IO.File]::ReadAllBytes(\$path).Length); \$scriptBlock = [ScriptBlock]::Create(\$text); & \$scriptBlock; if (\$LASTEXITCODE -is [int] -and \$LASTEXITCODE -ne 0) { exit \$LASTEXITCODE } } catch { Write-Host '[jenkins-powershell] failed:' \$_.Exception.Message; if (\$_.ScriptStackTrace) { Write-Host \$_.ScriptStackTrace }; exit 1 }" -exit /b %ERRORLEVEL% -""" -} - pipeline { agent none @@ -46,11 +22,11 @@ pipeline { string(name: 'COMMIT_HASH', defaultValue: '', description: '部署脚本来源 commit') string(name: 'SERVER_NAME', defaultValue: 'genarrative.example.com', description: '证书主域名；也作为 Nginx server_name 的第一个域名') string(name: 'SERVER_ALIASES', defaultValue: '', description: '可选，额外 Nginx server_name，多个用空格或逗号分隔，例如 www.genarrative.world') - string(name: 'PROVISION_DOWNLOADS_DIR', defaultValue: 'provision-tool-downloads', description: 'Windows 下载阶段暂存 SpacetimeDB/otelcol 安装包的工作区相对目录') + string(name: 'PROVISION_DOWNLOADS_DIR', defaultValue: 'provision-tool-downloads', description: 'Linux 预下载阶段暂存 SpacetimeDB/otelcol 安装包的工作区相对目录') string(name: 'PROVISION_TOOLS_DIR', defaultValue: 'provision-tools', description: '目标机工作区内由已下载安装包生成的工具包目录') - string(name: 'PROVISION_DOWNLOAD_PROXY', defaultValue: '', description: '可选，Windows 下载 SpacetimeDB 和 otelcol-contrib 时使用的代理地址，例如 http://127.0.0.1:7890；留空不设置代理') - string(name: 'SPACETIME_DOWNLOAD_ROOT', defaultValue: 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download', description: 'Windows 下载 SpacetimeDB Linux release tarball 的根地址；目标机不访问该地址') - string(name: 'SPACETIME_TARGET_HOST', defaultValue: 'x86_64-unknown-linux-gnu', description: '目标机 SpacetimeDB 预编译包 host triple，development/release Linux amd64 使用默认值') + string(name: 'PROVISION_DOWNLOAD_PROXY', defaultValue: '', description: '可选，Linux 预下载阶段下载 SpacetimeDB 和 otelcol-contrib 时使用的代理地址，例如 http://127.0.0.1:7890；留空不设置代理') + string(name: 'SPACETIME_DOWNLOAD_ROOT', defaultValue: 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download', description: 'Linux 预下载阶段使用的 SpacetimeDB Linux release tarball 根地址') + string(name: 'SPACETIME_TARGET_HOST', defaultValue: 'x86_64-unknown-linux-gnu', description: 'SpacetimeDB 预编译包 host triple，development/release Linux amd64 使用默认值') string(name: 'SPACETIME_ROOT', defaultValue: '/stdb', description: 'SpacetimeDB root-dir') string(name: 'RELEASE_ROOT', defaultValue: '/opt/genarrative/releases', description: 'release 根目录') string(name: 'CURRENT_LINK', defaultValue: '/opt/genarrative/current', description: '当前版本软链接') @@ -66,7 +42,7 @@ pipeline { stages { stage('Prepare') { agent { - label 'windows' + label 'linux && genarrative-build' } steps { script { @@ -134,258 +110,64 @@ pipeline { } } - stage('Download Provision Tool Archives') { + stage('Prepare Provision Tools') { agent { - label 'windows' + label 'linux && genarrative-build' } steps { script { - runWindowsPowerShell('server-provision-tool-downloads', ''' - $ErrorActionPreference = 'Stop' - [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 - - $downloadsDir = if ($env:PROVISION_DOWNLOADS_DIR) { $env:PROVISION_DOWNLOADS_DIR } else { 'provision-tool-downloads' } - $otelVersion = if ($env:OTELCOL_VERSION) { $env:OTELCOL_VERSION } else { '0.151.0' } - $prepareOtel = if ($env:ENABLE_OTELCOL) { $env:ENABLE_OTELCOL } else { 'true' } - $otelRoot = if ($env:OTELCOL_DOWNLOAD_ROOT) { $env:OTELCOL_DOWNLOAD_ROOT.TrimEnd('/') } else { 'https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download' } - $spacetimeDownloadRoot = if ($env:SPACETIME_DOWNLOAD_ROOT) { $env:SPACETIME_DOWNLOAD_ROOT.TrimEnd('/') } else { 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download' } - $spacetimeTargetHost = if ($env:SPACETIME_TARGET_HOST) { $env:SPACETIME_TARGET_HOST } else { 'x86_64-unknown-linux-gnu' } - $downloadProxy = if ($env:PROVISION_DOWNLOAD_PROXY) { $env:PROVISION_DOWNLOAD_PROXY } else { '' } - $workspace = (Get-Location).ProviderPath - if ([System.IO.Path]::IsPathRooted($downloadsDir)) { - throw "[prepare-provision-downloads] PROVISION_DOWNLOADS_DIR 只能是工作区内相对路径: ${downloadsDir}" - } - $downloadsDir = Join-Path $workspace $downloadsDir - Write-Host "[prepare-provision-downloads] Windows workspace: ${workspace}" - Write-Host "[prepare-provision-downloads] download dir: ${downloadsDir}" - - if (Test-Path -LiteralPath $downloadsDir) { - Write-Host "[prepare-provision-downloads] 复用已有下载目录: ${downloadsDir}" - } else { - New-Item -ItemType Directory -Force -Path $downloadsDir | Out-Null - Write-Host "[prepare-provision-downloads] 已创建下载目录: ${downloadsDir}" - } - - if ($downloadProxy) { - $env:HTTP_PROXY = $downloadProxy - $env:HTTPS_PROXY = $downloadProxy - $env:ALL_PROXY = $downloadProxy - Write-Host "[prepare-provision-downloads] 已配置 Windows 下载代理: $($downloadProxy -replace '://.*', '://***')" - } - - function Get-GithubReleaseAssetDigest { - param( - [Parameter(Mandatory=$true)][string]$Repository, - [Parameter(Mandatory=$true)][string]$ReleaseSelector, - [Parameter(Mandatory=$true)][string]$AssetName - ) - - $request = @{ - Uri = "https://api.github.com/repos/${Repository}/${ReleaseSelector}" - Headers = @{ - Accept = 'application/vnd.github+json' - 'User-Agent' = 'Genarrative-Server-Provision' - } - ErrorAction = 'Stop' - } - if ($downloadProxy) { - $request.Proxy = $downloadProxy - } - - Write-Host "[prepare-provision-downloads] 查询 GitHub digest: repo=${Repository} release=${ReleaseSelector} asset=${AssetName}" - $release = Invoke-RestMethod @request - $asset = $release.assets | Where-Object { $_.name -eq $AssetName } | Select-Object -First 1 - if (-not $asset) { - throw "[prepare-provision-downloads] GitHub release 未找到资产: ${Repository}/${AssetName}" - } - if (-not $asset.digest) { - throw "[prepare-provision-downloads] GitHub release 未返回 digest: ${Repository}/${AssetName}" - } - Write-Host "[prepare-provision-downloads] GitHub digest ${AssetName}: $($asset.digest)" - return $asset.digest - } - - function Get-FileDigest { - param( - [Parameter(Mandatory=$true)][string]$Path, - [Parameter(Mandatory=$true)][string]$Algorithm - ) - - $fileHash = Get-FileHash -Algorithm $Algorithm -LiteralPath $Path - return $fileHash.Hash.ToLowerInvariant() - } - - function Test-DownloadDigestMatch { - param( - [Parameter(Mandatory=$true)][string]$Path, - [Parameter(Mandatory=$true)][string]$ExpectedDigest - ) - - $parts = $ExpectedDigest.Split(':', 2) - if ($parts.Length -ne 2) { - throw "[prepare-provision-downloads] 无法解析 GitHub digest: ${ExpectedDigest}" - } - $algorithm = $parts[0].Trim().ToLowerInvariant() - $expectedHash = $parts[1].Trim().ToLowerInvariant() - if ($algorithm -ne 'sha256') { - throw "[prepare-provision-downloads] 暂不支持的 GitHub digest 算法: ${algorithm}" - } - $localHash = Get-FileDigest -Path $Path -Algorithm 'SHA256' - return $localHash -eq $expectedHash - } - - function Invoke-ProvisionDownload { - param( - [Parameter(Mandatory=$true)][string]$Label, - [Parameter(Mandatory=$true)][string]$Url, - [Parameter(Mandatory=$true)][string]$Output, - [string]$ExpectedDigest = '' - ) - - if ($ExpectedDigest) { - if (Test-Path -LiteralPath $Output) { - if (Test-DownloadDigestMatch -Path $Output -ExpectedDigest $ExpectedDigest) { - $existingItem = Get-Item -LiteralPath $Output - Write-Host "[prepare-provision-downloads] 已存在且校验一致，跳过下载: ${Label} bytes=$($existingItem.Length) path=${Output}" - return - } - Write-Host "[prepare-provision-downloads] 已存在但校验不一致，重新下载: ${Label} path=${Output}" - } - } - - Write-Host "[prepare-provision-downloads] 下载 ${Label}: ${Url}" - $tempOutput = "${Output}.download" - if (Test-Path -LiteralPath $tempOutput) { - $tempItem = Get-Item -LiteralPath $tempOutput - if ($ExpectedDigest -and $tempItem.Length -gt 0 -and (Test-DownloadDigestMatch -Path $tempOutput -ExpectedDigest $ExpectedDigest)) { - Move-Item -LiteralPath $tempOutput -Destination $Output -Force - $finalItem = Get-Item -LiteralPath $Output - Write-Host "[prepare-provision-downloads] 已复用校验通过的临时下载: ${Label} bytes=$($finalItem.Length) path=${Output}" - return - } - if ($tempItem.Length -gt 0) { - Write-Host "[prepare-provision-downloads] 发现未完成临时文件，后续尝试断点续传: ${Label} bytes=$($tempItem.Length) path=${tempOutput}" - } else { - Remove-Item -LiteralPath $tempOutput -Force - } - } - $curl = Get-Command curl.exe -ErrorAction SilentlyContinue - $maxAttempts = 8 - for ($attempt = 1; $attempt -le $maxAttempts; $attempt++) { - $resumeBytes = 0 - if (Test-Path -LiteralPath $tempOutput) { - $resumeBytes = (Get-Item -LiteralPath $tempOutput).Length - } - try { - if ($curl) { - $arguments = @('-fL', '--retry', '3', '--retry-delay', '3', '--retry-all-errors', '--connect-timeout', '30', '--speed-time', '60', '--speed-limit', '1024') - if ($resumeBytes -gt 0) { - $arguments += @('-C', '-') - Write-Host "[prepare-provision-downloads] curl 断点续传 ${Label}: attempt=${attempt}/${maxAttempts} resumeBytes=${resumeBytes}" - } else { - Write-Host "[prepare-provision-downloads] curl 下载 ${Label}: attempt=${attempt}/${maxAttempts}" - } - $arguments += @('-o', $tempOutput) - if ($downloadProxy) { - $arguments += @('--proxy', $downloadProxy) - } - $arguments += $Url - & $curl.Source @arguments - $exitCode = $LASTEXITCODE - if ($exitCode -ne 0) { - $currentBytes = if (Test-Path -LiteralPath $tempOutput) { (Get-Item -LiteralPath $tempOutput).Length } else { 0 } - Write-Host "[prepare-provision-downloads] curl 下载未完成: ${Label}, attempt=${attempt}/${maxAttempts}, exit=${exitCode}, tempBytes=${currentBytes}" - if ($attempt -lt $maxAttempts) { - Start-Sleep -Seconds ([Math]::Min(30, 3 * $attempt)) - continue - } - throw "[prepare-provision-downloads] curl 下载失败: ${Label}, exit=${exitCode}, temp=${tempOutput}" - } - } else { - Write-Host "[prepare-provision-downloads] Invoke-WebRequest 下载 ${Label}: attempt=${attempt}/${maxAttempts}" - if ($resumeBytes -gt 0) { - Write-Host "[prepare-provision-downloads] Invoke-WebRequest 不支持断点续传，删除临时文件后重新下载: ${Label}, bytes=${resumeBytes}" - Remove-Item -LiteralPath $tempOutput -Force - } - $parameters = @{ - Uri = $Url - OutFile = $tempOutput - UseBasicParsing = $true - } - if ($downloadProxy) { - $parameters.Proxy = $downloadProxy - } - Invoke-WebRequest @parameters - } - } catch { - $currentBytes = if (Test-Path -LiteralPath $tempOutput) { (Get-Item -LiteralPath $tempOutput).Length } else { 0 } - Write-Host "[prepare-provision-downloads] 下载尝试失败: ${Label}, attempt=${attempt}/${maxAttempts}, tempBytes=${currentBytes}, error=$($_.Exception.Message)" - if ($attempt -lt $maxAttempts) { - Start-Sleep -Seconds ([Math]::Min(30, 3 * $attempt)) - continue - } - throw - } - - if (-not (Test-Path -LiteralPath $tempOutput)) { - throw "[prepare-provision-downloads] 下载未生成临时文件: ${tempOutput}" - } - $item = Get-Item -LiteralPath $tempOutput - if ($item.Length -le 0) { - if ($attempt -lt $maxAttempts) { - Write-Host "[prepare-provision-downloads] 下载结果为空，将重试: ${Label}" - Start-Sleep -Seconds ([Math]::Min(30, 3 * $attempt)) - continue - } - throw "[prepare-provision-downloads] 下载结果为空: ${tempOutput}" - } - if ($ExpectedDigest) { - if (-not (Test-DownloadDigestMatch -Path $tempOutput -ExpectedDigest $ExpectedDigest)) { - Write-Host "[prepare-provision-downloads] 下载结果校验未通过，将继续重试: ${Label}, attempt=${attempt}/${maxAttempts}, tempBytes=$($item.Length)" - if ($attempt -lt $maxAttempts) { - Remove-Item -LiteralPath $tempOutput -Force - Start-Sleep -Seconds ([Math]::Min(30, 3 * $attempt)) - continue - } - throw "[prepare-provision-downloads] 下载结果校验失败: ${Label}, temp=${tempOutput}" - } - } - Move-Item -LiteralPath $tempOutput -Destination $Output -Force - $finalItem = Get-Item -LiteralPath $Output - Write-Host "[prepare-provision-downloads] 已下载 ${Label}: bytes=$($finalItem.Length) path=${Output}" - return - } - throw "[prepare-provision-downloads] 下载重试耗尽: ${Label}" - } - - $spacetimeArchiveName = "spacetime-${spacetimeTargetHost}.tar.gz" - $spacetimeArchiveUrl = "${spacetimeDownloadRoot}/${spacetimeArchiveName}" - $spacetimeArchiveDigest = Get-GithubReleaseAssetDigest -Repository 'clockworklabs/SpacetimeDB' -ReleaseSelector 'releases/latest' -AssetName $spacetimeArchiveName - Invoke-ProvisionDownload -Label "SpacetimeDB release tarball ${spacetimeTargetHost}" -Url $spacetimeArchiveUrl -Output (Join-Path $downloadsDir $spacetimeArchiveName) -ExpectedDigest $spacetimeArchiveDigest - - if ($prepareOtel -eq 'true') { - $otelArchiveName = "otelcol-contrib_${otelVersion}_linux_amd64.tar.gz" - $otelUrl = "${otelRoot}/v${otelVersion}/${otelArchiveName}" - $otelDigest = Get-GithubReleaseAssetDigest -Repository 'open-telemetry/opentelemetry-collector-releases' -ReleaseSelector "releases/tags/v${otelVersion}" -AssetName $otelArchiveName - Invoke-ProvisionDownload -Label "otelcol-contrib ${otelVersion} linux amd64" -Url $otelUrl -Output (Join-Path $downloadsDir $otelArchiveName) -ExpectedDigest $otelDigest - } else { - Write-Host "[prepare-provision-downloads] ENABLE_OTELCOL=${prepareOtel}，跳过 otelcol-contrib 下载。" - } - - $utf8NoBom = New-Object System.Text.UTF8Encoding $false - $manifest = @( - "spacetime release tarball ${spacetimeArchiveUrl}", - "spacetime target host ${spacetimeTargetHost}", - "otelcol-contrib ${otelVersion} prepare=${prepareOtel}" - ) - [System.IO.File]::WriteAllLines((Join-Path $downloadsDir 'DOWNLOADS-MANIFEST.txt'), $manifest, $utf8NoBom) - - Get-ChildItem -LiteralPath $downloadsDir | Sort-Object Name | ForEach-Object { - Write-Host "[prepare-provision-downloads] artifact $($_.Length) $($_.Name)" - } - ''') + def checkoutFromRemote = { String remoteUrl -> + checkout([ + $class: 'GitSCM', + branches: [[name: "*/${params.SOURCE_BRANCH}"]], + doGenerateSubmoduleConfigurations: false, + extensions: [ + [$class: 'CleanBeforeCheckout'], + [$class: 'CloneOption', shallow: true, depth: 1, noTags: true, timeout: 30, honorRefspec: true], + ], + userRemoteConfigs: [[url: remoteUrl, refspec: "+refs/heads/${params.SOURCE_BRANCH}:refs/remotes/origin/${params.SOURCE_BRANCH}"]], + ]) + } + try { + checkoutFromRemote(env.GIT_REMOTE_URL) + env.EFFECTIVE_GIT_REMOTE_URL = env.GIT_REMOTE_URL + } catch (error) { + echo "Git 主地址拉取失败: ${env.GIT_REMOTE_URL}，改用备用地址: ${env.GIT_REMOTE_FALLBACK_URL}" + checkoutFromRemote(env.GIT_REMOTE_FALLBACK_URL) + env.EFFECTIVE_GIT_REMOTE_URL = env.GIT_REMOTE_FALLBACK_URL + } } - stash name: 'server-provision-tool-downloads', includes: "${params.PROVISION_DOWNLOADS_DIR}/**", useDefaultExcludes: false + sh ''' + bash <<'BASH' + set -euo pipefail + chmod +x scripts/jenkins-checkout-source.sh + SOURCE_BRANCH="${SOURCE_BRANCH:-master}" \ + COMMIT_HASH="${COMMIT_HASH:-}" \ + GIT_REMOTE_URL="${EFFECTIVE_GIT_REMOTE_URL:-${GIT_REMOTE_URL}}" \ + GIT_REMOTE_FALLBACK_URL="${GIT_REMOTE_FALLBACK_URL:-}" \ + SOURCE_COMMIT_FILE=".jenkins-source-commit" \ + scripts/jenkins-checkout-source.sh +BASH + ''' + sh ''' + bash -lc ' + set -euo pipefail + chmod +x scripts/prepare-server-provision-tools.sh + PROVISION_TOOLS_DIR="${PROVISION_TOOLS_DIR:-provision-tools}" \ + PROVISION_DOWNLOADS_DIR="${PROVISION_DOWNLOADS_DIR:-provision-tool-downloads}" \ + OTELCOL_VERSION="${OTELCOL_VERSION:-0.151.0}" \ + PREPARE_OTELCOL="${ENABLE_OTELCOL:-true}" \ + PROVISION_DOWNLOAD_PROXY="${PROVISION_DOWNLOAD_PROXY:-}" \ + SPACETIME_DOWNLOAD_ROOT="${SPACETIME_DOWNLOAD_ROOT:-https://github.com/clockworklabs/SpacetimeDB/releases/latest/download}" \ + SPACETIME_TARGET_HOST="${SPACETIME_TARGET_HOST:-x86_64-unknown-linux-gnu}" \ + scripts/prepare-server-provision-tools.sh + ' + ''' + script { + env.SOURCE_COMMIT = readFile('.jenkins-source-commit').trim() + echo "Provision 工具包源码 commit=${env.SOURCE_COMMIT}" + } + stash name: 'server-provision-tools', includes: "${params.PROVISION_TOOLS_DIR}/**", useDefaultExcludes: false } } @@ -440,20 +222,10 @@ BASH label "${params.DEPLOY_TARGET == 'development' ? 'linux && genarrative-build' : 'linux && genarrative-release-deploy'}" } steps { - unstash 'server-provision-tool-downloads' + unstash 'server-provision-tools' sh ''' bash <<'BASH' set -euo pipefail - chmod +x scripts/prepare-server-provision-tools.sh - PROVISION_DOWNLOADS_DIR="${PROVISION_DOWNLOADS_DIR:-provision-tool-downloads}" \ - PROVISION_TOOLS_DIR="${PROVISION_TOOLS_DIR:-provision-tools}" \ - OTELCOL_VERSION="${OTELCOL_VERSION:-0.151.0}" \ - PREPARE_OTELCOL="${ENABLE_OTELCOL:-true}" \ - PROVISION_REQUIRE_LOCAL_DOWNLOADS="true" \ - SPACETIME_DOWNLOAD_ROOT="${SPACETIME_DOWNLOAD_ROOT:-https://github.com/clockworklabs/SpacetimeDB/releases/latest/download}" \ - SPACETIME_TARGET_HOST="${SPACETIME_TARGET_HOST:-x86_64-unknown-linux-gnu}" \ - scripts/prepare-server-provision-tools.sh - if [[ "${ENABLE_OTELCOL:-true}" == "true" ]]; then chmod +x "${PROVISION_TOOLS_DIR:-provision-tools}/otelcol-contrib" fi @@ -504,4 +276,4 @@ BASH echo "Server provision 完成: target=${params.DEPLOY_TARGET}, dryRun=${params.DRY_RUN}, nginxConfigMode=${params.NGINX_CONFIG_MODE}" } } -} \ No newline at end of file +} diff --git a/jenkins/Jenkinsfile.production-stdb-module-build b/jenkins/Jenkinsfile.production-stdb-module-build index 9d4ead23..6b6a964a 100644 --- a/jenkins/Jenkinsfile.production-stdb-module-build +++ b/jenkins/Jenkinsfile.production-stdb-module-build @@ -1,27 +1,6 @@ -def runWindowsPowerShell(String scriptName, String scriptBody) { - def scriptPath = ".jenkins-${scriptName}.ps1" - writeFile file: scriptPath, text: scriptBody, encoding: 'UTF-8' - bat label: "PowerShell ${scriptName}", script: """ -@echo off -setlocal -set "GENARRATIVE_POWERSHELL=%SystemRoot%\\System32\\WindowsPowerShell\\v1.0\\powershell.exe" -if not exist "%GENARRATIVE_POWERSHELL%" ( - echo [jenkins-powershell] powershell.exe not found: %GENARRATIVE_POWERSHELL% - exit /b 1 -) -echo [jenkins-powershell] user: -whoami -echo [jenkins-powershell] exe: %GENARRATIVE_POWERSHELL% -"%GENARRATIVE_POWERSHELL%" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -Command "\$path = '%CD%\\${scriptPath}'; \$text = [System.IO.File]::ReadAllText(\$path, [System.Text.Encoding]::UTF8); \$utf8Bom = New-Object System.Text.UTF8Encoding(\$true); [System.IO.File]::WriteAllText(\$path, \$text, \$utf8Bom)" -if errorlevel 1 exit /b %ERRORLEVEL% -"%GENARRATIVE_POWERSHELL%" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -File "%CD%\\${scriptPath}" -exit /b %ERRORLEVEL% -""" -} - pipeline { agent { - label 'windows' + label 'linux && genarrative-build' } options { @@ -31,12 +10,10 @@ pipeline { } environment { - GIT_REMOTE_URL = 'https://git.genarrative.world/GenarrativeAI/Genarrative.git' - CARGO_HOME = '${env.WORKSPACE_TMP}/cargo-home' - CARGO_TARGET_DIR = '${env.WORKSPACE_TMP}/cargo-target/prod-release' + GIT_REMOTE_URL = 'http://127.0.0.1:3000/GenarrativeAI/Genarrative.git' + GIT_REMOTE_FALLBACK_URL = 'https://git.genarrative.world/GenarrativeAI/Genarrative.git' CARGO_INCREMENTAL = '0' RUSTC_WRAPPER = 'sccache' - SCCACHE_DIR = '${env.USERPROFILE}\\.cache\\sccache-stdb-module' SCCACHE_CACHE_SIZE = '30G' } @@ -56,106 +33,42 @@ pipeline { stages { stage('Checkout') { steps { - checkout([ - $class: 'GitSCM', - branches: [[name: "*/${params.SOURCE_BRANCH}"]], - doGenerateSubmoduleConfigurations: false, - extensions: [ - [$class: 'CleanBeforeCheckout'], - [$class: 'CloneOption', shallow: true, depth: 1, noTags: true, timeout: 30, honorRefspec: true], - ], - userRemoteConfigs: [[url: "${GIT_REMOTE_URL}", refspec: "+refs/heads/${params.SOURCE_BRANCH}:refs/remotes/origin/${params.SOURCE_BRANCH}"]], - ]) script { - runWindowsPowerShell('stdb-checkout', ''' - $ErrorActionPreference = 'Stop' - $sourceBranch = if ($env:SOURCE_BRANCH) { $env:SOURCE_BRANCH } else { 'master' } - $commitHash = if ($env:COMMIT_HASH) { $env:COMMIT_HASH } else { '' } - $gitRemoteUrl = if ($env:GIT_REMOTE_URL) { $env:GIT_REMOTE_URL } else { 'https://git.genarrative.world/GenarrativeAI/Genarrative.git' } - - function Invoke-GitCommand { - param( - [string]$Label, - [string[]]$Arguments - ) - - Write-Host "[stdb-checkout] $Label" - & git @Arguments - $exitCode = $LASTEXITCODE - if ($exitCode -ne 0) { - throw "[stdb-checkout] $Label failed with exit code $exitCode" - } - } - - Write-Host "[stdb-checkout] sourceBranch: $sourceBranch" - Write-Host "[stdb-checkout] remote: $gitRemoteUrl" - $currentCommit = (git rev-parse HEAD).Trim() - if ($LASTEXITCODE -ne 0 -or -not $currentCommit) { - throw '[stdb-checkout] cannot resolve current HEAD' - } - Write-Host "[stdb-checkout] current HEAD: $currentCommit" - - if ($commitHash) { - Write-Host "[stdb-checkout] requested commit: $commitHash" - $resolvedCommit = (git rev-parse --verify "${commitHash}^{commit}" 2>$null).Trim() - if ($LASTEXITCODE -eq 0 -and $resolvedCommit -eq $currentCommit) { - Write-Host '[stdb-checkout] requested commit already matches Jenkins GitSCM checkout' - } else { - Invoke-GitCommand -Label 'fetch source branch history' -Arguments @( - 'fetch', - '--no-tags', - '--prune', - $gitRemoteUrl, - "+refs/heads/${sourceBranch}:refs/remotes/origin/${sourceBranch}" - ) - $isShallowRepository = (git rev-parse --is-shallow-repository 2>$null).Trim() - if ($LASTEXITCODE -ne 0) { - throw '[stdb-checkout] cannot determine whether repository is shallow' - } - if ($isShallowRepository -eq 'true') { - Invoke-GitCommand -Label 'deepen source branch history' -Arguments @( - 'fetch', - '--unshallow', - '--no-tags', - $gitRemoteUrl, - "+refs/heads/${sourceBranch}:refs/remotes/origin/${sourceBranch}" - ) - } - Invoke-GitCommand -Label 'validate source branch ref' -Arguments @( - 'cat-file', - '-e', - "refs/remotes/origin/${sourceBranch}^{commit}" - ) - Invoke-GitCommand -Label 'validate requested commit' -Arguments @( - 'cat-file', - '-e', - "${commitHash}^{commit}" - ) - $resolvedCommit = (git rev-parse --verify "${commitHash}^{commit}").Trim() - if ($LASTEXITCODE -ne 0 -or -not $resolvedCommit) { - throw "[stdb-checkout] cannot resolve requested commit: $commitHash" - } - Invoke-GitCommand -Label 'validate requested commit belongs to branch' -Arguments @( - 'merge-base', - '--is-ancestor', - $resolvedCommit, - "refs/remotes/origin/${sourceBranch}" - ) - Invoke-GitCommand -Label "checkout commit $resolvedCommit" -Arguments @( - 'checkout', - '--force', - $resolvedCommit - ) - } - } else { - Write-Host "[stdb-checkout] COMMIT_HASH empty, reusing Jenkins GitSCM checkout result" - } - - $resolvedCommit = (git rev-parse HEAD).Trim() - $utf8NoBom = New-Object System.Text.UTF8Encoding $false - [System.IO.File]::WriteAllText((Join-Path (Get-Location) '.jenkins-source-commit'), "$resolvedCommit`n", $utf8NoBom) - ''') - env.SOURCE_COMMIT = readFile('.jenkins-source-commit').replace('\uFEFF', '').trim() + def checkoutFromRemote = { String remoteUrl -> + checkout([ + $class: 'GitSCM', + branches: [[name: "*/${params.SOURCE_BRANCH}"]], + doGenerateSubmoduleConfigurations: false, + extensions: [ + [$class: 'CleanBeforeCheckout'], + [$class: 'CloneOption', shallow: true, depth: 1, noTags: true, timeout: 30, honorRefspec: true], + ], + userRemoteConfigs: [[url: remoteUrl, refspec: "+refs/heads/${params.SOURCE_BRANCH}:refs/remotes/origin/${params.SOURCE_BRANCH}"]], + ]) + } + try { + checkoutFromRemote(env.GIT_REMOTE_URL) + env.EFFECTIVE_GIT_REMOTE_URL = env.GIT_REMOTE_URL + } catch (error) { + echo "Git 主地址拉取失败: ${env.GIT_REMOTE_URL}，改用备用地址: ${env.GIT_REMOTE_FALLBACK_URL}" + checkoutFromRemote(env.GIT_REMOTE_FALLBACK_URL) + env.EFFECTIVE_GIT_REMOTE_URL = env.GIT_REMOTE_FALLBACK_URL + } + } + sh ''' + bash -lc ' + set -euo pipefail + chmod +x scripts/jenkins-checkout-source.sh + SOURCE_BRANCH="${SOURCE_BRANCH:-master}" \ + COMMIT_HASH="${COMMIT_HASH:-}" \ + GIT_REMOTE_URL="${EFFECTIVE_GIT_REMOTE_URL:-${GIT_REMOTE_URL}}" \ + GIT_REMOTE_FALLBACK_URL="${GIT_REMOTE_FALLBACK_URL:-}" \ + SOURCE_COMMIT_FILE=".jenkins-source-commit" \ + scripts/jenkins-checkout-source.sh + ' + ''' + script { + env.SOURCE_COMMIT = readFile('.jenkins-source-commit').trim() env.EFFECTIVE_BUILD_VERSION = params.BUILD_VERSION?.trim() ? params.BUILD_VERSION.trim() : env.BUILD_NUMBER } } @@ -165,45 +78,27 @@ pipeline { steps { script { def buildStep = { - runWindowsPowerShell('stdb-build', ''' - $ErrorActionPreference = 'Stop' - $workspaceTmp = if ($env:WORKSPACE_TMP) { $env:WORKSPACE_TMP } else { "$env:WORKSPACE@tmp" } - $env:CARGO_HOME = "$workspaceTmp/cargo-home" - $env:CARGO_TARGET_DIR = "$workspaceTmp/cargo-target/prod-release" - $env:SCCACHE_DIR = "$env:USERPROFILE/.cache/sccache-stdb-module" - $env:PATH = "$env:CARGO_HOME/bin;$env:PATH" - $gitBash = @( - $env:GENARRATIVE_BASH, - 'C:/Program Files/Git/bin/bash.exe', - 'C:/Program Files/Git/usr/bin/bash.exe', - 'C:/msys64/usr/bin/bash.exe', - 'bash' - ) | Where-Object { $_ -and (($_ -eq 'bash') -or (Test-Path $_)) } | Select-Object -First 1 - if (-not $gitBash) { - throw '[stdb-build] Windows 构建节点缺少 Git Bash，无法执行仓库现有生产构建脚本。请先安装 Git for Windows，并确保 bash 在 PATH 中。' - } - $env:GENARRATIVE_BASH = $gitBash - if (-not (Get-Command cargo -ErrorAction SilentlyContinue)) { - throw '[stdb-build] 缺少 cargo。请先在 Windows 构建节点安装 Rust 工具链，并确保 cargo 在 PATH 中。' - } - # sccache 只是可选缓存；PATH 中存在但不可执行时必须回退到 rustc。 - $sccacheCommand = Get-Command sccache -ErrorAction SilentlyContinue - $sccacheUsable = $false - if ($sccacheCommand) { - try { - & $sccacheCommand.Source --version | Out-Host - $sccacheUsable = $true - } catch { - Write-Host "[stdb-build] sccache 无法执行：$($_.Exception.Message)" - } - } - if (-not $sccacheUsable) { - Write-Host '[stdb-build] 未找到可用 sccache，改用 rustc 直接构建。' - Remove-Item Env:RUSTC_WRAPPER -ErrorAction SilentlyContinue - } - npm run build:production-release -- --component spacetime-module --name "$env:EFFECTIVE_BUILD_VERSION" + sh ''' + bash -lc ' + set -euo pipefail + workspace_tmp="${WORKSPACE_TMP:-${WORKSPACE}@tmp}" + export CARGO_HOME="${workspace_tmp}/cargo-home" + export CARGO_TARGET_DIR="${workspace_tmp}/cargo-target/prod-release" + export CARGO_INCREMENTAL=0 + export RUSTC_WRAPPER=sccache + export SCCACHE_DIR="${workspace_tmp}/sccache-stdb-module" + export SCCACHE_CACHE_SIZE=30G + mkdir -p "${CARGO_HOME}" "${CARGO_TARGET_DIR}" "${SCCACHE_DIR}" + chmod +x scripts/jenkins-prepare-cargo-env.sh + source scripts/jenkins-prepare-cargo-env.sh + if ! command -v sccache >/dev/null 2>&1; then + echo "[stdb-build] 未找到可用 sccache，改用 rustc 直接构建。" + unset RUSTC_WRAPPER + fi + SOURCE_BRANCH="${SOURCE_BRANCH}" SOURCE_COMMIT="${SOURCE_COMMIT}" \ + npm run build:production-release -- --component spacetime-module --name "${EFFECTIVE_BUILD_VERSION}" + ' ''' - ) } if (params.MIGRATION_BOOTSTRAP_SECRET_CREDENTIAL_ID?.trim()) { withCredentials([