diff --git a/.hermes/shared-memory/decision-log.md b/.hermes/shared-memory/decision-log.md index 3393cfe0..9896b079 100644 --- a/.hermes/shared-memory/decision-log.md +++ b/.hermes/shared-memory/decision-log.md @@ -87,20 +87,12 @@ - 验证方式：执行 `npm run container:config` 展开 compose 配置；需要真实运行时再执行 `npm run container:build`、`npm run container:up`、`npm run container:k6`，并结合容器 Nginx log 与 OTLP debug exporter 判断瓶颈。 - 关联文档：`deploy/container/README.md`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 -## 2026-05-18 生产 provision 改为构建机准备工具包再上传安装 +## 2026-05-19 生产 provision 改为 Windows 下载包后由目标机本地安装 -- 背景：目标 release 服务器无法访问 GitHub，之前的 server provision 默认仍假设 `spacetime` 和 `otelcol-contrib` 已经存在于目标机本地路径，和真实运维条件不符。 -- 决策：Jenkins 新增 `Prepare Provision Tools` 阶段，在 `linux && genarrative-build` 构建机执行 `scripts/prepare-server-provision-tools.sh`，通过官方 SpacetimeDB 安装入口和 OpenTelemetry release 包生成 `provision-tools/`，再用 `stash/unstash` 带到 release 部署 agent；`scripts/jenkins-server-provision.sh` 只从工作区工具包复制安装，不再要求目标机自己下载或预装二进制。 +- 背景：当前 `development` provision 目标实际就是 Linux agent `genarrative-build-01`，之前把 `Prepare Provision Tools` 放在 `linux && genarrative-build` 会让目标机自己连 GitHub 和 `install.spacetimedb.com`，违背“Windows 本机先下载再传到目标机”的运维要求。 +- 决策：`Genarrative-Server-Provision` 拆成 Windows 下载阶段和 Linux 目标机安装阶段。Windows 节点的 `Download Provision Tool Archives` 只下载 SpacetimeDB 官方安装脚本、Linux update installer 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz`，通过 `stash/unstash` 传到目标 Linux 节点；目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，只消费已下载件生成 `provision-tools/`，缺包直接失败，不回退外网下载。 - 影响范围：`jenkins/Jenkinsfile.production-server-provision`、`scripts/prepare-server-provision-tools.sh`、`scripts/jenkins-server-provision.sh`、生产运维文档。 -- 验证方式：Jenkins 构建机可完成工具包准备，release 部署 agent 只消费工作区文件；目标机不再依赖 GitHub 外网下载。 -- 关联文档：`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 - -## 2026-05-19 provision 工具包由 Jenkins 构建机本地准备后上传 - -- 背景：目标 release 机器不应自己下载 SpacetimeDB 或 `otelcol-contrib`，但 Jenkins 构建机可以先准备二进制工具包，再把结果带到目标部署 agent。 -- 决策：`jenkins/Jenkinsfile.production-server-provision` 不要求人工上传安装包；`Prepare Provision Tools` 阶段在 `linux && genarrative-build` 构建机本地执行 `scripts/prepare-server-provision-tools.sh`，下载 SpacetimeDB 官方安装器和 OpenTelemetry release 包并生成 `provision-tools/`，随后通过 `stash/unstash` 上传到 release 部署 agent。若构建机下载较慢，可通过 `PROVISION_DOWNLOAD_PROXY` 显式指定该构建机可访问的代理地址。 -- 影响范围：`jenkins/Jenkinsfile.production-server-provision`、`scripts/prepare-server-provision-tools.sh`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 -- 验证方式：Jenkins 日志应显示目标部署 agent 只消费 `server-provision-tools` stash；目标机不直接访问 `install.spacetimedb.com` 或 OpenTelemetry GitHub release 下载地址。 +- 验证方式：Jenkins 日志应先出现 Windows 节点的 `[prepare-provision-downloads]` 下载日志，再在 `genarrative-build-01` 上出现“使用已下载的 ...”日志；目标机不应出现直接访问 `install.spacetimedb.com` 或 OpenTelemetry GitHub release 下载地址的回退日志。 - 关联文档：`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 ## 2026-05-19 公开 gallery 入口发布限流以快拒绝保护后端 diff --git a/.hermes/shared-memory/pitfalls.md b/.hermes/shared-memory/pitfalls.md index 656466b7..46f4448f 100644 --- a/.hermes/shared-memory/pitfalls.md +++ b/.hermes/shared-memory/pitfalls.md @@ -744,6 +744,14 @@ - 验证：运行 `git ls-files --stage scripts/prepare-server-provision-tools.sh`，确认 mode 为 `100755`；重新跑 `Genarrative-Server-Provision` 时应进入工具下载/打包日志，而不是停在 `Permission denied`。 - 关联：`jenkins/Jenkinsfile.production-server-provision`、`scripts/prepare-server-provision-tools.sh`、`scripts/jenkins-checkout-source.sh`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 +## Server-Provision 下载阶段不要放回 genarrative-build-01 + +- 现象：`Genarrative-Server-Provision` 日志里 `Prepare Provision Tools` 显示 `Running on genarrative-build-01 in /root/...`，随后在该节点上下载 GitHub release 或 `install.spacetimedb.com` 失败。 +- 原因：`genarrative-build-01` 在当前 provision 流程里是 Linux 目标发布机/目标 agent，不是用户本地 Windows 下载环境；把下载阶段放在 `linux && genarrative-build` 等于让目标机自己外连。 +- 处理：下载必须发生在 Jenkins `windows` 节点的 `Download Provision Tool Archives` 阶段，先下载 SpacetimeDB Linux update installer 和 `otelcol-contrib` Linux amd64 包，再 `stash/unstash` 到目标 Linux 节点。目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，缺少下载件直接失败，不回退联网下载。 +- 验证：Jenkins 日志应先出现 `Running on ... windows` 和 `[prepare-provision-downloads] 下载 ...`，目标节点只出现 `[prepare-provision-tools] 使用已下载的 ...`；如果目标节点出现 `下载 otelcol-contrib:` 或 `下载 SpacetimeDB 官方安装器脚本:`，说明又回退到错误路径。 +- 关联：`jenkins/Jenkinsfile.production-server-provision`、`scripts/prepare-server-provision-tools.sh`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 + ## 个人任务 scope 不得扩成 work/site/module - 现象：个人任务配置为 `work` / `site` / `module` 后进度串桶或静默按 0 处理。 diff --git a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md index e18049d0..be4285e1 100644 --- a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md +++ b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md @@ -160,8 +160,8 @@ Windows Stdb module 构建流水线运行在 Jenkins `windows` 节点上。该 - `api-server` 生产模板默认 `GENARRATIVE_API_LISTEN_BACKLOG=1024`、`GENARRATIVE_API_WORKER_THREADS=4`；本地未设置 worker threads 时继续使用 Tokio 默认值。 - `GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512` 开启应用内 HTTP 并发背压；`GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=320`、`GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS=64`、`GENARRATIVE_API_ADMIN_MAX_CONCURRENT_REQUESTS=16` 分别限制公开列表、公开详情和后台 API 热路径。超过许可时直接返回 `429 Too Many Requests` 和 `Retry-After: 1`，`/healthz` 不受该限制。这些值不是 RPS 限速；如果压测中 429 上升但内存和 p95 收敛，说明背压正在保护进程。直连 `api-server` 的极高 RPS 压测若出现 `connection refused`，通常已经打到 TCP 监听 / accept 层，应同时检查 backlog、Nginx upstream keepalive 和前置限流。 - `genarrative-api.service` 设置 `LimitNOFILE=65535`、`TasksMax=2048`；上线后用 `systemctl show genarrative-api.service -p LimitNOFILE -p TasksMax` 和 `cat /proc/$(pidof api-server)/limits` 核对。 -- Server provision 不在目标机下载 SpacetimeDB 或 `otelcol-contrib`。Jenkins 的 `Prepare Provision Tools` 阶段在 `linux && genarrative-build` 构建机执行 `scripts/prepare-server-provision-tools.sh`，先在该构建机本地通过官方 SpacetimeDB 安装入口 `https://install.spacetimedb.com` 和 OpenTelemetry release 包生成 `provision-tools/`，再通过 `stash/unstash` 上传到 release 部署 agent。目标机上的 `scripts/jenkins-server-provision.sh` 只从该工作区工具包安装 `/stdb/spacetime`、`/stdb/bin/current/*` 和 `/usr/local/bin/otelcol-contrib`。注意 `scripts/jenkins-checkout-source.sh` 会执行 `git reset --hard` / `git clean`，因此被直接执行的新增脚本必须以 Git `100755` 模式提交，或在二次 checkout 之后再补 `chmod +x`。 -- 构建机下载慢时，在 `Genarrative-Server-Provision` 参数 `PROVISION_DOWNLOAD_PROXY` 填写构建机可访问的 HTTP 代理，例如 `http://:7890`。不要填写目标 release 机器视角的 `127.0.0.1`，除非代理确实运行在该构建机本机。 +- Server provision 不在目标机联网下载 SpacetimeDB 或 `otelcol-contrib`。`Genarrative-Server-Provision` 先在 Windows Jenkins 节点执行 `Download Provision Tool Archives`，把 `https://install.spacetimedb.com`、SpacetimeDB Linux update installer 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz` 先下载到工作区，再通过 `stash/unstash` 带到 `genarrative-build-01`；目标 Linux 节点上的 `scripts/prepare-server-provision-tools.sh` 只消费这些本地下载件生成 `provision-tools/`，再交给 `scripts/jenkins-server-provision.sh` 安装 `/stdb/spacetime`、`/stdb/bin/current/*` 和 `/usr/local/bin/otelcol-contrib`。注意 `scripts/jenkins-checkout-source.sh` 会执行 `git reset --hard` / `git clean`，因此被直接执行的新增脚本必须以 Git `100755` 模式提交，或在二次 checkout 之后再补 `chmod +x`。 +- Windows 下载阶段如果走代理，在 `Genarrative-Server-Provision` 参数 `PROVISION_DOWNLOAD_PROXY` 填写 Windows Jenkins 节点可访问的 HTTP 代理，例如 `http://127.0.0.1:7890`；不要填写目标 release 机器视角的 `127.0.0.1`，除非代理确实运行在该 Windows 节点本机。Linux 目标机阶段会强制要求使用本地下载件，缺少文件直接失败，不再回退到外网下载。 - `otelcol-contrib.service` 作为可选系统服务加入 provision，默认监听 `127.0.0.1:4317/4318` 并使用 `deploy/otelcol/genarrative-debug.yaml`。api-server 是否发送 OTLP 仍由 `GENARRATIVE_OTEL_ENABLED` 控制，服务 unit 见 `deploy/systemd/otelcol-contrib.service`。 - Nginx `/api/` 与 `/admin/api/` 通过 `genarrative_api` upstream 代理到 `127.0.0.1:8082`，upstream keepalive 为 64；`limit_conn` 负责连接 / 并发保护，`limit_req` 负责入口 RPS 快拒绝。当前模板把公开 gallery list 单独放到 `genarrative_gallery_rps`，默认 `rate=5000r/s`、`burst=4096`、`limit_conn=320`；公开详情和普通 API 放到 `genarrative_api_rps`，后台 API 放到 `genarrative_admin_rps`。压测时看 `/var/log/nginx/genarrative.access.log` 中的 `request_time`、`upstream_connect_time`、`upstream_header_time`、`upstream_response_time`、`upstream_status`、`request_id`。 - 作品列表 K6 脚本一次 iteration 默认请求两个公开接口，因此约 50 HTTP req/s 的目标命令使用 `SCENARIO=spike START_RPS=5 PEAK_RPS=25 HOLD=60s END_RPS=5 DETAIL_RATIO=0 npm run loadtest:k6:works`。 @@ -179,7 +179,7 @@ npm run container:k6 npm run container:down ``` -容器方案默认暴露 `http://127.0.0.1:18080`，`api-server` 在容器内监听 `0.0.0.0:8082`，Nginx 通过 `api-server:8082` upstream 反代 `/api/` 和 `/admin/api/`。SpacetimeDB 也纳入 compose，容器内由 `spacetimedb:3101` 提供服务，宿主机通过 `http://127.0.0.1:13101` 进行模块发布；Collector 镜像使用 `otel/opentelemetry-collector-contrib:0.151.0`。生产 provision 侧则通过 Jenkins 构建机准备的 `provision-tools/otelcol-contrib` 安装本机 `otelcol-contrib.service`，真实库名、token 和外部服务密钥只写本地 `deploy/container/api-server.env`，不提交 Git。完整拓扑、端口、k6 参数和 OTLP debug exporter 使用方法见 `deploy/container/README.md`。 +容器方案默认暴露 `http://127.0.0.1:18080`，`api-server` 在容器内监听 `0.0.0.0:8082`，Nginx 通过 `api-server:8082` upstream 反代 `/api/` 和 `/admin/api/`。SpacetimeDB 也纳入 compose，容器内由 `spacetimedb:3101` 提供服务，宿主机通过 `http://127.0.0.1:13101` 进行模块发布；Collector 镜像使用 `otel/opentelemetry-collector-contrib:0.151.0`。生产 provision 侧则通过 Windows Jenkins 下载件在目标 Linux 节点生成 `provision-tools/otelcol-contrib`，再安装本机 `otelcol-contrib.service`，真实库名、token 和外部服务密钥只写本地 `deploy/container/api-server.env`，不提交 Git。完整拓扑、端口、k6 参数和 OTLP debug exporter 使用方法见 `deploy/container/README.md`。 `npm run container:config` 默认只做 quiet 校验，避免把本地 env 中的 token 展开到终端；确需排查完整 compose 时再传 `-- --print`。 OpenTelemetry 现阶段默认开启 OTLP traces / metrics / logs，但本地日志与 Nginx 文件日志仍保留： diff --git a/jenkins/Jenkinsfile.production-server-provision b/jenkins/Jenkinsfile.production-server-provision index b78e2791..c324b3e0 100644 --- a/jenkins/Jenkinsfile.production-server-provision +++ b/jenkins/Jenkinsfile.production-server-provision @@ -1,3 +1,24 @@ +def runWindowsPowerShell(String scriptName, String scriptBody) { + def scriptPath = ".jenkins-${scriptName}.ps1" + writeFile file: scriptPath, text: scriptBody, encoding: 'UTF-8' + bat label: "PowerShell ${scriptName}", script: """ +@echo off +setlocal +set "GENARRATIVE_POWERSHELL=%SystemRoot%\\System32\\WindowsPowerShell\\v1.0\\powershell.exe" +if not exist "%GENARRATIVE_POWERSHELL%" ( + echo [jenkins-powershell] powershell.exe not found: %GENARRATIVE_POWERSHELL% + exit /b 1 +) +echo [jenkins-powershell] user: +whoami +echo [jenkins-powershell] exe: %GENARRATIVE_POWERSHELL% +"%GENARRATIVE_POWERSHELL%" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -Command "\$path = '%CD%\\${scriptPath}'; \$text = [System.IO.File]::ReadAllText(\$path, [System.Text.Encoding]::UTF8); \$utf8Bom = New-Object System.Text.UTF8Encoding(\$true); [System.IO.File]::WriteAllText(\$path, \$text, \$utf8Bom)" +if errorlevel 1 exit /b %ERRORLEVEL% +"%GENARRATIVE_POWERSHELL%" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -File "%CD%\\${scriptPath}" +exit /b %ERRORLEVEL% +""" +} + pipeline { agent none @@ -22,9 +43,12 @@ pipeline { string(name: 'COMMIT_HASH', defaultValue: '', description: '部署脚本来源 commit') string(name: 'SERVER_NAME', defaultValue: 'genarrative.example.com', description: '证书主域名；也作为 Nginx server_name 的第一个域名') string(name: 'SERVER_ALIASES', defaultValue: '', description: '可选，额外 Nginx server_name，多个用空格或逗号分隔，例如 www.genarrative.world') - string(name: 'PROVISION_TOOLS_DIR', defaultValue: 'provision-tools', description: '构建机准备并上传到目标机工作区的工具包目录') - string(name: 'PROVISION_DOWNLOAD_PROXY', defaultValue: '', description: '可选，构建机下载 SpacetimeDB 和 otelcol-contrib 时使用的代理地址，例如 http://proxy-host:7890；留空不设置代理') - string(name: 'SPACETIME_DOWNLOAD_ROOT', defaultValue: 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download', description: '构建机下载 SpacetimeDB 官方安装产物的根地址；目标机不访问该地址') + string(name: 'PROVISION_DOWNLOADS_DIR', defaultValue: 'provision-tool-downloads', description: 'Windows 下载阶段暂存 SpacetimeDB/otelcol 安装包的工作区相对目录') + string(name: 'PROVISION_TOOLS_DIR', defaultValue: 'provision-tools', description: '目标机工作区内由已下载安装包生成的工具包目录') + string(name: 'PROVISION_DOWNLOAD_PROXY', defaultValue: '', description: '可选，Windows 下载 SpacetimeDB 和 otelcol-contrib 时使用的代理地址，例如 http://127.0.0.1:7890；留空不设置代理') + string(name: 'SPACETIME_INSTALLER_URL', defaultValue: 'https://install.spacetimedb.com', description: 'Windows 下载 SpacetimeDB 官方安装脚本的地址；目标机不访问该地址') + string(name: 'SPACETIME_DOWNLOAD_ROOT', defaultValue: 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download', description: 'Windows 下载 SpacetimeDB Linux update installer 的根地址；目标机不访问该地址') + string(name: 'SPACETIME_TARGET_HOST', defaultValue: 'x86_64-unknown-linux-gnu', description: '目标机 SpacetimeDB 预编译包 host triple，development/release Linux amd64 使用默认值') string(name: 'SPACETIME_ROOT', defaultValue: '/stdb', description: 'SpacetimeDB root-dir') string(name: 'RELEASE_ROOT', defaultValue: '/opt/genarrative/releases', description: 'release 根目录') string(name: 'CURRENT_LINK', defaultValue: '/opt/genarrative/current', description: '当前版本软链接') @@ -40,7 +64,7 @@ pipeline { stages { stage('Prepare') { agent { - label 'linux && genarrative-build' + label 'windows' } steps { script { @@ -67,9 +91,20 @@ pipeline { if (!params.PROVISION_TOOLS_DIR?.trim()) { error('PROVISION_TOOLS_DIR 不能为空。') } - if (!(params.PROVISION_TOOLS_DIR.trim() ==~ /^[0-9A-Za-z._\/-]+$/) || params.PROVISION_TOOLS_DIR.startsWith('/') || params.PROVISION_TOOLS_DIR.contains('..')) { + if (!(params.PROVISION_TOOLS_DIR.trim() ==~ /^[0-9A-Za-z._\/-]+$/) || params.PROVISION_TOOLS_DIR.startsWith('/') || params.PROVISION_TOOLS_DIR.contains('..') || params.PROVISION_TOOLS_DIR.trim() == '.') { error("PROVISION_TOOLS_DIR 只能是工作区内的相对目录，不能包含绝对路径或连续点号: ${params.PROVISION_TOOLS_DIR}") } + if (!params.PROVISION_DOWNLOADS_DIR?.trim()) { + error('PROVISION_DOWNLOADS_DIR 不能为空。') + } + if (!(params.PROVISION_DOWNLOADS_DIR.trim() ==~ /^[0-9A-Za-z._\/-]+$/) || params.PROVISION_DOWNLOADS_DIR.startsWith('/') || params.PROVISION_DOWNLOADS_DIR.contains('..') || params.PROVISION_DOWNLOADS_DIR.trim() == '.') { + error("PROVISION_DOWNLOADS_DIR 只能是工作区内的相对目录，不能包含绝对路径或连续点号: ${params.PROVISION_DOWNLOADS_DIR}") + } + def provisionToolsDir = params.PROVISION_TOOLS_DIR.trim() + def provisionDownloadsDir = params.PROVISION_DOWNLOADS_DIR.trim() + if (provisionToolsDir == provisionDownloadsDir || provisionDownloadsDir.startsWith("${provisionToolsDir}/")) { + error("PROVISION_DOWNLOADS_DIR 不能等于或位于 PROVISION_TOOLS_DIR 内，否则目标机生成工具包时会删除下载缓存: ${provisionDownloadsDir}") + } def provisionDownloadProxy = params.PROVISION_DOWNLOAD_PROXY?.trim() if (provisionDownloadProxy && !(provisionDownloadProxy ==~ /^https?:\/\/\S+$/)) { error("PROVISION_DOWNLOAD_PROXY 只能填写 http:// 或 https:// 开头的代理地址，当前值: ${params.PROVISION_DOWNLOAD_PROXY}") @@ -77,9 +112,15 @@ pipeline { if (!(params.OTELCOL_VERSION?.trim() ==~ /^[0-9]+\.[0-9]+\.[0-9]+$/)) { error("OTELCOL_VERSION 格式应为 x.y.z: ${params.OTELCOL_VERSION}") } - if (!params.SPACETIME_DOWNLOAD_ROOT?.trim()) { + if (!(params.SPACETIME_INSTALLER_URL?.trim() ==~ /^https?:\/\/\S+$/)) { + error("SPACETIME_INSTALLER_URL 只能填写 http:// 或 https:// 开头的地址: ${params.SPACETIME_INSTALLER_URL}") + } + if (!(params.SPACETIME_DOWNLOAD_ROOT?.trim() ==~ /^https?:\/\/\S+$/)) { error('SPACETIME_DOWNLOAD_ROOT 不能为空。') } + if (!(params.SPACETIME_TARGET_HOST?.trim() ==~ /^[0-9A-Za-z._-]+$/)) { + error("SPACETIME_TARGET_HOST 只能包含字母、数字、点号、下划线和短横线: ${params.SPACETIME_TARGET_HOST}") + } def nginxMode = params.NGINX_CONFIG_MODE?.trim() if (!(nginxMode in ['none', 'production-https', 'development-http'])) { error("NGINX_CONFIG_MODE 只能是 none、production-https 或 development-http，当前值: ${params.NGINX_CONFIG_MODE}") @@ -94,65 +135,106 @@ pipeline { } } - stage('Prepare Provision Tools') { + stage('Download Provision Tool Archives') { agent { - label 'linux && genarrative-build' + label 'windows' } steps { script { - def checkoutFromRemote = { String remoteUrl -> - checkout([ - $class: 'GitSCM', - branches: [[name: "*/${params.SOURCE_BRANCH}"]], - doGenerateSubmoduleConfigurations: false, - extensions: [ - [$class: 'CleanBeforeCheckout'], - [$class: 'CloneOption', shallow: true, depth: 1, noTags: true, timeout: 30, honorRefspec: true], - ], - userRemoteConfigs: [[url: remoteUrl, refspec: "+refs/heads/${params.SOURCE_BRANCH}:refs/remotes/origin/${params.SOURCE_BRANCH}"]], - ]) - } - try { - checkoutFromRemote(env.GIT_REMOTE_URL) - env.EFFECTIVE_GIT_REMOTE_URL = env.GIT_REMOTE_URL - } catch (error) { - echo "Git 主地址拉取失败: ${env.GIT_REMOTE_URL}，改用备用地址: ${env.GIT_REMOTE_FALLBACK_URL}" - checkoutFromRemote(env.GIT_REMOTE_FALLBACK_URL) - env.EFFECTIVE_GIT_REMOTE_URL = env.GIT_REMOTE_FALLBACK_URL - } - } - sh ''' - bash <<'BASH' - set -euo pipefail - chmod +x scripts/jenkins-checkout-source.sh - SOURCE_BRANCH="${SOURCE_BRANCH:-master}" \ - COMMIT_HASH="${COMMIT_HASH:-}" \ - GIT_REMOTE_URL="${EFFECTIVE_GIT_REMOTE_URL:-${GIT_REMOTE_URL}}" \ - GIT_REMOTE_FALLBACK_URL="${GIT_REMOTE_FALLBACK_URL:-}" \ - SOURCE_COMMIT_FILE=".jenkins-source-commit" \ - scripts/jenkins-checkout-source.sh - # jenkins-checkout-source.sh 会 reset/clean 到目标 commit，前面的临时 chmod 可能被 Git mode 还原； - # 直接执行脚本前在二次 checkout 之后再补执行位，避免 Linux agent 报 Permission denied。 - chmod +x scripts/prepare-server-provision-tools.sh -BASH - ''' - sh ''' - bash <<'BASH' - set -euo pipefail + runWindowsPowerShell('server-provision-tool-downloads', ''' + $ErrorActionPreference = 'Stop' + [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 - PROVISION_TOOLS_DIR="${PROVISION_TOOLS_DIR:-provision-tools}" \ - OTELCOL_VERSION="${OTELCOL_VERSION:-0.151.0}" \ - PREPARE_OTELCOL="${ENABLE_OTELCOL:-true}" \ - PROVISION_DOWNLOAD_PROXY="${PROVISION_DOWNLOAD_PROXY:-}" \ - SPACETIME_DOWNLOAD_ROOT="${SPACETIME_DOWNLOAD_ROOT:-https://github.com/clockworklabs/SpacetimeDB/releases/latest/download}" \ - scripts/prepare-server-provision-tools.sh -BASH - ''' - script { - env.SOURCE_COMMIT = readFile('.jenkins-source-commit').trim() - echo "Provision 工具包已准备，源码 commit=${env.SOURCE_COMMIT}" + $downloadsDir = if ($env:PROVISION_DOWNLOADS_DIR) { $env:PROVISION_DOWNLOADS_DIR } else { 'provision-tool-downloads' } + $otelVersion = if ($env:OTELCOL_VERSION) { $env:OTELCOL_VERSION } else { '0.151.0' } + $prepareOtel = if ($env:ENABLE_OTELCOL) { $env:ENABLE_OTELCOL } else { 'true' } + $otelRoot = if ($env:OTELCOL_DOWNLOAD_ROOT) { $env:OTELCOL_DOWNLOAD_ROOT.TrimEnd('/') } else { 'https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download' } + $spacetimeInstallerUrl = if ($env:SPACETIME_INSTALLER_URL) { $env:SPACETIME_INSTALLER_URL } else { 'https://install.spacetimedb.com' } + $spacetimeDownloadRoot = if ($env:SPACETIME_DOWNLOAD_ROOT) { $env:SPACETIME_DOWNLOAD_ROOT.TrimEnd('/') } else { 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download' } + $spacetimeTargetHost = if ($env:SPACETIME_TARGET_HOST) { $env:SPACETIME_TARGET_HOST } else { 'x86_64-unknown-linux-gnu' } + $downloadProxy = if ($env:PROVISION_DOWNLOAD_PROXY) { $env:PROVISION_DOWNLOAD_PROXY } else { '' } + + if (Test-Path -LiteralPath $downloadsDir) { + Remove-Item -LiteralPath $downloadsDir -Recurse -Force + } + New-Item -ItemType Directory -Force -Path $downloadsDir | Out-Null + + if ($downloadProxy) { + $env:HTTP_PROXY = $downloadProxy + $env:HTTPS_PROXY = $downloadProxy + $env:ALL_PROXY = $downloadProxy + Write-Host "[prepare-provision-downloads] 已配置 Windows 下载代理: $($downloadProxy -replace '://.*', '://***')" + } + + function Invoke-ProvisionDownload { + param( + [Parameter(Mandatory=$true)][string]$Label, + [Parameter(Mandatory=$true)][string]$Url, + [Parameter(Mandatory=$true)][string]$Output + ) + + Write-Host "[prepare-provision-downloads] 下载 ${Label}: ${Url}" + $curl = Get-Command curl.exe -ErrorAction SilentlyContinue + if ($curl) { + $arguments = @('-fL', '--retry', '3', '--retry-delay', '2', '-o', $Output) + if ($downloadProxy) { + $arguments += @('--proxy', $downloadProxy) + } + $arguments += $Url + & $curl.Source @arguments + $exitCode = $LASTEXITCODE + if ($exitCode -ne 0) { + throw "[prepare-provision-downloads] curl 下载失败: ${Label}, exit=${exitCode}" + } + } else { + $parameters = @{ + Uri = $Url + OutFile = $Output + UseBasicParsing = $true + } + if ($downloadProxy) { + $parameters.Proxy = $downloadProxy + } + Invoke-WebRequest @parameters + } + + $item = Get-Item -LiteralPath $Output + if ($item.Length -le 0) { + throw "[prepare-provision-downloads] 下载结果为空: ${Output}" + } + Write-Host "[prepare-provision-downloads] 已下载 ${Label}: bytes=$($item.Length) path=${Output}" + } + + $installerPath = Join-Path $downloadsDir 'spacetime-install.sh' + Invoke-ProvisionDownload -Label 'SpacetimeDB install script' -Url $spacetimeInstallerUrl -Output $installerPath + + $spacetimeUpdateName = "spacetimedb-update-${spacetimeTargetHost}" + $spacetimeUpdateUrl = "${spacetimeDownloadRoot}/${spacetimeUpdateName}" + Invoke-ProvisionDownload -Label "SpacetimeDB Linux update installer ${spacetimeTargetHost}" -Url $spacetimeUpdateUrl -Output (Join-Path $downloadsDir $spacetimeUpdateName) + + if ($prepareOtel -eq 'true') { + $otelArchiveName = "otelcol-contrib_${otelVersion}_linux_amd64.tar.gz" + $otelUrl = "${otelRoot}/v${otelVersion}/${otelArchiveName}" + Invoke-ProvisionDownload -Label "otelcol-contrib ${otelVersion} linux amd64" -Url $otelUrl -Output (Join-Path $downloadsDir $otelArchiveName) + } else { + Write-Host "[prepare-provision-downloads] ENABLE_OTELCOL=${prepareOtel}，跳过 otelcol-contrib 下载。" + } + + $utf8NoBom = New-Object System.Text.UTF8Encoding $false + $manifest = @( + "spacetime installer ${spacetimeInstallerUrl}", + "spacetime update ${spacetimeDownloadRoot}/${spacetimeUpdateName}", + "spacetime target host ${spacetimeTargetHost}", + "otelcol-contrib ${otelVersion} prepare=${prepareOtel}" + ) + [System.IO.File]::WriteAllLines((Join-Path $downloadsDir 'DOWNLOADS-MANIFEST.txt'), $manifest, $utf8NoBom) + + Get-ChildItem -LiteralPath $downloadsDir | Sort-Object Name | ForEach-Object { + Write-Host "[prepare-provision-downloads] artifact $($_.Length) $($_.Name)" + } + ''') } - stash name: 'server-provision-tools', includes: "${params.PROVISION_TOOLS_DIR}/**", useDefaultExcludes: false + stash name: 'server-provision-tool-downloads', includes: "${params.PROVISION_DOWNLOADS_DIR}/**", useDefaultExcludes: false } } @@ -195,6 +277,10 @@ BASH scripts/jenkins-checkout-source.sh BASH ''' + script { + env.SOURCE_COMMIT = readFile('.jenkins-source-commit').trim() + echo "Provision 源码 commit=${env.SOURCE_COMMIT}" + } } } @@ -203,10 +289,21 @@ BASH label "${params.DEPLOY_TARGET == 'development' ? 'linux && genarrative-build' : 'linux && genarrative-release-deploy'}" } steps { - unstash 'server-provision-tools' + unstash 'server-provision-tool-downloads' sh ''' bash <<'BASH' set -euo pipefail + chmod +x scripts/prepare-server-provision-tools.sh + PROVISION_DOWNLOADS_DIR="${PROVISION_DOWNLOADS_DIR:-provision-tool-downloads}" \ + PROVISION_TOOLS_DIR="${PROVISION_TOOLS_DIR:-provision-tools}" \ + OTELCOL_VERSION="${OTELCOL_VERSION:-0.151.0}" \ + PREPARE_OTELCOL="${ENABLE_OTELCOL:-true}" \ + PROVISION_REQUIRE_LOCAL_DOWNLOADS="true" \ + SPACETIME_INSTALLER_URL="${SPACETIME_INSTALLER_URL:-https://install.spacetimedb.com}" \ + SPACETIME_DOWNLOAD_ROOT="${SPACETIME_DOWNLOAD_ROOT:-https://github.com/clockworklabs/SpacetimeDB/releases/latest/download}" \ + SPACETIME_TARGET_HOST="${SPACETIME_TARGET_HOST:-x86_64-unknown-linux-gnu}" \ + scripts/prepare-server-provision-tools.sh + if [[ "${ENABLE_OTELCOL:-true}" == "true" ]]; then chmod +x "${PROVISION_TOOLS_DIR:-provision-tools}/otelcol-contrib" fi diff --git a/scripts/prepare-server-provision-tools.sh b/scripts/prepare-server-provision-tools.sh index 5a19434a..e35fbe74 100755 --- a/scripts/prepare-server-provision-tools.sh +++ b/scripts/prepare-server-provision-tools.sh @@ -2,16 +2,23 @@ set -euo pipefail PROVISION_TOOLS_DIR="${PROVISION_TOOLS_DIR:-provision-tools}" +PROVISION_DOWNLOADS_DIR="${PROVISION_DOWNLOADS_DIR:-}" OTELCOL_VERSION="${OTELCOL_VERSION:-0.151.0}" PREPARE_OTELCOL="${PREPARE_OTELCOL:-${ENABLE_OTELCOL:-true}}" OTELCOL_DOWNLOAD_ROOT="${OTELCOL_DOWNLOAD_ROOT:-https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download}" +OTELCOL_ARCHIVE_PATH="${OTELCOL_ARCHIVE_PATH:-}" SPACETIME_INSTALLER_URL="${SPACETIME_INSTALLER_URL:-https://install.spacetimedb.com}" SPACETIME_DOWNLOAD_ROOT="${SPACETIME_DOWNLOAD_ROOT:-https://github.com/clockworklabs/SpacetimeDB/releases/latest/download}" +SPACETIME_TARGET_HOST="${SPACETIME_TARGET_HOST:-x86_64-unknown-linux-gnu}" +SPACETIME_INSTALLER_PATH="${SPACETIME_INSTALLER_PATH:-}" +SPACETIME_UPDATE_INSTALLER_PATH="${SPACETIME_UPDATE_INSTALLER_PATH:-}" PROVISION_DOWNLOAD_PROXY="${PROVISION_DOWNLOAD_PROXY:-}" PROVISION_NO_PROXY="${PROVISION_NO_PROXY:-127.0.0.1,localhost}" +PROVISION_REQUIRE_LOCAL_DOWNLOADS="${PROVISION_REQUIRE_LOCAL_DOWNLOADS:-false}" PROVISION_TOOLS_TMP_PARENT="${PROVISION_TOOLS_TMP_PARENT:-${WORKSPACE:-$(pwd)}/.tmp/server-provision-tools}" TMP_DIR_TO_CLEAN="" OTELCOL_SOURCE_DESCRIPTION="skipped" +SPACETIME_SOURCE_DESCRIPTION="unset" cleanup_tmp_dir() { if [[ -n "${TMP_DIR_TO_CLEAN}" ]]; then @@ -57,6 +64,19 @@ download_file() { fi } +validate_relative_dir() { + local label="$1" + local path="$2" + + if [[ -z "${path}" ]]; then + return + fi + if [[ "${path}" == /* || "${path}" == *..* || "${path}" == "." ]]; then + echo "[prepare-provision-tools] ${label} 只能是工作区内的相对路径: ${path}" >&2 + exit 1 + fi +} + make_spacetime_wrapper() { local target="$1" @@ -74,14 +94,32 @@ prepare_otelcol() { local archive="${tmp_dir}/otelcol-contrib.tar.gz" local extract_dir="${tmp_dir}/otelcol-contrib" local url="${OTELCOL_DOWNLOAD_ROOT}/v${OTELCOL_VERSION}/otelcol-contrib_${OTELCOL_VERSION}_linux_amd64.tar.gz" + local downloaded_archive="${PROVISION_DOWNLOADS_DIR}/otelcol-contrib_${OTELCOL_VERSION}_linux_amd64.tar.gz" + local source_archive="" local target="${PROVISION_TOOLS_DIR}/otelcol-contrib" require_cmd tar - echo "[prepare-provision-tools] 下载 otelcol-contrib: ${url}" + if [[ -n "${OTELCOL_ARCHIVE_PATH}" && -f "${OTELCOL_ARCHIVE_PATH}" ]]; then + source_archive="${OTELCOL_ARCHIVE_PATH}" + elif [[ -n "${PROVISION_DOWNLOADS_DIR}" && -f "${downloaded_archive}" ]]; then + source_archive="${downloaded_archive}" + fi + if [[ "${PROVISION_REQUIRE_LOCAL_DOWNLOADS}" == "true" && -z "${source_archive}" ]]; then + echo "[prepare-provision-tools] 要求使用 Windows 已下载的 otelcol-contrib 包，但未找到: ${downloaded_archive}" >&2 + exit 1 + fi + mkdir -p "${extract_dir}" - download_file "${url}" "${archive}" - OTELCOL_SOURCE_DESCRIPTION="download ${url}" + if [[ -n "${source_archive}" ]]; then + echo "[prepare-provision-tools] 使用已下载的 otelcol-contrib 包: ${source_archive}" + cp "${source_archive}" "${archive}" + OTELCOL_SOURCE_DESCRIPTION="local ${source_archive}" + else + echo "[prepare-provision-tools] 下载 otelcol-contrib: ${url}" + download_file "${url}" "${archive}" + OTELCOL_SOURCE_DESCRIPTION="download ${url}" + fi tar -xzf "${archive}" -C "${extract_dir}" if [[ ! -x "${extract_dir}/otelcol-contrib" ]]; then @@ -97,12 +135,51 @@ prepare_spacetime() { local tmp_dir="$1" local install_root="${tmp_dir}/spacetime-root" local target_dir="${PROVISION_TOOLS_DIR}/spacetime" + local update_name="spacetimedb-update-${SPACETIME_TARGET_HOST}" + local downloaded_update="${PROVISION_DOWNLOADS_DIR}/${update_name}" + local source_update="" + local prepared_update="${tmp_dir}/${update_name}" + local downloaded_installer="${PROVISION_DOWNLOADS_DIR}/spacetime-install.sh" + local source_installer="" - echo "[prepare-provision-tools] 使用官方安装器准备 SpacetimeDB: ${SPACETIME_INSTALLER_URL}" mkdir -p "${install_root}" - download_file "${SPACETIME_INSTALLER_URL}" "${tmp_dir}/spacetime-install.sh" - chmod 0755 "${tmp_dir}/spacetime-install.sh" - TMPDIR="${tmp_dir}" SPACETIME_DOWNLOAD_ROOT="${SPACETIME_DOWNLOAD_ROOT}" sh "${tmp_dir}/spacetime-install.sh" --root-dir "${install_root}" -y + if [[ -n "${SPACETIME_UPDATE_INSTALLER_PATH}" && -f "${SPACETIME_UPDATE_INSTALLER_PATH}" ]]; then + source_update="${SPACETIME_UPDATE_INSTALLER_PATH}" + elif [[ -n "${PROVISION_DOWNLOADS_DIR}" && -f "${downloaded_update}" ]]; then + source_update="${downloaded_update}" + fi + if [[ "${PROVISION_REQUIRE_LOCAL_DOWNLOADS}" == "true" && -z "${source_update}" ]]; then + echo "[prepare-provision-tools] 要求使用 Windows 已下载的 SpacetimeDB Linux update installer，但未找到: ${downloaded_update}" >&2 + exit 1 + fi + + if [[ -n "${source_update}" ]]; then + echo "[prepare-provision-tools] 使用已下载的 SpacetimeDB Linux update installer: ${source_update}" + cp "${source_update}" "${prepared_update}" + chmod 0755 "${prepared_update}" + TMPDIR="${tmp_dir}" "${prepared_update}" --root-dir "${install_root}" -y + SPACETIME_SOURCE_DESCRIPTION="local update ${source_update}" + else + if [[ -n "${SPACETIME_INSTALLER_PATH}" && -f "${SPACETIME_INSTALLER_PATH}" ]]; then + source_installer="${SPACETIME_INSTALLER_PATH}" + elif [[ -n "${PROVISION_DOWNLOADS_DIR}" && -f "${downloaded_installer}" ]]; then + source_installer="${downloaded_installer}" + fi + + if [[ "${PROVISION_REQUIRE_LOCAL_DOWNLOADS}" == "true" && -z "${source_installer}" ]]; then + echo "[prepare-provision-tools] 要求使用 Windows 已下载的 SpacetimeDB 官方安装器脚本，但未找到: ${downloaded_installer}" >&2 + exit 1 + elif [[ -n "${source_installer}" ]]; then + echo "[prepare-provision-tools] 使用已下载的 SpacetimeDB 官方安装器脚本: ${source_installer}" + cp "${source_installer}" "${tmp_dir}/spacetime-install.sh" + else + echo "[prepare-provision-tools] 下载 SpacetimeDB 官方安装器脚本: ${SPACETIME_INSTALLER_URL}" + download_file "${SPACETIME_INSTALLER_URL}" "${tmp_dir}/spacetime-install.sh" + fi + chmod 0755 "${tmp_dir}/spacetime-install.sh" + TMPDIR="${tmp_dir}" SPACETIME_DOWNLOAD_ROOT="${SPACETIME_DOWNLOAD_ROOT}" sh "${tmp_dir}/spacetime-install.sh" --root-dir "${install_root}" -y + SPACETIME_SOURCE_DESCRIPTION="installer ${SPACETIME_INSTALLER_URL}; download root ${SPACETIME_DOWNLOAD_ROOT}" + fi if [[ ! -x "${install_root}/bin/current/spacetimedb-cli" ]]; then echo "[prepare-provision-tools] SpacetimeDB 安装结果缺少 bin/current/spacetimedb-cli。" >&2 @@ -129,6 +206,13 @@ main() { require_cmd mktemp require_cmd rm + validate_relative_dir "PROVISION_TOOLS_DIR" "${PROVISION_TOOLS_DIR}" + validate_relative_dir "PROVISION_DOWNLOADS_DIR" "${PROVISION_DOWNLOADS_DIR}" + if [[ -n "${PROVISION_DOWNLOADS_DIR}" && "${PROVISION_DOWNLOADS_DIR%/}" == "${PROVISION_TOOLS_DIR%/}" ]]; then + echo "[prepare-provision-tools] PROVISION_DOWNLOADS_DIR 不能等于 PROVISION_TOOLS_DIR，否则会被清理: ${PROVISION_DOWNLOADS_DIR}" >&2 + exit 1 + fi + configure_download_proxy mkdir -p "${PROVISION_TOOLS_TMP_PARENT}" @@ -148,8 +232,9 @@ main() { cat >"${PROVISION_TOOLS_DIR}/MANIFEST.txt" <