diff --git a/.hermes/shared-memory/decision-log.md b/.hermes/shared-memory/decision-log.md index 02a76a14..2a788d08 100644 --- a/.hermes/shared-memory/decision-log.md +++ b/.hermes/shared-memory/decision-log.md @@ -90,10 +90,11 @@ ## 2026-05-19 生产 provision 改为 Windows 下载包后由目标机本地安装 - 背景：当前 `development` provision 目标实际就是 Linux agent `genarrative-build-01`，之前把 `Prepare Provision Tools` 放在 `linux && genarrative-build` 会让目标机自己连 GitHub 和 `install.spacetimedb.com`，违背“Windows 本机先下载再传到目标机”的运维要求。 -- 决策：`Genarrative-Server-Provision` 拆成 Windows 下载阶段和 Linux 目标机安装阶段。Windows 节点的 `Download Provision Tool Archives` 只下载 SpacetimeDB 官方安装脚本、Linux update installer 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz`，通过 `stash/unstash` 传到目标 Linux 节点；目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，只消费已下载件生成 `provision-tools/`，缺包直接失败，不回退外网下载。 +- 决策：`Genarrative-Server-Provision` 拆成 Windows 下载阶段和 Linux 目标机安装阶段。Windows 节点的 `Download Provision Tool Archives` 只下载 `spacetime-x86_64-unknown-linux-gnu.tar.gz` 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz`，通过 `stash/unstash` 传到目标 Linux 节点；目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，只消费已下载件生成 `provision-tools/`，缺包直接失败，不回退外网下载。 - 追加决策：Server-Provision 的 Windows helper 不再对 Jenkins `writeFile` 刚写出的 `.ps1` 做原地 UTF-8 BOM 重写，而是由显式 `powershell.exe` 按 UTF-8 读入脚本文本，并用 `ScriptBlock::Create(...)` 在内存中执行；这样既保留中文脚本内容，又避免同一个 workspace 脚本被立即重写时触发 `拒绝访问`。 +- 追加决策：GitHub release asset 的可用校验信息使用 `digest` 字段，实际是 `sha256:...`，不是 MD5；Windows 下载阶段先查 digest，再决定是否复用已有文件。 - 影响范围：`jenkins/Jenkinsfile.production-server-provision`、`scripts/prepare-server-provision-tools.sh`、`scripts/jenkins-server-provision.sh`、生产运维文档。 -- 验证方式：Jenkins 日志应先出现 Windows 节点的 `[jenkins-powershell] workspace:`、`[jenkins-powershell] loaded bytes:` 和 `[prepare-provision-downloads]` 下载日志，再在 `genarrative-build-01` 上出现“使用已下载的 ...”日志；目标机不应出现直接访问 `install.spacetimedb.com` 或 OpenTelemetry GitHub release 下载地址的回退日志。 +- 验证方式：Jenkins 日志应先出现 Windows 节点的 `[jenkins-powershell] workspace:`、`[jenkins-powershell] loaded bytes:` 和 `[prepare-provision-downloads]` 下载日志，再在 `genarrative-build-01` 上出现“使用已下载的 ...”日志；目标机不应出现直接访问 `install.spacetimedb.com` 或 OpenTelemetry GitHub release 下载地址的回退日志，且不再需要 `spacetimedb-update-*` 作为离线交付包。 - 关联文档：`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 ## 2026-05-19 公开 gallery 入口发布限流以快拒绝保护后端 diff --git a/.hermes/shared-memory/pitfalls.md b/.hermes/shared-memory/pitfalls.md index 5dcce27f..4a7d37cf 100644 --- a/.hermes/shared-memory/pitfalls.md +++ b/.hermes/shared-memory/pitfalls.md @@ -748,7 +748,7 @@ - 现象：`Genarrative-Server-Provision` 日志里 `Prepare Provision Tools` 显示 `Running on genarrative-build-01 in /root/...`，随后在该节点上下载 GitHub release 或 `install.spacetimedb.com` 失败。 - 原因：`genarrative-build-01` 在当前 provision 流程里是 Linux 目标发布机/目标 agent，不是用户本地 Windows 下载环境；把下载阶段放在 `linux && genarrative-build` 等于让目标机自己外连。 -- 处理：下载必须发生在 Jenkins `windows` 节点的 `Download Provision Tool Archives` 阶段，先下载 SpacetimeDB Linux update installer 和 `otelcol-contrib` Linux amd64 包，再 `stash/unstash` 到目标 Linux 节点。目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，缺少下载件直接失败，不回退联网下载。 +- 处理：下载必须发生在 Jenkins `windows` 节点的 `Download Provision Tool Archives` 阶段，先下载 SpacetimeDB Linux release tarball 和 `otelcol-contrib` Linux amd64 包，再 `stash/unstash` 到目标 Linux 节点。目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，缺少下载件直接失败，不回退联网下载。 - 验证：Jenkins 日志应先出现 `Running on ... windows` 和 `[prepare-provision-downloads] 下载 ...`，目标节点只出现 `[prepare-provision-tools] 使用已下载的 ...`；如果目标节点出现 `下载 otelcol-contrib:` 或 `下载 SpacetimeDB 官方安装器脚本:`，说明又回退到错误路径。 - 关联：`jenkins/Jenkinsfile.production-server-provision`、`scripts/prepare-server-provision-tools.sh`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 @@ -1042,10 +1042,10 @@ ## SpacetimeDB update installer 不要按带 host 后缀的下载文件名执行 -- 现象：Server-Provision 目标机阶段已经显示“使用已下载的 SpacetimeDB Linux update installer”，随后报 `Error: unknown command name for spacetimedb-update multicall binary: spacetimedb-update-x86_64-unknown-linux-gnu`。 -- 原因：下载资产名是 `spacetimedb-update-${SPACETIME_TARGET_HOST}`，但该二进制是 multicall 风格，会根据自己的启动名判断命令；直接用带 host 后缀的文件名执行会被识别成未知命令。 -- 处理：目标机准备工具包时可以继续从 Windows 下载 `spacetimedb-update-x86_64-unknown-linux-gnu`，但复制到临时执行路径时必须命名为 `spacetimedb-update`，再执行 `--root-dir ... -y`。 -- 验证：Jenkins 目标机日志不再出现 `unknown command name for spacetimedb-update multicall binary`，后续应继续检查 `bin/current/spacetimedb-cli` 和 `bin/current/spacetimedb-standalone` 是否生成。 +- 现象：Server-Provision 目标机阶段已经显示“使用已下载的 SpacetimeDB Linux update installer”，随后报 `Error: unexpected argument '-y' found` 或前置 `unknown command name for spacetimedb-update multicall binary`。 +- 原因：`spacetimedb-update-*` 不是当前离线交付的最终形态，GitHub release 页面真正可比较的缓存对象是 `spacetime-x86_64-unknown-linux-gnu.tar.gz` 这种 release tarball；GitHub release asset API 暴露的是 `digest` / SHA256，不是 MD5。 +- 处理：Windows 下载阶段应直接缓存 release tarball 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz`，目标机 `scripts/prepare-server-provision-tools.sh` 只解压本地 tarball 生成 `bin/current/spacetimedb-cli` 与 `bin/current/spacetimedb-standalone`，不要再把 update installer 当成最终离线包执行。 +- 验证：Jenkins 目标机日志不再出现 `unexpected argument '-y'`、`unknown command name for spacetimedb-update multicall binary`，后续应继续检查 `bin/current/spacetimedb-cli` 和 `bin/current/spacetimedb-standalone` 是否生成。 - 关联：`scripts/prepare-server-provision-tools.sh`、`jenkins/Jenkinsfile.production-server-provision`。 ## QQ 浏览器发现页推荐封面全不显示先查 aspect-ratio 兜底 diff --git a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md index b153e8bb..01f7deb2 100644 --- a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md +++ b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md @@ -160,7 +160,7 @@ Windows Stdb module 构建流水线运行在 Jenkins `windows` 节点上。该 - `api-server` 生产模板默认 `GENARRATIVE_API_LISTEN_BACKLOG=1024`、`GENARRATIVE_API_WORKER_THREADS=4`；本地未设置 worker threads 时继续使用 Tokio 默认值。 - `GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512` 开启应用内 HTTP 并发背压；`GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=320`、`GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS=64`、`GENARRATIVE_API_ADMIN_MAX_CONCURRENT_REQUESTS=16` 分别限制公开列表、公开详情和后台 API 热路径。超过许可时直接返回 `429 Too Many Requests` 和 `Retry-After: 1`，`/healthz` 不受该限制。这些值不是 RPS 限速；如果压测中 429 上升但内存和 p95 收敛，说明背压正在保护进程。直连 `api-server` 的极高 RPS 压测若出现 `connection refused`，通常已经打到 TCP 监听 / accept 层，应同时检查 backlog、Nginx upstream keepalive 和前置限流。 - `genarrative-api.service` 设置 `LimitNOFILE=65535`、`TasksMax=2048`；上线后用 `systemctl show genarrative-api.service -p LimitNOFILE -p TasksMax` 和 `cat /proc/$(pidof api-server)/limits` 核对。 -- Server provision 不在目标机联网下载 SpacetimeDB 或 `otelcol-contrib`。`Genarrative-Server-Provision` 先在 Windows Jenkins 节点执行 `Download Provision Tool Archives`，把 `https://install.spacetimedb.com`、SpacetimeDB Linux update installer 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz` 先下载到工作区，再通过 `stash/unstash` 带到 `genarrative-build-01`；目标 Linux 节点上的 `scripts/prepare-server-provision-tools.sh` 只消费这些本地下载件生成 `provision-tools/`，再交给 `scripts/jenkins-server-provision.sh` 安装 `/stdb/spacetime`、`/stdb/bin/current/*` 和 `/usr/local/bin/otelcol-contrib`。Windows 侧的 `runWindowsPowerShell(...)` 现在会先 `writeFile` 生成 UTF-8 `.ps1`，再直接把脚本文本读入内存并通过 `ScriptBlock::Create(...)` 执行，避免对同一个 workspace 脚本做原地 BOM 重写；此外，SpacetimeDB Linux update installer 需要以 `spacetimedb-update` 这个启动名执行，不能直接按下载文件名 `spacetimedb-update-x86_64-unknown-linux-gnu` 运行。排查时除了看下载日志，还要看 `[jenkins-powershell] workspace:`、`[jenkins-powershell] script:` 和 `[jenkins-powershell] loaded bytes:`。注意 `scripts/jenkins-checkout-source.sh` 会执行 `git reset --hard` / `git clean`，因此被直接执行的新增脚本必须以 Git `100755` 模式提交，或在二次 checkout 之后再补 `chmod +x`。 +- Server provision 不在目标机联网下载 SpacetimeDB 或 `otelcol-contrib`。`Genarrative-Server-Provision` 先在 Windows Jenkins 节点执行 `Download Provision Tool Archives`，把 `spacetime-x86_64-unknown-linux-gnu.tar.gz` 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz` 先下载到工作区，再通过 `stash/unstash` 带到 `genarrative-build-01`；Windows 下载前会先查 GitHub release asset 的 `digest` 字段做 SHA256 校验，已有本地文件且 digest 一致就直接复用，不再重复下载。目标 Linux 节点上的 `scripts/prepare-server-provision-tools.sh` 只消费这些本地下载件生成 `provision-tools/`，再交给 `scripts/jenkins-server-provision.sh` 安装 `/stdb/spacetime`、`/stdb/bin/current/*` 和 `/usr/local/bin/otelcol-contrib`。Windows 侧的 `runWindowsPowerShell(...)` 现在会先 `writeFile` 生成 UTF-8 `.ps1`，再直接把脚本文本读入内存并通过 `ScriptBlock::Create(...)` 执行，避免对同一个 workspace 脚本做原地 BOM 重写。排查时除了看下载日志，还要看 `[jenkins-powershell] workspace:`、`[jenkins-powershell] script:` 和 `[jenkins-powershell] loaded bytes:`。注意 `scripts/jenkins-checkout-source.sh` 会执行 `git reset --hard` / `git clean`，因此被直接执行的新增脚本必须以 Git `100755` 模式提交，或在二次 checkout 之后再补 `chmod +x`。 - Windows 下载阶段如果走代理，在 `Genarrative-Server-Provision` 参数 `PROVISION_DOWNLOAD_PROXY` 填写 Windows Jenkins 节点可访问的 HTTP 代理，例如 `http://127.0.0.1:7890`；不要填写目标 release 机器视角的 `127.0.0.1`，除非代理确实运行在该 Windows 节点本机。Linux 目标机阶段会强制要求使用本地下载件，缺少文件直接失败，不再回退到外网下载。 - `otelcol-contrib.service` 作为可选系统服务加入 provision，默认监听 `127.0.0.1:4317/4318` 并使用 `deploy/otelcol/genarrative-debug.yaml`。api-server 是否发送 OTLP 仍由 `GENARRATIVE_OTEL_ENABLED` 控制，服务 unit 见 `deploy/systemd/otelcol-contrib.service`。 - Nginx `/api/` 与 `/admin/api/` 通过 `genarrative_api` upstream 代理到 `127.0.0.1:8082`，upstream keepalive 为 64；`limit_conn` 负责连接 / 并发保护，`limit_req` 负责入口 RPS 快拒绝。当前模板把公开 gallery list 单独放到 `genarrative_gallery_rps`，默认 `rate=5000r/s`、`burst=4096`、`limit_conn=320`；公开详情和普通 API 放到 `genarrative_api_rps`，后台 API 放到 `genarrative_admin_rps`。压测时看 `/var/log/nginx/genarrative.access.log` 中的 `request_time`、`upstream_connect_time`、`upstream_header_time`、`upstream_response_time`、`upstream_status`、`request_id`。 diff --git a/jenkins/Jenkinsfile.production-server-provision b/jenkins/Jenkinsfile.production-server-provision index e95ef581..b317ebac 100644 --- a/jenkins/Jenkinsfile.production-server-provision +++ b/jenkins/Jenkinsfile.production-server-provision @@ -49,8 +49,7 @@ pipeline { string(name: 'PROVISION_DOWNLOADS_DIR', defaultValue: 'provision-tool-downloads', description: 'Windows 下载阶段暂存 SpacetimeDB/otelcol 安装包的工作区相对目录') string(name: 'PROVISION_TOOLS_DIR', defaultValue: 'provision-tools', description: '目标机工作区内由已下载安装包生成的工具包目录') string(name: 'PROVISION_DOWNLOAD_PROXY', defaultValue: '', description: '可选，Windows 下载 SpacetimeDB 和 otelcol-contrib 时使用的代理地址，例如 http://127.0.0.1:7890；留空不设置代理') - string(name: 'SPACETIME_INSTALLER_URL', defaultValue: 'https://install.spacetimedb.com', description: 'Windows 下载 SpacetimeDB 官方安装脚本的地址；目标机不访问该地址') - string(name: 'SPACETIME_DOWNLOAD_ROOT', defaultValue: 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download', description: 'Windows 下载 SpacetimeDB Linux update installer 的根地址；目标机不访问该地址') + string(name: 'SPACETIME_DOWNLOAD_ROOT', defaultValue: 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download', description: 'Windows 下载 SpacetimeDB Linux release tarball 的根地址；目标机不访问该地址') string(name: 'SPACETIME_TARGET_HOST', defaultValue: 'x86_64-unknown-linux-gnu', description: '目标机 SpacetimeDB 预编译包 host triple，development/release Linux amd64 使用默认值') string(name: 'SPACETIME_ROOT', defaultValue: '/stdb', description: 'SpacetimeDB root-dir') string(name: 'RELEASE_ROOT', defaultValue: '/opt/genarrative/releases', description: 'release 根目录') @@ -115,9 +114,6 @@ pipeline { if (!(params.OTELCOL_VERSION?.trim() ==~ /^[0-9]+\.[0-9]+\.[0-9]+$/)) { error("OTELCOL_VERSION 格式应为 x.y.z: ${params.OTELCOL_VERSION}") } - if (!(params.SPACETIME_INSTALLER_URL?.trim() ==~ /^https?:\/\/\S+$/)) { - error("SPACETIME_INSTALLER_URL 只能填写 http:// 或 https:// 开头的地址: ${params.SPACETIME_INSTALLER_URL}") - } if (!(params.SPACETIME_DOWNLOAD_ROOT?.trim() ==~ /^https?:\/\/\S+$/)) { error('SPACETIME_DOWNLOAD_ROOT 不能为空。') } @@ -152,7 +148,6 @@ pipeline { $otelVersion = if ($env:OTELCOL_VERSION) { $env:OTELCOL_VERSION } else { '0.151.0' } $prepareOtel = if ($env:ENABLE_OTELCOL) { $env:ENABLE_OTELCOL } else { 'true' } $otelRoot = if ($env:OTELCOL_DOWNLOAD_ROOT) { $env:OTELCOL_DOWNLOAD_ROOT.TrimEnd('/') } else { 'https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download' } - $spacetimeInstallerUrl = if ($env:SPACETIME_INSTALLER_URL) { $env:SPACETIME_INSTALLER_URL } else { 'https://install.spacetimedb.com' } $spacetimeDownloadRoot = if ($env:SPACETIME_DOWNLOAD_ROOT) { $env:SPACETIME_DOWNLOAD_ROOT.TrimEnd('/') } else { 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download' } $spacetimeTargetHost = if ($env:SPACETIME_TARGET_HOST) { $env:SPACETIME_TARGET_HOST } else { 'x86_64-unknown-linux-gnu' } $downloadProxy = if ($env:PROVISION_DOWNLOAD_PROXY) { $env:PROVISION_DOWNLOAD_PROXY } else { '' } @@ -165,11 +160,11 @@ pipeline { Write-Host "[prepare-provision-downloads] download dir: ${downloadsDir}" if (Test-Path -LiteralPath $downloadsDir) { - Write-Host "[prepare-provision-downloads] 清理旧下载目录: ${downloadsDir}" - Remove-Item -LiteralPath $downloadsDir -Recurse -Force + Write-Host "[prepare-provision-downloads] 复用已有下载目录: ${downloadsDir}" + } else { + New-Item -ItemType Directory -Force -Path $downloadsDir | Out-Null + Write-Host "[prepare-provision-downloads] 已创建下载目录: ${downloadsDir}" } - New-Item -ItemType Directory -Force -Path $downloadsDir | Out-Null - Write-Host "[prepare-provision-downloads] 已创建下载目录: ${downloadsDir}" if ($downloadProxy) { $env:HTTP_PROXY = $downloadProxy @@ -178,17 +173,94 @@ pipeline { Write-Host "[prepare-provision-downloads] 已配置 Windows 下载代理: $($downloadProxy -replace '://.*', '://***')" } + function Get-GithubReleaseAssetDigest { + param( + [Parameter(Mandatory=$true)][string]$Repository, + [Parameter(Mandatory=$true)][string]$ReleaseSelector, + [Parameter(Mandatory=$true)][string]$AssetName + ) + + $request = @{ + Uri = "https://api.github.com/repos/${Repository}/${ReleaseSelector}" + Headers = @{ + Accept = 'application/vnd.github+json' + 'User-Agent' = 'Genarrative-Server-Provision' + } + ErrorAction = 'Stop' + } + if ($downloadProxy) { + $request.Proxy = $downloadProxy + } + + Write-Host "[prepare-provision-downloads] 查询 GitHub digest: repo=${Repository} release=${ReleaseSelector} asset=${AssetName}" + $release = Invoke-RestMethod @request + $asset = $release.assets | Where-Object { $_.name -eq $AssetName } | Select-Object -First 1 + if (-not $asset) { + throw "[prepare-provision-downloads] GitHub release 未找到资产: ${Repository}/${AssetName}" + } + if (-not $asset.digest) { + throw "[prepare-provision-downloads] GitHub release 未返回 digest: ${Repository}/${AssetName}" + } + Write-Host "[prepare-provision-downloads] GitHub digest ${AssetName}: $($asset.digest)" + return $asset.digest + } + + function Get-FileDigest { + param( + [Parameter(Mandatory=$true)][string]$Path, + [Parameter(Mandatory=$true)][string]$Algorithm + ) + + $fileHash = Get-FileHash -Algorithm $Algorithm -LiteralPath $Path + return $fileHash.Hash.ToLowerInvariant() + } + + function Test-DownloadDigestMatch { + param( + [Parameter(Mandatory=$true)][string]$Path, + [Parameter(Mandatory=$true)][string]$ExpectedDigest + ) + + $parts = $ExpectedDigest.Split(':', 2) + if ($parts.Length -ne 2) { + throw "[prepare-provision-downloads] 无法解析 GitHub digest: ${ExpectedDigest}" + } + $algorithm = $parts[0].Trim().ToLowerInvariant() + $expectedHash = $parts[1].Trim().ToLowerInvariant() + if ($algorithm -ne 'sha256') { + throw "[prepare-provision-downloads] 暂不支持的 GitHub digest 算法: ${algorithm}" + } + $localHash = Get-FileDigest -Path $Path -Algorithm 'SHA256' + return $localHash -eq $expectedHash + } + function Invoke-ProvisionDownload { param( [Parameter(Mandatory=$true)][string]$Label, [Parameter(Mandatory=$true)][string]$Url, - [Parameter(Mandatory=$true)][string]$Output + [Parameter(Mandatory=$true)][string]$Output, + [string]$ExpectedDigest = '' ) + if ($ExpectedDigest) { + if (Test-Path -LiteralPath $Output) { + if (Test-DownloadDigestMatch -Path $Output -ExpectedDigest $ExpectedDigest) { + $existingItem = Get-Item -LiteralPath $Output + Write-Host "[prepare-provision-downloads] 已存在且校验一致，跳过下载: ${Label} bytes=$($existingItem.Length) path=${Output}" + return + } + Write-Host "[prepare-provision-downloads] 已存在但校验不一致，重新下载: ${Label} path=${Output}" + } + } + Write-Host "[prepare-provision-downloads] 下载 ${Label}: ${Url}" + $tempOutput = "${Output}.download" + if (Test-Path -LiteralPath $tempOutput) { + Remove-Item -LiteralPath $tempOutput -Force + } $curl = Get-Command curl.exe -ErrorAction SilentlyContinue if ($curl) { - $arguments = @('-fL', '--retry', '3', '--retry-delay', '2', '-o', $Output) + $arguments = @('-fL', '--retry', '3', '--retry-delay', '2', '-o', $tempOutput) if ($downloadProxy) { $arguments += @('--proxy', $downloadProxy) } @@ -196,46 +268,51 @@ pipeline { & $curl.Source @arguments $exitCode = $LASTEXITCODE if ($exitCode -ne 0) { - throw "[prepare-provision-downloads] curl 下载失败: ${Label}, exit=${exitCode}" + throw "[prepare-provision-downloads] curl 下载失败: ${Label}, exit=${exitCode}" + } + } else { + $parameters = @{ + Uri = $Url + OutFile = $tempOutput + UseBasicParsing = $true + } + if ($downloadProxy) { + $parameters.Proxy = $downloadProxy + } + Invoke-WebRequest @parameters } - } else { - $parameters = @{ - Uri = $Url - OutFile = $Output - UseBasicParsing = $true - } - if ($downloadProxy) { - $parameters.Proxy = $downloadProxy - } - Invoke-WebRequest @parameters - } - $item = Get-Item -LiteralPath $Output + $item = Get-Item -LiteralPath $tempOutput if ($item.Length -le 0) { - throw "[prepare-provision-downloads] 下载结果为空: ${Output}" + throw "[prepare-provision-downloads] 下载结果为空: ${tempOutput}" } - Write-Host "[prepare-provision-downloads] 已下载 ${Label}: bytes=$($item.Length) path=${Output}" + if ($ExpectedDigest) { + if (-not (Test-DownloadDigestMatch -Path $tempOutput -ExpectedDigest $ExpectedDigest)) { + throw "[prepare-provision-downloads] 下载结果校验失败: ${Label}" + } + } + Move-Item -LiteralPath $tempOutput -Destination $Output -Force + $finalItem = Get-Item -LiteralPath $Output + Write-Host "[prepare-provision-downloads] 已下载 ${Label}: bytes=$($finalItem.Length) path=${Output}" } - $installerPath = Join-Path $downloadsDir 'spacetime-install.sh' - Invoke-ProvisionDownload -Label 'SpacetimeDB install script' -Url $spacetimeInstallerUrl -Output $installerPath - - $spacetimeUpdateName = "spacetimedb-update-${spacetimeTargetHost}" - $spacetimeUpdateUrl = "${spacetimeDownloadRoot}/${spacetimeUpdateName}" - Invoke-ProvisionDownload -Label "SpacetimeDB Linux update installer ${spacetimeTargetHost}" -Url $spacetimeUpdateUrl -Output (Join-Path $downloadsDir $spacetimeUpdateName) + $spacetimeArchiveName = "spacetime-${spacetimeTargetHost}.tar.gz" + $spacetimeArchiveUrl = "${spacetimeDownloadRoot}/${spacetimeArchiveName}" + $spacetimeArchiveDigest = Get-GithubReleaseAssetDigest -Repository 'clockworklabs/SpacetimeDB' -ReleaseSelector 'releases/latest' -AssetName $spacetimeArchiveName + Invoke-ProvisionDownload -Label "SpacetimeDB release tarball ${spacetimeTargetHost}" -Url $spacetimeArchiveUrl -Output (Join-Path $downloadsDir $spacetimeArchiveName) -ExpectedDigest $spacetimeArchiveDigest if ($prepareOtel -eq 'true') { $otelArchiveName = "otelcol-contrib_${otelVersion}_linux_amd64.tar.gz" $otelUrl = "${otelRoot}/v${otelVersion}/${otelArchiveName}" - Invoke-ProvisionDownload -Label "otelcol-contrib ${otelVersion} linux amd64" -Url $otelUrl -Output (Join-Path $downloadsDir $otelArchiveName) + $otelDigest = Get-GithubReleaseAssetDigest -Repository 'open-telemetry/opentelemetry-collector-releases' -ReleaseSelector "releases/tags/v${otelVersion}" -AssetName $otelArchiveName + Invoke-ProvisionDownload -Label "otelcol-contrib ${otelVersion} linux amd64" -Url $otelUrl -Output (Join-Path $downloadsDir $otelArchiveName) -ExpectedDigest $otelDigest } else { Write-Host "[prepare-provision-downloads] ENABLE_OTELCOL=${prepareOtel}，跳过 otelcol-contrib 下载。" } $utf8NoBom = New-Object System.Text.UTF8Encoding $false $manifest = @( - "spacetime installer ${spacetimeInstallerUrl}", - "spacetime update ${spacetimeDownloadRoot}/${spacetimeUpdateName}", + "spacetime release tarball ${spacetimeArchiveUrl}", "spacetime target host ${spacetimeTargetHost}", "otelcol-contrib ${otelVersion} prepare=${prepareOtel}" ) @@ -311,7 +388,6 @@ BASH OTELCOL_VERSION="${OTELCOL_VERSION:-0.151.0}" \ PREPARE_OTELCOL="${ENABLE_OTELCOL:-true}" \ PROVISION_REQUIRE_LOCAL_DOWNLOADS="true" \ - SPACETIME_INSTALLER_URL="${SPACETIME_INSTALLER_URL:-https://install.spacetimedb.com}" \ SPACETIME_DOWNLOAD_ROOT="${SPACETIME_DOWNLOAD_ROOT:-https://github.com/clockworklabs/SpacetimeDB/releases/latest/download}" \ SPACETIME_TARGET_HOST="${SPACETIME_TARGET_HOST:-x86_64-unknown-linux-gnu}" \ scripts/prepare-server-provision-tools.sh diff --git a/scripts/prepare-server-provision-tools.sh b/scripts/prepare-server-provision-tools.sh index 2a607533..f9ff18e8 100755 --- a/scripts/prepare-server-provision-tools.sh +++ b/scripts/prepare-server-provision-tools.sh @@ -10,6 +10,7 @@ OTELCOL_ARCHIVE_PATH="${OTELCOL_ARCHIVE_PATH:-}" SPACETIME_INSTALLER_URL="${SPACETIME_INSTALLER_URL:-https://install.spacetimedb.com}" SPACETIME_DOWNLOAD_ROOT="${SPACETIME_DOWNLOAD_ROOT:-https://github.com/clockworklabs/SpacetimeDB/releases/latest/download}" SPACETIME_TARGET_HOST="${SPACETIME_TARGET_HOST:-x86_64-unknown-linux-gnu}" +SPACETIME_ARCHIVE_PATH="${SPACETIME_ARCHIVE_PATH:-}" SPACETIME_INSTALLER_PATH="${SPACETIME_INSTALLER_PATH:-}" SPACETIME_UPDATE_INSTALLER_PATH="${SPACETIME_UPDATE_INSTALLER_PATH:-}" PROVISION_DOWNLOAD_PROXY="${PROVISION_DOWNLOAD_PROXY:-}" @@ -135,6 +136,9 @@ prepare_spacetime() { local tmp_dir="$1" local install_root="${tmp_dir}/spacetime-root" local target_dir="${PROVISION_TOOLS_DIR}/spacetime" + local archive_name="spacetime-${SPACETIME_TARGET_HOST}.tar.gz" + local downloaded_archive="${PROVISION_DOWNLOADS_DIR}/${archive_name}" + local source_archive="" local update_name="spacetimedb-update-${SPACETIME_TARGET_HOST}" local downloaded_update="${PROVISION_DOWNLOADS_DIR}/${update_name}" local source_update="" @@ -143,17 +147,31 @@ prepare_spacetime() { local source_installer="" mkdir -p "${install_root}" - if [[ -n "${SPACETIME_UPDATE_INSTALLER_PATH}" && -f "${SPACETIME_UPDATE_INSTALLER_PATH}" ]]; then + if [[ -n "${SPACETIME_ARCHIVE_PATH}" && -f "${SPACETIME_ARCHIVE_PATH}" ]]; then + source_archive="${SPACETIME_ARCHIVE_PATH}" + elif [[ -n "${PROVISION_DOWNLOADS_DIR}" && -f "${downloaded_archive}" ]]; then + source_archive="${downloaded_archive}" + fi + + if [[ -n "${source_archive}" ]]; then + echo "[prepare-provision-tools] 使用已下载的 SpacetimeDB release tarball: ${source_archive}" + mkdir -p "${install_root}/bin/current" + tar -xzf "${source_archive}" -C "${install_root}/bin/current" + chmod 0755 "${install_root}/bin/current/spacetimedb-cli" "${install_root}/bin/current/spacetimedb-standalone" + SPACETIME_SOURCE_DESCRIPTION="local archive ${source_archive}" + elif [[ -n "${SPACETIME_UPDATE_INSTALLER_PATH}" && -f "${SPACETIME_UPDATE_INSTALLER_PATH}" ]]; then source_update="${SPACETIME_UPDATE_INSTALLER_PATH}" elif [[ -n "${PROVISION_DOWNLOADS_DIR}" && -f "${downloaded_update}" ]]; then source_update="${downloaded_update}" fi - if [[ "${PROVISION_REQUIRE_LOCAL_DOWNLOADS}" == "true" && -z "${source_update}" ]]; then - echo "[prepare-provision-tools] 要求使用 Windows 已下载的 SpacetimeDB Linux update installer，但未找到: ${downloaded_update}" >&2 + if [[ "${PROVISION_REQUIRE_LOCAL_DOWNLOADS}" == "true" && -z "${source_archive}" ]]; then + echo "[prepare-provision-tools] 要求使用 Windows 已下载的 SpacetimeDB release tarball，但未找到: ${downloaded_archive}" >&2 exit 1 fi - if [[ -n "${source_update}" ]]; then + if [[ -n "${source_archive}" ]]; then + : + elif [[ -n "${source_update}" ]]; then echo "[prepare-provision-tools] 使用已下载的 SpacetimeDB Linux update installer: ${source_update}" cp "${source_update}" "${prepared_update}" chmod 0755 "${prepared_update}" @@ -205,6 +223,7 @@ main() { require_cmd install require_cmd mktemp require_cmd rm + require_cmd tar validate_relative_dir "PROVISION_TOOLS_DIR" "${PROVISION_TOOLS_DIR}" validate_relative_dir "PROVISION_DOWNLOADS_DIR" "${PROVISION_DOWNLOADS_DIR}"