fix(jenkins): cache provision downloads by github digest
This commit is contained in:
@@ -90,10 +90,11 @@
|
||||
## 2026-05-19 生产 provision 改为 Windows 下载包后由目标机本地安装
|
||||
|
||||
- 背景：当前 `development` provision 目标实际就是 Linux agent `genarrative-build-01`，之前把 `Prepare Provision Tools` 放在 `linux && genarrative-build` 会让目标机自己连 GitHub 和 `install.spacetimedb.com`，违背“Windows 本机先下载再传到目标机”的运维要求。
|
||||
- 决策：`Genarrative-Server-Provision` 拆成 Windows 下载阶段和 Linux 目标机安装阶段。Windows 节点的 `Download Provision Tool Archives` 只下载 SpacetimeDB 官方安装脚本、Linux update installer 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz`，通过 `stash/unstash` 传到目标 Linux 节点；目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，只消费已下载件生成 `provision-tools/`，缺包直接失败，不回退外网下载。
|
||||
- 决策：`Genarrative-Server-Provision` 拆成 Windows 下载阶段和 Linux 目标机安装阶段。Windows 节点的 `Download Provision Tool Archives` 只下载 `spacetime-x86_64-unknown-linux-gnu.tar.gz` 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz`，通过 `stash/unstash` 传到目标 Linux 节点；目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，只消费已下载件生成 `provision-tools/`，缺包直接失败，不回退外网下载。
|
||||
- 追加决策：Server-Provision 的 Windows helper 不再对 Jenkins `writeFile` 刚写出的 `.ps1` 做原地 UTF-8 BOM 重写，而是由显式 `powershell.exe` 按 UTF-8 读入脚本文本，并用 `ScriptBlock::Create(...)` 在内存中执行；这样既保留中文脚本内容，又避免同一个 workspace 脚本被立即重写时触发 `拒绝访问`。
|
||||
- 追加决策：GitHub release asset 的可用校验信息使用 `digest` 字段，实际是 `sha256:...`，不是 MD5；Windows 下载阶段先查 digest，再决定是否复用已有文件。
|
||||
- 影响范围：`jenkins/Jenkinsfile.production-server-provision`、`scripts/prepare-server-provision-tools.sh`、`scripts/jenkins-server-provision.sh`、生产运维文档。
|
||||
- 验证方式：Jenkins 日志应先出现 Windows 节点的 `[jenkins-powershell] workspace:`、`[jenkins-powershell] loaded bytes:` 和 `[prepare-provision-downloads]` 下载日志，再在 `genarrative-build-01` 上出现“使用已下载的 ...”日志；目标机不应出现直接访问 `install.spacetimedb.com` 或 OpenTelemetry GitHub release 下载地址的回退日志。
|
||||
- 验证方式：Jenkins 日志应先出现 Windows 节点的 `[jenkins-powershell] workspace:`、`[jenkins-powershell] loaded bytes:` 和 `[prepare-provision-downloads]` 下载日志，再在 `genarrative-build-01` 上出现“使用已下载的 ...”日志；目标机不应出现直接访问 `install.spacetimedb.com` 或 OpenTelemetry GitHub release 下载地址的回退日志，且不再需要 `spacetimedb-update-*` 作为离线交付包。
|
||||
- 关联文档：`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。
|
||||
|
||||
## 2026-05-19 公开 gallery 入口发布限流以快拒绝保护后端
|
||||
|
||||
@@ -748,7 +748,7 @@
|
||||
|
||||
- 现象：`Genarrative-Server-Provision` 日志里 `Prepare Provision Tools` 显示 `Running on genarrative-build-01 in /root/...`，随后在该节点上下载 GitHub release 或 `install.spacetimedb.com` 失败。
|
||||
- 原因：`genarrative-build-01` 在当前 provision 流程里是 Linux 目标发布机/目标 agent，不是用户本地 Windows 下载环境；把下载阶段放在 `linux && genarrative-build` 等于让目标机自己外连。
|
||||
- 处理：下载必须发生在 Jenkins `windows` 节点的 `Download Provision Tool Archives` 阶段，先下载 SpacetimeDB Linux update installer 和 `otelcol-contrib` Linux amd64 包，再 `stash/unstash` 到目标 Linux 节点。目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，缺少下载件直接失败，不回退联网下载。
|
||||
- 处理：下载必须发生在 Jenkins `windows` 节点的 `Download Provision Tool Archives` 阶段，先下载 SpacetimeDB Linux release tarball 和 `otelcol-contrib` Linux amd64 包，再 `stash/unstash` 到目标 Linux 节点。目标机执行 `scripts/prepare-server-provision-tools.sh` 时设置 `PROVISION_REQUIRE_LOCAL_DOWNLOADS=true`，缺少下载件直接失败，不回退联网下载。
|
||||
- 验证：Jenkins 日志应先出现 `Running on ... windows` 和 `[prepare-provision-downloads] 下载 ...`，目标节点只出现 `[prepare-provision-tools] 使用已下载的 ...`；如果目标节点出现 `下载 otelcol-contrib:` 或 `下载 SpacetimeDB 官方安装器脚本:`，说明又回退到错误路径。
|
||||
- 关联：`jenkins/Jenkinsfile.production-server-provision`、`scripts/prepare-server-provision-tools.sh`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。
|
||||
|
||||
@@ -1042,10 +1042,10 @@
|
||||
|
||||
## SpacetimeDB update installer 不要按带 host 后缀的下载文件名执行
|
||||
|
||||
- 现象：Server-Provision 目标机阶段已经显示“使用已下载的 SpacetimeDB Linux update installer”，随后报 `Error: unknown command name for spacetimedb-update multicall binary: spacetimedb-update-x86_64-unknown-linux-gnu`。
|
||||
- 原因：下载资产名是 `spacetimedb-update-${SPACETIME_TARGET_HOST}`，但该二进制是 multicall 风格，会根据自己的启动名判断命令；直接用带 host 后缀的文件名执行会被识别成未知命令。
|
||||
- 处理：目标机准备工具包时可以继续从 Windows 下载 `spacetimedb-update-x86_64-unknown-linux-gnu`，但复制到临时执行路径时必须命名为 `spacetimedb-update`，再执行 `--root-dir ... -y`。
|
||||
- 验证：Jenkins 目标机日志不再出现 `unknown command name for spacetimedb-update multicall binary`，后续应继续检查 `bin/current/spacetimedb-cli` 和 `bin/current/spacetimedb-standalone` 是否生成。
|
||||
- 现象：Server-Provision 目标机阶段已经显示“使用已下载的 SpacetimeDB Linux update installer”，随后报 `Error: unexpected argument '-y' found` 或前置 `unknown command name for spacetimedb-update multicall binary`。
|
||||
- 原因：`spacetimedb-update-*` 不是当前离线交付的最终形态，GitHub release 页面真正可比较的缓存对象是 `spacetime-x86_64-unknown-linux-gnu.tar.gz` 这种 release tarball；GitHub release asset API 暴露的是 `digest` / SHA256，不是 MD5。
|
||||
- 处理：Windows 下载阶段应直接缓存 release tarball 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz`，目标机 `scripts/prepare-server-provision-tools.sh` 只解压本地 tarball 生成 `bin/current/spacetimedb-cli` 与 `bin/current/spacetimedb-standalone`，不要再把 update installer 当成最终离线包执行。
|
||||
- 验证：Jenkins 目标机日志不再出现 `unexpected argument '-y'`、`unknown command name for spacetimedb-update multicall binary`，后续应继续检查 `bin/current/spacetimedb-cli` 和 `bin/current/spacetimedb-standalone` 是否生成。
|
||||
- 关联：`scripts/prepare-server-provision-tools.sh`、`jenkins/Jenkinsfile.production-server-provision`。
|
||||
|
||||
## QQ 浏览器发现页推荐封面全不显示先查 aspect-ratio 兜底
|
||||
|
||||
@@ -160,7 +160,7 @@ Windows Stdb module 构建流水线运行在 Jenkins `windows` 节点上。该
|
||||
- `api-server` 生产模板默认 `GENARRATIVE_API_LISTEN_BACKLOG=1024`、`GENARRATIVE_API_WORKER_THREADS=4`；本地未设置 worker threads 时继续使用 Tokio 默认值。
|
||||
- `GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512` 开启应用内 HTTP 并发背压；`GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=320`、`GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS=64`、`GENARRATIVE_API_ADMIN_MAX_CONCURRENT_REQUESTS=16` 分别限制公开列表、公开详情和后台 API 热路径。超过许可时直接返回 `429 Too Many Requests` 和 `Retry-After: 1`，`/healthz` 不受该限制。这些值不是 RPS 限速；如果压测中 429 上升但内存和 p95 收敛，说明背压正在保护进程。直连 `api-server` 的极高 RPS 压测若出现 `connection refused`，通常已经打到 TCP 监听 / accept 层，应同时检查 backlog、Nginx upstream keepalive 和前置限流。
|
||||
- `genarrative-api.service` 设置 `LimitNOFILE=65535`、`TasksMax=2048`；上线后用 `systemctl show genarrative-api.service -p LimitNOFILE -p TasksMax` 和 `cat /proc/$(pidof api-server)/limits` 核对。
|
||||
- Server provision 不在目标机联网下载 SpacetimeDB 或 `otelcol-contrib`。`Genarrative-Server-Provision` 先在 Windows Jenkins 节点执行 `Download Provision Tool Archives`，把 `https://install.spacetimedb.com`、SpacetimeDB Linux update installer 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz` 先下载到工作区，再通过 `stash/unstash` 带到 `genarrative-build-01`；目标 Linux 节点上的 `scripts/prepare-server-provision-tools.sh` 只消费这些本地下载件生成 `provision-tools/`，再交给 `scripts/jenkins-server-provision.sh` 安装 `/stdb/spacetime`、`/stdb/bin/current/*` 和 `/usr/local/bin/otelcol-contrib`。Windows 侧的 `runWindowsPowerShell(...)` 现在会先 `writeFile` 生成 UTF-8 `.ps1`，再直接把脚本文本读入内存并通过 `ScriptBlock::Create(...)` 执行，避免对同一个 workspace 脚本做原地 BOM 重写；此外，SpacetimeDB Linux update installer 需要以 `spacetimedb-update` 这个启动名执行，不能直接按下载文件名 `spacetimedb-update-x86_64-unknown-linux-gnu` 运行。排查时除了看下载日志，还要看 `[jenkins-powershell] workspace:`、`[jenkins-powershell] script:` 和 `[jenkins-powershell] loaded bytes:`。注意 `scripts/jenkins-checkout-source.sh` 会执行 `git reset --hard` / `git clean`，因此被直接执行的新增脚本必须以 Git `100755` 模式提交，或在二次 checkout 之后再补 `chmod +x`。
|
||||
- Server provision 不在目标机联网下载 SpacetimeDB 或 `otelcol-contrib`。`Genarrative-Server-Provision` 先在 Windows Jenkins 节点执行 `Download Provision Tool Archives`，把 `spacetime-x86_64-unknown-linux-gnu.tar.gz` 和 `otelcol-contrib_0.151.0_linux_amd64.tar.gz` 先下载到工作区，再通过 `stash/unstash` 带到 `genarrative-build-01`；Windows 下载前会先查 GitHub release asset 的 `digest` 字段做 SHA256 校验，已有本地文件且 digest 一致就直接复用，不再重复下载。目标 Linux 节点上的 `scripts/prepare-server-provision-tools.sh` 只消费这些本地下载件生成 `provision-tools/`，再交给 `scripts/jenkins-server-provision.sh` 安装 `/stdb/spacetime`、`/stdb/bin/current/*` 和 `/usr/local/bin/otelcol-contrib`。Windows 侧的 `runWindowsPowerShell(...)` 现在会先 `writeFile` 生成 UTF-8 `.ps1`，再直接把脚本文本读入内存并通过 `ScriptBlock::Create(...)` 执行，避免对同一个 workspace 脚本做原地 BOM 重写。排查时除了看下载日志，还要看 `[jenkins-powershell] workspace:`、`[jenkins-powershell] script:` 和 `[jenkins-powershell] loaded bytes:`。注意 `scripts/jenkins-checkout-source.sh` 会执行 `git reset --hard` / `git clean`，因此被直接执行的新增脚本必须以 Git `100755` 模式提交，或在二次 checkout 之后再补 `chmod +x`。
|
||||
- Windows 下载阶段如果走代理，在 `Genarrative-Server-Provision` 参数 `PROVISION_DOWNLOAD_PROXY` 填写 Windows Jenkins 节点可访问的 HTTP 代理，例如 `http://127.0.0.1:7890`；不要填写目标 release 机器视角的 `127.0.0.1`，除非代理确实运行在该 Windows 节点本机。Linux 目标机阶段会强制要求使用本地下载件，缺少文件直接失败，不再回退到外网下载。
|
||||
- `otelcol-contrib.service` 作为可选系统服务加入 provision，默认监听 `127.0.0.1:4317/4318` 并使用 `deploy/otelcol/genarrative-debug.yaml`。api-server 是否发送 OTLP 仍由 `GENARRATIVE_OTEL_ENABLED` 控制，服务 unit 见 `deploy/systemd/otelcol-contrib.service`。
|
||||
- Nginx `/api/` 与 `/admin/api/` 通过 `genarrative_api` upstream 代理到 `127.0.0.1:8082`，upstream keepalive 为 64；`limit_conn` 负责连接 / 并发保护，`limit_req` 负责入口 RPS 快拒绝。当前模板把公开 gallery list 单独放到 `genarrative_gallery_rps`，默认 `rate=5000r/s`、`burst=4096`、`limit_conn=320`；公开详情和普通 API 放到 `genarrative_api_rps`，后台 API 放到 `genarrative_admin_rps`。压测时看 `/var/log/nginx/genarrative.access.log` 中的 `request_time`、`upstream_connect_time`、`upstream_header_time`、`upstream_response_time`、`upstream_status`、`request_id`。
|
||||
|
||||
@@ -49,8 +49,7 @@ pipeline {
|
||||
string(name: 'PROVISION_DOWNLOADS_DIR', defaultValue: 'provision-tool-downloads', description: 'Windows 下载阶段暂存 SpacetimeDB/otelcol 安装包的工作区相对目录')
|
||||
string(name: 'PROVISION_TOOLS_DIR', defaultValue: 'provision-tools', description: '目标机工作区内由已下载安装包生成的工具包目录')
|
||||
string(name: 'PROVISION_DOWNLOAD_PROXY', defaultValue: '', description: '可选，Windows 下载 SpacetimeDB 和 otelcol-contrib 时使用的代理地址，例如 http://127.0.0.1:7890；留空不设置代理')
|
||||
string(name: 'SPACETIME_INSTALLER_URL', defaultValue: 'https://install.spacetimedb.com', description: 'Windows 下载 SpacetimeDB 官方安装脚本的地址；目标机不访问该地址')
|
||||
string(name: 'SPACETIME_DOWNLOAD_ROOT', defaultValue: 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download', description: 'Windows 下载 SpacetimeDB Linux update installer 的根地址；目标机不访问该地址')
|
||||
string(name: 'SPACETIME_DOWNLOAD_ROOT', defaultValue: 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download', description: 'Windows 下载 SpacetimeDB Linux release tarball 的根地址；目标机不访问该地址')
|
||||
string(name: 'SPACETIME_TARGET_HOST', defaultValue: 'x86_64-unknown-linux-gnu', description: '目标机 SpacetimeDB 预编译包 host triple，development/release Linux amd64 使用默认值')
|
||||
string(name: 'SPACETIME_ROOT', defaultValue: '/stdb', description: 'SpacetimeDB root-dir')
|
||||
string(name: 'RELEASE_ROOT', defaultValue: '/opt/genarrative/releases', description: 'release 根目录')
|
||||
@@ -115,9 +114,6 @@ pipeline {
|
||||
if (!(params.OTELCOL_VERSION?.trim() ==~ /^[0-9]+\.[0-9]+\.[0-9]+$/)) {
|
||||
error("OTELCOL_VERSION 格式应为 x.y.z: ${params.OTELCOL_VERSION}")
|
||||
}
|
||||
if (!(params.SPACETIME_INSTALLER_URL?.trim() ==~ /^https?:\/\/\S+$/)) {
|
||||
error("SPACETIME_INSTALLER_URL 只能填写 http:// 或 https:// 开头的地址: ${params.SPACETIME_INSTALLER_URL}")
|
||||
}
|
||||
if (!(params.SPACETIME_DOWNLOAD_ROOT?.trim() ==~ /^https?:\/\/\S+$/)) {
|
||||
error('SPACETIME_DOWNLOAD_ROOT 不能为空。')
|
||||
}
|
||||
@@ -152,7 +148,6 @@ pipeline {
|
||||
$otelVersion = if ($env:OTELCOL_VERSION) { $env:OTELCOL_VERSION } else { '0.151.0' }
|
||||
$prepareOtel = if ($env:ENABLE_OTELCOL) { $env:ENABLE_OTELCOL } else { 'true' }
|
||||
$otelRoot = if ($env:OTELCOL_DOWNLOAD_ROOT) { $env:OTELCOL_DOWNLOAD_ROOT.TrimEnd('/') } else { 'https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download' }
|
||||
$spacetimeInstallerUrl = if ($env:SPACETIME_INSTALLER_URL) { $env:SPACETIME_INSTALLER_URL } else { 'https://install.spacetimedb.com' }
|
||||
$spacetimeDownloadRoot = if ($env:SPACETIME_DOWNLOAD_ROOT) { $env:SPACETIME_DOWNLOAD_ROOT.TrimEnd('/') } else { 'https://github.com/clockworklabs/SpacetimeDB/releases/latest/download' }
|
||||
$spacetimeTargetHost = if ($env:SPACETIME_TARGET_HOST) { $env:SPACETIME_TARGET_HOST } else { 'x86_64-unknown-linux-gnu' }
|
||||
$downloadProxy = if ($env:PROVISION_DOWNLOAD_PROXY) { $env:PROVISION_DOWNLOAD_PROXY } else { '' }
|
||||
@@ -165,11 +160,11 @@ pipeline {
|
||||
Write-Host "[prepare-provision-downloads] download dir: ${downloadsDir}"
|
||||
|
||||
if (Test-Path -LiteralPath $downloadsDir) {
|
||||
Write-Host "[prepare-provision-downloads] 清理旧下载目录: ${downloadsDir}"
|
||||
Remove-Item -LiteralPath $downloadsDir -Recurse -Force
|
||||
Write-Host "[prepare-provision-downloads] 复用已有下载目录: ${downloadsDir}"
|
||||
} else {
|
||||
New-Item -ItemType Directory -Force -Path $downloadsDir | Out-Null
|
||||
Write-Host "[prepare-provision-downloads] 已创建下载目录: ${downloadsDir}"
|
||||
}
|
||||
New-Item -ItemType Directory -Force -Path $downloadsDir | Out-Null
|
||||
Write-Host "[prepare-provision-downloads] 已创建下载目录: ${downloadsDir}"
|
||||
|
||||
if ($downloadProxy) {
|
||||
$env:HTTP_PROXY = $downloadProxy
|
||||
@@ -178,17 +173,94 @@ pipeline {
|
||||
Write-Host "[prepare-provision-downloads] 已配置 Windows 下载代理: $($downloadProxy -replace '://.*', '://***')"
|
||||
}
|
||||
|
||||
function Get-GithubReleaseAssetDigest {
|
||||
param(
|
||||
[Parameter(Mandatory=$true)][string]$Repository,
|
||||
[Parameter(Mandatory=$true)][string]$ReleaseSelector,
|
||||
[Parameter(Mandatory=$true)][string]$AssetName
|
||||
)
|
||||
|
||||
$request = @{
|
||||
Uri = "https://api.github.com/repos/${Repository}/${ReleaseSelector}"
|
||||
Headers = @{
|
||||
Accept = 'application/vnd.github+json'
|
||||
'User-Agent' = 'Genarrative-Server-Provision'
|
||||
}
|
||||
ErrorAction = 'Stop'
|
||||
}
|
||||
if ($downloadProxy) {
|
||||
$request.Proxy = $downloadProxy
|
||||
}
|
||||
|
||||
Write-Host "[prepare-provision-downloads] 查询 GitHub digest: repo=${Repository} release=${ReleaseSelector} asset=${AssetName}"
|
||||
$release = Invoke-RestMethod @request
|
||||
$asset = $release.assets | Where-Object { $_.name -eq $AssetName } | Select-Object -First 1
|
||||
if (-not $asset) {
|
||||
throw "[prepare-provision-downloads] GitHub release 未找到资产: ${Repository}/${AssetName}"
|
||||
}
|
||||
if (-not $asset.digest) {
|
||||
throw "[prepare-provision-downloads] GitHub release 未返回 digest: ${Repository}/${AssetName}"
|
||||
}
|
||||
Write-Host "[prepare-provision-downloads] GitHub digest ${AssetName}: $($asset.digest)"
|
||||
return $asset.digest
|
||||
}
|
||||
|
||||
function Get-FileDigest {
|
||||
param(
|
||||
[Parameter(Mandatory=$true)][string]$Path,
|
||||
[Parameter(Mandatory=$true)][string]$Algorithm
|
||||
)
|
||||
|
||||
$fileHash = Get-FileHash -Algorithm $Algorithm -LiteralPath $Path
|
||||
return $fileHash.Hash.ToLowerInvariant()
|
||||
}
|
||||
|
||||
function Test-DownloadDigestMatch {
|
||||
param(
|
||||
[Parameter(Mandatory=$true)][string]$Path,
|
||||
[Parameter(Mandatory=$true)][string]$ExpectedDigest
|
||||
)
|
||||
|
||||
$parts = $ExpectedDigest.Split(':', 2)
|
||||
if ($parts.Length -ne 2) {
|
||||
throw "[prepare-provision-downloads] 无法解析 GitHub digest: ${ExpectedDigest}"
|
||||
}
|
||||
$algorithm = $parts[0].Trim().ToLowerInvariant()
|
||||
$expectedHash = $parts[1].Trim().ToLowerInvariant()
|
||||
if ($algorithm -ne 'sha256') {
|
||||
throw "[prepare-provision-downloads] 暂不支持的 GitHub digest 算法: ${algorithm}"
|
||||
}
|
||||
$localHash = Get-FileDigest -Path $Path -Algorithm 'SHA256'
|
||||
return $localHash -eq $expectedHash
|
||||
}
|
||||
|
||||
function Invoke-ProvisionDownload {
|
||||
param(
|
||||
[Parameter(Mandatory=$true)][string]$Label,
|
||||
[Parameter(Mandatory=$true)][string]$Url,
|
||||
[Parameter(Mandatory=$true)][string]$Output
|
||||
[Parameter(Mandatory=$true)][string]$Output,
|
||||
[string]$ExpectedDigest = ''
|
||||
)
|
||||
|
||||
if ($ExpectedDigest) {
|
||||
if (Test-Path -LiteralPath $Output) {
|
||||
if (Test-DownloadDigestMatch -Path $Output -ExpectedDigest $ExpectedDigest) {
|
||||
$existingItem = Get-Item -LiteralPath $Output
|
||||
Write-Host "[prepare-provision-downloads] 已存在且校验一致，跳过下载: ${Label} bytes=$($existingItem.Length) path=${Output}"
|
||||
return
|
||||
}
|
||||
Write-Host "[prepare-provision-downloads] 已存在但校验不一致，重新下载: ${Label} path=${Output}"
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host "[prepare-provision-downloads] 下载 ${Label}: ${Url}"
|
||||
$tempOutput = "${Output}.download"
|
||||
if (Test-Path -LiteralPath $tempOutput) {
|
||||
Remove-Item -LiteralPath $tempOutput -Force
|
||||
}
|
||||
$curl = Get-Command curl.exe -ErrorAction SilentlyContinue
|
||||
if ($curl) {
|
||||
$arguments = @('-fL', '--retry', '3', '--retry-delay', '2', '-o', $Output)
|
||||
$arguments = @('-fL', '--retry', '3', '--retry-delay', '2', '-o', $tempOutput)
|
||||
if ($downloadProxy) {
|
||||
$arguments += @('--proxy', $downloadProxy)
|
||||
}
|
||||
@@ -196,46 +268,51 @@ pipeline {
|
||||
& $curl.Source @arguments
|
||||
$exitCode = $LASTEXITCODE
|
||||
if ($exitCode -ne 0) {
|
||||
throw "[prepare-provision-downloads] curl 下载失败: ${Label}, exit=${exitCode}"
|
||||
throw "[prepare-provision-downloads] curl 下载失败: ${Label}, exit=${exitCode}"
|
||||
}
|
||||
} else {
|
||||
$parameters = @{
|
||||
Uri = $Url
|
||||
OutFile = $tempOutput
|
||||
UseBasicParsing = $true
|
||||
}
|
||||
if ($downloadProxy) {
|
||||
$parameters.Proxy = $downloadProxy
|
||||
}
|
||||
Invoke-WebRequest @parameters
|
||||
}
|
||||
} else {
|
||||
$parameters = @{
|
||||
Uri = $Url
|
||||
OutFile = $Output
|
||||
UseBasicParsing = $true
|
||||
}
|
||||
if ($downloadProxy) {
|
||||
$parameters.Proxy = $downloadProxy
|
||||
}
|
||||
Invoke-WebRequest @parameters
|
||||
}
|
||||
|
||||
$item = Get-Item -LiteralPath $Output
|
||||
$item = Get-Item -LiteralPath $tempOutput
|
||||
if ($item.Length -le 0) {
|
||||
throw "[prepare-provision-downloads] 下载结果为空: ${Output}"
|
||||
throw "[prepare-provision-downloads] 下载结果为空: ${tempOutput}"
|
||||
}
|
||||
Write-Host "[prepare-provision-downloads] 已下载 ${Label}: bytes=$($item.Length) path=${Output}"
|
||||
if ($ExpectedDigest) {
|
||||
if (-not (Test-DownloadDigestMatch -Path $tempOutput -ExpectedDigest $ExpectedDigest)) {
|
||||
throw "[prepare-provision-downloads] 下载结果校验失败: ${Label}"
|
||||
}
|
||||
}
|
||||
Move-Item -LiteralPath $tempOutput -Destination $Output -Force
|
||||
$finalItem = Get-Item -LiteralPath $Output
|
||||
Write-Host "[prepare-provision-downloads] 已下载 ${Label}: bytes=$($finalItem.Length) path=${Output}"
|
||||
}
|
||||
|
||||
$installerPath = Join-Path $downloadsDir 'spacetime-install.sh'
|
||||
Invoke-ProvisionDownload -Label 'SpacetimeDB install script' -Url $spacetimeInstallerUrl -Output $installerPath
|
||||
|
||||
$spacetimeUpdateName = "spacetimedb-update-${spacetimeTargetHost}"
|
||||
$spacetimeUpdateUrl = "${spacetimeDownloadRoot}/${spacetimeUpdateName}"
|
||||
Invoke-ProvisionDownload -Label "SpacetimeDB Linux update installer ${spacetimeTargetHost}" -Url $spacetimeUpdateUrl -Output (Join-Path $downloadsDir $spacetimeUpdateName)
|
||||
$spacetimeArchiveName = "spacetime-${spacetimeTargetHost}.tar.gz"
|
||||
$spacetimeArchiveUrl = "${spacetimeDownloadRoot}/${spacetimeArchiveName}"
|
||||
$spacetimeArchiveDigest = Get-GithubReleaseAssetDigest -Repository 'clockworklabs/SpacetimeDB' -ReleaseSelector 'releases/latest' -AssetName $spacetimeArchiveName
|
||||
Invoke-ProvisionDownload -Label "SpacetimeDB release tarball ${spacetimeTargetHost}" -Url $spacetimeArchiveUrl -Output (Join-Path $downloadsDir $spacetimeArchiveName) -ExpectedDigest $spacetimeArchiveDigest
|
||||
|
||||
if ($prepareOtel -eq 'true') {
|
||||
$otelArchiveName = "otelcol-contrib_${otelVersion}_linux_amd64.tar.gz"
|
||||
$otelUrl = "${otelRoot}/v${otelVersion}/${otelArchiveName}"
|
||||
Invoke-ProvisionDownload -Label "otelcol-contrib ${otelVersion} linux amd64" -Url $otelUrl -Output (Join-Path $downloadsDir $otelArchiveName)
|
||||
$otelDigest = Get-GithubReleaseAssetDigest -Repository 'open-telemetry/opentelemetry-collector-releases' -ReleaseSelector "releases/tags/v${otelVersion}" -AssetName $otelArchiveName
|
||||
Invoke-ProvisionDownload -Label "otelcol-contrib ${otelVersion} linux amd64" -Url $otelUrl -Output (Join-Path $downloadsDir $otelArchiveName) -ExpectedDigest $otelDigest
|
||||
} else {
|
||||
Write-Host "[prepare-provision-downloads] ENABLE_OTELCOL=${prepareOtel}，跳过 otelcol-contrib 下载。"
|
||||
}
|
||||
|
||||
$utf8NoBom = New-Object System.Text.UTF8Encoding $false
|
||||
$manifest = @(
|
||||
"spacetime installer ${spacetimeInstallerUrl}",
|
||||
"spacetime update ${spacetimeDownloadRoot}/${spacetimeUpdateName}",
|
||||
"spacetime release tarball ${spacetimeArchiveUrl}",
|
||||
"spacetime target host ${spacetimeTargetHost}",
|
||||
"otelcol-contrib ${otelVersion} prepare=${prepareOtel}"
|
||||
)
|
||||
@@ -311,7 +388,6 @@ BASH
|
||||
OTELCOL_VERSION="${OTELCOL_VERSION:-0.151.0}" \
|
||||
PREPARE_OTELCOL="${ENABLE_OTELCOL:-true}" \
|
||||
PROVISION_REQUIRE_LOCAL_DOWNLOADS="true" \
|
||||
SPACETIME_INSTALLER_URL="${SPACETIME_INSTALLER_URL:-https://install.spacetimedb.com}" \
|
||||
SPACETIME_DOWNLOAD_ROOT="${SPACETIME_DOWNLOAD_ROOT:-https://github.com/clockworklabs/SpacetimeDB/releases/latest/download}" \
|
||||
SPACETIME_TARGET_HOST="${SPACETIME_TARGET_HOST:-x86_64-unknown-linux-gnu}" \
|
||||
scripts/prepare-server-provision-tools.sh
|
||||
|
||||
@@ -10,6 +10,7 @@ OTELCOL_ARCHIVE_PATH="${OTELCOL_ARCHIVE_PATH:-}"
|
||||
SPACETIME_INSTALLER_URL="${SPACETIME_INSTALLER_URL:-https://install.spacetimedb.com}"
|
||||
SPACETIME_DOWNLOAD_ROOT="${SPACETIME_DOWNLOAD_ROOT:-https://github.com/clockworklabs/SpacetimeDB/releases/latest/download}"
|
||||
SPACETIME_TARGET_HOST="${SPACETIME_TARGET_HOST:-x86_64-unknown-linux-gnu}"
|
||||
SPACETIME_ARCHIVE_PATH="${SPACETIME_ARCHIVE_PATH:-}"
|
||||
SPACETIME_INSTALLER_PATH="${SPACETIME_INSTALLER_PATH:-}"
|
||||
SPACETIME_UPDATE_INSTALLER_PATH="${SPACETIME_UPDATE_INSTALLER_PATH:-}"
|
||||
PROVISION_DOWNLOAD_PROXY="${PROVISION_DOWNLOAD_PROXY:-}"
|
||||
@@ -135,6 +136,9 @@ prepare_spacetime() {
|
||||
local tmp_dir="$1"
|
||||
local install_root="${tmp_dir}/spacetime-root"
|
||||
local target_dir="${PROVISION_TOOLS_DIR}/spacetime"
|
||||
local archive_name="spacetime-${SPACETIME_TARGET_HOST}.tar.gz"
|
||||
local downloaded_archive="${PROVISION_DOWNLOADS_DIR}/${archive_name}"
|
||||
local source_archive=""
|
||||
local update_name="spacetimedb-update-${SPACETIME_TARGET_HOST}"
|
||||
local downloaded_update="${PROVISION_DOWNLOADS_DIR}/${update_name}"
|
||||
local source_update=""
|
||||
@@ -143,17 +147,31 @@ prepare_spacetime() {
|
||||
local source_installer=""
|
||||
|
||||
mkdir -p "${install_root}"
|
||||
if [[ -n "${SPACETIME_UPDATE_INSTALLER_PATH}" && -f "${SPACETIME_UPDATE_INSTALLER_PATH}" ]]; then
|
||||
if [[ -n "${SPACETIME_ARCHIVE_PATH}" && -f "${SPACETIME_ARCHIVE_PATH}" ]]; then
|
||||
source_archive="${SPACETIME_ARCHIVE_PATH}"
|
||||
elif [[ -n "${PROVISION_DOWNLOADS_DIR}" && -f "${downloaded_archive}" ]]; then
|
||||
source_archive="${downloaded_archive}"
|
||||
fi
|
||||
|
||||
if [[ -n "${source_archive}" ]]; then
|
||||
echo "[prepare-provision-tools] 使用已下载的 SpacetimeDB release tarball: ${source_archive}"
|
||||
mkdir -p "${install_root}/bin/current"
|
||||
tar -xzf "${source_archive}" -C "${install_root}/bin/current"
|
||||
chmod 0755 "${install_root}/bin/current/spacetimedb-cli" "${install_root}/bin/current/spacetimedb-standalone"
|
||||
SPACETIME_SOURCE_DESCRIPTION="local archive ${source_archive}"
|
||||
elif [[ -n "${SPACETIME_UPDATE_INSTALLER_PATH}" && -f "${SPACETIME_UPDATE_INSTALLER_PATH}" ]]; then
|
||||
source_update="${SPACETIME_UPDATE_INSTALLER_PATH}"
|
||||
elif [[ -n "${PROVISION_DOWNLOADS_DIR}" && -f "${downloaded_update}" ]]; then
|
||||
source_update="${downloaded_update}"
|
||||
fi
|
||||
if [[ "${PROVISION_REQUIRE_LOCAL_DOWNLOADS}" == "true" && -z "${source_update}" ]]; then
|
||||
echo "[prepare-provision-tools] 要求使用 Windows 已下载的 SpacetimeDB Linux update installer，但未找到: ${downloaded_update}" >&2
|
||||
if [[ "${PROVISION_REQUIRE_LOCAL_DOWNLOADS}" == "true" && -z "${source_archive}" ]]; then
|
||||
echo "[prepare-provision-tools] 要求使用 Windows 已下载的 SpacetimeDB release tarball，但未找到: ${downloaded_archive}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -n "${source_update}" ]]; then
|
||||
if [[ -n "${source_archive}" ]]; then
|
||||
:
|
||||
elif [[ -n "${source_update}" ]]; then
|
||||
echo "[prepare-provision-tools] 使用已下载的 SpacetimeDB Linux update installer: ${source_update}"
|
||||
cp "${source_update}" "${prepared_update}"
|
||||
chmod 0755 "${prepared_update}"
|
||||
@@ -205,6 +223,7 @@ main() {
|
||||
require_cmd install
|
||||
require_cmd mktemp
|
||||
require_cmd rm
|
||||
require_cmd tar
|
||||
|
||||
validate_relative_dir "PROVISION_TOOLS_DIR" "${PROVISION_TOOLS_DIR}"
|
||||
validate_relative_dir "PROVISION_DOWNLOADS_DIR" "${PROVISION_DOWNLOADS_DIR}"
|
||||
|
||||
Reference in New Issue
Block a user