预热Gitea CI项目依赖缓存

将npm与Rust依赖下载缓存预热到固定CI镜像改为直接从Gitea checkout并增加网络重试收窄镜像构建上下文并补齐离线与运行时校验同步Runner切换边界和运维文档
2026-07-22 15:44:40 +08:00
parent 2e02c28d0a
commit f4f7d8ac20
11 changed files with 348 additions and 51 deletions
@@ -67,13 +67,13 @@ Linux Docker Engine 若要从宿主机 CLI 连到容器内服务，直接用 `ht
 ```bash
 bash scripts/gitea-ci-job-image.sh build
 bash scripts/gitea-ci-job-image.sh verify
-bash scripts/gitea-ci-job-image.sh export /仓库外受控路径/genarrative-gitea-project-ci-20260722.1.tar.zst
+bash scripts/gitea-ci-job-image.sh export /仓库外受控路径/genarrative-gitea-project-ci-20260722.2.tar.zst
 bash scripts/gitea-ci-job-image.sh load-runner
 ```

-默认构建 tag 为 `genarrative/gitea-project-ci:20260722.1`。脚本将 build context 固定为 `deploy/container`，不会把整个工作区发送给 Docker daemon；`build` 完成后会自动运行环境校验，`load-runner` 还会比对宿主和 runner 内层的完整 Image ID，并在内层执行 bwrap 与 Chrome headless canary。当前验证镜像约 `1.416 GB`，完整 Image ID 为 `sha256:62304a788905cec805bf0ae3638c2811bf530b2f5709523277d816edb4772942`。执行这些命令不要求必须使用 root，但执行账号必须有权访问宿主 Docker API 并管理 runner 容器；没有该权限时交给 runner 运维人员执行。
+默认构建 tag 为 `genarrative/gitea-project-ci:20260722.2`。脚本通过 NUL 分隔白名单 tar 流只发送 Dockerfile、checkout 脚本、npm manifests/lock 和 Cargo manifests/lock；当前构建 context 约 `1.638 MB`，不会把业务源码、素材或本地私密文件发送给 Docker daemon。镜像除固定工具链外，还按当前根 `package-lock.json`、`server-rs/Cargo.lock` 和桌面壳 `Cargo.lock` 预热 npm / Cargo 下载缓存，并在构建阶段执行离线完整性校验；不包含 `node_modules` 或 Cargo `target`。`build` 完成后会自动运行环境校验，`load-runner` 还会比对宿主和 runner 内层的完整 Image ID，并在内层执行 bwrap 与 Chrome headless canary。当前验证镜像约 `1.788 GB`，完整 Image ID 为 `sha256:548431a2529d325b5ab546f242799a4076f979779ee832a0871ac1af881a4946`。执行这些命令不要求必须使用 root，但执行账号必须有权访问宿主 Docker API 并管理 runner 容器；没有该权限时交给 runner 运维人员执行。

-runner 配置保留原 `ubuntu-latest` 映射，另外增加 `genarrative-ci:docker://sha256:62304a788905cec805bf0ae3638c2811bf530b2f5709523277d816edb4772942`。内层 Docker 数据必须持久化，`force_pull` 保持 `false`；该精确 Image ID 在内层不存在时 job 应直接失败，不回退到浮动 tag 或现场拉取。四个 job 开头均以 `GENARRATIVE_GITEA_CI_CHECK_RUNTIME=1` 执行 `scripts/check-gitea-ci-job-image.sh`，同时验工具链、bwrap 和 Chrome headless；它们仍各自运行 `npm ci` 以校验当前 lockfile 并隔离 PR 依赖，不把 `node_modules` 烘进镜像，本阶段也不启用 Actions cache。
+runner 配置保留原 `ubuntu-latest` 映射，另外增加 `genarrative-ci:docker://sha256:548431a2529d325b5ab546f242799a4076f979779ee832a0871ac1af881a4946`。内层 Docker 数据必须持久化，`force_pull` 保持 `false`；该精确 Image ID 在内层不存在时 job 应直接失败，不回退到浮动 tag 或现场拉取。四个 job 使用镜像内 `genarrative-gitea-checkout` 直接从当前 Gitea 拉取事件 commit，带 5 次有界重试，不再运行时下载 GitHub checkout action；随后以 `GENARRATIVE_GITEA_CI_CHECK_RUNTIME=1` 执行 `scripts/check-gitea-ci-job-image.sh`，同时校验工具链、缓存锁命中状态、bwrap 和 Chrome headless。它们仍各自运行干净的 `npm ci` 以校验当前 lockfile 并隔离 PR 依赖，但使用镜像内 npm cache 和 `prefer-offline`；锁文件新增依赖时允许经受控网络补齐，本阶段不启用共享 Actions cache。

 更新顺序固定为：

@@ -83,6 +83,8 @@ runner 配置保留原 `ubuntu-latest` 映射，另外增加 `genarrative-ci:doc
 4. 增加或替换 `genarrative-ci` 的精确 `docker://<Image ID>` 映射，然后执行 `docker restart --timeout 660 gitea-runner`。
 5. 重跑真实 PR 的四个 CI job；全部通过且隔离边界复核完成后，才能清理旧镜像。

+`docker restart --timeout 660` 只提供容器停止宽限，不是 Runner drain API；rootless DinD 的 supervisor 可能与 runner 同时停止内层 dockerd。重启前必须同时确认 Gitea 没有 `in_progress` run 且内层 `docker ps` 为空，不能依赖该 timeout 等待活跃 job。
+
 回滚时先把 workflow 的 `runs-on` 改回 `ubuntu-latest`，再恢复备份的 runner config 并用同一超时重启 runner。不要在真实 CI 验证前删除旧映射或旧镜像。

 ## 启动与验证
@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+workspace="${GITHUB_WORKSPACE:?GITHUB_WORKSPACE is required}"
+repository="${GITHUB_REPOSITORY:?GITHUB_REPOSITORY is required}"
+server_url="${GITHUB_SERVER_URL:?GITHUB_SERVER_URL is required}"
+event_path="${GITHUB_EVENT_PATH:?GITHUB_EVENT_PATH is required}"
+fetch_depth="${GENARRATIVE_GITEA_FETCH_DEPTH:-1}"
+repository_url="${GENARRATIVE_GITEA_REPOSITORY_URL:-${server_url%/}/${repository}.git}"
+
+if [[ ! "${fetch_depth}" =~ ^[0-9]+$ ]]; then
+  echo "GENARRATIVE_GITEA_FETCH_DEPTH must be a non-negative integer: ${fetch_depth}" >&2
+  exit 2
+fi
+if [[ -z "${workspace}" || "${workspace}" == '/' ]]; then
+  echo "refusing to clean unsafe workspace: ${workspace}" >&2
+  exit 2
+fi
+
+readarray -t event_checkout < <(
+  node - "${event_path}" <<'NODE'
+const fs = require('node:fs');
+
+const event = JSON.parse(fs.readFileSync(process.argv[2], 'utf8'));
+const targetSha = event.pull_request?.head?.sha ?? event.after ?? process.env.GITHUB_SHA ?? '';
+const baseRef = event.pull_request?.base?.ref ?? process.env.GITHUB_BASE_REF ?? 'master';
+process.stdout.write(`${targetSha}\n${baseRef}\n`);
+NODE
+)
+target_sha="${event_checkout[0]:-}"
+base_ref="${event_checkout[1]:-master}"
+
+if [[ ! "${target_sha}" =~ ^[0-9a-fA-F]{40,64}$ ]]; then
+  echo "event did not provide a valid checkout commit: ${target_sha}" >&2
+  exit 1
+fi
+git check-ref-format --branch "${base_ref}" >/dev/null
+
+mkdir -p "${workspace}"
+find "${workspace}" -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +
+git config --global --add safe.directory "${workspace}"
+git -c init.defaultBranch=master init "${workspace}"
+git -C "${workspace}" remote add origin "${repository_url}"
+git -C "${workspace}" config gc.auto 0
+git -C "${workspace}" config advice.detachedHead false
+
+fetch_args=(
+  -c protocol.version=2
+  fetch
+  --no-tags
+  --prune
+  --no-recurse-submodules
+)
+if [[ "${fetch_depth}" -gt 0 ]]; then
+  fetch_args+=(--depth="${fetch_depth}")
+fi
+fetch_refspecs=(
+  "+${target_sha}:refs/remotes/ci/head"
+)
+if [[ "${fetch_depth}" -eq 0 ]]; then
+  fetch_refspecs+=(
+    "+refs/heads/${base_ref}:refs/remotes/origin/${base_ref}"
+  )
+fi
+
+auth_header=''
+if [[ -n "${GENARRATIVE_GITEA_TOKEN:-}" ]]; then
+  auth_header="AUTHORIZATION: basic $(
+    printf 'x-access-token:%s' "${GENARRATIVE_GITEA_TOKEN}" | base64 -w 0
+  )"
+fi
+
+for attempt in $(seq 1 5); do
+  if [[ -n "${auth_header}" ]]; then
+    if GIT_CONFIG_COUNT=1 \
+      GIT_CONFIG_KEY_0="http.${repository_url%/}.extraheader" \
+      GIT_CONFIG_VALUE_0="${auth_header}" \
+      git -C "${workspace}" "${fetch_args[@]}" origin "${fetch_refspecs[@]}"; then
+      break
+    fi
+  elif git -C "${workspace}" "${fetch_args[@]}" origin "${fetch_refspecs[@]}"; then
+    break
+  fi
+  if [[ "${attempt}" -eq 5 ]]; then
+    echo 'Gitea checkout fetch failed after 5 attempts.' >&2
+    exit 1
+  fi
+  sleep $((attempt * 2))
+done
+
+git -C "${workspace}" checkout --force refs/remotes/ci/head
+git -C "${workspace}" clean -ffdx
+test "$(git -C "${workspace}" rev-parse HEAD)" = "${target_sha,,}"
+printf 'checked_out_commit=%s\n' "${target_sha,,}"
@@ -8,6 +8,35 @@ RUN rustup component add rustfmt \
    && cargo --version \
    && rustfmt --version

+FROM rust-toolchain AS rust-dependency-cache
+
+ENV CARGO_HTTP_MULTIPLEXING=false \
+    CARGO_NET_RETRY=10
+
+COPY server-rs /tmp/genarrative-cargo-cache/server-rs
+COPY apps/desktop-shell/src-tauri /tmp/genarrative-cargo-cache/desktop-shell
+
+RUN find /tmp/genarrative-cargo-cache -name Cargo.toml -exec dirname {} \; \
+        | while IFS= read -r crate_dir; do \
+            mkdir -p "${crate_dir}/src"; \
+            : > "${crate_dir}/src/lib.rs"; \
+            : > "${crate_dir}/src/main.rs"; \
+            : > "${crate_dir}/src/active.rs"; \
+        done \
+    && cargo fetch --locked \
+        --target x86_64-unknown-linux-gnu \
+        --manifest-path /tmp/genarrative-cargo-cache/server-rs/Cargo.toml \
+    && cargo fetch --locked \
+        --target x86_64-unknown-linux-gnu \
+        --manifest-path /tmp/genarrative-cargo-cache/desktop-shell/Cargo.toml \
+    && CARGO_NET_OFFLINE=true cargo fetch --locked \
+        --target x86_64-unknown-linux-gnu \
+        --manifest-path /tmp/genarrative-cargo-cache/server-rs/Cargo.toml \
+    && CARGO_NET_OFFLINE=true cargo fetch --locked \
+        --target x86_64-unknown-linux-gnu \
+        --manifest-path /tmp/genarrative-cargo-cache/desktop-shell/Cargo.toml \
+    && rm -rf /tmp/genarrative-cargo-cache
+
 FROM ${RUNNER_IMAGE}

 ARG NODE_VERSION=22.23.1
@@ -16,8 +45,8 @@ ARG GOOGLE_CHROME_VERSION=150.0.7871.181-1
 ARG GOOGLE_LINUX_SIGNING_KEY_FINGERPRINT=EB4C1BFD4F042F6DDDCCEC917721F63BD38B4796

 LABEL org.opencontainers.image.title="Genarrative Gitea CI job image" \
-    org.opencontainers.image.description="Ubuntu 24.04 CI image with Node 22, Rust 1.96, Chrome, and native build dependencies" \
-    org.opencontainers.image.version="2026.07.22.1"
+    org.opencontainers.image.description="Ubuntu 24.04 CI image with fixed toolchains and prewarmed npm/Cargo caches" \
+    org.opencontainers.image.version="2026.07.22.2"

 RUN test "$(dpkg --print-architecture)" = "amd64" \
    && find /etc/apt/sources.list.d -maxdepth 1 -type f \
@@ -80,8 +109,42 @@ RUN node_archive="node-v${NODE_VERSION}-linux-x64.tar.xz" \
    && ln -sfn /usr/local/lib/genarrative-node/bin/npx /usr/local/bin/npx \
    && ln -sfn /usr/local/lib/genarrative-node/bin/corepack /usr/local/bin/corepack

-COPY --from=rust-toolchain /usr/local/cargo /usr/local/cargo
-COPY --from=rust-toolchain /usr/local/rustup /usr/local/rustup
+COPY --from=rust-dependency-cache /usr/local/cargo /usr/local/cargo
+COPY --from=rust-dependency-cache /usr/local/rustup /usr/local/rustup
+
+ARG NPM_LOCK_SHA256
+ARG SERVER_RUST_LOCK_SHA256
+ARG DESKTOP_RUST_LOCK_SHA256
+
+COPY package.json package-lock.json /usr/local/share/genarrative-ci/npm/
+COPY server-rs/Cargo.lock /usr/local/share/genarrative-ci/locks/server-rs.Cargo.lock
+COPY apps/desktop-shell/src-tauri/Cargo.lock /usr/local/share/genarrative-ci/locks/desktop-shell.Cargo.lock
+COPY deploy/container/gitea-ci-checkout.sh /usr/local/bin/genarrative-gitea-checkout
+
+RUN test -n "${NPM_LOCK_SHA256}" \
+    && test -n "${SERVER_RUST_LOCK_SHA256}" \
+    && test -n "${DESKTOP_RUST_LOCK_SHA256}" \
+    && printf '%s  %s\n' \
+        "${NPM_LOCK_SHA256}" \
+        /usr/local/share/genarrative-ci/npm/package-lock.json \
+        | sha256sum --check --strict \
+    && printf '%s  %s\n' \
+        "${SERVER_RUST_LOCK_SHA256}" \
+        /usr/local/share/genarrative-ci/locks/server-rs.Cargo.lock \
+        | sha256sum --check --strict \
+    && printf '%s  %s\n' \
+        "${DESKTOP_RUST_LOCK_SHA256}" \
+        /usr/local/share/genarrative-ci/locks/desktop-shell.Cargo.lock \
+        | sha256sum --check --strict \
+    && chmod 0755 /usr/local/bin/genarrative-gitea-checkout \
+    && npm ci \
+        --ignore-scripts \
+        --no-audit \
+        --no-fund \
+        --prefer-offline \
+        --prefix /usr/local/share/genarrative-ci/npm \
+    && rm -rf /usr/local/share/genarrative-ci/npm/node_modules \
+    && npm cache verify

 RUN install -m 0755 /usr/local/cargo/bin/rustup /usr/local/bin/rustup \
    && for command_name in cargo cargo-fmt rustc rustdoc rustfmt; do \
@@ -89,6 +152,16 @@ RUN install -m 0755 /usr/local/cargo/bin/rustup /usr/local/bin/rustup \
    done

 ENV CARGO_HOME=/usr/local/cargo \
+    GENARRATIVE_GITEA_CI_DESKTOP_RUST_LOCK_SHA256=${DESKTOP_RUST_LOCK_SHA256} \
+    GENARRATIVE_GITEA_CI_NPM_LOCK_SHA256=${NPM_LOCK_SHA256} \
+    GENARRATIVE_GITEA_CI_SERVER_RUST_LOCK_SHA256=${SERVER_RUST_LOCK_SHA256} \
+    NPM_CONFIG_AUDIT=false \
+    NPM_CONFIG_FETCH_RETRIES=10 \
+    NPM_CONFIG_FETCH_RETRY_FACTOR=2 \
+    NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT=60000 \
+    NPM_CONFIG_FETCH_RETRY_MINTIMEOUT=2000 \
+    NPM_CONFIG_FUND=false \
+    NPM_CONFIG_PREFER_OFFLINE=true \
    RUSTUP_AUTO_INSTALL=0 \
    RUSTUP_HOME=/usr/local/rustup \
    PATH=/usr/local/lib/genarrative-node/bin:/usr/local/bin:/usr/local/cargo/bin:/opt/acttoolcache/node/24.18.0/x64/bin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
@@ -109,5 +182,9 @@ RUN node --version \
    && rustfmt --version \
    && google-chrome --version \
    && bwrap --version \
+    && test -x /usr/local/bin/genarrative-gitea-checkout \
+    && bash -n /usr/local/bin/genarrative-gitea-checkout \
+    && test -d /root/.npm/_cacache \
+    && test -d /usr/local/cargo/registry/cache \
    && rg --version \
    && ffmpeg -version | head -n 1
@@ -0,0 +1,18 @@
+**
+!deploy/
+!deploy/container/
+!deploy/container/gitea-ci-job.Dockerfile
+!deploy/container/gitea-ci-checkout.sh
+!package.json
+!package-lock.json
+!server-rs/
+!server-rs/Cargo.toml
+!server-rs/Cargo.lock
+!server-rs/crates/
+!server-rs/crates/*/
+!server-rs/crates/*/Cargo.toml
+!apps/
+!apps/desktop-shell/
+!apps/desktop-shell/src-tauri/
+!apps/desktop-shell/src-tauri/Cargo.toml
+!apps/desktop-shell/src-tauri/Cargo.lock