Merge branch 'master' into codex/game-distribution
Project CI / AI game creator shell Rust lane 1/2 (pull_request) Has been cancelled
Project CI / AI game creator shell Rust lane 2/2 (pull_request) Has been cancelled
Project CI / AI game creator shell Rust smoke (pull_request) Has been cancelled
Project CI / AI game creator shell Rust crates (pull_request) Has been cancelled
Project CI / Backend tests (pull_request) Has been cancelled
Project CI / Native shell tests (pull_request) Has been cancelled
Project CI / Frontend tests (pull_request) Has been cancelled
Project CI / Repository checks (pull_request) Has been cancelled
Project CI / AI game creator shell web tests (pull_request) Has been cancelled
Project CI / AI game creator shell Rust lane 1/2 (pull_request) Has been cancelled
Project CI / AI game creator shell Rust lane 2/2 (pull_request) Has been cancelled
Project CI / AI game creator shell Rust smoke (pull_request) Has been cancelled
Project CI / AI game creator shell Rust crates (pull_request) Has been cancelled
Project CI / Backend tests (pull_request) Has been cancelled
Project CI / Native shell tests (pull_request) Has been cancelled
Project CI / Frontend tests (pull_request) Has been cancelled
Project CI / Repository checks (pull_request) Has been cancelled
Project CI / AI game creator shell web tests (pull_request) Has been cancelled
This commit is contained in:
@@ -98,6 +98,12 @@ jobs:
|
||||
- name: Report isolated Rust compilation cache
|
||||
if: always()
|
||||
run: bash scripts/ci-rust-cache.sh report
|
||||
- name: Publish master Rust cache artifact
|
||||
continue-on-error: true
|
||||
if: ${{ !cancelled() && github.event_name == 'push' && github.ref == 'refs/heads/master' }}
|
||||
env:
|
||||
GENARRATIVE_GITEA_TOKEN: ${{ github.token }}
|
||||
run: python3 scripts/export-gitea-rust-cache.py
|
||||
|
||||
ai-game-creator-shell-rust-lane-2:
|
||||
name: AI game creator shell Rust lane 2/2
|
||||
@@ -141,6 +147,12 @@ jobs:
|
||||
- name: Report isolated Rust compilation cache
|
||||
if: always()
|
||||
run: bash scripts/ci-rust-cache.sh report
|
||||
- name: Publish master Rust cache artifact
|
||||
continue-on-error: true
|
||||
if: ${{ !cancelled() && github.event_name == 'push' && github.ref == 'refs/heads/master' }}
|
||||
env:
|
||||
GENARRATIVE_GITEA_TOKEN: ${{ github.token }}
|
||||
run: python3 scripts/export-gitea-rust-cache.py
|
||||
|
||||
# agent-run smoke 会 spawn `cargo run`(走壳自己的 manifest),同样不装 npm 依赖,
|
||||
# 单独一个 job,免得把已经压到 4 分钟级的片 job 拖长。
|
||||
@@ -183,6 +195,12 @@ jobs:
|
||||
- name: Report isolated Rust compilation cache
|
||||
if: always()
|
||||
run: bash scripts/ci-rust-cache.sh report
|
||||
- name: Publish master Rust cache artifact
|
||||
continue-on-error: true
|
||||
if: ${{ !cancelled() && github.event_name == 'push' && github.ref == 'refs/heads/master' }}
|
||||
env:
|
||||
GENARRATIVE_GITEA_TOKEN: ${{ github.token }}
|
||||
run: python3 scripts/export-gitea-rust-cache.py
|
||||
|
||||
# AGC 壳依赖的共享 / 平台和编辑器插件 crate 各自预热独立 manifest,再运行对应测试。
|
||||
ai-game-creator-shell-rust-crates:
|
||||
@@ -291,6 +309,12 @@ jobs:
|
||||
- name: Report isolated Rust compilation cache
|
||||
if: always()
|
||||
run: bash scripts/ci-rust-cache.sh report
|
||||
- name: Publish master Rust cache artifact
|
||||
continue-on-error: true
|
||||
if: ${{ !cancelled() && github.event_name == 'push' && github.ref == 'refs/heads/master' }}
|
||||
env:
|
||||
GENARRATIVE_GITEA_TOKEN: ${{ github.token }}
|
||||
run: python3 scripts/export-gitea-rust-cache.py
|
||||
|
||||
backend-tests:
|
||||
name: Backend tests
|
||||
@@ -380,6 +404,12 @@ jobs:
|
||||
- name: Report isolated Rust compilation cache
|
||||
if: always()
|
||||
run: bash scripts/ci-rust-cache.sh report
|
||||
- name: Publish master Rust cache artifact
|
||||
continue-on-error: true
|
||||
if: ${{ !cancelled() && github.event_name == 'push' && github.ref == 'refs/heads/master' }}
|
||||
env:
|
||||
GENARRATIVE_GITEA_TOKEN: ${{ github.token }}
|
||||
run: python3 scripts/export-gitea-rust-cache.py
|
||||
|
||||
# 客户端的壳级与契约门禁:静态契约断言、H5 / 微信 / 移动 / 桌面壳运行时门禁,
|
||||
# 以及依赖发布产物的构建 smoke。
|
||||
@@ -442,6 +472,12 @@ jobs:
|
||||
- name: Report isolated Rust compilation cache
|
||||
if: always()
|
||||
run: bash scripts/ci-rust-cache.sh report
|
||||
- name: Publish master Rust cache artifact
|
||||
continue-on-error: true
|
||||
if: ${{ !cancelled() && github.event_name == 'push' && github.ref == 'refs/heads/master' }}
|
||||
env:
|
||||
GENARRATIVE_GITEA_TOKEN: ${{ github.token }}
|
||||
run: python3 scripts/export-gitea-rust-cache.py
|
||||
|
||||
frontend-tests:
|
||||
name: Frontend tests
|
||||
@@ -524,6 +560,9 @@ jobs:
|
||||
- name: Install npm dependencies
|
||||
run: bash scripts/ci-npm-ci-with-retry.sh
|
||||
|
||||
- name: Validate CI cache maintenance behavior
|
||||
run: python3 -m unittest discover -s scripts -p 'test_gitea_cache_*.py'
|
||||
|
||||
- name: Run repository checks
|
||||
run: npm run check:repository-ci
|
||||
|
||||
|
||||
@@ -89,7 +89,7 @@ runner 配置保留原 `ubuntu-latest` 映射,`genarrative-ci` 继续映射到
|
||||
2. 执行 `load-runner`,确认该 ID 已进入 runner 内层 Docker。
|
||||
3. 确认没有活跃 job,将当前 runner config 备份到仓库外的受控位置;备份不得进入 Git,也不得在文档或日志中回显注册信息。
|
||||
4. 增加或替换 `genarrative-ci` 的精确 `docker://<Image ID>` 映射,然后执行 `docker restart --timeout 660 gitea-runner`。
|
||||
5. 重跑真实 PR 的四个 CI job;全部通过且隔离边界复核完成后,才能清理旧镜像。
|
||||
5. 用真实 CI 的九个 job 验证新镜像;全部通过且隔离边界复核完成后,才能清理旧镜像。
|
||||
|
||||
`docker restart --timeout 660` 只提供容器停止宽限,不是 Runner drain API;rootless DinD 的 supervisor 可能与 runner 同时停止内层 dockerd。重启前必须同时确认 Gitea 没有 `in_progress` run 且内层 `docker ps` 为空,不能依赖该 timeout 等待活跃 job。
|
||||
|
||||
@@ -97,9 +97,15 @@ runner 配置保留原 `ubuntu-latest` 映射,`genarrative-ci` 继续映射到
|
||||
|
||||
### Rust 测试组编译对象快照
|
||||
|
||||
`scripts/build-gitea-rust-cache.sh` 在已验证的 job 镜像上生成候选镜像,覆盖 AGC Rust 两条 lane、crates、agent-run smoke、Backend 和 Native shell 的桌面壳测试。Native shell 的 release build smoke 显式清空两个 wrapper,不消费测试快照。固定 sccache `0.18.0` 的 Linux x64 musl 归档并校验 SHA-256,维护者从 origin/master 的确定提交预热编译对象,PR job 没有生成/发布公共快照的权限。
|
||||
自动维护由宿主 systemd timer 调用 `scripts/maintain-gitea-rust-cache.py`,只管理 Gitea CI 测试镜像,不修改 Jenkins、生产发布、本地开发或客户端发行构建。六个 Rust job 仅在 master push 中导出本次 CI 新增的 sccache 对象;已命中的继承对象只上传新近使用时间,通过 Gitea 原生 V4 artifact 接口上传;PR 不发布。维护器选择已结束且六组产物完整的最新 master run,校验提交、任务尝试、工具链与来源镜像,与六组实际使用的同一镜像快照合并去重,并按新近使用时间限制快照总容量为 4 GiB,然后从无对象缓存基础镜像组装新镜像,**不重复执行 Cargo 预热编译,也不要求源 run 事先全绿**。缺组、取消或校验失败时保留现役版,不混合不同 run 的对象来假装完整快照。
|
||||
|
||||
基础镜像必须不含 `/opt/genarrative-ci/rust-cache`;脚本在拉取源码、下载工具和预热前执行只读、断网检查,发现已有对象快照就拒绝构建。不能在旧缓存镜像上删除目录再叠加新快照,删除操作不会释放旧镜像层。切换且真实 CI 验证通过后,人工定向清理更旧的缓存镜像与导出归档,保留当前版、一个回滚版及运行中 CI 使用的版本;对象缓存上限不覆盖这些宿主文件,不使用全局 prune。
|
||||
切换先通过专属入口阻断新的 FetchTask,确认已转发的领取请求全部收到完整上游响应,并检查入口持久化跟踪的已领取任务全部结束、内层 Docker 没有活动容器。任务终态必须依据 Runner 的执行结束及最终上报协议,不能由容器暂时为空、API 已取消或请求超时推断。有任务即恢复领取并延后,不停止任务;状态未知拒绝切换。维护器只需普通账号的 `write:repository` Token(包括查询、下载及定向删除 artifact),不访问全局 Runner 管理 API。切换后等待使用该 Image ID 的完整真实 master push CI 通过,才允许下一次升级及旧镜像清理;不会自动重跑失败用例或为了验收额外触发整轮 CI。首次接管的历史镜像默认不归自动清理管理。
|
||||
|
||||
维护状态、凭据、归档和配置备份保存在仓库外。当前版、回滚版、待验证候选、它们的基础镜像及容器引用的镜像均受保护。清理只针对维护器登记的专属 tag、完整 Image ID 和专用目录中的归档;禁止全局 prune。API、构建、验证或空闲检查失败时保留现役镜像与回滚资料,不以失败重跑制造全绿结果。
|
||||
|
||||
`scripts/build-gitea-rust-cache.sh` 仅用于首次缺少可消费快照时的人工 bootstrap,在已验证的 job 镜像上生成候选镜像,覆盖 AGC Rust 两条 lane、crates、agent-run smoke、Backend 和 Native shell 的桌面壳测试。Native shell 的 release build smoke 显式清空两个 wrapper,不消费测试快照。固定 sccache `0.18.0` 的 Linux x64 musl 归档并校验 SHA-256,维护者从 origin/master 的确定提交预热编译对象,PR job 没有生成/发布公共快照的权限。
|
||||
|
||||
基础镜像必须不含 `/opt/genarrative-ci/rust-cache`;脚本在拉取源码、下载工具和预热前执行只读、断网检查,发现已有对象快照就拒绝构建。不能在旧缓存镜像上删除目录再叠加新快照,删除操作不会释放旧镜像层。切换且真实 CI 验证通过后,维护器定向清理自己登记的更旧缓存镜像与导出归档,保留当前版、一个回滚版、所需基础镜像及容器引用的版本;接管前的试验镜像/归档仍由维护者确认后人工清理。对象缓存上限不覆盖这些宿主文件,不使用全局 prune,也不清理其它构建的 Docker build cache。
|
||||
|
||||
```bash
|
||||
bash scripts/build-gitea-rust-cache.sh genarrative/gitea-project-ci:20260920.2 genarrative/gitea-project-ci:rust-cache-candidate
|
||||
@@ -108,9 +114,52 @@ bash scripts/gitea-ci-job-image.sh export /仓库外受控路径/ci-rust-cache.t
|
||||
bash scripts/gitea-ci-job-image.sh load-runner genarrative/gitea-project-ci:rust-cache-candidate
|
||||
```
|
||||
|
||||
预热容器上限为 4 核、12 GiB,移除 capabilities,不挂宿主目录/socket,也不注入 Git/OSS/Jenkins 凭据。源码通过 `git archive` 复制,当前工作区、ignored 文件和 `.git` 不进入容器。最终从原镜像重新组装,仅复制 `/opt/genarrative-ci/rust-cache` 的 sccache、对象和来源元数据,不提交含源码/target 的预热容器;镜像本身的下载缓存与工具链校验保持原样。
|
||||
自动维护不调用预热脚本;以下人工 bootstrap 脚本接受第三个参数 `<完整 master SHA>`。脚本确认该 SHA 是抓取到的 master 祖先,源码和 `ci-rust-cache.sh` 均来自该提交;两参数人工调用仍默认使用最新 master。基础镜像的 `revision` 由同一份 build context 清单计算;输入变化时维护器先生成新的无对象缓存基础镜像。Rust 输入指纹保守包含代码、配置、资源、脚本和内嵌 skill,仅排除一般 `docs/` 及几个根说明文档,`docs/openapi/` 始终参与。
|
||||
|
||||
快照由固定 Image ID 分发,每个 job 仅修改容器自己的写时复制层,缓存上限 4 GiB,构建末尾输出实际对象体积,结束后不回传。`ci-rust-cache.sh prepare` 清空继承的 `SCCACHE_*` 远程配置,使用独立配置和 Unix socket;旧镜像、工具链不匹配或限时 wrapper 探测失败时使用直接 rustc,正式编译启用 sccache 的 server IO 错误回退。真实编译/测试失败保留非零退出码。`report` 输出命中统计并停止本 job daemon,分片日志输出独立编译耗时。sccache 0.18.0 的只读模式在 miss 后仍打包再拒绝写入,不能用它宣称零 miss 开销;普通 CI 继续只向容器层写入。
|
||||
缓存产物通过六个 `Publish master Rust cache artifact` 步骤发布,daemon 成功停止才导出。源 run 必须结束,且六组为 success/failure 并有完整上传证据;整轮或任一 Rust 组取消、旧 attempt、缺组、混用来源镜像均不采用。上传失败仅告警,不改变测试结论。每组仅导出相对其启动快照的新 key,命中只上报触达时间;宿主按真实 job 日志确认同一来源 Image ID,再复用其对象,不信任产物中的可执行文件。合并校验 tar/zip 路径、对象大小与 SHA256,同 key 内容冲突拒绝发布。sccache 可执行文件取自可信来源镜像,不从 artifact 执行代码。
|
||||
|
||||
候选已校验、导出并装载内层 Docker 后,可定向删除该 run 已收集的六份上传产物;旧镜像清理仍须等真实 CI 验证。专属缓存 artifact 保留 7 天,宿主也清理超过 7 天且源 run 已结束的遗留项,不删普通构建产物、run 或日志。Gitea 1.26.4 不清理未 finalized 的上传块:上传器使用专属双层编码块标识,宿主从 `artifact_storage_dir/tmp-upload/run-<id>-v4/` 定向清理本上传器的普通文件,要求所属仓库 master run 已结束超过 7 天且文件本身也超过 7 天。未知、年轻、符号链接或不属于本上传器的文件受到保护,不改数据库、不删除目录。已完成产物通过 API 删除;未完成上传留下的数据库元数据仍归 Gitea 管理。宿主的未完成下载/合并临时文件仅在登记的私有目录内清理。
|
||||
|
||||
### 自动维护首次部署与恢复
|
||||
|
||||
首次部署在本变更合入 master 后进行。维护脚本安装到 `/opt/genarrative-ci-cache/scripts/`;运行状态、源码专用 clone、日志和归档放在 `/var/lib/genarrative-ci-cache/`,token 放在 `/etc/genarrative-ci-cache/api-token`(root 所有、0600),均不进入 Git。复制 `gitea-ci-cache.config.example.json` 为该目录的 `config.json` 并核实仓库、Runner 容器与网关地址。Token 使用具有目标仓库 Actions 读写权限的普通账号,授予 `write:repository` 即可,不需要 `admin`;写操作仅定向删除本维护器的缓存 artifact,不重跑或取消任务。宿主运行账号还须能通过 `clone_url` 拉取专属 clone,API Token 不自动传给 Git。`artifact_storage_dir` 必须指向当前 Gitea 本地 ActionsArtifacts 存储的真实宿主路径(不是容器路径);按当前 `/data` bind mount 和 `APP_DATA_PATH`,模板为 `/opt/gitea-stack/data/gitea/gitea/actions_artifacts`。部署时由 root 核实路径与 Gitea `[actions.artifacts]`/storage 配置一致,不能指向其它目录;此残块清理实现只支持本地存储,外部对象存储需要另行适配。
|
||||
|
||||
领取入口由 `gitea-runner-fetch-gate.compose.yml` 启动,使用已验证且含 Python 3 的无对象缓存 CI Image ID(`GITEA_FETCH_GATE_IMAGE`)。它只连接现有 `gitea-actions` 内部网络,不发布宿主端口、不挂 Docker socket;仅转发 `/api/actions/` RPC,控制 socket 位于独立私有目录 `/var/lib/genarrative-ci-cache-gate/`。该目录必须由 root 持有且权限为 0700。普通 job 无法访问控制 socket。部署前先运行下述测试,不直接启用 timer。
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s scripts -p 'test_gitea_cache_*.py'
|
||||
install -d -m 700 /etc/genarrative-ci-cache /var/lib/genarrative-ci-cache /var/lib/genarrative-ci-cache-gate
|
||||
install -d /opt/genarrative-ci-cache/scripts
|
||||
install -m 755 scripts/maintain-gitea-rust-cache.py scripts/gitea-runner-fetch-gate.py /opt/genarrative-ci-cache/scripts/
|
||||
install -m 644 scripts/gitea_cache_snapshot.py scripts/gitea_cache_upload_cleanup.py /opt/genarrative-ci-cache/scripts/
|
||||
install -m 600 deploy/container/gitea-ci-cache.config.example.json /etc/genarrative-ci-cache/config.json
|
||||
# 由维护者写入 write:repository API token,并核实 config.json;不要在终端回显 token。
|
||||
# 设置 GITEA_FETCH_GATE_IMAGE 为已验证基础镜像的完整 Image ID 后:
|
||||
docker compose -f deploy/container/gitea-runner-fetch-gate.compose.yml up -d
|
||||
```
|
||||
|
||||
**首次接入或从不跟踪任务的旧网关升级,需要空闲维护窗口**:确认无活跃 CI 且暂停新 CI 触发,再备份现有 runner 配置和注册文件,在 runner 部署的 `GITEA_INSTANCE_URL` 及 `/data/.runner` 的 `address` 中改用 `http://gitea-runner-fetch-gate:8080`,保留其余注册字段。同时在 `/data/config.yaml` 的 `runner.envs` 中设置 `GENARRATIVE_GITEA_REPOSITORY_URL: "http://gitea:3000/GenarrativeAI/Genarrative.git"`,与维护器的 `repository_url` 一致,沿用现有 job 已可达的内部 Git 通道。此专用变量也覆盖旧 PR 的 checkout;不要用同名 GITHUB_SERVER_URL 环境变量代替,Runner 会再次覆盖它。按原流程重启并验证 runner 注册。
|
||||
|
||||
不能仅改磁盘文件却不让进程加载;维护器还会检查独立 checkout URL,并确认入口实际见到了当前容器本次启动后的 FetchTask 来源 IP。此一次接入不由维护器冒险猜测空闲,也不对运行中 CI 动手。实例 API 根地址仍使用配置中的 HTTPS Gitea 地址,不能指向只支持 runner RPC 的入口。上传脚本从独立 `GENARRATIVE_GITEA_REPOSITORY_URL` 推导真实 Gitea 地址,使用本任务临时凭据调用原生 V4 API;不依赖被网关覆盖的 `GITHUB_SERVER_URL` / `ACTIONS_RUNTIME_URL`。Gitea 1.26.4 的仓库 REST 下载接口只支持 V4,不能换回 V3 上传。宿主下载只跟随同一 HTTPS Gitea origin 的签名重定向,不转发长期 Token;如配置外部对象存储直出,需另行适配下载来源。
|
||||
|
||||
```bash
|
||||
# --apply 缺省时只检查 API、runner、master 和入口路径,不修改配置或镜像。
|
||||
python3 /opt/genarrative-ci-cache/scripts/maintain-gitea-rust-cache.py --config /etc/genarrative-ci-cache/config.json
|
||||
install -m 644 deploy/systemd/genarrative-ci-cache.service deploy/systemd/genarrative-ci-cache.timer /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now genarrative-ci-cache.timer
|
||||
journalctl -u genarrative-ci-cache.service -n 50
|
||||
```
|
||||
|
||||
timer 在上次执行结束后约 5 分钟再次检查,文件锁防止人工与定时执行重叠。构建日志位于状态目录 `artifacts/<SHA>/build.log`。同一 run 组装失败后不每 5 分钟重复消耗资源;新的完整 master run 到来后自动尝试,也可修复环境后显式运行 `--apply --retry`。切换事务及暂停归属先落状态文件;`ExecStopPost --resume` 恢复本维护器暂停的领取,下次执行再收敛中断的切换。其它人暂停的入口不由维护器擅自恢复;不修改 Gitea 的 Runner disabled 设置。
|
||||
|
||||
候选切换后的验收读取真实 **master push** 的完整九个 job,要求全部 success、每个 job 均使用目标 Image ID,六个 Rust job 有启用缓存、正命中数和零缓存错误。测试失败、旧 PR 缺 prepare、混用镜像或缺日志均不清理旧版,也不伪造“缓存已验收”。仍保留源代码失败需要修复的原始结果。
|
||||
|
||||
入口遇到“已发送 FetchTask,但上游响应未完整结束”会持久化 `uncertain` 并拒绝继续领取/自动切换;不会因为客户端的 5 秒超时就认定服务端事务已回滚。其它 RPC 和任务上报仍继续转发。维护者需先核实 Gitea 在途领取事务与该 runner 的任务全部收敛,在维护窗口停止入口,核实并修复它的 `tasks.json` 任务账本及 `uncertain` / `inflight` 标记后再启动并恢复领取。网关按 Gitea 1.26.4 / Runner 2.0.0 的 Connect Protobuf 协议,在最终日志确认及 Runner 执行结束后的最终任务上报确认后才清账;取消响应不等于进程停止,任务 ID 不按超时自动删除。禁止自动删这些标记绕过屏障。停用自动维护先 `systemctl disable --now genarrative-ci-cache.timer`;不要为了停 timer 停止运行中的 CI 容器。已接入的领取入口继续运行,不影响普通 CI。
|
||||
|
||||
人工 bootstrap 预热容器上限为 4 核、12 GiB,移除 capabilities,不挂宿主目录/socket,也不注入 Git/OSS/Jenkins 凭据。源码通过 `git archive` 复制,当前工作区、ignored 文件和 `.git` 不进入容器。最终从原镜像重新组装,仅复制 `/opt/genarrative-ci/rust-cache` 的 sccache、对象和来源元数据,不提交含源码/target 的预热容器;镜像本身的下载缓存与工具链校验保持原样。
|
||||
|
||||
快照由固定 Image ID 分发,每个 job 仅修改容器自己的写时复制层,缓存上限 4 GiB,只有 master push 在结束前回传新增对象;PR 不扫描基线、不打包、不回传。`ci-rust-cache.sh prepare` 清空继承的 `SCCACHE_*` 远程配置,使用独立配置和 Unix socket;旧镜像、工具链不匹配或限时 wrapper 探测失败时使用直接 rustc,正式编译启用 sccache 的 server IO 错误回退。真实编译/测试失败保留非零退出码。`report` 输出命中统计并停止本 job daemon,分片日志输出独立编译耗时。sccache 0.18.0 的只读模式在 miss 后仍打包再拒绝写入,不能用它宣称零 miss 开销;普通 CI 继续只向容器层写入。
|
||||
|
||||
生成候选不会改变 runner 配置。线上有活跃 CI 时禁止停止 job、重启 runner 或切换标签;只在确认空闲后按上节流程切换固定 Image ID。全组启用须重建完整快照;旧 AGC 单目标快照不能作为其它组预热成功的证据。按组用相同源码、独立干净 target 比较编译命中与耗时;分片测试内容和前置检查不同,不能直接用两条 lane 总耗时推断缓存收益。回滚时可移除各 job 的 prepare/report,或恢复无对象缓存的原基础镜像 ID,均不需要改变 incremental 或测试分片。
|
||||
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"api_url": "https://git.genarrative.world/git/api/v1",
|
||||
"repository": "GenarrativeAI/Genarrative",
|
||||
"repository_url": "http://gitea:3000/GenarrativeAI/Genarrative.git",
|
||||
"clone_url": "https://git.genarrative.world/git/GenarrativeAI/Genarrative.git",
|
||||
"runner_container": "gitea-runner",
|
||||
"token_file": "/etc/genarrative-ci-cache/api-token",
|
||||
"state_dir": "/var/lib/genarrative-ci-cache",
|
||||
"artifact_storage_dir": "/opt/gitea-stack/data/gitea/gitea/actions_artifacts",
|
||||
"gate_socket": "/var/lib/genarrative-ci-cache-gate/gate.sock",
|
||||
"gate_url": "http://gitea-runner-fetch-gate:8080"
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
# 只加入 runner 与 Gitea 共用的内部网络,不发布端口,不挂 Docker socket。
|
||||
services:
|
||||
gitea-runner-fetch-gate:
|
||||
image: ${GITEA_FETCH_GATE_IMAGE:?Set a verified no-object-cache CI image ID with Python 3}
|
||||
container_name: gitea-runner-fetch-gate
|
||||
restart: unless-stopped
|
||||
entrypoint: ['python3', '/app/gitea-runner-fetch-gate.py']
|
||||
environment:
|
||||
GITEA_RUNNER_UPSTREAM: http://gitea:3000
|
||||
read_only: true
|
||||
cap_drop: [ALL]
|
||||
security_opt: [no-new-privileges:true]
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
pids_limit: 128
|
||||
volumes:
|
||||
- /opt/genarrative-ci-cache/scripts/gitea-runner-fetch-gate.py:/app/gitea-runner-fetch-gate.py:ro
|
||||
- /var/lib/genarrative-ci-cache-gate:/control
|
||||
networks: [gitea-actions]
|
||||
|
||||
networks:
|
||||
gitea-actions:
|
||||
external: true
|
||||
@@ -0,0 +1,22 @@
|
||||
[Unit]
|
||||
Description=Refresh and retain trusted Gitea CI Rust cache images
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
Requires=docker.service
|
||||
ConditionPathExists=/etc/genarrative-ci-cache/config.json
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=root
|
||||
WorkingDirectory=/opt/genarrative-ci-cache
|
||||
UMask=0077
|
||||
ExecStart=/usr/bin/python3 /opt/genarrative-ci-cache/scripts/maintain-gitea-rust-cache.py --config /etc/genarrative-ci-cache/config.json --apply
|
||||
ExecStopPost=/usr/bin/python3 /opt/genarrative-ci-cache/scripts/maintain-gitea-rust-cache.py --config /etc/genarrative-ci-cache/config.json --resume
|
||||
TimeoutStartSec=3h
|
||||
TimeoutStopSec=60s
|
||||
Nice=10
|
||||
IOSchedulingClass=best-effort
|
||||
IOSchedulingPriority=7
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=Check for new Gitea CI compilation inputs
|
||||
|
||||
[Timer]
|
||||
OnBootSec=5min
|
||||
OnUnitInactiveSec=5min
|
||||
RandomizedDelaySec=30s
|
||||
Unit=genarrative-ci-cache.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -96,6 +96,10 @@ SpacetimeDB 任务统一先读取 `.codex/skills/genarrative-spacetimedb/SKILL.m
|
||||
|
||||
## Gitea CI 依赖闭合
|
||||
|
||||
AGC Rust 两条 lane、crates、smoke、Backend 和桌面壳测试使用镜像内可信 sccache 对象快照;Native shell release step 显式清空双 wrapper,前端/repository checks 不启用。维护者通过 `scripts/build-gitea-rust-cache.sh` 从远端 master 在限额、无宿主挂载的临时容器中按实际 cwd/profile/目标预热全部测试组,仅编译、不执行测试/应用;后端 workspace 与 spacetime-module 保持独立,AGC 的三个 cwd 入口之间清理预热 target,防止 fresh 判断漏产缓存键。最终镜像只追加 sccache、对象和来源元数据,不包含源码或 target。容量上限 4 GiB,不替代宿主旧镜像/归档清理。PR 只写当前容器层、不回传,不开放 Docker API/发布权限;继续禁用 incremental。`ci-rust-cache.sh` 在快照缺失、工具链不符或 wrapper 探测失败时直接编译,并隔离远程缓存配置和 daemon。分片日志记录编译耗时,收尾输出命中统计;两个 lane 的测试和前置检查不同,耗时差不是严格 A/B。线上存在活跃 CI 时不得重启 runner 或切换标签;全组启用前须刷新完整快照并逐组验证,详见开发运维文档。
|
||||
Gitea Rust 缓存自动维护由宿主 `genarrative-ci-cache.timer` 收集同一 master push run 六个 Rust job 的原生 V4 缓存产物,不重复执行 Cargo 预热。只传本轮新 key,命中对象只传使用时间;宿主与真实来源镜像对象合并、去重、按新近使用时间裁剪到 4 GiB,从无对象缓存基础镜像重新组装。源 run 不要求全绿,但取消、缺组、旧 attempt、未完成上传或混用来源镜像不得采用。网关暂停新 FetchTask、在途领取结束、持久化任务账本清空且内层活动容器为空才切换,不打断运行中的 CI。首次接入/升级网关需空闲窗口;Token 只需普通仓库 `write:repository`,不查管理员 API。候选装载后清理已收集 artifact,遗留项保留 7 天;真实 master CI 验证后才清理旧镜像,保留当前、一个回滚版、基础镜像及容器引用。部署入口见 `deploy/container/README.md`,合并代码不等于服务启用。
|
||||
|
||||
修改 Gitea workflow 的 job 显示名称、ID 或缓存导出组时,必须同步维护器的 `JOBS` / `RUST_JOB_IDS`;`test_gitea_cache_maintenance.py` 直接对照实际 workflow 检查全集和导出映射,避免自动刷新或镜像验收因名单漂移长期等待。维护器 `Api.request` 的 `method` 是必填关键字参数,GET 也必须显式指定,不根据 body 推断请求方法。
|
||||
|
||||
AGC Rust 两条 lane、crates、smoke、Backend 和桌面壳测试使用镜像内可信 sccache 对象快照;Native shell release step 显式清空双 wrapper,前端/repository checks 不启用。仅首次人工 bootstrap 时,维护者通过 `scripts/build-gitea-rust-cache.sh` 从远端 master 在限额、无宿主挂载的临时容器中按实际 cwd/profile/目标预热全部测试组,仅编译、不执行测试/应用;后端 workspace 与 spacetime-module 保持独立,AGC 的三个 cwd 入口之间清理预热 target,防止 fresh 判断漏产缓存键。最终镜像只追加 sccache、对象和来源元数据,不包含源码或 target。容量上限 4 GiB,不替代宿主旧镜像/归档清理。PR 只写当前容器层、不回传,不开放 Docker API/发布权限;继续禁用 incremental。`ci-rust-cache.sh` 在快照缺失、工具链不符或 wrapper 探测失败时直接编译,并隔离远程缓存配置和 daemon。分片日志记录编译耗时,收尾输出命中统计;两个 lane 的测试和前置检查不同,耗时差不是严格 A/B。线上存在活跃 CI 时不得重启 runner 或切换标签;全组启用前须刷新完整快照并逐组验证,详见开发运维文档。
|
||||
|
||||
`.gitea/workflows/project-ci.yml` 的客户端门禁拆成 lane 与功能 job,每个 job 只预热自己会构建的那几份依赖:`AI game creator shell Rust lane 1/2`、`lane 2/2` 各自预取一次 AGC 壳 manifest,并顺序运行两片 Rust bin 单测;`AI game creator shell Rust smoke` 同样只预取 AGC 壳 manifest(`agent-run` smoke 会用 `src-tauri/Cargo.toml` spawn `cargo run`),`AI game creator shell Rust crates` 预取 `server-rs/Cargo.toml` 与独立 crate,`Native shell tests` 预取桌面壳与 AGC 壳 manifest,`AI game creator shell web tests` 不触碰 Cargo,不预热。两条 Rust lane、smoke job 与 crates job 只用 cargo 与 node 内建模块,因此不执行 `npm ci`。两个被 `server-rs/Cargo.toml` 排除、且没有提交 `Cargo.lock` 的独立 crate(`agent-runtime-core`、`agent-runtime-orchestration`)只能在 `AI game creator shell Rust crates` 里用不带锁标志的 fetch。AGC 壳的 bin target 单测(约 2466 条)由 `apps/ai-game-creator-shell/scripts/run-rust-shell-test-shards.mjs` 编译后按 `--list` 名单分 4 片:每次分片调用用 `--shard-index=<i>` 只跑自己那片,片内保持 `--test-threads=1` 并使用独立 `TMPDIR`;两条 lane 之间并发,lane 内顺序运行两片,避免重复依赖预热和同一容器内多进程争抢。不要改回「一个 job 内多进程并行这几片」——同一容器里它们会争抢共享 `HOME`、target 目录与固定临时路径,实测比整套串行还慢。每个分片调用都会自校验「片并集等于全集且互斥」,因此改分片规则不会静默漏跑。Backend host workspace tests 使用 `cargo test --locked --workspace --exclude spacetime-module --no-fail-fast`,避免 `spacetime-module` 的 `spacetime-types` feature 统一污染普通领域 crate 的 host 测试;随后单独执行 `cargo test --locked -p spacetime-module --no-fail-fast`,由 `spacetime-module/src/active.rs` 在 host 测试构建期间提供仅测试期的 SpacetimeDB ABI 链接支持,使该 crate 的纯单元测试也纳入 Backend 门禁。`spacetime-module` 的 reducer / procedure 运行时行为仍必须通过真实 SpacetimeDB runtime/integration harness 验证,host 链接支持不得被当作运行时替身。Backend 另外执行 `cargo check --locked -p spacetime-module` 验证模块源码。AGC 壳检查还会运行 `platform-llm` 与 `shared-contracts` 的 server-rs workspace 测试,这些命令以及 AGC 壳测试必须带 `--locked`,避免在测试阶段重新解析 registry index;锁文件发生变化时应先更新受信任 CI 镜像缓存,再重跑门禁。
|
||||
|
||||
@@ -6004,5 +6004,7 @@ Cocos Creator 根目录由 `package.json.creator.version` 与普通 `assets/`
|
||||
- 处理:预热使用已核实的 Gitea 路径 `/workspace/GenarrativeAI/Genarrative`,并与分片运行器一样从 AGC `src-tauri` 启动 Cargo;保存 `workspace.txt`,路径不符时回退直接编译。wrapper 放在容器内固定路径,daemon 状态与 Unix socket 仍使用随机私有目录;预热环境与 workflow 的 Cargo 环境由定向契约测试核对。不要为命中率随意增加 `RUSTFLAGS`、改写源码路径或恢复共享可写 target。
|
||||
- 验证:相同源码、资源上限和独立干净 target 下分别记录无缓存、冷缓存、热缓存的编译耗时和 hit/miss;只有真实热命中有净收益才切换候选镜像。PR 的写入始终留在 job 容器层,公共快照仍由可信维护流程生成。
|
||||
- 统计:job 私有 daemon 设置 `SCCACHE_IDLE_TIMEOUT=0`,由 `report` 显式停止;最终测试 bin 的不可缓存编译或测试可能超过一分钟,短 idle timeout 会让 daemon 提前退出,结尾查询启动新 daemon 后误报零次请求。容器销毁仍会回收该 job 的全部进程。
|
||||
- 磁盘:快照构建拒绝含 `/opt/genarrative-ci/rust-cache` 的基础镜像,始终从无对象缓存的镜像重建;容器内删除旧对象不能释放 Docker 底层。对象缓存容量上限不涵盖宿主旧镜像及导出归档,切换验证后按运维文档人工保留当前版和一个回滚版,同时保护运行中 CI 使用的镜像。
|
||||
- 磁盘:快照构建拒绝含 `/opt/genarrative-ci/rust-cache` 的基础镜像,始终从无对象缓存的镜像重建;容器内删除旧对象不能释放 Docker 底层。对象缓存容量上限不涵盖宿主旧镜像及导出归档。自动维护只回收自己预先登记的 Image ID/tag 和专属归档,保留当前、一个回滚版、基础镜像及所有容器引用;内层按 ID 导入的镜像可能没有 tag,不能只按 tag 判断已清理。接管前历史试验版本仍需人工确认。
|
||||
- CI 产物清理:Gitea 1.26.4 的仓库 REST 仅列出 finalized/expired V4 artifact,内置到期清理不回收上传中断的 tmp-upload 分块。缓存上传块须带专属标识,宿主只清理目标仓库已结束且超过 7 天的 master run 中同样过期的自有普通文件,未知文件/符号链接保护,不改数据库或全局 prune。Artifact.workflow_run 仅含 ID/SHA,判断过期产物所属事件和状态须再读 run API,不能当作完整 run 使用。
|
||||
- 自动切换:Gitea 1.26.4 的 disabled 检查与 FetchTask 事务不原子,Runner 客户端超时不能证明服务端回滚,容器暂时为空也不能证明没有已领取任务。网关必须解析实际 Connect Protobuf/gzip,转发 FetchTask 结果前持久化任务 ID,仅在最终日志及执行清理后的最终 UpdateTask 确认后清账;取消响应不能提前释放。暂停新领取、在途为零、账本为零且内层活动容器为空才可切换,无需全局 Runner admin API。未知协议/响应或崩溃遗留标记停止切换;旧网关缺 active_tasks 不能默认零。首次接入与账本升级须空闲窗口。.runner 的 mtime 不证明地址已加载,应核验真实 FetchTask 来源及本次容器启动时间。
|
||||
- 扩展:预热所有 Rust 测试组时保留各自 cwd、profile、features 和锁策略;同一临时 target 的 Cargo fresh 不代表不同 cwd 都已生成缓存键,AGC 提示词契约、分片和 smoke 切换入口前清理预热 target。不要把 workspace 与 spacetime-module 合并成一次编译;Native shell release step 清空双 wrapper,避免将测试缓存扩展成发布缓存。当前 sccache 0.18.0 的 READ_ONLY 在 miss 后仍打包产物并产生 cache write error,不适合用来承诺“未命中无开销”。
|
||||
|
||||
@@ -334,16 +334,18 @@ master 日常交付必须禁止直接 push,只允许经 PR 在最近一次 Pro
|
||||
|
||||
#### Rust 测试组的隔离编译缓存
|
||||
|
||||
每次生成快照必须使用不含对象缓存的原始 CI 基础镜像;构建脚本在拉取源码和预热前拒绝已存在 `/opt/genarrative-ci/rust-cache` 的基础镜像,避免重复叠加不可释放的旧对象层。切换并完成真实 CI 验证后,按 `deploy/container/README.md` 的保留规则人工定向清理旧缓存镜像及归档,保护运行中 CI 使用的版本。
|
||||
每次生成快照必须使用不含对象缓存的原始 CI 基础镜像,避免叠加不可释放的旧层。宿主自动维护器 `scripts/maintain-gitea-rust-cache.py` 收集完整 master push CI 的六组 V4 增量产物,复用其真实来源镜像对象并合并去重,总容量 4 GiB,不重复执行 Cargo 预热编译。测试失败可收集,取消、缺组、未完成上传、旧 attempt 或混用来源镜像不可发布。候选校验、导出和装载后删除已收集 artifact,遗留项保留 7 天;切换后通过真实 master CI 才定向清理受管旧镜像/归档,保护当前、一个回滚版、基础镜像及容器引用。
|
||||
|
||||
AGC Rust 两条 lane、crates、agent-run smoke、Backend 和 Native shell 的桌面壳测试均启用 sccache。Native shell 的 release build smoke 显式清空两个 wrapper,保持发布构建原有 profile/features 与资源 staging;前端和 repository checks 不启用对象缓存。继续设置 `CARGO_INCREMENTAL=0`,不共享 target、不恢复 Actions 可写缓存。可信快照通过已有固定 Image ID 分发:镜像只增加固定版本的 sccache、编译对象和来源元数据,不包含源码、target 或凭据;容器写时复制层承接本 job 的新增对象,job 删除后丢弃,PR 没有 Docker API 或快照发布权限。该权限边界由 runner 基础设施保证,不能仅用 workflow 的分支条件替代。
|
||||
|
||||
维护者在受信任 checkout 中运行 `bash scripts/build-gitea-rust-cache.sh <已验证基础镜像> <候选镜像tag>`。脚本从 origin 获取 master 的确定提交,在无宿主目录挂载、无凭据且有 CPU/内存上限的临时容器中预热各组实际 Cargo 目标,只编译、不执行测试或应用。后端 workspace 仍排除 spacetime-module,再单独预热该模块,禁止为缓存合并 features。AGC 独立 crate 与插件沿用各自 manifest/锁文件策略;未提交锁文件的 crate 仍可能因新版本解析产生 miss。最终镜像从原基础镜像重新组装,只导出 sccache 对象,不提交预热容器。公共快照不接受 PR 上传,也不复用 Jenkins 发布缓存。工具链或系统依赖变化时重新生成快照,缓存命中仍由 sccache 的编译输入校验决定,不能省略 Cargo 构建。
|
||||
人工 bootstrap 在没有可消费快照时使用 `bash scripts/build-gitea-rust-cache.sh <已验证基础镜像> <候选镜像tag> [完整master-SHA]`;自动维护不调用它。bootstrap 固定 master 归档,在无宿主挂载、无凭据、有资源限额的容器中只编译预热,不执行测试/应用,Cargo features 和工作目录保持实际 CI 口径。日常更新由 master CI 导出新 key,命中继承对象只传使用时间;PR 不导出、不扫描。宿主验证来源、大小和哈希,不从上传产物执行程序,只从可信来源镜像复制固定 sccache。工具链或基础镜像输入变化时重建无对象缓存基础镜像,缓存工具链必须匹配。公共快照不接受 PR,不复用 Jenkins 发布缓存。
|
||||
|
||||
各 Rust job 在编译前执行 `scripts/ci-rust-cache.sh prepare`:检查快照与 rustc 身份,隔离 sccache 配置和 daemon,限时探测真实 wrapper。旧镜像没有快照或探测失败时保留空 wrapper,输出 fallback 原因;缓存故障不得把真实编译/测试失败改成成功,也不允许重跑整个测试组掩盖失败。结束时 `report` 输出命中统计;分片日志单独记录 Cargo 编译耗时。对象缓存上限为 4 GiB,快照构建完成后输出实际体积;最终测试 bin 的链接仍须执行。全组开启时必须生成覆盖全部目标的新快照,不能把旧 AGC 单目标快照当作后端/桌面壳的预热验收。
|
||||
|
||||
候选镜像沿用 `gitea-ci-job-image.sh verify/export/load-runner` 验证和装载。现役 CI 正在运行时只允许准备候选镜像,不停止 job、不重启 runner、不切换标签;待确认无运行中的 job 后再按上述镜像更新顺序切换。验收分别记录独立 target 的无缓存、冷缓存和热缓存构建耗时及命中率,并检查失效/故障回退、两容器写入互不影响。未完成真实 CI 验证前不能宣称提速或推广到其它 job。
|
||||
|
||||
自动维护首次部署及故障恢复统一见 `deploy/container/README.md`。timer 约每 5 分钟检查,文件锁串行执行,使用专属 clone/状态/归档目录;只管理 Gitea CI 测试镜像,Native shell release smoke 继续清空双 wrapper。凭据只需普通账号 `write:repository`,读取 run/job/日志/产物并定向删除缓存 artifact,不需要管理员权限。切换依赖 FetchTask 暂停与在途屏障、任务持久账本和内层 Docker 空闲;Gitea 显示 cancelled、disabled 或客户端超时都不能证明执行已经结束。网关依据最终日志与执行收尾后的最终任务上报确认释放任务。首次接入/旧网关升级须等待 CI 自然结束的空闲窗口,不在运行中的 CI 上试切;脚本合并不等于服务已部署。
|
||||
|
||||
Rust 对象 key 包含编译工作目录。预热必须使用已核实的 Gitea checkout 路径 `/workspace/GenarrativeAI/Genarrative`:后端、独立 crate、插件、桌面壳及提示词契约从仓库根目录启动 Cargo,AGC 分片从 `apps/ai-game-creator-shell/src-tauri` 启动,agent-run smoke 从 `apps/ai-game-creator-shell` 启动。切换 AGC 编译入口前清理该临时容器的 AGC target,防止 Cargo 的 fresh 判断跳过新 cwd 所需对象;CI 各 job 的 target 本来就独立。快照保存 `workspace.txt`,job 根路径不符时直接编译。不要仅设置 `SCCACHE_BASEDIRS` 就假定 Rust 可以跨 cwd 命中,也不为缓存改写 `RUSTFLAGS` 或源码路径语义。固定 sccache `0.18.0` 的基础设施错误码 `2` 会由 wrapper 回退执行本次 rustc;其它退出码原样返回,升级 sccache/Rust 时须复核此约定。
|
||||
|
||||
sccache `0.18.0` 还会 hash 大部分 `CARGO_*` 环境变量。因此 wrapper 固定在每个容器自己的 `/opt/genarrative-ci/rust-cache/rustc-wrapper`,prepare 重写 launcher 后才启用;daemon 状态与 socket 仍各自随机隔离。预热同步 workflow 的 `CARGO_INCREMENTAL`、`CARGO_HTTP_MULTIPLEXING`、`CARGO_NET_RETRY` 和 `CARGO_TERM_COLOR`,由现有 CI 契约测试验证一致;新增 Cargo 环境配置时须同步评估 cache key,不能只看快照文件是否存在。
|
||||
|
||||
@@ -5,6 +5,15 @@ set -euo pipefail
|
||||
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd -P)"
|
||||
base_ref="${1:?usage: build-gitea-rust-cache.sh <verified-base-image> <candidate-tag>}"
|
||||
candidate_tag="${2:?candidate image tag is required}"
|
||||
requested_source_commit="${3:-}"
|
||||
if [[ "$#" -gt 3 ]]; then
|
||||
echo 'usage: build-gitea-rust-cache.sh <verified-base-image> <candidate-tag> [master-commit-sha]' >&2
|
||||
exit 2
|
||||
fi
|
||||
if [[ -n "${requested_source_commit}" && ! "${requested_source_commit}" =~ ^[0-9a-fA-F]{40}$ ]]; then
|
||||
echo 'master-commit-sha must be a complete 40-character SHA.' >&2
|
||||
exit 2
|
||||
fi
|
||||
# 与实际 Gitea checkout 路径一致;Rust 对象 key 包含编译 cwd,不能随意换临时根。
|
||||
workspace=/workspace/GenarrativeAI/Genarrative
|
||||
[[ "${CI:-}" != true ]] || { echo 'Run on the trusted image builder, outside CI jobs.' >&2; exit 1; }
|
||||
@@ -22,7 +31,17 @@ bash "${repo_root}/scripts/gitea-ci-job-image.sh" verify "${base_id}"
|
||||
|
||||
# 只归档远端 master 的确定提交;不复制当前工作区或本地凭据。
|
||||
git -C "${repo_root}" fetch --no-tags origin refs/heads/master
|
||||
source_commit="$(git -C "${repo_root}" rev-parse FETCH_HEAD^{commit})"
|
||||
master_commit="$(git -C "${repo_root}" rev-parse FETCH_HEAD^{commit})"
|
||||
if [[ -n "${requested_source_commit}" ]]; then
|
||||
git -C "${repo_root}" cat-file -e "${requested_source_commit}^{commit}"
|
||||
if ! git -C "${repo_root}" merge-base --is-ancestor "${requested_source_commit}" "${master_commit}"; then
|
||||
echo "master-commit-sha is not contained in fetched master: ${requested_source_commit}" >&2
|
||||
exit 1
|
||||
fi
|
||||
source_commit="$(git -C "${repo_root}" rev-parse "${requested_source_commit}^{commit}")"
|
||||
else
|
||||
source_commit="${master_commit}"
|
||||
fi
|
||||
work_dir="$(mktemp -d "${TMPDIR:-/tmp}/gitea-rust-cache.XXXXXX")"
|
||||
container_id=''
|
||||
cleanup() {
|
||||
@@ -50,7 +69,7 @@ container_id="$(docker run --detach --cpus=4 --memory=12g --pids-limit=1024 \
|
||||
docker exec "${container_id}" mkdir -p "${workspace}" /opt/genarrative-ci/rust-cache/objects
|
||||
git -C "${repo_root}" archive "${source_commit}" | docker cp - "${container_id}:${workspace}"
|
||||
docker cp "${work_dir}/snapshot/." "${container_id}:/opt/genarrative-ci/rust-cache/"
|
||||
docker cp "${repo_root}/scripts/ci-rust-cache.sh" "${container_id}:/tmp/ci-rust-cache.sh"
|
||||
docker exec "${container_id}" cp "${workspace}/scripts/ci-rust-cache.sh" /tmp/ci-rust-cache.sh
|
||||
docker exec --interactive --workdir "${workspace}" "${container_id}" bash -s <<'WARM'
|
||||
set -euo pipefail
|
||||
rustc -vV > /opt/genarrative-ci/rust-cache/rustc.txt
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# 仅消费镜像内的可信快照;所有写入留在当前容器的可写层。
|
||||
# 消费镜像内可信快照;写入留在任务容器,master push 可在任务结束后导出。
|
||||
set -euo pipefail
|
||||
|
||||
cache_root="${GENARRATIVE_CI_RUST_CACHE_ROOT:-/opt/genarrative-ci/rust-cache}"
|
||||
@@ -22,7 +22,7 @@ configure_local_cache() {
|
||||
case "${1:-}" in
|
||||
prepare)
|
||||
: "${GITHUB_ENV:?GITHUB_ENV is required}"
|
||||
printf 'RUSTC_WRAPPER=\nCARGO_BUILD_RUSTC_WRAPPER=\nGENARRATIVE_CI_RUST_CACHE_STATE=\n' >> "${GITHUB_ENV}"
|
||||
printf 'RUSTC_WRAPPER=\nCARGO_BUILD_RUSTC_WRAPPER=\nGENARRATIVE_CI_RUST_CACHE_STATE=\nGENARRATIVE_CI_RUST_CACHE_EXPORT_READY=\n' >> "${GITHUB_ENV}"
|
||||
fallback() { printf '[rust-cache] mode=direct reason=%s\n' "$1"; exit 0; }
|
||||
[[ -x "${cache_binary}" && -d "${cache_root}/objects" && -f "${cache_root}/rustc.txt" && -f "${cache_root}/source-commit.txt" && -f "${cache_root}/workspace.txt" ]] \
|
||||
|| fallback snapshot-unavailable
|
||||
@@ -44,6 +44,10 @@ case "${1:-}" in
|
||||
fallback wrapper-probe-failed
|
||||
fi
|
||||
script_path="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)/$(basename "${BASH_SOURCE[0]}")"
|
||||
if [[ "${GITHUB_EVENT_NAME:-}" == push && "${GITHUB_REF:-}" == refs/heads/master ]]; then
|
||||
# 只记 key/大小/时间,不读取对象正文;PR 没有扫描与打包开销。
|
||||
python3 "$(dirname "${script_path}")/export-gitea-rust-cache.py" baseline "${state}/baseline.json"
|
||||
fi
|
||||
# wrapper 路径也参与 Rust cache key;固定容器内路径,隔离由 job 容器保证。
|
||||
wrapper_path="${cache_root}/rustc-wrapper"
|
||||
printf '#!/usr/bin/env bash\nexec bash %q "$@"\n' "${script_path}" > "${wrapper_path}"
|
||||
@@ -59,7 +63,13 @@ case "${1:-}" in
|
||||
if [[ -n "${state}" && -d "${state}" ]]; then
|
||||
configure_local_cache
|
||||
timeout --kill-after=2 5 "${cache_binary}" --show-stats || true
|
||||
timeout --kill-after=2 5 "${cache_binary}" --stop-server >/dev/null 2>&1 || true
|
||||
# 只有成功停服后才允许读取对象;失败不能把仍在写入的目录发布成完整快照。
|
||||
if timeout --kill-after=2 15 "${cache_binary}" --stop-server >/dev/null 2>&1; then
|
||||
if [[ "${GITHUB_EVENT_NAME:-}" == push && "${GITHUB_REF:-}" == refs/heads/master && ! -f "${state}/disabled" && -f "${state}/baseline.json" ]]; then
|
||||
mv -- "${state}/baseline.json" "${cache_root}/export-baseline.json"
|
||||
printf 'GENARRATIVE_CI_RUST_CACHE_EXPORT_READY=1\n' >> "${GITHUB_ENV}"
|
||||
fi
|
||||
fi
|
||||
rm -rf -- "${state}"
|
||||
else
|
||||
printf '[rust-cache] mode=direct\n'
|
||||
|
||||
@@ -35,7 +35,8 @@ function fixture(t) {
|
||||
`#!/bin/bash
|
||||
set -eu
|
||||
case "$1" in
|
||||
--stop-server|--show-stats) exit 0 ;;
|
||||
--stop-server) exit "\${STOP_FAILURE:-0}" ;;
|
||||
--show-stats) exit 0 ;;
|
||||
esac
|
||||
if [[ "$*" == *-vV ]]; then
|
||||
[[ "\${PROBE_FAILURE:-}" != 1 ]] || exit 1
|
||||
@@ -215,3 +216,25 @@ linuxTest('lost local cache state still invokes the real compiler', (t) => {
|
||||
const f = fixture(t);
|
||||
assert.equal(f.run(['/bin/bash', '-c', 'exit 43']).status, 43);
|
||||
});
|
||||
|
||||
linuxTest('only a cleanly stopped master cache can export objects', (t) => {
|
||||
const master = {
|
||||
GITHUB_EVENT_NAME: 'push',
|
||||
GITHUB_REF: 'refs/heads/master',
|
||||
};
|
||||
for (const stopFailure of ['0', '1']) {
|
||||
const f = fixture(t);
|
||||
const prepared = f.run(['prepare'], master);
|
||||
assert.equal(prepared.status, 0, prepared.stderr);
|
||||
const result = f.run(['report'], {
|
||||
...f.preparedEnv(),
|
||||
...master,
|
||||
STOP_FAILURE: stopFailure,
|
||||
});
|
||||
assert.equal(result.status, 0, result.stderr);
|
||||
assert.equal(
|
||||
f.preparedEnv().GENARRATIVE_CI_RUST_CACHE_EXPORT_READY,
|
||||
stopFailure === '0' ? '1' : '',
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -0,0 +1,274 @@
|
||||
#!/usr/bin/env python3
|
||||
"""只把 master push 的任务内 sccache 对象导出为 Gitea Actions 产物。"""
|
||||
|
||||
import base64
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import zipfile
|
||||
|
||||
|
||||
MAX_OBJECT_BYTES = 4 * 1024**3
|
||||
CHUNK_BYTES = 8 * 1024**2
|
||||
JOBS = {
|
||||
"ai-game-creator-shell-rust-lane-1",
|
||||
"ai-game-creator-shell-rust-lane-2",
|
||||
"ai-game-creator-shell-rust-smoke",
|
||||
"ai-game-creator-shell-rust-crates",
|
||||
"backend-tests",
|
||||
"native-shell-tests",
|
||||
}
|
||||
|
||||
|
||||
def object_path(name):
|
||||
parts = name.split("/")
|
||||
return (
|
||||
len(parts) == 3
|
||||
and re.fullmatch(r"[a-f0-9]{64}", parts[2]) is not None
|
||||
and parts[0] == parts[2][0]
|
||||
and parts[1] == parts[2][1]
|
||||
)
|
||||
|
||||
|
||||
class HashingReader:
|
||||
def __init__(self, stream):
|
||||
self.stream = stream
|
||||
self.digest = hashlib.sha256()
|
||||
|
||||
def read(self, size=-1):
|
||||
data = self.stream.read(size)
|
||||
self.digest.update(data)
|
||||
return data
|
||||
|
||||
|
||||
def scan_objects(root):
|
||||
objects = root / "objects"
|
||||
if objects.is_symlink() or not objects.is_dir():
|
||||
raise ValueError("cache object directory is unavailable")
|
||||
candidates = []
|
||||
for directory, directories, files in os.walk(objects, followlinks=False):
|
||||
directories[:] = [
|
||||
name for name in directories if not (Path(directory) / name).is_symlink()
|
||||
]
|
||||
for filename in files:
|
||||
path = Path(directory) / filename
|
||||
relative = path.relative_to(objects).as_posix()
|
||||
info = path.lstat()
|
||||
if object_path(relative) and stat.S_ISREG(info.st_mode):
|
||||
candidates.append((path, relative, info))
|
||||
return candidates
|
||||
|
||||
|
||||
def save_baseline(root, destination):
|
||||
# sccache 命中会更新 mtime,不能把时间变化当成内容变化,否则又变成全量上传。
|
||||
baseline = {
|
||||
relative: {"size": info.st_size, "mtime_ns": info.st_mtime_ns}
|
||||
for _, relative, info in scan_objects(root)
|
||||
}
|
||||
destination.write_text(json.dumps(baseline), encoding="utf-8")
|
||||
|
||||
|
||||
def pack_snapshot(root, destination, metadata, baseline, limit=MAX_OBJECT_BYTES):
|
||||
"""仅归档新 key;命中的已有对象只传递新近使用时间,不传输对象内容。"""
|
||||
candidates = scan_objects(root)
|
||||
candidates.sort(key=lambda item: (-item[2].st_mtime_ns, item[1]))
|
||||
entries = []
|
||||
touched = []
|
||||
total = 0
|
||||
with zipfile.ZipFile(destination, "w", compression=zipfile.ZIP_STORED) as bundle, \
|
||||
bundle.open("snapshot.tar", "w", force_zip64=True) as tar_stream, \
|
||||
tarfile.open(fileobj=tar_stream, mode="w|", format=tarfile.USTAR_FORMAT) as archive:
|
||||
for path, relative, info in candidates:
|
||||
previous = baseline.get(relative)
|
||||
if previous is not None and previous["size"] == info.st_size:
|
||||
if previous["mtime_ns"] != info.st_mtime_ns:
|
||||
touched.append({"path": "objects/" + relative, "mtime_ns": info.st_mtime_ns})
|
||||
continue
|
||||
if info.st_size <= 0 or total + info.st_size > limit:
|
||||
continue
|
||||
# report 已成功停止 daemon;打开后再核实对象,避免导出变化中的文件。
|
||||
descriptor = os.open(path, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0))
|
||||
with os.fdopen(descriptor, "rb") as stream:
|
||||
actual = os.fstat(stream.fileno())
|
||||
if not stat.S_ISREG(actual.st_mode) or (
|
||||
actual.st_size, actual.st_mtime_ns, actual.st_ino
|
||||
) != (info.st_size, info.st_mtime_ns, info.st_ino):
|
||||
raise ValueError("cache object changed during export")
|
||||
member = tarfile.TarInfo("objects/" + relative)
|
||||
member.size = info.st_size
|
||||
member.mode = 0o644
|
||||
member.mtime = int(info.st_mtime)
|
||||
reader = HashingReader(stream)
|
||||
archive.addfile(member, reader)
|
||||
entries.append({
|
||||
"path": member.name,
|
||||
"size": info.st_size,
|
||||
"sha256": reader.digest.hexdigest(),
|
||||
"mtime_ns": info.st_mtime_ns,
|
||||
})
|
||||
total += info.st_size
|
||||
manifest = dict(metadata, schema=1, mode="delta", objects=entries, touched=touched)
|
||||
data = json.dumps(manifest, ensure_ascii=False, sort_keys=True).encode("utf-8")
|
||||
member = tarfile.TarInfo("manifest.json")
|
||||
member.size = len(data)
|
||||
member.mode = 0o644
|
||||
archive.addfile(member, io.BytesIO(data))
|
||||
return len(entries), total
|
||||
|
||||
|
||||
class NoRedirect(urllib.request.HTTPRedirectHandler):
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
# runtime token 只允许发往明确配置的 Gitea 地址。
|
||||
return None
|
||||
|
||||
|
||||
def request(method, url, token, data=None, headers=None, attempts=3):
|
||||
opener = urllib.request.build_opener(NoRedirect())
|
||||
for attempt in range(attempts):
|
||||
req = urllib.request.Request(url, data=data, method=method, headers={
|
||||
**({"Authorization": "Bearer " + token} if token else {}),
|
||||
"Content-Type": "application/json",
|
||||
**(headers or {}),
|
||||
})
|
||||
try:
|
||||
with opener.open(req, timeout=120) as response:
|
||||
body = response.read(1024 * 1024)
|
||||
return json.loads(body) if body else {}
|
||||
except urllib.error.HTTPError as error:
|
||||
if error.code not in (408, 429, 500, 502, 503, 504) or attempt == attempts - 1:
|
||||
raise RuntimeError(f"artifact {method} failed: HTTP {error.code}") from None
|
||||
except (urllib.error.URLError, TimeoutError, ConnectionError):
|
||||
if attempt == attempts - 1:
|
||||
raise RuntimeError(f"artifact {method} connection failed") from None
|
||||
time.sleep(2 ** attempt)
|
||||
|
||||
|
||||
def artifact_base_url(env):
|
||||
# .runner.address 可指向仅支持 RPC 的领取网关,不能用 GITHUB_SERVER_URL。
|
||||
repository = env["GITHUB_REPOSITORY"]
|
||||
clone_url = env["GENARRATIVE_GITEA_REPOSITORY_URL"]
|
||||
suffix = "/" + repository + ".git"
|
||||
parsed = urllib.parse.urlsplit(clone_url)
|
||||
if (
|
||||
parsed.scheme not in ("http", "https")
|
||||
or not parsed.netloc
|
||||
or parsed.username or parsed.password or parsed.query or parsed.fragment
|
||||
or not parsed.path.endswith(suffix)
|
||||
):
|
||||
raise ValueError("a credential-free HTTP Gitea repository URL is required")
|
||||
return urllib.parse.urlunsplit((
|
||||
parsed.scheme, parsed.netloc, parsed.path[:-len(suffix)], "", ""
|
||||
))
|
||||
|
||||
|
||||
def upload_snapshot(path, name, run_id, base_url, token):
|
||||
"""Gitea 1.26.4 原生 v4:只有完成全部块并校验 SHA256 后才发布。"""
|
||||
producer_match = re.fullmatch(r"rust-cache-v1-([a-z0-9-]+)-attempt-(0|[1-9][0-9]*)", name)
|
||||
if not producer_match or producer_match[1] not in JOBS:
|
||||
raise ValueError("unexpected cache artifact name")
|
||||
producer = f"{producer_match[1]}:{producer_match[2]}"
|
||||
endpoint = f"{base_url}/twirp/github.actions.results.api.v1.ArtifactService"
|
||||
# Gitea 从任务凭据确定 job,只要求请求中的 run ID 与任务所属 run 一致。
|
||||
identity = {"workflowRunBackendId": str(run_id), "name": name}
|
||||
created = request("POST", endpoint + "/CreateArtifact", token, json.dumps({
|
||||
**identity, "version": 4,
|
||||
# Gitea 将剩余小时向下取整成天;多给一天,服务端才实际保留至少七天。
|
||||
"expiresAt": (datetime.now(timezone.utc) + timedelta(days=8)).isoformat(),
|
||||
}).encode())
|
||||
received = urllib.parse.urlsplit(created.get("signedUploadUrl", ""))
|
||||
expected_path = urllib.parse.urlsplit(endpoint).path + "/UploadArtifact"
|
||||
if not created.get("ok") or received.path != expected_path or not received.query:
|
||||
raise ValueError("unexpected artifact upload path")
|
||||
# 服务可能返回外部 AppURL;保留签名,但只连接配置的内部 Gitea 地址。
|
||||
upload_url = endpoint + "/UploadArtifact?" + received.query
|
||||
size = path.stat().st_size
|
||||
digest = hashlib.sha256()
|
||||
blocks = []
|
||||
with path.open("rb") as source:
|
||||
while chunk := source.read(CHUNK_BYTES):
|
||||
# 双层编码中的内层保留所属任务;宿主只回收带此前缀的过期未完成块。
|
||||
block = base64.b64encode(
|
||||
f"genarrative-rust-cache-v1:{producer}:{len(blocks):08d}".encode()
|
||||
).decode()
|
||||
blocks.append(block)
|
||||
digest.update(chunk)
|
||||
request("PUT", upload_url + "&" + urllib.parse.urlencode({
|
||||
"comp": "block", "blockid": block,
|
||||
}), None, chunk, {
|
||||
"Content-Type": "application/octet-stream",
|
||||
})
|
||||
blocklist = "<BlockList>" + "".join(f"<Latest>{block}</Latest>" for block in blocks) + "</BlockList>"
|
||||
request("PUT", upload_url + "&comp=blocklist", None, blocklist.encode(), {
|
||||
"Content-Type": "application/xml",
|
||||
})
|
||||
# Finalize 会消耗服务端块列表;响应不确定时不盲目重试、也不宣告成功。
|
||||
finalized = request("POST", endpoint + "/FinalizeArtifact", token, json.dumps({
|
||||
**identity, "size": str(size), "hash": "sha256:" + digest.hexdigest(),
|
||||
}).encode(), attempts=1)
|
||||
if not finalized.get("ok"):
|
||||
raise ValueError("artifact was not finalized")
|
||||
|
||||
|
||||
def export(env):
|
||||
# 双重限定:手工调用、PR 和其它分支不会扫描、打包或上传对象。
|
||||
if env.get("GITHUB_EVENT_NAME") != "push" or env.get("GITHUB_REF") != "refs/heads/master":
|
||||
return
|
||||
if env.get("GENARRATIVE_CI_RUST_CACHE_EXPORT_READY") != "1":
|
||||
print("[rust-cache] export skipped: cache daemon did not stop cleanly")
|
||||
return
|
||||
job = env.get("GITHUB_JOB")
|
||||
if job not in JOBS:
|
||||
raise ValueError("unexpected Rust cache job")
|
||||
sha = env["GITHUB_SHA"]
|
||||
if not re.fullmatch(r"[a-f0-9]{40}", sha):
|
||||
raise ValueError("a complete source SHA is required")
|
||||
run_id, attempt = int(env["GITHUB_RUN_ID"]), int(env["GITHUB_RUN_ATTEMPT"])
|
||||
if run_id <= 0 or attempt < 0:
|
||||
raise ValueError("invalid run identity")
|
||||
token = env.get("ACTIONS_RUNTIME_TOKEN") or env["GENARRATIVE_GITEA_TOKEN"]
|
||||
base_url = artifact_base_url(env)
|
||||
root = Path(env.get("GENARRATIVE_CI_RUST_CACHE_ROOT", "/opt/genarrative-ci/rust-cache"))
|
||||
baseline_path = root / "export-baseline.json"
|
||||
baseline = json.loads(baseline_path.read_text(encoding="utf-8"))
|
||||
rustc = subprocess.check_output(["rustc", "-vV"], text=True)
|
||||
workspace = str(Path.cwd().resolve())
|
||||
if rustc != (root / "rustc.txt").read_text() or workspace != (root / "workspace.txt").read_text().strip():
|
||||
raise ValueError("cache provenance changed after prepare")
|
||||
metadata = {
|
||||
"repository": env["GITHUB_REPOSITORY"],
|
||||
"run_id": run_id, "run_attempt": attempt, "job": job, "source_sha": sha,
|
||||
"rustc": rustc, "workspace": workspace,
|
||||
"inherited_source_sha": (root / "source-commit.txt").read_text().strip(),
|
||||
"sccache_version": subprocess.check_output([str(root / "sccache"), "--version"], text=True).strip(),
|
||||
}
|
||||
if (root / "base-image.txt").is_file():
|
||||
metadata["base_image"] = (root / "base-image.txt").read_text().strip()
|
||||
name = f"rust-cache-v1-{job}-attempt-{attempt}"
|
||||
try:
|
||||
with tempfile.TemporaryDirectory(prefix="ci-rust-cache-export-") as directory:
|
||||
path = Path(directory) / "snapshot.zip"
|
||||
count, size = pack_snapshot(root, path, metadata, baseline)
|
||||
upload_snapshot(path, name, run_id, base_url, token)
|
||||
finally:
|
||||
baseline_path.unlink(missing_ok=True)
|
||||
print(f"[rust-cache] artifact={name} objects={count} bytes={size} complete=true")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) == 3 and sys.argv[1] == "baseline":
|
||||
save_baseline(Path(os.environ.get("GENARRATIVE_CI_RUST_CACHE_ROOT", "/opt/genarrative-ci/rust-cache")), Path(sys.argv[2]))
|
||||
else:
|
||||
export(os.environ)
|
||||
@@ -39,6 +39,7 @@ usage() {
|
||||
cat <<'EOF'
|
||||
用法:
|
||||
bash scripts/gitea-ci-job-image.sh build
|
||||
bash scripts/gitea-ci-job-image.sh revision
|
||||
bash scripts/gitea-ci-job-image.sh verify [镜像引用]
|
||||
bash scripts/gitea-ci-job-image.sh load-runner [镜像引用]
|
||||
bash scripts/gitea-ci-job-image.sh export <归档路径> [镜像引用]
|
||||
@@ -65,40 +66,15 @@ verify_image() {
|
||||
|
||||
command_name="${1:-}"
|
||||
case "${command_name}" in
|
||||
build)
|
||||
image_revision="$(
|
||||
revision)
|
||||
# 与 build 的 IMAGE_REVISION 使用同一份输入顺序,用于维护器判断基础镜像是否过期。
|
||||
(
|
||||
cd "${repo_root}"
|
||||
{
|
||||
sha256sum \
|
||||
deploy/container/gitea-ci-job.Dockerfile \
|
||||
deploy/container/gitea-ci-job.Dockerfile.dockerignore \
|
||||
deploy/container/gitea-ci-checkout.sh \
|
||||
package.json \
|
||||
package-lock.json \
|
||||
apps/admin-web/package.json \
|
||||
apps/ai-game-creator-shell/package.json \
|
||||
apps/desktop-shell/package.json \
|
||||
apps/mobile-shell/package.json \
|
||||
apps/preview-deployer-web/package.json \
|
||||
packages/image-canvas-core/package.json \
|
||||
packages/image-canvas-react/package.json \
|
||||
packages/shared/package.json \
|
||||
tools/spine-json-export-validator/package.json \
|
||||
apps/ai-game-creator-shell/src-tauri/Cargo.toml \
|
||||
apps/ai-game-creator-shell/src-tauri/Cargo.lock \
|
||||
server-rs/Cargo.toml \
|
||||
server-rs/Cargo.lock \
|
||||
apps/desktop-shell/src-tauri/Cargo.toml \
|
||||
apps/desktop-shell/src-tauri/Cargo.lock
|
||||
find server-rs/crates plugins/agc-*-editor/native/*-editor-bridge \
|
||||
\( -name Cargo.toml -o -path 'plugins/agc-*-editor/native/*-editor-bridge/*' \) \
|
||||
-type f -print0 \
|
||||
| sort -z \
|
||||
| xargs -0 -r sha256sum
|
||||
} \
|
||||
| sha256sum \
|
||||
| awk '{ print $1 }'
|
||||
)"
|
||||
write_build_context_file_list | xargs -0 -r sha256sum | sha256sum | awk '{ print $1 }'
|
||||
)
|
||||
;;
|
||||
build)
|
||||
image_revision="$(bash "${BASH_SOURCE[0]}" revision)"
|
||||
npm_lock_sha256="$(sha256sum "${repo_root}/package-lock.json")"
|
||||
npm_lock_sha256="${npm_lock_sha256%% *}"
|
||||
server_rust_lock_sha256="$(sha256sum "${repo_root}/server-rs/Cargo.lock")"
|
||||
|
||||
@@ -0,0 +1,447 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Gitea Runner 领取屏障;控制 socket 仅供宿主缓存维护任务使用。"""
|
||||
|
||||
import http.client
|
||||
import http.server
|
||||
import gzip
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import socketserver
|
||||
import threading
|
||||
import time
|
||||
import urllib.parse
|
||||
import zlib
|
||||
|
||||
MAX_BODY = 32 * 1024 * 1024
|
||||
HOP_HEADERS = {
|
||||
"connection", "keep-alive", "proxy-authenticate", "proxy-authorization",
|
||||
"te", "trailer", "transfer-encoding", "upgrade", "host", "content-length",
|
||||
}
|
||||
RPC_PREFIX = "/api/actions/runner.v1.RunnerService/"
|
||||
|
||||
|
||||
def protobuf_fields(data):
|
||||
"""只解码已核对的 actions-proto-go v0.4.1 字段,不引入 protobuf 运行时。"""
|
||||
fields = {}
|
||||
offset = 0
|
||||
|
||||
def varint():
|
||||
nonlocal offset
|
||||
value = 0
|
||||
for shift in range(0, 70, 7):
|
||||
if offset >= len(data):
|
||||
raise ValueError("truncated protobuf")
|
||||
byte = data[offset]
|
||||
offset += 1
|
||||
value |= (byte & 127) << shift
|
||||
if byte < 128:
|
||||
return value
|
||||
raise ValueError("invalid protobuf varint")
|
||||
|
||||
while offset < len(data):
|
||||
tag = varint()
|
||||
number, wire = tag >> 3, tag & 7
|
||||
if not number:
|
||||
raise ValueError("invalid protobuf tag")
|
||||
if wire == 0:
|
||||
value = varint()
|
||||
elif wire in (1, 2, 5):
|
||||
length = varint() if wire == 2 else (8 if wire == 1 else 4)
|
||||
if offset + length > len(data):
|
||||
raise ValueError("truncated protobuf field")
|
||||
value = data[offset:offset + length]
|
||||
offset += length
|
||||
else:
|
||||
raise ValueError("unsupported protobuf wire type")
|
||||
fields.setdefault(number, []).append(value)
|
||||
return fields
|
||||
|
||||
|
||||
def single(fields, number, default=None):
|
||||
values = fields.get(number, [])
|
||||
if len(values) > 1:
|
||||
raise ValueError("ambiguous tracking field")
|
||||
return values[0] if values else default
|
||||
|
||||
|
||||
def rpc_fields(body, headers):
|
||||
encoding = headers.get("Content-Encoding", "identity").lower()
|
||||
if encoding == "gzip":
|
||||
with gzip.GzipFile(fileobj=io.BytesIO(body)) as stream:
|
||||
body = stream.read(MAX_BODY + 1)
|
||||
elif encoding != "identity":
|
||||
raise ValueError("unsupported RPC encoding")
|
||||
if len(body) > MAX_BODY:
|
||||
raise ValueError("decoded RPC too large")
|
||||
if headers.get("Content-Type", "").split(";", 1)[0] != "application/proto":
|
||||
raise ValueError("task tracking requires Connect protobuf")
|
||||
return protobuf_fields(body)
|
||||
|
||||
|
||||
def task_state(fields):
|
||||
nested = single(fields, 1)
|
||||
if not isinstance(nested, bytes):
|
||||
raise ValueError("missing task state")
|
||||
state = protobuf_fields(nested)
|
||||
task_id = positive_id(single(state, 1))
|
||||
result = single(state, 2, 0)
|
||||
if type(result) is not int or result not in range(5):
|
||||
raise ValueError("unknown task result")
|
||||
return task_id, result
|
||||
|
||||
|
||||
def positive_id(value):
|
||||
if type(value) is not int or not 0 < value < 2 ** 63:
|
||||
raise ValueError("invalid task ID")
|
||||
return str(value)
|
||||
|
||||
|
||||
class Gate:
|
||||
def __init__(self, directory):
|
||||
self.directory = Path(directory)
|
||||
self.directory.mkdir(parents=True, exist_ok=True)
|
||||
self.lock = threading.Lock()
|
||||
self.inflight = 0
|
||||
self.last_fetch_peer = None
|
||||
self.last_fetch_at = None
|
||||
self.tasks = {}
|
||||
self.uncertain = (self.directory / "uncertain").exists()
|
||||
try:
|
||||
if (self.directory / "tasks.json").exists():
|
||||
tasks = json.loads((self.directory / "tasks.json").read_text())
|
||||
if (not isinstance(tasks, dict) or any(
|
||||
positive_id(int(key)) != key or type(value) is not bool
|
||||
for key, value in tasks.items())):
|
||||
raise ValueError("invalid task ledger")
|
||||
self.tasks = tasks
|
||||
except (ValueError, TypeError, OSError):
|
||||
self.uncertain = True
|
||||
self.mark("uncertain")
|
||||
if (self.directory / "inflight").exists():
|
||||
self.uncertain = True
|
||||
self.mark("uncertain")
|
||||
self.paused = (self.directory / "paused").exists() or self.uncertain
|
||||
|
||||
def mark(self, name):
|
||||
with (self.directory / name).open("w", encoding="ascii") as stream:
|
||||
stream.write("1\n")
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
self.sync_directory()
|
||||
|
||||
def sync_directory(self):
|
||||
descriptor = os.open(self.directory, os.O_RDONLY | os.O_DIRECTORY)
|
||||
try:
|
||||
os.fsync(descriptor)
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
|
||||
def snapshot(self):
|
||||
return {"paused": self.paused, "inflight": self.inflight,
|
||||
"active_tasks": len(self.tasks), "task_ids": sorted(self.tasks),
|
||||
"uncertain": self.uncertain, "last_fetch_peer": self.last_fetch_peer,
|
||||
"last_fetch_at": self.last_fetch_at}
|
||||
|
||||
def save_tasks(self):
|
||||
temporary = self.directory / "tasks.json.tmp"
|
||||
with temporary.open("w", encoding="ascii") as stream:
|
||||
json.dump(self.tasks, stream, sort_keys=True)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temporary, self.directory / "tasks.json")
|
||||
self.sync_directory()
|
||||
|
||||
def fail_closed(self):
|
||||
with self.lock:
|
||||
self.uncertain = self.paused = True
|
||||
self.mark("uncertain")
|
||||
|
||||
def assigned(self, fields):
|
||||
task = single(fields, 1)
|
||||
if task is None:
|
||||
return
|
||||
if not isinstance(task, bytes):
|
||||
raise ValueError("invalid fetched task")
|
||||
task_id = positive_id(single(protobuf_fields(task), 1))
|
||||
with self.lock:
|
||||
self.tasks[task_id] = False
|
||||
# 必须先持久化,再把领取结果交给 runner;仅保存 ID,不保存 secrets。
|
||||
self.save_tasks()
|
||||
|
||||
def reported(self, method, request, response):
|
||||
if method == "UpdateLog":
|
||||
task_id = positive_id(single(request, 1))
|
||||
index = single(request, 2, 0)
|
||||
ack = single(response, 1, 0)
|
||||
no_more = single(request, 4, 0)
|
||||
if (type(index) is not int or type(ack) is not int
|
||||
or index < 0 or ack < 0 or no_more not in (0, 1)):
|
||||
raise ValueError("invalid log acknowledgement")
|
||||
finalized = no_more == 1 and ack == index + len(request.get(3, []))
|
||||
else:
|
||||
task_id, result = task_state(request)
|
||||
response_id, response_result = task_state(response)
|
||||
if response_id != task_id:
|
||||
raise ValueError("mismatched task acknowledgement")
|
||||
# 取消响应可能出现在任务执行途中,必须等 runner 自己报告终态。
|
||||
finalized = result != 0 and response_result != 0
|
||||
output_keys = {single(protobuf_fields(entry), 1, b"")
|
||||
for entry in request.get(2, [])}
|
||||
finalized = finalized and output_keys.issubset(set(response.get(2, [])))
|
||||
with self.lock:
|
||||
if task_id not in self.tasks:
|
||||
# 客户端可能未读到已成功发出的终态响应而重试;v2.0.0 仅在
|
||||
# executor 清理后的 Close 中发送终态,不为幂等重报增加墓碑账本。
|
||||
if method == "UpdateTask" and finalized:
|
||||
return
|
||||
raise ValueError("report for untracked task; idle bootstrap required")
|
||||
if method == "UpdateLog" and finalized:
|
||||
self.tasks[task_id] = True
|
||||
self.save_tasks()
|
||||
elif method == "UpdateTask" and finalized and self.tasks[task_id]:
|
||||
# act_runner Reporter.Close 在 executor 清理之后先封存日志再报终态。
|
||||
del self.tasks[task_id]
|
||||
self.save_tasks()
|
||||
|
||||
def record_fetch(self, peer):
|
||||
with self.lock:
|
||||
self.last_fetch_peer = peer
|
||||
self.last_fetch_at = time.time()
|
||||
|
||||
def control(self, action):
|
||||
with self.lock:
|
||||
if action == "pause":
|
||||
self.mark("paused")
|
||||
self.paused = True
|
||||
elif action == "resume":
|
||||
if self.uncertain:
|
||||
return {**self.snapshot(), "error": "upstream completion uncertain; operator recovery required"}
|
||||
(self.directory / "paused").unlink(missing_ok=True)
|
||||
self.sync_directory()
|
||||
self.paused = False
|
||||
elif action != "status":
|
||||
return {**self.snapshot(), "error": "unknown action"}
|
||||
return self.snapshot()
|
||||
|
||||
def enter(self):
|
||||
with self.lock:
|
||||
if self.paused or self.uncertain:
|
||||
return False
|
||||
# 必须先落盘再转发;崩溃后不能把遗留的领取请求误认为已完成。
|
||||
self.mark("inflight")
|
||||
self.inflight += 1
|
||||
return True
|
||||
|
||||
def leave(self, completed):
|
||||
with self.lock:
|
||||
self.inflight -= 1
|
||||
if not completed:
|
||||
self.uncertain = self.paused = True
|
||||
self.mark("uncertain")
|
||||
if self.inflight == 0 and not self.uncertain:
|
||||
(self.directory / "inflight").unlink(missing_ok=True)
|
||||
self.sync_directory()
|
||||
|
||||
|
||||
def read_body(stream, headers):
|
||||
"""解码 HTTP 请求;拒绝含糊 framing,不依赖下游连接关闭。"""
|
||||
encodings = headers.get_all("Transfer-Encoding", [])
|
||||
lengths = headers.get_all("Content-Length", [])
|
||||
if encodings and lengths:
|
||||
raise ValueError("ambiguous request framing")
|
||||
if len(lengths) > 1 or len(encodings) > 1:
|
||||
raise ValueError("duplicate request framing")
|
||||
|
||||
def exact(length):
|
||||
data = stream.read(length)
|
||||
if len(data) != length:
|
||||
raise ValueError("incomplete request body")
|
||||
return data
|
||||
|
||||
if not encodings:
|
||||
length = int(lengths[0]) if lengths else 0
|
||||
if not 0 <= length <= MAX_BODY:
|
||||
raise ValueError("request too large")
|
||||
return exact(length)
|
||||
if encodings[0].strip().lower() != "chunked":
|
||||
raise ValueError("unsupported transfer encoding")
|
||||
body = bytearray()
|
||||
while True:
|
||||
line = stream.readline(8193)
|
||||
if len(line) > 8192 or not line.endswith(b"\r\n"):
|
||||
raise ValueError("invalid chunk header")
|
||||
length = int(line.split(b";", 1)[0].strip(), 16)
|
||||
if length < 0 or len(body) + length > MAX_BODY:
|
||||
raise ValueError("request too large")
|
||||
if length == 0:
|
||||
trailer_size = 0
|
||||
while True:
|
||||
line = stream.readline(8193)
|
||||
trailer_size += len(line)
|
||||
if trailer_size > 8192 or not line.endswith(b"\r\n"):
|
||||
raise ValueError("invalid request trailers")
|
||||
if line == b"\r\n":
|
||||
return bytes(body)
|
||||
body.extend(exact(length))
|
||||
if exact(2) != b"\r\n":
|
||||
raise ValueError("invalid chunk terminator")
|
||||
|
||||
|
||||
class Proxy(http.server.BaseHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.1"
|
||||
|
||||
def log_message(self, *_args):
|
||||
pass # 不输出 RPC 认证头、请求内容或带认证信息的 URL。
|
||||
|
||||
def reply(self, status, body, headers=()):
|
||||
delivered = False
|
||||
try:
|
||||
self.send_response(status)
|
||||
excluded = HOP_HEADERS | {
|
||||
item.strip().lower() for key, value in headers if key.lower() == "connection"
|
||||
for item in value.split(",")
|
||||
}
|
||||
for key, value in headers:
|
||||
if key.lower() not in excluded:
|
||||
self.send_header(key, value)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.send_header("Connection", "close")
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
self.wfile.flush()
|
||||
delivered = True
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
self.close_connection = True
|
||||
return delivered
|
||||
|
||||
def do_POST(self):
|
||||
path = urllib.parse.urlsplit(self.path)
|
||||
if (path.scheme or path.netloc or not path.path.startswith("/api/actions/")
|
||||
or "%" in path.path or any(p in (".", "..") for p in path.path.split("/"))):
|
||||
self.reply(404, b"runner RPC only\n")
|
||||
return
|
||||
try:
|
||||
body = read_body(self.rfile, self.headers)
|
||||
except (ValueError, OSError):
|
||||
self.reply(400, b"invalid request body\n")
|
||||
return
|
||||
method = path.path.removeprefix(RPC_PREFIX) if path.path.startswith(RPC_PREFIX) else ""
|
||||
is_fetch = method == "FetchTask"
|
||||
if is_fetch:
|
||||
# 包括暂停时被拒绝的请求;只记录网络来源与时间以证明 daemon 路由。
|
||||
self.server.gate.record_fetch(self.client_address[0])
|
||||
if not self.server.gate.enter():
|
||||
self.reply(503, b"runner maintenance\n")
|
||||
return
|
||||
completed = False
|
||||
connection = None
|
||||
try:
|
||||
upstream = self.server.upstream
|
||||
cls = http.client.HTTPSConnection if upstream.scheme == "https" else http.client.HTTPConnection
|
||||
connection = cls(upstream.hostname, upstream.port, timeout=None)
|
||||
try:
|
||||
connection.connect()
|
||||
except OSError:
|
||||
# TCP/TLS 连接阶段尚未发送 RPC,不存在服务端分配事务。
|
||||
completed = True
|
||||
raise
|
||||
excluded = HOP_HEADERS | {
|
||||
name.strip().lower() for name in self.headers.get("Connection", "").split(",")
|
||||
}
|
||||
headers = {key: value for key, value in self.headers.items() if key.lower() not in excluded}
|
||||
headers["Content-Length"] = str(len(body))
|
||||
connection.request("POST", self.path, body=body, headers=headers)
|
||||
response = connection.getresponse()
|
||||
result = bytearray()
|
||||
oversized = False
|
||||
# 客户端即使断开,也继续读取上游完整响应,再释放领取屏障。
|
||||
while block := response.read(65536):
|
||||
if len(result) + len(block) <= MAX_BODY and not oversized:
|
||||
result.extend(block)
|
||||
else:
|
||||
oversized = True
|
||||
result.clear()
|
||||
if response.length not in (None, 0):
|
||||
raise http.client.IncompleteRead(bytes(result), response.length)
|
||||
completed = True
|
||||
if is_fetch:
|
||||
try:
|
||||
if oversized or response.status != 200:
|
||||
raise ValueError("unknown FetchTask result")
|
||||
self.server.gate.assigned(rpc_fields(bytes(result), response.headers))
|
||||
except (ValueError, TypeError, OSError, EOFError, zlib.error):
|
||||
self.server.gate.fail_closed()
|
||||
self.server.gate.leave(True)
|
||||
is_fetch = False
|
||||
if oversized:
|
||||
self.reply(502, b"upstream response too large\n")
|
||||
else:
|
||||
# 日志封存先记账再返回,避免 runner 立即发终态时抢先读到旧账本。
|
||||
if method == "UpdateLog" and response.status == 200:
|
||||
self.observe_report(method, body, bytes(result), response.headers)
|
||||
delivered = self.reply(response.status, bytes(result), response.getheaders())
|
||||
if method == "UpdateTask" and response.status == 200 and delivered:
|
||||
self.observe_report(method, body, bytes(result), response.headers)
|
||||
except (OSError, http.client.HTTPException, ValueError):
|
||||
self.reply(502, b"runner upstream unavailable\n")
|
||||
finally:
|
||||
if connection is not None:
|
||||
connection.close()
|
||||
if is_fetch:
|
||||
self.server.gate.leave(completed)
|
||||
|
||||
def observe_report(self, method, request, response, headers):
|
||||
try:
|
||||
self.server.gate.reported(method, rpc_fields(request, self.headers),
|
||||
rpc_fields(response, headers))
|
||||
except (ValueError, TypeError, OSError, EOFError, zlib.error):
|
||||
self.server.gate.fail_closed()
|
||||
|
||||
|
||||
class Control(socketserver.StreamRequestHandler):
|
||||
def handle(self):
|
||||
try:
|
||||
line = self.rfile.readline(4097)
|
||||
if len(line) > 4096 or not line.endswith(b"\n"):
|
||||
raise ValueError("invalid control request")
|
||||
request = json.loads(line)
|
||||
result = self.server.gate.control(request["action"])
|
||||
except (ValueError, KeyError, TypeError, OSError):
|
||||
result = {"error": "invalid control request"}
|
||||
self.wfile.write(json.dumps(result).encode("utf-8") + b"\n")
|
||||
|
||||
|
||||
class ControlServer(socketserver.ThreadingUnixStreamServer):
|
||||
daemon_threads = True
|
||||
|
||||
|
||||
def create_proxy(address, upstream, gate):
|
||||
parsed = urllib.parse.urlsplit(upstream)
|
||||
if (parsed.scheme not in ("http", "https") or not parsed.hostname
|
||||
or parsed.username or parsed.password or parsed.path not in ("", "/")
|
||||
or parsed.query or parsed.fragment):
|
||||
raise ValueError("upstream must be an HTTP(S) origin")
|
||||
server = http.server.ThreadingHTTPServer(address, Proxy)
|
||||
server.upstream = parsed
|
||||
server.gate = gate
|
||||
return server
|
||||
|
||||
|
||||
def main():
|
||||
gate = Gate(os.environ.get("GITEA_GATE_CONTROL_DIR", "/control"))
|
||||
socket_path = gate.directory / "gate.sock"
|
||||
socket_path.unlink(missing_ok=True)
|
||||
control = ControlServer(str(socket_path), Control)
|
||||
control.gate = gate
|
||||
os.chmod(socket_path, 0o600)
|
||||
proxy = create_proxy(("0.0.0.0", 8080), os.environ.get(
|
||||
"GITEA_RUNNER_UPSTREAM", "http://gitea:3000"), gate)
|
||||
threading.Thread(target=control.serve_forever, daemon=True).start()
|
||||
proxy.serve_forever()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,115 @@
|
||||
"""定向回收 Gitea 1.26.4 不会自动过期的本仓库缓存上传块。"""
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import math
|
||||
from pathlib import Path
|
||||
import re
|
||||
import stat
|
||||
|
||||
|
||||
JOBS = {
|
||||
"ai-game-creator-shell-rust-lane-1",
|
||||
"ai-game-creator-shell-rust-lane-2",
|
||||
"ai-game-creator-shell-rust-smoke",
|
||||
"ai-game-creator-shell-rust-crates",
|
||||
"backend-tests",
|
||||
"native-shell-tests",
|
||||
}
|
||||
CHUNK = re.compile(r"block-([1-9][0-9]*)-([1-9][0-9]*)-(0|[1-9][0-9]*)-([A-Za-z0-9_-]+={0,2})")
|
||||
BLOCKLIST = re.compile(r"([1-9][0-9]*)-([1-9][0-9]*)-blocklist")
|
||||
MARKER = re.compile(r"genarrative-rust-cache-v1:([a-z0-9-]+):(0|[1-9][0-9]*):([0-9]{8})")
|
||||
|
||||
|
||||
def decode_owner(encoded):
|
||||
"""Gitea 文件名编码为 URL-base64(生产者上传的标准 base64 blockid)。"""
|
||||
try:
|
||||
block_id = base64.b64decode(encoded, altchars=b"-_", validate=True)
|
||||
if base64.urlsafe_b64encode(block_id).decode() != encoded:
|
||||
return None
|
||||
marker = base64.b64decode(block_id, validate=True)
|
||||
if base64.b64encode(marker) != block_id:
|
||||
return None
|
||||
match = MARKER.fullmatch(marker.decode("ascii"))
|
||||
if match and match[1] in JOBS:
|
||||
return match[1], int(match[2])
|
||||
except (binascii.Error, UnicodeError, ValueError):
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def _unchanged_old_regular(path, previous, cutoff):
|
||||
try:
|
||||
current = path.lstat()
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
return (
|
||||
stat.S_ISREG(current.st_mode)
|
||||
and current.st_mtime < cutoff
|
||||
and (current.st_ino, current.st_size, current.st_mtime_ns)
|
||||
== (previous.st_ino, previous.st_size, previous.st_mtime_ns)
|
||||
)
|
||||
|
||||
|
||||
def cleanup_upload_chunks(storage_root: Path, eligible_runs: set[int], cutoff_timestamp: float) -> int:
|
||||
"""调用方先用仓库 API 确认这些 master run 已结束超过保留期限。"""
|
||||
storage_root = Path(storage_root)
|
||||
if not storage_root.is_absolute() or storage_root.resolve(strict=True) != storage_root:
|
||||
raise ValueError("artifact storage root must be an existing absolute real directory")
|
||||
if not stat.S_ISDIR(storage_root.lstat().st_mode):
|
||||
raise ValueError("artifact storage root must be a directory")
|
||||
if not math.isfinite(cutoff_timestamp) or cutoff_timestamp <= 0:
|
||||
raise ValueError("invalid artifact cleanup cutoff")
|
||||
if any(type(run_id) is not int or run_id <= 0 for run_id in eligible_runs):
|
||||
raise ValueError("invalid eligible run ID")
|
||||
temporary = storage_root / "tmp-upload"
|
||||
if not temporary.exists() and not temporary.is_symlink():
|
||||
return 0
|
||||
if temporary.is_symlink() or not temporary.is_dir():
|
||||
raise ValueError("artifact temporary path must be a real directory")
|
||||
|
||||
deleted = 0
|
||||
for run_id in sorted(eligible_runs):
|
||||
directory = temporary / f"run-{run_id}-v4"
|
||||
if directory.is_symlink() or not directory.is_dir():
|
||||
continue
|
||||
groups = {}
|
||||
unknown = False
|
||||
for path in directory.iterdir():
|
||||
chunk = CHUNK.fullmatch(path.name)
|
||||
blocklist = BLOCKLIST.fullmatch(path.name)
|
||||
match = chunk or blocklist
|
||||
if match and int(match[1]) == run_id:
|
||||
artifact_id = int(match[2])
|
||||
owner = decode_owner(chunk[4]) if chunk else None
|
||||
owned = owner is not None if chunk else True
|
||||
else:
|
||||
# 若仍能识别 artifact ID,仅保护该组;无法识别时保护整个 run。
|
||||
prefix = re.match(rf"(?:block-)?{run_id}-([1-9][0-9]*)-", path.name)
|
||||
if not prefix:
|
||||
unknown = True
|
||||
break
|
||||
artifact_id, owner, owned = int(prefix[1]), None, False
|
||||
group = groups.setdefault(artifact_id, {"files": [], "owners": set(), "safe": True})
|
||||
try:
|
||||
info = path.lstat()
|
||||
except FileNotFoundError:
|
||||
group["safe"] = False
|
||||
continue
|
||||
group["safe"] &= owned and stat.S_ISREG(info.st_mode) and info.st_mtime < cutoff_timestamp
|
||||
if owner:
|
||||
group["owners"].add(owner)
|
||||
group["files"].append((path, info))
|
||||
if unknown:
|
||||
continue
|
||||
for group in groups.values():
|
||||
if not group["safe"] or len(group["owners"]) != 1:
|
||||
continue
|
||||
files = group["files"]
|
||||
if not all(_unchanged_old_regular(path, info, cutoff_timestamp) for path, info in files):
|
||||
continue
|
||||
# 保留目录,不使用递归删除;没有所属块证明的孤立 blocklist 也不会删除。
|
||||
for path, _ in files:
|
||||
path.unlink()
|
||||
deleted += 1
|
||||
return deleted
|
||||
File diff suppressed because it is too large
Load Diff
@@ -142,6 +142,38 @@ function backendStepIndex(stepName: string) {
|
||||
}
|
||||
|
||||
describe('project CI workflow', () => {
|
||||
it('publishes cache deltas only after Rust reporting on non-cancelled master pushes', () => {
|
||||
const producers = [
|
||||
'ai-game-creator-shell-rust-lane-1',
|
||||
'ai-game-creator-shell-rust-lane-2',
|
||||
'ai-game-creator-shell-rust-smoke',
|
||||
'ai-game-creator-shell-rust-crates',
|
||||
'backend-tests',
|
||||
'native-shell-tests',
|
||||
];
|
||||
for (const job of jobNames) {
|
||||
if (!producers.includes(job)) {
|
||||
expect(jobSection(job)).not.toContain('export-gitea-rust-cache.py');
|
||||
continue;
|
||||
}
|
||||
const publish = stepSection(job, 'Publish master Rust cache artifact');
|
||||
expect(publish).toContain(
|
||||
"if: ${{ !cancelled() && github.event_name == 'push' && github.ref == 'refs/heads/master' }}",
|
||||
);
|
||||
expect(publish).toContain('GENARRATIVE_GITEA_TOKEN: ${{ github.token }}');
|
||||
expect(publish).toContain('continue-on-error: true');
|
||||
expect(publish).toContain(
|
||||
'run: python3 scripts/export-gitea-rust-cache.py',
|
||||
);
|
||||
const section = jobSection(job);
|
||||
expect(
|
||||
section.indexOf('Publish master Rust cache artifact'),
|
||||
).toBeGreaterThan(
|
||||
section.indexOf('run: bash scripts/ci-rust-cache.sh report'),
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it('uses isolated compilation caching for every Rust test job', () => {
|
||||
const firstRustSteps = {
|
||||
'ai-game-creator-shell-rust-lane-1':
|
||||
@@ -331,7 +363,8 @@ describe('project CI workflow', () => {
|
||||
const rustLock = 'apps/ai-game-creator-shell/src-tauri/Cargo.lock';
|
||||
|
||||
for (const workspaceManifest of workspaceManifests) {
|
||||
expect(imageBuildScript.split(workspaceManifest)).toHaveLength(3);
|
||||
// 构建上下文与 revision 共用一份清单,不重复枚举 manifest。
|
||||
expect(imageBuildScript.split(workspaceManifest)).toHaveLength(2);
|
||||
expect(imageDockerignore).toContain(`!${workspaceManifest}`);
|
||||
expect(imageDockerfile).toContain(
|
||||
`COPY ${workspaceManifest} /usr/local/share/genarrative-ci/npm/${workspaceManifest}`,
|
||||
@@ -339,8 +372,8 @@ describe('project CI workflow', () => {
|
||||
}
|
||||
|
||||
for (const [path, expectedCount] of [
|
||||
[rustManifest, 2],
|
||||
[rustLock, 3],
|
||||
[rustManifest, 1],
|
||||
[rustLock, 2],
|
||||
] as const) {
|
||||
expect(imageBuildScript.split(path)).toHaveLength(expectedCount + 1);
|
||||
expect(imageDockerignore).toContain(`!${path}`);
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Targeted regression tests for the trusted Gitea Rust-cache image builder."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import shlex
|
||||
import stat
|
||||
import subprocess
|
||||
import tempfile
|
||||
import textwrap
|
||||
import unittest
|
||||
|
||||
|
||||
REPOSITORY_ROOT = Path(__file__).resolve().parent.parent
|
||||
BUILDER_NAME = "build-gitea-rust-cache.sh"
|
||||
FIXED_SHA = "a" * 40
|
||||
MASTER_SHA = "b" * 40
|
||||
|
||||
|
||||
class GiteaRustCacheBuilderTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.temporary_directory = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
|
||||
self.root = Path(self.temporary_directory.name)
|
||||
self.repo = self.root / "repo"
|
||||
scripts = self.repo / "scripts"
|
||||
scripts.mkdir(parents=True)
|
||||
shutil.copy2(REPOSITORY_ROOT / "scripts" / BUILDER_NAME, scripts / BUILDER_NAME)
|
||||
(scripts / "gitea-ci-job-image.sh").write_text("#!/usr/bin/env bash\nexit 0\n")
|
||||
(scripts / "ci-rust-cache.sh").write_text("#!/usr/bin/env bash\nexit 0\n")
|
||||
self.trace = self.root / "trace.log"
|
||||
self.bin = self.root / "bin"
|
||||
self.bin.mkdir()
|
||||
self.write_fake_tools()
|
||||
self.execution_bin = self.wsl_path(self.bin)
|
||||
self.wsl_fake_root: str | None = None
|
||||
if os.name == "nt":
|
||||
self.wsl_fake_root = f"/tmp/gitea-cache-builder-{self.root.name}"
|
||||
subprocess.run(
|
||||
[
|
||||
"bash",
|
||||
"-c",
|
||||
"rm -rf {root}; mkdir -p {root}/bin; cp {source}/* {root}/bin/; chmod +x {root}/bin/*".format(
|
||||
root=shlex.quote(self.wsl_fake_root),
|
||||
source=shlex.quote(self.wsl_path(self.bin)),
|
||||
),
|
||||
],
|
||||
check=True,
|
||||
)
|
||||
self.execution_bin = f"{self.wsl_fake_root}/bin"
|
||||
|
||||
def tearDown(self) -> None:
|
||||
if self.wsl_fake_root is not None:
|
||||
subprocess.run(
|
||||
["bash", "-c", f"rm -rf {shlex.quote(self.wsl_fake_root)}"],
|
||||
check=False,
|
||||
)
|
||||
self.temporary_directory.cleanup()
|
||||
|
||||
def write_tool(self, name: str, source: str) -> None:
|
||||
target = self.bin / name
|
||||
target.write_bytes(textwrap.dedent(source).encode("utf-8"))
|
||||
target.chmod(target.stat().st_mode | stat.S_IXUSR)
|
||||
|
||||
def write_fake_tools(self) -> None:
|
||||
self.write_tool(
|
||||
"git",
|
||||
"""#!/usr/bin/env bash
|
||||
set -eu
|
||||
printf 'git:%s\\n' "$*" >> "$TRACE"
|
||||
if [[ "$1" == '-C' ]]; then shift 2; fi
|
||||
case "$1" in
|
||||
fetch|cat-file) exit 0 ;;
|
||||
merge-base)
|
||||
[[ "${FAKE_NON_ANCESTOR:-}" != 1 ]]
|
||||
exit
|
||||
;;
|
||||
rev-parse)
|
||||
if [[ "$2" == 'FETCH_HEAD^{commit}' ]]; then
|
||||
printf '%s\\n' "$FAKE_MASTER_SHA"
|
||||
else
|
||||
printf '%s\\n' "$FAKE_FIXED_SHA"
|
||||
fi
|
||||
;;
|
||||
archive) printf 'archived %s\\n' "$2" ;;
|
||||
*) echo "unexpected git command: $*" >&2; exit 9 ;;
|
||||
esac
|
||||
""",
|
||||
)
|
||||
self.write_tool(
|
||||
"docker",
|
||||
"""#!/usr/bin/env bash
|
||||
set -eu
|
||||
# Drain the archive pipeline before logging to avoid concurrent
|
||||
# appends to the shared trace on Windows/WSL filesystems.
|
||||
if [[ "$1" == cp && "$2" == '-' ]]; then
|
||||
cat >/dev/null
|
||||
fi
|
||||
printf 'docker:%s\\n' "$*" >> "$TRACE"
|
||||
if [[ "$1" == image && "$2" == inspect ]]; then
|
||||
printf 'sha256:%064d\\n' 0
|
||||
elif [[ "$1" == run && " $* " == *' --detach '* ]]; then
|
||||
printf 'fake-container\\n'
|
||||
elif [[ "$1" == exec && " $* " == *' bash -s '* ]]; then
|
||||
cat >/dev/null
|
||||
fi
|
||||
""",
|
||||
)
|
||||
self.write_tool(
|
||||
"curl",
|
||||
"""#!/usr/bin/env bash
|
||||
set -eu
|
||||
while [[ "$#" -gt 0 ]]; do
|
||||
if [[ "$1" == --output ]]; then touch "$2"; exit 0; fi
|
||||
shift
|
||||
done
|
||||
exit 9
|
||||
""",
|
||||
)
|
||||
self.write_tool("sha256sum", "#!/usr/bin/env bash\ncat >/dev/null\n")
|
||||
self.write_tool(
|
||||
"tar",
|
||||
"""#!/usr/bin/env bash
|
||||
set -eu
|
||||
while [[ "$#" -gt 0 ]]; do
|
||||
if [[ "$1" == --directory ]]; then
|
||||
mkdir -p "$2/sccache-v0.18.0-x86_64-unknown-linux-musl"
|
||||
printf '#!/usr/bin/env bash\\nexit 0\\n' > "$2/sccache-v0.18.0-x86_64-unknown-linux-musl/sccache"
|
||||
chmod +x "$2/sccache-v0.18.0-x86_64-unknown-linux-musl/sccache"
|
||||
exit 0
|
||||
fi
|
||||
shift
|
||||
done
|
||||
exit 9
|
||||
""",
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def wsl_path(path: Path) -> str:
|
||||
value = path.resolve().as_posix()
|
||||
if len(value) >= 3 and value[1] == ":":
|
||||
return f"/mnt/{value[0].lower()}{value[2:]}"
|
||||
return value
|
||||
|
||||
def run_builder(self, *arguments: str, non_ancestor: bool = False) -> subprocess.CompletedProcess[str]:
|
||||
script_path = self.wsl_path(self.repo / "scripts" / BUILDER_NAME)
|
||||
assignments = {
|
||||
"TRACE": self.wsl_path(self.trace),
|
||||
"FAKE_FIXED_SHA": FIXED_SHA,
|
||||
"FAKE_MASTER_SHA": MASTER_SHA,
|
||||
"FAKE_NON_ANCESTOR": "1" if non_ancestor else "",
|
||||
}
|
||||
exports = "; ".join(
|
||||
f"export {name}={shlex.quote(value)}" for name, value in assignments.items()
|
||||
)
|
||||
command = (
|
||||
f"unset CI; {exports}; export PATH={shlex.quote(self.execution_bin)}:\"$PATH\"; cd /; "
|
||||
f"exec bash {shlex.quote(script_path)} "
|
||||
+ " ".join(shlex.quote(argument) for argument in arguments)
|
||||
)
|
||||
return subprocess.run(
|
||||
["bash", "-c", command],
|
||||
env=os.environ,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
|
||||
def trace_text(self) -> str:
|
||||
return self.trace.read_bytes().decode("utf-8", errors="replace")
|
||||
|
||||
def test_archives_requested_master_ancestor_and_uses_its_helper(self) -> None:
|
||||
result = self.run_builder("trusted-base", "candidate", FIXED_SHA)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn(f"snapshot_source={FIXED_SHA}", result.stdout)
|
||||
trace = self.trace_text()
|
||||
self.assertIn(f"archive {FIXED_SHA}", trace)
|
||||
self.assertIn(
|
||||
"docker:exec fake-container cp "
|
||||
"/workspace/GenarrativeAI/Genarrative/scripts/ci-rust-cache.sh "
|
||||
"/tmp/ci-rust-cache.sh",
|
||||
trace,
|
||||
)
|
||||
|
||||
def test_rejects_commit_outside_fetched_master(self) -> None:
|
||||
result = self.run_builder("trusted-base", "candidate", FIXED_SHA, non_ancestor=True)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("not contained in fetched master", result.stderr)
|
||||
self.assertNotIn(" archive ", self.trace_text())
|
||||
|
||||
def test_rejects_non_full_sha_before_calling_external_tools(self) -> None:
|
||||
result = self.run_builder("trusted-base", "candidate", "abc123")
|
||||
self.assertEqual(result.returncode, 2)
|
||||
self.assertIn("complete 40-character SHA", result.stderr)
|
||||
self.assertFalse(self.trace.exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,205 @@
|
||||
"""master 产物的增量归档与 Gitea v4 上传协议回归。"""
|
||||
|
||||
import hashlib
|
||||
import base64
|
||||
from contextlib import redirect_stdout
|
||||
from datetime import datetime, timezone
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tarfile
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
import urllib.parse
|
||||
import zipfile
|
||||
|
||||
from gitea_cache_upload_cleanup import decode_owner
|
||||
|
||||
|
||||
spec = importlib.util.spec_from_file_location("cache_export", Path(__file__).with_name("export-gitea-rust-cache.py"))
|
||||
exporter = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(exporter)
|
||||
|
||||
|
||||
class ExportTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
(self.root / "objects").mkdir()
|
||||
|
||||
def object(self, key, value, mtime):
|
||||
path = self.root / "objects" / key[0] / key[1] / key
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(value)
|
||||
os.utime(path, ns=(mtime * 10**9, mtime * 10**9))
|
||||
return path
|
||||
|
||||
def unpack(self, baseline, limit=1024):
|
||||
destination = self.root / "snapshot.zip"
|
||||
result = exporter.pack_snapshot(self.root, destination, {"job": "fixture"}, baseline, limit)
|
||||
with zipfile.ZipFile(destination) as bundle:
|
||||
self.assertEqual(bundle.namelist(), ["snapshot.tar"])
|
||||
with tarfile.open(fileobj=io.BytesIO(bundle.read("snapshot.tar"))) as archive:
|
||||
files = {member.name: archive.extractfile(member).read() for member in archive}
|
||||
return result, json.loads(files.pop("manifest.json")), files
|
||||
|
||||
def test_delta_omits_hit_bytes_preserves_recency_and_exports_new_keys(self):
|
||||
inherited = self.object("a" * 64, b"old", 10)
|
||||
baseline_path = self.root / "baseline.json"
|
||||
exporter.save_baseline(self.root, baseline_path)
|
||||
baseline = json.loads(baseline_path.read_text())
|
||||
# sccache 对命中对象更新 mtime,不能因此重复上传已有对象。
|
||||
os.utime(inherited, ns=(20 * 10**9, 20 * 10**9))
|
||||
self.object("b" * 64, b"new", 30)
|
||||
result, manifest, files = self.unpack(baseline)
|
||||
key = "objects/b/b/" + "b" * 64
|
||||
self.assertEqual(result, (1, 3))
|
||||
self.assertEqual(files, {key: b"new"})
|
||||
self.assertEqual(manifest["mode"], "delta")
|
||||
self.assertEqual(manifest["touched"], [{"path": "objects/a/a/" + "a" * 64, "mtime_ns": 20 * 10**9}])
|
||||
self.assertEqual(manifest["objects"][0]["sha256"], hashlib.sha256(b"new").hexdigest())
|
||||
|
||||
def test_empty_delta_is_valid_and_changed_size_is_exported(self):
|
||||
path = self.object("a" * 64, b"old", 10)
|
||||
baseline = {"a/a/" + "a" * 64: {"size": 3, "mtime_ns": 10 * 10**9}}
|
||||
self.assertEqual(self.unpack(baseline)[0], (0, 0))
|
||||
path.write_bytes(b"replacement")
|
||||
self.assertEqual(self.unpack(baseline)[0], (1, 11))
|
||||
|
||||
def test_capacity_prefers_newest_objects_and_excludes_unrelated_files(self):
|
||||
self.object("a" * 64, b"old", 10)
|
||||
self.object("b" * 64, b"new", 20)
|
||||
(self.root / "objects" / "credentials").write_bytes(b"secret")
|
||||
(self.root / "sccache").write_bytes(b"binary")
|
||||
result, _, files = self.unpack({}, limit=3)
|
||||
self.assertEqual(result, (1, 3))
|
||||
self.assertEqual(list(files.values()), [b"new"])
|
||||
|
||||
@unittest.skipUnless(os.name == "posix", "symlink fixture requires POSIX")
|
||||
def test_symlink_objects_and_subdirectories_are_not_exported(self):
|
||||
outside = self.root / "secret"
|
||||
outside.write_bytes(b"secret")
|
||||
linked = self.object("a" * 64, b"old", 10)
|
||||
linked.unlink()
|
||||
linked.symlink_to(outside)
|
||||
(self.root / "objects" / "b").symlink_to(self.root, target_is_directory=True)
|
||||
self.assertEqual(self.unpack({})[0], (0, 0))
|
||||
|
||||
def test_non_master_and_unclean_shutdown_do_not_scan_or_upload(self):
|
||||
with patch.object(exporter, "scan_objects", side_effect=AssertionError("must not scan")), \
|
||||
patch.object(exporter, "upload_snapshot", side_effect=AssertionError("must not upload")):
|
||||
exporter.export({"GITHUB_EVENT_NAME": "pull_request", "GITHUB_REF": "refs/heads/master"})
|
||||
exporter.export({"GITHUB_EVENT_NAME": "push", "GITHUB_REF": "refs/heads/feature"})
|
||||
exporter.export({"GITHUB_EVENT_NAME": "push", "GITHUB_REF": "refs/heads/master"})
|
||||
|
||||
def test_internal_repository_route_bypasses_rpc_only_gateway(self):
|
||||
env = {
|
||||
"GITHUB_REPOSITORY": "owner/repo",
|
||||
"GITHUB_SERVER_URL": "http://fetch-gate:8080",
|
||||
"GENARRATIVE_GITEA_REPOSITORY_URL": "http://gitea:3000/sub/owner/repo.git",
|
||||
}
|
||||
self.assertEqual(exporter.artifact_base_url(env), "http://gitea:3000/sub")
|
||||
env["GENARRATIVE_GITEA_REPOSITORY_URL"] = "http://token@gitea:3000/owner/repo.git"
|
||||
with self.assertRaises(ValueError):
|
||||
exporter.artifact_base_url(env)
|
||||
|
||||
def test_native_v4_chunks_finalize_with_verified_size_and_hash(self):
|
||||
calls = []
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
def do_POST(self):
|
||||
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
|
||||
calls.append((self.command, self.path, dict(self.headers), body))
|
||||
payload = {"ok": True}
|
||||
if self.path.endswith("CreateArtifact"):
|
||||
payload["signedUploadUrl"] = "https://external.invalid/twirp/github.actions.results.api.v1.ArtifactService/UploadArtifact?sig=private&taskID=4&artifactID=8"
|
||||
else:
|
||||
payload["artifactId"] = "8"
|
||||
self.send_response(200)
|
||||
self.end_headers()
|
||||
self.wfile.write(json.dumps(payload).encode())
|
||||
|
||||
def do_PUT(self):
|
||||
body = self.rfile.read(int(self.headers["Content-Length"]))
|
||||
calls.append((self.command, self.path, dict(self.headers), body))
|
||||
self.send_response(201)
|
||||
self.end_headers()
|
||||
self.wfile.write(b'"created"')
|
||||
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
path = self.root / "test.zip"
|
||||
path.write_bytes(b"123456789")
|
||||
with patch.object(exporter, "CHUNK_BYTES", 4):
|
||||
exporter.upload_snapshot(path, "rust-cache-v1-backend-tests-attempt-1", 123, f"http://127.0.0.1:{server.server_port}", "ephemeral")
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
thread.join()
|
||||
self.assertEqual(len(calls), 6)
|
||||
self.assertEqual(calls[0][3]["workflowRunBackendId"], "123")
|
||||
remaining = datetime.fromisoformat(calls[0][3]["expiresAt"]) - datetime.now(timezone.utc)
|
||||
self.assertGreater(remaining.total_seconds(), 7 * 24 * 3600)
|
||||
self.assertLessEqual(remaining.total_seconds(), 8 * 24 * 3600)
|
||||
blocks = calls[1:4]
|
||||
self.assertEqual(b"".join(call[3] for call in blocks), b"123456789")
|
||||
self.assertEqual(len({urllib.parse.parse_qs(urllib.parse.urlsplit(call[1]).query)["blockid"][0] for call in blocks}), 3)
|
||||
for call in blocks:
|
||||
block_id = urllib.parse.parse_qs(urllib.parse.urlsplit(call[1]).query)["blockid"][0]
|
||||
gitea_filename_suffix = base64.urlsafe_b64encode(block_id.encode()).decode()
|
||||
self.assertEqual(decode_owner(gitea_filename_suffix), ("backend-tests", 1))
|
||||
self.assertTrue(all("Authorization" not in call[2] for call in calls[1:5]))
|
||||
self.assertEqual(calls[-1][2]["Authorization"], "Bearer ephemeral")
|
||||
self.assertEqual(calls[-1][3]["size"], "9")
|
||||
self.assertEqual(calls[-1][3]["hash"], "sha256:" + hashlib.sha256(b"123456789").hexdigest())
|
||||
|
||||
def test_upload_failure_never_finalizes_an_incomplete_artifact(self):
|
||||
path = self.root / "test.zip"
|
||||
path.write_bytes(b"data")
|
||||
with patch.object(exporter, "request", side_effect=[
|
||||
{"ok": True, "signedUploadUrl": "http://gitea/twirp/github.actions.results.api.v1.ArtifactService/UploadArtifact?sig=x"},
|
||||
RuntimeError("upload failed"),
|
||||
]) as request:
|
||||
with self.assertRaisesRegex(RuntimeError, "upload failed"):
|
||||
exporter.upload_snapshot(path, "rust-cache-v1-backend-tests-attempt-1", 123, "http://gitea", "ephemeral")
|
||||
self.assertEqual(request.call_count, 2)
|
||||
|
||||
def test_failed_optional_export_never_emits_host_completion_marker(self):
|
||||
(self.root / "export-baseline.json").write_text("{}")
|
||||
(self.root / "rustc.txt").write_text("rustc fixture\n")
|
||||
(self.root / "workspace.txt").write_text(str(Path.cwd().resolve()))
|
||||
(self.root / "source-commit.txt").write_text("a" * 40)
|
||||
self.object("b" * 64, b"new", 20)
|
||||
env = {
|
||||
"GITHUB_EVENT_NAME": "push", "GITHUB_REF": "refs/heads/master",
|
||||
"GENARRATIVE_CI_RUST_CACHE_EXPORT_READY": "1",
|
||||
"GITHUB_JOB": "backend-tests", "GITHUB_SHA": "b" * 40,
|
||||
"GITHUB_RUN_ID": "123", "GITHUB_RUN_ATTEMPT": "1",
|
||||
"GENARRATIVE_GITEA_TOKEN": "ephemeral",
|
||||
"GITHUB_REPOSITORY": "owner/repo",
|
||||
"GENARRATIVE_GITEA_REPOSITORY_URL": "http://gitea/owner/repo.git",
|
||||
"GENARRATIVE_CI_RUST_CACHE_ROOT": str(self.root),
|
||||
}
|
||||
output = io.StringIO()
|
||||
with patch.object(exporter.subprocess, "check_output", side_effect=["rustc fixture\n", "sccache 0.18.0\n"]), \
|
||||
patch.object(exporter, "upload_snapshot", side_effect=RuntimeError("upload failed")), \
|
||||
redirect_stdout(output):
|
||||
with self.assertRaisesRegex(RuntimeError, "upload failed"):
|
||||
exporter.export(env)
|
||||
self.assertNotIn("complete=true", output.getvalue())
|
||||
self.assertFalse((self.root / "export-baseline.json").exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,347 @@
|
||||
"""使用真实 HTTP/socket 验证领取屏障,不连接线上 Gitea。"""
|
||||
|
||||
import http.client
|
||||
import http.server
|
||||
import gzip
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import socket
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
|
||||
if os.name != "posix":
|
||||
raise unittest.SkipTest("领取屏障使用 Linux Unix socket 和目录 fsync;在 Linux/WSL 运行")
|
||||
|
||||
SPEC = importlib.util.spec_from_file_location(
|
||||
"gitea_cache_gate", Path(__file__).with_name("gitea-runner-fetch-gate.py"))
|
||||
MODULE = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(MODULE)
|
||||
FETCH = "/api/actions/runner.v1.RunnerService/FetchTask"
|
||||
UPDATE = "/api/actions/runner.v1.RunnerService/UpdateTask"
|
||||
LOG = "/api/actions/runner.v1.RunnerService/UpdateLog"
|
||||
PING = "/api/actions/ping.v1.PingService/Ping"
|
||||
|
||||
|
||||
def varint(value):
|
||||
data = bytearray()
|
||||
while value > 127:
|
||||
data.append((value & 127) | 128)
|
||||
value >>= 7
|
||||
data.append(value)
|
||||
return bytes(data)
|
||||
|
||||
|
||||
def field(number, value):
|
||||
# actions-proto-go v0.4.1: task/state ID=1; result=2; log task_id=1,
|
||||
# index=2, rows=3, no_more=4; log response ack_index=1.
|
||||
if isinstance(value, bytes):
|
||||
return varint(number * 8 + 2) + varint(len(value)) + value
|
||||
return varint(number * 8) + varint(value)
|
||||
|
||||
|
||||
def state(task_id, result=0):
|
||||
return field(1, field(1, task_id) + field(2, result))
|
||||
|
||||
|
||||
def core_state(state):
|
||||
return {key: state[key] for key in ("paused", "inflight", "uncertain")}
|
||||
|
||||
|
||||
class Upstream(http.server.BaseHTTPRequestHandler):
|
||||
def log_message(self, *_args):
|
||||
pass
|
||||
|
||||
def do_POST(self):
|
||||
data = self.rfile.read(int(self.headers.get("Content-Length", 0)))
|
||||
self.server.requests.append((self.path, data))
|
||||
if self.path.endswith("/Redirect"):
|
||||
self.send_response(302)
|
||||
self.send_header("Location", "/api/actions/runner.v1.RunnerService/FetchTask")
|
||||
self.send_header("Content-Length", "0")
|
||||
self.end_headers()
|
||||
return
|
||||
if self.path == FETCH:
|
||||
self.server.entered.set()
|
||||
self.server.release.wait(10)
|
||||
if self.server.truncated and self.path == FETCH:
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Length", "100")
|
||||
self.end_headers()
|
||||
self.wfile.write(b"incomplete")
|
||||
self.close_connection = True
|
||||
return
|
||||
body = self.server.responses.get(self.path, b"")
|
||||
if self.path == UPDATE and self.path not in self.server.responses:
|
||||
body = state(*map(int, MODULE.task_state(MODULE.protobuf_fields(data))))
|
||||
if self.path == LOG and self.path not in self.server.responses:
|
||||
fields = MODULE.protobuf_fields(data)
|
||||
body = field(1, MODULE.single(fields, 2, 0) + len(fields.get(3, [])))
|
||||
self.send_response(self.server.statuses.get(self.path, 200))
|
||||
self.send_header("Content-Type", self.server.content_type)
|
||||
if self.server.compressed:
|
||||
body = gzip.compress(body)
|
||||
self.send_header("Content-Encoding", "gzip")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
|
||||
class GateTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.gate = MODULE.Gate(self.temp.name)
|
||||
self.upstream = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Upstream)
|
||||
self.upstream.requests = []
|
||||
self.upstream.entered = threading.Event()
|
||||
self.upstream.release = threading.Event()
|
||||
self.upstream.truncated = False
|
||||
self.upstream.responses = {}
|
||||
self.upstream.statuses = {}
|
||||
self.upstream.content_type = "application/proto"
|
||||
self.upstream.compressed = False
|
||||
self.proxy = MODULE.create_proxy(("127.0.0.1", 0),
|
||||
f"http://127.0.0.1:{self.upstream.server_port}", self.gate)
|
||||
self.servers = [self.upstream, self.proxy]
|
||||
for server in self.servers:
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
|
||||
def tearDown(self):
|
||||
self.upstream.release.set()
|
||||
for server in reversed(self.servers):
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
self.temp.cleanup()
|
||||
|
||||
def request(self, path, body=b"", chunked=False):
|
||||
connection = http.client.HTTPConnection("127.0.0.1", self.proxy.server_port, timeout=5)
|
||||
try:
|
||||
if chunked:
|
||||
connection.request("POST", path, body=[body], encode_chunked=True,
|
||||
headers={"Content-Type": "application/proto"})
|
||||
else:
|
||||
connection.request("POST", path, body=body,
|
||||
headers={"Content-Type": "application/proto"})
|
||||
response = connection.getresponse()
|
||||
result = response.status, response.read()
|
||||
return result
|
||||
finally:
|
||||
connection.close()
|
||||
|
||||
def wait_for(self, predicate):
|
||||
deadline = time.monotonic() + 3
|
||||
while time.monotonic() < deadline:
|
||||
if predicate():
|
||||
return
|
||||
time.sleep(0.01)
|
||||
self.fail("condition did not become true")
|
||||
|
||||
def start_fetch(self):
|
||||
self.result = []
|
||||
thread = threading.Thread(target=lambda: self.result.append(self.request(FETCH)))
|
||||
thread.start()
|
||||
self.assertTrue(self.upstream.entered.wait(3))
|
||||
return thread
|
||||
|
||||
def test_pause_holds_inflight_and_allows_reporting(self):
|
||||
thread = self.start_fetch()
|
||||
self.assertEqual(core_state(self.gate.control("pause")),
|
||||
{"paused": True, "inflight": 1, "uncertain": False})
|
||||
self.assertEqual(self.request(FETCH)[0], 503)
|
||||
self.assertEqual(self.request(PING)[0], 200)
|
||||
self.assertEqual(sum(path == FETCH for path, _ in self.upstream.requests), 1)
|
||||
self.upstream.release.set()
|
||||
thread.join(3)
|
||||
self.assertFalse(thread.is_alive())
|
||||
self.assertEqual(self.result[0][0], 200)
|
||||
self.assertEqual(self.gate.control("status")["inflight"], 0)
|
||||
self.assertEqual(self.gate.control("resume")["paused"], False)
|
||||
self.assertEqual(self.request(FETCH)[0], 200)
|
||||
|
||||
def test_disconnected_client_does_not_release_upstream_request(self):
|
||||
client = socket.create_connection(self.proxy.server_address, timeout=3)
|
||||
client.sendall(f"POST {FETCH} HTTP/1.1\r\nHost: localhost\r\nContent-Length: 2\r\n\r\n{{}}".encode())
|
||||
self.assertTrue(self.upstream.entered.wait(3))
|
||||
client.close()
|
||||
state = self.gate.control("pause")
|
||||
self.assertEqual(state["inflight"], 1)
|
||||
self.upstream.release.set()
|
||||
self.wait_for(lambda: self.gate.control("status")["inflight"] == 0)
|
||||
self.assertFalse(self.gate.control("status")["uncertain"])
|
||||
|
||||
def test_truncated_upstream_latches_uncertainty_across_restart(self):
|
||||
self.upstream.truncated = True
|
||||
self.upstream.release.set()
|
||||
self.assertEqual(self.request(FETCH)[0], 502)
|
||||
self.wait_for(lambda: self.gate.control("status")["uncertain"])
|
||||
self.assertTrue(self.gate.control("resume")["paused"])
|
||||
restored = MODULE.Gate(self.temp.name)
|
||||
self.assertTrue(restored.control("status")["uncertain"])
|
||||
self.assertFalse(restored.enter())
|
||||
|
||||
def test_crashed_inflight_is_not_treated_as_successful_drain(self):
|
||||
self.assertTrue(self.gate.enter())
|
||||
restored = MODULE.Gate(self.temp.name)
|
||||
self.assertTrue(restored.control("status")["uncertain"])
|
||||
self.assertIn("error", restored.control("resume"))
|
||||
self.gate.leave(True)
|
||||
|
||||
def test_pause_marker_survives_restart(self):
|
||||
self.gate.control("pause")
|
||||
restored = MODULE.Gate(self.temp.name)
|
||||
self.assertEqual(core_state(restored.control("status")),
|
||||
{"paused": True, "inflight": 0, "uncertain": False})
|
||||
restored.control("resume")
|
||||
self.assertFalse(MODULE.Gate(self.temp.name).control("status")["paused"])
|
||||
|
||||
def test_chunked_body_is_decoded_and_non_rpc_path_rejected(self):
|
||||
self.assertEqual(self.request(PING, b"ping", chunked=True)[0], 200)
|
||||
self.assertEqual(self.upstream.requests[-1], (PING, b"ping"))
|
||||
self.assertEqual(self.request("/api/v1/repos")[0], 404)
|
||||
self.assertEqual(self.request("/api/actions/../v1/repos")[0], 404)
|
||||
|
||||
def test_upstream_redirect_is_not_followed(self):
|
||||
self.assertEqual(self.request("/api/actions/runner.v1.RunnerService/Redirect")[0], 302)
|
||||
self.assertFalse(self.upstream.entered.is_set())
|
||||
self.assertEqual(len(self.upstream.requests), 1)
|
||||
|
||||
def test_connection_refused_before_rpc_does_not_latch_uncertainty(self):
|
||||
with socket.socket() as unused:
|
||||
unused.bind(("127.0.0.1", 0))
|
||||
port = unused.getsockname()[1]
|
||||
self.proxy.upstream = MODULE.urllib.parse.urlsplit(f"http://127.0.0.1:{port}")
|
||||
self.assertEqual(self.request(FETCH)[0], 502)
|
||||
self.assertEqual(core_state(self.gate.control("status")),
|
||||
{"paused": False, "inflight": 0, "uncertain": False})
|
||||
|
||||
@unittest.skipUnless(hasattr(socket, "AF_UNIX"), "Unix control socket required")
|
||||
def test_control_socket_uses_newline_json(self):
|
||||
path = str(Path(self.temp.name) / "gate.sock")
|
||||
control = MODULE.ControlServer(path, MODULE.Control)
|
||||
control.gate = self.gate
|
||||
self.servers.append(control)
|
||||
threading.Thread(target=control.serve_forever, daemon=True).start()
|
||||
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client:
|
||||
client.connect(path)
|
||||
client.sendall(b'{"action":"pause"}\n')
|
||||
payload = client.makefile("rb").readline()
|
||||
self.assertEqual(core_state(json.loads(payload)),
|
||||
{"paused": True, "inflight": 0, "uncertain": False})
|
||||
|
||||
def test_paused_fetch_records_live_route_and_restart_discards_proof(self):
|
||||
self.gate.control("pause")
|
||||
before = time.time()
|
||||
self.assertEqual(self.request(FETCH)[0], 503)
|
||||
state = self.gate.control("status")
|
||||
self.assertEqual(state["last_fetch_peer"], "127.0.0.1")
|
||||
self.assertGreaterEqual(state["last_fetch_at"], before)
|
||||
self.assertEqual(self.upstream.requests, [])
|
||||
self.assertEqual(self.request(PING)[0], 200)
|
||||
self.assertEqual(self.gate.control("status")["last_fetch_at"], state["last_fetch_at"])
|
||||
restarted = MODULE.Gate(self.temp.name).control("status")
|
||||
self.assertIsNone(restarted["last_fetch_peer"])
|
||||
self.assertIsNone(restarted["last_fetch_at"])
|
||||
|
||||
def fetch_task(self, task_id=42):
|
||||
self.upstream.responses[FETCH] = field(1, field(1, task_id)) + field(2, 10)
|
||||
self.upstream.release.set()
|
||||
self.assertEqual(self.request(FETCH)[0], 200)
|
||||
self.assertEqual(self.gate.control("status")["task_ids"], [str(task_id)])
|
||||
|
||||
def finalize_log(self, task_id=42):
|
||||
self.assertEqual(self.request(LOG, field(1, task_id) + field(2, 3)
|
||||
+ field(3, b"log row") + field(4, 1))[0], 200)
|
||||
self.wait_for(lambda: self.gate.tasks.get(str(task_id)) is True)
|
||||
|
||||
def test_assigned_task_survives_pause_until_runner_finished_reporting(self):
|
||||
self.fetch_task()
|
||||
self.assertEqual(self.gate.control("pause")["active_tasks"], 1)
|
||||
self.assertEqual(self.request(FETCH)[0], 503)
|
||||
self.assertEqual(self.request(UPDATE, state(42, 1))[0], 200)
|
||||
self.assertEqual(self.gate.control("status")["active_tasks"], 1)
|
||||
self.finalize_log()
|
||||
self.assertEqual(self.gate.control("status")["active_tasks"], 1)
|
||||
self.assertEqual(self.request(UPDATE, state(42, 1))[0], 200)
|
||||
self.wait_for(lambda: self.gate.control("status")["active_tasks"] == 0)
|
||||
self.assertFalse(self.gate.control("status")["uncertain"])
|
||||
# 终态响应在客户端超时后重试,不应再次阻断后续任务领取。
|
||||
self.request(UPDATE, state(42, 1))
|
||||
self.assertFalse(self.gate.control("status")["uncertain"])
|
||||
|
||||
def test_compressed_fetch_and_task_ledger_survive_restart(self):
|
||||
self.upstream.compressed = True
|
||||
self.fetch_task()
|
||||
restored = MODULE.Gate(self.temp.name)
|
||||
self.assertEqual(restored.control("status")["task_ids"], ["42"])
|
||||
self.assertFalse(restored.control("status")["uncertain"])
|
||||
self.proxy.gate = self.gate = restored
|
||||
self.finalize_log()
|
||||
self.request(UPDATE, state(42, 2))
|
||||
self.wait_for(lambda: self.gate.control("status")["active_tasks"] == 0)
|
||||
self.assertEqual(MODULE.Gate(self.temp.name).control("status")["active_tasks"], 0)
|
||||
|
||||
def test_server_cancellation_does_not_retire_executing_task(self):
|
||||
self.fetch_task()
|
||||
self.upstream.responses[UPDATE] = state(42, 3)
|
||||
self.request(UPDATE, state(42))
|
||||
self.assertEqual(self.gate.control("status")["active_tasks"], 1)
|
||||
self.finalize_log()
|
||||
self.request(UPDATE, state(42))
|
||||
self.assertEqual(self.gate.control("status")["active_tasks"], 1)
|
||||
self.request(UPDATE, state(42, 3))
|
||||
self.wait_for(lambda: self.gate.control("status")["active_tasks"] == 0)
|
||||
|
||||
def test_partial_final_log_acknowledgement_keeps_task_active(self):
|
||||
self.fetch_task()
|
||||
self.upstream.responses[LOG] = field(1, 3)
|
||||
self.request(LOG, field(1, 42) + field(2, 3) + field(3, b"row") + field(4, 1))
|
||||
self.request(UPDATE, state(42, 1))
|
||||
self.assertEqual(self.gate.control("status")["active_tasks"], 1)
|
||||
del self.upstream.responses[LOG]
|
||||
self.finalize_log()
|
||||
self.request(UPDATE, state(42, 1))
|
||||
self.wait_for(lambda: self.gate.control("status")["active_tasks"] == 0)
|
||||
|
||||
def test_final_state_waits_for_success_and_output_acknowledgement(self):
|
||||
self.fetch_task()
|
||||
self.finalize_log()
|
||||
self.upstream.statuses[UPDATE] = 500
|
||||
self.request(UPDATE, state(42, 1))
|
||||
self.assertEqual(self.gate.control("status")["active_tasks"], 1)
|
||||
self.upstream.statuses[UPDATE] = 200
|
||||
output = field(2, field(1, b"key") + field(2, b"value"))
|
||||
self.request(UPDATE, state(42, 1) + output)
|
||||
self.assertEqual(self.gate.control("status")["active_tasks"], 1)
|
||||
self.upstream.responses[UPDATE] = state(42, 1) + field(2, b"key")
|
||||
self.request(UPDATE, state(42, 1) + output)
|
||||
self.wait_for(lambda: self.gate.control("status")["active_tasks"] == 0)
|
||||
|
||||
def test_unknown_or_invalid_fetch_result_prevents_idle(self):
|
||||
self.upstream.release.set()
|
||||
self.upstream.responses[FETCH] = b"not protobuf"
|
||||
self.assertEqual(self.request(FETCH)[0], 200)
|
||||
self.assertTrue(self.gate.control("status")["uncertain"])
|
||||
self.assertTrue(MODULE.Gate(self.temp.name).control("status")["uncertain"])
|
||||
|
||||
def test_task_report_without_observed_assignment_requires_idle_bootstrap(self):
|
||||
self.assertEqual(self.request(UPDATE, state(42))[0], 200)
|
||||
self.wait_for(lambda: self.gate.control("status")["uncertain"])
|
||||
self.assertIn("error", self.gate.control("resume"))
|
||||
|
||||
def test_disconnected_fetch_client_still_leaves_assigned_task_busy(self):
|
||||
self.upstream.responses[FETCH] = field(1, field(1, 42))
|
||||
client = socket.create_connection(self.proxy.server_address, timeout=3)
|
||||
client.sendall(f"POST {FETCH} HTTP/1.1\r\nHost: localhost\r\nContent-Length: 0\r\n\r\n".encode())
|
||||
self.assertTrue(self.upstream.entered.wait(3))
|
||||
client.close()
|
||||
self.gate.control("pause")
|
||||
self.upstream.release.set()
|
||||
self.wait_for(lambda: self.gate.control("status")["inflight"] == 0)
|
||||
self.assertEqual(self.gate.control("status")["task_ids"], ["42"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,240 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Tests for streamed Gitea sccache snapshot validation and merging."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import os
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
import zipfile
|
||||
|
||||
|
||||
SCRIPT = Path(__file__).with_name("gitea_cache_snapshot.py")
|
||||
SPEC = importlib.util.spec_from_file_location("gitea_cache_snapshot", SCRIPT)
|
||||
assert SPEC and SPEC.loader
|
||||
snapshot = importlib.util.module_from_spec(SPEC)
|
||||
sys.modules[SPEC.name] = snapshot
|
||||
SPEC.loader.exec_module(snapshot)
|
||||
|
||||
REPOSITORY = "team/project"
|
||||
SOURCE_SHA = "a" * 40
|
||||
INHERITED_SOURCE_SHA = "c" * 40
|
||||
BASE_IMAGE = "sha256:" + "b" * 64
|
||||
RUSTC = "rustc 1.90.0\nbinary: rustc\n"
|
||||
WORKSPACE = "/workspace/GenarrativeAI/Genarrative"
|
||||
SCCACHE_VERSION = "0.18.0"
|
||||
|
||||
|
||||
def object_path(character: str) -> str:
|
||||
key = character * 64
|
||||
return f"objects/{key[0]}/{key[1]}/{key}"
|
||||
|
||||
|
||||
class SnapshotMergeTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.temporary_directory = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
|
||||
self.root = Path(self.temporary_directory.name).resolve()
|
||||
self.base = self.root / "base-objects"
|
||||
self.base.mkdir()
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.temporary_directory.cleanup()
|
||||
|
||||
def identity(self, job: str, run_id: int) -> object:
|
||||
return snapshot.ArtifactIdentity(REPOSITORY, run_id, 1, job, SOURCE_SHA)
|
||||
|
||||
def archive(
|
||||
self,
|
||||
name: str,
|
||||
job: str,
|
||||
run_id: int,
|
||||
objects: list[tuple[str, bytes, int]],
|
||||
*,
|
||||
prefix: bool = True,
|
||||
manifest_update=None,
|
||||
extra_member: tuple[str, bytes, str] | None = None,
|
||||
touched: list[tuple[str, int]] | None = None,
|
||||
) -> object:
|
||||
rows = [
|
||||
{
|
||||
"path": path,
|
||||
"size": len(contents),
|
||||
"sha256": hashlib.sha256(contents).hexdigest(),
|
||||
"mtime_ns": mtime_ns,
|
||||
}
|
||||
for path, contents, mtime_ns in objects
|
||||
]
|
||||
manifest = {
|
||||
"schema": 1,
|
||||
"repository": REPOSITORY,
|
||||
"run_id": run_id,
|
||||
"run_attempt": 1,
|
||||
"job": job,
|
||||
"source_sha": SOURCE_SHA,
|
||||
"base_image": BASE_IMAGE,
|
||||
"rustc": RUSTC,
|
||||
"workspace": WORKSPACE,
|
||||
"sccache_version": SCCACHE_VERSION,
|
||||
"mode": "delta",
|
||||
"inherited_source_sha": INHERITED_SOURCE_SHA,
|
||||
"objects": rows,
|
||||
"touched": [
|
||||
{"path": path, "mtime_ns": mtime_ns}
|
||||
for path, mtime_ns in (touched or [])
|
||||
],
|
||||
}
|
||||
if manifest_update is not None:
|
||||
manifest_update(manifest)
|
||||
|
||||
tar_bytes = io.BytesIO()
|
||||
with tarfile.open(fileobj=tar_bytes, mode="w") as bundle:
|
||||
for path, contents, _ in objects:
|
||||
info = tarfile.TarInfo(path)
|
||||
info.size = len(contents)
|
||||
bundle.addfile(info, io.BytesIO(contents))
|
||||
if extra_member is not None:
|
||||
path, contents, kind = extra_member
|
||||
info = tarfile.TarInfo(path)
|
||||
if kind == "symlink":
|
||||
info.type = tarfile.SYMTYPE
|
||||
info.linkname = "manifest.json"
|
||||
bundle.addfile(info)
|
||||
else:
|
||||
info.size = len(contents)
|
||||
bundle.addfile(info, io.BytesIO(contents))
|
||||
manifest_bytes = json.dumps(manifest, separators=(",", ":")).encode()
|
||||
info = tarfile.TarInfo("manifest.json")
|
||||
info.size = len(manifest_bytes)
|
||||
bundle.addfile(info, io.BytesIO(manifest_bytes))
|
||||
|
||||
archive = self.root / f"{name}.zip"
|
||||
member = f"{name}/snapshot.tar" if prefix else "snapshot.tar"
|
||||
with zipfile.ZipFile(archive, "w", compression=zipfile.ZIP_STORED) as bundle:
|
||||
bundle.writestr(member, tar_bytes.getvalue())
|
||||
return snapshot.ArtifactInput(archive, self.identity(job, run_id))
|
||||
|
||||
def base_object(self, path: str, contents: bytes, mtime_ns: int) -> None:
|
||||
target = self.base.joinpath(*Path(path).parts[1:])
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_bytes(contents)
|
||||
os.utime(target, ns=(mtime_ns, mtime_ns))
|
||||
|
||||
def merge(self, inputs, output=None, maximum=1024 ** 3):
|
||||
return snapshot.merge_snapshots(
|
||||
inputs,
|
||||
output or self.root / "merged",
|
||||
base_objects=self.base,
|
||||
expected_inherited_source_sha=INHERITED_SOURCE_SHA,
|
||||
max_combined_bytes=maximum,
|
||||
)
|
||||
|
||||
def test_merges_groups_deduplicates_newest_mtime_and_prunes_to_bound(self) -> None:
|
||||
first_path = object_path("a")
|
||||
second_path = object_path("b")
|
||||
third_path = object_path("c")
|
||||
inherited_path = object_path("d")
|
||||
self.base_object(inherited_path, b"dd", 50)
|
||||
inputs = [
|
||||
self.archive("first", "Backend tests", 41, [
|
||||
(first_path, b"aaaa", 100),
|
||||
(second_path, b"bbbb", 200),
|
||||
]),
|
||||
self.archive("second", "Native shell tests", 41, [
|
||||
(first_path, b"aaaa", 300),
|
||||
(third_path, b"ccc", 250),
|
||||
], prefix=False, touched=[(inherited_path, 400)]),
|
||||
]
|
||||
|
||||
output = self.root / "merged"
|
||||
result = self.merge(inputs, output, maximum=9)
|
||||
|
||||
self.assertEqual((result.object_count, result.total_bytes), (3, 9))
|
||||
self.assertEqual((output / inherited_path).read_bytes(), b"dd")
|
||||
self.assertEqual((output / first_path).read_bytes(), b"aaaa")
|
||||
self.assertEqual((output / third_path).read_bytes(), b"ccc")
|
||||
self.assertFalse((output / second_path).exists())
|
||||
self.assertEqual((output / "rustc.txt").read_text(), RUSTC)
|
||||
self.assertEqual((output / "workspace.txt").read_text(), WORKSPACE + "\n")
|
||||
self.assertEqual((output / "source-commit.txt").read_text(), SOURCE_SHA + "\n")
|
||||
self.assertEqual((output / "base-image.txt").read_text(), BASE_IMAGE + "\n")
|
||||
self.assertFalse((output / "sccache").exists())
|
||||
|
||||
def test_rejects_traversal_member_without_creating_output(self) -> None:
|
||||
item = self.archive(
|
||||
"traversal", "Backend tests", 42, [],
|
||||
extra_member=("../outside", b"bad", "file"),
|
||||
)
|
||||
output = self.root / "merged"
|
||||
with self.assertRaisesRegex(snapshot.SnapshotError, "invalid sccache object path"):
|
||||
self.merge([item], output)
|
||||
self.assertFalse(output.exists())
|
||||
self.assertFalse((self.root / "outside").exists())
|
||||
|
||||
def test_rejects_checksum_mismatch(self) -> None:
|
||||
path = object_path("d")
|
||||
|
||||
def corrupt(manifest):
|
||||
manifest["objects"][0]["sha256"] = "0" * 64
|
||||
|
||||
item = self.archive(
|
||||
"checksum", "Backend tests", 43, [(path, b"content", 1)],
|
||||
manifest_update=corrupt,
|
||||
)
|
||||
with self.assertRaisesRegex(snapshot.SnapshotError, "checksums do not match"):
|
||||
self.merge([item])
|
||||
|
||||
def test_rejects_manifest_identity_mismatch(self) -> None:
|
||||
item = self.archive(
|
||||
"identity", "Backend tests", 44, [],
|
||||
manifest_update=lambda manifest: manifest.update(run_id=999),
|
||||
)
|
||||
with self.assertRaisesRegex(snapshot.SnapshotError, "run_id does not match"):
|
||||
self.merge([item])
|
||||
|
||||
def test_rejects_conflicting_duplicate_key(self) -> None:
|
||||
path = object_path("e")
|
||||
inputs = [
|
||||
self.archive("conflict-one", "Backend tests", 45, [(path, b"one", 1)]),
|
||||
self.archive("conflict-two", "Native shell tests", 45, [(path, b"two", 2)]),
|
||||
]
|
||||
with self.assertRaisesRegex(snapshot.SnapshotError, "conflicting content"):
|
||||
self.merge(inputs)
|
||||
|
||||
def test_rejects_delta_that_conflicts_with_inherited_key(self) -> None:
|
||||
path = object_path("f")
|
||||
self.base_object(path, b"base", 1)
|
||||
item = self.archive("base-conflict", "Backend tests", 46, [(path, b"evil", 2)])
|
||||
with self.assertRaisesRegex(snapshot.SnapshotError, "conflicting content"):
|
||||
self.merge([item])
|
||||
|
||||
def test_accepts_empty_delta(self) -> None:
|
||||
item = self.archive("empty", "Backend tests", 47, [])
|
||||
output = self.root / "merged"
|
||||
result = self.merge([item], output)
|
||||
self.assertEqual((result.object_count, result.total_bytes), (0, 0))
|
||||
self.assertEqual(list((output / "objects").iterdir()), [])
|
||||
|
||||
def test_rejects_noncanonical_base_entry(self) -> None:
|
||||
(self.base / "unexpected").write_bytes(b"not an object")
|
||||
item = self.archive("base-path", "Backend tests", 48, [])
|
||||
with self.assertRaisesRegex(snapshot.SnapshotError, "noncanonical entry"):
|
||||
self.merge([item])
|
||||
|
||||
def test_rejects_tar_links(self) -> None:
|
||||
item = self.archive(
|
||||
"link", "Backend tests", 49, [],
|
||||
extra_member=(object_path("9"), b"", "symlink"),
|
||||
)
|
||||
with self.assertRaisesRegex(snapshot.SnapshotError, "regular file"):
|
||||
self.merge([item])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,90 @@
|
||||
"""缓存上传残块清理只触碰已确认结束的过期自有文件。"""
|
||||
|
||||
import base64
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from gitea_cache_upload_cleanup import cleanup_upload_chunks, decode_owner
|
||||
|
||||
|
||||
class UploadCleanupTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name).resolve()
|
||||
|
||||
def file(self, name, run=12, age=10):
|
||||
directory = self.root / "tmp-upload" / f"run-{run}-v4"
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
path = directory / name
|
||||
path.write_bytes(b"block data")
|
||||
os.utime(path, (age, age))
|
||||
return path
|
||||
|
||||
def block(self, artifact=34, run=12, age=10, marker="genarrative-rust-cache-v1:backend-tests:1:00000000"):
|
||||
encoded = base64.urlsafe_b64encode(base64.b64encode(marker.encode())).decode()
|
||||
return self.file(f"block-{run}-{artifact}-10-{encoded}", run, age)
|
||||
|
||||
def test_old_owned_blocks_and_associated_blocklist_are_deleted(self):
|
||||
block = self.block()
|
||||
blocklist = self.file("12-34-blocklist")
|
||||
self.assertEqual(cleanup_upload_chunks(self.root, {12}, 20), 2)
|
||||
self.assertFalse(block.exists())
|
||||
self.assertFalse(blocklist.exists())
|
||||
self.assertTrue(block.parent.is_dir())
|
||||
|
||||
def test_other_runs_artifacts_and_unproven_blocklists_remain(self):
|
||||
own = self.block()
|
||||
other_run = self.block(run=13)
|
||||
other_artifact = self.block(artifact=35, marker="unrelated-upload")
|
||||
unproven = self.file("12-36-blocklist")
|
||||
self.assertEqual(cleanup_upload_chunks(self.root, {12}, 20), 1)
|
||||
self.assertFalse(own.exists())
|
||||
self.assertTrue(all(path.exists() for path in (other_run, other_artifact, unproven)))
|
||||
|
||||
def test_young_or_unknown_member_protects_entire_artifact_group(self):
|
||||
own = self.block()
|
||||
young = self.file("12-34-blocklist", age=20)
|
||||
other = self.block(artifact=35)
|
||||
unknown = self.file("block-12-35-invalid-size-unknown")
|
||||
self.assertEqual(cleanup_upload_chunks(self.root, {12}, 20), 0)
|
||||
self.assertTrue(all(path.exists() for path in (own, young, other, unknown)))
|
||||
|
||||
def test_completely_unknown_file_protects_entire_run(self):
|
||||
block = self.block()
|
||||
self.file("unknown")
|
||||
self.assertEqual(cleanup_upload_chunks(self.root, {12}, 20), 0)
|
||||
self.assertTrue(block.exists())
|
||||
|
||||
@unittest.skipUnless(os.name == "posix", "symlink fixture requires POSIX")
|
||||
def test_symlinks_never_traversed_or_deleted(self):
|
||||
block = self.block()
|
||||
outside = self.root / "secret"
|
||||
outside.write_text("keep")
|
||||
(block.parent / "12-34-blocklist").symlink_to(outside)
|
||||
(self.root / "tmp-upload" / "run-13-v4").symlink_to(block.parent, target_is_directory=True)
|
||||
self.assertEqual(cleanup_upload_chunks(self.root, {12, 13}, 20), 0)
|
||||
self.assertTrue(block.exists())
|
||||
self.assertEqual(outside.read_text(), "keep")
|
||||
linked = self.root / "root-link"
|
||||
linked.symlink_to(self.root, target_is_directory=True)
|
||||
with self.assertRaises(ValueError):
|
||||
cleanup_upload_chunks(linked, {12}, 20)
|
||||
|
||||
def test_root_missing_or_relative_is_an_error(self):
|
||||
with self.assertRaises(FileNotFoundError):
|
||||
cleanup_upload_chunks(self.root / "missing", {12}, 20)
|
||||
with self.assertRaises(ValueError):
|
||||
cleanup_upload_chunks(Path("relative"), {12}, 20)
|
||||
|
||||
def test_noncanonical_encoding_and_unknown_job_are_not_owned(self):
|
||||
for marker in ("genarrative-rust-cache-v1:other-job:1:00000000", "genarrative-rust-cache-v1:backend-tests:01:00000000"):
|
||||
encoded = base64.urlsafe_b64encode(base64.b64encode(marker.encode())).decode()
|
||||
self.assertIsNone(decode_owner(encoded))
|
||||
self.assertIsNone(decode_owner("not-base64"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user