修复 CI 缓存镜像组装的基础镜像引用
Project CI / Backend tests (pull_request) Has been cancelled
Project CI / Native shell tests (pull_request) Has been cancelled
Project CI / Frontend tests (pull_request) Has been cancelled
Project CI / Repository checks (pull_request) Has been cancelled
Project CI / AI game creator shell Rust smoke (pull_request) Has been cancelled
Project CI / AI game creator shell Rust crates (pull_request) Has been cancelled
Project CI / AI game creator shell Rust lane 1/2 (pull_request) Has been cancelled
Project CI / AI game creator shell Rust lane 2/2 (pull_request) Has been cancelled
Project CI / AI game creator shell web tests (pull_request) Has been cancelled
Project CI / AI game creator shell Rust crates (push) Successful in 1m18s
Project CI / AI game creator shell Rust smoke (push) Successful in 1m50s
Project CI / Backend tests (push) Successful in 3m40s
Project CI / Frontend tests (push) Successful in 2m0s
Project CI / AI game creator shell Rust lane 2/2 (push) Successful in 7m13s
Project CI / Native shell tests (push) Successful in 5m47s
Project CI / AI game creator shell Rust lane 1/2 (push) Successful in 8m34s
Project CI / AI game creator shell web tests (push) Successful in 1m32s
Project CI / Repository checks (push) Successful in 1m55s
Project CI / Backend tests (pull_request) Has been cancelled
Project CI / Native shell tests (pull_request) Has been cancelled
Project CI / Frontend tests (pull_request) Has been cancelled
Project CI / Repository checks (pull_request) Has been cancelled
Project CI / AI game creator shell Rust smoke (pull_request) Has been cancelled
Project CI / AI game creator shell Rust crates (pull_request) Has been cancelled
Project CI / AI game creator shell Rust lane 1/2 (pull_request) Has been cancelled
Project CI / AI game creator shell Rust lane 2/2 (pull_request) Has been cancelled
Project CI / AI game creator shell web tests (pull_request) Has been cancelled
Project CI / AI game creator shell Rust crates (push) Successful in 1m18s
Project CI / AI game creator shell Rust smoke (push) Successful in 1m50s
Project CI / Backend tests (push) Successful in 3m40s
Project CI / Frontend tests (push) Successful in 2m0s
Project CI / AI game creator shell Rust lane 2/2 (push) Successful in 7m13s
Project CI / Native shell tests (push) Successful in 5m47s
Project CI / AI game creator shell Rust lane 1/2 (push) Successful in 8m34s
Project CI / AI game creator shell web tests (push) Successful in 1m32s
Project CI / Repository checks (push) Successful in 1m55s
使用临时本地标签和宿主默认 builder 组装缓存镜像 组装成功或失败后清理临时标签,保持基础镜像清理归属 补充基础镜像复用、重建及失败清理测试 同步部署文档和项目共享记忆
This commit was merged in pull request #484.
This commit is contained in:
@@ -120,6 +120,8 @@ runner 配置保留原 `ubuntu-latest` 映射,`genarrative-ci` 继续映射到
|
||||
|
||||
维护 journal 分阶段记录来源 run、基础镜像重建或复用、artifact 下载、对象合并、镜像组装校验、导出、载入及空闲等待;长操作记录开始和结束耗时,失败输出对应私有 `artifacts/<source-sha>/build.log` 路径。构建的详细下载与 Docker 输出仍只写该日志,不回显 Token、命令环境或认证配置。
|
||||
|
||||
快照组装使用宿主 `default` Docker builder,读取已载入宿主的无对象缓存基础镜像;不能使用无法直接读取宿主镜像的独立 docker-container builder。BuildKit 的 `FROM` 不接受裸 `sha256:<Image ID>`,因此维护器在持锁期间将确定的基础 Image ID 绑定到专用临时 `assembly-base` tag,并在组装成功或失败后去除该 tag;来源元数据及 runner 配置仍使用完整 Image ID。临时别名不登记为受管基础镜像,不改变历史镜像的清理归属;进程被强杀时最多残留这一个别名,下次组装会重新绑定并清除。
|
||||
|
||||
切换先通过专属入口阻断新的 FetchTask,确认已转发的领取请求全部收到完整上游响应,并检查入口持久化跟踪的已领取任务全部结束、内层 Docker 没有活动容器。任务终态必须依据 Runner 的执行结束及最终上报协议,不能由容器暂时为空、API 已取消或请求超时推断。有任务即恢复领取并延后,不停止任务;状态未知拒绝切换。维护器只需普通账号的 `write:repository` Token(包括查询、下载及定向删除 artifact),不访问全局 Runner 管理 API。切换后等待使用该 Image ID 的完整真实 master push CI 通过,才允许下一次升级及旧镜像清理;不会自动重跑失败用例或为了验收额外触发整轮 CI。首次接管的历史镜像默认不归自动清理管理。
|
||||
|
||||
维护状态、凭据、归档和配置备份保存在仓库外。当前版、回滚版、待验证候选、它们的基础镜像及容器引用的镜像均受保护。清理只针对维护器登记的专属 tag、完整 Image ID 和专用目录中的归档;禁止全局 prune。API、构建、验证或空闲检查失败时保留现役镜像与回滚资料,不以失败重跑制造全绿结果。
|
||||
|
||||
@@ -100,6 +100,8 @@ SpacetimeDB 任务统一先读取 `.codex/skills/genarrative-spacetimedb/SKILL.m
|
||||
|
||||
Buildx 0.30.1 的 `inspect` 不支持 `--format`,builder 驱动校验读取普通输出的 `Driver:` 字段。相关命令须在宿主真实插件上验证;测试替身应拒绝不支持的参数,避免把模拟命令成功误当兼容性证据。
|
||||
|
||||
BuildKit 会把 Dockerfile 中的裸 `FROM sha256:<Image ID>` 当作远程镜像名;快照组装须在维护锁内把可信基础 Image ID 绑定到专用临时 tag,并显式使用能读取宿主镜像的 `default` Docker builder。成功或失败后去 tag,元数据和 runner 仍固定完整 Image ID;不要复用受管 `base_tag` 给历史基础镜像打别名,以免改变自动清理归属。
|
||||
|
||||
Gitea 基础镜像通过专用 `genarrative-ci-images` Buildx builder 持久复用 Cargo/npm 下载缓存;稳定 cache mount 与 commit、lock 哈希无关,以 `sharing=locked` 隔离并发写入,仅供可信宿主构建、不开放给 PR。最终镜像显式物化当前依赖下载快照,仍不包含 node_modules/target 或上一版 sccache 层。首次可用 `seed-downloads` 从可信完整 Image ID 提取包缓存,操作账号须与维护服务一致;部署要求及 builder GC 空间目标见 `deploy/container/README.md`。构建上下文必须覆盖 AGC vendor 与编辑器 bridge 的全部本地 path manifest,普通源码变化不应使依赖层失效。维护 journal 提供阶段耗时和失败 build.log 定位。
|
||||
|
||||
Gitea Rust 缓存自动维护由宿主 `genarrative-ci-cache.timer` 收集同一 master push run 六个 Rust job 的原生 V4 缓存产物,不重复执行 Cargo 预热。只传本轮新 key,命中对象只传使用时间;宿主与真实来源镜像对象合并、去重、按新近使用时间裁剪到 4 GiB,从无对象缓存基础镜像重新组装。源 run 不要求全绿,但取消、缺组、旧 attempt、未完成上传或混用来源镜像不得采用。网关暂停新 FetchTask、在途领取结束、持久化任务账本清空且内层活动容器为空才切换,不打断运行中的 CI。首次接入/升级网关需空闲窗口;Token 只需普通仓库 `write:repository`,不查管理员 API。候选装载后清理已收集 artifact,遗留项保留 7 天;真实 master CI 验证后才清理旧镜像,保留当前、一个回滚版、基础镜像及容器引用。部署入口见 `deploy/container/README.md`,合并代码不等于服务启用。
|
||||
|
||||
@@ -519,14 +519,22 @@ class Maintenance:
|
||||
shutil.copyfile(inherited / "sccache", snapshot / "sccache")
|
||||
(snapshot / "sccache").chmod(0o755)
|
||||
(snapshot / "base-image.txt").write_text(base + "\n")
|
||||
# BuildKit 的 FROM 不接受裸 Image ID;维护锁内使用一个临时本地别名。
|
||||
# 别名不登记为 owned base,避免把接管前的基础镜像纳入自动清理。
|
||||
assembly_base = "genarrative/gitea-project-ci:assembly-base"
|
||||
(work / "Dockerfile").write_text(
|
||||
f"FROM {base}\nCOPY snapshot/ /opt/genarrative-ci/rust-cache/\n"
|
||||
f"FROM {assembly_base}\nCOPY snapshot/ /opt/genarrative-ci/rust-cache/\n"
|
||||
f'LABEL world.genarrative.ci.rust-cache-source="{sha}"\n'
|
||||
f'LABEL world.genarrative.ci.rust-cache-base="{base}"\n')
|
||||
(work / ".dockerignore").write_text("**\n!Dockerfile\n!snapshot/\n!snapshot/**\n")
|
||||
with operation("assemble cache candidate image", build_log=build_log):
|
||||
with build_log.open("a") as out:
|
||||
self.docker("build", "--pull=false", "--tag", tag, str(work), output=out, timeout=1800)
|
||||
self.docker("image", "tag", base, assembly_base)
|
||||
try:
|
||||
with build_log.open("a") as out:
|
||||
self.docker("build", "--builder", "default", "--pull=false", "--tag", tag,
|
||||
str(work), output=out, timeout=1800)
|
||||
finally:
|
||||
self.docker("image", "rm", assembly_base)
|
||||
self.build_command(build_log, "gitea-ci-job-image.sh", "verify", tag, env=env,
|
||||
description="verify cache candidate image")
|
||||
image = self.image_info(tag)["Id"]
|
||||
|
||||
@@ -374,15 +374,27 @@ class GiteaCacheMaintenanceTest(unittest.TestCase):
|
||||
self.assertIsNone(self.select_source(instance))
|
||||
|
||||
def test_assembly_uses_ci_objects_and_trusted_binary_without_warming_compiler(self):
|
||||
self.assert_assembly()
|
||||
|
||||
def test_assembly_uses_rebuilt_local_base(self):
|
||||
self.assert_assembly(rebuild=True)
|
||||
|
||||
def test_failed_assembly_removes_temporary_base_alias(self):
|
||||
self.assert_assembly(build_fails=True)
|
||||
|
||||
def assert_assembly(self, *, rebuild=False, build_fails=False):
|
||||
instance, _, _, _ = self.source_fixture()
|
||||
source = self.select_source(instance)
|
||||
shell_calls, docker_calls = [], []
|
||||
base_tag = "genarrative/gitea-project-ci:base-auto-" + SHA
|
||||
assembly_base = "genarrative/gitea-project-ci:assembly-base"
|
||||
instance.build_command = lambda log, script, *args, **kw: shell_calls.append((script, args))
|
||||
def info(image, inner=False):
|
||||
return {"Id": IMAGE if image.startswith("genarrative/") else image, "Config": {"Labels": {
|
||||
image_id = BASE_IMAGE if image == base_tag else IMAGE if image.startswith("genarrative/") else image
|
||||
return {"Id": image_id, "Config": {"Labels": {
|
||||
"world.genarrative.ci.rust-cache-source": "b" * 40,
|
||||
"world.genarrative.ci.rust-cache-base": BASE_IMAGE,
|
||||
"com.genarrative.ci.definition-sha256": "definition",
|
||||
"com.genarrative.ci.definition-sha256": "old-definition" if rebuild else "definition",
|
||||
}}}
|
||||
instance.image_info = info
|
||||
def docker(*args, **kw):
|
||||
@@ -395,6 +407,16 @@ class GiteaCacheMaintenanceTest(unittest.TestCase):
|
||||
(root / "sccache").write_bytes(b"trusted binary")
|
||||
if args[-2:] == ("rustc", "-vV"):
|
||||
return "rustc test\n"
|
||||
if args[0] == "build":
|
||||
self.assertEqual(args[1:5], ("--builder", "default", "--pull=false", "--tag"))
|
||||
work = Path(args[-1])
|
||||
self.assertTrue((work / "Dockerfile").read_text().startswith(f"FROM {assembly_base}\n"))
|
||||
self.assertIn(f'world.genarrative.ci.rust-cache-base="{BASE_IMAGE}"',
|
||||
(work / "Dockerfile").read_text())
|
||||
self.assertEqual((work / "snapshot/base-image.txt").read_text(), BASE_IMAGE + "\n")
|
||||
self.assertEqual(docker_calls[-2], ("image", "tag", BASE_IMAGE, assembly_base))
|
||||
if build_fails:
|
||||
raise RuntimeError("assembly failed")
|
||||
return ""
|
||||
instance.docker = docker
|
||||
instance.api.download = lambda path, destination, expected_size: destination.write_bytes(b"download")
|
||||
@@ -407,10 +429,22 @@ class GiteaCacheMaintenanceTest(unittest.TestCase):
|
||||
"workspace": "/workspace/team/project", "base_image": BASE_IMAGE})()
|
||||
with patch.object(maintenance_module, "command", return_value="definition\n"), \
|
||||
patch.object(maintenance_module, "merge_snapshots", merge):
|
||||
instance.build(source)
|
||||
self.assertEqual(shell_calls, [("gitea-ci-job-image.sh", ("verify", "genarrative/gitea-project-ci:rust-cache-auto-" + SHA))])
|
||||
if build_fails:
|
||||
with self.assertRaisesRegex(RuntimeError, "assembly failed"):
|
||||
instance.build(source)
|
||||
else:
|
||||
instance.build(source)
|
||||
expected_calls = [("gitea-ci-job-image.sh", ("build",))] if rebuild else []
|
||||
if not build_fails:
|
||||
expected_calls.append(("gitea-ci-job-image.sh", ("verify", "genarrative/gitea-project-ci:rust-cache-auto-" + SHA)))
|
||||
self.assertEqual(shell_calls, expected_calls)
|
||||
self.assertIn(("create", OLD_IMAGE), docker_calls)
|
||||
self.assertIn(("rm", "--volumes", "temporary-copy-container"), docker_calls)
|
||||
self.assertEqual(docker_calls[-1], ("image", "rm", assembly_base))
|
||||
self.assertEqual(instance.state.get("bases", {}), {BASE_IMAGE: base_tag} if rebuild else {})
|
||||
if build_fails:
|
||||
self.assertIsNone(instance.state.get("candidate"))
|
||||
return
|
||||
self.assertEqual(instance.state["candidate"], IMAGE)
|
||||
self.assertEqual(instance.version(IMAGE)["run_id"], 44)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user