限制 dev 发布目录与 Jenkins 构建暂存的历史堆积
Project CI / AI game creator shell Rust crates (push) Successful in 1m26s
Project CI / AI game creator shell Rust smoke (push) Successful in 1m53s
Project CI / Backend tests (push) Successful in 3m47s
Project CI / AI game creator shell Rust lane 2/2 (push) Has been cancelled
Project CI / Repository checks (push) Has been cancelled
Project CI / Frontend tests (push) Has been cancelled
Project CI / AI game creator shell web tests (push) Has been cancelled
Project CI / AI game creator shell Rust lane 1/2 (push) Has been cancelled
Project CI / Native shell tests (push) Has been cancelled

- production-api-deploy.sh 新增 --keep-releases(默认 2):发布成功后保留 current 目标与最近 1 个历史 release,只清理同时含 api-server 或 web 标记的旧目录,清理失败只告警、不改变发布结论
- production-api-deploy.sh 新增 prune_old_releases 与参数校验:非整数或小于 1 的 --keep-releases 在任何动作之前失败
- check-production-api-deploy 增加默认值、显式值和非法值三类夹具,并断言非法参数不会提升 release
- check-production-ops 增加 release 保留策略与三个部署 Job 暂存清理的合同
- Api Deploy、Web Deploy、Stdb Publish 三个 Jenkinsfile 在部署或发布成功后只保留最近 2 个 build/<version> 暂存,失败时保留暂存便于诊断和重跑
- 内联清理片段避开 Groovy 反斜杠转义,并在 set -euo pipefail 下对空 build 目录安全(if [ -d build ] 守卫加兜底)
- 开发运维文档补充 release 与 Jenkins 暂存的保留上限口径
- pitfalls 记录 dev 根盘写满的两处根因(发布与暂存无上限、JNLP 幽灵 agent 刷 syslog)以及 Jenkins 内联 shell 的两个坑
This commit is contained in:
kdletters
2026-09-29 07:19:00 +08:00
parent 589f3582ae
commit 16ff10111f
8 changed files with 235 additions and 2 deletions
+51 -1
View File
@@ -5,7 +5,7 @@ set -euo pipefail
usage() {
cat <<'EOF'
用法:
./scripts/deploy/production-api-deploy.sh --source-dir build/<version> [--version <version>] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--pingora-service genarrative-pingora-gateway.service] [--require-pingora-gateway] [--bgfilter-worker-service genarrative-bgfilter-worker.service] [--bgfilter-worker-health-url <url>] [--bgfilter-worker-env-file /etc/genarrative/bgfilter-worker.env] [--no-bgfilter-worker] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--worker-controller-service genarrative-external-generation-controller.service] [--controller-env-file /etc/genarrative/external-generation-controller.env] [--no-worker-controller] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--worker-env-file /etc/genarrative/external-generation-worker.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101] [--keep-maintenance-mode]
./scripts/deploy/production-api-deploy.sh --source-dir build/<version> [--version <version>] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--pingora-service genarrative-pingora-gateway.service] [--require-pingora-gateway] [--bgfilter-worker-service genarrative-bgfilter-worker.service] [--bgfilter-worker-health-url <url>] [--bgfilter-worker-env-file /etc/genarrative/bgfilter-worker.env] [--no-bgfilter-worker] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--worker-controller-service genarrative-external-generation-controller.service] [--controller-env-file /etc/genarrative/external-generation-controller.env] [--no-worker-controller] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--worker-env-file /etc/genarrative/external-generation-worker.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101] [--keep-maintenance-mode] [--keep-releases 2]
说明:
进入维护模式,校验并发布 api-server 单文件,更新 current 链接,重启 systemd 服务并执行 readiness 检查。
@@ -16,6 +16,7 @@ usage() {
若发布包包含 pingora-gateway,或传入 --require-pingora-gateway,部署脚本会要求 release manifest、二进制与 checksum 一致,再在 current 链接切换后先复核 systemd/env 仍是本机高端口 shadow 配置,启动或重启 Pingora 影子服务并复核 active。
默认在 readiness 通过后退出维护模式;传入 --keep-maintenance-mode 时保留维护文件,供人工验收后再恢复公网。
current 链接切换前失败时会退出本次打开的维护模式;current 链接切换后失败时保留维护模式,避免暴露半发布版本。
发布成功后按 --keep-releases(默认 2)保留 current 目标与最近的历史 release 目录,清理其它含 api-server 或 web 的旧发布目录,避免目标机根盘被历史发布撑满;该清理失败只告警,不改变本次发布结论。
EOF
}
@@ -1177,6 +1178,7 @@ DEPLOY_COMPLETED=0
PINGORA_INCLUDED=0
REQUIRE_PINGORA_GATEWAY=0
KEEP_MAINTENANCE_MODE=0
KEEP_RELEASES=2
MAINTENANCE_ENABLED_BY_DEPLOY=0
MAINTENANCE_FILE="${GENARRATIVE_MAINTENANCE_FILE:-/var/lib/genarrative/maintenance/enabled}"
CURRENT_LINK_SWITCHED=0
@@ -1222,6 +1224,10 @@ while [[ $# -gt 0 ]]; do
KEEP_MAINTENANCE_MODE=1
shift
;;
--keep-releases)
KEEP_RELEASES="${2:?缺少 --keep-releases 的值}"
shift 2
;;
--worker-service-pattern)
WORKER_SERVICE_PATTERN="${2:?缺少 --worker-service-pattern 的值}"
shift 2
@@ -1290,6 +1296,10 @@ done
require_argument "${SOURCE_DIR}" "--source-dir"
require_absolute_path "${RELEASE_ROOT}" "--release-root"
require_absolute_path "${CURRENT_LINK}" "--current-link"
if [[ ! "${KEEP_RELEASES}" =~ ^[0-9]+$ ]] || (( KEEP_RELEASES < 1 )); then
echo "[production-api-deploy] --keep-releases 必须是大于等于 1 的整数: ${KEEP_RELEASES}" >&2
exit 1
fi
require_absolute_path "${API_ENV_FILE}" "--api-env-file"
if [[ -n "${WORKER_ENV_FILE}" ]]; then
require_absolute_path "${WORKER_ENV_FILE}" "--worker-env-file"
@@ -1359,6 +1369,44 @@ cleanup_rendered_systemd_unit() {
RENDERED_SYSTEMD_UNIT_FILE=""
}
# 保留 current 目标与最近 keep_total-1 个其它发布目录。
# 只清理同时不是符号链接、名称不以点开头且含 api-server 或 web 的目录,
# 避免误删发布根目录下不属于发布产物的目录。
prune_old_releases() {
local keep_total="$1"
local current_target=""
local keep_others=$(( keep_total - 1 ))
local kept_others=0
local candidate resolved
if [[ -L "${CURRENT_LINK}" ]]; then
current_target="$(readlink -f "${CURRENT_LINK}" 2>/dev/null || true)"
fi
while IFS= read -r candidate; do
[[ -n "${candidate}" ]] || continue
case "${candidate}" in
*[[:space:]]*) continue ;;
esac
[[ -d "${candidate}" && ! -L "${candidate}" ]] || continue
[[ -e "${candidate}/api-server" || -e "${candidate}/web" ]] || continue
resolved="$(readlink -f "${candidate}")"
if [[ -n "${current_target}" && "${resolved}" == "${current_target}" ]]; then
continue
fi
if (( kept_others < keep_others )); then
kept_others=$(( kept_others + 1 ))
continue
fi
echo "[production-api-deploy] 清理历史 release: ${candidate}"
rm -rf --one-file-system "${resolved}"
done < <(
find "${RELEASE_ROOT}" -mindepth 1 -maxdepth 1 -type d -name '[!.]*' -printf '%T@ %p\n' 2>/dev/null |
sort -rn |
cut -d' ' -f2-
)
}
on_exit() {
local exit_code=$?
cleanup_rendered_systemd_unit
@@ -1697,6 +1745,8 @@ for _ in {1..30}; do
bash "${SCRIPT_DIR}/maintenance-off.sh"
fi
DEPLOY_COMPLETED=1
prune_old_releases "${KEEP_RELEASES}" ||
echo "[production-api-deploy] 历史 release 清理失败(仅影响磁盘占用,不影响本次发布结论)" >&2
echo "[production-api-deploy] 完成: ${RELEASE_DIR}/api-server"
exit 0
fi