限制 dev 发布目录与 Jenkins 构建暂存的历史堆积
Project CI / AI game creator shell Rust crates (push) Successful in 1m26s
Project CI / AI game creator shell Rust smoke (push) Successful in 1m53s
Project CI / Backend tests (push) Successful in 3m47s
Project CI / AI game creator shell Rust lane 2/2 (push) Has been cancelled
Project CI / Repository checks (push) Has been cancelled
Project CI / Frontend tests (push) Has been cancelled
Project CI / AI game creator shell web tests (push) Has been cancelled
Project CI / AI game creator shell Rust lane 1/2 (push) Has been cancelled
Project CI / Native shell tests (push) Has been cancelled
Project CI / AI game creator shell Rust crates (push) Successful in 1m26s
Project CI / AI game creator shell Rust smoke (push) Successful in 1m53s
Project CI / Backend tests (push) Successful in 3m47s
Project CI / AI game creator shell Rust lane 2/2 (push) Has been cancelled
Project CI / Repository checks (push) Has been cancelled
Project CI / Frontend tests (push) Has been cancelled
Project CI / AI game creator shell web tests (push) Has been cancelled
Project CI / AI game creator shell Rust lane 1/2 (push) Has been cancelled
Project CI / Native shell tests (push) Has been cancelled
- production-api-deploy.sh 新增 --keep-releases(默认 2):发布成功后保留 current 目标与最近 1 个历史 release,只清理同时含 api-server 或 web 标记的旧目录,清理失败只告警、不改变发布结论 - production-api-deploy.sh 新增 prune_old_releases 与参数校验:非整数或小于 1 的 --keep-releases 在任何动作之前失败 - check-production-api-deploy 增加默认值、显式值和非法值三类夹具,并断言非法参数不会提升 release - check-production-ops 增加 release 保留策略与三个部署 Job 暂存清理的合同 - Api Deploy、Web Deploy、Stdb Publish 三个 Jenkinsfile 在部署或发布成功后只保留最近 2 个 build/<version> 暂存,失败时保留暂存便于诊断和重跑 - 内联清理片段避开 Groovy 反斜杠转义,并在 set -euo pipefail 下对空 build 目录安全(if [ -d build ] 守卫加兜底) - 开发运维文档补充 release 与 Jenkins 暂存的保留上限口径 - pitfalls 记录 dev 根盘写满的两处根因(发布与暂存无上限、JNLP 幽灵 agent 刷 syslog)以及 Jenkins 内联 shell 的两个坑
This commit is contained in:
@@ -5,7 +5,7 @@ set -euo pipefail
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
用法:
|
||||
./scripts/deploy/production-api-deploy.sh --source-dir build/<version> [--version <version>] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--pingora-service genarrative-pingora-gateway.service] [--require-pingora-gateway] [--bgfilter-worker-service genarrative-bgfilter-worker.service] [--bgfilter-worker-health-url <url>] [--bgfilter-worker-env-file /etc/genarrative/bgfilter-worker.env] [--no-bgfilter-worker] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--worker-controller-service genarrative-external-generation-controller.service] [--controller-env-file /etc/genarrative/external-generation-controller.env] [--no-worker-controller] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--worker-env-file /etc/genarrative/external-generation-worker.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101] [--keep-maintenance-mode]
|
||||
./scripts/deploy/production-api-deploy.sh --source-dir build/<version> [--version <version>] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--pingora-service genarrative-pingora-gateway.service] [--require-pingora-gateway] [--bgfilter-worker-service genarrative-bgfilter-worker.service] [--bgfilter-worker-health-url <url>] [--bgfilter-worker-env-file /etc/genarrative/bgfilter-worker.env] [--no-bgfilter-worker] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--worker-controller-service genarrative-external-generation-controller.service] [--controller-env-file /etc/genarrative/external-generation-controller.env] [--no-worker-controller] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--worker-env-file /etc/genarrative/external-generation-worker.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101] [--keep-maintenance-mode] [--keep-releases 2]
|
||||
|
||||
说明:
|
||||
进入维护模式,校验并发布 api-server 单文件,更新 current 链接,重启 systemd 服务并执行 readiness 检查。
|
||||
@@ -16,6 +16,7 @@ usage() {
|
||||
若发布包包含 pingora-gateway,或传入 --require-pingora-gateway,部署脚本会要求 release manifest、二进制与 checksum 一致,再在 current 链接切换后先复核 systemd/env 仍是本机高端口 shadow 配置,启动或重启 Pingora 影子服务并复核 active。
|
||||
默认在 readiness 通过后退出维护模式;传入 --keep-maintenance-mode 时保留维护文件,供人工验收后再恢复公网。
|
||||
current 链接切换前失败时会退出本次打开的维护模式;current 链接切换后失败时保留维护模式,避免暴露半发布版本。
|
||||
发布成功后按 --keep-releases(默认 2)保留 current 目标与最近的历史 release 目录,清理其它含 api-server 或 web 的旧发布目录,避免目标机根盘被历史发布撑满;该清理失败只告警,不改变本次发布结论。
|
||||
EOF
|
||||
}
|
||||
|
||||
@@ -1177,6 +1178,7 @@ DEPLOY_COMPLETED=0
|
||||
PINGORA_INCLUDED=0
|
||||
REQUIRE_PINGORA_GATEWAY=0
|
||||
KEEP_MAINTENANCE_MODE=0
|
||||
KEEP_RELEASES=2
|
||||
MAINTENANCE_ENABLED_BY_DEPLOY=0
|
||||
MAINTENANCE_FILE="${GENARRATIVE_MAINTENANCE_FILE:-/var/lib/genarrative/maintenance/enabled}"
|
||||
CURRENT_LINK_SWITCHED=0
|
||||
@@ -1222,6 +1224,10 @@ while [[ $# -gt 0 ]]; do
|
||||
KEEP_MAINTENANCE_MODE=1
|
||||
shift
|
||||
;;
|
||||
--keep-releases)
|
||||
KEEP_RELEASES="${2:?缺少 --keep-releases 的值}"
|
||||
shift 2
|
||||
;;
|
||||
--worker-service-pattern)
|
||||
WORKER_SERVICE_PATTERN="${2:?缺少 --worker-service-pattern 的值}"
|
||||
shift 2
|
||||
@@ -1290,6 +1296,10 @@ done
|
||||
require_argument "${SOURCE_DIR}" "--source-dir"
|
||||
require_absolute_path "${RELEASE_ROOT}" "--release-root"
|
||||
require_absolute_path "${CURRENT_LINK}" "--current-link"
|
||||
if [[ ! "${KEEP_RELEASES}" =~ ^[0-9]+$ ]] || (( KEEP_RELEASES < 1 )); then
|
||||
echo "[production-api-deploy] --keep-releases 必须是大于等于 1 的整数: ${KEEP_RELEASES}" >&2
|
||||
exit 1
|
||||
fi
|
||||
require_absolute_path "${API_ENV_FILE}" "--api-env-file"
|
||||
if [[ -n "${WORKER_ENV_FILE}" ]]; then
|
||||
require_absolute_path "${WORKER_ENV_FILE}" "--worker-env-file"
|
||||
@@ -1359,6 +1369,44 @@ cleanup_rendered_systemd_unit() {
|
||||
RENDERED_SYSTEMD_UNIT_FILE=""
|
||||
}
|
||||
|
||||
# 保留 current 目标与最近 keep_total-1 个其它发布目录。
|
||||
# 只清理同时不是符号链接、名称不以点开头且含 api-server 或 web 的目录,
|
||||
# 避免误删发布根目录下不属于发布产物的目录。
|
||||
prune_old_releases() {
|
||||
local keep_total="$1"
|
||||
local current_target=""
|
||||
local keep_others=$(( keep_total - 1 ))
|
||||
local kept_others=0
|
||||
local candidate resolved
|
||||
|
||||
if [[ -L "${CURRENT_LINK}" ]]; then
|
||||
current_target="$(readlink -f "${CURRENT_LINK}" 2>/dev/null || true)"
|
||||
fi
|
||||
|
||||
while IFS= read -r candidate; do
|
||||
[[ -n "${candidate}" ]] || continue
|
||||
case "${candidate}" in
|
||||
*[[:space:]]*) continue ;;
|
||||
esac
|
||||
[[ -d "${candidate}" && ! -L "${candidate}" ]] || continue
|
||||
[[ -e "${candidate}/api-server" || -e "${candidate}/web" ]] || continue
|
||||
resolved="$(readlink -f "${candidate}")"
|
||||
if [[ -n "${current_target}" && "${resolved}" == "${current_target}" ]]; then
|
||||
continue
|
||||
fi
|
||||
if (( kept_others < keep_others )); then
|
||||
kept_others=$(( kept_others + 1 ))
|
||||
continue
|
||||
fi
|
||||
echo "[production-api-deploy] 清理历史 release: ${candidate}"
|
||||
rm -rf --one-file-system "${resolved}"
|
||||
done < <(
|
||||
find "${RELEASE_ROOT}" -mindepth 1 -maxdepth 1 -type d -name '[!.]*' -printf '%T@ %p\n' 2>/dev/null |
|
||||
sort -rn |
|
||||
cut -d' ' -f2-
|
||||
)
|
||||
}
|
||||
|
||||
on_exit() {
|
||||
local exit_code=$?
|
||||
cleanup_rendered_systemd_unit
|
||||
@@ -1697,6 +1745,8 @@ for _ in {1..30}; do
|
||||
bash "${SCRIPT_DIR}/maintenance-off.sh"
|
||||
fi
|
||||
DEPLOY_COMPLETED=1
|
||||
prune_old_releases "${KEEP_RELEASES}" ||
|
||||
echo "[production-api-deploy] 历史 release 清理失败(仅影响磁盘占用,不影响本次发布结论)" >&2
|
||||
echo "[production-api-deploy] 完成: ${RELEASE_DIR}/api-server"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user