Harden production publish flow #5

Merged
kdletters merged 26 commits from codex/publish-flow into master 2026-05-03 03:47:14 +08:00
4 changed files with 140 additions and 19 deletions
Showing only changes of commit 6c519970b4 - Show all commits

View File

@@ -0,0 +1,87 @@
# 开发服无域名时使用的 HTTP 入口,只允许用于 DEPLOY_TARGET=development。
# 没有域名时,将 SERVER_NAME 填为开发机 IP 或临时主机名。
# 生产 release 仍必须使用 genarrative.conf 的 HTTPS 配置。
server {
listen 80;
server_name genarrative.example.com;
root /srv/genarrative/web;
index index.html;
include /etc/nginx/snippets/genarrative-maintenance.conf;
location ^~ /admin/api/ {
default_type application/json;
if ($genarrative_maintenance) {
return 503 '{"ok":false,"error":{"code":"MAINTENANCE","message":"服务维护中"}}';
}
proxy_pass http://127.0.0.1:8082/admin/api/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Request-Id $request_id;
}
location = /admin {
return 301 /admin/;
}
location ^~ /admin/assets/ {
try_files $uri =404;
}
location ^~ /admin/ {
error_page 503 /maintenance.html;
if ($genarrative_maintenance) {
return 503;
}
try_files $uri $uri/ /admin/index.html;
}
location ^~ /assets/ {
try_files $uri =404;
}
# 开发服也不恢复旧一体化 API、生成资源代理和健康检查公网入口。
location ~ ^/(api|generated-|healthz) {
return 404;
}
# 仅开放前端 SpacetimeDB SDK 运行所需的最小公网路由。
location ~ ^/v1/database/[^/]+/subscribe$ {
proxy_pass http://127.0.0.1:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_read_timeout 3600s;
}
location ^~ /v1/identity {
proxy_pass http://127.0.0.1:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
}
location ^~ /v1/ {
return 404;
}
location / {
error_page 503 /maintenance.html;
if ($genarrative_maintenance) {
return 503;
}
try_files $uri $uri/ /index.html;
}
}

View File

@@ -9,6 +9,7 @@
- `deploy/systemd/spacetimedb.service`
- `deploy/systemd/genarrative-api.service`
- `deploy/nginx/genarrative.conf`
- `deploy/nginx/genarrative-dev-http.conf`
- `deploy/nginx/snippets/genarrative-maintenance.conf`
- `deploy/env/api-server.env.example`
- `scripts/deploy/maintenance-on.sh`
@@ -93,12 +94,12 @@
## Nginx 规则
只保留必要入口
生产正式入口只保留必要路由
- `/`:主站静态页面。
- `/admin/`:后台前端静态页面,后台构建时使用 `/admin/` 作为 base path。
- `/admin/api/`:反向代理到 `http://127.0.0.1:8082/admin/api/`
- HTTP 到 HTTPS只保留 301 重定向。
- HTTP 到 HTTPS`production-https` 模式只保留 301 重定向。
- `/maintenance.html`:维护中页面。
移除这些公网反向代理:
@@ -110,6 +111,11 @@
SpacetimeDB 公网路由默认保持收敛,只按实际前端 SDK 需要暴露最小集合。禁止开放可远程发布数据库或管理实例的通用入口。
Nginx 配置文件分为两类:
- `deploy/nginx/genarrative.conf`:生产正式域名 HTTPS 配置,`genarrative.example.com` 只是占位域名,安装时必须替换为真实 `SERVER_NAME`,并要求 `/etc/letsencrypt/live/<SERVER_NAME>/fullchain.pem``privkey.pem` 已存在。
- `deploy/nginx/genarrative-dev-http.conf`:开发服无域名时的 HTTP-only 配置,只允许 `DEPLOY_TARGET=development` 使用。没有域名时,`SERVER_NAME` 填开发机 IP 或临时主机名。它仍复用同一套静态目录、后台 API 反代和 SpacetimeDB SDK 最小公网路由,不恢复旧 `/api/*``/generated-*` 或公网 `/healthz`
## 维护模式
维护模式由 `/var/lib/genarrative/maintenance/enabled` 控制:
@@ -164,6 +170,7 @@ build/<version>/
│ │ └─ genarrative-api.service
│ ├─ nginx/
│ │ ├─ genarrative.conf
│ │ ├─ genarrative-dev-http.conf
│ │ └─ snippets/genarrative-maintenance.conf
│ └─ env/api-server.env.example
└─ README.md
@@ -322,11 +329,11 @@ WASM_SOURCE="${CARGO_TARGET_DIR}/wasm32-unknown-unknown/release/spacetime_module
并发与清理规则:
- Rust 构建 Job 建议使用 `disableConcurrentBuilds()`或对共享 `CARGO_TARGET_DIR` 加 Jenkins `lock`,避免多个同包 release 构建同时写入同一最终产物路径。
- 同一个 Rust 构建 Job 建议使用 `disableConcurrentBuilds()`避免同一组件的多个 release 构建同时写入同一最终产物路径。
- 如果 Linux agent 未安装 `sccache`Rust 构建流水线必须自动取消 `RUSTC_WRAPPER`,回退到直接使用 `rustc`,不能因为缺少可选缓存工具阻断真实构建。
- 生产发布流水线只能消费 `build/<version>/` 或 Jenkins 归档产物,不允许从共享 `cargo-target` 目录直接发布。
- `SCCACHE_CACHE_SIZE` 必须设置上限,避免编译缓存无限增长。
-`/var/cache/genarrative-build/cargo-target` 或数据盘对应目录配置定期清理,建议保留最近 14 到 30 天。
-`/var/cache/genarrative-build/*/cargo-target` 或数据盘对应目录配置定期清理,建议保留最近 14 到 30 天。
- Jenkins Job 必须配置构建记录和归档产物保留策略,避免历史 release 包长期堆积。
### 统一源码版本参数
@@ -374,9 +381,9 @@ WASM_SOURCE="${CARGO_TARGET_DIR}/wasm32-unknown-unknown/release/spacetime_module
该流水线属于高风险操作,默认要求人工确认后执行。
已落地的 Jenkinsfile 为 `jenkins/Jenkinsfile.production-server-provision`。该流水线默认 `DRY_RUN=true`只打印将执行的初始化动作真正写入系统用户、目录、systemd、环境文件并启动服务时必须设置 `DRY_RUN=false` 且勾选 `CONFIRM_PROVISION`。当 `DEPLOY_TARGET=release` 时,还必须勾选 `CONFIRM_RELEASE_DEPLOY_AGENT`,并通过 `linux && genarrative-release-deploy` 调度到独立 release 部署 agent。
首次真实初始化默认保持 `INSTALL_NGINX_CONFIG=false`先完成系统用户、目录、SpacetimeDB、systemd unit 与 `/etc/genarrative/api-server.env` 落盘。等真实域名确定,并且目标机已经存在 `/etc/letsencrypt/live/<SERVER_NAME>/fullchain.pem``/etc/letsencrypt/live/<SERVER_NAME>/privkey.pem` 后,再把 `SERVER_NAME``genarrative.example.com` 改成真实域名,并设置 `INSTALL_NGINX_CONFIG=true` 安装 Nginx HTTPS 配置。流水线会拒绝在真实初始化中用占位域名写入 Nginx 配置,也会在证书缺失时提前失败
首次真实初始化默认保持 `NGINX_CONFIG_MODE=none`先完成系统用户、目录、SpacetimeDB、systemd unit 与 `/etc/genarrative/api-server.env` 落盘。开发服没有域名时,使用 `DEPLOY_TARGET=development` + `NGINX_CONFIG_MODE=development-http` 安装 `deploy/nginx/genarrative-dev-http.conf`,并把 `SERVER_NAME` 填为开发机 IP 或临时主机名。等正式域名确定,并且目标机已经存在 `/etc/letsencrypt/live/<SERVER_NAME>/fullchain.pem``/etc/letsencrypt/live/<SERVER_NAME>/privkey.pem` 后,再把 `SERVER_NAME` 改成真实域名,并设置 `NGINX_CONFIG_MODE=production-https` 安装 Nginx HTTPS 配置。流水线会拒绝 release 目标安装 `development-http`,也会拒绝用占位域名或缺失证书安装 `production-https`
若误用占位域名执行过真实初始化,失败通常发生在 `nginx -t`,错误表现为找不到 `/etc/letsencrypt/live/genarrative.example.com/fullchain.pem``privkey.pem`。新版初始化在 `INSTALL_NGINX_CONFIG=false` 时会检测并禁用上一轮留下的占位域名 Nginx 配置,避免它继续影响后续 `nginx -t`
若误用占位域名执行过真实初始化,失败通常发生在 `nginx -t`,错误表现为找不到 `/etc/letsencrypt/live/genarrative.example.com/fullchain.pem``privkey.pem`。新版初始化在 `NGINX_CONFIG_MODE=none` 时会检测并禁用上一轮留下的占位域名 Nginx 配置,避免它继续影响后续 `nginx -t`
### Web Build / Deploy
@@ -510,6 +517,7 @@ WASM_SOURCE="${CARGO_TARGET_DIR}/wasm32-unknown-unknown/release/spacetime_module
- [x] `deploy/systemd/spacetimedb.service`
- [x] `deploy/systemd/genarrative-api.service`
- [x] `deploy/nginx/genarrative.conf`
- [x] `deploy/nginx/genarrative-dev-http.conf`
- [x] `deploy/nginx/snippets/genarrative-maintenance.conf`
- [x] `deploy/env/api-server.env.example`
- [x] `scripts/deploy/maintenance-on.sh`

View File

@@ -26,7 +26,7 @@ pipeline {
string(name: 'WEB_LINK', defaultValue: '/srv/genarrative/web', description: 'Nginx 静态站点目录或软链接')
string(name: 'API_ENV_FILE', defaultValue: '/etc/genarrative/api-server.env', description: 'api-server 环境文件')
string(name: 'API_PORT', defaultValue: '8082', description: 'api-server 本机监听端口')
booleanParam(name: 'INSTALL_NGINX_CONFIG', defaultValue: false, description: '安装 Nginx 配置并执行 nginx -t首次初始化且证书未准备好时保持关闭')
choice(name: 'NGINX_CONFIG_MODE', choices: ['none', 'production-https', 'development-http'], description: 'Nginx 配置模式;开发服无域名时选 development-httprelease 正式入口选 production-https')
booleanParam(name: 'ENABLE_SERVICES', defaultValue: true, description: '启用并启动 spacetimedb 与 api-server systemd 服务')
}
@@ -49,8 +49,15 @@ pipeline {
if (!params.SPACETIME_BIN_SOURCE?.trim()) {
error('SPACETIME_BIN_SOURCE 不能为空。')
}
if (!params.DRY_RUN && params.INSTALL_NGINX_CONFIG && params.SERVER_NAME?.trim() == 'genarrative.example.com') {
error('真实初始化安装 Nginx 配置时必须把 SERVER_NAME 改成真实域名,不能使用 genarrative.example.com 占位值。证书未准备好时请先保持 INSTALL_NGINX_CONFIG=false。')
def nginxMode = params.NGINX_CONFIG_MODE?.trim()
if (!(nginxMode in ['none', 'production-https', 'development-http'])) {
error("NGINX_CONFIG_MODE 只能是 none、production-https 或 development-http当前值: ${params.NGINX_CONFIG_MODE}")
}
if (params.DEPLOY_TARGET == 'release' && nginxMode == 'development-http') {
error('release 目标禁止安装 development-http Nginx 配置;无证书初始化请使用 NGINX_CONFIG_MODE=none。')
}
if (!params.DRY_RUN && nginxMode == 'production-https' && params.SERVER_NAME?.trim() == 'genarrative.example.com') {
error('真实初始化安装 Nginx 配置时必须把 SERVER_NAME 改成真实域名,不能使用 genarrative.example.com 占位值。证书未准备好时请先保持 NGINX_CONFIG_MODE=none。')
}
}
}
@@ -116,10 +123,14 @@ pipeline {
fi
}
render_nginx_config() {
render_nginx_https_config() {
sed "s/genarrative.example.com/${SERVER_NAME}/g" deploy/nginx/genarrative.conf
}
render_nginx_development_http_config() {
sed "s/genarrative.example.com/${SERVER_NAME}/g" deploy/nginx/genarrative-dev-http.conf
}
render_api_env_example() {
sed \
-e "s|^GENARRATIVE_API_PORT=.*|GENARRATIVE_API_PORT=${API_PORT}|" \
@@ -130,12 +141,12 @@ pipeline {
validate_nginx_tls() {
local cert_dir="/etc/letsencrypt/live/${SERVER_NAME}"
if [[ "${SERVER_NAME}" == "genarrative.example.com" ]]; then
echo "[server-provision] SERVER_NAME 仍是占位域名,拒绝写入 Nginx HTTPS 配置。请填写真实域名,或先设置 INSTALL_NGINX_CONFIG=false。" >&2
echo "[server-provision] SERVER_NAME 仍是占位域名,拒绝写入 Nginx HTTPS 配置。请填写真实域名,或先设置 NGINX_CONFIG_MODE=none。" >&2
exit 1
fi
if [[ ! -f "${cert_dir}/fullchain.pem" || ! -f "${cert_dir}/privkey.pem" ]]; then
echo "[server-provision] 未找到 Nginx HTTPS 证书: ${cert_dir}/fullchain.pem 或 ${cert_dir}/privkey.pem" >&2
echo "[server-provision] 请先完成证书申请,或首次初始化时设置 INSTALL_NGINX_CONFIG=false避免写入无法通过 nginx -t 的配置。" >&2
echo "[server-provision] 请先完成证书申请,或首次初始化时设置 NGINX_CONFIG_MODE=none避免写入无法通过 nginx -t 的配置。" >&2
exit 1
fi
}
@@ -143,12 +154,22 @@ pipeline {
install_nginx_config_with_rollback() {
local config_target="/etc/nginx/conf.d/genarrative.conf"
local snippet_target="/etc/nginx/snippets/genarrative-maintenance.conf"
local config_source
local rendered_config rendered_snippet config_backup snippet_backup
local had_config="false"
local had_snippet="false"
run_cmd mkdir -p /etc/nginx/snippets /etc/nginx/conf.d
echo "+ render deploy/nginx/genarrative.conf -> ${config_target}"
if [[ "${NGINX_CONFIG_MODE}" == "production-https" ]]; then
config_source="deploy/nginx/genarrative.conf"
elif [[ "${NGINX_CONFIG_MODE}" == "development-http" ]]; then
config_source="deploy/nginx/genarrative-dev-http.conf"
else
echo "[server-provision] NGINX_CONFIG_MODE=${NGINX_CONFIG_MODE} 不需要安装 Nginx 配置。"
return
fi
echo "+ render ${config_source} -> ${config_target}"
echo "+ install -m 0644 deploy/nginx/snippets/genarrative-maintenance.conf ${snippet_target}"
if [[ "${DRY_RUN}" == "true" ]]; then
@@ -156,12 +177,16 @@ pipeline {
return
fi
validate_nginx_tls
rendered_config="$(mktemp)"
rendered_snippet="$(mktemp)"
config_backup="$(mktemp)"
snippet_backup="$(mktemp)"
render_nginx_config >"${rendered_config}"
if [[ "${NGINX_CONFIG_MODE}" == "production-https" ]]; then
validate_nginx_tls
render_nginx_https_config >"${rendered_config}"
else
render_nginx_development_http_config >"${rendered_config}"
fi
cp deploy/nginx/snippets/genarrative-maintenance.conf "${rendered_snippet}"
if [[ -f "${config_target}" ]]; then
@@ -244,13 +269,14 @@ pipeline {
require_path deploy/systemd/spacetimedb.service
require_path deploy/systemd/genarrative-api.service
require_path deploy/nginx/genarrative.conf
require_path deploy/nginx/genarrative-dev-http.conf
require_path deploy/nginx/snippets/genarrative-maintenance.conf
require_path deploy/env/api-server.env.example
require_path scripts/deploy/maintenance-on.sh
require_path scripts/deploy/maintenance-off.sh
require_path scripts/deploy/maintenance-status.sh
echo "[server-provision] target=${DEPLOY_TARGET}, dry_run=${DRY_RUN}, source_commit=$(cat .jenkins-source-commit)"
echo "[server-provision] target=${DEPLOY_TARGET}, dry_run=${DRY_RUN}, nginx_config_mode=${NGINX_CONFIG_MODE}, source_commit=$(cat .jenkins-source-commit)"
run_cmd id
run_cmd mkdir -p "${SPACETIME_ROOT}" "${RELEASE_ROOT}" "$(dirname "${CURRENT_LINK}")" "$(dirname "${WEB_LINK}")" /etc/genarrative /var/lib/genarrative/maintenance /var/lib/genarrative/auth
@@ -299,7 +325,7 @@ pipeline {
echo "[server-provision] 已存在环境文件,保留不覆盖: ${API_ENV_FILE}"
fi
if [[ "${INSTALL_NGINX_CONFIG}" == "true" ]]; then
if [[ "${NGINX_CONFIG_MODE}" != "none" ]]; then
install_nginx_config_with_rollback
else
cleanup_placeholder_nginx_config
@@ -325,7 +351,7 @@ pipeline {
post {
success {
echo "Server provision 完成: target=${params.DEPLOY_TARGET}, dryRun=${params.DRY_RUN}"
echo "Server provision 完成: target=${params.DEPLOY_TARGET}, dryRun=${params.DRY_RUN}, nginxConfigMode=${params.NGINX_CONFIG_MODE}"
}
}
}

View File

@@ -399,7 +399,7 @@ cat >"${TARGET_DIR}/README.md" <<EOF
- \`*.sha256\`:发布产物 checksum用于部署前校验。
- \`release-manifest.json\`:发布版本、源码 commit 与产物清单。
- \`scripts/\`:维护模式脚本、数据库导入导出脚本和迁移授权脚本。
- \`deploy/\`systemd、Nginx 和生产环境变量示例。
- \`deploy/\`systemd、Nginx 和生产环境变量示例\`deploy/nginx/genarrative-dev-http.conf\` 仅供无域名开发服初始化使用
## 生产部署口径