补齐 Pingora TLS 证书同步

新增 Pingora TLS 证书同步脚本,默认 dry-run,apply 时原子复制证书到私有目录并复核服务用户可读。

将证书同步脚本纳入 API release 构建、部署复制和 current release 自审清单。

补充发布包、deploy、自审和脚本自身门禁,覆盖缺失脚本、symlink 目标与权限边界。

更新 Pingora env 示例、运维文档和项目踩坑记录,明确不要放宽 Let’s Encrypt 原路径权限。
This commit is contained in:
2026-06-17 22:40:05 +08:00
parent 206ea48878
commit 057522fc2e
13 changed files with 841 additions and 12 deletions
+2 -1
View File
@@ -46,7 +46,8 @@
- Pingora 网关默认仍只监听本机 shadow 端口,不绑定公网 `80/443`
- 需要评估直连入口时,显式配置 `GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN``GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE``GENARRATIVE_PINGORA_GATEWAY_TLS_KEY_FILE`;网关只读取现有证书文件,不负责 Certbot / ACME 自动续期。
- 需要 HTTP 到 HTTPS 跳转时,再显式配置 `GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_LISTEN`;该入口除 `/.well-known/acme-challenge/*` 外统一 301 到 HTTPSACME challenge 仍从 `GENARRATIVE_PINGORA_GATEWAY_ACME_ROOT` 精确读取。
- API release 必须把 `api-server.sha256`、可选 `pingora-gateway.sha256``scripts/check-pingora-release-readiness.mjs``scripts/check-pingora-canary-live.mjs``scripts/check-pingora-canary-access-log-parity.mjs``scripts/ops/pingora-current-release-audit.mjs``scripts/ops/pingora-cutover-status-snapshot.mjs``scripts/ops/pingora-cutover-evidence-bundle.mjs``scripts/check-pingora-direct-preflight.mjs``scripts/check-pingora-direct-live.mjs``scripts/check-production-health-patrol-env.mjs``scripts/deploy/pingora-health-patrol-env-switch.mjs``deploy/systemd/``deploy/env/``deploy/pingora/` 一起复制到 `/opt/genarrative/current`current release 自审、直连启用、health patrol env 切换、canary live、access log 对账、cutover 状态快照和证据包脚本都从 current release 推导依赖路径,切换窗口不得依赖 Jenkins 工作区或目标机源码 checkout。
- API release 必须把 `api-server.sha256`、可选 `pingora-gateway.sha256``scripts/check-pingora-release-readiness.mjs``scripts/check-pingora-canary-live.mjs``scripts/check-pingora-canary-access-log-parity.mjs``scripts/ops/pingora-current-release-audit.mjs``scripts/ops/pingora-cutover-status-snapshot.mjs``scripts/ops/pingora-cutover-evidence-bundle.mjs``scripts/check-pingora-direct-preflight.mjs``scripts/check-pingora-direct-live.mjs``scripts/check-production-health-patrol-env.mjs``scripts/deploy/pingora-health-patrol-env-switch.mjs``scripts/deploy/pingora-tls-cert-sync.mjs``deploy/systemd/``deploy/env/``deploy/pingora/` 一起复制到 `/opt/genarrative/current`current release 自审、直连启用、health patrol env 切换、TLS 证书同步、canary live、access log 对账、cutover 状态快照和证据包脚本都从 current release 推导依赖路径,切换窗口不得依赖 Jenkins 工作区或目标机源码 checkout。
- 目标机如果使用 Certbot / Lets Encrypt 证书,不要直接放宽 `/etc/letsencrypt/live``archive` 的目录 / 私钥权限。先用随包脚本把 root 可读的 live symlink 解析并复制到 Pingora 私有目录,例如 `sudo -n node -- /opt/genarrative/current/scripts/deploy/pingora-tls-cert-sync.mjs --apply --source-cert-file /etc/letsencrypt/live/<域名>/fullchain.pem --source-key-file /etc/letsencrypt/live/<域名>/privkey.pem --target-dir /etc/genarrative/pingora-tls/<域名>`;脚本默认 dry-run`--apply` 才原子写入 `fullchain.pem` / `privkey.pem`,目标目录默认 `root:genarrative 0750`,文件默认 `root:genarrative 0640`,并复核 `genarrative` 服务用户可读。随后把 `GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE` / `TLS_KEY_FILE` 指向 `/etc/genarrative/pingora-tls/<域名>/` 下的副本。
- 默认 systemd service 以 `genarrative` 非 root 用户运行,不具备绑定 `80/443` 的能力。Server-Provision 只安装 `/etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf` 作为人工启用模板;切直连低端口时必须先确认 `genarrative` 可读证书文件、Nginx 已释放 `80/443`,再执行 `/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh --apply --preflight-env-file /etc/genarrative/pingora-gateway.env --preflight-check-cert-readable --preflight-check-service-env-file --preflight-check-service-user-cert-readable --preflight-check-service-binary-executable --preflight-check-ports-free --direct-https-base-url https://127.0.0.1 --direct-http-base-url http://127.0.0.1 --direct-host <域名> --direct-redirect-host <域名或host:port> --direct-spacetime-database <库名> --direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`;脚本会先执行随包 current release 自审,确认 `pingora-gateway` 可执行且 systemd `ExecStart` 指向 current release,失败时不会安装 drop-in。自审通过后才安装 drop-in、`systemctl daemon-reload` 并重启 Pingora;脚本会用 `systemctl cat` 确认 `CAP_NET_BIND_SERVICE` drop-in 已进入 systemd 最终配置,用 `systemctl show ... ExecStart` 确认最终 service 指向随包主 service 模板中的 current release `pingora-gateway`,用 `systemctl is-active` 确认 Pingora active,并执行 direct live smoke,强制覆盖 HTTPS / HTTP redirect / ACME / WSS 101 和 Pingora access log request_id 落盘。
- `npm run check:pingora-gateway-smoke` 会用临时自签证书覆盖 TLS 直连、HTTP redirect 和 WSS subscribe 行为;正式公网切换前仍必须先走 Nginx canary 和 release readiness 门禁。
- 目标机直连入口启用后先运行 `npm run check:pingora-direct-preflight -- --env-file /etc/genarrative/pingora-gateway.env --require-live-env --systemd-cat --check-cert-readable --check-service-env-file --check-service-user-cert-readable --check-service-binary-executable --check-ports-free`,再运行 `npm run check:pingora-direct-live`,检查 HTTPS 根路径、HTTP/2 ALPN、代表性 API、SpacetimeDB identity、WSS subscribe 握手、generated 拒绝、公开 `/healthz` 拒绝、可选 shadow probe,以及 HTTP 301 / ACME challenge;真实 SpacetimeDB 对 `GET /v1/identity` 返回 `405 Method Not Allowed` 属于可接受语义,direct live 只把它作为路径转发代表,不要求该 GET 创建 identityrelease readiness 可用 `--require-direct --direct-https-base-url https://127.0.0.1 --direct-http-base-url http://127.0.0.1 --direct-host <域名> --direct-redirect-host <域名或host:port> --direct-spacetime-database <库名> --direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log --direct-health-patrol-env-file /etc/genarrative/health-patrol.env --direct-preflight-env-file /etc/genarrative/pingora-gateway.env --direct-preflight-systemd --direct-preflight-check-cert-readable --direct-preflight-check-service-env-file --direct-preflight-check-service-user-cert-readable --direct-preflight-check-service-binary-executable --direct-preflight-check-ports-free` 强制纳入同一条门禁,漏掉 direct HTTP base URL、正式域名 Host/SNI、redirect Location host、目标 SpacetimeDB 库、Pingora access log request_id 落盘、health patrol env direct 模式复核、systemd drop-in 生效检查、service EnvironmentFile 一致性检查、当前用户 / systemd 服务用户证书可读检查、service 二进制可执行检查或 80/443 端口释放检查都会直接失败。
+5 -3
View File
@@ -4,11 +4,13 @@
GENARRATIVE_PINGORA_GATEWAY_LISTEN=127.0.0.1:18081
# 可选:直连 HTTPS 入口。默认关闭;启用时必须同时配置 cert/key,并把 FORWARDED_PROTO 设为 https。
# cert/key 必须允许 genarrative 用户读取;80/443 必须先从 Nginx 或其它进程释放
# cert/key 必须允许 genarrative 用户读取;不要直接 chmod Let's Encrypt live/archive 原始目录
# 可先用随包 scripts/deploy/pingora-tls-cert-sync.mjs 把证书同步到 /etc/genarrative/pingora-tls/<host>/。
# 80/443 必须先从 Nginx 或其它进程释放;高端口 loopback 演练可先使用 127.0.0.1:18443 / 127.0.0.1:18080。
# 如果要绑定 443/80,必须另行人工启用 systemd direct-entry drop-in 授予 CAP_NET_BIND_SERVICE。
# GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN=0.0.0.0:18443
# GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE=/etc/letsencrypt/live/genarrative.world/fullchain.pem
# GENARRATIVE_PINGORA_GATEWAY_TLS_KEY_FILE=/etc/letsencrypt/live/genarrative.world/privkey.pem
# GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE=/etc/genarrative/pingora-tls/genarrative.world/fullchain.pem
# GENARRATIVE_PINGORA_GATEWAY_TLS_KEY_FILE=/etc/genarrative/pingora-tls/genarrative.world/privkey.pem
# 可选:直连 HTTP 只做 301 到 HTTPS;启用时必须已经配置 TLS_LISTENACME challenge 仍从 ACME_ROOT 静态读取。
# GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_LISTEN=0.0.0.0:18080
# GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_TARGET_SCHEME=https
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+1
View File
@@ -44,6 +44,7 @@
"check:pingora-direct-enable": "node scripts/check-pingora-direct-enable.mjs",
"check:pingora-direct-rollback": "node scripts/check-pingora-direct-rollback.mjs",
"check:pingora-health-patrol-env-switch": "node scripts/check-pingora-health-patrol-env-switch.mjs",
"check:pingora-tls-cert-sync": "node scripts/check-pingora-tls-cert-sync.mjs",
"check:pingora-current-release-audit": "node scripts/check-pingora-current-release-audit.mjs",
"check:pingora-cutover-status-snapshot": "node scripts/check-pingora-cutover-status-snapshot.mjs",
"check:pingora-cutover-evidence-bundle": "node scripts/check-pingora-cutover-evidence-bundle.mjs",
+3 -1
View File
@@ -521,6 +521,7 @@ cp "${SCRIPT_DIR}/deploy/maintenance-off.sh" "${TARGET_DIR}/scripts/deploy/maint
cp "${SCRIPT_DIR}/deploy/pingora-direct-enable.sh" "${TARGET_DIR}/scripts/deploy/pingora-direct-enable.sh"
cp "${SCRIPT_DIR}/deploy/pingora-direct-rollback.sh" "${TARGET_DIR}/scripts/deploy/pingora-direct-rollback.sh"
cp "${SCRIPT_DIR}/deploy/pingora-health-patrol-env-switch.mjs" "${TARGET_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs"
cp "${SCRIPT_DIR}/deploy/pingora-tls-cert-sync.mjs" "${TARGET_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs"
cp "${SCRIPT_DIR}/deploy/jenkins-inbound-agent-start.sh" "${TARGET_DIR}/scripts/jenkins-inbound-agent-start.sh"
cp "${SCRIPT_DIR}/deploy/install-jenkins-inbound-agent.sh" "${TARGET_DIR}/scripts/install-jenkins-inbound-agent.sh"
cp "${SCRIPT_DIR}/deploy/jenkins-agent-reverse-tunnel.ps1" "${TARGET_DIR}/scripts/jenkins-agent-reverse-tunnel.ps1"
@@ -535,6 +536,7 @@ chmod +x \
"${TARGET_DIR}/scripts/deploy/pingora-direct-enable.sh" \
"${TARGET_DIR}/scripts/deploy/pingora-direct-rollback.sh" \
"${TARGET_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" \
"${TARGET_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" \
"${TARGET_DIR}/scripts/jenkins-inbound-agent-start.sh" \
"${TARGET_DIR}/scripts/install-jenkins-inbound-agent.sh"
@@ -578,7 +580,7 @@ cat >"${TARGET_DIR}/README.md" <<EOF
- \`migration-bootstrap-secret.txt\`:构建 \`spacetime_module.wasm\` 时注入的迁移引导密钥,仅用于创建首个迁移操作员;请作为敏感文件保存到 Jenkins Secret Text,授权完成后不要长期留在公开归档中。
- \`*.sha256\`:发布产物 checksum,用于部署前校验。
- \`release-manifest.json\`:发布版本、源码 commit 与产物清单。
- \`scripts/\`:维护模式脚本、数据库导入导出脚本、数据库 OSS 备份脚本、生产健康巡检脚本、Pingora release readiness 聚合门禁、直连启用 / 回退 / health patrol env 切换 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审 / 状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本、迁移授权脚本和 Jenkins inbound agent systemd 安装脚本。
- \`scripts/\`:维护模式脚本、数据库导入导出脚本、数据库 OSS 备份脚本、生产健康巡检脚本、Pingora release readiness 聚合门禁、直连启用 / 回退 / health patrol env 切换 / TLS 证书同步 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审 / 状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本、迁移授权脚本和 Jenkins inbound agent systemd 安装脚本。
- \`scripts/deploy/production-api-deploy.sh\`API Deploy 执行入口;同目录的 \`maintenance-on.sh\` / \`maintenance-off.sh\` 必须来自同一发布包。
- \`deploy/\`systemd、Nginx 和生产环境变量示例;\`deploy/nginx/genarrative-dev-http.conf\` 仅供无域名开发服初始化使用。
@@ -66,6 +66,11 @@ function assertScriptShape() {
'scripts/ops/pingora-cutover-evidence-bundle.mjs',
'current release 自审必须检查切换证据包脚本。',
);
assertIncludes(
content,
'scripts/deploy/pingora-tls-cert-sync.mjs',
'current release 自审必须检查 TLS 证书同步脚本。',
);
assertIncludes(
content,
'scripts/ops/pingora-current-release-audit.mjs',
@@ -154,6 +159,12 @@ function assertCompleteReleasePasses() {
if (!canaryLiveArtifact || canaryLiveArtifact.status !== 'OK') {
failures.push('完整 release 自审必须确认 canary live smoke 脚本存在。');
}
const tlsCertSyncArtifact = payload.artifacts.find(
(artifact) => artifact.path === 'scripts/deploy/pingora-tls-cert-sync.mjs',
);
if (!tlsCertSyncArtifact || tlsCertSyncArtifact.status !== 'OK') {
failures.push('完整 release 自审必须确认 TLS 证书同步脚本存在且可执行。');
}
}
function assertChecksumMismatchFails() {
@@ -453,6 +464,7 @@ function prepareReleaseRoot(releaseRoot, options) {
'scripts/deploy/pingora-direct-enable.sh',
'scripts/deploy/pingora-direct-rollback.sh',
'scripts/deploy/pingora-health-patrol-env-switch.mjs',
'scripts/deploy/pingora-tls-cert-sync.mjs',
'deploy/systemd/genarrative-pingora-gateway.service',
'deploy/systemd/genarrative-pingora-gateway-direct-entry.conf',
'deploy/nginx/snippets/genarrative-pingora-canary.conf',
@@ -476,6 +488,7 @@ function prepareReleaseRoot(releaseRoot, options) {
'scripts/deploy/pingora-direct-enable.sh',
'scripts/deploy/pingora-direct-rollback.sh',
'scripts/deploy/pingora-health-patrol-env-switch.mjs',
'scripts/deploy/pingora-tls-cert-sync.mjs',
]) {
chmodExecutable(path.join(releaseRoot, file));
}
+281
View File
@@ -0,0 +1,281 @@
#!/usr/bin/env node
import { spawnSync } from 'node:child_process';
import {
existsSync,
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
statSync,
symlinkSync,
writeFileSync,
} from 'node:fs';
import { tmpdir } from 'node:os';
import path from 'node:path';
const SCRIPT = 'scripts/deploy/pingora-tls-cert-sync.mjs';
const failures = [];
const tmpRoot = mkdtempSync(
path.join(tmpdir(), 'genarrative-pingora-tls-cert-sync-'),
);
try {
main();
} finally {
rmSync(tmpRoot, { recursive: true, force: true });
}
if (failures.length > 0) {
console.error('[check:pingora-tls-cert-sync] FAILED');
for (const failure of failures) {
console.error(`- ${failure}`);
}
process.exit(1);
}
console.log('[check:pingora-tls-cert-sync] OK');
function main() {
assertDryRunAcceptsCertbotLiveSymlinks();
assertApplyCopiesToPrivateTarget();
assertRejectsRelativeAndRootPaths();
assertRejectsControlCharactersBeforeApply();
assertRejectsSymlinkTargetDirectory();
assertRejectsSymlinkTargetFiles();
assertRejectsWorldReadableModes();
}
function assertDryRunAcceptsCertbotLiveSymlinks() {
const fixture = prepareFixture('dry-run-live-symlink');
const result = runSync(fixture, []);
assertStatus(result, 0, 'dry-run 应接受 Certbot live symlink 源。');
if (existsSync(fixture.targetCert) || existsSync(fixture.targetKey)) {
failures.push('dry-run 不应写入目标证书文件。');
}
const payload = parseJson(result.stdout, 'dry-run JSON 输出');
assertEqual(payload.mode, 'dry-run', 'dry-run JSON mode 应为 dry-run。');
assertEqual(
payload.files?.[0]?.source?.sourceIsSymlink,
true,
'dry-run JSON 应标记源证书来自 symlink。',
);
}
function assertApplyCopiesToPrivateTarget() {
const fixture = prepareFixture('apply-copy', { symlinkSources: false });
const result = runSync(fixture, ['--apply'], {
serviceUser: `#${process.getuid?.() ?? 0}`,
serviceGroup: `#${process.getgid?.() ?? 0}`,
});
assertStatus(result, 0, 'apply 应复制证书到目标目录。');
assertFileContent(fixture.targetCert, 'fixture fullchain\n');
assertFileContent(fixture.targetKey, 'fixture private key\n');
assertMode(fixture.targetDir, 0o750, '目标目录权限应为 0750。');
assertMode(fixture.targetCert, 0o640, '目标证书权限应为 0640。');
assertMode(fixture.targetKey, 0o640, '目标私钥权限应为 0640。');
}
function assertRejectsRelativeAndRootPaths() {
const fixture = prepareFixture('reject-paths');
const relative = runRaw([
'--source-cert-file',
'relative/fullchain.pem',
'--source-key-file',
fixture.sourceKey,
'--target-dir',
fixture.targetDir,
]);
assertNonZero(relative, '相对源证书路径必须失败。');
assertIncludes(
`${relative.stdout}\n${relative.stderr}`,
'--source-cert-file 必须是绝对路径',
'相对路径必须给出明确错误。',
);
const root = runRaw([
'--source-cert-file',
fixture.sourceCert,
'--source-key-file',
fixture.sourceKey,
'--target-dir',
'/',
]);
assertNonZero(root, 'target-dir 指向文件系统根目录必须失败。');
assertIncludes(
`${root.stdout}\n${root.stderr}`,
'--target-dir 不能是文件系统根目录',
'根目录目标必须给出明确错误。',
);
}
function assertRejectsControlCharactersBeforeApply() {
const fixture = prepareFixture('reject-control');
const result = runRaw([
'--source-cert-file',
`${fixture.sourceCert}\n--apply`,
'--source-key-file',
fixture.sourceKey,
'--target-dir',
fixture.targetDir,
]);
assertNonZero(result, '带换行的源证书路径必须失败。');
assertIncludes(
`${result.stdout}\n${result.stderr}`,
'--source-cert-file 不能包含换行或 NUL 字符',
'控制字符路径必须给出明确错误。',
);
if (existsSync(fixture.targetCert) || existsSync(fixture.targetKey)) {
failures.push('控制字符参数失败时不应写入目标证书。');
}
}
function assertRejectsSymlinkTargetDirectory() {
const fixture = prepareFixture('reject-target-dir-symlink');
const realTarget = path.join(fixture.root, 'real-target');
mkdirSync(realTarget, { recursive: true });
rmSync(fixture.targetDir, { recursive: true, force: true });
symlinkSync(realTarget, fixture.targetDir);
const result = runSync(fixture, []);
assertNonZero(result, 'target-dir 是 symlink 时必须失败。');
assertIncludes(
`${result.stdout}\n${result.stderr}`,
'--target-dir 已存在路径不能包含符号链接',
'target-dir symlink 必须给出明确错误。',
);
}
function assertRejectsSymlinkTargetFiles() {
const fixture = prepareFixture('reject-target-file-symlink');
mkdirSync(fixture.targetDir, { recursive: true });
const outside = path.join(fixture.root, 'outside-fullchain.pem');
writeFileSync(outside, 'outside\n', 'utf8');
symlinkSync(outside, fixture.targetCert);
const result = runSync(fixture, []);
assertNonZero(result, '目标证书是 symlink 时必须失败。');
assertIncludes(
`${result.stdout}\n${result.stderr}`,
'目标 fullchain.pem 不能是符号链接',
'目标文件 symlink 必须给出明确错误。',
);
}
function assertRejectsWorldReadableModes() {
const fixture = prepareFixture('reject-world-readable-mode');
const result = runSync(fixture, ['--file-mode', '0644']);
assertNonZero(result, 'world-readable file mode 必须失败。');
assertIncludes(
`${result.stdout}\n${result.stderr}`,
'--file-mode 不能允许 other 读取证书或私钥',
'world-readable file mode 必须给出明确错误。',
);
}
function prepareFixture(name, options = {}) {
const root = path.join(tmpRoot, name);
const archiveDir = path.join(root, 'archive');
const liveDir = path.join(root, 'live');
const targetDir = path.join(root, 'target');
mkdirSync(archiveDir, { recursive: true });
mkdirSync(liveDir, { recursive: true });
writeFileSync(path.join(archiveDir, 'fullchain1.pem'), 'fixture fullchain\n', 'utf8');
writeFileSync(path.join(archiveDir, 'privkey1.pem'), 'fixture private key\n', 'utf8');
const symlinkSources = options.symlinkSources !== false;
const sourceCert = symlinkSources
? path.join(liveDir, 'fullchain.pem')
: path.join(archiveDir, 'fullchain1.pem');
const sourceKey = symlinkSources
? path.join(liveDir, 'privkey.pem')
: path.join(archiveDir, 'privkey1.pem');
if (symlinkSources) {
symlinkSync('../archive/fullchain1.pem', sourceCert);
symlinkSync('../archive/privkey1.pem', sourceKey);
}
return {
root,
sourceCert,
sourceKey,
targetDir,
targetCert: path.join(targetDir, 'fullchain.pem'),
targetKey: path.join(targetDir, 'privkey.pem'),
};
}
function runSync(fixture, args = [], options = {}) {
return runRaw([
'--source-cert-file',
fixture.sourceCert,
'--source-key-file',
fixture.sourceKey,
'--target-dir',
fixture.targetDir,
'--service-user',
options.serviceUser || `#${process.getuid?.() ?? 0}`,
'--service-group',
options.serviceGroup || `#${process.getgid?.() ?? 0}`,
'--json',
...args,
]);
}
function runRaw(args) {
return spawnSync('node', [SCRIPT, ...args], {
cwd: process.cwd(),
encoding: 'utf8',
});
}
function parseJson(text, label) {
try {
return JSON.parse(text);
} catch (error) {
failures.push(`${label} 不是合法 JSON: ${error.message}\n${text}`);
return {};
}
}
function assertStatus(result, expected, reason) {
if ((result.status ?? 0) !== expected) {
failures.push(
`${reason} 实际退出码 ${result.status}\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`,
);
}
}
function assertNonZero(result, reason) {
if ((result.status ?? 0) === 0) {
failures.push(`${reason}\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`);
}
}
function assertIncludes(value, expected, reason) {
if (!String(value).includes(expected)) {
failures.push(`${reason} 缺少: ${expected}`);
}
}
function assertEqual(actual, expected, reason) {
if (actual !== expected) {
failures.push(`${reason} 实际 ${JSON.stringify(actual)},预期 ${JSON.stringify(expected)}`);
}
}
function assertFileContent(filePath, expected) {
if (!existsSync(filePath)) {
failures.push(`缺少文件: ${filePath}`);
return;
}
const actual = readFileSync(filePath, 'utf8');
if (actual !== expected) {
failures.push(`${filePath} 内容不符合预期。`);
}
}
function assertMode(filePath, expected, reason) {
const mode = statSync(filePath).mode & 0o777;
if (mode !== expected) {
failures.push(`${reason} 实际 0${mode.toString(8)},预期 0${expected.toString(8)}`);
}
}
+32
View File
@@ -62,6 +62,7 @@ function main() {
assertMissingHealthPatrolEnvCheckFails();
assertMissingPingoraReleaseReadinessFails();
assertMissingPingoraHealthPatrolEnvSwitchFails();
assertMissingPingoraTlsCertSyncFails();
assertMissingEnvExamplesFails();
assertMissingPingoraDirectCheckFails();
assertMissingPingoraCanaryLiveFails();
@@ -105,6 +106,10 @@ function assertDeployCopiesPingoraDirectReleaseDependencies() {
),
'current release 必须包含 Pingora health patrol env 切换脚本。',
);
assertFileExists(
path.join(releaseDir, 'scripts/deploy/pingora-tls-cert-sync.mjs'),
'current release 必须包含 Pingora TLS 证书同步脚本。',
);
assertFileExists(
path.join(releaseDir, 'scripts/check-pingora-direct-preflight.mjs'),
'current release 必须包含 Pingora 直连预检脚本。',
@@ -885,6 +890,29 @@ function assertMissingPingoraHealthPatrolEnvSwitchFails() {
}
}
function assertMissingPingoraTlsCertSyncFails() {
const fixture = prepareFixture('missing-pingora-tls-cert-sync');
rmSync(
path.join(
fixture.sourceDir,
'scripts/deploy/pingora-tls-cert-sync.mjs',
),
);
const result = runDeploy(fixture);
if (result.status === 0) {
failures.push('发布产物缺少 Pingora TLS 证书同步脚本时部署必须失败。');
}
assertIncludes(
result.stderr,
'发布产物缺少 Pingora TLS 证书同步脚本',
'缺少 Pingora TLS 证书同步脚本时必须给出明确错误。',
);
if (!existsSync(fixture.maintenanceFile)) {
failures.push('部署失败时必须保持维护模式。');
}
}
function assertMissingEnvExamplesFails() {
const fixture = prepareFixture('missing-env-examples');
rmSync(path.join(fixture.sourceDir, 'deploy/env'), {
@@ -1020,6 +1048,10 @@ function prepareFixture(name) {
'scripts/deploy/pingora-health-patrol-env-switch.mjs',
path.join(sourceDir, 'scripts/deploy/pingora-health-patrol-env-switch.mjs'),
);
copyFile(
'scripts/deploy/pingora-tls-cert-sync.mjs',
path.join(sourceDir, 'scripts/deploy/pingora-tls-cert-sync.mjs'),
);
copyFile(
'scripts/check-production-health-patrol-env.mjs',
path.join(sourceDir, 'scripts/check-production-health-patrol-env.mjs'),
+9 -1
View File
@@ -116,6 +116,14 @@ function assertApiReleaseContainsPingoraDirectDependencies() {
),
'Pingora health patrol env 切换脚本必须可执行。',
);
assertFileExists(
path.join(releaseDir, 'scripts/deploy/pingora-tls-cert-sync.mjs'),
'API release 必须包含 Pingora TLS 证书同步脚本。',
);
assertExecutable(
path.join(releaseDir, 'scripts/deploy/pingora-tls-cert-sync.mjs'),
'Pingora TLS 证书同步脚本必须可执行。',
);
assertFileExists(
path.join(releaseDir, 'scripts/check-production-health-patrol-env.mjs'),
'API release 必须包含健康巡检 env 复核脚本。',
@@ -246,7 +254,7 @@ function assertApiReleaseContainsPingoraDirectDependencies() {
);
assertIncludes(
releaseReadme,
'Pingora release readiness 聚合门禁、直连启用 / 回退 / health patrol env 切换 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审 / 状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本',
'Pingora release readiness 聚合门禁、直连启用 / 回退 / health patrol env 切换 / TLS 证书同步 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审 / 状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本',
'API release README 必须说明随包携带 Pingora 直连脚本。',
);
assertIncludes(
+470
View File
@@ -0,0 +1,470 @@
#!/usr/bin/env node
import { randomBytes } from 'node:crypto';
import {
accessSync,
chmodSync,
chownSync,
constants,
copyFileSync,
existsSync,
lstatSync,
mkdirSync,
realpathSync,
renameSync,
rmSync,
statSync,
} from 'node:fs';
import { execFileSync, spawnSync } from 'node:child_process';
import path from 'node:path';
import { userInfo } from 'node:os';
const DEFAULT_SERVICE_USER = 'genarrative';
const DEFAULT_SERVICE_GROUP = 'genarrative';
const DEFAULT_DIRECTORY_MODE = '0750';
const DEFAULT_FILE_MODE = '0640';
const config = parseArgs(process.argv.slice(2));
const result = run();
if (config.json) {
console.log(`${JSON.stringify(result, null, 2)}\n`);
} else {
const mode = config.apply ? 'apply' : 'dry-run';
console.log(`[pingora-tls-cert-sync] ${mode} OK`);
console.log(`[pingora-tls-cert-sync] target=${result.target.directory}`);
for (const item of result.files) {
console.log(
`[pingora-tls-cert-sync] ${item.kind}: ${item.source.path} -> ${item.target.path}`,
);
}
}
function usage() {
console.log(`Usage:
node scripts/deploy/pingora-tls-cert-sync.mjs --source-cert-file <path> --source-key-file <path> --target-dir <path> [options]
Options:
--source-cert-file <path> 已存在的证书链文件,通常是 Let's Encrypt live/fullchain.pem。
--source-key-file <path> 已存在的私钥文件,通常是 Let's Encrypt live/privkey.pem。
--target-dir <path> Pingora 私有 TLS 目录,例如 /etc/genarrative/pingora-tls/<host>。
--service-user <user|uid> 目标服务用户,默认 genarrative。
--service-group <group|gid>
目标服务组,默认 genarrative。
--directory-mode <mode> 目标目录权限,默认 0750。
--file-mode <mode> 证书和私钥权限,默认 0640。
--apply 执行复制;默认只做 dry-run 校验。
--json 输出 JSON。
该脚本只把现有证书复制到 Pingora 私有目录,不修改 Let's Encrypt / Certbot / Nginx 原始文件。
源路径可以是 Certbot live symlink,但解析后的目标必须是普通文件;目标目录和目标文件不允许是 symlink。
`);
}
function parseArgs(argv) {
const result = {
sourceCertFile:
process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_SOURCE_CERT_FILE || '',
sourceKeyFile:
process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_SOURCE_KEY_FILE || '',
targetDir: process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_TARGET_DIR || '',
serviceUser:
process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_SERVICE_USER ||
DEFAULT_SERVICE_USER,
serviceGroup:
process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_SERVICE_GROUP ||
DEFAULT_SERVICE_GROUP,
directoryMode:
process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_DIRECTORY_MODE ||
DEFAULT_DIRECTORY_MODE,
fileMode:
process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_FILE_MODE ||
DEFAULT_FILE_MODE,
apply: readBoolEnv(
process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_APPLY,
false,
'GENARRATIVE_PINGORA_TLS_CERT_SYNC_APPLY',
),
json: false,
};
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
switch (arg) {
case '-h':
case '--help':
usage();
process.exit(0);
break;
case '--source-cert-file':
result.sourceCertFile = requireValue(argv, ++index, arg);
break;
case '--source-key-file':
result.sourceKeyFile = requireValue(argv, ++index, arg);
break;
case '--target-dir':
result.targetDir = requireValue(argv, ++index, arg);
break;
case '--service-user':
result.serviceUser = requireValue(argv, ++index, arg);
break;
case '--service-group':
result.serviceGroup = requireValue(argv, ++index, arg);
break;
case '--directory-mode':
result.directoryMode = requireValue(argv, ++index, arg);
break;
case '--file-mode':
result.fileMode = requireValue(argv, ++index, arg);
break;
case '--apply':
result.apply = true;
break;
case '--json':
result.json = true;
break;
default:
throw new Error(`未知参数: ${arg}`);
}
}
validateSafeAbsoluteFilePath(
result.sourceCertFile,
'--source-cert-file',
);
validateSafeAbsoluteFilePath(result.sourceKeyFile, '--source-key-file');
validateSafeAbsoluteDirectoryPath(result.targetDir, '--target-dir');
validateIdentity(result.serviceUser, '--service-user');
validateIdentity(result.serviceGroup, '--service-group');
result.directoryModeNumber = parseMode(
result.directoryMode,
'--directory-mode',
);
result.fileModeNumber = parseMode(result.fileMode, '--file-mode');
if ((result.fileModeNumber & 0o007) !== 0) {
throw new Error('--file-mode 不能允许 other 读取证书或私钥。');
}
if ((result.directoryModeNumber & 0o007) !== 0) {
throw new Error('--directory-mode 不能允许 other 进入 TLS 私有目录。');
}
return result;
}
function requireValue(argv, index, flag) {
const value = argv[index];
if (!value || value.startsWith('--')) {
throw new Error(`${flag} 缺少参数值`);
}
return value;
}
function readBoolEnv(raw, fallback, label) {
if (raw === undefined || raw === null || String(raw).trim() === '') {
return fallback;
}
validateNoControlCharacters(raw, label);
const normalized = String(raw).trim().toLowerCase();
if (['1', 'true', 'yes', 'on'].includes(normalized)) {
return true;
}
if (['0', 'false', 'no', 'off'].includes(normalized)) {
return false;
}
throw new Error(`${label} 必须是布尔值 true/false 或 1/0。`);
}
function validateSafeAbsoluteFilePath(value, label) {
validateNoControlCharacters(value, label);
if (!path.isAbsolute(value)) {
throw new Error(`${label} 必须是绝对路径。`);
}
if (isFilesystemRootPath(value)) {
throw new Error(`${label} 不能是文件系统根目录。`);
}
}
function validateSafeAbsoluteDirectoryPath(value, label) {
validateSafeAbsoluteFilePath(value, label);
if (path.basename(path.resolve(value)) === '..') {
throw new Error(`${label} 不能以 .. 结尾。`);
}
}
function validateNoControlCharacters(value, label) {
if (/[\0\r\n]/u.test(String(value))) {
throw new Error(`${label} 不能包含换行或 NUL 字符。`);
}
}
function validateIdentity(value, label) {
validateNoControlCharacters(value, label);
if (!/^[A-Za-z0-9_.#-]+$/u.test(String(value))) {
throw new Error(`${label} 只能包含字母、数字、点、下划线、短横线或 #。`);
}
}
function parseMode(value, label) {
validateNoControlCharacters(value, label);
const text = String(value).trim();
if (!/^0?[0-7]{3}$/u.test(text)) {
throw new Error(`${label} 必须是三位八进制权限,例如 0640。`);
}
return Number.parseInt(text, 8);
}
function isFilesystemRootPath(value) {
const resolved = path.resolve(String(value));
return resolved === path.parse(resolved).root;
}
function run() {
const certSource = inspectSourceFile(
config.sourceCertFile,
'--source-cert-file',
);
const keySource = inspectSourceFile(config.sourceKeyFile, '--source-key-file');
const certTarget = path.join(config.targetDir, 'fullchain.pem');
const keyTarget = path.join(config.targetDir, 'privkey.pem');
const owner = config.apply
? resolveOwner(config.serviceUser, config.serviceGroup)
: { uid: null, gid: null };
assertNoSymlinkAncestors(config.targetDir, '--target-dir');
assertTargetPath(certTarget, '目标 fullchain.pem');
assertTargetPath(keyTarget, '目标 privkey.pem');
if (existsSync(config.targetDir)) {
assertTargetDirectory(config.targetDir);
}
const planned = {
mode: config.apply ? 'apply' : 'dry-run',
target: {
directory: config.targetDir,
user: config.serviceUser,
group: config.serviceGroup,
uid: owner.uid,
gid: owner.gid,
directoryMode: formatMode(config.directoryModeNumber),
fileMode: formatMode(config.fileModeNumber),
},
files: [
{
kind: 'cert',
source: certSource,
target: { path: certTarget },
},
{
kind: 'key',
source: keySource,
target: { path: keyTarget },
},
],
serviceUserReadable: false,
};
if (!config.apply) {
return planned;
}
mkdirSync(config.targetDir, { recursive: true });
assertTargetDirectory(config.targetDir);
chmodSync(config.targetDir, config.directoryModeNumber);
applyOwner(config.targetDir, owner);
copyAtomic(certSource.resolvedPath, certTarget, owner);
copyAtomic(keySource.resolvedPath, keyTarget, owner);
planned.serviceUserReadable = assertServiceUserReadable(
config.serviceUser,
[certTarget, keyTarget],
);
return planned;
}
function inspectSourceFile(filePath, label) {
const linkStat = lstatSync(filePath);
const sourceIsSymlink = linkStat.isSymbolicLink();
const resolvedPath = realpathSync(filePath);
const fileStat = statSync(resolvedPath);
if (!fileStat.isFile()) {
throw new Error(`${label} 解析后必须是普通文件: ${resolvedPath}`);
}
return {
path: filePath,
resolvedPath,
sourceIsSymlink,
sizeBytes: fileStat.size,
};
}
function assertTargetDirectory(directory) {
const linkStat = lstatSync(directory);
if (linkStat.isSymbolicLink()) {
throw new Error(`--target-dir 不能是符号链接: ${directory}`);
}
if (!linkStat.isDirectory()) {
throw new Error(`--target-dir 已存在但不是目录: ${directory}`);
}
}
function assertNoSymlinkAncestors(targetDir, label) {
const resolved = path.resolve(targetDir);
const parsed = path.parse(resolved);
let current = parsed.root;
for (const part of resolved.slice(parsed.root.length).split(path.sep)) {
if (!part) {
continue;
}
current = path.join(current, part);
if (!existsSync(current)) {
break;
}
const linkStat = lstatSync(current);
if (linkStat.isSymbolicLink()) {
throw new Error(`${label} 已存在路径不能包含符号链接: ${current}`);
}
if (!linkStat.isDirectory()) {
throw new Error(`${label} 已存在父路径不是目录: ${current}`);
}
}
}
function assertTargetPath(filePath, label) {
if (!existsSync(filePath)) {
return;
}
const linkStat = lstatSync(filePath);
if (linkStat.isSymbolicLink()) {
throw new Error(`${label} 不能是符号链接: ${filePath}`);
}
if (!linkStat.isFile()) {
throw new Error(`${label} 已存在但不是普通文件: ${filePath}`);
}
}
function resolveOwner(user, group) {
const current = currentIdentity();
const uid = resolveUserId(user, current);
const gid = resolveGroupId(group, current, user);
if (!isCurrentUserRoot()) {
if (uid !== current.uid || gid !== current.gid) {
throw new Error(
`当前用户不是 root,不能把证书授权给 ${user}:${group};请用 sudo -n 执行 --apply。`,
);
}
}
return { uid, gid };
}
function resolveUserId(user, current) {
if (/^#?\d+$/u.test(user)) {
return Number.parseInt(user.replace(/^#/u, ''), 10);
}
if (user === current.username) {
return current.uid;
}
return Number.parseInt(
execFileSync('id', ['-u', user], { encoding: 'utf8' }).trim(),
10,
);
}
function resolveGroupId(group, current, user) {
if (/^#?\d+$/u.test(group)) {
return Number.parseInt(group.replace(/^#/u, ''), 10);
}
if (group === current.groupname || group === String(current.gid)) {
return current.gid;
}
const getent = spawnSync('getent', ['group', group], { encoding: 'utf8' });
if (getent.status === 0 && getent.stdout.trim()) {
const parts = getent.stdout.trim().split(':');
return Number.parseInt(parts[2], 10);
}
if (group === user) {
return Number.parseInt(
execFileSync('id', ['-g', user], { encoding: 'utf8' }).trim(),
10,
);
}
throw new Error(`无法解析服务组: ${group}`);
}
function currentIdentity() {
const uid = typeof process.getuid === 'function' ? process.getuid() : -1;
const gid = typeof process.getgid === 'function' ? process.getgid() : -1;
let username = '';
try {
username = userInfo().username;
} catch {
username = process.env.USER || process.env.LOGNAME || '';
}
return {
uid,
gid,
username,
groupname: '',
};
}
function isCurrentUserRoot() {
return typeof process.getuid === 'function' && process.getuid() === 0;
}
function applyOwner(filePath, owner) {
const current = currentIdentity();
if (!isCurrentUserRoot() && owner.uid === current.uid && owner.gid === current.gid) {
return;
}
chownSync(filePath, owner.uid, owner.gid);
}
function copyAtomic(sourcePath, targetPath, owner) {
const tempPath = path.join(
path.dirname(targetPath),
`.${path.basename(targetPath)}.tmp-${process.pid}-${randomBytes(4).toString('hex')}`,
);
try {
copyFileSync(sourcePath, tempPath, constants.COPYFILE_EXCL);
chmodSync(tempPath, config.fileModeNumber);
applyOwner(tempPath, owner);
renameSync(tempPath, targetPath);
chmodSync(targetPath, config.fileModeNumber);
applyOwner(targetPath, owner);
} catch (error) {
rmSync(tempPath, { force: true });
throw error;
}
}
function assertServiceUserReadable(serviceUser, files) {
if (/^#?\d+$/u.test(serviceUser)) {
for (const file of files) {
accessSync(file, constants.R_OK);
}
return true;
}
const current = currentIdentity();
if (serviceUser === current.username && !isCurrentUserRoot()) {
for (const file of files) {
accessSync(file, constants.R_OK);
}
return true;
}
for (const file of files) {
const check = spawnSync('sudo', ['-n', '-u', serviceUser, 'test', '-r', file], {
encoding: 'utf8',
});
if (check.status !== 0) {
const detail = (check.stderr || check.stdout || '').trim();
throw new Error(
`服务用户 ${serviceUser} 不可读: ${file}${detail ? ` (${detail})` : ''}`,
);
}
}
return true;
}
function formatMode(mode) {
return `0${mode.toString(8).padStart(3, '0')}`;
}
+7
View File
@@ -620,6 +620,7 @@ PINGORA_RELEASE_READINESS_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-rel
PINGORA_ENABLE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-direct-enable.sh"
PINGORA_ROLLBACK_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-direct-rollback.sh"
PINGORA_HEALTH_PATROL_ENV_SWITCH_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs"
PINGORA_TLS_CERT_SYNC_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs"
PINGORA_PREFLIGHT_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-direct-preflight.mjs"
PINGORA_LIVE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-direct-live.mjs"
PINGORA_CANARY_LIVE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-canary-live.mjs"
@@ -707,6 +708,12 @@ if [[ ! -f "${PINGORA_HEALTH_PATROL_ENV_SWITCH_SCRIPT_SOURCE}" ]]; then
fi
cp "${PINGORA_HEALTH_PATROL_ENV_SWITCH_SCRIPT_SOURCE}" "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs"
chmod 0755 "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs"
if [[ ! -f "${PINGORA_TLS_CERT_SYNC_SCRIPT_SOURCE}" ]]; then
echo "[production-api-deploy] 发布产物缺少 Pingora TLS 证书同步脚本: ${SOURCE_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" >&2
exit 1
fi
cp "${PINGORA_TLS_CERT_SYNC_SCRIPT_SOURCE}" "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs"
chmod 0755 "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs"
if [[ ! -f "${PINGORA_PREFLIGHT_SCRIPT_SOURCE}" ]]; then
echo "[production-api-deploy] 发布产物缺少 Pingora 直连预检脚本: ${SOURCE_DIR}/scripts/check-pingora-direct-preflight.mjs" >&2
exit 1
@@ -22,6 +22,7 @@ const REQUIRED_ARTIFACTS = [
{ path: 'scripts/deploy/pingora-direct-enable.sh', executable: true },
{ path: 'scripts/deploy/pingora-direct-rollback.sh', executable: true },
{ path: 'scripts/deploy/pingora-health-patrol-env-switch.mjs', executable: true },
{ path: 'scripts/deploy/pingora-tls-cert-sync.mjs', executable: true },
{ path: 'deploy/systemd/genarrative-pingora-gateway.service' },
{ path: 'deploy/systemd/genarrative-pingora-gateway-direct-entry.conf' },
{ path: 'deploy/nginx/snippets/genarrative-pingora-canary.conf' },