kdletters
a7337c67a1
修复 release 内存持续增长问题。 本次范围： - 收口备份扫描与历史维护的内存峰值。 - 限制外部生成 worker 脱管任务的实际并发。 - 为 API 内存态和历史数据增加有界留存。 验收： - 定向测试、cargo 检查和生产运维门禁通过。 - 备份不再触发全局 OOM。 - worker/API/SpacetimeDB 内存曲线在空闲期停止单调增长。 Reviewed-on: http://192.168.35.82/git/GenarrativeAI/Genarrative/pulls/203 Co-authored-by: kdletters <kdletters@qq.com> Co-committed-by: kdletters <kdletters@qq.com>
30 lines
2.1 KiB
Desktop File
30 lines
2.1 KiB
Desktop File
[Unit]
|
|
Description=Genarrative SpacetimeDB OSS Backup
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
User=root
|
|
Group=root
|
|
WorkingDirectory=/opt/genarrative/current
|
|
EnvironmentFile=/etc/genarrative/api-server.env
|
|
ExecStart=/usr/bin/node -- /opt/genarrative/current/scripts/database-backup-to-oss.mjs --env-file /etc/genarrative/api-server.env --stop-service spacetimedb.service --restart-service-after genarrative-api.service --restart-service-after genarrative-external-generation-worker@1.service --restart-service-after genarrative-external-generation-controller.service
|
|
|
|
# 备份脚本必须受独立内存上限保护，不能因目录扫描异常拖垮整台 release 主机。
|
|
Environment=NODE_OPTIONS=--max-old-space-size=768
|
|
Environment=GENARRATIVE_DATABASE_BACKUP_STOP_MARKER=/var/lib/genarrative/database-backups/.spacetimedb-stopped
|
|
MemoryHigh=768M
|
|
MemoryMax=1G
|
|
OOMPolicy=stop
|
|
|
|
# 主进程可能在停库后被 MemoryMax/OOMPolicy 强制终止，JS finally 无法执行；
|
|
# 仅当备份脚本留下停库 marker 且本次 service 非正常成功时，由 systemd 兜底恢复全部依赖服务。
|
|
ExecStopPost=/bin/sh -c 'if [ "${SERVICE_RESULT}" != "success" ] && [ -f "${GENARRATIVE_DATABASE_BACKUP_STOP_MARKER}" ]; then systemctl start spacetimedb.service; systemctl restart genarrative-api.service; systemctl restart genarrative-external-generation-worker@1.service; systemctl restart genarrative-external-generation-controller.service; if systemctl is-active --quiet spacetimedb.service && systemctl is-active --quiet genarrative-api.service && systemctl is-active --quiet genarrative-external-generation-worker@1.service && systemctl is-active --quiet genarrative-external-generation-controller.service; then rm -f "${GENARRATIVE_DATABASE_BACKUP_STOP_MARKER}"; fi; fi'
|
|
|
|
# 备份需要停止 / 启动 spacetimedb.service，并读取 /stdb、写入 /var/lib/genarrative/database-backups。
|
|
# 停止 SpacetimeDB 会连带停止 Requires 它的 API / worker / controller，冷备份后必须显式拉起。
|
|
PrivateTmp=true
|
|
ProtectSystem=full
|
|
ReadWritePaths=/stdb /var/lib/genarrative
|