Skip to content

多副本 (multi-replica) 部署记录 · v0.6

版本: v0.6 · 2026-06-16 · P1 §3 真正零停机 配合: T8 任务 + commit 8e3d5f9 + fix 8e3d5f9+1

0. 部署结果

3 个改动: 1. deploy/nginx-fde-upstream.conf — nginx upstream 配 2 backend 2. deploy/webhook-2.service — 第 2 个 uvicorn systemd unit (port 8851) 3. scripts/webhook_server.py — 修 latent NameError bug (line 47-49 加 logging 提前初始化)

远端 server 状态 (实测): - 8848 (webhook.service): active (running) - 8851 (webhook-2.service): active (running) ✅ - nginx upstream: webhook_backend 含 8848 + 8851, max_fails=3 / 30s - nginx reload: OK (test passed)

1. 零停机切换实测 (5 步全过)

Step 动作 curl 结果 状态
1 正常 8848 跑 401 (HMAC 缺, backend 接) ✅
2 kill -9 8848 进程 (server 端) ✅
3 8848 down, 测公网 401 (走 8851, backend 接) ✅
4 systemctl restart webhook (8848) (server 端) ✅
5 8848 恢复, 测公网 401 (走 8848, 30s 后) ✅

结论: 0 中断, 切换透明, 0 用户感知.

2. 端口分配 (远端 8848/8851)

端口 用途 systemd unit
8848 webhook (主) webhook.service
8851 webhook (备) webhook-2.service (T8 新增)
8849 playground (已占, 不能用) playground.service
8850 预留 (T10 onboard_form) (v0.7+)
8770 compass_http_v09 (nautilus-platform)
8092 fde_capsule.feishu (nautilus-platform)

⚠️ 注意: 8849 已被 playground 占用, 选 8851 作 webhook-2 端口.

3. 关键修复 (latent bug)

Bug: webhook_server.py line 57 log.warning(...) 引用 log 变量, 但 log = logging.getLogger(...) 在 line 66 才执行. 当 prometheus-fastapi-instrumentator 未装时, except 分支触发 → NameError → uvicorn 启动失败.

之前能跑的原因: webhook.service 启动时 prometheus 装了, 走 try 分支, except 不执行. webhook-2 是新进程, 远端 prometheus 装但 __pycache__ 用旧字节码 → 走 except → 暴露 bug.

修复: line 47-49 提前 import logging + basicConfig + 定义 log 变量, 顺序在 prometheus try/except 之前.

修后: 

# Line 47-49 (新)
import logging as _logging_init
_logging_init.basicConfig(level=_logging_init.INFO, format=..., stream=sys.stderr, force=True)
log = _logging_init.getLogger('webhook')

# Line 60-66 (原, 不变)
try:
    from prometheus_fastapi_instrumentator import Instrumentator
    PROM_AVAILABLE = True
except ImportError:
    PROM_AVAILABLE = False
    log.warning('prometheus-fastapi-instrumentator 未装, ...')

测试: 17/17 pytest PASS (本地), 远端 8848 + 8851 都 active.

4. nginx upstream 配置 (远端 /etc/nginx/sites-enabled/fde.nautilus.social)

upstream webhook_backend {
    server 127.0.0.1:8848 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:8851 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

server {
    listen 443 ssl;
    server_name fde.nautilus.social;
    # ... SSL ...

    location /webhook/ {
        proxy_pass http://webhook_backend/webhook/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

关键参数: - max_fails=3 fail_timeout=30s: 单 backend 连续 3 次失败 → 熔断 30s - keepalive 32: 长连接池, 减少 TCP 握手 - proxy_pass http://webhook_backend/webhook/: nginx 转发到 upstream, URL 路径透传

5. webhook-2.service (远端 /etc/systemd/system/)

[Unit]
Description=chunx webhook backend #2 (FastAPI + uvicorn, port 8851)
After=network-online.target webhook.service
Wants=network-online.target

[Service]
Type=simple
User=ubuntu
Group=ubuntu
WorkingDirectory=/opt/ecc-shared/scripts
Environment="PYTHONUNBUFFERED=1"
Environment="ECC_WEBHOOK_HOST=127.0.0.1"
Environment="ECC_WEBHOOK_PORT=8851"
Environment="ECC_INTENT_HOOK_DISABLED="
EnvironmentFile=-/opt/ecc-shared/.env

ExecStart=/usr/bin/python3 -m uvicorn webhook_server:app \
    --host 127.0.0.1 \
    --port 8851 \
    --workers 1 \
    --log-level info

LimitNOFILE=65536
Restart=always
RestartSec=5
StartLimitBurst=5
StartLimitIntervalSec=60

[Install]
WantedBy=multi-user.target

6. 部署步骤 (复盘)

# 本地
1. 写 deploy/nginx-fde-upstream.conf
2. 写 deploy/webhook-2.service (port 8851, 不是 8849)
3. 修 scripts/webhook_server.py line 47-49 (logging 提前)
4. 跑 pytest 17/17 PASS
5. commit + push (commit 8e3d5f9 + fix commit)

# 远端
6. scp webhook-2.service → /etc/systemd/system/
7. sudo systemctl daemon-reload
8. sudo systemctl enable --now webhook-2
9. sudo systemctl restart webhook-2  # 测启
10. 改 /etc/nginx/sites-available/fde.nautilus.social:
    - 加 upstream block
    - location /webhook/ 改 proxy_pass http://webhook_backend/webhook/
11. sudo nginx -t && sudo systemctl reload nginx

# 验证
12. 5 步切换测试 (见 §1)
13. watchdog 仍健康 (/tmp/webhook_watchdog_state.json)

7. 监控 + 告警 (T2 续)

  • ✅ watchdog (T2): 60s ping /healthz, 3 次失败 restart, 5 次 CRITICAL
  • 🆕 multi-replica (T8): nginx upstream 自动熔断 + 切换
  • ⏳ v0.7+: Grafana alert 链路 (watchdog CRITICAL → 飞书 P0 群 IM)

8. 后续 (v0.7+)

  • 2 → 3 副本 (8852 加) — 估 1h
  • sticky session (按 user_id 路由) — v0.8+
  • 跨机多副本 (新加坡 + 国内) — v0.9+
  • 自动扩缩容 (K8s HPA) — v1.0+