deploy_cutover.py
6.53 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
"""c_deploy_supervisor_cutover (03-C §3.15 / 04-P §4.6 / design doc §4 step
5): the blue-green switch itself -- old Live stops, the candidate (already
staged by U-037, already health-checked by U-038) takes over the production
port and database.
**`LIVE_PORT` (new MVP-fixed parameter, found missing while building this)**:
05 §5.9 lists every OTHER Deploy Supervisor timing/threshold constant but
never actually names a fixed production port -- reasonable, since the
console's own frontend/app.js only ever makes same-origin relative fetch()
calls (whatever port serves index.html is naturally used for /api/... too),
so nothing in the running system needed one internally. Cutover is the one
place that genuinely does: successive deployments need to keep landing on
the SAME port so a human bookmark/shortcut to `http://127.0.0.1:LIVE_PORT`
keeps working across self-modifications. Same "MVP-fixed draft value, not
user-configurable" treatment as every other constant in that table.
**Rebind strategy (documented judgment call, not escalated -- stdlib
constraint, not an SDD ambiguity)**: `http.server` has no live "rebind this
already-listening socket to a different port/DB" operation. "候选切换到生产
端口/生产库" is implemented as stop-candidate-then-relaunch-from-the-same-
already-verified checkout directory bound to LIVE_PORT/PROD_DB_PATH, rather
than an in-place socket rebind -- functionally identical from the outside
(same code now serving the production port/db) and the only mechanism
stdlib's `http.server` actually offers.
**REQ-06 breakpoint recovery is relied on, not reimplemented here**: any
`work_node_execution` in flight when Live stops resumes correctly on restart
via the existing `resume_work_graph_run`/`run_work_node` machinery (U-013/
U-021) -- this module does nothing special for "in-flight work", it just
stops one process and starts another, which REQ-06 was built to already
survive regardless of what caused the interruption.
**Known adjacent gap, out of scope here**: `backend_server.py` doesn't
currently invoke `scheduler.run_scheduler_once()` anywhere -- there's no
in-process background dispatch loop running inside the Live server today,
so "在途工作" is presently more theoretical than real. Not this Unit's job
to fix (cutover's contract is the process swap, not the scheduler wiring
gap); flagged so a future Unit doesn't rediscover it from scratch.
"""
import os
import signal
import time
from supervisor import deploy_health_check, deploy_stage, deploy_supervisor
from src import paths
LIVE_PORT = 8420
class CutoverFailedError(Exception):
"""CUTOVER_FAILED -- 03-C §3.15: the restarted candidate failed to bind
the production port/db during cutover. Not retried -- the caller routes
this to `deployment.status='blocked_pending_human'`, already applied by
the time this is raised."""
def _stop_process(pid, timeout=10):
"""Uses `reap_if_exited`, not plain `is_alive`, throughout -- a process
this module itself launched via `deploy_stage._launch_candidate`
(subprocess.Popen) needs to actually be reaped for `is_alive()` to ever
report it as gone; found for real (2026-08-02) via a test that hung
forever otherwise (see `deploy_supervisor.is_alive`'s own docstring)."""
if pid is None or deploy_supervisor.reap_if_exited(pid):
return
os.kill(pid, signal.SIGTERM)
deadline = time.time() + timeout
while time.time() < deadline:
if deploy_supervisor.reap_if_exited(pid):
return
time.sleep(0.1)
os.kill(pid, signal.SIGKILL)
deploy_supervisor.reap_if_exited(pid)
def _wait_for_live_http(port, timeout, sleep=time.sleep, now=time.time):
deadline = now() + timeout
while now() < deadline:
if deploy_health_check.check_http_health(port, timeout=2)["status"] == "ok":
return True
sleep(0.2)
return False
def cutover(
conn, deployment_id, live_pid, candidate_pid, candidate_instance_dir,
prod_db_path=None, live_port=LIVE_PORT, python_executable=None,
stop_timeout=10, bind_timeout=15, sleep=time.sleep, now=time.time,
):
"""Only callable after `c_deploy_supervisor_health_check` returned
`health_check_passed` for `deployment_id` (03-C's own precondition;
not re-checked here -- the caller, Deploy Supervisor's own poll loop,
is what enforces the state-machine ordering, same layering
`c_deploy_supervisor_stage` uses).
`live_pid` may be None (nothing was Live yet -- the very first
deployment). Returns {status: 'live', live_pid, live_port, cutover_at}.
Raises CutoverFailedError (deployment already marked
blocked_pending_human) if the restarted candidate never binds
`live_port`.
"""
prod_db_path = prod_db_path or paths.db_path()
conn.execute("UPDATE deployment SET status = 'cutover_in_progress' WHERE deployment_id = ?", (deployment_id,))
conn.commit()
try:
# Live stops accepting new triggers by simply stopping -- in-flight
# work_node_execution resumes via REQ-06 machinery on next dispatch,
# not this module's concern (see module docstring).
_stop_process(live_pid, timeout=stop_timeout)
# The health-check-mode candidate (bound to candidate_port/
# candidate_db_path) is retired too -- its job was verification,
# not serving traffic.
_stop_process(candidate_pid, timeout=stop_timeout)
new_live_process = deploy_stage._launch_candidate(
candidate_instance_dir, live_port, prod_db_path, python_executable
)
if not _wait_for_live_http(live_port, bind_timeout, sleep=sleep, now=now):
raise CutoverFailedError(
"candidate for deployment {} never bound live_port {} within {}s".format(
deployment_id, live_port, bind_timeout
)
)
except CutoverFailedError:
conn.execute(
"UPDATE deployment SET status = 'blocked_pending_human' WHERE deployment_id = ?", (deployment_id,)
)
conn.commit()
raise
except Exception as e:
conn.execute(
"UPDATE deployment SET status = 'blocked_pending_human' WHERE deployment_id = ?", (deployment_id,)
)
conn.commit()
raise CutoverFailedError(
"cutover failed for deployment {}: {}".format(deployment_id, e)
) from e
conn.execute(
"UPDATE deployment SET status = 'live', cutover_at = CURRENT_TIMESTAMP WHERE deployment_id = ?",
(deployment_id,),
)
conn.commit()
return {"status": "live", "live_pid": new_live_process.pid, "live_port": live_port}