fix: preserve safe stop and explicit forced interruption lifecycle

This commit is contained in:
阿白叔 committed 2026-10-26 16:39:21 +08:00
1 parent b939101565
commit 9c03cd5900
29 files changed
+1829 -86

No files matched your search

@@ -0,0 +1,62 @@
# Safe User Stop Implementation Plan
> **For agentic workers:** use superpowers:subagent-driven-development or executing-plans; continue within the user-approved scope without another approval round.
**Goal:** 用户停止只在安全边界收尾;写入中的无进度不自动杀进程;明确强制中断有保护证据和恢复入口。
**Architecture:** 保留现有 Event/队列边界和独立监督进程,加入独立的强制停止意图、等待状态反馈;退出正常性与物理资源复核分开。使用既有任务保护与批次记录,不新增数据库版本。
**Tech Stack:** Python 3.12、Windows pywin32、PySide6、VBScript、pytest。
**Spec:** `docs/superpowers/specs/2026-10-26-safe-user-stop.md`
## Global Constraints
- 不修改原工程、中心库或未跟踪 feedback;真实测试只复制 fixture。
- 不放宽身份、写前保护、禁止不确定重放的门禁。
- 普通停止与强制结束独立,确认前不得产生强制意图。
- 不通过无限创建桌面或删除旧保护绕过异常。
- 不发布、不推送;本地分支 `codex/day0-safe-stop`。
## Task 1: 退出协议与后代进程
Files: `isolated_cad.py`, `isolated_desktop.py`, `tests/v1_1/test_isolated_cad.py`, `test_isolated_desktop.py`.
- [x] 先复现 closing 记录失败跳过 Quit、父进程自然退出后仍立即终止子进程的测试。
- [x] 诊断写入失败不跳过满足身份/空工程条件的 Quit;将记录失败独立保留并报告,禁止伪报 clean。
- [x] 根进程正常退出时,给 Job 后代 2 秒有界自然退出窗口;到期再强制并记录 `forced_termination`, `termination_reason`, `forced_descendant_pids`。根进程仍活着的异常清理仍有界。
- [x] 保留所有权和退出确认,测试自然退出与强制升级两条路径。
## Task 2: 停止、等待与强制中断
Files: new `stop_control.py`; `worker_monitor.py`, `project_session.py`, `batch_runtime.py`, `batch_supervisor.py`, `execution_feedback.py`; associated tests.
Interface:
```python
class StopControl(Event):
force_event: Event
def request_force(self):
self.set()
self.force_event.set()
```
- [x] 首先测试 unsafe 保存阶段超时仍保留 worker,安全阶段仍有 watchdog,显式 force 才结束并产生 uncertain。
- [x] `ProgressWatch` 提供 `waiting_for_safe_completion`;新进度可清除提示,不把心跳当作写入推进。
- [x] 监督父进程持久化停止意图;force 先写保护与中断记录,再只终止已拥有的监督进程 Job,不使用名字或不可信 PID 清理。
- [x] 业务完成但退出异常仍通过独立错误返回,附带已核验业务结果;纯桌面延迟且退出正常可继续使用现有复核恢复。
- [x] 普通停止在当前项收尾后暂停;会话 close 不得因普通等待超时反向强杀仍在写入的 worker。
## Task 3: 保存边界与界面
Files: `QuickPartReplace.vbs`, `ui/run_panel.py`, `ui/main_window.py`, `ui/execution_feedback.py`, protocol serializer if required; VBS/UI tests.
- [x] VBS 用真实函数的失败先行测试覆盖保存已发出但工程关闭失败;保存请求前设置“可能保存”状态,独立记录保存返回/回读/关闭,不再声称未保存。
- [x] 正常停止显示等待当前项收尾;unsafe 超时显示保存/核验尚未确认。
- [x] 提供与普通暂停分离的强制结束按钮,确认文案明确结果待核验;取消确认没有副作用。
- [x] 关闭窗口在运行/等待期间保持打开;任务失败后取消自动关闭,让用户看见恢复信息。
## Task 4: 集成验证与审查
- [x] 运行相关 VBS、monitor、supervisor、queue、UI、资源测试及完整发布门禁范围。
- [x] 实机空 CAD 自然退出、两工程副本正常保存及独立回读;停止/强制结束采用受控替身或独立进程,明确验证边界。
- [x] 独立审查关注:不确定写入重放、等待时 GUI 退出杀进程、force 身份校验与持久化失败、结果错误被吞、已完成项保留。
- [x] 更新操作文档和验收记录,交付本地可审查修改及测试证据。
@@ -0,0 +1,9 @@
# 用户停止与安全收尾(已批准)
用户已在本会话批准:普通停止冻结后续调度;未修改可取消;当前已修改项完成校验、保存、回读后停止;保存或修改结果不明保持保护,不自动重放或盲目保存/丢弃;关闭窗口等待收尾;强制结束必须由用户明确选择,记录中断并在重启后核验。
业务结果、应用正常退出、强制清理和物理资源状态分别记录。资源消失不抹去应用异常。保存中无进度超时进入等待/提示,不自动终止。提供明确确认的强制结束出口;其范围仅为本程序当前监督进程及自有 Job。
普通停止以当前替换项及其必要收尾为边界,不启动同工程下一项。保护记录、批次身份和进程创建时间核验继续有效。先保存中断证据再执行工具内强制终止;持久化失败时拒绝工具内强制操作并反馈错误。系统外强杀/掉电依赖已有写前保护与重启核验,不承诺可在终止时写入日志。
保持源码版本,不发布包、不关闭远端事故工单。真实工程验证只用全新副本。
@@ -0,0 +1,134 @@
# Day0:保存、关闭、退出全链路排查
2026-10-26,北京时间。源码基线 `b9391015650bf5c43a77a017c5484e9a8f352266`。承接[资源事故分析](2026-10-26-day0-resource-release-analysis.md),本轮以核心应用完成条件为主线。
## 结论
发现并验证了四个具体的生命周期问题:
1. **CAD 主进程正常退出后,程序会立即终止 Job 中的剩余辅助进程,没有先给予其自然退出窗口。** 本机实测 `vnsd.exe` 再等约 0.813 秒即可自然退出;这个等待可以避免该次强制终止。
2. **退出阶段的记录写入失败会跳过正常 Quit。** 随后仍进入自有进程清理。用生产退出控制流和替身 CAD 复现。
3. **保存调用返回、工程关闭失败时,“已保存”标记仍可能没有设置。** 实际 VBS 函数配合替身 CAD 复现,说明保存与关闭的状态记录边界混在一起。
4. **物理资源复核通过后,上层可能返回已完成业务结果,同时不再向调用方抛出正常退出失败。** 生产运行时记录可以出现 `cad_shutdown.clean=false`、资源状态 `released`、监督记录 outcome 为 `error`,但函数返回 completed 任务结果。日志证据保留,调用方的返回语义却丢失了退出异常。
这些问题与“只盯桌面释放”的分析不同:资源消失不能代表应用正常退出,更不能取代数据保存确认。
同时,两份全新工程副本的真实替换、保存、退出及新 CAD 实例独立回读均通过,未发现本轮保存数据丢失。上述问题不是已经证明的工单 #1 唯一根因;原附件中的 #1 属于读取后的收尾失败,#3 尚未启动 CAD。现场版本、实际触发路径和完整诊断仍须对照。
## 一、实机正常链路
使用已有 Audio_D2128、Audio_T605A fixture 的全新副本,生产服务入口执行 BOM 导入及两工程队列,每工程仅 RV05:`4100-CA0000-T7 → 4100-CA0000-T0`,保留原生目标 Value 策略。原始 fixture 未修改。
证据目录:`build/day0-resource-analysis-20261026/live-chain-bbd9fba7ce444e3d8e5627a6f4a385d0/`。汇总:`build/day0-resource-analysis-20261026/live-chain-summary.json`。进程时间线为同级 `live-chain-bbd9fba7ce444e3d8e5627a6f4a385d0-process-timeline.jsonl`。
| 检查 | 本次结果 |
| --- | --- |
| BOM 导入 | 2 行通过,3.875 秒 |
| 顺序执行 | 两任务 completed,155.843 秒,共用一个执行 CAD |
| 保存关闭图页调用 | D2128 约 0.32 秒,T605A 约 0.21 秒 |
| 保存后的 CloseProject | 两次返回 True,耗时约 1.68 / 1.33 秒;均先观察到 ProjectClosed 事件,再观察到调用返回 |
| 同实例重开回读 | 每工程完成内容核验后才发布 saved_ok,随后关闭工程 |
| CAD 自然退出 | 本轮五个 CAD 实例(前读 2、执行 1、后读 2)主进程均正常退出码 0,无 CAD 主进程 terminate_request |
| CAD 退出等待 | Quit helper 成功后的主进程等待约 0.234–0.390 秒,不是整个 Quit 的总耗时 |
| 退出后的独立回读 | 每工程用新 CAD 实例读取;57 器件、103 引脚连接,56 个非目标器件不变,目标身份、位置、连接和目标料号正确 |
| 原件 | 两个源工程目录哈希不变,中央库 `.lmc` 哈希不变;未宣称整个库目录全部文件无变化 |
| 前台与资源 | 观察器未发现自有 CAD 进入输入桌面或成为前台,已记录 CAD 退出和桌面消失 |
这轮运行的是源码服务层、真实 CAD 和加入边界日志的 VBS 副本;没有修改生产源码。不是完整冻结 GUI/监督进程验收,时序日志可能增加少量开销,耗时不能作为性能承诺。
第一轮记录 `live-chain-366340e7a6a942ccadaf2e1eedd9c4f4` 因探针复制 VBS 时引入 UTF-8 BOM,导致第 1 行编译失败,替换前停止。该失败属于探针,不是产品故障;保留原记录和阻断状态,修正编码后使用另一组全新副本,未重放该次失败任务。
## 二、已验证问题与含义
### A. 辅助进程尚未自然退出,就调用 TerminateJobObject
位置:`isolated_desktop.py:380` 的 `_Win32Process.close`。处理顺序为:确认/停止根进程 → 收集后代 → **TerminateJobObject** → 最多等待 5 秒确认进程退出。因此这 5 秒是强制终止后的确认窗口,不是辅助进程正常收尾的宽限期。
两工程实机中五次 CAD 主进程退出后,Job 都仍有一个成员。原时间线的映像查询使用了本机 pywin32 未提供的函数,查询错误已保留;不能凭 PID 将五个成员全部命名。独立 CIM 快照将最后一次回读的 CAD PID 55500、子进程 PID 45368 对应到实际 `vnsd.exe`。
随后单独进行空 CAD 实验,改用 Win32 `QueryFullProcessImageNameW` 确认成员身份。仅推迟探针中对已正常退出 CAD 的 Job 清理,最长观察预算 75 秒,未打开任何工程:
- CAD PID 60332 正常退出码 0。
- 此时 `vnsd.exe` PID 34960 仍存活。
- 约 **0.813 秒**后该辅助进程自行退出,随后原清理函数运行时已没有需终止的后代。
- 最终自有资源全部释放,桌面检查耗时 0 秒,整个关闭阶段约 2.625 秒。
证据:`build/day0-resource-analysis-20261026/empty-cad-descendants-9fd000b5f0af48dc82e12b6b034d2917/report.json`;探针 `probe_descendant_exit.py` 位于同级分析目录。
既有原生进程研究将 `vnsd.exe` 识别为许可证辅助程序,另有工程数据库 `iCDBNetServer.exe` 和交叉探查 `xprobSvr.exe`。本次空 CAD 仅验证 vnsd 的自然退出,不证明数据库提交被终止,也不证明它就是工单现场残留桌面的持有者。
建议方向:对已经正常退出的根进程,其明确自有的后代先进行有界自然退出等待;到期才进入可审计的强制清理。区分根进程、数据库辅助进程、许可证辅助进程;记录终止对象、原因及发生时的业务阶段。不能只把等待常量变大,也不能对不明进程进行清理。
### B. 收尾记录的 I/O 错误会改变应用退出方式
位置:`isolated_cad.py:207`。正常 Quit 前的 `_record(state='closing')` 与 Quit 在同一个 try 内,记录失败会直接跳入异常处理,接着 finally 仍关闭自有资源。
故障注入:仅让这一次 closing 记录写入抛 PermissionError,其他记录正常。结果为:没有发出 Quit;写入 uncertain;调用资源 close。真实 CAD 由替身代替,因此本项证明控制流缺陷,没有实际杀死用户 CAD。
含义:目录权限、同步软件/安全软件的文件访问冲突、磁盘问题等不只影响日志,还可能改变应用本来可以正常退出的路径。上述环境来源只是候选,没有确定具体软件责任。
建议方向:隔离诊断/资源收尾记录错误与正常退出协议;写前业务保护失败仍应禁止写入,但不能混同为“所有诊断写入失败都必须跳过 Quit”。退出请求仍须满足实例身份、工程状态等原有条件。
### C. 保存状态在工程关闭成功之后才记录
位置:`QuickPartReplace.vbs:934`。调用 `QPCloseBatchProject(TargetPath, True)` 返回后才执行 `QP_BatchSaveCommitted=True`;这个子程序内部既保存图页,又关闭工程。
用真实 `QPFinishBatchItem` / `QPCloseBatchProject` 函数与 CAD 形状替身运行:图页的 SaveChanges=True 调用已经返回,随后工程拒绝关闭。观察到 `save_called=True`、`commit_flag=0`、错误已抛出,没有发布成功 checkpoint。
因此,这个标记不能独立表达“保存已发出/可能完成,而工程关闭失败”。某些后续错误文案依赖它,可能把已涉及保存的故障描述为未保存。`failed_native_result_uncertain` 仍会依据原生修改证据阻断重试,本轮没有发现借此自动重放替换;问题主要在恢复判断所需的状态粒度和事实描述。
建议方向:拆分 save_requested、save_returned、saved_readback_verified、project_closed 等证据;保存请求开始后发生异常应保留“可能已保存”的状态,不能因 CloseProject 失败回退成“未保存”。
### D. 资源消失后,正常退出异常不再向调用方抛出
位置:`batch_supervisor.py:159` 的 `_finish_response`,及 `batch_resources.py:117` 的重查逻辑。
用生产 BatchCadRuntime 创建合法资源记录,替身 CAD 在退出时抛出 Quit 超时,已保存业务结果独立保留;模拟拥有资源的监督进程结束后,再调用生产资源复核及返回函数。结果:
- 会话先 blocked;物理桌面已不存在后变为 released。
- `cad_shutdown.clean=false` 和 `orderly_shutdown` 异常仍保留在 JSON。
- 监督记录 `state=finished`、`outcome=error`。
- 调用方收到含 completed 条目的 TaskQueueResult,没有收到退出异常。
这是状态返回语义不一致,不是原始错误日志被删除,也不证明已独立核验的保存结果应当被作废。应保留正确业务结果,同时显式暴露“业务完成、应用异常退出/强制清理”的收尾状态,不能仅凭物理资源消失将异常归为恢复成功。
上述 B/C/D 复现实物:`build/day0-resource-analysis-20261026/fault-chains-907a534cc6bb441e8cddf7670dd04aac/`,脚本 `probe_fault_chains.py`。较早的临时探针曾因日志文件句柄尚未关闭而无法删除临时目录;后续改为保留独立证据目录并关闭日志句柄,此探针清理错误不计入产品结论。
## 三、仍需验证的高影响风险
| 风险 | 已知机制 | 尚不能断言的部分 |
| --- | --- | --- |
| 慢保存被当成无响应 | 保存前仅发一次阶段进度;默认 150 秒请求停止、300 秒判无响应,safe_stop=False 不阻止监督器升级。保存没有类似原生快照的进度旁路 | 没有真实慢保存达到 300 秒的实验,不证明现场曾在保存中被终止 |
| Quit/关闭响应延迟 | Quit helper 20 秒;成功后 CAD 15 秒;checkpoint 20 秒;项目会话关闭另有 60 秒宽限 | 本次正常样例均未超过这些时限,不能据此证明大工程也安全或必须统一加长 |
| 同实例回读可能不足以覆盖持久性 | 生产路径在同一个 CAD 实例中重开回读 | 本次新实例读回通过,未发现缓存掩盖丢失;仍缺断电、数据库服务异常和大型工程覆盖 |
| 关闭工程返回值被忽略 | 本机 TypeLib 表明 CloseProject 返回 Boolean;VBS 常直接调用并用工程路径为空作为补充判断 | 本次调用均为 True,尚未观察到 False 且空路径的真实异常,不能将其直接确定为事故根因 |
| GUI/监督进程异常终止 | Job 关闭会清理受控进程,不能保证正在进行的保存完成 | 正常点击关闭时 GUI 会等待当前 worker,不能把所有 GUI 关闭行为描述成直接杀进程。崩溃/强制结束另需故障实验 |
| 数据库和跨进程协作收尾 | 既有研究存在 iCDBNetServer、xprobSvr 等进程;仅确认 viewdraw 退出不足以描述全部应用状态 | 本轮没有捕获这些服务器在未提交数据时被终止;不能把 vnsd 的观察套用成数据库丢失证据 |
## 四、厂商接口核对的重要限制
实际安装的 7.9.5 `viewdraw.tlb` 已离线核对,结果保存在 `cad-lifecycle-typelib.json`:
- `IVdApp.CloseProject` 返回 Boolean。
- `IVdApp.Quit`、`IVdSchematicSheetDocument.Close(SaveChanges, FileName)` 返回 void,不能虚构它们有保存成功布尔返回值。
- `IVdDoc` 有 Saved 属性,但不能据此假定当前 `IVdSchematicSheetDocument` 也有同样的属性契约。
- 事件接口包含 BeforeDocumentSave、AfterDocumentSave、ProjectClosed。
真实 VBS 日志已订阅上述事件:本轮 ProjectClosed 被正常观测,但在两次 SaveChanges=True 路径中**没有观测到 BeforeDocumentSave/AfterDocumentSave**。因此不能因为类型库列出了保存事件,就把等待该事件直接设为当前原理图保存路径的必经条件。需要核实它适用的文档类型、接口和触发条件。
本轮读取的安装帮助未取得足以证明上述调用异步/同步持久化语义的详细契约。类型库提供方法签名,不能替代持久化保证。
## 五、验证与修改范围
本轮补充回归:
```text
python -m pytest -q tests/integration/test_quick_value_modes.py tests/v1_1/test_project_session.py tests/v1_1/test_failed_native_recovery_guard.py tests/v1_1/test_supervisor_result.py tests/v1_1/test_ui_operation_lifecycle.py
94 passed in 34.11s
```
上一轮已执行 IsolatedCad 和 worker monitor 的 60 项验证,本轮不重复当成新结果。测试通过说明已有行为与保护仍按测试约定运行,不证明上述异常路径合理。
仅增加分析报告和 `build/day0-resource-analysis-20261026/` 下的诊断脚本/证据。生产 Python、正式 VBS、版本号及发布包未修改,未向远端工单发消息或关闭工单。
下一轮修改应优先针对已复现的四个问题,并保留原有身份校验、写前保护和禁止不确定重放。验收必须分别报告数据结果、工程关闭结果、应用退出方式、后代进程退出方式和最终资源状态,不能汇成一个“释放成功”。
@@ -0,0 +1,140 @@
# Day0:资源释放工单原因分析
日期:2026-10-26(北京时间)。严重级别依据用户本次升级说明;未修改远端工单标签或状态。分析源码基线:`b9391015650bf5c43a77a017c5484e9a8f352266`。
后续已完成[核心应用生命周期专项排查](2026-10-26-application-lifecycle-investigation.md):包含两工程真实保存与独立读回、vnsd 自然退出对照及三类异常控制流复现。后续结论以该记录注明的证据和边界为准。
## 判断
工单 #1 的现场根因仍未闭环。根据用户对核心应用生命周期的补充,排查主线调整为“业务操作完成 → 保存确认 → 工程关闭确认 → 应用正常退出 → 进程及辅助资源释放 → 桌面回收”。桌面是隐藏操作的基础设施,其消失不能替代业务完成或进程正常退出证明。第一轮对桌面末端症状的侧重不足以覆盖保存与异常退出风险。
对已经观察到的桌面残留,需要继续追查自有清理范围以外的引用,包括本程序未登记的额外引用、无窗口线程、其他进程持有的桌面句柄和由系统或服务创建的辅助进程。这里的“范围以外”不等于已经证明是第三方软件造成,也不能将其提前确定为整个事故的业务根因。
工单 #3 在 CAD 未启动时出现同类残留,说明 CAD Quit 或 COM 释放不足不能解释全部现场现象。两个工单具有相同末端症状,但尚不能断言具体持有者或底层根因相同。
Beta 3 的进程隔离、批次复用和资源复核覆盖了部分故障机制。产品仍缺少现场持有者的完整识别能力,缺少对应现场版本的复测成功回执,不能凭本机通过将事故关闭。
## 补充核查:核心应用完成和退出链路
本节优先于后文仅针对桌面残留的原因排序。关闭桌面句柄不会自动结束进程;当前实现实际先清理受控进程、再关闭桌面句柄。应分别证明数据保存成功、工程关闭成功、应用正常退出、强制终止是否发生和资源是否消失。
### 正常路径已有保存回读,但完成语义仍需核实
- `QuickPartReplace.vbs:863` 的 `QPCloseBatchProject` 调用 `Document.Close(True, "")` 保存关闭图页,随后调用 `CloseProject`,立即查询工程路径为空。
- `QuickPartReplace.vbs:934` 起会重新打开同一工程、校验目标及非目标内容,之后才给出 `saved_ok`。不能把现有程序描述为“完全没有等待或没有保存核验”。
- COM 调用返回是现有的等待边界,但当前函数没有单独的保存完成事件、持久化提交状态或忙碌状态等待。是否需要增加这些机制,必须核对该版本 CAD 的 API 契约及真实行为;不能仅因缺少 sleep 就断言保存是异步或保存未完成。
- 生产路径是在同一个 CAD 实例中重开工程进行回读。这提供了内容核验,但若要证明进程退出后的持久性,应增加正常退出后的独立新实例读取实验,并核查实际数据库服务的提交语义。不能仅凭同实例回读断言存在缓存问题,也不能将开发验收中的独立回读当成每次生产执行都有此保障。
- 图页/工程关闭后立即查询一次状态,没有针对“调用已返回、关闭状态稍后才就绪”的有界状态等待。若厂商允许这种延迟,可能把仍在正常收尾的应用判为失败;目前还没有确认其 API 是这种行为。
### 明确存在的风险路径:应用未确认正常退出时进入强制清理
`isolated_cad.py:207` 的退出流程为:Quit helper 最多等待 20 秒;helper 成功才进一步等待 CAD 进程最多 15 秒。Quit helper 超时或失败时会跳过这次 CAD 正常退出等待,记录 uncertain,然后仍执行 `desktop.close()`。在批次模式中这里首先是 lease 的进程清理,随后才是批次拥有者的桌面关闭;底层 `_Win32Process.close()` 对存活自有进程/Job 有终止逻辑。
因此存在需要验证的链路:应用仍忙或关闭被拒绝 → 固定时限/单次状态检查失败 → 保护标记 → 强制终止自有进程 → 后续桌面释放检查。保护标记能避免无依据重放,不能保证被终止前未完成的保存可以安全完成。强制终止后的“资源清理成功”也不能用作“业务正常完成”的证据。
Quit helper 在发现工程仍打开时会拒绝退出,避免隐式保存或丢弃;但外层依然会进入自有进程清理兜底,所以必须检查当时是否有未完成的写操作。业务状态、退出方式和资源状态需要分开判断。
### 保存期间的无进度监控需要专项验证
`QuickPartReplace.vbs:874` 在进入保存关闭前发出一次 `save_and_close` 进度。若后续阻塞在 COM 调用中,脚本不能继续发送阶段进度。当前原生侧进度旁路只处理 `native_snapshot_read`,没有相同的保存进度旁路。
默认无进度时限为 150 秒。`ProgressWatch.action()` 在 150 秒请求停止、300 秒判定 unresponsive;这个判定不因 `safe_stop=False` 而停止升级。VBS 会在安全边界才响应协作停止,但监督器仍可在后续超时中终止 worker,并进入 CAD 收尾。真实慢保存是否能达到这个条件尚未实测;普通进程存活或无条件心跳也不能当成保存正在推进的证据。
### 本轮验证与证据边界
- `python -m pytest -q tests/v1_1/test_isolated_cad.py tests/trial_task_center/test_worker_monitor.py`:**60 passed,16.22 秒**。包括控制流替身和真实辅助进程测试,没有真实 CAD 慢保存实验。
- 对生产 `IsolatedCadSession` 注入模拟 Quit helper 超时:确认等待参数为 20 秒、CAD wait 未被调用、uncertain 标记已写入、之后进入资源 close。未终止真实 CAD。
- 对生产 `ProgressWatch` 输入模拟 `save_and_close / safe_stop=False`:150 秒为 request_stop,300 秒为 unresponsive。
- 证据:`build/day0-resource-analysis-20261026/application-lifecycle-probe.json`。这是控制流证据,不能写成现场已发生保存中断或数据丢失。
- 原始 #1 是库存读取后的收尾失败,四次日志记录 Quit helper/CAD 正常退出;#3 尚未启动 CAD。当前附件没有证明替换保存被提前打断。上述生命周期风险必须调查,但不能直接取代这两组现场证据。
优先实验应改为:在工程副本上记录保存调用前后、关闭调用前后、回读结果、Quit 请求与返回、CAD 自然退出、强制终止请求及原因的完整时间线;分别验证正常保存、长时间忙碌、关闭延迟和退出拒绝。数据正确性及退出契约查清后,再处理桌面末端残留。无需通过开启可见桌面或无界等待来替代这些完成条件。
## 已核实的证据
### 工单 #1:V1.1.3,四次资源释放确认失败
原件位置:`build/issue-1-20261021/evidence/`。四份失败的 `private-desktop.json` 一致记录:
- CAD 和退出辅助程序的退出码均为 0。
- 所记录进程的 `job_active_processes=0`,已知后代句柄均已确认退出,自有进程及 Job 句柄关闭。
- `desktop_handle_closed=true`,但 5 秒后 `released=false`。
- 四次清理来自同一个主程序 PID 14608,调用线程不完全相同,调用线程桌面为 `Default`。
- 另有一份采集中途取消的记录停在 `ready`,没有最终收尾证据;不能算作释放成功,也不能直接算作第五次同因泄漏。
这些证据降低了“已登记 CAD 主进程没有退出”“完全没等待终止”“已登记桌面句柄没有关闭”的优先级。它们不能证明进程所有线程、所有其他桌面句柄或所有系统辅助进程都已释放。
### 工单 #3:Beta 2,CAD 未启动也残留
本地 `build/issue-3-20261023/application.log` 与本次重新读取的远端附件逐字节一致。SHA256:`28d164039831973b7f6596a081d305421131f883514feae2fd652ff90ec54241`。
| 北京时间 | 日志位置 | 事实 |
| --- | --- | --- |
| 10-23 14:29:13 | 第 2 行 | 运行版本 `1.1.4-beta.2`,Windows 基础 build 26200 |
| 14:29:34 | 第 14 行 | 开始整批 BOM 检查 |
| 14:30:01 | 第 669 行 | 开始批末释放 |
| 14:30:06 | 第 1098 行 | 桌面句柄关闭,桌面仍存在;CAD PID 为空,显示 CAD 未启动;窗口列表为空 |
| 同一次失败的异常链 | 第 1102、1116 行及后续堆栈 | 首发错误为内部 attempt 目录创建的 `FileNotFoundError / WinError 3`,失败路径长 265 字符;释放错误发生在异常收尾期间 |
| 14:30:47 | 第 1155 行 | 用户请求重新清理资源 |
| 14:30:54 | 第 2013 行 | 同一个桌面仍存在,句柄关闭、CAD 未启动、窗口列表为空 |
前后失败样本相隔约 48 秒。未连续观测期间的状态,但相同随机桌面名两次均被枚举,明显削弱了“只需略微增加 5 秒超时”的解释。
`cleanup.processes=[]` 仅表示该次最终清理列表为空。原生 CLI 在批内执行后会从活动列表移除,不能据此说整批从未启动过子进程。
长路径是已记录的首发业务错误;桌面残留是收尾时另一个故障。修复路径问题不自动证明桌面引用来源已修复。
## 桌面残留的原因排序和判别办法
“优先级”表示调查顺序,不表示统计概率。
| 原因候选 | 优先级及证据 | 如何区分 |
| --- | --- | --- |
| 本程序的其他线程或额外桌面句柄仍持有引用 | 高;旧版 GUI 生命周期长,关闭登记句柄不能排除额外引用。四次 #1 来自同一主程序 | 现场保留桌面名,由独立观察进程在监督进程退出前后、主程序正常退出前后枚举;配合句柄和线程归属快照。随某进程退出而消失可缩小范围,但仍需持有者证据,不能单凭时间相关性定责 |
| Job 外的无窗口辅助线程/进程 | 高;完全符合“Job 清空但桌面存在”。历史受控实验发现过无窗口 Python 和 conhost 线程 | 对目标桌面扫描全局线程,记录 PID/TID、映像、创建时间、会话及 Job 归属;与原生 CLI 启停时间对齐。历史 conhost 只是实验线索,现场未确认 |
| 其他进程仅持有 OpenDesktop 句柄 | 高;可在不启动 CAD、没有窗口和目标线程的情况下复现相同症状 | 在线程扫描之外查询 Desktop 类型句柄。已有外部句柄实验复现阻断,持有者释放后可恢复;具体现场持有者未知 |
| 原生 CLI、运行库、服务代理触发额外引用 | 中高;#3 没启动 CAD,但此前走过原生读取链 | 在现场副本依次运行空桌面、简单子进程、单个 icdb2ascii 读取、多个读取、CAD 回退,找出首次出现残留的阶段。普通子进程默认受 Job 约束,不能笼统断言所有后代都逃逸 |
| 安全软件、录屏、远程协助、桌面管理组件等参与桌面访问 | 中,属于上一类的来源候选;当前没有具体软件或 PID 证据 | 先采集持有者或模块证据,再做单因素环境对照;没有证据时不能指定某个软件为元凶 |
| Windows USER 引用延迟回收/系统环境差异 | 中低;短暂延迟有受控实验支持,但现场约 48 秒后的重查仍失败 | 使用独立进程记录 0、1、5、15、60 秒存在性,同时记录所有者退出状态。本机和 #3 均为基础 build 26200,仍需对照累计更新、运行库和注入组件,不能只按系统大版本归因 |
| 句柄继承、Job 接管或异常清理分支缺陷 | 中低,需保留;当前代码在恢复运行前分配 Job,重定向使用明确的继承句柄清单,已有后代退出和句柄关闭失败测试 | 检查实际运行 EXE 哈希、Job 分配结果、句柄继承标志、失败时保留的所有权记录。当前源码机制不能倒推所有旧版或现场程序完全相同 |
| 历史资源记录损坏或残留,形成逻辑阻断 | 对“升级后仍不能用”须优先区分;代码会保守阻断坏记录 | 对比当前物理桌面/进程与 `session.json`、`supervisor.json` 的状态。若物理对象已消失而记录无法校验,是证据恢复问题;不应归为仍有物理泄漏。原始 #1/#3 明确枚举到桌面,坏记录不能解释其最初残留 |
| CAD COM 引用未释放 | 对 CAD 路径仍可查;作为统一解释优先级低 | 仅在已启动 CAD 的场景核对 COM、helper 与 CAD 生命周期。#3 的 CAD 未启动是其无法解释全部现象的直接证据 |
| 工程数量上限、桌面堆耗尽、纯 BOM 匹配错误 | 当前证据弱 | 查桌面创建失败、内存资源错误和随批量增长的资源曲线。当前主要失败在销毁确认;BOM 行不匹配是独立业务结果,不能据此改资源配额或放宽匹配 |
微软文档确认:关闭一个桌面句柄与销毁所有引用不是一回事;线程的桌面连接不依赖存在可见窗口;Job 默认覆盖普通 CreateProcess 子进程,但不是整个系统的进程集合。例如 WMI `Win32_Process.Create` 创建的进程不属于这类默认继承范围。此机制说明需要查 Job 外所有者,不代表本软件已经被证明使用了 WMI 启动逃逸进程。
## 当前实现为何仍可能卡住
1. **判定严格,但归属诊断不足。** `isolated_desktop.py:66` 通过枚举确认桌面是否消失;失败时只收集窗口。`isolated_desktop.py:565` 明确将外部句柄持有者标为 unknown。空窗口不是无引用证明。独立工具 `tools/diagnose_private_desktop.py` 能查无窗口线程,但不能列出所有仅持有句柄的进程,权限错误也必须保留。
2. **监督进程隔离只解决其覆盖的所有权。** `batch_supervisor.py:279` 等待并关闭受控 Job,随后 `_finish_response` 复核资源。这能够处理监督进程自身存活期间的引用;其他长寿命进程的引用不会因该监督进程退出而消失。
3. **重新清理有能力边界。** `batch_resources.py:117` 对仍有 live owner 的会话可重试其关闭;没有 live owner 时只确认桌面消失和历史 PID 身份,不会接管或终止不明持有者。Beta 3 常见的是监督进程已经退出后的复核分支。反复点击不会凭空释放外部引用。
4. **未释放门禁扩大可用性影响。** `guard_pending_resources` 阻止同一数据根的新批次。这一保护避免在旧资源不明时继续创建桌面或重放业务,却使一个释放问题阻断后续使用。恢复机制和持有者识别必须共同补齐。
5. **损坏记录不会自动修复。** 无法校验的记录保持 blocked;无 session_id 时跳过自动资源操作。这是证据不足时的保护行为,但意味着资源已消失仍可能需要审计式人工核对。不能通过删除保护文件掩盖该状态。
6. **诊断包不等于实时取证。** 当前导出函数主要打包已有文本,未自动执行全局线程或句柄扫描。上限为单文件 4 MiB、总计 32 MiB、512 个文件,超限写入 omissions;必须检查 manifest,不能把未导出的记录当成从未发生。
## 本次新验证
运行环境:Python 3.12.10、Windows 11 基础 build 26200。以下测试未打开或替换用户工程。
- `python -m pytest -q tests/v1_1/test_isolated_desktop.py tests/v1_1/test_desktop_diagnostics.py tests/v1_1/test_batch_runtime_windows.py tests/v1_1/test_batch_supervisor.py`:**44 passed,87.89 秒**。包含真实 Windows 资源测试与替身测试,覆盖无窗口引用、延迟释放、后代进程、诊断边界、监督恢复等;不等同于现场完整 EXE 验收。
- `python -m pytest -q tests/v1_1/test_batch_runtime.py::test_unknown_resource_evidence_never_authorizes_cleanup tests/v1_1/test_batch_runtime.py::test_restart_reconciles_only_confirmed_disappearance_without_replacement`:**4 passed,1.22 秒**。确认坏证据继续阻断、重启后仅复核的代码行为;这两类为模拟边界测试。
- 资源循环探针 `build/day0-resource-analysis-20261026/probe_cycles.py`:20 次空桌面、20 次带重定向的简单 Python 子进程,**40/40 正常释放**。自有桌面残留 0,观察前后没有新增桌面,调用线程桌面未改变。两组的释放枚举检查最大耗时均约 0.016 秒,该数字不包含完整子进程启动耗时。
- 循环证据:`build/day0-resource-analysis-20261026/cycles-6819874fa1b04649a6dbb9c6b1992034.json`。它是源码 Python 基础路径实验,没有覆盖 Qt GUI、冻结 EXE、实际 icdb2ascii、CAD 或现场常驻组件,不排除这些组合中的问题。
这批结果降低了本机基础资源封装普遍失效的可能性,不能证明现场根因已修复,也不能证明 102 个原工程全部可用。
## 下一步应取得什么证据
1. **先确认事故实际版本和状态。** 当前收到的两组原件属于 V1.1.3 与 Beta 2;没有 Beta 3 的新失败诊断包。记录现场 EXE 哈希、是否完整配套 `_internal`、原始 data 是否迁移、故障是再次产生桌面还是旧记录阻断。Beta 3 公开 EXE 的历史记录哈希为 `df08cd328f6f4c9019f232c845e09627551e7b2d24790a466d53efcd291622d1`,以实际安装文件核对。
2. **残留发生时立即取证。** 在原机器、原登录会话,从 session 记录取得准确桌面名;运行独立线程诊断并查询 Desktop 句柄持有者,保留 PID、创建时间、映像、线程、会话、Job 和时间戳。仅复制旧日志到开发机无法找回消失的 Windows 引用。
3. **独立观察所有者退出前后。** 先保存结果及诊断,再观察监督进程退出、应用正常退出后的同名桌面变化。桌面随监督进程退出消失,优先追监督进程内部引用;随 GUI 退出消失,追 GUI 或其关联组件;GUI 退出后仍存在,追其他进程或系统引用。上述变化用于缩小范围,最终需结合持有者证据确认因果。
4. **按最小场景逐层复现。** 空桌面 → 简单 worker → 单次原生 CLI → 多次原生 CLI → CAD 回退 → 批量混合。每层记录创建数量、已退出进程、最终桌面状态。实际读取使用工程副本,首次失败就保留证据,避免反复业务执行污染现场。
5. **修复后验证恢复结果。** 对已识别的持有者修复生命周期;完成正常、异常、取消、重启和跨批次检查,确认只清理资源时业务结果不变、没有重复替换,随后在现场原目录和原批量范围验收。只有现场复测通过并留存回执,才具备关闭工单的依据。
技术交付首先补齐业务完成与应用退出的分阶段证据,并专项验证等待与强制清理边界;对完成业务后仍残留的桌面,再补自动捕获持有者的能力。单纯延长等待、反复要求重新清理或继续发布未取得现场证据的候选包,都不足以建立闭环。
## 关联记录
- [工单 #1](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/1)、[工单 #3](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/3)。
- [首轮工单分析](2026-10-21-issue-1-bom-import.md)、[Windows 生命周期受控实验](2026-10-21-windows-desktop-lifecycle.md)、[外部句柄持有者实验](2026-10-23-issue-3-external-holder.md)、[Beta 3 验收边界](2026-10-23-issue-3-safe-recovery.md)。
- 微软:[CloseDesktop](https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-closedesktop)、[线程连接桌面的规则](https://learn.microsoft.com/en-us/windows/win32/winstation/thread-connection-to-a-desktop)、[Job Objects](https://learn.microsoft.com/en-us/windows/win32/procthread/job-objects)。本次重新核对官方页面。
@@ -0,0 +1,59 @@
# 用户停止与安全收尾实施验收
日期:2026-10-26;分支:`codex/day0-safe-stop`。实现用户在本会话批准的[停止流程](../superpowers/specs/2026-10-26-safe-user-stop.md),对应[全链路调查](2026-10-26-application-lifecycle-investigation.md)。
## 实现行为
- 普通暂停和关闭窗口不启动下一项,当前写入、保存和回读安全完成后停止。当前项并不等于整个工程。
- 有写入可能的阶段长期没有进度时显示等待,不自动把超时升级成杀进程。纯只读采集保留有界取消。
- 写入失败、保存结果不确定或 Quit 被拒绝/超时且 CAD 仍存活时,交互执行保留应用与工程,等待处理,不自动保存或丢弃。
- “强制结束…”独立确认;先持久化业务保护及批次中断证据,再终止明确自有的监督进程及 Job。证据写入或 fsync 失败时拒绝该次 force,界面保留重试能力。
- 保存请求、保存调用返回、保存回读和工程关闭分别记录,工程关闭失败不会丢失此前保存边界。
- 收尾记录写入失败不跳过满足身份检查的正常 Quit。主进程自然退出后给予自有后代 2 秒自然退出窗口,之后的强制清理单独记录。
- 资源消失不吞掉异常退出;调用方收到独立收尾错误,已核验业务结果仍附带保留。
- 等待期间可独立导出诊断包。关闭窗口在操作结束前不退出;失败/强制结束后取消自动关闭,保留恢复提示。
## 自动验证与审查
针对安全边界先运行失败用例再实现,包括慢保存不会被 watchdog 终止、明确 force 的保护先行、保护写入失败保留进程、已有文件也重新 fsync、保存后工程关闭拒绝、窗口关闭不触发 force、确认取消无副作用。
真实辅助进程测试覆盖监督子进程强制中断、控制回调异常时保持子进程存活、拒绝第一次保护写入后仍允许后续明确重试、资源清理不重放业务。CAD 调用在这些故障测试中使用替身,不能据此声称真实保存中断无损。
退出协议独立审查修复了部分子进程查询失败时丢失已知强制终止 PID 的问题。整体验证另外修复:只读入口没有 force 出口却进入无限等待、证据首次 fsync 失败后因文件存在被误放行、已执行 force 被诊断异常降级为普通停止、控制通信异常绕过确认清理、直接执行 force 失败没有反馈,以及 force 意图与监控异常并发的竞态。复审未发现新增阻断问题。
首次完整实现回归 **1063 passed,1 条既有 pywinauto STA 警告,363.82 秒**。之后完成审查修正,最终扩大回归 **1147 passed,1 条既有 pywinauto STA 警告,325.64 秒**:
```powershell
python -m pytest -q tests/v1_1 tests/trial_task_center/test_application.py tests/trial_task_center/test_execution_safety.py tests/part_replace/test_verification.py tests/integration/test_quick_value_modes.py tests/integration/test_quick_session_startup.py tests/integration/test_quick_part_replace_contract.py tests/integration/test_automatic_replacement_regression.py tests/trial_task_center/test_worker_monitor.py tests/trial_task_center/test_batch.py
```
本命令包含项目发布脚本要求的源码测试范围,并补充此次受影响的 VBS、worker monitor 与批次测试。`git diff --check` 与新探针/停止模块语法检查通过。
两个旧 VBS fixture 在基线 `b939101` 也失败:未填 owned project path 且错误期望 no-match 保存、图页清单断言未使用规范化路径。修正的是 fixture 和断言,没有放宽生产所有权或保存规则;脚本契约组 72 项通过。
## 真实保存期间请求普通停止
工具:[probe_safe_stop_live.py](../../tools/probe_safe_stop_live.py)。本轮输出为 `build/safe-stop-live-a9113545bf/`,报告 `report.json`,最终资源复核 `final-resource-check.json`。
入口为源码 supervised service → 真实独立 batch worker → 生产 VBS → 真实 CAD,使用两份全新 Audio_D2128/Audio_T605A 副本。监听第一工程的生产 `save_requested` 日志后只设置普通停止 Event,不设置 force。此触发处于保存请求附近,不能声称精确暂停在 COM 内某一条指令。
| 检查 | 结果 |
| --- | --- |
| 第一个工程 | 当前 RV05 替换、保存和回读完成,任务 completed |
| 第二个工程 | 保持 pending,没有继续替换 |
| 队列 | paused=true,未请求强制结束 |
| 第一工程新 CAD 独立读回 | 57 器件、103 引脚连接;目标 RV05 为新料号,56 个非目标器件保持;目标身份、位置和连接保持 |
| 第二工程新 CAD 独立读回 | 57 个器件的完整属性及连接图与执行前一致 |
| 执行批次退出 | orderly_shutdown=true、clean=true、forced_termination=false |
| 资源 | 资源记录全部 released,最终枚举确认其桌面消失,没有本轮残留 CAD |
| 原件 | 两个源工程目录哈希不变,中央库 `.lmc` 哈希不变 |
没有重放任何已经完成的替换;测试用的 before/after 独立读取使用新 CAD 实例。
## 边界
本轮没有构建或发布新 EXE,未做完整冻结 GUI 的保存中强制结束、掉电或原工单全量现场复测。GUI 行为使用 Qt 测试验证,真实保存停止使用源码监督入口验证;二者不能混写为冻结 GUI 全流程验收。
强制结束为避免切换任务竞态,保守标记本批可能涉及的目标;已完成业务证据和结果保留,但先前已完成任务也可能需要恢复核验。不能清除保护来继续。
保持应用等待不代表保存结果一定可以恢复。外部任务管理器强杀/系统掉电没有安全收尾保证,重启必须依赖原先写前证据核验。工单现场具体桌面持有者仍未查明,本实现不能当作现场根因已经闭环。
+2
View File
@@ -1,5 +1,7 @@
# R8:进度式超时与分阶段引脚索引
> 本文保留 R8 历史行为。2026-10-26 起,有写入可能的阶段无进度超时改为等待安全收尾,只有明确确认的强制结束才中断;当前停止与恢复规则见[停止、强制结束与恢复](safe-stop-and-recovery.md)。只读采集继续采用有界取消。
## 改变的行为
- 150 秒不再是整个任务的总时限,而是连续没有有效进度的观察窗口。有有效进度时可以继续执行超过 150 秒。
+30
View File
@@ -0,0 +1,30 @@
# 停止、强制结束与恢复
执行期间点击“完成当前项后暂停”,程序不再启动下一替换项。已经进入修改或保存的当前项会继续完成必要校验、保存、重开回读和工程关闭,然后正常退出 CAD,并等待自有辅助进程收尾。关闭主窗口也遵循这个流程,操作仍在进行时窗口保持打开。
“当前项”是当前图页和替换规则对应的执行单元,不是整个工程或整个批次。已经保存核验的结果保留,未开始的任务保持未执行。
## 保存或修改长时间没有进度
有写入可能的阶段,程序不会仅因超过无进度时限就自动终止应用。界面显示保存或核验尚未确认,可以继续等待并导出诊断包。等待不会启动下一项,也不会自动清除保护记录。
如果操作已经失败且保存结果不确定,程序保留当前应用和工程现场,不再自动保存或丢弃。应先导出诊断、保留原始工程与备份,再决定是否需要强制结束。此时“继续等待”不表示自动恢复执行。
纯读取的 BOM/库存采集仍采用有界取消;不存在写入需要完成时,不要求用户无限等待。确认未修改的启动失败也保留既有有界停止机制。
## 明确选择强制结束
“强制结束…”与普通暂停分开。确认框明确提示:当前操作可能正在修改或保存,强制结束可能留下未保存或部分保存结果,之后必须先核验恢复。
确认后,程序先持久化中断保护和批次记录,再终止自己拥有的监督进程及 Job。保护记录无法可靠保存时,不执行这次强制结束,界面提示原因并允许重试或导出诊断。取消确认没有终止副作用。
为避免强制请求与工程切换同时发生而漏记目标,强制结束会保守保护本批可能涉及的任务。先前已保存结果不会被覆盖或重放,但这些任务也可能显示需复核或被保留在当前列表。不要删除保护文件绕过核验。
## 程序重启之后
- 资源复核只处理进程和桌面状态,不重新替换。
- 结果不确定的任务必须使用“核验并恢复”,按照已有快照和当前实际工程核对。
- 已保存核验结果继续保留;资源清理成功不自动解除业务保护,也不自动继续队列。
- 如果应用没有正常退出或后代进程需要强制清理,会单独报告收尾异常,不能把“资源已释放”理解为“保存和退出均正常”。
任务管理器强杀、系统关机或掉电可能不给程序写最终日志的机会。这些情况依赖修改前已经落盘的证据及下次启动的中断核验;程序不能保证被外部强杀时完成当前保存。
+21 -2
View File
@@ -28,7 +28,7 @@ Dim QP_ComponentUidIndex, QP_ComponentRefdesIndex, QP_ComponentLocationIndex
Dim QP_ComponentIndexContext
Dim QP_NativeCallActive
Dim QP_BatchMode, QP_OwnedProject, QP_BatchState, QP_BatchCode, QP_BatchMessage
Dim QP_BatchSaveCommitted
Dim QP_BatchSaveCommitted, QP_BatchSaveRequested
Dim QP_DiscoveryMode, QP_TargetSchematic, QP_TargetSheet
Dim QP_FailureNumber, QP_FailureSource, QP_FailureDescription
Dim QP_NcSyncSummary
@@ -60,6 +60,7 @@ QP_BatchMode = False
QP_OwnedProject = False
QP_BatchState = ""
QP_BatchSaveCommitted = False
QP_BatchSaveRequested = False
QP_DiscoveryMode = False
QP_TargetSchematic = ""
QP_TargetSheet = ""
@@ -110,6 +111,10 @@ Else
QP_FailureDescription = Err.Description
If QP_DiscoveryMode Then QPCleanupDiscoveryFailure
FailureState = "hard_failure"
If QP_BatchSaveRequested Then
FailureState = "batch_saved_unverified"
QP_FailureDescription = "save was requested; persisted outcome requires reconciliation. " & QP_FailureDescription
End If
CloseError = ""
If QP_BatchMode Or QP_SessionMode Then
QPDiscardFailedBatchProject CloseError
@@ -318,6 +323,7 @@ Sub QPSessionNext()
QP_BatchMode = True
QP_BatchState = ""
QP_BatchSaveCommitted = False
QP_BatchSaveRequested = False
QP_NativeMutationPossible = False
If Trim(QPReadUtf8(QP_Fso.BuildPath(QP_Root, "batch.request"))) <> QP_ExpectedJobId Then _
Err.Raise vbObjectError + 873, "QuickPartReplace.ProjectSession", "next batch authorization mismatch"
@@ -872,12 +878,21 @@ Sub QPCloseBatchProject(ByVal ExpectedPath, ByVal SaveChanges)
If Documents.Count <> 1 Then Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "open sheet set changed; no automatic close"
Set Document = Documents.Item(1)
QPProgress "save_and_close", 0, 1
If SaveChanges Then
QP_BatchSaveRequested = True
QPLog QP_Root, "INFO", "save_requested job=" & QP_CurrentJobId
End If
Document.Close CBool(SaveChanges), ""
If SaveChanges Then
QP_BatchSaveCommitted = True
QPLog QP_Root, "INFO", "save_returned job=" & QP_CurrentJobId
End If
If QPCanonicalPath(QPProjectPath()) <> QPCanonicalPath(ExpectedPath) Then _
Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "project changed during sheet save"
QP_App.CloseProject
If Len(CStr(QP_App.GetProjectData().GetProjectFilePath())) > 0 Then _
Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "project did not close; queue stopped"
QPLog QP_Root, "INFO", "project_closed job=" & QP_CurrentJobId
QP_OwnedProject = False
QP_OwnedProjectPath = ""
QPInvalidatePinIndex
@@ -888,6 +903,10 @@ Sub QPDiscardFailedBatchProject(ByRef ErrorDescription)
Dim ErrorNumber, ErrorSource, Description
ErrorDescription = ""
If (Not QP_BatchMode And Not QP_SessionMode) Or Not QP_OwnedProject Or QP_App Is Nothing Then Exit Sub
If QP_NativeMutationPossible Or QP_BatchSaveRequested Or QP_BatchSaveCommitted Then
ErrorDescription = "write or save outcome requires reconciliation; owned project retained without automatic save or discard"
Exit Sub
End If
On Error Resume Next
QPCloseBatchProject QP_OwnedProjectPath, False
ErrorNumber = Err.Number
@@ -932,7 +951,6 @@ Sub QPFinishBatchItem()
If FinalState = "restored_ok" Then
QPMarkBatchMutation Command
QPCloseBatchProject TargetPath, True
QP_BatchSaveCommitted = True
ElseIf Not QP_SessionMode Then
QPCloseBatchProject TargetPath, False
End If
@@ -952,6 +970,7 @@ Sub QPFinishBatchItem()
QPVerifyRestoredPlans Command, Records, Plans, ValueStates
If Not QP_SessionMode Then QPCloseBatchProject TargetPath, False
FinalState = "saved_ok"
QPLog QP_Root, "INFO", "saved_readback_verified job=" & QP_CurrentJobId
If QP_SessionMode Then
QP_BatchMessage = QP_BatchMessage & " Saved and reopened for readback; owned project checkpoint verified."
Else
@@ -66,10 +66,11 @@ class _DesktopLease:
for process in reversed(self.processes):
try:
self.owner._release_process(process, desktop=self.desktop, generation=self.generation)
records.append(getattr(process, 'cleanup_record', {}))
except Exception as error:
errors.append(str(error))
remaining.append(process)
finally:
records.append(getattr(process, 'cleanup_record', {}))
self.processes = list(reversed(remaining))
self.cleanup_record = {'scope': 'operation', 'processes': records,
'workers_closed': not errors, 'desktop_retained_by_batch': True, 'errors': errors}
@@ -99,6 +100,7 @@ class BatchCadRuntime:
self._retry_counts = {}
self._project_attempts = {}
self._logging = None
self._active_evidence = None
def bind_logs(self, context, task_ids):
"""Keep every participating task's final release outcome visible."""
@@ -116,6 +118,14 @@ class BatchCadRuntime:
return self
def __exit__(self, error_type, error, traceback):
if error is not None and self._active_evidence is not None:
attempts = [item for values in self._project_attempts.values() for item in values]
try:
self._retain_uncertain(self._active_evidence, None, attempts, error)
except WorkerMonitorError:
if not getattr(getattr(self.stop_event, 'force_event', None), 'is_set', lambda: False)():
raise
# Explicit force has durable protection; cleanup is now allowed.
try:
self.close()
except Exception as cleanup:
@@ -190,7 +200,8 @@ class BatchCadRuntime:
started = time.monotonic()
self._save(cad_start_requested=True)
cad = factory(self.path.parent / f'generation-{self.generation + (self.desktop is None)}',
abort_event=abort_event, desktop=lease)
abort_event=abort_event, desktop=lease,
**({'stop_control': self.stop_event} if hasattr(self.stop_event, 'force_event') else {}))
self.cad = cad # Retain cleanup ownership even if startup fails.
try:
cad.__enter__()
@@ -247,6 +258,7 @@ class BatchCadRuntime:
from .isolated_cad import PrivateCadStartupError, is_identity_conflict
from .worker_monitor import run_monitored_process
root = canonical_path(command[3])
self._active_evidence = root
while True:
check_cancelled(abort_event or self.stop_event)
attempt_id = uuid4().hex
@@ -272,7 +284,11 @@ class BatchCadRuntime:
def start_worker(launch, **options):
return cad.start_worker(launch, extra_environment=environment, **options)
result = run_monitored_process(command, timeout, abort_event=abort_event,
cooperative_abort=cooperative_abort, process_factory=start_worker)
cooperative_abort=cooperative_abort, process_factory=start_worker,
**({'force_control': self.stop_event} if hasattr(self.stop_event, 'force_event') else {}))
if result.returncode:
self._retain_uncertain(root, project, expected,
WorkerMonitorError(result.stderr or 'worker failed after possible mutation'))
self._checkpoint(root, project, attempt_id)
if result.returncode not in (2, 3):
return result
@@ -294,6 +310,7 @@ class BatchCadRuntime:
and not (abort_event is not None and abort_event.is_set())
and not (self.stop_event is not None and self.stop_event.is_set()))
if not safe:
self._retain_uncertain(root, project, expected, failure)
raise failure
key = str(project).casefold()
self._retry_counts[key] = 1
@@ -302,6 +319,13 @@ class BatchCadRuntime:
self.close(rebuilding=True) # Prove disappearance before creating another.
self.desktop = self.cad = None
def _retain_uncertain(self, root, project, expected, error):
from .batch_attempt import can_retry_attempt
from .stop_control import wait_for_resolution
attempts = self._project_attempts.get(str(project).casefold(), expected)
if attempts and not all(can_retry_attempt(path, record) for path, record in attempts):
wait_for_resolution(self.stop_event, root, error)
def recover_native_failure(self, project, evidence, reason):
"""One read-only recovery, based on durable source and worker proof."""
from .batch_attempt import can_retry_attempt
@@ -387,7 +411,8 @@ class BatchCadRuntime:
cad_record = cad_record if isinstance(cad_record, dict) else {}
self.record['cad_shutdown'] = {key: cad_record[key] for key in (
'state', 'clean', 'cad_exit_code', 'quit_helper_exit_code', 'cleanup_errors',
'shutdown_seconds') if key in cad_record}
'shutdown_seconds', 'orderly_shutdown', 'diagnostic_errors', 'forced_termination',
'forced_root_termination', 'forced_descendant_pids', 'termination_reason') if key in cad_record}
if self.desktop is not None:
try:
self.desktop.close()
@@ -24,27 +24,31 @@ _WRITE_OPERATIONS = frozenset({'execute_task_queue', 'execute_quick_run_queue',
def _mark_interrupted(root, journal, reason):
from .batch_resources import write_record
from .batch_resources import write_record, _valid_record
from .stop_control import protect_interruption
journal.update(state='blocked', outcome='interrupted', error=str(reason))
if journal['operation'] in _WRITE_OPERATIONS:
# A missing final result is not proof of no mutation. Publish the same
# protection the existing recovery UI understands, without overwriting
# any earlier, more specific marker.
journal['business_uncertain'] = True
for task_id in journal['task_ids']:
task_ids = journal['task_ids']
resource_path = Path(root) / 'batch-resources' / journal['batch_id'] / 'session.json'
try:
resource = json.loads(internal_path(resource_path).read_text(encoding='utf-8'))
if (not journal.get('protect_all_targets') and _valid_record(resource, resource_path)
and resource.get('active_task') in task_ids):
task_ids = [resource['active_task']]
except (OSError, ValueError, TypeError):
pass # Without valid active ownership, protect every possible target.
journal['protected_task_ids'] = list(task_ids)
for task_id in task_ids:
allowed = {canonical_path(root), canonical_path(Path(root) / 'tasks' / task_id)}
actual = journal.get('uncertainty_roots', {}).get(task_id)
roots = {canonical_path(actual)} if actual and canonical_path(actual) in allowed else allowed
for evidence in roots:
marker = internal_path(evidence / 'quick-run/uncertain.flag')
marker.parent.mkdir(parents=True, exist_ok=True)
try:
with marker.open('x', encoding='utf-8') as stream:
stream.write('reason=supervisor_interrupted; batch=' + journal['batch_id'] + '; manual reconciliation required\n')
stream.flush()
os.fsync(stream.fileno())
except FileExistsError:
pass
protect_interruption(evidence / 'quick-run',
'reason=supervisor_interrupted; batch=' + journal['batch_id'] + '; manual reconciliation required\n')
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal)
@@ -53,6 +57,61 @@ def _worker_context(root):
_WORKER_ROOT = canonical_path(root)
def _force_owned_supervisor(root, journal, process):
"""Persist uncertainty before terminating the exact process/Job we own."""
journal['stop_mode'] = 'forced'
journal['force_requested_at'] = time.time()
# The running worker can cross a task boundary while evidence is written.
# Protect all possible targets before force; keep their saved results intact.
journal['protect_all_targets'] = True
_mark_interrupted(root, journal, '用户明确强制结束;写入与保存结果须核验,禁止自动重放。')
try:
process.terminate()
except Exception as error:
if not getattr(process, 'cleanup_record', {}).get('forced_termination'):
raise
journal['force_cleanup_error'] = str(error)
try:
from .batch_resources import write_record
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal)
except OSError:
pass # The forced intent and uncertainty were already durably saved.
def _retain_supervisor_after_control_error(root, journal, process, control, error):
"""Do not turn a GUI/transport exception into an unconfirmed Job kill."""
from .batch_resources import write_record
control.set()
control.control_error = '控制通信异常,已请求安全停止并保留应用;可导出诊断或明确强制结束:' + str(error)
journal['protect_all_targets'] = True
try:
_mark_interrupted(root, journal, error)
except Exception as failure:
control.control_error += ';中断证据尚未落盘:' + str(failure)
try:
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'cancel.json',
{'version': VERSION, 'batch_id': journal['batch_id'], 'attempt_id': journal['attempt_id']})
except OSError:
pass # Keep ownership; failure to request stop is not permission to kill.
while True:
if control.force_event.is_set():
try:
_force_owned_supervisor(root, journal, process)
except Exception as failure:
control.force_event.clear()
control.control_error = '强制结束未确认,应用仍保留:' + str(failure)
else:
return True
try:
process.wait(timeout=.2)
return False
except subprocess.TimeoutExpired:
pass
except Exception as failure:
control.control_error = '无法确认应用退出,保留所有权等待处理:' + str(failure)
threading.Event().wait(.2)
def _file_lock(handle, acquire):
from .task_queue import _os_lock
_os_lock(handle, acquire)
@@ -170,6 +229,19 @@ def _finish_response(root, journal, response, context):
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal)
if related:
raise resource_pending_error(journal['operation'], result, error, primary)
resource_path = internal_path(Path(root) / 'batch-resources' / journal['batch_id'] / 'session.json')
try:
resource = json.loads(resource_path.read_text(encoding='utf-8'))
except FileNotFoundError:
resource = {}
shutdown = resource.get('cad_shutdown') or {}
if (shutdown.get('orderly_shutdown') is False or shutdown.get('forced_termination')
or shutdown.get('diagnostic_errors')
or any(item.get('stage') != 'desktop_release' for item in shutdown.get('cleanup_errors', ()))):
failure = TaskCenterError('application_shutdown_failed',
'已核验业务结果保留,但应用未正常收尾;请查看退出诊断,不能将资源消失视为正常退出。')
failure.completed_result, failure.cleanup_error = result, error
raise failure
if response.get('business_succeeded'):
if response.get('error'):
context.logger.info('supervisor_resource_recovery_passed batch_id=%s business_replayed=false', journal['batch_id'])
@@ -186,6 +258,7 @@ def _finish_response(root, journal, response, context):
def run_supervised(service, operation, arguments, *, stop_event=None, on_progress=None, on_phase=None):
from .batch_resources import write_record, process_identity
from .isolated_desktop import _Win32Api
from .stop_control import StopControl
root = canonical_path(service._logging.root)
if stop_event is not None and stop_event.is_set():
raise TaskCenterError('operation_cancelled', '本批操作已取消,未启动监督进程。')
@@ -216,13 +289,15 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres
journal['parent_process'] = process_identity(parent)
write_record(request, {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id,
'root': str(root), 'database': str(canonical_path(service._store.path)),
'operation': operation, 'arguments': encode(arguments), 'task_ids': journal['task_ids']})
'operation': operation, 'arguments': encode(arguments), 'task_ids': journal['task_ids'],
'interactive_stop': isinstance(stop_event, StopControl)})
journal['request_sha256'] = hashlib.sha256(internal_path(request).read_bytes()).hexdigest()
write_record(directory / 'supervisor.json', journal)
gate, messages = MessageGate(batch_id, attempt_id), queue.Queue()
api, process, reader = _Win32Api(), None, None
sent_cancel, last_heartbeat = False, time.monotonic()
transport_error = None
force_accepted = False
def receive():
try:
while line := process.stdout.readline(1024 * 1024):
@@ -242,9 +317,26 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres
reader = threading.Thread(target=receive, daemon=True)
reader.start()
while process.poll() is None or not messages.empty():
force = getattr(stop_event, 'force_event', None)
if force is not None and force.is_set() and process.poll() is None:
try:
_force_owned_supervisor(root, journal, process)
except Exception as error:
force.clear()
stop_event.control_error = '强制结束未执行或未确认;请保留窗口和诊断:' + str(error)
service._logging.logger.error('force_stop_failed batch_id=%s error=%s', batch_id, error)
else:
force_accepted = True
break
if stop_event is not None and stop_event.is_set() and not sent_cancel:
write_record(directory / 'cancel.json', {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id})
sent_cancel = True
try:
write_record(directory / 'cancel.json', {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id})
journal.update(stop_mode='safe', stop_requested_at=time.time())
write_record(directory / 'supervisor.json', journal)
sent_cancel = True
except OSError as error:
if hasattr(stop_event, 'control_error'):
stop_event.control_error = '停止请求尚未成功记录,正在重试:' + str(error)
try:
value = messages.get(timeout=.1)
except queue.Empty:
@@ -264,6 +356,7 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres
state = decode(value['payload'])
service._task_queue_feedback = state['queue']
service._batch_feedback = state['batch']
service._supervisor_waiting = bool(state.get('waiting_for_resolution'))
if on_phase is not None and state['phase'] in {'recycling', 'closing'}:
from .bom_import_state import BomImportPhase
on_phase(BomImportPhase.RECOVERING if state['phase'] == 'recycling' else BomImportPhase.RELEASING,
@@ -271,9 +364,12 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres
process.wait(timeout=15)
except Exception as error:
transport_error = error
if process is not None and isinstance(stop_event, StopControl):
force_accepted = _retain_supervisor_after_control_error(root, journal, process, stop_event, error)
finally:
service._task_queue_feedback = None
service._batch_feedback = None
service._supervisor_waiting = False
try:
if process is not None:
process.close() # Waits, and confines termination to the owned Job.
@@ -285,6 +381,14 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres
'batch_resources_pending', f'监督进程错误:{transport_error};收尾错误:{error}')
if reader is not None:
reader.join(3)
if force_accepted:
from .batch_resources import cleanup_pending_resources
try:
cleanup_pending_resources(root, logging_context=service._logging)
except Exception as error:
service._logging.logger.error('forced_stop_cleanup_pending batch_id=%s error=%s', batch_id, error)
raise TaskCenterError('user_forced_stop',
'已按确认强制结束;已保存的执行证据保留,当前结果待核验。请先核验恢复,不要重复替换。')
try:
try:
response = json.loads(internal_path(directory / 'result.json').read_text(encoding='utf-8'))
@@ -75,7 +75,7 @@ def run(request_path):
request = json.loads(request_bytes)
batch_id, attempt_id = request['batch_id'], request['attempt_id']
if (request.get('version') != VERSION or UUID(batch_id).hex != batch_id or UUID(attempt_id).hex != attempt_id
or request['operation'] not in OPERATIONS):
or request['operation'] not in OPERATIONS or type(request.get('interactive_stop', False)) is not bool):
raise ValueError('Invalid batch supervisor request')
root = canonical_path(request['root'])
directory = root / 'batch-resources' / batch_id
@@ -83,7 +83,9 @@ def run(request_path):
raise ValueError('Supervisor request ownership path mismatch')
_admit_request(directory, request, request_bytes)
_worker_context(root)
stop, finished, output_lock = threading.Event(), threading.Event(), threading.Lock()
from .stop_control import StopControl
stop = StopControl() if request.get('interactive_stop', False) else threading.Event()
finished, output_lock = threading.Event(), threading.Lock()
kind = {'inspect_bom_batch': 'bom_import', 'preview_bom_batch': 'bom_import',
'execute_task_queue': 'task_queue', 'execute_quick_run_queue': 'quick_run_queue'}.get(request['operation'], request['operation'])
runtime = BatchCadRuntime(root, kind, stop, session_id=batch_id)
@@ -113,7 +115,9 @@ def run(request_path):
emit('heartbeat')
feedback = getattr(service, '_task_queue_feedback', None)
emit('state', encode({'phase': runtime.record['state'], 'queue': feedback,
'batch': service.batch_feedback()}))
'batch': service.batch_feedback(),
'waiting_for_resolution': getattr(stop, 'waiting_for_resolution', False),
'resolution_error': getattr(stop, 'resolution_error', '')}))
except Exception:
stop.set() # Lost control evidence cannot authorize new work.
finished.wait(1)
@@ -13,6 +13,7 @@ class ExecutionFeedback:
message: str = '正在准备执行'
stage: str = ''
stage_seconds: float = 0.0
waiting_for_safe_completion: bool = False
def read_feedback(task_id, position, total, message, root: Path, started_after: float):
@@ -27,15 +28,16 @@ def read_feedback(task_id, position, total, message, root: Path, started_after:
except (OSError, ValueError, TypeError, AttributeError):
pass
path = root / 'quick-run' / 'worker-progress.json'
stage, seconds = '', 0.0
stage, seconds, waiting = '', 0.0, False
try:
modified = path.stat().st_mtime
if modified >= started_after:
record = json.loads(path.read_text(encoding='utf-8'))
waiting = record.get('waiting_for_safe_completion') is True
candidate = record.get('stage', '')
if candidate and candidate != 'finished':
seconds = float(record.get('stage_seconds', {}).get(candidate, 0)) + max(0, time.time() - modified)
stage = candidate
except (OSError, ValueError, TypeError, AttributeError):
pass # Display feedback is optional; it never changes execution state.
return ExecutionFeedback(task_id, position, total, message, stage, seconds)
return ExecutionFeedback(task_id, position, total, message, stage, seconds, waiting)
@@ -87,7 +87,7 @@ def _designer_executable():
class IsolatedCadSession:
"""The supervisor never attaches to COM on the user's input desktop."""
def __init__(self, root, *, abort_event=None, desktop=None):
def __init__(self, root, *, abort_event=None, desktop=None, stop_control=None):
self.root = canonical_path(root)
self.token = uuid4().hex
self.borrowed_desktop = desktop is not None
@@ -96,6 +96,7 @@ class IsolatedCadSession:
self.binding = None
self.started = time.monotonic()
self.abort_event = abort_event
self.stop_control = stop_control
self.record = {'desktop': self.desktop.name, 'token': self.token, 'state': 'starting'}
def _record(self, **values):
@@ -206,52 +207,115 @@ class IsolatedCadSession:
def __exit__(self, error_type, error, traceback):
closing_started = time.monotonic()
cleanup_error = None
cleanup_errors = []
diagnostic_errors = []
failures = []
exit_codes = {}
closing = None
def remember(stage, failure):
cleanup_errors.append({'stage': stage, 'message': str(failure)})
message = '; '.join(f"{item['stage']}: {item['message']}" for item in cleanup_errors)
self._uncertain(message, cleanup_errors=cleanup_errors,
failures.append(failure)
def diagnose(stage, action):
try:
action()
except Exception as failure:
diagnostic_errors.append({'stage': stage, 'message': str(failure)})
failures.append(failure)
self.record['clean'] = False
def failure_message():
return '; '.join(f"{item['stage']}: {item['message']}"
for item in cleanup_errors + diagnostic_errors)
def persist_uncertain():
message = failure_message() or str(error)
self.record.update(state='uncertain', clean=False, error=message,
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors,
shutdown_seconds=time.monotonic() - closing_started,
desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes)
diagnose('uncertain_record', lambda: self._uncertain(message,
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors,
clean=False, orderly_shutdown=self.record['orderly_shutdown'],
shutdown_seconds=time.monotonic() - closing_started,
desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes))
self.record.update(clean=False, orderly_shutdown=False, forced_termination=False,
forced_root_termination=False, forced_descendant_pids=[], termination_reason=None,
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors)
diagnose('closing_record', lambda: self._record(state='closing', clean=False,
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors))
try:
self._record(state='closing', cleanup_errors=cleanup_errors)
if error is not None:
self._uncertain(str(error))
if self.cad is not None and self.cad.poll() is not None:
raise WorkerMonitorError('owned CAD exited before the authorized quit request')
if self.cad is not None and self.binding is not None:
closing = self.desktop.start_process(_helper_command('quit', self.root / 'private-desktop.json', self.cad.pid),
env=_child_environment(self.binding.environment()), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
try:
quit_code = closing.wait(timeout=20)
exit_codes['quit_helper_exit_code'] = quit_code
if quit_code:
raise WorkerMonitorError('owned CAD refused orderly exit: ' + closing.stderr.read().decode('utf-8', errors='replace'))
cad_code = self.cad.wait(timeout=15)
exit_codes['cad_exit_code'] = cad_code
if cad_code != 0:
raise WorkerMonitorError('owned CAD returned a nonzero exit code after quit')
finally:
closing.close()
quit_code = closing.wait(timeout=20)
exit_codes['quit_helper_exit_code'] = quit_code
if quit_code:
raise WorkerMonitorError('owned CAD refused orderly exit: ' + closing.stderr.read().decode('utf-8', errors='replace'))
cad_code = self.cad.wait(timeout=15)
exit_codes['cad_exit_code'] = cad_code
if cad_code != 0:
raise WorkerMonitorError('owned CAD returned a nonzero exit code after quit')
self.record['orderly_shutdown'] = True
except Exception as failure:
cleanup_error = failure
remember('orderly_shutdown', failure)
finally:
if error is not None or failures:
# Evidence failure must never prevent the identity-checked Quit
# above or release of the handles we own below.
persist_uncertain()
if self.stop_control is not None and cleanup_errors and self.cad is not None:
cad_running = True
try:
cad_running = self.cad.poll() is None
except Exception as failure:
remember('cad_exit_check', failure)
if cad_running:
# A refused/timed-out Quit may still be saving or closing.
# Keep both CAD and its helper owned and alive until the
# interactive caller confirms force, including ordinary stop.
from .stop_control import wait_for_resolution
try:
wait_for_resolution(self.stop_control, self.root, failure_message())
except WorkerMonitorError as failure:
force_event = getattr(self.stop_control, 'force_event', None)
if force_event is None or not force_event.is_set():
raise
remember('forced_stop', failure)
if closing is not None:
try:
closing.close()
except Exception as failure:
remember('quit_helper_release', failure)
try:
self.desktop.close()
self._record(state='session_closed' if self.borrowed_desktop else 'released',
desktop_retained_by_batch=self.borrowed_desktop,
clean=error is None and cleanup_error is None,
shutdown_seconds=time.monotonic() - closing_started,
desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes)
except Exception as failure:
cleanup_error = failure
remember('desktop_release', failure)
if cleanup_error is not None:
message = '; '.join(f"{item['stage']}: {item['message']}" for item in cleanup_errors)
raise WorkerMonitorError('private CAD cleanup requires reconciliation: ' + message) from cleanup_error
desktop_cleanup = getattr(self.desktop, 'cleanup_record', None) or {}
forced = [item for item in desktop_cleanup.get('processes', []) if item.get('forced_termination')]
self.record.update(forced_termination=bool(forced),
forced_root_termination=any(item.get('forced_root_termination') for item in forced),
termination_reason=next((item.get('termination_reason') for item in forced
if item.get('termination_reason')), None),
forced_descendant_pids=sorted({pid for item in forced for pid in item.get('forced_descendant_pids', [])}))
if forced:
remember('forced_cleanup', WorkerMonitorError('owned processes required forced termination'))
if error is not None or failures:
persist_uncertain()
else:
diagnose('released_record', lambda: self._record(
state='session_closed' if self.borrowed_desktop else 'released',
desktop_retained_by_batch=self.borrowed_desktop, clean=True,
shutdown_seconds=time.monotonic() - closing_started,
desktop_cleanup=desktop_cleanup, **exit_codes))
if failures:
persist_uncertain()
if failures:
raise WorkerMonitorError('private CAD cleanup requires reconciliation: ' + failure_message()) from failures[0]
def _uncertain(self, message, **details):
from .quick_run import _write_uncertain_flag
@@ -23,6 +23,7 @@ from .errors import TaskCenterError
DESKTOP_RELEASE_TIMEOUT_SECONDS = 5.0
DESCENDANT_EXIT_GRACE_SECONDS = 2.0
def _native_handle(handle):
@@ -192,7 +193,10 @@ def _stop_and_wait(process):
errors.append(error)
if running:
try:
process.terminate()
if isinstance(process, _Win32Process):
process.terminate(reason='root_process_active')
else:
process.terminate()
except Exception as error:
errors.append(error)
try:
@@ -313,7 +317,9 @@ class _Win32Process:
self.job_assigned = False
self.returncode = None
self.stdout = self.stderr = None
self.cleanup_record = {'pid': self.pid}
self.cleanup_record = {'pid': self.pid, 'forced_termination': False,
'forced_root_termination': False, 'forced_descendant_pids': [],
'termination_reason': None}
def poll(self):
if self.returncode is not None or self.handle is None:
@@ -340,34 +346,59 @@ class _Win32Process:
self.returncode = self._process_api.GetExitCodeProcess(self.handle)
return self.returncode
def terminate(self):
def terminate(self, *, reason='explicit_termination'):
if self.handle is None:
return
errors = []
root_active = True
try:
root_active = self.poll() is None
except Exception as error:
errors.append(error)
self.cleanup_record.setdefault('root_exit_before_cleanup', not root_active)
if not self.job_assigned:
self._process_api.TerminateProcess(self.handle, 1)
self._record_termination(reason, root_active, ())
_raise_cleanup_errors(errors)
return
errors = []
descendants = set()
try:
self._remember_descendants()
# Keep confirmed members of this snapshot even if opening a later
# member fails; the Job termination below still includes them.
self._remember_descendants(active=descendants)
except Exception as error:
errors.append(error)
try:
self._job_api.TerminateJobObject(self.job, 1)
self._record_termination(reason, root_active, descendants)
except Exception:
self._process_api.TerminateProcess(self.handle, 1)
self._record_termination(reason, root_active, ())
_raise_cleanup_errors(errors)
kill = terminate
def _remember_descendants(self):
def _record_termination(self, reason, root_active, descendants):
self.cleanup_record.update(forced_termination=True,
forced_root_termination=self.cleanup_record['forced_root_termination'] or root_active,
forced_descendant_pids=sorted(set(self.cleanup_record['forced_descendant_pids']) | set(descendants)),
termination_reason=self.cleanup_record['termination_reason'] or reason)
def _remember_descendants(self, *, active=None):
if active is None:
active = set()
if self._open_process is None or self.job is None:
return
return active
for pid in self._job_api.QueryInformationJobObject(self.job, self._job_api.JobObjectBasicProcessIdList):
if pid == self.pid or pid in self._descendant_pids:
if pid == self.pid:
continue
if pid in self._descendant_pids:
active.add(pid)
continue
try:
self._descendant_handles.append(self._open_process(0x100000, False, pid))
self._descendant_pids.add(pid)
active.add(pid)
except Exception as error:
# pywintypes.error is not an OSError. A child that exited
# between the job snapshot and OpenProcess reports error 87.
@@ -376,22 +407,47 @@ class _Win32Process:
code = error.args[0]
if code != 87:
raise
return active
def close(self):
errors = []
job_empty = not self.job_assigned or self.job is None
if self.handle is not None:
if 'root_exit_before_cleanup' not in self.cleanup_record:
try:
self.cleanup_record['root_exit_before_cleanup'] = self.poll() is not None
except Exception as error:
errors.append(error)
errors.extend(_stop_and_wait(self))
deadline = time.monotonic() + 5.0
if self.job is not None and self.job_assigned:
try:
# A completed root can leave grandchildren holding the desktop.
self._remember_descendants()
self._job_api.TerminateJobObject(self.job, 1)
grace_started = time.monotonic()
grace_deadline = grace_started + DESCENDANT_EXIT_GRACE_SECONDS
# A completed root may leave children finishing normal shutdown.
# An empty job takes no grace delay; an active root was already
# stopped above and does not receive this normal-exit allowance.
while True:
active = self._job_api.QueryInformationJobObject(
self.job, self._job_api.JobObjectBasicAccountingInformation)['ActiveProcesses']
self.cleanup_record['job_active_processes'] = active
if not active:
job_empty = True
break
self._remember_descendants()
if (self.cleanup_record.get('root_exit_before_cleanup') is not True
or self.cleanup_record['forced_termination'] or time.monotonic() >= grace_deadline):
self.terminate(reason='descendant_grace_expired')
break
time.sleep(min(0.01, max(0, grace_deadline - time.monotonic())))
self.cleanup_record['descendant_grace_seconds'] = time.monotonic() - grace_started
deadline = time.monotonic() + 5.0
while True:
active = self._job_api.QueryInformationJobObject(
self.job, self._job_api.JobObjectBasicAccountingInformation)['ActiveProcesses']
self.cleanup_record['job_active_processes'] = active
if not active:
job_empty = True
break
if time.monotonic() >= deadline:
raise TimeoutError('owned job descendants did not exit within 5 seconds')
@@ -417,6 +473,10 @@ class _Win32Process:
self._descendant_handles = retained
for name in ('job', 'thread', 'handle'):
handle = getattr(self, name)
if name in ('job', 'handle') and not job_empty:
# Closing a KILL_ON_JOB_CLOSE job would silently force remaining
# children and discard the capability needed for cleanup retry.
continue
if name == 'handle' and self.returncode is None:
# Keep the only termination/wait capability until cleanup can retry.
continue
@@ -111,6 +111,13 @@ class ProjectSessionWorker:
return
if not self._done.is_set():
_write_control(self.root / 'project-session-next.tsv', f'{self.token}\tclose')
from .stop_control import StopControl
interactive = isinstance(getattr(self.runtime, 'stop_event', None), StopControl)
if interactive:
# The monitor owns the safe/unsafe distinction. Do not
# turn a GUI close or ordinary pause into a timed kill.
while not self._done.wait(.1):
pass
close_grace = 60 if self.private_desktop else min(30, max(10, self._timeout))
if not self._done.wait(close_grace):
self._abort.set()
@@ -925,6 +925,9 @@ class TaskCenterService:
message = batch[2] if batch else '正在准备执行'
value = read_feedback(task_id, position, total, message,
child._quick_run_preparer.data_root, started_after)
if getattr(self, '_supervisor_waiting', False):
from dataclasses import replace
value = replace(value, stage='awaiting_reconciliation', waiting_for_safe_completion=True)
if read_control is not None:
read_control.checkpoint()
return value
@@ -951,12 +954,12 @@ class TaskCenterService:
return delegated.has_interrupted_quick_run(task_id)
return self._quick_run_preparer.has_interrupted(task_id)
def recover_quick_run(self, task_id: str, *, confirmed: bool = False, runtime=None) -> QuickRunStepResult:
def recover_quick_run(self, task_id: str, *, confirmed: bool = False, runtime=None, stop_event=None) -> QuickRunStepResult:
if runtime is None and self._supervised:
return self._supervise('recover_quick_run', {'task_id': task_id, 'confirmed': confirmed})
return self._supervise('recover_quick_run', {'task_id': task_id, 'confirmed': confirmed}, stop_event=stop_event)
if runtime is None:
return self._owned_cad_operation('recover_quick_run', lambda owned:
self.recover_quick_run(task_id, confirmed=confirmed, runtime=owned), task_id=task_id)
self.recover_quick_run(task_id, confirmed=confirmed, runtime=owned), stop_event, task_id=task_id)
delegated = self._project_task_service(task_id)
if delegated is not self:
return delegated.recover_quick_run(task_id, confirmed=confirmed, runtime=runtime)
@@ -0,0 +1,78 @@
"""Separate ordinary stop intent from an explicitly confirmed forced interruption."""
from threading import Event
import json
import os
from .path_access import internal_path
class StopControl(Event):
def __init__(self):
super().__init__()
self.force_event = Event()
self.control_error = ''
self.waiting_for_resolution = False
self.resolution_error = ''
def request_force(self):
self.set()
self.control_error = ''
self.force_event.set()
def protect_interruption(root, reason):
"""Create or re-sync a marker without overwriting prior protection evidence."""
root = internal_path(root)
root.mkdir(parents=True, exist_ok=True)
marker = root / 'uncertain.flag'
try:
stream = marker.open('x', encoding='utf-8')
except FileExistsError:
# A previous create/write/fsync may have failed. Existence alone does
# not establish durable evidence; preserve its bytes but sync again.
with marker.open('r+b') as existing:
os.fsync(existing.fileno())
else:
with stream:
stream.write(str(reason))
stream.flush()
os.fsync(stream.fileno())
def wait_for_resolution(control, root, error):
"""Interactive owners retain a possibly changed project until explicit force.
The GUI-side supervisor can also terminate its own Job after durable force
evidence. Ordinary cancellation never releases this wait.
"""
if not isinstance(control, StopControl):
return
from .worker_monitor import WorkerMonitorError, _publish_progress
root = internal_path(root)
control.set()
control.waiting_for_resolution = True
control.resolution_error = str(error)
def protect():
protect_interruption(root, 'operation_interrupted: ' + str(error))
try:
protect()
except OSError as failure:
control.control_error = '保护记录写入失败,保留应用等待处理:' + str(failure)
try:
record = json.loads((root / 'worker-progress.json').read_text(encoding='utf-8'))
if not isinstance(record, dict):
record = {}
except (OSError, ValueError):
record = {}
record.update(stage='awaiting_reconciliation', waiting_for_safe_completion=True,
resolution_error=str(error), stop_requested=True, safe_stop=False)
_publish_progress(root / 'worker-progress.json', record)
while True:
if control.force_event.wait(.1):
try:
protect()
except OSError as failure:
control.force_event.clear()
control.control_error = '强制结束未执行,保护记录无法保存:' + str(failure)
continue
raise WorkerMonitorError('user forced stop of uncertain operation; reconciliation required')
@@ -15,6 +15,7 @@ STAGES = {
'restore_attribute_fields': '恢复属性字段', 'restore_attributes': '恢复属性',
'verify_restoration': '核验恢复结果', 'verify_attribute_fields': '核验属性字段',
'save_and_close': '保存并关闭工程', 'verify_saved_project': '回读已保存工程',
'awaiting_reconciliation': '当前结果待核验,已保留工程现场',
}
@@ -22,4 +23,6 @@ def format_feedback(value, task_name, elapsed, stopping):
stage = STAGES.get(value.stage, '')
details = f'{stage}({value.stage_seconds:.1f} 秒)' if stage else value.message
prefix = '等待当前步骤保存核验后暂停 · ' if stopping else ''
if value.waiting_for_safe_completion:
prefix = '当前保存或核验尚未确认,未自动结束应用;可继续等待、导出诊断,或确认强制结束后核验 · '
return f'{prefix}任务 {value.position}/{value.total}:{task_name} · {details} · 总计 {elapsed} 秒'
@@ -1,6 +1,7 @@
"""Single-project task presentation; persistence and CAD remain in the service."""
from __future__ import annotations
from threading import Event
from ..stop_control import StopControl
from shiboken6 import isValid
import time
from PySide6.QtCore import QElapsedTimer, QSettings, QSignalBlocker, QThreadPool, QTimer, Qt
@@ -39,6 +40,7 @@ class TrialTaskCenterWindow(QMainWindow):
self._tasks = ()
self._known_task_groups = {}
self._workers = set()
self._diagnostic_worker = None
self._closed = False
self._batch_stop = None
self._active_quick_run_task_id = None
@@ -88,6 +90,7 @@ class TrialTaskCenterWindow(QMainWindow):
self._log_presenter = TaskLogPresenter(service, self.run_panel, self)
self.run_panel.quick_run_button.clicked.connect(self._prepare_quick_run)
self.run_panel.stop_queue_button.clicked.connect(self._pause_queue)
self.run_panel.force_stop_button.clicked.connect(self._force_batch_stop)
self.run_panel.retry_button.clicked.connect(self._retry_failed)
self.run_panel.recover_button.clicked.connect(self._recover_quick_run)
self.run_panel.cleanup_resources_button.clicked.connect(self._cleanup_batch_resources)
@@ -337,6 +340,10 @@ class TrialTaskCenterWindow(QMainWindow):
self.run_panel.history_button.setEnabled(bool(context and not busy))
self.run_panel.stop_queue_button.setVisible(self._batch_stop is not None)
self.run_panel.stop_queue_button.setEnabled(bool(self._batch_stop and not self._batch_stop.is_set()))
self.run_panel.force_stop_button.setVisible(self._batch_stop is not None)
self.run_panel.force_stop_button.setEnabled(bool(self._batch_stop and
not self._batch_stop.force_event.is_set()))
self.run_panel.diagnostics_button.setEnabled(self._diagnostic_worker is None)
self.run_panel.copy_value_check.setEnabled(state.editable)
rule_editable = state.editable and not group_unsaved
self.mapping_table.set_rules_editable(rule_editable)
@@ -571,6 +578,8 @@ class TrialTaskCenterWindow(QMainWindow):
self._finish_run()
self._metadata_changed(None)
failures = sum(e.state in {'failed', 'blocked'} for e in result.entries)
if failures:
self._close_pending = False
self._show_task_log(force=True)
self._set_operator_message(f'本批任务已暂停。失败/阻断 {failures} 项。' if result.paused else f'本批任务已结束。失败 {failures} 项。',
'warning' if result.paused or failures else 'success')
@@ -578,7 +587,7 @@ class TrialTaskCenterWindow(QMainWindow):
self.task_list.queue_button)
def _begin_run(self, task_id):
self._batch_stop = Event()
self._batch_stop = StopControl()
self._active_quick_run_task_id = task_id
self._run_started_at = time.time()
self._quick_run_elapsed.start()
@@ -601,6 +610,24 @@ class TrialTaskCenterWindow(QMainWindow):
self._set_operator_message('已请求暂停;当前项安全收尾后停止。', 'warning')
self._update_gates()
def _force_batch_stop(self):
control = self._batch_stop
if control is None or control.force_event.is_set():
return
if QMessageBox.question(self, '强制结束当前操作',
'当前操作可能正在修改或保存工程。强制结束可能留下未保存或部分保存的结果。\n\n'
'已完成的执行证据会保留;中断后必须先核验恢复,不能直接重复替换。'
'建议先导出诊断包。确定强制结束本程序当前操作吗?',
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
QMessageBox.StandardButton.No) != QMessageBox.StandardButton.Yes:
return
if self._batch_stop is not control:
return # Completion/new run while the confirmation dialog was open.
self._close_pending = False
control.request_force()
self._set_operator_message('已请求强制结束;正在记录中断证据,结果将进入待核验。', 'warning')
self._update_gates()
def _show_batch_result(self, result):
owner = self._active_quick_run_task_id or self._task_id
name = next((t.name for t in self._tasks if t.task_id == owner), owner)
@@ -609,6 +636,8 @@ class TrialTaskCenterWindow(QMainWindow):
if owner == self._task_id:
self._show_task_log(force=True)
p = result.progress
if p.failed_items:
self._close_pending = False
self._set_operator_message(f"任务「{name}」{'已暂停' if result.paused else '已结束'}:已保存 {p.saved_items},失败 {p.failed_items},未命中 {p.no_match_items}。",
'warning' if result.paused or p.failed_items else 'success')
@@ -637,10 +666,31 @@ class TrialTaskCenterWindow(QMainWindow):
from pathlib import Path
name = 'DxPartReplace-诊断-' + time.strftime('%Y%m%d-%H%M%S') + '.zip'
value, _ = QFileDialog.getSaveFileName(self, '导出脱敏诊断包', name, 'ZIP 文件 (*.zip)')
if value:
self._run_async(lambda: self.service.export_diagnostics(Path(value)),
lambda path: self._set_operator_message(f'脱敏诊断包已导出:{path}', 'success'),
self.run_panel.diagnostics_button)
if not value or self._diagnostic_worker is not None:
return
# A read-only support export remains available while CAD is waiting.
worker = ServiceWorker(lambda: self.service.export_diagnostics(Path(value)))
self._diagnostic_worker = worker
self._workers.add(worker)
def exported(path):
if not self._closed:
self._set_operator_message(f'脱敏诊断包已导出:{path}', 'success')
def failed(error):
self._close_pending = False
if not self._closed:
self._show_error(error)
def finished():
self._workers.discard(worker)
self._diagnostic_worker = None
if not self._closed:
self._update_gates()
if self._close_pending and not self._workers:
QTimer.singleShot(0, self.close)
worker.signals.succeeded.connect(exported)
worker.signals.failed.connect(failed)
worker.signals.finished.connect(finished)
self._update_gates()
QThreadPool.globalInstance().start(worker)
def _retry_failed(self):
if self.run_panel.retry_button.isEnabled():
@@ -700,7 +750,8 @@ class TrialTaskCenterWindow(QMainWindow):
self._finish_run()
self._metadata_changed(None)
self._set_operator_message(result.message)
self._run_async(lambda: self.service.recover_quick_run(task_id, confirmed=True), done,
stop = self._batch_stop
self._run_async(lambda: self.service.recover_quick_run(task_id, confirmed=True, stop_event=stop), done,
self.run_panel.recover_button, expected_task_id=task_id)
def _confirm_quick_run(self, scope, selected_projects, enabled_rules):
@@ -725,6 +776,9 @@ class TrialTaskCenterWindow(QMainWindow):
self.run_panel.execution_status.setText(format_feedback(
value, name, self._quick_run_elapsed.elapsed() // 1000,
bool(self._batch_stop and self._batch_stop.is_set())))
if self._batch_stop and self._batch_stop.control_error:
self._set_operator_message(self._batch_stop.control_error, 'warning')
self._update_gates()
self._feedback_query.request(lambda control: self.service.execution_feedback(task_id, started_after=started, read_control=control),
received, lambda _: None, cancellable=True)
if self.run_panel.toggle_log_button.isChecked():
@@ -799,6 +853,7 @@ class TrialTaskCenterWindow(QMainWindow):
callback(result)
except Exception as error:
operation_error = error
self._close_pending = False
self.operation_state = self.operation_state.finish(error)
self._show_error(error)
def failed(error):
@@ -4,6 +4,8 @@ import json
import re
ERROR_TEXT = {
'user_forced_stop': '操作已强制中断,结果待核验。',
'application_shutdown_failed': '业务结果已保留,但应用收尾异常。',
"batch_resources_pending": "后台资源尚未确认释放,请使用“重新清理资源”后再开始下一批。",
"supervisor_busy": "另一批操作正在执行,请先安全收尾。",
"supervisor_interrupted": "批次监督连接异常,保护记录已保留,没有自动重放替换。",
@@ -23,6 +23,8 @@ class RunPanel(QWidget):
self.busy_indicator.hide()
self.quick_run_button = QPushButton('执行当前任务', objectName='quickRunButton')
self.stop_queue_button = QPushButton('完成当前项后暂停', objectName='stopQueueButton')
self.force_stop_button = QPushButton('强制结束…', objectName='forceStopButton')
self.force_stop_button.setToolTip('异常情况下使用;会中断当前操作,结果必须重新核验。')
self.retry_button = QPushButton('重试失败项', objectName='retryTaskItems')
self.recover_button = QPushButton('核验并恢复', objectName='recoverQuickRunButton')
self.cleanup_resources_button = QPushButton('重新清理资源', objectName='cleanupBatchResources')
@@ -42,7 +44,7 @@ class RunPanel(QWidget):
self.toggle_log_button.toggled.connect(self._toggle_log)
buttons = QHBoxLayout()
for button in (self.quick_run_button, self.retry_button, self.recover_button, self.cleanup_resources_button, self.diagnostics_button,
self.new_round_button, self.history_button, self.stop_queue_button):
self.new_round_button, self.history_button, self.stop_queue_button, self.force_stop_button):
buttons.addWidget(button)
buttons.addStretch()
buttons.addWidget(self.toggle_log_button)
@@ -35,6 +35,7 @@ class ProgressWatch:
self.done = 0
self.total = 0
self.safe_stop = True
self.waiting_for_safe_completion = False
self.started = started
self.stage_started = started
self.stage_seconds: dict[str, float] = {}
@@ -67,6 +68,7 @@ class ProgressWatch:
self.native_sequence = 0
self.last_progress = now
self.stage, self.done, self.total, self.safe_stop = fields[4], done, total, bool(safe)
self.waiting_for_safe_completion = False
return True
def accept_native(self, line: bytes, now: float) -> bool:
@@ -102,6 +104,11 @@ class ProgressWatch:
self.stop_requested = True
return 'request_stop'
if self.stop_requested and idle >= self.idle_timeout * 2:
if not self.safe_stop:
if not self.waiting_for_safe_completion:
self.waiting_for_safe_completion = True
return 'await_safe_completion'
return None
return 'unresponsive'
return None
@@ -109,7 +116,7 @@ class ProgressWatch:
now = time.monotonic() if now is None else now
stages = dict(self.stage_seconds)
stages[self.stage] = stages.get(self.stage, 0.0) + max(0.0, now - self.stage_started)
return {name: getattr(self, name) for name in ('token', 'sequence', 'stage', 'done', 'total', 'safe_stop', 'stop_requested')} | {
return {name: getattr(self, name) for name in ('token', 'sequence', 'stage', 'done', 'total', 'safe_stop', 'stop_requested', 'waiting_for_safe_completion')} | {
'job_id': self.current_job_id,
'native_progress_frames': self.native_progress_frames,
'elapsed_seconds': max(0.0, now - self.started), 'stage_seconds': stages}
@@ -124,9 +131,8 @@ def _atomic_text(path: Path, text: str) -> None:
def _publish_progress(path: Path, snapshot: dict) -> None:
try:
_atomic_text(path, json.dumps(snapshot, ensure_ascii=False))
except OSError as error:
if not isinstance(error, PermissionError) and getattr(error, 'winerror', None) not in {5, 32, 33}:
raise
except OSError:
pass # Display telemetry failure must not terminate a possible CAD write.
def _decode(data: bytes) -> str:
@@ -190,7 +196,7 @@ class _OwnedWorkers:
def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abort_event=None,
process_factory=None, cooperative_abort=False) -> subprocess.CompletedProcess[str]:
process_factory=None, cooperative_abort=False, force_event=None, force_control=None) -> subprocess.CompletedProcess[str]:
root = Path(command[3]).resolve()
job_id = command[-1]
token = uuid4().hex
@@ -200,8 +206,11 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
native_progress_path = root / f'monitor-{token}.native-progress'
root.mkdir(parents=True, exist_ok=True)
watch = ProgressWatch(token, job_id, idle_timeout, time.monotonic())
if force_control is not None:
force_event = force_control.force_event
owned = _OwnedWorkers(token, {command[0], command[4]})
process = None
durable_force_accepted = False
threads = []
messages = queue.Queue(maxsize=1024)
stop_readers = threading.Event()
@@ -244,15 +253,32 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
_publish_progress(progress_path, watch.snapshot())
_atomic_text(ready, token)
while process.poll() is None:
if force_event is not None and force_event.is_set():
from .stop_control import protect_interruption
try:
protect_interruption(root, 'user_forced_stop')
except OSError as error:
force_event.clear()
if force_control is not None:
force_control.control_error = '强制结束未执行,保护记录无法保存:' + str(error)
watch.waiting_for_safe_completion = True
_publish_progress(progress_path, watch.snapshot())
else:
durable_force_accepted = True
raise WorkerMonitorError('user forced stop; execution requires reconciliation')
if abort_event is not None and abort_event.is_set():
if not cooperative_abort:
if not cooperative_abort and watch.safe_stop:
raise WorkerMonitorError('project session supervisor requested worker cleanup')
if not watch.stop_requested:
# Read-only callers request the VBS safe boundary first.
# The existing idle watchdog still bounds a stalled COM call.
watch.stop_requested = True
watch.last_progress = time.monotonic()
_atomic_text(stop, token)
try:
_atomic_text(stop, token)
except OSError:
if watch.safe_stop:
raise
_publish_progress(progress_path, watch.snapshot())
try:
receive(*messages.get(timeout=min(0.1, idle_timeout / 10)))
@@ -273,7 +299,17 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
break
action = watch.action(time.monotonic())
if action == 'request_stop':
_atomic_text(stop, token)
try:
_atomic_text(stop, token)
except OSError:
if watch.safe_stop:
raise
_publish_progress(progress_path, watch.snapshot())
elif action == 'await_safe_completion':
if cooperative_abort:
# This entry point is explicitly read-only. A blocked COM
# read has no write to finish and retains its bounded cancel.
raise subprocess.TimeoutExpired(command, idle_timeout * 2)
_publish_progress(progress_path, watch.snapshot())
elif action == 'unresponsive':
raise subprocess.TimeoutExpired(command, idle_timeout * 2)
@@ -287,6 +323,15 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
return subprocess.CompletedProcess(command, process.returncode, _decode(output[0]), _decode(output[1]))
except Exception as error:
if process is not None:
if (not watch.safe_stop and not cooperative_abort and force_control is not None
and not durable_force_accepted):
from .stop_control import wait_for_resolution
try:
wait_for_resolution(force_control, root, error)
except WorkerMonitorError:
if not force_event.is_set():
raise
durable_force_accepted = True
failures = owned.stop_workers(process)
process.wait(timeout=10)
for thread in threads:
@@ -175,13 +175,13 @@ Set QP_App=FakeApp
FakeApp.ProjectPath=Target
QP_CurrentJobId="batch-job": QP_ExpectedJobId="batch-job"
QPWriteUtf8 QP_Fso.BuildPath(QP_Root,"command.tsv"),"task_id" & vbTab & "job_id" & vbTab & "project_path" & vbTab & "old_part" & vbTab & "new_part" & vbTab & "value_mode" & vbCrLf & "task" & vbTab & "batch-job" & vbTab & Target & vbTab & "OLD" & vbTab & "NEW" & vbTab & "preserve" & vbCrLf
QP_BatchMode=True: QP_OwnedProject=True
QP_BatchMode=True: QP_OwnedProject=True: QP_OwnedProjectPath=Target
QPWriteStatus QP_Root,QP_CurrentJobId,"no_match","","not found"
Require Not QP_Fso.FileExists(QP_Fso.BuildPath(QP_Root,"status.tsv")),"published success before close"
Require QPReadState(QP_Fso.BuildPath(QP_Root,"status.tsv"),QP_CurrentJobId)="no_match","pending no-match not visible to worker"
QPFinishBatchItem
Require FakeApp.CloseCount=1,"project not closed exactly once"
Require FakeApp.SaveCount=1,"sheet not closed with saving allowed"
Require FakeApp.SaveCount=0,"no-match must close without saving"
Require InStr(QPReadUtf8(QP_Fso.BuildPath(QP_Root,"status.tsv")),"batch_closed")>0,"no completion proof"
''')
@@ -368,8 +368,8 @@ Require FakeApp.ProjectPath = "", "discovery left the project open"
Require FakeApp.Documents.OpenCount = 0, "discovery opened a page"
Dim Manifest
Manifest = QPReadUtf8(QP_Fso.BuildPath(QP_Root, "sheets.tsv"))
Require InStr(Manifest, Target & vbTab & "Main" & vbTab & "1") > 0, "sheet one missing"
Require InStr(Manifest, Target & vbTab & "Main" & vbTab & "2") > 0, "sheet two missing"
Require InStr(Manifest, LCase(Target) & vbTab & "Main" & vbTab & "1") > 0, "sheet one missing"
Require InStr(Manifest, LCase(Target) & vbTab & "Main" & vbTab & "2") > 0, "sheet two missing"
''')
@@ -253,6 +253,31 @@ End Sub''', routines)
'Sub QPVerifyUntouched(ByRef Records)\n' + injection)
def test_save_then_project_close_refusal_retains_save_boundary(tmp_path):
run_vbs(tmp_path, 'part_number', '''
Dim FullChecks
FullChecks=0
Command.Add "project_path",QPProjectPath()
QP_BatchMode=True: QP_SessionMode=True
QP_SessionToken=String(32,"b")
QP_OwnedProject=True: QP_OwnedProjectPath=QPProjectPath()
QPRestoreAfterOperatorReplace Command,SnapshotPath
QP_App.RefuseClose=True
On Error Resume Next
QPFinishBatchItem
Failure=Err.Number
Err.Clear
On Error GoTo 0
Require Failure<>0,"refused close must stop the operation"
Require QP_BatchSaveCommitted,"returned save was lost when project close failed"
QP_App.RefuseClose=False
QPDiscardFailedBatchProject CloseError
Require Len(CloseError)>0,"uncertain save must not discard the open project"
Require Len(QPProjectPath())>0,"failed saved project was silently discarded"
Require Not QP_Fso.FileExists(QP_Fso.BuildPath(QP_Root,"project-session-result.tsv")),"failure published success"
''', routines_transform=_batch_readback_fixture)
@pytest.mark.parametrize('fault', ['', 'before_save', 'after_reopen'])
def test_owned_batch_checks_full_state_at_save_and_disk_boundaries(tmp_path, fault):
run_vbs(tmp_path, 'part_number', '''
+117
View File
@@ -1,10 +1,12 @@
import io
import json
import os
import subprocess
import weakref
from pathlib import Path
from threading import Event
from threading import Timer
from threading import Thread
from types import SimpleNamespace
from traceback import extract_tb
@@ -194,6 +196,121 @@ def test_clean_release_records_helper_and_cad_exit_codes(monkeypatch, tmp_path):
assert record['cleanup_errors'] == []
@pytest.mark.parametrize('persistent', [False, True])
def test_closing_journal_failure_still_quits_and_reports_diagnostic_error(monkeypatch, tmp_path, persistent):
events, cad = fake_desktop(monkeypatch, tmp_path)
session = runtime.IsolatedCadSession(tmp_path)
session.__enter__()
record = session._record
def failing_record(**values):
if persistent or values.get('state') == 'closing':
raise PermissionError('closing journal locked')
record(**values)
session._record = failing_record
with pytest.raises(WorkerMonitorError, match='closing journal locked'):
session.__exit__(None, None, None)
assert 'quit' in events and cad.returncode == 0
assert session.record['orderly_shutdown'] is True
assert session.record['clean'] is False
assert session.record['diagnostic_errors'][0]['stage'] == 'closing_record'
assert (tmp_path / 'uncertain.flag').exists()
@pytest.mark.parametrize('helper_failure', ['refused', 'timeout'])
def test_interactive_quit_failure_retains_cad_and_helper_until_explicit_force(monkeypatch, tmp_path, helper_failure):
from dx_productivity.trial_task_center.stop_control import StopControl
from dx_productivity.trial_task_center import worker_monitor
events, cad = fake_desktop(monkeypatch, tmp_path, quit_code=1, cad_exit_code=None)
control = StopControl()
session = runtime.IsolatedCadSession(tmp_path, stop_control=control)
session.__enter__()
start = session.desktop.start_process
def start_quit(command, **options):
helper = start(command, **options)
if command[0] == 'quit':
if helper_failure == 'timeout':
helper.wait = lambda timeout: (_ for _ in ()).throw(subprocess.TimeoutExpired('quit', timeout))
helper.poll = lambda: None
helper.close = lambda: events.append('close-quit-helper')
return helper
session.desktop.start_process = start_quit
waiting, finished = Event(), Event()
publish = worker_monitor._publish_progress
def observe_wait(path, snapshot):
publish(path, snapshot)
if snapshot.get('stage') == 'awaiting_reconciliation':
waiting.set()
monkeypatch.setattr(worker_monitor, '_publish_progress', observe_wait)
failures = []
def close_session():
try:
session.__exit__(None, None, None)
except Exception as failure:
failures.append(failure)
finally:
finished.set()
thread = Thread(target=close_session)
thread.start()
try:
assert waiting.wait(1), 'failed Quit must enter protected waiting before resource release'
control.set()
assert not finished.wait(.05)
assert cad.poll() is None
assert 'close-quit-helper' not in events
assert not any(isinstance(event, tuple) and event[0] == 'release' for event in events)
assert (tmp_path / 'uncertain.flag').exists()
control.request_force()
assert finished.wait(2)
assert 'close-quit-helper' in events and ('release', True) in events
assert cad.poll() == 1
assert isinstance(failures[0], WorkerMonitorError)
assert 'forced' in str(failures[0])
assert session.record['orderly_shutdown'] is False and session.record['clean'] is False
finally:
control.request_force()
thread.join(3)
@pytest.mark.parametrize('failure', ['cad_exited', 'diagnostic'])
def test_interactive_exit_does_not_hold_completed_cad_for_resolution(monkeypatch, tmp_path, failure):
from dx_productivity.trial_task_center.stop_control import StopControl
from dx_productivity.trial_task_center import stop_control
events, _cad = fake_desktop(monkeypatch, tmp_path, quit_code=1 if failure == 'cad_exited' else 0)
session = runtime.IsolatedCadSession(tmp_path, stop_control=StopControl())
session.__enter__()
monkeypatch.setattr(stop_control, 'wait_for_resolution', lambda *_args: pytest.fail('exited CAD must not wait'))
if failure == 'diagnostic':
record = session._record
def failed_journal(**values):
if values.get('state') == 'closing':
raise PermissionError('closing diagnostic unavailable')
record(**values)
session._record = failed_journal
with pytest.raises(WorkerMonitorError):
session.__exit__(None, None, None)
assert ('release', True) in events
def test_forced_descendant_cleanup_is_not_reported_as_clean_orderly_shutdown(monkeypatch, tmp_path):
fake_desktop(monkeypatch, tmp_path)
session = runtime.IsolatedCadSession(tmp_path)
session.__enter__()
release = session.desktop.close
def forced_release():
release()
session.desktop.cleanup_record = {'processes': [{'pid': 123, 'forced_termination': True,
'forced_root_termination': False, 'termination_reason': 'descendant_grace_expired',
'forced_descendant_pids': [456]}]}
session.desktop.close = forced_release
with pytest.raises(WorkerMonitorError, match='forced'):
session.__exit__(None, None, None)
record = json.loads((tmp_path / 'private-desktop.json').read_text())
assert record['clean'] is False and record['forced_termination'] is True
assert record['orderly_shutdown'] is True
assert record['forced_descendant_pids'] == [456]
assert record['termination_reason'] == 'descendant_grace_expired'
def test_exit_code_diagnostics_do_not_interrupt_the_normal_cad_wait(monkeypatch, tmp_path):
_, cad = fake_desktop(monkeypatch, tmp_path)
waited = []
+145
View File
@@ -367,6 +367,99 @@ def test_descendant_handles_are_owned_before_terminating_the_job():
assert ('close', 'child') in events
@pytest.mark.parametrize('child_exit_after', [0.5, None])
def test_completed_root_gives_owned_descendants_bounded_natural_exit_grace(monkeypatch, child_exit_after):
from dx_productivity.trial_task_center import isolated_desktop as module
api, events, state, *_ = native_api()
clock = [0.0]
monkeypatch.setattr(module.time, 'monotonic', lambda: clock[0])
monkeypatch.setattr(module.time, 'sleep', lambda seconds: clock.__setitem__(0, clock[0] + seconds))
state['running'] = False
child = Handle('child', events)
child_running = lambda: not any(event[0] == 'terminate' for event in events) and (
child_exit_after is None or clock[0] < child_exit_after)
api.job.JobObjectBasicProcessIdList = 3
api.job.QueryInformationJobObject = lambda job, kind: (
{'BasicLimitInformation': {'LimitFlags': 0}} if kind == 9 else
([43] if child_running() else []) if kind == 3 else {'ActiveProcesses': int(child_running())})
api.open_process = lambda *_args: child
root_wait = api.event.WaitForSingleObject
api.event.WaitForSingleObject = lambda handle, timeout: (
258 if child_running() else 0) if handle is child else root_wait(handle, timeout)
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
process.close()
assert child.closed
if child_exit_after is not None:
assert child_exit_after <= clock[0] < 2
assert ('terminate',) not in events
assert process.cleanup_record['forced_termination'] is False
assert process.cleanup_record['forced_descendant_pids'] == []
else:
assert 2 <= clock[0] < 2.1
assert ('terminate',) in events
assert process.cleanup_record['forced_termination'] is True
assert process.cleanup_record['forced_root_termination'] is False
assert process.cleanup_record['forced_descendant_pids'] == [43]
assert process.cleanup_record['termination_reason'] == 'descendant_grace_expired'
def test_completed_root_without_descendants_has_no_grace_delay(monkeypatch):
from dx_productivity.trial_task_center import isolated_desktop as module
api, events, state, *_ = native_api()
state['running'] = False
monkeypatch.setattr(module.time, 'sleep', lambda seconds: pytest.fail('empty job must not wait'))
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
process.close()
assert ('terminate',) not in events
assert process.cleanup_record['forced_termination'] is False
def test_active_root_cleanup_is_forced_without_descendant_grace(monkeypatch):
from dx_productivity.trial_task_center import isolated_desktop as module
api, events, *_ = native_api()
monkeypatch.setattr(module.time, 'sleep', lambda seconds: pytest.fail('active root must be bounded immediately'))
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
process.close()
assert process.cleanup_record['forced_termination'] is True
assert process.cleanup_record['forced_root_termination'] is True
assert process.cleanup_record['termination_reason'] == 'root_process_active'
def test_native_poll_error_still_terminates_owned_root_and_records_force():
api, events, state, *_ = native_api()
wait = api.event.WaitForSingleObject
def failing_poll(handle, timeout):
if timeout == 0 and state['running']:
raise OSError('native poll failed')
return wait(handle, timeout)
api.event.WaitForSingleObject = failing_poll
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
with pytest.raises(TaskCenterError, match='native poll failed'):
process.close()
assert state['running'] is False
assert process.cleanup_record['forced_root_termination'] is True
def test_failed_job_cleanup_retains_ownership_until_retry(monkeypatch):
from dx_productivity.trial_task_center import isolated_desktop as module
api, events, state, root, job = native_api()
clock = [0.0]
monkeypatch.setattr(module.time, 'monotonic', lambda: clock[0])
monkeypatch.setattr(module.time, 'sleep', lambda seconds: clock.__setitem__(0, clock[0] + seconds))
api.job.TerminateJobObject = lambda *_args: (_ for _ in ()).throw(OSError('job termination failed'))
query = api.job.QueryInformationJobObject
child_running = [True]
api.job.QueryInformationJobObject = lambda handle, kind: (
{'ActiveProcesses': int(state['running']) + int(child_running[0])} if kind == 1 else query(handle, kind))
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
with pytest.raises(TaskCenterError, match='descendants did not exit'):
process.close()
assert not job.closed and process.job is job
child_running[0] = False
process.close()
assert job.closed
@pytest.mark.parametrize('error_code', [87, 5])
def test_native_descendant_open_error_only_ignores_already_exited_process(error_code):
# pywintypes.error derives directly from Exception, rather than OSError.
@@ -391,6 +484,30 @@ def test_native_descendant_open_error_only_ignores_already_exited_process(error_
assert caught.value is error
def test_forced_job_preserves_partial_current_descendant_snapshot_after_open_failure():
api, events, state, *_ = native_api()
api.job.JobObjectBasicProcessIdList = 3
query = api.job.QueryInformationJobObject
snapshot = [45]
api.job.QueryInformationJobObject = lambda job, kind: list(snapshot) if kind == 3 else query(job, kind)
def open_child(_access, _inherit, pid):
if pid == 44:
raise OSError(5, 'child access denied')
return Handle('child-' + str(pid), events)
api.open_process = open_child
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
# PID 45 was observed earlier but is no longer part of the active Job.
process._remember_descendants()
snapshot[:] = [42, 43, 44]
with pytest.raises(TaskCenterError, match='child access denied'):
process.close()
assert state['running'] is False
assert process.cleanup_record['forced_termination'] is True
assert process.cleanup_record['forced_descendant_pids'] == [43]
assert process.cleanup_record['descendant_pids'] == [43, 45]
assert process.cleanup_record['job_handle_closed'] is True
def test_unconfirmed_descendant_exit_retains_handle_for_cleanup_retry():
api, events, state, *_ = native_api()
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
@@ -430,9 +547,37 @@ def test_native_private_job_kills_descendant_after_root_exit(parent_exits):
if parent_exits:
assert process.wait(timeout=5) == 0
assert win32event.WaitForSingleObject(child_handle, 0) == 0
assert process.cleanup_record['forced_termination'] is True
assert process.cleanup_record['forced_root_termination'] is (not parent_exits)
assert child_pid in process.cleanup_record['forced_descendant_pids']
with pytest.raises(pywintypes.error):
unexpected = win32service.OpenDesktop(name, 0, False, 1)
unexpected.CloseDesktop()
finally:
if child_handle is not None:
child_handle.Close()
@pytest.mark.skipif(os.name != 'nt', reason='native Windows descendant shutdown grace')
def test_native_private_job_allows_descendant_to_finish_after_root_exit():
import win32api
import win32event
import win32process
child_handle = None
try:
with PrivateDesktop('dx-test-' + uuid4().hex, enabled=True) as desktop:
process = desktop.start_process((sys.executable, '-c',
"import subprocess,sys; p=subprocess.Popen([sys.executable,'-c','import time;time.sleep(0.5)']); print(p.pid,flush=True)"),
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
child_pid = int(process.stdout.readline())
child_handle = win32api.OpenProcess(0x100000 | 0x1000, False, child_pid)
assert process.wait(timeout=5) == 0
assert win32event.WaitForSingleObject(child_handle, 0) == 0
assert win32process.GetExitCodeProcess(child_handle) == 0
assert process.cleanup_record['forced_termination'] is False
assert process.cleanup_record['forced_descendant_pids'] == []
assert desktop.cleanup_record['desktop_confirmation']['released'] is True
finally:
if child_handle is not None:
child_handle.Close()
+75
View File
@@ -0,0 +1,75 @@
from threading import Event
from PySide6.QtGui import QCloseEvent
from PySide6.QtWidgets import QMessageBox
from test_ui_remediation import window
from test_ui_batch_retry import open_project
def test_pause_and_window_close_never_request_force(window, qtbot, tmp_path):
open_project(window, qtbot, tmp_path)
window._begin_run(window._task_id)
worker = object()
window._workers.add(worker)
try:
event = QCloseEvent()
window.closeEvent(event)
assert not event.isAccepted()
assert window._batch_stop.is_set()
assert not window._batch_stop.force_event.is_set()
assert not window._closed
finally:
window._workers.discard(worker)
window._close_pending = False
window._finish_run()
def test_force_requires_confirmation_and_keeps_window_for_reconciliation(window, qtbot, tmp_path, monkeypatch):
open_project(window, qtbot, tmp_path)
window._begin_run(window._task_id)
control = window._batch_stop
try:
monkeypatch.setattr(QMessageBox, 'question', lambda *args: QMessageBox.StandardButton.No)
window._force_batch_stop()
assert not control.is_set() and not control.force_event.is_set()
monkeypatch.setattr(QMessageBox, 'question', lambda *args: QMessageBox.StandardButton.Yes)
window._close_pending = True
window._force_batch_stop()
assert control.is_set() and control.force_event.is_set()
assert not window._close_pending
finally:
window._finish_run()
def test_uncertain_wait_is_visible_in_execution_feedback(tmp_path):
import json
from dx_productivity.trial_task_center.execution_feedback import read_feedback
from dx_productivity.trial_task_center.ui.execution_feedback import format_feedback
root = tmp_path / 'quick-run'
root.mkdir()
(root / 'worker-progress.json').write_text(json.dumps({
'stage': 'save_and_close', 'waiting_for_safe_completion': True, 'stage_seconds': {}}))
feedback = read_feedback('task', 1, 1, '', tmp_path, 0)
text = format_feedback(feedback, 'board', 400, True)
assert '尚未确认' in text and '强制' in text
def test_diagnostic_export_during_wait_does_not_finish_or_cancel_execution(window, qtbot, tmp_path, monkeypatch):
from PySide6.QtWidgets import QFileDialog
open_project(window, qtbot, tmp_path)
window._begin_run(window._task_id)
control = window._batch_stop
owner = object()
window._workers.add(owner)
destination = tmp_path / 'support.zip'
monkeypatch.setattr(QFileDialog, 'getSaveFileName', lambda *args: (str(destination), ''))
try:
window._export_diagnostics()
qtbot.waitUntil(lambda: window._diagnostic_worker is None, timeout=5000)
assert destination.is_file()
assert owner in window._workers and window._batch_stop is control
assert not control.is_set() and not control.force_event.is_set()
finally:
window._workers.discard(owner)
window._finish_run()
+343
View File
@@ -0,0 +1,343 @@
import json
from pathlib import Path
import subprocess
import sys
from threading import Event, Thread
import pytest
from dx_productivity.trial_task_center.worker_monitor import ProgressWatch, run_monitored_process
def test_unsafe_save_timeout_waits_for_completion_and_recovers_on_progress():
watch = ProgressWatch('token', 'job', 10, 0)
assert watch.accept(b'DX_PROGRESS\ttoken\tjob\t1\tsave_and_close\t0\t1\t0\n', 0)
assert watch.action(10) == 'request_stop'
assert watch.action(20) == 'await_safe_completion'
assert watch.snapshot(now=20)['waiting_for_safe_completion'] is True
assert watch.action(200) != 'unresponsive'
assert watch.accept(b'DX_PROGRESS\ttoken\tjob\t2\tverify_saved_project\t1\t1\t0\n', 201)
assert watch.snapshot(now=201)['waiting_for_safe_completion'] is False
def test_real_slow_save_is_not_killed_by_inactivity(tmp_path):
script = tmp_path / 'save.py'
script.write_text('''import sys,time,pathlib
root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5]
while not (root/f'monitor-{token}.ready').exists():time.sleep(.01)
print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tsave_and_close\\t0\\t1\\t0',flush=True)
time.sleep(1.5)
(root/'saved.txt').write_text('saved and independently verified')
''', encoding='utf-8')
command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job')
result = run_monitored_process(command, .3)
assert result.returncode == 0
assert (tmp_path / 'saved.txt').read_text() == 'saved and independently verified'
def test_readonly_native_wait_keeps_bounded_cancellation_without_force_ui(tmp_path):
script = tmp_path / 'read.py'
script.write_text('''import sys,time,pathlib
root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5]
while not (root/f'monitor-{token}.ready').exists():time.sleep(.01)
print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tnative_snapshot_read\\t0\\t1\\t0',flush=True)
time.sleep(1.5)
''', encoding='utf-8')
command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job')
with pytest.raises(subprocess.TimeoutExpired):
run_monitored_process(command, .3, cooperative_abort=True)
def test_explicit_force_stops_unsafe_worker_with_protection(tmp_path):
script = tmp_path / 'save.py'
script.write_text('''import sys,time,pathlib
root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5]
while not (root/f'monitor-{token}.ready').exists():time.sleep(.01)
print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tsave_and_close\\t0\\t1\\t0',flush=True)
(root/'started').touch()
time.sleep(30)
(root/'unexpected_completed').touch()
''', encoding='utf-8')
force = Event()
def request():
for _ in range(200):
if (tmp_path / 'started').exists():
force.set()
return
Event().wait(.01)
requester = Thread(target=request)
requester.start()
command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job')
try:
with pytest.raises(Exception, match='forced'):
run_monitored_process(command, .3, force_event=force)
assert (tmp_path / 'uncertain.flag').exists()
assert not (tmp_path / 'unexpected_completed').exists()
finally:
requester.join(3)
def test_abnormal_cad_shutdown_is_not_hidden_by_physical_release(tmp_path):
from dx_productivity.trial_task_center.batch_supervisor import _finish_response
from dx_productivity.trial_task_center.batch_resources import write_record
from dx_productivity.trial_task_center.logging_setup import configure_logging
from dx_productivity.trial_task_center.supervisor_protocol import encode, encode_error
from dx_productivity.trial_task_center.errors import TaskCenterError
root = tmp_path
journal = {'operation': 'probe', 'batch_id': 'a' * 32, 'state': 'blocked'}
write_record(root / 'batch-resources' / journal['batch_id'] / 'session.json', {
'version': 1, 'session_id': 'a' * 32, 'kind': 'probe', 'state': 'released',
'generation': 1, 'desktop': 'dx-' + 'b' * 32, 'processes': [], 'cad_pid': None,
'cad_guid': None, 'checkpoints': {}, 'retries': {}, 'timings': [],
'cad_shutdown': {'clean': False, 'cleanup_errors': [{'stage': 'orderly_shutdown', 'message': 'Quit timed out'}]}})
response = {'business_succeeded': True, 'status': 'error', 'result': encode({'verified': True}),
'error': encode_error(TaskCenterError('batch_resources_pending', 'Quit timed out'))}
with pytest.raises(TaskCenterError) as caught:
_finish_response(root, journal, response, configure_logging(root))
assert caught.value.code == 'application_shutdown_failed'
assert caught.value.completed_result == {'verified': True}
assert journal['resource_recheck']['pending'] is False
def test_force_cannot_terminate_before_interruption_evidence_is_durable(tmp_path, monkeypatch):
from types import SimpleNamespace
from dx_productivity.trial_task_center import batch_supervisor as module
calls = []
process = SimpleNamespace(terminate=lambda: calls.append('terminate'))
journal = {'operation': 'execute_task_queue', 'batch_id': 'a' * 32, 'attempt_id': 'b' * 32,
'task_ids': ['task'], 'state': 'running'}
monkeypatch.setattr(module, '_mark_interrupted', lambda *args: (_ for _ in ()).throw(PermissionError('denied')))
with pytest.raises(PermissionError):
module._force_owned_supervisor(tmp_path, journal, process)
assert calls == []
def test_force_already_performed_remains_forced_when_exit_diagnostics_fail(tmp_path):
from types import SimpleNamespace
from dx_productivity.trial_task_center.batch_supervisor import _force_owned_supervisor
record = {}
def terminate():
record['forced_termination'] = True
raise OSError('child enumeration failed after force')
process = SimpleNamespace(terminate=terminate, cleanup_record=record)
journal = {'operation': 'execute_task_queue', 'batch_id': 'a' * 32, 'attempt_id': 'b' * 32,
'task_ids': ['task'], 'state': 'running'}
_force_owned_supervisor(tmp_path, journal, process)
saved = json.loads((tmp_path / 'batch-resources' / ('a' * 32) / 'supervisor.json').read_text(encoding='utf-8'))
assert saved['stop_mode'] == 'forced'
assert 'enumeration' in saved['force_cleanup_error']
def test_interruption_targets_active_project_without_invalidating_previous_tasks(tmp_path):
from dx_productivity.trial_task_center.batch_supervisor import _mark_interrupted
from dx_productivity.trial_task_center.batch_resources import write_record
batch = 'a' * 32
write_record(tmp_path / 'batch-resources' / batch / 'session.json', {
'version': 1, 'session_id': batch, 'kind': 'task_queue', 'state': 'ready',
'generation': 1, 'desktop': 'dx-' + 'b' * 32, 'processes': [], 'cad_pid': 123,
'cad_guid': 'owned', 'checkpoints': {}, 'retries': {}, 'timings': [], 'active_task': 'current'})
journal = {'operation': 'execute_task_queue', 'batch_id': batch, 'attempt_id': 'c' * 32,
'task_ids': ['previous', 'current'], 'state': 'running'}
_mark_interrupted(tmp_path, journal, 'forced stop')
assert (tmp_path / 'tasks/current/quick-run/uncertain.flag').exists()
assert not (tmp_path / 'tasks/previous/quick-run/uncertain.flag').exists()
@pytest.mark.parametrize('interactive', [False, True])
def test_supervisor_only_holds_for_resolution_when_caller_has_force_control(tmp_path, monkeypatch, interactive):
import textwrap
from dx_productivity.trial_task_center import batch_supervisor as module
from dx_productivity.trial_task_center.stop_control import StopControl
from dx_productivity.trial_task_center.service import TaskCenterService
from dx_productivity.trial_task_center.store import TaskStore
from dx_productivity.trial_task_center.logging_setup import configure_logging
bootstrap = textwrap.dedent('''
import sys,json
from pathlib import Path
from dx_productivity.trial_task_center import batch_worker,batch_runtime
from dx_productivity.trial_task_center.stop_control import StopControl
original=batch_runtime.BatchCadRuntime
def runtime(*args,**kwargs):
result=original(*args,**kwargs)
Path(sys.argv[1]).with_name('interactive-witness.json').write_text(json.dumps(isinstance(result.stop_event,StopControl)))
return result
batch_runtime.BatchCadRuntime=runtime
batch_worker.run(Path(sys.argv[1]))
''')
monkeypatch.setattr(module, '_command', lambda request: (sys.executable, '-c', bootstrap, str(request)))
store = TaskStore(tmp_path / 'tasks.db')
try:
service = TaskCenterService(store, configure_logging(tmp_path))
module.run_supervised(service, 'probe', {}, stop_event=StopControl() if interactive else Event())
witness = next((tmp_path / 'batch-resources').glob('*/interactive-witness.json'))
assert json.loads(witness.read_text()) is interactive
finally:
store.close()
def test_uncertain_operation_keeps_owner_until_explicit_force(tmp_path):
from dx_productivity.trial_task_center.stop_control import StopControl, wait_for_resolution
control = StopControl()
failures = []
def owner():
try:
wait_for_resolution(control, tmp_path, 'save result unknown')
except Exception as error:
failures.append(error)
thread = Thread(target=owner)
thread.start()
try:
for _ in range(100):
if (tmp_path / 'worker-progress.json').exists():
break
Event().wait(.01)
control.set() # Ordinary stop never authorizes discarding a changed project.
thread.join(.15)
assert thread.is_alive()
assert (tmp_path / 'uncertain.flag').exists()
assert json.loads((tmp_path / 'worker-progress.json').read_text())['waiting_for_safe_completion']
control.request_force()
thread.join(3)
assert not thread.is_alive()
assert 'forced' in str(failures[0])
finally:
control.request_force()
thread.join(3)
def test_failed_protection_fsync_is_retried_before_force_is_allowed(tmp_path, monkeypatch):
from dx_productivity.trial_task_center import stop_control as module
control = module.StopControl()
original = module.os.fsync
attempts, failures = [], []
def denied(fd):
attempts.append(fd)
raise OSError('evidence fsync denied')
monkeypatch.setattr(module.os, 'fsync', denied)
def owner():
try:
module.wait_for_resolution(control, tmp_path, 'save unknown')
except Exception as error:
failures.append(error)
thread = Thread(target=owner)
thread.start()
try:
for _ in range(100):
if control.control_error:
break
Event().wait(.01)
assert control.control_error
control.request_force()
thread.join(.25)
assert thread.is_alive(), 'a file left by failed fsync must not authorize force'
assert len(attempts) >= 2
monkeypatch.setattr(module.os, 'fsync', original)
control.request_force()
thread.join(3)
assert failures and 'forced' in str(failures[0])
finally:
monkeypatch.setattr(module.os, 'fsync', original)
control.request_force()
thread.join(3)
@pytest.mark.parametrize('deny_first,transport_failure', [(False, False), (True, False), (False, True)])
def test_real_supervisor_force_preserves_protection_and_never_replays(tmp_path, monkeypatch, deny_first, transport_failure):
import os
import textwrap
import time
import win32api
import win32event
from dx_productivity.trial_task_center import batch_supervisor as module
from dx_productivity.trial_task_center.stop_control import StopControl
from dx_productivity.trial_task_center.service import TaskCenterService
from dx_productivity.trial_task_center.store import TaskStore
from dx_productivity.trial_task_center.logging_setup import configure_logging
from dx_productivity.trial_task_center.errors import TaskCenterError
store = TaskStore(tmp_path / 'tasks.db')
service = TaskCenterService(store, configure_logging(tmp_path), supervised=True)
task = store.create_task('force fixture')
evidence = service._project_task_service(task.task_id)._quick_run_preparer.data_root / 'quick-run'
witness = evidence / 'started'
bootstrap = textwrap.dedent('''
import sys,os,time,json
from pathlib import Path
from dx_productivity.trial_task_center import batch_worker
from dx_productivity.trial_task_center.service import TaskCenterService
def business(self, task_id, **kwargs):
root=self._project_task_service(task_id)._quick_run_preparer.data_root/'quick-run'
root.mkdir(parents=True,exist_ok=True)
(root/'started').write_text(str(os.getpid()))
request=json.loads(Path(sys.argv[1]).read_text(encoding='utf-8'))
print(json.dumps({'version':1,'batch_id':request['batch_id'],'attempt_id':request['attempt_id'],
'generation':1,'sequence':100,'type':'progress','payload':'fixture executing'}),flush=True)
time.sleep(30)
(root/'unexpected_completed').touch()
TaskCenterService.execute_quick_run=business
batch_worker.run(Path(sys.argv[1]))
''')
original_command = module._command
monkeypatch.setattr(module, '_command', lambda request:
(sys.executable, '-c', bootstrap, str(request))
if json.loads(request.read_text())['operation'] == 'execute_quick_run' else original_command(request))
original_mark = module._mark_interrupted
denied = []
def mark(*args):
if deny_first and not denied:
denied.append(True)
raise PermissionError('force evidence denied once')
return original_mark(*args)
monkeypatch.setattr(module, '_mark_interrupted', mark)
control = StopControl()
driver_errors, survived = [], []
def request():
try:
deadline = time.monotonic() + 15
while not witness.exists() and time.monotonic() < deadline:
time.sleep(.02)
assert witness.exists()
pid = int(witness.read_text())
if transport_failure:
while not control.control_error and time.monotonic() < deadline:
time.sleep(.02)
assert control.control_error
handle = win32api.OpenProcess(0x100000, False, pid)
try:
survived.append(win32event.WaitForSingleObject(handle, 0) == 258)
finally:
handle.Close()
control.request_force()
if deny_first:
while not control.control_error and time.monotonic() < deadline:
time.sleep(.02)
assert control.control_error
handle = win32api.OpenProcess(0x100000, False, pid)
try:
survived.append(win32event.WaitForSingleObject(handle, 0) == 258)
finally:
handle.Close()
control.request_force()
except Exception as error:
driver_errors.append(error)
thread = Thread(target=request)
thread.start()
try:
with pytest.raises(TaskCenterError) as caught:
module.run_supervised(service, 'execute_quick_run', {'task_id': task.task_id}, stop_event=control,
on_progress=(lambda _: (_ for _ in ()).throw(RuntimeError('UI feedback failed'))) if transport_failure else None)
assert not driver_errors
assert caught.value.code == 'user_forced_stop'
assert (evidence / 'uncertain.flag').exists()
marker = (evidence / 'uncertain.flag').read_bytes()
assert not (evidence / 'unexpected_completed').exists()
journals = [json.loads(p.read_text(encoding='utf-8')) for p in (tmp_path / 'batch-resources').glob('*/supervisor.json')]
forced = next(r for r in journals if r['operation'] == 'execute_quick_run')
assert forced['stop_mode'] == 'forced' and forced['business_uncertain'] is True
assert service.cleanup_batch_resources() == ()
assert (evidence / 'uncertain.flag').read_bytes() == marker
assert not (evidence / 'unexpected_completed').exists()
if deny_first or transport_failure:
assert survived == [True]
finally:
thread.join(3)
store.close()
+129
View File
@@ -0,0 +1,129 @@
"""Request ordinary stop during save on disposable projects; never replay a run."""
from dataclasses import asdict
import hashlib
import json
from pathlib import Path
import re
import shutil
import sys
from threading import Event, Thread
import time
from uuid import uuid4
REPO = Path(__file__).resolve().parents[1]
sys.path[:0] = [str(REPO / 'src'), str(REPO / 'tools')]
from probe_batch_runtime_live import hashes, compare_saved, save, OLD, NEW, NAMES
from dx_productivity.trial_task_center.isolated_cad import IsolatedCadSession
from dx_productivity.trial_task_center.logging_setup import configure_logging
from dx_productivity.trial_task_center.service import TaskCenterService
from dx_productivity.trial_task_center.store import TaskStore
from dx_productivity.trial_task_center.bom_plan import write_bom_template
from dx_productivity.trial_task_center.stop_control import StopControl
from dx_productivity.trial_task_center.task_queue import execute_task_queue
def main():
import argparse
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--source-root', required=True, type=Path)
parser.add_argument('--library', required=True, type=Path)
parser.add_argument('--run-native', action='store_true', required=True)
args = parser.parse_args()
output = REPO / 'build' / ('safe-stop-live-' + uuid4().hex[:10])
output.mkdir()
source_root, library = args.source_root.resolve(strict=True), args.library.resolve(strict=True)
source_hashes = {name: hashes(source_root / name) for name in NAMES}
library_hash = hashlib.sha256(library.read_bytes()).hexdigest()
projects = []
for name in NAMES:
folder = output / 'projects' / name
shutil.copytree(source_root / name, folder)
project = folder / (name + '.prj')
text, count = re.subn(r'(?m)^KEY CentralLibrary .*$', lambda _: f'KEY CentralLibrary "{library.as_posix()}"',
project.read_text(encoding='utf-8-sig'))
assert count == 1
project.write_text(text, encoding='utf-8')
projects.append(project)
report = {'scope': 'source supervised service, production VBS and real CAD; no GUI or frozen EXE',
'output': str(output), 'projects': list(map(str, projects))}
print(json.dumps(report, ensure_ascii=False), flush=True)
store = TaskStore(output / 'tasks.db')
service = TaskCenterService(store, configure_logging(output / 'data'), supervised=True)
control, finished = StopControl(), Event()
watcher = None
def inspect(project, sheets, label):
evidence = output / (label + '-' + project.stem)
evidence.mkdir()
spec, result = evidence / 'spec.json', evidence / 'graph.json'
save(spec, {'project': str(project), 'sheets': sorted(sheets), 'destination': str(result)})
with IsolatedCadSession(evidence) as cad:
child = cad.start_worker((sys.executable, str(REPO / 'tools/probe_batch_runtime_live.py'),
'--repo', str(REPO), '--inspect-file', str(spec), '--run-native'), stdout=-1, stderr=-1)
try:
code = child.wait(timeout=180)
(evidence / 'worker-output.txt').write_bytes(child.stdout.read() + b'\n' + child.stderr.read())
assert code == 0, (evidence / 'worker-output.txt').read_text(errors='replace')
finally:
child.close()
return json.loads(result.read_text(encoding='utf-8'))
try:
tasks = service.create_project_tasks(output / 'projects', selected_paths=tuple(projects))
bom = write_bom_template(output / 'input.xlsx', ({'project_path': str(p), 'old_part_number': OLD,
'new_part_number': NEW, 'refdes': 'RV05', 'enabled': True, 'sync_nc': True} for p in projects))
imported = service.inspect_bom_batch(tasks[0].task_id, bom)
assert imported.passed_rows == 2 and imported.preview is not None
service.save_bom_batch(imported.preview, {p.task_id: p.rows for p in imported.preview.projects})
before, sheet_sets = {}, {}
for project in projects:
rows = next(p.rows for p in imported.preview.projects if p.project_path.resolve() == project.resolve())
sheet_sets[project.stem] = {(row.target.schematic_name, row.target.sheet_number) for row in rows}
before[project.stem] = inspect(project, sheet_sets[project.stem], 'before')
def stop_during_save():
while not finished.wait(.01):
for path in (output / 'data/tasks').glob('*/quick-run/quick-part-replace.log'):
try:
text = path.read_text(encoding='utf-8-sig')
except OSError:
continue
if 'save_requested job=' in text:
report['stop_requested'] = {'time_ns': time.time_ns(), 'log': str(path),
'trigger': 'production save_requested log; ordinary stop only'}
control.set()
return
watcher = Thread(target=stop_during_save, daemon=True)
watcher.start()
result = execute_task_queue(service, tuple(task.task_id for task in tasks), stop_event=control)
finished.set()
watcher.join(3)
report['queue_result'] = asdict(result)
assert report.get('stop_requested') and not control.force_event.is_set()
assert result.paused and [row.state for row in result.entries] == ['completed', 'pending'], result
report['comparisons'] = []
for index, project in enumerate(projects):
after = inspect(project, sheet_sets[project.stem], 'after')
if index == 0:
report['comparisons'].append(compare_saved(before[project.stem], after))
else:
assert after == before[project.stem], 'second project was changed after stop'
report['comparisons'].append({'unchanged': True, 'components': len(after)})
assert not service.pending_batch_resources()
assert not list((output / 'data').rglob('uncertain.flag'))
report['verified'] = True
except BaseException as error:
import traceback
report.update(error=repr(error), traceback=traceback.format_exc())
raise
finally:
finished.set()
if watcher is not None:
watcher.join(3)
store.close()
report['source_unchanged'] = all(hashes(source_root / name) == source_hashes[name] for name in NAMES)
report['library_lmc_unchanged'] = hashlib.sha256(library.read_bytes()).hexdigest() == library_hash
save(output / 'report.json', report)
print(json.dumps(report, ensure_ascii=False), flush=True)
return 0 if report.get('verified') and report['source_unchanged'] and report['library_lmc_unchanged'] else 1
if __name__ == '__main__':
raise SystemExit(main())