diff --git a/docs/superpowers/plans/2026-10-26-safe-user-stop.md b/docs/superpowers/plans/2026-10-26-safe-user-stop.md new file mode 100644 index 0000000..f3b2154 --- /dev/null +++ b/docs/superpowers/plans/2026-10-26-safe-user-stop.md @@ -0,0 +1,62 @@ +# Safe User Stop Implementation Plan + +> **For agentic workers:** use superpowers:subagent-driven-development or executing-plans; continue within the user-approved scope without another approval round. + +**Goal:** 用户停止只在安全边界收尾;写入中的无进度不自动杀进程;明确强制中断有保护证据和恢复入口。 + +**Architecture:** 保留现有 Event/队列边界和独立监督进程,加入独立的强制停止意图、等待状态反馈;退出正常性与物理资源复核分开。使用既有任务保护与批次记录,不新增数据库版本。 + +**Tech Stack:** Python 3.12、Windows pywin32、PySide6、VBScript、pytest。 + +**Spec:** `docs/superpowers/specs/2026-10-26-safe-user-stop.md` + +## Global Constraints + +- 不修改原工程、中心库或未跟踪 feedback;真实测试只复制 fixture。 +- 不放宽身份、写前保护、禁止不确定重放的门禁。 +- 普通停止与强制结束独立,确认前不得产生强制意图。 +- 不通过无限创建桌面或删除旧保护绕过异常。 +- 不发布、不推送;本地分支 `codex/day0-safe-stop`。 + +## Task 1: 退出协议与后代进程 + +Files: `isolated_cad.py`, `isolated_desktop.py`, `tests/v1_1/test_isolated_cad.py`, `test_isolated_desktop.py`. + +- [x] 先复现 closing 记录失败跳过 Quit、父进程自然退出后仍立即终止子进程的测试。 +- [x] 诊断写入失败不跳过满足身份/空工程条件的 Quit;将记录失败独立保留并报告,禁止伪报 clean。 +- [x] 根进程正常退出时,给 Job 后代 2 秒有界自然退出窗口;到期再强制并记录 `forced_termination`, `termination_reason`, `forced_descendant_pids`。根进程仍活着的异常清理仍有界。 +- [x] 保留所有权和退出确认,测试自然退出与强制升级两条路径。 + +## Task 2: 停止、等待与强制中断 + +Files: new `stop_control.py`; `worker_monitor.py`, `project_session.py`, `batch_runtime.py`, `batch_supervisor.py`, `execution_feedback.py`; associated tests. + +Interface: +```python +class StopControl(Event): + force_event: Event + def request_force(self): + self.set() + self.force_event.set() +``` +- [x] 首先测试 unsafe 保存阶段超时仍保留 worker,安全阶段仍有 watchdog,显式 force 才结束并产生 uncertain。 +- [x] `ProgressWatch` 提供 `waiting_for_safe_completion`;新进度可清除提示,不把心跳当作写入推进。 +- [x] 监督父进程持久化停止意图;force 先写保护与中断记录,再只终止已拥有的监督进程 Job,不使用名字或不可信 PID 清理。 +- [x] 业务完成但退出异常仍通过独立错误返回,附带已核验业务结果;纯桌面延迟且退出正常可继续使用现有复核恢复。 +- [x] 普通停止在当前项收尾后暂停;会话 close 不得因普通等待超时反向强杀仍在写入的 worker。 + +## Task 3: 保存边界与界面 + +Files: `QuickPartReplace.vbs`, `ui/run_panel.py`, `ui/main_window.py`, `ui/execution_feedback.py`, protocol serializer if required; VBS/UI tests. + +- [x] VBS 用真实函数的失败先行测试覆盖保存已发出但工程关闭失败;保存请求前设置“可能保存”状态,独立记录保存返回/回读/关闭,不再声称未保存。 +- [x] 正常停止显示等待当前项收尾;unsafe 超时显示保存/核验尚未确认。 +- [x] 提供与普通暂停分离的强制结束按钮,确认文案明确结果待核验;取消确认没有副作用。 +- [x] 关闭窗口在运行/等待期间保持打开;任务失败后取消自动关闭,让用户看见恢复信息。 + +## Task 4: 集成验证与审查 + +- [x] 运行相关 VBS、monitor、supervisor、queue、UI、资源测试及完整发布门禁范围。 +- [x] 实机空 CAD 自然退出、两工程副本正常保存及独立回读;停止/强制结束采用受控替身或独立进程,明确验证边界。 +- [x] 独立审查关注:不确定写入重放、等待时 GUI 退出杀进程、force 身份校验与持久化失败、结果错误被吞、已完成项保留。 +- [x] 更新操作文档和验收记录,交付本地可审查修改及测试证据。 diff --git a/docs/superpowers/specs/2026-10-26-safe-user-stop.md b/docs/superpowers/specs/2026-10-26-safe-user-stop.md new file mode 100644 index 0000000..7ca44e7 --- /dev/null +++ b/docs/superpowers/specs/2026-10-26-safe-user-stop.md @@ -0,0 +1,9 @@ +# 用户停止与安全收尾(已批准) + +用户已在本会话批准:普通停止冻结后续调度;未修改可取消;当前已修改项完成校验、保存、回读后停止;保存或修改结果不明保持保护,不自动重放或盲目保存/丢弃;关闭窗口等待收尾;强制结束必须由用户明确选择,记录中断并在重启后核验。 + +业务结果、应用正常退出、强制清理和物理资源状态分别记录。资源消失不抹去应用异常。保存中无进度超时进入等待/提示,不自动终止。提供明确确认的强制结束出口;其范围仅为本程序当前监督进程及自有 Job。 + +普通停止以当前替换项及其必要收尾为边界,不启动同工程下一项。保护记录、批次身份和进程创建时间核验继续有效。先保存中断证据再执行工具内强制终止;持久化失败时拒绝工具内强制操作并反馈错误。系统外强杀/掉电依赖已有写前保护与重启核验,不承诺可在终止时写入日志。 + +保持源码版本,不发布包、不关闭远端事故工单。真实工程验证只用全新副本。 diff --git a/docs/testing/2026-10-26-application-lifecycle-investigation.md b/docs/testing/2026-10-26-application-lifecycle-investigation.md new file mode 100644 index 0000000..b4018b8 --- /dev/null +++ b/docs/testing/2026-10-26-application-lifecycle-investigation.md @@ -0,0 +1,134 @@ +# Day0:保存、关闭、退出全链路排查 + +2026-10-26,北京时间。源码基线 `b9391015650bf5c43a77a017c5484e9a8f352266`。承接[资源事故分析](2026-10-26-day0-resource-release-analysis.md),本轮以核心应用完成条件为主线。 + +## 结论 + +发现并验证了四个具体的生命周期问题: + +1. **CAD 主进程正常退出后,程序会立即终止 Job 中的剩余辅助进程,没有先给予其自然退出窗口。** 本机实测 `vnsd.exe` 再等约 0.813 秒即可自然退出;这个等待可以避免该次强制终止。 +2. **退出阶段的记录写入失败会跳过正常 Quit。** 随后仍进入自有进程清理。用生产退出控制流和替身 CAD 复现。 +3. **保存调用返回、工程关闭失败时,“已保存”标记仍可能没有设置。** 实际 VBS 函数配合替身 CAD 复现,说明保存与关闭的状态记录边界混在一起。 +4. **物理资源复核通过后,上层可能返回已完成业务结果,同时不再向调用方抛出正常退出失败。** 生产运行时记录可以出现 `cad_shutdown.clean=false`、资源状态 `released`、监督记录 outcome 为 `error`,但函数返回 completed 任务结果。日志证据保留,调用方的返回语义却丢失了退出异常。 + +这些问题与“只盯桌面释放”的分析不同:资源消失不能代表应用正常退出,更不能取代数据保存确认。 + +同时,两份全新工程副本的真实替换、保存、退出及新 CAD 实例独立回读均通过,未发现本轮保存数据丢失。上述问题不是已经证明的工单 #1 唯一根因;原附件中的 #1 属于读取后的收尾失败,#3 尚未启动 CAD。现场版本、实际触发路径和完整诊断仍须对照。 + +## 一、实机正常链路 + +使用已有 Audio_D2128、Audio_T605A fixture 的全新副本,生产服务入口执行 BOM 导入及两工程队列,每工程仅 RV05:`4100-CA0000-T7 → 4100-CA0000-T0`,保留原生目标 Value 策略。原始 fixture 未修改。 + +证据目录:`build/day0-resource-analysis-20261026/live-chain-bbd9fba7ce444e3d8e5627a6f4a385d0/`。汇总:`build/day0-resource-analysis-20261026/live-chain-summary.json`。进程时间线为同级 `live-chain-bbd9fba7ce444e3d8e5627a6f4a385d0-process-timeline.jsonl`。 + +| 检查 | 本次结果 | +| --- | --- | +| BOM 导入 | 2 行通过,3.875 秒 | +| 顺序执行 | 两任务 completed,155.843 秒,共用一个执行 CAD | +| 保存关闭图页调用 | D2128 约 0.32 秒,T605A 约 0.21 秒 | +| 保存后的 CloseProject | 两次返回 True,耗时约 1.68 / 1.33 秒;均先观察到 ProjectClosed 事件,再观察到调用返回 | +| 同实例重开回读 | 每工程完成内容核验后才发布 saved_ok,随后关闭工程 | +| CAD 自然退出 | 本轮五个 CAD 实例(前读 2、执行 1、后读 2)主进程均正常退出码 0,无 CAD 主进程 terminate_request | +| CAD 退出等待 | Quit helper 成功后的主进程等待约 0.234–0.390 秒,不是整个 Quit 的总耗时 | +| 退出后的独立回读 | 每工程用新 CAD 实例读取;57 器件、103 引脚连接,56 个非目标器件不变,目标身份、位置、连接和目标料号正确 | +| 原件 | 两个源工程目录哈希不变,中央库 `.lmc` 哈希不变;未宣称整个库目录全部文件无变化 | +| 前台与资源 | 观察器未发现自有 CAD 进入输入桌面或成为前台,已记录 CAD 退出和桌面消失 | + +这轮运行的是源码服务层、真实 CAD 和加入边界日志的 VBS 副本;没有修改生产源码。不是完整冻结 GUI/监督进程验收,时序日志可能增加少量开销,耗时不能作为性能承诺。 + +第一轮记录 `live-chain-366340e7a6a942ccadaf2e1eedd9c4f4` 因探针复制 VBS 时引入 UTF-8 BOM,导致第 1 行编译失败,替换前停止。该失败属于探针,不是产品故障;保留原记录和阻断状态,修正编码后使用另一组全新副本,未重放该次失败任务。 + +## 二、已验证问题与含义 + +### A. 辅助进程尚未自然退出,就调用 TerminateJobObject + +位置:`isolated_desktop.py:380` 的 `_Win32Process.close`。处理顺序为:确认/停止根进程 → 收集后代 → **TerminateJobObject** → 最多等待 5 秒确认进程退出。因此这 5 秒是强制终止后的确认窗口,不是辅助进程正常收尾的宽限期。 + +两工程实机中五次 CAD 主进程退出后,Job 都仍有一个成员。原时间线的映像查询使用了本机 pywin32 未提供的函数,查询错误已保留;不能凭 PID 将五个成员全部命名。独立 CIM 快照将最后一次回读的 CAD PID 55500、子进程 PID 45368 对应到实际 `vnsd.exe`。 + +随后单独进行空 CAD 实验,改用 Win32 `QueryFullProcessImageNameW` 确认成员身份。仅推迟探针中对已正常退出 CAD 的 Job 清理,最长观察预算 75 秒,未打开任何工程: + +- CAD PID 60332 正常退出码 0。 +- 此时 `vnsd.exe` PID 34960 仍存活。 +- 约 **0.813 秒**后该辅助进程自行退出,随后原清理函数运行时已没有需终止的后代。 +- 最终自有资源全部释放,桌面检查耗时 0 秒,整个关闭阶段约 2.625 秒。 + +证据:`build/day0-resource-analysis-20261026/empty-cad-descendants-9fd000b5f0af48dc82e12b6b034d2917/report.json`;探针 `probe_descendant_exit.py` 位于同级分析目录。 + +既有原生进程研究将 `vnsd.exe` 识别为许可证辅助程序,另有工程数据库 `iCDBNetServer.exe` 和交叉探查 `xprobSvr.exe`。本次空 CAD 仅验证 vnsd 的自然退出,不证明数据库提交被终止,也不证明它就是工单现场残留桌面的持有者。 + +建议方向:对已经正常退出的根进程,其明确自有的后代先进行有界自然退出等待;到期才进入可审计的强制清理。区分根进程、数据库辅助进程、许可证辅助进程;记录终止对象、原因及发生时的业务阶段。不能只把等待常量变大,也不能对不明进程进行清理。 + +### B. 收尾记录的 I/O 错误会改变应用退出方式 + +位置:`isolated_cad.py:207`。正常 Quit 前的 `_record(state='closing')` 与 Quit 在同一个 try 内,记录失败会直接跳入异常处理,接着 finally 仍关闭自有资源。 + +故障注入:仅让这一次 closing 记录写入抛 PermissionError,其他记录正常。结果为:没有发出 Quit;写入 uncertain;调用资源 close。真实 CAD 由替身代替,因此本项证明控制流缺陷,没有实际杀死用户 CAD。 + +含义:目录权限、同步软件/安全软件的文件访问冲突、磁盘问题等不只影响日志,还可能改变应用本来可以正常退出的路径。上述环境来源只是候选,没有确定具体软件责任。 + +建议方向:隔离诊断/资源收尾记录错误与正常退出协议;写前业务保护失败仍应禁止写入,但不能混同为“所有诊断写入失败都必须跳过 Quit”。退出请求仍须满足实例身份、工程状态等原有条件。 + +### C. 保存状态在工程关闭成功之后才记录 + +位置:`QuickPartReplace.vbs:934`。调用 `QPCloseBatchProject(TargetPath, True)` 返回后才执行 `QP_BatchSaveCommitted=True`;这个子程序内部既保存图页,又关闭工程。 + +用真实 `QPFinishBatchItem` / `QPCloseBatchProject` 函数与 CAD 形状替身运行:图页的 SaveChanges=True 调用已经返回,随后工程拒绝关闭。观察到 `save_called=True`、`commit_flag=0`、错误已抛出,没有发布成功 checkpoint。 + +因此,这个标记不能独立表达“保存已发出/可能完成,而工程关闭失败”。某些后续错误文案依赖它,可能把已涉及保存的故障描述为未保存。`failed_native_result_uncertain` 仍会依据原生修改证据阻断重试,本轮没有发现借此自动重放替换;问题主要在恢复判断所需的状态粒度和事实描述。 + +建议方向:拆分 save_requested、save_returned、saved_readback_verified、project_closed 等证据;保存请求开始后发生异常应保留“可能已保存”的状态,不能因 CloseProject 失败回退成“未保存”。 + +### D. 资源消失后,正常退出异常不再向调用方抛出 + +位置:`batch_supervisor.py:159` 的 `_finish_response`,及 `batch_resources.py:117` 的重查逻辑。 + +用生产 BatchCadRuntime 创建合法资源记录,替身 CAD 在退出时抛出 Quit 超时,已保存业务结果独立保留;模拟拥有资源的监督进程结束后,再调用生产资源复核及返回函数。结果: + +- 会话先 blocked;物理桌面已不存在后变为 released。 +- `cad_shutdown.clean=false` 和 `orderly_shutdown` 异常仍保留在 JSON。 +- 监督记录 `state=finished`、`outcome=error`。 +- 调用方收到含 completed 条目的 TaskQueueResult,没有收到退出异常。 + +这是状态返回语义不一致,不是原始错误日志被删除,也不证明已独立核验的保存结果应当被作废。应保留正确业务结果,同时显式暴露“业务完成、应用异常退出/强制清理”的收尾状态,不能仅凭物理资源消失将异常归为恢复成功。 + +上述 B/C/D 复现实物:`build/day0-resource-analysis-20261026/fault-chains-907a534cc6bb441e8cddf7670dd04aac/`,脚本 `probe_fault_chains.py`。较早的临时探针曾因日志文件句柄尚未关闭而无法删除临时目录;后续改为保留独立证据目录并关闭日志句柄,此探针清理错误不计入产品结论。 + +## 三、仍需验证的高影响风险 + +| 风险 | 已知机制 | 尚不能断言的部分 | +| --- | --- | --- | +| 慢保存被当成无响应 | 保存前仅发一次阶段进度;默认 150 秒请求停止、300 秒判无响应,safe_stop=False 不阻止监督器升级。保存没有类似原生快照的进度旁路 | 没有真实慢保存达到 300 秒的实验,不证明现场曾在保存中被终止 | +| Quit/关闭响应延迟 | Quit helper 20 秒;成功后 CAD 15 秒;checkpoint 20 秒;项目会话关闭另有 60 秒宽限 | 本次正常样例均未超过这些时限,不能据此证明大工程也安全或必须统一加长 | +| 同实例回读可能不足以覆盖持久性 | 生产路径在同一个 CAD 实例中重开回读 | 本次新实例读回通过,未发现缓存掩盖丢失;仍缺断电、数据库服务异常和大型工程覆盖 | +| 关闭工程返回值被忽略 | 本机 TypeLib 表明 CloseProject 返回 Boolean;VBS 常直接调用并用工程路径为空作为补充判断 | 本次调用均为 True,尚未观察到 False 且空路径的真实异常,不能将其直接确定为事故根因 | +| GUI/监督进程异常终止 | Job 关闭会清理受控进程,不能保证正在进行的保存完成 | 正常点击关闭时 GUI 会等待当前 worker,不能把所有 GUI 关闭行为描述成直接杀进程。崩溃/强制结束另需故障实验 | +| 数据库和跨进程协作收尾 | 既有研究存在 iCDBNetServer、xprobSvr 等进程;仅确认 viewdraw 退出不足以描述全部应用状态 | 本轮没有捕获这些服务器在未提交数据时被终止;不能把 vnsd 的观察套用成数据库丢失证据 | + +## 四、厂商接口核对的重要限制 + +实际安装的 7.9.5 `viewdraw.tlb` 已离线核对,结果保存在 `cad-lifecycle-typelib.json`: + +- `IVdApp.CloseProject` 返回 Boolean。 +- `IVdApp.Quit`、`IVdSchematicSheetDocument.Close(SaveChanges, FileName)` 返回 void,不能虚构它们有保存成功布尔返回值。 +- `IVdDoc` 有 Saved 属性,但不能据此假定当前 `IVdSchematicSheetDocument` 也有同样的属性契约。 +- 事件接口包含 BeforeDocumentSave、AfterDocumentSave、ProjectClosed。 + +真实 VBS 日志已订阅上述事件:本轮 ProjectClosed 被正常观测,但在两次 SaveChanges=True 路径中**没有观测到 BeforeDocumentSave/AfterDocumentSave**。因此不能因为类型库列出了保存事件,就把等待该事件直接设为当前原理图保存路径的必经条件。需要核实它适用的文档类型、接口和触发条件。 + +本轮读取的安装帮助未取得足以证明上述调用异步/同步持久化语义的详细契约。类型库提供方法签名,不能替代持久化保证。 + +## 五、验证与修改范围 + +本轮补充回归: + +```text +python -m pytest -q tests/integration/test_quick_value_modes.py tests/v1_1/test_project_session.py tests/v1_1/test_failed_native_recovery_guard.py tests/v1_1/test_supervisor_result.py tests/v1_1/test_ui_operation_lifecycle.py +94 passed in 34.11s +``` + +上一轮已执行 IsolatedCad 和 worker monitor 的 60 项验证,本轮不重复当成新结果。测试通过说明已有行为与保护仍按测试约定运行,不证明上述异常路径合理。 + +仅增加分析报告和 `build/day0-resource-analysis-20261026/` 下的诊断脚本/证据。生产 Python、正式 VBS、版本号及发布包未修改,未向远端工单发消息或关闭工单。 + +下一轮修改应优先针对已复现的四个问题,并保留原有身份校验、写前保护和禁止不确定重放。验收必须分别报告数据结果、工程关闭结果、应用退出方式、后代进程退出方式和最终资源状态,不能汇成一个“释放成功”。 diff --git a/docs/testing/2026-10-26-day0-resource-release-analysis.md b/docs/testing/2026-10-26-day0-resource-release-analysis.md new file mode 100644 index 0000000..e728723 --- /dev/null +++ b/docs/testing/2026-10-26-day0-resource-release-analysis.md @@ -0,0 +1,140 @@ +# Day0:资源释放工单原因分析 + +日期:2026-10-26(北京时间)。严重级别依据用户本次升级说明;未修改远端工单标签或状态。分析源码基线:`b9391015650bf5c43a77a017c5484e9a8f352266`。 + +后续已完成[核心应用生命周期专项排查](2026-10-26-application-lifecycle-investigation.md):包含两工程真实保存与独立读回、vnsd 自然退出对照及三类异常控制流复现。后续结论以该记录注明的证据和边界为准。 + +## 判断 + +工单 #1 的现场根因仍未闭环。根据用户对核心应用生命周期的补充,排查主线调整为“业务操作完成 → 保存确认 → 工程关闭确认 → 应用正常退出 → 进程及辅助资源释放 → 桌面回收”。桌面是隐藏操作的基础设施,其消失不能替代业务完成或进程正常退出证明。第一轮对桌面末端症状的侧重不足以覆盖保存与异常退出风险。 + +对已经观察到的桌面残留,需要继续追查自有清理范围以外的引用,包括本程序未登记的额外引用、无窗口线程、其他进程持有的桌面句柄和由系统或服务创建的辅助进程。这里的“范围以外”不等于已经证明是第三方软件造成,也不能将其提前确定为整个事故的业务根因。 + +工单 #3 在 CAD 未启动时出现同类残留,说明 CAD Quit 或 COM 释放不足不能解释全部现场现象。两个工单具有相同末端症状,但尚不能断言具体持有者或底层根因相同。 + +Beta 3 的进程隔离、批次复用和资源复核覆盖了部分故障机制。产品仍缺少现场持有者的完整识别能力,缺少对应现场版本的复测成功回执,不能凭本机通过将事故关闭。 + +## 补充核查:核心应用完成和退出链路 + +本节优先于后文仅针对桌面残留的原因排序。关闭桌面句柄不会自动结束进程;当前实现实际先清理受控进程、再关闭桌面句柄。应分别证明数据保存成功、工程关闭成功、应用正常退出、强制终止是否发生和资源是否消失。 + +### 正常路径已有保存回读,但完成语义仍需核实 + +- `QuickPartReplace.vbs:863` 的 `QPCloseBatchProject` 调用 `Document.Close(True, "")` 保存关闭图页,随后调用 `CloseProject`,立即查询工程路径为空。 +- `QuickPartReplace.vbs:934` 起会重新打开同一工程、校验目标及非目标内容,之后才给出 `saved_ok`。不能把现有程序描述为“完全没有等待或没有保存核验”。 +- COM 调用返回是现有的等待边界,但当前函数没有单独的保存完成事件、持久化提交状态或忙碌状态等待。是否需要增加这些机制,必须核对该版本 CAD 的 API 契约及真实行为;不能仅因缺少 sleep 就断言保存是异步或保存未完成。 +- 生产路径是在同一个 CAD 实例中重开工程进行回读。这提供了内容核验,但若要证明进程退出后的持久性,应增加正常退出后的独立新实例读取实验,并核查实际数据库服务的提交语义。不能仅凭同实例回读断言存在缓存问题,也不能将开发验收中的独立回读当成每次生产执行都有此保障。 +- 图页/工程关闭后立即查询一次状态,没有针对“调用已返回、关闭状态稍后才就绪”的有界状态等待。若厂商允许这种延迟,可能把仍在正常收尾的应用判为失败;目前还没有确认其 API 是这种行为。 + +### 明确存在的风险路径:应用未确认正常退出时进入强制清理 + +`isolated_cad.py:207` 的退出流程为:Quit helper 最多等待 20 秒;helper 成功才进一步等待 CAD 进程最多 15 秒。Quit helper 超时或失败时会跳过这次 CAD 正常退出等待,记录 uncertain,然后仍执行 `desktop.close()`。在批次模式中这里首先是 lease 的进程清理,随后才是批次拥有者的桌面关闭;底层 `_Win32Process.close()` 对存活自有进程/Job 有终止逻辑。 + +因此存在需要验证的链路:应用仍忙或关闭被拒绝 → 固定时限/单次状态检查失败 → 保护标记 → 强制终止自有进程 → 后续桌面释放检查。保护标记能避免无依据重放,不能保证被终止前未完成的保存可以安全完成。强制终止后的“资源清理成功”也不能用作“业务正常完成”的证据。 + +Quit helper 在发现工程仍打开时会拒绝退出,避免隐式保存或丢弃;但外层依然会进入自有进程清理兜底,所以必须检查当时是否有未完成的写操作。业务状态、退出方式和资源状态需要分开判断。 + +### 保存期间的无进度监控需要专项验证 + +`QuickPartReplace.vbs:874` 在进入保存关闭前发出一次 `save_and_close` 进度。若后续阻塞在 COM 调用中,脚本不能继续发送阶段进度。当前原生侧进度旁路只处理 `native_snapshot_read`,没有相同的保存进度旁路。 + +默认无进度时限为 150 秒。`ProgressWatch.action()` 在 150 秒请求停止、300 秒判定 unresponsive;这个判定不因 `safe_stop=False` 而停止升级。VBS 会在安全边界才响应协作停止,但监督器仍可在后续超时中终止 worker,并进入 CAD 收尾。真实慢保存是否能达到这个条件尚未实测;普通进程存活或无条件心跳也不能当成保存正在推进的证据。 + +### 本轮验证与证据边界 + +- `python -m pytest -q tests/v1_1/test_isolated_cad.py tests/trial_task_center/test_worker_monitor.py`:**60 passed,16.22 秒**。包括控制流替身和真实辅助进程测试,没有真实 CAD 慢保存实验。 +- 对生产 `IsolatedCadSession` 注入模拟 Quit helper 超时:确认等待参数为 20 秒、CAD wait 未被调用、uncertain 标记已写入、之后进入资源 close。未终止真实 CAD。 +- 对生产 `ProgressWatch` 输入模拟 `save_and_close / safe_stop=False`:150 秒为 request_stop,300 秒为 unresponsive。 +- 证据:`build/day0-resource-analysis-20261026/application-lifecycle-probe.json`。这是控制流证据,不能写成现场已发生保存中断或数据丢失。 +- 原始 #1 是库存读取后的收尾失败,四次日志记录 Quit helper/CAD 正常退出;#3 尚未启动 CAD。当前附件没有证明替换保存被提前打断。上述生命周期风险必须调查,但不能直接取代这两组现场证据。 + +优先实验应改为:在工程副本上记录保存调用前后、关闭调用前后、回读结果、Quit 请求与返回、CAD 自然退出、强制终止请求及原因的完整时间线;分别验证正常保存、长时间忙碌、关闭延迟和退出拒绝。数据正确性及退出契约查清后,再处理桌面末端残留。无需通过开启可见桌面或无界等待来替代这些完成条件。 + +## 已核实的证据 + +### 工单 #1:V1.1.3,四次资源释放确认失败 + +原件位置:`build/issue-1-20261021/evidence/`。四份失败的 `private-desktop.json` 一致记录: + +- CAD 和退出辅助程序的退出码均为 0。 +- 所记录进程的 `job_active_processes=0`,已知后代句柄均已确认退出,自有进程及 Job 句柄关闭。 +- `desktop_handle_closed=true`,但 5 秒后 `released=false`。 +- 四次清理来自同一个主程序 PID 14608,调用线程不完全相同,调用线程桌面为 `Default`。 +- 另有一份采集中途取消的记录停在 `ready`,没有最终收尾证据;不能算作释放成功,也不能直接算作第五次同因泄漏。 + +这些证据降低了“已登记 CAD 主进程没有退出”“完全没等待终止”“已登记桌面句柄没有关闭”的优先级。它们不能证明进程所有线程、所有其他桌面句柄或所有系统辅助进程都已释放。 + +### 工单 #3:Beta 2,CAD 未启动也残留 + +本地 `build/issue-3-20261023/application.log` 与本次重新读取的远端附件逐字节一致。SHA256:`28d164039831973b7f6596a081d305421131f883514feae2fd652ff90ec54241`。 + +| 北京时间 | 日志位置 | 事实 | +| --- | --- | --- | +| 10-23 14:29:13 | 第 2 行 | 运行版本 `1.1.4-beta.2`,Windows 基础 build 26200 | +| 14:29:34 | 第 14 行 | 开始整批 BOM 检查 | +| 14:30:01 | 第 669 行 | 开始批末释放 | +| 14:30:06 | 第 1098 行 | 桌面句柄关闭,桌面仍存在;CAD PID 为空,显示 CAD 未启动;窗口列表为空 | +| 同一次失败的异常链 | 第 1102、1116 行及后续堆栈 | 首发错误为内部 attempt 目录创建的 `FileNotFoundError / WinError 3`,失败路径长 265 字符;释放错误发生在异常收尾期间 | +| 14:30:47 | 第 1155 行 | 用户请求重新清理资源 | +| 14:30:54 | 第 2013 行 | 同一个桌面仍存在,句柄关闭、CAD 未启动、窗口列表为空 | + +前后失败样本相隔约 48 秒。未连续观测期间的状态,但相同随机桌面名两次均被枚举,明显削弱了“只需略微增加 5 秒超时”的解释。 + +`cleanup.processes=[]` 仅表示该次最终清理列表为空。原生 CLI 在批内执行后会从活动列表移除,不能据此说整批从未启动过子进程。 + +长路径是已记录的首发业务错误;桌面残留是收尾时另一个故障。修复路径问题不自动证明桌面引用来源已修复。 + +## 桌面残留的原因排序和判别办法 + +“优先级”表示调查顺序,不表示统计概率。 + +| 原因候选 | 优先级及证据 | 如何区分 | +| --- | --- | --- | +| 本程序的其他线程或额外桌面句柄仍持有引用 | 高;旧版 GUI 生命周期长,关闭登记句柄不能排除额外引用。四次 #1 来自同一主程序 | 现场保留桌面名,由独立观察进程在监督进程退出前后、主程序正常退出前后枚举;配合句柄和线程归属快照。随某进程退出而消失可缩小范围,但仍需持有者证据,不能单凭时间相关性定责 | +| Job 外的无窗口辅助线程/进程 | 高;完全符合“Job 清空但桌面存在”。历史受控实验发现过无窗口 Python 和 conhost 线程 | 对目标桌面扫描全局线程,记录 PID/TID、映像、创建时间、会话及 Job 归属;与原生 CLI 启停时间对齐。历史 conhost 只是实验线索,现场未确认 | +| 其他进程仅持有 OpenDesktop 句柄 | 高;可在不启动 CAD、没有窗口和目标线程的情况下复现相同症状 | 在线程扫描之外查询 Desktop 类型句柄。已有外部句柄实验复现阻断,持有者释放后可恢复;具体现场持有者未知 | +| 原生 CLI、运行库、服务代理触发额外引用 | 中高;#3 没启动 CAD,但此前走过原生读取链 | 在现场副本依次运行空桌面、简单子进程、单个 icdb2ascii 读取、多个读取、CAD 回退,找出首次出现残留的阶段。普通子进程默认受 Job 约束,不能笼统断言所有后代都逃逸 | +| 安全软件、录屏、远程协助、桌面管理组件等参与桌面访问 | 中,属于上一类的来源候选;当前没有具体软件或 PID 证据 | 先采集持有者或模块证据,再做单因素环境对照;没有证据时不能指定某个软件为元凶 | +| Windows USER 引用延迟回收/系统环境差异 | 中低;短暂延迟有受控实验支持,但现场约 48 秒后的重查仍失败 | 使用独立进程记录 0、1、5、15、60 秒存在性,同时记录所有者退出状态。本机和 #3 均为基础 build 26200,仍需对照累计更新、运行库和注入组件,不能只按系统大版本归因 | +| 句柄继承、Job 接管或异常清理分支缺陷 | 中低,需保留;当前代码在恢复运行前分配 Job,重定向使用明确的继承句柄清单,已有后代退出和句柄关闭失败测试 | 检查实际运行 EXE 哈希、Job 分配结果、句柄继承标志、失败时保留的所有权记录。当前源码机制不能倒推所有旧版或现场程序完全相同 | +| 历史资源记录损坏或残留,形成逻辑阻断 | 对“升级后仍不能用”须优先区分;代码会保守阻断坏记录 | 对比当前物理桌面/进程与 `session.json`、`supervisor.json` 的状态。若物理对象已消失而记录无法校验,是证据恢复问题;不应归为仍有物理泄漏。原始 #1/#3 明确枚举到桌面,坏记录不能解释其最初残留 | +| CAD COM 引用未释放 | 对 CAD 路径仍可查;作为统一解释优先级低 | 仅在已启动 CAD 的场景核对 COM、helper 与 CAD 生命周期。#3 的 CAD 未启动是其无法解释全部现象的直接证据 | +| 工程数量上限、桌面堆耗尽、纯 BOM 匹配错误 | 当前证据弱 | 查桌面创建失败、内存资源错误和随批量增长的资源曲线。当前主要失败在销毁确认;BOM 行不匹配是独立业务结果,不能据此改资源配额或放宽匹配 | + +微软文档确认:关闭一个桌面句柄与销毁所有引用不是一回事;线程的桌面连接不依赖存在可见窗口;Job 默认覆盖普通 CreateProcess 子进程,但不是整个系统的进程集合。例如 WMI `Win32_Process.Create` 创建的进程不属于这类默认继承范围。此机制说明需要查 Job 外所有者,不代表本软件已经被证明使用了 WMI 启动逃逸进程。 + +## 当前实现为何仍可能卡住 + +1. **判定严格,但归属诊断不足。** `isolated_desktop.py:66` 通过枚举确认桌面是否消失;失败时只收集窗口。`isolated_desktop.py:565` 明确将外部句柄持有者标为 unknown。空窗口不是无引用证明。独立工具 `tools/diagnose_private_desktop.py` 能查无窗口线程,但不能列出所有仅持有句柄的进程,权限错误也必须保留。 +2. **监督进程隔离只解决其覆盖的所有权。** `batch_supervisor.py:279` 等待并关闭受控 Job,随后 `_finish_response` 复核资源。这能够处理监督进程自身存活期间的引用;其他长寿命进程的引用不会因该监督进程退出而消失。 +3. **重新清理有能力边界。** `batch_resources.py:117` 对仍有 live owner 的会话可重试其关闭;没有 live owner 时只确认桌面消失和历史 PID 身份,不会接管或终止不明持有者。Beta 3 常见的是监督进程已经退出后的复核分支。反复点击不会凭空释放外部引用。 +4. **未释放门禁扩大可用性影响。** `guard_pending_resources` 阻止同一数据根的新批次。这一保护避免在旧资源不明时继续创建桌面或重放业务,却使一个释放问题阻断后续使用。恢复机制和持有者识别必须共同补齐。 +5. **损坏记录不会自动修复。** 无法校验的记录保持 blocked;无 session_id 时跳过自动资源操作。这是证据不足时的保护行为,但意味着资源已消失仍可能需要审计式人工核对。不能通过删除保护文件掩盖该状态。 +6. **诊断包不等于实时取证。** 当前导出函数主要打包已有文本,未自动执行全局线程或句柄扫描。上限为单文件 4 MiB、总计 32 MiB、512 个文件,超限写入 omissions;必须检查 manifest,不能把未导出的记录当成从未发生。 + +## 本次新验证 + +运行环境:Python 3.12.10、Windows 11 基础 build 26200。以下测试未打开或替换用户工程。 + +- `python -m pytest -q tests/v1_1/test_isolated_desktop.py tests/v1_1/test_desktop_diagnostics.py tests/v1_1/test_batch_runtime_windows.py tests/v1_1/test_batch_supervisor.py`:**44 passed,87.89 秒**。包含真实 Windows 资源测试与替身测试,覆盖无窗口引用、延迟释放、后代进程、诊断边界、监督恢复等;不等同于现场完整 EXE 验收。 +- `python -m pytest -q tests/v1_1/test_batch_runtime.py::test_unknown_resource_evidence_never_authorizes_cleanup tests/v1_1/test_batch_runtime.py::test_restart_reconciles_only_confirmed_disappearance_without_replacement`:**4 passed,1.22 秒**。确认坏证据继续阻断、重启后仅复核的代码行为;这两类为模拟边界测试。 +- 资源循环探针 `build/day0-resource-analysis-20261026/probe_cycles.py`:20 次空桌面、20 次带重定向的简单 Python 子进程,**40/40 正常释放**。自有桌面残留 0,观察前后没有新增桌面,调用线程桌面未改变。两组的释放枚举检查最大耗时均约 0.016 秒,该数字不包含完整子进程启动耗时。 +- 循环证据:`build/day0-resource-analysis-20261026/cycles-6819874fa1b04649a6dbb9c6b1992034.json`。它是源码 Python 基础路径实验,没有覆盖 Qt GUI、冻结 EXE、实际 icdb2ascii、CAD 或现场常驻组件,不排除这些组合中的问题。 + +这批结果降低了本机基础资源封装普遍失效的可能性,不能证明现场根因已修复,也不能证明 102 个原工程全部可用。 + +## 下一步应取得什么证据 + +1. **先确认事故实际版本和状态。** 当前收到的两组原件属于 V1.1.3 与 Beta 2;没有 Beta 3 的新失败诊断包。记录现场 EXE 哈希、是否完整配套 `_internal`、原始 data 是否迁移、故障是再次产生桌面还是旧记录阻断。Beta 3 公开 EXE 的历史记录哈希为 `df08cd328f6f4c9019f232c845e09627551e7b2d24790a466d53efcd291622d1`,以实际安装文件核对。 +2. **残留发生时立即取证。** 在原机器、原登录会话,从 session 记录取得准确桌面名;运行独立线程诊断并查询 Desktop 句柄持有者,保留 PID、创建时间、映像、线程、会话、Job 和时间戳。仅复制旧日志到开发机无法找回消失的 Windows 引用。 +3. **独立观察所有者退出前后。** 先保存结果及诊断,再观察监督进程退出、应用正常退出后的同名桌面变化。桌面随监督进程退出消失,优先追监督进程内部引用;随 GUI 退出消失,追 GUI 或其关联组件;GUI 退出后仍存在,追其他进程或系统引用。上述变化用于缩小范围,最终需结合持有者证据确认因果。 +4. **按最小场景逐层复现。** 空桌面 → 简单 worker → 单次原生 CLI → 多次原生 CLI → CAD 回退 → 批量混合。每层记录创建数量、已退出进程、最终桌面状态。实际读取使用工程副本,首次失败就保留证据,避免反复业务执行污染现场。 +5. **修复后验证恢复结果。** 对已识别的持有者修复生命周期;完成正常、异常、取消、重启和跨批次检查,确认只清理资源时业务结果不变、没有重复替换,随后在现场原目录和原批量范围验收。只有现场复测通过并留存回执,才具备关闭工单的依据。 + +技术交付首先补齐业务完成与应用退出的分阶段证据,并专项验证等待与强制清理边界;对完成业务后仍残留的桌面,再补自动捕获持有者的能力。单纯延长等待、反复要求重新清理或继续发布未取得现场证据的候选包,都不足以建立闭环。 + +## 关联记录 + +- [工单 #1](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/1)、[工单 #3](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/3)。 +- [首轮工单分析](2026-10-21-issue-1-bom-import.md)、[Windows 生命周期受控实验](2026-10-21-windows-desktop-lifecycle.md)、[外部句柄持有者实验](2026-10-23-issue-3-external-holder.md)、[Beta 3 验收边界](2026-10-23-issue-3-safe-recovery.md)。 +- 微软:[CloseDesktop](https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-closedesktop)、[线程连接桌面的规则](https://learn.microsoft.com/en-us/windows/win32/winstation/thread-connection-to-a-desktop)、[Job Objects](https://learn.microsoft.com/en-us/windows/win32/procthread/job-objects)。本次重新核对官方页面。 diff --git a/docs/testing/2026-10-26-safe-stop-acceptance.md b/docs/testing/2026-10-26-safe-stop-acceptance.md new file mode 100644 index 0000000..0737de8 --- /dev/null +++ b/docs/testing/2026-10-26-safe-stop-acceptance.md @@ -0,0 +1,59 @@ +# 用户停止与安全收尾实施验收 + +日期:2026-10-26;分支:`codex/day0-safe-stop`。实现用户在本会话批准的[停止流程](../superpowers/specs/2026-10-26-safe-user-stop.md),对应[全链路调查](2026-10-26-application-lifecycle-investigation.md)。 + +## 实现行为 + +- 普通暂停和关闭窗口不启动下一项,当前写入、保存和回读安全完成后停止。当前项并不等于整个工程。 +- 有写入可能的阶段长期没有进度时显示等待,不自动把超时升级成杀进程。纯只读采集保留有界取消。 +- 写入失败、保存结果不确定或 Quit 被拒绝/超时且 CAD 仍存活时,交互执行保留应用与工程,等待处理,不自动保存或丢弃。 +- “强制结束…”独立确认;先持久化业务保护及批次中断证据,再终止明确自有的监督进程及 Job。证据写入或 fsync 失败时拒绝该次 force,界面保留重试能力。 +- 保存请求、保存调用返回、保存回读和工程关闭分别记录,工程关闭失败不会丢失此前保存边界。 +- 收尾记录写入失败不跳过满足身份检查的正常 Quit。主进程自然退出后给予自有后代 2 秒自然退出窗口,之后的强制清理单独记录。 +- 资源消失不吞掉异常退出;调用方收到独立收尾错误,已核验业务结果仍附带保留。 +- 等待期间可独立导出诊断包。关闭窗口在操作结束前不退出;失败/强制结束后取消自动关闭,保留恢复提示。 + +## 自动验证与审查 + +针对安全边界先运行失败用例再实现,包括慢保存不会被 watchdog 终止、明确 force 的保护先行、保护写入失败保留进程、已有文件也重新 fsync、保存后工程关闭拒绝、窗口关闭不触发 force、确认取消无副作用。 + +真实辅助进程测试覆盖监督子进程强制中断、控制回调异常时保持子进程存活、拒绝第一次保护写入后仍允许后续明确重试、资源清理不重放业务。CAD 调用在这些故障测试中使用替身,不能据此声称真实保存中断无损。 + +退出协议独立审查修复了部分子进程查询失败时丢失已知强制终止 PID 的问题。整体验证另外修复:只读入口没有 force 出口却进入无限等待、证据首次 fsync 失败后因文件存在被误放行、已执行 force 被诊断异常降级为普通停止、控制通信异常绕过确认清理、直接执行 force 失败没有反馈,以及 force 意图与监控异常并发的竞态。复审未发现新增阻断问题。 + +首次完整实现回归 **1063 passed,1 条既有 pywinauto STA 警告,363.82 秒**。之后完成审查修正,最终扩大回归 **1147 passed,1 条既有 pywinauto STA 警告,325.64 秒**: + +```powershell +python -m pytest -q tests/v1_1 tests/trial_task_center/test_application.py tests/trial_task_center/test_execution_safety.py tests/part_replace/test_verification.py tests/integration/test_quick_value_modes.py tests/integration/test_quick_session_startup.py tests/integration/test_quick_part_replace_contract.py tests/integration/test_automatic_replacement_regression.py tests/trial_task_center/test_worker_monitor.py tests/trial_task_center/test_batch.py +``` + +本命令包含项目发布脚本要求的源码测试范围,并补充此次受影响的 VBS、worker monitor 与批次测试。`git diff --check` 与新探针/停止模块语法检查通过。 + +两个旧 VBS fixture 在基线 `b939101` 也失败:未填 owned project path 且错误期望 no-match 保存、图页清单断言未使用规范化路径。修正的是 fixture 和断言,没有放宽生产所有权或保存规则;脚本契约组 72 项通过。 + +## 真实保存期间请求普通停止 + +工具:[probe_safe_stop_live.py](../../tools/probe_safe_stop_live.py)。本轮输出为 `build/safe-stop-live-a9113545bf/`,报告 `report.json`,最终资源复核 `final-resource-check.json`。 + +入口为源码 supervised service → 真实独立 batch worker → 生产 VBS → 真实 CAD,使用两份全新 Audio_D2128/Audio_T605A 副本。监听第一工程的生产 `save_requested` 日志后只设置普通停止 Event,不设置 force。此触发处于保存请求附近,不能声称精确暂停在 COM 内某一条指令。 + +| 检查 | 结果 | +| --- | --- | +| 第一个工程 | 当前 RV05 替换、保存和回读完成,任务 completed | +| 第二个工程 | 保持 pending,没有继续替换 | +| 队列 | paused=true,未请求强制结束 | +| 第一工程新 CAD 独立读回 | 57 器件、103 引脚连接;目标 RV05 为新料号,56 个非目标器件保持;目标身份、位置和连接保持 | +| 第二工程新 CAD 独立读回 | 57 个器件的完整属性及连接图与执行前一致 | +| 执行批次退出 | orderly_shutdown=true、clean=true、forced_termination=false | +| 资源 | 资源记录全部 released,最终枚举确认其桌面消失,没有本轮残留 CAD | +| 原件 | 两个源工程目录哈希不变,中央库 `.lmc` 哈希不变 | + +没有重放任何已经完成的替换;测试用的 before/after 独立读取使用新 CAD 实例。 + +## 边界 + +本轮没有构建或发布新 EXE,未做完整冻结 GUI 的保存中强制结束、掉电或原工单全量现场复测。GUI 行为使用 Qt 测试验证,真实保存停止使用源码监督入口验证;二者不能混写为冻结 GUI 全流程验收。 + +强制结束为避免切换任务竞态,保守标记本批可能涉及的目标;已完成业务证据和结果保留,但先前已完成任务也可能需要恢复核验。不能清除保护来继续。 + +保持应用等待不代表保存结果一定可以恢复。外部任务管理器强杀/系统掉电没有安全收尾保证,重启必须依赖原先写前证据核验。工单现场具体桌面持有者仍未查明,本实现不能当作现场根因已经闭环。 diff --git a/docs/user/progress-timeout.md b/docs/user/progress-timeout.md index 6696010..b448fd5 100644 --- a/docs/user/progress-timeout.md +++ b/docs/user/progress-timeout.md @@ -1,5 +1,7 @@ # R8:进度式超时与分阶段引脚索引 +> 本文保留 R8 历史行为。2026-10-26 起,有写入可能的阶段无进度超时改为等待安全收尾,只有明确确认的强制结束才中断;当前停止与恢复规则见[停止、强制结束与恢复](safe-stop-and-recovery.md)。只读采集继续采用有界取消。 + ## 改变的行为 - 150 秒不再是整个任务的总时限,而是连续没有有效进度的观察窗口。有有效进度时可以继续执行超过 150 秒。 diff --git a/docs/user/safe-stop-and-recovery.md b/docs/user/safe-stop-and-recovery.md new file mode 100644 index 0000000..da1e96a --- /dev/null +++ b/docs/user/safe-stop-and-recovery.md @@ -0,0 +1,30 @@ +# 停止、强制结束与恢复 + +执行期间点击“完成当前项后暂停”,程序不再启动下一替换项。已经进入修改或保存的当前项会继续完成必要校验、保存、重开回读和工程关闭,然后正常退出 CAD,并等待自有辅助进程收尾。关闭主窗口也遵循这个流程,操作仍在进行时窗口保持打开。 + +“当前项”是当前图页和替换规则对应的执行单元,不是整个工程或整个批次。已经保存核验的结果保留,未开始的任务保持未执行。 + +## 保存或修改长时间没有进度 + +有写入可能的阶段,程序不会仅因超过无进度时限就自动终止应用。界面显示保存或核验尚未确认,可以继续等待并导出诊断包。等待不会启动下一项,也不会自动清除保护记录。 + +如果操作已经失败且保存结果不确定,程序保留当前应用和工程现场,不再自动保存或丢弃。应先导出诊断、保留原始工程与备份,再决定是否需要强制结束。此时“继续等待”不表示自动恢复执行。 + +纯读取的 BOM/库存采集仍采用有界取消;不存在写入需要完成时,不要求用户无限等待。确认未修改的启动失败也保留既有有界停止机制。 + +## 明确选择强制结束 + +“强制结束…”与普通暂停分开。确认框明确提示:当前操作可能正在修改或保存,强制结束可能留下未保存或部分保存结果,之后必须先核验恢复。 + +确认后,程序先持久化中断保护和批次记录,再终止自己拥有的监督进程及 Job。保护记录无法可靠保存时,不执行这次强制结束,界面提示原因并允许重试或导出诊断。取消确认没有终止副作用。 + +为避免强制请求与工程切换同时发生而漏记目标,强制结束会保守保护本批可能涉及的任务。先前已保存结果不会被覆盖或重放,但这些任务也可能显示需复核或被保留在当前列表。不要删除保护文件绕过核验。 + +## 程序重启之后 + +- 资源复核只处理进程和桌面状态,不重新替换。 +- 结果不确定的任务必须使用“核验并恢复”,按照已有快照和当前实际工程核对。 +- 已保存核验结果继续保留;资源清理成功不自动解除业务保护,也不自动继续队列。 +- 如果应用没有正常退出或后代进程需要强制清理,会单独报告收尾异常,不能把“资源已释放”理解为“保存和退出均正常”。 + +任务管理器强杀、系统关机或掉电可能不给程序写最终日志的机会。这些情况依赖修改前已经落盘的证据及下次启动的中断核验;程序不能保证被外部强杀时完成当前保存。 diff --git a/integration/dxdesigner795/QuickPartReplace.vbs b/integration/dxdesigner795/QuickPartReplace.vbs index 7dd3e9d..b3b493d 100644 --- a/integration/dxdesigner795/QuickPartReplace.vbs +++ b/integration/dxdesigner795/QuickPartReplace.vbs @@ -28,7 +28,7 @@ Dim QP_ComponentUidIndex, QP_ComponentRefdesIndex, QP_ComponentLocationIndex Dim QP_ComponentIndexContext Dim QP_NativeCallActive Dim QP_BatchMode, QP_OwnedProject, QP_BatchState, QP_BatchCode, QP_BatchMessage -Dim QP_BatchSaveCommitted +Dim QP_BatchSaveCommitted, QP_BatchSaveRequested Dim QP_DiscoveryMode, QP_TargetSchematic, QP_TargetSheet Dim QP_FailureNumber, QP_FailureSource, QP_FailureDescription Dim QP_NcSyncSummary @@ -60,6 +60,7 @@ QP_BatchMode = False QP_OwnedProject = False QP_BatchState = "" QP_BatchSaveCommitted = False +QP_BatchSaveRequested = False QP_DiscoveryMode = False QP_TargetSchematic = "" QP_TargetSheet = "" @@ -110,6 +111,10 @@ Else QP_FailureDescription = Err.Description If QP_DiscoveryMode Then QPCleanupDiscoveryFailure FailureState = "hard_failure" + If QP_BatchSaveRequested Then + FailureState = "batch_saved_unverified" + QP_FailureDescription = "save was requested; persisted outcome requires reconciliation. " & QP_FailureDescription + End If CloseError = "" If QP_BatchMode Or QP_SessionMode Then QPDiscardFailedBatchProject CloseError @@ -318,6 +323,7 @@ Sub QPSessionNext() QP_BatchMode = True QP_BatchState = "" QP_BatchSaveCommitted = False + QP_BatchSaveRequested = False QP_NativeMutationPossible = False If Trim(QPReadUtf8(QP_Fso.BuildPath(QP_Root, "batch.request"))) <> QP_ExpectedJobId Then _ Err.Raise vbObjectError + 873, "QuickPartReplace.ProjectSession", "next batch authorization mismatch" @@ -872,12 +878,21 @@ Sub QPCloseBatchProject(ByVal ExpectedPath, ByVal SaveChanges) If Documents.Count <> 1 Then Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "open sheet set changed; no automatic close" Set Document = Documents.Item(1) QPProgress "save_and_close", 0, 1 + If SaveChanges Then + QP_BatchSaveRequested = True + QPLog QP_Root, "INFO", "save_requested job=" & QP_CurrentJobId + End If Document.Close CBool(SaveChanges), "" + If SaveChanges Then + QP_BatchSaveCommitted = True + QPLog QP_Root, "INFO", "save_returned job=" & QP_CurrentJobId + End If If QPCanonicalPath(QPProjectPath()) <> QPCanonicalPath(ExpectedPath) Then _ Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "project changed during sheet save" QP_App.CloseProject If Len(CStr(QP_App.GetProjectData().GetProjectFilePath())) > 0 Then _ Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "project did not close; queue stopped" + QPLog QP_Root, "INFO", "project_closed job=" & QP_CurrentJobId QP_OwnedProject = False QP_OwnedProjectPath = "" QPInvalidatePinIndex @@ -888,6 +903,10 @@ Sub QPDiscardFailedBatchProject(ByRef ErrorDescription) Dim ErrorNumber, ErrorSource, Description ErrorDescription = "" If (Not QP_BatchMode And Not QP_SessionMode) Or Not QP_OwnedProject Or QP_App Is Nothing Then Exit Sub + If QP_NativeMutationPossible Or QP_BatchSaveRequested Or QP_BatchSaveCommitted Then + ErrorDescription = "write or save outcome requires reconciliation; owned project retained without automatic save or discard" + Exit Sub + End If On Error Resume Next QPCloseBatchProject QP_OwnedProjectPath, False ErrorNumber = Err.Number @@ -932,7 +951,6 @@ Sub QPFinishBatchItem() If FinalState = "restored_ok" Then QPMarkBatchMutation Command QPCloseBatchProject TargetPath, True - QP_BatchSaveCommitted = True ElseIf Not QP_SessionMode Then QPCloseBatchProject TargetPath, False End If @@ -952,6 +970,7 @@ Sub QPFinishBatchItem() QPVerifyRestoredPlans Command, Records, Plans, ValueStates If Not QP_SessionMode Then QPCloseBatchProject TargetPath, False FinalState = "saved_ok" + QPLog QP_Root, "INFO", "saved_readback_verified job=" & QP_CurrentJobId If QP_SessionMode Then QP_BatchMessage = QP_BatchMessage & " Saved and reopened for readback; owned project checkpoint verified." Else diff --git a/src/dx_productivity/trial_task_center/batch_runtime.py b/src/dx_productivity/trial_task_center/batch_runtime.py index 7ad2fa7..953cfc2 100644 --- a/src/dx_productivity/trial_task_center/batch_runtime.py +++ b/src/dx_productivity/trial_task_center/batch_runtime.py @@ -66,10 +66,11 @@ class _DesktopLease: for process in reversed(self.processes): try: self.owner._release_process(process, desktop=self.desktop, generation=self.generation) - records.append(getattr(process, 'cleanup_record', {})) except Exception as error: errors.append(str(error)) remaining.append(process) + finally: + records.append(getattr(process, 'cleanup_record', {})) self.processes = list(reversed(remaining)) self.cleanup_record = {'scope': 'operation', 'processes': records, 'workers_closed': not errors, 'desktop_retained_by_batch': True, 'errors': errors} @@ -99,6 +100,7 @@ class BatchCadRuntime: self._retry_counts = {} self._project_attempts = {} self._logging = None + self._active_evidence = None def bind_logs(self, context, task_ids): """Keep every participating task's final release outcome visible.""" @@ -116,6 +118,14 @@ class BatchCadRuntime: return self def __exit__(self, error_type, error, traceback): + if error is not None and self._active_evidence is not None: + attempts = [item for values in self._project_attempts.values() for item in values] + try: + self._retain_uncertain(self._active_evidence, None, attempts, error) + except WorkerMonitorError: + if not getattr(getattr(self.stop_event, 'force_event', None), 'is_set', lambda: False)(): + raise + # Explicit force has durable protection; cleanup is now allowed. try: self.close() except Exception as cleanup: @@ -190,7 +200,8 @@ class BatchCadRuntime: started = time.monotonic() self._save(cad_start_requested=True) cad = factory(self.path.parent / f'generation-{self.generation + (self.desktop is None)}', - abort_event=abort_event, desktop=lease) + abort_event=abort_event, desktop=lease, + **({'stop_control': self.stop_event} if hasattr(self.stop_event, 'force_event') else {})) self.cad = cad # Retain cleanup ownership even if startup fails. try: cad.__enter__() @@ -247,6 +258,7 @@ class BatchCadRuntime: from .isolated_cad import PrivateCadStartupError, is_identity_conflict from .worker_monitor import run_monitored_process root = canonical_path(command[3]) + self._active_evidence = root while True: check_cancelled(abort_event or self.stop_event) attempt_id = uuid4().hex @@ -272,7 +284,11 @@ class BatchCadRuntime: def start_worker(launch, **options): return cad.start_worker(launch, extra_environment=environment, **options) result = run_monitored_process(command, timeout, abort_event=abort_event, - cooperative_abort=cooperative_abort, process_factory=start_worker) + cooperative_abort=cooperative_abort, process_factory=start_worker, + **({'force_control': self.stop_event} if hasattr(self.stop_event, 'force_event') else {})) + if result.returncode: + self._retain_uncertain(root, project, expected, + WorkerMonitorError(result.stderr or 'worker failed after possible mutation')) self._checkpoint(root, project, attempt_id) if result.returncode not in (2, 3): return result @@ -294,6 +310,7 @@ class BatchCadRuntime: and not (abort_event is not None and abort_event.is_set()) and not (self.stop_event is not None and self.stop_event.is_set())) if not safe: + self._retain_uncertain(root, project, expected, failure) raise failure key = str(project).casefold() self._retry_counts[key] = 1 @@ -302,6 +319,13 @@ class BatchCadRuntime: self.close(rebuilding=True) # Prove disappearance before creating another. self.desktop = self.cad = None + def _retain_uncertain(self, root, project, expected, error): + from .batch_attempt import can_retry_attempt + from .stop_control import wait_for_resolution + attempts = self._project_attempts.get(str(project).casefold(), expected) + if attempts and not all(can_retry_attempt(path, record) for path, record in attempts): + wait_for_resolution(self.stop_event, root, error) + def recover_native_failure(self, project, evidence, reason): """One read-only recovery, based on durable source and worker proof.""" from .batch_attempt import can_retry_attempt @@ -387,7 +411,8 @@ class BatchCadRuntime: cad_record = cad_record if isinstance(cad_record, dict) else {} self.record['cad_shutdown'] = {key: cad_record[key] for key in ( 'state', 'clean', 'cad_exit_code', 'quit_helper_exit_code', 'cleanup_errors', - 'shutdown_seconds') if key in cad_record} + 'shutdown_seconds', 'orderly_shutdown', 'diagnostic_errors', 'forced_termination', + 'forced_root_termination', 'forced_descendant_pids', 'termination_reason') if key in cad_record} if self.desktop is not None: try: self.desktop.close() diff --git a/src/dx_productivity/trial_task_center/batch_supervisor.py b/src/dx_productivity/trial_task_center/batch_supervisor.py index 7a9be5a..45c4a56 100644 --- a/src/dx_productivity/trial_task_center/batch_supervisor.py +++ b/src/dx_productivity/trial_task_center/batch_supervisor.py @@ -24,27 +24,31 @@ _WRITE_OPERATIONS = frozenset({'execute_task_queue', 'execute_quick_run_queue', def _mark_interrupted(root, journal, reason): - from .batch_resources import write_record + from .batch_resources import write_record, _valid_record + from .stop_control import protect_interruption journal.update(state='blocked', outcome='interrupted', error=str(reason)) if journal['operation'] in _WRITE_OPERATIONS: # A missing final result is not proof of no mutation. Publish the same # protection the existing recovery UI understands, without overwriting # any earlier, more specific marker. journal['business_uncertain'] = True - for task_id in journal['task_ids']: + task_ids = journal['task_ids'] + resource_path = Path(root) / 'batch-resources' / journal['batch_id'] / 'session.json' + try: + resource = json.loads(internal_path(resource_path).read_text(encoding='utf-8')) + if (not journal.get('protect_all_targets') and _valid_record(resource, resource_path) + and resource.get('active_task') in task_ids): + task_ids = [resource['active_task']] + except (OSError, ValueError, TypeError): + pass # Without valid active ownership, protect every possible target. + journal['protected_task_ids'] = list(task_ids) + for task_id in task_ids: allowed = {canonical_path(root), canonical_path(Path(root) / 'tasks' / task_id)} actual = journal.get('uncertainty_roots', {}).get(task_id) roots = {canonical_path(actual)} if actual and canonical_path(actual) in allowed else allowed for evidence in roots: - marker = internal_path(evidence / 'quick-run/uncertain.flag') - marker.parent.mkdir(parents=True, exist_ok=True) - try: - with marker.open('x', encoding='utf-8') as stream: - stream.write('reason=supervisor_interrupted; batch=' + journal['batch_id'] + '; manual reconciliation required\n') - stream.flush() - os.fsync(stream.fileno()) - except FileExistsError: - pass + protect_interruption(evidence / 'quick-run', + 'reason=supervisor_interrupted; batch=' + journal['batch_id'] + '; manual reconciliation required\n') write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal) @@ -53,6 +57,61 @@ def _worker_context(root): _WORKER_ROOT = canonical_path(root) +def _force_owned_supervisor(root, journal, process): + """Persist uncertainty before terminating the exact process/Job we own.""" + journal['stop_mode'] = 'forced' + journal['force_requested_at'] = time.time() + # The running worker can cross a task boundary while evidence is written. + # Protect all possible targets before force; keep their saved results intact. + journal['protect_all_targets'] = True + _mark_interrupted(root, journal, '用户明确强制结束;写入与保存结果须核验,禁止自动重放。') + try: + process.terminate() + except Exception as error: + if not getattr(process, 'cleanup_record', {}).get('forced_termination'): + raise + journal['force_cleanup_error'] = str(error) + try: + from .batch_resources import write_record + write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal) + except OSError: + pass # The forced intent and uncertainty were already durably saved. + + +def _retain_supervisor_after_control_error(root, journal, process, control, error): + """Do not turn a GUI/transport exception into an unconfirmed Job kill.""" + from .batch_resources import write_record + control.set() + control.control_error = '控制通信异常,已请求安全停止并保留应用;可导出诊断或明确强制结束:' + str(error) + journal['protect_all_targets'] = True + try: + _mark_interrupted(root, journal, error) + except Exception as failure: + control.control_error += ';中断证据尚未落盘:' + str(failure) + try: + write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'cancel.json', + {'version': VERSION, 'batch_id': journal['batch_id'], 'attempt_id': journal['attempt_id']}) + except OSError: + pass # Keep ownership; failure to request stop is not permission to kill. + while True: + if control.force_event.is_set(): + try: + _force_owned_supervisor(root, journal, process) + except Exception as failure: + control.force_event.clear() + control.control_error = '强制结束未确认,应用仍保留:' + str(failure) + else: + return True + try: + process.wait(timeout=.2) + return False + except subprocess.TimeoutExpired: + pass + except Exception as failure: + control.control_error = '无法确认应用退出,保留所有权等待处理:' + str(failure) + threading.Event().wait(.2) + + def _file_lock(handle, acquire): from .task_queue import _os_lock _os_lock(handle, acquire) @@ -170,6 +229,19 @@ def _finish_response(root, journal, response, context): write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal) if related: raise resource_pending_error(journal['operation'], result, error, primary) + resource_path = internal_path(Path(root) / 'batch-resources' / journal['batch_id'] / 'session.json') + try: + resource = json.loads(resource_path.read_text(encoding='utf-8')) + except FileNotFoundError: + resource = {} + shutdown = resource.get('cad_shutdown') or {} + if (shutdown.get('orderly_shutdown') is False or shutdown.get('forced_termination') + or shutdown.get('diagnostic_errors') + or any(item.get('stage') != 'desktop_release' for item in shutdown.get('cleanup_errors', ()))): + failure = TaskCenterError('application_shutdown_failed', + '已核验业务结果保留,但应用未正常收尾;请查看退出诊断,不能将资源消失视为正常退出。') + failure.completed_result, failure.cleanup_error = result, error + raise failure if response.get('business_succeeded'): if response.get('error'): context.logger.info('supervisor_resource_recovery_passed batch_id=%s business_replayed=false', journal['batch_id']) @@ -186,6 +258,7 @@ def _finish_response(root, journal, response, context): def run_supervised(service, operation, arguments, *, stop_event=None, on_progress=None, on_phase=None): from .batch_resources import write_record, process_identity from .isolated_desktop import _Win32Api + from .stop_control import StopControl root = canonical_path(service._logging.root) if stop_event is not None and stop_event.is_set(): raise TaskCenterError('operation_cancelled', '本批操作已取消,未启动监督进程。') @@ -216,13 +289,15 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres journal['parent_process'] = process_identity(parent) write_record(request, {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id, 'root': str(root), 'database': str(canonical_path(service._store.path)), - 'operation': operation, 'arguments': encode(arguments), 'task_ids': journal['task_ids']}) + 'operation': operation, 'arguments': encode(arguments), 'task_ids': journal['task_ids'], + 'interactive_stop': isinstance(stop_event, StopControl)}) journal['request_sha256'] = hashlib.sha256(internal_path(request).read_bytes()).hexdigest() write_record(directory / 'supervisor.json', journal) gate, messages = MessageGate(batch_id, attempt_id), queue.Queue() api, process, reader = _Win32Api(), None, None sent_cancel, last_heartbeat = False, time.monotonic() transport_error = None + force_accepted = False def receive(): try: while line := process.stdout.readline(1024 * 1024): @@ -242,9 +317,26 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres reader = threading.Thread(target=receive, daemon=True) reader.start() while process.poll() is None or not messages.empty(): + force = getattr(stop_event, 'force_event', None) + if force is not None and force.is_set() and process.poll() is None: + try: + _force_owned_supervisor(root, journal, process) + except Exception as error: + force.clear() + stop_event.control_error = '强制结束未执行或未确认;请保留窗口和诊断:' + str(error) + service._logging.logger.error('force_stop_failed batch_id=%s error=%s', batch_id, error) + else: + force_accepted = True + break if stop_event is not None and stop_event.is_set() and not sent_cancel: - write_record(directory / 'cancel.json', {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id}) - sent_cancel = True + try: + write_record(directory / 'cancel.json', {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id}) + journal.update(stop_mode='safe', stop_requested_at=time.time()) + write_record(directory / 'supervisor.json', journal) + sent_cancel = True + except OSError as error: + if hasattr(stop_event, 'control_error'): + stop_event.control_error = '停止请求尚未成功记录,正在重试:' + str(error) try: value = messages.get(timeout=.1) except queue.Empty: @@ -264,6 +356,7 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres state = decode(value['payload']) service._task_queue_feedback = state['queue'] service._batch_feedback = state['batch'] + service._supervisor_waiting = bool(state.get('waiting_for_resolution')) if on_phase is not None and state['phase'] in {'recycling', 'closing'}: from .bom_import_state import BomImportPhase on_phase(BomImportPhase.RECOVERING if state['phase'] == 'recycling' else BomImportPhase.RELEASING, @@ -271,9 +364,12 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres process.wait(timeout=15) except Exception as error: transport_error = error + if process is not None and isinstance(stop_event, StopControl): + force_accepted = _retain_supervisor_after_control_error(root, journal, process, stop_event, error) finally: service._task_queue_feedback = None service._batch_feedback = None + service._supervisor_waiting = False try: if process is not None: process.close() # Waits, and confines termination to the owned Job. @@ -285,6 +381,14 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres 'batch_resources_pending', f'监督进程错误:{transport_error};收尾错误:{error}') if reader is not None: reader.join(3) + if force_accepted: + from .batch_resources import cleanup_pending_resources + try: + cleanup_pending_resources(root, logging_context=service._logging) + except Exception as error: + service._logging.logger.error('forced_stop_cleanup_pending batch_id=%s error=%s', batch_id, error) + raise TaskCenterError('user_forced_stop', + '已按确认强制结束;已保存的执行证据保留,当前结果待核验。请先核验恢复,不要重复替换。') try: try: response = json.loads(internal_path(directory / 'result.json').read_text(encoding='utf-8')) diff --git a/src/dx_productivity/trial_task_center/batch_worker.py b/src/dx_productivity/trial_task_center/batch_worker.py index e9195b7..0468406 100644 --- a/src/dx_productivity/trial_task_center/batch_worker.py +++ b/src/dx_productivity/trial_task_center/batch_worker.py @@ -75,7 +75,7 @@ def run(request_path): request = json.loads(request_bytes) batch_id, attempt_id = request['batch_id'], request['attempt_id'] if (request.get('version') != VERSION or UUID(batch_id).hex != batch_id or UUID(attempt_id).hex != attempt_id - or request['operation'] not in OPERATIONS): + or request['operation'] not in OPERATIONS or type(request.get('interactive_stop', False)) is not bool): raise ValueError('Invalid batch supervisor request') root = canonical_path(request['root']) directory = root / 'batch-resources' / batch_id @@ -83,7 +83,9 @@ def run(request_path): raise ValueError('Supervisor request ownership path mismatch') _admit_request(directory, request, request_bytes) _worker_context(root) - stop, finished, output_lock = threading.Event(), threading.Event(), threading.Lock() + from .stop_control import StopControl + stop = StopControl() if request.get('interactive_stop', False) else threading.Event() + finished, output_lock = threading.Event(), threading.Lock() kind = {'inspect_bom_batch': 'bom_import', 'preview_bom_batch': 'bom_import', 'execute_task_queue': 'task_queue', 'execute_quick_run_queue': 'quick_run_queue'}.get(request['operation'], request['operation']) runtime = BatchCadRuntime(root, kind, stop, session_id=batch_id) @@ -113,7 +115,9 @@ def run(request_path): emit('heartbeat') feedback = getattr(service, '_task_queue_feedback', None) emit('state', encode({'phase': runtime.record['state'], 'queue': feedback, - 'batch': service.batch_feedback()})) + 'batch': service.batch_feedback(), + 'waiting_for_resolution': getattr(stop, 'waiting_for_resolution', False), + 'resolution_error': getattr(stop, 'resolution_error', '')})) except Exception: stop.set() # Lost control evidence cannot authorize new work. finished.wait(1) diff --git a/src/dx_productivity/trial_task_center/execution_feedback.py b/src/dx_productivity/trial_task_center/execution_feedback.py index da5cffc..d68a2dd 100644 --- a/src/dx_productivity/trial_task_center/execution_feedback.py +++ b/src/dx_productivity/trial_task_center/execution_feedback.py @@ -13,6 +13,7 @@ class ExecutionFeedback: message: str = '正在准备执行' stage: str = '' stage_seconds: float = 0.0 + waiting_for_safe_completion: bool = False def read_feedback(task_id, position, total, message, root: Path, started_after: float): @@ -27,15 +28,16 @@ def read_feedback(task_id, position, total, message, root: Path, started_after: except (OSError, ValueError, TypeError, AttributeError): pass path = root / 'quick-run' / 'worker-progress.json' - stage, seconds = '', 0.0 + stage, seconds, waiting = '', 0.0, False try: modified = path.stat().st_mtime if modified >= started_after: record = json.loads(path.read_text(encoding='utf-8')) + waiting = record.get('waiting_for_safe_completion') is True candidate = record.get('stage', '') if candidate and candidate != 'finished': seconds = float(record.get('stage_seconds', {}).get(candidate, 0)) + max(0, time.time() - modified) stage = candidate except (OSError, ValueError, TypeError, AttributeError): pass # Display feedback is optional; it never changes execution state. - return ExecutionFeedback(task_id, position, total, message, stage, seconds) + return ExecutionFeedback(task_id, position, total, message, stage, seconds, waiting) diff --git a/src/dx_productivity/trial_task_center/isolated_cad.py b/src/dx_productivity/trial_task_center/isolated_cad.py index bb3916a..2313635 100644 --- a/src/dx_productivity/trial_task_center/isolated_cad.py +++ b/src/dx_productivity/trial_task_center/isolated_cad.py @@ -87,7 +87,7 @@ def _designer_executable(): class IsolatedCadSession: """The supervisor never attaches to COM on the user's input desktop.""" - def __init__(self, root, *, abort_event=None, desktop=None): + def __init__(self, root, *, abort_event=None, desktop=None, stop_control=None): self.root = canonical_path(root) self.token = uuid4().hex self.borrowed_desktop = desktop is not None @@ -96,6 +96,7 @@ class IsolatedCadSession: self.binding = None self.started = time.monotonic() self.abort_event = abort_event + self.stop_control = stop_control self.record = {'desktop': self.desktop.name, 'token': self.token, 'state': 'starting'} def _record(self, **values): @@ -206,52 +207,115 @@ class IsolatedCadSession: def __exit__(self, error_type, error, traceback): closing_started = time.monotonic() - cleanup_error = None cleanup_errors = [] + diagnostic_errors = [] + failures = [] exit_codes = {} + closing = None + def remember(stage, failure): cleanup_errors.append({'stage': stage, 'message': str(failure)}) - message = '; '.join(f"{item['stage']}: {item['message']}" for item in cleanup_errors) - self._uncertain(message, cleanup_errors=cleanup_errors, + failures.append(failure) + + def diagnose(stage, action): + try: + action() + except Exception as failure: + diagnostic_errors.append({'stage': stage, 'message': str(failure)}) + failures.append(failure) + self.record['clean'] = False + + def failure_message(): + return '; '.join(f"{item['stage']}: {item['message']}" + for item in cleanup_errors + diagnostic_errors) + + def persist_uncertain(): + message = failure_message() or str(error) + self.record.update(state='uncertain', clean=False, error=message, + cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors, shutdown_seconds=time.monotonic() - closing_started, desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes) + diagnose('uncertain_record', lambda: self._uncertain(message, + cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors, + clean=False, orderly_shutdown=self.record['orderly_shutdown'], + shutdown_seconds=time.monotonic() - closing_started, + desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes)) + + self.record.update(clean=False, orderly_shutdown=False, forced_termination=False, + forced_root_termination=False, forced_descendant_pids=[], termination_reason=None, + cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors) + diagnose('closing_record', lambda: self._record(state='closing', clean=False, + cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors)) try: - self._record(state='closing', cleanup_errors=cleanup_errors) - if error is not None: - self._uncertain(str(error)) if self.cad is not None and self.cad.poll() is not None: raise WorkerMonitorError('owned CAD exited before the authorized quit request') if self.cad is not None and self.binding is not None: closing = self.desktop.start_process(_helper_command('quit', self.root / 'private-desktop.json', self.cad.pid), env=_child_environment(self.binding.environment()), stdout=subprocess.PIPE, stderr=subprocess.PIPE) - try: - quit_code = closing.wait(timeout=20) - exit_codes['quit_helper_exit_code'] = quit_code - if quit_code: - raise WorkerMonitorError('owned CAD refused orderly exit: ' + closing.stderr.read().decode('utf-8', errors='replace')) - cad_code = self.cad.wait(timeout=15) - exit_codes['cad_exit_code'] = cad_code - if cad_code != 0: - raise WorkerMonitorError('owned CAD returned a nonzero exit code after quit') - finally: - closing.close() + quit_code = closing.wait(timeout=20) + exit_codes['quit_helper_exit_code'] = quit_code + if quit_code: + raise WorkerMonitorError('owned CAD refused orderly exit: ' + closing.stderr.read().decode('utf-8', errors='replace')) + cad_code = self.cad.wait(timeout=15) + exit_codes['cad_exit_code'] = cad_code + if cad_code != 0: + raise WorkerMonitorError('owned CAD returned a nonzero exit code after quit') + self.record['orderly_shutdown'] = True except Exception as failure: - cleanup_error = failure remember('orderly_shutdown', failure) finally: + if error is not None or failures: + # Evidence failure must never prevent the identity-checked Quit + # above or release of the handles we own below. + persist_uncertain() + if self.stop_control is not None and cleanup_errors and self.cad is not None: + cad_running = True + try: + cad_running = self.cad.poll() is None + except Exception as failure: + remember('cad_exit_check', failure) + if cad_running: + # A refused/timed-out Quit may still be saving or closing. + # Keep both CAD and its helper owned and alive until the + # interactive caller confirms force, including ordinary stop. + from .stop_control import wait_for_resolution + try: + wait_for_resolution(self.stop_control, self.root, failure_message()) + except WorkerMonitorError as failure: + force_event = getattr(self.stop_control, 'force_event', None) + if force_event is None or not force_event.is_set(): + raise + remember('forced_stop', failure) + if closing is not None: + try: + closing.close() + except Exception as failure: + remember('quit_helper_release', failure) try: self.desktop.close() - self._record(state='session_closed' if self.borrowed_desktop else 'released', - desktop_retained_by_batch=self.borrowed_desktop, - clean=error is None and cleanup_error is None, - shutdown_seconds=time.monotonic() - closing_started, - desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes) except Exception as failure: - cleanup_error = failure remember('desktop_release', failure) - if cleanup_error is not None: - message = '; '.join(f"{item['stage']}: {item['message']}" for item in cleanup_errors) - raise WorkerMonitorError('private CAD cleanup requires reconciliation: ' + message) from cleanup_error + desktop_cleanup = getattr(self.desktop, 'cleanup_record', None) or {} + forced = [item for item in desktop_cleanup.get('processes', []) if item.get('forced_termination')] + self.record.update(forced_termination=bool(forced), + forced_root_termination=any(item.get('forced_root_termination') for item in forced), + termination_reason=next((item.get('termination_reason') for item in forced + if item.get('termination_reason')), None), + forced_descendant_pids=sorted({pid for item in forced for pid in item.get('forced_descendant_pids', [])})) + if forced: + remember('forced_cleanup', WorkerMonitorError('owned processes required forced termination')) + if error is not None or failures: + persist_uncertain() + else: + diagnose('released_record', lambda: self._record( + state='session_closed' if self.borrowed_desktop else 'released', + desktop_retained_by_batch=self.borrowed_desktop, clean=True, + shutdown_seconds=time.monotonic() - closing_started, + desktop_cleanup=desktop_cleanup, **exit_codes)) + if failures: + persist_uncertain() + if failures: + raise WorkerMonitorError('private CAD cleanup requires reconciliation: ' + failure_message()) from failures[0] def _uncertain(self, message, **details): from .quick_run import _write_uncertain_flag diff --git a/src/dx_productivity/trial_task_center/isolated_desktop.py b/src/dx_productivity/trial_task_center/isolated_desktop.py index c1561a2..978119f 100644 --- a/src/dx_productivity/trial_task_center/isolated_desktop.py +++ b/src/dx_productivity/trial_task_center/isolated_desktop.py @@ -23,6 +23,7 @@ from .errors import TaskCenterError DESKTOP_RELEASE_TIMEOUT_SECONDS = 5.0 +DESCENDANT_EXIT_GRACE_SECONDS = 2.0 def _native_handle(handle): @@ -192,7 +193,10 @@ def _stop_and_wait(process): errors.append(error) if running: try: - process.terminate() + if isinstance(process, _Win32Process): + process.terminate(reason='root_process_active') + else: + process.terminate() except Exception as error: errors.append(error) try: @@ -313,7 +317,9 @@ class _Win32Process: self.job_assigned = False self.returncode = None self.stdout = self.stderr = None - self.cleanup_record = {'pid': self.pid} + self.cleanup_record = {'pid': self.pid, 'forced_termination': False, + 'forced_root_termination': False, 'forced_descendant_pids': [], + 'termination_reason': None} def poll(self): if self.returncode is not None or self.handle is None: @@ -340,34 +346,59 @@ class _Win32Process: self.returncode = self._process_api.GetExitCodeProcess(self.handle) return self.returncode - def terminate(self): + def terminate(self, *, reason='explicit_termination'): if self.handle is None: return + errors = [] + root_active = True + try: + root_active = self.poll() is None + except Exception as error: + errors.append(error) + self.cleanup_record.setdefault('root_exit_before_cleanup', not root_active) if not self.job_assigned: self._process_api.TerminateProcess(self.handle, 1) + self._record_termination(reason, root_active, ()) + _raise_cleanup_errors(errors) return - errors = [] + descendants = set() try: - self._remember_descendants() + # Keep confirmed members of this snapshot even if opening a later + # member fails; the Job termination below still includes them. + self._remember_descendants(active=descendants) except Exception as error: errors.append(error) try: self._job_api.TerminateJobObject(self.job, 1) + self._record_termination(reason, root_active, descendants) except Exception: self._process_api.TerminateProcess(self.handle, 1) + self._record_termination(reason, root_active, ()) _raise_cleanup_errors(errors) kill = terminate - def _remember_descendants(self): + def _record_termination(self, reason, root_active, descendants): + self.cleanup_record.update(forced_termination=True, + forced_root_termination=self.cleanup_record['forced_root_termination'] or root_active, + forced_descendant_pids=sorted(set(self.cleanup_record['forced_descendant_pids']) | set(descendants)), + termination_reason=self.cleanup_record['termination_reason'] or reason) + + def _remember_descendants(self, *, active=None): + if active is None: + active = set() if self._open_process is None or self.job is None: - return + return active for pid in self._job_api.QueryInformationJobObject(self.job, self._job_api.JobObjectBasicProcessIdList): - if pid == self.pid or pid in self._descendant_pids: + if pid == self.pid: + continue + if pid in self._descendant_pids: + active.add(pid) continue try: self._descendant_handles.append(self._open_process(0x100000, False, pid)) self._descendant_pids.add(pid) + active.add(pid) except Exception as error: # pywintypes.error is not an OSError. A child that exited # between the job snapshot and OpenProcess reports error 87. @@ -376,22 +407,47 @@ class _Win32Process: code = error.args[0] if code != 87: raise + return active def close(self): errors = [] + job_empty = not self.job_assigned or self.job is None if self.handle is not None: + if 'root_exit_before_cleanup' not in self.cleanup_record: + try: + self.cleanup_record['root_exit_before_cleanup'] = self.poll() is not None + except Exception as error: + errors.append(error) errors.extend(_stop_and_wait(self)) deadline = time.monotonic() + 5.0 if self.job is not None and self.job_assigned: try: - # A completed root can leave grandchildren holding the desktop. - self._remember_descendants() - self._job_api.TerminateJobObject(self.job, 1) + grace_started = time.monotonic() + grace_deadline = grace_started + DESCENDANT_EXIT_GRACE_SECONDS + # A completed root may leave children finishing normal shutdown. + # An empty job takes no grace delay; an active root was already + # stopped above and does not receive this normal-exit allowance. while True: active = self._job_api.QueryInformationJobObject( self.job, self._job_api.JobObjectBasicAccountingInformation)['ActiveProcesses'] self.cleanup_record['job_active_processes'] = active if not active: + job_empty = True + break + self._remember_descendants() + if (self.cleanup_record.get('root_exit_before_cleanup') is not True + or self.cleanup_record['forced_termination'] or time.monotonic() >= grace_deadline): + self.terminate(reason='descendant_grace_expired') + break + time.sleep(min(0.01, max(0, grace_deadline - time.monotonic()))) + self.cleanup_record['descendant_grace_seconds'] = time.monotonic() - grace_started + deadline = time.monotonic() + 5.0 + while True: + active = self._job_api.QueryInformationJobObject( + self.job, self._job_api.JobObjectBasicAccountingInformation)['ActiveProcesses'] + self.cleanup_record['job_active_processes'] = active + if not active: + job_empty = True break if time.monotonic() >= deadline: raise TimeoutError('owned job descendants did not exit within 5 seconds') @@ -417,6 +473,10 @@ class _Win32Process: self._descendant_handles = retained for name in ('job', 'thread', 'handle'): handle = getattr(self, name) + if name in ('job', 'handle') and not job_empty: + # Closing a KILL_ON_JOB_CLOSE job would silently force remaining + # children and discard the capability needed for cleanup retry. + continue if name == 'handle' and self.returncode is None: # Keep the only termination/wait capability until cleanup can retry. continue diff --git a/src/dx_productivity/trial_task_center/project_session.py b/src/dx_productivity/trial_task_center/project_session.py index a346d59..879d7d6 100644 --- a/src/dx_productivity/trial_task_center/project_session.py +++ b/src/dx_productivity/trial_task_center/project_session.py @@ -111,6 +111,13 @@ class ProjectSessionWorker: return if not self._done.is_set(): _write_control(self.root / 'project-session-next.tsv', f'{self.token}\tclose') + from .stop_control import StopControl + interactive = isinstance(getattr(self.runtime, 'stop_event', None), StopControl) + if interactive: + # The monitor owns the safe/unsafe distinction. Do not + # turn a GUI close or ordinary pause into a timed kill. + while not self._done.wait(.1): + pass close_grace = 60 if self.private_desktop else min(30, max(10, self._timeout)) if not self._done.wait(close_grace): self._abort.set() diff --git a/src/dx_productivity/trial_task_center/service.py b/src/dx_productivity/trial_task_center/service.py index ce87502..7948ac7 100644 --- a/src/dx_productivity/trial_task_center/service.py +++ b/src/dx_productivity/trial_task_center/service.py @@ -925,6 +925,9 @@ class TaskCenterService: message = batch[2] if batch else '正在准备执行' value = read_feedback(task_id, position, total, message, child._quick_run_preparer.data_root, started_after) + if getattr(self, '_supervisor_waiting', False): + from dataclasses import replace + value = replace(value, stage='awaiting_reconciliation', waiting_for_safe_completion=True) if read_control is not None: read_control.checkpoint() return value @@ -951,12 +954,12 @@ class TaskCenterService: return delegated.has_interrupted_quick_run(task_id) return self._quick_run_preparer.has_interrupted(task_id) - def recover_quick_run(self, task_id: str, *, confirmed: bool = False, runtime=None) -> QuickRunStepResult: + def recover_quick_run(self, task_id: str, *, confirmed: bool = False, runtime=None, stop_event=None) -> QuickRunStepResult: if runtime is None and self._supervised: - return self._supervise('recover_quick_run', {'task_id': task_id, 'confirmed': confirmed}) + return self._supervise('recover_quick_run', {'task_id': task_id, 'confirmed': confirmed}, stop_event=stop_event) if runtime is None: return self._owned_cad_operation('recover_quick_run', lambda owned: - self.recover_quick_run(task_id, confirmed=confirmed, runtime=owned), task_id=task_id) + self.recover_quick_run(task_id, confirmed=confirmed, runtime=owned), stop_event, task_id=task_id) delegated = self._project_task_service(task_id) if delegated is not self: return delegated.recover_quick_run(task_id, confirmed=confirmed, runtime=runtime) diff --git a/src/dx_productivity/trial_task_center/stop_control.py b/src/dx_productivity/trial_task_center/stop_control.py new file mode 100644 index 0000000..7f016b9 --- /dev/null +++ b/src/dx_productivity/trial_task_center/stop_control.py @@ -0,0 +1,78 @@ +"""Separate ordinary stop intent from an explicitly confirmed forced interruption.""" +from threading import Event +import json +import os + +from .path_access import internal_path + + +class StopControl(Event): + def __init__(self): + super().__init__() + self.force_event = Event() + self.control_error = '' + self.waiting_for_resolution = False + self.resolution_error = '' + + def request_force(self): + self.set() + self.control_error = '' + self.force_event.set() + + +def protect_interruption(root, reason): + """Create or re-sync a marker without overwriting prior protection evidence.""" + root = internal_path(root) + root.mkdir(parents=True, exist_ok=True) + marker = root / 'uncertain.flag' + try: + stream = marker.open('x', encoding='utf-8') + except FileExistsError: + # A previous create/write/fsync may have failed. Existence alone does + # not establish durable evidence; preserve its bytes but sync again. + with marker.open('r+b') as existing: + os.fsync(existing.fileno()) + else: + with stream: + stream.write(str(reason)) + stream.flush() + os.fsync(stream.fileno()) + + +def wait_for_resolution(control, root, error): + """Interactive owners retain a possibly changed project until explicit force. + + The GUI-side supervisor can also terminate its own Job after durable force + evidence. Ordinary cancellation never releases this wait. + """ + if not isinstance(control, StopControl): + return + from .worker_monitor import WorkerMonitorError, _publish_progress + root = internal_path(root) + control.set() + control.waiting_for_resolution = True + control.resolution_error = str(error) + def protect(): + protect_interruption(root, 'operation_interrupted: ' + str(error)) + try: + protect() + except OSError as failure: + control.control_error = '保护记录写入失败,保留应用等待处理:' + str(failure) + try: + record = json.loads((root / 'worker-progress.json').read_text(encoding='utf-8')) + if not isinstance(record, dict): + record = {} + except (OSError, ValueError): + record = {} + record.update(stage='awaiting_reconciliation', waiting_for_safe_completion=True, + resolution_error=str(error), stop_requested=True, safe_stop=False) + _publish_progress(root / 'worker-progress.json', record) + while True: + if control.force_event.wait(.1): + try: + protect() + except OSError as failure: + control.force_event.clear() + control.control_error = '强制结束未执行,保护记录无法保存:' + str(failure) + continue + raise WorkerMonitorError('user forced stop of uncertain operation; reconciliation required') diff --git a/src/dx_productivity/trial_task_center/ui/execution_feedback.py b/src/dx_productivity/trial_task_center/ui/execution_feedback.py index 1516721..ede87b0 100644 --- a/src/dx_productivity/trial_task_center/ui/execution_feedback.py +++ b/src/dx_productivity/trial_task_center/ui/execution_feedback.py @@ -15,6 +15,7 @@ STAGES = { 'restore_attribute_fields': '恢复属性字段', 'restore_attributes': '恢复属性', 'verify_restoration': '核验恢复结果', 'verify_attribute_fields': '核验属性字段', 'save_and_close': '保存并关闭工程', 'verify_saved_project': '回读已保存工程', + 'awaiting_reconciliation': '当前结果待核验,已保留工程现场', } @@ -22,4 +23,6 @@ def format_feedback(value, task_name, elapsed, stopping): stage = STAGES.get(value.stage, '') details = f'{stage}({value.stage_seconds:.1f} 秒)' if stage else value.message prefix = '等待当前步骤保存核验后暂停 · ' if stopping else '' + if value.waiting_for_safe_completion: + prefix = '当前保存或核验尚未确认,未自动结束应用;可继续等待、导出诊断,或确认强制结束后核验 · ' return f'{prefix}任务 {value.position}/{value.total}:{task_name} · {details} · 总计 {elapsed} 秒' diff --git a/src/dx_productivity/trial_task_center/ui/main_window.py b/src/dx_productivity/trial_task_center/ui/main_window.py index bee7608..2b4ff64 100644 --- a/src/dx_productivity/trial_task_center/ui/main_window.py +++ b/src/dx_productivity/trial_task_center/ui/main_window.py @@ -1,6 +1,7 @@ """Single-project task presentation; persistence and CAD remain in the service.""" from __future__ import annotations from threading import Event +from ..stop_control import StopControl from shiboken6 import isValid import time from PySide6.QtCore import QElapsedTimer, QSettings, QSignalBlocker, QThreadPool, QTimer, Qt @@ -39,6 +40,7 @@ class TrialTaskCenterWindow(QMainWindow): self._tasks = () self._known_task_groups = {} self._workers = set() + self._diagnostic_worker = None self._closed = False self._batch_stop = None self._active_quick_run_task_id = None @@ -88,6 +90,7 @@ class TrialTaskCenterWindow(QMainWindow): self._log_presenter = TaskLogPresenter(service, self.run_panel, self) self.run_panel.quick_run_button.clicked.connect(self._prepare_quick_run) self.run_panel.stop_queue_button.clicked.connect(self._pause_queue) + self.run_panel.force_stop_button.clicked.connect(self._force_batch_stop) self.run_panel.retry_button.clicked.connect(self._retry_failed) self.run_panel.recover_button.clicked.connect(self._recover_quick_run) self.run_panel.cleanup_resources_button.clicked.connect(self._cleanup_batch_resources) @@ -337,6 +340,10 @@ class TrialTaskCenterWindow(QMainWindow): self.run_panel.history_button.setEnabled(bool(context and not busy)) self.run_panel.stop_queue_button.setVisible(self._batch_stop is not None) self.run_panel.stop_queue_button.setEnabled(bool(self._batch_stop and not self._batch_stop.is_set())) + self.run_panel.force_stop_button.setVisible(self._batch_stop is not None) + self.run_panel.force_stop_button.setEnabled(bool(self._batch_stop and + not self._batch_stop.force_event.is_set())) + self.run_panel.diagnostics_button.setEnabled(self._diagnostic_worker is None) self.run_panel.copy_value_check.setEnabled(state.editable) rule_editable = state.editable and not group_unsaved self.mapping_table.set_rules_editable(rule_editable) @@ -571,6 +578,8 @@ class TrialTaskCenterWindow(QMainWindow): self._finish_run() self._metadata_changed(None) failures = sum(e.state in {'failed', 'blocked'} for e in result.entries) + if failures: + self._close_pending = False self._show_task_log(force=True) self._set_operator_message(f'本批任务已暂停。失败/阻断 {failures} 项。' if result.paused else f'本批任务已结束。失败 {failures} 项。', 'warning' if result.paused or failures else 'success') @@ -578,7 +587,7 @@ class TrialTaskCenterWindow(QMainWindow): self.task_list.queue_button) def _begin_run(self, task_id): - self._batch_stop = Event() + self._batch_stop = StopControl() self._active_quick_run_task_id = task_id self._run_started_at = time.time() self._quick_run_elapsed.start() @@ -601,6 +610,24 @@ class TrialTaskCenterWindow(QMainWindow): self._set_operator_message('已请求暂停;当前项安全收尾后停止。', 'warning') self._update_gates() + def _force_batch_stop(self): + control = self._batch_stop + if control is None or control.force_event.is_set(): + return + if QMessageBox.question(self, '强制结束当前操作', + '当前操作可能正在修改或保存工程。强制结束可能留下未保存或部分保存的结果。\n\n' + '已完成的执行证据会保留;中断后必须先核验恢复,不能直接重复替换。' + '建议先导出诊断包。确定强制结束本程序当前操作吗?', + QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No, + QMessageBox.StandardButton.No) != QMessageBox.StandardButton.Yes: + return + if self._batch_stop is not control: + return # Completion/new run while the confirmation dialog was open. + self._close_pending = False + control.request_force() + self._set_operator_message('已请求强制结束;正在记录中断证据,结果将进入待核验。', 'warning') + self._update_gates() + def _show_batch_result(self, result): owner = self._active_quick_run_task_id or self._task_id name = next((t.name for t in self._tasks if t.task_id == owner), owner) @@ -609,6 +636,8 @@ class TrialTaskCenterWindow(QMainWindow): if owner == self._task_id: self._show_task_log(force=True) p = result.progress + if p.failed_items: + self._close_pending = False self._set_operator_message(f"任务「{name}」{'已暂停' if result.paused else '已结束'}:已保存 {p.saved_items},失败 {p.failed_items},未命中 {p.no_match_items}。", 'warning' if result.paused or p.failed_items else 'success') @@ -637,10 +666,31 @@ class TrialTaskCenterWindow(QMainWindow): from pathlib import Path name = 'DxPartReplace-诊断-' + time.strftime('%Y%m%d-%H%M%S') + '.zip' value, _ = QFileDialog.getSaveFileName(self, '导出脱敏诊断包', name, 'ZIP 文件 (*.zip)') - if value: - self._run_async(lambda: self.service.export_diagnostics(Path(value)), - lambda path: self._set_operator_message(f'脱敏诊断包已导出:{path}', 'success'), - self.run_panel.diagnostics_button) + if not value or self._diagnostic_worker is not None: + return + # A read-only support export remains available while CAD is waiting. + worker = ServiceWorker(lambda: self.service.export_diagnostics(Path(value))) + self._diagnostic_worker = worker + self._workers.add(worker) + def exported(path): + if not self._closed: + self._set_operator_message(f'脱敏诊断包已导出:{path}', 'success') + def failed(error): + self._close_pending = False + if not self._closed: + self._show_error(error) + def finished(): + self._workers.discard(worker) + self._diagnostic_worker = None + if not self._closed: + self._update_gates() + if self._close_pending and not self._workers: + QTimer.singleShot(0, self.close) + worker.signals.succeeded.connect(exported) + worker.signals.failed.connect(failed) + worker.signals.finished.connect(finished) + self._update_gates() + QThreadPool.globalInstance().start(worker) def _retry_failed(self): if self.run_panel.retry_button.isEnabled(): @@ -700,7 +750,8 @@ class TrialTaskCenterWindow(QMainWindow): self._finish_run() self._metadata_changed(None) self._set_operator_message(result.message) - self._run_async(lambda: self.service.recover_quick_run(task_id, confirmed=True), done, + stop = self._batch_stop + self._run_async(lambda: self.service.recover_quick_run(task_id, confirmed=True, stop_event=stop), done, self.run_panel.recover_button, expected_task_id=task_id) def _confirm_quick_run(self, scope, selected_projects, enabled_rules): @@ -725,6 +776,9 @@ class TrialTaskCenterWindow(QMainWindow): self.run_panel.execution_status.setText(format_feedback( value, name, self._quick_run_elapsed.elapsed() // 1000, bool(self._batch_stop and self._batch_stop.is_set()))) + if self._batch_stop and self._batch_stop.control_error: + self._set_operator_message(self._batch_stop.control_error, 'warning') + self._update_gates() self._feedback_query.request(lambda control: self.service.execution_feedback(task_id, started_after=started, read_control=control), received, lambda _: None, cancellable=True) if self.run_panel.toggle_log_button.isChecked(): @@ -799,6 +853,7 @@ class TrialTaskCenterWindow(QMainWindow): callback(result) except Exception as error: operation_error = error + self._close_pending = False self.operation_state = self.operation_state.finish(error) self._show_error(error) def failed(error): diff --git a/src/dx_productivity/trial_task_center/ui/messages.py b/src/dx_productivity/trial_task_center/ui/messages.py index 2d11df5..e343b5b 100644 --- a/src/dx_productivity/trial_task_center/ui/messages.py +++ b/src/dx_productivity/trial_task_center/ui/messages.py @@ -4,6 +4,8 @@ import json import re ERROR_TEXT = { + 'user_forced_stop': '操作已强制中断,结果待核验。', + 'application_shutdown_failed': '业务结果已保留,但应用收尾异常。', "batch_resources_pending": "后台资源尚未确认释放,请使用“重新清理资源”后再开始下一批。", "supervisor_busy": "另一批操作正在执行,请先安全收尾。", "supervisor_interrupted": "批次监督连接异常,保护记录已保留,没有自动重放替换。", diff --git a/src/dx_productivity/trial_task_center/ui/run_panel.py b/src/dx_productivity/trial_task_center/ui/run_panel.py index 150f024..d5149c4 100644 --- a/src/dx_productivity/trial_task_center/ui/run_panel.py +++ b/src/dx_productivity/trial_task_center/ui/run_panel.py @@ -23,6 +23,8 @@ class RunPanel(QWidget): self.busy_indicator.hide() self.quick_run_button = QPushButton('执行当前任务', objectName='quickRunButton') self.stop_queue_button = QPushButton('完成当前项后暂停', objectName='stopQueueButton') + self.force_stop_button = QPushButton('强制结束…', objectName='forceStopButton') + self.force_stop_button.setToolTip('异常情况下使用;会中断当前操作,结果必须重新核验。') self.retry_button = QPushButton('重试失败项', objectName='retryTaskItems') self.recover_button = QPushButton('核验并恢复', objectName='recoverQuickRunButton') self.cleanup_resources_button = QPushButton('重新清理资源', objectName='cleanupBatchResources') @@ -42,7 +44,7 @@ class RunPanel(QWidget): self.toggle_log_button.toggled.connect(self._toggle_log) buttons = QHBoxLayout() for button in (self.quick_run_button, self.retry_button, self.recover_button, self.cleanup_resources_button, self.diagnostics_button, - self.new_round_button, self.history_button, self.stop_queue_button): + self.new_round_button, self.history_button, self.stop_queue_button, self.force_stop_button): buttons.addWidget(button) buttons.addStretch() buttons.addWidget(self.toggle_log_button) diff --git a/src/dx_productivity/trial_task_center/worker_monitor.py b/src/dx_productivity/trial_task_center/worker_monitor.py index a5d27fd..c6ac3fd 100644 --- a/src/dx_productivity/trial_task_center/worker_monitor.py +++ b/src/dx_productivity/trial_task_center/worker_monitor.py @@ -35,6 +35,7 @@ class ProgressWatch: self.done = 0 self.total = 0 self.safe_stop = True + self.waiting_for_safe_completion = False self.started = started self.stage_started = started self.stage_seconds: dict[str, float] = {} @@ -67,6 +68,7 @@ class ProgressWatch: self.native_sequence = 0 self.last_progress = now self.stage, self.done, self.total, self.safe_stop = fields[4], done, total, bool(safe) + self.waiting_for_safe_completion = False return True def accept_native(self, line: bytes, now: float) -> bool: @@ -102,6 +104,11 @@ class ProgressWatch: self.stop_requested = True return 'request_stop' if self.stop_requested and idle >= self.idle_timeout * 2: + if not self.safe_stop: + if not self.waiting_for_safe_completion: + self.waiting_for_safe_completion = True + return 'await_safe_completion' + return None return 'unresponsive' return None @@ -109,7 +116,7 @@ class ProgressWatch: now = time.monotonic() if now is None else now stages = dict(self.stage_seconds) stages[self.stage] = stages.get(self.stage, 0.0) + max(0.0, now - self.stage_started) - return {name: getattr(self, name) for name in ('token', 'sequence', 'stage', 'done', 'total', 'safe_stop', 'stop_requested')} | { + return {name: getattr(self, name) for name in ('token', 'sequence', 'stage', 'done', 'total', 'safe_stop', 'stop_requested', 'waiting_for_safe_completion')} | { 'job_id': self.current_job_id, 'native_progress_frames': self.native_progress_frames, 'elapsed_seconds': max(0.0, now - self.started), 'stage_seconds': stages} @@ -124,9 +131,8 @@ def _atomic_text(path: Path, text: str) -> None: def _publish_progress(path: Path, snapshot: dict) -> None: try: _atomic_text(path, json.dumps(snapshot, ensure_ascii=False)) - except OSError as error: - if not isinstance(error, PermissionError) and getattr(error, 'winerror', None) not in {5, 32, 33}: - raise + except OSError: + pass # Display telemetry failure must not terminate a possible CAD write. def _decode(data: bytes) -> str: @@ -190,7 +196,7 @@ class _OwnedWorkers: def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abort_event=None, - process_factory=None, cooperative_abort=False) -> subprocess.CompletedProcess[str]: + process_factory=None, cooperative_abort=False, force_event=None, force_control=None) -> subprocess.CompletedProcess[str]: root = Path(command[3]).resolve() job_id = command[-1] token = uuid4().hex @@ -200,8 +206,11 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor native_progress_path = root / f'monitor-{token}.native-progress' root.mkdir(parents=True, exist_ok=True) watch = ProgressWatch(token, job_id, idle_timeout, time.monotonic()) + if force_control is not None: + force_event = force_control.force_event owned = _OwnedWorkers(token, {command[0], command[4]}) process = None + durable_force_accepted = False threads = [] messages = queue.Queue(maxsize=1024) stop_readers = threading.Event() @@ -244,15 +253,32 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor _publish_progress(progress_path, watch.snapshot()) _atomic_text(ready, token) while process.poll() is None: + if force_event is not None and force_event.is_set(): + from .stop_control import protect_interruption + try: + protect_interruption(root, 'user_forced_stop') + except OSError as error: + force_event.clear() + if force_control is not None: + force_control.control_error = '强制结束未执行,保护记录无法保存:' + str(error) + watch.waiting_for_safe_completion = True + _publish_progress(progress_path, watch.snapshot()) + else: + durable_force_accepted = True + raise WorkerMonitorError('user forced stop; execution requires reconciliation') if abort_event is not None and abort_event.is_set(): - if not cooperative_abort: + if not cooperative_abort and watch.safe_stop: raise WorkerMonitorError('project session supervisor requested worker cleanup') if not watch.stop_requested: # Read-only callers request the VBS safe boundary first. # The existing idle watchdog still bounds a stalled COM call. watch.stop_requested = True watch.last_progress = time.monotonic() - _atomic_text(stop, token) + try: + _atomic_text(stop, token) + except OSError: + if watch.safe_stop: + raise _publish_progress(progress_path, watch.snapshot()) try: receive(*messages.get(timeout=min(0.1, idle_timeout / 10))) @@ -273,7 +299,17 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor break action = watch.action(time.monotonic()) if action == 'request_stop': - _atomic_text(stop, token) + try: + _atomic_text(stop, token) + except OSError: + if watch.safe_stop: + raise + _publish_progress(progress_path, watch.snapshot()) + elif action == 'await_safe_completion': + if cooperative_abort: + # This entry point is explicitly read-only. A blocked COM + # read has no write to finish and retains its bounded cancel. + raise subprocess.TimeoutExpired(command, idle_timeout * 2) _publish_progress(progress_path, watch.snapshot()) elif action == 'unresponsive': raise subprocess.TimeoutExpired(command, idle_timeout * 2) @@ -287,6 +323,15 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor return subprocess.CompletedProcess(command, process.returncode, _decode(output[0]), _decode(output[1])) except Exception as error: if process is not None: + if (not watch.safe_stop and not cooperative_abort and force_control is not None + and not durable_force_accepted): + from .stop_control import wait_for_resolution + try: + wait_for_resolution(force_control, root, error) + except WorkerMonitorError: + if not force_event.is_set(): + raise + durable_force_accepted = True failures = owned.stop_workers(process) process.wait(timeout=10) for thread in threads: diff --git a/tests/integration/test_quick_session_startup.py b/tests/integration/test_quick_session_startup.py index 3e81ffb..37dfc7b 100644 --- a/tests/integration/test_quick_session_startup.py +++ b/tests/integration/test_quick_session_startup.py @@ -175,13 +175,13 @@ Set QP_App=FakeApp FakeApp.ProjectPath=Target QP_CurrentJobId="batch-job": QP_ExpectedJobId="batch-job" QPWriteUtf8 QP_Fso.BuildPath(QP_Root,"command.tsv"),"task_id" & vbTab & "job_id" & vbTab & "project_path" & vbTab & "old_part" & vbTab & "new_part" & vbTab & "value_mode" & vbCrLf & "task" & vbTab & "batch-job" & vbTab & Target & vbTab & "OLD" & vbTab & "NEW" & vbTab & "preserve" & vbCrLf -QP_BatchMode=True: QP_OwnedProject=True +QP_BatchMode=True: QP_OwnedProject=True: QP_OwnedProjectPath=Target QPWriteStatus QP_Root,QP_CurrentJobId,"no_match","","not found" Require Not QP_Fso.FileExists(QP_Fso.BuildPath(QP_Root,"status.tsv")),"published success before close" Require QPReadState(QP_Fso.BuildPath(QP_Root,"status.tsv"),QP_CurrentJobId)="no_match","pending no-match not visible to worker" QPFinishBatchItem Require FakeApp.CloseCount=1,"project not closed exactly once" -Require FakeApp.SaveCount=1,"sheet not closed with saving allowed" +Require FakeApp.SaveCount=0,"no-match must close without saving" Require InStr(QPReadUtf8(QP_Fso.BuildPath(QP_Root,"status.tsv")),"batch_closed")>0,"no completion proof" ''') @@ -368,8 +368,8 @@ Require FakeApp.ProjectPath = "", "discovery left the project open" Require FakeApp.Documents.OpenCount = 0, "discovery opened a page" Dim Manifest Manifest = QPReadUtf8(QP_Fso.BuildPath(QP_Root, "sheets.tsv")) -Require InStr(Manifest, Target & vbTab & "Main" & vbTab & "1") > 0, "sheet one missing" -Require InStr(Manifest, Target & vbTab & "Main" & vbTab & "2") > 0, "sheet two missing" +Require InStr(Manifest, LCase(Target) & vbTab & "Main" & vbTab & "1") > 0, "sheet one missing" +Require InStr(Manifest, LCase(Target) & vbTab & "Main" & vbTab & "2") > 0, "sheet two missing" ''') diff --git a/tests/integration/test_quick_value_modes.py b/tests/integration/test_quick_value_modes.py index 3172da9..27648cc 100644 --- a/tests/integration/test_quick_value_modes.py +++ b/tests/integration/test_quick_value_modes.py @@ -253,6 +253,31 @@ End Sub''', routines) 'Sub QPVerifyUntouched(ByRef Records)\n' + injection) +def test_save_then_project_close_refusal_retains_save_boundary(tmp_path): + run_vbs(tmp_path, 'part_number', ''' +Dim FullChecks +FullChecks=0 +Command.Add "project_path",QPProjectPath() +QP_BatchMode=True: QP_SessionMode=True +QP_SessionToken=String(32,"b") +QP_OwnedProject=True: QP_OwnedProjectPath=QPProjectPath() +QPRestoreAfterOperatorReplace Command,SnapshotPath +QP_App.RefuseClose=True +On Error Resume Next +QPFinishBatchItem +Failure=Err.Number +Err.Clear +On Error GoTo 0 +Require Failure<>0,"refused close must stop the operation" +Require QP_BatchSaveCommitted,"returned save was lost when project close failed" +QP_App.RefuseClose=False +QPDiscardFailedBatchProject CloseError +Require Len(CloseError)>0,"uncertain save must not discard the open project" +Require Len(QPProjectPath())>0,"failed saved project was silently discarded" +Require Not QP_Fso.FileExists(QP_Fso.BuildPath(QP_Root,"project-session-result.tsv")),"failure published success" +''', routines_transform=_batch_readback_fixture) + + @pytest.mark.parametrize('fault', ['', 'before_save', 'after_reopen']) def test_owned_batch_checks_full_state_at_save_and_disk_boundaries(tmp_path, fault): run_vbs(tmp_path, 'part_number', ''' diff --git a/tests/v1_1/test_isolated_cad.py b/tests/v1_1/test_isolated_cad.py index 69c9ac1..99f9fc4 100644 --- a/tests/v1_1/test_isolated_cad.py +++ b/tests/v1_1/test_isolated_cad.py @@ -1,10 +1,12 @@ import io import json import os +import subprocess import weakref from pathlib import Path from threading import Event from threading import Timer +from threading import Thread from types import SimpleNamespace from traceback import extract_tb @@ -194,6 +196,121 @@ def test_clean_release_records_helper_and_cad_exit_codes(monkeypatch, tmp_path): assert record['cleanup_errors'] == [] +@pytest.mark.parametrize('persistent', [False, True]) +def test_closing_journal_failure_still_quits_and_reports_diagnostic_error(monkeypatch, tmp_path, persistent): + events, cad = fake_desktop(monkeypatch, tmp_path) + session = runtime.IsolatedCadSession(tmp_path) + session.__enter__() + record = session._record + def failing_record(**values): + if persistent or values.get('state') == 'closing': + raise PermissionError('closing journal locked') + record(**values) + session._record = failing_record + with pytest.raises(WorkerMonitorError, match='closing journal locked'): + session.__exit__(None, None, None) + assert 'quit' in events and cad.returncode == 0 + assert session.record['orderly_shutdown'] is True + assert session.record['clean'] is False + assert session.record['diagnostic_errors'][0]['stage'] == 'closing_record' + assert (tmp_path / 'uncertain.flag').exists() + + +@pytest.mark.parametrize('helper_failure', ['refused', 'timeout']) +def test_interactive_quit_failure_retains_cad_and_helper_until_explicit_force(monkeypatch, tmp_path, helper_failure): + from dx_productivity.trial_task_center.stop_control import StopControl + from dx_productivity.trial_task_center import worker_monitor + events, cad = fake_desktop(monkeypatch, tmp_path, quit_code=1, cad_exit_code=None) + control = StopControl() + session = runtime.IsolatedCadSession(tmp_path, stop_control=control) + session.__enter__() + start = session.desktop.start_process + def start_quit(command, **options): + helper = start(command, **options) + if command[0] == 'quit': + if helper_failure == 'timeout': + helper.wait = lambda timeout: (_ for _ in ()).throw(subprocess.TimeoutExpired('quit', timeout)) + helper.poll = lambda: None + helper.close = lambda: events.append('close-quit-helper') + return helper + session.desktop.start_process = start_quit + waiting, finished = Event(), Event() + publish = worker_monitor._publish_progress + def observe_wait(path, snapshot): + publish(path, snapshot) + if snapshot.get('stage') == 'awaiting_reconciliation': + waiting.set() + monkeypatch.setattr(worker_monitor, '_publish_progress', observe_wait) + failures = [] + def close_session(): + try: + session.__exit__(None, None, None) + except Exception as failure: + failures.append(failure) + finally: + finished.set() + thread = Thread(target=close_session) + thread.start() + try: + assert waiting.wait(1), 'failed Quit must enter protected waiting before resource release' + control.set() + assert not finished.wait(.05) + assert cad.poll() is None + assert 'close-quit-helper' not in events + assert not any(isinstance(event, tuple) and event[0] == 'release' for event in events) + assert (tmp_path / 'uncertain.flag').exists() + control.request_force() + assert finished.wait(2) + assert 'close-quit-helper' in events and ('release', True) in events + assert cad.poll() == 1 + assert isinstance(failures[0], WorkerMonitorError) + assert 'forced' in str(failures[0]) + assert session.record['orderly_shutdown'] is False and session.record['clean'] is False + finally: + control.request_force() + thread.join(3) + + +@pytest.mark.parametrize('failure', ['cad_exited', 'diagnostic']) +def test_interactive_exit_does_not_hold_completed_cad_for_resolution(monkeypatch, tmp_path, failure): + from dx_productivity.trial_task_center.stop_control import StopControl + from dx_productivity.trial_task_center import stop_control + events, _cad = fake_desktop(monkeypatch, tmp_path, quit_code=1 if failure == 'cad_exited' else 0) + session = runtime.IsolatedCadSession(tmp_path, stop_control=StopControl()) + session.__enter__() + monkeypatch.setattr(stop_control, 'wait_for_resolution', lambda *_args: pytest.fail('exited CAD must not wait')) + if failure == 'diagnostic': + record = session._record + def failed_journal(**values): + if values.get('state') == 'closing': + raise PermissionError('closing diagnostic unavailable') + record(**values) + session._record = failed_journal + with pytest.raises(WorkerMonitorError): + session.__exit__(None, None, None) + assert ('release', True) in events + + +def test_forced_descendant_cleanup_is_not_reported_as_clean_orderly_shutdown(monkeypatch, tmp_path): + fake_desktop(monkeypatch, tmp_path) + session = runtime.IsolatedCadSession(tmp_path) + session.__enter__() + release = session.desktop.close + def forced_release(): + release() + session.desktop.cleanup_record = {'processes': [{'pid': 123, 'forced_termination': True, + 'forced_root_termination': False, 'termination_reason': 'descendant_grace_expired', + 'forced_descendant_pids': [456]}]} + session.desktop.close = forced_release + with pytest.raises(WorkerMonitorError, match='forced'): + session.__exit__(None, None, None) + record = json.loads((tmp_path / 'private-desktop.json').read_text()) + assert record['clean'] is False and record['forced_termination'] is True + assert record['orderly_shutdown'] is True + assert record['forced_descendant_pids'] == [456] + assert record['termination_reason'] == 'descendant_grace_expired' + + def test_exit_code_diagnostics_do_not_interrupt_the_normal_cad_wait(monkeypatch, tmp_path): _, cad = fake_desktop(monkeypatch, tmp_path) waited = [] diff --git a/tests/v1_1/test_isolated_desktop.py b/tests/v1_1/test_isolated_desktop.py index 19b10ac..eab707c 100644 --- a/tests/v1_1/test_isolated_desktop.py +++ b/tests/v1_1/test_isolated_desktop.py @@ -367,6 +367,99 @@ def test_descendant_handles_are_owned_before_terminating_the_job(): assert ('close', 'child') in events +@pytest.mark.parametrize('child_exit_after', [0.5, None]) +def test_completed_root_gives_owned_descendants_bounded_natural_exit_grace(monkeypatch, child_exit_after): + from dx_productivity.trial_task_center import isolated_desktop as module + api, events, state, *_ = native_api() + clock = [0.0] + monkeypatch.setattr(module.time, 'monotonic', lambda: clock[0]) + monkeypatch.setattr(module.time, 'sleep', lambda seconds: clock.__setitem__(0, clock[0] + seconds)) + state['running'] = False + child = Handle('child', events) + child_running = lambda: not any(event[0] == 'terminate' for event in events) and ( + child_exit_after is None or clock[0] < child_exit_after) + api.job.JobObjectBasicProcessIdList = 3 + api.job.QueryInformationJobObject = lambda job, kind: ( + {'BasicLimitInformation': {'LimitFlags': 0}} if kind == 9 else + ([43] if child_running() else []) if kind == 3 else {'ActiveProcesses': int(child_running())}) + api.open_process = lambda *_args: child + root_wait = api.event.WaitForSingleObject + api.event.WaitForSingleObject = lambda handle, timeout: ( + 258 if child_running() else 0) if handle is child else root_wait(handle, timeout) + process = api.create_process(('python.exe',), 'winsta0\\private', None, None) + process.close() + assert child.closed + if child_exit_after is not None: + assert child_exit_after <= clock[0] < 2 + assert ('terminate',) not in events + assert process.cleanup_record['forced_termination'] is False + assert process.cleanup_record['forced_descendant_pids'] == [] + else: + assert 2 <= clock[0] < 2.1 + assert ('terminate',) in events + assert process.cleanup_record['forced_termination'] is True + assert process.cleanup_record['forced_root_termination'] is False + assert process.cleanup_record['forced_descendant_pids'] == [43] + assert process.cleanup_record['termination_reason'] == 'descendant_grace_expired' + + +def test_completed_root_without_descendants_has_no_grace_delay(monkeypatch): + from dx_productivity.trial_task_center import isolated_desktop as module + api, events, state, *_ = native_api() + state['running'] = False + monkeypatch.setattr(module.time, 'sleep', lambda seconds: pytest.fail('empty job must not wait')) + process = api.create_process(('python.exe',), 'winsta0\\private', None, None) + process.close() + assert ('terminate',) not in events + assert process.cleanup_record['forced_termination'] is False + + +def test_active_root_cleanup_is_forced_without_descendant_grace(monkeypatch): + from dx_productivity.trial_task_center import isolated_desktop as module + api, events, *_ = native_api() + monkeypatch.setattr(module.time, 'sleep', lambda seconds: pytest.fail('active root must be bounded immediately')) + process = api.create_process(('python.exe',), 'winsta0\\private', None, None) + process.close() + assert process.cleanup_record['forced_termination'] is True + assert process.cleanup_record['forced_root_termination'] is True + assert process.cleanup_record['termination_reason'] == 'root_process_active' + + +def test_native_poll_error_still_terminates_owned_root_and_records_force(): + api, events, state, *_ = native_api() + wait = api.event.WaitForSingleObject + def failing_poll(handle, timeout): + if timeout == 0 and state['running']: + raise OSError('native poll failed') + return wait(handle, timeout) + api.event.WaitForSingleObject = failing_poll + process = api.create_process(('python.exe',), 'winsta0\\private', None, None) + with pytest.raises(TaskCenterError, match='native poll failed'): + process.close() + assert state['running'] is False + assert process.cleanup_record['forced_root_termination'] is True + + +def test_failed_job_cleanup_retains_ownership_until_retry(monkeypatch): + from dx_productivity.trial_task_center import isolated_desktop as module + api, events, state, root, job = native_api() + clock = [0.0] + monkeypatch.setattr(module.time, 'monotonic', lambda: clock[0]) + monkeypatch.setattr(module.time, 'sleep', lambda seconds: clock.__setitem__(0, clock[0] + seconds)) + api.job.TerminateJobObject = lambda *_args: (_ for _ in ()).throw(OSError('job termination failed')) + query = api.job.QueryInformationJobObject + child_running = [True] + api.job.QueryInformationJobObject = lambda handle, kind: ( + {'ActiveProcesses': int(state['running']) + int(child_running[0])} if kind == 1 else query(handle, kind)) + process = api.create_process(('python.exe',), 'winsta0\\private', None, None) + with pytest.raises(TaskCenterError, match='descendants did not exit'): + process.close() + assert not job.closed and process.job is job + child_running[0] = False + process.close() + assert job.closed + + @pytest.mark.parametrize('error_code', [87, 5]) def test_native_descendant_open_error_only_ignores_already_exited_process(error_code): # pywintypes.error derives directly from Exception, rather than OSError. @@ -391,6 +484,30 @@ def test_native_descendant_open_error_only_ignores_already_exited_process(error_ assert caught.value is error +def test_forced_job_preserves_partial_current_descendant_snapshot_after_open_failure(): + api, events, state, *_ = native_api() + api.job.JobObjectBasicProcessIdList = 3 + query = api.job.QueryInformationJobObject + snapshot = [45] + api.job.QueryInformationJobObject = lambda job, kind: list(snapshot) if kind == 3 else query(job, kind) + def open_child(_access, _inherit, pid): + if pid == 44: + raise OSError(5, 'child access denied') + return Handle('child-' + str(pid), events) + api.open_process = open_child + process = api.create_process(('python.exe',), 'winsta0\\private', None, None) + # PID 45 was observed earlier but is no longer part of the active Job. + process._remember_descendants() + snapshot[:] = [42, 43, 44] + with pytest.raises(TaskCenterError, match='child access denied'): + process.close() + assert state['running'] is False + assert process.cleanup_record['forced_termination'] is True + assert process.cleanup_record['forced_descendant_pids'] == [43] + assert process.cleanup_record['descendant_pids'] == [43, 45] + assert process.cleanup_record['job_handle_closed'] is True + + def test_unconfirmed_descendant_exit_retains_handle_for_cleanup_retry(): api, events, state, *_ = native_api() process = api.create_process(('python.exe',), 'winsta0\\private', None, None) @@ -430,9 +547,37 @@ def test_native_private_job_kills_descendant_after_root_exit(parent_exits): if parent_exits: assert process.wait(timeout=5) == 0 assert win32event.WaitForSingleObject(child_handle, 0) == 0 + assert process.cleanup_record['forced_termination'] is True + assert process.cleanup_record['forced_root_termination'] is (not parent_exits) + assert child_pid in process.cleanup_record['forced_descendant_pids'] with pytest.raises(pywintypes.error): unexpected = win32service.OpenDesktop(name, 0, False, 1) unexpected.CloseDesktop() finally: if child_handle is not None: child_handle.Close() + + +@pytest.mark.skipif(os.name != 'nt', reason='native Windows descendant shutdown grace') +def test_native_private_job_allows_descendant_to_finish_after_root_exit(): + import win32api + import win32event + import win32process + + child_handle = None + try: + with PrivateDesktop('dx-test-' + uuid4().hex, enabled=True) as desktop: + process = desktop.start_process((sys.executable, '-c', + "import subprocess,sys; p=subprocess.Popen([sys.executable,'-c','import time;time.sleep(0.5)']); print(p.pid,flush=True)"), + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + child_pid = int(process.stdout.readline()) + child_handle = win32api.OpenProcess(0x100000 | 0x1000, False, child_pid) + assert process.wait(timeout=5) == 0 + assert win32event.WaitForSingleObject(child_handle, 0) == 0 + assert win32process.GetExitCodeProcess(child_handle) == 0 + assert process.cleanup_record['forced_termination'] is False + assert process.cleanup_record['forced_descendant_pids'] == [] + assert desktop.cleanup_record['desktop_confirmation']['released'] is True + finally: + if child_handle is not None: + child_handle.Close() diff --git a/tests/v1_1/test_safe_stop_ui.py b/tests/v1_1/test_safe_stop_ui.py new file mode 100644 index 0000000..07aada3 --- /dev/null +++ b/tests/v1_1/test_safe_stop_ui.py @@ -0,0 +1,75 @@ +from threading import Event + +from PySide6.QtGui import QCloseEvent +from PySide6.QtWidgets import QMessageBox + +from test_ui_remediation import window +from test_ui_batch_retry import open_project + + +def test_pause_and_window_close_never_request_force(window, qtbot, tmp_path): + open_project(window, qtbot, tmp_path) + window._begin_run(window._task_id) + worker = object() + window._workers.add(worker) + try: + event = QCloseEvent() + window.closeEvent(event) + assert not event.isAccepted() + assert window._batch_stop.is_set() + assert not window._batch_stop.force_event.is_set() + assert not window._closed + finally: + window._workers.discard(worker) + window._close_pending = False + window._finish_run() + + +def test_force_requires_confirmation_and_keeps_window_for_reconciliation(window, qtbot, tmp_path, monkeypatch): + open_project(window, qtbot, tmp_path) + window._begin_run(window._task_id) + control = window._batch_stop + try: + monkeypatch.setattr(QMessageBox, 'question', lambda *args: QMessageBox.StandardButton.No) + window._force_batch_stop() + assert not control.is_set() and not control.force_event.is_set() + monkeypatch.setattr(QMessageBox, 'question', lambda *args: QMessageBox.StandardButton.Yes) + window._close_pending = True + window._force_batch_stop() + assert control.is_set() and control.force_event.is_set() + assert not window._close_pending + finally: + window._finish_run() + + +def test_uncertain_wait_is_visible_in_execution_feedback(tmp_path): + import json + from dx_productivity.trial_task_center.execution_feedback import read_feedback + from dx_productivity.trial_task_center.ui.execution_feedback import format_feedback + root = tmp_path / 'quick-run' + root.mkdir() + (root / 'worker-progress.json').write_text(json.dumps({ + 'stage': 'save_and_close', 'waiting_for_safe_completion': True, 'stage_seconds': {}})) + feedback = read_feedback('task', 1, 1, '', tmp_path, 0) + text = format_feedback(feedback, 'board', 400, True) + assert '尚未确认' in text and '强制' in text + + +def test_diagnostic_export_during_wait_does_not_finish_or_cancel_execution(window, qtbot, tmp_path, monkeypatch): + from PySide6.QtWidgets import QFileDialog + open_project(window, qtbot, tmp_path) + window._begin_run(window._task_id) + control = window._batch_stop + owner = object() + window._workers.add(owner) + destination = tmp_path / 'support.zip' + monkeypatch.setattr(QFileDialog, 'getSaveFileName', lambda *args: (str(destination), '')) + try: + window._export_diagnostics() + qtbot.waitUntil(lambda: window._diagnostic_worker is None, timeout=5000) + assert destination.is_file() + assert owner in window._workers and window._batch_stop is control + assert not control.is_set() and not control.force_event.is_set() + finally: + window._workers.discard(owner) + window._finish_run() diff --git a/tests/v1_1/test_safe_user_stop.py b/tests/v1_1/test_safe_user_stop.py new file mode 100644 index 0000000..50f0f3f --- /dev/null +++ b/tests/v1_1/test_safe_user_stop.py @@ -0,0 +1,343 @@ +import json +from pathlib import Path +import subprocess +import sys +from threading import Event, Thread + +import pytest + +from dx_productivity.trial_task_center.worker_monitor import ProgressWatch, run_monitored_process + + +def test_unsafe_save_timeout_waits_for_completion_and_recovers_on_progress(): + watch = ProgressWatch('token', 'job', 10, 0) + assert watch.accept(b'DX_PROGRESS\ttoken\tjob\t1\tsave_and_close\t0\t1\t0\n', 0) + assert watch.action(10) == 'request_stop' + assert watch.action(20) == 'await_safe_completion' + assert watch.snapshot(now=20)['waiting_for_safe_completion'] is True + assert watch.action(200) != 'unresponsive' + assert watch.accept(b'DX_PROGRESS\ttoken\tjob\t2\tverify_saved_project\t1\t1\t0\n', 201) + assert watch.snapshot(now=201)['waiting_for_safe_completion'] is False + + +def test_real_slow_save_is_not_killed_by_inactivity(tmp_path): + script = tmp_path / 'save.py' + script.write_text('''import sys,time,pathlib +root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5] +while not (root/f'monitor-{token}.ready').exists():time.sleep(.01) +print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tsave_and_close\\t0\\t1\\t0',flush=True) +time.sleep(1.5) +(root/'saved.txt').write_text('saved and independently verified') +''', encoding='utf-8') + command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job') + result = run_monitored_process(command, .3) + assert result.returncode == 0 + assert (tmp_path / 'saved.txt').read_text() == 'saved and independently verified' + + +def test_readonly_native_wait_keeps_bounded_cancellation_without_force_ui(tmp_path): + script = tmp_path / 'read.py' + script.write_text('''import sys,time,pathlib +root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5] +while not (root/f'monitor-{token}.ready').exists():time.sleep(.01) +print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tnative_snapshot_read\\t0\\t1\\t0',flush=True) +time.sleep(1.5) +''', encoding='utf-8') + command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job') + with pytest.raises(subprocess.TimeoutExpired): + run_monitored_process(command, .3, cooperative_abort=True) + + +def test_explicit_force_stops_unsafe_worker_with_protection(tmp_path): + script = tmp_path / 'save.py' + script.write_text('''import sys,time,pathlib +root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5] +while not (root/f'monitor-{token}.ready').exists():time.sleep(.01) +print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tsave_and_close\\t0\\t1\\t0',flush=True) +(root/'started').touch() +time.sleep(30) +(root/'unexpected_completed').touch() +''', encoding='utf-8') + force = Event() + def request(): + for _ in range(200): + if (tmp_path / 'started').exists(): + force.set() + return + Event().wait(.01) + requester = Thread(target=request) + requester.start() + command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job') + try: + with pytest.raises(Exception, match='forced'): + run_monitored_process(command, .3, force_event=force) + assert (tmp_path / 'uncertain.flag').exists() + assert not (tmp_path / 'unexpected_completed').exists() + finally: + requester.join(3) + + +def test_abnormal_cad_shutdown_is_not_hidden_by_physical_release(tmp_path): + from dx_productivity.trial_task_center.batch_supervisor import _finish_response + from dx_productivity.trial_task_center.batch_resources import write_record + from dx_productivity.trial_task_center.logging_setup import configure_logging + from dx_productivity.trial_task_center.supervisor_protocol import encode, encode_error + from dx_productivity.trial_task_center.errors import TaskCenterError + root = tmp_path + journal = {'operation': 'probe', 'batch_id': 'a' * 32, 'state': 'blocked'} + write_record(root / 'batch-resources' / journal['batch_id'] / 'session.json', { + 'version': 1, 'session_id': 'a' * 32, 'kind': 'probe', 'state': 'released', + 'generation': 1, 'desktop': 'dx-' + 'b' * 32, 'processes': [], 'cad_pid': None, + 'cad_guid': None, 'checkpoints': {}, 'retries': {}, 'timings': [], + 'cad_shutdown': {'clean': False, 'cleanup_errors': [{'stage': 'orderly_shutdown', 'message': 'Quit timed out'}]}}) + response = {'business_succeeded': True, 'status': 'error', 'result': encode({'verified': True}), + 'error': encode_error(TaskCenterError('batch_resources_pending', 'Quit timed out'))} + with pytest.raises(TaskCenterError) as caught: + _finish_response(root, journal, response, configure_logging(root)) + assert caught.value.code == 'application_shutdown_failed' + assert caught.value.completed_result == {'verified': True} + assert journal['resource_recheck']['pending'] is False + + +def test_force_cannot_terminate_before_interruption_evidence_is_durable(tmp_path, monkeypatch): + from types import SimpleNamespace + from dx_productivity.trial_task_center import batch_supervisor as module + calls = [] + process = SimpleNamespace(terminate=lambda: calls.append('terminate')) + journal = {'operation': 'execute_task_queue', 'batch_id': 'a' * 32, 'attempt_id': 'b' * 32, + 'task_ids': ['task'], 'state': 'running'} + monkeypatch.setattr(module, '_mark_interrupted', lambda *args: (_ for _ in ()).throw(PermissionError('denied'))) + with pytest.raises(PermissionError): + module._force_owned_supervisor(tmp_path, journal, process) + assert calls == [] + + +def test_force_already_performed_remains_forced_when_exit_diagnostics_fail(tmp_path): + from types import SimpleNamespace + from dx_productivity.trial_task_center.batch_supervisor import _force_owned_supervisor + record = {} + def terminate(): + record['forced_termination'] = True + raise OSError('child enumeration failed after force') + process = SimpleNamespace(terminate=terminate, cleanup_record=record) + journal = {'operation': 'execute_task_queue', 'batch_id': 'a' * 32, 'attempt_id': 'b' * 32, + 'task_ids': ['task'], 'state': 'running'} + _force_owned_supervisor(tmp_path, journal, process) + saved = json.loads((tmp_path / 'batch-resources' / ('a' * 32) / 'supervisor.json').read_text(encoding='utf-8')) + assert saved['stop_mode'] == 'forced' + assert 'enumeration' in saved['force_cleanup_error'] + + +def test_interruption_targets_active_project_without_invalidating_previous_tasks(tmp_path): + from dx_productivity.trial_task_center.batch_supervisor import _mark_interrupted + from dx_productivity.trial_task_center.batch_resources import write_record + batch = 'a' * 32 + write_record(tmp_path / 'batch-resources' / batch / 'session.json', { + 'version': 1, 'session_id': batch, 'kind': 'task_queue', 'state': 'ready', + 'generation': 1, 'desktop': 'dx-' + 'b' * 32, 'processes': [], 'cad_pid': 123, + 'cad_guid': 'owned', 'checkpoints': {}, 'retries': {}, 'timings': [], 'active_task': 'current'}) + journal = {'operation': 'execute_task_queue', 'batch_id': batch, 'attempt_id': 'c' * 32, + 'task_ids': ['previous', 'current'], 'state': 'running'} + _mark_interrupted(tmp_path, journal, 'forced stop') + assert (tmp_path / 'tasks/current/quick-run/uncertain.flag').exists() + assert not (tmp_path / 'tasks/previous/quick-run/uncertain.flag').exists() + + +@pytest.mark.parametrize('interactive', [False, True]) +def test_supervisor_only_holds_for_resolution_when_caller_has_force_control(tmp_path, monkeypatch, interactive): + import textwrap + from dx_productivity.trial_task_center import batch_supervisor as module + from dx_productivity.trial_task_center.stop_control import StopControl + from dx_productivity.trial_task_center.service import TaskCenterService + from dx_productivity.trial_task_center.store import TaskStore + from dx_productivity.trial_task_center.logging_setup import configure_logging + bootstrap = textwrap.dedent(''' + import sys,json + from pathlib import Path + from dx_productivity.trial_task_center import batch_worker,batch_runtime + from dx_productivity.trial_task_center.stop_control import StopControl + original=batch_runtime.BatchCadRuntime + def runtime(*args,**kwargs): + result=original(*args,**kwargs) + Path(sys.argv[1]).with_name('interactive-witness.json').write_text(json.dumps(isinstance(result.stop_event,StopControl))) + return result + batch_runtime.BatchCadRuntime=runtime + batch_worker.run(Path(sys.argv[1])) + ''') + monkeypatch.setattr(module, '_command', lambda request: (sys.executable, '-c', bootstrap, str(request))) + store = TaskStore(tmp_path / 'tasks.db') + try: + service = TaskCenterService(store, configure_logging(tmp_path)) + module.run_supervised(service, 'probe', {}, stop_event=StopControl() if interactive else Event()) + witness = next((tmp_path / 'batch-resources').glob('*/interactive-witness.json')) + assert json.loads(witness.read_text()) is interactive + finally: + store.close() + + +def test_uncertain_operation_keeps_owner_until_explicit_force(tmp_path): + from dx_productivity.trial_task_center.stop_control import StopControl, wait_for_resolution + control = StopControl() + failures = [] + def owner(): + try: + wait_for_resolution(control, tmp_path, 'save result unknown') + except Exception as error: + failures.append(error) + thread = Thread(target=owner) + thread.start() + try: + for _ in range(100): + if (tmp_path / 'worker-progress.json').exists(): + break + Event().wait(.01) + control.set() # Ordinary stop never authorizes discarding a changed project. + thread.join(.15) + assert thread.is_alive() + assert (tmp_path / 'uncertain.flag').exists() + assert json.loads((tmp_path / 'worker-progress.json').read_text())['waiting_for_safe_completion'] + control.request_force() + thread.join(3) + assert not thread.is_alive() + assert 'forced' in str(failures[0]) + finally: + control.request_force() + thread.join(3) + + +def test_failed_protection_fsync_is_retried_before_force_is_allowed(tmp_path, monkeypatch): + from dx_productivity.trial_task_center import stop_control as module + control = module.StopControl() + original = module.os.fsync + attempts, failures = [], [] + def denied(fd): + attempts.append(fd) + raise OSError('evidence fsync denied') + monkeypatch.setattr(module.os, 'fsync', denied) + def owner(): + try: + module.wait_for_resolution(control, tmp_path, 'save unknown') + except Exception as error: + failures.append(error) + thread = Thread(target=owner) + thread.start() + try: + for _ in range(100): + if control.control_error: + break + Event().wait(.01) + assert control.control_error + control.request_force() + thread.join(.25) + assert thread.is_alive(), 'a file left by failed fsync must not authorize force' + assert len(attempts) >= 2 + monkeypatch.setattr(module.os, 'fsync', original) + control.request_force() + thread.join(3) + assert failures and 'forced' in str(failures[0]) + finally: + monkeypatch.setattr(module.os, 'fsync', original) + control.request_force() + thread.join(3) + + +@pytest.mark.parametrize('deny_first,transport_failure', [(False, False), (True, False), (False, True)]) +def test_real_supervisor_force_preserves_protection_and_never_replays(tmp_path, monkeypatch, deny_first, transport_failure): + import os + import textwrap + import time + import win32api + import win32event + from dx_productivity.trial_task_center import batch_supervisor as module + from dx_productivity.trial_task_center.stop_control import StopControl + from dx_productivity.trial_task_center.service import TaskCenterService + from dx_productivity.trial_task_center.store import TaskStore + from dx_productivity.trial_task_center.logging_setup import configure_logging + from dx_productivity.trial_task_center.errors import TaskCenterError + store = TaskStore(tmp_path / 'tasks.db') + service = TaskCenterService(store, configure_logging(tmp_path), supervised=True) + task = store.create_task('force fixture') + evidence = service._project_task_service(task.task_id)._quick_run_preparer.data_root / 'quick-run' + witness = evidence / 'started' + bootstrap = textwrap.dedent(''' + import sys,os,time,json + from pathlib import Path + from dx_productivity.trial_task_center import batch_worker + from dx_productivity.trial_task_center.service import TaskCenterService + def business(self, task_id, **kwargs): + root=self._project_task_service(task_id)._quick_run_preparer.data_root/'quick-run' + root.mkdir(parents=True,exist_ok=True) + (root/'started').write_text(str(os.getpid())) + request=json.loads(Path(sys.argv[1]).read_text(encoding='utf-8')) + print(json.dumps({'version':1,'batch_id':request['batch_id'],'attempt_id':request['attempt_id'], + 'generation':1,'sequence':100,'type':'progress','payload':'fixture executing'}),flush=True) + time.sleep(30) + (root/'unexpected_completed').touch() + TaskCenterService.execute_quick_run=business + batch_worker.run(Path(sys.argv[1])) + ''') + original_command = module._command + monkeypatch.setattr(module, '_command', lambda request: + (sys.executable, '-c', bootstrap, str(request)) + if json.loads(request.read_text())['operation'] == 'execute_quick_run' else original_command(request)) + original_mark = module._mark_interrupted + denied = [] + def mark(*args): + if deny_first and not denied: + denied.append(True) + raise PermissionError('force evidence denied once') + return original_mark(*args) + monkeypatch.setattr(module, '_mark_interrupted', mark) + control = StopControl() + driver_errors, survived = [], [] + def request(): + try: + deadline = time.monotonic() + 15 + while not witness.exists() and time.monotonic() < deadline: + time.sleep(.02) + assert witness.exists() + pid = int(witness.read_text()) + if transport_failure: + while not control.control_error and time.monotonic() < deadline: + time.sleep(.02) + assert control.control_error + handle = win32api.OpenProcess(0x100000, False, pid) + try: + survived.append(win32event.WaitForSingleObject(handle, 0) == 258) + finally: + handle.Close() + control.request_force() + if deny_first: + while not control.control_error and time.monotonic() < deadline: + time.sleep(.02) + assert control.control_error + handle = win32api.OpenProcess(0x100000, False, pid) + try: + survived.append(win32event.WaitForSingleObject(handle, 0) == 258) + finally: + handle.Close() + control.request_force() + except Exception as error: + driver_errors.append(error) + thread = Thread(target=request) + thread.start() + try: + with pytest.raises(TaskCenterError) as caught: + module.run_supervised(service, 'execute_quick_run', {'task_id': task.task_id}, stop_event=control, + on_progress=(lambda _: (_ for _ in ()).throw(RuntimeError('UI feedback failed'))) if transport_failure else None) + assert not driver_errors + assert caught.value.code == 'user_forced_stop' + assert (evidence / 'uncertain.flag').exists() + marker = (evidence / 'uncertain.flag').read_bytes() + assert not (evidence / 'unexpected_completed').exists() + journals = [json.loads(p.read_text(encoding='utf-8')) for p in (tmp_path / 'batch-resources').glob('*/supervisor.json')] + forced = next(r for r in journals if r['operation'] == 'execute_quick_run') + assert forced['stop_mode'] == 'forced' and forced['business_uncertain'] is True + assert service.cleanup_batch_resources() == () + assert (evidence / 'uncertain.flag').read_bytes() == marker + assert not (evidence / 'unexpected_completed').exists() + if deny_first or transport_failure: + assert survived == [True] + finally: + thread.join(3) + store.close() diff --git a/tools/probe_safe_stop_live.py b/tools/probe_safe_stop_live.py new file mode 100644 index 0000000..40cc097 --- /dev/null +++ b/tools/probe_safe_stop_live.py @@ -0,0 +1,129 @@ +"""Request ordinary stop during save on disposable projects; never replay a run.""" +from dataclasses import asdict +import hashlib +import json +from pathlib import Path +import re +import shutil +import sys +from threading import Event, Thread +import time +from uuid import uuid4 + +REPO = Path(__file__).resolve().parents[1] +sys.path[:0] = [str(REPO / 'src'), str(REPO / 'tools')] +from probe_batch_runtime_live import hashes, compare_saved, save, OLD, NEW, NAMES +from dx_productivity.trial_task_center.isolated_cad import IsolatedCadSession +from dx_productivity.trial_task_center.logging_setup import configure_logging +from dx_productivity.trial_task_center.service import TaskCenterService +from dx_productivity.trial_task_center.store import TaskStore +from dx_productivity.trial_task_center.bom_plan import write_bom_template +from dx_productivity.trial_task_center.stop_control import StopControl +from dx_productivity.trial_task_center.task_queue import execute_task_queue + + +def main(): + import argparse + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--source-root', required=True, type=Path) + parser.add_argument('--library', required=True, type=Path) + parser.add_argument('--run-native', action='store_true', required=True) + args = parser.parse_args() + output = REPO / 'build' / ('safe-stop-live-' + uuid4().hex[:10]) + output.mkdir() + source_root, library = args.source_root.resolve(strict=True), args.library.resolve(strict=True) + source_hashes = {name: hashes(source_root / name) for name in NAMES} + library_hash = hashlib.sha256(library.read_bytes()).hexdigest() + projects = [] + for name in NAMES: + folder = output / 'projects' / name + shutil.copytree(source_root / name, folder) + project = folder / (name + '.prj') + text, count = re.subn(r'(?m)^KEY CentralLibrary .*$', lambda _: f'KEY CentralLibrary "{library.as_posix()}"', + project.read_text(encoding='utf-8-sig')) + assert count == 1 + project.write_text(text, encoding='utf-8') + projects.append(project) + report = {'scope': 'source supervised service, production VBS and real CAD; no GUI or frozen EXE', + 'output': str(output), 'projects': list(map(str, projects))} + print(json.dumps(report, ensure_ascii=False), flush=True) + store = TaskStore(output / 'tasks.db') + service = TaskCenterService(store, configure_logging(output / 'data'), supervised=True) + control, finished = StopControl(), Event() + watcher = None + def inspect(project, sheets, label): + evidence = output / (label + '-' + project.stem) + evidence.mkdir() + spec, result = evidence / 'spec.json', evidence / 'graph.json' + save(spec, {'project': str(project), 'sheets': sorted(sheets), 'destination': str(result)}) + with IsolatedCadSession(evidence) as cad: + child = cad.start_worker((sys.executable, str(REPO / 'tools/probe_batch_runtime_live.py'), + '--repo', str(REPO), '--inspect-file', str(spec), '--run-native'), stdout=-1, stderr=-1) + try: + code = child.wait(timeout=180) + (evidence / 'worker-output.txt').write_bytes(child.stdout.read() + b'\n' + child.stderr.read()) + assert code == 0, (evidence / 'worker-output.txt').read_text(errors='replace') + finally: + child.close() + return json.loads(result.read_text(encoding='utf-8')) + try: + tasks = service.create_project_tasks(output / 'projects', selected_paths=tuple(projects)) + bom = write_bom_template(output / 'input.xlsx', ({'project_path': str(p), 'old_part_number': OLD, + 'new_part_number': NEW, 'refdes': 'RV05', 'enabled': True, 'sync_nc': True} for p in projects)) + imported = service.inspect_bom_batch(tasks[0].task_id, bom) + assert imported.passed_rows == 2 and imported.preview is not None + service.save_bom_batch(imported.preview, {p.task_id: p.rows for p in imported.preview.projects}) + before, sheet_sets = {}, {} + for project in projects: + rows = next(p.rows for p in imported.preview.projects if p.project_path.resolve() == project.resolve()) + sheet_sets[project.stem] = {(row.target.schematic_name, row.target.sheet_number) for row in rows} + before[project.stem] = inspect(project, sheet_sets[project.stem], 'before') + def stop_during_save(): + while not finished.wait(.01): + for path in (output / 'data/tasks').glob('*/quick-run/quick-part-replace.log'): + try: + text = path.read_text(encoding='utf-8-sig') + except OSError: + continue + if 'save_requested job=' in text: + report['stop_requested'] = {'time_ns': time.time_ns(), 'log': str(path), + 'trigger': 'production save_requested log; ordinary stop only'} + control.set() + return + watcher = Thread(target=stop_during_save, daemon=True) + watcher.start() + result = execute_task_queue(service, tuple(task.task_id for task in tasks), stop_event=control) + finished.set() + watcher.join(3) + report['queue_result'] = asdict(result) + assert report.get('stop_requested') and not control.force_event.is_set() + assert result.paused and [row.state for row in result.entries] == ['completed', 'pending'], result + report['comparisons'] = [] + for index, project in enumerate(projects): + after = inspect(project, sheet_sets[project.stem], 'after') + if index == 0: + report['comparisons'].append(compare_saved(before[project.stem], after)) + else: + assert after == before[project.stem], 'second project was changed after stop' + report['comparisons'].append({'unchanged': True, 'components': len(after)}) + assert not service.pending_batch_resources() + assert not list((output / 'data').rglob('uncertain.flag')) + report['verified'] = True + except BaseException as error: + import traceback + report.update(error=repr(error), traceback=traceback.format_exc()) + raise + finally: + finished.set() + if watcher is not None: + watcher.join(3) + store.close() + report['source_unchanged'] = all(hashes(source_root / name) == source_hashes[name] for name in NAMES) + report['library_lmc_unchanged'] = hashlib.sha256(library.read_bytes()).hexdigest() == library_hash + save(output / 'report.json', report) + print(json.dumps(report, ensure_ascii=False), flush=True) + return 0 if report.get('verified') and report['source_unchanged'] and report['library_lmc_unchanged'] else 1 + + +if __name__ == '__main__': + raise SystemExit(main())