Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
12895e9868 | ||
|
|
9c03cd5900 |
No files matched your search
@@ -4,11 +4,11 @@
|
||||
|
||||
面向 **Windows + DxDesigner / Expedition Enterprise 7.9.5** 的器件批量替换工具。一个工程对应一个任务,可以只替换同料号下的部分位号;支持直接编辑和 Excel BOM 导入,同批工程按顺序执行。
|
||||
|
||||
**最新版本:V1.1.4 Beta 3(预发布)** · [下载便携包](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/download/v1.1.4-beta.3/DxPartReplace-1.1.4-beta.3.zip) · [版本说明与附件](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/tag/v1.1.4-beta.3) · [完整使用说明](docs/user/v1.1.0-project-tasks.md)
|
||||
**最新版本:V1.1.4 Beta 4(预发布)** · [下载便携包](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/download/v1.1.4-beta.4/DxPartReplace-1.1.4-beta.4.zip) · [版本说明与附件](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/tag/v1.1.4-beta.4) · [完整使用说明](docs/user/v1.1.0-project-tasks.md)
|
||||
|
||||
所有发布均标记为 **Beta 预发布**。V1.1.4 Beta 1 因 BOM 导入缺少 `win32timezone` 已撤回,请使用 Beta 3;旧 Git 标签保留供回溯。升级前备份 `data`,完整更新 EXE 与 `_internal`,不要只替换 EXE。
|
||||
所有发布均标记为 **Beta 预发布**。V1.1.4 Beta 1 因 BOM 导入缺少 `win32timezone` 已撤回,请使用 Beta 4;旧 Git 标签保留供回溯。升级前备份 `data`,完整更新 EXE 与 `_internal`,不要只替换 EXE。
|
||||
|
||||
Beta 3 面向工单 #1/#3 复测:修复内部长路径访问,将后台资源移交独立批次监督进程,补齐安全重建、释放诊断与可选关闭归档。[本版变更和复测说明](docs/releases/v1.1.4-beta.3.md)。
|
||||
Beta 4 面向停止与资源收尾复测:普通暂停等待当前项保存核验,写入中无进度不再自动杀进程;新增明确确认的强制结束,保留中断证据与恢复入口。[本版变更和复测说明](docs/releases/v1.1.4-beta.4.md)。
|
||||
|
||||
本版提供[目标封装选择](docs/user/target-cell-selection.md):目标封装可留空,此时工具不主动覆写封装,可后续与中心库同步;需要指定时,在料号规则中读取并选择合法封装。BOM 保持原 11 列。
|
||||
|
||||
@@ -20,7 +20,7 @@ Beta 3 面向工单 #1/#3 复测:修复内部长路径访问,将后台资
|
||||
|
||||
### 第 1 步:解压并打开应用
|
||||
|
||||
[下载 V1.1.4 Beta 3 便携包](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/download/v1.1.4-beta.3/DxPartReplace-1.1.4-beta.3.zip),**完整解压**,双击 `DxPartReplace.exe`。保留它与 `_internal` 的相对位置,无需安装 Python;本机需要可用的 DxDesigner、许可证和中央库。
|
||||
[下载 V1.1.4 Beta 4 便携包](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/download/v1.1.4-beta.4/DxPartReplace-1.1.4-beta.4.zip),**完整解压**,双击 `DxPartReplace.exe`。保留它与 `_internal` 的相对位置,无需安装 Python;本机需要可用的 DxDesigner、许可证和中央库。
|
||||
|
||||
打开后看到下图主窗口。首次使用请准备完整的工程副本,执行前保存并关闭待处理工程。
|
||||
|
||||
@@ -133,6 +133,8 @@ Beta 3 面向工单 #1/#3 复测:修复内部长路径访问,将后台资
|
||||
|
||||
运行期间可以在主桌面查看其他文件,请勿打开或修改本次正在处理的同一工程。需要暂停时,点击 **完成当前项后暂停**,等待当前步骤安全收尾。
|
||||
|
||||
Beta 4 在保存或核验尚未确认时保留应用等待,可先 **导出诊断包**。只有明确接受中断后需要核验时,才选择 **强制结束…**;之后先核验恢复,不能直接重复替换。详见[停止、强制结束与恢复](docs/user/safe-stop-and-recovery.md)。
|
||||
|
||||
**完成标志:** 任务显示已完成,底部提示本批执行结束。还要打开结果核对,不能只看进度条。
|
||||
|
||||
### 第 12 步:查看实际修改结果与日志
|
||||
@@ -196,7 +198,7 @@ DxPartReplace/
|
||||
|
||||
## 验证范围与已知限制
|
||||
|
||||
V1.1.4 Beta 3 新增批次后台资源管理:一次 BOM 导入或顺序执行共用一个私有桌面,CAD 按需启动并在同批工程间复用,批次结束统一退出。若提示 **资源释放待处理**,已保存核验的结果保留,请点击 **重新清理资源**;该操作只清理后台资源,不会再次替换。旧的工程保护锁仍按“核验并恢复”处理。设计、异常边界与实机结果见[批次运行时验收记录](docs/testing/2026-10-21-batch-private-runtime.md)。
|
||||
V1.1.4 Beta 4 沿用批次后台资源管理:一次 BOM 导入或顺序执行共用一个私有桌面,CAD 按需启动并在同批工程间复用,批次结束统一退出。若提示 **资源释放待处理**,已保存核验的结果保留,请点击 **重新清理资源**;该操作只清理后台资源,不会再次替换。旧的工程保护锁仍按“核验并恢复”处理。设计、异常边界与实机结果见[批次运行时验收记录](docs/testing/2026-10-21-batch-private-runtime.md)。
|
||||
|
||||
资源释放会追加日志:在 **替换结果 / 日志 → 完整日志** 查看本批开始释放、释放成功/失败及重新清理的结果。独立文件位于 `data/batch-resources/<批次ID>/resource-release.log`,记录批次、桌面、CAD PID、退出码、释放耗时、句柄关闭和桌面消失的分别确认,以及失败原因;同时保留 `session.json` 的原始释放证明。重启后重新检查也会追加记录,不覆盖原失败日志。
|
||||
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# V1.1.4 Beta 4 构建与发布验证
|
||||
|
||||
2026-10-26,用户授权发布下一版 Beta。版本 `1.1.4-beta.4`,Python 元数据 `1.1.4b4`,Windows 文件版本 `1.1.4.4`,预发布标志开启。业务修改基于 `9c03cd5`;本轮产品代码只调整版本标识。
|
||||
|
||||
## 构建门禁
|
||||
|
||||
- `packaging/Build-DxPartReplace.ps1`:退出码 0,源码门禁 **923 passed,232.69 秒**,一条既有 pywinauto STA 警告。
|
||||
- 打包配置及外部验收脚本检查:**17 passed,11.08 秒**。
|
||||
- 构建产物在完整冻结 EXE 主进程中启动独立监督进程、读取真实进程创建时间并确认自有进程/Job/桌面释放,再完成 GUI 启动与正常关闭检查。
|
||||
- 冻结的 92 个 `dx_productivity` 模块及应用入口逐项对照源码,递归标准化 code filename 后 code object 一致。包含新的 `stop_control` 模块;两份 VBS 在包装阶段另做哈希比对。
|
||||
- EXE SHA256:`ee64d983f6f0457bac2bb16225ee8a8044753b5bf7779e498549c61d8ed4d13b`。
|
||||
|
||||
构建日志:`build/release-1.1.4-beta.4-build.log`。源码比对:`build/release-1.1.4-beta.4/frozen-source-check.json`。实现阶段扩大测试 1147 项通过的范围见[安全停止验收](../testing/2026-10-26-safe-stop-acceptance.md),不与本轮构建门禁数量混算。
|
||||
|
||||
## 冻结监督进程实机验证
|
||||
|
||||
本轮用源码控制器驱动上述正式 EXE 的 `--batch-worker`,执行 BOM 导入与真实 CAD 队列;批内 helper 和 VBS 使用冻结产物。复制两个全新工程,不重放之前完成的替换。
|
||||
|
||||
在第一个工程的生产 `save_requested` 日志出现时只请求普通停止,结果为第一任务 completed、第二任务 pending、队列 paused。新 CAD 实例独立读回确认:第一工程 57 器件、103 引脚连接正确,56 个非目标器件保持;第二工程 57 器件完整属性/连接图与执行前一致。
|
||||
|
||||
执行批次 `orderly_shutdown=true`、`clean=true`、`forced_termination=false`,CAD 与 Quit helper 退出码均为 0,资源状态 released。源工程目录哈希和中央库 `.lmc` 哈希不变。
|
||||
|
||||
证据:`build/release-1.1.4-beta.4/frozen-save-stop.json` 与 `build/safe-stop-live-9894f95db1/`。后者由通用 source 探针生成,保留了其原始 scope 文本;本轮实际冻结进程路由、调用请求和 EXE 哈希以 companion `frozen-save-stop.json` 为准。前后独立读取使用源码读取器,不将其说成全部入口均冻结,也不将该测试说成完整 GUI 交互停止。
|
||||
|
||||
## 包装与公开附件
|
||||
|
||||
便携包只从 EXE、完整 `_internal`、仓库教程及文档生成,附空白 11 列 BOM 模板和历史窗口 GIF;不带 `data`、任务数据库、用户工程、工单附件或现场日志。
|
||||
|
||||
包装脚本要求本轮冻结停止验证的 EXE 哈希与被打包文件一致,检查 ZIP CRC、解压路径、所有程序文件 SHA256,并在清除源码 Python 环境变量后执行解压出来的完整 EXE smoke。ZIP 大小、SHA256、程序文件清单与解压验证结果由公开 `release-manifest-1.1.4-beta.4.json` 记录。
|
||||
|
||||
附件:`DxPartReplace-1.1.4-beta.4.zip`、`SHA256SUMS-1.1.4-beta.4.txt`、`release-manifest-1.1.4-beta.4.json`、`BOM-template.xlsx`、`window-tutorial-1.1.4-beta.4.gif`。GIF 沿用 1.1.2 基础教程,新停止功能以包内说明和界面为准。
|
||||
|
||||
发布流程为先创建草稿,上传后逐项回下载核对大小和 SHA256,全部一致才公开为 Beta 预发布。远端响应与附件校验证据保存在 `build/release-1.1.4-beta.4/`;认证信息不写入文件或日志。本轮没有授权给工单发消息,因此不发布工单评论、不关闭工单,也不修改旧版本附件。
|
||||
|
||||
## 现场边界
|
||||
|
||||
仍未完成原工单 102 个工程的全量现场复测,也没有完整冻结 GUI 的真实保存中强杀或掉电无损验收。工单现场桌面残留持有者尚未查明。本版本提供已验证的停止与收尾改进,不能据此宣布现场事故根因已经根治。
|
||||
@@ -0,0 +1,41 @@
|
||||
# DxPartReplace V1.1.4 Beta 4
|
||||
|
||||
2026-10-26,**Beta 预发布/安全停止与应用收尾复测版**。关联工单 [#1](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/1)、[#3](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/3)。
|
||||
|
||||
## 下载与升级
|
||||
|
||||
完整解压 `DxPartReplace-1.1.4-beta.4.zip`,运行 `DxPartReplace.exe`。无需安装 Python;保留 EXE 和 `_internal` 的相对位置,本机需要可用的 DxDesigner、许可证及中央库。
|
||||
|
||||
请在新目录解压,使用工程副本复测。沿用旧任务前,先安全退出旧程序并备份完整 `data`;不要同时用两个版本打开同一数据目录。不要删除原有保护记录来绕过核验。
|
||||
|
||||
便携包包含中文教程、空白 11 列 BOM 模板、历史窗口操作 GIF、停止与恢复说明及文件校验清单。GIF 沿用 1.1.2 基础操作,新版停止与强制结束功能以实际界面及[停止说明](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/src/tag/v1.1.4-beta.4/docs/user/safe-stop-and-recovery.md)为准。
|
||||
|
||||
## 本版修改
|
||||
|
||||
- 普通“完成当前项后暂停”只停止后续调度,当前替换项完成必要校验、保存、回读和关闭后再停止;关闭窗口也等待收尾。
|
||||
- 有写入可能的阶段长时间没有进度时保留应用等待,不再仅凭超时自动杀进程;纯只读采集仍采用有界取消。
|
||||
- 新增独立的“强制结束…”确认。先可靠保存中断保护和批次证据,再终止本程序拥有的进程;保护写入失败时拒绝本次强制结束并显示原因。
|
||||
- 当前修改或保存结果不确定、应用拒绝退出时保留现场,禁止盲目保存、丢弃和重放。等待期间可导出诊断包。
|
||||
- 保存请求、保存调用返回、保存回读和工程关闭分别记录,避免保存后关闭失败被误记为未保存。
|
||||
- 修复收尾记录写入失败跳过正常 Quit 的路径;CAD 主进程正常退出后,给自有辅助进程短暂自然退出机会,再决定是否需要强制清理。
|
||||
- 正常退出、强制终止和物理资源释放分别报告。资源消失不再掩盖应用退出异常,已经核验的业务结果继续保留。
|
||||
- 控制通信异常不会直接授权终止正在运行的交互批次;强制结束及进程退出诊断记录保持一致。
|
||||
|
||||
## 复测建议
|
||||
|
||||
1. 新建两个工程副本的少量位号任务,导入 BOM 并保存计划。
|
||||
2. 在第一项执行或保存时点击普通暂停,核对当前项完成保存回读,下一项保持未执行;重新打开工程核对料号和连接。
|
||||
3. 对长时间等待先导出诊断。仅在接受当前结果需要核验时确认“强制结束…”;重开后先“核验并恢复”,不要直接重复替换。
|
||||
4. 如仍有异常,保留版本、触发步骤、诊断 ZIP 和失败工程名称。资源待处理时的清理不会重新执行替换。
|
||||
|
||||
为避免强制请求与任务切换同时发生而漏记目标,本版强制结束会保守保护本批可能涉及的任务。先前已完成任务也可能需要复核,但其保存结果和执行证据不被覆盖或重放。
|
||||
|
||||
## 验证范围与已知边界
|
||||
|
||||
实现阶段扩展回归 **1147 项通过**。源码监督入口在两份新工程副本上验证了保存期间请求普通停止:第一工程 completed,第二工程 pending;新 CAD 独立读回确认第一工程 57 器件、103 引脚连接正确、56 个非目标器件不变,第二工程完整图未变化。源工程及中央库 `.lmc` 哈希不变,执行批次正常退出且没有强制清理。
|
||||
|
||||
本轮发布门禁 923 项通过,打包与验收脚本检查 17 项通过。正式 EXE 的冻结监督进程再次完成真实 CAD 保存期间普通停止及新实例独立读回;当前项完成、下一项未执行,正常退出且无强制终止。该实机入口由源码控制器驱动,不等同于完整 GUI 交互验收。
|
||||
|
||||
发布构建、解压后完整 EXE 启动、冻结监督进程及资源释放检查另见[Beta 4 构建记录](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/src/tag/v1.1.4-beta.4/docs/releases/v1.1.4-beta.4-build.md)和附件 manifest。源码/受控辅助进程故障测试不等同于完整冻结 GUI 下的真实 CAD 强制中断或掉电无损验证。
|
||||
|
||||
**现场资源残留根因仍未全部闭环。** 原工单全量 102 个工程及特定系统组合仍需现场复测,本版不宣称所有环境已根治,不自动关闭关联工单。完整范围见[实施验收](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/src/tag/v1.1.4-beta.4/docs/testing/2026-10-26-safe-stop-acceptance.md)。
|
||||
@@ -0,0 +1,62 @@
|
||||
# Safe User Stop Implementation Plan
|
||||
|
||||
> **For agentic workers:** use superpowers:subagent-driven-development or executing-plans; continue within the user-approved scope without another approval round.
|
||||
|
||||
**Goal:** 用户停止只在安全边界收尾;写入中的无进度不自动杀进程;明确强制中断有保护证据和恢复入口。
|
||||
|
||||
**Architecture:** 保留现有 Event/队列边界和独立监督进程,加入独立的强制停止意图、等待状态反馈;退出正常性与物理资源复核分开。使用既有任务保护与批次记录,不新增数据库版本。
|
||||
|
||||
**Tech Stack:** Python 3.12、Windows pywin32、PySide6、VBScript、pytest。
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-10-26-safe-user-stop.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- 不修改原工程、中心库或未跟踪 feedback;真实测试只复制 fixture。
|
||||
- 不放宽身份、写前保护、禁止不确定重放的门禁。
|
||||
- 普通停止与强制结束独立,确认前不得产生强制意图。
|
||||
- 不通过无限创建桌面或删除旧保护绕过异常。
|
||||
- 不发布、不推送;本地分支 `codex/day0-safe-stop`。
|
||||
|
||||
## Task 1: 退出协议与后代进程
|
||||
|
||||
Files: `isolated_cad.py`, `isolated_desktop.py`, `tests/v1_1/test_isolated_cad.py`, `test_isolated_desktop.py`.
|
||||
|
||||
- [x] 先复现 closing 记录失败跳过 Quit、父进程自然退出后仍立即终止子进程的测试。
|
||||
- [x] 诊断写入失败不跳过满足身份/空工程条件的 Quit;将记录失败独立保留并报告,禁止伪报 clean。
|
||||
- [x] 根进程正常退出时,给 Job 后代 2 秒有界自然退出窗口;到期再强制并记录 `forced_termination`, `termination_reason`, `forced_descendant_pids`。根进程仍活着的异常清理仍有界。
|
||||
- [x] 保留所有权和退出确认,测试自然退出与强制升级两条路径。
|
||||
|
||||
## Task 2: 停止、等待与强制中断
|
||||
|
||||
Files: new `stop_control.py`; `worker_monitor.py`, `project_session.py`, `batch_runtime.py`, `batch_supervisor.py`, `execution_feedback.py`; associated tests.
|
||||
|
||||
Interface:
|
||||
```python
|
||||
class StopControl(Event):
|
||||
force_event: Event
|
||||
def request_force(self):
|
||||
self.set()
|
||||
self.force_event.set()
|
||||
```
|
||||
- [x] 首先测试 unsafe 保存阶段超时仍保留 worker,安全阶段仍有 watchdog,显式 force 才结束并产生 uncertain。
|
||||
- [x] `ProgressWatch` 提供 `waiting_for_safe_completion`;新进度可清除提示,不把心跳当作写入推进。
|
||||
- [x] 监督父进程持久化停止意图;force 先写保护与中断记录,再只终止已拥有的监督进程 Job,不使用名字或不可信 PID 清理。
|
||||
- [x] 业务完成但退出异常仍通过独立错误返回,附带已核验业务结果;纯桌面延迟且退出正常可继续使用现有复核恢复。
|
||||
- [x] 普通停止在当前项收尾后暂停;会话 close 不得因普通等待超时反向强杀仍在写入的 worker。
|
||||
|
||||
## Task 3: 保存边界与界面
|
||||
|
||||
Files: `QuickPartReplace.vbs`, `ui/run_panel.py`, `ui/main_window.py`, `ui/execution_feedback.py`, protocol serializer if required; VBS/UI tests.
|
||||
|
||||
- [x] VBS 用真实函数的失败先行测试覆盖保存已发出但工程关闭失败;保存请求前设置“可能保存”状态,独立记录保存返回/回读/关闭,不再声称未保存。
|
||||
- [x] 正常停止显示等待当前项收尾;unsafe 超时显示保存/核验尚未确认。
|
||||
- [x] 提供与普通暂停分离的强制结束按钮,确认文案明确结果待核验;取消确认没有副作用。
|
||||
- [x] 关闭窗口在运行/等待期间保持打开;任务失败后取消自动关闭,让用户看见恢复信息。
|
||||
|
||||
## Task 4: 集成验证与审查
|
||||
|
||||
- [x] 运行相关 VBS、monitor、supervisor、queue、UI、资源测试及完整发布门禁范围。
|
||||
- [x] 实机空 CAD 自然退出、两工程副本正常保存及独立回读;停止/强制结束采用受控替身或独立进程,明确验证边界。
|
||||
- [x] 独立审查关注:不确定写入重放、等待时 GUI 退出杀进程、force 身份校验与持久化失败、结果错误被吞、已完成项保留。
|
||||
- [x] 更新操作文档和验收记录,交付本地可审查修改及测试证据。
|
||||
@@ -0,0 +1,9 @@
|
||||
# 用户停止与安全收尾(已批准)
|
||||
|
||||
用户已在本会话批准:普通停止冻结后续调度;未修改可取消;当前已修改项完成校验、保存、回读后停止;保存或修改结果不明保持保护,不自动重放或盲目保存/丢弃;关闭窗口等待收尾;强制结束必须由用户明确选择,记录中断并在重启后核验。
|
||||
|
||||
业务结果、应用正常退出、强制清理和物理资源状态分别记录。资源消失不抹去应用异常。保存中无进度超时进入等待/提示,不自动终止。提供明确确认的强制结束出口;其范围仅为本程序当前监督进程及自有 Job。
|
||||
|
||||
普通停止以当前替换项及其必要收尾为边界,不启动同工程下一项。保护记录、批次身份和进程创建时间核验继续有效。先保存中断证据再执行工具内强制终止;持久化失败时拒绝工具内强制操作并反馈错误。系统外强杀/掉电依赖已有写前保护与重启核验,不承诺可在终止时写入日志。
|
||||
|
||||
保持源码版本,不发布包、不关闭远端事故工单。真实工程验证只用全新副本。
|
||||
@@ -0,0 +1,134 @@
|
||||
# Day0:保存、关闭、退出全链路排查
|
||||
|
||||
2026-10-26,北京时间。源码基线 `b9391015650bf5c43a77a017c5484e9a8f352266`。承接[资源事故分析](2026-10-26-day0-resource-release-analysis.md),本轮以核心应用完成条件为主线。
|
||||
|
||||
## 结论
|
||||
|
||||
发现并验证了四个具体的生命周期问题:
|
||||
|
||||
1. **CAD 主进程正常退出后,程序会立即终止 Job 中的剩余辅助进程,没有先给予其自然退出窗口。** 本机实测 `vnsd.exe` 再等约 0.813 秒即可自然退出;这个等待可以避免该次强制终止。
|
||||
2. **退出阶段的记录写入失败会跳过正常 Quit。** 随后仍进入自有进程清理。用生产退出控制流和替身 CAD 复现。
|
||||
3. **保存调用返回、工程关闭失败时,“已保存”标记仍可能没有设置。** 实际 VBS 函数配合替身 CAD 复现,说明保存与关闭的状态记录边界混在一起。
|
||||
4. **物理资源复核通过后,上层可能返回已完成业务结果,同时不再向调用方抛出正常退出失败。** 生产运行时记录可以出现 `cad_shutdown.clean=false`、资源状态 `released`、监督记录 outcome 为 `error`,但函数返回 completed 任务结果。日志证据保留,调用方的返回语义却丢失了退出异常。
|
||||
|
||||
这些问题与“只盯桌面释放”的分析不同:资源消失不能代表应用正常退出,更不能取代数据保存确认。
|
||||
|
||||
同时,两份全新工程副本的真实替换、保存、退出及新 CAD 实例独立回读均通过,未发现本轮保存数据丢失。上述问题不是已经证明的工单 #1 唯一根因;原附件中的 #1 属于读取后的收尾失败,#3 尚未启动 CAD。现场版本、实际触发路径和完整诊断仍须对照。
|
||||
|
||||
## 一、实机正常链路
|
||||
|
||||
使用已有 Audio_D2128、Audio_T605A fixture 的全新副本,生产服务入口执行 BOM 导入及两工程队列,每工程仅 RV05:`4100-CA0000-T7 → 4100-CA0000-T0`,保留原生目标 Value 策略。原始 fixture 未修改。
|
||||
|
||||
证据目录:`build/day0-resource-analysis-20261026/live-chain-bbd9fba7ce444e3d8e5627a6f4a385d0/`。汇总:`build/day0-resource-analysis-20261026/live-chain-summary.json`。进程时间线为同级 `live-chain-bbd9fba7ce444e3d8e5627a6f4a385d0-process-timeline.jsonl`。
|
||||
|
||||
| 检查 | 本次结果 |
|
||||
| --- | --- |
|
||||
| BOM 导入 | 2 行通过,3.875 秒 |
|
||||
| 顺序执行 | 两任务 completed,155.843 秒,共用一个执行 CAD |
|
||||
| 保存关闭图页调用 | D2128 约 0.32 秒,T605A 约 0.21 秒 |
|
||||
| 保存后的 CloseProject | 两次返回 True,耗时约 1.68 / 1.33 秒;均先观察到 ProjectClosed 事件,再观察到调用返回 |
|
||||
| 同实例重开回读 | 每工程完成内容核验后才发布 saved_ok,随后关闭工程 |
|
||||
| CAD 自然退出 | 本轮五个 CAD 实例(前读 2、执行 1、后读 2)主进程均正常退出码 0,无 CAD 主进程 terminate_request |
|
||||
| CAD 退出等待 | Quit helper 成功后的主进程等待约 0.234–0.390 秒,不是整个 Quit 的总耗时 |
|
||||
| 退出后的独立回读 | 每工程用新 CAD 实例读取;57 器件、103 引脚连接,56 个非目标器件不变,目标身份、位置、连接和目标料号正确 |
|
||||
| 原件 | 两个源工程目录哈希不变,中央库 `.lmc` 哈希不变;未宣称整个库目录全部文件无变化 |
|
||||
| 前台与资源 | 观察器未发现自有 CAD 进入输入桌面或成为前台,已记录 CAD 退出和桌面消失 |
|
||||
|
||||
这轮运行的是源码服务层、真实 CAD 和加入边界日志的 VBS 副本;没有修改生产源码。不是完整冻结 GUI/监督进程验收,时序日志可能增加少量开销,耗时不能作为性能承诺。
|
||||
|
||||
第一轮记录 `live-chain-366340e7a6a942ccadaf2e1eedd9c4f4` 因探针复制 VBS 时引入 UTF-8 BOM,导致第 1 行编译失败,替换前停止。该失败属于探针,不是产品故障;保留原记录和阻断状态,修正编码后使用另一组全新副本,未重放该次失败任务。
|
||||
|
||||
## 二、已验证问题与含义
|
||||
|
||||
### A. 辅助进程尚未自然退出,就调用 TerminateJobObject
|
||||
|
||||
位置:`isolated_desktop.py:380` 的 `_Win32Process.close`。处理顺序为:确认/停止根进程 → 收集后代 → **TerminateJobObject** → 最多等待 5 秒确认进程退出。因此这 5 秒是强制终止后的确认窗口,不是辅助进程正常收尾的宽限期。
|
||||
|
||||
两工程实机中五次 CAD 主进程退出后,Job 都仍有一个成员。原时间线的映像查询使用了本机 pywin32 未提供的函数,查询错误已保留;不能凭 PID 将五个成员全部命名。独立 CIM 快照将最后一次回读的 CAD PID 55500、子进程 PID 45368 对应到实际 `vnsd.exe`。
|
||||
|
||||
随后单独进行空 CAD 实验,改用 Win32 `QueryFullProcessImageNameW` 确认成员身份。仅推迟探针中对已正常退出 CAD 的 Job 清理,最长观察预算 75 秒,未打开任何工程:
|
||||
|
||||
- CAD PID 60332 正常退出码 0。
|
||||
- 此时 `vnsd.exe` PID 34960 仍存活。
|
||||
- 约 **0.813 秒**后该辅助进程自行退出,随后原清理函数运行时已没有需终止的后代。
|
||||
- 最终自有资源全部释放,桌面检查耗时 0 秒,整个关闭阶段约 2.625 秒。
|
||||
|
||||
证据:`build/day0-resource-analysis-20261026/empty-cad-descendants-9fd000b5f0af48dc82e12b6b034d2917/report.json`;探针 `probe_descendant_exit.py` 位于同级分析目录。
|
||||
|
||||
既有原生进程研究将 `vnsd.exe` 识别为许可证辅助程序,另有工程数据库 `iCDBNetServer.exe` 和交叉探查 `xprobSvr.exe`。本次空 CAD 仅验证 vnsd 的自然退出,不证明数据库提交被终止,也不证明它就是工单现场残留桌面的持有者。
|
||||
|
||||
建议方向:对已经正常退出的根进程,其明确自有的后代先进行有界自然退出等待;到期才进入可审计的强制清理。区分根进程、数据库辅助进程、许可证辅助进程;记录终止对象、原因及发生时的业务阶段。不能只把等待常量变大,也不能对不明进程进行清理。
|
||||
|
||||
### B. 收尾记录的 I/O 错误会改变应用退出方式
|
||||
|
||||
位置:`isolated_cad.py:207`。正常 Quit 前的 `_record(state='closing')` 与 Quit 在同一个 try 内,记录失败会直接跳入异常处理,接着 finally 仍关闭自有资源。
|
||||
|
||||
故障注入:仅让这一次 closing 记录写入抛 PermissionError,其他记录正常。结果为:没有发出 Quit;写入 uncertain;调用资源 close。真实 CAD 由替身代替,因此本项证明控制流缺陷,没有实际杀死用户 CAD。
|
||||
|
||||
含义:目录权限、同步软件/安全软件的文件访问冲突、磁盘问题等不只影响日志,还可能改变应用本来可以正常退出的路径。上述环境来源只是候选,没有确定具体软件责任。
|
||||
|
||||
建议方向:隔离诊断/资源收尾记录错误与正常退出协议;写前业务保护失败仍应禁止写入,但不能混同为“所有诊断写入失败都必须跳过 Quit”。退出请求仍须满足实例身份、工程状态等原有条件。
|
||||
|
||||
### C. 保存状态在工程关闭成功之后才记录
|
||||
|
||||
位置:`QuickPartReplace.vbs:934`。调用 `QPCloseBatchProject(TargetPath, True)` 返回后才执行 `QP_BatchSaveCommitted=True`;这个子程序内部既保存图页,又关闭工程。
|
||||
|
||||
用真实 `QPFinishBatchItem` / `QPCloseBatchProject` 函数与 CAD 形状替身运行:图页的 SaveChanges=True 调用已经返回,随后工程拒绝关闭。观察到 `save_called=True`、`commit_flag=0`、错误已抛出,没有发布成功 checkpoint。
|
||||
|
||||
因此,这个标记不能独立表达“保存已发出/可能完成,而工程关闭失败”。某些后续错误文案依赖它,可能把已涉及保存的故障描述为未保存。`failed_native_result_uncertain` 仍会依据原生修改证据阻断重试,本轮没有发现借此自动重放替换;问题主要在恢复判断所需的状态粒度和事实描述。
|
||||
|
||||
建议方向:拆分 save_requested、save_returned、saved_readback_verified、project_closed 等证据;保存请求开始后发生异常应保留“可能已保存”的状态,不能因 CloseProject 失败回退成“未保存”。
|
||||
|
||||
### D. 资源消失后,正常退出异常不再向调用方抛出
|
||||
|
||||
位置:`batch_supervisor.py:159` 的 `_finish_response`,及 `batch_resources.py:117` 的重查逻辑。
|
||||
|
||||
用生产 BatchCadRuntime 创建合法资源记录,替身 CAD 在退出时抛出 Quit 超时,已保存业务结果独立保留;模拟拥有资源的监督进程结束后,再调用生产资源复核及返回函数。结果:
|
||||
|
||||
- 会话先 blocked;物理桌面已不存在后变为 released。
|
||||
- `cad_shutdown.clean=false` 和 `orderly_shutdown` 异常仍保留在 JSON。
|
||||
- 监督记录 `state=finished`、`outcome=error`。
|
||||
- 调用方收到含 completed 条目的 TaskQueueResult,没有收到退出异常。
|
||||
|
||||
这是状态返回语义不一致,不是原始错误日志被删除,也不证明已独立核验的保存结果应当被作废。应保留正确业务结果,同时显式暴露“业务完成、应用异常退出/强制清理”的收尾状态,不能仅凭物理资源消失将异常归为恢复成功。
|
||||
|
||||
上述 B/C/D 复现实物:`build/day0-resource-analysis-20261026/fault-chains-907a534cc6bb441e8cddf7670dd04aac/`,脚本 `probe_fault_chains.py`。较早的临时探针曾因日志文件句柄尚未关闭而无法删除临时目录;后续改为保留独立证据目录并关闭日志句柄,此探针清理错误不计入产品结论。
|
||||
|
||||
## 三、仍需验证的高影响风险
|
||||
|
||||
| 风险 | 已知机制 | 尚不能断言的部分 |
|
||||
| --- | --- | --- |
|
||||
| 慢保存被当成无响应 | 保存前仅发一次阶段进度;默认 150 秒请求停止、300 秒判无响应,safe_stop=False 不阻止监督器升级。保存没有类似原生快照的进度旁路 | 没有真实慢保存达到 300 秒的实验,不证明现场曾在保存中被终止 |
|
||||
| Quit/关闭响应延迟 | Quit helper 20 秒;成功后 CAD 15 秒;checkpoint 20 秒;项目会话关闭另有 60 秒宽限 | 本次正常样例均未超过这些时限,不能据此证明大工程也安全或必须统一加长 |
|
||||
| 同实例回读可能不足以覆盖持久性 | 生产路径在同一个 CAD 实例中重开回读 | 本次新实例读回通过,未发现缓存掩盖丢失;仍缺断电、数据库服务异常和大型工程覆盖 |
|
||||
| 关闭工程返回值被忽略 | 本机 TypeLib 表明 CloseProject 返回 Boolean;VBS 常直接调用并用工程路径为空作为补充判断 | 本次调用均为 True,尚未观察到 False 且空路径的真实异常,不能将其直接确定为事故根因 |
|
||||
| GUI/监督进程异常终止 | Job 关闭会清理受控进程,不能保证正在进行的保存完成 | 正常点击关闭时 GUI 会等待当前 worker,不能把所有 GUI 关闭行为描述成直接杀进程。崩溃/强制结束另需故障实验 |
|
||||
| 数据库和跨进程协作收尾 | 既有研究存在 iCDBNetServer、xprobSvr 等进程;仅确认 viewdraw 退出不足以描述全部应用状态 | 本轮没有捕获这些服务器在未提交数据时被终止;不能把 vnsd 的观察套用成数据库丢失证据 |
|
||||
|
||||
## 四、厂商接口核对的重要限制
|
||||
|
||||
实际安装的 7.9.5 `viewdraw.tlb` 已离线核对,结果保存在 `cad-lifecycle-typelib.json`:
|
||||
|
||||
- `IVdApp.CloseProject` 返回 Boolean。
|
||||
- `IVdApp.Quit`、`IVdSchematicSheetDocument.Close(SaveChanges, FileName)` 返回 void,不能虚构它们有保存成功布尔返回值。
|
||||
- `IVdDoc` 有 Saved 属性,但不能据此假定当前 `IVdSchematicSheetDocument` 也有同样的属性契约。
|
||||
- 事件接口包含 BeforeDocumentSave、AfterDocumentSave、ProjectClosed。
|
||||
|
||||
真实 VBS 日志已订阅上述事件:本轮 ProjectClosed 被正常观测,但在两次 SaveChanges=True 路径中**没有观测到 BeforeDocumentSave/AfterDocumentSave**。因此不能因为类型库列出了保存事件,就把等待该事件直接设为当前原理图保存路径的必经条件。需要核实它适用的文档类型、接口和触发条件。
|
||||
|
||||
本轮读取的安装帮助未取得足以证明上述调用异步/同步持久化语义的详细契约。类型库提供方法签名,不能替代持久化保证。
|
||||
|
||||
## 五、验证与修改范围
|
||||
|
||||
本轮补充回归:
|
||||
|
||||
```text
|
||||
python -m pytest -q tests/integration/test_quick_value_modes.py tests/v1_1/test_project_session.py tests/v1_1/test_failed_native_recovery_guard.py tests/v1_1/test_supervisor_result.py tests/v1_1/test_ui_operation_lifecycle.py
|
||||
94 passed in 34.11s
|
||||
```
|
||||
|
||||
上一轮已执行 IsolatedCad 和 worker monitor 的 60 项验证,本轮不重复当成新结果。测试通过说明已有行为与保护仍按测试约定运行,不证明上述异常路径合理。
|
||||
|
||||
仅增加分析报告和 `build/day0-resource-analysis-20261026/` 下的诊断脚本/证据。生产 Python、正式 VBS、版本号及发布包未修改,未向远端工单发消息或关闭工单。
|
||||
|
||||
下一轮修改应优先针对已复现的四个问题,并保留原有身份校验、写前保护和禁止不确定重放。验收必须分别报告数据结果、工程关闭结果、应用退出方式、后代进程退出方式和最终资源状态,不能汇成一个“释放成功”。
|
||||
@@ -0,0 +1,140 @@
|
||||
# Day0:资源释放工单原因分析
|
||||
|
||||
日期:2026-10-26(北京时间)。严重级别依据用户本次升级说明;未修改远端工单标签或状态。分析源码基线:`b9391015650bf5c43a77a017c5484e9a8f352266`。
|
||||
|
||||
后续已完成[核心应用生命周期专项排查](2026-10-26-application-lifecycle-investigation.md):包含两工程真实保存与独立读回、vnsd 自然退出对照及三类异常控制流复现。后续结论以该记录注明的证据和边界为准。
|
||||
|
||||
## 判断
|
||||
|
||||
工单 #1 的现场根因仍未闭环。根据用户对核心应用生命周期的补充,排查主线调整为“业务操作完成 → 保存确认 → 工程关闭确认 → 应用正常退出 → 进程及辅助资源释放 → 桌面回收”。桌面是隐藏操作的基础设施,其消失不能替代业务完成或进程正常退出证明。第一轮对桌面末端症状的侧重不足以覆盖保存与异常退出风险。
|
||||
|
||||
对已经观察到的桌面残留,需要继续追查自有清理范围以外的引用,包括本程序未登记的额外引用、无窗口线程、其他进程持有的桌面句柄和由系统或服务创建的辅助进程。这里的“范围以外”不等于已经证明是第三方软件造成,也不能将其提前确定为整个事故的业务根因。
|
||||
|
||||
工单 #3 在 CAD 未启动时出现同类残留,说明 CAD Quit 或 COM 释放不足不能解释全部现场现象。两个工单具有相同末端症状,但尚不能断言具体持有者或底层根因相同。
|
||||
|
||||
Beta 3 的进程隔离、批次复用和资源复核覆盖了部分故障机制。产品仍缺少现场持有者的完整识别能力,缺少对应现场版本的复测成功回执,不能凭本机通过将事故关闭。
|
||||
|
||||
## 补充核查:核心应用完成和退出链路
|
||||
|
||||
本节优先于后文仅针对桌面残留的原因排序。关闭桌面句柄不会自动结束进程;当前实现实际先清理受控进程、再关闭桌面句柄。应分别证明数据保存成功、工程关闭成功、应用正常退出、强制终止是否发生和资源是否消失。
|
||||
|
||||
### 正常路径已有保存回读,但完成语义仍需核实
|
||||
|
||||
- `QuickPartReplace.vbs:863` 的 `QPCloseBatchProject` 调用 `Document.Close(True, "")` 保存关闭图页,随后调用 `CloseProject`,立即查询工程路径为空。
|
||||
- `QuickPartReplace.vbs:934` 起会重新打开同一工程、校验目标及非目标内容,之后才给出 `saved_ok`。不能把现有程序描述为“完全没有等待或没有保存核验”。
|
||||
- COM 调用返回是现有的等待边界,但当前函数没有单独的保存完成事件、持久化提交状态或忙碌状态等待。是否需要增加这些机制,必须核对该版本 CAD 的 API 契约及真实行为;不能仅因缺少 sleep 就断言保存是异步或保存未完成。
|
||||
- 生产路径是在同一个 CAD 实例中重开工程进行回读。这提供了内容核验,但若要证明进程退出后的持久性,应增加正常退出后的独立新实例读取实验,并核查实际数据库服务的提交语义。不能仅凭同实例回读断言存在缓存问题,也不能将开发验收中的独立回读当成每次生产执行都有此保障。
|
||||
- 图页/工程关闭后立即查询一次状态,没有针对“调用已返回、关闭状态稍后才就绪”的有界状态等待。若厂商允许这种延迟,可能把仍在正常收尾的应用判为失败;目前还没有确认其 API 是这种行为。
|
||||
|
||||
### 明确存在的风险路径:应用未确认正常退出时进入强制清理
|
||||
|
||||
`isolated_cad.py:207` 的退出流程为:Quit helper 最多等待 20 秒;helper 成功才进一步等待 CAD 进程最多 15 秒。Quit helper 超时或失败时会跳过这次 CAD 正常退出等待,记录 uncertain,然后仍执行 `desktop.close()`。在批次模式中这里首先是 lease 的进程清理,随后才是批次拥有者的桌面关闭;底层 `_Win32Process.close()` 对存活自有进程/Job 有终止逻辑。
|
||||
|
||||
因此存在需要验证的链路:应用仍忙或关闭被拒绝 → 固定时限/单次状态检查失败 → 保护标记 → 强制终止自有进程 → 后续桌面释放检查。保护标记能避免无依据重放,不能保证被终止前未完成的保存可以安全完成。强制终止后的“资源清理成功”也不能用作“业务正常完成”的证据。
|
||||
|
||||
Quit helper 在发现工程仍打开时会拒绝退出,避免隐式保存或丢弃;但外层依然会进入自有进程清理兜底,所以必须检查当时是否有未完成的写操作。业务状态、退出方式和资源状态需要分开判断。
|
||||
|
||||
### 保存期间的无进度监控需要专项验证
|
||||
|
||||
`QuickPartReplace.vbs:874` 在进入保存关闭前发出一次 `save_and_close` 进度。若后续阻塞在 COM 调用中,脚本不能继续发送阶段进度。当前原生侧进度旁路只处理 `native_snapshot_read`,没有相同的保存进度旁路。
|
||||
|
||||
默认无进度时限为 150 秒。`ProgressWatch.action()` 在 150 秒请求停止、300 秒判定 unresponsive;这个判定不因 `safe_stop=False` 而停止升级。VBS 会在安全边界才响应协作停止,但监督器仍可在后续超时中终止 worker,并进入 CAD 收尾。真实慢保存是否能达到这个条件尚未实测;普通进程存活或无条件心跳也不能当成保存正在推进的证据。
|
||||
|
||||
### 本轮验证与证据边界
|
||||
|
||||
- `python -m pytest -q tests/v1_1/test_isolated_cad.py tests/trial_task_center/test_worker_monitor.py`:**60 passed,16.22 秒**。包括控制流替身和真实辅助进程测试,没有真实 CAD 慢保存实验。
|
||||
- 对生产 `IsolatedCadSession` 注入模拟 Quit helper 超时:确认等待参数为 20 秒、CAD wait 未被调用、uncertain 标记已写入、之后进入资源 close。未终止真实 CAD。
|
||||
- 对生产 `ProgressWatch` 输入模拟 `save_and_close / safe_stop=False`:150 秒为 request_stop,300 秒为 unresponsive。
|
||||
- 证据:`build/day0-resource-analysis-20261026/application-lifecycle-probe.json`。这是控制流证据,不能写成现场已发生保存中断或数据丢失。
|
||||
- 原始 #1 是库存读取后的收尾失败,四次日志记录 Quit helper/CAD 正常退出;#3 尚未启动 CAD。当前附件没有证明替换保存被提前打断。上述生命周期风险必须调查,但不能直接取代这两组现场证据。
|
||||
|
||||
优先实验应改为:在工程副本上记录保存调用前后、关闭调用前后、回读结果、Quit 请求与返回、CAD 自然退出、强制终止请求及原因的完整时间线;分别验证正常保存、长时间忙碌、关闭延迟和退出拒绝。数据正确性及退出契约查清后,再处理桌面末端残留。无需通过开启可见桌面或无界等待来替代这些完成条件。
|
||||
|
||||
## 已核实的证据
|
||||
|
||||
### 工单 #1:V1.1.3,四次资源释放确认失败
|
||||
|
||||
原件位置:`build/issue-1-20261021/evidence/`。四份失败的 `private-desktop.json` 一致记录:
|
||||
|
||||
- CAD 和退出辅助程序的退出码均为 0。
|
||||
- 所记录进程的 `job_active_processes=0`,已知后代句柄均已确认退出,自有进程及 Job 句柄关闭。
|
||||
- `desktop_handle_closed=true`,但 5 秒后 `released=false`。
|
||||
- 四次清理来自同一个主程序 PID 14608,调用线程不完全相同,调用线程桌面为 `Default`。
|
||||
- 另有一份采集中途取消的记录停在 `ready`,没有最终收尾证据;不能算作释放成功,也不能直接算作第五次同因泄漏。
|
||||
|
||||
这些证据降低了“已登记 CAD 主进程没有退出”“完全没等待终止”“已登记桌面句柄没有关闭”的优先级。它们不能证明进程所有线程、所有其他桌面句柄或所有系统辅助进程都已释放。
|
||||
|
||||
### 工单 #3:Beta 2,CAD 未启动也残留
|
||||
|
||||
本地 `build/issue-3-20261023/application.log` 与本次重新读取的远端附件逐字节一致。SHA256:`28d164039831973b7f6596a081d305421131f883514feae2fd652ff90ec54241`。
|
||||
|
||||
| 北京时间 | 日志位置 | 事实 |
|
||||
| --- | --- | --- |
|
||||
| 10-23 14:29:13 | 第 2 行 | 运行版本 `1.1.4-beta.2`,Windows 基础 build 26200 |
|
||||
| 14:29:34 | 第 14 行 | 开始整批 BOM 检查 |
|
||||
| 14:30:01 | 第 669 行 | 开始批末释放 |
|
||||
| 14:30:06 | 第 1098 行 | 桌面句柄关闭,桌面仍存在;CAD PID 为空,显示 CAD 未启动;窗口列表为空 |
|
||||
| 同一次失败的异常链 | 第 1102、1116 行及后续堆栈 | 首发错误为内部 attempt 目录创建的 `FileNotFoundError / WinError 3`,失败路径长 265 字符;释放错误发生在异常收尾期间 |
|
||||
| 14:30:47 | 第 1155 行 | 用户请求重新清理资源 |
|
||||
| 14:30:54 | 第 2013 行 | 同一个桌面仍存在,句柄关闭、CAD 未启动、窗口列表为空 |
|
||||
|
||||
前后失败样本相隔约 48 秒。未连续观测期间的状态,但相同随机桌面名两次均被枚举,明显削弱了“只需略微增加 5 秒超时”的解释。
|
||||
|
||||
`cleanup.processes=[]` 仅表示该次最终清理列表为空。原生 CLI 在批内执行后会从活动列表移除,不能据此说整批从未启动过子进程。
|
||||
|
||||
长路径是已记录的首发业务错误;桌面残留是收尾时另一个故障。修复路径问题不自动证明桌面引用来源已修复。
|
||||
|
||||
## 桌面残留的原因排序和判别办法
|
||||
|
||||
“优先级”表示调查顺序,不表示统计概率。
|
||||
|
||||
| 原因候选 | 优先级及证据 | 如何区分 |
|
||||
| --- | --- | --- |
|
||||
| 本程序的其他线程或额外桌面句柄仍持有引用 | 高;旧版 GUI 生命周期长,关闭登记句柄不能排除额外引用。四次 #1 来自同一主程序 | 现场保留桌面名,由独立观察进程在监督进程退出前后、主程序正常退出前后枚举;配合句柄和线程归属快照。随某进程退出而消失可缩小范围,但仍需持有者证据,不能单凭时间相关性定责 |
|
||||
| Job 外的无窗口辅助线程/进程 | 高;完全符合“Job 清空但桌面存在”。历史受控实验发现过无窗口 Python 和 conhost 线程 | 对目标桌面扫描全局线程,记录 PID/TID、映像、创建时间、会话及 Job 归属;与原生 CLI 启停时间对齐。历史 conhost 只是实验线索,现场未确认 |
|
||||
| 其他进程仅持有 OpenDesktop 句柄 | 高;可在不启动 CAD、没有窗口和目标线程的情况下复现相同症状 | 在线程扫描之外查询 Desktop 类型句柄。已有外部句柄实验复现阻断,持有者释放后可恢复;具体现场持有者未知 |
|
||||
| 原生 CLI、运行库、服务代理触发额外引用 | 中高;#3 没启动 CAD,但此前走过原生读取链 | 在现场副本依次运行空桌面、简单子进程、单个 icdb2ascii 读取、多个读取、CAD 回退,找出首次出现残留的阶段。普通子进程默认受 Job 约束,不能笼统断言所有后代都逃逸 |
|
||||
| 安全软件、录屏、远程协助、桌面管理组件等参与桌面访问 | 中,属于上一类的来源候选;当前没有具体软件或 PID 证据 | 先采集持有者或模块证据,再做单因素环境对照;没有证据时不能指定某个软件为元凶 |
|
||||
| Windows USER 引用延迟回收/系统环境差异 | 中低;短暂延迟有受控实验支持,但现场约 48 秒后的重查仍失败 | 使用独立进程记录 0、1、5、15、60 秒存在性,同时记录所有者退出状态。本机和 #3 均为基础 build 26200,仍需对照累计更新、运行库和注入组件,不能只按系统大版本归因 |
|
||||
| 句柄继承、Job 接管或异常清理分支缺陷 | 中低,需保留;当前代码在恢复运行前分配 Job,重定向使用明确的继承句柄清单,已有后代退出和句柄关闭失败测试 | 检查实际运行 EXE 哈希、Job 分配结果、句柄继承标志、失败时保留的所有权记录。当前源码机制不能倒推所有旧版或现场程序完全相同 |
|
||||
| 历史资源记录损坏或残留,形成逻辑阻断 | 对“升级后仍不能用”须优先区分;代码会保守阻断坏记录 | 对比当前物理桌面/进程与 `session.json`、`supervisor.json` 的状态。若物理对象已消失而记录无法校验,是证据恢复问题;不应归为仍有物理泄漏。原始 #1/#3 明确枚举到桌面,坏记录不能解释其最初残留 |
|
||||
| CAD COM 引用未释放 | 对 CAD 路径仍可查;作为统一解释优先级低 | 仅在已启动 CAD 的场景核对 COM、helper 与 CAD 生命周期。#3 的 CAD 未启动是其无法解释全部现象的直接证据 |
|
||||
| 工程数量上限、桌面堆耗尽、纯 BOM 匹配错误 | 当前证据弱 | 查桌面创建失败、内存资源错误和随批量增长的资源曲线。当前主要失败在销毁确认;BOM 行不匹配是独立业务结果,不能据此改资源配额或放宽匹配 |
|
||||
|
||||
微软文档确认:关闭一个桌面句柄与销毁所有引用不是一回事;线程的桌面连接不依赖存在可见窗口;Job 默认覆盖普通 CreateProcess 子进程,但不是整个系统的进程集合。例如 WMI `Win32_Process.Create` 创建的进程不属于这类默认继承范围。此机制说明需要查 Job 外所有者,不代表本软件已经被证明使用了 WMI 启动逃逸进程。
|
||||
|
||||
## 当前实现为何仍可能卡住
|
||||
|
||||
1. **判定严格,但归属诊断不足。** `isolated_desktop.py:66` 通过枚举确认桌面是否消失;失败时只收集窗口。`isolated_desktop.py:565` 明确将外部句柄持有者标为 unknown。空窗口不是无引用证明。独立工具 `tools/diagnose_private_desktop.py` 能查无窗口线程,但不能列出所有仅持有句柄的进程,权限错误也必须保留。
|
||||
2. **监督进程隔离只解决其覆盖的所有权。** `batch_supervisor.py:279` 等待并关闭受控 Job,随后 `_finish_response` 复核资源。这能够处理监督进程自身存活期间的引用;其他长寿命进程的引用不会因该监督进程退出而消失。
|
||||
3. **重新清理有能力边界。** `batch_resources.py:117` 对仍有 live owner 的会话可重试其关闭;没有 live owner 时只确认桌面消失和历史 PID 身份,不会接管或终止不明持有者。Beta 3 常见的是监督进程已经退出后的复核分支。反复点击不会凭空释放外部引用。
|
||||
4. **未释放门禁扩大可用性影响。** `guard_pending_resources` 阻止同一数据根的新批次。这一保护避免在旧资源不明时继续创建桌面或重放业务,却使一个释放问题阻断后续使用。恢复机制和持有者识别必须共同补齐。
|
||||
5. **损坏记录不会自动修复。** 无法校验的记录保持 blocked;无 session_id 时跳过自动资源操作。这是证据不足时的保护行为,但意味着资源已消失仍可能需要审计式人工核对。不能通过删除保护文件掩盖该状态。
|
||||
6. **诊断包不等于实时取证。** 当前导出函数主要打包已有文本,未自动执行全局线程或句柄扫描。上限为单文件 4 MiB、总计 32 MiB、512 个文件,超限写入 omissions;必须检查 manifest,不能把未导出的记录当成从未发生。
|
||||
|
||||
## 本次新验证
|
||||
|
||||
运行环境:Python 3.12.10、Windows 11 基础 build 26200。以下测试未打开或替换用户工程。
|
||||
|
||||
- `python -m pytest -q tests/v1_1/test_isolated_desktop.py tests/v1_1/test_desktop_diagnostics.py tests/v1_1/test_batch_runtime_windows.py tests/v1_1/test_batch_supervisor.py`:**44 passed,87.89 秒**。包含真实 Windows 资源测试与替身测试,覆盖无窗口引用、延迟释放、后代进程、诊断边界、监督恢复等;不等同于现场完整 EXE 验收。
|
||||
- `python -m pytest -q tests/v1_1/test_batch_runtime.py::test_unknown_resource_evidence_never_authorizes_cleanup tests/v1_1/test_batch_runtime.py::test_restart_reconciles_only_confirmed_disappearance_without_replacement`:**4 passed,1.22 秒**。确认坏证据继续阻断、重启后仅复核的代码行为;这两类为模拟边界测试。
|
||||
- 资源循环探针 `build/day0-resource-analysis-20261026/probe_cycles.py`:20 次空桌面、20 次带重定向的简单 Python 子进程,**40/40 正常释放**。自有桌面残留 0,观察前后没有新增桌面,调用线程桌面未改变。两组的释放枚举检查最大耗时均约 0.016 秒,该数字不包含完整子进程启动耗时。
|
||||
- 循环证据:`build/day0-resource-analysis-20261026/cycles-6819874fa1b04649a6dbb9c6b1992034.json`。它是源码 Python 基础路径实验,没有覆盖 Qt GUI、冻结 EXE、实际 icdb2ascii、CAD 或现场常驻组件,不排除这些组合中的问题。
|
||||
|
||||
这批结果降低了本机基础资源封装普遍失效的可能性,不能证明现场根因已修复,也不能证明 102 个原工程全部可用。
|
||||
|
||||
## 下一步应取得什么证据
|
||||
|
||||
1. **先确认事故实际版本和状态。** 当前收到的两组原件属于 V1.1.3 与 Beta 2;没有 Beta 3 的新失败诊断包。记录现场 EXE 哈希、是否完整配套 `_internal`、原始 data 是否迁移、故障是再次产生桌面还是旧记录阻断。Beta 3 公开 EXE 的历史记录哈希为 `df08cd328f6f4c9019f232c845e09627551e7b2d24790a466d53efcd291622d1`,以实际安装文件核对。
|
||||
2. **残留发生时立即取证。** 在原机器、原登录会话,从 session 记录取得准确桌面名;运行独立线程诊断并查询 Desktop 句柄持有者,保留 PID、创建时间、映像、线程、会话、Job 和时间戳。仅复制旧日志到开发机无法找回消失的 Windows 引用。
|
||||
3. **独立观察所有者退出前后。** 先保存结果及诊断,再观察监督进程退出、应用正常退出后的同名桌面变化。桌面随监督进程退出消失,优先追监督进程内部引用;随 GUI 退出消失,追 GUI 或其关联组件;GUI 退出后仍存在,追其他进程或系统引用。上述变化用于缩小范围,最终需结合持有者证据确认因果。
|
||||
4. **按最小场景逐层复现。** 空桌面 → 简单 worker → 单次原生 CLI → 多次原生 CLI → CAD 回退 → 批量混合。每层记录创建数量、已退出进程、最终桌面状态。实际读取使用工程副本,首次失败就保留证据,避免反复业务执行污染现场。
|
||||
5. **修复后验证恢复结果。** 对已识别的持有者修复生命周期;完成正常、异常、取消、重启和跨批次检查,确认只清理资源时业务结果不变、没有重复替换,随后在现场原目录和原批量范围验收。只有现场复测通过并留存回执,才具备关闭工单的依据。
|
||||
|
||||
技术交付首先补齐业务完成与应用退出的分阶段证据,并专项验证等待与强制清理边界;对完成业务后仍残留的桌面,再补自动捕获持有者的能力。单纯延长等待、反复要求重新清理或继续发布未取得现场证据的候选包,都不足以建立闭环。
|
||||
|
||||
## 关联记录
|
||||
|
||||
- [工单 #1](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/1)、[工单 #3](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/3)。
|
||||
- [首轮工单分析](2026-10-21-issue-1-bom-import.md)、[Windows 生命周期受控实验](2026-10-21-windows-desktop-lifecycle.md)、[外部句柄持有者实验](2026-10-23-issue-3-external-holder.md)、[Beta 3 验收边界](2026-10-23-issue-3-safe-recovery.md)。
|
||||
- 微软:[CloseDesktop](https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-closedesktop)、[线程连接桌面的规则](https://learn.microsoft.com/en-us/windows/win32/winstation/thread-connection-to-a-desktop)、[Job Objects](https://learn.microsoft.com/en-us/windows/win32/procthread/job-objects)。本次重新核对官方页面。
|
||||
@@ -0,0 +1,59 @@
|
||||
# 用户停止与安全收尾实施验收
|
||||
|
||||
日期:2026-10-26;分支:`codex/day0-safe-stop`。实现用户在本会话批准的[停止流程](../superpowers/specs/2026-10-26-safe-user-stop.md),对应[全链路调查](2026-10-26-application-lifecycle-investigation.md)。
|
||||
|
||||
## 实现行为
|
||||
|
||||
- 普通暂停和关闭窗口不启动下一项,当前写入、保存和回读安全完成后停止。当前项并不等于整个工程。
|
||||
- 有写入可能的阶段长期没有进度时显示等待,不自动把超时升级成杀进程。纯只读采集保留有界取消。
|
||||
- 写入失败、保存结果不确定或 Quit 被拒绝/超时且 CAD 仍存活时,交互执行保留应用与工程,等待处理,不自动保存或丢弃。
|
||||
- “强制结束…”独立确认;先持久化业务保护及批次中断证据,再终止明确自有的监督进程及 Job。证据写入或 fsync 失败时拒绝该次 force,界面保留重试能力。
|
||||
- 保存请求、保存调用返回、保存回读和工程关闭分别记录,工程关闭失败不会丢失此前保存边界。
|
||||
- 收尾记录写入失败不跳过满足身份检查的正常 Quit。主进程自然退出后给予自有后代 2 秒自然退出窗口,之后的强制清理单独记录。
|
||||
- 资源消失不吞掉异常退出;调用方收到独立收尾错误,已核验业务结果仍附带保留。
|
||||
- 等待期间可独立导出诊断包。关闭窗口在操作结束前不退出;失败/强制结束后取消自动关闭,保留恢复提示。
|
||||
|
||||
## 自动验证与审查
|
||||
|
||||
针对安全边界先运行失败用例再实现,包括慢保存不会被 watchdog 终止、明确 force 的保护先行、保护写入失败保留进程、已有文件也重新 fsync、保存后工程关闭拒绝、窗口关闭不触发 force、确认取消无副作用。
|
||||
|
||||
真实辅助进程测试覆盖监督子进程强制中断、控制回调异常时保持子进程存活、拒绝第一次保护写入后仍允许后续明确重试、资源清理不重放业务。CAD 调用在这些故障测试中使用替身,不能据此声称真实保存中断无损。
|
||||
|
||||
退出协议独立审查修复了部分子进程查询失败时丢失已知强制终止 PID 的问题。整体验证另外修复:只读入口没有 force 出口却进入无限等待、证据首次 fsync 失败后因文件存在被误放行、已执行 force 被诊断异常降级为普通停止、控制通信异常绕过确认清理、直接执行 force 失败没有反馈,以及 force 意图与监控异常并发的竞态。复审未发现新增阻断问题。
|
||||
|
||||
首次完整实现回归 **1063 passed,1 条既有 pywinauto STA 警告,363.82 秒**。之后完成审查修正,最终扩大回归 **1147 passed,1 条既有 pywinauto STA 警告,325.64 秒**:
|
||||
|
||||
```powershell
|
||||
python -m pytest -q tests/v1_1 tests/trial_task_center/test_application.py tests/trial_task_center/test_execution_safety.py tests/part_replace/test_verification.py tests/integration/test_quick_value_modes.py tests/integration/test_quick_session_startup.py tests/integration/test_quick_part_replace_contract.py tests/integration/test_automatic_replacement_regression.py tests/trial_task_center/test_worker_monitor.py tests/trial_task_center/test_batch.py
|
||||
```
|
||||
|
||||
本命令包含项目发布脚本要求的源码测试范围,并补充此次受影响的 VBS、worker monitor 与批次测试。`git diff --check` 与新探针/停止模块语法检查通过。
|
||||
|
||||
两个旧 VBS fixture 在基线 `b939101` 也失败:未填 owned project path 且错误期望 no-match 保存、图页清单断言未使用规范化路径。修正的是 fixture 和断言,没有放宽生产所有权或保存规则;脚本契约组 72 项通过。
|
||||
|
||||
## 真实保存期间请求普通停止
|
||||
|
||||
工具:[probe_safe_stop_live.py](../../tools/probe_safe_stop_live.py)。本轮输出为 `build/safe-stop-live-a9113545bf/`,报告 `report.json`,最终资源复核 `final-resource-check.json`。
|
||||
|
||||
入口为源码 supervised service → 真实独立 batch worker → 生产 VBS → 真实 CAD,使用两份全新 Audio_D2128/Audio_T605A 副本。监听第一工程的生产 `save_requested` 日志后只设置普通停止 Event,不设置 force。此触发处于保存请求附近,不能声称精确暂停在 COM 内某一条指令。
|
||||
|
||||
| 检查 | 结果 |
|
||||
| --- | --- |
|
||||
| 第一个工程 | 当前 RV05 替换、保存和回读完成,任务 completed |
|
||||
| 第二个工程 | 保持 pending,没有继续替换 |
|
||||
| 队列 | paused=true,未请求强制结束 |
|
||||
| 第一工程新 CAD 独立读回 | 57 器件、103 引脚连接;目标 RV05 为新料号,56 个非目标器件保持;目标身份、位置和连接保持 |
|
||||
| 第二工程新 CAD 独立读回 | 57 个器件的完整属性及连接图与执行前一致 |
|
||||
| 执行批次退出 | orderly_shutdown=true、clean=true、forced_termination=false |
|
||||
| 资源 | 资源记录全部 released,最终枚举确认其桌面消失,没有本轮残留 CAD |
|
||||
| 原件 | 两个源工程目录哈希不变,中央库 `.lmc` 哈希不变 |
|
||||
|
||||
没有重放任何已经完成的替换;测试用的 before/after 独立读取使用新 CAD 实例。
|
||||
|
||||
## 边界
|
||||
|
||||
本轮没有构建或发布新 EXE,未做完整冻结 GUI 的保存中强制结束、掉电或原工单全量现场复测。GUI 行为使用 Qt 测试验证,真实保存停止使用源码监督入口验证;二者不能混写为冻结 GUI 全流程验收。
|
||||
|
||||
强制结束为避免切换任务竞态,保守标记本批可能涉及的目标;已完成业务证据和结果保留,但先前已完成任务也可能需要恢复核验。不能清除保护来继续。
|
||||
|
||||
保持应用等待不代表保存结果一定可以恢复。外部任务管理器强杀/系统掉电没有安全收尾保证,重启必须依赖原先写前证据核验。工单现场具体桌面持有者仍未查明,本实现不能当作现场根因已经闭环。
|
||||
@@ -1,5 +1,7 @@
|
||||
# R8:进度式超时与分阶段引脚索引
|
||||
|
||||
> 本文保留 R8 历史行为。2026-10-26 起,有写入可能的阶段无进度超时改为等待安全收尾,只有明确确认的强制结束才中断;当前停止与恢复规则见[停止、强制结束与恢复](safe-stop-and-recovery.md)。只读采集继续采用有界取消。
|
||||
|
||||
## 改变的行为
|
||||
|
||||
- 150 秒不再是整个任务的总时限,而是连续没有有效进度的观察窗口。有有效进度时可以继续执行超过 150 秒。
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
# 停止、强制结束与恢复
|
||||
|
||||
执行期间点击“完成当前项后暂停”,程序不再启动下一替换项。已经进入修改或保存的当前项会继续完成必要校验、保存、重开回读和工程关闭,然后正常退出 CAD,并等待自有辅助进程收尾。关闭主窗口也遵循这个流程,操作仍在进行时窗口保持打开。
|
||||
|
||||
“当前项”是当前图页和替换规则对应的执行单元,不是整个工程或整个批次。已经保存核验的结果保留,未开始的任务保持未执行。
|
||||
|
||||
## 保存或修改长时间没有进度
|
||||
|
||||
有写入可能的阶段,程序不会仅因超过无进度时限就自动终止应用。界面显示保存或核验尚未确认,可以继续等待并导出诊断包。等待不会启动下一项,也不会自动清除保护记录。
|
||||
|
||||
如果操作已经失败且保存结果不确定,程序保留当前应用和工程现场,不再自动保存或丢弃。应先导出诊断、保留原始工程与备份,再决定是否需要强制结束。此时“继续等待”不表示自动恢复执行。
|
||||
|
||||
纯读取的 BOM/库存采集仍采用有界取消;不存在写入需要完成时,不要求用户无限等待。确认未修改的启动失败也保留既有有界停止机制。
|
||||
|
||||
## 明确选择强制结束
|
||||
|
||||
“强制结束…”与普通暂停分开。确认框明确提示:当前操作可能正在修改或保存,强制结束可能留下未保存或部分保存结果,之后必须先核验恢复。
|
||||
|
||||
确认后,程序先持久化中断保护和批次记录,再终止自己拥有的监督进程及 Job。保护记录无法可靠保存时,不执行这次强制结束,界面提示原因并允许重试或导出诊断。取消确认没有终止副作用。
|
||||
|
||||
为避免强制请求与工程切换同时发生而漏记目标,强制结束会保守保护本批可能涉及的任务。先前已保存结果不会被覆盖或重放,但这些任务也可能显示需复核或被保留在当前列表。不要删除保护文件绕过核验。
|
||||
|
||||
## 程序重启之后
|
||||
|
||||
- 资源复核只处理进程和桌面状态,不重新替换。
|
||||
- 结果不确定的任务必须使用“核验并恢复”,按照已有快照和当前实际工程核对。
|
||||
- 已保存核验结果继续保留;资源清理成功不自动解除业务保护,也不自动继续队列。
|
||||
- 如果应用没有正常退出或后代进程需要强制清理,会单独报告收尾异常,不能把“资源已释放”理解为“保存和退出均正常”。
|
||||
|
||||
任务管理器强杀、系统关机或掉电可能不给程序写最终日志的机会。这些情况依赖修改前已经落盘的证据及下次启动的中断核验;程序不能保证被外部强杀时完成当前保存。
|
||||
@@ -28,7 +28,7 @@ Dim QP_ComponentUidIndex, QP_ComponentRefdesIndex, QP_ComponentLocationIndex
|
||||
Dim QP_ComponentIndexContext
|
||||
Dim QP_NativeCallActive
|
||||
Dim QP_BatchMode, QP_OwnedProject, QP_BatchState, QP_BatchCode, QP_BatchMessage
|
||||
Dim QP_BatchSaveCommitted
|
||||
Dim QP_BatchSaveCommitted, QP_BatchSaveRequested
|
||||
Dim QP_DiscoveryMode, QP_TargetSchematic, QP_TargetSheet
|
||||
Dim QP_FailureNumber, QP_FailureSource, QP_FailureDescription
|
||||
Dim QP_NcSyncSummary
|
||||
@@ -60,6 +60,7 @@ QP_BatchMode = False
|
||||
QP_OwnedProject = False
|
||||
QP_BatchState = ""
|
||||
QP_BatchSaveCommitted = False
|
||||
QP_BatchSaveRequested = False
|
||||
QP_DiscoveryMode = False
|
||||
QP_TargetSchematic = ""
|
||||
QP_TargetSheet = ""
|
||||
@@ -110,6 +111,10 @@ Else
|
||||
QP_FailureDescription = Err.Description
|
||||
If QP_DiscoveryMode Then QPCleanupDiscoveryFailure
|
||||
FailureState = "hard_failure"
|
||||
If QP_BatchSaveRequested Then
|
||||
FailureState = "batch_saved_unverified"
|
||||
QP_FailureDescription = "save was requested; persisted outcome requires reconciliation. " & QP_FailureDescription
|
||||
End If
|
||||
CloseError = ""
|
||||
If QP_BatchMode Or QP_SessionMode Then
|
||||
QPDiscardFailedBatchProject CloseError
|
||||
@@ -318,6 +323,7 @@ Sub QPSessionNext()
|
||||
QP_BatchMode = True
|
||||
QP_BatchState = ""
|
||||
QP_BatchSaveCommitted = False
|
||||
QP_BatchSaveRequested = False
|
||||
QP_NativeMutationPossible = False
|
||||
If Trim(QPReadUtf8(QP_Fso.BuildPath(QP_Root, "batch.request"))) <> QP_ExpectedJobId Then _
|
||||
Err.Raise vbObjectError + 873, "QuickPartReplace.ProjectSession", "next batch authorization mismatch"
|
||||
@@ -872,12 +878,21 @@ Sub QPCloseBatchProject(ByVal ExpectedPath, ByVal SaveChanges)
|
||||
If Documents.Count <> 1 Then Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "open sheet set changed; no automatic close"
|
||||
Set Document = Documents.Item(1)
|
||||
QPProgress "save_and_close", 0, 1
|
||||
If SaveChanges Then
|
||||
QP_BatchSaveRequested = True
|
||||
QPLog QP_Root, "INFO", "save_requested job=" & QP_CurrentJobId
|
||||
End If
|
||||
Document.Close CBool(SaveChanges), ""
|
||||
If SaveChanges Then
|
||||
QP_BatchSaveCommitted = True
|
||||
QPLog QP_Root, "INFO", "save_returned job=" & QP_CurrentJobId
|
||||
End If
|
||||
If QPCanonicalPath(QPProjectPath()) <> QPCanonicalPath(ExpectedPath) Then _
|
||||
Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "project changed during sheet save"
|
||||
QP_App.CloseProject
|
||||
If Len(CStr(QP_App.GetProjectData().GetProjectFilePath())) > 0 Then _
|
||||
Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "project did not close; queue stopped"
|
||||
QPLog QP_Root, "INFO", "project_closed job=" & QP_CurrentJobId
|
||||
QP_OwnedProject = False
|
||||
QP_OwnedProjectPath = ""
|
||||
QPInvalidatePinIndex
|
||||
@@ -888,6 +903,10 @@ Sub QPDiscardFailedBatchProject(ByRef ErrorDescription)
|
||||
Dim ErrorNumber, ErrorSource, Description
|
||||
ErrorDescription = ""
|
||||
If (Not QP_BatchMode And Not QP_SessionMode) Or Not QP_OwnedProject Or QP_App Is Nothing Then Exit Sub
|
||||
If QP_NativeMutationPossible Or QP_BatchSaveRequested Or QP_BatchSaveCommitted Then
|
||||
ErrorDescription = "write or save outcome requires reconciliation; owned project retained without automatic save or discard"
|
||||
Exit Sub
|
||||
End If
|
||||
On Error Resume Next
|
||||
QPCloseBatchProject QP_OwnedProjectPath, False
|
||||
ErrorNumber = Err.Number
|
||||
@@ -932,7 +951,6 @@ Sub QPFinishBatchItem()
|
||||
If FinalState = "restored_ok" Then
|
||||
QPMarkBatchMutation Command
|
||||
QPCloseBatchProject TargetPath, True
|
||||
QP_BatchSaveCommitted = True
|
||||
ElseIf Not QP_SessionMode Then
|
||||
QPCloseBatchProject TargetPath, False
|
||||
End If
|
||||
@@ -952,6 +970,7 @@ Sub QPFinishBatchItem()
|
||||
QPVerifyRestoredPlans Command, Records, Plans, ValueStates
|
||||
If Not QP_SessionMode Then QPCloseBatchProject TargetPath, False
|
||||
FinalState = "saved_ok"
|
||||
QPLog QP_Root, "INFO", "saved_readback_verified job=" & QP_CurrentJobId
|
||||
If QP_SessionMode Then
|
||||
QP_BatchMessage = QP_BatchMessage & " Saved and reopened for readback; owned project checkpoint verified."
|
||||
Else
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# UTF-8
|
||||
VSVersionInfo(
|
||||
ffi=FixedFileInfo(
|
||||
filevers=(1, 1, 4, 3),
|
||||
prodvers=(1, 1, 4, 3),
|
||||
filevers=(1, 1, 4, 4),
|
||||
prodvers=(1, 1, 4, 4),
|
||||
mask=0x3f,
|
||||
flags=0x2,
|
||||
OS=0x40004,
|
||||
@@ -15,11 +15,11 @@ VSVersionInfo(
|
||||
StringTable('040904B0', [
|
||||
StringStruct('CompanyName', 'DxProductivity'),
|
||||
StringStruct('FileDescription', 'DxDesigner Part Replacement'),
|
||||
StringStruct('FileVersion', '1.1.4.3'),
|
||||
StringStruct('FileVersion', '1.1.4.4'),
|
||||
StringStruct('InternalName', 'DxPartReplace'),
|
||||
StringStruct('OriginalFilename', 'DxPartReplace.exe'),
|
||||
StringStruct('ProductName', 'DxDesigner Part Replacement'),
|
||||
StringStruct('ProductVersion', '1.1.4-beta.3')
|
||||
StringStruct('ProductVersion', '1.1.4-beta.4')
|
||||
])
|
||||
]),
|
||||
VarFileInfo([VarStruct('Translation', [1033, 1200])])
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "dx-productivity"
|
||||
version = "1.1.4b3"
|
||||
version = "1.1.4b4"
|
||||
requires-python = ">=3.12,<3.13"
|
||||
dependencies = ["PySide6>=6.7,<7", "pywin32>=306", "pywinauto>=0.6.9,<1", "openpyxl>=3.1,<4"]
|
||||
|
||||
|
||||
@@ -66,10 +66,11 @@ class _DesktopLease:
|
||||
for process in reversed(self.processes):
|
||||
try:
|
||||
self.owner._release_process(process, desktop=self.desktop, generation=self.generation)
|
||||
records.append(getattr(process, 'cleanup_record', {}))
|
||||
except Exception as error:
|
||||
errors.append(str(error))
|
||||
remaining.append(process)
|
||||
finally:
|
||||
records.append(getattr(process, 'cleanup_record', {}))
|
||||
self.processes = list(reversed(remaining))
|
||||
self.cleanup_record = {'scope': 'operation', 'processes': records,
|
||||
'workers_closed': not errors, 'desktop_retained_by_batch': True, 'errors': errors}
|
||||
@@ -99,6 +100,7 @@ class BatchCadRuntime:
|
||||
self._retry_counts = {}
|
||||
self._project_attempts = {}
|
||||
self._logging = None
|
||||
self._active_evidence = None
|
||||
|
||||
def bind_logs(self, context, task_ids):
|
||||
"""Keep every participating task's final release outcome visible."""
|
||||
@@ -116,6 +118,14 @@ class BatchCadRuntime:
|
||||
return self
|
||||
|
||||
def __exit__(self, error_type, error, traceback):
|
||||
if error is not None and self._active_evidence is not None:
|
||||
attempts = [item for values in self._project_attempts.values() for item in values]
|
||||
try:
|
||||
self._retain_uncertain(self._active_evidence, None, attempts, error)
|
||||
except WorkerMonitorError:
|
||||
if not getattr(getattr(self.stop_event, 'force_event', None), 'is_set', lambda: False)():
|
||||
raise
|
||||
# Explicit force has durable protection; cleanup is now allowed.
|
||||
try:
|
||||
self.close()
|
||||
except Exception as cleanup:
|
||||
@@ -190,7 +200,8 @@ class BatchCadRuntime:
|
||||
started = time.monotonic()
|
||||
self._save(cad_start_requested=True)
|
||||
cad = factory(self.path.parent / f'generation-{self.generation + (self.desktop is None)}',
|
||||
abort_event=abort_event, desktop=lease)
|
||||
abort_event=abort_event, desktop=lease,
|
||||
**({'stop_control': self.stop_event} if hasattr(self.stop_event, 'force_event') else {}))
|
||||
self.cad = cad # Retain cleanup ownership even if startup fails.
|
||||
try:
|
||||
cad.__enter__()
|
||||
@@ -247,6 +258,7 @@ class BatchCadRuntime:
|
||||
from .isolated_cad import PrivateCadStartupError, is_identity_conflict
|
||||
from .worker_monitor import run_monitored_process
|
||||
root = canonical_path(command[3])
|
||||
self._active_evidence = root
|
||||
while True:
|
||||
check_cancelled(abort_event or self.stop_event)
|
||||
attempt_id = uuid4().hex
|
||||
@@ -272,7 +284,11 @@ class BatchCadRuntime:
|
||||
def start_worker(launch, **options):
|
||||
return cad.start_worker(launch, extra_environment=environment, **options)
|
||||
result = run_monitored_process(command, timeout, abort_event=abort_event,
|
||||
cooperative_abort=cooperative_abort, process_factory=start_worker)
|
||||
cooperative_abort=cooperative_abort, process_factory=start_worker,
|
||||
**({'force_control': self.stop_event} if hasattr(self.stop_event, 'force_event') else {}))
|
||||
if result.returncode:
|
||||
self._retain_uncertain(root, project, expected,
|
||||
WorkerMonitorError(result.stderr or 'worker failed after possible mutation'))
|
||||
self._checkpoint(root, project, attempt_id)
|
||||
if result.returncode not in (2, 3):
|
||||
return result
|
||||
@@ -294,6 +310,7 @@ class BatchCadRuntime:
|
||||
and not (abort_event is not None and abort_event.is_set())
|
||||
and not (self.stop_event is not None and self.stop_event.is_set()))
|
||||
if not safe:
|
||||
self._retain_uncertain(root, project, expected, failure)
|
||||
raise failure
|
||||
key = str(project).casefold()
|
||||
self._retry_counts[key] = 1
|
||||
@@ -302,6 +319,13 @@ class BatchCadRuntime:
|
||||
self.close(rebuilding=True) # Prove disappearance before creating another.
|
||||
self.desktop = self.cad = None
|
||||
|
||||
def _retain_uncertain(self, root, project, expected, error):
|
||||
from .batch_attempt import can_retry_attempt
|
||||
from .stop_control import wait_for_resolution
|
||||
attempts = self._project_attempts.get(str(project).casefold(), expected)
|
||||
if attempts and not all(can_retry_attempt(path, record) for path, record in attempts):
|
||||
wait_for_resolution(self.stop_event, root, error)
|
||||
|
||||
def recover_native_failure(self, project, evidence, reason):
|
||||
"""One read-only recovery, based on durable source and worker proof."""
|
||||
from .batch_attempt import can_retry_attempt
|
||||
@@ -387,7 +411,8 @@ class BatchCadRuntime:
|
||||
cad_record = cad_record if isinstance(cad_record, dict) else {}
|
||||
self.record['cad_shutdown'] = {key: cad_record[key] for key in (
|
||||
'state', 'clean', 'cad_exit_code', 'quit_helper_exit_code', 'cleanup_errors',
|
||||
'shutdown_seconds') if key in cad_record}
|
||||
'shutdown_seconds', 'orderly_shutdown', 'diagnostic_errors', 'forced_termination',
|
||||
'forced_root_termination', 'forced_descendant_pids', 'termination_reason') if key in cad_record}
|
||||
if self.desktop is not None:
|
||||
try:
|
||||
self.desktop.close()
|
||||
|
||||
@@ -24,27 +24,31 @@ _WRITE_OPERATIONS = frozenset({'execute_task_queue', 'execute_quick_run_queue',
|
||||
|
||||
|
||||
def _mark_interrupted(root, journal, reason):
|
||||
from .batch_resources import write_record
|
||||
from .batch_resources import write_record, _valid_record
|
||||
from .stop_control import protect_interruption
|
||||
journal.update(state='blocked', outcome='interrupted', error=str(reason))
|
||||
if journal['operation'] in _WRITE_OPERATIONS:
|
||||
# A missing final result is not proof of no mutation. Publish the same
|
||||
# protection the existing recovery UI understands, without overwriting
|
||||
# any earlier, more specific marker.
|
||||
journal['business_uncertain'] = True
|
||||
for task_id in journal['task_ids']:
|
||||
task_ids = journal['task_ids']
|
||||
resource_path = Path(root) / 'batch-resources' / journal['batch_id'] / 'session.json'
|
||||
try:
|
||||
resource = json.loads(internal_path(resource_path).read_text(encoding='utf-8'))
|
||||
if (not journal.get('protect_all_targets') and _valid_record(resource, resource_path)
|
||||
and resource.get('active_task') in task_ids):
|
||||
task_ids = [resource['active_task']]
|
||||
except (OSError, ValueError, TypeError):
|
||||
pass # Without valid active ownership, protect every possible target.
|
||||
journal['protected_task_ids'] = list(task_ids)
|
||||
for task_id in task_ids:
|
||||
allowed = {canonical_path(root), canonical_path(Path(root) / 'tasks' / task_id)}
|
||||
actual = journal.get('uncertainty_roots', {}).get(task_id)
|
||||
roots = {canonical_path(actual)} if actual and canonical_path(actual) in allowed else allowed
|
||||
for evidence in roots:
|
||||
marker = internal_path(evidence / 'quick-run/uncertain.flag')
|
||||
marker.parent.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
with marker.open('x', encoding='utf-8') as stream:
|
||||
stream.write('reason=supervisor_interrupted; batch=' + journal['batch_id'] + '; manual reconciliation required\n')
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
except FileExistsError:
|
||||
pass
|
||||
protect_interruption(evidence / 'quick-run',
|
||||
'reason=supervisor_interrupted; batch=' + journal['batch_id'] + '; manual reconciliation required\n')
|
||||
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal)
|
||||
|
||||
|
||||
@@ -53,6 +57,61 @@ def _worker_context(root):
|
||||
_WORKER_ROOT = canonical_path(root)
|
||||
|
||||
|
||||
def _force_owned_supervisor(root, journal, process):
|
||||
"""Persist uncertainty before terminating the exact process/Job we own."""
|
||||
journal['stop_mode'] = 'forced'
|
||||
journal['force_requested_at'] = time.time()
|
||||
# The running worker can cross a task boundary while evidence is written.
|
||||
# Protect all possible targets before force; keep their saved results intact.
|
||||
journal['protect_all_targets'] = True
|
||||
_mark_interrupted(root, journal, '用户明确强制结束;写入与保存结果须核验,禁止自动重放。')
|
||||
try:
|
||||
process.terminate()
|
||||
except Exception as error:
|
||||
if not getattr(process, 'cleanup_record', {}).get('forced_termination'):
|
||||
raise
|
||||
journal['force_cleanup_error'] = str(error)
|
||||
try:
|
||||
from .batch_resources import write_record
|
||||
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal)
|
||||
except OSError:
|
||||
pass # The forced intent and uncertainty were already durably saved.
|
||||
|
||||
|
||||
def _retain_supervisor_after_control_error(root, journal, process, control, error):
|
||||
"""Do not turn a GUI/transport exception into an unconfirmed Job kill."""
|
||||
from .batch_resources import write_record
|
||||
control.set()
|
||||
control.control_error = '控制通信异常,已请求安全停止并保留应用;可导出诊断或明确强制结束:' + str(error)
|
||||
journal['protect_all_targets'] = True
|
||||
try:
|
||||
_mark_interrupted(root, journal, error)
|
||||
except Exception as failure:
|
||||
control.control_error += ';中断证据尚未落盘:' + str(failure)
|
||||
try:
|
||||
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'cancel.json',
|
||||
{'version': VERSION, 'batch_id': journal['batch_id'], 'attempt_id': journal['attempt_id']})
|
||||
except OSError:
|
||||
pass # Keep ownership; failure to request stop is not permission to kill.
|
||||
while True:
|
||||
if control.force_event.is_set():
|
||||
try:
|
||||
_force_owned_supervisor(root, journal, process)
|
||||
except Exception as failure:
|
||||
control.force_event.clear()
|
||||
control.control_error = '强制结束未确认,应用仍保留:' + str(failure)
|
||||
else:
|
||||
return True
|
||||
try:
|
||||
process.wait(timeout=.2)
|
||||
return False
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
except Exception as failure:
|
||||
control.control_error = '无法确认应用退出,保留所有权等待处理:' + str(failure)
|
||||
threading.Event().wait(.2)
|
||||
|
||||
|
||||
def _file_lock(handle, acquire):
|
||||
from .task_queue import _os_lock
|
||||
_os_lock(handle, acquire)
|
||||
@@ -170,6 +229,19 @@ def _finish_response(root, journal, response, context):
|
||||
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal)
|
||||
if related:
|
||||
raise resource_pending_error(journal['operation'], result, error, primary)
|
||||
resource_path = internal_path(Path(root) / 'batch-resources' / journal['batch_id'] / 'session.json')
|
||||
try:
|
||||
resource = json.loads(resource_path.read_text(encoding='utf-8'))
|
||||
except FileNotFoundError:
|
||||
resource = {}
|
||||
shutdown = resource.get('cad_shutdown') or {}
|
||||
if (shutdown.get('orderly_shutdown') is False or shutdown.get('forced_termination')
|
||||
or shutdown.get('diagnostic_errors')
|
||||
or any(item.get('stage') != 'desktop_release' for item in shutdown.get('cleanup_errors', ()))):
|
||||
failure = TaskCenterError('application_shutdown_failed',
|
||||
'已核验业务结果保留,但应用未正常收尾;请查看退出诊断,不能将资源消失视为正常退出。')
|
||||
failure.completed_result, failure.cleanup_error = result, error
|
||||
raise failure
|
||||
if response.get('business_succeeded'):
|
||||
if response.get('error'):
|
||||
context.logger.info('supervisor_resource_recovery_passed batch_id=%s business_replayed=false', journal['batch_id'])
|
||||
@@ -186,6 +258,7 @@ def _finish_response(root, journal, response, context):
|
||||
def run_supervised(service, operation, arguments, *, stop_event=None, on_progress=None, on_phase=None):
|
||||
from .batch_resources import write_record, process_identity
|
||||
from .isolated_desktop import _Win32Api
|
||||
from .stop_control import StopControl
|
||||
root = canonical_path(service._logging.root)
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
raise TaskCenterError('operation_cancelled', '本批操作已取消,未启动监督进程。')
|
||||
@@ -216,13 +289,15 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres
|
||||
journal['parent_process'] = process_identity(parent)
|
||||
write_record(request, {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id,
|
||||
'root': str(root), 'database': str(canonical_path(service._store.path)),
|
||||
'operation': operation, 'arguments': encode(arguments), 'task_ids': journal['task_ids']})
|
||||
'operation': operation, 'arguments': encode(arguments), 'task_ids': journal['task_ids'],
|
||||
'interactive_stop': isinstance(stop_event, StopControl)})
|
||||
journal['request_sha256'] = hashlib.sha256(internal_path(request).read_bytes()).hexdigest()
|
||||
write_record(directory / 'supervisor.json', journal)
|
||||
gate, messages = MessageGate(batch_id, attempt_id), queue.Queue()
|
||||
api, process, reader = _Win32Api(), None, None
|
||||
sent_cancel, last_heartbeat = False, time.monotonic()
|
||||
transport_error = None
|
||||
force_accepted = False
|
||||
def receive():
|
||||
try:
|
||||
while line := process.stdout.readline(1024 * 1024):
|
||||
@@ -242,9 +317,26 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres
|
||||
reader = threading.Thread(target=receive, daemon=True)
|
||||
reader.start()
|
||||
while process.poll() is None or not messages.empty():
|
||||
force = getattr(stop_event, 'force_event', None)
|
||||
if force is not None and force.is_set() and process.poll() is None:
|
||||
try:
|
||||
_force_owned_supervisor(root, journal, process)
|
||||
except Exception as error:
|
||||
force.clear()
|
||||
stop_event.control_error = '强制结束未执行或未确认;请保留窗口和诊断:' + str(error)
|
||||
service._logging.logger.error('force_stop_failed batch_id=%s error=%s', batch_id, error)
|
||||
else:
|
||||
force_accepted = True
|
||||
break
|
||||
if stop_event is not None and stop_event.is_set() and not sent_cancel:
|
||||
write_record(directory / 'cancel.json', {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id})
|
||||
sent_cancel = True
|
||||
try:
|
||||
write_record(directory / 'cancel.json', {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id})
|
||||
journal.update(stop_mode='safe', stop_requested_at=time.time())
|
||||
write_record(directory / 'supervisor.json', journal)
|
||||
sent_cancel = True
|
||||
except OSError as error:
|
||||
if hasattr(stop_event, 'control_error'):
|
||||
stop_event.control_error = '停止请求尚未成功记录,正在重试:' + str(error)
|
||||
try:
|
||||
value = messages.get(timeout=.1)
|
||||
except queue.Empty:
|
||||
@@ -264,6 +356,7 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres
|
||||
state = decode(value['payload'])
|
||||
service._task_queue_feedback = state['queue']
|
||||
service._batch_feedback = state['batch']
|
||||
service._supervisor_waiting = bool(state.get('waiting_for_resolution'))
|
||||
if on_phase is not None and state['phase'] in {'recycling', 'closing'}:
|
||||
from .bom_import_state import BomImportPhase
|
||||
on_phase(BomImportPhase.RECOVERING if state['phase'] == 'recycling' else BomImportPhase.RELEASING,
|
||||
@@ -271,9 +364,12 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres
|
||||
process.wait(timeout=15)
|
||||
except Exception as error:
|
||||
transport_error = error
|
||||
if process is not None and isinstance(stop_event, StopControl):
|
||||
force_accepted = _retain_supervisor_after_control_error(root, journal, process, stop_event, error)
|
||||
finally:
|
||||
service._task_queue_feedback = None
|
||||
service._batch_feedback = None
|
||||
service._supervisor_waiting = False
|
||||
try:
|
||||
if process is not None:
|
||||
process.close() # Waits, and confines termination to the owned Job.
|
||||
@@ -285,6 +381,14 @@ def run_supervised(service, operation, arguments, *, stop_event=None, on_progres
|
||||
'batch_resources_pending', f'监督进程错误:{transport_error};收尾错误:{error}')
|
||||
if reader is not None:
|
||||
reader.join(3)
|
||||
if force_accepted:
|
||||
from .batch_resources import cleanup_pending_resources
|
||||
try:
|
||||
cleanup_pending_resources(root, logging_context=service._logging)
|
||||
except Exception as error:
|
||||
service._logging.logger.error('forced_stop_cleanup_pending batch_id=%s error=%s', batch_id, error)
|
||||
raise TaskCenterError('user_forced_stop',
|
||||
'已按确认强制结束;已保存的执行证据保留,当前结果待核验。请先核验恢复,不要重复替换。')
|
||||
try:
|
||||
try:
|
||||
response = json.loads(internal_path(directory / 'result.json').read_text(encoding='utf-8'))
|
||||
|
||||
@@ -75,7 +75,7 @@ def run(request_path):
|
||||
request = json.loads(request_bytes)
|
||||
batch_id, attempt_id = request['batch_id'], request['attempt_id']
|
||||
if (request.get('version') != VERSION or UUID(batch_id).hex != batch_id or UUID(attempt_id).hex != attempt_id
|
||||
or request['operation'] not in OPERATIONS):
|
||||
or request['operation'] not in OPERATIONS or type(request.get('interactive_stop', False)) is not bool):
|
||||
raise ValueError('Invalid batch supervisor request')
|
||||
root = canonical_path(request['root'])
|
||||
directory = root / 'batch-resources' / batch_id
|
||||
@@ -83,7 +83,9 @@ def run(request_path):
|
||||
raise ValueError('Supervisor request ownership path mismatch')
|
||||
_admit_request(directory, request, request_bytes)
|
||||
_worker_context(root)
|
||||
stop, finished, output_lock = threading.Event(), threading.Event(), threading.Lock()
|
||||
from .stop_control import StopControl
|
||||
stop = StopControl() if request.get('interactive_stop', False) else threading.Event()
|
||||
finished, output_lock = threading.Event(), threading.Lock()
|
||||
kind = {'inspect_bom_batch': 'bom_import', 'preview_bom_batch': 'bom_import',
|
||||
'execute_task_queue': 'task_queue', 'execute_quick_run_queue': 'quick_run_queue'}.get(request['operation'], request['operation'])
|
||||
runtime = BatchCadRuntime(root, kind, stop, session_id=batch_id)
|
||||
@@ -113,7 +115,9 @@ def run(request_path):
|
||||
emit('heartbeat')
|
||||
feedback = getattr(service, '_task_queue_feedback', None)
|
||||
emit('state', encode({'phase': runtime.record['state'], 'queue': feedback,
|
||||
'batch': service.batch_feedback()}))
|
||||
'batch': service.batch_feedback(),
|
||||
'waiting_for_resolution': getattr(stop, 'waiting_for_resolution', False),
|
||||
'resolution_error': getattr(stop, 'resolution_error', '')}))
|
||||
except Exception:
|
||||
stop.set() # Lost control evidence cannot authorize new work.
|
||||
finished.wait(1)
|
||||
|
||||
@@ -13,6 +13,7 @@ class ExecutionFeedback:
|
||||
message: str = '正在准备执行'
|
||||
stage: str = ''
|
||||
stage_seconds: float = 0.0
|
||||
waiting_for_safe_completion: bool = False
|
||||
|
||||
|
||||
def read_feedback(task_id, position, total, message, root: Path, started_after: float):
|
||||
@@ -27,15 +28,16 @@ def read_feedback(task_id, position, total, message, root: Path, started_after:
|
||||
except (OSError, ValueError, TypeError, AttributeError):
|
||||
pass
|
||||
path = root / 'quick-run' / 'worker-progress.json'
|
||||
stage, seconds = '', 0.0
|
||||
stage, seconds, waiting = '', 0.0, False
|
||||
try:
|
||||
modified = path.stat().st_mtime
|
||||
if modified >= started_after:
|
||||
record = json.loads(path.read_text(encoding='utf-8'))
|
||||
waiting = record.get('waiting_for_safe_completion') is True
|
||||
candidate = record.get('stage', '')
|
||||
if candidate and candidate != 'finished':
|
||||
seconds = float(record.get('stage_seconds', {}).get(candidate, 0)) + max(0, time.time() - modified)
|
||||
stage = candidate
|
||||
except (OSError, ValueError, TypeError, AttributeError):
|
||||
pass # Display feedback is optional; it never changes execution state.
|
||||
return ExecutionFeedback(task_id, position, total, message, stage, seconds)
|
||||
return ExecutionFeedback(task_id, position, total, message, stage, seconds, waiting)
|
||||
@@ -87,7 +87,7 @@ def _designer_executable():
|
||||
|
||||
class IsolatedCadSession:
|
||||
"""The supervisor never attaches to COM on the user's input desktop."""
|
||||
def __init__(self, root, *, abort_event=None, desktop=None):
|
||||
def __init__(self, root, *, abort_event=None, desktop=None, stop_control=None):
|
||||
self.root = canonical_path(root)
|
||||
self.token = uuid4().hex
|
||||
self.borrowed_desktop = desktop is not None
|
||||
@@ -96,6 +96,7 @@ class IsolatedCadSession:
|
||||
self.binding = None
|
||||
self.started = time.monotonic()
|
||||
self.abort_event = abort_event
|
||||
self.stop_control = stop_control
|
||||
self.record = {'desktop': self.desktop.name, 'token': self.token, 'state': 'starting'}
|
||||
|
||||
def _record(self, **values):
|
||||
@@ -206,52 +207,115 @@ class IsolatedCadSession:
|
||||
|
||||
def __exit__(self, error_type, error, traceback):
|
||||
closing_started = time.monotonic()
|
||||
cleanup_error = None
|
||||
cleanup_errors = []
|
||||
diagnostic_errors = []
|
||||
failures = []
|
||||
exit_codes = {}
|
||||
closing = None
|
||||
|
||||
def remember(stage, failure):
|
||||
cleanup_errors.append({'stage': stage, 'message': str(failure)})
|
||||
message = '; '.join(f"{item['stage']}: {item['message']}" for item in cleanup_errors)
|
||||
self._uncertain(message, cleanup_errors=cleanup_errors,
|
||||
failures.append(failure)
|
||||
|
||||
def diagnose(stage, action):
|
||||
try:
|
||||
action()
|
||||
except Exception as failure:
|
||||
diagnostic_errors.append({'stage': stage, 'message': str(failure)})
|
||||
failures.append(failure)
|
||||
self.record['clean'] = False
|
||||
|
||||
def failure_message():
|
||||
return '; '.join(f"{item['stage']}: {item['message']}"
|
||||
for item in cleanup_errors + diagnostic_errors)
|
||||
|
||||
def persist_uncertain():
|
||||
message = failure_message() or str(error)
|
||||
self.record.update(state='uncertain', clean=False, error=message,
|
||||
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors,
|
||||
shutdown_seconds=time.monotonic() - closing_started,
|
||||
desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes)
|
||||
diagnose('uncertain_record', lambda: self._uncertain(message,
|
||||
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors,
|
||||
clean=False, orderly_shutdown=self.record['orderly_shutdown'],
|
||||
shutdown_seconds=time.monotonic() - closing_started,
|
||||
desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes))
|
||||
|
||||
self.record.update(clean=False, orderly_shutdown=False, forced_termination=False,
|
||||
forced_root_termination=False, forced_descendant_pids=[], termination_reason=None,
|
||||
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors)
|
||||
diagnose('closing_record', lambda: self._record(state='closing', clean=False,
|
||||
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors))
|
||||
try:
|
||||
self._record(state='closing', cleanup_errors=cleanup_errors)
|
||||
if error is not None:
|
||||
self._uncertain(str(error))
|
||||
if self.cad is not None and self.cad.poll() is not None:
|
||||
raise WorkerMonitorError('owned CAD exited before the authorized quit request')
|
||||
if self.cad is not None and self.binding is not None:
|
||||
closing = self.desktop.start_process(_helper_command('quit', self.root / 'private-desktop.json', self.cad.pid),
|
||||
env=_child_environment(self.binding.environment()), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
try:
|
||||
quit_code = closing.wait(timeout=20)
|
||||
exit_codes['quit_helper_exit_code'] = quit_code
|
||||
if quit_code:
|
||||
raise WorkerMonitorError('owned CAD refused orderly exit: ' + closing.stderr.read().decode('utf-8', errors='replace'))
|
||||
cad_code = self.cad.wait(timeout=15)
|
||||
exit_codes['cad_exit_code'] = cad_code
|
||||
if cad_code != 0:
|
||||
raise WorkerMonitorError('owned CAD returned a nonzero exit code after quit')
|
||||
finally:
|
||||
closing.close()
|
||||
quit_code = closing.wait(timeout=20)
|
||||
exit_codes['quit_helper_exit_code'] = quit_code
|
||||
if quit_code:
|
||||
raise WorkerMonitorError('owned CAD refused orderly exit: ' + closing.stderr.read().decode('utf-8', errors='replace'))
|
||||
cad_code = self.cad.wait(timeout=15)
|
||||
exit_codes['cad_exit_code'] = cad_code
|
||||
if cad_code != 0:
|
||||
raise WorkerMonitorError('owned CAD returned a nonzero exit code after quit')
|
||||
self.record['orderly_shutdown'] = True
|
||||
except Exception as failure:
|
||||
cleanup_error = failure
|
||||
remember('orderly_shutdown', failure)
|
||||
finally:
|
||||
if error is not None or failures:
|
||||
# Evidence failure must never prevent the identity-checked Quit
|
||||
# above or release of the handles we own below.
|
||||
persist_uncertain()
|
||||
if self.stop_control is not None and cleanup_errors and self.cad is not None:
|
||||
cad_running = True
|
||||
try:
|
||||
cad_running = self.cad.poll() is None
|
||||
except Exception as failure:
|
||||
remember('cad_exit_check', failure)
|
||||
if cad_running:
|
||||
# A refused/timed-out Quit may still be saving or closing.
|
||||
# Keep both CAD and its helper owned and alive until the
|
||||
# interactive caller confirms force, including ordinary stop.
|
||||
from .stop_control import wait_for_resolution
|
||||
try:
|
||||
wait_for_resolution(self.stop_control, self.root, failure_message())
|
||||
except WorkerMonitorError as failure:
|
||||
force_event = getattr(self.stop_control, 'force_event', None)
|
||||
if force_event is None or not force_event.is_set():
|
||||
raise
|
||||
remember('forced_stop', failure)
|
||||
if closing is not None:
|
||||
try:
|
||||
closing.close()
|
||||
except Exception as failure:
|
||||
remember('quit_helper_release', failure)
|
||||
try:
|
||||
self.desktop.close()
|
||||
self._record(state='session_closed' if self.borrowed_desktop else 'released',
|
||||
desktop_retained_by_batch=self.borrowed_desktop,
|
||||
clean=error is None and cleanup_error is None,
|
||||
shutdown_seconds=time.monotonic() - closing_started,
|
||||
desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes)
|
||||
except Exception as failure:
|
||||
cleanup_error = failure
|
||||
remember('desktop_release', failure)
|
||||
if cleanup_error is not None:
|
||||
message = '; '.join(f"{item['stage']}: {item['message']}" for item in cleanup_errors)
|
||||
raise WorkerMonitorError('private CAD cleanup requires reconciliation: ' + message) from cleanup_error
|
||||
desktop_cleanup = getattr(self.desktop, 'cleanup_record', None) or {}
|
||||
forced = [item for item in desktop_cleanup.get('processes', []) if item.get('forced_termination')]
|
||||
self.record.update(forced_termination=bool(forced),
|
||||
forced_root_termination=any(item.get('forced_root_termination') for item in forced),
|
||||
termination_reason=next((item.get('termination_reason') for item in forced
|
||||
if item.get('termination_reason')), None),
|
||||
forced_descendant_pids=sorted({pid for item in forced for pid in item.get('forced_descendant_pids', [])}))
|
||||
if forced:
|
||||
remember('forced_cleanup', WorkerMonitorError('owned processes required forced termination'))
|
||||
if error is not None or failures:
|
||||
persist_uncertain()
|
||||
else:
|
||||
diagnose('released_record', lambda: self._record(
|
||||
state='session_closed' if self.borrowed_desktop else 'released',
|
||||
desktop_retained_by_batch=self.borrowed_desktop, clean=True,
|
||||
shutdown_seconds=time.monotonic() - closing_started,
|
||||
desktop_cleanup=desktop_cleanup, **exit_codes))
|
||||
if failures:
|
||||
persist_uncertain()
|
||||
if failures:
|
||||
raise WorkerMonitorError('private CAD cleanup requires reconciliation: ' + failure_message()) from failures[0]
|
||||
|
||||
def _uncertain(self, message, **details):
|
||||
from .quick_run import _write_uncertain_flag
|
||||
|
||||
@@ -23,6 +23,7 @@ from .errors import TaskCenterError
|
||||
|
||||
|
||||
DESKTOP_RELEASE_TIMEOUT_SECONDS = 5.0
|
||||
DESCENDANT_EXIT_GRACE_SECONDS = 2.0
|
||||
|
||||
|
||||
def _native_handle(handle):
|
||||
@@ -192,7 +193,10 @@ def _stop_and_wait(process):
|
||||
errors.append(error)
|
||||
if running:
|
||||
try:
|
||||
process.terminate()
|
||||
if isinstance(process, _Win32Process):
|
||||
process.terminate(reason='root_process_active')
|
||||
else:
|
||||
process.terminate()
|
||||
except Exception as error:
|
||||
errors.append(error)
|
||||
try:
|
||||
@@ -313,7 +317,9 @@ class _Win32Process:
|
||||
self.job_assigned = False
|
||||
self.returncode = None
|
||||
self.stdout = self.stderr = None
|
||||
self.cleanup_record = {'pid': self.pid}
|
||||
self.cleanup_record = {'pid': self.pid, 'forced_termination': False,
|
||||
'forced_root_termination': False, 'forced_descendant_pids': [],
|
||||
'termination_reason': None}
|
||||
|
||||
def poll(self):
|
||||
if self.returncode is not None or self.handle is None:
|
||||
@@ -340,34 +346,59 @@ class _Win32Process:
|
||||
self.returncode = self._process_api.GetExitCodeProcess(self.handle)
|
||||
return self.returncode
|
||||
|
||||
def terminate(self):
|
||||
def terminate(self, *, reason='explicit_termination'):
|
||||
if self.handle is None:
|
||||
return
|
||||
errors = []
|
||||
root_active = True
|
||||
try:
|
||||
root_active = self.poll() is None
|
||||
except Exception as error:
|
||||
errors.append(error)
|
||||
self.cleanup_record.setdefault('root_exit_before_cleanup', not root_active)
|
||||
if not self.job_assigned:
|
||||
self._process_api.TerminateProcess(self.handle, 1)
|
||||
self._record_termination(reason, root_active, ())
|
||||
_raise_cleanup_errors(errors)
|
||||
return
|
||||
errors = []
|
||||
descendants = set()
|
||||
try:
|
||||
self._remember_descendants()
|
||||
# Keep confirmed members of this snapshot even if opening a later
|
||||
# member fails; the Job termination below still includes them.
|
||||
self._remember_descendants(active=descendants)
|
||||
except Exception as error:
|
||||
errors.append(error)
|
||||
try:
|
||||
self._job_api.TerminateJobObject(self.job, 1)
|
||||
self._record_termination(reason, root_active, descendants)
|
||||
except Exception:
|
||||
self._process_api.TerminateProcess(self.handle, 1)
|
||||
self._record_termination(reason, root_active, ())
|
||||
_raise_cleanup_errors(errors)
|
||||
|
||||
kill = terminate
|
||||
|
||||
def _remember_descendants(self):
|
||||
def _record_termination(self, reason, root_active, descendants):
|
||||
self.cleanup_record.update(forced_termination=True,
|
||||
forced_root_termination=self.cleanup_record['forced_root_termination'] or root_active,
|
||||
forced_descendant_pids=sorted(set(self.cleanup_record['forced_descendant_pids']) | set(descendants)),
|
||||
termination_reason=self.cleanup_record['termination_reason'] or reason)
|
||||
|
||||
def _remember_descendants(self, *, active=None):
|
||||
if active is None:
|
||||
active = set()
|
||||
if self._open_process is None or self.job is None:
|
||||
return
|
||||
return active
|
||||
for pid in self._job_api.QueryInformationJobObject(self.job, self._job_api.JobObjectBasicProcessIdList):
|
||||
if pid == self.pid or pid in self._descendant_pids:
|
||||
if pid == self.pid:
|
||||
continue
|
||||
if pid in self._descendant_pids:
|
||||
active.add(pid)
|
||||
continue
|
||||
try:
|
||||
self._descendant_handles.append(self._open_process(0x100000, False, pid))
|
||||
self._descendant_pids.add(pid)
|
||||
active.add(pid)
|
||||
except Exception as error:
|
||||
# pywintypes.error is not an OSError. A child that exited
|
||||
# between the job snapshot and OpenProcess reports error 87.
|
||||
@@ -376,22 +407,47 @@ class _Win32Process:
|
||||
code = error.args[0]
|
||||
if code != 87:
|
||||
raise
|
||||
return active
|
||||
|
||||
def close(self):
|
||||
errors = []
|
||||
job_empty = not self.job_assigned or self.job is None
|
||||
if self.handle is not None:
|
||||
if 'root_exit_before_cleanup' not in self.cleanup_record:
|
||||
try:
|
||||
self.cleanup_record['root_exit_before_cleanup'] = self.poll() is not None
|
||||
except Exception as error:
|
||||
errors.append(error)
|
||||
errors.extend(_stop_and_wait(self))
|
||||
deadline = time.monotonic() + 5.0
|
||||
if self.job is not None and self.job_assigned:
|
||||
try:
|
||||
# A completed root can leave grandchildren holding the desktop.
|
||||
self._remember_descendants()
|
||||
self._job_api.TerminateJobObject(self.job, 1)
|
||||
grace_started = time.monotonic()
|
||||
grace_deadline = grace_started + DESCENDANT_EXIT_GRACE_SECONDS
|
||||
# A completed root may leave children finishing normal shutdown.
|
||||
# An empty job takes no grace delay; an active root was already
|
||||
# stopped above and does not receive this normal-exit allowance.
|
||||
while True:
|
||||
active = self._job_api.QueryInformationJobObject(
|
||||
self.job, self._job_api.JobObjectBasicAccountingInformation)['ActiveProcesses']
|
||||
self.cleanup_record['job_active_processes'] = active
|
||||
if not active:
|
||||
job_empty = True
|
||||
break
|
||||
self._remember_descendants()
|
||||
if (self.cleanup_record.get('root_exit_before_cleanup') is not True
|
||||
or self.cleanup_record['forced_termination'] or time.monotonic() >= grace_deadline):
|
||||
self.terminate(reason='descendant_grace_expired')
|
||||
break
|
||||
time.sleep(min(0.01, max(0, grace_deadline - time.monotonic())))
|
||||
self.cleanup_record['descendant_grace_seconds'] = time.monotonic() - grace_started
|
||||
deadline = time.monotonic() + 5.0
|
||||
while True:
|
||||
active = self._job_api.QueryInformationJobObject(
|
||||
self.job, self._job_api.JobObjectBasicAccountingInformation)['ActiveProcesses']
|
||||
self.cleanup_record['job_active_processes'] = active
|
||||
if not active:
|
||||
job_empty = True
|
||||
break
|
||||
if time.monotonic() >= deadline:
|
||||
raise TimeoutError('owned job descendants did not exit within 5 seconds')
|
||||
@@ -417,6 +473,10 @@ class _Win32Process:
|
||||
self._descendant_handles = retained
|
||||
for name in ('job', 'thread', 'handle'):
|
||||
handle = getattr(self, name)
|
||||
if name in ('job', 'handle') and not job_empty:
|
||||
# Closing a KILL_ON_JOB_CLOSE job would silently force remaining
|
||||
# children and discard the capability needed for cleanup retry.
|
||||
continue
|
||||
if name == 'handle' and self.returncode is None:
|
||||
# Keep the only termination/wait capability until cleanup can retry.
|
||||
continue
|
||||
|
||||
@@ -111,6 +111,13 @@ class ProjectSessionWorker:
|
||||
return
|
||||
if not self._done.is_set():
|
||||
_write_control(self.root / 'project-session-next.tsv', f'{self.token}\tclose')
|
||||
from .stop_control import StopControl
|
||||
interactive = isinstance(getattr(self.runtime, 'stop_event', None), StopControl)
|
||||
if interactive:
|
||||
# The monitor owns the safe/unsafe distinction. Do not
|
||||
# turn a GUI close or ordinary pause into a timed kill.
|
||||
while not self._done.wait(.1):
|
||||
pass
|
||||
close_grace = 60 if self.private_desktop else min(30, max(10, self._timeout))
|
||||
if not self._done.wait(close_grace):
|
||||
self._abort.set()
|
||||
|
||||
@@ -925,6 +925,9 @@ class TaskCenterService:
|
||||
message = batch[2] if batch else '正在准备执行'
|
||||
value = read_feedback(task_id, position, total, message,
|
||||
child._quick_run_preparer.data_root, started_after)
|
||||
if getattr(self, '_supervisor_waiting', False):
|
||||
from dataclasses import replace
|
||||
value = replace(value, stage='awaiting_reconciliation', waiting_for_safe_completion=True)
|
||||
if read_control is not None:
|
||||
read_control.checkpoint()
|
||||
return value
|
||||
@@ -951,12 +954,12 @@ class TaskCenterService:
|
||||
return delegated.has_interrupted_quick_run(task_id)
|
||||
return self._quick_run_preparer.has_interrupted(task_id)
|
||||
|
||||
def recover_quick_run(self, task_id: str, *, confirmed: bool = False, runtime=None) -> QuickRunStepResult:
|
||||
def recover_quick_run(self, task_id: str, *, confirmed: bool = False, runtime=None, stop_event=None) -> QuickRunStepResult:
|
||||
if runtime is None and self._supervised:
|
||||
return self._supervise('recover_quick_run', {'task_id': task_id, 'confirmed': confirmed})
|
||||
return self._supervise('recover_quick_run', {'task_id': task_id, 'confirmed': confirmed}, stop_event=stop_event)
|
||||
if runtime is None:
|
||||
return self._owned_cad_operation('recover_quick_run', lambda owned:
|
||||
self.recover_quick_run(task_id, confirmed=confirmed, runtime=owned), task_id=task_id)
|
||||
self.recover_quick_run(task_id, confirmed=confirmed, runtime=owned), stop_event, task_id=task_id)
|
||||
delegated = self._project_task_service(task_id)
|
||||
if delegated is not self:
|
||||
return delegated.recover_quick_run(task_id, confirmed=confirmed, runtime=runtime)
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
"""Separate ordinary stop intent from an explicitly confirmed forced interruption."""
|
||||
from threading import Event
|
||||
import json
|
||||
import os
|
||||
|
||||
from .path_access import internal_path
|
||||
|
||||
|
||||
class StopControl(Event):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.force_event = Event()
|
||||
self.control_error = ''
|
||||
self.waiting_for_resolution = False
|
||||
self.resolution_error = ''
|
||||
|
||||
def request_force(self):
|
||||
self.set()
|
||||
self.control_error = ''
|
||||
self.force_event.set()
|
||||
|
||||
|
||||
def protect_interruption(root, reason):
|
||||
"""Create or re-sync a marker without overwriting prior protection evidence."""
|
||||
root = internal_path(root)
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
marker = root / 'uncertain.flag'
|
||||
try:
|
||||
stream = marker.open('x', encoding='utf-8')
|
||||
except FileExistsError:
|
||||
# A previous create/write/fsync may have failed. Existence alone does
|
||||
# not establish durable evidence; preserve its bytes but sync again.
|
||||
with marker.open('r+b') as existing:
|
||||
os.fsync(existing.fileno())
|
||||
else:
|
||||
with stream:
|
||||
stream.write(str(reason))
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
|
||||
|
||||
def wait_for_resolution(control, root, error):
|
||||
"""Interactive owners retain a possibly changed project until explicit force.
|
||||
|
||||
The GUI-side supervisor can also terminate its own Job after durable force
|
||||
evidence. Ordinary cancellation never releases this wait.
|
||||
"""
|
||||
if not isinstance(control, StopControl):
|
||||
return
|
||||
from .worker_monitor import WorkerMonitorError, _publish_progress
|
||||
root = internal_path(root)
|
||||
control.set()
|
||||
control.waiting_for_resolution = True
|
||||
control.resolution_error = str(error)
|
||||
def protect():
|
||||
protect_interruption(root, 'operation_interrupted: ' + str(error))
|
||||
try:
|
||||
protect()
|
||||
except OSError as failure:
|
||||
control.control_error = '保护记录写入失败,保留应用等待处理:' + str(failure)
|
||||
try:
|
||||
record = json.loads((root / 'worker-progress.json').read_text(encoding='utf-8'))
|
||||
if not isinstance(record, dict):
|
||||
record = {}
|
||||
except (OSError, ValueError):
|
||||
record = {}
|
||||
record.update(stage='awaiting_reconciliation', waiting_for_safe_completion=True,
|
||||
resolution_error=str(error), stop_requested=True, safe_stop=False)
|
||||
_publish_progress(root / 'worker-progress.json', record)
|
||||
while True:
|
||||
if control.force_event.wait(.1):
|
||||
try:
|
||||
protect()
|
||||
except OSError as failure:
|
||||
control.force_event.clear()
|
||||
control.control_error = '强制结束未执行,保护记录无法保存:' + str(failure)
|
||||
continue
|
||||
raise WorkerMonitorError('user forced stop of uncertain operation; reconciliation required')
|
||||
@@ -15,6 +15,7 @@ STAGES = {
|
||||
'restore_attribute_fields': '恢复属性字段', 'restore_attributes': '恢复属性',
|
||||
'verify_restoration': '核验恢复结果', 'verify_attribute_fields': '核验属性字段',
|
||||
'save_and_close': '保存并关闭工程', 'verify_saved_project': '回读已保存工程',
|
||||
'awaiting_reconciliation': '当前结果待核验,已保留工程现场',
|
||||
}
|
||||
|
||||
|
||||
@@ -22,4 +23,6 @@ def format_feedback(value, task_name, elapsed, stopping):
|
||||
stage = STAGES.get(value.stage, '')
|
||||
details = f'{stage}({value.stage_seconds:.1f} 秒)' if stage else value.message
|
||||
prefix = '等待当前步骤保存核验后暂停 · ' if stopping else ''
|
||||
if value.waiting_for_safe_completion:
|
||||
prefix = '当前保存或核验尚未确认,未自动结束应用;可继续等待、导出诊断,或确认强制结束后核验 · '
|
||||
return f'{prefix}任务 {value.position}/{value.total}:{task_name} · {details} · 总计 {elapsed} 秒'
|
||||
@@ -1,6 +1,7 @@
|
||||
"""Single-project task presentation; persistence and CAD remain in the service."""
|
||||
from __future__ import annotations
|
||||
from threading import Event
|
||||
from ..stop_control import StopControl
|
||||
from shiboken6 import isValid
|
||||
import time
|
||||
from PySide6.QtCore import QElapsedTimer, QSettings, QSignalBlocker, QThreadPool, QTimer, Qt
|
||||
@@ -39,6 +40,7 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self._tasks = ()
|
||||
self._known_task_groups = {}
|
||||
self._workers = set()
|
||||
self._diagnostic_worker = None
|
||||
self._closed = False
|
||||
self._batch_stop = None
|
||||
self._active_quick_run_task_id = None
|
||||
@@ -88,6 +90,7 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self._log_presenter = TaskLogPresenter(service, self.run_panel, self)
|
||||
self.run_panel.quick_run_button.clicked.connect(self._prepare_quick_run)
|
||||
self.run_panel.stop_queue_button.clicked.connect(self._pause_queue)
|
||||
self.run_panel.force_stop_button.clicked.connect(self._force_batch_stop)
|
||||
self.run_panel.retry_button.clicked.connect(self._retry_failed)
|
||||
self.run_panel.recover_button.clicked.connect(self._recover_quick_run)
|
||||
self.run_panel.cleanup_resources_button.clicked.connect(self._cleanup_batch_resources)
|
||||
@@ -337,6 +340,10 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self.run_panel.history_button.setEnabled(bool(context and not busy))
|
||||
self.run_panel.stop_queue_button.setVisible(self._batch_stop is not None)
|
||||
self.run_panel.stop_queue_button.setEnabled(bool(self._batch_stop and not self._batch_stop.is_set()))
|
||||
self.run_panel.force_stop_button.setVisible(self._batch_stop is not None)
|
||||
self.run_panel.force_stop_button.setEnabled(bool(self._batch_stop and
|
||||
not self._batch_stop.force_event.is_set()))
|
||||
self.run_panel.diagnostics_button.setEnabled(self._diagnostic_worker is None)
|
||||
self.run_panel.copy_value_check.setEnabled(state.editable)
|
||||
rule_editable = state.editable and not group_unsaved
|
||||
self.mapping_table.set_rules_editable(rule_editable)
|
||||
@@ -571,6 +578,8 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self._finish_run()
|
||||
self._metadata_changed(None)
|
||||
failures = sum(e.state in {'failed', 'blocked'} for e in result.entries)
|
||||
if failures:
|
||||
self._close_pending = False
|
||||
self._show_task_log(force=True)
|
||||
self._set_operator_message(f'本批任务已暂停。失败/阻断 {failures} 项。' if result.paused else f'本批任务已结束。失败 {failures} 项。',
|
||||
'warning' if result.paused or failures else 'success')
|
||||
@@ -578,7 +587,7 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self.task_list.queue_button)
|
||||
|
||||
def _begin_run(self, task_id):
|
||||
self._batch_stop = Event()
|
||||
self._batch_stop = StopControl()
|
||||
self._active_quick_run_task_id = task_id
|
||||
self._run_started_at = time.time()
|
||||
self._quick_run_elapsed.start()
|
||||
@@ -601,6 +610,24 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self._set_operator_message('已请求暂停;当前项安全收尾后停止。', 'warning')
|
||||
self._update_gates()
|
||||
|
||||
def _force_batch_stop(self):
|
||||
control = self._batch_stop
|
||||
if control is None or control.force_event.is_set():
|
||||
return
|
||||
if QMessageBox.question(self, '强制结束当前操作',
|
||||
'当前操作可能正在修改或保存工程。强制结束可能留下未保存或部分保存的结果。\n\n'
|
||||
'已完成的执行证据会保留;中断后必须先核验恢复,不能直接重复替换。'
|
||||
'建议先导出诊断包。确定强制结束本程序当前操作吗?',
|
||||
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
|
||||
QMessageBox.StandardButton.No) != QMessageBox.StandardButton.Yes:
|
||||
return
|
||||
if self._batch_stop is not control:
|
||||
return # Completion/new run while the confirmation dialog was open.
|
||||
self._close_pending = False
|
||||
control.request_force()
|
||||
self._set_operator_message('已请求强制结束;正在记录中断证据,结果将进入待核验。', 'warning')
|
||||
self._update_gates()
|
||||
|
||||
def _show_batch_result(self, result):
|
||||
owner = self._active_quick_run_task_id or self._task_id
|
||||
name = next((t.name for t in self._tasks if t.task_id == owner), owner)
|
||||
@@ -609,6 +636,8 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
if owner == self._task_id:
|
||||
self._show_task_log(force=True)
|
||||
p = result.progress
|
||||
if p.failed_items:
|
||||
self._close_pending = False
|
||||
self._set_operator_message(f"任务「{name}」{'已暂停' if result.paused else '已结束'}:已保存 {p.saved_items},失败 {p.failed_items},未命中 {p.no_match_items}。",
|
||||
'warning' if result.paused or p.failed_items else 'success')
|
||||
|
||||
@@ -637,10 +666,31 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
from pathlib import Path
|
||||
name = 'DxPartReplace-诊断-' + time.strftime('%Y%m%d-%H%M%S') + '.zip'
|
||||
value, _ = QFileDialog.getSaveFileName(self, '导出脱敏诊断包', name, 'ZIP 文件 (*.zip)')
|
||||
if value:
|
||||
self._run_async(lambda: self.service.export_diagnostics(Path(value)),
|
||||
lambda path: self._set_operator_message(f'脱敏诊断包已导出:{path}', 'success'),
|
||||
self.run_panel.diagnostics_button)
|
||||
if not value or self._diagnostic_worker is not None:
|
||||
return
|
||||
# A read-only support export remains available while CAD is waiting.
|
||||
worker = ServiceWorker(lambda: self.service.export_diagnostics(Path(value)))
|
||||
self._diagnostic_worker = worker
|
||||
self._workers.add(worker)
|
||||
def exported(path):
|
||||
if not self._closed:
|
||||
self._set_operator_message(f'脱敏诊断包已导出:{path}', 'success')
|
||||
def failed(error):
|
||||
self._close_pending = False
|
||||
if not self._closed:
|
||||
self._show_error(error)
|
||||
def finished():
|
||||
self._workers.discard(worker)
|
||||
self._diagnostic_worker = None
|
||||
if not self._closed:
|
||||
self._update_gates()
|
||||
if self._close_pending and not self._workers:
|
||||
QTimer.singleShot(0, self.close)
|
||||
worker.signals.succeeded.connect(exported)
|
||||
worker.signals.failed.connect(failed)
|
||||
worker.signals.finished.connect(finished)
|
||||
self._update_gates()
|
||||
QThreadPool.globalInstance().start(worker)
|
||||
|
||||
def _retry_failed(self):
|
||||
if self.run_panel.retry_button.isEnabled():
|
||||
@@ -700,7 +750,8 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self._finish_run()
|
||||
self._metadata_changed(None)
|
||||
self._set_operator_message(result.message)
|
||||
self._run_async(lambda: self.service.recover_quick_run(task_id, confirmed=True), done,
|
||||
stop = self._batch_stop
|
||||
self._run_async(lambda: self.service.recover_quick_run(task_id, confirmed=True, stop_event=stop), done,
|
||||
self.run_panel.recover_button, expected_task_id=task_id)
|
||||
|
||||
def _confirm_quick_run(self, scope, selected_projects, enabled_rules):
|
||||
@@ -725,6 +776,9 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self.run_panel.execution_status.setText(format_feedback(
|
||||
value, name, self._quick_run_elapsed.elapsed() // 1000,
|
||||
bool(self._batch_stop and self._batch_stop.is_set())))
|
||||
if self._batch_stop and self._batch_stop.control_error:
|
||||
self._set_operator_message(self._batch_stop.control_error, 'warning')
|
||||
self._update_gates()
|
||||
self._feedback_query.request(lambda control: self.service.execution_feedback(task_id, started_after=started, read_control=control),
|
||||
received, lambda _: None, cancellable=True)
|
||||
if self.run_panel.toggle_log_button.isChecked():
|
||||
@@ -799,6 +853,7 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
callback(result)
|
||||
except Exception as error:
|
||||
operation_error = error
|
||||
self._close_pending = False
|
||||
self.operation_state = self.operation_state.finish(error)
|
||||
self._show_error(error)
|
||||
def failed(error):
|
||||
|
||||
@@ -4,6 +4,8 @@ import json
|
||||
import re
|
||||
|
||||
ERROR_TEXT = {
|
||||
'user_forced_stop': '操作已强制中断,结果待核验。',
|
||||
'application_shutdown_failed': '业务结果已保留,但应用收尾异常。',
|
||||
"batch_resources_pending": "后台资源尚未确认释放,请使用“重新清理资源”后再开始下一批。",
|
||||
"supervisor_busy": "另一批操作正在执行,请先安全收尾。",
|
||||
"supervisor_interrupted": "批次监督连接异常,保护记录已保留,没有自动重放替换。",
|
||||
|
||||
@@ -23,6 +23,8 @@ class RunPanel(QWidget):
|
||||
self.busy_indicator.hide()
|
||||
self.quick_run_button = QPushButton('执行当前任务', objectName='quickRunButton')
|
||||
self.stop_queue_button = QPushButton('完成当前项后暂停', objectName='stopQueueButton')
|
||||
self.force_stop_button = QPushButton('强制结束…', objectName='forceStopButton')
|
||||
self.force_stop_button.setToolTip('异常情况下使用;会中断当前操作,结果必须重新核验。')
|
||||
self.retry_button = QPushButton('重试失败项', objectName='retryTaskItems')
|
||||
self.recover_button = QPushButton('核验并恢复', objectName='recoverQuickRunButton')
|
||||
self.cleanup_resources_button = QPushButton('重新清理资源', objectName='cleanupBatchResources')
|
||||
@@ -42,7 +44,7 @@ class RunPanel(QWidget):
|
||||
self.toggle_log_button.toggled.connect(self._toggle_log)
|
||||
buttons = QHBoxLayout()
|
||||
for button in (self.quick_run_button, self.retry_button, self.recover_button, self.cleanup_resources_button, self.diagnostics_button,
|
||||
self.new_round_button, self.history_button, self.stop_queue_button):
|
||||
self.new_round_button, self.history_button, self.stop_queue_button, self.force_stop_button):
|
||||
buttons.addWidget(button)
|
||||
buttons.addStretch()
|
||||
buttons.addWidget(self.toggle_log_button)
|
||||
|
||||
@@ -1 +1 @@
|
||||
VERSION = "1.1.4-beta.3"
|
||||
VERSION = "1.1.4-beta.4"
|
||||
@@ -35,6 +35,7 @@ class ProgressWatch:
|
||||
self.done = 0
|
||||
self.total = 0
|
||||
self.safe_stop = True
|
||||
self.waiting_for_safe_completion = False
|
||||
self.started = started
|
||||
self.stage_started = started
|
||||
self.stage_seconds: dict[str, float] = {}
|
||||
@@ -67,6 +68,7 @@ class ProgressWatch:
|
||||
self.native_sequence = 0
|
||||
self.last_progress = now
|
||||
self.stage, self.done, self.total, self.safe_stop = fields[4], done, total, bool(safe)
|
||||
self.waiting_for_safe_completion = False
|
||||
return True
|
||||
|
||||
def accept_native(self, line: bytes, now: float) -> bool:
|
||||
@@ -102,6 +104,11 @@ class ProgressWatch:
|
||||
self.stop_requested = True
|
||||
return 'request_stop'
|
||||
if self.stop_requested and idle >= self.idle_timeout * 2:
|
||||
if not self.safe_stop:
|
||||
if not self.waiting_for_safe_completion:
|
||||
self.waiting_for_safe_completion = True
|
||||
return 'await_safe_completion'
|
||||
return None
|
||||
return 'unresponsive'
|
||||
return None
|
||||
|
||||
@@ -109,7 +116,7 @@ class ProgressWatch:
|
||||
now = time.monotonic() if now is None else now
|
||||
stages = dict(self.stage_seconds)
|
||||
stages[self.stage] = stages.get(self.stage, 0.0) + max(0.0, now - self.stage_started)
|
||||
return {name: getattr(self, name) for name in ('token', 'sequence', 'stage', 'done', 'total', 'safe_stop', 'stop_requested')} | {
|
||||
return {name: getattr(self, name) for name in ('token', 'sequence', 'stage', 'done', 'total', 'safe_stop', 'stop_requested', 'waiting_for_safe_completion')} | {
|
||||
'job_id': self.current_job_id,
|
||||
'native_progress_frames': self.native_progress_frames,
|
||||
'elapsed_seconds': max(0.0, now - self.started), 'stage_seconds': stages}
|
||||
@@ -124,9 +131,8 @@ def _atomic_text(path: Path, text: str) -> None:
|
||||
def _publish_progress(path: Path, snapshot: dict) -> None:
|
||||
try:
|
||||
_atomic_text(path, json.dumps(snapshot, ensure_ascii=False))
|
||||
except OSError as error:
|
||||
if not isinstance(error, PermissionError) and getattr(error, 'winerror', None) not in {5, 32, 33}:
|
||||
raise
|
||||
except OSError:
|
||||
pass # Display telemetry failure must not terminate a possible CAD write.
|
||||
|
||||
|
||||
def _decode(data: bytes) -> str:
|
||||
@@ -190,7 +196,7 @@ class _OwnedWorkers:
|
||||
|
||||
|
||||
def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abort_event=None,
|
||||
process_factory=None, cooperative_abort=False) -> subprocess.CompletedProcess[str]:
|
||||
process_factory=None, cooperative_abort=False, force_event=None, force_control=None) -> subprocess.CompletedProcess[str]:
|
||||
root = Path(command[3]).resolve()
|
||||
job_id = command[-1]
|
||||
token = uuid4().hex
|
||||
@@ -200,8 +206,11 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
|
||||
native_progress_path = root / f'monitor-{token}.native-progress'
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
watch = ProgressWatch(token, job_id, idle_timeout, time.monotonic())
|
||||
if force_control is not None:
|
||||
force_event = force_control.force_event
|
||||
owned = _OwnedWorkers(token, {command[0], command[4]})
|
||||
process = None
|
||||
durable_force_accepted = False
|
||||
threads = []
|
||||
messages = queue.Queue(maxsize=1024)
|
||||
stop_readers = threading.Event()
|
||||
@@ -244,15 +253,32 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
|
||||
_publish_progress(progress_path, watch.snapshot())
|
||||
_atomic_text(ready, token)
|
||||
while process.poll() is None:
|
||||
if force_event is not None and force_event.is_set():
|
||||
from .stop_control import protect_interruption
|
||||
try:
|
||||
protect_interruption(root, 'user_forced_stop')
|
||||
except OSError as error:
|
||||
force_event.clear()
|
||||
if force_control is not None:
|
||||
force_control.control_error = '强制结束未执行,保护记录无法保存:' + str(error)
|
||||
watch.waiting_for_safe_completion = True
|
||||
_publish_progress(progress_path, watch.snapshot())
|
||||
else:
|
||||
durable_force_accepted = True
|
||||
raise WorkerMonitorError('user forced stop; execution requires reconciliation')
|
||||
if abort_event is not None and abort_event.is_set():
|
||||
if not cooperative_abort:
|
||||
if not cooperative_abort and watch.safe_stop:
|
||||
raise WorkerMonitorError('project session supervisor requested worker cleanup')
|
||||
if not watch.stop_requested:
|
||||
# Read-only callers request the VBS safe boundary first.
|
||||
# The existing idle watchdog still bounds a stalled COM call.
|
||||
watch.stop_requested = True
|
||||
watch.last_progress = time.monotonic()
|
||||
_atomic_text(stop, token)
|
||||
try:
|
||||
_atomic_text(stop, token)
|
||||
except OSError:
|
||||
if watch.safe_stop:
|
||||
raise
|
||||
_publish_progress(progress_path, watch.snapshot())
|
||||
try:
|
||||
receive(*messages.get(timeout=min(0.1, idle_timeout / 10)))
|
||||
@@ -273,7 +299,17 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
|
||||
break
|
||||
action = watch.action(time.monotonic())
|
||||
if action == 'request_stop':
|
||||
_atomic_text(stop, token)
|
||||
try:
|
||||
_atomic_text(stop, token)
|
||||
except OSError:
|
||||
if watch.safe_stop:
|
||||
raise
|
||||
_publish_progress(progress_path, watch.snapshot())
|
||||
elif action == 'await_safe_completion':
|
||||
if cooperative_abort:
|
||||
# This entry point is explicitly read-only. A blocked COM
|
||||
# read has no write to finish and retains its bounded cancel.
|
||||
raise subprocess.TimeoutExpired(command, idle_timeout * 2)
|
||||
_publish_progress(progress_path, watch.snapshot())
|
||||
elif action == 'unresponsive':
|
||||
raise subprocess.TimeoutExpired(command, idle_timeout * 2)
|
||||
@@ -287,6 +323,15 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
|
||||
return subprocess.CompletedProcess(command, process.returncode, _decode(output[0]), _decode(output[1]))
|
||||
except Exception as error:
|
||||
if process is not None:
|
||||
if (not watch.safe_stop and not cooperative_abort and force_control is not None
|
||||
and not durable_force_accepted):
|
||||
from .stop_control import wait_for_resolution
|
||||
try:
|
||||
wait_for_resolution(force_control, root, error)
|
||||
except WorkerMonitorError:
|
||||
if not force_event.is_set():
|
||||
raise
|
||||
durable_force_accepted = True
|
||||
failures = owned.stop_workers(process)
|
||||
process.wait(timeout=10)
|
||||
for thread in threads:
|
||||
|
||||
@@ -175,13 +175,13 @@ Set QP_App=FakeApp
|
||||
FakeApp.ProjectPath=Target
|
||||
QP_CurrentJobId="batch-job": QP_ExpectedJobId="batch-job"
|
||||
QPWriteUtf8 QP_Fso.BuildPath(QP_Root,"command.tsv"),"task_id" & vbTab & "job_id" & vbTab & "project_path" & vbTab & "old_part" & vbTab & "new_part" & vbTab & "value_mode" & vbCrLf & "task" & vbTab & "batch-job" & vbTab & Target & vbTab & "OLD" & vbTab & "NEW" & vbTab & "preserve" & vbCrLf
|
||||
QP_BatchMode=True: QP_OwnedProject=True
|
||||
QP_BatchMode=True: QP_OwnedProject=True: QP_OwnedProjectPath=Target
|
||||
QPWriteStatus QP_Root,QP_CurrentJobId,"no_match","","not found"
|
||||
Require Not QP_Fso.FileExists(QP_Fso.BuildPath(QP_Root,"status.tsv")),"published success before close"
|
||||
Require QPReadState(QP_Fso.BuildPath(QP_Root,"status.tsv"),QP_CurrentJobId)="no_match","pending no-match not visible to worker"
|
||||
QPFinishBatchItem
|
||||
Require FakeApp.CloseCount=1,"project not closed exactly once"
|
||||
Require FakeApp.SaveCount=1,"sheet not closed with saving allowed"
|
||||
Require FakeApp.SaveCount=0,"no-match must close without saving"
|
||||
Require InStr(QPReadUtf8(QP_Fso.BuildPath(QP_Root,"status.tsv")),"batch_closed")>0,"no completion proof"
|
||||
''')
|
||||
|
||||
@@ -368,8 +368,8 @@ Require FakeApp.ProjectPath = "", "discovery left the project open"
|
||||
Require FakeApp.Documents.OpenCount = 0, "discovery opened a page"
|
||||
Dim Manifest
|
||||
Manifest = QPReadUtf8(QP_Fso.BuildPath(QP_Root, "sheets.tsv"))
|
||||
Require InStr(Manifest, Target & vbTab & "Main" & vbTab & "1") > 0, "sheet one missing"
|
||||
Require InStr(Manifest, Target & vbTab & "Main" & vbTab & "2") > 0, "sheet two missing"
|
||||
Require InStr(Manifest, LCase(Target) & vbTab & "Main" & vbTab & "1") > 0, "sheet one missing"
|
||||
Require InStr(Manifest, LCase(Target) & vbTab & "Main" & vbTab & "2") > 0, "sheet two missing"
|
||||
''')
|
||||
|
||||
|
||||
|
||||
@@ -253,6 +253,31 @@ End Sub''', routines)
|
||||
'Sub QPVerifyUntouched(ByRef Records)\n' + injection)
|
||||
|
||||
|
||||
def test_save_then_project_close_refusal_retains_save_boundary(tmp_path):
|
||||
run_vbs(tmp_path, 'part_number', '''
|
||||
Dim FullChecks
|
||||
FullChecks=0
|
||||
Command.Add "project_path",QPProjectPath()
|
||||
QP_BatchMode=True: QP_SessionMode=True
|
||||
QP_SessionToken=String(32,"b")
|
||||
QP_OwnedProject=True: QP_OwnedProjectPath=QPProjectPath()
|
||||
QPRestoreAfterOperatorReplace Command,SnapshotPath
|
||||
QP_App.RefuseClose=True
|
||||
On Error Resume Next
|
||||
QPFinishBatchItem
|
||||
Failure=Err.Number
|
||||
Err.Clear
|
||||
On Error GoTo 0
|
||||
Require Failure<>0,"refused close must stop the operation"
|
||||
Require QP_BatchSaveCommitted,"returned save was lost when project close failed"
|
||||
QP_App.RefuseClose=False
|
||||
QPDiscardFailedBatchProject CloseError
|
||||
Require Len(CloseError)>0,"uncertain save must not discard the open project"
|
||||
Require Len(QPProjectPath())>0,"failed saved project was silently discarded"
|
||||
Require Not QP_Fso.FileExists(QP_Fso.BuildPath(QP_Root,"project-session-result.tsv")),"failure published success"
|
||||
''', routines_transform=_batch_readback_fixture)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('fault', ['', 'before_save', 'after_reopen'])
|
||||
def test_owned_batch_checks_full_state_at_save_and_disk_boundaries(tmp_path, fault):
|
||||
run_vbs(tmp_path, 'part_number', '''
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import weakref
|
||||
from pathlib import Path
|
||||
from threading import Event
|
||||
from threading import Timer
|
||||
from threading import Thread
|
||||
from types import SimpleNamespace
|
||||
from traceback import extract_tb
|
||||
|
||||
@@ -194,6 +196,121 @@ def test_clean_release_records_helper_and_cad_exit_codes(monkeypatch, tmp_path):
|
||||
assert record['cleanup_errors'] == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize('persistent', [False, True])
|
||||
def test_closing_journal_failure_still_quits_and_reports_diagnostic_error(monkeypatch, tmp_path, persistent):
|
||||
events, cad = fake_desktop(monkeypatch, tmp_path)
|
||||
session = runtime.IsolatedCadSession(tmp_path)
|
||||
session.__enter__()
|
||||
record = session._record
|
||||
def failing_record(**values):
|
||||
if persistent or values.get('state') == 'closing':
|
||||
raise PermissionError('closing journal locked')
|
||||
record(**values)
|
||||
session._record = failing_record
|
||||
with pytest.raises(WorkerMonitorError, match='closing journal locked'):
|
||||
session.__exit__(None, None, None)
|
||||
assert 'quit' in events and cad.returncode == 0
|
||||
assert session.record['orderly_shutdown'] is True
|
||||
assert session.record['clean'] is False
|
||||
assert session.record['diagnostic_errors'][0]['stage'] == 'closing_record'
|
||||
assert (tmp_path / 'uncertain.flag').exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize('helper_failure', ['refused', 'timeout'])
|
||||
def test_interactive_quit_failure_retains_cad_and_helper_until_explicit_force(monkeypatch, tmp_path, helper_failure):
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
from dx_productivity.trial_task_center import worker_monitor
|
||||
events, cad = fake_desktop(monkeypatch, tmp_path, quit_code=1, cad_exit_code=None)
|
||||
control = StopControl()
|
||||
session = runtime.IsolatedCadSession(tmp_path, stop_control=control)
|
||||
session.__enter__()
|
||||
start = session.desktop.start_process
|
||||
def start_quit(command, **options):
|
||||
helper = start(command, **options)
|
||||
if command[0] == 'quit':
|
||||
if helper_failure == 'timeout':
|
||||
helper.wait = lambda timeout: (_ for _ in ()).throw(subprocess.TimeoutExpired('quit', timeout))
|
||||
helper.poll = lambda: None
|
||||
helper.close = lambda: events.append('close-quit-helper')
|
||||
return helper
|
||||
session.desktop.start_process = start_quit
|
||||
waiting, finished = Event(), Event()
|
||||
publish = worker_monitor._publish_progress
|
||||
def observe_wait(path, snapshot):
|
||||
publish(path, snapshot)
|
||||
if snapshot.get('stage') == 'awaiting_reconciliation':
|
||||
waiting.set()
|
||||
monkeypatch.setattr(worker_monitor, '_publish_progress', observe_wait)
|
||||
failures = []
|
||||
def close_session():
|
||||
try:
|
||||
session.__exit__(None, None, None)
|
||||
except Exception as failure:
|
||||
failures.append(failure)
|
||||
finally:
|
||||
finished.set()
|
||||
thread = Thread(target=close_session)
|
||||
thread.start()
|
||||
try:
|
||||
assert waiting.wait(1), 'failed Quit must enter protected waiting before resource release'
|
||||
control.set()
|
||||
assert not finished.wait(.05)
|
||||
assert cad.poll() is None
|
||||
assert 'close-quit-helper' not in events
|
||||
assert not any(isinstance(event, tuple) and event[0] == 'release' for event in events)
|
||||
assert (tmp_path / 'uncertain.flag').exists()
|
||||
control.request_force()
|
||||
assert finished.wait(2)
|
||||
assert 'close-quit-helper' in events and ('release', True) in events
|
||||
assert cad.poll() == 1
|
||||
assert isinstance(failures[0], WorkerMonitorError)
|
||||
assert 'forced' in str(failures[0])
|
||||
assert session.record['orderly_shutdown'] is False and session.record['clean'] is False
|
||||
finally:
|
||||
control.request_force()
|
||||
thread.join(3)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('failure', ['cad_exited', 'diagnostic'])
|
||||
def test_interactive_exit_does_not_hold_completed_cad_for_resolution(monkeypatch, tmp_path, failure):
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
from dx_productivity.trial_task_center import stop_control
|
||||
events, _cad = fake_desktop(monkeypatch, tmp_path, quit_code=1 if failure == 'cad_exited' else 0)
|
||||
session = runtime.IsolatedCadSession(tmp_path, stop_control=StopControl())
|
||||
session.__enter__()
|
||||
monkeypatch.setattr(stop_control, 'wait_for_resolution', lambda *_args: pytest.fail('exited CAD must not wait'))
|
||||
if failure == 'diagnostic':
|
||||
record = session._record
|
||||
def failed_journal(**values):
|
||||
if values.get('state') == 'closing':
|
||||
raise PermissionError('closing diagnostic unavailable')
|
||||
record(**values)
|
||||
session._record = failed_journal
|
||||
with pytest.raises(WorkerMonitorError):
|
||||
session.__exit__(None, None, None)
|
||||
assert ('release', True) in events
|
||||
|
||||
|
||||
def test_forced_descendant_cleanup_is_not_reported_as_clean_orderly_shutdown(monkeypatch, tmp_path):
|
||||
fake_desktop(monkeypatch, tmp_path)
|
||||
session = runtime.IsolatedCadSession(tmp_path)
|
||||
session.__enter__()
|
||||
release = session.desktop.close
|
||||
def forced_release():
|
||||
release()
|
||||
session.desktop.cleanup_record = {'processes': [{'pid': 123, 'forced_termination': True,
|
||||
'forced_root_termination': False, 'termination_reason': 'descendant_grace_expired',
|
||||
'forced_descendant_pids': [456]}]}
|
||||
session.desktop.close = forced_release
|
||||
with pytest.raises(WorkerMonitorError, match='forced'):
|
||||
session.__exit__(None, None, None)
|
||||
record = json.loads((tmp_path / 'private-desktop.json').read_text())
|
||||
assert record['clean'] is False and record['forced_termination'] is True
|
||||
assert record['orderly_shutdown'] is True
|
||||
assert record['forced_descendant_pids'] == [456]
|
||||
assert record['termination_reason'] == 'descendant_grace_expired'
|
||||
|
||||
|
||||
def test_exit_code_diagnostics_do_not_interrupt_the_normal_cad_wait(monkeypatch, tmp_path):
|
||||
_, cad = fake_desktop(monkeypatch, tmp_path)
|
||||
waited = []
|
||||
|
||||
@@ -367,6 +367,99 @@ def test_descendant_handles_are_owned_before_terminating_the_job():
|
||||
assert ('close', 'child') in events
|
||||
|
||||
|
||||
@pytest.mark.parametrize('child_exit_after', [0.5, None])
|
||||
def test_completed_root_gives_owned_descendants_bounded_natural_exit_grace(monkeypatch, child_exit_after):
|
||||
from dx_productivity.trial_task_center import isolated_desktop as module
|
||||
api, events, state, *_ = native_api()
|
||||
clock = [0.0]
|
||||
monkeypatch.setattr(module.time, 'monotonic', lambda: clock[0])
|
||||
monkeypatch.setattr(module.time, 'sleep', lambda seconds: clock.__setitem__(0, clock[0] + seconds))
|
||||
state['running'] = False
|
||||
child = Handle('child', events)
|
||||
child_running = lambda: not any(event[0] == 'terminate' for event in events) and (
|
||||
child_exit_after is None or clock[0] < child_exit_after)
|
||||
api.job.JobObjectBasicProcessIdList = 3
|
||||
api.job.QueryInformationJobObject = lambda job, kind: (
|
||||
{'BasicLimitInformation': {'LimitFlags': 0}} if kind == 9 else
|
||||
([43] if child_running() else []) if kind == 3 else {'ActiveProcesses': int(child_running())})
|
||||
api.open_process = lambda *_args: child
|
||||
root_wait = api.event.WaitForSingleObject
|
||||
api.event.WaitForSingleObject = lambda handle, timeout: (
|
||||
258 if child_running() else 0) if handle is child else root_wait(handle, timeout)
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
process.close()
|
||||
assert child.closed
|
||||
if child_exit_after is not None:
|
||||
assert child_exit_after <= clock[0] < 2
|
||||
assert ('terminate',) not in events
|
||||
assert process.cleanup_record['forced_termination'] is False
|
||||
assert process.cleanup_record['forced_descendant_pids'] == []
|
||||
else:
|
||||
assert 2 <= clock[0] < 2.1
|
||||
assert ('terminate',) in events
|
||||
assert process.cleanup_record['forced_termination'] is True
|
||||
assert process.cleanup_record['forced_root_termination'] is False
|
||||
assert process.cleanup_record['forced_descendant_pids'] == [43]
|
||||
assert process.cleanup_record['termination_reason'] == 'descendant_grace_expired'
|
||||
|
||||
|
||||
def test_completed_root_without_descendants_has_no_grace_delay(monkeypatch):
|
||||
from dx_productivity.trial_task_center import isolated_desktop as module
|
||||
api, events, state, *_ = native_api()
|
||||
state['running'] = False
|
||||
monkeypatch.setattr(module.time, 'sleep', lambda seconds: pytest.fail('empty job must not wait'))
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
process.close()
|
||||
assert ('terminate',) not in events
|
||||
assert process.cleanup_record['forced_termination'] is False
|
||||
|
||||
|
||||
def test_active_root_cleanup_is_forced_without_descendant_grace(monkeypatch):
|
||||
from dx_productivity.trial_task_center import isolated_desktop as module
|
||||
api, events, *_ = native_api()
|
||||
monkeypatch.setattr(module.time, 'sleep', lambda seconds: pytest.fail('active root must be bounded immediately'))
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
process.close()
|
||||
assert process.cleanup_record['forced_termination'] is True
|
||||
assert process.cleanup_record['forced_root_termination'] is True
|
||||
assert process.cleanup_record['termination_reason'] == 'root_process_active'
|
||||
|
||||
|
||||
def test_native_poll_error_still_terminates_owned_root_and_records_force():
|
||||
api, events, state, *_ = native_api()
|
||||
wait = api.event.WaitForSingleObject
|
||||
def failing_poll(handle, timeout):
|
||||
if timeout == 0 and state['running']:
|
||||
raise OSError('native poll failed')
|
||||
return wait(handle, timeout)
|
||||
api.event.WaitForSingleObject = failing_poll
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
with pytest.raises(TaskCenterError, match='native poll failed'):
|
||||
process.close()
|
||||
assert state['running'] is False
|
||||
assert process.cleanup_record['forced_root_termination'] is True
|
||||
|
||||
|
||||
def test_failed_job_cleanup_retains_ownership_until_retry(monkeypatch):
|
||||
from dx_productivity.trial_task_center import isolated_desktop as module
|
||||
api, events, state, root, job = native_api()
|
||||
clock = [0.0]
|
||||
monkeypatch.setattr(module.time, 'monotonic', lambda: clock[0])
|
||||
monkeypatch.setattr(module.time, 'sleep', lambda seconds: clock.__setitem__(0, clock[0] + seconds))
|
||||
api.job.TerminateJobObject = lambda *_args: (_ for _ in ()).throw(OSError('job termination failed'))
|
||||
query = api.job.QueryInformationJobObject
|
||||
child_running = [True]
|
||||
api.job.QueryInformationJobObject = lambda handle, kind: (
|
||||
{'ActiveProcesses': int(state['running']) + int(child_running[0])} if kind == 1 else query(handle, kind))
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
with pytest.raises(TaskCenterError, match='descendants did not exit'):
|
||||
process.close()
|
||||
assert not job.closed and process.job is job
|
||||
child_running[0] = False
|
||||
process.close()
|
||||
assert job.closed
|
||||
|
||||
|
||||
@pytest.mark.parametrize('error_code', [87, 5])
|
||||
def test_native_descendant_open_error_only_ignores_already_exited_process(error_code):
|
||||
# pywintypes.error derives directly from Exception, rather than OSError.
|
||||
@@ -391,6 +484,30 @@ def test_native_descendant_open_error_only_ignores_already_exited_process(error_
|
||||
assert caught.value is error
|
||||
|
||||
|
||||
def test_forced_job_preserves_partial_current_descendant_snapshot_after_open_failure():
|
||||
api, events, state, *_ = native_api()
|
||||
api.job.JobObjectBasicProcessIdList = 3
|
||||
query = api.job.QueryInformationJobObject
|
||||
snapshot = [45]
|
||||
api.job.QueryInformationJobObject = lambda job, kind: list(snapshot) if kind == 3 else query(job, kind)
|
||||
def open_child(_access, _inherit, pid):
|
||||
if pid == 44:
|
||||
raise OSError(5, 'child access denied')
|
||||
return Handle('child-' + str(pid), events)
|
||||
api.open_process = open_child
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
# PID 45 was observed earlier but is no longer part of the active Job.
|
||||
process._remember_descendants()
|
||||
snapshot[:] = [42, 43, 44]
|
||||
with pytest.raises(TaskCenterError, match='child access denied'):
|
||||
process.close()
|
||||
assert state['running'] is False
|
||||
assert process.cleanup_record['forced_termination'] is True
|
||||
assert process.cleanup_record['forced_descendant_pids'] == [43]
|
||||
assert process.cleanup_record['descendant_pids'] == [43, 45]
|
||||
assert process.cleanup_record['job_handle_closed'] is True
|
||||
|
||||
|
||||
def test_unconfirmed_descendant_exit_retains_handle_for_cleanup_retry():
|
||||
api, events, state, *_ = native_api()
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
@@ -430,9 +547,37 @@ def test_native_private_job_kills_descendant_after_root_exit(parent_exits):
|
||||
if parent_exits:
|
||||
assert process.wait(timeout=5) == 0
|
||||
assert win32event.WaitForSingleObject(child_handle, 0) == 0
|
||||
assert process.cleanup_record['forced_termination'] is True
|
||||
assert process.cleanup_record['forced_root_termination'] is (not parent_exits)
|
||||
assert child_pid in process.cleanup_record['forced_descendant_pids']
|
||||
with pytest.raises(pywintypes.error):
|
||||
unexpected = win32service.OpenDesktop(name, 0, False, 1)
|
||||
unexpected.CloseDesktop()
|
||||
finally:
|
||||
if child_handle is not None:
|
||||
child_handle.Close()
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name != 'nt', reason='native Windows descendant shutdown grace')
|
||||
def test_native_private_job_allows_descendant_to_finish_after_root_exit():
|
||||
import win32api
|
||||
import win32event
|
||||
import win32process
|
||||
|
||||
child_handle = None
|
||||
try:
|
||||
with PrivateDesktop('dx-test-' + uuid4().hex, enabled=True) as desktop:
|
||||
process = desktop.start_process((sys.executable, '-c',
|
||||
"import subprocess,sys; p=subprocess.Popen([sys.executable,'-c','import time;time.sleep(0.5)']); print(p.pid,flush=True)"),
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
child_pid = int(process.stdout.readline())
|
||||
child_handle = win32api.OpenProcess(0x100000 | 0x1000, False, child_pid)
|
||||
assert process.wait(timeout=5) == 0
|
||||
assert win32event.WaitForSingleObject(child_handle, 0) == 0
|
||||
assert win32process.GetExitCodeProcess(child_handle) == 0
|
||||
assert process.cleanup_record['forced_termination'] is False
|
||||
assert process.cleanup_record['forced_descendant_pids'] == []
|
||||
assert desktop.cleanup_record['desktop_confirmation']['released'] is True
|
||||
finally:
|
||||
if child_handle is not None:
|
||||
child_handle.Close()
|
||||
@@ -0,0 +1,75 @@
|
||||
from threading import Event
|
||||
|
||||
from PySide6.QtGui import QCloseEvent
|
||||
from PySide6.QtWidgets import QMessageBox
|
||||
|
||||
from test_ui_remediation import window
|
||||
from test_ui_batch_retry import open_project
|
||||
|
||||
|
||||
def test_pause_and_window_close_never_request_force(window, qtbot, tmp_path):
|
||||
open_project(window, qtbot, tmp_path)
|
||||
window._begin_run(window._task_id)
|
||||
worker = object()
|
||||
window._workers.add(worker)
|
||||
try:
|
||||
event = QCloseEvent()
|
||||
window.closeEvent(event)
|
||||
assert not event.isAccepted()
|
||||
assert window._batch_stop.is_set()
|
||||
assert not window._batch_stop.force_event.is_set()
|
||||
assert not window._closed
|
||||
finally:
|
||||
window._workers.discard(worker)
|
||||
window._close_pending = False
|
||||
window._finish_run()
|
||||
|
||||
|
||||
def test_force_requires_confirmation_and_keeps_window_for_reconciliation(window, qtbot, tmp_path, monkeypatch):
|
||||
open_project(window, qtbot, tmp_path)
|
||||
window._begin_run(window._task_id)
|
||||
control = window._batch_stop
|
||||
try:
|
||||
monkeypatch.setattr(QMessageBox, 'question', lambda *args: QMessageBox.StandardButton.No)
|
||||
window._force_batch_stop()
|
||||
assert not control.is_set() and not control.force_event.is_set()
|
||||
monkeypatch.setattr(QMessageBox, 'question', lambda *args: QMessageBox.StandardButton.Yes)
|
||||
window._close_pending = True
|
||||
window._force_batch_stop()
|
||||
assert control.is_set() and control.force_event.is_set()
|
||||
assert not window._close_pending
|
||||
finally:
|
||||
window._finish_run()
|
||||
|
||||
|
||||
def test_uncertain_wait_is_visible_in_execution_feedback(tmp_path):
|
||||
import json
|
||||
from dx_productivity.trial_task_center.execution_feedback import read_feedback
|
||||
from dx_productivity.trial_task_center.ui.execution_feedback import format_feedback
|
||||
root = tmp_path / 'quick-run'
|
||||
root.mkdir()
|
||||
(root / 'worker-progress.json').write_text(json.dumps({
|
||||
'stage': 'save_and_close', 'waiting_for_safe_completion': True, 'stage_seconds': {}}))
|
||||
feedback = read_feedback('task', 1, 1, '', tmp_path, 0)
|
||||
text = format_feedback(feedback, 'board', 400, True)
|
||||
assert '尚未确认' in text and '强制' in text
|
||||
|
||||
|
||||
def test_diagnostic_export_during_wait_does_not_finish_or_cancel_execution(window, qtbot, tmp_path, monkeypatch):
|
||||
from PySide6.QtWidgets import QFileDialog
|
||||
open_project(window, qtbot, tmp_path)
|
||||
window._begin_run(window._task_id)
|
||||
control = window._batch_stop
|
||||
owner = object()
|
||||
window._workers.add(owner)
|
||||
destination = tmp_path / 'support.zip'
|
||||
monkeypatch.setattr(QFileDialog, 'getSaveFileName', lambda *args: (str(destination), ''))
|
||||
try:
|
||||
window._export_diagnostics()
|
||||
qtbot.waitUntil(lambda: window._diagnostic_worker is None, timeout=5000)
|
||||
assert destination.is_file()
|
||||
assert owner in window._workers and window._batch_stop is control
|
||||
assert not control.is_set() and not control.force_event.is_set()
|
||||
finally:
|
||||
window._workers.discard(owner)
|
||||
window._finish_run()
|
||||
@@ -0,0 +1,343 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
from threading import Event, Thread
|
||||
|
||||
import pytest
|
||||
|
||||
from dx_productivity.trial_task_center.worker_monitor import ProgressWatch, run_monitored_process
|
||||
|
||||
|
||||
def test_unsafe_save_timeout_waits_for_completion_and_recovers_on_progress():
|
||||
watch = ProgressWatch('token', 'job', 10, 0)
|
||||
assert watch.accept(b'DX_PROGRESS\ttoken\tjob\t1\tsave_and_close\t0\t1\t0\n', 0)
|
||||
assert watch.action(10) == 'request_stop'
|
||||
assert watch.action(20) == 'await_safe_completion'
|
||||
assert watch.snapshot(now=20)['waiting_for_safe_completion'] is True
|
||||
assert watch.action(200) != 'unresponsive'
|
||||
assert watch.accept(b'DX_PROGRESS\ttoken\tjob\t2\tverify_saved_project\t1\t1\t0\n', 201)
|
||||
assert watch.snapshot(now=201)['waiting_for_safe_completion'] is False
|
||||
|
||||
|
||||
def test_real_slow_save_is_not_killed_by_inactivity(tmp_path):
|
||||
script = tmp_path / 'save.py'
|
||||
script.write_text('''import sys,time,pathlib
|
||||
root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5]
|
||||
while not (root/f'monitor-{token}.ready').exists():time.sleep(.01)
|
||||
print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tsave_and_close\\t0\\t1\\t0',flush=True)
|
||||
time.sleep(1.5)
|
||||
(root/'saved.txt').write_text('saved and independently verified')
|
||||
''', encoding='utf-8')
|
||||
command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job')
|
||||
result = run_monitored_process(command, .3)
|
||||
assert result.returncode == 0
|
||||
assert (tmp_path / 'saved.txt').read_text() == 'saved and independently verified'
|
||||
|
||||
|
||||
def test_readonly_native_wait_keeps_bounded_cancellation_without_force_ui(tmp_path):
|
||||
script = tmp_path / 'read.py'
|
||||
script.write_text('''import sys,time,pathlib
|
||||
root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5]
|
||||
while not (root/f'monitor-{token}.ready').exists():time.sleep(.01)
|
||||
print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tnative_snapshot_read\\t0\\t1\\t0',flush=True)
|
||||
time.sleep(1.5)
|
||||
''', encoding='utf-8')
|
||||
command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job')
|
||||
with pytest.raises(subprocess.TimeoutExpired):
|
||||
run_monitored_process(command, .3, cooperative_abort=True)
|
||||
|
||||
|
||||
def test_explicit_force_stops_unsafe_worker_with_protection(tmp_path):
|
||||
script = tmp_path / 'save.py'
|
||||
script.write_text('''import sys,time,pathlib
|
||||
root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5]
|
||||
while not (root/f'monitor-{token}.ready').exists():time.sleep(.01)
|
||||
print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tsave_and_close\\t0\\t1\\t0',flush=True)
|
||||
(root/'started').touch()
|
||||
time.sleep(30)
|
||||
(root/'unexpected_completed').touch()
|
||||
''', encoding='utf-8')
|
||||
force = Event()
|
||||
def request():
|
||||
for _ in range(200):
|
||||
if (tmp_path / 'started').exists():
|
||||
force.set()
|
||||
return
|
||||
Event().wait(.01)
|
||||
requester = Thread(target=request)
|
||||
requester.start()
|
||||
command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job')
|
||||
try:
|
||||
with pytest.raises(Exception, match='forced'):
|
||||
run_monitored_process(command, .3, force_event=force)
|
||||
assert (tmp_path / 'uncertain.flag').exists()
|
||||
assert not (tmp_path / 'unexpected_completed').exists()
|
||||
finally:
|
||||
requester.join(3)
|
||||
|
||||
|
||||
def test_abnormal_cad_shutdown_is_not_hidden_by_physical_release(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import _finish_response
|
||||
from dx_productivity.trial_task_center.batch_resources import write_record
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.supervisor_protocol import encode, encode_error
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
root = tmp_path
|
||||
journal = {'operation': 'probe', 'batch_id': 'a' * 32, 'state': 'blocked'}
|
||||
write_record(root / 'batch-resources' / journal['batch_id'] / 'session.json', {
|
||||
'version': 1, 'session_id': 'a' * 32, 'kind': 'probe', 'state': 'released',
|
||||
'generation': 1, 'desktop': 'dx-' + 'b' * 32, 'processes': [], 'cad_pid': None,
|
||||
'cad_guid': None, 'checkpoints': {}, 'retries': {}, 'timings': [],
|
||||
'cad_shutdown': {'clean': False, 'cleanup_errors': [{'stage': 'orderly_shutdown', 'message': 'Quit timed out'}]}})
|
||||
response = {'business_succeeded': True, 'status': 'error', 'result': encode({'verified': True}),
|
||||
'error': encode_error(TaskCenterError('batch_resources_pending', 'Quit timed out'))}
|
||||
with pytest.raises(TaskCenterError) as caught:
|
||||
_finish_response(root, journal, response, configure_logging(root))
|
||||
assert caught.value.code == 'application_shutdown_failed'
|
||||
assert caught.value.completed_result == {'verified': True}
|
||||
assert journal['resource_recheck']['pending'] is False
|
||||
|
||||
|
||||
def test_force_cannot_terminate_before_interruption_evidence_is_durable(tmp_path, monkeypatch):
|
||||
from types import SimpleNamespace
|
||||
from dx_productivity.trial_task_center import batch_supervisor as module
|
||||
calls = []
|
||||
process = SimpleNamespace(terminate=lambda: calls.append('terminate'))
|
||||
journal = {'operation': 'execute_task_queue', 'batch_id': 'a' * 32, 'attempt_id': 'b' * 32,
|
||||
'task_ids': ['task'], 'state': 'running'}
|
||||
monkeypatch.setattr(module, '_mark_interrupted', lambda *args: (_ for _ in ()).throw(PermissionError('denied')))
|
||||
with pytest.raises(PermissionError):
|
||||
module._force_owned_supervisor(tmp_path, journal, process)
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_force_already_performed_remains_forced_when_exit_diagnostics_fail(tmp_path):
|
||||
from types import SimpleNamespace
|
||||
from dx_productivity.trial_task_center.batch_supervisor import _force_owned_supervisor
|
||||
record = {}
|
||||
def terminate():
|
||||
record['forced_termination'] = True
|
||||
raise OSError('child enumeration failed after force')
|
||||
process = SimpleNamespace(terminate=terminate, cleanup_record=record)
|
||||
journal = {'operation': 'execute_task_queue', 'batch_id': 'a' * 32, 'attempt_id': 'b' * 32,
|
||||
'task_ids': ['task'], 'state': 'running'}
|
||||
_force_owned_supervisor(tmp_path, journal, process)
|
||||
saved = json.loads((tmp_path / 'batch-resources' / ('a' * 32) / 'supervisor.json').read_text(encoding='utf-8'))
|
||||
assert saved['stop_mode'] == 'forced'
|
||||
assert 'enumeration' in saved['force_cleanup_error']
|
||||
|
||||
|
||||
def test_interruption_targets_active_project_without_invalidating_previous_tasks(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import _mark_interrupted
|
||||
from dx_productivity.trial_task_center.batch_resources import write_record
|
||||
batch = 'a' * 32
|
||||
write_record(tmp_path / 'batch-resources' / batch / 'session.json', {
|
||||
'version': 1, 'session_id': batch, 'kind': 'task_queue', 'state': 'ready',
|
||||
'generation': 1, 'desktop': 'dx-' + 'b' * 32, 'processes': [], 'cad_pid': 123,
|
||||
'cad_guid': 'owned', 'checkpoints': {}, 'retries': {}, 'timings': [], 'active_task': 'current'})
|
||||
journal = {'operation': 'execute_task_queue', 'batch_id': batch, 'attempt_id': 'c' * 32,
|
||||
'task_ids': ['previous', 'current'], 'state': 'running'}
|
||||
_mark_interrupted(tmp_path, journal, 'forced stop')
|
||||
assert (tmp_path / 'tasks/current/quick-run/uncertain.flag').exists()
|
||||
assert not (tmp_path / 'tasks/previous/quick-run/uncertain.flag').exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize('interactive', [False, True])
|
||||
def test_supervisor_only_holds_for_resolution_when_caller_has_force_control(tmp_path, monkeypatch, interactive):
|
||||
import textwrap
|
||||
from dx_productivity.trial_task_center import batch_supervisor as module
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
bootstrap = textwrap.dedent('''
|
||||
import sys,json
|
||||
from pathlib import Path
|
||||
from dx_productivity.trial_task_center import batch_worker,batch_runtime
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
original=batch_runtime.BatchCadRuntime
|
||||
def runtime(*args,**kwargs):
|
||||
result=original(*args,**kwargs)
|
||||
Path(sys.argv[1]).with_name('interactive-witness.json').write_text(json.dumps(isinstance(result.stop_event,StopControl)))
|
||||
return result
|
||||
batch_runtime.BatchCadRuntime=runtime
|
||||
batch_worker.run(Path(sys.argv[1]))
|
||||
''')
|
||||
monkeypatch.setattr(module, '_command', lambda request: (sys.executable, '-c', bootstrap, str(request)))
|
||||
store = TaskStore(tmp_path / 'tasks.db')
|
||||
try:
|
||||
service = TaskCenterService(store, configure_logging(tmp_path))
|
||||
module.run_supervised(service, 'probe', {}, stop_event=StopControl() if interactive else Event())
|
||||
witness = next((tmp_path / 'batch-resources').glob('*/interactive-witness.json'))
|
||||
assert json.loads(witness.read_text()) is interactive
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
def test_uncertain_operation_keeps_owner_until_explicit_force(tmp_path):
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl, wait_for_resolution
|
||||
control = StopControl()
|
||||
failures = []
|
||||
def owner():
|
||||
try:
|
||||
wait_for_resolution(control, tmp_path, 'save result unknown')
|
||||
except Exception as error:
|
||||
failures.append(error)
|
||||
thread = Thread(target=owner)
|
||||
thread.start()
|
||||
try:
|
||||
for _ in range(100):
|
||||
if (tmp_path / 'worker-progress.json').exists():
|
||||
break
|
||||
Event().wait(.01)
|
||||
control.set() # Ordinary stop never authorizes discarding a changed project.
|
||||
thread.join(.15)
|
||||
assert thread.is_alive()
|
||||
assert (tmp_path / 'uncertain.flag').exists()
|
||||
assert json.loads((tmp_path / 'worker-progress.json').read_text())['waiting_for_safe_completion']
|
||||
control.request_force()
|
||||
thread.join(3)
|
||||
assert not thread.is_alive()
|
||||
assert 'forced' in str(failures[0])
|
||||
finally:
|
||||
control.request_force()
|
||||
thread.join(3)
|
||||
|
||||
|
||||
def test_failed_protection_fsync_is_retried_before_force_is_allowed(tmp_path, monkeypatch):
|
||||
from dx_productivity.trial_task_center import stop_control as module
|
||||
control = module.StopControl()
|
||||
original = module.os.fsync
|
||||
attempts, failures = [], []
|
||||
def denied(fd):
|
||||
attempts.append(fd)
|
||||
raise OSError('evidence fsync denied')
|
||||
monkeypatch.setattr(module.os, 'fsync', denied)
|
||||
def owner():
|
||||
try:
|
||||
module.wait_for_resolution(control, tmp_path, 'save unknown')
|
||||
except Exception as error:
|
||||
failures.append(error)
|
||||
thread = Thread(target=owner)
|
||||
thread.start()
|
||||
try:
|
||||
for _ in range(100):
|
||||
if control.control_error:
|
||||
break
|
||||
Event().wait(.01)
|
||||
assert control.control_error
|
||||
control.request_force()
|
||||
thread.join(.25)
|
||||
assert thread.is_alive(), 'a file left by failed fsync must not authorize force'
|
||||
assert len(attempts) >= 2
|
||||
monkeypatch.setattr(module.os, 'fsync', original)
|
||||
control.request_force()
|
||||
thread.join(3)
|
||||
assert failures and 'forced' in str(failures[0])
|
||||
finally:
|
||||
monkeypatch.setattr(module.os, 'fsync', original)
|
||||
control.request_force()
|
||||
thread.join(3)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('deny_first,transport_failure', [(False, False), (True, False), (False, True)])
|
||||
def test_real_supervisor_force_preserves_protection_and_never_replays(tmp_path, monkeypatch, deny_first, transport_failure):
|
||||
import os
|
||||
import textwrap
|
||||
import time
|
||||
import win32api
|
||||
import win32event
|
||||
from dx_productivity.trial_task_center import batch_supervisor as module
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
store = TaskStore(tmp_path / 'tasks.db')
|
||||
service = TaskCenterService(store, configure_logging(tmp_path), supervised=True)
|
||||
task = store.create_task('force fixture')
|
||||
evidence = service._project_task_service(task.task_id)._quick_run_preparer.data_root / 'quick-run'
|
||||
witness = evidence / 'started'
|
||||
bootstrap = textwrap.dedent('''
|
||||
import sys,os,time,json
|
||||
from pathlib import Path
|
||||
from dx_productivity.trial_task_center import batch_worker
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
def business(self, task_id, **kwargs):
|
||||
root=self._project_task_service(task_id)._quick_run_preparer.data_root/'quick-run'
|
||||
root.mkdir(parents=True,exist_ok=True)
|
||||
(root/'started').write_text(str(os.getpid()))
|
||||
request=json.loads(Path(sys.argv[1]).read_text(encoding='utf-8'))
|
||||
print(json.dumps({'version':1,'batch_id':request['batch_id'],'attempt_id':request['attempt_id'],
|
||||
'generation':1,'sequence':100,'type':'progress','payload':'fixture executing'}),flush=True)
|
||||
time.sleep(30)
|
||||
(root/'unexpected_completed').touch()
|
||||
TaskCenterService.execute_quick_run=business
|
||||
batch_worker.run(Path(sys.argv[1]))
|
||||
''')
|
||||
original_command = module._command
|
||||
monkeypatch.setattr(module, '_command', lambda request:
|
||||
(sys.executable, '-c', bootstrap, str(request))
|
||||
if json.loads(request.read_text())['operation'] == 'execute_quick_run' else original_command(request))
|
||||
original_mark = module._mark_interrupted
|
||||
denied = []
|
||||
def mark(*args):
|
||||
if deny_first and not denied:
|
||||
denied.append(True)
|
||||
raise PermissionError('force evidence denied once')
|
||||
return original_mark(*args)
|
||||
monkeypatch.setattr(module, '_mark_interrupted', mark)
|
||||
control = StopControl()
|
||||
driver_errors, survived = [], []
|
||||
def request():
|
||||
try:
|
||||
deadline = time.monotonic() + 15
|
||||
while not witness.exists() and time.monotonic() < deadline:
|
||||
time.sleep(.02)
|
||||
assert witness.exists()
|
||||
pid = int(witness.read_text())
|
||||
if transport_failure:
|
||||
while not control.control_error and time.monotonic() < deadline:
|
||||
time.sleep(.02)
|
||||
assert control.control_error
|
||||
handle = win32api.OpenProcess(0x100000, False, pid)
|
||||
try:
|
||||
survived.append(win32event.WaitForSingleObject(handle, 0) == 258)
|
||||
finally:
|
||||
handle.Close()
|
||||
control.request_force()
|
||||
if deny_first:
|
||||
while not control.control_error and time.monotonic() < deadline:
|
||||
time.sleep(.02)
|
||||
assert control.control_error
|
||||
handle = win32api.OpenProcess(0x100000, False, pid)
|
||||
try:
|
||||
survived.append(win32event.WaitForSingleObject(handle, 0) == 258)
|
||||
finally:
|
||||
handle.Close()
|
||||
control.request_force()
|
||||
except Exception as error:
|
||||
driver_errors.append(error)
|
||||
thread = Thread(target=request)
|
||||
thread.start()
|
||||
try:
|
||||
with pytest.raises(TaskCenterError) as caught:
|
||||
module.run_supervised(service, 'execute_quick_run', {'task_id': task.task_id}, stop_event=control,
|
||||
on_progress=(lambda _: (_ for _ in ()).throw(RuntimeError('UI feedback failed'))) if transport_failure else None)
|
||||
assert not driver_errors
|
||||
assert caught.value.code == 'user_forced_stop'
|
||||
assert (evidence / 'uncertain.flag').exists()
|
||||
marker = (evidence / 'uncertain.flag').read_bytes()
|
||||
assert not (evidence / 'unexpected_completed').exists()
|
||||
journals = [json.loads(p.read_text(encoding='utf-8')) for p in (tmp_path / 'batch-resources').glob('*/supervisor.json')]
|
||||
forced = next(r for r in journals if r['operation'] == 'execute_quick_run')
|
||||
assert forced['stop_mode'] == 'forced' and forced['business_uncertain'] is True
|
||||
assert service.cleanup_batch_resources() == ()
|
||||
assert (evidence / 'uncertain.flag').read_bytes() == marker
|
||||
assert not (evidence / 'unexpected_completed').exists()
|
||||
if deny_first or transport_failure:
|
||||
assert survived == [True]
|
||||
finally:
|
||||
thread.join(3)
|
||||
store.close()
|
||||
@@ -0,0 +1,129 @@
|
||||
"""Request ordinary stop during save on disposable projects; never replay a run."""
|
||||
from dataclasses import asdict
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
from threading import Event, Thread
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
sys.path[:0] = [str(REPO / 'src'), str(REPO / 'tools')]
|
||||
from probe_batch_runtime_live import hashes, compare_saved, save, OLD, NEW, NAMES
|
||||
from dx_productivity.trial_task_center.isolated_cad import IsolatedCadSession
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.bom_plan import write_bom_template
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
from dx_productivity.trial_task_center.task_queue import execute_task_queue
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--source-root', required=True, type=Path)
|
||||
parser.add_argument('--library', required=True, type=Path)
|
||||
parser.add_argument('--run-native', action='store_true', required=True)
|
||||
args = parser.parse_args()
|
||||
output = REPO / 'build' / ('safe-stop-live-' + uuid4().hex[:10])
|
||||
output.mkdir()
|
||||
source_root, library = args.source_root.resolve(strict=True), args.library.resolve(strict=True)
|
||||
source_hashes = {name: hashes(source_root / name) for name in NAMES}
|
||||
library_hash = hashlib.sha256(library.read_bytes()).hexdigest()
|
||||
projects = []
|
||||
for name in NAMES:
|
||||
folder = output / 'projects' / name
|
||||
shutil.copytree(source_root / name, folder)
|
||||
project = folder / (name + '.prj')
|
||||
text, count = re.subn(r'(?m)^KEY CentralLibrary .*$', lambda _: f'KEY CentralLibrary "{library.as_posix()}"',
|
||||
project.read_text(encoding='utf-8-sig'))
|
||||
assert count == 1
|
||||
project.write_text(text, encoding='utf-8')
|
||||
projects.append(project)
|
||||
report = {'scope': 'source supervised service, production VBS and real CAD; no GUI or frozen EXE',
|
||||
'output': str(output), 'projects': list(map(str, projects))}
|
||||
print(json.dumps(report, ensure_ascii=False), flush=True)
|
||||
store = TaskStore(output / 'tasks.db')
|
||||
service = TaskCenterService(store, configure_logging(output / 'data'), supervised=True)
|
||||
control, finished = StopControl(), Event()
|
||||
watcher = None
|
||||
def inspect(project, sheets, label):
|
||||
evidence = output / (label + '-' + project.stem)
|
||||
evidence.mkdir()
|
||||
spec, result = evidence / 'spec.json', evidence / 'graph.json'
|
||||
save(spec, {'project': str(project), 'sheets': sorted(sheets), 'destination': str(result)})
|
||||
with IsolatedCadSession(evidence) as cad:
|
||||
child = cad.start_worker((sys.executable, str(REPO / 'tools/probe_batch_runtime_live.py'),
|
||||
'--repo', str(REPO), '--inspect-file', str(spec), '--run-native'), stdout=-1, stderr=-1)
|
||||
try:
|
||||
code = child.wait(timeout=180)
|
||||
(evidence / 'worker-output.txt').write_bytes(child.stdout.read() + b'\n' + child.stderr.read())
|
||||
assert code == 0, (evidence / 'worker-output.txt').read_text(errors='replace')
|
||||
finally:
|
||||
child.close()
|
||||
return json.loads(result.read_text(encoding='utf-8'))
|
||||
try:
|
||||
tasks = service.create_project_tasks(output / 'projects', selected_paths=tuple(projects))
|
||||
bom = write_bom_template(output / 'input.xlsx', ({'project_path': str(p), 'old_part_number': OLD,
|
||||
'new_part_number': NEW, 'refdes': 'RV05', 'enabled': True, 'sync_nc': True} for p in projects))
|
||||
imported = service.inspect_bom_batch(tasks[0].task_id, bom)
|
||||
assert imported.passed_rows == 2 and imported.preview is not None
|
||||
service.save_bom_batch(imported.preview, {p.task_id: p.rows for p in imported.preview.projects})
|
||||
before, sheet_sets = {}, {}
|
||||
for project in projects:
|
||||
rows = next(p.rows for p in imported.preview.projects if p.project_path.resolve() == project.resolve())
|
||||
sheet_sets[project.stem] = {(row.target.schematic_name, row.target.sheet_number) for row in rows}
|
||||
before[project.stem] = inspect(project, sheet_sets[project.stem], 'before')
|
||||
def stop_during_save():
|
||||
while not finished.wait(.01):
|
||||
for path in (output / 'data/tasks').glob('*/quick-run/quick-part-replace.log'):
|
||||
try:
|
||||
text = path.read_text(encoding='utf-8-sig')
|
||||
except OSError:
|
||||
continue
|
||||
if 'save_requested job=' in text:
|
||||
report['stop_requested'] = {'time_ns': time.time_ns(), 'log': str(path),
|
||||
'trigger': 'production save_requested log; ordinary stop only'}
|
||||
control.set()
|
||||
return
|
||||
watcher = Thread(target=stop_during_save, daemon=True)
|
||||
watcher.start()
|
||||
result = execute_task_queue(service, tuple(task.task_id for task in tasks), stop_event=control)
|
||||
finished.set()
|
||||
watcher.join(3)
|
||||
report['queue_result'] = asdict(result)
|
||||
assert report.get('stop_requested') and not control.force_event.is_set()
|
||||
assert result.paused and [row.state for row in result.entries] == ['completed', 'pending'], result
|
||||
report['comparisons'] = []
|
||||
for index, project in enumerate(projects):
|
||||
after = inspect(project, sheet_sets[project.stem], 'after')
|
||||
if index == 0:
|
||||
report['comparisons'].append(compare_saved(before[project.stem], after))
|
||||
else:
|
||||
assert after == before[project.stem], 'second project was changed after stop'
|
||||
report['comparisons'].append({'unchanged': True, 'components': len(after)})
|
||||
assert not service.pending_batch_resources()
|
||||
assert not list((output / 'data').rglob('uncertain.flag'))
|
||||
report['verified'] = True
|
||||
except BaseException as error:
|
||||
import traceback
|
||||
report.update(error=repr(error), traceback=traceback.format_exc())
|
||||
raise
|
||||
finally:
|
||||
finished.set()
|
||||
if watcher is not None:
|
||||
watcher.join(3)
|
||||
store.close()
|
||||
report['source_unchanged'] = all(hashes(source_root / name) == source_hashes[name] for name in NAMES)
|
||||
report['library_lmc_unchanged'] = hashlib.sha256(library.read_bytes()).hexdigest() == library_hash
|
||||
save(output / 'report.json', report)
|
||||
print(json.dumps(report, ensure_ascii=False), flush=True)
|
||||
return 0 if report.get('verified') and report['source_unchanged'] and report['library_lmc_unchanged'] else 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
Reference in new issue
Block a user