Compare commits
23
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
12895e9868 | ||
|
|
9c03cd5900 | ||
|
|
b939101565 | ||
|
|
ae4e0909a6 | ||
|
|
530e768a5b | ||
|
|
3367bb6454 | ||
|
|
2a90201bfa | ||
|
|
b776ce58eb | ||
|
|
ffc0548e1f | ||
|
|
264fee5127 | ||
|
|
3b8a93e12c | ||
|
|
cf879c9265 | ||
|
|
01d1b54caa | ||
|
|
f3d7423d02 | ||
|
|
62f143fe9f | ||
|
|
9837ca581b | ||
|
|
c787ef325b | ||
|
|
3c4e6e7e37 | ||
|
|
ba5b13667b | ||
|
|
1baf908ba4 | ||
|
|
6d0b92c515 | ||
|
|
7d5ea814ec | ||
|
|
b21960cda1 |
No files matched your search
@@ -4,9 +4,11 @@
|
||||
|
||||
面向 **Windows + DxDesigner / Expedition Enterprise 7.9.5** 的器件批量替换工具。一个工程对应一个任务,可以只替换同料号下的部分位号;支持直接编辑和 Excel BOM 导入,同批工程按顺序执行。
|
||||
|
||||
**最新版本:V1.1.4 Beta 2(预发布)** · [下载便携包](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/download/v1.1.4-beta.2/DxPartReplace-1.1.4-beta.2.zip) · [版本说明与附件](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/tag/v1.1.4-beta.2) · [完整使用说明](docs/user/v1.1.0-project-tasks.md)
|
||||
**最新版本:V1.1.4 Beta 4(预发布)** · [下载便携包](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/download/v1.1.4-beta.4/DxPartReplace-1.1.4-beta.4.zip) · [版本说明与附件](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/tag/v1.1.4-beta.4) · [完整使用说明](docs/user/v1.1.0-project-tasks.md)
|
||||
|
||||
所有发布均标记为 **Beta 预发布**。V1.1.4 Beta 1 因 BOM 导入缺少 `win32timezone` 已撤回,请使用 Beta 2;旧 Git 标签保留供回溯。升级前备份 `data`,完整更新 EXE 与 `_internal`,不要只替换 EXE。
|
||||
所有发布均标记为 **Beta 预发布**。V1.1.4 Beta 1 因 BOM 导入缺少 `win32timezone` 已撤回,请使用 Beta 4;旧 Git 标签保留供回溯。升级前备份 `data`,完整更新 EXE 与 `_internal`,不要只替换 EXE。
|
||||
|
||||
Beta 4 面向停止与资源收尾复测:普通暂停等待当前项保存核验,写入中无进度不再自动杀进程;新增明确确认的强制结束,保留中断证据与恢复入口。[本版变更和复测说明](docs/releases/v1.1.4-beta.4.md)。
|
||||
|
||||
本版提供[目标封装选择](docs/user/target-cell-selection.md):目标封装可留空,此时工具不主动覆写封装,可后续与中心库同步;需要指定时,在料号规则中读取并选择合法封装。BOM 保持原 11 列。
|
||||
|
||||
@@ -18,7 +20,7 @@
|
||||
|
||||
### 第 1 步:解压并打开应用
|
||||
|
||||
[下载 V1.1.4 Beta 2 便携包](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/download/v1.1.4-beta.2/DxPartReplace-1.1.4-beta.2.zip),**完整解压**,双击 `DxPartReplace.exe`。保留它与 `_internal` 的相对位置,无需安装 Python;本机需要可用的 DxDesigner、许可证和中央库。
|
||||
[下载 V1.1.4 Beta 4 便携包](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/download/v1.1.4-beta.4/DxPartReplace-1.1.4-beta.4.zip),**完整解压**,双击 `DxPartReplace.exe`。保留它与 `_internal` 的相对位置,无需安装 Python;本机需要可用的 DxDesigner、许可证和中央库。
|
||||
|
||||
打开后看到下图主窗口。首次使用请准备完整的工程副本,执行前保存并关闭待处理工程。
|
||||
|
||||
@@ -131,6 +133,8 @@
|
||||
|
||||
运行期间可以在主桌面查看其他文件,请勿打开或修改本次正在处理的同一工程。需要暂停时,点击 **完成当前项后暂停**,等待当前步骤安全收尾。
|
||||
|
||||
Beta 4 在保存或核验尚未确认时保留应用等待,可先 **导出诊断包**。只有明确接受中断后需要核验时,才选择 **强制结束…**;之后先核验恢复,不能直接重复替换。详见[停止、强制结束与恢复](docs/user/safe-stop-and-recovery.md)。
|
||||
|
||||
**完成标志:** 任务显示已完成,底部提示本批执行结束。还要打开结果核对,不能只看进度条。
|
||||
|
||||
### 第 12 步:查看实际修改结果与日志
|
||||
@@ -194,7 +198,7 @@ DxPartReplace/
|
||||
|
||||
## 验证范围与已知限制
|
||||
|
||||
V1.1.4 Beta 2 新增批次后台资源管理:一次 BOM 导入或顺序执行共用一个私有桌面,CAD 按需启动并在同批工程间复用,批次结束统一退出。若提示 **资源释放待处理**,已保存核验的结果保留,请点击 **重新清理资源**;该操作只清理后台资源,不会再次替换。旧的工程保护锁仍按“核验并恢复”处理。设计、异常边界与实机结果见[批次运行时验收记录](docs/testing/2026-10-21-batch-private-runtime.md)。
|
||||
V1.1.4 Beta 4 沿用批次后台资源管理:一次 BOM 导入或顺序执行共用一个私有桌面,CAD 按需启动并在同批工程间复用,批次结束统一退出。若提示 **资源释放待处理**,已保存核验的结果保留,请点击 **重新清理资源**;该操作只清理后台资源,不会再次替换。旧的工程保护锁仍按“核验并恢复”处理。设计、异常边界与实机结果见[批次运行时验收记录](docs/testing/2026-10-21-batch-private-runtime.md)。
|
||||
|
||||
资源释放会追加日志:在 **替换结果 / 日志 → 完整日志** 查看本批开始释放、释放成功/失败及重新清理的结果。独立文件位于 `data/batch-resources/<批次ID>/resource-release.log`,记录批次、桌面、CAD PID、退出码、释放耗时、句柄关闭和桌面消失的分别确认,以及失败原因;同时保留 `session.json` 的原始释放证明。重启后重新检查也会追加记录,不覆盖原失败日志。
|
||||
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# DxPartReplace V1.1.4 Beta 3
|
||||
|
||||
2026-10-23,**Beta 预发布/工单现场复测版**,对应工单 [#1](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/1)、[#3](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/3)。
|
||||
|
||||
## 下载与升级
|
||||
|
||||
完整解压 `DxPartReplace-1.1.4-beta.3.zip`,运行 `DxPartReplace.exe`,无需安装 Python。保留 EXE 与 `_internal` 的相对位置;本机需要可用的 DxDesigner、许可证和中央库。
|
||||
|
||||
建议在新目录解压,用工程副本复测。需要沿用任务时,先关闭旧程序、备份旧 `data`,再完整复制数据目录;不同版本不能同时打开同一数据目录。旧保护锁保留,不能通过删除保护记录来重试。
|
||||
|
||||
包内包含中文图文教程、窗口 GIF、空白 11 列 BOM 模板、发布清单。教程沿用旧版已录制的基本操作,新增加的恢复与归档选项以本版界面为准。
|
||||
|
||||
## 本版修改
|
||||
|
||||
- 修复内部长路径访问:缩短批次 attempt 目录,补齐扩展路径、兼容工作目录、深目录工程扫描、日志和保护记录访问;不要求修改系统注册表。
|
||||
- 后台资源由独立批次监督进程持有,批内复用桌面与按需启动的 CAD,批末退出后由主界面独立复核。主界面不持有私有桌面或 COM 对象。
|
||||
- 只有持久化证据证明尚未修改、工程身份一致、旧资源已释放,才允许每工程自动重试一次;修改后、保存核验失败或结果不确定时不重放。
|
||||
- 自有 CAD 启动期间退出时及时检测,避免探针继续空等 120 秒。真实连续启动故障的检测与收尾阶段约 0.25–0.33 秒,完整任务仍受 CAD 启动、替换及保存回读耗时影响。
|
||||
- 分别保留首发错误、清理错误与 BOM 逐行报告。清理资源不重新执行已完成替换;恢复请求和消息检查批次、代次及尝试身份。
|
||||
- 补充自有进程/句柄归属诊断,提供“导出诊断包”,包含独立的批次释放日志并脱敏。
|
||||
- 增加“正常关闭时归档并清空当前任务列表”,默认关闭。启用后安全任务移至回收站;存在保护、中断或未释放资源的任务保留,不物理删除工程、日志及执行证据。
|
||||
- 持久化只读采集缓存,绑定工程指纹及完整载荷摘要;损坏缓存隔离后重新读取。
|
||||
|
||||
## 工单复测步骤
|
||||
|
||||
1. 使用原问题目录长度及工程副本,创建任务并导入原 BOM;确认通过、失败、未检查行数和逐行原因都能显示,失败后再次导入会刷新状态。
|
||||
2. 先选择两个工程的少量位号,保存计划并顺序执行;核对目标料号、未选位号、保存重开结果及释放日志,再扩大批次。
|
||||
3. 遇到资源待处理时,使用“重新清理资源”,确认不会重复替换。仍失败请点击“导出诊断包”,将 ZIP、版本及触发步骤附到原工单。
|
||||
4. 需要关闭后清空列表时,启用正常关闭归档选项;重开后可从回收站恢复。保护或待核验任务应继续保留。
|
||||
|
||||
## 验证范围与已知边界
|
||||
|
||||
实现阶段完成两份新工程副本的完整 EXE BOM 流程、真实替换、保存重开与独立属性/连接对照:每工程 57 器件、103 引脚,56 个非目标器件不变;顺序执行共一个桌面、一次 CAD 启动。另完成深中文路径的冻结 EXE 检查、真实外部桌面持有者、标记写入失败、连续启动失败及写后监督进程崩溃的测试。
|
||||
|
||||
故障实机入口部分为源码非 GUI parent+真实冻结 supervisor/helper/VBS,不等同于完整 GUI 硬退出或机器掉电测试。不同候选的具体哈希、命令和范围见[实施与验收记录](../testing/2026-10-23-issue-3-safe-recovery.md)。本发布包重新构建并检查冻结源码一致性、解压启动及资源释放,结果写入发布清单。
|
||||
|
||||
**仍需现场验证:** 关闭系统长路径支持且没有 8.3 的组合环境尚未完整验收;原工单 102 个工程没有完成原工程实机替换,其中 96 个连报告回放也未覆盖,见[逐工程清单](../testing/2026-10-23-issue-3-uncovered-projects.md)。工单 #1 现场残留桌面的具体持有者仍未确定,不能宣称根因已根治。两个工单保持打开,等待现场反馈。
|
||||
@@ -0,0 +1,37 @@
|
||||
# V1.1.4 Beta 4 构建与发布验证
|
||||
|
||||
2026-10-26,用户授权发布下一版 Beta。版本 `1.1.4-beta.4`,Python 元数据 `1.1.4b4`,Windows 文件版本 `1.1.4.4`,预发布标志开启。业务修改基于 `9c03cd5`;本轮产品代码只调整版本标识。
|
||||
|
||||
## 构建门禁
|
||||
|
||||
- `packaging/Build-DxPartReplace.ps1`:退出码 0,源码门禁 **923 passed,232.69 秒**,一条既有 pywinauto STA 警告。
|
||||
- 打包配置及外部验收脚本检查:**17 passed,11.08 秒**。
|
||||
- 构建产物在完整冻结 EXE 主进程中启动独立监督进程、读取真实进程创建时间并确认自有进程/Job/桌面释放,再完成 GUI 启动与正常关闭检查。
|
||||
- 冻结的 92 个 `dx_productivity` 模块及应用入口逐项对照源码,递归标准化 code filename 后 code object 一致。包含新的 `stop_control` 模块;两份 VBS 在包装阶段另做哈希比对。
|
||||
- EXE SHA256:`ee64d983f6f0457bac2bb16225ee8a8044753b5bf7779e498549c61d8ed4d13b`。
|
||||
|
||||
构建日志:`build/release-1.1.4-beta.4-build.log`。源码比对:`build/release-1.1.4-beta.4/frozen-source-check.json`。实现阶段扩大测试 1147 项通过的范围见[安全停止验收](../testing/2026-10-26-safe-stop-acceptance.md),不与本轮构建门禁数量混算。
|
||||
|
||||
## 冻结监督进程实机验证
|
||||
|
||||
本轮用源码控制器驱动上述正式 EXE 的 `--batch-worker`,执行 BOM 导入与真实 CAD 队列;批内 helper 和 VBS 使用冻结产物。复制两个全新工程,不重放之前完成的替换。
|
||||
|
||||
在第一个工程的生产 `save_requested` 日志出现时只请求普通停止,结果为第一任务 completed、第二任务 pending、队列 paused。新 CAD 实例独立读回确认:第一工程 57 器件、103 引脚连接正确,56 个非目标器件保持;第二工程 57 器件完整属性/连接图与执行前一致。
|
||||
|
||||
执行批次 `orderly_shutdown=true`、`clean=true`、`forced_termination=false`,CAD 与 Quit helper 退出码均为 0,资源状态 released。源工程目录哈希和中央库 `.lmc` 哈希不变。
|
||||
|
||||
证据:`build/release-1.1.4-beta.4/frozen-save-stop.json` 与 `build/safe-stop-live-9894f95db1/`。后者由通用 source 探针生成,保留了其原始 scope 文本;本轮实际冻结进程路由、调用请求和 EXE 哈希以 companion `frozen-save-stop.json` 为准。前后独立读取使用源码读取器,不将其说成全部入口均冻结,也不将该测试说成完整 GUI 交互停止。
|
||||
|
||||
## 包装与公开附件
|
||||
|
||||
便携包只从 EXE、完整 `_internal`、仓库教程及文档生成,附空白 11 列 BOM 模板和历史窗口 GIF;不带 `data`、任务数据库、用户工程、工单附件或现场日志。
|
||||
|
||||
包装脚本要求本轮冻结停止验证的 EXE 哈希与被打包文件一致,检查 ZIP CRC、解压路径、所有程序文件 SHA256,并在清除源码 Python 环境变量后执行解压出来的完整 EXE smoke。ZIP 大小、SHA256、程序文件清单与解压验证结果由公开 `release-manifest-1.1.4-beta.4.json` 记录。
|
||||
|
||||
附件:`DxPartReplace-1.1.4-beta.4.zip`、`SHA256SUMS-1.1.4-beta.4.txt`、`release-manifest-1.1.4-beta.4.json`、`BOM-template.xlsx`、`window-tutorial-1.1.4-beta.4.gif`。GIF 沿用 1.1.2 基础教程,新停止功能以包内说明和界面为准。
|
||||
|
||||
发布流程为先创建草稿,上传后逐项回下载核对大小和 SHA256,全部一致才公开为 Beta 预发布。远端响应与附件校验证据保存在 `build/release-1.1.4-beta.4/`;认证信息不写入文件或日志。本轮没有授权给工单发消息,因此不发布工单评论、不关闭工单,也不修改旧版本附件。
|
||||
|
||||
## 现场边界
|
||||
|
||||
仍未完成原工单 102 个工程的全量现场复测,也没有完整冻结 GUI 的真实保存中强杀或掉电无损验收。工单现场桌面残留持有者尚未查明。本版本提供已验证的停止与收尾改进,不能据此宣布现场事故根因已经根治。
|
||||
@@ -0,0 +1,41 @@
|
||||
# DxPartReplace V1.1.4 Beta 4
|
||||
|
||||
2026-10-26,**Beta 预发布/安全停止与应用收尾复测版**。关联工单 [#1](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/1)、[#3](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/3)。
|
||||
|
||||
## 下载与升级
|
||||
|
||||
完整解压 `DxPartReplace-1.1.4-beta.4.zip`,运行 `DxPartReplace.exe`。无需安装 Python;保留 EXE 和 `_internal` 的相对位置,本机需要可用的 DxDesigner、许可证及中央库。
|
||||
|
||||
请在新目录解压,使用工程副本复测。沿用旧任务前,先安全退出旧程序并备份完整 `data`;不要同时用两个版本打开同一数据目录。不要删除原有保护记录来绕过核验。
|
||||
|
||||
便携包包含中文教程、空白 11 列 BOM 模板、历史窗口操作 GIF、停止与恢复说明及文件校验清单。GIF 沿用 1.1.2 基础操作,新版停止与强制结束功能以实际界面及[停止说明](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/src/tag/v1.1.4-beta.4/docs/user/safe-stop-and-recovery.md)为准。
|
||||
|
||||
## 本版修改
|
||||
|
||||
- 普通“完成当前项后暂停”只停止后续调度,当前替换项完成必要校验、保存、回读和关闭后再停止;关闭窗口也等待收尾。
|
||||
- 有写入可能的阶段长时间没有进度时保留应用等待,不再仅凭超时自动杀进程;纯只读采集仍采用有界取消。
|
||||
- 新增独立的“强制结束…”确认。先可靠保存中断保护和批次证据,再终止本程序拥有的进程;保护写入失败时拒绝本次强制结束并显示原因。
|
||||
- 当前修改或保存结果不确定、应用拒绝退出时保留现场,禁止盲目保存、丢弃和重放。等待期间可导出诊断包。
|
||||
- 保存请求、保存调用返回、保存回读和工程关闭分别记录,避免保存后关闭失败被误记为未保存。
|
||||
- 修复收尾记录写入失败跳过正常 Quit 的路径;CAD 主进程正常退出后,给自有辅助进程短暂自然退出机会,再决定是否需要强制清理。
|
||||
- 正常退出、强制终止和物理资源释放分别报告。资源消失不再掩盖应用退出异常,已经核验的业务结果继续保留。
|
||||
- 控制通信异常不会直接授权终止正在运行的交互批次;强制结束及进程退出诊断记录保持一致。
|
||||
|
||||
## 复测建议
|
||||
|
||||
1. 新建两个工程副本的少量位号任务,导入 BOM 并保存计划。
|
||||
2. 在第一项执行或保存时点击普通暂停,核对当前项完成保存回读,下一项保持未执行;重新打开工程核对料号和连接。
|
||||
3. 对长时间等待先导出诊断。仅在接受当前结果需要核验时确认“强制结束…”;重开后先“核验并恢复”,不要直接重复替换。
|
||||
4. 如仍有异常,保留版本、触发步骤、诊断 ZIP 和失败工程名称。资源待处理时的清理不会重新执行替换。
|
||||
|
||||
为避免强制请求与任务切换同时发生而漏记目标,本版强制结束会保守保护本批可能涉及的任务。先前已完成任务也可能需要复核,但其保存结果和执行证据不被覆盖或重放。
|
||||
|
||||
## 验证范围与已知边界
|
||||
|
||||
实现阶段扩展回归 **1147 项通过**。源码监督入口在两份新工程副本上验证了保存期间请求普通停止:第一工程 completed,第二工程 pending;新 CAD 独立读回确认第一工程 57 器件、103 引脚连接正确、56 个非目标器件不变,第二工程完整图未变化。源工程及中央库 `.lmc` 哈希不变,执行批次正常退出且没有强制清理。
|
||||
|
||||
本轮发布门禁 923 项通过,打包与验收脚本检查 17 项通过。正式 EXE 的冻结监督进程再次完成真实 CAD 保存期间普通停止及新实例独立读回;当前项完成、下一项未执行,正常退出且无强制终止。该实机入口由源码控制器驱动,不等同于完整 GUI 交互验收。
|
||||
|
||||
发布构建、解压后完整 EXE 启动、冻结监督进程及资源释放检查另见[Beta 4 构建记录](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/src/tag/v1.1.4-beta.4/docs/releases/v1.1.4-beta.4-build.md)和附件 manifest。源码/受控辅助进程故障测试不等同于完整冻结 GUI 下的真实 CAD 强制中断或掉电无损验证。
|
||||
|
||||
**现场资源残留根因仍未全部闭环。** 原工单全量 102 个工程及特定系统组合仍需现场复测,本版不宣称所有环境已根治,不自动关闭关联工单。完整范围见[实施验收](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/src/tag/v1.1.4-beta.4/docs/testing/2026-10-26-safe-stop-acceptance.md)。
|
||||
@@ -0,0 +1,62 @@
|
||||
# Safe User Stop Implementation Plan
|
||||
|
||||
> **For agentic workers:** use superpowers:subagent-driven-development or executing-plans; continue within the user-approved scope without another approval round.
|
||||
|
||||
**Goal:** 用户停止只在安全边界收尾;写入中的无进度不自动杀进程;明确强制中断有保护证据和恢复入口。
|
||||
|
||||
**Architecture:** 保留现有 Event/队列边界和独立监督进程,加入独立的强制停止意图、等待状态反馈;退出正常性与物理资源复核分开。使用既有任务保护与批次记录,不新增数据库版本。
|
||||
|
||||
**Tech Stack:** Python 3.12、Windows pywin32、PySide6、VBScript、pytest。
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-10-26-safe-user-stop.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- 不修改原工程、中心库或未跟踪 feedback;真实测试只复制 fixture。
|
||||
- 不放宽身份、写前保护、禁止不确定重放的门禁。
|
||||
- 普通停止与强制结束独立,确认前不得产生强制意图。
|
||||
- 不通过无限创建桌面或删除旧保护绕过异常。
|
||||
- 不发布、不推送;本地分支 `codex/day0-safe-stop`。
|
||||
|
||||
## Task 1: 退出协议与后代进程
|
||||
|
||||
Files: `isolated_cad.py`, `isolated_desktop.py`, `tests/v1_1/test_isolated_cad.py`, `test_isolated_desktop.py`.
|
||||
|
||||
- [x] 先复现 closing 记录失败跳过 Quit、父进程自然退出后仍立即终止子进程的测试。
|
||||
- [x] 诊断写入失败不跳过满足身份/空工程条件的 Quit;将记录失败独立保留并报告,禁止伪报 clean。
|
||||
- [x] 根进程正常退出时,给 Job 后代 2 秒有界自然退出窗口;到期再强制并记录 `forced_termination`, `termination_reason`, `forced_descendant_pids`。根进程仍活着的异常清理仍有界。
|
||||
- [x] 保留所有权和退出确认,测试自然退出与强制升级两条路径。
|
||||
|
||||
## Task 2: 停止、等待与强制中断
|
||||
|
||||
Files: new `stop_control.py`; `worker_monitor.py`, `project_session.py`, `batch_runtime.py`, `batch_supervisor.py`, `execution_feedback.py`; associated tests.
|
||||
|
||||
Interface:
|
||||
```python
|
||||
class StopControl(Event):
|
||||
force_event: Event
|
||||
def request_force(self):
|
||||
self.set()
|
||||
self.force_event.set()
|
||||
```
|
||||
- [x] 首先测试 unsafe 保存阶段超时仍保留 worker,安全阶段仍有 watchdog,显式 force 才结束并产生 uncertain。
|
||||
- [x] `ProgressWatch` 提供 `waiting_for_safe_completion`;新进度可清除提示,不把心跳当作写入推进。
|
||||
- [x] 监督父进程持久化停止意图;force 先写保护与中断记录,再只终止已拥有的监督进程 Job,不使用名字或不可信 PID 清理。
|
||||
- [x] 业务完成但退出异常仍通过独立错误返回,附带已核验业务结果;纯桌面延迟且退出正常可继续使用现有复核恢复。
|
||||
- [x] 普通停止在当前项收尾后暂停;会话 close 不得因普通等待超时反向强杀仍在写入的 worker。
|
||||
|
||||
## Task 3: 保存边界与界面
|
||||
|
||||
Files: `QuickPartReplace.vbs`, `ui/run_panel.py`, `ui/main_window.py`, `ui/execution_feedback.py`, protocol serializer if required; VBS/UI tests.
|
||||
|
||||
- [x] VBS 用真实函数的失败先行测试覆盖保存已发出但工程关闭失败;保存请求前设置“可能保存”状态,独立记录保存返回/回读/关闭,不再声称未保存。
|
||||
- [x] 正常停止显示等待当前项收尾;unsafe 超时显示保存/核验尚未确认。
|
||||
- [x] 提供与普通暂停分离的强制结束按钮,确认文案明确结果待核验;取消确认没有副作用。
|
||||
- [x] 关闭窗口在运行/等待期间保持打开;任务失败后取消自动关闭,让用户看见恢复信息。
|
||||
|
||||
## Task 4: 集成验证与审查
|
||||
|
||||
- [x] 运行相关 VBS、monitor、supervisor、queue、UI、资源测试及完整发布门禁范围。
|
||||
- [x] 实机空 CAD 自然退出、两工程副本正常保存及独立回读;停止/强制结束采用受控替身或独立进程,明确验证边界。
|
||||
- [x] 独立审查关注:不确定写入重放、等待时 GUI 退出杀进程、force 身份校验与持久化失败、结果错误被吞、已完成项保留。
|
||||
- [x] 更新操作文档和验收记录,交付本地可审查修改及测试证据。
|
||||
@@ -0,0 +1,9 @@
|
||||
# 用户停止与安全收尾(已批准)
|
||||
|
||||
用户已在本会话批准:普通停止冻结后续调度;未修改可取消;当前已修改项完成校验、保存、回读后停止;保存或修改结果不明保持保护,不自动重放或盲目保存/丢弃;关闭窗口等待收尾;强制结束必须由用户明确选择,记录中断并在重启后核验。
|
||||
|
||||
业务结果、应用正常退出、强制清理和物理资源状态分别记录。资源消失不抹去应用异常。保存中无进度超时进入等待/提示,不自动终止。提供明确确认的强制结束出口;其范围仅为本程序当前监督进程及自有 Job。
|
||||
|
||||
普通停止以当前替换项及其必要收尾为边界,不启动同工程下一项。保护记录、批次身份和进程创建时间核验继续有效。先保存中断证据再执行工具内强制终止;持久化失败时拒绝工具内强制操作并反馈错误。系统外强杀/掉电依赖已有写前保护与重启核验,不承诺可在终止时写入日志。
|
||||
|
||||
保持源码版本,不发布包、不关闭远端事故工单。真实工程验证只用全新副本。
|
||||
@@ -0,0 +1,25 @@
|
||||
# V1.1.4 Beta 3 现场复测包构建记录
|
||||
|
||||
用户在实现验收后授权“压缩发布一版本发给工单回去测试”。本轮交付为工单 #1/#3 的 Beta 复测包,不将此前未满足的严格现场条件写为已解决,不自动关闭工单。
|
||||
|
||||
## 发布构建
|
||||
|
||||
版本字符串 `1.1.4-beta.3`,Python 项目版本 `1.1.4b3`,Windows 文件版本 `1.1.4.3`,Windows 预发布标志启用。
|
||||
|
||||
- `packaging/Build-DxPartReplace.ps1` 退出码 0,源码发布门禁 **889 passed**,195.14 秒;一条既有 pywinauto STA 警告。原始构建输出:`build/release-1.1.4-beta.3-build.log`。
|
||||
- 打包配置与验收脚本检查 **17 passed**,6.11 秒。
|
||||
- 构建门禁在完整冻结 EXE 中启动独立批次监督进程,读取真实进程创建时间,确认 Job/进程/桌面释放,再完成 GUI 启动和正常退出检查。
|
||||
- EXE SHA256:`df08cd328f6f4c9019f232c845e09627551e7b2d24790a466d53efcd291622d1`。
|
||||
- 与上一实现验收候选 `f38411a2…` 比较 91 个冻结 `dx_productivity` 模块,只有 `version` 模块变化;应用入口一致。比较递归标准化 code filename 后的 Python code object,不以时间戳或文件名推断源码一致。
|
||||
|
||||
完整实现、两工程替换和故障注入的候选哈希及范围见[实施验收](2026-10-23-issue-3-safe-recovery.md)。本轮没有对已完成工程重放替换。
|
||||
|
||||
## 压缩与分发校验
|
||||
|
||||
便携包只从发布 EXE、完整 `_internal` 和仓库教程组装,附空白 11 列 BOM 模板、窗口 GIF、清单;不从运行目录打包 `data`、工程、数据库或现场日志。
|
||||
|
||||
发布准备脚本校验冻结模块与当前源码、应用入口、两份 VBS 的一致性,生成程序文件 SHA256 清单。ZIP 校验 CRC、解压路径和所有程序文件哈希,再在清除源码 Python 环境变量后运行解压出来的完整 EXE。具体 ZIP 大小、SHA256、文件数与解压启动结果记录在发布附件 `release-manifest-1.1.4-beta.3.json`,公开校验文件为 `SHA256SUMS-1.1.4-beta.3.txt`。
|
||||
|
||||
先上传为草稿,下载回验全部附件的大小和 SHA256,再标记为 Beta 预发布。工单回复包含版本、下载地址、复测步骤和诊断包反馈入口;工单保持打开。运行时 API 返回与下载校验留存于 `build/release-1.1.4-beta.3`,不将认证凭据写入文件或输出。
|
||||
|
||||
发布说明:[V1.1.4 Beta 3](../releases/v1.1.4-beta.3.md)。尚未覆盖的系统组合、原工程和现场持有者问题继续以该说明为准。
|
||||
@@ -0,0 +1,27 @@
|
||||
# Issue 3:外部 OpenDesktop 句柄实机验收
|
||||
|
||||
2026-10-23,在 Windows 本机运行 `python tools/probe_external_desktop_holder.py`,退出码为 0。探针使用真实 `BatchCadRuntime` / `PrivateDesktop`,只创建独立 Python 持有者,不启动 CAD,不打开工程,不执行替换。
|
||||
|
||||
真实证据保存在 `build/issue-3-external-holder/run-d0a0af0e6a154c65910fb5f29a645e61/evidence.json`,同时保留该目录下真实 `data/batch-resources/*/session.json`、`business-result.json` 和资源释放日志。每次运行创建全新的 `run-*` 目录,避免覆盖既有证据。
|
||||
|
||||
本次探针 PID 为 37744;桌面为 `dx-d346d3ab310f470c86ffd9d32ccbef4a`;独立持有者 PID 为 18136,使用 `OpenDesktop(..., DESKTOP_READOBJECTS)` 获得句柄 412。桌面所有者记录句柄 552 的创建与成功关闭,包含 PID 37744、原生线程 ID 22468。持有者仍在默认桌面,无测试窗口。
|
||||
|
||||
| 检查 | 实机结果 |
|
||||
| --- | --- |
|
||||
| 首次批次清理 | 自有桌面句柄关闭,但枚举仍发现桌面;等待 5 秒后 `blocked` |
|
||||
| 残留窗口 | `remaining_windows=[]`,仍然阻断,没有据此推断资源已释放 |
|
||||
| 持有者存活 | 首次失败与持有期间资源重试后,持有者均仍在运行 |
|
||||
| 新批次门禁 | `BatchCadRuntime.__enter__` 抛出 `batch_resources_pending`;候选批次 generation=0,未创建桌面 |
|
||||
| 持有中仅重试资源 | `cleanup_pending_resources` 再次确认阻断,没有启动 CAD 或业务操作 |
|
||||
| 持有者自行释放 | 通过探针持有的 stdin 发送 `release`;持有者调用 `CloseDesktop`,正常退出码 0 |
|
||||
| 释放后仅重试资源 | `cleanup_pending_resources` 返回空元组,session 状态为 `released`,桌面枚举确认消失 |
|
||||
| 业务结果 | 合成的已核验 `business-result.json` 逐字节保持一致;内存业务结果亦保持一致 |
|
||||
| CAD 与重放 | CAD 工厂调用次数 0;`cad_start_requested=false`、`cad_pid=null`、`processes=[]`;没有业务重放入口调用 |
|
||||
| 输入桌面 | 前后一致,未切换输入桌面 |
|
||||
| 释放后的准入 | 后续批次可通过准入检查,但探针不启动其桌面/CAD或业务 |
|
||||
|
||||
业务结果前后 SHA256 均为 `80507674ce6b5c1d11419bd6e7b2fdfea3c5a6697cb01d628abf232615e9dc7e`。本轮总耗时 10.609 秒,主要为两次有界的 5 秒释放确认。
|
||||
|
||||
诊断中的外部持有者仍为 `unknown`,未识别具体外部句柄持有 PID。上述 PID 18136 仅由探针自身创建进程返回值与辅助进程握手确认,不能视为产品诊断识别能力。空窗口列表也不能证明没有外部句柄持有者。
|
||||
|
||||
这轮覆盖的是资源门禁与资源清理 API 的真实 Windows 句柄路径。业务结果是独立合成验收夹具,不等同于真实 CAD 工程结果,也不替代打包 EXE 的交互验收。未修改 `src`。
|
||||
@@ -0,0 +1,33 @@
|
||||
# Issue 3:新版冻结 EXE 两次启动故障,最多恢复一次
|
||||
|
||||
2026-10-23,在 CAD 独占窗口运行 `tools/probe_frozen_startup_recovery.py --failures 2`,实机退出码 0。使用另一份全新 `Audio_D2128` 副本,不复用已完成替换或 postwrite 崩溃工程。未修改生产源码/VBS、未降低生产超时。
|
||||
|
||||
冻结 EXE SHA256 为 `3b6a8608bac9d88f93ac71487d5c32647af0894c639d624f380f60345d27d34a`,源码基线为 264fee5。本例入口仍为**源码非 GUI supervised parent + 冻结 worker/helper/VBS**;独立完整图读取为源码只读 inspector,在独立私有 CAD 中运行。
|
||||
|
||||
真实证据目录为 `build/issue-3-frozen-startup-recovery/run-275ef3111c8d423caa4242b9045e9c59`。
|
||||
|
||||
两次终止均使用同一安全约束:仅本轮 request/session/generation 的 CAD PID;核对 Windows 创建时间与拥有者记录相同、私有桌面相同、starting 状态、无 GUID/bootstrap/cad-ready、attempt=prepared;然后只终止该自有 CAD。未终止任何未知或外部进程。
|
||||
|
||||
| 检查 | 实机结果 |
|
||||
| --- | --- |
|
||||
| 第一代故障 | 自有 CAD PID 16192,创建时间 `2026-10-23 08:53:02.663000+00:00`,退出码 71 |
|
||||
| 第二代故障 | 自有 CAD PID 31544,创建时间 `2026-10-23 08:53:04.657000+00:00`,退出码 71 |
|
||||
| 提前退出检测 | reason=`owned private CAD exited during startup: exit code 71`,未等待 bootstrap 总超时 |
|
||||
| 故障阶段耗时 | 生产 `project_worker` 记录分别为 0.328 秒、0.250 秒;包含该阶段的启动/检测/本地清理,不是孤立检测函数的精确计时 |
|
||||
| 唯一预算 | execution generation=2;retries 只有一条且 `count=1`;第二次失败后任务 blocked/paused,无第三代 |
|
||||
| 两代 proof | 都保持 prepared;前后字节与 SHA256 不变;没有 mutation_possible |
|
||||
| 换代安全顺序 | 第一桌面确认消失、released_at 先于第二桌面创建 time_ns |
|
||||
| 业务执行 | 没有 `saved_ok`,未进入正式替换 |
|
||||
| 完整图对照 | before 与 after 结构完全一致:1 图页、57 器件、103 引脚,包括全部已读取属性、位置、方向、缩放、引脚编号/位置及 net UID/name |
|
||||
| RV05 | 仍为 `4100-CA0000-T7`、Value `0R`,没有修改 |
|
||||
| 清理 | 两代资源均确认释放,最终 session `released`,无 pending resources/supervisors |
|
||||
| 用户桌面 | 输入桌面未变化,自有 CAD 未成为前台或出现在输入桌面 |
|
||||
| 原始输入 | 原工程目录与中央库 LMC 哈希不变 |
|
||||
|
||||
第一代 prepared SHA256 为 `89ae4b40d87359a429f38afc9c653c39dab6722a8a2d2526d881aea8cec54f0e`;第二代为 `8d9fe3a23c1dfaf629db066dd6858f91c45103c4d08691f265f27fffa7e282da`。两者分别与自身执行前字节比较一致。
|
||||
|
||||
整个探针耗时 233.891 秒,包含两次独立私有 CAD 的完整图读取。此前旧 EXE SHA `4e251627…` 的同类第一代故障阶段耗时 120.187 秒;新版此处为 0.328 秒。不能把两轮探针总耗时直接视为性能对照:旧轮还完成了一次真实替换,本轮连续失败后没有替换。
|
||||
|
||||
主要证据为 `report.json`(`verified=true`)、`fault-observations.json`(两次准确身份与 prepared 快照)、`before.json`、`after.json`、`desktop-observation.json`、`data/batch-resources/*/session.json`(generation/retry/timings/release)及两代 attempt JSON。
|
||||
|
||||
本轮验证新版提前检测与一次恢复预算用尽后停止。旧 EXE 的首次故障后恢复成功并保存替换已在独立文档记录;没有再次对已完成工程运行替换,也没有将旧 EXE 成功结果改称为新版同一用例通过。完整 GUI 故障注入与机器掉电仍未覆盖。
|
||||
@@ -0,0 +1,38 @@
|
||||
# Issue 3:真实 native 写入后冻结监督进程硬退出
|
||||
|
||||
2026-10-23,`tools/probe_frozen_postwrite_crash.py` 在两份全新工程副本上运行,最终实机退出码 0。冻结 EXE SHA256 为 `3b6a8608bac9d88f93ac71487d5c32647af0894c639d624f380f60345d27d34a`,源码基线为 264fee5。本例是**源码非 GUI parent + 冻结 supervisor/helper/VBS**,不是完整 GUI 故障注入。
|
||||
|
||||
真实通过证据目录为 `build/issue-3-frozen-postwrite-crash/run-c26f51c28caf47e1bf3f612801af9f33`。工程 `Audio_D2128`、`Audio_T605A` 从原 checkout 样例分别复制;不触碰正常 UI、marker 写保护或启动恢复验收副本。生产意图均为 RV05 的 `4100-CA0000-T7 → 4100-CA0000-T0`。
|
||||
|
||||
测试 parent 使用 `TaskCenterService(supervised=True)`,仅在测试进程中覆盖 `_command`,使生产请求调用真实冻结 EXE `--batch-worker`。没有修改生产源码、VBS 或添加生产故障开关。
|
||||
|
||||
监视器仅接受第一任务 helper 原子发布的 `native_replace_applied` / `native_replace_verified` 报告,并同时要求非空 replaced UID、对应 attempt `mutation_possible`、execution session generation=1 / retries 为空、下一任务仍 pending、没有可信最终 business-result。随后打开 supervisor journal 中的进程,核对其 PID 与创建时间,在仍存活时仅硬退出该明确自有冻结监督进程。
|
||||
|
||||
本轮触发瞬间为:
|
||||
|
||||
- `stage=native_replace_verified`,`replaced_uids=["$2I2014"]`;没有使用 `native_replace_started` 冒充写后证据。
|
||||
- mutation marker 为 `mutation_possible`,SHA256 `7f7cfa6dd6cc991289e3e883a74d17e507a88424a44aee1ab818528047137502`。
|
||||
- supervisor PID 38916,创建时间 `2026-10-23 08:48:49.752000+00:00`,与 journal 身份一致。
|
||||
- batch ID `07a50287d6b840b3b160b35c035eb3f1`。
|
||||
|
||||
| 检查 | 实机结果 |
|
||||
| --- | --- |
|
||||
| 正写后硬退出 | 已触发;保存完整 native report、marker、队列及进程身份后 `TerminateProcess` 该自有 supervisor |
|
||||
| parent 业务反馈 | `supervisor_interrupted`;没有可信 business-result,不伪报成功 |
|
||||
| 自动重复 | session generation=1、retries 为空、唯一 attempt;没有重新执行业务 |
|
||||
| 下一任务 | 第一任务 running、第二任务 pending;第二工程没有执行 status 文件,执行前后目录完整文件哈希一致 |
|
||||
| 保护发布 | 两参与任务均保留 `reason=supervisor_interrupted` 的 uncertain.flag |
|
||||
| 重启边界 | 关闭原 parent 的 store 后,启动全新 Python parent 进程打开原数据目录 |
|
||||
| 新 parent 操作 | 只启动冻结 `cleanup_batch_resources` 请求;没有 BOM 或队列执行请求 |
|
||||
| 资源复核 | execution session `released`;supervisor journal `finished / outcome=interrupted` |
|
||||
| 业务保护 | 新 parent 启动与资源清理后,uncertain.flag 内容、mutation marker 字节及其 SHA256 均未变化 |
|
||||
| 外部输入 | 原始两工程目录与库 LMC 哈希不变 |
|
||||
| 桌面 | 所有自有 CAD 确认退出、私有桌面确认消失;输入桌面前后一致,自有 CAD 不成为前台 |
|
||||
|
||||
“第二工程未启动”指执行队列未进入第二任务;BOM 导入仍按生产流程读取两工程。通过轮首工程导入时 native CLI 曾遇到路径 `WinError 5`,生产流程转入只读私有 CAD 读取并完成库存核验。这发生在故障监视启用之前,未额外注入或修改生产处理。总耗时 187.343 秒包含这一导入路径。
|
||||
|
||||
首轮 `run-39fa3d839de54a748992f3c06e86f61c` 的测试监视器断言失败,仅发合作取消;第一任务正常保存后停止,第二任务 pending,没有硬退出。该轮保留为失败证据,不计为 postwrite 故障通过,也未重放或复用其已完成工程。旧监视器未记录具体断言栈,不能据此证明确切失败行。工具随后改进可正常变化阶段的等待处理,并保存详细异常栈;上述通过轮来自另一份全新副本。
|
||||
|
||||
主要证据:`postwrite-crash.json`(真实触发快照)、`report.json`(`verified=true`)、`restart-resource-recheck.json`(全新 parent 的 resource-only 验证)、`inputs.json`(第二工程执行前哈希)、`desktop-observation.json`、生产队列与资源 journal、原 uncertain.flag。
|
||||
|
||||
本例没有声称崩溃后的第一工程是完整成功结果,也未自动打开第一工程做修复或再次替换。真实 native 写入已经发生,最终保存/恢复结果缺少可信业务收尾证据,因此保留保护并等待人工核验符合预期。完整 GUI 硬退出、系统/机器掉电,以及任意时点崩溃仍不在本例覆盖范围。
|
||||
@@ -0,0 +1,32 @@
|
||||
# Issue 3:冻结监督进程中的首次 CAD 启动失败恢复
|
||||
|
||||
2026-10-23,运行 `tools/probe_frozen_startup_recovery.py --failures 1`,实机退出码 0。使用独立全新 `Audio_D2128` 副本,与正常 UI 验收及 marker 写保护验收目录隔离。
|
||||
|
||||
本轮冻结 EXE SHA256 为 `4e251627d1ed6f22786de6febec971f889b6b1f2f69c86f19ccdeebd0d41ed95`。该版本首次启动故障等待约 120 秒的行为如实保留;后续检测 CAD 提前退出的源码/EXE 优化不得套用本轮耗时。
|
||||
|
||||
本轮是**源码非 GUI parent + 真实冻结 worker/helper/VBS**。parent 创建 `TaskCenterService(supervised=True)`,测试中仅替换 `batch_supervisor._command`,使 BOM 与队列请求均启动该冻结 EXE 的 `--batch-worker` 模式。不修改生产源码、生产 VBS 或增加注入开关。完整图读取为独立源码只读 inspector,在独立私有 CAD 中运行。该混合入口不等同于完整 GUI 故障注入。
|
||||
|
||||
真实证据目录为 `build/issue-3-frozen-startup-recovery/run-b266b219d8374db3b926d8a24a7dcc2f`。
|
||||
|
||||
故障线程仅监视本轮 `execute_task_queue` 的 session/私有 CAD 记录。终止前交叉核对 batch/session/generation、桌面名、记录中的 CAD PID、进程创建时间、`state=starting`、无 GUID、无 bootstrap 绑定、无 cad-ready checkpoint,以及对应 `prepared` attempt。所有条件满足后,只终止该明确自有 CAD;不附着外部 COM,不终止未知或外部进程。
|
||||
|
||||
| 检查 | 真实结果 |
|
||||
| --- | --- |
|
||||
| 第一次自有 CAD | PID 40836,创建时间 `2026-10-23 08:32:26.507000+00:00`;确认尚未授权工程 worker 后终止 |
|
||||
| 第一次 attempt | `prepared` 保持不变;SHA256 `79c88c6b81d3488952aa154e50ef878cca1d68462cadb91e8ceb0bad5ea08737` 未变化 |
|
||||
| 第一次桌面 | `dx-3e9ef3b9fd75476d9e7b8825bc6172be` 释放确认成功 |
|
||||
| 安全换代顺序 | 第一代 `released_at` 早于第二代桌面 handle-create `time_ns`;观察第二代时桌面枚举中旧桌面已经消失 |
|
||||
| 第二代 CAD | PID 33116,创建时间 `2026-10-23 08:34:27.099000+00:00`,新桌面 `dx-89dbf9636ccd461db04a95ef1df03dfa` |
|
||||
| 重试预算 | execution session generation=2,项目 retries 只有一条且 `count=1` |
|
||||
| 正式业务结果 | 一次 `saved_ok`,任务 completed;没有重复完成或再次替换 |
|
||||
| 独立图对照 | 57 器件、103 引脚;56 个非目标器件完全一致;RV05 原 UID `$2I2014`、位置/方向/缩放与连接保持不变 |
|
||||
| 目标结果 | 料号 `4100-CA0000-T0`,Part Label `0R`,保留 native target 的 Value 属性不存在状态 |
|
||||
| 清理 | session `released`;所有自有 CAD 退出、桌面消失;无 pending resource/supervisor |
|
||||
| 用户桌面 | 输入桌面未变化;自有 CAD 未成为前台或出现在输入桌面 |
|
||||
| 原始输入 | 原始工程目录与中央库 LMC 哈希保持不变 |
|
||||
|
||||
本轮总耗时 349.859 秒,包含独立前后图读取。第一代 `project_worker` 耗时 120.187 秒,记录的重试原因为 `private CAD bootstrap timed out`;第二代 CAD 启动 18.188 秒,第二代 `project_worker` 89.750 秒。未降低生产超时,也未把等待失败降为成功。
|
||||
|
||||
主要证据:`report.json`(`verified=true`)、`fault-observations.json`、`before.json`、`after.json`、`desktop-observation.json`、两代私有 CAD 记录,以及 `data/batch-resources/*/{request.json,supervisor.json,session.json,result.json,business-result.json}`。
|
||||
|
||||
本轮只验证首次 prewrite 启动失败后安全恢复一次。连续两次启动失败、优化后提前退出检测的完整冻结重测,以及完整 GUI 故障注入不在本轮覆盖范围;postwrite 监督进程崩溃使用另外的新副本单独验收。
|
||||
@@ -0,0 +1,44 @@
|
||||
# Issue 3:真实写保护使 mutation marker 发布失败
|
||||
|
||||
2026-10-23,在 Windows 本机独占 CAD 验收窗口,使用全新复制的 `Audio_D2128` 工程运行源码生产 `TaskCenterService → execute_task_queue → QuickPartReplace.vbs → native_helper`。替换意图为 RV05 的 `4100-CA0000-T7 → 4100-CA0000-T0`,不勾选复制料号到 Value。中央库为 `mentor_lib.lmc`。未修改 `src`、生产 VBS 或根代理的 `build/issue-3-ui-acceptance`。
|
||||
|
||||
实际工作目录为 `build/issue-3-marker-failure/run-e361ef90cd1d4145b1c68cdfb067f4a6`。工程从原 checkout 的 `build/cbb-bom-import-live-20261015/projects/Audio_D2128` 单独复制。执行前检查没有其他 CAD 实机测试进程,并与冻结 UI 验收协调串行进行。
|
||||
|
||||
测试进程仅包装 `BatchCadRuntime._prepare_attempts`:生产方法返回并发布 `prepared` 记录后,对该 JSON 调用真实 `win32file.CreateFile`,以 `GENERIC_READ`、`FILE_SHARE_READ | FILE_SHARE_WRITE` 打开,明确不允许 `FILE_SHARE_DELETE`。该句柄不妨碍读取 prepared 记录,但阻止 helper 的 `os.replace` 原子发布。句柄保持到该次生产队列退出,并在 `finally` 关闭;没有添加生产故障开关。
|
||||
|
||||
实际证据:
|
||||
|
||||
| 检查 | 结果 |
|
||||
| --- | --- |
|
||||
| prepared 发布与读取 | 成功;真实测试句柄 1524,可读取原 JSON |
|
||||
| helper 首次发布 mutation marker | `PermissionError: [WinError 5]`,拒绝原子替换 |
|
||||
| 真实调用栈 | `run_native → mark_snapshot_mutation → mark_mutation_possible → _publish → os.replace` |
|
||||
| 原生提交边界 | 失败发生于 `mark_snapshot_mutation`,在原生 Replace 按钮 `click(...)` 之前 |
|
||||
| VBS 状态 | `awaiting_native_replace → batch_failed`,失败工程关闭且未保存 |
|
||||
| 队列结果 | `blocked`、`paused=true`,没有 `saved_ok` |
|
||||
| 重复执行 | `_prepare_attempts` 只调用 1 次;本批 `retries={}`;没有再次执行替换 |
|
||||
| 标记内容 | 保持 `state=prepared`,执行前后 SHA256 完全一致 |
|
||||
| 资源 | 批次 session 为 `released`;CAD 退出与桌面消失均已确认 |
|
||||
| 独立回读 | 私有 CAD 枚举工程全部图页,并读取所有器件属性、位置、方向、缩放及引脚位置、编号、net UID/name;前后结构化数据完全相同 |
|
||||
| 对照范围 | 1 图页 `Audio-D2128:S905L CVBS-LR`,57 器件,103 引脚 |
|
||||
| RV05 | UID `$2I2014`、料号 `4100-CA0000-T7`、Value `0R`,属性及两条引脚连接保持原样 |
|
||||
| 原始输入 | 源工程目录完整文件哈希与中央库 LMC 哈希保持不变 |
|
||||
| 可见性 | 输入桌面前后均为 Default;自有 CAD 未出现在输入桌面或成为前台 |
|
||||
|
||||
prepared 文件前后 SHA256 为 `a2ee4d65d953613c50fa14fb49a22159a7d4cc3b9c3685281ffb672962c41d17`。
|
||||
|
||||
native report 的 `stage=native_replace_started` 在源码中位于 marker 写入之前,不能仅凭该 stage 判断已提交原生修改。本轮以真实异常调用栈、prepared 标记未变化、未保存状态,以及完整原理图前后独立对照共同判定。
|
||||
|
||||
首轮工具错误地要求出现后置 VBS guard 文案 `durable mutation proof failed`。实际上 native helper 更早的 marker guard 已经失败,因此该工具文案断言退出码为 1;原始 `report.json` 保留此失败,未删除或改写。修正测试识别真实 helper `_publish/os.replace` 异常后,只对同一次已执行工程运行 `--verify-existing` 只读回读,退出码为 0;该分支没有导入或队列执行调用,没有重放业务。
|
||||
|
||||
主要证据文件:
|
||||
|
||||
- `report.json`:生产队列结果、唯一 attempt、真实拒绝删除共享句柄、prepared 前后哈希、原输入不变及首轮工具断言失败记录。
|
||||
- `data/tasks/*/quick-run/status.tsv` 与 `snapshot-*.tsv.native.json`:真实生产状态、WinError 5 与调用栈。
|
||||
- `data/batch-resources/*/session.json`:释放确认与空重试记录。
|
||||
- `before.json`、`readback-only/after.json`:完整结构化读取结果;JSON 对象内容比较一致,属性枚举顺序可能不同,不以文本字节哈希代替结构比较。
|
||||
- `readback-only/verification.json`:`verified=true`、`graph_identical=true`、`marker_unchanged=true`、`resources_released=true`、`business_replayed=false`。
|
||||
- 两阶段的 `desktop-observation.json`:真实 CAD 退出、桌面消失及前台/输入桌面观察。
|
||||
- `source-under-test.json`:本轮源代码与 VBS 哈希清单。
|
||||
|
||||
本例覆盖源码服务/队列/真实 VBS/helper/真实私有 CAD 的实际 Windows 写保护失败。通过测试进程包装方法实施故障注入,没有覆盖冻结 EXE 的同类注入;冻结 UI 正常流程由另一轮验收单独覆盖,不混记为本例通过。
|
||||
@@ -0,0 +1,102 @@
|
||||
# 工单 1/3:长路径与安全自恢复实施、验收记录
|
||||
|
||||
后续用户已授权提供 Beta 3 工单复测包,见[发布构建记录](2026-10-23-beta3-release.md)。下文保留实现阶段各候选的验证边界,不将现场未覆盖项改记为完成。
|
||||
|
||||
本轮在 `codex/issue-3-safe-recovery` 实施,不发布版本、不清除旧保护锁。工程与中心库原件只用于复制和哈希核对。
|
||||
|
||||
## 已实施
|
||||
|
||||
- 写入前记录改存 `data/batch-resources/<批次>/attempts/g<代次>/<尝试>/`。目录、原子写入和兼容入口预检在桌面创建前完成。Python 使用扩展路径,CAD/VBS 保留短路径或受控目录联接入口;身份仍用规范路径。
|
||||
- GUI 通过同一 EXE 的隐藏工作模式启动独立监督进程。桌面、CAD、CLI 和工作 Job 属于监督进程;GUI 只持有监督进程自己的 Job。SQLite 连接不跨进程传递。
|
||||
- 请求有版本、批次与尝试 ID、请求摘要、父子 PID/创建时间和一次性 claim。消息检查序号、代次;结果持久化后通知。缺失或不可信的写结果保留实际任务证据目录的保护标记,不从头执行。
|
||||
- GUI 修改与监督进程准入使用同一跨进程互斥。启动恢复持锁到 runner 初始化结束;已占用则只查询。监督进程构造 runner 时不再次执行启动恢复。
|
||||
- 业务结果在收尾前另存;资源失败保留结果。BOM 的两个预览引用均撤销,防止资源未安全释放时保存部分计划。界面分别显示首发错误与释放错误。
|
||||
- 持久化缓存绑定工程完整指纹与载荷摘要;损坏缓存隔离后重新读取。跨监督进程可复用命中的只读采集结果。
|
||||
- 日志分为 GUI、监督进程和逐批释放日志;资源记录包含自有进程创建时间、句柄归属与关闭结果。诊断包有大小边界并脱敏。
|
||||
- 正常关闭归档默认关闭。启用后使用现有回收站事务;未释放、保护或待核验任务保留。崩溃不归档。
|
||||
|
||||
## 安全裁定与审查
|
||||
|
||||
Ruling: 不为桌面残留创建更多桌面 — 释放未证实时保持阻断 — 外部持有者必须释放后才准入。
|
||||
|
||||
Ruling: 心跳丢失只记通信异常 — 不提供“未修改”证明 — 不能授权重放。
|
||||
|
||||
Ruling: 保持现有 Beta 2 版本标识,仅生成未发布验收产物 — 本轮没有发布授权 — 候选哈希和分支用于区分,不能与公开下载混用。
|
||||
|
||||
独立审查发现并修正:冻结 windowed EXE 没有 `sys.stdout`;保留请求可被重复调用;BOM 清理失败仍带预览;旧式任务保护根目录不一致;启动恢复只瞬时检查锁导致多窗口竞争。最后对 `f3d7423` 和 `c787ef3` 的独立复审未发现新增实质安全问题。
|
||||
|
||||
## 验收边界
|
||||
|
||||
本机 `LongPathsEnabled=1`,尚无关闭系统长路径支持且禁用 8.3 的独立 Windows 环境。扩展路径、真实深目录及强制兼容入口测试只能证明其覆盖的分支,不能替代该环境的完整冻结 EXE 验收。
|
||||
|
||||
工单原始多工程文件不齐备。报告回放只验证已有原生报告对应的行,不算原批量工程替换实机验收。故障注入中的替身测试与真实进程/CAD 测试分别记录,不相互替代。
|
||||
|
||||
外部持有者真实测试见 [独立记录](2026-10-23-issue-3-external-holder.md):保持阻断、持有者退出后仅资源清理恢复、业务结果未变化;产品仍不能识别任意外部持有者。不得据此宣称工单 1 根因已根治。
|
||||
|
||||
## 最终验证
|
||||
|
||||
最终源码整合回归:**896 passed,1 条既有 pywinauto STA 警告,205.29 秒**。命令为 `python -m pytest -q tests/v1_1 tests/trial_task_center/test_application.py tests/trial_task_center/test_service.py tests/trial_task_center/test_execution_safety.py tests/part_replace/test_verification.py`;原始输出在 `build/issue-3-final-source-regression.log`。工作树差异检查通过。
|
||||
|
||||
源码构建门禁 885 项通过。实际冻结深目录检查额外发现 `CreateProcess` 不能使用过长工作目录:首次 274 字符数据目录启动失败,保留在 `build/issue-3-frozen-deep`。修正为经过身份核验的兼容工作目录,并保留首发 transport error;22 项深目录/监督/结果相关回归通过。重建候选后,280 字符数据目录的完整 EXE smoke 成功启动监督进程,创建并释放桌面、Job 和自有进程;证据为 `build/issue-3-frozen-deep-report.json`。
|
||||
|
||||
未发布候选 EXE SHA256:`4e251627d1ed6f22786de6febec971f889b6b1f2f69c86f19ccdeebd0d41ed95`。完整交互证据保留在 `build/issue-3-ui-acceptance`,不是源码服务替代 UI。两份全新副本为 Audio_D2128 和 Audio_T605A,其中前者工程路径含中文、长度 287 字符。
|
||||
|
||||
| 完整 EXE 交互检查 | 结果 |
|
||||
| --- | --- |
|
||||
| 创建两任务、导出默认 11 列模板并填入替换项 | 通过 |
|
||||
| 含错误位号 BOM | 通过 2/失败 1/未检查 0;8.172 秒;未部分保存 |
|
||||
| 修正后 BOM | 通过 2/失败 0/未检查 0;4.891 秒 |
|
||||
| 资源与缓存 | 首次导入原生 CLI,两工程一个桌面、CAD 未启动;第二次命中持久化缓存,不创建桌面 |
|
||||
| 计划与搜索 | 四条目标记录,每工程仅 RV05 启用;RV11 未启用;搜索不改变批次范围 |
|
||||
| 顺序替换、保存、关闭重开回读 | 两任务 completed,每工程一条 saved_ok 事件;175.984 秒 |
|
||||
| 执行资源 | 一个桌面、一代会话、一次 CAD 启动,无重建;最终进程退出、句柄关闭和桌面消失均确认 |
|
||||
| 内部写前证据路径 | 两个 attempt JSON,规范路径最长 254 字符;迁移后不再随工程历史目录嵌套增长 |
|
||||
| 主桌面 | 独立观察未发现自有 CAD 成为前台或进入输入桌面,持续文件读取成功 |
|
||||
| 独立工程读回 | 每工程 57 器件、103 引脚;56 个非目标器件全部字段一致,目标的身份、位置和连接保持,RV05 新料号与 Value/Part Label 为预期值 |
|
||||
| 原件 | 两源工程整目录哈希不变,中心库 `.lmc` 哈希不变;未声称完成整个库目录前后哈希对照 |
|
||||
| 正常关闭与重开 | 归档开关默认关闭;两次正常关闭后两 completed 任务保留;重开仍能查看实际结果 |
|
||||
| 界面诊断导出 | 成功导出 30 个文件,包含释放日志,不包含 DB、工程、BOM、EXE 或中心库文件 |
|
||||
|
||||
与已记录的 Beta 2 同组 Audio_D2128/Audio_T605A、每工程 RV05、开启 Part Label→Value 的一次交互观测对比:
|
||||
|
||||
| 候选 | 执行总耗时 | CAD 启动 | 执行桌面创建 |
|
||||
| --- | --- | --- | --- |
|
||||
| 公开 Beta 2 的既有验收 | 209.406 秒 | 1 | 1 |
|
||||
| 本轮隔离监督候选完整 GUI 验收 | 175.984 秒 | 1 | 1 |
|
||||
|
||||
本轮另包含一个 287 字符工程路径。两次观测的缓存、机器负载及目录不同,没有统计重复,因此不能据此承诺稳定加速比例;主要收益是 GUI 与资源所有权分离,以及释放后独立复核。基线出处为 `2026-10-23-issue-2-frozen-runtime.md`。
|
||||
|
||||
独立读回证据为 `graph-before-report.json`、`graph-after-report.json` 和逐工程 `graph-read`;主桌面观察为 `desktop-observation-execute.json`。默认关闭证据为 `normal-close-report.json`。诊断探针曾在异步 ZIP 生成完毕前读取文件而失败,随后只读验证已完成的同一 ZIP 通过,未重新导出。
|
||||
|
||||
工单 BOM 报告回放为 `build/issue-3-bom-report-replay/report.json`:428 个原任务的副本中,仅现有 6 工程原生采集报告覆盖 31 行,20 通过、11 失败,与原行/工程/状态一致;0 资源创建、0 计划写入、原证据哈希不变。另有 406 行/96 工程缺少采集证据,未覆盖。
|
||||
|
||||
真实标记持久化失败检查见 [标记失败实机记录](2026-10-23-issue-3-marker-write-failure.md):生产 helper 因 deny-delete 持有句柄在原子发布时返回 WinError 5,唯一尝试终止,无 saved_ok;只读回读确认 57 元件、103 引脚完全一致。该故障注入以源码 parent/helper 运行,不替代冻结 GUI 的全部故障案例。
|
||||
|
||||
跨路径共享重试预算、不同工程独立预算以及真实子进程硬退出前后结果保护的证据边界见 [监督安全回归](2026-10-23-supervisor-safety-regressions.md)。
|
||||
|
||||
首次真实冻结启动故障恢复见 [冻结启动记录](2026-10-23-issue-3-frozen-startup-recovery.md):一代故障只重建一次,旧桌面消失后才创建二代,最终 saved_ok 恰一次且完整图对照通过。该实验发现 CAD 已退出后仍等待探针约 120 秒;后续 `264fee5` 在等待期间检查明确自有 CAD 的退出码,及时进入同一清理路径,未增加或绕过重试预算。新增用例先失败再通过,相关 65 项回归通过,独立审查确认没有跳过清理或复用死进程。
|
||||
|
||||
最后三行启动检测优化后的候选 SHA256 为 `3b6a8608bac9d88f93ac71487d5c32647af0894c639d624f380f60345d27d34a`,280 字符数据目录的冻结监督、Job/进程身份及桌面释放再次通过,证据在 `build/issue-3-final-frozen-runtime.json`。上表完整 GUI 交互属于前一候选 `4e251627…`,不能混淆两个产物的实测范围;之后的源码更改仅限 CAD 启动等待检测。
|
||||
|
||||
该候选的真实写后硬退出检查已通过,详见 [写后崩溃与仅资源恢复](2026-10-23-issue-3-frozen-postwrite-crash.md):在 `native_replace_verified`、明确 mutation marker 和自有监督 PID/创建时间均核对后终止监督进程;未知业务结果未伪报成功,无重试,第二工程仍 pending 且文件哈希不变。全新 parent 只请求资源清理,保护与 marker 字节保留。该例仍是源码非 GUI parent+真实冻结 supervisor/helper/VBS,不将其描述为 GUI 硬退出或掉电验收。
|
||||
|
||||
同候选 [连续两次启动失败实机记录](2026-10-23-issue-3-frozen-double-startup-failure.md) 通过:两次故障分别在 0.328/0.250 秒被检测,generation=2、重试 count=1,第二次失败后停止,没有第三代或 saved_ok。两个 prepared 记录字节不变,57 元件/103 引脚完整图前后一致。总 233.891 秒包含前后独立 CAD 图读取,不能把故障检测耗时当成全流程耗时。
|
||||
|
||||
## 最终路径补漏与冻结产物
|
||||
|
||||
`3367bb6` 只补扫描和归档的文件访问边界。受限长路径分支的 failing-first 测试证实:原扫描不能访问深工程,原孤立保护检测可能把没有 handoff 的受保护任务归档。修正后深工程可扫描,孤立保护任务 retained,保护文件未变化。30 项相关回归通过;独立审查确认公开路径身份、祖先/reparse/containment 检查未放宽。
|
||||
|
||||
该补丁后的最终未发布 EXE SHA256 为 `f38411a2d91f30461e1441249120b461a7f56ad3f2a0c4a0a95a817037fd9798`。最终冻结验证:
|
||||
|
||||
- 285 字符数据目录 smoke 通过真实监督、进程身份、Job 与私有桌面释放;`build/issue-3-final-path-frozen-runtime.json`。
|
||||
- 完整 GUI 在 296 字符数据目录扫描 302/152 字符工程路径,创建两任务并导出含两工程的 11 列默认模板;公开路径无扩展前缀泄漏,归档默认关闭,正常退出后任务保留,全程没有 CAD/私有桌面资源;`build/issue-3-final-scan-ui/report.json`。
|
||||
- 完整替换与故障注入对应上文注明的前两候选。最后三个模块只改扫描/孤立保护的 I/O,未再次重放已完成替换;不同候选的证据分别保留,不混写为同一个产物的整套验收。
|
||||
|
||||
## 未满足的严格验收条件
|
||||
|
||||
- 未取得 LongPathsEnabled=0 且 8.3 不可用的独立系统,不能宣称该组合环境完整冻结 EXE 验收完成。
|
||||
- 工单原工程的真实批量替换未覆盖,逐工程/原 Excel 行号见 [102 个原工程清单](2026-10-23-issue-3-uncovered-projects.md);其中 96 个连报告回放也未覆盖。
|
||||
- 工单 1 现场桌面残留的具体持有者仍未确定。新隔离设计与自有/人工外部持有者的释放测试通过,但不能宣称现场根因已根治。
|
||||
- 故障实机入口为源码非 GUI parent+冻结 worker/helper/VBS;没有完整 GUI 硬退出或掉电验收。原库只核对 `.lmc` 哈希,未完成整个库目录内容的前后哈希基线。
|
||||
|
||||
因此本轮代码与证据可审查,仍不标为“所有严格验收条件完成”,不自动发布版本或清除旧保护。
|
||||
@@ -0,0 +1,110 @@
|
||||
# 工单 BOM 原工程实机未覆盖清单
|
||||
|
||||
本清单按原 BOM 工程路径身份分组,只展示文件名与不可逆标识,避免发布客户目录。102 个有替换行的原工程均未取得可用于本轮完整实机替换的工程文件;其中 6 个仅有原生报告覆盖的 31 行回放。两个 Audio 全新样例的实机验收不能替代下列原工程验收。
|
||||
|
||||
其余 96 个工程的 406 行连报告回放也未覆盖。Excel 行号对应原 `reported-bom.xlsx`,不是删减后的新编号。
|
||||
|
||||
| 工程文件/路径身份摘要 | 原 BOM Excel 行 | 报告回放行数 | 原工程实机 |
|
||||
| --- | --- | --- | --- |
|
||||
| 1gphy_rtl8211f.prj · 9a3147d0e1 | 3, 4, 5, 6 | 4 | 未覆盖 |
|
||||
| 1gphy_rtl8211f_a.prj · 03b0761631 | 7, 8, 9, 10 | 4 | 未覆盖 |
|
||||
| 1gphy_yt8511c.prj · 2a63ba7f0c | 11, 12, 13, 14, 15 | 5 | 未覆盖 |
|
||||
| 2g5phy_gpy211.prj · 9f2961f1db | 30, 31, 32, 33, 34, 35 | 6 | 未覆盖 |
|
||||
| 2gphy_zx5201.prj · b704fb0870 | 16, 17, 18, 19 | 4 | 未覆盖 |
|
||||
| audio_d2128.prj · f5c5728ea2 | 37, 38, 39, 40, 41, 42, 43 | 0 | 未覆盖 |
|
||||
| audio_d2128_esd.prj · 0786c112bd | 44, 45, 46, 47, 48, 49, 50 | 0 | 未覆盖 |
|
||||
| audio_d2128_surge_esd.prj · 1326530805 | 51, 52, 53, 54, 55, 56, 57, 58 | 0 | 未覆盖 |
|
||||
| audio_t605a.prj · 6249e3b8e4 | 60, 61, 62, 63, 64, 65, 66 | 0 | 未覆盖 |
|
||||
| bob.prj · 1a40673ab4 | 67, 68, 69, 70, 71 | 0 | 未覆盖 |
|
||||
| bob_shield.prj · fc8d09c983 | 72, 73, 74 | 0 | 未覆盖 |
|
||||
| bob_xpon.prj · 943ccbc2f7 | 77, 78, 79 | 0 | 未覆盖 |
|
||||
| bt_rtl8761.prj · 6fdc0b230e | 80, 81, 82, 83, 84, 85, 86, 87 | 0 | 未覆盖 |
|
||||
| ca_card_high.prj · 6116093290 | 91, 92, 93, 94, 95, 96, 97 | 0 | 未覆盖 |
|
||||
| ca_card_low.prj · abd489c265 | 98, 99, 100 | 0 | 未覆盖 |
|
||||
| ca_nds_msd5s69.prj · 312be1ad3f | 101, 102, 103 | 0 | 未覆盖 |
|
||||
| dvbt_ocp.prj · 4c5e90a1be | 312, 313 | 0 | 未覆盖 |
|
||||
| dvbt_ocp_od_gate.prj · 7adda0c253 | 314, 315 | 0 | 未覆盖 |
|
||||
| ethernet_1g.prj · da6029740f | 322 | 0 | 未覆盖 |
|
||||
| ethernet_a.prj · 00d8b04fa6 | 324, 325 | 0 | 未覆盖 |
|
||||
| ethernet_a_emc.prj · ab351a144d | 326, 327 | 0 | 未覆盖 |
|
||||
| ethernet_v.prj · 1e78d4ad40 | 328, 329 | 0 | 未覆盖 |
|
||||
| ethernet_v_camera.prj · def269650f | 330, 331 | 0 | 未覆盖 |
|
||||
| fem_kct8529h.prj · 13586535a9 | 337, 338, 339, 340, 341, 342, 343 | 0 | 未覆盖 |
|
||||
| fem_sky85717.prj · 8c12fd9b42 | 353, 354, 355, 356, 357, 358, 359 | 0 | 未覆盖 |
|
||||
| ldo1117_to252.prj · 053f19cdcd | 367 | 0 | 未覆盖 |
|
||||
| ldo1117_to252_0.prj · 3c4b4ecc60 | 368 | 0 | 未覆盖 |
|
||||
| ldo1117_to252_a.prj · bbc4b07fe2 | 369 | 0 | 未覆盖 |
|
||||
| ldo1117_to252_b.prj · 1a1cdc8bfe | 370 | 0 | 未覆盖 |
|
||||
| ldo1117_to252_c.prj · 9f80a38c68 | 371 | 0 | 未覆盖 |
|
||||
| ldo1117_to252_d.prj · c5e2a54807 | 372 | 0 | 未覆盖 |
|
||||
| ldo_1117.prj · 4229b7c0a0 | 373 | 0 | 未覆盖 |
|
||||
| ldo_1117_1v5.prj · 3e886188ae | 374 | 0 | 未覆盖 |
|
||||
| ldo_1117_1v5.prj · ed5dace4ba | 375 | 0 | 未覆盖 |
|
||||
| ldo_1117_1v5.prj · 32c52f6437 | 376 | 0 | 未覆盖 |
|
||||
| ldo_1117_1v8.prj · 9ca917fe09 | 377 | 0 | 未覆盖 |
|
||||
| ldo_1117_1v8.prj · 009ab46382 | 378 | 0 | 未覆盖 |
|
||||
| ldo_1117_1v8.prj · fe1ea827e2 | 379 | 0 | 未覆盖 |
|
||||
| ldo_1117_3v3.prj · d594fc61a9 | 380 | 0 | 未覆盖 |
|
||||
| ldo_1117_3v3.prj · 56024698a7 | 381 | 0 | 未覆盖 |
|
||||
| ldo_1117_3v3.prj · 9a4d3ba5d0 | 382 | 0 | 未覆盖 |
|
||||
| ldo_1117_a.prj · 1c64ec79aa | 383 | 0 | 未覆盖 |
|
||||
| ldo_1117_b.prj · bcb4b4017a | 384 | 0 | 未覆盖 |
|
||||
| ldo_1117_c.prj · 8bcbbfbafe | 385 | 0 | 未覆盖 |
|
||||
| ldo_1117_d.prj · 546091705e | 386 | 0 | 未覆盖 |
|
||||
| ldo_s2032_sot23.prj · 65da610a4c | 392, 393, 394 | 0 | 未覆盖 |
|
||||
| ldo_s2032_sot23_1v5.prj · 986a46e1b8 | 395, 396, 397 | 0 | 未覆盖 |
|
||||
| ldo_s2032_sot23_1v5_a.prj · 65333edf53 | 398, 399, 400 | 0 | 未覆盖 |
|
||||
| ldo_s2032_sot23_1v8.prj · 151ac70306 | 401, 402, 403 | 0 | 未覆盖 |
|
||||
| ldo_s2032_sot23_1v8_a.prj · d65baa1d59 | 404, 405, 406 | 0 | 未覆盖 |
|
||||
| ldo_s2032_sot23_1v8_b.prj · f43fc74710 | 407, 408, 409 | 0 | 未覆盖 |
|
||||
| ldo_s2032_sot23_2v5.prj · 3b8ebfee28 | 410, 411, 412 | 0 | 未覆盖 |
|
||||
| ldo_s2032_sot23_2v5_a.prj · 76a6f10c07 | 413, 414, 415 | 0 | 未覆盖 |
|
||||
| ldo_s2032_sot23_a.prj · 9afe3eed88 | 416, 417, 418 | 0 | 未覆盖 |
|
||||
| ldo_wl2811ea_sot23_1v5.prj · 03b146975a | 419, 420, 421 | 0 | 未覆盖 |
|
||||
| ldo_wl2811ea_sot23_1v8.prj · 9152c392ee | 422, 423, 424 | 0 | 未覆盖 |
|
||||
| ldo_wl2811ea_sot23_2v5.prj · b701d5b2e4 | 425, 426, 427 | 0 | 未覆盖 |
|
||||
| ldo_wl2811ea_sot23_adj.prj · c9f627eb7d | 428, 429, 430 | 0 | 未覆盖 |
|
||||
| lnb_mp8124.prj · c10e7bd2c2 | 437, 438, 439, 440, 441 | 0 | 未覆盖 |
|
||||
| lnb_mp8124_nodec.prj · 3a4685a09b | 432, 433, 434, 435, 436 | 0 | 未覆盖 |
|
||||
| lnb_tps65233.prj · 7cebbb12e6 | 442, 443, 444, 445, 446, 447, 448 | 0 | 未覆盖 |
|
||||
| mos-integrated-nmos.prj · bb34f6614a | 450 | 0 | 未覆盖 |
|
||||
| nandflash_bga63.prj · 31807ad105 | 461, 462 | 0 | 未覆盖 |
|
||||
| nandflash_tsop48.prj · 3b2f668c42 | 463 | 0 | 未覆盖 |
|
||||
| nor_flash_sop8.prj · 2aa42f724f | 464 | 0 | 未覆盖 |
|
||||
| ovp_circuit.prj · 0219c48734 | 518 | 0 | 未覆盖 |
|
||||
| pef7071_phy.prj · 008a1a6507 | 20, 21, 22, 23, 24, 25, 26, 27 | 8 | 未覆盖 |
|
||||
| rtc5638he.prj · 778fb484d5 | 344, 345, 346, 347, 348, 349, 350, 351, 352 | 0 | 未覆盖 |
|
||||
| rtl8197x.prj · 86fcfef21c | 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529 | 0 | 未覆盖 |
|
||||
| sd_card_high_cost_esd.prj · cda8210a8c | 530, 531, 532 | 0 | 未覆盖 |
|
||||
| sd_card_low_cost_esd.prj · ef0d2bb5f5 | 533, 534, 535 | 0 | 未覆盖 |
|
||||
| sensor_gc2063_sc3335_sc200ai.prj · 981cd40966 | 536, 537, 538 | 0 | 未覆盖 |
|
||||
| speaker_class-d-audio-amplifier.prj · 206b1c3026 | 539, 540, 541 | 0 | 未覆盖 |
|
||||
| spi_nand_flash.prj · fafa5d4252 | 542 | 0 | 未覆盖 |
|
||||
| t18250_in_loop_filter_shield.prj · e4f2dc6748 | 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553 | 0 | 未覆盖 |
|
||||
| telephony_interface_voice_si32192.prj · 24086dd2ab | 554, 555 | 0 | 未覆盖 |
|
||||
| telephony_interface_voice_si32280.prj · 178a144ca1 | 556, 557, 558, 559, 560, 561, 562, 563, 564 | 0 | 未覆盖 |
|
||||
| tuner+demod_dvb-c_m88rc6800.prj · 0f840071d7 | 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575 | 0 | 未覆盖 |
|
||||
| tuner_atbm253.prj · 110dc6f5c0 | 587, 588, 589, 590, 591 | 0 | 未覆盖 |
|
||||
| tuner_demod_mxl248.prj · 26594f8a75 | 577, 578, 579, 580, 581, 582, 583, 584, 585, 586 | 0 | 未覆盖 |
|
||||
| tuner_demod_mxl248.prj · e299962e81 | 592, 593, 594, 595, 596, 597, 598, 599, 600, 601 | 0 | 未覆盖 |
|
||||
| tuner_mxl608_shield.prj · cf17607708 | 602, 603, 604, 605, 606, 607, 608, 609, 610 | 0 | 未覆盖 |
|
||||
| tuner_r58150.prj · 537ed3b2c1 | 611, 612, 613, 614, 615, 616 | 0 | 未覆盖 |
|
||||
| tuner_r836.prj · cca63e4916 | 617, 618, 619, 620, 621, 622, 623 | 0 | 未覆盖 |
|
||||
| tuner_r836_a.prj · e41cc11d66 | 624, 625, 626, 627, 628, 629, 630 | 0 | 未覆盖 |
|
||||
| tuner_r836_h_cbb.prj · 882836fa9e | 631, 632, 633, 634, 635, 636, 637, 638, 639, 640 | 0 | 未覆盖 |
|
||||
| tuner_rda5815_dip+snaky_lines.prj · 78145e0625 | 641, 642, 643, 644, 645, 646, 647, 648 | 0 | 未覆盖 |
|
||||
| tuner_rda5815_double_tuner.prj · 662d874266 | 649, 650, 651, 652, 653, 654, 655, 656 | 0 | 未覆盖 |
|
||||
| tuner_rda5815_smd+gdt+filter.prj · 6f23c09e9e | 657, 658, 659, 660, 661, 662, 663, 664 | 0 | 未覆盖 |
|
||||
| tuner_rda5815_smd+gdt.prj · 9583c3ea34 | 679, 680, 681, 682, 683, 684, 685 | 0 | 未覆盖 |
|
||||
| tuner_rda5815_smd+snaky_lines.prj · e08bababf6 | 504, 505, 506, 507, 508, 509, 510 | 0 | 未覆盖 |
|
||||
| tuner_rda5815_smd+snaky_lines.prj · f9a967caf1 | 665, 666, 667, 668, 669, 670, 671 | 0 | 未覆盖 |
|
||||
| tuner_rda5815_smd+snaky_lines_a.prj · 6b2d71dc48 | 490, 491, 492, 493, 494, 495, 496 | 0 | 未覆盖 |
|
||||
| tuner_rda5815_smd+snaky_lines_a.prj · 5e6177a7de | 497, 498, 499, 500, 501, 502, 503 | 0 | 未覆盖 |
|
||||
| tuner_rda5815_smd+snaky_lines_a.prj · fd2fe146f9 | 672, 673, 674, 675, 676, 677, 678 | 0 | 未覆盖 |
|
||||
| tuner_t18250.prj · 72c882b7d6 | 686, 687, 688, 689, 690, 691, 692, 693, 694 | 0 | 未覆盖 |
|
||||
| wifi-8192fr-2l.prj · be5eb866d0 | 722, 723 | 0 | 未覆盖 |
|
||||
| wifi-rtl8188ftv-1t1r-2l.prj · e33a176a9b | 724, 725, 726, 727 | 0 | 未覆盖 |
|
||||
| wifi_8192er.prj · d5c71c558b | 729, 730, 731, 732 | 0 | 未覆盖 |
|
||||
| wifi_rtl8192fc.prj · d5b6659208 | 733, 734, 735, 736, 737 | 0 | 未覆盖 |
|
||||
| wifi_rtl8723du.prj · 88e300dc79 | 738, 739, 740, 741 | 0 | 未覆盖 |
|
||||
| xview.prj · 460b5b6322 | 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753 | 0 | 未覆盖 |
|
||||
@@ -0,0 +1,13 @@
|
||||
# Supervisor safety regression evidence boundaries
|
||||
|
||||
These automated tests do not open CAD or certify schematic property writes.
|
||||
|
||||
| Test | Real behavior exercised | Replaced behavior |
|
||||
| --- | --- | --- |
|
||||
| `test_mock_native_read_recovery_spends_same_project_worker_retry_budget` | Durable source hashes and attempt proof, runtime rebuild decision, shared native/worker project budget | Desktop/CAD ownership and worker failure |
|
||||
| `test_mock_different_projects_each_get_one_retry_without_accumulating_budget` | Two project identities, one rebuild per project, successful execution cannot replenish a spent budget | Desktop/CAD ownership and monitored worker execution |
|
||||
| `test_real_supervisor_process_crash_reconciles_without_replaying_mock_business` | Separate Python child, production admission and one-shot claim, process creation identity, hard `os._exit`, durable saved-result fallback, uncertainty protection at legacy/project-task evidence roots, separate cleanup child | Business replacement writes a durable witness instead of using CAD; no private desktop or CAD resources are acquired |
|
||||
|
||||
The process test covers both an unknown outcome before publication and a completed business result published before a crash. Cleanup must neither execute the witness operation again nor alter an earlier completed result. Unknown outcomes retain their original protection marker after reconciliation; proven completed outcomes do not acquire an uncertainty marker.
|
||||
|
||||
Remaining physical acceptance: inject failures in the frozen executable before and after the actual first CAD mutation, prove marker/helper durability failure prevents every production write/save path, and confirm actual CAD/desktop release and independent schematic readback. The tests above do not provide that evidence.
|
||||
@@ -0,0 +1,134 @@
|
||||
# Day0:保存、关闭、退出全链路排查
|
||||
|
||||
2026-10-26,北京时间。源码基线 `b9391015650bf5c43a77a017c5484e9a8f352266`。承接[资源事故分析](2026-10-26-day0-resource-release-analysis.md),本轮以核心应用完成条件为主线。
|
||||
|
||||
## 结论
|
||||
|
||||
发现并验证了四个具体的生命周期问题:
|
||||
|
||||
1. **CAD 主进程正常退出后,程序会立即终止 Job 中的剩余辅助进程,没有先给予其自然退出窗口。** 本机实测 `vnsd.exe` 再等约 0.813 秒即可自然退出;这个等待可以避免该次强制终止。
|
||||
2. **退出阶段的记录写入失败会跳过正常 Quit。** 随后仍进入自有进程清理。用生产退出控制流和替身 CAD 复现。
|
||||
3. **保存调用返回、工程关闭失败时,“已保存”标记仍可能没有设置。** 实际 VBS 函数配合替身 CAD 复现,说明保存与关闭的状态记录边界混在一起。
|
||||
4. **物理资源复核通过后,上层可能返回已完成业务结果,同时不再向调用方抛出正常退出失败。** 生产运行时记录可以出现 `cad_shutdown.clean=false`、资源状态 `released`、监督记录 outcome 为 `error`,但函数返回 completed 任务结果。日志证据保留,调用方的返回语义却丢失了退出异常。
|
||||
|
||||
这些问题与“只盯桌面释放”的分析不同:资源消失不能代表应用正常退出,更不能取代数据保存确认。
|
||||
|
||||
同时,两份全新工程副本的真实替换、保存、退出及新 CAD 实例独立回读均通过,未发现本轮保存数据丢失。上述问题不是已经证明的工单 #1 唯一根因;原附件中的 #1 属于读取后的收尾失败,#3 尚未启动 CAD。现场版本、实际触发路径和完整诊断仍须对照。
|
||||
|
||||
## 一、实机正常链路
|
||||
|
||||
使用已有 Audio_D2128、Audio_T605A fixture 的全新副本,生产服务入口执行 BOM 导入及两工程队列,每工程仅 RV05:`4100-CA0000-T7 → 4100-CA0000-T0`,保留原生目标 Value 策略。原始 fixture 未修改。
|
||||
|
||||
证据目录:`build/day0-resource-analysis-20261026/live-chain-bbd9fba7ce444e3d8e5627a6f4a385d0/`。汇总:`build/day0-resource-analysis-20261026/live-chain-summary.json`。进程时间线为同级 `live-chain-bbd9fba7ce444e3d8e5627a6f4a385d0-process-timeline.jsonl`。
|
||||
|
||||
| 检查 | 本次结果 |
|
||||
| --- | --- |
|
||||
| BOM 导入 | 2 行通过,3.875 秒 |
|
||||
| 顺序执行 | 两任务 completed,155.843 秒,共用一个执行 CAD |
|
||||
| 保存关闭图页调用 | D2128 约 0.32 秒,T605A 约 0.21 秒 |
|
||||
| 保存后的 CloseProject | 两次返回 True,耗时约 1.68 / 1.33 秒;均先观察到 ProjectClosed 事件,再观察到调用返回 |
|
||||
| 同实例重开回读 | 每工程完成内容核验后才发布 saved_ok,随后关闭工程 |
|
||||
| CAD 自然退出 | 本轮五个 CAD 实例(前读 2、执行 1、后读 2)主进程均正常退出码 0,无 CAD 主进程 terminate_request |
|
||||
| CAD 退出等待 | Quit helper 成功后的主进程等待约 0.234–0.390 秒,不是整个 Quit 的总耗时 |
|
||||
| 退出后的独立回读 | 每工程用新 CAD 实例读取;57 器件、103 引脚连接,56 个非目标器件不变,目标身份、位置、连接和目标料号正确 |
|
||||
| 原件 | 两个源工程目录哈希不变,中央库 `.lmc` 哈希不变;未宣称整个库目录全部文件无变化 |
|
||||
| 前台与资源 | 观察器未发现自有 CAD 进入输入桌面或成为前台,已记录 CAD 退出和桌面消失 |
|
||||
|
||||
这轮运行的是源码服务层、真实 CAD 和加入边界日志的 VBS 副本;没有修改生产源码。不是完整冻结 GUI/监督进程验收,时序日志可能增加少量开销,耗时不能作为性能承诺。
|
||||
|
||||
第一轮记录 `live-chain-366340e7a6a942ccadaf2e1eedd9c4f4` 因探针复制 VBS 时引入 UTF-8 BOM,导致第 1 行编译失败,替换前停止。该失败属于探针,不是产品故障;保留原记录和阻断状态,修正编码后使用另一组全新副本,未重放该次失败任务。
|
||||
|
||||
## 二、已验证问题与含义
|
||||
|
||||
### A. 辅助进程尚未自然退出,就调用 TerminateJobObject
|
||||
|
||||
位置:`isolated_desktop.py:380` 的 `_Win32Process.close`。处理顺序为:确认/停止根进程 → 收集后代 → **TerminateJobObject** → 最多等待 5 秒确认进程退出。因此这 5 秒是强制终止后的确认窗口,不是辅助进程正常收尾的宽限期。
|
||||
|
||||
两工程实机中五次 CAD 主进程退出后,Job 都仍有一个成员。原时间线的映像查询使用了本机 pywin32 未提供的函数,查询错误已保留;不能凭 PID 将五个成员全部命名。独立 CIM 快照将最后一次回读的 CAD PID 55500、子进程 PID 45368 对应到实际 `vnsd.exe`。
|
||||
|
||||
随后单独进行空 CAD 实验,改用 Win32 `QueryFullProcessImageNameW` 确认成员身份。仅推迟探针中对已正常退出 CAD 的 Job 清理,最长观察预算 75 秒,未打开任何工程:
|
||||
|
||||
- CAD PID 60332 正常退出码 0。
|
||||
- 此时 `vnsd.exe` PID 34960 仍存活。
|
||||
- 约 **0.813 秒**后该辅助进程自行退出,随后原清理函数运行时已没有需终止的后代。
|
||||
- 最终自有资源全部释放,桌面检查耗时 0 秒,整个关闭阶段约 2.625 秒。
|
||||
|
||||
证据:`build/day0-resource-analysis-20261026/empty-cad-descendants-9fd000b5f0af48dc82e12b6b034d2917/report.json`;探针 `probe_descendant_exit.py` 位于同级分析目录。
|
||||
|
||||
既有原生进程研究将 `vnsd.exe` 识别为许可证辅助程序,另有工程数据库 `iCDBNetServer.exe` 和交叉探查 `xprobSvr.exe`。本次空 CAD 仅验证 vnsd 的自然退出,不证明数据库提交被终止,也不证明它就是工单现场残留桌面的持有者。
|
||||
|
||||
建议方向:对已经正常退出的根进程,其明确自有的后代先进行有界自然退出等待;到期才进入可审计的强制清理。区分根进程、数据库辅助进程、许可证辅助进程;记录终止对象、原因及发生时的业务阶段。不能只把等待常量变大,也不能对不明进程进行清理。
|
||||
|
||||
### B. 收尾记录的 I/O 错误会改变应用退出方式
|
||||
|
||||
位置:`isolated_cad.py:207`。正常 Quit 前的 `_record(state='closing')` 与 Quit 在同一个 try 内,记录失败会直接跳入异常处理,接着 finally 仍关闭自有资源。
|
||||
|
||||
故障注入:仅让这一次 closing 记录写入抛 PermissionError,其他记录正常。结果为:没有发出 Quit;写入 uncertain;调用资源 close。真实 CAD 由替身代替,因此本项证明控制流缺陷,没有实际杀死用户 CAD。
|
||||
|
||||
含义:目录权限、同步软件/安全软件的文件访问冲突、磁盘问题等不只影响日志,还可能改变应用本来可以正常退出的路径。上述环境来源只是候选,没有确定具体软件责任。
|
||||
|
||||
建议方向:隔离诊断/资源收尾记录错误与正常退出协议;写前业务保护失败仍应禁止写入,但不能混同为“所有诊断写入失败都必须跳过 Quit”。退出请求仍须满足实例身份、工程状态等原有条件。
|
||||
|
||||
### C. 保存状态在工程关闭成功之后才记录
|
||||
|
||||
位置:`QuickPartReplace.vbs:934`。调用 `QPCloseBatchProject(TargetPath, True)` 返回后才执行 `QP_BatchSaveCommitted=True`;这个子程序内部既保存图页,又关闭工程。
|
||||
|
||||
用真实 `QPFinishBatchItem` / `QPCloseBatchProject` 函数与 CAD 形状替身运行:图页的 SaveChanges=True 调用已经返回,随后工程拒绝关闭。观察到 `save_called=True`、`commit_flag=0`、错误已抛出,没有发布成功 checkpoint。
|
||||
|
||||
因此,这个标记不能独立表达“保存已发出/可能完成,而工程关闭失败”。某些后续错误文案依赖它,可能把已涉及保存的故障描述为未保存。`failed_native_result_uncertain` 仍会依据原生修改证据阻断重试,本轮没有发现借此自动重放替换;问题主要在恢复判断所需的状态粒度和事实描述。
|
||||
|
||||
建议方向:拆分 save_requested、save_returned、saved_readback_verified、project_closed 等证据;保存请求开始后发生异常应保留“可能已保存”的状态,不能因 CloseProject 失败回退成“未保存”。
|
||||
|
||||
### D. 资源消失后,正常退出异常不再向调用方抛出
|
||||
|
||||
位置:`batch_supervisor.py:159` 的 `_finish_response`,及 `batch_resources.py:117` 的重查逻辑。
|
||||
|
||||
用生产 BatchCadRuntime 创建合法资源记录,替身 CAD 在退出时抛出 Quit 超时,已保存业务结果独立保留;模拟拥有资源的监督进程结束后,再调用生产资源复核及返回函数。结果:
|
||||
|
||||
- 会话先 blocked;物理桌面已不存在后变为 released。
|
||||
- `cad_shutdown.clean=false` 和 `orderly_shutdown` 异常仍保留在 JSON。
|
||||
- 监督记录 `state=finished`、`outcome=error`。
|
||||
- 调用方收到含 completed 条目的 TaskQueueResult,没有收到退出异常。
|
||||
|
||||
这是状态返回语义不一致,不是原始错误日志被删除,也不证明已独立核验的保存结果应当被作废。应保留正确业务结果,同时显式暴露“业务完成、应用异常退出/强制清理”的收尾状态,不能仅凭物理资源消失将异常归为恢复成功。
|
||||
|
||||
上述 B/C/D 复现实物:`build/day0-resource-analysis-20261026/fault-chains-907a534cc6bb441e8cddf7670dd04aac/`,脚本 `probe_fault_chains.py`。较早的临时探针曾因日志文件句柄尚未关闭而无法删除临时目录;后续改为保留独立证据目录并关闭日志句柄,此探针清理错误不计入产品结论。
|
||||
|
||||
## 三、仍需验证的高影响风险
|
||||
|
||||
| 风险 | 已知机制 | 尚不能断言的部分 |
|
||||
| --- | --- | --- |
|
||||
| 慢保存被当成无响应 | 保存前仅发一次阶段进度;默认 150 秒请求停止、300 秒判无响应,safe_stop=False 不阻止监督器升级。保存没有类似原生快照的进度旁路 | 没有真实慢保存达到 300 秒的实验,不证明现场曾在保存中被终止 |
|
||||
| Quit/关闭响应延迟 | Quit helper 20 秒;成功后 CAD 15 秒;checkpoint 20 秒;项目会话关闭另有 60 秒宽限 | 本次正常样例均未超过这些时限,不能据此证明大工程也安全或必须统一加长 |
|
||||
| 同实例回读可能不足以覆盖持久性 | 生产路径在同一个 CAD 实例中重开回读 | 本次新实例读回通过,未发现缓存掩盖丢失;仍缺断电、数据库服务异常和大型工程覆盖 |
|
||||
| 关闭工程返回值被忽略 | 本机 TypeLib 表明 CloseProject 返回 Boolean;VBS 常直接调用并用工程路径为空作为补充判断 | 本次调用均为 True,尚未观察到 False 且空路径的真实异常,不能将其直接确定为事故根因 |
|
||||
| GUI/监督进程异常终止 | Job 关闭会清理受控进程,不能保证正在进行的保存完成 | 正常点击关闭时 GUI 会等待当前 worker,不能把所有 GUI 关闭行为描述成直接杀进程。崩溃/强制结束另需故障实验 |
|
||||
| 数据库和跨进程协作收尾 | 既有研究存在 iCDBNetServer、xprobSvr 等进程;仅确认 viewdraw 退出不足以描述全部应用状态 | 本轮没有捕获这些服务器在未提交数据时被终止;不能把 vnsd 的观察套用成数据库丢失证据 |
|
||||
|
||||
## 四、厂商接口核对的重要限制
|
||||
|
||||
实际安装的 7.9.5 `viewdraw.tlb` 已离线核对,结果保存在 `cad-lifecycle-typelib.json`:
|
||||
|
||||
- `IVdApp.CloseProject` 返回 Boolean。
|
||||
- `IVdApp.Quit`、`IVdSchematicSheetDocument.Close(SaveChanges, FileName)` 返回 void,不能虚构它们有保存成功布尔返回值。
|
||||
- `IVdDoc` 有 Saved 属性,但不能据此假定当前 `IVdSchematicSheetDocument` 也有同样的属性契约。
|
||||
- 事件接口包含 BeforeDocumentSave、AfterDocumentSave、ProjectClosed。
|
||||
|
||||
真实 VBS 日志已订阅上述事件:本轮 ProjectClosed 被正常观测,但在两次 SaveChanges=True 路径中**没有观测到 BeforeDocumentSave/AfterDocumentSave**。因此不能因为类型库列出了保存事件,就把等待该事件直接设为当前原理图保存路径的必经条件。需要核实它适用的文档类型、接口和触发条件。
|
||||
|
||||
本轮读取的安装帮助未取得足以证明上述调用异步/同步持久化语义的详细契约。类型库提供方法签名,不能替代持久化保证。
|
||||
|
||||
## 五、验证与修改范围
|
||||
|
||||
本轮补充回归:
|
||||
|
||||
```text
|
||||
python -m pytest -q tests/integration/test_quick_value_modes.py tests/v1_1/test_project_session.py tests/v1_1/test_failed_native_recovery_guard.py tests/v1_1/test_supervisor_result.py tests/v1_1/test_ui_operation_lifecycle.py
|
||||
94 passed in 34.11s
|
||||
```
|
||||
|
||||
上一轮已执行 IsolatedCad 和 worker monitor 的 60 项验证,本轮不重复当成新结果。测试通过说明已有行为与保护仍按测试约定运行,不证明上述异常路径合理。
|
||||
|
||||
仅增加分析报告和 `build/day0-resource-analysis-20261026/` 下的诊断脚本/证据。生产 Python、正式 VBS、版本号及发布包未修改,未向远端工单发消息或关闭工单。
|
||||
|
||||
下一轮修改应优先针对已复现的四个问题,并保留原有身份校验、写前保护和禁止不确定重放。验收必须分别报告数据结果、工程关闭结果、应用退出方式、后代进程退出方式和最终资源状态,不能汇成一个“释放成功”。
|
||||
@@ -0,0 +1,140 @@
|
||||
# Day0:资源释放工单原因分析
|
||||
|
||||
日期:2026-10-26(北京时间)。严重级别依据用户本次升级说明;未修改远端工单标签或状态。分析源码基线:`b9391015650bf5c43a77a017c5484e9a8f352266`。
|
||||
|
||||
后续已完成[核心应用生命周期专项排查](2026-10-26-application-lifecycle-investigation.md):包含两工程真实保存与独立读回、vnsd 自然退出对照及三类异常控制流复现。后续结论以该记录注明的证据和边界为准。
|
||||
|
||||
## 判断
|
||||
|
||||
工单 #1 的现场根因仍未闭环。根据用户对核心应用生命周期的补充,排查主线调整为“业务操作完成 → 保存确认 → 工程关闭确认 → 应用正常退出 → 进程及辅助资源释放 → 桌面回收”。桌面是隐藏操作的基础设施,其消失不能替代业务完成或进程正常退出证明。第一轮对桌面末端症状的侧重不足以覆盖保存与异常退出风险。
|
||||
|
||||
对已经观察到的桌面残留,需要继续追查自有清理范围以外的引用,包括本程序未登记的额外引用、无窗口线程、其他进程持有的桌面句柄和由系统或服务创建的辅助进程。这里的“范围以外”不等于已经证明是第三方软件造成,也不能将其提前确定为整个事故的业务根因。
|
||||
|
||||
工单 #3 在 CAD 未启动时出现同类残留,说明 CAD Quit 或 COM 释放不足不能解释全部现场现象。两个工单具有相同末端症状,但尚不能断言具体持有者或底层根因相同。
|
||||
|
||||
Beta 3 的进程隔离、批次复用和资源复核覆盖了部分故障机制。产品仍缺少现场持有者的完整识别能力,缺少对应现场版本的复测成功回执,不能凭本机通过将事故关闭。
|
||||
|
||||
## 补充核查:核心应用完成和退出链路
|
||||
|
||||
本节优先于后文仅针对桌面残留的原因排序。关闭桌面句柄不会自动结束进程;当前实现实际先清理受控进程、再关闭桌面句柄。应分别证明数据保存成功、工程关闭成功、应用正常退出、强制终止是否发生和资源是否消失。
|
||||
|
||||
### 正常路径已有保存回读,但完成语义仍需核实
|
||||
|
||||
- `QuickPartReplace.vbs:863` 的 `QPCloseBatchProject` 调用 `Document.Close(True, "")` 保存关闭图页,随后调用 `CloseProject`,立即查询工程路径为空。
|
||||
- `QuickPartReplace.vbs:934` 起会重新打开同一工程、校验目标及非目标内容,之后才给出 `saved_ok`。不能把现有程序描述为“完全没有等待或没有保存核验”。
|
||||
- COM 调用返回是现有的等待边界,但当前函数没有单独的保存完成事件、持久化提交状态或忙碌状态等待。是否需要增加这些机制,必须核对该版本 CAD 的 API 契约及真实行为;不能仅因缺少 sleep 就断言保存是异步或保存未完成。
|
||||
- 生产路径是在同一个 CAD 实例中重开工程进行回读。这提供了内容核验,但若要证明进程退出后的持久性,应增加正常退出后的独立新实例读取实验,并核查实际数据库服务的提交语义。不能仅凭同实例回读断言存在缓存问题,也不能将开发验收中的独立回读当成每次生产执行都有此保障。
|
||||
- 图页/工程关闭后立即查询一次状态,没有针对“调用已返回、关闭状态稍后才就绪”的有界状态等待。若厂商允许这种延迟,可能把仍在正常收尾的应用判为失败;目前还没有确认其 API 是这种行为。
|
||||
|
||||
### 明确存在的风险路径:应用未确认正常退出时进入强制清理
|
||||
|
||||
`isolated_cad.py:207` 的退出流程为:Quit helper 最多等待 20 秒;helper 成功才进一步等待 CAD 进程最多 15 秒。Quit helper 超时或失败时会跳过这次 CAD 正常退出等待,记录 uncertain,然后仍执行 `desktop.close()`。在批次模式中这里首先是 lease 的进程清理,随后才是批次拥有者的桌面关闭;底层 `_Win32Process.close()` 对存活自有进程/Job 有终止逻辑。
|
||||
|
||||
因此存在需要验证的链路:应用仍忙或关闭被拒绝 → 固定时限/单次状态检查失败 → 保护标记 → 强制终止自有进程 → 后续桌面释放检查。保护标记能避免无依据重放,不能保证被终止前未完成的保存可以安全完成。强制终止后的“资源清理成功”也不能用作“业务正常完成”的证据。
|
||||
|
||||
Quit helper 在发现工程仍打开时会拒绝退出,避免隐式保存或丢弃;但外层依然会进入自有进程清理兜底,所以必须检查当时是否有未完成的写操作。业务状态、退出方式和资源状态需要分开判断。
|
||||
|
||||
### 保存期间的无进度监控需要专项验证
|
||||
|
||||
`QuickPartReplace.vbs:874` 在进入保存关闭前发出一次 `save_and_close` 进度。若后续阻塞在 COM 调用中,脚本不能继续发送阶段进度。当前原生侧进度旁路只处理 `native_snapshot_read`,没有相同的保存进度旁路。
|
||||
|
||||
默认无进度时限为 150 秒。`ProgressWatch.action()` 在 150 秒请求停止、300 秒判定 unresponsive;这个判定不因 `safe_stop=False` 而停止升级。VBS 会在安全边界才响应协作停止,但监督器仍可在后续超时中终止 worker,并进入 CAD 收尾。真实慢保存是否能达到这个条件尚未实测;普通进程存活或无条件心跳也不能当成保存正在推进的证据。
|
||||
|
||||
### 本轮验证与证据边界
|
||||
|
||||
- `python -m pytest -q tests/v1_1/test_isolated_cad.py tests/trial_task_center/test_worker_monitor.py`:**60 passed,16.22 秒**。包括控制流替身和真实辅助进程测试,没有真实 CAD 慢保存实验。
|
||||
- 对生产 `IsolatedCadSession` 注入模拟 Quit helper 超时:确认等待参数为 20 秒、CAD wait 未被调用、uncertain 标记已写入、之后进入资源 close。未终止真实 CAD。
|
||||
- 对生产 `ProgressWatch` 输入模拟 `save_and_close / safe_stop=False`:150 秒为 request_stop,300 秒为 unresponsive。
|
||||
- 证据:`build/day0-resource-analysis-20261026/application-lifecycle-probe.json`。这是控制流证据,不能写成现场已发生保存中断或数据丢失。
|
||||
- 原始 #1 是库存读取后的收尾失败,四次日志记录 Quit helper/CAD 正常退出;#3 尚未启动 CAD。当前附件没有证明替换保存被提前打断。上述生命周期风险必须调查,但不能直接取代这两组现场证据。
|
||||
|
||||
优先实验应改为:在工程副本上记录保存调用前后、关闭调用前后、回读结果、Quit 请求与返回、CAD 自然退出、强制终止请求及原因的完整时间线;分别验证正常保存、长时间忙碌、关闭延迟和退出拒绝。数据正确性及退出契约查清后,再处理桌面末端残留。无需通过开启可见桌面或无界等待来替代这些完成条件。
|
||||
|
||||
## 已核实的证据
|
||||
|
||||
### 工单 #1:V1.1.3,四次资源释放确认失败
|
||||
|
||||
原件位置:`build/issue-1-20261021/evidence/`。四份失败的 `private-desktop.json` 一致记录:
|
||||
|
||||
- CAD 和退出辅助程序的退出码均为 0。
|
||||
- 所记录进程的 `job_active_processes=0`,已知后代句柄均已确认退出,自有进程及 Job 句柄关闭。
|
||||
- `desktop_handle_closed=true`,但 5 秒后 `released=false`。
|
||||
- 四次清理来自同一个主程序 PID 14608,调用线程不完全相同,调用线程桌面为 `Default`。
|
||||
- 另有一份采集中途取消的记录停在 `ready`,没有最终收尾证据;不能算作释放成功,也不能直接算作第五次同因泄漏。
|
||||
|
||||
这些证据降低了“已登记 CAD 主进程没有退出”“完全没等待终止”“已登记桌面句柄没有关闭”的优先级。它们不能证明进程所有线程、所有其他桌面句柄或所有系统辅助进程都已释放。
|
||||
|
||||
### 工单 #3:Beta 2,CAD 未启动也残留
|
||||
|
||||
本地 `build/issue-3-20261023/application.log` 与本次重新读取的远端附件逐字节一致。SHA256:`28d164039831973b7f6596a081d305421131f883514feae2fd652ff90ec54241`。
|
||||
|
||||
| 北京时间 | 日志位置 | 事实 |
|
||||
| --- | --- | --- |
|
||||
| 10-23 14:29:13 | 第 2 行 | 运行版本 `1.1.4-beta.2`,Windows 基础 build 26200 |
|
||||
| 14:29:34 | 第 14 行 | 开始整批 BOM 检查 |
|
||||
| 14:30:01 | 第 669 行 | 开始批末释放 |
|
||||
| 14:30:06 | 第 1098 行 | 桌面句柄关闭,桌面仍存在;CAD PID 为空,显示 CAD 未启动;窗口列表为空 |
|
||||
| 同一次失败的异常链 | 第 1102、1116 行及后续堆栈 | 首发错误为内部 attempt 目录创建的 `FileNotFoundError / WinError 3`,失败路径长 265 字符;释放错误发生在异常收尾期间 |
|
||||
| 14:30:47 | 第 1155 行 | 用户请求重新清理资源 |
|
||||
| 14:30:54 | 第 2013 行 | 同一个桌面仍存在,句柄关闭、CAD 未启动、窗口列表为空 |
|
||||
|
||||
前后失败样本相隔约 48 秒。未连续观测期间的状态,但相同随机桌面名两次均被枚举,明显削弱了“只需略微增加 5 秒超时”的解释。
|
||||
|
||||
`cleanup.processes=[]` 仅表示该次最终清理列表为空。原生 CLI 在批内执行后会从活动列表移除,不能据此说整批从未启动过子进程。
|
||||
|
||||
长路径是已记录的首发业务错误;桌面残留是收尾时另一个故障。修复路径问题不自动证明桌面引用来源已修复。
|
||||
|
||||
## 桌面残留的原因排序和判别办法
|
||||
|
||||
“优先级”表示调查顺序,不表示统计概率。
|
||||
|
||||
| 原因候选 | 优先级及证据 | 如何区分 |
|
||||
| --- | --- | --- |
|
||||
| 本程序的其他线程或额外桌面句柄仍持有引用 | 高;旧版 GUI 生命周期长,关闭登记句柄不能排除额外引用。四次 #1 来自同一主程序 | 现场保留桌面名,由独立观察进程在监督进程退出前后、主程序正常退出前后枚举;配合句柄和线程归属快照。随某进程退出而消失可缩小范围,但仍需持有者证据,不能单凭时间相关性定责 |
|
||||
| Job 外的无窗口辅助线程/进程 | 高;完全符合“Job 清空但桌面存在”。历史受控实验发现过无窗口 Python 和 conhost 线程 | 对目标桌面扫描全局线程,记录 PID/TID、映像、创建时间、会话及 Job 归属;与原生 CLI 启停时间对齐。历史 conhost 只是实验线索,现场未确认 |
|
||||
| 其他进程仅持有 OpenDesktop 句柄 | 高;可在不启动 CAD、没有窗口和目标线程的情况下复现相同症状 | 在线程扫描之外查询 Desktop 类型句柄。已有外部句柄实验复现阻断,持有者释放后可恢复;具体现场持有者未知 |
|
||||
| 原生 CLI、运行库、服务代理触发额外引用 | 中高;#3 没启动 CAD,但此前走过原生读取链 | 在现场副本依次运行空桌面、简单子进程、单个 icdb2ascii 读取、多个读取、CAD 回退,找出首次出现残留的阶段。普通子进程默认受 Job 约束,不能笼统断言所有后代都逃逸 |
|
||||
| 安全软件、录屏、远程协助、桌面管理组件等参与桌面访问 | 中,属于上一类的来源候选;当前没有具体软件或 PID 证据 | 先采集持有者或模块证据,再做单因素环境对照;没有证据时不能指定某个软件为元凶 |
|
||||
| Windows USER 引用延迟回收/系统环境差异 | 中低;短暂延迟有受控实验支持,但现场约 48 秒后的重查仍失败 | 使用独立进程记录 0、1、5、15、60 秒存在性,同时记录所有者退出状态。本机和 #3 均为基础 build 26200,仍需对照累计更新、运行库和注入组件,不能只按系统大版本归因 |
|
||||
| 句柄继承、Job 接管或异常清理分支缺陷 | 中低,需保留;当前代码在恢复运行前分配 Job,重定向使用明确的继承句柄清单,已有后代退出和句柄关闭失败测试 | 检查实际运行 EXE 哈希、Job 分配结果、句柄继承标志、失败时保留的所有权记录。当前源码机制不能倒推所有旧版或现场程序完全相同 |
|
||||
| 历史资源记录损坏或残留,形成逻辑阻断 | 对“升级后仍不能用”须优先区分;代码会保守阻断坏记录 | 对比当前物理桌面/进程与 `session.json`、`supervisor.json` 的状态。若物理对象已消失而记录无法校验,是证据恢复问题;不应归为仍有物理泄漏。原始 #1/#3 明确枚举到桌面,坏记录不能解释其最初残留 |
|
||||
| CAD COM 引用未释放 | 对 CAD 路径仍可查;作为统一解释优先级低 | 仅在已启动 CAD 的场景核对 COM、helper 与 CAD 生命周期。#3 的 CAD 未启动是其无法解释全部现象的直接证据 |
|
||||
| 工程数量上限、桌面堆耗尽、纯 BOM 匹配错误 | 当前证据弱 | 查桌面创建失败、内存资源错误和随批量增长的资源曲线。当前主要失败在销毁确认;BOM 行不匹配是独立业务结果,不能据此改资源配额或放宽匹配 |
|
||||
|
||||
微软文档确认:关闭一个桌面句柄与销毁所有引用不是一回事;线程的桌面连接不依赖存在可见窗口;Job 默认覆盖普通 CreateProcess 子进程,但不是整个系统的进程集合。例如 WMI `Win32_Process.Create` 创建的进程不属于这类默认继承范围。此机制说明需要查 Job 外所有者,不代表本软件已经被证明使用了 WMI 启动逃逸进程。
|
||||
|
||||
## 当前实现为何仍可能卡住
|
||||
|
||||
1. **判定严格,但归属诊断不足。** `isolated_desktop.py:66` 通过枚举确认桌面是否消失;失败时只收集窗口。`isolated_desktop.py:565` 明确将外部句柄持有者标为 unknown。空窗口不是无引用证明。独立工具 `tools/diagnose_private_desktop.py` 能查无窗口线程,但不能列出所有仅持有句柄的进程,权限错误也必须保留。
|
||||
2. **监督进程隔离只解决其覆盖的所有权。** `batch_supervisor.py:279` 等待并关闭受控 Job,随后 `_finish_response` 复核资源。这能够处理监督进程自身存活期间的引用;其他长寿命进程的引用不会因该监督进程退出而消失。
|
||||
3. **重新清理有能力边界。** `batch_resources.py:117` 对仍有 live owner 的会话可重试其关闭;没有 live owner 时只确认桌面消失和历史 PID 身份,不会接管或终止不明持有者。Beta 3 常见的是监督进程已经退出后的复核分支。反复点击不会凭空释放外部引用。
|
||||
4. **未释放门禁扩大可用性影响。** `guard_pending_resources` 阻止同一数据根的新批次。这一保护避免在旧资源不明时继续创建桌面或重放业务,却使一个释放问题阻断后续使用。恢复机制和持有者识别必须共同补齐。
|
||||
5. **损坏记录不会自动修复。** 无法校验的记录保持 blocked;无 session_id 时跳过自动资源操作。这是证据不足时的保护行为,但意味着资源已消失仍可能需要审计式人工核对。不能通过删除保护文件掩盖该状态。
|
||||
6. **诊断包不等于实时取证。** 当前导出函数主要打包已有文本,未自动执行全局线程或句柄扫描。上限为单文件 4 MiB、总计 32 MiB、512 个文件,超限写入 omissions;必须检查 manifest,不能把未导出的记录当成从未发生。
|
||||
|
||||
## 本次新验证
|
||||
|
||||
运行环境:Python 3.12.10、Windows 11 基础 build 26200。以下测试未打开或替换用户工程。
|
||||
|
||||
- `python -m pytest -q tests/v1_1/test_isolated_desktop.py tests/v1_1/test_desktop_diagnostics.py tests/v1_1/test_batch_runtime_windows.py tests/v1_1/test_batch_supervisor.py`:**44 passed,87.89 秒**。包含真实 Windows 资源测试与替身测试,覆盖无窗口引用、延迟释放、后代进程、诊断边界、监督恢复等;不等同于现场完整 EXE 验收。
|
||||
- `python -m pytest -q tests/v1_1/test_batch_runtime.py::test_unknown_resource_evidence_never_authorizes_cleanup tests/v1_1/test_batch_runtime.py::test_restart_reconciles_only_confirmed_disappearance_without_replacement`:**4 passed,1.22 秒**。确认坏证据继续阻断、重启后仅复核的代码行为;这两类为模拟边界测试。
|
||||
- 资源循环探针 `build/day0-resource-analysis-20261026/probe_cycles.py`:20 次空桌面、20 次带重定向的简单 Python 子进程,**40/40 正常释放**。自有桌面残留 0,观察前后没有新增桌面,调用线程桌面未改变。两组的释放枚举检查最大耗时均约 0.016 秒,该数字不包含完整子进程启动耗时。
|
||||
- 循环证据:`build/day0-resource-analysis-20261026/cycles-6819874fa1b04649a6dbb9c6b1992034.json`。它是源码 Python 基础路径实验,没有覆盖 Qt GUI、冻结 EXE、实际 icdb2ascii、CAD 或现场常驻组件,不排除这些组合中的问题。
|
||||
|
||||
这批结果降低了本机基础资源封装普遍失效的可能性,不能证明现场根因已修复,也不能证明 102 个原工程全部可用。
|
||||
|
||||
## 下一步应取得什么证据
|
||||
|
||||
1. **先确认事故实际版本和状态。** 当前收到的两组原件属于 V1.1.3 与 Beta 2;没有 Beta 3 的新失败诊断包。记录现场 EXE 哈希、是否完整配套 `_internal`、原始 data 是否迁移、故障是再次产生桌面还是旧记录阻断。Beta 3 公开 EXE 的历史记录哈希为 `df08cd328f6f4c9019f232c845e09627551e7b2d24790a466d53efcd291622d1`,以实际安装文件核对。
|
||||
2. **残留发生时立即取证。** 在原机器、原登录会话,从 session 记录取得准确桌面名;运行独立线程诊断并查询 Desktop 句柄持有者,保留 PID、创建时间、映像、线程、会话、Job 和时间戳。仅复制旧日志到开发机无法找回消失的 Windows 引用。
|
||||
3. **独立观察所有者退出前后。** 先保存结果及诊断,再观察监督进程退出、应用正常退出后的同名桌面变化。桌面随监督进程退出消失,优先追监督进程内部引用;随 GUI 退出消失,追 GUI 或其关联组件;GUI 退出后仍存在,追其他进程或系统引用。上述变化用于缩小范围,最终需结合持有者证据确认因果。
|
||||
4. **按最小场景逐层复现。** 空桌面 → 简单 worker → 单次原生 CLI → 多次原生 CLI → CAD 回退 → 批量混合。每层记录创建数量、已退出进程、最终桌面状态。实际读取使用工程副本,首次失败就保留证据,避免反复业务执行污染现场。
|
||||
5. **修复后验证恢复结果。** 对已识别的持有者修复生命周期;完成正常、异常、取消、重启和跨批次检查,确认只清理资源时业务结果不变、没有重复替换,随后在现场原目录和原批量范围验收。只有现场复测通过并留存回执,才具备关闭工单的依据。
|
||||
|
||||
技术交付首先补齐业务完成与应用退出的分阶段证据,并专项验证等待与强制清理边界;对完成业务后仍残留的桌面,再补自动捕获持有者的能力。单纯延长等待、反复要求重新清理或继续发布未取得现场证据的候选包,都不足以建立闭环。
|
||||
|
||||
## 关联记录
|
||||
|
||||
- [工单 #1](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/1)、[工单 #3](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/issues/3)。
|
||||
- [首轮工单分析](2026-10-21-issue-1-bom-import.md)、[Windows 生命周期受控实验](2026-10-21-windows-desktop-lifecycle.md)、[外部句柄持有者实验](2026-10-23-issue-3-external-holder.md)、[Beta 3 验收边界](2026-10-23-issue-3-safe-recovery.md)。
|
||||
- 微软:[CloseDesktop](https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-closedesktop)、[线程连接桌面的规则](https://learn.microsoft.com/en-us/windows/win32/winstation/thread-connection-to-a-desktop)、[Job Objects](https://learn.microsoft.com/en-us/windows/win32/procthread/job-objects)。本次重新核对官方页面。
|
||||
@@ -0,0 +1,59 @@
|
||||
# 用户停止与安全收尾实施验收
|
||||
|
||||
日期:2026-10-26;分支:`codex/day0-safe-stop`。实现用户在本会话批准的[停止流程](../superpowers/specs/2026-10-26-safe-user-stop.md),对应[全链路调查](2026-10-26-application-lifecycle-investigation.md)。
|
||||
|
||||
## 实现行为
|
||||
|
||||
- 普通暂停和关闭窗口不启动下一项,当前写入、保存和回读安全完成后停止。当前项并不等于整个工程。
|
||||
- 有写入可能的阶段长期没有进度时显示等待,不自动把超时升级成杀进程。纯只读采集保留有界取消。
|
||||
- 写入失败、保存结果不确定或 Quit 被拒绝/超时且 CAD 仍存活时,交互执行保留应用与工程,等待处理,不自动保存或丢弃。
|
||||
- “强制结束…”独立确认;先持久化业务保护及批次中断证据,再终止明确自有的监督进程及 Job。证据写入或 fsync 失败时拒绝该次 force,界面保留重试能力。
|
||||
- 保存请求、保存调用返回、保存回读和工程关闭分别记录,工程关闭失败不会丢失此前保存边界。
|
||||
- 收尾记录写入失败不跳过满足身份检查的正常 Quit。主进程自然退出后给予自有后代 2 秒自然退出窗口,之后的强制清理单独记录。
|
||||
- 资源消失不吞掉异常退出;调用方收到独立收尾错误,已核验业务结果仍附带保留。
|
||||
- 等待期间可独立导出诊断包。关闭窗口在操作结束前不退出;失败/强制结束后取消自动关闭,保留恢复提示。
|
||||
|
||||
## 自动验证与审查
|
||||
|
||||
针对安全边界先运行失败用例再实现,包括慢保存不会被 watchdog 终止、明确 force 的保护先行、保护写入失败保留进程、已有文件也重新 fsync、保存后工程关闭拒绝、窗口关闭不触发 force、确认取消无副作用。
|
||||
|
||||
真实辅助进程测试覆盖监督子进程强制中断、控制回调异常时保持子进程存活、拒绝第一次保护写入后仍允许后续明确重试、资源清理不重放业务。CAD 调用在这些故障测试中使用替身,不能据此声称真实保存中断无损。
|
||||
|
||||
退出协议独立审查修复了部分子进程查询失败时丢失已知强制终止 PID 的问题。整体验证另外修复:只读入口没有 force 出口却进入无限等待、证据首次 fsync 失败后因文件存在被误放行、已执行 force 被诊断异常降级为普通停止、控制通信异常绕过确认清理、直接执行 force 失败没有反馈,以及 force 意图与监控异常并发的竞态。复审未发现新增阻断问题。
|
||||
|
||||
首次完整实现回归 **1063 passed,1 条既有 pywinauto STA 警告,363.82 秒**。之后完成审查修正,最终扩大回归 **1147 passed,1 条既有 pywinauto STA 警告,325.64 秒**:
|
||||
|
||||
```powershell
|
||||
python -m pytest -q tests/v1_1 tests/trial_task_center/test_application.py tests/trial_task_center/test_execution_safety.py tests/part_replace/test_verification.py tests/integration/test_quick_value_modes.py tests/integration/test_quick_session_startup.py tests/integration/test_quick_part_replace_contract.py tests/integration/test_automatic_replacement_regression.py tests/trial_task_center/test_worker_monitor.py tests/trial_task_center/test_batch.py
|
||||
```
|
||||
|
||||
本命令包含项目发布脚本要求的源码测试范围,并补充此次受影响的 VBS、worker monitor 与批次测试。`git diff --check` 与新探针/停止模块语法检查通过。
|
||||
|
||||
两个旧 VBS fixture 在基线 `b939101` 也失败:未填 owned project path 且错误期望 no-match 保存、图页清单断言未使用规范化路径。修正的是 fixture 和断言,没有放宽生产所有权或保存规则;脚本契约组 72 项通过。
|
||||
|
||||
## 真实保存期间请求普通停止
|
||||
|
||||
工具:[probe_safe_stop_live.py](../../tools/probe_safe_stop_live.py)。本轮输出为 `build/safe-stop-live-a9113545bf/`,报告 `report.json`,最终资源复核 `final-resource-check.json`。
|
||||
|
||||
入口为源码 supervised service → 真实独立 batch worker → 生产 VBS → 真实 CAD,使用两份全新 Audio_D2128/Audio_T605A 副本。监听第一工程的生产 `save_requested` 日志后只设置普通停止 Event,不设置 force。此触发处于保存请求附近,不能声称精确暂停在 COM 内某一条指令。
|
||||
|
||||
| 检查 | 结果 |
|
||||
| --- | --- |
|
||||
| 第一个工程 | 当前 RV05 替换、保存和回读完成,任务 completed |
|
||||
| 第二个工程 | 保持 pending,没有继续替换 |
|
||||
| 队列 | paused=true,未请求强制结束 |
|
||||
| 第一工程新 CAD 独立读回 | 57 器件、103 引脚连接;目标 RV05 为新料号,56 个非目标器件保持;目标身份、位置和连接保持 |
|
||||
| 第二工程新 CAD 独立读回 | 57 个器件的完整属性及连接图与执行前一致 |
|
||||
| 执行批次退出 | orderly_shutdown=true、clean=true、forced_termination=false |
|
||||
| 资源 | 资源记录全部 released,最终枚举确认其桌面消失,没有本轮残留 CAD |
|
||||
| 原件 | 两个源工程目录哈希不变,中央库 `.lmc` 哈希不变 |
|
||||
|
||||
没有重放任何已经完成的替换;测试用的 before/after 独立读取使用新 CAD 实例。
|
||||
|
||||
## 边界
|
||||
|
||||
本轮没有构建或发布新 EXE,未做完整冻结 GUI 的保存中强制结束、掉电或原工单全量现场复测。GUI 行为使用 Qt 测试验证,真实保存停止使用源码监督入口验证;二者不能混写为冻结 GUI 全流程验收。
|
||||
|
||||
强制结束为避免切换任务竞态,保守标记本批可能涉及的目标;已完成业务证据和结果保留,但先前已完成任务也可能需要恢复核验。不能清除保护来继续。
|
||||
|
||||
保持应用等待不代表保存结果一定可以恢复。外部任务管理器强杀/系统掉电没有安全收尾保证,重启必须依赖原先写前证据核验。工单现场具体桌面持有者仍未查明,本实现不能当作现场根因已经闭环。
|
||||
@@ -1,5 +1,7 @@
|
||||
# R8:进度式超时与分阶段引脚索引
|
||||
|
||||
> 本文保留 R8 历史行为。2026-10-26 起,有写入可能的阶段无进度超时改为等待安全收尾,只有明确确认的强制结束才中断;当前停止与恢复规则见[停止、强制结束与恢复](safe-stop-and-recovery.md)。只读采集继续采用有界取消。
|
||||
|
||||
## 改变的行为
|
||||
|
||||
- 150 秒不再是整个任务的总时限,而是连续没有有效进度的观察窗口。有有效进度时可以继续执行超过 150 秒。
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
# 停止、强制结束与恢复
|
||||
|
||||
执行期间点击“完成当前项后暂停”,程序不再启动下一替换项。已经进入修改或保存的当前项会继续完成必要校验、保存、重开回读和工程关闭,然后正常退出 CAD,并等待自有辅助进程收尾。关闭主窗口也遵循这个流程,操作仍在进行时窗口保持打开。
|
||||
|
||||
“当前项”是当前图页和替换规则对应的执行单元,不是整个工程或整个批次。已经保存核验的结果保留,未开始的任务保持未执行。
|
||||
|
||||
## 保存或修改长时间没有进度
|
||||
|
||||
有写入可能的阶段,程序不会仅因超过无进度时限就自动终止应用。界面显示保存或核验尚未确认,可以继续等待并导出诊断包。等待不会启动下一项,也不会自动清除保护记录。
|
||||
|
||||
如果操作已经失败且保存结果不确定,程序保留当前应用和工程现场,不再自动保存或丢弃。应先导出诊断、保留原始工程与备份,再决定是否需要强制结束。此时“继续等待”不表示自动恢复执行。
|
||||
|
||||
纯读取的 BOM/库存采集仍采用有界取消;不存在写入需要完成时,不要求用户无限等待。确认未修改的启动失败也保留既有有界停止机制。
|
||||
|
||||
## 明确选择强制结束
|
||||
|
||||
“强制结束…”与普通暂停分开。确认框明确提示:当前操作可能正在修改或保存,强制结束可能留下未保存或部分保存结果,之后必须先核验恢复。
|
||||
|
||||
确认后,程序先持久化中断保护和批次记录,再终止自己拥有的监督进程及 Job。保护记录无法可靠保存时,不执行这次强制结束,界面提示原因并允许重试或导出诊断。取消确认没有终止副作用。
|
||||
|
||||
为避免强制请求与工程切换同时发生而漏记目标,强制结束会保守保护本批可能涉及的任务。先前已保存结果不会被覆盖或重放,但这些任务也可能显示需复核或被保留在当前列表。不要删除保护文件绕过核验。
|
||||
|
||||
## 程序重启之后
|
||||
|
||||
- 资源复核只处理进程和桌面状态,不重新替换。
|
||||
- 结果不确定的任务必须使用“核验并恢复”,按照已有快照和当前实际工程核对。
|
||||
- 已保存核验结果继续保留;资源清理成功不自动解除业务保护,也不自动继续队列。
|
||||
- 如果应用没有正常退出或后代进程需要强制清理,会单独报告收尾异常,不能把“资源已释放”理解为“保存和退出均正常”。
|
||||
|
||||
任务管理器强杀、系统关机或掉电可能不给程序写最终日志的机会。这些情况依赖修改前已经落盘的证据及下次启动的中断核验;程序不能保证被外部强杀时完成当前保存。
|
||||
@@ -28,7 +28,7 @@ Dim QP_ComponentUidIndex, QP_ComponentRefdesIndex, QP_ComponentLocationIndex
|
||||
Dim QP_ComponentIndexContext
|
||||
Dim QP_NativeCallActive
|
||||
Dim QP_BatchMode, QP_OwnedProject, QP_BatchState, QP_BatchCode, QP_BatchMessage
|
||||
Dim QP_BatchSaveCommitted
|
||||
Dim QP_BatchSaveCommitted, QP_BatchSaveRequested
|
||||
Dim QP_DiscoveryMode, QP_TargetSchematic, QP_TargetSheet
|
||||
Dim QP_FailureNumber, QP_FailureSource, QP_FailureDescription
|
||||
Dim QP_NcSyncSummary
|
||||
@@ -60,6 +60,7 @@ QP_BatchMode = False
|
||||
QP_OwnedProject = False
|
||||
QP_BatchState = ""
|
||||
QP_BatchSaveCommitted = False
|
||||
QP_BatchSaveRequested = False
|
||||
QP_DiscoveryMode = False
|
||||
QP_TargetSchematic = ""
|
||||
QP_TargetSheet = ""
|
||||
@@ -110,6 +111,10 @@ Else
|
||||
QP_FailureDescription = Err.Description
|
||||
If QP_DiscoveryMode Then QPCleanupDiscoveryFailure
|
||||
FailureState = "hard_failure"
|
||||
If QP_BatchSaveRequested Then
|
||||
FailureState = "batch_saved_unverified"
|
||||
QP_FailureDescription = "save was requested; persisted outcome requires reconciliation. " & QP_FailureDescription
|
||||
End If
|
||||
CloseError = ""
|
||||
If QP_BatchMode Or QP_SessionMode Then
|
||||
QPDiscardFailedBatchProject CloseError
|
||||
@@ -318,6 +323,7 @@ Sub QPSessionNext()
|
||||
QP_BatchMode = True
|
||||
QP_BatchState = ""
|
||||
QP_BatchSaveCommitted = False
|
||||
QP_BatchSaveRequested = False
|
||||
QP_NativeMutationPossible = False
|
||||
If Trim(QPReadUtf8(QP_Fso.BuildPath(QP_Root, "batch.request"))) <> QP_ExpectedJobId Then _
|
||||
Err.Raise vbObjectError + 873, "QuickPartReplace.ProjectSession", "next batch authorization mismatch"
|
||||
@@ -872,12 +878,21 @@ Sub QPCloseBatchProject(ByVal ExpectedPath, ByVal SaveChanges)
|
||||
If Documents.Count <> 1 Then Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "open sheet set changed; no automatic close"
|
||||
Set Document = Documents.Item(1)
|
||||
QPProgress "save_and_close", 0, 1
|
||||
If SaveChanges Then
|
||||
QP_BatchSaveRequested = True
|
||||
QPLog QP_Root, "INFO", "save_requested job=" & QP_CurrentJobId
|
||||
End If
|
||||
Document.Close CBool(SaveChanges), ""
|
||||
If SaveChanges Then
|
||||
QP_BatchSaveCommitted = True
|
||||
QPLog QP_Root, "INFO", "save_returned job=" & QP_CurrentJobId
|
||||
End If
|
||||
If QPCanonicalPath(QPProjectPath()) <> QPCanonicalPath(ExpectedPath) Then _
|
||||
Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "project changed during sheet save"
|
||||
QP_App.CloseProject
|
||||
If Len(CStr(QP_App.GetProjectData().GetProjectFilePath())) > 0 Then _
|
||||
Err.Raise vbObjectError + 871, "QuickPartReplace.Batch", "project did not close; queue stopped"
|
||||
QPLog QP_Root, "INFO", "project_closed job=" & QP_CurrentJobId
|
||||
QP_OwnedProject = False
|
||||
QP_OwnedProjectPath = ""
|
||||
QPInvalidatePinIndex
|
||||
@@ -888,6 +903,10 @@ Sub QPDiscardFailedBatchProject(ByRef ErrorDescription)
|
||||
Dim ErrorNumber, ErrorSource, Description
|
||||
ErrorDescription = ""
|
||||
If (Not QP_BatchMode And Not QP_SessionMode) Or Not QP_OwnedProject Or QP_App Is Nothing Then Exit Sub
|
||||
If QP_NativeMutationPossible Or QP_BatchSaveRequested Or QP_BatchSaveCommitted Then
|
||||
ErrorDescription = "write or save outcome requires reconciliation; owned project retained without automatic save or discard"
|
||||
Exit Sub
|
||||
End If
|
||||
On Error Resume Next
|
||||
QPCloseBatchProject QP_OwnedProjectPath, False
|
||||
ErrorNumber = Err.Number
|
||||
@@ -932,7 +951,6 @@ Sub QPFinishBatchItem()
|
||||
If FinalState = "restored_ok" Then
|
||||
QPMarkBatchMutation Command
|
||||
QPCloseBatchProject TargetPath, True
|
||||
QP_BatchSaveCommitted = True
|
||||
ElseIf Not QP_SessionMode Then
|
||||
QPCloseBatchProject TargetPath, False
|
||||
End If
|
||||
@@ -952,6 +970,7 @@ Sub QPFinishBatchItem()
|
||||
QPVerifyRestoredPlans Command, Records, Plans, ValueStates
|
||||
If Not QP_SessionMode Then QPCloseBatchProject TargetPath, False
|
||||
FinalState = "saved_ok"
|
||||
QPLog QP_Root, "INFO", "saved_readback_verified job=" & QP_CurrentJobId
|
||||
If QP_SessionMode Then
|
||||
QP_BatchMessage = QP_BatchMessage & " Saved and reopened for readback; owned project checkpoint verified."
|
||||
Else
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# UTF-8
|
||||
VSVersionInfo(
|
||||
ffi=FixedFileInfo(
|
||||
filevers=(1, 1, 4, 2),
|
||||
prodvers=(1, 1, 4, 2),
|
||||
filevers=(1, 1, 4, 4),
|
||||
prodvers=(1, 1, 4, 4),
|
||||
mask=0x3f,
|
||||
flags=0x2,
|
||||
OS=0x40004,
|
||||
@@ -15,11 +15,11 @@ VSVersionInfo(
|
||||
StringTable('040904B0', [
|
||||
StringStruct('CompanyName', 'DxProductivity'),
|
||||
StringStruct('FileDescription', 'DxDesigner Part Replacement'),
|
||||
StringStruct('FileVersion', '1.1.4.2'),
|
||||
StringStruct('FileVersion', '1.1.4.4'),
|
||||
StringStruct('InternalName', 'DxPartReplace'),
|
||||
StringStruct('OriginalFilename', 'DxPartReplace.exe'),
|
||||
StringStruct('ProductName', 'DxDesigner Part Replacement'),
|
||||
StringStruct('ProductVersion', '1.1.4-beta.2')
|
||||
StringStruct('ProductVersion', '1.1.4-beta.4')
|
||||
])
|
||||
]),
|
||||
VarFileInfo([VarStruct('Translation', [1033, 1200])])
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "dx-productivity"
|
||||
version = "1.1.4b2"
|
||||
version = "1.1.4b4"
|
||||
requires-python = ">=3.12,<3.13"
|
||||
dependencies = ["PySide6>=6.7,<7", "pywin32>=306", "pywinauto>=0.6.9,<1", "openpyxl>=3.1,<4"]
|
||||
|
||||
|
||||
@@ -48,6 +48,7 @@ def configure_application_identity(application: QApplication) -> None:
|
||||
def _parse_arguments(argv: Sequence[str] | None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(prog="DxPartReplace")
|
||||
parser.add_argument("--smoke-test", action="store_true", help=argparse.SUPPRESS)
|
||||
parser.add_argument('--batch-worker', type=Path, help=argparse.SUPPRESS)
|
||||
parser.add_argument("--data-root", type=Path, help=argparse.SUPPRESS)
|
||||
parser.add_argument("--native-helper", action="store_true", help=argparse.SUPPRESS)
|
||||
parser.add_argument("--snapshot", type=Path, help=argparse.SUPPRESS)
|
||||
@@ -119,25 +120,24 @@ def _flush(context: LoggingContext) -> None:
|
||||
|
||||
def _smoke_batch_runtime(data_root: Path, context: LoggingContext) -> None:
|
||||
"""Exercise process tracking inside the frozen supervisor before releasing."""
|
||||
from dx_productivity.trial_task_center.batch_runtime import BatchCadRuntime
|
||||
|
||||
with BatchCadRuntime(data_root, 'packaged_runtime_smoke') as runtime:
|
||||
runtime.bind_logs(context, ())
|
||||
with runtime.native_scope(data_root / 'runtime-smoke') as lease:
|
||||
command = str(Path(os.environ['SystemRoot']) / 'System32/cmd.exe')
|
||||
process = lease.start_process((command, '/d', '/c', 'exit', '0'))
|
||||
if not runtime.record['processes'][0]['creation_time']:
|
||||
raise RuntimeError('Smoke process identity has no creation time')
|
||||
if process.wait(timeout=15) != 0:
|
||||
raise RuntimeError('Smoke worker did not exit successfully')
|
||||
if runtime.record['state'] != 'released':
|
||||
raise RuntimeError('Smoke resources were not released')
|
||||
context.logger.info('packaged_runtime_smoke_passed batch_id=%s', runtime.session_id)
|
||||
from dx_productivity.trial_task_center.batch_supervisor import run_supervised
|
||||
with_store = TaskStore(data_root / 'task-center.db')
|
||||
try:
|
||||
service = TaskCenterService(with_store, context, recover_on_startup=False)
|
||||
result = run_supervised(service, 'runtime_smoke', {})
|
||||
if not result['process_identity_read']:
|
||||
raise RuntimeError('Smoke worker identity was not checked')
|
||||
finally:
|
||||
with_store.close()
|
||||
context.logger.info('packaged_runtime_smoke_passed supervisor_pid=%s', result['pid'])
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
"""Run the Qt shell, returning a conventional process status code."""
|
||||
arguments = _parse_arguments(argv)
|
||||
if arguments.batch_worker is not None:
|
||||
from dx_productivity.trial_task_center.batch_worker import run
|
||||
return run(arguments.batch_worker)
|
||||
if arguments.mark_batch_attempt:
|
||||
from dx_productivity.trial_task_center.native_helper import run_attempt_marker
|
||||
return run_attempt_marker(job_id=arguments.attempt_job, task_id=arguments.attempt_task,
|
||||
@@ -171,6 +171,7 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
_smoke_batch_runtime(data_root, logging_context)
|
||||
store = TaskStore(database_path)
|
||||
service = TaskCenterService(store, logging_context)
|
||||
service._supervised = True
|
||||
qt_application = QApplication([sys.executable, *(argv or ())])
|
||||
configure_application_identity(qt_application)
|
||||
window = _load_window()(service)
|
||||
|
||||
@@ -8,15 +8,16 @@ import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import tempfile
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
from .path_access import canonical_path, internal_path, retry_sharing_violation
|
||||
|
||||
|
||||
_IDENTITY = ('session_id', 'generation', 'attempt_id', 'task_id', 'job_id', 'project_path')
|
||||
|
||||
|
||||
def _project(value):
|
||||
return os.path.normcase(os.path.abspath(value))
|
||||
return os.path.normcase(str(canonical_path(value)))
|
||||
|
||||
|
||||
def _validate(record):
|
||||
@@ -37,22 +38,14 @@ def _validate(record):
|
||||
|
||||
|
||||
def _publish(path, record):
|
||||
path = Path(path)
|
||||
path = internal_path(path)
|
||||
descriptor, temporary = tempfile.mkstemp(prefix=f'.{path.name}.', suffix='.tmp', dir=path.parent)
|
||||
try:
|
||||
with os.fdopen(descriptor, 'w', encoding='utf-8', newline='') as handle:
|
||||
json.dump(record, handle, ensure_ascii=False, indent=2)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
deadline = time.monotonic() + 1
|
||||
while True:
|
||||
try:
|
||||
os.replace(temporary, path)
|
||||
break
|
||||
except PermissionError:
|
||||
if time.monotonic() >= deadline:
|
||||
raise
|
||||
time.sleep(.02)
|
||||
retry_sharing_violation(lambda: os.replace(temporary, path))
|
||||
# POSIX needs a directory sync for the atomic rename; Windows does not
|
||||
# expose fsync on directories through the CRT.
|
||||
if os.name != 'nt':
|
||||
@@ -69,13 +62,13 @@ def prepare_attempt(path, *, session_id, generation, task_id, job_id, project_pa
|
||||
record = _validate(dict(schema_version=1, session_id=str(session_id), generation=generation,
|
||||
task_id=str(task_id), job_id=str(job_id), project_path=_project(project_path),
|
||||
attempt_id=uuid4().hex if attempt_id is None else attempt_id, state='prepared'))
|
||||
Path(path).parent.mkdir(parents=True, exist_ok=True)
|
||||
retry_sharing_violation(lambda: internal_path(path).parent.mkdir(parents=True, exist_ok=True))
|
||||
_publish(path, record)
|
||||
return record
|
||||
|
||||
|
||||
def read_attempt(path):
|
||||
return _validate(json.loads(Path(path).read_text(encoding='utf-8')))
|
||||
return _validate(json.loads(internal_path(path).read_text(encoding='utf-8')))
|
||||
|
||||
|
||||
def _bound_attempt(path=None, *, job_id=None, task_id=None, project_path=None):
|
||||
@@ -86,7 +79,7 @@ def _bound_attempt(path=None, *, job_id=None, task_id=None, project_path=None):
|
||||
raw_path = os.environ.get('DXPART_ATTEMPT_FILE') if path is None else os.fspath(path)
|
||||
if not raw_path or not Path(raw_path).is_absolute():
|
||||
raise ValueError('invalid batch attempt file')
|
||||
path = Path(raw_path)
|
||||
path = internal_path(raw_path)
|
||||
job_id = str(job_id) if job_id is not None else os.environ.get('DXPART_JOB_ID')
|
||||
task_id = str(task_id) if task_id is not None else os.environ.get('DXPART_TASK_ID')
|
||||
if path.is_dir():
|
||||
|
||||
@@ -5,9 +5,11 @@ import os
|
||||
from pathlib import Path
|
||||
|
||||
from .logging_setup import LoggingContext, append_task_event, configure_logging
|
||||
from .path_access import internal_path
|
||||
|
||||
|
||||
def _append_line(path, line):
|
||||
path = internal_path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open('a', encoding='utf-8', newline='\n') as stream:
|
||||
stream.write(line + '\n')
|
||||
|
||||
@@ -9,6 +9,7 @@ import time
|
||||
from uuid import uuid4
|
||||
|
||||
from .errors import TaskCenterError
|
||||
from .path_access import canonical_path, internal_path, retry_sharing_violation
|
||||
|
||||
|
||||
# Recovery retains owned native handles after failed cleanup. This registry is
|
||||
@@ -17,29 +18,24 @@ LIVE_RESOURCES = {}
|
||||
|
||||
|
||||
class BatchResourceCleanupError(TaskCenterError):
|
||||
def __init__(self, message, *, completed_result=None):
|
||||
def __init__(self, message, *, completed_result=None, primary_error=None, cleanup_error=None):
|
||||
super().__init__('batch_resources_pending', message)
|
||||
self.completed_result = completed_result
|
||||
self.primary_error = primary_error
|
||||
self.cleanup_error = cleanup_error
|
||||
self.bom_report = getattr(primary_error, "bom_report", None)
|
||||
|
||||
|
||||
def write_record(path, record):
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path = internal_path(path)
|
||||
retry_sharing_violation(lambda: path.parent.mkdir(parents=True, exist_ok=True))
|
||||
temporary = path.with_name(path.name + '.' + uuid4().hex + '.tmp')
|
||||
try:
|
||||
with temporary.open('w', encoding='utf-8', newline='\n') as stream:
|
||||
json.dump(record, stream, ensure_ascii=False, indent=2)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
deadline = time.monotonic() + 1.0
|
||||
while True:
|
||||
try:
|
||||
os.replace(temporary, path)
|
||||
break
|
||||
except PermissionError:
|
||||
if time.monotonic() >= deadline:
|
||||
raise
|
||||
time.sleep(.02)
|
||||
retry_sharing_violation(lambda: os.replace(temporary, path))
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
|
||||
@@ -70,9 +66,10 @@ def _valid_record(record, path):
|
||||
|
||||
def pending_resources(root):
|
||||
pending = []
|
||||
for path in sorted((Path(root).resolve() / 'batch-resources').glob('*/session.json')):
|
||||
for path in sorted(internal_path(canonical_path(root) / 'batch-resources').glob('*/session.json')):
|
||||
path = canonical_path(path)
|
||||
try:
|
||||
record = json.loads(path.read_text(encoding='utf-8'))
|
||||
record = json.loads(internal_path(path).read_text(encoding='utf-8'))
|
||||
if not _valid_record(record, path):
|
||||
raise ValueError('invalid resource ownership record')
|
||||
if record.get('state') != 'released':
|
||||
@@ -169,11 +166,11 @@ def cleanup_pending_resources(root, *, logging_context=None):
|
||||
|
||||
|
||||
def project_checkpoint_released(root):
|
||||
root = Path(root).resolve()
|
||||
root = canonical_path(root)
|
||||
try:
|
||||
checkpoint = json.loads((root / 'project-checkpoint.json').read_text(encoding='utf-8'))
|
||||
checkpoint = json.loads(internal_path(root / 'project-checkpoint.json').read_text(encoding='utf-8'))
|
||||
path = Path(checkpoint['resource_record']).resolve()
|
||||
record = json.loads(path.read_text(encoding='utf-8'))
|
||||
record = json.loads(internal_path(path).read_text(encoding='utf-8'))
|
||||
key = str(root).casefold()
|
||||
return (_valid_record(record, path) and checkpoint.get('version') == 1
|
||||
and type(checkpoint.get('generation')) is int
|
||||
|
||||
@@ -6,11 +6,12 @@ import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
from uuid import uuid4
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from .batch_resources import (BatchResourceCleanupError, LIVE_RESOURCES, cleanup_pending_resources,
|
||||
guard_pending_resources, pending_resources, process_identity, project_checkpoint_released, write_record)
|
||||
from .errors import TaskCenterError
|
||||
from .path_access import canonical_path, internal_path, legacy_path
|
||||
from .isolated_desktop import PrivateDesktop
|
||||
from .operation_control import check_cancelled
|
||||
from .worker_monitor import WorkerMonitorError
|
||||
@@ -43,8 +44,12 @@ class _DesktopLease:
|
||||
return {'session_id': self.owner.session_id, 'generation': self.generation,
|
||||
'attempt_id': self.attempt_id}
|
||||
|
||||
def __exit__(self, *_exc):
|
||||
self.close()
|
||||
def __exit__(self, error_type, error, traceback):
|
||||
try:
|
||||
self.close()
|
||||
except BatchResourceCleanupError as cleanup:
|
||||
raise BatchResourceCleanupError(cleanup.message, primary_error=error,
|
||||
cleanup_error=cleanup) from cleanup
|
||||
|
||||
def start_process(self, command, **options):
|
||||
available = self.owner.record['state'] == 'ready' or (
|
||||
@@ -61,10 +66,11 @@ class _DesktopLease:
|
||||
for process in reversed(self.processes):
|
||||
try:
|
||||
self.owner._release_process(process, desktop=self.desktop, generation=self.generation)
|
||||
records.append(getattr(process, 'cleanup_record', {}))
|
||||
except Exception as error:
|
||||
errors.append(str(error))
|
||||
remaining.append(process)
|
||||
finally:
|
||||
records.append(getattr(process, 'cleanup_record', {}))
|
||||
self.processes = list(reversed(remaining))
|
||||
self.cleanup_record = {'scope': 'operation', 'processes': records,
|
||||
'workers_closed': not errors, 'desktop_retained_by_batch': True, 'errors': errors}
|
||||
@@ -78,10 +84,10 @@ class _DesktopLease:
|
||||
|
||||
|
||||
class BatchCadRuntime:
|
||||
def __init__(self, root, kind, stop_event=None, *, desktop_factory=None, cad_factory=None):
|
||||
self.root = Path(root).resolve()
|
||||
def __init__(self, root, kind, stop_event=None, *, desktop_factory=None, cad_factory=None, session_id=None):
|
||||
self.root = canonical_path(root)
|
||||
self.kind, self.stop_event = kind, stop_event
|
||||
self.session_id = uuid4().hex
|
||||
self.session_id = uuid4().hex if session_id is None else UUID(str(session_id)).hex
|
||||
self.path = self.root / 'batch-resources' / self.session_id / 'session.json'
|
||||
self.desktop_factory = desktop_factory or PrivateDesktop
|
||||
self.cad_factory = cad_factory
|
||||
@@ -94,6 +100,7 @@ class BatchCadRuntime:
|
||||
self._retry_counts = {}
|
||||
self._project_attempts = {}
|
||||
self._logging = None
|
||||
self._active_evidence = None
|
||||
|
||||
def bind_logs(self, context, task_ids):
|
||||
"""Keep every participating task's final release outcome visible."""
|
||||
@@ -111,11 +118,31 @@ class BatchCadRuntime:
|
||||
return self
|
||||
|
||||
def __exit__(self, error_type, error, traceback):
|
||||
if error is not None and self._active_evidence is not None:
|
||||
attempts = [item for values in self._project_attempts.values() for item in values]
|
||||
try:
|
||||
self._retain_uncertain(self._active_evidence, None, attempts, error)
|
||||
except WorkerMonitorError:
|
||||
if not getattr(getattr(self.stop_event, 'force_event', None), 'is_set', lambda: False)():
|
||||
raise
|
||||
# Explicit force has durable protection; cleanup is now allowed.
|
||||
try:
|
||||
self.close()
|
||||
except Exception as cleanup:
|
||||
raise BatchResourceCleanupError('批次资源释放待处理;已核验任务结果保留。' + str(cleanup),
|
||||
completed_result=self.result) from cleanup
|
||||
def detail(failure):
|
||||
return {'type': type(failure).__name__, 'message': str(failure),
|
||||
'code': getattr(failure, 'code', None)}
|
||||
self.record['primary_error'] = detail(error) if error is not None else None
|
||||
self.record['cleanup_error'] = detail(cleanup)
|
||||
try:
|
||||
self._save()
|
||||
except Exception:
|
||||
pass # Journal failure must not replace either original failure.
|
||||
context = ('BOM 导入读取已停止;未生成可执行预览。'
|
||||
if self.kind == 'bom_import' else
|
||||
'已核验任务结果保留。' if self.result is not None else '本批操作已停止。')
|
||||
raise BatchResourceCleanupError('批次资源释放待处理;' + context + str(cleanup),
|
||||
completed_result=self.result, primary_error=error, cleanup_error=cleanup) from cleanup
|
||||
|
||||
def _save(self, **values):
|
||||
record = dict(self.record, **values)
|
||||
@@ -173,7 +200,8 @@ class BatchCadRuntime:
|
||||
started = time.monotonic()
|
||||
self._save(cad_start_requested=True)
|
||||
cad = factory(self.path.parent / f'generation-{self.generation + (self.desktop is None)}',
|
||||
abort_event=abort_event, desktop=lease)
|
||||
abort_event=abort_event, desktop=lease,
|
||||
**({'stop_control': self.stop_event} if hasattr(self.stop_event, 'force_event') else {}))
|
||||
self.cad = cad # Retain cleanup ownership even if startup fails.
|
||||
try:
|
||||
cad.__enter__()
|
||||
@@ -188,26 +216,26 @@ class BatchCadRuntime:
|
||||
self._save(state='ready', cad_pid=cad.binding.pid, cad_guid=cad.binding.guid)
|
||||
return self.cad
|
||||
|
||||
def _prepare_attempts(self, root, command, attempt_id):
|
||||
def _prepare_attempts(self, root, command, attempt_id, generation):
|
||||
from .batch_attempt import prepare_attempt
|
||||
from .quick_run import _read_commands
|
||||
rows = _read_commands(root / 'command.tsv')
|
||||
rows = _read_commands(internal_path(root / 'command.tsv'))
|
||||
selected_rows = [row for row in rows if row[1] == command[-1]]
|
||||
if len(selected_rows) != 1 or len({row[1] for row in rows}) != len(rows):
|
||||
raise TaskCenterError('target_identity', '批次工作命令身份缺失或重复,已停止。')
|
||||
selected = selected_rows[0]
|
||||
project = Path(selected[2]).resolve()
|
||||
self._save(active_project=str(project), active_task=selected[0], active_job=selected[1])
|
||||
directory = root / 'batch-attempts' / (self.session_id + f'-g{self.generation}-' + attempt_id)
|
||||
project = canonical_path(selected[2])
|
||||
self.record.update(active_project=str(project), active_task=selected[0], active_job=selected[1])
|
||||
directory = self.path.parent / 'attempts' / f'g{generation}' / attempt_id
|
||||
expected = []
|
||||
for row in rows:
|
||||
if Path(row[2]).resolve() == project:
|
||||
if canonical_path(row[2]) == project:
|
||||
path = directory / (row[1] + '.json')
|
||||
record = prepare_attempt(path, session_id=self.session_id, generation=self.generation,
|
||||
record = prepare_attempt(path, session_id=self.session_id, generation=generation,
|
||||
task_id=row[0], job_id=row[1], project_path=str(project), attempt_id=attempt_id)
|
||||
expected.append((path, record))
|
||||
self._project_attempts.setdefault(str(project).casefold(), []).extend(expected)
|
||||
return project, directory, expected
|
||||
return project, legacy_path(directory), expected
|
||||
|
||||
def _can_retry(self, project, expected):
|
||||
from .batch_attempt import can_retry_attempt
|
||||
@@ -229,18 +257,22 @@ class BatchCadRuntime:
|
||||
from .discovery_cache import project_fingerprint
|
||||
from .isolated_cad import PrivateCadStartupError, is_identity_conflict
|
||||
from .worker_monitor import run_monitored_process
|
||||
root = Path(command[3]).resolve()
|
||||
self._ensure_desktop(abort_event)
|
||||
root = canonical_path(command[3])
|
||||
self._active_evidence = root
|
||||
while True:
|
||||
check_cancelled(abort_event or self.stop_event)
|
||||
attempt_id = uuid4().hex
|
||||
# No previous empty-project certificate may authorize continuation
|
||||
# once another operation could attach or open a project.
|
||||
self.record['checkpoints'].pop(str(root).casefold(), None)
|
||||
self._save(active_attempt=attempt_id)
|
||||
project, directory, expected = self._prepare_attempts(root, command, attempt_id)
|
||||
self.record['active_attempt'] = attempt_id
|
||||
if self.generation:
|
||||
self._save()
|
||||
generation = self.generation + (self.desktop is None)
|
||||
project, directory, expected = self._prepare_attempts(root, command, attempt_id, generation)
|
||||
self._ensure_desktop(abort_event)
|
||||
before = project_fingerprint(project, self.root)
|
||||
status = root / 'status.tsv'
|
||||
status = internal_path(root / 'status.tsv')
|
||||
status_before = status.read_bytes() if status.exists() else None
|
||||
started = time.monotonic()
|
||||
failure = None
|
||||
@@ -252,7 +284,11 @@ class BatchCadRuntime:
|
||||
def start_worker(launch, **options):
|
||||
return cad.start_worker(launch, extra_environment=environment, **options)
|
||||
result = run_monitored_process(command, timeout, abort_event=abort_event,
|
||||
cooperative_abort=cooperative_abort, process_factory=start_worker)
|
||||
cooperative_abort=cooperative_abort, process_factory=start_worker,
|
||||
**({'force_control': self.stop_event} if hasattr(self.stop_event, 'force_event') else {}))
|
||||
if result.returncode:
|
||||
self._retain_uncertain(root, project, expected,
|
||||
WorkerMonitorError(result.stderr or 'worker failed after possible mutation'))
|
||||
self._checkpoint(root, project, attempt_id)
|
||||
if result.returncode not in (2, 3):
|
||||
return result
|
||||
@@ -270,10 +306,11 @@ class BatchCadRuntime:
|
||||
# available for a failed read, use the stricter full fingerprint.
|
||||
safe = (not is_identity_conflict(failure) and before is not None and before == after
|
||||
and status_before == status_after and self._can_retry(project, expected)
|
||||
and not (root / 'recovery.request').exists()
|
||||
and not internal_path(root / 'recovery.request').exists()
|
||||
and not (abort_event is not None and abort_event.is_set())
|
||||
and not (self.stop_event is not None and self.stop_event.is_set()))
|
||||
if not safe:
|
||||
self._retain_uncertain(root, project, expected, failure)
|
||||
raise failure
|
||||
key = str(project).casefold()
|
||||
self._retry_counts[key] = 1
|
||||
@@ -281,18 +318,24 @@ class BatchCadRuntime:
|
||||
self._save(state='recycling')
|
||||
self.close(rebuilding=True) # Prove disappearance before creating another.
|
||||
self.desktop = self.cad = None
|
||||
self._ensure_desktop(abort_event)
|
||||
|
||||
def _retain_uncertain(self, root, project, expected, error):
|
||||
from .batch_attempt import can_retry_attempt
|
||||
from .stop_control import wait_for_resolution
|
||||
attempts = self._project_attempts.get(str(project).casefold(), expected)
|
||||
if attempts and not all(can_retry_attempt(path, record) for path, record in attempts):
|
||||
wait_for_resolution(self.stop_event, root, error)
|
||||
|
||||
def recover_native_failure(self, project, evidence, reason):
|
||||
"""One read-only recovery, based on durable source and worker proof."""
|
||||
from .batch_attempt import can_retry_attempt
|
||||
project, evidence = Path(project).resolve(), Path(evidence).resolve()
|
||||
project, evidence = canonical_path(project), canonical_path(evidence)
|
||||
key = str(project).casefold()
|
||||
if (self._retry_counts.get(key) or self.record['state'] == 'blocked'
|
||||
or self.stop_event is not None and self.stop_event.is_set()):
|
||||
return False
|
||||
try:
|
||||
proof = json.loads((evidence / 'native-retry-proof.json').read_text(encoding='utf-8'))
|
||||
proof = json.loads(internal_path(evidence / 'native-retry-proof.json').read_text(encoding='utf-8'))
|
||||
sources = proof['source_hashes']
|
||||
operation = self.record['native_operations'][str(evidence)]
|
||||
if (proof['version'] != 1 or proof['native_read_only'] is not True
|
||||
@@ -302,8 +345,8 @@ class BatchCadRuntime:
|
||||
or proof['workers_closed'] is not True or proof['project_path'] != str(project)
|
||||
or proof['evidence_root'] != str(evidence) or str(project) not in sources
|
||||
or len(sources) != 2
|
||||
or not all(Path(name).resolve().is_relative_to(project.parent) for name in sources)
|
||||
or not all(hashlib.sha256(Path(name).read_bytes()).hexdigest() == digest
|
||||
or not all(canonical_path(name).is_relative_to(project.parent) for name in sources)
|
||||
or not all(hashlib.sha256(internal_path(name).read_bytes()).hexdigest() == digest
|
||||
for name, digest in sources.items())
|
||||
or not all(can_retry_attempt(path, record)
|
||||
for path, record in self._project_attempts.get(key, ()))):
|
||||
@@ -368,7 +411,8 @@ class BatchCadRuntime:
|
||||
cad_record = cad_record if isinstance(cad_record, dict) else {}
|
||||
self.record['cad_shutdown'] = {key: cad_record[key] for key in (
|
||||
'state', 'clean', 'cad_exit_code', 'quit_helper_exit_code', 'cleanup_errors',
|
||||
'shutdown_seconds') if key in cad_record}
|
||||
'shutdown_seconds', 'orderly_shutdown', 'diagnostic_errors', 'forced_termination',
|
||||
'forced_root_termination', 'forced_descendant_pids', 'termination_reason') if key in cad_record}
|
||||
if self.desktop is not None:
|
||||
try:
|
||||
self.desktop.close()
|
||||
|
||||
@@ -0,0 +1,427 @@
|
||||
"""GUI-side batch admission, process supervision and resource-only reconciliation."""
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager
|
||||
import json
|
||||
import hashlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import queue
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
from .errors import TaskCenterError
|
||||
from .path_access import canonical_path, internal_path, legacy_path
|
||||
from .supervisor_protocol import VERSION, MessageGate, valid_journal, encode, decode, decode_error
|
||||
|
||||
_WORKER_ROOT = None
|
||||
_LAUNCH_LOCK = threading.Lock()
|
||||
_LIVE_SUPERVISORS = {}
|
||||
_WRITE_OPERATIONS = frozenset({'execute_task_queue', 'execute_quick_run_queue', 'execute_quick_run', 'recover_quick_run'})
|
||||
|
||||
|
||||
def _mark_interrupted(root, journal, reason):
|
||||
from .batch_resources import write_record, _valid_record
|
||||
from .stop_control import protect_interruption
|
||||
journal.update(state='blocked', outcome='interrupted', error=str(reason))
|
||||
if journal['operation'] in _WRITE_OPERATIONS:
|
||||
# A missing final result is not proof of no mutation. Publish the same
|
||||
# protection the existing recovery UI understands, without overwriting
|
||||
# any earlier, more specific marker.
|
||||
journal['business_uncertain'] = True
|
||||
task_ids = journal['task_ids']
|
||||
resource_path = Path(root) / 'batch-resources' / journal['batch_id'] / 'session.json'
|
||||
try:
|
||||
resource = json.loads(internal_path(resource_path).read_text(encoding='utf-8'))
|
||||
if (not journal.get('protect_all_targets') and _valid_record(resource, resource_path)
|
||||
and resource.get('active_task') in task_ids):
|
||||
task_ids = [resource['active_task']]
|
||||
except (OSError, ValueError, TypeError):
|
||||
pass # Without valid active ownership, protect every possible target.
|
||||
journal['protected_task_ids'] = list(task_ids)
|
||||
for task_id in task_ids:
|
||||
allowed = {canonical_path(root), canonical_path(Path(root) / 'tasks' / task_id)}
|
||||
actual = journal.get('uncertainty_roots', {}).get(task_id)
|
||||
roots = {canonical_path(actual)} if actual and canonical_path(actual) in allowed else allowed
|
||||
for evidence in roots:
|
||||
protect_interruption(evidence / 'quick-run',
|
||||
'reason=supervisor_interrupted; batch=' + journal['batch_id'] + '; manual reconciliation required\n')
|
||||
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal)
|
||||
|
||||
|
||||
def _worker_context(root):
|
||||
global _WORKER_ROOT
|
||||
_WORKER_ROOT = canonical_path(root)
|
||||
|
||||
|
||||
def _force_owned_supervisor(root, journal, process):
|
||||
"""Persist uncertainty before terminating the exact process/Job we own."""
|
||||
journal['stop_mode'] = 'forced'
|
||||
journal['force_requested_at'] = time.time()
|
||||
# The running worker can cross a task boundary while evidence is written.
|
||||
# Protect all possible targets before force; keep their saved results intact.
|
||||
journal['protect_all_targets'] = True
|
||||
_mark_interrupted(root, journal, '用户明确强制结束;写入与保存结果须核验,禁止自动重放。')
|
||||
try:
|
||||
process.terminate()
|
||||
except Exception as error:
|
||||
if not getattr(process, 'cleanup_record', {}).get('forced_termination'):
|
||||
raise
|
||||
journal['force_cleanup_error'] = str(error)
|
||||
try:
|
||||
from .batch_resources import write_record
|
||||
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal)
|
||||
except OSError:
|
||||
pass # The forced intent and uncertainty were already durably saved.
|
||||
|
||||
|
||||
def _retain_supervisor_after_control_error(root, journal, process, control, error):
|
||||
"""Do not turn a GUI/transport exception into an unconfirmed Job kill."""
|
||||
from .batch_resources import write_record
|
||||
control.set()
|
||||
control.control_error = '控制通信异常,已请求安全停止并保留应用;可导出诊断或明确强制结束:' + str(error)
|
||||
journal['protect_all_targets'] = True
|
||||
try:
|
||||
_mark_interrupted(root, journal, error)
|
||||
except Exception as failure:
|
||||
control.control_error += ';中断证据尚未落盘:' + str(failure)
|
||||
try:
|
||||
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'cancel.json',
|
||||
{'version': VERSION, 'batch_id': journal['batch_id'], 'attempt_id': journal['attempt_id']})
|
||||
except OSError:
|
||||
pass # Keep ownership; failure to request stop is not permission to kill.
|
||||
while True:
|
||||
if control.force_event.is_set():
|
||||
try:
|
||||
_force_owned_supervisor(root, journal, process)
|
||||
except Exception as failure:
|
||||
control.force_event.clear()
|
||||
control.control_error = '强制结束未确认,应用仍保留:' + str(failure)
|
||||
else:
|
||||
return True
|
||||
try:
|
||||
process.wait(timeout=.2)
|
||||
return False
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
except Exception as failure:
|
||||
control.control_error = '无法确认应用退出,保留所有权等待处理:' + str(failure)
|
||||
threading.Event().wait(.2)
|
||||
|
||||
|
||||
def _file_lock(handle, acquire):
|
||||
from .task_queue import _os_lock
|
||||
_os_lock(handle, acquire)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def supervisor_lease(root):
|
||||
root = internal_path(root)
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
if not _LAUNCH_LOCK.acquire(blocking=False):
|
||||
raise TaskCenterError('supervisor_busy', '另一批操作正在运行,请先安全收尾。')
|
||||
handle = None
|
||||
acquired = False
|
||||
try:
|
||||
handle = (root / '.batch-supervisor.execution.lock').open('a+b')
|
||||
if handle.seek(0, os.SEEK_END) == 0:
|
||||
handle.write(b'\0')
|
||||
handle.flush()
|
||||
try:
|
||||
_file_lock(handle, True)
|
||||
acquired = True
|
||||
except OSError as error:
|
||||
raise TaskCenterError('supervisor_busy', '另一窗口的批次监督进程尚未结束。') from error
|
||||
yield
|
||||
finally:
|
||||
if handle is not None:
|
||||
if acquired:
|
||||
_file_lock(handle, False)
|
||||
handle.close()
|
||||
_LAUNCH_LOCK.release()
|
||||
|
||||
|
||||
def assert_supervisor_idle(root):
|
||||
if _WORKER_ROOT == canonical_path(root):
|
||||
return
|
||||
with supervisor_lease(root):
|
||||
pass
|
||||
|
||||
|
||||
@contextmanager
|
||||
def mutation_lease(root):
|
||||
if _WORKER_ROOT == canonical_path(root):
|
||||
yield
|
||||
else:
|
||||
with supervisor_lease(root):
|
||||
yield
|
||||
|
||||
|
||||
def pending_supervisors(root):
|
||||
from .batch_resources import _process_gone
|
||||
pending = []
|
||||
for path in (internal_path(root) / 'batch-resources').glob('*/supervisor.json'):
|
||||
try:
|
||||
value = json.loads(path.read_text(encoding='utf-8'))
|
||||
if not valid_journal(value, path.parent.name):
|
||||
raise ValueError('invalid supervisor ownership record')
|
||||
if value.get('state') == 'finished':
|
||||
continue
|
||||
identity = value.get('process') or value.get('parent_process')
|
||||
if (value.get('state') in {'starting', 'running'} and identity and not _process_gone(identity)
|
||||
and value.get('parent_process') and not _process_gone(value['parent_process'])):
|
||||
continue
|
||||
pending.append(dict(value, record_path=str(canonical_path(path)), state='blocked'))
|
||||
except (OSError, ValueError, TypeError, KeyError) as error:
|
||||
pending.append({'state': 'blocked', 'record_path': str(canonical_path(path)), 'error': str(error)})
|
||||
return tuple(pending)
|
||||
|
||||
|
||||
def reconcile_supervisors(root):
|
||||
"""Recheck orphan ownership only; this never launches or replays a job."""
|
||||
from .batch_resources import _process_gone, pending_resources, write_record
|
||||
physical = pending_resources(root)
|
||||
for record in pending_supervisors(root):
|
||||
identity = record.get('process')
|
||||
if not record.get('batch_id') or (identity and not _process_gone(identity)):
|
||||
continue
|
||||
if record.get('outcome') not in {'interrupted', 'completed_resources_pending'}:
|
||||
_mark_interrupted(root, record, 'Supervisor ended without a reconciled final result')
|
||||
if any(item.get('session_id') == record['batch_id'] or not item.get('session_id') for item in physical):
|
||||
continue
|
||||
path = Path(record.pop('record_path'))
|
||||
outcome = 'completed' if record.get('outcome') == 'completed_resources_pending' else 'interrupted'
|
||||
record.update(state='finished', reconciled=True, outcome=outcome, reconciled_at=time.time())
|
||||
write_record(path, record)
|
||||
return pending_supervisors(root)
|
||||
|
||||
|
||||
def _command(request):
|
||||
if bool(getattr(sys, 'frozen', False)):
|
||||
return (sys.executable, '--batch-worker', str(request))
|
||||
return (sys.executable, '-m', 'dx_productivity.trial_task_center.batch_worker', '--request', str(request))
|
||||
|
||||
|
||||
def _environment():
|
||||
environment = dict(os.environ)
|
||||
if bool(getattr(sys, 'frozen', False)):
|
||||
for key in ('PYTHONPATH', 'PYTHONHOME'):
|
||||
environment.pop(key, None)
|
||||
else:
|
||||
environment['PYTHONPATH'] = str(Path(__file__).resolve().parents[2])
|
||||
return environment
|
||||
|
||||
|
||||
def _finish_response(root, journal, response, context):
|
||||
"""A supervisor exit may release its last USER reference. Never repeat work."""
|
||||
from .batch_resources import cleanup_pending_resources, write_record
|
||||
from .supervisor_result import validated_outcome, resource_pending_error
|
||||
result, error, primary = validated_outcome(journal['operation'], response)
|
||||
pending = cleanup_pending_resources(root, logging_context=context)
|
||||
related = tuple(item for item in pending if item.get('session_id') == journal['batch_id'] or not item.get('session_id'))
|
||||
journal['state'] = 'blocked' if related else 'finished'
|
||||
journal['outcome'] = ('completed_resources_pending' if related and response.get('business_succeeded')
|
||||
else response.get('status', 'interrupted'))
|
||||
journal['resource_recheck'] = {'pending': bool(related), 'checked_at': time.time()}
|
||||
write_record(Path(root) / 'batch-resources' / journal['batch_id'] / 'supervisor.json', journal)
|
||||
if related:
|
||||
raise resource_pending_error(journal['operation'], result, error, primary)
|
||||
resource_path = internal_path(Path(root) / 'batch-resources' / journal['batch_id'] / 'session.json')
|
||||
try:
|
||||
resource = json.loads(resource_path.read_text(encoding='utf-8'))
|
||||
except FileNotFoundError:
|
||||
resource = {}
|
||||
shutdown = resource.get('cad_shutdown') or {}
|
||||
if (shutdown.get('orderly_shutdown') is False or shutdown.get('forced_termination')
|
||||
or shutdown.get('diagnostic_errors')
|
||||
or any(item.get('stage') != 'desktop_release' for item in shutdown.get('cleanup_errors', ()))):
|
||||
failure = TaskCenterError('application_shutdown_failed',
|
||||
'已核验业务结果保留,但应用未正常收尾;请查看退出诊断,不能将资源消失视为正常退出。')
|
||||
failure.completed_result, failure.cleanup_error = result, error
|
||||
raise failure
|
||||
if response.get('business_succeeded'):
|
||||
if response.get('error'):
|
||||
context.logger.info('supervisor_resource_recovery_passed batch_id=%s business_replayed=false', journal['batch_id'])
|
||||
return result
|
||||
if response.get('operation_error'):
|
||||
if error is not None and hasattr(error, 'bom_report') and not hasattr(primary, 'bom_report'):
|
||||
primary.bom_report = error.bom_report
|
||||
raise primary
|
||||
if response.get('error'):
|
||||
raise error
|
||||
raise TaskCenterError('supervisor_interrupted', '批次监督进程异常退出;未重放任务,请检查已有结果和保护记录。')
|
||||
|
||||
|
||||
def run_supervised(service, operation, arguments, *, stop_event=None, on_progress=None, on_phase=None):
|
||||
from .batch_resources import write_record, process_identity
|
||||
from .isolated_desktop import _Win32Api
|
||||
from .stop_control import StopControl
|
||||
root = canonical_path(service._logging.root)
|
||||
if stop_event is not None and stop_event.is_set():
|
||||
raise TaskCenterError('operation_cancelled', '本批操作已取消,未启动监督进程。')
|
||||
with supervisor_lease(root):
|
||||
if operation == 'cleanup_batch_resources':
|
||||
for key, (owned_api, owned_process) in tuple(_LIVE_SUPERVISORS.items()):
|
||||
if key.parent.parent == root / 'batch-resources':
|
||||
owned_process.close()
|
||||
owned_api.close_pending()
|
||||
_LIVE_SUPERVISORS.pop(key)
|
||||
if operation != 'cleanup_batch_resources' and pending_supervisors(root):
|
||||
raise TaskCenterError('batch_resources_pending', '上一监督进程尚未完成资源复核,请先重新清理资源。')
|
||||
batch_id, attempt_id = uuid4().hex, uuid4().hex
|
||||
directory = root / 'batch-resources' / batch_id
|
||||
request = directory / 'request.json'
|
||||
journal = {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id,
|
||||
'operation': operation, 'state': 'starting', 'started_at': time.time(),
|
||||
'task_ids': list(arguments.get('task_ids', ())) or ([arguments['task_id']] if arguments.get('task_id') else []),
|
||||
'process': None}
|
||||
if any(not isinstance(value, str) or Path(value).name != value or value in {'.', '..'}
|
||||
for value in journal['task_ids']):
|
||||
raise TaskCenterError('target_identity', '监督进程任务编号无效。')
|
||||
journal['uncertainty_roots'] = {task_id: str(canonical_path(
|
||||
service._project_task_service(task_id)._quick_run_preparer.data_root))
|
||||
for task_id in journal['task_ids'] if operation in _WRITE_OPERATIONS}
|
||||
import win32api
|
||||
parent = type('ParentIdentity', (), {'pid': os.getpid(), 'handle': win32api.GetCurrentProcess()})()
|
||||
journal['parent_process'] = process_identity(parent)
|
||||
write_record(request, {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id,
|
||||
'root': str(root), 'database': str(canonical_path(service._store.path)),
|
||||
'operation': operation, 'arguments': encode(arguments), 'task_ids': journal['task_ids'],
|
||||
'interactive_stop': isinstance(stop_event, StopControl)})
|
||||
journal['request_sha256'] = hashlib.sha256(internal_path(request).read_bytes()).hexdigest()
|
||||
write_record(directory / 'supervisor.json', journal)
|
||||
gate, messages = MessageGate(batch_id, attempt_id), queue.Queue()
|
||||
api, process, reader = _Win32Api(), None, None
|
||||
sent_cancel, last_heartbeat = False, time.monotonic()
|
||||
transport_error = None
|
||||
force_accepted = False
|
||||
def receive():
|
||||
try:
|
||||
while line := process.stdout.readline(1024 * 1024):
|
||||
try:
|
||||
messages.put(json.loads(line))
|
||||
except (ValueError, UnicodeError):
|
||||
messages.put({'type': 'invalid'})
|
||||
finally:
|
||||
messages.put(None)
|
||||
try:
|
||||
# A normal input-desktop supervisor owns its private-desktop children.
|
||||
# Only the supervisor, never the GUI, creates a private desktop.
|
||||
process = api.create_process(_command(internal_path(request)), None, legacy_path(root),
|
||||
_environment(), stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
|
||||
journal.update(state='running', process=process_identity(process))
|
||||
write_record(directory / 'supervisor.json', journal)
|
||||
reader = threading.Thread(target=receive, daemon=True)
|
||||
reader.start()
|
||||
while process.poll() is None or not messages.empty():
|
||||
force = getattr(stop_event, 'force_event', None)
|
||||
if force is not None and force.is_set() and process.poll() is None:
|
||||
try:
|
||||
_force_owned_supervisor(root, journal, process)
|
||||
except Exception as error:
|
||||
force.clear()
|
||||
stop_event.control_error = '强制结束未执行或未确认;请保留窗口和诊断:' + str(error)
|
||||
service._logging.logger.error('force_stop_failed batch_id=%s error=%s', batch_id, error)
|
||||
else:
|
||||
force_accepted = True
|
||||
break
|
||||
if stop_event is not None and stop_event.is_set() and not sent_cancel:
|
||||
try:
|
||||
write_record(directory / 'cancel.json', {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id})
|
||||
journal.update(stop_mode='safe', stop_requested_at=time.time())
|
||||
write_record(directory / 'supervisor.json', journal)
|
||||
sent_cancel = True
|
||||
except OSError as error:
|
||||
if hasattr(stop_event, 'control_error'):
|
||||
stop_event.control_error = '停止请求尚未成功记录,正在重试:' + str(error)
|
||||
try:
|
||||
value = messages.get(timeout=.1)
|
||||
except queue.Empty:
|
||||
if time.monotonic() - last_heartbeat > 15:
|
||||
service._logging.logger.warning('supervisor_heartbeat_delayed batch_id=%s; no automatic replay', batch_id)
|
||||
last_heartbeat = time.monotonic()
|
||||
continue
|
||||
if value is None or not gate.accept(value):
|
||||
continue
|
||||
last_heartbeat = time.monotonic()
|
||||
if value['type'] == 'progress' and on_progress is not None:
|
||||
on_progress(value['payload'])
|
||||
elif value['type'] == 'phase' and on_phase is not None:
|
||||
payload = decode(value['payload'])
|
||||
on_phase(*payload)
|
||||
elif value['type'] == 'state':
|
||||
state = decode(value['payload'])
|
||||
service._task_queue_feedback = state['queue']
|
||||
service._batch_feedback = state['batch']
|
||||
service._supervisor_waiting = bool(state.get('waiting_for_resolution'))
|
||||
if on_phase is not None and state['phase'] in {'recycling', 'closing'}:
|
||||
from .bom_import_state import BomImportPhase
|
||||
on_phase(BomImportPhase.RECOVERING if state['phase'] == 'recycling' else BomImportPhase.RELEASING,
|
||||
'正在安全恢复读取会话…' if state['phase'] == 'recycling' else '正在释放后台资源…')
|
||||
process.wait(timeout=15)
|
||||
except Exception as error:
|
||||
transport_error = error
|
||||
if process is not None and isinstance(stop_event, StopControl):
|
||||
force_accepted = _retain_supervisor_after_control_error(root, journal, process, stop_event, error)
|
||||
finally:
|
||||
service._task_queue_feedback = None
|
||||
service._batch_feedback = None
|
||||
service._supervisor_waiting = False
|
||||
try:
|
||||
if process is not None:
|
||||
process.close() # Waits, and confines termination to the owned Job.
|
||||
api.close_pending()
|
||||
except Exception as error:
|
||||
if process is not None:
|
||||
_LIVE_SUPERVISORS[directory / 'supervisor.json'] = (api, process)
|
||||
transport_error = error if transport_error is None else TaskCenterError(
|
||||
'batch_resources_pending', f'监督进程错误:{transport_error};收尾错误:{error}')
|
||||
if reader is not None:
|
||||
reader.join(3)
|
||||
if force_accepted:
|
||||
from .batch_resources import cleanup_pending_resources
|
||||
try:
|
||||
cleanup_pending_resources(root, logging_context=service._logging)
|
||||
except Exception as error:
|
||||
service._logging.logger.error('forced_stop_cleanup_pending batch_id=%s error=%s', batch_id, error)
|
||||
raise TaskCenterError('user_forced_stop',
|
||||
'已按确认强制结束;已保存的执行证据保留,当前结果待核验。请先核验恢复,不要重复替换。')
|
||||
try:
|
||||
try:
|
||||
response = json.loads(internal_path(directory / 'result.json').read_text(encoding='utf-8'))
|
||||
except FileNotFoundError:
|
||||
saved = json.loads(internal_path(directory / 'business-result.json').read_text(encoding='utf-8'))
|
||||
if (saved.get('version') != VERSION or saved.get('batch_id') != batch_id
|
||||
or saved.get('attempt_id') != attempt_id or not saved['payload'].get('business_succeeded')):
|
||||
raise ValueError('Untrusted saved business result')
|
||||
response = dict(saved, type='result', sequence=gate.sequence + 1)
|
||||
service._logging.logger.warning('supervisor_final_missing_saved_result_retained batch_id=%s', batch_id)
|
||||
if (response.get('version') != VERSION or response.get('batch_id') != batch_id
|
||||
or response.get('attempt_id') != attempt_id or response.get('type') != 'result'
|
||||
or type(response.get('sequence')) is not int or response['sequence'] < gate.sequence
|
||||
or type(response.get('generation')) is not int or response['generation'] < gate.generation):
|
||||
raise ValueError('stale or invalid final supervisor result')
|
||||
from .supervisor_result import validated_outcome, resource_pending_error
|
||||
decoded_result, decoded_error, decoded_primary = validated_outcome(operation, response['payload'])
|
||||
except (OSError, ValueError, KeyError, TypeError, AttributeError) as error:
|
||||
_mark_interrupted(root, journal, transport_error or error)
|
||||
# Recheck physical resources even when final business results are
|
||||
# missing; this never authorizes repeating the interrupted job.
|
||||
from .batch_resources import cleanup_pending_resources
|
||||
cleanup_pending_resources(root, logging_context=service._logging)
|
||||
failure = TaskCenterError('supervisor_interrupted',
|
||||
'批次监督进程未留下可信结果;保留保护记录,不自动重试。' +
|
||||
(f' 首发原因:{transport_error}' if transport_error else ''))
|
||||
failure.primary_error = transport_error or error
|
||||
raise failure from (transport_error or error)
|
||||
if transport_error is not None:
|
||||
if response['payload']['business_succeeded']:
|
||||
journal.update(state='blocked', outcome='completed_resources_pending', error=str(transport_error))
|
||||
write_record(directory / 'supervisor.json', journal)
|
||||
else:
|
||||
_mark_interrupted(root, journal, transport_error)
|
||||
raise resource_pending_error(operation, decoded_result, transport_error, decoded_primary)
|
||||
return _finish_response(root, journal, response['payload'], service._logging)
|
||||
@@ -0,0 +1,199 @@
|
||||
"""One request, one resource-owning supervisor process, no Qt application."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import traceback
|
||||
from uuid import UUID
|
||||
|
||||
from .path_access import internal_path, canonical_path
|
||||
from .supervisor_protocol import VERSION, OPERATIONS, valid_journal, encode, decode, encode_error
|
||||
|
||||
def _output_stream():
|
||||
if sys.stdout is not None:
|
||||
return sys.stdout, False
|
||||
# Windowed PyInstaller deliberately sets sys.stdout=None. The supervisor
|
||||
# still has the explicit inherited pipe from STARTUPINFO; duplicate it so
|
||||
# closing this Python stream cannot close an unrelated borrowed handle.
|
||||
import msvcrt
|
||||
import win32api
|
||||
import win32con
|
||||
source = win32api.GetStdHandle(win32con.STD_OUTPUT_HANDLE)
|
||||
handle = win32api.DuplicateHandle(win32api.GetCurrentProcess(), source,
|
||||
win32api.GetCurrentProcess(), 0, False, win32con.DUPLICATE_SAME_ACCESS)
|
||||
descriptor = msvcrt.open_osfhandle(handle.Detach(), os.O_WRONLY | os.O_BINARY)
|
||||
return os.fdopen(descriptor, 'w', encoding='utf-8', buffering=1), True
|
||||
|
||||
|
||||
def _admit_request(directory, request, request_bytes):
|
||||
from .batch_resources import _process_gone, process_identity
|
||||
import win32api
|
||||
deadline = time.monotonic() + 5
|
||||
while True:
|
||||
journal = json.loads(internal_path(directory / 'supervisor.json').read_text(encoding='utf-8'))
|
||||
if (not valid_journal(journal, request['batch_id'])
|
||||
or journal.get('attempt_id') != request['attempt_id']
|
||||
or journal.get('state') not in {'starting', 'running'}
|
||||
or journal.get('request_sha256') != hashlib.sha256(request_bytes).hexdigest()
|
||||
or not journal.get('parent_process', {}).get('creation_time')
|
||||
or _process_gone(journal['parent_process'])):
|
||||
raise ValueError('Stale or unowned supervisor request')
|
||||
if journal.get('process'):
|
||||
break
|
||||
if time.monotonic() >= deadline:
|
||||
raise ValueError('Supervisor launch identity was not published')
|
||||
time.sleep(.02)
|
||||
own = process_identity(type('WorkerIdentity', (), {'pid': os.getpid(), 'handle': win32api.GetCurrentProcess()})())
|
||||
if journal['process'] != own:
|
||||
raise ValueError('Supervisor request belongs to another process')
|
||||
# Atomic exclusive publication makes requests one-shot, including retained
|
||||
# requests invoked manually or racing a delayed/duplicate launch.
|
||||
with internal_path(directory / 'claim.json').open('x', encoding='utf-8') as stream:
|
||||
json.dump({'version': VERSION, 'batch_id': request['batch_id'], 'attempt_id': request['attempt_id'],
|
||||
'process': own}, stream)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
|
||||
|
||||
def run(request_path):
|
||||
from .batch_resources import write_record
|
||||
from .batch_runtime import BatchCadRuntime
|
||||
from .batch_supervisor import _worker_context
|
||||
from .logging_setup import configure_logging
|
||||
from .service import TaskCenterService
|
||||
from .store import TaskStore
|
||||
|
||||
output, owns_output = _output_stream()
|
||||
request_path = canonical_path(request_path, strict=True)
|
||||
request_bytes = internal_path(request_path).read_bytes()
|
||||
request = json.loads(request_bytes)
|
||||
batch_id, attempt_id = request['batch_id'], request['attempt_id']
|
||||
if (request.get('version') != VERSION or UUID(batch_id).hex != batch_id or UUID(attempt_id).hex != attempt_id
|
||||
or request['operation'] not in OPERATIONS or type(request.get('interactive_stop', False)) is not bool):
|
||||
raise ValueError('Invalid batch supervisor request')
|
||||
root = canonical_path(request['root'])
|
||||
directory = root / 'batch-resources' / batch_id
|
||||
if request_path != directory / 'request.json':
|
||||
raise ValueError('Supervisor request ownership path mismatch')
|
||||
_admit_request(directory, request, request_bytes)
|
||||
_worker_context(root)
|
||||
from .stop_control import StopControl
|
||||
stop = StopControl() if request.get('interactive_stop', False) else threading.Event()
|
||||
finished, output_lock = threading.Event(), threading.Lock()
|
||||
kind = {'inspect_bom_batch': 'bom_import', 'preview_bom_batch': 'bom_import',
|
||||
'execute_task_queue': 'task_queue', 'execute_quick_run_queue': 'quick_run_queue'}.get(request['operation'], request['operation'])
|
||||
runtime = BatchCadRuntime(root, kind, stop, session_id=batch_id)
|
||||
sequence = 0
|
||||
def emit(kind, payload=None, *, persist=False):
|
||||
nonlocal sequence
|
||||
with output_lock:
|
||||
sequence += 1
|
||||
envelope = {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id,
|
||||
'generation': max(1, runtime.generation), 'sequence': sequence,
|
||||
'type': kind, 'payload': payload}
|
||||
if persist:
|
||||
write_record(directory / 'result.json', envelope)
|
||||
try:
|
||||
output.write(json.dumps(envelope, ensure_ascii=True, allow_nan=False) + '\n')
|
||||
output.flush()
|
||||
except (BrokenPipeError, OSError):
|
||||
stop.set()
|
||||
def monitor():
|
||||
while not finished.is_set():
|
||||
try:
|
||||
cancel = internal_path(directory / 'cancel.json')
|
||||
if cancel.is_file():
|
||||
value = json.loads(cancel.read_text(encoding='utf-8'))
|
||||
if value == {'version': VERSION, 'batch_id': batch_id, 'attempt_id': attempt_id}:
|
||||
stop.set()
|
||||
emit('heartbeat')
|
||||
feedback = getattr(service, '_task_queue_feedback', None)
|
||||
emit('state', encode({'phase': runtime.record['state'], 'queue': feedback,
|
||||
'batch': service.batch_feedback(),
|
||||
'waiting_for_resolution': getattr(stop, 'waiting_for_resolution', False),
|
||||
'resolution_error': getattr(stop, 'resolution_error', '')}))
|
||||
except Exception:
|
||||
stop.set() # Lost control evidence cannot authorize new work.
|
||||
finished.wait(1)
|
||||
context = configure_logging(root, log_path=directory / 'supervisor.log')
|
||||
store = TaskStore(Path(request['database']))
|
||||
service = TaskCenterService(store, context, recover_on_startup=False)
|
||||
thread = threading.Thread(target=monitor, daemon=True)
|
||||
thread.start()
|
||||
response = {'business_succeeded': False, 'status': 'error'}
|
||||
try:
|
||||
arguments = decode(request['arguments'])
|
||||
operation = request['operation']
|
||||
def progress(message):
|
||||
emit('progress', str(message))
|
||||
def phase(value, message):
|
||||
emit('phase', encode((value, str(message))))
|
||||
if operation == 'cleanup_batch_resources':
|
||||
result = service.cleanup_batch_resources()
|
||||
response.update(status='ok', business_succeeded=True, result=encode(result))
|
||||
else:
|
||||
with runtime:
|
||||
runtime.bind_logs(context, request['task_ids'])
|
||||
try:
|
||||
if operation == 'probe':
|
||||
result = {'pid': os.getpid(), 'resource_state': 'not_acquired'}
|
||||
elif operation == 'runtime_smoke':
|
||||
with runtime.native_scope(directory / 'runtime-smoke') as lease:
|
||||
child = lease.start_process((str(Path(os.environ['SystemRoot'])/'System32/cmd.exe'), '/d', '/c', 'exit', '0'))
|
||||
if not runtime.record['processes'][0]['creation_time'] or child.wait(timeout=15):
|
||||
raise RuntimeError('Supervisor process-identity smoke failed')
|
||||
result = {'pid': os.getpid(), 'process_identity_read': True}
|
||||
elif operation == 'execute_task_queue':
|
||||
from .task_queue import execute_task_queue
|
||||
result = execute_task_queue(service, **arguments, stop_event=stop, runtime=runtime)
|
||||
elif operation in {'inspect_bom_batch', 'preview_bom_batch'}:
|
||||
controls = {'stop_event': stop, 'on_progress': progress, 'runtime': runtime}
|
||||
if operation == 'inspect_bom_batch':
|
||||
controls['on_phase'] = phase
|
||||
result = getattr(service, operation)(**arguments, **controls)
|
||||
elif operation == 'collect_target_inventory':
|
||||
result = service.collect_target_inventory(**arguments, stop_event=stop, on_progress=progress, runtime=runtime)
|
||||
elif operation == 'execute_quick_run_queue':
|
||||
result = service.execute_quick_run_queue(**arguments, stop_event=stop, runtime=runtime)
|
||||
elif operation == 'target_cell_choices':
|
||||
result = service.target_cell_choices(**arguments)
|
||||
else:
|
||||
result = getattr(service, operation)(**arguments, runtime=runtime)
|
||||
runtime.result = result
|
||||
response.update(status='ok', business_succeeded=True, result=encode(result))
|
||||
write_record(directory / 'business-result.json', {'version': VERSION, 'batch_id': batch_id,
|
||||
'attempt_id': attempt_id, 'generation': max(1, runtime.generation), 'payload': response})
|
||||
if operation == 'inspect_bom_batch':
|
||||
from .bom_import_state import BomImportPhase
|
||||
phase(BomImportPhase.RELEASING, '本批位号核验结束,正在释放后台资源…')
|
||||
except Exception as error:
|
||||
response['operation_error'] = encode_error(error)
|
||||
raise
|
||||
except Exception as error:
|
||||
context.report_exception('batch_supervisor', error)
|
||||
response.update(status='error', error=encode_error(error))
|
||||
finally:
|
||||
finished.set()
|
||||
thread.join(3)
|
||||
store.close()
|
||||
emit('result', response, persist=True)
|
||||
if owns_output:
|
||||
output.close()
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--request', required=True, type=Path)
|
||||
args = parser.parse_args()
|
||||
return run(args.request)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -17,6 +17,8 @@ class BomImportPhase(Enum):
|
||||
VALIDATING = 'validating'
|
||||
INVENTORY = 'inventory'
|
||||
MATCHING = 'matching'
|
||||
RECOVERING = 'recovering'
|
||||
RELEASING = 'releasing'
|
||||
CANCELLING = 'cancelling'
|
||||
PREVIEW_READY = 'preview_ready'
|
||||
REJECTED = 'rejected'
|
||||
@@ -30,6 +32,11 @@ _NEXT = {
|
||||
BomImportPhase.INVENTORY: {BomImportPhase.INVENTORY, BomImportPhase.MATCHING},
|
||||
BomImportPhase.MATCHING: {BomImportPhase.MATCHING, BomImportPhase.INVENTORY},
|
||||
}
|
||||
_RECOVERY_PHASES = {BomImportPhase.RECOVERING, BomImportPhase.RELEASING}
|
||||
for _phase in tuple(_NEXT):
|
||||
_NEXT[_phase] |= _RECOVERY_PHASES
|
||||
for _phase in _RECOVERY_PHASES:
|
||||
_NEXT[_phase] = _RECOVERY_PHASES | {BomImportPhase.INVENTORY, BomImportPhase.MATCHING}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""Allowlisted, bounded support export; never packages project or CAD inputs."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import stat
|
||||
import zipfile
|
||||
|
||||
from .path_access import canonical_path, internal_path
|
||||
|
||||
MAX_FILE_BYTES = 4 * 1024 * 1024
|
||||
MAX_BUNDLE_BYTES = 32 * 1024 * 1024
|
||||
MAX_FILES = 512
|
||||
_BATCH_NAMES = {'session.json', 'supervisor.json', 'request.json', 'result.json',
|
||||
'business-result.json', 'supervisor.log', 'resource-release.log', 'cancel.json'}
|
||||
_QUICK_NAMES = {'private-desktop.json', 'checkpoint.json', 'status.json',
|
||||
'uncertain.flag', 'resolved-uncertain.flag', 'protection.json', 'cleanup.json',
|
||||
'project-session.json', 'project-checkpoint.json', 'status.tsv'}
|
||||
_SECRET_KEY = re.compile(r'(?:password|passwd|(?:client[_-]?)?secret|credentials?|authorization|api[_-]?key|access[_-]?token|refresh[_-]?token|token)', re.I)
|
||||
_USER_KEY = re.compile(r'(?:user(?:[_-]?name)?|login[_-]?name)', re.I)
|
||||
_SECRET_TEXT = re.compile(r'''(?ix)(\b(?:password|passwd|(?:client[_-]?)?secret|credentials?|authorization|api[_-]?key|access[_-]?token|refresh[_-]?token|token)\b["']?\s*[=:]\s*)(?:"[^"\r\n]*"|'[^'\r\n]*'|[^\s,;}]+)''')
|
||||
|
||||
|
||||
def _scrub_text(text):
|
||||
# Scrub other users as well as the exporting user's home, including JSON
|
||||
# escaped Windows paths. Preserve suffixes useful for fault diagnosis.
|
||||
text = re.sub(r'(?i)(?:[A-Z]:[\\/]+Users[\\/]+|/Users/|/home/)[^\\/\s"\']+', '<USER_HOME>', text)
|
||||
home = str(Path.home())
|
||||
for prefix in {home, home.replace('\\', '/'), home.replace('\\', '\\\\')}:
|
||||
text = re.sub(re.escape(prefix), '<USER_HOME>', text, flags=re.I)
|
||||
username = os.environ.get('USERNAME') or os.environ.get('USER')
|
||||
if username and len(username) > 2:
|
||||
text = re.sub(r'(?<![\w])' + re.escape(username) + r'(?![\w])', '<USER>', text, flags=re.I)
|
||||
text = re.sub(r'(?i)\bBearer\s+[A-Za-z0-9._~+/-]+=*', 'Bearer <REDACTED>', text)
|
||||
text = _SECRET_TEXT.sub(r'\1<REDACTED>', text)
|
||||
text = re.sub(r'''(?ix)(\b(?:user(?:[_-]?name)?|login[_-]?name)\b["']?\s*[=:]\s*)(?:"[^"\r\n]*"|'[^'\r\n]*'|[^\s,;}]+)''', r'\1<USER>', text)
|
||||
text = re.sub(r'(?i)(https?://)[^\s/@:]+:[^\s/@]+@', r'\1<REDACTED>@', text)
|
||||
return text
|
||||
|
||||
|
||||
def _scrub_json(value):
|
||||
if isinstance(value, dict):
|
||||
return {_scrub_text(str(key)): ('<REDACTED>' if _SECRET_KEY.fullmatch(str(key))
|
||||
else '<USER>' if _USER_KEY.fullmatch(str(key)) else _scrub_json(item))
|
||||
for key, item in value.items()}
|
||||
if isinstance(value, list):
|
||||
return [_scrub_json(item) for item in value]
|
||||
return _scrub_text(value) if isinstance(value, str) else value
|
||||
|
||||
|
||||
def _allowed(parts):
|
||||
if len(parts) == 2 and parts[0] == 'logs':
|
||||
return bool(re.fullmatch(r'[^/]+\.log(?:\.\d+)?', parts[-1]))
|
||||
if len(parts) == 3 and parts[0] == 'batch-resources':
|
||||
return parts[-1] in _BATCH_NAMES
|
||||
if len(parts) >= 5 and parts[0] == 'batch-resources' and parts[2] == 'attempts':
|
||||
return parts[-1].endswith('.json')
|
||||
if len(parts) >= 2 and parts[0] == 'attempts':
|
||||
return parts[-1].endswith('.json')
|
||||
if len(parts) == 3 and parts[0] == 'tasks':
|
||||
return bool(re.fullmatch(r'events\.jsonl(?:\.\d+)?', parts[-1]))
|
||||
if len(parts) >= 2 and parts[0] in {'queue', 'queues', 'task-queues'}:
|
||||
return parts[-1].endswith('.json')
|
||||
if len(parts) >= 2 and any(value in parts for value in ('quick-run', 'quickrun', 'quick-run-history')):
|
||||
return parts[-1] in _QUICK_NAMES
|
||||
return False
|
||||
|
||||
|
||||
def export_diagnostic_bundle(data_root, destination: Path) -> Path:
|
||||
"""Write a new ZIP, excluding links, binaries, inputs and credential configs.
|
||||
|
||||
This is a point-in-time best-effort read: active writers may produce partial
|
||||
log lines. Unsupported/unreadable items are documented, never dereferenced.
|
||||
"""
|
||||
root, destination = canonical_path(data_root), canonical_path(destination)
|
||||
if destination.suffix.lower() != '.zip':
|
||||
raise ValueError('diagnostic destination must have a .zip suffix')
|
||||
manifest = {'version': 1, 'included': [], 'omissions': [], 'unknowns': [
|
||||
'External desktop handle holders are unknown; empty window/thread lists do not prove release.',
|
||||
'Active logs and metadata are point-in-time reads; no source CAD, libraries, BOM, database or environment configuration is included.'],
|
||||
'limits': {'max_file_bytes': MAX_FILE_BYTES, 'max_bundle_bytes': MAX_BUNDLE_BYTES, 'max_files': MAX_FILES}}
|
||||
total = 0
|
||||
physical = internal_path(root)
|
||||
# os.walk does not descend symlinks; explicitly prune Windows junctions too.
|
||||
with zipfile.ZipFile(internal_path(destination), 'x', compression=zipfile.ZIP_DEFLATED) as archive:
|
||||
for directory, folders, files in os.walk(physical, followlinks=False):
|
||||
base = Path(directory)
|
||||
safe_folders = []
|
||||
for name in sorted(folders):
|
||||
entry = base / name
|
||||
relative = entry.relative_to(physical).as_posix()
|
||||
info = entry.lstat()
|
||||
if stat.S_ISLNK(info.st_mode) or getattr(info, 'st_file_attributes', 0) & 0x400:
|
||||
manifest['omissions'].append({'path': _scrub_text(relative), 'reason': 'link_or_reparse_point'})
|
||||
elif base == physical and name not in {'logs', 'batch-resources', 'attempts', 'tasks', 'queue', 'queues', 'task-queues', 'quick-run', 'quick-run-history'}:
|
||||
manifest['omissions'].append({'path': _scrub_text(relative), 'reason': 'outside_allowlist'})
|
||||
else:
|
||||
safe_folders.append(name)
|
||||
folders[:] = safe_folders
|
||||
for name in sorted(files):
|
||||
path = base / name
|
||||
relative = path.relative_to(physical)
|
||||
label = _scrub_text(relative.as_posix())
|
||||
reason = None
|
||||
try:
|
||||
info = path.lstat()
|
||||
if not _allowed(relative.parts):
|
||||
reason = 'outside_allowlist'
|
||||
elif stat.S_ISLNK(info.st_mode) or getattr(info, 'st_file_attributes', 0) & 0x400 or info.st_nlink != 1:
|
||||
reason = 'link_or_reparse_point'
|
||||
elif not stat.S_ISREG(info.st_mode):
|
||||
reason = 'not_regular_file'
|
||||
elif info.st_size > MAX_FILE_BYTES or total + info.st_size > MAX_BUNDLE_BYTES or len(manifest['included']) >= MAX_FILES:
|
||||
reason = 'size_or_count_limit'
|
||||
else:
|
||||
with path.open('rb') as source:
|
||||
payload = source.read(MAX_FILE_BYTES + 1)
|
||||
if len(payload) > MAX_FILE_BYTES or b'\x00' in payload:
|
||||
reason = 'binary_or_size_limit'
|
||||
else:
|
||||
text = payload.decode('utf-8-sig')
|
||||
if relative.suffix == '.json':
|
||||
text = json.dumps(_scrub_json(json.loads(text)), ensure_ascii=False, indent=2)
|
||||
elif '.jsonl' in relative.name:
|
||||
text = '\n'.join(json.dumps(_scrub_json(json.loads(line)), ensure_ascii=False)
|
||||
for line in text.splitlines() if line.strip()) + '\n'
|
||||
else:
|
||||
text = _scrub_text(text)
|
||||
archive.writestr(label, text)
|
||||
total += len(payload)
|
||||
manifest['included'].append(label)
|
||||
except (OSError, UnicodeError, ValueError):
|
||||
reason = 'unreadable_or_invalid_text'
|
||||
if reason:
|
||||
manifest['omissions'].append({'path': label, 'reason': reason})
|
||||
archive.writestr('manifest.json', json.dumps(manifest, ensure_ascii=False, indent=2))
|
||||
return destination
|
||||
@@ -130,6 +130,66 @@ class DiscoveryCache:
|
||||
def invalidate(self, project: Path):
|
||||
with self.lock:
|
||||
self._entries.pop(_key(project), None)
|
||||
self._disk_path(project).unlink(missing_ok=True)
|
||||
|
||||
def _disk_path(self, project):
|
||||
from .path_access import internal_path
|
||||
return internal_path(Path(self.data_root) / 'discovery-cache' /
|
||||
(hashlib.sha256(_key(project).encode('utf-8')).hexdigest() + '.json'))
|
||||
|
||||
def _disk_entry(self, project, stamp):
|
||||
import json
|
||||
from uuid import uuid4
|
||||
from .supervisor_protocol import decode
|
||||
path = self._disk_path(project)
|
||||
if stamp is None or not path.exists():
|
||||
return None
|
||||
try:
|
||||
if path.stat().st_size > 32 * 1024 * 1024:
|
||||
raise ValueError('inventory cache exceeds size limit')
|
||||
record = json.loads(path.read_text(encoding='utf-8'))
|
||||
checksum = record.pop('record_sha256')
|
||||
if (record.get('version') != 2 or not isinstance(checksum, str)
|
||||
or checksum != hashlib.sha256(json.dumps(record, ensure_ascii=True, sort_keys=True).encode()).hexdigest()
|
||||
or type(record.get('collect_components')) is not bool
|
||||
or not isinstance(record.get('parts'), list)
|
||||
or not all(isinstance(part, str) for part in record['parts'])):
|
||||
raise ValueError('invalid discovery cache coverage or identity')
|
||||
payload = json.dumps(record['discovery'], ensure_ascii=True, sort_keys=True).encode('utf-8')
|
||||
if record['payload_sha256'] != hashlib.sha256(payload).hexdigest():
|
||||
raise ValueError('invalid discovery cache')
|
||||
if record['project'] != _key(project) or record['fingerprint'] != hashlib.sha256(repr(stamp).encode()).hexdigest():
|
||||
return None
|
||||
discovery = decode(record['discovery'])
|
||||
from .quick_run_protocol import ProjectDiscovery
|
||||
if not isinstance(discovery, ProjectDiscovery):
|
||||
raise ValueError('unexpected discovery cache type')
|
||||
_discovery_identity(discovery)
|
||||
return stamp, frozenset(record['parts']), record['collect_components'], discovery
|
||||
except (OSError, ValueError, TypeError, KeyError):
|
||||
try:
|
||||
path.replace(path.with_name(path.name + '.bad-' + uuid4().hex))
|
||||
except OSError:
|
||||
pass # A bad cache is never authoritative, even if quarantine fails.
|
||||
return None
|
||||
|
||||
def _save_disk(self, project, stamp, requested, components, discovery):
|
||||
import json
|
||||
from .batch_resources import write_record
|
||||
from .supervisor_protocol import encode
|
||||
try:
|
||||
payload = encode(discovery)
|
||||
record = {'version': 2, 'project': _key(project),
|
||||
'fingerprint': hashlib.sha256(repr(stamp).encode()).hexdigest(), 'parts': sorted(requested),
|
||||
'collect_components': components, 'discovery': payload,
|
||||
'payload_sha256': hashlib.sha256(json.dumps(payload, ensure_ascii=True, sort_keys=True).encode()).hexdigest()}
|
||||
record['record_sha256'] = hashlib.sha256(json.dumps(record, ensure_ascii=True, sort_keys=True).encode()).hexdigest()
|
||||
write_record(self._disk_path(project), record)
|
||||
files = sorted(self._disk_path(project).parent.glob('*.json'), key=lambda path: path.stat().st_mtime_ns, reverse=True)
|
||||
for stale in files[self.capacity:]:
|
||||
stale.unlink(missing_ok=True)
|
||||
except (OSError, ValueError, TypeError):
|
||||
pass # Cache persistence is optional, unlike execution write-ahead proof.
|
||||
|
||||
def discover(self, project, parts, collect_components, loader, *, stop_event=None):
|
||||
with self.lock:
|
||||
@@ -140,7 +200,7 @@ class DiscoveryCache:
|
||||
key = _key(project)
|
||||
requested = frozenset(part.casefold() for part in parts)
|
||||
before = fingerprint()
|
||||
entry = self._entries.pop(key, None)
|
||||
entry = self._entries.pop(key, None) or self._disk_entry(project, before)
|
||||
if entry is not None:
|
||||
stamp, covered, has_components, discovery = entry
|
||||
if (before is not None and before == stamp and requested <= covered
|
||||
@@ -197,6 +257,7 @@ class DiscoveryCache:
|
||||
and (len(getattr(discovery, 'components', ())) + len(sheets)
|
||||
+ sum(len(parts) for parts in matches.values())) <= 100000):
|
||||
self._entries[key] = (after, requested, collect_components, deepcopy(discovery))
|
||||
self._save_disk(project, after, requested, collect_components, discovery)
|
||||
while len(self._entries) > self.capacity:
|
||||
self._entries.popitem(last=False)
|
||||
return discovery
|
||||
|
||||
@@ -13,6 +13,7 @@ class ExecutionFeedback:
|
||||
message: str = '正在准备执行'
|
||||
stage: str = ''
|
||||
stage_seconds: float = 0.0
|
||||
waiting_for_safe_completion: bool = False
|
||||
|
||||
|
||||
def read_feedback(task_id, position, total, message, root: Path, started_after: float):
|
||||
@@ -27,15 +28,16 @@ def read_feedback(task_id, position, total, message, root: Path, started_after:
|
||||
except (OSError, ValueError, TypeError, AttributeError):
|
||||
pass
|
||||
path = root / 'quick-run' / 'worker-progress.json'
|
||||
stage, seconds = '', 0.0
|
||||
stage, seconds, waiting = '', 0.0, False
|
||||
try:
|
||||
modified = path.stat().st_mtime
|
||||
if modified >= started_after:
|
||||
record = json.loads(path.read_text(encoding='utf-8'))
|
||||
waiting = record.get('waiting_for_safe_completion') is True
|
||||
candidate = record.get('stage', '')
|
||||
if candidate and candidate != 'finished':
|
||||
seconds = float(record.get('stage_seconds', {}).get(candidate, 0)) + max(0, time.time() - modified)
|
||||
stage = candidate
|
||||
except (OSError, ValueError, TypeError, AttributeError):
|
||||
pass # Display feedback is optional; it never changes execution state.
|
||||
return ExecutionFeedback(task_id, position, total, message, stage, seconds)
|
||||
return ExecutionFeedback(task_id, position, total, message, stage, seconds, waiting)
|
||||
@@ -12,6 +12,7 @@ from uuid import uuid4
|
||||
from .cad_identity import CadBinding, current_desktop_name, verify_application, project_file_path
|
||||
from .isolated_desktop import PrivateDesktop
|
||||
from .worker_monitor import WorkerMonitorError
|
||||
from .path_access import internal_path, canonical_path, retry_sharing_violation
|
||||
|
||||
|
||||
class PrivateCadStartupError(WorkerMonitorError):
|
||||
@@ -47,6 +48,7 @@ def is_identity_conflict(error):
|
||||
|
||||
|
||||
def _helper_command(action, evidence, pid):
|
||||
evidence = internal_path(evidence)
|
||||
if getattr(sys, 'frozen', False):
|
||||
return (sys.executable, '--private-cad-action', action, '--private-cad-evidence', str(evidence),
|
||||
'--private-cad-pid', str(pid))
|
||||
@@ -64,15 +66,7 @@ def _replace_record(temporary, target):
|
||||
# Windows readers can temporarily deny delete access to the old JSON.
|
||||
# Keep the complete old record until replacement succeeds, with a finite
|
||||
# retry budget so persistent permissions failures still reach the caller.
|
||||
deadline = time.monotonic() + 1.0
|
||||
while True:
|
||||
try:
|
||||
temporary.replace(target)
|
||||
return
|
||||
except PermissionError:
|
||||
if time.monotonic() >= deadline:
|
||||
raise
|
||||
time.sleep(0.02)
|
||||
retry_sharing_violation(lambda: internal_path(temporary).replace(internal_path(target)), target=target)
|
||||
|
||||
|
||||
def _designer_executable():
|
||||
@@ -93,8 +87,8 @@ def _designer_executable():
|
||||
|
||||
class IsolatedCadSession:
|
||||
"""The supervisor never attaches to COM on the user's input desktop."""
|
||||
def __init__(self, root, *, abort_event=None, desktop=None):
|
||||
self.root = Path(root).resolve()
|
||||
def __init__(self, root, *, abort_event=None, desktop=None, stop_control=None):
|
||||
self.root = canonical_path(root)
|
||||
self.token = uuid4().hex
|
||||
self.borrowed_desktop = desktop is not None
|
||||
self.desktop = desktop if desktop is not None else PrivateDesktop('dx-' + self.token, enabled=True)
|
||||
@@ -102,16 +96,17 @@ class IsolatedCadSession:
|
||||
self.binding = None
|
||||
self.started = time.monotonic()
|
||||
self.abort_event = abort_event
|
||||
self.stop_control = stop_control
|
||||
self.record = {'desktop': self.desktop.name, 'token': self.token, 'state': 'starting'}
|
||||
|
||||
def _record(self, **values):
|
||||
self.record.update(values)
|
||||
self.root.mkdir(parents=True, exist_ok=True)
|
||||
target = self.root / 'private-desktop.json'
|
||||
internal_path(self.root).mkdir(parents=True, exist_ok=True)
|
||||
target = internal_path(self.root / 'private-desktop.json')
|
||||
temporary = target.with_suffix('.tmp')
|
||||
temporary.write_text(json.dumps(self.record, ensure_ascii=False, indent=2), encoding='utf-8')
|
||||
_replace_record(temporary, target)
|
||||
history = self.root / 'private-desktop-sessions'
|
||||
history = internal_path(self.root / 'private-desktop-sessions')
|
||||
history.mkdir(exist_ok=True)
|
||||
archived = history / (self.token + '.json')
|
||||
pending = archived.with_suffix('.tmp')
|
||||
@@ -134,6 +129,9 @@ class IsolatedCadSession:
|
||||
deadline = time.monotonic() + 120
|
||||
while probe.poll() is None:
|
||||
self._check_abort()
|
||||
cad_exit = self.cad.poll()
|
||||
if cad_exit is not None:
|
||||
raise WorkerMonitorError(f'owned private CAD exited during startup: exit code {cad_exit}')
|
||||
if time.monotonic() >= deadline:
|
||||
raise WorkerMonitorError('private CAD bootstrap timed out')
|
||||
time.sleep(0.05)
|
||||
@@ -144,7 +142,7 @@ class IsolatedCadSession:
|
||||
finally:
|
||||
probe.close()
|
||||
try:
|
||||
result = json.loads(bootstrap.read_text(encoding='utf-8'))
|
||||
result = json.loads(internal_path(bootstrap).read_text(encoding='utf-8'))
|
||||
binding = CadBinding(**result)
|
||||
if (type(binding.pid) is not int or not isinstance(binding.guid, str) or not binding.guid
|
||||
or (binding.pid, binding.desktop, binding.token) != (self.cad.pid, self.desktop.name, self.token)):
|
||||
@@ -185,7 +183,7 @@ class IsolatedCadSession:
|
||||
if self.binding is None or self.cad.poll() is not None:
|
||||
raise WorkerMonitorError('owned CAD unavailable at project checkpoint')
|
||||
evidence = Path(evidence)
|
||||
evidence.parent.mkdir(parents=True, exist_ok=True)
|
||||
internal_path(evidence.parent).mkdir(parents=True, exist_ok=True)
|
||||
process = self.start_worker(_helper_command('checkpoint', evidence, self.cad.pid),
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
try:
|
||||
@@ -195,7 +193,7 @@ class IsolatedCadSession:
|
||||
raise failure_type('project close checkpoint failed: ' +
|
||||
process.stderr.read().decode('utf-8', errors='replace'))
|
||||
try:
|
||||
result = json.loads(evidence.read_text(encoding='utf-8'))
|
||||
result = json.loads(internal_path(evidence).read_text(encoding='utf-8'))
|
||||
if (not isinstance(result, dict) or result.get('binding') != asdict(self.binding)
|
||||
or result.get('project_path') != '' or result.get('project_closed') is not True):
|
||||
raise ValueError('project close checkpoint identity mismatch')
|
||||
@@ -209,52 +207,115 @@ class IsolatedCadSession:
|
||||
|
||||
def __exit__(self, error_type, error, traceback):
|
||||
closing_started = time.monotonic()
|
||||
cleanup_error = None
|
||||
cleanup_errors = []
|
||||
diagnostic_errors = []
|
||||
failures = []
|
||||
exit_codes = {}
|
||||
closing = None
|
||||
|
||||
def remember(stage, failure):
|
||||
cleanup_errors.append({'stage': stage, 'message': str(failure)})
|
||||
message = '; '.join(f"{item['stage']}: {item['message']}" for item in cleanup_errors)
|
||||
self._uncertain(message, cleanup_errors=cleanup_errors,
|
||||
failures.append(failure)
|
||||
|
||||
def diagnose(stage, action):
|
||||
try:
|
||||
action()
|
||||
except Exception as failure:
|
||||
diagnostic_errors.append({'stage': stage, 'message': str(failure)})
|
||||
failures.append(failure)
|
||||
self.record['clean'] = False
|
||||
|
||||
def failure_message():
|
||||
return '; '.join(f"{item['stage']}: {item['message']}"
|
||||
for item in cleanup_errors + diagnostic_errors)
|
||||
|
||||
def persist_uncertain():
|
||||
message = failure_message() or str(error)
|
||||
self.record.update(state='uncertain', clean=False, error=message,
|
||||
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors,
|
||||
shutdown_seconds=time.monotonic() - closing_started,
|
||||
desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes)
|
||||
diagnose('uncertain_record', lambda: self._uncertain(message,
|
||||
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors,
|
||||
clean=False, orderly_shutdown=self.record['orderly_shutdown'],
|
||||
shutdown_seconds=time.monotonic() - closing_started,
|
||||
desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes))
|
||||
|
||||
self.record.update(clean=False, orderly_shutdown=False, forced_termination=False,
|
||||
forced_root_termination=False, forced_descendant_pids=[], termination_reason=None,
|
||||
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors)
|
||||
diagnose('closing_record', lambda: self._record(state='closing', clean=False,
|
||||
cleanup_errors=cleanup_errors, diagnostic_errors=diagnostic_errors))
|
||||
try:
|
||||
self._record(state='closing', cleanup_errors=cleanup_errors)
|
||||
if error is not None:
|
||||
self._uncertain(str(error))
|
||||
if self.cad is not None and self.cad.poll() is not None:
|
||||
raise WorkerMonitorError('owned CAD exited before the authorized quit request')
|
||||
if self.cad is not None and self.binding is not None:
|
||||
closing = self.desktop.start_process(_helper_command('quit', self.root / 'private-desktop.json', self.cad.pid),
|
||||
env=_child_environment(self.binding.environment()), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
try:
|
||||
quit_code = closing.wait(timeout=20)
|
||||
exit_codes['quit_helper_exit_code'] = quit_code
|
||||
if quit_code:
|
||||
raise WorkerMonitorError('owned CAD refused orderly exit: ' + closing.stderr.read().decode('utf-8', errors='replace'))
|
||||
cad_code = self.cad.wait(timeout=15)
|
||||
exit_codes['cad_exit_code'] = cad_code
|
||||
if cad_code != 0:
|
||||
raise WorkerMonitorError('owned CAD returned a nonzero exit code after quit')
|
||||
finally:
|
||||
closing.close()
|
||||
quit_code = closing.wait(timeout=20)
|
||||
exit_codes['quit_helper_exit_code'] = quit_code
|
||||
if quit_code:
|
||||
raise WorkerMonitorError('owned CAD refused orderly exit: ' + closing.stderr.read().decode('utf-8', errors='replace'))
|
||||
cad_code = self.cad.wait(timeout=15)
|
||||
exit_codes['cad_exit_code'] = cad_code
|
||||
if cad_code != 0:
|
||||
raise WorkerMonitorError('owned CAD returned a nonzero exit code after quit')
|
||||
self.record['orderly_shutdown'] = True
|
||||
except Exception as failure:
|
||||
cleanup_error = failure
|
||||
remember('orderly_shutdown', failure)
|
||||
finally:
|
||||
if error is not None or failures:
|
||||
# Evidence failure must never prevent the identity-checked Quit
|
||||
# above or release of the handles we own below.
|
||||
persist_uncertain()
|
||||
if self.stop_control is not None and cleanup_errors and self.cad is not None:
|
||||
cad_running = True
|
||||
try:
|
||||
cad_running = self.cad.poll() is None
|
||||
except Exception as failure:
|
||||
remember('cad_exit_check', failure)
|
||||
if cad_running:
|
||||
# A refused/timed-out Quit may still be saving or closing.
|
||||
# Keep both CAD and its helper owned and alive until the
|
||||
# interactive caller confirms force, including ordinary stop.
|
||||
from .stop_control import wait_for_resolution
|
||||
try:
|
||||
wait_for_resolution(self.stop_control, self.root, failure_message())
|
||||
except WorkerMonitorError as failure:
|
||||
force_event = getattr(self.stop_control, 'force_event', None)
|
||||
if force_event is None or not force_event.is_set():
|
||||
raise
|
||||
remember('forced_stop', failure)
|
||||
if closing is not None:
|
||||
try:
|
||||
closing.close()
|
||||
except Exception as failure:
|
||||
remember('quit_helper_release', failure)
|
||||
try:
|
||||
self.desktop.close()
|
||||
self._record(state='session_closed' if self.borrowed_desktop else 'released',
|
||||
desktop_retained_by_batch=self.borrowed_desktop,
|
||||
clean=error is None and cleanup_error is None,
|
||||
shutdown_seconds=time.monotonic() - closing_started,
|
||||
desktop_cleanup=getattr(self.desktop, 'cleanup_record', None), **exit_codes)
|
||||
except Exception as failure:
|
||||
cleanup_error = failure
|
||||
remember('desktop_release', failure)
|
||||
if cleanup_error is not None:
|
||||
message = '; '.join(f"{item['stage']}: {item['message']}" for item in cleanup_errors)
|
||||
raise WorkerMonitorError('private CAD cleanup requires reconciliation: ' + message) from cleanup_error
|
||||
desktop_cleanup = getattr(self.desktop, 'cleanup_record', None) or {}
|
||||
forced = [item for item in desktop_cleanup.get('processes', []) if item.get('forced_termination')]
|
||||
self.record.update(forced_termination=bool(forced),
|
||||
forced_root_termination=any(item.get('forced_root_termination') for item in forced),
|
||||
termination_reason=next((item.get('termination_reason') for item in forced
|
||||
if item.get('termination_reason')), None),
|
||||
forced_descendant_pids=sorted({pid for item in forced for pid in item.get('forced_descendant_pids', [])}))
|
||||
if forced:
|
||||
remember('forced_cleanup', WorkerMonitorError('owned processes required forced termination'))
|
||||
if error is not None or failures:
|
||||
persist_uncertain()
|
||||
else:
|
||||
diagnose('released_record', lambda: self._record(
|
||||
state='session_closed' if self.borrowed_desktop else 'released',
|
||||
desktop_retained_by_batch=self.borrowed_desktop, clean=True,
|
||||
shutdown_seconds=time.monotonic() - closing_started,
|
||||
desktop_cleanup=desktop_cleanup, **exit_codes))
|
||||
if failures:
|
||||
persist_uncertain()
|
||||
if failures:
|
||||
raise WorkerMonitorError('private CAD cleanup requires reconciliation: ' + failure_message()) from failures[0]
|
||||
|
||||
def _uncertain(self, message, **details):
|
||||
from .quick_run import _write_uncertain_flag
|
||||
@@ -294,7 +355,7 @@ def _run_native_action(action, evidence, expected_pid):
|
||||
# COM startup can block inside a native call, so keep diagnostics outside
|
||||
# the redirected pipes and retain them when the supervisor ends the child.
|
||||
import faulthandler
|
||||
with Path(str(evidence) + '.fault.txt').open('a', encoding='utf-8') as trace:
|
||||
with internal_path(str(evidence) + '.fault.txt').open('a', encoding='utf-8') as trace:
|
||||
faulthandler.dump_traceback_later(20, repeat=True, file=trace)
|
||||
try:
|
||||
return _native_action(action, evidence, expected_pid)
|
||||
@@ -339,7 +400,7 @@ def _native_action(action, evidence, expected_pid):
|
||||
verify_application(app, binding, require_view=False)
|
||||
except ValueError as error:
|
||||
raise CadIdentityConflict(str(error)) from error
|
||||
Path(evidence).write_text(json.dumps(asdict(binding)), encoding='utf-8')
|
||||
internal_path(evidence).write_text(json.dumps(asdict(binding)), encoding='utf-8')
|
||||
elif action in {'quit', 'checkpoint'}:
|
||||
try:
|
||||
binding = CadBinding.from_environment()
|
||||
|
||||
@@ -8,6 +8,7 @@ from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from contextlib import ExitStack
|
||||
from copy import deepcopy
|
||||
import ctypes
|
||||
from ctypes import wintypes
|
||||
import math
|
||||
@@ -22,6 +23,20 @@ from .errors import TaskCenterError
|
||||
|
||||
|
||||
DESKTOP_RELEASE_TIMEOUT_SECONDS = 5.0
|
||||
DESCENDANT_EXIT_GRACE_SECONDS = 2.0
|
||||
|
||||
|
||||
def _native_handle(handle):
|
||||
try:
|
||||
return int(handle) if handle is not None else None
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _resource_event(operation, handle, success, error=None):
|
||||
return {'operation': operation, 'handle': _native_handle(handle), 'success': success,
|
||||
'pid': os.getpid(), 'thread_id': get_native_id(), 'time_ns': time.time_ns(),
|
||||
'error': str(error) if error is not None else None}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
@@ -178,7 +193,10 @@ def _stop_and_wait(process):
|
||||
errors.append(error)
|
||||
if running:
|
||||
try:
|
||||
process.terminate()
|
||||
if isinstance(process, _Win32Process):
|
||||
process.terminate(reason='root_process_active')
|
||||
else:
|
||||
process.terminate()
|
||||
except Exception as error:
|
||||
errors.append(error)
|
||||
try:
|
||||
@@ -299,7 +317,9 @@ class _Win32Process:
|
||||
self.job_assigned = False
|
||||
self.returncode = None
|
||||
self.stdout = self.stderr = None
|
||||
self.cleanup_record = {'pid': self.pid}
|
||||
self.cleanup_record = {'pid': self.pid, 'forced_termination': False,
|
||||
'forced_root_termination': False, 'forced_descendant_pids': [],
|
||||
'termination_reason': None}
|
||||
|
||||
def poll(self):
|
||||
if self.returncode is not None or self.handle is None:
|
||||
@@ -326,34 +346,59 @@ class _Win32Process:
|
||||
self.returncode = self._process_api.GetExitCodeProcess(self.handle)
|
||||
return self.returncode
|
||||
|
||||
def terminate(self):
|
||||
def terminate(self, *, reason='explicit_termination'):
|
||||
if self.handle is None:
|
||||
return
|
||||
errors = []
|
||||
root_active = True
|
||||
try:
|
||||
root_active = self.poll() is None
|
||||
except Exception as error:
|
||||
errors.append(error)
|
||||
self.cleanup_record.setdefault('root_exit_before_cleanup', not root_active)
|
||||
if not self.job_assigned:
|
||||
self._process_api.TerminateProcess(self.handle, 1)
|
||||
self._record_termination(reason, root_active, ())
|
||||
_raise_cleanup_errors(errors)
|
||||
return
|
||||
errors = []
|
||||
descendants = set()
|
||||
try:
|
||||
self._remember_descendants()
|
||||
# Keep confirmed members of this snapshot even if opening a later
|
||||
# member fails; the Job termination below still includes them.
|
||||
self._remember_descendants(active=descendants)
|
||||
except Exception as error:
|
||||
errors.append(error)
|
||||
try:
|
||||
self._job_api.TerminateJobObject(self.job, 1)
|
||||
self._record_termination(reason, root_active, descendants)
|
||||
except Exception:
|
||||
self._process_api.TerminateProcess(self.handle, 1)
|
||||
self._record_termination(reason, root_active, ())
|
||||
_raise_cleanup_errors(errors)
|
||||
|
||||
kill = terminate
|
||||
|
||||
def _remember_descendants(self):
|
||||
def _record_termination(self, reason, root_active, descendants):
|
||||
self.cleanup_record.update(forced_termination=True,
|
||||
forced_root_termination=self.cleanup_record['forced_root_termination'] or root_active,
|
||||
forced_descendant_pids=sorted(set(self.cleanup_record['forced_descendant_pids']) | set(descendants)),
|
||||
termination_reason=self.cleanup_record['termination_reason'] or reason)
|
||||
|
||||
def _remember_descendants(self, *, active=None):
|
||||
if active is None:
|
||||
active = set()
|
||||
if self._open_process is None or self.job is None:
|
||||
return
|
||||
return active
|
||||
for pid in self._job_api.QueryInformationJobObject(self.job, self._job_api.JobObjectBasicProcessIdList):
|
||||
if pid == self.pid or pid in self._descendant_pids:
|
||||
if pid == self.pid:
|
||||
continue
|
||||
if pid in self._descendant_pids:
|
||||
active.add(pid)
|
||||
continue
|
||||
try:
|
||||
self._descendant_handles.append(self._open_process(0x100000, False, pid))
|
||||
self._descendant_pids.add(pid)
|
||||
active.add(pid)
|
||||
except Exception as error:
|
||||
# pywintypes.error is not an OSError. A child that exited
|
||||
# between the job snapshot and OpenProcess reports error 87.
|
||||
@@ -362,22 +407,47 @@ class _Win32Process:
|
||||
code = error.args[0]
|
||||
if code != 87:
|
||||
raise
|
||||
return active
|
||||
|
||||
def close(self):
|
||||
errors = []
|
||||
job_empty = not self.job_assigned or self.job is None
|
||||
if self.handle is not None:
|
||||
if 'root_exit_before_cleanup' not in self.cleanup_record:
|
||||
try:
|
||||
self.cleanup_record['root_exit_before_cleanup'] = self.poll() is not None
|
||||
except Exception as error:
|
||||
errors.append(error)
|
||||
errors.extend(_stop_and_wait(self))
|
||||
deadline = time.monotonic() + 5.0
|
||||
if self.job is not None and self.job_assigned:
|
||||
try:
|
||||
# A completed root can leave grandchildren holding the desktop.
|
||||
self._remember_descendants()
|
||||
self._job_api.TerminateJobObject(self.job, 1)
|
||||
grace_started = time.monotonic()
|
||||
grace_deadline = grace_started + DESCENDANT_EXIT_GRACE_SECONDS
|
||||
# A completed root may leave children finishing normal shutdown.
|
||||
# An empty job takes no grace delay; an active root was already
|
||||
# stopped above and does not receive this normal-exit allowance.
|
||||
while True:
|
||||
active = self._job_api.QueryInformationJobObject(
|
||||
self.job, self._job_api.JobObjectBasicAccountingInformation)['ActiveProcesses']
|
||||
self.cleanup_record['job_active_processes'] = active
|
||||
if not active:
|
||||
job_empty = True
|
||||
break
|
||||
self._remember_descendants()
|
||||
if (self.cleanup_record.get('root_exit_before_cleanup') is not True
|
||||
or self.cleanup_record['forced_termination'] or time.monotonic() >= grace_deadline):
|
||||
self.terminate(reason='descendant_grace_expired')
|
||||
break
|
||||
time.sleep(min(0.01, max(0, grace_deadline - time.monotonic())))
|
||||
self.cleanup_record['descendant_grace_seconds'] = time.monotonic() - grace_started
|
||||
deadline = time.monotonic() + 5.0
|
||||
while True:
|
||||
active = self._job_api.QueryInformationJobObject(
|
||||
self.job, self._job_api.JobObjectBasicAccountingInformation)['ActiveProcesses']
|
||||
self.cleanup_record['job_active_processes'] = active
|
||||
if not active:
|
||||
job_empty = True
|
||||
break
|
||||
if time.monotonic() >= deadline:
|
||||
raise TimeoutError('owned job descendants did not exit within 5 seconds')
|
||||
@@ -403,6 +473,10 @@ class _Win32Process:
|
||||
self._descendant_handles = retained
|
||||
for name in ('job', 'thread', 'handle'):
|
||||
handle = getattr(self, name)
|
||||
if name in ('job', 'handle') and not job_empty:
|
||||
# Closing a KILL_ON_JOB_CLOSE job would silently force remaining
|
||||
# children and discard the capability needed for cleanup retry.
|
||||
continue
|
||||
if name == 'handle' and self.returncode is None:
|
||||
# Keep the only termination/wait capability until cleanup can retry.
|
||||
continue
|
||||
@@ -435,6 +509,8 @@ class PrivateDesktop:
|
||||
self._processes: list[subprocess.Popen[str]] = []
|
||||
self._api = api
|
||||
self.cleanup_record = None
|
||||
self._desktop_handle_history = []
|
||||
self._process_history = []
|
||||
|
||||
def __enter__(self) -> "PrivateDesktop":
|
||||
if not self.enabled:
|
||||
@@ -451,7 +527,12 @@ class PrivateDesktop:
|
||||
if os.name != "nt" and self._api is None:
|
||||
return
|
||||
self._api = self._api or _Win32Api()
|
||||
self._handle = self._api.create_desktop(self.name)
|
||||
try:
|
||||
self._handle = self._api.create_desktop(self.name)
|
||||
except Exception as error:
|
||||
self._desktop_handle_history.append(_resource_event('create', None, False, error))
|
||||
raise
|
||||
self._desktop_handle_history.append(_resource_event('create', self._handle, self._handle is not None))
|
||||
|
||||
def start_process(
|
||||
self,
|
||||
@@ -471,8 +552,20 @@ class PrivateDesktop:
|
||||
options = {} if stdout is None and stderr is None else {'stdout': stdout, 'stderr': stderr}
|
||||
process = self._api.create_process(command, build_startup_info(self.name).lpDesktop, cwd, env, **options)
|
||||
self._processes.append(process)
|
||||
self._process_history.append({'pid': getattr(process, 'pid', None),
|
||||
'owner_pid': os.getpid(), 'owner_thread_id': get_native_id(),
|
||||
'started_time_ns': time.time_ns(), 'process_handle': _native_handle(getattr(process, 'handle', None)),
|
||||
'job_handle': _native_handle(getattr(process, 'job', None)), 'forgotten': False})
|
||||
return process
|
||||
|
||||
def _remember_process_cleanup(self, process, *, forgotten=False):
|
||||
for record in reversed(self._process_history):
|
||||
if record['pid'] == getattr(process, 'pid', None) and not record['forgotten']:
|
||||
record['cleanup'] = deepcopy(getattr(process, 'cleanup_record', {}))
|
||||
record['forgotten'] = forgotten
|
||||
record['observed_time_ns'] = time.time_ns()
|
||||
break
|
||||
|
||||
def forget_closed_process(self, process):
|
||||
"""Drop a worker only after its owning wrapper confirmed all cleanup."""
|
||||
record = getattr(process, 'cleanup_record', {})
|
||||
@@ -482,6 +575,7 @@ class PrivateDesktop:
|
||||
or record.get('errors') or process.poll() is None):
|
||||
raise OSError('cannot forget a worker with unconfirmed owned resources')
|
||||
self._processes = [item for item in self._processes if item is not process]
|
||||
self._remember_process_cleanup(process, forgotten=True)
|
||||
|
||||
def close(self) -> None:
|
||||
errors, remaining = [], []
|
||||
@@ -497,6 +591,7 @@ class PrivateDesktop:
|
||||
failed = True
|
||||
self.cleanup_record['processes'].append(getattr(process, 'cleanup_record', {
|
||||
'pid': getattr(process, 'pid', None), 'exit_code': getattr(process, 'returncode', None)}))
|
||||
self._remember_process_cleanup(process)
|
||||
if failed:
|
||||
remaining.append(process)
|
||||
self._processes = list(reversed(remaining))
|
||||
@@ -508,11 +603,15 @@ class PrivateDesktop:
|
||||
errors.append(error)
|
||||
close_desktop = getattr(self._handle, "CloseDesktop", None)
|
||||
if close_desktop:
|
||||
event = _resource_event('close', self._handle, False)
|
||||
try:
|
||||
close_desktop()
|
||||
event['success'] = True
|
||||
self._handle = None
|
||||
except Exception as error:
|
||||
errors.append(error)
|
||||
event['error'] = str(error)
|
||||
self._desktop_handle_history.append(event)
|
||||
confirm = getattr(self._api, 'confirm_desktop_released', None)
|
||||
if confirm and self._handle is None:
|
||||
try:
|
||||
@@ -521,5 +620,50 @@ class PrivateDesktop:
|
||||
errors.append(error)
|
||||
self.cleanup_record.update(desktop_handle_closed=self._handle is None,
|
||||
desktop_confirmation=getattr(self._api, 'desktop_release_check', None),
|
||||
desktop_handle_history=deepcopy(self._desktop_handle_history),
|
||||
process_history=deepcopy(self._process_history),
|
||||
diagnostic_scope={'external_handle_holders': 'unknown',
|
||||
'thread_desktops': 'unknown unless observed through residual windows; no exhaustive holder scan',
|
||||
'empty_windows_prove_no_holders': False},
|
||||
errors=[str(error) for error in errors])
|
||||
if errors:
|
||||
self.cleanup_record['owned_resource_diagnostics'] = self._owned_resource_diagnostics()
|
||||
_raise_cleanup_errors(errors)
|
||||
|
||||
def _owned_resource_diagnostics(self):
|
||||
"""Bounded queries of our handles only; never terminate/query foreign PIDs.
|
||||
|
||||
GetThreadDesktop can introduce borrowed USER references. Do not run a
|
||||
global thread scan in this long-lived owner while retrying release.
|
||||
Window observations identify only window-owning threads, not every
|
||||
thread or an external OpenDesktop handle holder.
|
||||
"""
|
||||
resources = []
|
||||
for process in self._processes:
|
||||
item = {'pid': getattr(process, 'pid', None),
|
||||
'process_handle': _native_handle(getattr(process, 'handle', None)),
|
||||
'job_handle': _native_handle(getattr(process, 'job', None)),
|
||||
'thread_handle': _native_handle(getattr(process, 'thread', None)),
|
||||
'cleanup': deepcopy(getattr(process, 'cleanup_record', {}))}
|
||||
try:
|
||||
item['exit_code'] = process.poll()
|
||||
except Exception as error:
|
||||
item['poll_error'] = str(error)
|
||||
job = getattr(process, 'job', None)
|
||||
job_api = getattr(process, '_job_api', None)
|
||||
if job is not None and job_api is not None:
|
||||
try:
|
||||
item['job_accounting'] = job_api.QueryInformationJobObject(
|
||||
job, job_api.JobObjectBasicAccountingInformation)
|
||||
except Exception as error:
|
||||
item['job_query_error'] = str(error)
|
||||
resources.append(item)
|
||||
confirmation = getattr(self._api, 'desktop_release_check', None) or {}
|
||||
return {'owned_processes': resources,
|
||||
'pending_launch_cleanup_count': len(getattr(self._api, '_pending_cleanup', [])),
|
||||
'residual_windows': deepcopy(confirmation.get('remaining_windows')),
|
||||
'window_query_error': confirmation.get('window_query_error'),
|
||||
'external_handle_holders': 'unknown',
|
||||
'thread_desktops': {'observations': deepcopy(confirmation.get('remaining_windows', [])),
|
||||
'complete': False, 'unobserved_threads': 'unknown'},
|
||||
'empty_windows_prove_no_holders': False}
|
||||
@@ -11,6 +11,7 @@ import tempfile
|
||||
from threading import RLock
|
||||
|
||||
from .errors import TaskCenterError
|
||||
from .path_access import canonical_path, internal_path, retry_sharing_violation, strip_extended_prefix
|
||||
|
||||
|
||||
def _short_path(path: Path) -> str | None:
|
||||
@@ -18,8 +19,8 @@ def _short_path(path: Path) -> str | None:
|
||||
function.argtypes = [ctypes.c_wchar_p, ctypes.c_wchar_p, ctypes.c_uint]
|
||||
function.restype = ctypes.c_uint
|
||||
buffer = ctypes.create_unicode_buffer(32768)
|
||||
count = function(str(path), buffer, len(buffer))
|
||||
return buffer.value if count and count < len(buffer) else None
|
||||
count = function(str(internal_path(path)), buffer, len(buffer))
|
||||
return strip_extended_prefix(buffer.value) if count and count < len(buffer) else None
|
||||
|
||||
|
||||
def _fits(root) -> bool:
|
||||
@@ -34,12 +35,13 @@ class _AliasPool:
|
||||
self.lock = RLock()
|
||||
|
||||
def alias(self, root: Path) -> Path:
|
||||
root = canonical_path(root, strict=True)
|
||||
import _winapi
|
||||
with self.lock:
|
||||
if self.directory is None:
|
||||
local = Path(os.environ.get('LOCALAPPDATA') or tempfile.gettempdir())
|
||||
parent = local / 'DxProductivity' / 'path-links'
|
||||
parent.mkdir(parents=True, exist_ok=True)
|
||||
internal_path(parent).mkdir(parents=True, exist_ok=True)
|
||||
self.directory = Path(tempfile.mkdtemp(prefix='dxp-', dir=parent))
|
||||
key = hashlib.sha256(os.path.normcase(str(root)).encode('utf-8')).hexdigest()[:20]
|
||||
link = self.directory / key
|
||||
@@ -50,7 +52,7 @@ class _AliasPool:
|
||||
# junction contains no copies and never changes the target.
|
||||
_winapi.CreateJunction(str(root), str(link))
|
||||
self.links[link] = root
|
||||
if not link.is_junction() or link.resolve(strict=True) != root:
|
||||
if not link.is_junction() or canonical_path(link, strict=True) != root:
|
||||
raise OSError('compatibility directory no longer points to the confirmed evidence')
|
||||
return link
|
||||
|
||||
@@ -60,7 +62,7 @@ class _AliasPool:
|
||||
for link, target in tuple(self.links.items()):
|
||||
try:
|
||||
if link.is_junction():
|
||||
if link.resolve() != target:
|
||||
if canonical_path(link) != target:
|
||||
raise OSError('refusing to remove a redirected compatibility directory')
|
||||
link.rmdir() # Remove only the link, never the target tree.
|
||||
elif link.exists():
|
||||
@@ -88,11 +90,12 @@ atexit.register(cleanup_worker_aliases)
|
||||
|
||||
|
||||
def worker_paths(script: Path, root: Path) -> tuple[str, str]:
|
||||
script, root = canonical_path(script), canonical_path(root)
|
||||
if os.name == 'nt':
|
||||
_prepare_project_access(root)
|
||||
if os.name != 'nt' or _fits(root):
|
||||
return str(script), str(root)
|
||||
canonical = root.resolve(strict=True)
|
||||
canonical = canonical_path(internal_path(root), strict=True)
|
||||
short = _short_path(canonical)
|
||||
if short and _fits(short):
|
||||
return str(Path(short) / script.name), short
|
||||
@@ -106,7 +109,7 @@ def worker_paths(script: Path, root: Path) -> tuple[str, str]:
|
||||
|
||||
def _prepare_project_access(root: Path):
|
||||
"""Keep command identities canonical while giving legacy CAD short entry paths."""
|
||||
command = root / 'command.tsv'
|
||||
command = internal_path(root / 'command.tsv')
|
||||
if not command.is_file():
|
||||
return
|
||||
rows = []
|
||||
@@ -117,32 +120,32 @@ def _prepare_project_access(root: Path):
|
||||
if not value or value in seen:
|
||||
continue
|
||||
seen.add(value)
|
||||
project = Path(value).resolve(strict=False)
|
||||
if (_fits(project.parent) and len(str(project)) < 260) or not project.is_file():
|
||||
project = canonical_path(internal_path(value))
|
||||
if (_fits(project.parent) and len(str(project)) < 260) or not internal_path(project).is_file():
|
||||
continue
|
||||
short = _short_path(project)
|
||||
if short and _fits(Path(short).parent) and len(short) < 260:
|
||||
access = Path(short)
|
||||
else:
|
||||
access = _aliases.alias(project.parent) / project.name
|
||||
if not _fits(access.parent) or len(str(access)) >= 260 or access.resolve(strict=True) != project:
|
||||
if not _fits(access.parent) or len(str(access)) >= 260 or canonical_path(access, strict=True) != project:
|
||||
raise TaskCenterError('quick_run_path_length', '工程短路径入口不能与原工程身份对应,未启动 CAD 操作。')
|
||||
pair = (str(project), str(access))
|
||||
if pair not in rows:
|
||||
rows.append(pair)
|
||||
target = root / 'project-access.tsv'
|
||||
target = internal_path(root / 'project-access.tsv')
|
||||
if rows or target.exists():
|
||||
temporary = target.with_suffix('.tmp')
|
||||
with temporary.open('w', encoding='utf-8', newline='') as stream:
|
||||
writer = csv.writer(stream, delimiter='\t', lineterminator='\n')
|
||||
writer.writerow(('project_path', 'access_path'))
|
||||
writer.writerows(rows)
|
||||
temporary.replace(target)
|
||||
retry_sharing_violation(lambda: temporary.replace(target))
|
||||
|
||||
|
||||
def validate_project_access(root: Path):
|
||||
"""Reject disappeared/redirected entry paths before and after a native worker."""
|
||||
target = Path(root) / 'project-access.tsv'
|
||||
target = internal_path(Path(root) / 'project-access.tsv')
|
||||
if not target.is_file():
|
||||
return
|
||||
with target.open(encoding='utf-8-sig', newline='') as stream:
|
||||
@@ -151,7 +154,7 @@ def validate_project_access(root: Path):
|
||||
raise TaskCenterError('target_identity', '工程路径入口清单格式已改变')
|
||||
for row in reader:
|
||||
try:
|
||||
same = Path(row['access_path']).resolve(strict=True) == Path(row['project_path']).resolve(strict=True)
|
||||
same = canonical_path(internal_path(row['access_path']), strict=True) == canonical_path(internal_path(row['project_path']), strict=True)
|
||||
except (OSError, ValueError, KeyError):
|
||||
same = False
|
||||
if not same:
|
||||
|
||||
@@ -18,6 +18,7 @@ import traceback
|
||||
from typing import Any, Mapping
|
||||
|
||||
from .errors import TaskCenterError
|
||||
from .path_access import internal_path
|
||||
from .retention import rotate_text_log
|
||||
|
||||
|
||||
@@ -75,24 +76,28 @@ class LoggingContext:
|
||||
return reported
|
||||
|
||||
|
||||
def configure_logging(root: Path) -> LoggingContext:
|
||||
def configure_logging(root: Path, *, log_path: Path | None = None) -> LoggingContext:
|
||||
"""Create storage directories and a UTF-8 rotating application log handler."""
|
||||
storage_root = Path(root).resolve()
|
||||
log_path = storage_root / "logs" / "application.log"
|
||||
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
(storage_root / "tasks").mkdir(parents=True, exist_ok=True)
|
||||
from .path_access import canonical_path, internal_path
|
||||
storage_root = canonical_path(root)
|
||||
separate_log = log_path is not None
|
||||
log_path = canonical_path(log_path) if separate_log else storage_root / "logs" / "application.log"
|
||||
internal_path(log_path.parent).mkdir(parents=True, exist_ok=True)
|
||||
internal_path(storage_root / "tasks").mkdir(parents=True, exist_ok=True)
|
||||
logger_name = "dx_productivity.trial_task_center." + hashlib.sha256(
|
||||
str(storage_root.resolve()).encode("utf-8")
|
||||
).hexdigest()[:16]
|
||||
if separate_log:
|
||||
logger_name += '.' + hashlib.sha256(str(log_path).encode('utf-8')).hexdigest()[:12]
|
||||
logger = logging.getLogger(logger_name)
|
||||
logger.setLevel(logging.DEBUG)
|
||||
logger.propagate = False
|
||||
if not any(
|
||||
isinstance(handler, RotatingFileHandler) and Path(handler.baseFilename) == log_path
|
||||
isinstance(handler, RotatingFileHandler) and canonical_path(handler.baseFilename) == log_path
|
||||
for handler in logger.handlers
|
||||
):
|
||||
handler = RotatingFileHandler(
|
||||
log_path,
|
||||
internal_path(log_path),
|
||||
maxBytes=_MAX_LOG_BYTES,
|
||||
backupCount=_BACKUP_COUNT,
|
||||
encoding="utf-8",
|
||||
@@ -151,11 +156,11 @@ def append_task_event(
|
||||
encoded = json.dumps(event, ensure_ascii=False, separators=(",", ":"), allow_nan=False) + "\n"
|
||||
try:
|
||||
rotate_text_log(
|
||||
path,
|
||||
internal_path(path),
|
||||
max_bytes=_MAX_EVENT_LOG_BYTES,
|
||||
backup_count=_EVENT_BACKUP_COUNT,
|
||||
)
|
||||
with path.open("a", encoding="utf-8", newline="\n") as handle:
|
||||
with internal_path(path).open("a", encoding="utf-8", newline="\n") as handle:
|
||||
handle.write(encoded)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
@@ -183,7 +188,7 @@ def append_task_event(
|
||||
|
||||
def _event_file_stamp(path):
|
||||
try:
|
||||
info = path.stat()
|
||||
info = internal_path(path).stat()
|
||||
return info.st_dev, info.st_ino, info.st_size, info.st_mtime_ns, info.st_ctime_ns
|
||||
except OSError:
|
||||
return None
|
||||
@@ -226,15 +231,15 @@ def _task_directory(context: LoggingContext, task_id: str) -> Path:
|
||||
if not task_id or Path(task_id).name != task_id or task_id in {".", ".."}:
|
||||
raise TaskCenterError("task_id", "task event requires a safe task identifier")
|
||||
path = context.root / "tasks" / task_id
|
||||
if not path.is_dir():
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
if not internal_path(path).is_dir():
|
||||
internal_path(path).mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
|
||||
def _next_event_sequence(path: Path) -> int:
|
||||
try:
|
||||
for candidate in (path, path.with_name(f"{path.name}.1")):
|
||||
if not candidate.exists():
|
||||
if not internal_path(candidate).exists():
|
||||
continue
|
||||
lines = _tail_utf8_lines(candidate, 1)
|
||||
if lines:
|
||||
@@ -248,7 +253,7 @@ def _next_event_sequence(path: Path) -> int:
|
||||
def _tail_utf8_lines(path: Path, limit: int) -> tuple[str, ...]:
|
||||
if limit < 1:
|
||||
return ()
|
||||
with Path(path).open("rb") as handle:
|
||||
with internal_path(Path(path)).open("rb") as handle:
|
||||
handle.seek(0, os.SEEK_END)
|
||||
cursor = handle.tell()
|
||||
chunks = []
|
||||
|
||||
@@ -7,11 +7,13 @@ import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
from .errors import TaskCenterError
|
||||
from .path_access import canonical_path, internal_path, retry_sharing_violation
|
||||
from .isolated_desktop import PrivateDesktop
|
||||
from .native_inventory_protocol import NativeInventoryUnsupported, read_ascii_discovery, read_native_text
|
||||
from .operation_control import check_cancelled, report_progress
|
||||
@@ -27,20 +29,20 @@ class _NativeInventoryProcessError(NativeInventoryUnsupported):
|
||||
|
||||
|
||||
def _hash(path):
|
||||
with path.open('rb') as stream:
|
||||
with internal_path(path).open('rb') as stream:
|
||||
return hashlib.file_digest(stream, 'sha256').hexdigest()
|
||||
|
||||
|
||||
def _verify_sources(hashes):
|
||||
try:
|
||||
if any(path.resolve(strict=True) != path or _hash(path) != digest for path, digest in hashes.items()):
|
||||
if any(canonical_path(internal_path(path), strict=True) != path or _hash(path) != digest for path, digest in hashes.items()):
|
||||
raise OSError('source identity changed')
|
||||
except OSError as error:
|
||||
raise TaskCenterError('target_identity', '原生采集期间源工程或数据库已变化或无法核验,请重新导入') from error
|
||||
|
||||
|
||||
def _project_options(project):
|
||||
text = read_native_text(project)
|
||||
text = read_native_text(internal_path(project))
|
||||
def option(name, default=None):
|
||||
values = set(re.findall(r'^KEY ' + name + r' "([^"\r\n]*)"\s*$', text, re.MULTILINE))
|
||||
if not values and default is not None:
|
||||
@@ -54,8 +56,8 @@ def _project_options(project):
|
||||
snapshot = option('FrontEndSnapshot', 'DxD')
|
||||
if snapshot.casefold() != 'dxd':
|
||||
raise NativeInventoryUnsupported('当前工程不是已验证的 DxD 数据库快照')
|
||||
database = (project.parent / option('iCDBDir') / 'icdb.dat').resolve(strict=True)
|
||||
if not database.is_relative_to(project.parent) or not database.is_file():
|
||||
database = canonical_path(internal_path(project.parent / option('iCDBDir') / 'icdb.dat'), strict=True)
|
||||
if not database.is_relative_to(project.parent) or not internal_path(database).is_file():
|
||||
raise NativeInventoryUnsupported('数据库不在当前工程目录内')
|
||||
return text, block, snapshot, database
|
||||
|
||||
@@ -83,7 +85,7 @@ class NativeInventoryReader:
|
||||
def read(self, project, old_part_numbers, *, stop_event=None, on_progress=None, runtime=None):
|
||||
check_cancelled(stop_event)
|
||||
try:
|
||||
project = Path(project).resolve(strict=True)
|
||||
project = canonical_path(internal_path(project), strict=True)
|
||||
text, block, snapshot, database = _project_options(project)
|
||||
tools = self._tools()
|
||||
except (NativeInventoryUnsupported, OSError, ImportError, UnicodeError, TaskCenterError) as error:
|
||||
@@ -99,13 +101,13 @@ class NativeInventoryReader:
|
||||
|
||||
def _read(self, project, text, block, snapshot, database, tools, old_part_numbers, *, stop_event, on_progress, runtime=None, allow_retry=False):
|
||||
token = uuid4().hex
|
||||
evidence = self.data_root / 'quick-run-history' / ('native-inventory-' + token)
|
||||
evidence.mkdir(parents=True, exist_ok=False)
|
||||
evidence = internal_path(self.data_root / 'quick-run-history' / ('native-inventory-' + token))
|
||||
retry_sharing_violation(lambda: evidence.mkdir(parents=True, exist_ok=False))
|
||||
base = Path(self.temporary_root or Path(os.environ.get('LOCALAPPDATA') or tempfile.gettempdir())
|
||||
/ 'DxProductivity' / 'native-inventory').resolve()
|
||||
base.mkdir(parents=True, exist_ok=True)
|
||||
workspace = Path(tempfile.mkdtemp(prefix='dx-cli-', dir=base)).resolve()
|
||||
desktop = (runtime.native_scope(evidence) if runtime is not None
|
||||
desktop = (runtime.native_scope(canonical_path(evidence)) if runtime is not None
|
||||
else self.desktop_factory('dx-cli-' + token, enabled=True))
|
||||
record = {'backend': 'native_cli', 'project': str(project), 'state': 'reading',
|
||||
'workspace': str(workspace), 'tools': []}
|
||||
@@ -115,12 +117,12 @@ class NativeInventoryReader:
|
||||
source_hashes = {}
|
||||
try:
|
||||
source_hashes = {project: _hash(project), database: _hash(database)}
|
||||
if read_native_text(project) != text:
|
||||
if read_native_text(internal_path(project)) != text:
|
||||
raise TaskCenterError('target_identity', '创建原生采集快照时工程配置已变化,请重新导入')
|
||||
source = workspace / 'input.prj'
|
||||
target_database = workspace / 'database'
|
||||
target_database.mkdir()
|
||||
shutil.copyfile(database, target_database / 'icdb.dat')
|
||||
shutil.copyfile(internal_path(database), target_database / 'icdb.dat')
|
||||
# No server address files or running database directory are copied.
|
||||
text = re.sub(r'^KEY iCDBDir "[^"\r\n]*"\s*$',
|
||||
'KEY iCDBDir "database"', text, flags=re.MULTILINE)
|
||||
@@ -145,17 +147,21 @@ class NativeInventoryReader:
|
||||
self._run_tool(desktop, command, workspace, environment, output,
|
||||
record, stop_event)
|
||||
finally:
|
||||
primary_error = sys.exception()
|
||||
try:
|
||||
desktop.close()
|
||||
clean = True
|
||||
except Exception as error:
|
||||
record.update(state='uncertain', cleanup_error=str(error))
|
||||
record.update(state='uncertain', cleanup_error=str(error),
|
||||
primary_error=str(primary_error) if primary_error is not None else None)
|
||||
try:
|
||||
(evidence / 'uncertain.flag').write_text(str(error), encoding='utf-8')
|
||||
except OSError as marker_error:
|
||||
record['uncertain_marker_error'] = str(marker_error)
|
||||
raise TaskCenterError('inventory_cleanup',
|
||||
f'原生采集进程未能确认安全释放,请保留诊断记录:{evidence}') from error
|
||||
failure = TaskCenterError('inventory_cleanup',
|
||||
f'原生采集进程未能确认安全释放,请保留诊断记录:{canonical_path(evidence)}')
|
||||
failure.primary_error, failure.cleanup_error = primary_error, error
|
||||
raise failure from error
|
||||
_verify_sources(source_hashes)
|
||||
check_cancelled(stop_event)
|
||||
result = read_ascii_discovery(project, output / 'scope.txt', old_part_numbers,
|
||||
@@ -178,7 +184,7 @@ class NativeInventoryReader:
|
||||
write_record(evidence / 'native-retry-proof.json', {
|
||||
'version': 1, 'native_read_only': True, 'project_path': str(project),
|
||||
'source_hashes': {str(path): digest for path, digest in source_hashes.items()},
|
||||
'workers_closed': True, 'evidence_root': str(evidence.resolve()),
|
||||
'workers_closed': True, 'evidence_root': str(canonical_path(evidence)),
|
||||
'binding': dict(desktop.identity),
|
||||
})
|
||||
_verify_sources(source_hashes)
|
||||
@@ -248,11 +254,15 @@ class NativeInventoryReader:
|
||||
if metric['exit_code'] != 0:
|
||||
raise NativeInventoryUnsupported(f'{name} 返回退出码 {metric["exit_code"]}')
|
||||
finally:
|
||||
primary_error = sys.exception()
|
||||
metric['seconds'] = time.monotonic() - started
|
||||
if process is not None and callable(getattr(process, 'close', None)):
|
||||
try:
|
||||
process.close()
|
||||
except Exception as error:
|
||||
record.update(state='uncertain', cleanup_error=str(error))
|
||||
raise TaskCenterError('inventory_cleanup',
|
||||
f'原生读取进程及其子进程未能确认退出:{error}') from error
|
||||
record.update(state='uncertain', cleanup_error=str(error),
|
||||
primary_error=str(primary_error) if primary_error is not None else None)
|
||||
failure = TaskCenterError('inventory_cleanup',
|
||||
f'原生读取进程及其子进程未能确认退出:{error}')
|
||||
failure.primary_error, failure.cleanup_error = primary_error, error
|
||||
raise failure from error
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Filesystem access paths never replace canonical project/resource identities."""
|
||||
from __future__ import annotations
|
||||
|
||||
import ntpath
|
||||
import os
|
||||
from pathlib import Path
|
||||
import time
|
||||
|
||||
|
||||
def strip_extended_prefix(value):
|
||||
value = os.fspath(value)
|
||||
if value.startswith('\\\\?\\UNC\\'):
|
||||
return '\\\\' + value[8:]
|
||||
if value.startswith('\\\\?\\'):
|
||||
return value[4:]
|
||||
return value
|
||||
|
||||
|
||||
def windows_extended_path(value):
|
||||
value = os.fspath(value)
|
||||
if value.startswith('\\\\?\\'):
|
||||
return value
|
||||
value = ntpath.abspath(value)
|
||||
return '\\\\?\\UNC\\' + value[2:] if value.startswith('\\\\') else '\\\\?\\' + value
|
||||
|
||||
|
||||
def canonical_path(value, *, strict=False):
|
||||
return Path(strip_extended_prefix(internal_path(value).resolve(strict=strict)))
|
||||
|
||||
|
||||
def internal_path(value):
|
||||
"""Use Unicode extended Windows paths only at internal filesystem boundaries."""
|
||||
return Path(windows_extended_path(value)) if os.name == 'nt' else Path(value)
|
||||
|
||||
|
||||
def _sharing_denial(error, target):
|
||||
if getattr(error, 'winerror', None) in (32, 33):
|
||||
return True
|
||||
if os.name != 'nt' or getattr(error, 'winerror', None) != 5 or target is None:
|
||||
return False
|
||||
# MoveFileEx can report ACCESS_DENIED for a reader denying delete sharing.
|
||||
# Confirm that specific condition, rather than retrying arbitrary ACL errors.
|
||||
import ctypes
|
||||
from ctypes import wintypes
|
||||
kernel = ctypes.WinDLL('kernel32', use_last_error=True)
|
||||
create = kernel.CreateFileW
|
||||
create.argtypes = (wintypes.LPCWSTR, wintypes.DWORD, wintypes.DWORD, ctypes.c_void_p,
|
||||
wintypes.DWORD, wintypes.DWORD, wintypes.HANDLE)
|
||||
create.restype = wintypes.HANDLE
|
||||
handle = create(str(internal_path(target)), 0x10000, 7, None, 3, 0, None)
|
||||
if handle == ctypes.c_void_p(-1).value:
|
||||
return ctypes.get_last_error() in (32, 33)
|
||||
close = kernel.CloseHandle
|
||||
close.argtypes = (wintypes.HANDLE,)
|
||||
close(handle)
|
||||
return False
|
||||
|
||||
|
||||
def retry_sharing_violation(operation, *, target=None, sleep=time.sleep):
|
||||
"""At most three attempts and 150ms delay; access denial is never replayed."""
|
||||
for attempt in range(3):
|
||||
try:
|
||||
return operation()
|
||||
except OSError as error:
|
||||
if attempt == 2 or not _sharing_denial(error, target):
|
||||
raise
|
||||
sleep(.05 * (attempt + 1))
|
||||
|
||||
|
||||
def legacy_path(value):
|
||||
"""Give CAD/WSH a short entry while retaining canonical identity elsewhere."""
|
||||
from .legacy_paths import _aliases, _fits, _short_path
|
||||
path = canonical_path(value, strict=True)
|
||||
if os.name != 'nt' or (_fits(path.parent) and len(str(path)) < 260):
|
||||
return path
|
||||
short = _short_path(internal_path(path))
|
||||
if short and _fits(Path(short).parent) and len(short) < 260:
|
||||
access = Path(short)
|
||||
elif path.is_dir():
|
||||
access = _aliases.alias(path)
|
||||
else:
|
||||
access = _aliases.alias(path.parent) / path.name
|
||||
if canonical_path(access, strict=True) != path:
|
||||
raise OSError('compatibility entry does not identify the canonical path')
|
||||
return access
|
||||
@@ -10,6 +10,7 @@ import subprocess
|
||||
from uuid import uuid4
|
||||
|
||||
from .errors import TaskCenterError
|
||||
from .path_access import canonical_path, internal_path, retry_sharing_violation
|
||||
from .quick_run_protocol import ProjectDiscovery, _encode_vbs_field
|
||||
from .operation_control import check_cancelled, report_progress
|
||||
|
||||
@@ -64,13 +65,13 @@ def discover_project(
|
||||
) -> ProjectDiscovery:
|
||||
"""Enumerate sheets and requested source parts while the owned project is open."""
|
||||
check_cancelled(stop_event)
|
||||
project = project_path.resolve(strict=True)
|
||||
project = canonical_path(internal_path(project_path), strict=True)
|
||||
token = uuid4().hex
|
||||
job_id = "discover_" + token
|
||||
root = data_root / "quick-run-history" / ("sheet-discovery-" + token)
|
||||
root.mkdir(parents=True, exist_ok=False)
|
||||
root = internal_path(data_root / "quick-run-history" / ("sheet-discovery-" + token))
|
||||
retry_sharing_violation(lambda: root.mkdir(parents=True, exist_ok=False))
|
||||
script = root / "QuickPartReplace.vbs"
|
||||
shutil.copyfile(script_source, script)
|
||||
shutil.copyfile(internal_path(script_source), script)
|
||||
(root / "command.tsv").write_text(
|
||||
"task_id\tjob_id\tproject_path\told_part\tnew_part\tvalue_mode\r\n"
|
||||
f"sheet_discovery\t{job_id}\t{project}\tDISCOVERY_OLD\tDISCOVERY_NEW\tpreserve\r\n",
|
||||
@@ -93,7 +94,7 @@ def discover_project(
|
||||
)
|
||||
if collect_components:
|
||||
(root / "component-inventory.request").write_text(job_id, encoding="ascii")
|
||||
script_argument, root_argument = runtime.worker_paths(script, root)
|
||||
script_argument, root_argument = runtime.worker_paths(canonical_path(script), canonical_path(root))
|
||||
command = (
|
||||
str(runtime.cscript_resolver()),
|
||||
"//nologo",
|
||||
|
||||
@@ -111,6 +111,13 @@ class ProjectSessionWorker:
|
||||
return
|
||||
if not self._done.is_set():
|
||||
_write_control(self.root / 'project-session-next.tsv', f'{self.token}\tclose')
|
||||
from .stop_control import StopControl
|
||||
interactive = isinstance(getattr(self.runtime, 'stop_event', None), StopControl)
|
||||
if interactive:
|
||||
# The monitor owns the safe/unsafe distinction. Do not
|
||||
# turn a GUI close or ordinary pause into a timed kill.
|
||||
while not self._done.wait(.1):
|
||||
pass
|
||||
close_grace = 60 if self.private_desktop else min(30, max(10, self._timeout))
|
||||
if not self._done.wait(close_grace):
|
||||
self._abort.set()
|
||||
|
||||
@@ -11,19 +11,20 @@ import stat
|
||||
from uuid import uuid4
|
||||
|
||||
from .errors import TaskCenterError
|
||||
from .path_access import canonical_path, internal_path, strip_extended_prefix
|
||||
from .models import TaskRecord, TaskStatus
|
||||
from .scanner import _is_schematic_project, _reject_reparse
|
||||
from .store import TaskStore
|
||||
|
||||
|
||||
def _validated_root(root: Path, excluded_roots: tuple[Path, ...]) -> tuple[Path, tuple[Path, ...]]:
|
||||
root = Path(os.path.abspath(root))
|
||||
root = Path(os.path.abspath(strip_extended_prefix(root)))
|
||||
for ancestor in (*reversed(root.parents), root):
|
||||
_reject_reparse(ancestor.lstat(), ancestor)
|
||||
root = root.resolve(strict=True)
|
||||
if not root.is_dir():
|
||||
_reject_reparse(internal_path(ancestor).lstat(), ancestor)
|
||||
root = canonical_path(root, strict=True)
|
||||
if not internal_path(root).is_dir():
|
||||
raise TaskCenterError("discovery_root", "project root is not a directory")
|
||||
excluded = tuple(Path(path).resolve(strict=False) for path in excluded_roots)
|
||||
excluded = tuple(canonical_path(path) for path in excluded_roots)
|
||||
if any(root.is_relative_to(blocked) for blocked in excluded):
|
||||
raise TaskCenterError("discovery_root", "project root is inside an excluded directory")
|
||||
return root, excluded
|
||||
@@ -49,9 +50,9 @@ def scan_project_directory(root: Path, *, excluded_roots: tuple[Path, ...] = (),
|
||||
while pending:
|
||||
check_cancelled()
|
||||
directory = pending.pop()
|
||||
_reject_reparse(directory.lstat(), directory)
|
||||
_reject_reparse(internal_path(directory).lstat(), directory)
|
||||
children = []
|
||||
with os.scandir(directory) as entries:
|
||||
with os.scandir(internal_path(directory)) as entries:
|
||||
for entry in entries:
|
||||
check_cancelled()
|
||||
children.append(entry)
|
||||
@@ -59,7 +60,7 @@ def scan_project_directory(root: Path, *, excluded_roots: tuple[Path, ...] = (),
|
||||
folders: list[Path] = []
|
||||
for entry in sorted(children, key=lambda item: item.name.casefold()):
|
||||
check_cancelled()
|
||||
path = Path(entry.path)
|
||||
path = Path(strip_extended_prefix(entry.path))
|
||||
if any(path.is_relative_to(blocked) for blocked in excluded):
|
||||
continue
|
||||
details = entry.stat(follow_symlinks=False)
|
||||
@@ -67,7 +68,7 @@ def scan_project_directory(root: Path, *, excluded_roots: tuple[Path, ...] = (),
|
||||
if stat.S_ISDIR(details.st_mode):
|
||||
folders.append(path)
|
||||
elif stat.S_ISREG(details.st_mode) and path.suffix.casefold() == ".prj" and _is_schematic_project(path):
|
||||
projects.append(path.resolve(strict=True))
|
||||
projects.append(canonical_path(path, strict=True))
|
||||
found.extend(projects)
|
||||
visited += 1
|
||||
if on_progress is not None:
|
||||
@@ -82,15 +83,15 @@ def _validate_selected(root: Path, selected_paths: tuple[Path, ...], excluded: t
|
||||
selected: set[Path] = set()
|
||||
ancestor_projects: dict[Path, bool] = {}
|
||||
for candidate in selected_paths:
|
||||
path = Path(os.path.abspath(candidate))
|
||||
path = Path(os.path.abspath(strip_extended_prefix(candidate)))
|
||||
if not path.is_relative_to(root) or any(path.is_relative_to(blocked) for blocked in excluded):
|
||||
raise TaskCenterError("project_selection", "selected project is outside discovery results")
|
||||
relative = path.relative_to(root)
|
||||
cursor = root
|
||||
for part in relative.parts:
|
||||
cursor /= part
|
||||
_reject_reparse(cursor.lstat(), cursor)
|
||||
if not path.is_file() or path.suffix.casefold() != ".prj" or not _is_schematic_project(path):
|
||||
_reject_reparse(internal_path(cursor).lstat(), cursor)
|
||||
if not internal_path(path).is_file() or path.suffix.casefold() != ".prj" or not _is_schematic_project(path):
|
||||
raise TaskCenterError("project_selection", "selected file is not a schematic project")
|
||||
# A project in any ancestor directory stops discovery below that directory.
|
||||
directory = path.parent
|
||||
@@ -98,9 +99,9 @@ def _validate_selected(root: Path, selected_paths: tuple[Path, ...], excluded: t
|
||||
directory = directory.parent
|
||||
if directory not in ancestor_projects:
|
||||
has_project = False
|
||||
with os.scandir(directory) as entries:
|
||||
with os.scandir(internal_path(directory)) as entries:
|
||||
for entry in entries:
|
||||
sibling = Path(entry.path)
|
||||
sibling = Path(strip_extended_prefix(entry.path))
|
||||
if sibling.suffix.casefold() != ".prj" or any(sibling.is_relative_to(b) for b in excluded):
|
||||
continue
|
||||
details = entry.stat(follow_symlinks=False)
|
||||
@@ -111,7 +112,7 @@ def _validate_selected(root: Path, selected_paths: tuple[Path, ...], excluded: t
|
||||
ancestor_projects[directory] = has_project
|
||||
if ancestor_projects[directory]:
|
||||
raise TaskCenterError("project_selection", "selected project is hidden by an ancestor project")
|
||||
selected.add(path.resolve(strict=True))
|
||||
selected.add(canonical_path(path, strict=True))
|
||||
return tuple(sorted(selected, key=lambda path: str(path).casefold()))
|
||||
|
||||
|
||||
@@ -120,7 +121,7 @@ def create_project_tasks(store: TaskStore, root: Path, *, selected_paths: tuple[
|
||||
root, excluded = _validated_root(root, excluded_roots)
|
||||
discovered = (scan_project_directory(root, excluded_roots=excluded) if selected_paths is None
|
||||
else _validate_selected(root, selected_paths, excluded))
|
||||
active_paths = {project.path.resolve(strict=False) for task in store.list_tasks()
|
||||
active_paths = {canonical_path(project.path) for task in store.list_tasks()
|
||||
if task.status not in {TaskStatus.DELETED, TaskStatus.COMPLETED}
|
||||
for project in store.list_projects(task.task_id)}
|
||||
pending = [path for path in discovered if path not in active_paths]
|
||||
|
||||
@@ -20,6 +20,7 @@ from typing import Protocol
|
||||
from uuid import uuid4
|
||||
|
||||
from .errors import TaskCenterError
|
||||
from .path_access import internal_path
|
||||
from .models import ProjectRecord, ProjectStatus, RuleRecord
|
||||
from .retention import maintain_quick_run_storage, rotate_text_log
|
||||
from .worker_runtime import copy_worker_script
|
||||
@@ -112,22 +113,22 @@ class QuickRunPreparer:
|
||||
self._validate_fields(selected_projects, enabled_rules)
|
||||
_validate_target_scope(selected_projects, enabled_rules, sheet_targets)
|
||||
run_token = _safe_run_token(self._run_token_factory())
|
||||
maintain_quick_run_storage(self._data_root)
|
||||
maintain_quick_run_storage(_io_path(self._data_root))
|
||||
|
||||
root = self._data_root / "quick-run"
|
||||
script_path = root / "QuickPartReplace.vbs"
|
||||
command_path = root / "command.tsv"
|
||||
status_path = root / "status.tsv"
|
||||
log_path = root / "quick-part-replace.log"
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
_io_path(root).mkdir(parents=True, exist_ok=True)
|
||||
_assert_no_pending_status(status_path, log_path)
|
||||
copy_worker_script(source, script_path)
|
||||
if sheet_targets is not None:
|
||||
manifest_path = root / "sheets.tsv"
|
||||
manifest_path.write_text(_sheet_manifest_text(selected_projects, sheet_targets), encoding="utf-8", newline="")
|
||||
_io_path(manifest_path).write_text(_sheet_manifest_text(selected_projects, sheet_targets), encoding="utf-8", newline="")
|
||||
else:
|
||||
(root / "sheets.tsv").unlink(missing_ok=True)
|
||||
command_path.write_text(
|
||||
(_io_path(root / "sheets.tsv")).unlink(missing_ok=True)
|
||||
_io_path(command_path).write_text(
|
||||
_command_text(
|
||||
task_id,
|
||||
run_token,
|
||||
@@ -144,7 +145,7 @@ class QuickRunPreparer:
|
||||
discovery_matches,
|
||||
discovery_failures,
|
||||
)
|
||||
status_path.write_text(
|
||||
_io_path(status_path).write_text(
|
||||
initial_status,
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
@@ -157,7 +158,7 @@ class QuickRunPreparer:
|
||||
root = self._data_root / "quick-run"
|
||||
command_path = root / "command.tsv"
|
||||
status_path = root / "status.tsv"
|
||||
if not command_path.is_file() or not status_path.is_file():
|
||||
if not _io_path(command_path).is_file() or not _io_path(status_path).is_file():
|
||||
return None
|
||||
commands = _read_commands(command_path)
|
||||
if {command_task_id for command_task_id, *_ in commands} != {task_id}:
|
||||
@@ -220,7 +221,7 @@ class QuickRunPreparer:
|
||||
if discovery_matches is not None:
|
||||
discovery_matches[project_key] = {_MISSING_PROJECT_SHEET: frozenset()}
|
||||
except TaskCenterError:
|
||||
if project.path.exists():
|
||||
if _io_path(project.path).exists():
|
||||
raise
|
||||
project_key = _path_key(project.path)
|
||||
targets = (_MISSING_PROJECT_SHEET,)
|
||||
@@ -277,7 +278,7 @@ class QuickRunPreparer:
|
||||
project_keys_by_index = tuple(_path_key(command[2]) for command in commands)
|
||||
status_stat = _required_stat(handoff.status_path, "quick_run_status", "cannot inspect quick-run status")
|
||||
manifest_path = handoff.command_path.with_name("sheets.tsv")
|
||||
command_lines = handoff.command_path.read_text(encoding='utf-8-sig').splitlines()
|
||||
command_lines = _io_path(handoff.command_path).read_text(encoding='utf-8-sig').splitlines()
|
||||
job_column = command_lines[0].split('\t').index('job_id')
|
||||
target_command_rows = ({line.split('\t')[job_column]: line for line in command_lines[1:] if line}
|
||||
if {'target_scope', 'target_cell'} & set(command_lines[0].split('\t')) else {})
|
||||
@@ -334,7 +335,7 @@ class QuickRunPreparer:
|
||||
) -> QuickRunStepResult:
|
||||
"""Prepare or resume a queue, then execute one replacement rule end to end."""
|
||||
uncertain_path = self._data_root / "quick-run" / "uncertain.flag"
|
||||
if uncertain_path.exists():
|
||||
if _io_path(uncertain_path).exists():
|
||||
raise TaskCenterError("quick_run_uncertain", "a previous quick-run worker timed out or returned an uncorrelated status; reconcile the project before retrying")
|
||||
if self._process_runner is _run_process and not batch_mode:
|
||||
from .single_step import prepare_private_single_step
|
||||
@@ -342,7 +343,7 @@ class QuickRunPreparer:
|
||||
**({'runtime': runtime} if runtime is not None else {}))
|
||||
batch_mode = True
|
||||
current_root = self._data_root / "quick-run"
|
||||
if batch_session is None and self.evidence_path(task_id) == current_root and (current_root / "status.tsv").is_file():
|
||||
if batch_session is None and self.evidence_path(task_id) == current_root and (_io_path(current_root / "status.tsv")).is_file():
|
||||
recorded = _read_status_details(current_root / "status.tsv")
|
||||
latest_states = {job_id: state for job_id, state, _, _ in recorded}
|
||||
if any(state == BATCH_SAVED_UNVERIFIED_STATE for state in latest_states.values()):
|
||||
@@ -406,9 +407,9 @@ class QuickRunPreparer:
|
||||
if batch_mode:
|
||||
if batch_session is not None:
|
||||
current_command_stamp = _write_current_command(batch_session, selected_command)
|
||||
batch_request.write_text(selected_command[1], encoding="ascii")
|
||||
_io_path(batch_request).write_text(selected_command[1], encoding="ascii")
|
||||
else:
|
||||
batch_request.unlink(missing_ok=True)
|
||||
_io_path(batch_request).unlink(missing_ok=True)
|
||||
from .worker_monitor import WorkerMonitorError
|
||||
|
||||
self.invalidate_discovery(Path(selected_command[2]))
|
||||
@@ -540,7 +541,7 @@ class QuickRunPreparer:
|
||||
raise TaskCenterError("quick_run_execution", f"unexpected quick-run state: {state}")
|
||||
if batch_mode and state == "saved_ok":
|
||||
try:
|
||||
handoff.command_path.with_name("current-state.tsv").unlink(missing_ok=True)
|
||||
_io_path(handoff.command_path.with_name("current-state.tsv")).unlink(missing_ok=True)
|
||||
except OSError as error:
|
||||
_append_log(
|
||||
handoff.log_path,
|
||||
@@ -612,7 +613,7 @@ class QuickRunPreparer:
|
||||
root = self._data_root / "quick-run"
|
||||
command_path = root / "command.tsv"
|
||||
status_path = root / "status.tsv"
|
||||
if not command_path.is_file() or not status_path.is_file():
|
||||
if not _io_path(command_path).is_file() or not _io_path(status_path).is_file():
|
||||
return None
|
||||
try:
|
||||
commands = _read_commands(command_path)
|
||||
@@ -634,7 +635,7 @@ class QuickRunPreparer:
|
||||
_archive_obsolete_handoff(self._data_root, root)
|
||||
return None
|
||||
if _queue_value_mode(command_path) != value_mode:
|
||||
if statuses or any(root.glob("snapshot-*")) or _queue_has_mutation_evidence(root):
|
||||
if statuses or any(_io_path(root).glob("snapshot-*")) or _queue_has_mutation_evidence(root):
|
||||
raise TaskCenterError("quick_run_pending", "recorded Value policy differs; reconcile the old queue before creating a new task")
|
||||
_archive_obsolete_handoff(self._data_root, root)
|
||||
return None
|
||||
@@ -772,7 +773,7 @@ class QuickRunPreparer:
|
||||
return self._data_root
|
||||
|
||||
def _required_script_source(self) -> Path:
|
||||
if not self._script_source.is_file():
|
||||
if not _io_path(self._script_source).is_file():
|
||||
raise TaskCenterError("quick_run_script_missing", "QuickPartReplace.vbs runtime resource is missing")
|
||||
return self._script_source
|
||||
|
||||
@@ -1074,10 +1075,10 @@ def _assert_no_pending_status(status_path: Path, log_path: Path) -> None:
|
||||
|
||||
|
||||
def _read_status(status_path: Path) -> tuple[tuple[str, str], ...]:
|
||||
if not status_path.exists():
|
||||
if not _io_path(status_path).exists():
|
||||
return ()
|
||||
try:
|
||||
lines = status_path.read_text(encoding="utf-8-sig").splitlines()
|
||||
lines = _io_path(status_path).read_text(encoding="utf-8-sig").splitlines()
|
||||
except (OSError, UnicodeError) as error:
|
||||
raise TaskCenterError("quick_run_status", "cannot read quick-run status") from error
|
||||
if not lines or lines[0].split("\t") != ["job_id", "state", "code", "message"]:
|
||||
@@ -1098,17 +1099,17 @@ def _read_status(status_path: Path) -> tuple[tuple[str, str], ...]:
|
||||
|
||||
|
||||
def _append_log(log_path: Path, entry: str, *, check_rotation: bool = True) -> None:
|
||||
if not log_path.parent.is_dir():
|
||||
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
if not _io_path(log_path.parent).is_dir():
|
||||
_io_path(log_path.parent).mkdir(parents=True, exist_ok=True)
|
||||
if check_rotation:
|
||||
rotate_text_log(log_path)
|
||||
with log_path.open("a", encoding="utf-8", newline="\n") as handle:
|
||||
rotate_text_log(_io_path(log_path))
|
||||
with _io_path(log_path).open("a", encoding="utf-8", newline="\n") as handle:
|
||||
handle.write(entry + "\n")
|
||||
|
||||
|
||||
def _write_uncertain_flag(path: Path, reason: str = "manual_reconciliation_required") -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("w", encoding="utf-8", newline="\n") as handle:
|
||||
_io_path(path.parent).mkdir(parents=True, exist_ok=True)
|
||||
with _io_path(path).open("w", encoding="utf-8", newline="\n") as handle:
|
||||
handle.write(f"reason={_log_text(reason)}; manual reconciliation required\n")
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
@@ -1120,14 +1121,14 @@ def _states_are_complete(statuses: tuple[tuple[str, str, str, str], ...]) -> boo
|
||||
|
||||
|
||||
def _queue_has_mutation_evidence(root: Path) -> bool:
|
||||
if (root / "uncertain.flag").exists():
|
||||
if (_io_path(root / "uncertain.flag")).exists():
|
||||
return True
|
||||
for report_path in root.glob("snapshot-*.tsv.native.json"):
|
||||
for report_path in _io_path(root).glob("snapshot-*.tsv.native.json"):
|
||||
rollback_marker = Path(str(report_path)[:-len(".json")] + ".rolled-back")
|
||||
if rollback_marker.is_file():
|
||||
if _io_path(rollback_marker).is_file():
|
||||
continue
|
||||
try:
|
||||
report = json.loads(report_path.read_text(encoding="utf-8"))
|
||||
report = json.loads(_io_path(report_path).read_text(encoding="utf-8"))
|
||||
except (OSError, UnicodeError, json.JSONDecodeError):
|
||||
return True
|
||||
if report.get("ok") is True or report.get("stage") in {"native_replace_started", "native_replace_applied", "pin_promotion_started", "native_replace_verified"}:
|
||||
@@ -1140,12 +1141,12 @@ def _rollback_marker_verified(root: Path, job_id: str) -> bool:
|
||||
snapshot = root / f"snapshot-{job_id}.tsv"
|
||||
report_path = root / f"snapshot-{job_id}.tsv.native.json"
|
||||
try:
|
||||
report = json.loads(report_path.read_text(encoding="utf-8"))
|
||||
report = json.loads(_io_path(report_path).read_text(encoding="utf-8"))
|
||||
return (
|
||||
snapshot.is_file()
|
||||
_io_path(snapshot).is_file()
|
||||
and isinstance(report, dict)
|
||||
and report.get("stage") in {"native_replace_started", "native_replace_applied", "pin_promotion_started", "native_replace_verified"}
|
||||
and marker.read_text(encoding="utf-8-sig").strip() == "native replacement rollback verified"
|
||||
and _io_path(marker).read_text(encoding="utf-8-sig").strip() == "native replacement rollback verified"
|
||||
)
|
||||
except (OSError, UnicodeError, json.JSONDecodeError):
|
||||
return False
|
||||
@@ -1156,7 +1157,7 @@ def _recover_pre_mutation_hard_failures(
|
||||
commands: tuple[tuple[str, str, str, str, str], ...],
|
||||
statuses: tuple[tuple[str, str, str, str], ...],
|
||||
) -> tuple[tuple[str, str, str, str], ...]:
|
||||
if (root / "uncertain.flag").exists():
|
||||
if (_io_path(root / "uncertain.flag")).exists():
|
||||
return statuses
|
||||
command_ids = {job_id for _, job_id, *_ in commands}
|
||||
final_states = {job_id: state for job_id, state, _, _ in statuses}
|
||||
@@ -1176,7 +1177,7 @@ def _recover_pre_mutation_hard_failures(
|
||||
try:
|
||||
safe_job_id = _safe_run_token(job_id)
|
||||
report_path = root / f"snapshot-{safe_job_id}.tsv.native.json"
|
||||
report = json.loads(report_path.read_text(encoding="utf-8"))
|
||||
report = json.loads(_io_path(report_path).read_text(encoding="utf-8"))
|
||||
except (TaskCenterError, OSError, UnicodeError, json.JSONDecodeError):
|
||||
continue
|
||||
if (
|
||||
@@ -1204,26 +1205,26 @@ def _write_status_details(path: Path, statuses: tuple[tuple[str, str, str, str],
|
||||
text = "job_id\tstate\tcode\tmessage\r\n"
|
||||
text += "".join("\t".join(_encode_vbs_field(field) for field in entry) + "\r\n" for entry in statuses)
|
||||
try:
|
||||
temporary.write_text(text, encoding="utf-8", newline="")
|
||||
os.replace(temporary, path)
|
||||
_io_path(temporary).write_text(text, encoding="utf-8", newline="")
|
||||
os.replace(_io_path(temporary), _io_path(path))
|
||||
except OSError as error:
|
||||
temporary.unlink(missing_ok=True)
|
||||
_io_path(temporary).unlink(missing_ok=True)
|
||||
raise TaskCenterError("quick_run_status", "cannot recover pre-mutation quick-run status") from error
|
||||
|
||||
|
||||
def _archive_obsolete_handoff(data_root: Path, root: Path) -> Path:
|
||||
timestamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S-%f")
|
||||
destination = data_root / "quick-run-history" / f"obsolete-{timestamp}-{uuid4().hex[:8]}"
|
||||
destination.mkdir(parents=True, exist_ok=False)
|
||||
for entry in tuple(root.iterdir()):
|
||||
shutil.move(str(entry), destination / entry.name)
|
||||
_io_path(destination).mkdir(parents=True, exist_ok=False)
|
||||
for entry in tuple(_io_path(root).iterdir()):
|
||||
shutil.move(str(_io_path(entry)), _io_path(destination / entry.name))
|
||||
_append_log(destination / "quick-part-replace.log", "quick_run_obsolete_queue_archived")
|
||||
return destination
|
||||
|
||||
|
||||
def _read_commands(path: Path) -> tuple[tuple[str, str, str, str, str], ...]:
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8-sig").splitlines()
|
||||
lines = _io_path(path).read_text(encoding="utf-8-sig").splitlines()
|
||||
except (OSError, UnicodeError) as error:
|
||||
raise TaskCenterError("quick_run_command", "cannot read quick-run command") from error
|
||||
if not lines:
|
||||
@@ -1248,7 +1249,7 @@ def _read_commands(path: Path) -> tuple[tuple[str, str, str, str, str], ...]:
|
||||
|
||||
def _read_command_sheet_targets(path: Path) -> dict[str, tuple[str, str]] | None:
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8-sig").splitlines()
|
||||
lines = _io_path(path).read_text(encoding="utf-8-sig").splitlines()
|
||||
header = lines[0].split("\t")
|
||||
names = ("job_id", "schematic_name", "sheet_number")
|
||||
present = tuple(name in header for name in names[1:])
|
||||
@@ -1278,7 +1279,7 @@ def _read_sheet_manifest(
|
||||
expected_projects: tuple[ProjectRecord, ...] | None = None,
|
||||
) -> dict[str, tuple[tuple[str, str], ...]]:
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8-sig").splitlines()
|
||||
lines = _io_path(path).read_text(encoding="utf-8-sig").splitlines()
|
||||
if not lines or lines[0].split("\t") != ["project_path", "schematic_name", "sheet_number"]:
|
||||
raise ValueError("invalid manifest header")
|
||||
result: dict[str, list[tuple[str, str]]] = {}
|
||||
@@ -1319,7 +1320,7 @@ def _read_discovery_inventory(
|
||||
sheets: tuple[tuple[str, str], ...],
|
||||
) -> dict[tuple[str, str], frozenset[str]]:
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8-sig").splitlines()
|
||||
lines = _io_path(path).read_text(encoding="utf-8-sig").splitlines()
|
||||
if not lines or lines[0].split("\t") != ["project_path", "schematic_name", "sheet_number", "old_part"]:
|
||||
raise ValueError("invalid inventory header")
|
||||
expected_project = _path_key(project_path)
|
||||
@@ -1389,7 +1390,7 @@ def _initial_discovery_status(
|
||||
return rows[0] + "\r\n"
|
||||
commands = _read_commands(command_path)
|
||||
sheets = _read_command_sheet_targets(command_path)
|
||||
targeted = "target_scope" in command_path.read_text(encoding="utf-8-sig").splitlines()[0].split("\t")
|
||||
targeted = "target_scope" in _io_path(command_path).read_text(encoding="utf-8-sig").splitlines()[0].split("\t")
|
||||
if sheets is None:
|
||||
raise TaskCenterError("quick_run_sheet_discovery", "discovery inventory requires page-addressed commands")
|
||||
possible_by_sheet: dict[tuple[str, str, str], set[str]] = {}
|
||||
@@ -1435,12 +1436,12 @@ def _queue_matches_inputs(
|
||||
expected_cells = {str(getattr(rule, "target_cell", "")) for rule in rules}
|
||||
if actual_cells != expected_cells:
|
||||
return False
|
||||
if "target_scope" in command_path.read_text(encoding="utf-8-sig").splitlines()[0].split("\t"):
|
||||
if "target_scope" in _io_path(command_path).read_text(encoding="utf-8-sig").splitlines()[0].split("\t"):
|
||||
sheets = _read_command_sheet_targets(command_path)
|
||||
manifest = _read_sheet_manifest(command_path.with_name("sheets.tsv"), projects) if sheets is not None else None
|
||||
expected = _command_text("expected", "expected", projects, rules, base_mode,
|
||||
sheet_targets=manifest).splitlines()
|
||||
actual = command_path.read_text(encoding="utf-8-sig").splitlines()
|
||||
actual = _io_path(command_path).read_text(encoding="utf-8-sig").splitlines()
|
||||
return (expected[0] == actual[0] and len(expected) == len(actual) and
|
||||
all(expected_row.split("\t")[2:] == actual_row.split("\t")[2:]
|
||||
for expected_row, actual_row in zip(expected[1:], actual[1:])))
|
||||
@@ -1476,10 +1477,10 @@ def _queue_matches_inputs(
|
||||
|
||||
|
||||
def _read_status_details(path: Path) -> tuple[tuple[str, str, str, str], ...]:
|
||||
if not path.exists():
|
||||
if not _io_path(path).exists():
|
||||
return ()
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8-sig").splitlines()
|
||||
lines = _io_path(path).read_text(encoding="utf-8-sig").splitlines()
|
||||
except (OSError, UnicodeError) as error:
|
||||
raise TaskCenterError("quick_run_status", "cannot read quick-run status") from error
|
||||
if not lines or lines[0].split("\t") != ["job_id", "state", "code", "message"]:
|
||||
@@ -1502,7 +1503,7 @@ def _open_status_for_read(path: Path):
|
||||
deadline = time.monotonic() + 1.0
|
||||
while True:
|
||||
try:
|
||||
return path.open('rb')
|
||||
return _io_path(path).open('rb')
|
||||
except PermissionError:
|
||||
if time.monotonic() >= deadline:
|
||||
raise
|
||||
@@ -1544,7 +1545,7 @@ def _read_appended_status_details(
|
||||
|
||||
def _required_stat(path: Path, code: str, message: str) -> os.stat_result:
|
||||
try:
|
||||
return path.stat()
|
||||
return _io_path(path).stat()
|
||||
except OSError as error:
|
||||
raise TaskCenterError(code, message) from error
|
||||
|
||||
@@ -1593,7 +1594,7 @@ def _write_current_command(
|
||||
state = session.latest.get(job_id)
|
||||
state_path = path.with_name("current-state.tsv")
|
||||
if state is None:
|
||||
state_path.unlink(missing_ok=True)
|
||||
_io_path(state_path).unlink(missing_ok=True)
|
||||
else:
|
||||
_write_atomic_queue_text(state_path,
|
||||
"job_id\tstate\tcode\tmessage\r\n" + "\t".join(
|
||||
@@ -1637,7 +1638,7 @@ def _write_current_command(
|
||||
)
|
||||
if session.next_index == 0 or (previous_state is not None and previous_state[0] != "no_match"):
|
||||
try:
|
||||
state_path.unlink(missing_ok=True)
|
||||
_io_path(state_path).unlink(missing_ok=True)
|
||||
except OSError as error:
|
||||
raise TaskCenterError("quick_run_status", "cannot clear the previous batch state") from error
|
||||
else:
|
||||
@@ -1657,21 +1658,21 @@ def _write_current_command(
|
||||
def _write_atomic_queue_text(path: Path, text: str, code: str, message: str) -> None:
|
||||
temporary = path.with_name(f".{path.name}.{uuid4().hex}.tmp")
|
||||
try:
|
||||
with temporary.open("w", encoding="utf-8", newline="") as handle:
|
||||
with _io_path(temporary).open("w", encoding="utf-8", newline="") as handle:
|
||||
handle.write(text)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
deadline = time.monotonic() + 1.0
|
||||
while True:
|
||||
try:
|
||||
os.replace(temporary, path)
|
||||
os.replace(_io_path(temporary), _io_path(path))
|
||||
break
|
||||
except PermissionError:
|
||||
if time.monotonic() >= deadline:
|
||||
raise
|
||||
time.sleep(.02)
|
||||
except OSError as error:
|
||||
temporary.unlink(missing_ok=True)
|
||||
_io_path(temporary).unlink(missing_ok=True)
|
||||
raise TaskCenterError(code, message) from error
|
||||
|
||||
|
||||
@@ -1752,7 +1753,7 @@ def _resolve_cscript(windows_root: Path | None = None, is_64_bit: bool | None =
|
||||
if is_64_bit is None:
|
||||
is_64_bit = sys.maxsize > 2**32 or bool(os.environ.get("PROCESSOR_ARCHITEW6432"))
|
||||
candidate = windows / ("SysWOW64" if is_64_bit else "System32") / "cscript.exe"
|
||||
if candidate.is_file():
|
||||
if _io_path(candidate).is_file():
|
||||
return candidate
|
||||
raise TaskCenterError("quick_run_cscript", "Windows Script Host cscript.exe was not found")
|
||||
|
||||
@@ -1818,7 +1819,7 @@ def _value_mode_matches_option(recorded_mode: str, enabled: bool) -> bool:
|
||||
|
||||
def _queue_value_mode(path: Path) -> str:
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8-sig").splitlines()
|
||||
lines = _io_path(path).read_text(encoding="utf-8-sig").splitlines()
|
||||
header = lines[0].split("\t")
|
||||
if "value_mode" not in header:
|
||||
return "legacy_part_label"
|
||||
@@ -1836,7 +1837,7 @@ def _queue_value_mode(path: Path) -> str:
|
||||
|
||||
def _read_command_cells(path: Path) -> dict[str, str]:
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8-sig").splitlines()
|
||||
lines = _io_path(path).read_text(encoding="utf-8-sig").splitlines()
|
||||
header = lines[0].split("\t")
|
||||
if "target_cell" not in header:
|
||||
return {}
|
||||
@@ -1861,7 +1862,7 @@ def _read_command_cells(path: Path) -> dict[str, str]:
|
||||
|
||||
def _read_command_options(path: Path) -> dict[str, tuple[str, str, bool, str]]:
|
||||
try:
|
||||
lines = path.read_text(encoding="utf-8-sig").splitlines()
|
||||
lines = _io_path(path).read_text(encoding="utf-8-sig").splitlines()
|
||||
if not lines:
|
||||
raise ValueError("empty queue")
|
||||
header = lines[0].split("\t")
|
||||
@@ -1977,3 +1978,9 @@ def _validate_target_scope(projects, rules, sheet_targets) -> None:
|
||||
for target in rule.targets:
|
||||
if (_path_key(target.project_path), target.schematic_name, target.sheet_number) not in available:
|
||||
raise TaskCenterError("target_identity", "selected target sheet is missing from discovery")
|
||||
|
||||
|
||||
def _io_path(path):
|
||||
# Snapshot parsers also accept an immutable cached reader. Preserve that
|
||||
# protocol; converting it to Path would bypass the coherent read snapshot.
|
||||
return internal_path(path) if isinstance(path, (str, os.PathLike)) else path
|
||||
@@ -4,6 +4,7 @@ from dataclasses import dataclass, replace
|
||||
import json
|
||||
from pathlib import Path
|
||||
from .errors import TaskCenterError
|
||||
from .path_access import canonical_path, internal_path
|
||||
from .quick_run_protocol import QuickRunProgress
|
||||
|
||||
|
||||
@@ -19,9 +20,9 @@ def failed_native_result_uncertain(root, commands, statuses, *, files=None):
|
||||
continue
|
||||
path = root / f'snapshot-{job}.tsv.native.json'
|
||||
try:
|
||||
report = json.loads(files.read(path) if files is not None else path.read_text(encoding='utf-8-sig'))
|
||||
report = json.loads(files.read(path) if files is not None else internal_path(path).read_text(encoding='utf-8-sig'))
|
||||
except (OSError, UnicodeError, json.JSONDecodeError):
|
||||
if job in awaiting or path.exists():
|
||||
if job in awaiting or internal_path(path).exists():
|
||||
return True
|
||||
continue
|
||||
if not isinstance(report, dict):
|
||||
@@ -56,7 +57,7 @@ class _ReadFiles:
|
||||
def stamp(path):
|
||||
self.checkpoint()
|
||||
try:
|
||||
stat = path.stat()
|
||||
stat = internal_path(path).stat()
|
||||
return (stat.st_dev, stat.st_ino, stat.st_size, stat.st_mtime_ns, stat.st_ctime_ns)
|
||||
except OSError:
|
||||
return None
|
||||
@@ -64,7 +65,7 @@ class _ReadFiles:
|
||||
try:
|
||||
# Include sidecar evidence used by hard-failure recovery as well as
|
||||
# command/status and flags; creation/removal changes the root stamp.
|
||||
entries = tuple(root.iterdir())
|
||||
entries = tuple(internal_path(root).iterdir())
|
||||
except OSError:
|
||||
entries = ()
|
||||
result = {path.name: stamp(path) for path in entries}
|
||||
@@ -85,7 +86,7 @@ class _ReadFiles:
|
||||
self.checkpoint()
|
||||
if path not in self.texts:
|
||||
chunks = []
|
||||
with path.open('r', encoding='utf-8-sig') as source:
|
||||
with internal_path(path).open('r', encoding='utf-8-sig') as source:
|
||||
while True:
|
||||
self.checkpoint()
|
||||
chunk = source.read(65536)
|
||||
@@ -110,7 +111,7 @@ class _CachedPath:
|
||||
|
||||
def exists(self):
|
||||
self.files.checkpoint()
|
||||
return self.path.exists()
|
||||
return internal_path(self.path).exists()
|
||||
|
||||
|
||||
def read_snapshot(preparer, task_id, projects, rules, *, copy_part_number_to_value=False,
|
||||
@@ -148,12 +149,13 @@ def _read_snapshot_once(preparer, task_id, projects, rules, *, copy_part_number_
|
||||
def candidates():
|
||||
yield current
|
||||
files.checkpoint()
|
||||
yield from sorted((path for path in history.glob('obsolete-*') if path.is_dir()), reverse=True)
|
||||
yield from sorted((canonical_path(path) for path in internal_path(history).glob('obsolete-*')
|
||||
if internal_path(path).is_dir()), reverse=True)
|
||||
for candidate in candidates():
|
||||
files.checkpoint()
|
||||
files.track_root(candidate)
|
||||
command = candidate / 'command.tsv'
|
||||
if not command.is_file() or not (candidate / 'status.tsv').is_file():
|
||||
if not internal_path(command).is_file() or not (internal_path(candidate / 'status.tsv')).is_file():
|
||||
continue
|
||||
try:
|
||||
parsed = api._read_commands(files.path(command))
|
||||
@@ -168,7 +170,7 @@ def _read_snapshot_once(preparer, task_id, projects, rules, *, copy_part_number_
|
||||
if root is None:
|
||||
files.checkpoint()
|
||||
return QuickRunSnapshot(progress, False, None, False, 'selection')
|
||||
interrupted = root == current and (root / 'uncertain.flag').is_file()
|
||||
interrupted = root == current and (internal_path(root / 'uncertain.flag')).is_file()
|
||||
phase = 'selection'
|
||||
try:
|
||||
details = api._read_status_details(files.path(root / 'status.tsv'))
|
||||
|
||||
@@ -56,10 +56,13 @@ class TrialRunner:
|
||||
logging_context: LoggingContext,
|
||||
*,
|
||||
probe_source_resolver: Callable[[TaskRecord, ProjectRecord], Path] | None = None,
|
||||
recover_on_startup: bool = True,
|
||||
) -> None:
|
||||
self._store = store
|
||||
self._logging = logging_context
|
||||
self._probe_source_resolver = probe_source_resolver or self._default_probe_source
|
||||
if not recover_on_startup:
|
||||
return # Per-batch processes must not replay application-start recovery.
|
||||
migration_recoveries = self._store.pending_migration_recoveries()
|
||||
for recovery in migration_recoveries:
|
||||
self._event_transition(
|
||||
|
||||
@@ -8,10 +8,11 @@ import re
|
||||
import stat
|
||||
|
||||
from .errors import TaskCenterError
|
||||
from .path_access import canonical_path, internal_path, strip_extended_prefix
|
||||
|
||||
|
||||
def _path_identity(path: Path) -> str:
|
||||
normalized = os.path.normpath(str(path))
|
||||
normalized = os.path.normpath(strip_extended_prefix(path))
|
||||
return os.path.normcase(normalized) if os.name == "nt" else normalized
|
||||
|
||||
|
||||
@@ -22,7 +23,7 @@ _ICDB_DIRECTORY = re.compile(rb'^\s*KEY\s+iCDBDir\s+"[^"]+"\s*$', re.IGNORECASE
|
||||
|
||||
|
||||
def _is_schematic_project(path: Path) -> bool:
|
||||
with path.open("rb") as project_file:
|
||||
with internal_path(path).open("rb") as project_file:
|
||||
contents = project_file.read(_PROJECT_HEADER_LIMIT)
|
||||
return bool(_DXD_FRONT_END.search(contents) and _ICDB_DIRECTORY.search(contents))
|
||||
|
||||
@@ -46,13 +47,13 @@ def _lexical_path(path: Path) -> str:
|
||||
|
||||
def scan_projects(root: Path, *, excluded_roots: tuple[Path, ...] = ()) -> tuple[Path, ...]:
|
||||
"""Discover DxDesigner schematic projects without selecting or modifying them."""
|
||||
root = Path(root)
|
||||
root = Path(strip_extended_prefix(root))
|
||||
try:
|
||||
if not root.exists() or not root.is_dir():
|
||||
if not internal_path(root).exists() or not internal_path(root).is_dir():
|
||||
raise TaskCenterError("discovery_root", f"discovery root is not a directory: {root}")
|
||||
root_details = root.lstat()
|
||||
root_details = internal_path(root).lstat()
|
||||
_reject_reparse(root_details, root)
|
||||
canonical_root = root.resolve()
|
||||
canonical_root = canonical_path(root)
|
||||
excluded = tuple(_lexical_path(Path(path)) for path in excluded_roots)
|
||||
root_key = _lexical_path(root)
|
||||
if any(root_key == blocked or root_key.startswith(blocked + os.sep) for blocked in excluded):
|
||||
@@ -66,10 +67,10 @@ def scan_projects(root: Path, *, excluded_roots: tuple[Path, ...] = ()) -> tuple
|
||||
visited: set[tuple[int, int]] = set()
|
||||
|
||||
def read_entries(directory: Path) -> tuple[tuple[Path, os.stat_result], ...]:
|
||||
with os.scandir(directory) as entries:
|
||||
with os.scandir(internal_path(directory)) as entries:
|
||||
result: list[tuple[Path, os.stat_result]] = []
|
||||
for entry in sorted(entries, key=lambda item: item.name.casefold()):
|
||||
candidate = Path(entry.path)
|
||||
candidate = Path(strip_extended_prefix(entry.path))
|
||||
if is_excluded(candidate):
|
||||
continue
|
||||
entry_details = entry.stat(follow_symlinks=False)
|
||||
@@ -84,7 +85,7 @@ def scan_projects(root: Path, *, excluded_roots: tuple[Path, ...] = ()) -> tuple
|
||||
and candidate.suffix.casefold() == ".prj"
|
||||
and _is_schematic_project(candidate)
|
||||
):
|
||||
canonical = candidate.resolve()
|
||||
canonical = canonical_path(candidate)
|
||||
candidates.setdefault(_path_identity(canonical), canonical)
|
||||
|
||||
if candidates:
|
||||
@@ -94,10 +95,10 @@ def scan_projects(root: Path, *, excluded_roots: tuple[Path, ...] = ()) -> tuple
|
||||
directory: Path,
|
||||
directory_entries: tuple[tuple[Path, os.stat_result], ...] | None = None,
|
||||
) -> None:
|
||||
canonical_directory = directory.resolve()
|
||||
canonical_directory = canonical_path(directory)
|
||||
if not _contained(canonical_root, canonical_directory):
|
||||
raise TaskCenterError("discovery_containment", f"discovery path escapes its root: {directory}")
|
||||
details = directory.lstat()
|
||||
details = internal_path(directory).lstat()
|
||||
_reject_reparse(details, directory)
|
||||
identity = details.st_dev, details.st_ino
|
||||
if identity in visited:
|
||||
@@ -112,7 +113,7 @@ def scan_projects(root: Path, *, excluded_roots: tuple[Path, ...] = ()) -> tuple
|
||||
and candidate.suffix.casefold() == ".prj"
|
||||
and _is_schematic_project(candidate)
|
||||
):
|
||||
canonical = candidate.resolve()
|
||||
canonical = canonical_path(candidate)
|
||||
if not _contained(canonical_root, canonical):
|
||||
raise TaskCenterError("discovery_containment", f"project path escapes its root: {candidate}")
|
||||
candidates.setdefault(_path_identity(canonical), canonical)
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from contextlib import contextmanager
|
||||
from contextlib import contextmanager, ExitStack
|
||||
from datetime import datetime
|
||||
import os
|
||||
from pathlib import Path
|
||||
@@ -35,12 +35,26 @@ _QUICK_RUN_THREAD_LOCK = threading.Lock()
|
||||
|
||||
@contextmanager
|
||||
def _quick_run_lease(root: Path):
|
||||
from .batch_supervisor import mutation_lease
|
||||
try:
|
||||
with mutation_lease(root):
|
||||
with _quick_run_lease_owned(root):
|
||||
yield
|
||||
except TaskCenterError as error:
|
||||
if error.code == 'supervisor_busy':
|
||||
raise TaskCenterError('quick_run_busy', '另一批操作正在执行,请先安全暂停。') from error
|
||||
raise
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _quick_run_lease_owned(root: Path):
|
||||
if not _QUICK_RUN_THREAD_LOCK.acquire(blocking=False):
|
||||
raise TaskCenterError("quick_run_busy", "another quick-run operation is already executing")
|
||||
handle = None
|
||||
locked = False
|
||||
try:
|
||||
lock_path = root / ".quick-run.execution.lock"
|
||||
from .path_access import internal_path
|
||||
lock_path = internal_path(root / ".quick-run.execution.lock")
|
||||
lock_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
handle = lock_path.open("a+b")
|
||||
if handle.seek(0, os.SEEK_END) == 0:
|
||||
@@ -95,12 +109,29 @@ class TaskCenterService:
|
||||
probe_source_resolver: Callable[[TaskRecord, ProjectRecord], Path] | None = None,
|
||||
quick_run_preparer: QuickRunPreparer | None = None,
|
||||
runner: TrialRunner | None = None,
|
||||
supervised: bool = False,
|
||||
recover_on_startup: bool = True,
|
||||
) -> None:
|
||||
self._store = store
|
||||
self._logging = logging_context
|
||||
self._path_opener = path_opener
|
||||
self._clock = clock
|
||||
self._runner = runner or TrialRunner(store, logging_context, probe_source_resolver=probe_source_resolver)
|
||||
self._supervised = supervised
|
||||
with ExitStack() as startup:
|
||||
if runner is None and recover_on_startup:
|
||||
from .batch_supervisor import mutation_lease, pending_supervisors
|
||||
try:
|
||||
startup.enter_context(mutation_lease(logging_context.root))
|
||||
except TaskCenterError as error:
|
||||
if error.code != 'supervisor_busy':
|
||||
raise
|
||||
recover_on_startup = False
|
||||
if recover_on_startup and pending_supervisors(logging_context.root):
|
||||
recover_on_startup = False
|
||||
# Keep admission for the whole recovery, not just an idle probe:
|
||||
# another window must not start a batch while startup rewrites state.
|
||||
self._runner = runner or TrialRunner(store, logging_context, probe_source_resolver=probe_source_resolver,
|
||||
**({'recover_on_startup': False} if not recover_on_startup else {}))
|
||||
self._quick_run_preparer = quick_run_preparer or QuickRunPreparer(logging_context.root)
|
||||
self._batch_feedback = None
|
||||
self._project_facades: dict[str, TaskCenterService] = {}
|
||||
@@ -143,6 +174,9 @@ class TaskCenterService:
|
||||
lambda: export_template(self._store, task_id, path))
|
||||
|
||||
def preview_bom_batch(self, task_id: str, path: Path, *, stop_event=None, on_progress=None, runtime=None):
|
||||
if runtime is None and self._supervised:
|
||||
return self._supervise('preview_bom_batch', {'task_id': task_id, 'path': path},
|
||||
stop_event=stop_event, on_progress=on_progress)
|
||||
from .bom_batch import preview_batch
|
||||
return self._idle_quick_run_operation('preview_bom_batch', task_id,
|
||||
lambda: preview_batch(self, task_id, path, **control_options(stop_event, on_progress),
|
||||
@@ -187,6 +221,7 @@ class TaskCenterService:
|
||||
# the database. A facade shares the initialized runner instead.
|
||||
child = TaskCenterService(self._store, self._logging, path_opener=self._path_opener,
|
||||
clock=self._clock, quick_run_preparer=child_preparer, runner=self._runner)
|
||||
child._supervised = self._supervised
|
||||
self._project_facades[task_id] = child
|
||||
return self._project_facades[task_id]
|
||||
|
||||
@@ -361,6 +396,8 @@ class TaskCenterService:
|
||||
|
||||
def target_cell_choices(self, task_id: str, new_part_number: str) -> tuple[str, ...]:
|
||||
"""Read candidates for the captured batch; reject a changed project scope."""
|
||||
if self._supervised:
|
||||
return self._supervise('target_cell_choices', {'task_id': task_id, 'new_part_number': new_part_number})
|
||||
from .cell_catalog import read_project_cell_choices
|
||||
part_number = new_part_number.strip()
|
||||
projects = self.target_cell_projects(task_id)
|
||||
@@ -390,6 +427,9 @@ class TaskCenterService:
|
||||
|
||||
def collect_target_inventory(self, task_id: str, old_part_numbers: tuple[str, ...] | None = None,
|
||||
*, stop_event=None, on_progress=None, runtime=None):
|
||||
if runtime is None and self._supervised:
|
||||
return self._supervise('collect_target_inventory', {'task_id': task_id, 'old_part_numbers': old_part_numbers},
|
||||
stop_event=stop_event, on_progress=on_progress)
|
||||
if runtime is None:
|
||||
return self._owned_cad_operation('collect_inventory', lambda owned:
|
||||
self.collect_target_inventory(task_id, old_part_numbers,
|
||||
@@ -562,6 +602,8 @@ class TaskCenterService:
|
||||
return self._operation("expected_probe_path", task_id, expected_path)
|
||||
|
||||
def prepare_quick_run(self, task_id: str, *, runtime=None) -> QuickRunResult:
|
||||
if runtime is None and self._supervised:
|
||||
return self._supervise('prepare_quick_run', {'task_id': task_id})
|
||||
"""Create the legacy VBS handoff without starting a task-center trial."""
|
||||
if runtime is None:
|
||||
return self._owned_cad_operation('prepare_quick_run', lambda owned:
|
||||
@@ -636,6 +678,8 @@ class TaskCenterService:
|
||||
|
||||
def execute_quick_run(self, task_id: str, *, expected_item: tuple[str, str, str, str] | None = None, runtime=None) -> QuickRunStepResult:
|
||||
"""Execute one replacement rule through restoration and verification."""
|
||||
if runtime is None and self._supervised:
|
||||
return self._supervise('execute_quick_run', {'task_id': task_id, 'expected_item': expected_item})
|
||||
if runtime is None:
|
||||
return self._owned_cad_operation('quick_run_step', lambda owned:
|
||||
self.execute_quick_run(task_id, expected_item=expected_item, runtime=owned), task_id=task_id)
|
||||
@@ -679,6 +723,10 @@ class TaskCenterService:
|
||||
reuse_project_session: bool = True,
|
||||
runtime=None,
|
||||
):
|
||||
if runtime is None and self._supervised:
|
||||
return self._supervise('execute_quick_run_queue', {'task_id': task_id, 'expected_item': expected_item,
|
||||
'continue_after_backup_failure': continue_after_backup_failure,
|
||||
'reuse_project_session': reuse_project_session}, stop_event=stop_event)
|
||||
if runtime is None:
|
||||
from .batch_runtime import BatchCadRuntime
|
||||
with BatchCadRuntime(self._logging.root, 'quick_run_queue', stop_event) as owned_runtime:
|
||||
@@ -735,6 +783,8 @@ class TaskCenterService:
|
||||
|
||||
def prepare_failed_quick_run(self, task_id: str, *, runtime=None) -> QuickRunProgress:
|
||||
"""Reset only failed projects and discard the finished queue handoff."""
|
||||
if runtime is None and self._supervised:
|
||||
return self._supervise('prepare_failed_quick_run', {'task_id': task_id})
|
||||
if runtime is None:
|
||||
return self._owned_cad_operation('prepare_failed_quick_run', lambda owned:
|
||||
self.prepare_failed_quick_run(task_id, runtime=owned), task_id=task_id)
|
||||
@@ -816,10 +866,21 @@ class TaskCenterService:
|
||||
|
||||
def pending_batch_resources(self):
|
||||
from .batch_runtime import pending_resources
|
||||
return pending_resources(self._logging.root)
|
||||
from .batch_supervisor import pending_supervisors
|
||||
return (*pending_resources(self._logging.root), *pending_supervisors(self._logging.root))
|
||||
|
||||
def export_diagnostics(self, destination: Path) -> Path:
|
||||
from .diagnostics_bundle import export_diagnostic_bundle
|
||||
return self._operation('export_diagnostics', None,
|
||||
lambda: export_diagnostic_bundle(self._logging.root, destination))
|
||||
|
||||
def cleanup_batch_resources(self):
|
||||
"""Retry owned-resource release without opening or changing any project."""
|
||||
if self._supervised:
|
||||
result = self._supervise('cleanup_batch_resources', {})
|
||||
from .batch_supervisor import reconcile_supervisors
|
||||
reconcile_supervisors(self._logging.root)
|
||||
return self.pending_batch_resources()
|
||||
from .batch_runtime import cleanup_pending_resources
|
||||
from .task_queue import assert_task_queue_idle
|
||||
assert_task_queue_idle(self._logging.root)
|
||||
@@ -828,6 +889,9 @@ class TaskCenterService:
|
||||
lambda: cleanup_pending_resources(self._logging.root, logging_context=self._logging))
|
||||
|
||||
def inspect_bom_batch(self, task_id: str, path: Path, *, stop_event=None, on_progress=None, on_phase=None, runtime=None):
|
||||
if runtime is None and self._supervised:
|
||||
return self._supervise('inspect_bom_batch', {'task_id': task_id, 'path': path},
|
||||
stop_event=stop_event, on_progress=on_progress, on_phase=on_phase)
|
||||
from .bom_import_report import inspect_batch
|
||||
return self._idle_quick_run_operation('inspect_bom_batch', task_id,
|
||||
lambda: inspect_batch(self, task_id, path, on_phase=on_phase, runtime=runtime,
|
||||
@@ -861,6 +925,9 @@ class TaskCenterService:
|
||||
message = batch[2] if batch else '正在准备执行'
|
||||
value = read_feedback(task_id, position, total, message,
|
||||
child._quick_run_preparer.data_root, started_after)
|
||||
if getattr(self, '_supervisor_waiting', False):
|
||||
from dataclasses import replace
|
||||
value = replace(value, stage='awaiting_reconciliation', waiting_for_safe_completion=True)
|
||||
if read_control is not None:
|
||||
read_control.checkpoint()
|
||||
return value
|
||||
@@ -887,10 +954,12 @@ class TaskCenterService:
|
||||
return delegated.has_interrupted_quick_run(task_id)
|
||||
return self._quick_run_preparer.has_interrupted(task_id)
|
||||
|
||||
def recover_quick_run(self, task_id: str, *, confirmed: bool = False, runtime=None) -> QuickRunStepResult:
|
||||
def recover_quick_run(self, task_id: str, *, confirmed: bool = False, runtime=None, stop_event=None) -> QuickRunStepResult:
|
||||
if runtime is None and self._supervised:
|
||||
return self._supervise('recover_quick_run', {'task_id': task_id, 'confirmed': confirmed}, stop_event=stop_event)
|
||||
if runtime is None:
|
||||
return self._owned_cad_operation('recover_quick_run', lambda owned:
|
||||
self.recover_quick_run(task_id, confirmed=confirmed, runtime=owned), task_id=task_id)
|
||||
self.recover_quick_run(task_id, confirmed=confirmed, runtime=owned), stop_event, task_id=task_id)
|
||||
delegated = self._project_task_service(task_id)
|
||||
if delegated is not self:
|
||||
return delegated.recover_quick_run(task_id, confirmed=confirmed, runtime=runtime)
|
||||
@@ -1046,6 +1115,11 @@ class TaskCenterService:
|
||||
self._logging.logger.info("operation_complete=%s task_id=%s at=%s", name, task_id or "", completed_timestamp)
|
||||
return result
|
||||
|
||||
def _supervise(self, operation, arguments, **controls):
|
||||
from .batch_supervisor import run_supervised
|
||||
return self._operation(operation, arguments.get('task_id'),
|
||||
lambda: run_supervised(self, operation, arguments, **controls))
|
||||
|
||||
def _idle_quick_run_operation(
|
||||
self,
|
||||
name: str,
|
||||
@@ -1080,6 +1154,8 @@ class TaskCenterService:
|
||||
self._execution_rules(task_id),
|
||||
copy_part_number_to_value=task.copy_part_number_to_value,
|
||||
)
|
||||
if self._supervised:
|
||||
return progress # Only the batch process persists live execution state.
|
||||
by_path = {os.path.normcase(os.path.abspath(str(project.path))): project.project_id for project in projects}
|
||||
project_by_id = {project.project_id: project for project in projects}
|
||||
summary_by_id = {
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
"""Separate ordinary stop intent from an explicitly confirmed forced interruption."""
|
||||
from threading import Event
|
||||
import json
|
||||
import os
|
||||
|
||||
from .path_access import internal_path
|
||||
|
||||
|
||||
class StopControl(Event):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.force_event = Event()
|
||||
self.control_error = ''
|
||||
self.waiting_for_resolution = False
|
||||
self.resolution_error = ''
|
||||
|
||||
def request_force(self):
|
||||
self.set()
|
||||
self.control_error = ''
|
||||
self.force_event.set()
|
||||
|
||||
|
||||
def protect_interruption(root, reason):
|
||||
"""Create or re-sync a marker without overwriting prior protection evidence."""
|
||||
root = internal_path(root)
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
marker = root / 'uncertain.flag'
|
||||
try:
|
||||
stream = marker.open('x', encoding='utf-8')
|
||||
except FileExistsError:
|
||||
# A previous create/write/fsync may have failed. Existence alone does
|
||||
# not establish durable evidence; preserve its bytes but sync again.
|
||||
with marker.open('r+b') as existing:
|
||||
os.fsync(existing.fileno())
|
||||
else:
|
||||
with stream:
|
||||
stream.write(str(reason))
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
|
||||
|
||||
def wait_for_resolution(control, root, error):
|
||||
"""Interactive owners retain a possibly changed project until explicit force.
|
||||
|
||||
The GUI-side supervisor can also terminate its own Job after durable force
|
||||
evidence. Ordinary cancellation never releases this wait.
|
||||
"""
|
||||
if not isinstance(control, StopControl):
|
||||
return
|
||||
from .worker_monitor import WorkerMonitorError, _publish_progress
|
||||
root = internal_path(root)
|
||||
control.set()
|
||||
control.waiting_for_resolution = True
|
||||
control.resolution_error = str(error)
|
||||
def protect():
|
||||
protect_interruption(root, 'operation_interrupted: ' + str(error))
|
||||
try:
|
||||
protect()
|
||||
except OSError as failure:
|
||||
control.control_error = '保护记录写入失败,保留应用等待处理:' + str(failure)
|
||||
try:
|
||||
record = json.loads((root / 'worker-progress.json').read_text(encoding='utf-8'))
|
||||
if not isinstance(record, dict):
|
||||
record = {}
|
||||
except (OSError, ValueError):
|
||||
record = {}
|
||||
record.update(stage='awaiting_reconciliation', waiting_for_safe_completion=True,
|
||||
resolution_error=str(error), stop_requested=True, safe_stop=False)
|
||||
_publish_progress(root / 'worker-progress.json', record)
|
||||
while True:
|
||||
if control.force_event.wait(.1):
|
||||
try:
|
||||
protect()
|
||||
except OSError as failure:
|
||||
control.force_event.clear()
|
||||
control.control_error = '强制结束未执行,保护记录无法保存:' + str(failure)
|
||||
continue
|
||||
raise WorkerMonitorError('user forced stop of uncertain operation; reconciliation required')
|
||||
@@ -125,11 +125,12 @@ class TaskStore:
|
||||
|
||||
def __init__(self, path: Path) -> None:
|
||||
self.path = Path(path)
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
from .path_access import internal_path
|
||||
internal_path(self.path.parent).mkdir(parents=True, exist_ok=True)
|
||||
self._lock = threading.RLock()
|
||||
self._transaction_depth = 0
|
||||
self._savepoint_index = 0
|
||||
self._connection = sqlite3.connect(self.path, isolation_level=None, check_same_thread=False)
|
||||
self._connection = sqlite3.connect(internal_path(self.path), isolation_level=None, check_same_thread=False)
|
||||
self._connection.row_factory = sqlite3.Row
|
||||
with self._lock:
|
||||
self._connection.execute("PRAGMA foreign_keys = ON")
|
||||
@@ -418,6 +419,18 @@ class TaskStore:
|
||||
raise TaskCenterError("task_running", "pause a running task before deleting it")
|
||||
return self.set_task_status(task_id, TaskStatus.DELETED)
|
||||
|
||||
def archive_tasks(self, task_ids: Sequence[str]) -> tuple[TaskRecord, ...]:
|
||||
"""Move a safe close-session task set to the recycle bin in one commit."""
|
||||
with self.transaction():
|
||||
archived = []
|
||||
for task_id in task_ids:
|
||||
task = self.get_task(task_id)
|
||||
if task.status in {TaskStatus.RUNNING, TaskStatus.DELETED}:
|
||||
raise TaskCenterError("archive_task_state", "task cannot be archived in its current state")
|
||||
archived.append(self.set_task_status(task_id, TaskStatus.DELETED,
|
||||
error_id=task.error_id, summary=task.summary))
|
||||
return tuple(archived)
|
||||
|
||||
def record_quick_run_progress(
|
||||
self,
|
||||
task_id: str,
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Versioned, allowlisted JSON values exchanged with the batch supervisor."""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import fields, is_dataclass
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
import re
|
||||
|
||||
from .errors import TaskCenterError
|
||||
|
||||
VERSION = 1
|
||||
OPERATIONS = frozenset({'probe', 'runtime_smoke', 'inspect_bom_batch', 'preview_bom_batch',
|
||||
'collect_target_inventory', 'execute_task_queue', 'execute_quick_run_queue', 'execute_quick_run',
|
||||
'prepare_quick_run', 'prepare_failed_quick_run', 'recover_quick_run', 'cleanup_batch_resources',
|
||||
'target_cell_choices'})
|
||||
|
||||
|
||||
def valid_journal(value, batch_id):
|
||||
def identity(item):
|
||||
return (isinstance(item, dict) and type(item.get('pid')) is int and item['pid'] > 0
|
||||
and isinstance(item.get('creation_time'), str) and bool(item['creation_time']))
|
||||
return (isinstance(value, dict) and value.get('version') == VERSION and value.get('batch_id') == batch_id
|
||||
and isinstance(batch_id, str) and re.fullmatch('[0-9a-f]{32}', batch_id) is not None
|
||||
and isinstance(value.get('attempt_id'), str) and re.fullmatch('[0-9a-f]{32}', value['attempt_id']) is not None
|
||||
and value.get('operation') in OPERATIONS and value.get('state') in {'starting', 'running', 'blocked', 'finished'}
|
||||
and isinstance(value.get('task_ids'), list) and all(isinstance(v, str) and v and Path(v).name == v
|
||||
and v not in {'.', '..'} for v in value['task_ids'])
|
||||
and identity(value.get('parent_process')) and (value.get('process') is None or identity(value['process']))
|
||||
and isinstance(value.get('request_sha256'), str) and re.fullmatch('[0-9a-f]{64}', value['request_sha256']) is not None)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _types():
|
||||
from . import models, target_plan, quick_run_protocol, bom_batch, bom_import_report, task_queue, bom_import_state, batch
|
||||
modules = (models, target_plan, quick_run_protocol, bom_batch, bom_import_report, task_queue, bom_import_state, batch)
|
||||
return {value.__module__ + '.' + value.__name__: value
|
||||
for module in modules for value in vars(module).values()
|
||||
if isinstance(value, type) and value.__module__ == module.__name__
|
||||
and (is_dataclass(value) or issubclass(value, Enum))}
|
||||
|
||||
|
||||
def encode(value):
|
||||
if isinstance(value, Enum):
|
||||
return {'type': 'enum', 'name': type(value).__module__ + '.' + type(value).__name__, 'value': value.value}
|
||||
if value is None or type(value) in (str, int, float, bool):
|
||||
return value
|
||||
if isinstance(value, Path):
|
||||
return {'type': 'path', 'value': str(value)}
|
||||
if isinstance(value, datetime):
|
||||
return {'type': 'datetime', 'value': value.isoformat()}
|
||||
if type(value) in (tuple, list, frozenset):
|
||||
return {'type': type(value).__name__, 'items': [encode(item) for item in value]}
|
||||
if isinstance(value, dict):
|
||||
return {'type': 'dict', 'items': [[encode(key), encode(item)] for key, item in value.items()]}
|
||||
if is_dataclass(value):
|
||||
name = type(value).__module__ + '.' + type(value).__name__
|
||||
if name in _types():
|
||||
return {'type': 'object', 'name': name,
|
||||
'fields': {field.name: encode(getattr(value, field.name)) for field in fields(value)}}
|
||||
raise ValueError('Unsupported supervisor value: ' + type(value).__name__)
|
||||
|
||||
|
||||
def decode(value):
|
||||
if value is None or type(value) in (str, int, float, bool):
|
||||
return value
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError('Invalid supervisor value')
|
||||
kind = value.get('type')
|
||||
if kind == 'path':
|
||||
return Path(value['value'])
|
||||
if kind == 'datetime':
|
||||
return datetime.fromisoformat(value['value'])
|
||||
if kind in ('tuple', 'list', 'frozenset'):
|
||||
return {'tuple': tuple, 'list': list, 'frozenset': frozenset}[kind](decode(v) for v in value['items'])
|
||||
if kind == 'dict':
|
||||
return {decode(k): decode(v) for k, v in value['items']}
|
||||
cls = _types().get(value.get('name'))
|
||||
if cls is None:
|
||||
raise ValueError('Unknown supervisor type')
|
||||
if kind == 'enum' and issubclass(cls, Enum):
|
||||
return cls(value['value'])
|
||||
if kind == 'object' and is_dataclass(cls):
|
||||
if set(value['fields']) != {f.name for f in fields(cls)}:
|
||||
raise ValueError('Supervisor record fields do not match schema')
|
||||
return cls(**{key: decode(v) for key, v in value['fields'].items()})
|
||||
raise ValueError('Invalid supervisor type')
|
||||
|
||||
|
||||
def encode_error(error, *, depth=0):
|
||||
result = {'code': getattr(error, 'code', 'supervisor_operation'),
|
||||
'message': getattr(error, 'message', str(error)),
|
||||
'error_id': getattr(error, 'error_id', None)}
|
||||
for key in ('bom_report', 'completed_result'):
|
||||
if getattr(error, key, None) is not None:
|
||||
result[key] = encode(getattr(error, key))
|
||||
if depth < 3:
|
||||
for key in ('primary_error', 'cleanup_error'):
|
||||
other = getattr(error, key, None)
|
||||
if other is not None and other is not error:
|
||||
result[key] = encode_error(other, depth=depth + 1)
|
||||
return result
|
||||
|
||||
|
||||
def decode_error(value):
|
||||
error = TaskCenterError(value['code'], value['message'], error_id=value.get('error_id'))
|
||||
for key in ('bom_report', 'completed_result'):
|
||||
if key in value:
|
||||
setattr(error, key, decode(value[key]))
|
||||
for key in ('primary_error', 'cleanup_error'):
|
||||
if key in value:
|
||||
setattr(error, key, decode_error(value[key]))
|
||||
return error
|
||||
|
||||
|
||||
class MessageGate:
|
||||
def __init__(self, batch_id, attempt_id):
|
||||
self.batch_id, self.attempt_id = batch_id, attempt_id
|
||||
self.sequence, self.generation = 0, 1
|
||||
|
||||
def accept(self, value):
|
||||
if (not isinstance(value, dict) or value.get('version') != VERSION
|
||||
or value.get('batch_id') != self.batch_id or value.get('attempt_id') != self.attempt_id
|
||||
or type(value.get('sequence')) is not int or value['sequence'] <= self.sequence
|
||||
or type(value.get('generation')) is not int or value['generation'] < self.generation
|
||||
or value.get('type') not in {'progress', 'phase', 'heartbeat', 'result', 'state'}):
|
||||
return False
|
||||
self.sequence, self.generation = value['sequence'], value['generation']
|
||||
return True
|
||||
@@ -0,0 +1,50 @@
|
||||
"""Validate business evidence before changing any supervisor safety gate."""
|
||||
from .supervisor_protocol import decode, decode_error
|
||||
|
||||
|
||||
def validated_outcome(operation, body):
|
||||
from .batch import BatchRunResult
|
||||
from .bom_import_report import BomImportReport
|
||||
from .bom_batch import BomBatchPreview
|
||||
from .quick_run_protocol import QuickRunResult, QuickRunStepResult, QuickRunProgress
|
||||
from .target_plan import TargetPlanRow
|
||||
from .task_queue import TaskQueueResult
|
||||
if (not isinstance(body, dict) or type(body.get('business_succeeded')) is not bool
|
||||
or body.get('status') not in {'ok', 'error'}):
|
||||
raise ValueError('Invalid supervisor business outcome')
|
||||
result = decode(body['result']) if body['business_succeeded'] else None
|
||||
expected = {'inspect_bom_batch': BomImportReport, 'preview_bom_batch': BomBatchPreview,
|
||||
'execute_task_queue': TaskQueueResult, 'execute_quick_run_queue': BatchRunResult,
|
||||
'execute_quick_run': QuickRunStepResult, 'recover_quick_run': QuickRunStepResult,
|
||||
'prepare_quick_run': QuickRunResult, 'prepare_failed_quick_run': QuickRunProgress,
|
||||
'collect_target_inventory': tuple, 'target_cell_choices': tuple,
|
||||
'cleanup_batch_resources': tuple, 'probe': dict, 'runtime_smoke': dict}
|
||||
if body['business_succeeded'] and not isinstance(result, expected[operation]):
|
||||
raise ValueError('Supervisor result type does not match requested operation')
|
||||
if body['business_succeeded'] and operation == 'collect_target_inventory' and not all(isinstance(v, TargetPlanRow) for v in result):
|
||||
raise ValueError('Invalid inventory plan result')
|
||||
errors = []
|
||||
for name in ('error', 'operation_error'):
|
||||
value = body.get(name)
|
||||
if value is not None and (not isinstance(value, dict) or not isinstance(value.get('code'), str)
|
||||
or not isinstance(value.get('message'), str)):
|
||||
raise ValueError('Invalid supervisor failure')
|
||||
errors.append(decode_error(value) if value is not None else None)
|
||||
if not body['business_succeeded'] and not any(errors):
|
||||
raise ValueError('Failed supervisor outcome has no failure evidence')
|
||||
return result, *errors
|
||||
|
||||
|
||||
def resource_pending_error(operation, result, error, primary_error=None):
|
||||
from dataclasses import replace
|
||||
from .batch_resources import BatchResourceCleanupError
|
||||
from .bom_import_report import BomImportReport
|
||||
if isinstance(result, BomImportReport):
|
||||
result = replace(result, preview=None)
|
||||
failure = BatchResourceCleanupError('后台资源仍待清理;报告和已保存核验结果保留。' + (str(error) if error else ''),
|
||||
completed_result=result, primary_error=primary_error, cleanup_error=error)
|
||||
if isinstance(result, BomImportReport):
|
||||
failure.bom_report = result
|
||||
elif getattr(error, 'bom_report', None) is not None:
|
||||
failure.bom_report = replace(error.bom_report, preview=None)
|
||||
return failure
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Recoverable task-list archival performed only during an opted-in normal close."""
|
||||
from dataclasses import dataclass
|
||||
|
||||
from .models import ProjectStatus, TaskStatus
|
||||
from .path_access import internal_path
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ArchiveReport:
|
||||
archived_ids: tuple[str, ...]
|
||||
retained: tuple[tuple[str, str], ...] = ()
|
||||
|
||||
@property
|
||||
def retained_ids(self):
|
||||
return tuple(task_id for task_id, _ in self.retained)
|
||||
|
||||
|
||||
def archive_completed_session(service):
|
||||
"""Soft-delete safe tasks atomically; preserve recovery tasks and all files."""
|
||||
return service._idle_quick_run_operation('archive_completed_session', None,
|
||||
lambda: _archive(service))
|
||||
|
||||
|
||||
def _archive(service):
|
||||
store = service._store
|
||||
pending = service.pending_batch_resources()
|
||||
owned = set()
|
||||
unknown = False
|
||||
for record in pending:
|
||||
task_ids = record.get('log_task_ids') or record.get('task_ids')
|
||||
if not isinstance(task_ids, (list, tuple)) or not task_ids or any(
|
||||
not isinstance(task_id, str) or not task_id for task_id in task_ids):
|
||||
unknown = True
|
||||
else:
|
||||
owned.update(task_ids)
|
||||
retained, safe = [], []
|
||||
with store.transaction():
|
||||
for task in store.list_tasks():
|
||||
reason = None
|
||||
if unknown or task.task_id in owned:
|
||||
reason = '后台资源尚未确认释放;重新打开后使用“重新清理资源”。'
|
||||
elif task.status is TaskStatus.RUNNING:
|
||||
reason = '任务仍在执行状态;重新打开后暂停或核验恢复。'
|
||||
elif any(project.status in {ProjectStatus.CAPTURING_BEFORE, ProjectStatus.AWAITING_OPERATOR,
|
||||
ProjectStatus.CAPTURING_AFTER, ProjectStatus.VERIFYING}
|
||||
for project in store.list_projects(task.task_id)):
|
||||
reason = '人工流程尚未结束;重新打开后完成核验。'
|
||||
elif (service.has_interrupted_quick_run(task.task_id) or
|
||||
internal_path(service._project_task_service(task.task_id)._quick_run_preparer.data_root
|
||||
/ 'quick-run' / 'uncertain.flag').exists()):
|
||||
reason = '存在中断、保护或待核验结果;重新打开后使用“核验并恢复”。'
|
||||
if reason:
|
||||
retained.append((task.task_id, f'{task.name}:{reason}'))
|
||||
else:
|
||||
safe.append(task.task_id)
|
||||
store.archive_tasks(safe)
|
||||
return ArchiveReport(tuple(safe), tuple(retained))
|
||||
@@ -47,6 +47,8 @@ def _os_lock(handle, acquire):
|
||||
|
||||
|
||||
def _lock_file(root: Path):
|
||||
from .path_access import internal_path
|
||||
root = internal_path(root)
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
handle = (root / '.task-queue.execution.lock').open('a+b')
|
||||
if handle.seek(0, os.SEEK_END) == 0:
|
||||
@@ -57,6 +59,13 @@ def _lock_file(root: Path):
|
||||
|
||||
def assert_task_queue_idle(root: Path) -> None:
|
||||
"""Reject external mutations while a queue owns CAD; its runner may call services."""
|
||||
from .batch_supervisor import assert_supervisor_idle
|
||||
try:
|
||||
assert_supervisor_idle(root)
|
||||
except TaskCenterError as error:
|
||||
if error.code != 'supervisor_busy':
|
||||
raise
|
||||
raise TaskCenterError('quick_run_busy', '另一批操作正在执行,请先安全暂停。') from error
|
||||
if _QUEUE_OWNER == get_ident():
|
||||
return
|
||||
if _QUEUE_LOCK.locked():
|
||||
@@ -122,6 +131,8 @@ def _save(path: Path, state: dict) -> None:
|
||||
def execute_task_queue(service, task_ids: tuple[str, ...], *, stop_event: Event,
|
||||
queue_id: str | None = None, runtime=None) -> TaskQueueResult:
|
||||
"""Execute the frozen order; uncertain state stops before the next project."""
|
||||
if runtime is None and getattr(service, '_supervised', False):
|
||||
return service._supervise('execute_task_queue', {'task_ids': task_ids, 'queue_id': queue_id}, stop_event=stop_event)
|
||||
if runtime is None:
|
||||
from .batch_runtime import BatchCadRuntime
|
||||
with BatchCadRuntime(service._logging.root, 'task_queue', stop_event) as owned_runtime:
|
||||
|
||||
@@ -15,6 +15,7 @@ STAGES = {
|
||||
'restore_attribute_fields': '恢复属性字段', 'restore_attributes': '恢复属性',
|
||||
'verify_restoration': '核验恢复结果', 'verify_attribute_fields': '核验属性字段',
|
||||
'save_and_close': '保存并关闭工程', 'verify_saved_project': '回读已保存工程',
|
||||
'awaiting_reconciliation': '当前结果待核验,已保留工程现场',
|
||||
}
|
||||
|
||||
|
||||
@@ -22,4 +23,6 @@ def format_feedback(value, task_name, elapsed, stopping):
|
||||
stage = STAGES.get(value.stage, '')
|
||||
details = f'{stage}({value.stage_seconds:.1f} 秒)' if stage else value.message
|
||||
prefix = '等待当前步骤保存核验后暂停 · ' if stopping else ''
|
||||
if value.waiting_for_safe_completion:
|
||||
prefix = '当前保存或核验尚未确认,未自动结束应用;可继续等待、导出诊断,或确认强制结束后核验 · '
|
||||
return f'{prefix}任务 {value.position}/{value.total}:{task_name} · {details} · 总计 {elapsed} 秒'
|
||||
@@ -1,10 +1,11 @@
|
||||
"""Single-project task presentation; persistence and CAD remain in the service."""
|
||||
from __future__ import annotations
|
||||
from threading import Event
|
||||
from ..stop_control import StopControl
|
||||
from shiboken6 import isValid
|
||||
import time
|
||||
from PySide6.QtCore import QElapsedTimer, QSettings, QSignalBlocker, QThreadPool, QTimer, Qt
|
||||
from PySide6.QtWidgets import (QApplication, QDialog, QHBoxLayout, QMainWindow, QMessageBox,
|
||||
from PySide6.QtWidgets import (QApplication, QCheckBox, QDialog, QFileDialog, QHBoxLayout, QMainWindow, QMessageBox,
|
||||
QPushButton, QLabel, QSplitter, QTabWidget, QVBoxLayout, QWidget)
|
||||
from ..errors import TaskCenterError
|
||||
from ..models import TaskStatus
|
||||
@@ -12,7 +13,7 @@ from ..version import VERSION
|
||||
from .dialogs import _TaskDialog, _MappingDialog
|
||||
from .execution_state import execution_state
|
||||
from .mapping_table import MappingTable
|
||||
from .messages import ERROR_TEXT, _format_live_log_line
|
||||
from .messages import ERROR_TEXT, _format_live_log_line, format_operation_error
|
||||
from .project_plan import DirectoryTaskDialog, ProjectPlanPanel
|
||||
from .project_summary import ProjectSummary
|
||||
from .run_panel import RunPanel
|
||||
@@ -31,12 +32,15 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self._close_plan_confirmer = close_plan_confirmer or self._confirm_close_plans
|
||||
self._close_pending = False
|
||||
self._close_saving = False
|
||||
self._close_archived = False
|
||||
self._archive_close_notice = ''
|
||||
self._quick_run_confirmer = quick_run_confirmer or self._confirm_quick_run
|
||||
self._task_id = None
|
||||
self._context = None
|
||||
self._tasks = ()
|
||||
self._known_task_groups = {}
|
||||
self._workers = set()
|
||||
self._diagnostic_worker = None
|
||||
self._closed = False
|
||||
self._batch_stop = None
|
||||
self._active_quick_run_task_id = None
|
||||
@@ -86,16 +90,24 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self._log_presenter = TaskLogPresenter(service, self.run_panel, self)
|
||||
self.run_panel.quick_run_button.clicked.connect(self._prepare_quick_run)
|
||||
self.run_panel.stop_queue_button.clicked.connect(self._pause_queue)
|
||||
self.run_panel.force_stop_button.clicked.connect(self._force_batch_stop)
|
||||
self.run_panel.retry_button.clicked.connect(self._retry_failed)
|
||||
self.run_panel.recover_button.clicked.connect(self._recover_quick_run)
|
||||
self.run_panel.cleanup_resources_button.clicked.connect(self._cleanup_batch_resources)
|
||||
self.run_panel.diagnostics_button.clicked.connect(self._export_diagnostics)
|
||||
self.run_panel.new_round_button.clicked.connect(self._start_new_round)
|
||||
self.run_panel.history_button.clicked.connect(self._open_round_history)
|
||||
self.run_panel.copy_value_check.toggled.connect(self._value_option_changed)
|
||||
self.run_panel.reset_layout_button.clicked.connect(self._reset_layout)
|
||||
self.run_panel.toggle_log_button.toggled.connect(self._log_visibility_changed)
|
||||
self.archive_on_close_check = QCheckBox('正常关闭时归档并清空当前任务列表(可从回收站恢复)',
|
||||
objectName='archiveOnNormalClose')
|
||||
self.archive_on_close_check.setToolTip('默认关闭。仅归档安全任务,保留计划、日志和文件;待核验或资源未释放的任务保留在当前列表。')
|
||||
self._build_layout()
|
||||
self._layout_settings = QSettings(str(service.layout_settings_path), QSettings.Format.IniFormat)
|
||||
self.archive_on_close_check.setChecked(self._layout_settings.value(
|
||||
'tasks/archive_on_close', False, type=bool))
|
||||
self.archive_on_close_check.toggled.connect(self._set_archive_on_close)
|
||||
sort_index = self.task_list.sort.findData(self._layout_settings.value('tasks/sort', 'execution'))
|
||||
if sort_index >= 0:
|
||||
self.task_list.sort.setCurrentIndex(sort_index)
|
||||
@@ -106,6 +118,8 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
widget.restoreState(saved)
|
||||
self._refresh_task_list()
|
||||
self._update_gates()
|
||||
if getattr(service, '_supervised', False):
|
||||
QTimer.singleShot(0, self._startup_resource_recheck)
|
||||
|
||||
@property
|
||||
def _loading(self):
|
||||
@@ -144,6 +158,7 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
details_layout = QVBoxLayout(details)
|
||||
details_layout.setContentsMargins(0, 0, 0, 0)
|
||||
details_layout.addWidget(self.run_panel.copy_value_check)
|
||||
details_layout.addWidget(self.archive_on_close_check)
|
||||
details_layout.addWidget(self.detail_splitter, 1)
|
||||
self.workspace_splitter = QSplitter(Qt.Orientation.Horizontal)
|
||||
self.workspace_splitter.setChildrenCollapsible(False)
|
||||
@@ -177,6 +192,20 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
box.exec()
|
||||
return 'save' if box.clickedButton() is save else 'discard' if box.clickedButton() is discard else 'cancel'
|
||||
|
||||
def _set_archive_on_close(self, enabled):
|
||||
self._layout_settings.setValue('tasks/archive_on_close', bool(enabled))
|
||||
self._layout_settings.sync()
|
||||
|
||||
def _session_archived(self, report):
|
||||
self._close_archived = True
|
||||
self._close_pending = True
|
||||
if report.retained:
|
||||
self._archive_close_notice = ('以下任务未归档,重新打开后仍保留在任务列表,可继续恢复:\n'
|
||||
+ '\n'.join(reason for _, reason in report.retained))
|
||||
self._set_operator_message(self._archive_close_notice, 'warning')
|
||||
if self.isVisible() and QApplication.platformName().casefold() != 'offscreen':
|
||||
QMessageBox.warning(self, '已保留待恢复任务', self._archive_close_notice)
|
||||
|
||||
def closeEvent(self, event):
|
||||
if self._close_saving:
|
||||
event.ignore()
|
||||
@@ -206,6 +235,12 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
event.ignore()
|
||||
return
|
||||
self.plan_panel.discard_all()
|
||||
if self.archive_on_close_check.isChecked() and not self._close_archived:
|
||||
from ..task_archive import archive_completed_session
|
||||
event.ignore()
|
||||
self._close_saving = True
|
||||
self._run_async(lambda: archive_completed_session(self.service), self._session_archived)
|
||||
return
|
||||
for name, widget in self._layout_widgets().items():
|
||||
self._layout_settings.setValue(f'single_project/{name}', widget.saveState())
|
||||
self._layout_settings.setValue('tasks/sort', self.task_list.sort.currentData())
|
||||
@@ -305,6 +340,10 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self.run_panel.history_button.setEnabled(bool(context and not busy))
|
||||
self.run_panel.stop_queue_button.setVisible(self._batch_stop is not None)
|
||||
self.run_panel.stop_queue_button.setEnabled(bool(self._batch_stop and not self._batch_stop.is_set()))
|
||||
self.run_panel.force_stop_button.setVisible(self._batch_stop is not None)
|
||||
self.run_panel.force_stop_button.setEnabled(bool(self._batch_stop and
|
||||
not self._batch_stop.force_event.is_set()))
|
||||
self.run_panel.diagnostics_button.setEnabled(self._diagnostic_worker is None)
|
||||
self.run_panel.copy_value_check.setEnabled(state.editable)
|
||||
rule_editable = state.editable and not group_unsaved
|
||||
self.mapping_table.set_rules_editable(rule_editable)
|
||||
@@ -539,6 +578,8 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self._finish_run()
|
||||
self._metadata_changed(None)
|
||||
failures = sum(e.state in {'failed', 'blocked'} for e in result.entries)
|
||||
if failures:
|
||||
self._close_pending = False
|
||||
self._show_task_log(force=True)
|
||||
self._set_operator_message(f'本批任务已暂停。失败/阻断 {failures} 项。' if result.paused else f'本批任务已结束。失败 {failures} 项。',
|
||||
'warning' if result.paused or failures else 'success')
|
||||
@@ -546,7 +587,7 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self.task_list.queue_button)
|
||||
|
||||
def _begin_run(self, task_id):
|
||||
self._batch_stop = Event()
|
||||
self._batch_stop = StopControl()
|
||||
self._active_quick_run_task_id = task_id
|
||||
self._run_started_at = time.time()
|
||||
self._quick_run_elapsed.start()
|
||||
@@ -569,6 +610,24 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self._set_operator_message('已请求暂停;当前项安全收尾后停止。', 'warning')
|
||||
self._update_gates()
|
||||
|
||||
def _force_batch_stop(self):
|
||||
control = self._batch_stop
|
||||
if control is None or control.force_event.is_set():
|
||||
return
|
||||
if QMessageBox.question(self, '强制结束当前操作',
|
||||
'当前操作可能正在修改或保存工程。强制结束可能留下未保存或部分保存的结果。\n\n'
|
||||
'已完成的执行证据会保留;中断后必须先核验恢复,不能直接重复替换。'
|
||||
'建议先导出诊断包。确定强制结束本程序当前操作吗?',
|
||||
QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No,
|
||||
QMessageBox.StandardButton.No) != QMessageBox.StandardButton.Yes:
|
||||
return
|
||||
if self._batch_stop is not control:
|
||||
return # Completion/new run while the confirmation dialog was open.
|
||||
self._close_pending = False
|
||||
control.request_force()
|
||||
self._set_operator_message('已请求强制结束;正在记录中断证据,结果将进入待核验。', 'warning')
|
||||
self._update_gates()
|
||||
|
||||
def _show_batch_result(self, result):
|
||||
owner = self._active_quick_run_task_id or self._task_id
|
||||
name = next((t.name for t in self._tasks if t.task_id == owner), owner)
|
||||
@@ -577,6 +636,8 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
if owner == self._task_id:
|
||||
self._show_task_log(force=True)
|
||||
p = result.progress
|
||||
if p.failed_items:
|
||||
self._close_pending = False
|
||||
self._set_operator_message(f"任务「{name}」{'已暂停' if result.paused else '已结束'}:已保存 {p.saved_items},失败 {p.failed_items},未命中 {p.no_match_items}。",
|
||||
'warning' if result.paused or p.failed_items else 'success')
|
||||
|
||||
@@ -591,6 +652,46 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self._run_async(self.service.cleanup_batch_resources, done,
|
||||
self.run_panel.cleanup_resources_button)
|
||||
|
||||
def _startup_resource_recheck(self):
|
||||
if self._closed:
|
||||
return
|
||||
if self._workers or self._loading:
|
||||
QTimer.singleShot(250, self._startup_resource_recheck)
|
||||
return
|
||||
if self.service.pending_batch_resources():
|
||||
self._update_gates()
|
||||
self._cleanup_batch_resources() # Resource-only; never resumes a queue.
|
||||
|
||||
def _export_diagnostics(self):
|
||||
from pathlib import Path
|
||||
name = 'DxPartReplace-诊断-' + time.strftime('%Y%m%d-%H%M%S') + '.zip'
|
||||
value, _ = QFileDialog.getSaveFileName(self, '导出脱敏诊断包', name, 'ZIP 文件 (*.zip)')
|
||||
if not value or self._diagnostic_worker is not None:
|
||||
return
|
||||
# A read-only support export remains available while CAD is waiting.
|
||||
worker = ServiceWorker(lambda: self.service.export_diagnostics(Path(value)))
|
||||
self._diagnostic_worker = worker
|
||||
self._workers.add(worker)
|
||||
def exported(path):
|
||||
if not self._closed:
|
||||
self._set_operator_message(f'脱敏诊断包已导出:{path}', 'success')
|
||||
def failed(error):
|
||||
self._close_pending = False
|
||||
if not self._closed:
|
||||
self._show_error(error)
|
||||
def finished():
|
||||
self._workers.discard(worker)
|
||||
self._diagnostic_worker = None
|
||||
if not self._closed:
|
||||
self._update_gates()
|
||||
if self._close_pending and not self._workers:
|
||||
QTimer.singleShot(0, self.close)
|
||||
worker.signals.succeeded.connect(exported)
|
||||
worker.signals.failed.connect(failed)
|
||||
worker.signals.finished.connect(finished)
|
||||
self._update_gates()
|
||||
QThreadPool.globalInstance().start(worker)
|
||||
|
||||
def _retry_failed(self):
|
||||
if self.run_panel.retry_button.isEnabled():
|
||||
task_id = self._task_id
|
||||
@@ -649,7 +750,8 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self._finish_run()
|
||||
self._metadata_changed(None)
|
||||
self._set_operator_message(result.message)
|
||||
self._run_async(lambda: self.service.recover_quick_run(task_id, confirmed=True), done,
|
||||
stop = self._batch_stop
|
||||
self._run_async(lambda: self.service.recover_quick_run(task_id, confirmed=True, stop_event=stop), done,
|
||||
self.run_panel.recover_button, expected_task_id=task_id)
|
||||
|
||||
def _confirm_quick_run(self, scope, selected_projects, enabled_rules):
|
||||
@@ -674,6 +776,9 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
self.run_panel.execution_status.setText(format_feedback(
|
||||
value, name, self._quick_run_elapsed.elapsed() // 1000,
|
||||
bool(self._batch_stop and self._batch_stop.is_set())))
|
||||
if self._batch_stop and self._batch_stop.control_error:
|
||||
self._set_operator_message(self._batch_stop.control_error, 'warning')
|
||||
self._update_gates()
|
||||
self._feedback_query.request(lambda control: self.service.execution_feedback(task_id, started_after=started, read_control=control),
|
||||
received, lambda _: None, cancellable=True)
|
||||
if self.run_panel.toggle_log_button.isChecked():
|
||||
@@ -748,6 +853,7 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
callback(result)
|
||||
except Exception as error:
|
||||
operation_error = error
|
||||
self._close_pending = False
|
||||
self.operation_state = self.operation_state.finish(error)
|
||||
self._show_error(error)
|
||||
def failed(error):
|
||||
@@ -783,7 +889,7 @@ class TrialTaskCenterWindow(QMainWindow):
|
||||
|
||||
def _show_error(self, error, *, owner_task_id=None, **_unused):
|
||||
stable = error if isinstance(error, TaskCenterError) else self.service.report_ui_error(error)
|
||||
message = f'{ERROR_TEXT.get(stable.code, "操作未完成")}\n{stable.message}\n错误编号:{stable.error_id}'
|
||||
message = format_operation_error(stable)
|
||||
if owner_task_id:
|
||||
name = next((t.name for t in self._tasks if t.task_id == owner_task_id), owner_task_id)
|
||||
message = f'任务「{name}」:{message}'
|
||||
|
||||
@@ -4,7 +4,11 @@ import json
|
||||
import re
|
||||
|
||||
ERROR_TEXT = {
|
||||
"batch_resources_pending": "已核验的替换结果已保留。后台资源尚未释放,请使用“重新清理资源”后再开始下一批。",
|
||||
'user_forced_stop': '操作已强制中断,结果待核验。',
|
||||
'application_shutdown_failed': '业务结果已保留,但应用收尾异常。',
|
||||
"batch_resources_pending": "后台资源尚未确认释放,请使用“重新清理资源”后再开始下一批。",
|
||||
"supervisor_busy": "另一批操作正在执行,请先安全收尾。",
|
||||
"supervisor_interrupted": "批次监督连接异常,保护记录已保留,没有自动重放替换。",
|
||||
"inventory_session": "原生读取会话异常,安全重试未完成,已停止本批处理。请查看诊断记录并排除原因后重试。",
|
||||
"task_required": "请先选择任务。",
|
||||
"directory_required": "请先选择工程目录。",
|
||||
@@ -50,6 +54,28 @@ ERROR_TEXT = {
|
||||
"task_name": "任务名称为空或已经存在。",
|
||||
}
|
||||
|
||||
|
||||
def format_operation_error(error):
|
||||
title = ERROR_TEXT.get(error.code, '操作未完成')
|
||||
report = getattr(error, 'bom_report', None)
|
||||
if report is not None:
|
||||
title = 'BOM 导入未完成,校验报告已保留。' + title
|
||||
result = getattr(error, 'completed_result', None)
|
||||
if (getattr(result, 'state', None) == 'saved_ok'
|
||||
or getattr(getattr(result, 'progress', None), 'saved_items', 0)
|
||||
or any(getattr(entry, 'state', None) == 'completed' for entry in getattr(result, 'entries', ()))):
|
||||
title += ' 已保存核验的结果保留,不会重新替换。'
|
||||
lines = [title]
|
||||
for label, name, limit in (('首次错误', 'primary_error', 180), ('收尾错误', 'cleanup_error', 140)):
|
||||
failure = getattr(error, name, None)
|
||||
if failure is not None:
|
||||
text = getattr(failure, 'message', str(failure)).replace('\n', ' ')
|
||||
lines.append(f'{label}:{text[:limit]}' + ('…' if len(text) > limit else ''))
|
||||
if len(lines) == 1:
|
||||
lines.append(error.message[:260])
|
||||
lines.append('错误编号:' + error.error_id)
|
||||
return '\n'.join(lines)
|
||||
|
||||
DIALOG_ERROR_TEXT = {
|
||||
"duplicate_source": "该旧料号已经存在,请修改现有规则或输入其他旧料号。",
|
||||
"mapping_chain": "目标料号同时是另一条规则的旧料号,请拆分任务后再执行。",
|
||||
|
||||
@@ -23,10 +23,14 @@ class RunPanel(QWidget):
|
||||
self.busy_indicator.hide()
|
||||
self.quick_run_button = QPushButton('执行当前任务', objectName='quickRunButton')
|
||||
self.stop_queue_button = QPushButton('完成当前项后暂停', objectName='stopQueueButton')
|
||||
self.force_stop_button = QPushButton('强制结束…', objectName='forceStopButton')
|
||||
self.force_stop_button.setToolTip('异常情况下使用;会中断当前操作,结果必须重新核验。')
|
||||
self.retry_button = QPushButton('重试失败项', objectName='retryTaskItems')
|
||||
self.recover_button = QPushButton('核验并恢复', objectName='recoverQuickRunButton')
|
||||
self.cleanup_resources_button = QPushButton('重新清理资源', objectName='cleanupBatchResources')
|
||||
self.cleanup_resources_button.setToolTip('只释放本程序拥有的后台资源,不执行工程替换。')
|
||||
self.diagnostics_button = QPushButton('导出诊断包', objectName='exportDiagnosticBundle')
|
||||
self.diagnostics_button.setToolTip('导出脱敏日志与资源记录,不包含工程、中心库、BOM 或任务数据库。')
|
||||
self.new_round_button = QPushButton('开始新一轮', objectName='startNewRoundButton')
|
||||
self.history_button = QPushButton('查看轮次证据', objectName='roundHistoryButton')
|
||||
self.copy_value_check = QCheckBox('Value 策略:替换后将 Part Label 写入 Value(逐行自定义优先)', objectName='copyPartNumberToValue')
|
||||
@@ -39,8 +43,8 @@ class RunPanel(QWidget):
|
||||
self.log_dialog = None
|
||||
self.toggle_log_button.toggled.connect(self._toggle_log)
|
||||
buttons = QHBoxLayout()
|
||||
for button in (self.quick_run_button, self.retry_button, self.recover_button, self.cleanup_resources_button,
|
||||
self.new_round_button, self.history_button, self.stop_queue_button):
|
||||
for button in (self.quick_run_button, self.retry_button, self.recover_button, self.cleanup_resources_button, self.diagnostics_button,
|
||||
self.new_round_button, self.history_button, self.stop_queue_button, self.force_stop_button):
|
||||
buttons.addWidget(button)
|
||||
buttons.addStretch()
|
||||
buttons.addWidget(self.toggle_log_button)
|
||||
|
||||
@@ -1 +1 @@
|
||||
VERSION = "1.1.4-beta.2"
|
||||
VERSION = "1.1.4-beta.4"
|
||||
@@ -35,6 +35,7 @@ class ProgressWatch:
|
||||
self.done = 0
|
||||
self.total = 0
|
||||
self.safe_stop = True
|
||||
self.waiting_for_safe_completion = False
|
||||
self.started = started
|
||||
self.stage_started = started
|
||||
self.stage_seconds: dict[str, float] = {}
|
||||
@@ -67,6 +68,7 @@ class ProgressWatch:
|
||||
self.native_sequence = 0
|
||||
self.last_progress = now
|
||||
self.stage, self.done, self.total, self.safe_stop = fields[4], done, total, bool(safe)
|
||||
self.waiting_for_safe_completion = False
|
||||
return True
|
||||
|
||||
def accept_native(self, line: bytes, now: float) -> bool:
|
||||
@@ -102,6 +104,11 @@ class ProgressWatch:
|
||||
self.stop_requested = True
|
||||
return 'request_stop'
|
||||
if self.stop_requested and idle >= self.idle_timeout * 2:
|
||||
if not self.safe_stop:
|
||||
if not self.waiting_for_safe_completion:
|
||||
self.waiting_for_safe_completion = True
|
||||
return 'await_safe_completion'
|
||||
return None
|
||||
return 'unresponsive'
|
||||
return None
|
||||
|
||||
@@ -109,7 +116,7 @@ class ProgressWatch:
|
||||
now = time.monotonic() if now is None else now
|
||||
stages = dict(self.stage_seconds)
|
||||
stages[self.stage] = stages.get(self.stage, 0.0) + max(0.0, now - self.stage_started)
|
||||
return {name: getattr(self, name) for name in ('token', 'sequence', 'stage', 'done', 'total', 'safe_stop', 'stop_requested')} | {
|
||||
return {name: getattr(self, name) for name in ('token', 'sequence', 'stage', 'done', 'total', 'safe_stop', 'stop_requested', 'waiting_for_safe_completion')} | {
|
||||
'job_id': self.current_job_id,
|
||||
'native_progress_frames': self.native_progress_frames,
|
||||
'elapsed_seconds': max(0.0, now - self.started), 'stage_seconds': stages}
|
||||
@@ -124,9 +131,8 @@ def _atomic_text(path: Path, text: str) -> None:
|
||||
def _publish_progress(path: Path, snapshot: dict) -> None:
|
||||
try:
|
||||
_atomic_text(path, json.dumps(snapshot, ensure_ascii=False))
|
||||
except OSError as error:
|
||||
if not isinstance(error, PermissionError) and getattr(error, 'winerror', None) not in {5, 32, 33}:
|
||||
raise
|
||||
except OSError:
|
||||
pass # Display telemetry failure must not terminate a possible CAD write.
|
||||
|
||||
|
||||
def _decode(data: bytes) -> str:
|
||||
@@ -190,7 +196,7 @@ class _OwnedWorkers:
|
||||
|
||||
|
||||
def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abort_event=None,
|
||||
process_factory=None, cooperative_abort=False) -> subprocess.CompletedProcess[str]:
|
||||
process_factory=None, cooperative_abort=False, force_event=None, force_control=None) -> subprocess.CompletedProcess[str]:
|
||||
root = Path(command[3]).resolve()
|
||||
job_id = command[-1]
|
||||
token = uuid4().hex
|
||||
@@ -200,8 +206,11 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
|
||||
native_progress_path = root / f'monitor-{token}.native-progress'
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
watch = ProgressWatch(token, job_id, idle_timeout, time.monotonic())
|
||||
if force_control is not None:
|
||||
force_event = force_control.force_event
|
||||
owned = _OwnedWorkers(token, {command[0], command[4]})
|
||||
process = None
|
||||
durable_force_accepted = False
|
||||
threads = []
|
||||
messages = queue.Queue(maxsize=1024)
|
||||
stop_readers = threading.Event()
|
||||
@@ -244,15 +253,32 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
|
||||
_publish_progress(progress_path, watch.snapshot())
|
||||
_atomic_text(ready, token)
|
||||
while process.poll() is None:
|
||||
if force_event is not None and force_event.is_set():
|
||||
from .stop_control import protect_interruption
|
||||
try:
|
||||
protect_interruption(root, 'user_forced_stop')
|
||||
except OSError as error:
|
||||
force_event.clear()
|
||||
if force_control is not None:
|
||||
force_control.control_error = '强制结束未执行,保护记录无法保存:' + str(error)
|
||||
watch.waiting_for_safe_completion = True
|
||||
_publish_progress(progress_path, watch.snapshot())
|
||||
else:
|
||||
durable_force_accepted = True
|
||||
raise WorkerMonitorError('user forced stop; execution requires reconciliation')
|
||||
if abort_event is not None and abort_event.is_set():
|
||||
if not cooperative_abort:
|
||||
if not cooperative_abort and watch.safe_stop:
|
||||
raise WorkerMonitorError('project session supervisor requested worker cleanup')
|
||||
if not watch.stop_requested:
|
||||
# Read-only callers request the VBS safe boundary first.
|
||||
# The existing idle watchdog still bounds a stalled COM call.
|
||||
watch.stop_requested = True
|
||||
watch.last_progress = time.monotonic()
|
||||
_atomic_text(stop, token)
|
||||
try:
|
||||
_atomic_text(stop, token)
|
||||
except OSError:
|
||||
if watch.safe_stop:
|
||||
raise
|
||||
_publish_progress(progress_path, watch.snapshot())
|
||||
try:
|
||||
receive(*messages.get(timeout=min(0.1, idle_timeout / 10)))
|
||||
@@ -273,7 +299,17 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
|
||||
break
|
||||
action = watch.action(time.monotonic())
|
||||
if action == 'request_stop':
|
||||
_atomic_text(stop, token)
|
||||
try:
|
||||
_atomic_text(stop, token)
|
||||
except OSError:
|
||||
if watch.safe_stop:
|
||||
raise
|
||||
_publish_progress(progress_path, watch.snapshot())
|
||||
elif action == 'await_safe_completion':
|
||||
if cooperative_abort:
|
||||
# This entry point is explicitly read-only. A blocked COM
|
||||
# read has no write to finish and retains its bounded cancel.
|
||||
raise subprocess.TimeoutExpired(command, idle_timeout * 2)
|
||||
_publish_progress(progress_path, watch.snapshot())
|
||||
elif action == 'unresponsive':
|
||||
raise subprocess.TimeoutExpired(command, idle_timeout * 2)
|
||||
@@ -287,6 +323,15 @@ def run_monitored_process(command: tuple[str, ...], idle_timeout: float, *, abor
|
||||
return subprocess.CompletedProcess(command, process.returncode, _decode(output[0]), _decode(output[1]))
|
||||
except Exception as error:
|
||||
if process is not None:
|
||||
if (not watch.safe_stop and not cooperative_abort and force_control is not None
|
||||
and not durable_force_accepted):
|
||||
from .stop_control import wait_for_resolution
|
||||
try:
|
||||
wait_for_resolution(force_control, root, error)
|
||||
except WorkerMonitorError:
|
||||
if not force_event.is_set():
|
||||
raise
|
||||
durable_force_accepted = True
|
||||
failures = owned.stop_workers(process)
|
||||
process.wait(timeout=10)
|
||||
for thread in threads:
|
||||
|
||||
@@ -2,9 +2,11 @@
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
|
||||
from .path_access import internal_path
|
||||
|
||||
|
||||
def copy_worker_script(source: Path, destination: Path) -> None:
|
||||
source, destination = Path(source), Path(destination)
|
||||
source, destination = internal_path(source), internal_path(destination)
|
||||
if source.resolve() == destination.resolve():
|
||||
return
|
||||
shutil.copyfile(source, destination)
|
||||
|
||||
@@ -175,13 +175,13 @@ Set QP_App=FakeApp
|
||||
FakeApp.ProjectPath=Target
|
||||
QP_CurrentJobId="batch-job": QP_ExpectedJobId="batch-job"
|
||||
QPWriteUtf8 QP_Fso.BuildPath(QP_Root,"command.tsv"),"task_id" & vbTab & "job_id" & vbTab & "project_path" & vbTab & "old_part" & vbTab & "new_part" & vbTab & "value_mode" & vbCrLf & "task" & vbTab & "batch-job" & vbTab & Target & vbTab & "OLD" & vbTab & "NEW" & vbTab & "preserve" & vbCrLf
|
||||
QP_BatchMode=True: QP_OwnedProject=True
|
||||
QP_BatchMode=True: QP_OwnedProject=True: QP_OwnedProjectPath=Target
|
||||
QPWriteStatus QP_Root,QP_CurrentJobId,"no_match","","not found"
|
||||
Require Not QP_Fso.FileExists(QP_Fso.BuildPath(QP_Root,"status.tsv")),"published success before close"
|
||||
Require QPReadState(QP_Fso.BuildPath(QP_Root,"status.tsv"),QP_CurrentJobId)="no_match","pending no-match not visible to worker"
|
||||
QPFinishBatchItem
|
||||
Require FakeApp.CloseCount=1,"project not closed exactly once"
|
||||
Require FakeApp.SaveCount=1,"sheet not closed with saving allowed"
|
||||
Require FakeApp.SaveCount=0,"no-match must close without saving"
|
||||
Require InStr(QPReadUtf8(QP_Fso.BuildPath(QP_Root,"status.tsv")),"batch_closed")>0,"no completion proof"
|
||||
''')
|
||||
|
||||
@@ -368,8 +368,8 @@ Require FakeApp.ProjectPath = "", "discovery left the project open"
|
||||
Require FakeApp.Documents.OpenCount = 0, "discovery opened a page"
|
||||
Dim Manifest
|
||||
Manifest = QPReadUtf8(QP_Fso.BuildPath(QP_Root, "sheets.tsv"))
|
||||
Require InStr(Manifest, Target & vbTab & "Main" & vbTab & "1") > 0, "sheet one missing"
|
||||
Require InStr(Manifest, Target & vbTab & "Main" & vbTab & "2") > 0, "sheet two missing"
|
||||
Require InStr(Manifest, LCase(Target) & vbTab & "Main" & vbTab & "1") > 0, "sheet one missing"
|
||||
Require InStr(Manifest, LCase(Target) & vbTab & "Main" & vbTab & "2") > 0, "sheet two missing"
|
||||
''')
|
||||
|
||||
|
||||
|
||||
@@ -253,6 +253,31 @@ End Sub''', routines)
|
||||
'Sub QPVerifyUntouched(ByRef Records)\n' + injection)
|
||||
|
||||
|
||||
def test_save_then_project_close_refusal_retains_save_boundary(tmp_path):
|
||||
run_vbs(tmp_path, 'part_number', '''
|
||||
Dim FullChecks
|
||||
FullChecks=0
|
||||
Command.Add "project_path",QPProjectPath()
|
||||
QP_BatchMode=True: QP_SessionMode=True
|
||||
QP_SessionToken=String(32,"b")
|
||||
QP_OwnedProject=True: QP_OwnedProjectPath=QPProjectPath()
|
||||
QPRestoreAfterOperatorReplace Command,SnapshotPath
|
||||
QP_App.RefuseClose=True
|
||||
On Error Resume Next
|
||||
QPFinishBatchItem
|
||||
Failure=Err.Number
|
||||
Err.Clear
|
||||
On Error GoTo 0
|
||||
Require Failure<>0,"refused close must stop the operation"
|
||||
Require QP_BatchSaveCommitted,"returned save was lost when project close failed"
|
||||
QP_App.RefuseClose=False
|
||||
QPDiscardFailedBatchProject CloseError
|
||||
Require Len(CloseError)>0,"uncertain save must not discard the open project"
|
||||
Require Len(QPProjectPath())>0,"failed saved project was silently discarded"
|
||||
Require Not QP_Fso.FileExists(QP_Fso.BuildPath(QP_Root,"project-session-result.tsv")),"failure published success"
|
||||
''', routines_transform=_batch_readback_fixture)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('fault', ['', 'before_save', 'after_reopen'])
|
||||
def test_owned_batch_checks_full_state_at_save_and_disk_boundaries(tmp_path, fault):
|
||||
run_vbs(tmp_path, 'part_number', '''
|
||||
|
||||
@@ -208,6 +208,10 @@ def test_runtime_smoke_tracks_real_process_and_releases_on_failure(tmp_path, mon
|
||||
else:
|
||||
_smoke_batch_runtime(tmp_path, configure_logging(tmp_path))
|
||||
paths = list((tmp_path / 'batch-resources').glob('*/session.json'))
|
||||
if missing_dependency:
|
||||
# The GUI now verifies supervisor identity before launching anything.
|
||||
assert not paths
|
||||
return
|
||||
assert len(paths) == 1
|
||||
record = json.loads(paths[0].read_text(encoding='utf-8'))
|
||||
assert record['state'] == 'released' and record['cad_pid'] is None
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
from pathlib import Path
|
||||
from dx_productivity.trial_task_center.path_access import strip_extended_prefix
|
||||
import pytest
|
||||
from dx_productivity.trial_task_center.models import ProjectRecord, ProjectStatus, RuleRecord
|
||||
from dx_productivity.trial_task_center.quick_run import QuickRunPreparer
|
||||
@@ -21,7 +22,7 @@ def test_snapshot_reads_each_file_once_and_uses_same_status(tmp_path, monkeypatc
|
||||
opens = []
|
||||
original = Path.open
|
||||
def tracked(path, *args, **kwargs):
|
||||
if path.parent == handoff.command_path.parent:
|
||||
if Path(strip_extended_prefix(path)).parent == handoff.command_path.parent:
|
||||
opens.append(path.name)
|
||||
return original(path, *args, **kwargs)
|
||||
monkeypatch.setattr(Path, 'open', tracked)
|
||||
@@ -48,7 +49,7 @@ def test_snapshot_cancellation_stops_before_next_file_and_never_writes(tmp_path,
|
||||
def tracked(path, *args, **kwargs):
|
||||
opened.append(path.name)
|
||||
result = original(path, *args, **kwargs)
|
||||
if path == handoff.command_path:
|
||||
if Path(strip_extended_prefix(path)) == handoff.command_path:
|
||||
control.cancelled = True
|
||||
return result
|
||||
monkeypatch.setattr(Path, 'open', tracked)
|
||||
@@ -91,7 +92,7 @@ def test_snapshot_retries_when_uncertain_flag_is_created_during_status_read(tmp_
|
||||
injected = False
|
||||
def racing_open(path, *args, **kwargs):
|
||||
nonlocal injected
|
||||
if path == handoff.status_path and args and args[0] == 'r' and not injected:
|
||||
if Path(strip_extended_prefix(path)) == handoff.status_path and args and args[0] == 'r' and not injected:
|
||||
injected = True
|
||||
(path.parent / 'uncertain.flag').write_text('uncertain', encoding='utf-8')
|
||||
return original(path, *args, **kwargs)
|
||||
@@ -108,7 +109,7 @@ def test_snapshot_retries_command_and_status_as_one_generation(tmp_path, monkeyp
|
||||
injected = False
|
||||
def racing_open(path, *args, **kwargs):
|
||||
nonlocal injected
|
||||
if path == handoff.status_path and args and args[0] == 'r' and not injected:
|
||||
if Path(strip_extended_prefix(path)) == handoff.status_path and args and args[0] == 'r' and not injected:
|
||||
injected = True
|
||||
handoff.command_path.write_text(command.replace(job, 'new-generation'), encoding='utf-8')
|
||||
handoff.status_path.write_text('job_id\tstate\tcode\tmessage\nnew-generation\tno_match\t\tno match\n', encoding='utf-8')
|
||||
@@ -126,7 +127,7 @@ def test_snapshot_continuous_changes_block_continuation_after_bounded_retries(tm
|
||||
reads = 0
|
||||
def racing_open(path, *args, **kwargs):
|
||||
nonlocal reads
|
||||
if path == handoff.status_path and args and args[0] == 'r':
|
||||
if Path(strip_extended_prefix(path)) == handoff.status_path and args and args[0] == 'r':
|
||||
reads += 1
|
||||
(path.parent / 'changing.txt').write_text('x' * reads, encoding='utf-8')
|
||||
return original(path, *args, **kwargs)
|
||||
|
||||
@@ -7,6 +7,7 @@ import subprocess
|
||||
import pytest
|
||||
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
from dx_productivity.trial_task_center.path_access import strip_extended_prefix
|
||||
from dx_productivity.trial_task_center.scanner import scan_projects
|
||||
|
||||
|
||||
@@ -58,7 +59,7 @@ def test_scan_projects_stops_at_selected_project_directory_when_it_contains_prj_
|
||||
real_scandir = os.scandir
|
||||
|
||||
def reject_nested(path: str | bytes | int) -> object:
|
||||
if Path(path) != root:
|
||||
if Path(strip_extended_prefix(path)) != root:
|
||||
raise AssertionError(f"unexpected nested scan: {path}")
|
||||
return real_scandir(path)
|
||||
|
||||
@@ -84,7 +85,7 @@ def test_scan_projects_reads_batch_root_once_before_recursive_fallback(
|
||||
|
||||
def count_root(path: str | bytes | int) -> object:
|
||||
nonlocal root_reads
|
||||
if Path(path) == root:
|
||||
if Path(strip_extended_prefix(path)) == root:
|
||||
root_reads += 1
|
||||
return real_scandir(path)
|
||||
|
||||
@@ -138,7 +139,7 @@ def test_scan_projects_turns_inaccessible_subtree_errors_into_explicit_task_erro
|
||||
real_scandir = __import__("os").scandir
|
||||
|
||||
def deny(path: str | bytes | int) -> object:
|
||||
if Path(path) == denied:
|
||||
if Path(strip_extended_prefix(path)) == denied:
|
||||
raise PermissionError("denied subtree")
|
||||
return real_scandir(path)
|
||||
|
||||
|
||||
@@ -161,3 +161,39 @@ def test_native_failure_does_not_reuse_previous_attempt_transaction(prepared, tm
|
||||
assert report['stage'] == 'preflight' and 'transaction' not in report
|
||||
assert report['batch_attempt']['job_id'] == 'job-a'
|
||||
assert attempts.read_attempt(path)['state'] == 'prepared'
|
||||
|
||||
def test_attempt_write_handles_long_unicode_evidence_and_keeps_canonical_identity(tmp_path, monkeypatch):
|
||||
from pathlib import Path
|
||||
directory = tmp_path / ('long-' + 'x' * 110) / ('more-' + 'y' * 80)
|
||||
path = directory / 'attempt.json'
|
||||
real_mkdir = Path.mkdir
|
||||
def strict_mkdir(target, *args, **kwargs):
|
||||
if len(str(target)) >= 260 and not str(target).startswith(chr(92) * 2 + '?'):
|
||||
raise OSError('legacy MAX_PATH')
|
||||
return real_mkdir(target, *args, **kwargs)
|
||||
monkeypatch.setattr(Path, 'mkdir', strict_mkdir)
|
||||
project = tmp_path / 'board.prj'
|
||||
record = attempts.prepare_attempt(path, session_id='session', generation=1,
|
||||
task_id='task', job_id='job', project_path=project)
|
||||
assert record['project_path'] == os.path.normcase(str(project.resolve()))
|
||||
assert attempts.can_retry_attempt(path, record)
|
||||
assert not str(record['project_path']).startswith(chr(92) * 2 + '?')
|
||||
|
||||
|
||||
def test_publication_retries_sharing_violation_only_and_keeps_original_record(tmp_path, monkeypatch):
|
||||
path = tmp_path / 'attempt.json'
|
||||
first = attempts.prepare_attempt(path, session_id='session', generation=1,
|
||||
task_id='task', job_id='job', project_path=tmp_path / 'board.prj')
|
||||
calls = []
|
||||
def fail(source, target):
|
||||
calls.append((source, target))
|
||||
error = PermissionError('sharing violation')
|
||||
error.winerror = 32
|
||||
raise error
|
||||
monkeypatch.setattr(os, 'replace', fail)
|
||||
with pytest.raises(PermissionError):
|
||||
attempts.prepare_attempt(path, session_id='session', generation=2,
|
||||
task_id='task', job_id='job', project_path=tmp_path / 'board.prj')
|
||||
assert len(calls) == 3
|
||||
assert attempts.read_attempt(path) == first
|
||||
assert list(tmp_path.iterdir()) == [path]
|
||||
@@ -199,7 +199,7 @@ def test_possible_mutation_prevents_automatic_replay(tmp_path, monkeypatch):
|
||||
calls = []
|
||||
def monitor(*args, **kwargs):
|
||||
calls.append(1)
|
||||
path = next((Path(command[3]) / 'batch-attempts').rglob('job.json'))
|
||||
path = next((runtime.path.parent / 'attempts').rglob('job.json'))
|
||||
record = json.loads(path.read_text())
|
||||
record['state'] = 'mutation_possible'
|
||||
path.write_text(json.dumps(record))
|
||||
@@ -228,7 +228,7 @@ def test_previous_checkpoint_invalidated_before_new_worker(tmp_path, monkeypatch
|
||||
raise WorkerMonitorError('cannot certify empty project')
|
||||
runtime.cad.checkpoint = fail_checkpoint
|
||||
# No retry after evidence has been marked as possibly modified.
|
||||
path = next((Path(command[3]) / 'batch-attempts').rglob('job.json'))
|
||||
path = next((runtime.path.parent / 'attempts').rglob('job.json'))
|
||||
record = json.loads(path.read_text())
|
||||
record['state'] = 'mutation_possible'
|
||||
path.write_text(json.dumps(record))
|
||||
@@ -247,6 +247,9 @@ def test_attempt_publication_failure_never_launches_worker(tmp_path, monkeypatch
|
||||
with runtime, pytest.raises(OSError):
|
||||
runtime.run_worker(command, 5)
|
||||
assert not Cad.created
|
||||
assert not Desktop.created
|
||||
from dx_productivity.trial_task_center.batch_resources import pending_resources
|
||||
assert pending_resources(runtime.root) == ()
|
||||
|
||||
|
||||
def test_boundary_pause_does_not_abort_current_cad_startup(tmp_path, monkeypatch):
|
||||
@@ -364,6 +367,69 @@ def test_native_rebuild_requires_strict_source_and_worker_proof(tmp_path):
|
||||
assert len(Desktop.created) == 2
|
||||
|
||||
|
||||
def test_mock_native_read_recovery_spends_same_project_worker_retry_budget(tmp_path, monkeypatch):
|
||||
"""Real durable proof/runtime logic; desktop, CAD and worker failure are fakes."""
|
||||
import hashlib
|
||||
from dx_productivity.trial_task_center.worker_monitor import WorkerMonitorError
|
||||
command, project = command_for(tmp_path)
|
||||
database = project.parent / 'database.dat'
|
||||
database.write_bytes(b'unchanged database')
|
||||
evidence = Path(command[3]) / 'native'
|
||||
evidence.mkdir()
|
||||
calls = []
|
||||
def fail_worker(*args, **kwargs):
|
||||
calls.append(1)
|
||||
raise WorkerMonitorError('prewrite failure after native recovery')
|
||||
runtime = worker_runtime(tmp_path, monkeypatch, fail_worker)
|
||||
with runtime:
|
||||
with runtime.native_scope(evidence) as scope:
|
||||
pass
|
||||
proof = {'version': 1, 'native_read_only': True, 'workers_closed': True,
|
||||
'binding': scope.identity, 'project_path': str(project.resolve()),
|
||||
'evidence_root': str(evidence.resolve()),
|
||||
'source_hashes': {str(path.resolve()): hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
for path in (project, database)}}
|
||||
(evidence / 'native-retry-proof.json').write_text(json.dumps(proof))
|
||||
assert runtime.recover_native_failure(project, evidence, 'read timeout')
|
||||
assert len(Desktop.created) == 2 and Desktop.created[0].released
|
||||
with pytest.raises(WorkerMonitorError, match='after native recovery'):
|
||||
runtime.run_worker(command, 5)
|
||||
assert calls == [1]
|
||||
assert len(Desktop.created) == 2
|
||||
assert not runtime.recover_native_failure(project, evidence, 'another read timeout')
|
||||
assert all(desktop.released for desktop in Desktop.created)
|
||||
assert runtime.record['retries'][str(project.resolve()).casefold()]['count'] == 1
|
||||
|
||||
|
||||
def test_mock_different_projects_each_get_one_retry_without_accumulating_budget(tmp_path, monkeypatch):
|
||||
"""Real per-project accounting; fake CAD/desktop and unchanged prewrite failures."""
|
||||
import subprocess
|
||||
from dx_productivity.trial_task_center.worker_monitor import WorkerMonitorError
|
||||
one, project_one = command_for(tmp_path)
|
||||
two, project_two = command_for(tmp_path, 'two')
|
||||
calls = {one[3]: 0, two[3]: 0}
|
||||
def monitor(command, *args, **kwargs):
|
||||
calls[command[3]] += 1
|
||||
if calls[command[3]] == 2:
|
||||
return subprocess.CompletedProcess(command, 0, '', '')
|
||||
raise WorkerMonitorError('unchanged prewrite failure')
|
||||
runtime = worker_runtime(tmp_path, monkeypatch, monitor)
|
||||
with runtime:
|
||||
runtime.run_worker(one, 5)
|
||||
assert len(Desktop.created) == 2
|
||||
runtime.run_worker(two, 5)
|
||||
assert len(Desktop.created) == 3 # Each project gets its own single rebuild.
|
||||
for command in (one, two):
|
||||
with pytest.raises(WorkerMonitorError, match='prewrite failure'):
|
||||
runtime.run_worker(command, 5)
|
||||
assert len(Desktop.created) == 3 # Success cannot replenish either budget.
|
||||
assert calls == {one[3]: 3, two[3]: 3}
|
||||
assert set(runtime.record['retries']) == {
|
||||
str(project_one.resolve()).casefold(), str(project_two.resolve()).casefold()}
|
||||
assert all(value['count'] == 1 for value in runtime.record['retries'].values())
|
||||
assert all(desktop.released for desktop in Desktop.created)
|
||||
|
||||
|
||||
def test_release_publication_failure_can_be_reconciled_without_replaying(tmp_path, monkeypatch):
|
||||
from dx_productivity.trial_task_center import batch_runtime
|
||||
runtime = batch_runtime.BatchCadRuntime(tmp_path, 'execution', desktop_factory=Desktop)
|
||||
@@ -498,3 +564,45 @@ def test_identity_conflict_never_rebuilds_or_launches_another_worker(tmp_path, m
|
||||
assert len(workers) == (1 if conflict.startswith('checkpoint') or conflict == 'worker_identity' else 0)
|
||||
assert runtime.generation == 1 and not runtime.record['retries']
|
||||
assert Desktop.created[0].released
|
||||
|
||||
|
||||
def test_request_session_identity_is_uuid_bound(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_runtime import BatchCadRuntime
|
||||
session = '12345678-1234-5678-1234-567812345678'
|
||||
runtime = BatchCadRuntime(tmp_path, 'bom_import', session_id=session)
|
||||
assert runtime.session_id == session.replace('-', '')
|
||||
with pytest.raises(ValueError):
|
||||
BatchCadRuntime(tmp_path, 'bom_import', session_id='../escape')
|
||||
|
||||
|
||||
def test_cleanup_preserves_primary_bom_error_and_report(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_runtime import BatchCadRuntime
|
||||
primary = TaskCenterError('bom_inspection', 'read failed')
|
||||
primary.bom_report = object()
|
||||
Desktop.fail_release = True
|
||||
with pytest.raises(TaskCenterError) as caught:
|
||||
with BatchCadRuntime(tmp_path, 'bom_import', desktop_factory=Desktop) as runtime:
|
||||
with runtime.native_scope(tmp_path / 'operation'):
|
||||
pass
|
||||
raise primary
|
||||
error = caught.value
|
||||
assert error.primary_error is primary
|
||||
assert error.cleanup_error is not None
|
||||
assert error.bom_report is primary.bom_report
|
||||
assert 'BOM' in error.message and '已核验任务结果保留' not in error.message
|
||||
assert runtime.record['primary_error']['code'] == primary.code
|
||||
assert runtime.record['cleanup_error']
|
||||
Desktop.fail_release = False
|
||||
from dx_productivity.trial_task_center.batch_resources import cleanup_pending_resources
|
||||
cleanup_pending_resources(tmp_path)
|
||||
|
||||
|
||||
def test_attempts_live_under_short_session_root(tmp_path, monkeypatch):
|
||||
import subprocess
|
||||
command, _ = command_for(tmp_path)
|
||||
runtime = worker_runtime(tmp_path, monkeypatch, lambda command, *a, **k:
|
||||
subprocess.CompletedProcess(command, 0, '', ''))
|
||||
with runtime:
|
||||
runtime.run_worker(command, 5)
|
||||
assert list((runtime.path.parent / 'attempts').rglob('job.json'))
|
||||
assert not (Path(command[3]) / 'batch-attempts').exists()
|
||||
@@ -0,0 +1,278 @@
|
||||
from pathlib import Path
|
||||
import os
|
||||
import json
|
||||
import subprocess
|
||||
import pytest
|
||||
|
||||
|
||||
def test_supervisor_runs_in_other_process_and_releases_gate(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import run_supervised, assert_supervisor_idle
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
store = TaskStore(tmp_path/'task-center.db')
|
||||
try:
|
||||
service = TaskCenterService(store, configure_logging(tmp_path))
|
||||
result = run_supervised(service, 'probe', {})
|
||||
assert result['pid'] != os.getpid()
|
||||
assert result['resource_state'] == 'not_acquired'
|
||||
assert_supervisor_idle(tmp_path)
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
def test_mutation_guard_refuses_owned_launch_gate(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import supervisor_lease, assert_supervisor_idle
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
with supervisor_lease(tmp_path):
|
||||
with pytest.raises(TaskCenterError, match='supervisor_busy'):
|
||||
assert_supervisor_idle(tmp_path)
|
||||
|
||||
|
||||
def test_supervisor_deep_root_uses_compatible_process_working_directory(tmp_path, monkeypatch):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import run_supervised
|
||||
from dx_productivity.trial_task_center.isolated_desktop import _Win32Api
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.path_access import canonical_path
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
root = tmp_path
|
||||
while len(str(root)) < 270:
|
||||
root = root / '中文深目录abcdef0123456789'
|
||||
root.mkdir(parents=True)
|
||||
actual = _Win32Api.create_process
|
||||
launched = []
|
||||
def create(api, command, desktop, cwd, *args, **kwargs):
|
||||
assert len(str(cwd)) < 260
|
||||
assert canonical_path(cwd) == root
|
||||
launched.append(True)
|
||||
return actual(api, command, desktop, cwd, *args, **kwargs)
|
||||
monkeypatch.setattr(_Win32Api, 'create_process', create)
|
||||
store = TaskStore(root/'task-center.db')
|
||||
try:
|
||||
service = TaskCenterService(store, configure_logging(root))
|
||||
assert run_supervised(service, 'probe', {})['pid'] != os.getpid()
|
||||
assert launched == [True]
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
def test_child_error_returns_original_code_without_acquiring_desktop(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import run_supervised
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
store = TaskStore(tmp_path/'task-center.db')
|
||||
try:
|
||||
service = TaskCenterService(store, configure_logging(tmp_path))
|
||||
with pytest.raises(TaskCenterError) as caught:
|
||||
run_supervised(service, 'inspect_bom_batch', {'task_id':'missing','path':tmp_path/'absent.xlsx'})
|
||||
assert caught.value.code != 'supervisor_protocol'
|
||||
assert not list((tmp_path/'batch-resources').glob('*/session.json'))
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
def test_completed_worker_request_cannot_be_replayed(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import run_supervised, _command, _environment
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
store = TaskStore(tmp_path/'task-center.db')
|
||||
try:
|
||||
service = TaskCenterService(store, configure_logging(tmp_path))
|
||||
run_supervised(service, 'probe', {})
|
||||
request = next((tmp_path/'batch-resources').glob('*/request.json'))
|
||||
claim = request.with_name('claim.json').read_bytes()
|
||||
result = request.with_name('result.json').read_bytes()
|
||||
duplicate = subprocess.run(_command(request), env=_environment(), capture_output=True, timeout=30,
|
||||
creationflags=subprocess.CREATE_NO_WINDOW)
|
||||
assert duplicate.returncode != 0
|
||||
assert request.with_name('claim.json').read_bytes() == claim
|
||||
assert request.with_name('result.json').read_bytes() == result
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
def test_missing_write_result_preserves_protection_marker(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import _mark_interrupted
|
||||
journal = {'version': 1, 'batch_id': 'a'*32, 'attempt_id':'b'*32,
|
||||
'operation':'execute_task_queue', 'task_ids':['task'], 'state':'running'}
|
||||
_mark_interrupted(tmp_path, journal, 'worker exited unexpectedly')
|
||||
marker = tmp_path/'tasks/task/quick-run/uncertain.flag'
|
||||
assert marker.exists()
|
||||
marker.write_text('original stronger reason')
|
||||
_mark_interrupted(tmp_path, journal, 'another cleanup failure')
|
||||
assert marker.read_text() == 'original stronger reason'
|
||||
|
||||
|
||||
def test_cancelled_request_does_not_launch_supervisor(tmp_path):
|
||||
from threading import Event
|
||||
from types import SimpleNamespace
|
||||
from dx_productivity.trial_task_center.batch_supervisor import run_supervised
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
stop = Event(); stop.set()
|
||||
with pytest.raises(TaskCenterError, match='operation_cancelled'):
|
||||
run_supervised(SimpleNamespace(_logging=SimpleNamespace(root=tmp_path)), 'probe', {}, stop_event=stop)
|
||||
assert not (tmp_path/'batch-resources').exists()
|
||||
|
||||
|
||||
def test_startup_recovery_holds_admission_until_runner_initialized(tmp_path, monkeypatch):
|
||||
from dx_productivity.trial_task_center import service as module
|
||||
from dx_productivity.trial_task_center.batch_supervisor import assert_supervisor_idle
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
checked = []
|
||||
def runner(*args, **kwargs):
|
||||
with pytest.raises(TaskCenterError, match='supervisor_busy'):
|
||||
assert_supervisor_idle(tmp_path)
|
||||
checked.append(True)
|
||||
return object()
|
||||
monkeypatch.setattr(module, 'TrialRunner', runner)
|
||||
store = TaskStore(tmp_path/'task-center.db')
|
||||
try:
|
||||
module.TaskCenterService(store, configure_logging(tmp_path))
|
||||
assert checked == [True]
|
||||
assert_supervisor_idle(tmp_path)
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
def test_crashed_legacy_task_protects_actual_root_without_replay(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import _mark_interrupted
|
||||
journal = {'version': 1, 'batch_id': 'a'*32, 'attempt_id': 'b'*32,
|
||||
'operation': 'execute_task_queue', 'task_ids': ['legacy'], 'state': 'running',
|
||||
'uncertainty_roots': {'legacy': str(tmp_path)}}
|
||||
_mark_interrupted(tmp_path, journal, 'missing final result')
|
||||
assert (tmp_path/'quick-run/uncertain.flag').exists()
|
||||
assert not (tmp_path/'tasks/legacy/quick-run/uncertain.flag').exists()
|
||||
|
||||
|
||||
def test_parent_job_cleanup_failure_preserves_result_and_can_retry_resources_only(tmp_path, monkeypatch):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import run_supervised
|
||||
from dx_productivity.trial_task_center.isolated_desktop import _Win32Api
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
original = _Win32Api.create_process
|
||||
injected = []
|
||||
def create(api, *args, **kwargs):
|
||||
process = original(api, *args, **kwargs)
|
||||
if not injected:
|
||||
injected.append(process)
|
||||
close = process.close
|
||||
attempts = []
|
||||
def interrupted_close():
|
||||
if not attempts:
|
||||
attempts.append(1)
|
||||
raise OSError('injected parent Job close failure')
|
||||
close()
|
||||
process.close = interrupted_close
|
||||
return process
|
||||
monkeypatch.setattr(_Win32Api, 'create_process', create)
|
||||
store = TaskStore(tmp_path/'task-center.db')
|
||||
try:
|
||||
service = TaskCenterService(store, configure_logging(tmp_path), supervised=True)
|
||||
with pytest.raises(TaskCenterError) as caught:
|
||||
run_supervised(service, 'probe', {})
|
||||
assert caught.value.code == 'batch_resources_pending'
|
||||
assert caught.value.completed_result['pid'] != os.getpid()
|
||||
saved = next((tmp_path/'batch-resources').glob('*/business-result.json'))
|
||||
before = saved.read_bytes()
|
||||
assert service.cleanup_batch_resources() == ()
|
||||
assert saved.read_bytes() == before
|
||||
assert not service.pending_batch_resources()
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.mark.parametrize('bound_task', [False, True], ids=['legacy-root', 'project-task-root'])
|
||||
@pytest.mark.parametrize('saved_success', [False, True], ids=['unknown-write', 'saved-business-success'])
|
||||
def test_real_supervisor_process_crash_reconciles_without_replaying_mock_business(
|
||||
tmp_path, monkeypatch, bound_task, saved_success):
|
||||
"""Real child admission, hard exit, result proof and cleanup; no CAD/desktop.
|
||||
|
||||
Only the child business method is replaced with a durable witness write.
|
||||
This tests process-crash orchestration, not actual schematic write safety.
|
||||
"""
|
||||
import sys
|
||||
import textwrap
|
||||
from dx_productivity.trial_task_center import batch_supervisor as supervisor
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
|
||||
store = TaskStore(tmp_path / 'task-center.db')
|
||||
try:
|
||||
service = TaskCenterService(store, configure_logging(tmp_path), supervised=True)
|
||||
task = store.create_task('crash fixture')
|
||||
if bound_task:
|
||||
project_path = tmp_path / 'fixture.prj'
|
||||
project_path.write_text('placeholder; never opened in CAD')
|
||||
project = store.sync_projects(task.task_id, tmp_path, (project_path,))[0]
|
||||
store.bind_project_task(task.task_id, project.project_id, 'crash-fixture', 0)
|
||||
evidence = service._project_task_service(task.task_id)._quick_run_preparer.data_root
|
||||
witness = evidence / 'quick-run/crash-business-witness.txt'
|
||||
previous = supervisor.run_supervised(service, 'probe', {})
|
||||
previous_files = {path: path.read_bytes() for path in
|
||||
(tmp_path / 'batch-resources').glob('*/*result.json')}
|
||||
original_command = supervisor._command
|
||||
bootstrap = textwrap.dedent('''
|
||||
import os, sys
|
||||
from pathlib import Path
|
||||
from dx_productivity.trial_task_center import batch_worker, batch_resources
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.quick_run_protocol import QuickRunStepResult
|
||||
saved_success = sys.argv[2] == 'yes'
|
||||
def business(self, task_id, **kwargs):
|
||||
root = self._project_task_service(task_id)._quick_run_preparer.data_root / 'quick-run'
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
with (root / 'crash-business-witness.txt').open('a', encoding='utf-8') as stream:
|
||||
stream.write(str(os.getpid()) + '\\n')
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
if not saved_success:
|
||||
os._exit(73)
|
||||
return QuickRunStepResult(root/'worker.vbs', root/'command.tsv', root/'status.tsv',
|
||||
root/'events.log', 'restored_ok', 'mock business completed')
|
||||
TaskCenterService.execute_quick_run = business
|
||||
original_write = batch_resources.write_record
|
||||
def write_then_crash(path, value):
|
||||
original_write(path, value)
|
||||
if Path(path).name == 'business-result.json':
|
||||
os._exit(74)
|
||||
batch_resources.write_record = write_then_crash
|
||||
batch_worker.run(Path(sys.argv[1]))
|
||||
''')
|
||||
def command(request):
|
||||
if json.loads(request.read_text(encoding='utf-8'))['operation'] == 'execute_quick_run':
|
||||
return (sys.executable, '-c', bootstrap, str(request), 'yes' if saved_success else 'no')
|
||||
return original_command(request)
|
||||
monkeypatch.setattr(supervisor, '_command', command)
|
||||
if saved_success:
|
||||
result = supervisor.run_supervised(service, 'execute_quick_run', {'task_id': task.task_id})
|
||||
assert result.state == 'restored_ok'
|
||||
else:
|
||||
with pytest.raises(TaskCenterError) as caught:
|
||||
supervisor.run_supervised(service, 'execute_quick_run', {'task_id': task.task_id})
|
||||
assert caught.value.code == 'supervisor_interrupted'
|
||||
executions = witness.read_text().splitlines()
|
||||
assert len(executions) == 1 and int(executions[0]) not in {os.getpid(), previous['pid']}
|
||||
marker = evidence / 'quick-run/uncertain.flag'
|
||||
assert marker.exists() is (not saved_success)
|
||||
other_root = tmp_path if bound_task else tmp_path / 'tasks' / task.task_id
|
||||
assert not (other_root / 'quick-run/uncertain.flag').exists()
|
||||
protection = marker.read_bytes() if marker.exists() else None
|
||||
assert not list((tmp_path / 'batch-resources').glob('*/session.json'))
|
||||
# A new real cleanup supervisor performs resource reconciliation only.
|
||||
assert service.cleanup_batch_resources() == ()
|
||||
assert not service.pending_batch_resources()
|
||||
assert witness.read_text().splitlines() == executions
|
||||
assert (marker.read_bytes() if marker.exists() else None) == protection
|
||||
assert all(path.read_bytes() == value for path, value in previous_files.items())
|
||||
finally:
|
||||
store.close()
|
||||
@@ -0,0 +1,104 @@
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from dx_productivity.trial_task_center.path_access import internal_path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def deep_root(tmp_path, monkeypatch):
|
||||
if os.name != 'nt':
|
||||
pytest.skip('real Windows long-path regression')
|
||||
root = tmp_path / ('design-' + 'x' * 100) / ('revision-' + 'y' * 100)
|
||||
assert len(str(root)) > 260
|
||||
# Exercise real Windows files while reproducing a host with MAX_PATH-bound
|
||||
# nonextended access, regardless of this developer machine's policy.
|
||||
originals = {name: getattr(Path, name) for name in ('mkdir', 'open', 'stat')}
|
||||
for name, original in originals.items():
|
||||
def strict(path, *args, _original=original, **kwargs):
|
||||
if len(str(path)) >= 260 and not str(path).startswith(chr(92) * 2 + '?'):
|
||||
raise OSError(206, 'legacy MAX_PATH access')
|
||||
return _original(path, *args, **kwargs)
|
||||
monkeypatch.setattr(Path, name, strict)
|
||||
return root
|
||||
|
||||
|
||||
def test_deep_task_event_keeps_public_root_and_rotates_readable_events(deep_root, monkeypatch):
|
||||
from dx_productivity.trial_task_center import logging_setup
|
||||
context = logging_setup.configure_logging(deep_root)
|
||||
monkeypatch.setattr(logging_setup, '_MAX_EVENT_LOG_BYTES', 1)
|
||||
try:
|
||||
for operation in ('first', 'second', 'third'):
|
||||
logging_setup.append_task_event(context, level='INFO', task_id='task', operation=operation)
|
||||
path = deep_root / 'tasks/task/events.jsonl'
|
||||
assert context.root == deep_root
|
||||
assert context.application_log_path == deep_root / 'logs/application.log'
|
||||
assert json.loads(internal_path(path).read_text(encoding='utf-8'))['sequence'] == 3
|
||||
assert json.loads(internal_path(path.with_name('events.jsonl.1')).read_text(encoding='utf-8'))['sequence'] == 2
|
||||
finally:
|
||||
for handler in tuple(context.logger.handlers):
|
||||
handler.close()
|
||||
context.logger.removeHandler(handler)
|
||||
|
||||
|
||||
def test_deep_quick_run_prepare_publish_read_archive_keeps_canonical_results(deep_root, tmp_path):
|
||||
from dx_productivity.trial_task_center import quick_run
|
||||
from dx_productivity.trial_task_center.models import ProjectRecord, ProjectStatus, RuleRecord
|
||||
source = tmp_path / 'worker.vbs'
|
||||
source.write_text('runtime', encoding='utf-8')
|
||||
adapter = source.with_name('NativeSnapshotReader.vbs')
|
||||
adapter.write_text('adapter', encoding='utf-8')
|
||||
project = tmp_path / 'board.prj'
|
||||
project.touch()
|
||||
record = ProjectRecord('project', 'task', 'board', project, tmp_path, True, True, ProjectStatus.PENDING, 0)
|
||||
rule = RuleRecord('rule', 'task', 0, 'OLD', 'NEW', True)
|
||||
preparer = quick_run.QuickRunPreparer(deep_root, script_source=source, run_token_factory=lambda: 'run')
|
||||
handoff = preparer.prepare('task', (record,), (rule,))
|
||||
assert preparer.data_root == deep_root
|
||||
assert handoff.command_path == deep_root / 'quick-run/command.tsv'
|
||||
assert internal_path(handoff.script_path).read_text(encoding='utf-8') == 'runtime\nadapter'
|
||||
assert quick_run._read_commands(handoff.command_path)[0][2] == str(project)
|
||||
assert preparer.read_snapshot('task', (record,), (rule,)).evidence_path == deep_root / 'quick-run'
|
||||
statuses = (('run_0', 'no_match', '', 'done'),)
|
||||
quick_run._write_status_details(handoff.status_path, statuses)
|
||||
assert quick_run._read_status_details(handoff.status_path) == statuses
|
||||
quick_run._write_atomic_queue_text(handoff.command_path.parent / 'current-state.tsv', 'published', 'test', 'test')
|
||||
quick_run._write_uncertain_flag(handoff.command_path.parent / 'uncertain.flag')
|
||||
assert quick_run._queue_has_mutation_evidence(handoff.command_path.parent)
|
||||
archive = preparer.archive_current_handoff('task')
|
||||
assert archive.parent == deep_root / 'quick-run-history'
|
||||
assert quick_run._read_commands(archive / 'command.tsv')[0][2] == str(project)
|
||||
assert quick_run._read_status_details(archive / 'status.tsv') == statuses
|
||||
|
||||
assert preparer.read_snapshot('task', (record,), (rule,)).evidence_path == archive
|
||||
|
||||
def test_deep_native_recovery_keeps_canonical_proof_and_release_logs(deep_root):
|
||||
import hashlib
|
||||
from dx_productivity.trial_task_center.batch_runtime import BatchCadRuntime
|
||||
from test_batch_runtime import Desktop
|
||||
Desktop.created = []
|
||||
Desktop.fail_release = False
|
||||
project = deep_root / 'source/board.prj'
|
||||
database = project.with_name('icdb.dat')
|
||||
internal_path(project.parent).mkdir(parents=True)
|
||||
internal_path(project).write_bytes(b'project')
|
||||
internal_path(database).write_bytes(b'database')
|
||||
evidence = deep_root / 'quick-run-history/native-read'
|
||||
internal_path(evidence).mkdir(parents=True)
|
||||
with BatchCadRuntime(deep_root, 'bom_import', desktop_factory=Desktop) as owner:
|
||||
with owner.native_scope(evidence) as scope:
|
||||
pass
|
||||
proof = {'version': 1, 'native_read_only': True, 'workers_closed': True,
|
||||
'binding': scope.identity, 'project_path': str(project), 'evidence_root': str(evidence),
|
||||
'source_hashes': {str(path): hashlib.sha256(internal_path(path).read_bytes()).hexdigest()
|
||||
for path in (project, database)}}
|
||||
internal_path(evidence / 'native-retry-proof.json').write_text(json.dumps(proof), encoding='utf-8')
|
||||
assert owner.recover_native_failure(project, evidence, 'timeout')
|
||||
assert not owner.recover_native_failure(project, evidence, 'second timeout')
|
||||
assert all(desktop.released for desktop in Desktop.created)
|
||||
assert not owner.record.get('release_log_errors')
|
||||
release = owner.path.with_name('resource-release.log')
|
||||
assert 'resource-release' in str(release)
|
||||
assert 'released' in internal_path(release).read_text(encoding='utf-8')
|
||||
@@ -0,0 +1,80 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from dx_productivity.trial_task_center.path_access import internal_path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def deep_windows_tree(tmp_path, monkeypatch):
|
||||
if os.name != 'nt':
|
||||
pytest.skip('real Windows long paths')
|
||||
root = tmp_path / ('projects-' + 'x' * 105) / ('revision-' + 'y' * 100)
|
||||
assert len(str(root)) > 260
|
||||
internal_path(root).mkdir(parents=True)
|
||||
def legacy_failure(path):
|
||||
return len(os.fspath(path)) >= 260 and not os.fspath(path).startswith(chr(92) * 2 + '?')
|
||||
for name in ('stat', 'open', 'resolve'):
|
||||
original = getattr(Path, name)
|
||||
def strict(path, *args, _original=original, **kwargs):
|
||||
if legacy_failure(path):
|
||||
raise FileNotFoundError(2, 'MAX_PATH host cannot access unprefixed deep path')
|
||||
return _original(path, *args, **kwargs)
|
||||
monkeypatch.setattr(Path, name, strict)
|
||||
original_scandir = os.scandir
|
||||
def scandir(path):
|
||||
if not isinstance(path, int) and legacy_failure(path):
|
||||
raise FileNotFoundError(2, 'MAX_PATH host cannot scan unprefixed deep path')
|
||||
return original_scandir(path)
|
||||
monkeypatch.setattr(os, 'scandir', scandir)
|
||||
return root
|
||||
|
||||
|
||||
def test_deep_schematic_scan_keeps_canonical_projects_exclusions_and_progress(deep_windows_tree):
|
||||
from dx_productivity.trial_task_center.scanner import scan_projects
|
||||
from dx_productivity.trial_task_center.project_tasks import scan_project_directory, _validate_selected
|
||||
root = deep_windows_tree
|
||||
project = root / 'selected/board.prj'
|
||||
excluded = root / 'excluded'
|
||||
for path in (project, excluded / 'hidden.prj'):
|
||||
internal_path(path.parent).mkdir(parents=True, exist_ok=True)
|
||||
internal_path(path).write_text('KEY FrontEndSnapshot "DxD"\nKEY iCDBDir "database"\n', encoding='utf-8')
|
||||
assert scan_projects(root, excluded_roots=(excluded,)) == (project,)
|
||||
progress = []
|
||||
assert scan_project_directory(root, excluded_roots=(excluded,), on_progress=lambda directory, *_: progress.append(directory)) == (project,)
|
||||
assert progress and all(path.is_relative_to(root) for path in progress)
|
||||
assert _validate_selected(root, (project,), (excluded,)) == (project,)
|
||||
|
||||
|
||||
def test_deep_orphan_uncertainty_flag_retains_task_without_handoff(deep_windows_tree, tmp_path, monkeypatch):
|
||||
from types import SimpleNamespace
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.task_archive import archive_completed_session
|
||||
store = TaskStore(tmp_path / 'tasks.db')
|
||||
context = configure_logging(tmp_path / 'logs')
|
||||
service = TaskCenterService(store, context)
|
||||
task = service.create_task('protected')
|
||||
root = deep_windows_tree
|
||||
marker = root / 'quick-run/uncertain.flag'
|
||||
internal_path(marker.parent).mkdir(parents=True)
|
||||
internal_path(marker).write_text('needs verification', encoding='utf-8')
|
||||
monkeypatch.setattr(service, 'pending_batch_resources', lambda: ())
|
||||
monkeypatch.setattr(service, 'has_interrupted_quick_run', lambda task_id: False)
|
||||
facade = SimpleNamespace(_quick_run_preparer=SimpleNamespace(data_root=root))
|
||||
monkeypatch.setattr(service, '_project_task_service', lambda task_id: facade)
|
||||
try:
|
||||
report = archive_completed_session(service)
|
||||
assert report.archived_ids == ()
|
||||
assert report.retained_ids == (task.task_id,)
|
||||
assert [item.task_id for item in service.tasks()] == [task.task_id]
|
||||
assert internal_path(marker).read_text(encoding='utf-8') == 'needs verification'
|
||||
assert not internal_path(marker.with_name('command.tsv')).exists()
|
||||
finally:
|
||||
store.close()
|
||||
for handler in tuple(context.logger.handlers):
|
||||
handler.close()
|
||||
context.logger.removeHandler(handler)
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import zipfile
|
||||
|
||||
import pytest
|
||||
|
||||
from dx_productivity.trial_task_center import diagnostics_bundle as bundle
|
||||
|
||||
|
||||
def test_bundle_allowlist_redacts_secrets_and_home_and_omits_inputs(tmp_path):
|
||||
root = tmp_path / 'data'
|
||||
session = root / 'batch-resources' / 'batch1'
|
||||
session.mkdir(parents=True)
|
||||
(root / 'logs').mkdir()
|
||||
(root / 'logs/application.log.1').write_text('user=C:\\Users\\Alice token=super-secret password="a b"', encoding='utf-8')
|
||||
(session / 'session.json').write_text(json.dumps({'pid': 123, 'api_key': 'hidden', 'input': 'C:\\Users\\Alice\\work\\bom.xlsx'}))
|
||||
(session / 'source.xlsx').write_bytes(b'private BOM')
|
||||
(root / 'task-center.db').write_bytes(b'private DB')
|
||||
target = tmp_path / 'support.zip'
|
||||
assert bundle.export_diagnostic_bundle(root, target) == target
|
||||
with zipfile.ZipFile(target) as archive:
|
||||
text = '\n'.join(archive.read(name).decode('utf-8') for name in archive.namelist())
|
||||
assert 'super-secret' not in text and 'hidden' not in text and 'a b' not in text
|
||||
assert 'Alice' not in text and 'private BOM' not in text and 'private DB' not in text
|
||||
assert json.loads(archive.read('batch-resources/batch1/session.json'))['pid'] == 123
|
||||
manifest = json.loads(archive.read('manifest.json'))
|
||||
assert manifest['omissions'] and manifest['unknowns']
|
||||
with pytest.raises(FileExistsError):
|
||||
bundle.export_diagnostic_bundle(root, target)
|
||||
|
||||
|
||||
def test_bundle_rejects_non_zip_destination(tmp_path):
|
||||
with pytest.raises(ValueError):
|
||||
bundle.export_diagnostic_bundle(tmp_path, tmp_path / 'output.txt')
|
||||
|
||||
|
||||
def test_bundle_does_not_follow_links_or_export_binary_logs(tmp_path):
|
||||
root = tmp_path / 'data'
|
||||
(root / 'logs').mkdir(parents=True)
|
||||
(root / 'logs/binary.log').write_bytes(b'\x00\xff\x00')
|
||||
external = tmp_path / 'secret.log'
|
||||
external.write_text('outside data root')
|
||||
try:
|
||||
(root / 'logs/link.log').symlink_to(external)
|
||||
except OSError:
|
||||
pass
|
||||
output = bundle.export_diagnostic_bundle(root, tmp_path / 'support.zip')
|
||||
with zipfile.ZipFile(output) as archive:
|
||||
assert 'logs/binary.log' not in archive.namelist()
|
||||
assert 'logs/link.log' not in archive.namelist()
|
||||
|
||||
|
||||
def test_bundle_redacts_nested_client_secret_and_preserves_support_ids(tmp_path):
|
||||
root = tmp_path / 'data'
|
||||
directory = root / 'attempts'
|
||||
directory.mkdir(parents=True)
|
||||
(directory / 'attempt.json').write_text(json.dumps({'client_secret': 'oauth-private',
|
||||
'support_id': 'DXT-123', 'username': 'ExternalUser',
|
||||
'nested': {'authorization': 'Bearer a.b.c'}}))
|
||||
output = bundle.export_diagnostic_bundle(root, tmp_path / 'support.zip')
|
||||
with zipfile.ZipFile(output) as archive:
|
||||
text = archive.read('attempts/attempt.json').decode()
|
||||
assert 'oauth-private' not in text and 'a.b.c' not in text
|
||||
assert 'ExternalUser' not in text
|
||||
assert 'DXT-123' in text
|
||||
|
||||
|
||||
def test_bundle_excludes_hardlinks_and_redacts_jsonl_credentials(tmp_path):
|
||||
import os
|
||||
root = tmp_path / 'data'
|
||||
logs = root / 'logs'
|
||||
logs.mkdir(parents=True)
|
||||
external = tmp_path / 'private.log'
|
||||
external.write_text('private content')
|
||||
os.link(external, logs / 'linked.log')
|
||||
tasks = root / 'tasks' / 'task1'
|
||||
tasks.mkdir(parents=True)
|
||||
(tasks / 'events.jsonl').write_text(json.dumps({'token': 'private-token'}) + '\n')
|
||||
output = bundle.export_diagnostic_bundle(root, tmp_path / 'support.zip')
|
||||
with zipfile.ZipFile(output) as archive:
|
||||
assert 'logs/linked.log' not in archive.namelist()
|
||||
assert 'private-token' not in archive.read('tasks/task1/events.jsonl').decode()
|
||||
assert json.loads(archive.read('tasks/task1/events.jsonl'))['token'] == '<REDACTED>'
|
||||
|
||||
|
||||
def test_bundle_size_limit_omits_oversized_files(tmp_path, monkeypatch):
|
||||
(tmp_path / 'logs').mkdir()
|
||||
(tmp_path / 'logs/application.log').write_text('too large for configured limit')
|
||||
monkeypatch.setattr(bundle, 'MAX_FILE_BYTES', 4)
|
||||
output = bundle.export_diagnostic_bundle(tmp_path, tmp_path / 'support.zip')
|
||||
with zipfile.ZipFile(output) as archive:
|
||||
assert 'logs/application.log' not in archive.namelist()
|
||||
assert any(row['reason'] == 'size_or_count_limit' for row in json.loads(archive.read('manifest.json'))['omissions'])
|
||||
|
||||
|
||||
def test_bundle_includes_real_nested_attempt_queue_and_protection_locations(tmp_path):
|
||||
names = ['batch-resources/batch1/attempts/g1/attempt1/job1.json',
|
||||
'task-queues/queue1.json', 'tasks/task1/quick-run/project-checkpoint.json',
|
||||
'quick-run-history/recovery1/uncertain.flag']
|
||||
for name in names:
|
||||
path = tmp_path / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text('{}')
|
||||
output = bundle.export_diagnostic_bundle(tmp_path, tmp_path / 'support.zip')
|
||||
with zipfile.ZipFile(output) as archive:
|
||||
assert set(names).issubset(archive.namelist())
|
||||
|
||||
|
||||
def test_plain_log_authorization_redacts_bearer_value(tmp_path):
|
||||
(tmp_path / 'logs').mkdir()
|
||||
(tmp_path / 'logs/application.log').write_text('Authorization=Bearer private-bearer-value\n')
|
||||
output = bundle.export_diagnostic_bundle(tmp_path, tmp_path / 'support.zip')
|
||||
with zipfile.ZipFile(output) as archive:
|
||||
assert 'private-bearer-value' not in archive.read('logs/application.log').decode()
|
||||
@@ -1,10 +1,12 @@
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import weakref
|
||||
from pathlib import Path
|
||||
from threading import Event
|
||||
from threading import Timer
|
||||
from threading import Thread
|
||||
from types import SimpleNamespace
|
||||
from traceback import extract_tb
|
||||
|
||||
@@ -194,6 +196,121 @@ def test_clean_release_records_helper_and_cad_exit_codes(monkeypatch, tmp_path):
|
||||
assert record['cleanup_errors'] == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize('persistent', [False, True])
|
||||
def test_closing_journal_failure_still_quits_and_reports_diagnostic_error(monkeypatch, tmp_path, persistent):
|
||||
events, cad = fake_desktop(monkeypatch, tmp_path)
|
||||
session = runtime.IsolatedCadSession(tmp_path)
|
||||
session.__enter__()
|
||||
record = session._record
|
||||
def failing_record(**values):
|
||||
if persistent or values.get('state') == 'closing':
|
||||
raise PermissionError('closing journal locked')
|
||||
record(**values)
|
||||
session._record = failing_record
|
||||
with pytest.raises(WorkerMonitorError, match='closing journal locked'):
|
||||
session.__exit__(None, None, None)
|
||||
assert 'quit' in events and cad.returncode == 0
|
||||
assert session.record['orderly_shutdown'] is True
|
||||
assert session.record['clean'] is False
|
||||
assert session.record['diagnostic_errors'][0]['stage'] == 'closing_record'
|
||||
assert (tmp_path / 'uncertain.flag').exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize('helper_failure', ['refused', 'timeout'])
|
||||
def test_interactive_quit_failure_retains_cad_and_helper_until_explicit_force(monkeypatch, tmp_path, helper_failure):
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
from dx_productivity.trial_task_center import worker_monitor
|
||||
events, cad = fake_desktop(monkeypatch, tmp_path, quit_code=1, cad_exit_code=None)
|
||||
control = StopControl()
|
||||
session = runtime.IsolatedCadSession(tmp_path, stop_control=control)
|
||||
session.__enter__()
|
||||
start = session.desktop.start_process
|
||||
def start_quit(command, **options):
|
||||
helper = start(command, **options)
|
||||
if command[0] == 'quit':
|
||||
if helper_failure == 'timeout':
|
||||
helper.wait = lambda timeout: (_ for _ in ()).throw(subprocess.TimeoutExpired('quit', timeout))
|
||||
helper.poll = lambda: None
|
||||
helper.close = lambda: events.append('close-quit-helper')
|
||||
return helper
|
||||
session.desktop.start_process = start_quit
|
||||
waiting, finished = Event(), Event()
|
||||
publish = worker_monitor._publish_progress
|
||||
def observe_wait(path, snapshot):
|
||||
publish(path, snapshot)
|
||||
if snapshot.get('stage') == 'awaiting_reconciliation':
|
||||
waiting.set()
|
||||
monkeypatch.setattr(worker_monitor, '_publish_progress', observe_wait)
|
||||
failures = []
|
||||
def close_session():
|
||||
try:
|
||||
session.__exit__(None, None, None)
|
||||
except Exception as failure:
|
||||
failures.append(failure)
|
||||
finally:
|
||||
finished.set()
|
||||
thread = Thread(target=close_session)
|
||||
thread.start()
|
||||
try:
|
||||
assert waiting.wait(1), 'failed Quit must enter protected waiting before resource release'
|
||||
control.set()
|
||||
assert not finished.wait(.05)
|
||||
assert cad.poll() is None
|
||||
assert 'close-quit-helper' not in events
|
||||
assert not any(isinstance(event, tuple) and event[0] == 'release' for event in events)
|
||||
assert (tmp_path / 'uncertain.flag').exists()
|
||||
control.request_force()
|
||||
assert finished.wait(2)
|
||||
assert 'close-quit-helper' in events and ('release', True) in events
|
||||
assert cad.poll() == 1
|
||||
assert isinstance(failures[0], WorkerMonitorError)
|
||||
assert 'forced' in str(failures[0])
|
||||
assert session.record['orderly_shutdown'] is False and session.record['clean'] is False
|
||||
finally:
|
||||
control.request_force()
|
||||
thread.join(3)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('failure', ['cad_exited', 'diagnostic'])
|
||||
def test_interactive_exit_does_not_hold_completed_cad_for_resolution(monkeypatch, tmp_path, failure):
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
from dx_productivity.trial_task_center import stop_control
|
||||
events, _cad = fake_desktop(monkeypatch, tmp_path, quit_code=1 if failure == 'cad_exited' else 0)
|
||||
session = runtime.IsolatedCadSession(tmp_path, stop_control=StopControl())
|
||||
session.__enter__()
|
||||
monkeypatch.setattr(stop_control, 'wait_for_resolution', lambda *_args: pytest.fail('exited CAD must not wait'))
|
||||
if failure == 'diagnostic':
|
||||
record = session._record
|
||||
def failed_journal(**values):
|
||||
if values.get('state') == 'closing':
|
||||
raise PermissionError('closing diagnostic unavailable')
|
||||
record(**values)
|
||||
session._record = failed_journal
|
||||
with pytest.raises(WorkerMonitorError):
|
||||
session.__exit__(None, None, None)
|
||||
assert ('release', True) in events
|
||||
|
||||
|
||||
def test_forced_descendant_cleanup_is_not_reported_as_clean_orderly_shutdown(monkeypatch, tmp_path):
|
||||
fake_desktop(monkeypatch, tmp_path)
|
||||
session = runtime.IsolatedCadSession(tmp_path)
|
||||
session.__enter__()
|
||||
release = session.desktop.close
|
||||
def forced_release():
|
||||
release()
|
||||
session.desktop.cleanup_record = {'processes': [{'pid': 123, 'forced_termination': True,
|
||||
'forced_root_termination': False, 'termination_reason': 'descendant_grace_expired',
|
||||
'forced_descendant_pids': [456]}]}
|
||||
session.desktop.close = forced_release
|
||||
with pytest.raises(WorkerMonitorError, match='forced'):
|
||||
session.__exit__(None, None, None)
|
||||
record = json.loads((tmp_path / 'private-desktop.json').read_text())
|
||||
assert record['clean'] is False and record['forced_termination'] is True
|
||||
assert record['orderly_shutdown'] is True
|
||||
assert record['forced_descendant_pids'] == [456]
|
||||
assert record['termination_reason'] == 'descendant_grace_expired'
|
||||
|
||||
|
||||
def test_exit_code_diagnostics_do_not_interrupt_the_normal_cad_wait(monkeypatch, tmp_path):
|
||||
_, cad = fake_desktop(monkeypatch, tmp_path)
|
||||
waited = []
|
||||
@@ -236,6 +353,26 @@ def test_slow_startup_is_bounded_and_cleanup_remains_owned(monkeypatch, tmp_path
|
||||
assert ('release', False) in events
|
||||
|
||||
|
||||
def test_owned_cad_exit_does_not_wait_for_unresponsive_startup_probe(monkeypatch, tmp_path):
|
||||
events, cad = fake_desktop(monkeypatch, tmp_path)
|
||||
clock = [0.0]
|
||||
monkeypatch.setattr(runtime.time, 'monotonic', lambda: clock[0])
|
||||
monkeypatch.setattr(runtime.time, 'sleep', lambda seconds: clock.__setitem__(0, clock[0] + seconds))
|
||||
original = runtime.PrivateDesktop.start_process
|
||||
def start(self, command, **kwargs):
|
||||
process = original(self, command, **kwargs)
|
||||
if command[0] == 'probe':
|
||||
cad.returncode = 73
|
||||
process.poll = lambda: None
|
||||
return process
|
||||
monkeypatch.setattr(runtime.PrivateDesktop, 'start_process', start)
|
||||
with pytest.raises(runtime.PrivateCadStartupError, match='exited during startup.*73'):
|
||||
with runtime.IsolatedCadSession(tmp_path):
|
||||
pytest.fail('exited CAD must never enter project work')
|
||||
assert clock[0] < 1
|
||||
assert events.index('close-helper') < events.index(('release', False))
|
||||
|
||||
|
||||
def test_repeated_session_preserves_prior_release_evidence(monkeypatch, tmp_path):
|
||||
fake_desktop(monkeypatch, tmp_path)
|
||||
with runtime.IsolatedCadSession(tmp_path) as first:
|
||||
|
||||
@@ -68,6 +68,75 @@ def test_startup_info_always_targets_private_desktop():
|
||||
assert info.lpDesktop == 'winsta0\\dx-private-test'
|
||||
|
||||
|
||||
def test_desktop_records_native_handle_before_close_and_failed_retry():
|
||||
class Desktop:
|
||||
fail = True
|
||||
def __int__(self):
|
||||
return 761
|
||||
def CloseDesktop(self):
|
||||
if self.fail:
|
||||
self.fail = False
|
||||
raise OSError('busy')
|
||||
handle = Desktop()
|
||||
desktop = PrivateDesktop('private', enabled=True,
|
||||
api=SimpleNamespace(create_desktop=lambda _: handle))
|
||||
desktop.__enter__()
|
||||
with pytest.raises(TaskCenterError):
|
||||
desktop.close()
|
||||
desktop.close()
|
||||
history = desktop.cleanup_record['desktop_handle_history']
|
||||
assert [event['operation'] for event in history] == ['create', 'close', 'close']
|
||||
assert [event['success'] for event in history] == [True, False, True]
|
||||
assert all(event['handle'] == 761 and event['pid'] == os.getpid() and event['thread_id'] for event in history)
|
||||
|
||||
|
||||
def test_desktop_preserves_forgotten_worker_history_and_unknown_holder_scope():
|
||||
api, _, state, *_ = native_api()
|
||||
api.confirm_desktop_released = lambda _: None
|
||||
desktop = PrivateDesktop('private', enabled=True, api=api)
|
||||
desktop._handle = SimpleNamespace(CloseDesktop=lambda: None)
|
||||
process = desktop.start_process(('python.exe',))
|
||||
process.close()
|
||||
desktop.forget_closed_process(process)
|
||||
desktop.close()
|
||||
history = desktop.cleanup_record['process_history']
|
||||
assert history[0]['pid'] == 42 and history[0]['cleanup']['process_handle_closed']
|
||||
assert history[0]['forgotten'] is True
|
||||
assert desktop.cleanup_record['diagnostic_scope']['external_handle_holders'] == 'unknown'
|
||||
|
||||
|
||||
def test_failed_release_reports_windowless_holder_unknown_and_owned_job_failure():
|
||||
api, _, _, *_ = native_api()
|
||||
api._pending_cleanup = []
|
||||
def confirm(_):
|
||||
api.desktop_release_check = {'released': False, 'remaining_windows': []}
|
||||
raise OSError('still exists')
|
||||
api.confirm_desktop_released = confirm
|
||||
desktop = PrivateDesktop('private', enabled=True, api=api)
|
||||
desktop._handle = SimpleNamespace(CloseDesktop=lambda: None)
|
||||
process = desktop.start_process(('python.exe',))
|
||||
process.close = lambda: (_ for _ in ()).throw(OSError('owned handles retained'))
|
||||
with pytest.raises(TaskCenterError):
|
||||
desktop.close()
|
||||
report = desktop.cleanup_record['owned_resource_diagnostics']
|
||||
assert report['residual_windows'] == []
|
||||
assert report['external_handle_holders'] == 'unknown'
|
||||
assert report['thread_desktops']['complete'] is False
|
||||
assert report['owned_processes'][0]['pid'] == 42
|
||||
assert report['owned_processes'][0]['job_accounting']['ActiveProcesses'] == 0
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name != 'nt', reason='native Windows resource history')
|
||||
def test_real_helper_retains_completed_lifecycle_after_forget():
|
||||
with PrivateDesktop('dx-test-' + uuid4().hex, enabled=True) as desktop:
|
||||
process = desktop.start_process((sys.executable, '-c', 'pass'))
|
||||
assert process.wait(timeout=5) == 0
|
||||
process.close()
|
||||
desktop.forget_closed_process(process)
|
||||
assert desktop.cleanup_record['process_history'][0]['pid'] == process.pid
|
||||
assert desktop.cleanup_record['desktop_handle_history'][0]['handle'] > 0
|
||||
|
||||
|
||||
def test_private_desktop_never_falls_back_to_input_desktop(monkeypatch):
|
||||
desktop = PrivateDesktop('dx-private-test', enabled=True)
|
||||
monkeypatch.setattr(desktop, '_create', lambda: None)
|
||||
@@ -298,6 +367,99 @@ def test_descendant_handles_are_owned_before_terminating_the_job():
|
||||
assert ('close', 'child') in events
|
||||
|
||||
|
||||
@pytest.mark.parametrize('child_exit_after', [0.5, None])
|
||||
def test_completed_root_gives_owned_descendants_bounded_natural_exit_grace(monkeypatch, child_exit_after):
|
||||
from dx_productivity.trial_task_center import isolated_desktop as module
|
||||
api, events, state, *_ = native_api()
|
||||
clock = [0.0]
|
||||
monkeypatch.setattr(module.time, 'monotonic', lambda: clock[0])
|
||||
monkeypatch.setattr(module.time, 'sleep', lambda seconds: clock.__setitem__(0, clock[0] + seconds))
|
||||
state['running'] = False
|
||||
child = Handle('child', events)
|
||||
child_running = lambda: not any(event[0] == 'terminate' for event in events) and (
|
||||
child_exit_after is None or clock[0] < child_exit_after)
|
||||
api.job.JobObjectBasicProcessIdList = 3
|
||||
api.job.QueryInformationJobObject = lambda job, kind: (
|
||||
{'BasicLimitInformation': {'LimitFlags': 0}} if kind == 9 else
|
||||
([43] if child_running() else []) if kind == 3 else {'ActiveProcesses': int(child_running())})
|
||||
api.open_process = lambda *_args: child
|
||||
root_wait = api.event.WaitForSingleObject
|
||||
api.event.WaitForSingleObject = lambda handle, timeout: (
|
||||
258 if child_running() else 0) if handle is child else root_wait(handle, timeout)
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
process.close()
|
||||
assert child.closed
|
||||
if child_exit_after is not None:
|
||||
assert child_exit_after <= clock[0] < 2
|
||||
assert ('terminate',) not in events
|
||||
assert process.cleanup_record['forced_termination'] is False
|
||||
assert process.cleanup_record['forced_descendant_pids'] == []
|
||||
else:
|
||||
assert 2 <= clock[0] < 2.1
|
||||
assert ('terminate',) in events
|
||||
assert process.cleanup_record['forced_termination'] is True
|
||||
assert process.cleanup_record['forced_root_termination'] is False
|
||||
assert process.cleanup_record['forced_descendant_pids'] == [43]
|
||||
assert process.cleanup_record['termination_reason'] == 'descendant_grace_expired'
|
||||
|
||||
|
||||
def test_completed_root_without_descendants_has_no_grace_delay(monkeypatch):
|
||||
from dx_productivity.trial_task_center import isolated_desktop as module
|
||||
api, events, state, *_ = native_api()
|
||||
state['running'] = False
|
||||
monkeypatch.setattr(module.time, 'sleep', lambda seconds: pytest.fail('empty job must not wait'))
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
process.close()
|
||||
assert ('terminate',) not in events
|
||||
assert process.cleanup_record['forced_termination'] is False
|
||||
|
||||
|
||||
def test_active_root_cleanup_is_forced_without_descendant_grace(monkeypatch):
|
||||
from dx_productivity.trial_task_center import isolated_desktop as module
|
||||
api, events, *_ = native_api()
|
||||
monkeypatch.setattr(module.time, 'sleep', lambda seconds: pytest.fail('active root must be bounded immediately'))
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
process.close()
|
||||
assert process.cleanup_record['forced_termination'] is True
|
||||
assert process.cleanup_record['forced_root_termination'] is True
|
||||
assert process.cleanup_record['termination_reason'] == 'root_process_active'
|
||||
|
||||
|
||||
def test_native_poll_error_still_terminates_owned_root_and_records_force():
|
||||
api, events, state, *_ = native_api()
|
||||
wait = api.event.WaitForSingleObject
|
||||
def failing_poll(handle, timeout):
|
||||
if timeout == 0 and state['running']:
|
||||
raise OSError('native poll failed')
|
||||
return wait(handle, timeout)
|
||||
api.event.WaitForSingleObject = failing_poll
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
with pytest.raises(TaskCenterError, match='native poll failed'):
|
||||
process.close()
|
||||
assert state['running'] is False
|
||||
assert process.cleanup_record['forced_root_termination'] is True
|
||||
|
||||
|
||||
def test_failed_job_cleanup_retains_ownership_until_retry(monkeypatch):
|
||||
from dx_productivity.trial_task_center import isolated_desktop as module
|
||||
api, events, state, root, job = native_api()
|
||||
clock = [0.0]
|
||||
monkeypatch.setattr(module.time, 'monotonic', lambda: clock[0])
|
||||
monkeypatch.setattr(module.time, 'sleep', lambda seconds: clock.__setitem__(0, clock[0] + seconds))
|
||||
api.job.TerminateJobObject = lambda *_args: (_ for _ in ()).throw(OSError('job termination failed'))
|
||||
query = api.job.QueryInformationJobObject
|
||||
child_running = [True]
|
||||
api.job.QueryInformationJobObject = lambda handle, kind: (
|
||||
{'ActiveProcesses': int(state['running']) + int(child_running[0])} if kind == 1 else query(handle, kind))
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
with pytest.raises(TaskCenterError, match='descendants did not exit'):
|
||||
process.close()
|
||||
assert not job.closed and process.job is job
|
||||
child_running[0] = False
|
||||
process.close()
|
||||
assert job.closed
|
||||
|
||||
|
||||
@pytest.mark.parametrize('error_code', [87, 5])
|
||||
def test_native_descendant_open_error_only_ignores_already_exited_process(error_code):
|
||||
# pywintypes.error derives directly from Exception, rather than OSError.
|
||||
@@ -322,6 +484,30 @@ def test_native_descendant_open_error_only_ignores_already_exited_process(error_
|
||||
assert caught.value is error
|
||||
|
||||
|
||||
def test_forced_job_preserves_partial_current_descendant_snapshot_after_open_failure():
|
||||
api, events, state, *_ = native_api()
|
||||
api.job.JobObjectBasicProcessIdList = 3
|
||||
query = api.job.QueryInformationJobObject
|
||||
snapshot = [45]
|
||||
api.job.QueryInformationJobObject = lambda job, kind: list(snapshot) if kind == 3 else query(job, kind)
|
||||
def open_child(_access, _inherit, pid):
|
||||
if pid == 44:
|
||||
raise OSError(5, 'child access denied')
|
||||
return Handle('child-' + str(pid), events)
|
||||
api.open_process = open_child
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
# PID 45 was observed earlier but is no longer part of the active Job.
|
||||
process._remember_descendants()
|
||||
snapshot[:] = [42, 43, 44]
|
||||
with pytest.raises(TaskCenterError, match='child access denied'):
|
||||
process.close()
|
||||
assert state['running'] is False
|
||||
assert process.cleanup_record['forced_termination'] is True
|
||||
assert process.cleanup_record['forced_descendant_pids'] == [43]
|
||||
assert process.cleanup_record['descendant_pids'] == [43, 45]
|
||||
assert process.cleanup_record['job_handle_closed'] is True
|
||||
|
||||
|
||||
def test_unconfirmed_descendant_exit_retains_handle_for_cleanup_retry():
|
||||
api, events, state, *_ = native_api()
|
||||
process = api.create_process(('python.exe',), 'winsta0\\private', None, None)
|
||||
@@ -361,9 +547,37 @@ def test_native_private_job_kills_descendant_after_root_exit(parent_exits):
|
||||
if parent_exits:
|
||||
assert process.wait(timeout=5) == 0
|
||||
assert win32event.WaitForSingleObject(child_handle, 0) == 0
|
||||
assert process.cleanup_record['forced_termination'] is True
|
||||
assert process.cleanup_record['forced_root_termination'] is (not parent_exits)
|
||||
assert child_pid in process.cleanup_record['forced_descendant_pids']
|
||||
with pytest.raises(pywintypes.error):
|
||||
unexpected = win32service.OpenDesktop(name, 0, False, 1)
|
||||
unexpected.CloseDesktop()
|
||||
finally:
|
||||
if child_handle is not None:
|
||||
child_handle.Close()
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.name != 'nt', reason='native Windows descendant shutdown grace')
|
||||
def test_native_private_job_allows_descendant_to_finish_after_root_exit():
|
||||
import win32api
|
||||
import win32event
|
||||
import win32process
|
||||
|
||||
child_handle = None
|
||||
try:
|
||||
with PrivateDesktop('dx-test-' + uuid4().hex, enabled=True) as desktop:
|
||||
process = desktop.start_process((sys.executable, '-c',
|
||||
"import subprocess,sys; p=subprocess.Popen([sys.executable,'-c','import time;time.sleep(0.5)']); print(p.pid,flush=True)"),
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
child_pid = int(process.stdout.readline())
|
||||
child_handle = win32api.OpenProcess(0x100000 | 0x1000, False, child_pid)
|
||||
assert process.wait(timeout=5) == 0
|
||||
assert win32event.WaitForSingleObject(child_handle, 0) == 0
|
||||
assert win32process.GetExitCodeProcess(child_handle) == 0
|
||||
assert process.cleanup_record['forced_termination'] is False
|
||||
assert process.cleanup_record['forced_descendant_pids'] == []
|
||||
assert desktop.cleanup_record['desktop_confirmation']['released'] is True
|
||||
finally:
|
||||
if child_handle is not None:
|
||||
child_handle.Close()
|
||||
@@ -231,6 +231,8 @@ def test_native_cleanup_failure_blocks_fallback_and_keeps_recovery_evidence(tmp_
|
||||
with pytest.raises(TaskCenterError) as caught:
|
||||
reader.read(project, ('OLD',), stop_event=stop)
|
||||
assert caught.value.code == 'inventory_cleanup'
|
||||
assert caught.value.primary_error.code == 'operation_cancelled'
|
||||
assert isinstance(caught.value.cleanup_error, OSError)
|
||||
assert released == [True] and list((tmp_path / 'evidence').rglob('uncertain.flag'))
|
||||
assert list((tmp_path / 'temporary').iterdir())
|
||||
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
import pytest
|
||||
|
||||
|
||||
def test_windows_extended_paths_keep_canonical_identity():
|
||||
from dx_productivity.trial_task_center.path_access import windows_extended_path, strip_extended_prefix
|
||||
slash = chr(92)
|
||||
local = 'C:' + slash + ('nested' + slash) * 45 + 'file.json'
|
||||
assert windows_extended_path(local) == slash * 2 + '?' + slash + local
|
||||
assert strip_extended_prefix(windows_extended_path(local)) == local
|
||||
unc = slash * 2 + 'server' + slash + 'share' + slash + 'file.json'
|
||||
assert windows_extended_path(unc) == slash * 2 + '?' + slash + 'UNC' + unc[1:]
|
||||
assert strip_extended_prefix(windows_extended_path(unc)) == unc
|
||||
|
||||
|
||||
def test_sharing_retry_is_bounded_and_does_not_retry_permission_denied():
|
||||
from dx_productivity.trial_task_center.path_access import retry_sharing_violation
|
||||
calls = []
|
||||
def operation():
|
||||
calls.append(1)
|
||||
error = PermissionError('sharing violation')
|
||||
error.winerror = 32
|
||||
raise error
|
||||
with pytest.raises(PermissionError):
|
||||
retry_sharing_violation(operation, sleep=lambda delay: None)
|
||||
assert len(calls) == 3
|
||||
calls.clear()
|
||||
def denied():
|
||||
calls.append(1)
|
||||
raise PermissionError('access denied')
|
||||
with pytest.raises(PermissionError):
|
||||
retry_sharing_violation(denied, sleep=lambda delay: None)
|
||||
assert len(calls) == 1
|
||||
|
||||
def test_discovery_uses_extended_filesystem_access_before_legacy_launch(tmp_path, monkeypatch):
|
||||
import os
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from dx_productivity.trial_task_center.project_discovery import discover_project, DiscoveryRuntime
|
||||
if os.name != 'nt':
|
||||
pytest.skip('Windows filesystem boundary')
|
||||
data = tmp_path / ('nested-' + 'a' * 90) / ('more-' + 'b' * 70)
|
||||
project = tmp_path / 'board.prj'
|
||||
project.write_text('project')
|
||||
script = tmp_path / 'worker.vbs'
|
||||
script.write_text('script')
|
||||
real_mkdir = Path.mkdir
|
||||
def strict_mkdir(path, *args, **kwargs):
|
||||
if len(str(path)) >= 260 and not str(path).startswith(chr(92) * 2 + '?'):
|
||||
raise OSError('legacy MAX_PATH filesystem access')
|
||||
return real_mkdir(path, *args, **kwargs)
|
||||
monkeypatch.setattr(Path, 'mkdir', strict_mkdir)
|
||||
reached = []
|
||||
def launch(*args):
|
||||
reached.append(1)
|
||||
raise subprocess.SubprocessError('stop at launch')
|
||||
runtime = DiscoveryRuntime(launch, lambda: Path('cscript'), lambda s, r: (str(s), str(r)),
|
||||
lambda: (), str, lambda *a: (), lambda *a: {}, 5)
|
||||
with pytest.raises(Exception):
|
||||
discover_project(project, ('OLD',), data_root=data, script_source=script, runtime=runtime)
|
||||
assert reached, 'long-path filesystem preparation failed before worker boundary'
|
||||
|
||||
def test_native_short_path_never_passes_extended_prefix_to_wsh(tmp_path):
|
||||
import os
|
||||
from dx_productivity.trial_task_center.legacy_paths import _short_path
|
||||
if os.name != 'nt':
|
||||
pytest.skip('Windows short-path boundary')
|
||||
result = _short_path(tmp_path)
|
||||
if result is None:
|
||||
pytest.skip('filesystem does not provide short paths')
|
||||
assert not result.startswith(chr(92) * 2 + '?')
|
||||
@@ -0,0 +1,61 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import pytest
|
||||
|
||||
|
||||
def test_new_process_cache_instance_reuses_only_verified_identity(tmp_path):
|
||||
from dx_productivity.trial_task_center.discovery_cache import DiscoveryCache
|
||||
from dx_productivity.trial_task_center.quick_run_protocol import ProjectDiscovery
|
||||
project = tmp_path/'project/demo.prj'
|
||||
project.parent.mkdir()
|
||||
project.write_text('project')
|
||||
root = tmp_path/'data'
|
||||
discovery = ProjectDiscovery((('main','1'),), {('main','1'):frozenset({'OLD'})})
|
||||
calls = []
|
||||
def loader():
|
||||
calls.append(1)
|
||||
return discovery
|
||||
assert DiscoveryCache(root).discover(project, ('OLD',), False, loader) == discovery
|
||||
assert DiscoveryCache(root).discover(project, ('OLD',), False, loader) == discovery
|
||||
assert len(calls) == 1
|
||||
project.write_text('changed')
|
||||
DiscoveryCache(root).discover(project, ('OLD',), False, loader)
|
||||
assert len(calls) == 2
|
||||
|
||||
|
||||
def test_corrupted_disk_cache_is_quarantined_and_read_once(tmp_path):
|
||||
from dx_productivity.trial_task_center.discovery_cache import DiscoveryCache
|
||||
from dx_productivity.trial_task_center.quick_run_protocol import ProjectDiscovery
|
||||
project = tmp_path/'p/demo.prj';project.parent.mkdir();project.write_text('project')
|
||||
root=tmp_path/'data'
|
||||
discovery=ProjectDiscovery((('main','1'),),{('main','1'):frozenset({'OLD'})})
|
||||
DiscoveryCache(root).discover(project,('OLD',),False,lambda:discovery)
|
||||
paths=list((root/'discovery-cache').glob('*.json'))
|
||||
assert len(paths)==1
|
||||
paths[0].write_text('{broken')
|
||||
calls=[]
|
||||
def loader():
|
||||
calls.append(1);return discovery
|
||||
assert DiscoveryCache(root).discover(project,('OLD',),False,loader)==discovery
|
||||
assert calls==[1]
|
||||
assert list((root/'discovery-cache').glob('*.bad-*'))
|
||||
|
||||
|
||||
def test_corrupted_cache_coverage_cannot_authorize_incomplete_reuse(tmp_path):
|
||||
from dx_productivity.trial_task_center.discovery_cache import DiscoveryCache
|
||||
from dx_productivity.trial_task_center.quick_run_protocol import ProjectDiscovery
|
||||
project = tmp_path/'p/demo.prj'; project.parent.mkdir(); project.write_text('project')
|
||||
root = tmp_path/'data'
|
||||
discovery = ProjectDiscovery((('main','1'),), {('main','1'): frozenset({'OLD'})})
|
||||
DiscoveryCache(root).discover(project, ('OLD',), False, lambda: discovery)
|
||||
path = next((root/'discovery-cache').glob('*.json'))
|
||||
value = json.loads(path.read_text())
|
||||
value['parts'].append('other')
|
||||
path.write_text(json.dumps(value))
|
||||
calls = []
|
||||
def loader():
|
||||
calls.append(1)
|
||||
return discovery
|
||||
DiscoveryCache(root).discover(project, ('OTHER',), False, loader)
|
||||
assert calls == [1]
|
||||
assert list((root/'discovery-cache').glob('*.bad-*'))
|
||||
@@ -12,7 +12,8 @@ def test_project_access_resolves_each_distinct_input_once_per_preparation(tmp_pa
|
||||
(tmp_path/'command.tsv').write_text('project_path\n'+(str(project)+'\n')*1000,encoding='utf8')
|
||||
original=Path.resolve;calls=[]
|
||||
def resolve(path,*args,**kwargs):
|
||||
if path==project:calls.append(path)
|
||||
from dx_productivity.trial_task_center.path_access import strip_extended_prefix
|
||||
if Path(strip_extended_prefix(path))==project:calls.append(path)
|
||||
return original(path,*args,**kwargs)
|
||||
monkeypatch.setattr(Path,'resolve',resolve)
|
||||
legacy_paths._prepare_project_access(tmp_path)
|
||||
@@ -38,7 +39,9 @@ def test_existing_log_directories_avoid_mkdir_and_recreate_after_removal(tmp_pat
|
||||
directory=tmp_path/'tasks/task';directory.mkdir(parents=True)
|
||||
original=Path.mkdir;calls=[]
|
||||
def mkdir(path,*args,**kwargs):
|
||||
if path==directory:calls.append(path)
|
||||
from dx_productivity.trial_task_center.path_access import strip_extended_prefix
|
||||
canonical = Path(strip_extended_prefix(path))
|
||||
if canonical==directory:calls.append(canonical)
|
||||
return original(path,*args,**kwargs)
|
||||
monkeypatch.setattr(Path,'mkdir',mkdir)
|
||||
assert logging_setup._task_directory(context,'task')==directory
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
from threading import Event
|
||||
|
||||
from PySide6.QtGui import QCloseEvent
|
||||
from PySide6.QtWidgets import QMessageBox
|
||||
|
||||
from test_ui_remediation import window
|
||||
from test_ui_batch_retry import open_project
|
||||
|
||||
|
||||
def test_pause_and_window_close_never_request_force(window, qtbot, tmp_path):
|
||||
open_project(window, qtbot, tmp_path)
|
||||
window._begin_run(window._task_id)
|
||||
worker = object()
|
||||
window._workers.add(worker)
|
||||
try:
|
||||
event = QCloseEvent()
|
||||
window.closeEvent(event)
|
||||
assert not event.isAccepted()
|
||||
assert window._batch_stop.is_set()
|
||||
assert not window._batch_stop.force_event.is_set()
|
||||
assert not window._closed
|
||||
finally:
|
||||
window._workers.discard(worker)
|
||||
window._close_pending = False
|
||||
window._finish_run()
|
||||
|
||||
|
||||
def test_force_requires_confirmation_and_keeps_window_for_reconciliation(window, qtbot, tmp_path, monkeypatch):
|
||||
open_project(window, qtbot, tmp_path)
|
||||
window._begin_run(window._task_id)
|
||||
control = window._batch_stop
|
||||
try:
|
||||
monkeypatch.setattr(QMessageBox, 'question', lambda *args: QMessageBox.StandardButton.No)
|
||||
window._force_batch_stop()
|
||||
assert not control.is_set() and not control.force_event.is_set()
|
||||
monkeypatch.setattr(QMessageBox, 'question', lambda *args: QMessageBox.StandardButton.Yes)
|
||||
window._close_pending = True
|
||||
window._force_batch_stop()
|
||||
assert control.is_set() and control.force_event.is_set()
|
||||
assert not window._close_pending
|
||||
finally:
|
||||
window._finish_run()
|
||||
|
||||
|
||||
def test_uncertain_wait_is_visible_in_execution_feedback(tmp_path):
|
||||
import json
|
||||
from dx_productivity.trial_task_center.execution_feedback import read_feedback
|
||||
from dx_productivity.trial_task_center.ui.execution_feedback import format_feedback
|
||||
root = tmp_path / 'quick-run'
|
||||
root.mkdir()
|
||||
(root / 'worker-progress.json').write_text(json.dumps({
|
||||
'stage': 'save_and_close', 'waiting_for_safe_completion': True, 'stage_seconds': {}}))
|
||||
feedback = read_feedback('task', 1, 1, '', tmp_path, 0)
|
||||
text = format_feedback(feedback, 'board', 400, True)
|
||||
assert '尚未确认' in text and '强制' in text
|
||||
|
||||
|
||||
def test_diagnostic_export_during_wait_does_not_finish_or_cancel_execution(window, qtbot, tmp_path, monkeypatch):
|
||||
from PySide6.QtWidgets import QFileDialog
|
||||
open_project(window, qtbot, tmp_path)
|
||||
window._begin_run(window._task_id)
|
||||
control = window._batch_stop
|
||||
owner = object()
|
||||
window._workers.add(owner)
|
||||
destination = tmp_path / 'support.zip'
|
||||
monkeypatch.setattr(QFileDialog, 'getSaveFileName', lambda *args: (str(destination), ''))
|
||||
try:
|
||||
window._export_diagnostics()
|
||||
qtbot.waitUntil(lambda: window._diagnostic_worker is None, timeout=5000)
|
||||
assert destination.is_file()
|
||||
assert owner in window._workers and window._batch_stop is control
|
||||
assert not control.is_set() and not control.force_event.is_set()
|
||||
finally:
|
||||
window._workers.discard(owner)
|
||||
window._finish_run()
|
||||
@@ -0,0 +1,343 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
from threading import Event, Thread
|
||||
|
||||
import pytest
|
||||
|
||||
from dx_productivity.trial_task_center.worker_monitor import ProgressWatch, run_monitored_process
|
||||
|
||||
|
||||
def test_unsafe_save_timeout_waits_for_completion_and_recovers_on_progress():
|
||||
watch = ProgressWatch('token', 'job', 10, 0)
|
||||
assert watch.accept(b'DX_PROGRESS\ttoken\tjob\t1\tsave_and_close\t0\t1\t0\n', 0)
|
||||
assert watch.action(10) == 'request_stop'
|
||||
assert watch.action(20) == 'await_safe_completion'
|
||||
assert watch.snapshot(now=20)['waiting_for_safe_completion'] is True
|
||||
assert watch.action(200) != 'unresponsive'
|
||||
assert watch.accept(b'DX_PROGRESS\ttoken\tjob\t2\tverify_saved_project\t1\t1\t0\n', 201)
|
||||
assert watch.snapshot(now=201)['waiting_for_safe_completion'] is False
|
||||
|
||||
|
||||
def test_real_slow_save_is_not_killed_by_inactivity(tmp_path):
|
||||
script = tmp_path / 'save.py'
|
||||
script.write_text('''import sys,time,pathlib
|
||||
root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5]
|
||||
while not (root/f'monitor-{token}.ready').exists():time.sleep(.01)
|
||||
print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tsave_and_close\\t0\\t1\\t0',flush=True)
|
||||
time.sleep(1.5)
|
||||
(root/'saved.txt').write_text('saved and independently verified')
|
||||
''', encoding='utf-8')
|
||||
command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job')
|
||||
result = run_monitored_process(command, .3)
|
||||
assert result.returncode == 0
|
||||
assert (tmp_path / 'saved.txt').read_text() == 'saved and independently verified'
|
||||
|
||||
|
||||
def test_readonly_native_wait_keeps_bounded_cancellation_without_force_ui(tmp_path):
|
||||
script = tmp_path / 'read.py'
|
||||
script.write_text('''import sys,time,pathlib
|
||||
root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5]
|
||||
while not (root/f'monitor-{token}.ready').exists():time.sleep(.01)
|
||||
print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tnative_snapshot_read\\t0\\t1\\t0',flush=True)
|
||||
time.sleep(1.5)
|
||||
''', encoding='utf-8')
|
||||
command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job')
|
||||
with pytest.raises(subprocess.TimeoutExpired):
|
||||
run_monitored_process(command, .3, cooperative_abort=True)
|
||||
|
||||
|
||||
def test_explicit_force_stops_unsafe_worker_with_protection(tmp_path):
|
||||
script = tmp_path / 'save.py'
|
||||
script.write_text('''import sys,time,pathlib
|
||||
root=pathlib.Path(sys.argv[1]);job=sys.argv[4];token=sys.argv[5]
|
||||
while not (root/f'monitor-{token}.ready').exists():time.sleep(.01)
|
||||
print(f'DX_PROGRESS\\t{token}\\t{job}\\t1\\tsave_and_close\\t0\\t1\\t0',flush=True)
|
||||
(root/'started').touch()
|
||||
time.sleep(30)
|
||||
(root/'unexpected_completed').touch()
|
||||
''', encoding='utf-8')
|
||||
force = Event()
|
||||
def request():
|
||||
for _ in range(200):
|
||||
if (tmp_path / 'started').exists():
|
||||
force.set()
|
||||
return
|
||||
Event().wait(.01)
|
||||
requester = Thread(target=request)
|
||||
requester.start()
|
||||
command = (sys.executable, '-u', str(script), str(tmp_path), sys.executable, '', 'job')
|
||||
try:
|
||||
with pytest.raises(Exception, match='forced'):
|
||||
run_monitored_process(command, .3, force_event=force)
|
||||
assert (tmp_path / 'uncertain.flag').exists()
|
||||
assert not (tmp_path / 'unexpected_completed').exists()
|
||||
finally:
|
||||
requester.join(3)
|
||||
|
||||
|
||||
def test_abnormal_cad_shutdown_is_not_hidden_by_physical_release(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import _finish_response
|
||||
from dx_productivity.trial_task_center.batch_resources import write_record
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.supervisor_protocol import encode, encode_error
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
root = tmp_path
|
||||
journal = {'operation': 'probe', 'batch_id': 'a' * 32, 'state': 'blocked'}
|
||||
write_record(root / 'batch-resources' / journal['batch_id'] / 'session.json', {
|
||||
'version': 1, 'session_id': 'a' * 32, 'kind': 'probe', 'state': 'released',
|
||||
'generation': 1, 'desktop': 'dx-' + 'b' * 32, 'processes': [], 'cad_pid': None,
|
||||
'cad_guid': None, 'checkpoints': {}, 'retries': {}, 'timings': [],
|
||||
'cad_shutdown': {'clean': False, 'cleanup_errors': [{'stage': 'orderly_shutdown', 'message': 'Quit timed out'}]}})
|
||||
response = {'business_succeeded': True, 'status': 'error', 'result': encode({'verified': True}),
|
||||
'error': encode_error(TaskCenterError('batch_resources_pending', 'Quit timed out'))}
|
||||
with pytest.raises(TaskCenterError) as caught:
|
||||
_finish_response(root, journal, response, configure_logging(root))
|
||||
assert caught.value.code == 'application_shutdown_failed'
|
||||
assert caught.value.completed_result == {'verified': True}
|
||||
assert journal['resource_recheck']['pending'] is False
|
||||
|
||||
|
||||
def test_force_cannot_terminate_before_interruption_evidence_is_durable(tmp_path, monkeypatch):
|
||||
from types import SimpleNamespace
|
||||
from dx_productivity.trial_task_center import batch_supervisor as module
|
||||
calls = []
|
||||
process = SimpleNamespace(terminate=lambda: calls.append('terminate'))
|
||||
journal = {'operation': 'execute_task_queue', 'batch_id': 'a' * 32, 'attempt_id': 'b' * 32,
|
||||
'task_ids': ['task'], 'state': 'running'}
|
||||
monkeypatch.setattr(module, '_mark_interrupted', lambda *args: (_ for _ in ()).throw(PermissionError('denied')))
|
||||
with pytest.raises(PermissionError):
|
||||
module._force_owned_supervisor(tmp_path, journal, process)
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_force_already_performed_remains_forced_when_exit_diagnostics_fail(tmp_path):
|
||||
from types import SimpleNamespace
|
||||
from dx_productivity.trial_task_center.batch_supervisor import _force_owned_supervisor
|
||||
record = {}
|
||||
def terminate():
|
||||
record['forced_termination'] = True
|
||||
raise OSError('child enumeration failed after force')
|
||||
process = SimpleNamespace(terminate=terminate, cleanup_record=record)
|
||||
journal = {'operation': 'execute_task_queue', 'batch_id': 'a' * 32, 'attempt_id': 'b' * 32,
|
||||
'task_ids': ['task'], 'state': 'running'}
|
||||
_force_owned_supervisor(tmp_path, journal, process)
|
||||
saved = json.loads((tmp_path / 'batch-resources' / ('a' * 32) / 'supervisor.json').read_text(encoding='utf-8'))
|
||||
assert saved['stop_mode'] == 'forced'
|
||||
assert 'enumeration' in saved['force_cleanup_error']
|
||||
|
||||
|
||||
def test_interruption_targets_active_project_without_invalidating_previous_tasks(tmp_path):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import _mark_interrupted
|
||||
from dx_productivity.trial_task_center.batch_resources import write_record
|
||||
batch = 'a' * 32
|
||||
write_record(tmp_path / 'batch-resources' / batch / 'session.json', {
|
||||
'version': 1, 'session_id': batch, 'kind': 'task_queue', 'state': 'ready',
|
||||
'generation': 1, 'desktop': 'dx-' + 'b' * 32, 'processes': [], 'cad_pid': 123,
|
||||
'cad_guid': 'owned', 'checkpoints': {}, 'retries': {}, 'timings': [], 'active_task': 'current'})
|
||||
journal = {'operation': 'execute_task_queue', 'batch_id': batch, 'attempt_id': 'c' * 32,
|
||||
'task_ids': ['previous', 'current'], 'state': 'running'}
|
||||
_mark_interrupted(tmp_path, journal, 'forced stop')
|
||||
assert (tmp_path / 'tasks/current/quick-run/uncertain.flag').exists()
|
||||
assert not (tmp_path / 'tasks/previous/quick-run/uncertain.flag').exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize('interactive', [False, True])
|
||||
def test_supervisor_only_holds_for_resolution_when_caller_has_force_control(tmp_path, monkeypatch, interactive):
|
||||
import textwrap
|
||||
from dx_productivity.trial_task_center import batch_supervisor as module
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
bootstrap = textwrap.dedent('''
|
||||
import sys,json
|
||||
from pathlib import Path
|
||||
from dx_productivity.trial_task_center import batch_worker,batch_runtime
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
original=batch_runtime.BatchCadRuntime
|
||||
def runtime(*args,**kwargs):
|
||||
result=original(*args,**kwargs)
|
||||
Path(sys.argv[1]).with_name('interactive-witness.json').write_text(json.dumps(isinstance(result.stop_event,StopControl)))
|
||||
return result
|
||||
batch_runtime.BatchCadRuntime=runtime
|
||||
batch_worker.run(Path(sys.argv[1]))
|
||||
''')
|
||||
monkeypatch.setattr(module, '_command', lambda request: (sys.executable, '-c', bootstrap, str(request)))
|
||||
store = TaskStore(tmp_path / 'tasks.db')
|
||||
try:
|
||||
service = TaskCenterService(store, configure_logging(tmp_path))
|
||||
module.run_supervised(service, 'probe', {}, stop_event=StopControl() if interactive else Event())
|
||||
witness = next((tmp_path / 'batch-resources').glob('*/interactive-witness.json'))
|
||||
assert json.loads(witness.read_text()) is interactive
|
||||
finally:
|
||||
store.close()
|
||||
|
||||
|
||||
def test_uncertain_operation_keeps_owner_until_explicit_force(tmp_path):
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl, wait_for_resolution
|
||||
control = StopControl()
|
||||
failures = []
|
||||
def owner():
|
||||
try:
|
||||
wait_for_resolution(control, tmp_path, 'save result unknown')
|
||||
except Exception as error:
|
||||
failures.append(error)
|
||||
thread = Thread(target=owner)
|
||||
thread.start()
|
||||
try:
|
||||
for _ in range(100):
|
||||
if (tmp_path / 'worker-progress.json').exists():
|
||||
break
|
||||
Event().wait(.01)
|
||||
control.set() # Ordinary stop never authorizes discarding a changed project.
|
||||
thread.join(.15)
|
||||
assert thread.is_alive()
|
||||
assert (tmp_path / 'uncertain.flag').exists()
|
||||
assert json.loads((tmp_path / 'worker-progress.json').read_text())['waiting_for_safe_completion']
|
||||
control.request_force()
|
||||
thread.join(3)
|
||||
assert not thread.is_alive()
|
||||
assert 'forced' in str(failures[0])
|
||||
finally:
|
||||
control.request_force()
|
||||
thread.join(3)
|
||||
|
||||
|
||||
def test_failed_protection_fsync_is_retried_before_force_is_allowed(tmp_path, monkeypatch):
|
||||
from dx_productivity.trial_task_center import stop_control as module
|
||||
control = module.StopControl()
|
||||
original = module.os.fsync
|
||||
attempts, failures = [], []
|
||||
def denied(fd):
|
||||
attempts.append(fd)
|
||||
raise OSError('evidence fsync denied')
|
||||
monkeypatch.setattr(module.os, 'fsync', denied)
|
||||
def owner():
|
||||
try:
|
||||
module.wait_for_resolution(control, tmp_path, 'save unknown')
|
||||
except Exception as error:
|
||||
failures.append(error)
|
||||
thread = Thread(target=owner)
|
||||
thread.start()
|
||||
try:
|
||||
for _ in range(100):
|
||||
if control.control_error:
|
||||
break
|
||||
Event().wait(.01)
|
||||
assert control.control_error
|
||||
control.request_force()
|
||||
thread.join(.25)
|
||||
assert thread.is_alive(), 'a file left by failed fsync must not authorize force'
|
||||
assert len(attempts) >= 2
|
||||
monkeypatch.setattr(module.os, 'fsync', original)
|
||||
control.request_force()
|
||||
thread.join(3)
|
||||
assert failures and 'forced' in str(failures[0])
|
||||
finally:
|
||||
monkeypatch.setattr(module.os, 'fsync', original)
|
||||
control.request_force()
|
||||
thread.join(3)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('deny_first,transport_failure', [(False, False), (True, False), (False, True)])
|
||||
def test_real_supervisor_force_preserves_protection_and_never_replays(tmp_path, monkeypatch, deny_first, transport_failure):
|
||||
import os
|
||||
import textwrap
|
||||
import time
|
||||
import win32api
|
||||
import win32event
|
||||
from dx_productivity.trial_task_center import batch_supervisor as module
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
store = TaskStore(tmp_path / 'tasks.db')
|
||||
service = TaskCenterService(store, configure_logging(tmp_path), supervised=True)
|
||||
task = store.create_task('force fixture')
|
||||
evidence = service._project_task_service(task.task_id)._quick_run_preparer.data_root / 'quick-run'
|
||||
witness = evidence / 'started'
|
||||
bootstrap = textwrap.dedent('''
|
||||
import sys,os,time,json
|
||||
from pathlib import Path
|
||||
from dx_productivity.trial_task_center import batch_worker
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
def business(self, task_id, **kwargs):
|
||||
root=self._project_task_service(task_id)._quick_run_preparer.data_root/'quick-run'
|
||||
root.mkdir(parents=True,exist_ok=True)
|
||||
(root/'started').write_text(str(os.getpid()))
|
||||
request=json.loads(Path(sys.argv[1]).read_text(encoding='utf-8'))
|
||||
print(json.dumps({'version':1,'batch_id':request['batch_id'],'attempt_id':request['attempt_id'],
|
||||
'generation':1,'sequence':100,'type':'progress','payload':'fixture executing'}),flush=True)
|
||||
time.sleep(30)
|
||||
(root/'unexpected_completed').touch()
|
||||
TaskCenterService.execute_quick_run=business
|
||||
batch_worker.run(Path(sys.argv[1]))
|
||||
''')
|
||||
original_command = module._command
|
||||
monkeypatch.setattr(module, '_command', lambda request:
|
||||
(sys.executable, '-c', bootstrap, str(request))
|
||||
if json.loads(request.read_text())['operation'] == 'execute_quick_run' else original_command(request))
|
||||
original_mark = module._mark_interrupted
|
||||
denied = []
|
||||
def mark(*args):
|
||||
if deny_first and not denied:
|
||||
denied.append(True)
|
||||
raise PermissionError('force evidence denied once')
|
||||
return original_mark(*args)
|
||||
monkeypatch.setattr(module, '_mark_interrupted', mark)
|
||||
control = StopControl()
|
||||
driver_errors, survived = [], []
|
||||
def request():
|
||||
try:
|
||||
deadline = time.monotonic() + 15
|
||||
while not witness.exists() and time.monotonic() < deadline:
|
||||
time.sleep(.02)
|
||||
assert witness.exists()
|
||||
pid = int(witness.read_text())
|
||||
if transport_failure:
|
||||
while not control.control_error and time.monotonic() < deadline:
|
||||
time.sleep(.02)
|
||||
assert control.control_error
|
||||
handle = win32api.OpenProcess(0x100000, False, pid)
|
||||
try:
|
||||
survived.append(win32event.WaitForSingleObject(handle, 0) == 258)
|
||||
finally:
|
||||
handle.Close()
|
||||
control.request_force()
|
||||
if deny_first:
|
||||
while not control.control_error and time.monotonic() < deadline:
|
||||
time.sleep(.02)
|
||||
assert control.control_error
|
||||
handle = win32api.OpenProcess(0x100000, False, pid)
|
||||
try:
|
||||
survived.append(win32event.WaitForSingleObject(handle, 0) == 258)
|
||||
finally:
|
||||
handle.Close()
|
||||
control.request_force()
|
||||
except Exception as error:
|
||||
driver_errors.append(error)
|
||||
thread = Thread(target=request)
|
||||
thread.start()
|
||||
try:
|
||||
with pytest.raises(TaskCenterError) as caught:
|
||||
module.run_supervised(service, 'execute_quick_run', {'task_id': task.task_id}, stop_event=control,
|
||||
on_progress=(lambda _: (_ for _ in ()).throw(RuntimeError('UI feedback failed'))) if transport_failure else None)
|
||||
assert not driver_errors
|
||||
assert caught.value.code == 'user_forced_stop'
|
||||
assert (evidence / 'uncertain.flag').exists()
|
||||
marker = (evidence / 'uncertain.flag').read_bytes()
|
||||
assert not (evidence / 'unexpected_completed').exists()
|
||||
journals = [json.loads(p.read_text(encoding='utf-8')) for p in (tmp_path / 'batch-resources').glob('*/supervisor.json')]
|
||||
forced = next(r for r in journals if r['operation'] == 'execute_quick_run')
|
||||
assert forced['stop_mode'] == 'forced' and forced['business_uncertain'] is True
|
||||
assert service.cleanup_batch_resources() == ()
|
||||
assert (evidence / 'uncertain.flag').read_bytes() == marker
|
||||
assert not (evidence / 'unexpected_completed').exists()
|
||||
if deny_first or transport_failure:
|
||||
assert survived == [True]
|
||||
finally:
|
||||
thread.join(3)
|
||||
store.close()
|
||||
@@ -0,0 +1,54 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import pytest
|
||||
|
||||
|
||||
def test_wire_roundtrip_preserves_bom_preview_and_identity(tmp_path):
|
||||
from dx_productivity.trial_task_center.supervisor_protocol import encode, decode
|
||||
from dx_productivity.trial_task_center.bom_import_report import BomImportReport, BomRowResult
|
||||
from dx_productivity.trial_task_center.bom_batch import BomBatchPreview, BomProjectPreview
|
||||
from dx_productivity.trial_task_center.target_plan import ComponentTarget, TargetPlanRow
|
||||
target = ComponentTarget(tmp_path/'中文.prj', 'Main', '1', 'uid', 'R1', 'OLD', '1K')
|
||||
row = TargetPlanRow(target, True, 'NEW')
|
||||
preview = BomBatchPreview('task', (('task', str(target.project_path)),),
|
||||
(BomProjectPreview('task', target.project_path, (), (target,), (row,)),))
|
||||
report = BomImportReport((BomRowResult(2, str(target.project_path), '中文', 'passed'),), (), preview)
|
||||
assert decode(json.loads(json.dumps(encode(report)))) == report
|
||||
|
||||
|
||||
def test_protocol_rejects_unknown_types_and_late_results():
|
||||
from dx_productivity.trial_task_center.supervisor_protocol import decode, MessageGate
|
||||
with pytest.raises(ValueError):
|
||||
decode({'type': 'object', 'name': 'os.system', 'fields': {}})
|
||||
gate = MessageGate('batch', 'attempt')
|
||||
current = dict(version=1, batch_id='batch', attempt_id='attempt', generation=1, sequence=1, type='heartbeat', payload=None)
|
||||
assert gate.accept(current)
|
||||
assert not gate.accept(current)
|
||||
assert not gate.accept(dict(current, sequence=2, attempt_id='old'))
|
||||
assert not gate.accept(dict(current, sequence=2, generation=0))
|
||||
assert gate.accept(dict(current, sequence=2, generation=2))
|
||||
assert not gate.accept(dict(current, sequence=3, generation=1))
|
||||
|
||||
|
||||
def test_wire_error_retains_bom_report_and_primary_cleanup():
|
||||
from dx_productivity.trial_task_center.supervisor_protocol import encode_error, decode_error
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
from dx_productivity.trial_task_center.bom_import_report import BomImportReport
|
||||
error = TaskCenterError('batch_resources_pending', '收尾失败', error_id='support-id')
|
||||
error.primary_error = TaskCenterError('path_access', '目录创建失败')
|
||||
error.cleanup_error = TaskCenterError('private_desktop_cleanup', '桌面残留')
|
||||
error.bom_report = BomImportReport((), (), errors=('原始问题',))
|
||||
actual = decode_error(json.loads(json.dumps(encode_error(error))))
|
||||
assert actual.error_id == 'support-id'
|
||||
assert actual.primary_error.code == 'path_access'
|
||||
assert actual.cleanup_error.code == 'private_desktop_cleanup'
|
||||
assert actual.bom_report.errors == ('原始问题',)
|
||||
|
||||
|
||||
def test_single_task_batch_result_wire_roundtrip():
|
||||
from dx_productivity.trial_task_center.supervisor_protocol import encode, decode
|
||||
from dx_productivity.trial_task_center.batch import BatchRunResult
|
||||
from dx_productivity.trial_task_center.quick_run_protocol import QuickRunProgress
|
||||
progress = QuickRunProgress(1,1,None,None,None,(),True,saved_items=1)
|
||||
actual = BatchRunResult(progress,False,item_messages=('saved',))
|
||||
assert decode(encode(actual)) == actual
|
||||
@@ -0,0 +1,23 @@
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.mark.parametrize('body', [None, {}, {'status':'ok','business_succeeded':'yes','result':'bad'},
|
||||
{'status':'ok','business_succeeded':True,'result':'wrong type'}])
|
||||
def test_invalid_result_cannot_finish_resource_gate(tmp_path, body):
|
||||
from dx_productivity.trial_task_center.batch_supervisor import _finish_response
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
journal={'operation':'probe','batch_id':'a'*32,'state':'blocked'}
|
||||
with pytest.raises((ValueError, KeyError, TypeError)):
|
||||
_finish_response(tmp_path,journal,body,configure_logging(tmp_path))
|
||||
assert journal['state']=='blocked'
|
||||
assert not (tmp_path/'batch-resources').exists()
|
||||
|
||||
|
||||
def test_pending_cleanup_cannot_expose_bom_preview():
|
||||
from dx_productivity.trial_task_center.bom_import_report import BomImportReport
|
||||
from dx_productivity.trial_task_center.bom_batch import BomBatchPreview
|
||||
from dx_productivity.trial_task_center.supervisor_result import resource_pending_error
|
||||
report=BomImportReport((),(),BomBatchPreview('task',(),()))
|
||||
error=resource_pending_error('inspect_bom_batch',report,OSError('held desktop'))
|
||||
assert error.completed_result.preview is None
|
||||
assert error.bom_report.preview is None
|
||||
@@ -0,0 +1,90 @@
|
||||
from pathlib import Path
|
||||
import pytest
|
||||
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.models import TaskStatus
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def service(tmp_path):
|
||||
store = TaskStore(tmp_path / 'tasks.db')
|
||||
yield TaskCenterService(store, configure_logging(tmp_path / 'data'))
|
||||
store.close()
|
||||
|
||||
|
||||
def archive(service):
|
||||
from dx_productivity.trial_task_center.task_archive import archive_completed_session
|
||||
return archive_completed_session(service)
|
||||
|
||||
|
||||
def test_close_archive_preserves_files_rules_and_completed_restore(service):
|
||||
task = service.create_task('Done')
|
||||
service.add_rule(task.task_id, 'OLD', 'NEW')
|
||||
store = service._store
|
||||
store.set_task_status(task.task_id, TaskStatus.RUNNING)
|
||||
store.set_task_status(task.task_id, TaskStatus.COMPLETED, summary='verified')
|
||||
sentinel = service._logging.root / 'keep.log'
|
||||
sentinel.write_text('evidence')
|
||||
report = archive(service)
|
||||
assert report.archived_ids == (task.task_id,)
|
||||
assert not report.retained
|
||||
assert not service.tasks()
|
||||
assert sentinel.read_text() == 'evidence'
|
||||
restored = service.restore_task(task.task_id)
|
||||
assert restored.status is TaskStatus.COMPLETED
|
||||
assert store.list_rules(task.task_id)[0].old_part_number == 'OLD'
|
||||
|
||||
|
||||
def test_archive_retains_owned_and_uncertain_tasks_but_archives_safe_tasks(service, monkeypatch):
|
||||
owned, uncertain, safe = [service.create_task(name) for name in ('owned', 'uncertain', 'safe')]
|
||||
monkeypatch.setattr(service, 'pending_batch_resources', lambda: ({'log_task_ids': [owned.task_id]},))
|
||||
monkeypatch.setattr(service, 'has_interrupted_quick_run', lambda task_id: task_id == uncertain.task_id)
|
||||
report = archive(service)
|
||||
assert report.archived_ids == (safe.task_id,)
|
||||
assert set(report.retained_ids) == {owned.task_id, uncertain.task_id}
|
||||
assert set(t.task_id for t in service.tasks()) == set(report.retained_ids)
|
||||
|
||||
|
||||
def test_unknown_resource_ownership_retains_every_task(service, monkeypatch):
|
||||
task = service.create_task('safe')
|
||||
monkeypatch.setattr(service, 'pending_batch_resources', lambda: ({'state': 'blocked'},))
|
||||
report = archive(service)
|
||||
assert not report.archived_ids
|
||||
assert report.retained_ids == (task.task_id,)
|
||||
|
||||
|
||||
def test_archive_failure_rolls_back_every_task(service, monkeypatch):
|
||||
first, second = [service.create_task(name) for name in ('first', 'second')]
|
||||
original = service._store.set_task_status
|
||||
calls = []
|
||||
def fail(task_id, *args, **kwargs):
|
||||
calls.append(task_id)
|
||||
if len(calls) == 2:
|
||||
raise TaskCenterError('write_failed', 'disk unavailable')
|
||||
return original(task_id, *args, **kwargs)
|
||||
monkeypatch.setattr(service._store, 'set_task_status', fail)
|
||||
with pytest.raises(TaskCenterError):
|
||||
archive(service)
|
||||
assert {t.task_id for t in service.tasks()} == {first.task_id, second.task_id}
|
||||
|
||||
def test_orphan_uncertainty_flag_retains_task_without_handoff(service):
|
||||
task = service.create_task('protected')
|
||||
root = service._project_task_service(task.task_id)._quick_run_preparer.data_root / 'quick-run'
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
marker = root / 'uncertain.flag'
|
||||
marker.write_text('needs verification')
|
||||
report = archive(service)
|
||||
assert report.retained_ids == (task.task_id,)
|
||||
assert marker.read_text() == 'needs verification'
|
||||
|
||||
|
||||
def test_running_task_is_retained_and_other_tasks_archive(service):
|
||||
running = service.create_task('running')
|
||||
safe = service.create_task('draft')
|
||||
service._store.set_task_status(running.task_id, TaskStatus.RUNNING)
|
||||
report = archive(service)
|
||||
assert report.archived_ids == (safe.task_id,)
|
||||
assert report.retained_ids == (running.task_id,)
|
||||
@@ -0,0 +1,54 @@
|
||||
import os
|
||||
os.environ.setdefault('QT_QPA_PLATFORM', 'offscreen')
|
||||
|
||||
from PySide6.QtGui import QCloseEvent
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
from dx_productivity.trial_task_center.models import TaskStatus
|
||||
from test_ui_remediation import window
|
||||
|
||||
|
||||
def test_archive_close_setting_defaults_off_and_persists(window):
|
||||
assert not window.archive_on_close_check.isChecked()
|
||||
window.archive_on_close_check.setChecked(True)
|
||||
assert window._layout_settings.value('tasks/archive_on_close', False, type=bool)
|
||||
window.archive_on_close_check.setChecked(False)
|
||||
|
||||
|
||||
def test_enabled_close_archives_after_unsaved_plan_cancel_is_resolved(window, qtbot):
|
||||
task = window.service.create_task('normal close')
|
||||
window.archive_on_close_check.setChecked(True)
|
||||
window.plan_panel._drafts['hidden'] = ()
|
||||
window._close_plan_confirmer = lambda: 'cancel'
|
||||
event = QCloseEvent()
|
||||
window.closeEvent(event)
|
||||
assert not event.isAccepted()
|
||||
assert window.service._store.get_task(task.task_id).status is TaskStatus.DRAFT
|
||||
window._close_plan_confirmer = lambda: 'discard'
|
||||
window.closeEvent(QCloseEvent())
|
||||
qtbot.waitUntil(lambda: window._closed)
|
||||
assert window.service._store.get_task(task.task_id).status is TaskStatus.DELETED
|
||||
|
||||
|
||||
def test_archive_write_failure_keeps_window_open_and_all_tasks(window, qtbot, monkeypatch):
|
||||
task = window.service.create_task('keep')
|
||||
window.archive_on_close_check.setChecked(True)
|
||||
def fail(_):
|
||||
raise TaskCenterError('write_failed', 'archive disk unavailable')
|
||||
monkeypatch.setattr(window.service._store, 'archive_tasks', fail)
|
||||
window.closeEvent(QCloseEvent())
|
||||
qtbot.waitUntil(lambda: not window._workers)
|
||||
assert not window._closed
|
||||
assert window.service._store.get_task(task.task_id).status is TaskStatus.DRAFT
|
||||
assert 'archive disk unavailable' in window._message
|
||||
window.archive_on_close_check.setChecked(False)
|
||||
|
||||
|
||||
def test_close_notice_retains_recovery_tasks(window, qtbot, monkeypatch):
|
||||
task = window.service.create_task('recover this')
|
||||
window.archive_on_close_check.setChecked(True)
|
||||
monkeypatch.setattr(window.service, 'has_interrupted_quick_run', lambda _: True)
|
||||
window.closeEvent(QCloseEvent())
|
||||
qtbot.waitUntil(lambda: window._closed)
|
||||
assert window.service._store.get_task(task.task_id).status is TaskStatus.DRAFT
|
||||
assert 'recover this' in window._archive_close_notice
|
||||
assert '核验并恢复' in window._archive_close_notice
|
||||
@@ -0,0 +1,142 @@
|
||||
"""Windows-only real external OpenDesktop reference acceptance (never starts CAD)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'src'))
|
||||
|
||||
from dx_productivity.trial_task_center.batch_resources import (
|
||||
BatchResourceCleanupError, cleanup_pending_resources, pending_resources,
|
||||
)
|
||||
from dx_productivity.trial_task_center.batch_runtime import BatchCadRuntime
|
||||
from dx_productivity.trial_task_center.cad_identity import current_desktop_name
|
||||
from dx_productivity.trial_task_center.isolated_desktop import _Win32Api
|
||||
|
||||
|
||||
HOLDER = '''import json,os,sys,win32service
|
||||
handle=win32service.OpenDesktop(sys.argv[1],0,False,1)
|
||||
print(json.dumps({'state':'holding','pid':os.getpid(),'handle':int(handle)}),flush=True)
|
||||
assert sys.stdin.readline().strip()=='release'
|
||||
handle.CloseDesktop()
|
||||
print(json.dumps({'state':'released','pid':os.getpid()}),flush=True)
|
||||
'''
|
||||
|
||||
|
||||
def probe(output):
|
||||
if os.name != 'nt':
|
||||
raise RuntimeError('Windows is required')
|
||||
output = Path(output).resolve() / ('run-' + uuid4().hex)
|
||||
root = output / 'data'
|
||||
output.mkdir(parents=True)
|
||||
started = time.monotonic()
|
||||
before_desktop = current_desktop_name()
|
||||
cad_requests = []
|
||||
def forbidden_cad(*args, **kwargs):
|
||||
cad_requests.append(True)
|
||||
raise AssertionError('This resource-only probe must not create CAD')
|
||||
runtime = BatchCadRuntime(root, 'external_holder_acceptance', cad_factory=forbidden_cad)
|
||||
runtime.__enter__()
|
||||
desktop = runtime._ensure_desktop()
|
||||
result_path = runtime.path.parent / 'business-result.json'
|
||||
original_result = {'kind': 'acceptance_fixture', 'status': 'verified',
|
||||
'completed': True, 'saved_projects': 0, 'note': 'Synthetic result, no CAD/project operation'}
|
||||
original_bytes = json.dumps(original_result, ensure_ascii=False, indent=2).encode('utf-8')
|
||||
result_path.write_bytes(original_bytes)
|
||||
runtime.result = original_result
|
||||
digest = hashlib.sha256(original_bytes).hexdigest()
|
||||
holder = None
|
||||
report = {'version': 1, 'probe_pid': os.getpid(), 'desktop': desktop.name,
|
||||
'result_sha256_before': digest, 'events': [], 'cad_requests': cad_requests,
|
||||
'holder_identification': {'diagnostics_identified_external_handle_holder': False,
|
||||
'external_handle_holders': 'unknown',
|
||||
'fixture_pid_source': 'PID returned by subprocess.Popen and confirmed by fixture handshake; not inferred from diagnostics'}}
|
||||
try:
|
||||
holder = subprocess.Popen((sys.executable, '-u', '-c', HOLDER, desktop.name),
|
||||
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
text=True, encoding='utf-8', creationflags=subprocess.CREATE_NO_WINDOW)
|
||||
handshake = json.loads(holder.stdout.readline())
|
||||
assert handshake['pid'] == holder.pid and handshake['state'] == 'holding'
|
||||
report['holder_fixture'] = handshake
|
||||
try:
|
||||
runtime.close()
|
||||
raise AssertionError('cleanup incorrectly succeeded with external holder')
|
||||
except BatchResourceCleanupError as failure:
|
||||
assert failure.completed_result == original_result
|
||||
report['events'].append({'phase': 'initial_cleanup', 'blocked': True,
|
||||
'error': str(failure), 'holder_still_running': holder.poll() is None})
|
||||
assert holder.poll() is None
|
||||
blocked = pending_resources(root)
|
||||
assert len(blocked) == 1 and blocked[0]['state'] == 'blocked'
|
||||
assert blocked[0]['cleanup']['desktop_confirmation']['remaining_windows'] == []
|
||||
assert blocked[0]['cleanup']['desktop_handle_closed']
|
||||
report['blocked_resource_record'] = blocked[0]
|
||||
candidate = BatchCadRuntime(root, 'must_be_blocked', cad_factory=forbidden_cad)
|
||||
try:
|
||||
candidate.__enter__()
|
||||
raise AssertionError('new batch bypassed pending resource guard')
|
||||
except BatchResourceCleanupError as failure:
|
||||
report['events'].append({'phase': 'new_batch', 'blocked': True, 'error': str(failure)})
|
||||
assert candidate.generation == 0 and candidate.desktop is None
|
||||
pending = cleanup_pending_resources(root)
|
||||
assert pending and pending[0]['state'] == 'blocked' and holder.poll() is None
|
||||
report['events'].append({'phase': 'resource_only_retry_while_held',
|
||||
'blocked': True, 'holder_still_running': True})
|
||||
assert result_path.read_bytes() == original_bytes
|
||||
holder.stdin.write('release\n')
|
||||
holder.stdin.flush()
|
||||
released = json.loads(holder.stdout.readline())
|
||||
assert released['state'] == 'released' and released['pid'] == holder.pid
|
||||
assert holder.wait(timeout=5) == 0
|
||||
remaining = cleanup_pending_resources(root)
|
||||
assert remaining == ()
|
||||
final = json.loads(runtime.path.read_text(encoding='utf-8'))
|
||||
assert final['state'] == 'released'
|
||||
assert final['cad_pid'] is None and final['cad_start_requested'] is False
|
||||
assert final['processes'] == [] and cad_requests == []
|
||||
assert result_path.read_bytes() == original_bytes and runtime.result == original_result
|
||||
assert current_desktop_name() == before_desktop
|
||||
listed = _Win32Api().service.GetProcessWindowStation().EnumDesktops()
|
||||
assert desktop.name.casefold() not in {name.casefold() for name in listed}
|
||||
# Admission may be checked again, but no batch resources or business work
|
||||
# are started by this independent acceptance probe.
|
||||
with BatchCadRuntime(root, 'post_release_admission', cad_factory=forbidden_cad) as admitted:
|
||||
assert admitted.generation == 0 and admitted.desktop is None
|
||||
report.update(success=True, final_resource_record=final,
|
||||
result_sha256_after=hashlib.sha256(result_path.read_bytes()).hexdigest(),
|
||||
business_result_unchanged=True, cad_started=False, business_replayed=False,
|
||||
desktop_released=True, input_desktop_unchanged=True,
|
||||
elapsed_seconds=time.monotonic() - started)
|
||||
report['events'].append({'phase': 'resource_only_retry_after_holder_release',
|
||||
'released': True, 'business_result_unchanged': True})
|
||||
except Exception as failure:
|
||||
report.update(success=False, failure=repr(failure))
|
||||
raise
|
||||
finally:
|
||||
if holder is not None:
|
||||
if holder.poll() is None:
|
||||
# Only this explicitly created helper may be stopped on probe failure.
|
||||
holder.terminate()
|
||||
holder.wait(timeout=5)
|
||||
for stream in (holder.stdin, holder.stdout, holder.stderr):
|
||||
if stream is not None:
|
||||
stream.close()
|
||||
try:
|
||||
runtime.close(reason='probe_finally')
|
||||
finally:
|
||||
(output / 'evidence.json').write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding='utf-8')
|
||||
return output / 'evidence.json'
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--output', type=Path, default=Path('build/issue-3-external-holder'))
|
||||
args = parser.parse_args()
|
||||
print(probe(args.output))
|
||||
@@ -0,0 +1,273 @@
|
||||
"""Crash one certified owned frozen supervisor after actual native replacement.
|
||||
|
||||
Hybrid source non-GUI parent / frozen worker/helper/VBS. No production injection
|
||||
switches. New disposable two-project copies only; never resumes business.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from threading import Event, Thread
|
||||
import time
|
||||
import traceback
|
||||
from uuid import uuid4
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(REPO / 'src'))
|
||||
sys.path.insert(0, str(REPO / 'tools'))
|
||||
from probe_batch_runtime_live import hashes, save
|
||||
from probe_frozen_startup_recovery import read_json
|
||||
|
||||
|
||||
def protection(root):
|
||||
return {str(path.relative_to(root)): path.read_bytes().hex()
|
||||
for path in root.rglob('uncertain.flag')}
|
||||
|
||||
|
||||
class PostwriteCrash:
|
||||
def __init__(self, root, output, task_ids, cancel):
|
||||
self.root, self.output, self.task_ids, self.cancel = root, output, task_ids, cancel
|
||||
self.stop = Event()
|
||||
self.crash = None
|
||||
self.errors = []
|
||||
|
||||
def scan(self):
|
||||
import win32api
|
||||
import win32event
|
||||
import win32process
|
||||
for path in (self.root / 'tasks' / self.task_ids[0] / 'quick-run').glob('*.native.json'):
|
||||
native = read_json(path)
|
||||
if not native or native.get('stage') not in {'native_replace_applied', 'native_replace_verified'}:
|
||||
continue
|
||||
assert native.get('replaced_uids'), 'no observed replacement identity'
|
||||
for request_path in (self.root / 'batch-resources').glob('*/request.json'):
|
||||
request = read_json(request_path)
|
||||
if not request or request['operation'] != 'execute_task_queue':
|
||||
continue
|
||||
directory = request_path.parent
|
||||
journal, session = read_json(directory / 'supervisor.json'), read_json(directory / 'session.json')
|
||||
assert journal and session and journal['state'] == 'running', 'supervisor not running'
|
||||
assert journal['batch_id'] == request['batch_id'] == session['session_id'] == directory.name
|
||||
assert journal['task_ids'] == list(self.task_ids)
|
||||
assert session['generation'] == 1 and session['retries'] == {}, 'unexpected generation/retry'
|
||||
markers = list(directory.glob('attempts/**/*.json'))
|
||||
assert len(markers) == 1, 'unexpected marker count'
|
||||
marker = read_json(markers[0])
|
||||
assert marker and marker['state'] == 'mutation_possible'
|
||||
assert marker['task_id'] == self.task_ids[0]
|
||||
if (directory / 'result.json').exists() or (directory / 'business-result.json').exists():
|
||||
return # Too late for the requested missing-final-result fault.
|
||||
queues = [read_json(item) for item in (self.root / 'task-queues').glob('*.json')]
|
||||
assert len(queues) == 1 and queues[0]['entries'][self.task_ids[1]]['state'] == 'pending'
|
||||
assert not (self.root / 'tasks' / self.task_ids[1] / 'quick-run' / 'status.tsv').exists()
|
||||
identity = journal['process']
|
||||
assert identity and identity['creation_time']
|
||||
handle = win32api.OpenProcess(0x1000 | 0x100000 | 1, False, identity['pid'])
|
||||
try:
|
||||
created = str(win32process.GetProcessTimes(handle)['CreationTime'])
|
||||
assert created == identity['creation_time'], 'supervisor PID creation time mismatch'
|
||||
assert win32event.WaitForSingleObject(handle, 0) == 258
|
||||
latest = read_json(path)
|
||||
# If the verified report advanced, no assumption is made
|
||||
# from an earlier native_replace_started observation.
|
||||
if not latest or latest.get('stage') not in {'native_replace_applied', 'native_replace_verified'}:
|
||||
return # applied can advance to pin promotion between reads.
|
||||
assert read_json(markers[0])['state'] == 'mutation_possible'
|
||||
for status in (self.root / 'tasks').glob('*/quick-run/status.tsv'):
|
||||
if '\tsaved_ok\t' in status.read_text(encoding='utf-8-sig'):
|
||||
return # Keep the already completed fixture; never replay it.
|
||||
self.crash = {'time_ns': time.time_ns(), 'supervisor_identity': identity,
|
||||
'creation_time_observed': created, 'batch_id': directory.name,
|
||||
'native_report': latest, 'marker': marker, 'marker_path': str(markers[0]),
|
||||
'marker_sha256': hashlib.sha256(markers[0].read_bytes()).hexdigest(),
|
||||
'queue_before_crash': queues[0], 'terminated_owned_supervisor_only': True}
|
||||
save(self.output / 'postwrite-crash.json', self.crash)
|
||||
win32process.TerminateProcess(handle, 81)
|
||||
assert win32event.WaitForSingleObject(handle, 5000) == 0
|
||||
finally:
|
||||
handle.Close()
|
||||
return
|
||||
|
||||
def _watch(self):
|
||||
while not self.stop.wait(.005):
|
||||
try:
|
||||
self.scan()
|
||||
if self.crash is not None:
|
||||
return
|
||||
except Exception as error:
|
||||
self.errors.append({'error': repr(error), 'traceback': traceback.format_exc()})
|
||||
self.cancel.set()
|
||||
save(self.output / 'watcher-errors.json', self.errors)
|
||||
return
|
||||
|
||||
def __enter__(self):
|
||||
self.thread = Thread(target=self._watch, daemon=True)
|
||||
self.thread.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *_):
|
||||
self.stop.set()
|
||||
self.thread.join(5)
|
||||
assert not self.thread.is_alive()
|
||||
|
||||
|
||||
def recheck_existing(output):
|
||||
"""Fresh parent process: resource-only frozen request, never resumes queue."""
|
||||
from dx_productivity.trial_task_center import batch_supervisor
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
output = output.resolve(strict=True)
|
||||
assert output.is_relative_to(REPO / 'build/issue-3-frozen-postwrite-crash')
|
||||
inputs = read_json(output / 'inputs.json')
|
||||
root = output / 'data'
|
||||
before = protection(root)
|
||||
assert before
|
||||
crash = read_json(output / 'postwrite-crash.json')
|
||||
marker = Path(crash['marker_path'])
|
||||
assert read_json(marker)['state'] == 'mutation_possible'
|
||||
launch = []
|
||||
def command(request):
|
||||
launch.append(read_json(request)['operation'])
|
||||
return (inputs['executable'], '--batch-worker', str(request))
|
||||
original_command = batch_supervisor._command
|
||||
batch_supervisor._command = command
|
||||
store = TaskStore(output / 'tasks.db')
|
||||
try:
|
||||
service = TaskCenterService(store, configure_logging(root), supervised=True)
|
||||
assert protection(root) == before, 'startup cleared protection'
|
||||
pending = service.cleanup_batch_resources()
|
||||
assert not pending
|
||||
assert protection(root) == before, 'resource cleanup cleared business protection'
|
||||
assert hashlib.sha256(marker.read_bytes()).hexdigest() == crash['marker_sha256']
|
||||
assert launch == ['cleanup_batch_resources']
|
||||
assert hashes(Path(inputs['projects'][1]).parent) == inputs['second_project_before_execution']
|
||||
execution = read_json(root / 'batch-resources' / crash['batch_id'] / 'session.json')
|
||||
assert execution['generation'] == 1 and execution['retries'] == {}
|
||||
assert execution['state'] == 'released'
|
||||
assert len(list((root / 'batch-resources' / crash['batch_id']).glob('attempts/**/*.json'))) == 1
|
||||
save(output / 'restart-resource-recheck.json', {'verified': True, 'launch_operations': launch,
|
||||
'pending_resources': pending, 'protection_preserved': True, 'marker_preserved': True,
|
||||
'business_replayed': False, 'second_project_unchanged': True, 'execution_session': execution,
|
||||
'supervisor': read_json(root / 'batch-resources' / crash['batch_id'] / 'supervisor.json')})
|
||||
finally:
|
||||
batch_supervisor._command = original_command
|
||||
store.close()
|
||||
|
||||
|
||||
def run(source_root, library, executable, base):
|
||||
from dx_productivity.trial_task_center import batch_supervisor
|
||||
from dx_productivity.trial_task_center.bom_plan import write_bom_template
|
||||
from dx_productivity.trial_task_center.errors import TaskCenterError
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.task_queue import execute_task_queue
|
||||
from private_desktop_observer import PrivateDesktopObserver
|
||||
source_root, library, executable = [item.resolve(strict=True) for item in (source_root, library, executable)]
|
||||
output = base.resolve() / ('run-' + uuid4().hex)
|
||||
assert output.is_relative_to(REPO / 'build/issue-3-frozen-postwrite-crash')
|
||||
output.mkdir(parents=True, exist_ok=False)
|
||||
names, projects, source_hashes = ('Audio_D2128', 'Audio_T605A'), [], {}
|
||||
for name in names:
|
||||
source_hashes[name] = hashes(source_root / name)
|
||||
target = output / 'projects' / name
|
||||
shutil.copytree(source_root / name, target)
|
||||
project = target / (name + '.prj')
|
||||
text, count = re.subn(r'(?m)^KEY CentralLibrary .*$', lambda _: f'KEY CentralLibrary "{library.as_posix()}"',
|
||||
project.read_text(encoding='utf-8-sig'))
|
||||
assert count == 1
|
||||
project.write_text(text, encoding='utf-8')
|
||||
projects.append(project)
|
||||
library_hash = hashlib.sha256(library.read_bytes()).hexdigest()
|
||||
report = {'mode': 'source_non_gui_parent_frozen_worker_postwrite_crash', 'gui_parent_covered': False,
|
||||
'frozen_executable_sha256': hashlib.sha256(executable.read_bytes()).hexdigest(), 'launches': []}
|
||||
original_command = batch_supervisor._command
|
||||
def command(request):
|
||||
report['launches'].append(read_json(request))
|
||||
return (str(executable), '--batch-worker', str(request))
|
||||
batch_supervisor._command = command
|
||||
store = TaskStore(output / 'tasks.db')
|
||||
service = TaskCenterService(store, configure_logging(output / 'data'), supervised=True)
|
||||
started = time.monotonic()
|
||||
try:
|
||||
with PrivateDesktopObserver(output):
|
||||
tasks = service.create_project_tasks(output / 'projects', selected_paths=tuple(projects))
|
||||
by_project = {service.projects(task.task_id)[0].path.resolve(): task for task in tasks}
|
||||
ordered = tuple(by_project[project.resolve()].task_id for project in projects)
|
||||
bom = write_bom_template(output / 'input.xlsx', ({'project_path': str(project),
|
||||
'old_part_number': '4100-CA0000-T7', 'new_part_number': '4100-CA0000-T0',
|
||||
'refdes': 'RV05', 'enabled': True, 'sync_nc': True} for project in projects))
|
||||
imported = service.inspect_bom_batch(ordered[0], bom, on_progress=lambda text: print(text, flush=True))
|
||||
assert imported.passed_rows == 2 and imported.preview is not None
|
||||
service.save_bom_batch(imported.preview, {item.task_id: item.rows for item in imported.preview.projects})
|
||||
save(output / 'inputs.json', {'executable': str(executable), 'projects': list(map(str, projects)),
|
||||
'task_ids': ordered, 'second_project_before_execution': hashes(projects[1].parent)})
|
||||
cancel = Event()
|
||||
print('Frozen two-project queue: wait for actual applied/verified then crash own supervisor', flush=True)
|
||||
with PostwriteCrash(output / 'data', output, ordered, cancel) as injector:
|
||||
try:
|
||||
result = execute_task_queue(service, ordered, stop_event=cancel)
|
||||
report['unexpected_result'] = repr(result)
|
||||
except TaskCenterError as failure:
|
||||
report['operation_error'] = {'code': failure.code, 'message': failure.message}
|
||||
assert not injector.errors and injector.crash is not None
|
||||
assert report.get('operation_error', {}).get('code') == 'supervisor_interrupted'
|
||||
root = output / 'data'
|
||||
assert protection(root)
|
||||
execution = root / 'batch-resources' / injector.crash['batch_id']
|
||||
session = read_json(execution / 'session.json')
|
||||
assert session['generation'] == 1 and session['retries'] == {}
|
||||
assert len(list(execution.glob('attempts/**/*.json'))) == 1
|
||||
assert not (root / 'tasks' / ordered[1] / 'quick-run' / 'status.tsv').exists()
|
||||
queues = [read_json(path) for path in (root / 'task-queues').glob('*.json')]
|
||||
assert len(queues) == 1 and queues[0]['entries'][ordered[1]]['state'] == 'pending'
|
||||
report['queue_after_crash'] = queues[0]
|
||||
assert not (execution / 'business-result.json').exists()
|
||||
store.close()
|
||||
print('Fresh parent process resource-only recheck', flush=True)
|
||||
completed = subprocess.run((sys.executable, str(Path(__file__).resolve()), '--recheck-existing', str(output)),
|
||||
capture_output=True, timeout=90)
|
||||
(output / 'restart-output.txt').write_bytes(completed.stdout + b'\n' + completed.stderr)
|
||||
assert completed.returncode == 0, (output / 'restart-output.txt').read_text(errors='replace')
|
||||
observation = read_json(output / 'desktop-observation.json')
|
||||
assert not observation['observer_errors']
|
||||
assert all(observation['cad_exit_confirmed'].values()) and all(observation['desktop_absent'].values())
|
||||
assert not observation['owned_cad_became_foreground'] and not observation['owned_cad_on_input_desktop']
|
||||
assert observation['input_desktop_before'] == observation['input_desktop_after']
|
||||
report.update(verified=True, business_replayed=False, next_project_started=False,
|
||||
restart_resource_recheck=read_json(output / 'restart-resource-recheck.json'))
|
||||
except BaseException as error:
|
||||
report.update(verified=False, error=repr(error), traceback=traceback.format_exc())
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
batch_supervisor._command = original_command
|
||||
store.close()
|
||||
report['source_unchanged'] = all(hashes(source_root / name) == source_hashes[name] for name in names)
|
||||
report['library_unchanged'] = hashlib.sha256(library.read_bytes()).hexdigest() == library_hash
|
||||
report['elapsed_seconds'] = time.monotonic() - started
|
||||
save(output / 'report.json', report)
|
||||
print(json.dumps({'output': str(output), **{key: report.get(key) for key in
|
||||
('verified', 'error', 'elapsed_seconds')}}, ensure_ascii=False), flush=True)
|
||||
return 0 if report.get('verified') and report['source_unchanged'] and report['library_unchanged'] else 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--source-root', type=Path)
|
||||
parser.add_argument('--library', type=Path)
|
||||
parser.add_argument('--executable', type=Path, default=REPO / 'dist/DxPartReplace/DxPartReplace.exe')
|
||||
parser.add_argument('--output', type=Path, default=REPO / 'build/issue-3-frozen-postwrite-crash')
|
||||
parser.add_argument('--recheck-existing', type=Path)
|
||||
args = parser.parse_args()
|
||||
if args.recheck_existing:
|
||||
recheck_existing(args.recheck_existing)
|
||||
else:
|
||||
assert args.source_root and args.library
|
||||
raise SystemExit(run(args.source_root, args.library, args.executable, args.output))
|
||||
@@ -0,0 +1,271 @@
|
||||
"""Hybrid source parent / frozen supervisor startup-failure CAD acceptance.
|
||||
|
||||
Prepare without CAD with --help. Run only after obtaining the shared CAD test
|
||||
window. Production source/VBS are unchanged; parent _command alone is replaced.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from dataclasses import asdict
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from threading import Event, Thread
|
||||
import time
|
||||
import traceback
|
||||
from uuid import uuid4
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(REPO / 'src'))
|
||||
sys.path.insert(0, str(REPO / 'tools'))
|
||||
from probe_batch_runtime_live import hashes, compare_saved, save
|
||||
|
||||
|
||||
def read_json(path):
|
||||
try:
|
||||
return json.loads(path.read_text(encoding='utf-8'))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
class StartupFault:
|
||||
"""Stop only certified own viewdraw PIDs while their session is starting."""
|
||||
def __init__(self, root, output, failures, cancel):
|
||||
self.root, self.output, self.failures, self.cancel = root, output, failures, cancel
|
||||
self.stop = Event()
|
||||
self.faults, self.observations, self.errors = [], [], []
|
||||
self.seen = set()
|
||||
|
||||
def scan(self):
|
||||
import win32api
|
||||
import win32event
|
||||
import win32process
|
||||
from dx_productivity.trial_task_center.isolated_desktop import _Win32Api
|
||||
from dx_productivity.trial_task_center.path_access import internal_path
|
||||
for request in (self.root / 'batch-resources').glob('*/request.json'):
|
||||
value = read_json(request)
|
||||
if not value or value['operation'] != 'execute_task_queue':
|
||||
continue
|
||||
directory = request.parent
|
||||
session = read_json(directory / 'session.json')
|
||||
journal = read_json(directory / 'supervisor.json')
|
||||
if not session or not journal or not journal.get('process'):
|
||||
continue
|
||||
assert session['session_id'] == directory.name == value['batch_id']
|
||||
for record_path in directory.glob('generation-*/private-desktop.json'):
|
||||
record = read_json(record_path)
|
||||
if not record or not record.get('cad_pid'):
|
||||
continue
|
||||
pid = record['cad_pid']
|
||||
key = (directory.name, record['token'])
|
||||
if key in self.seen:
|
||||
continue
|
||||
generation = int(record_path.parent.name.removeprefix('generation-'))
|
||||
if generation != session['generation'] or record['desktop'] != session['desktop']:
|
||||
continue
|
||||
identities = [item for item in session['processes'] if item['pid'] == pid]
|
||||
if not identities:
|
||||
continue
|
||||
assert len(identities) == 1 and identities[0]['creation_time']
|
||||
identity = identities[0]
|
||||
handle = win32api.OpenProcess(0x1000 | 0x100000 | 1, False, pid)
|
||||
try:
|
||||
actual_created = str(win32process.GetProcessTimes(handle)['CreationTime'])
|
||||
assert actual_created == identity['creation_time'], 'PID creation time mismatch'
|
||||
listed = _Win32Api().service.GetProcessWindowStation().EnumDesktops()
|
||||
observation = {'generation': generation, 'pid': pid,
|
||||
'creation_time': actual_created, 'desktop': record['desktop'],
|
||||
'listed_desktops': [name for name in listed if name.startswith('dx-')],
|
||||
'state': record['state'], 'time_ns': time.time_ns()}
|
||||
self.observations.append(observation)
|
||||
if generation <= self.failures:
|
||||
# Recheck the same record immediately before the one
|
||||
# authorized termination. No COM attachment is needed.
|
||||
current = read_json(record_path)
|
||||
live_session = read_json(directory / 'session.json')
|
||||
assert current and live_session
|
||||
assert current['state'] == 'starting' and 'guid' not in current
|
||||
assert current['cad_pid'] == pid and current['token'] == record['token']
|
||||
assert current['desktop'] == live_session['desktop']
|
||||
assert live_session['generation'] == generation
|
||||
assert not list(record_path.parent.glob('cad-binding-*.json'))
|
||||
assert not list(directory.parent.parent.rglob('cad-ready-*.json'))
|
||||
prepared = list(directory.glob(f'attempts/g{generation}/**/*.json'))
|
||||
assert len(prepared) == 1
|
||||
marker = read_json(prepared[0])
|
||||
assert marker and marker['state'] == 'prepared'
|
||||
# The initial record is written immediately after
|
||||
# launch, before bootstrap/project worker authorization.
|
||||
assert win32event.WaitForSingleObject(handle, 0) == 258
|
||||
win32process.TerminateProcess(handle, 71)
|
||||
assert win32event.WaitForSingleObject(handle, 5000) == 0
|
||||
self.faults.append({**observation, 'terminated_owned_startup_only': True,
|
||||
'marker': str(prepared[0]), 'marker_before': marker,
|
||||
'marker_sha256_before': hashlib.sha256(prepared[0].read_bytes()).hexdigest()})
|
||||
self.seen.add(key)
|
||||
save(self.output / 'fault-observations.json', {'faults': self.faults,
|
||||
'observations': self.observations, 'errors': self.errors})
|
||||
finally:
|
||||
handle.Close()
|
||||
|
||||
def _watch(self):
|
||||
while not self.stop.wait(.02):
|
||||
try:
|
||||
self.scan()
|
||||
except Exception as error:
|
||||
self.errors.append(repr(error))
|
||||
self.cancel.set()
|
||||
save(self.output / 'fault-observations.json', {'faults': self.faults,
|
||||
'observations': self.observations, 'errors': self.errors})
|
||||
return
|
||||
|
||||
def __enter__(self):
|
||||
self.thread = Thread(target=self._watch, daemon=True)
|
||||
self.thread.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *_):
|
||||
self.stop.set()
|
||||
self.thread.join(5)
|
||||
assert not self.thread.is_alive()
|
||||
|
||||
|
||||
def run(source, library, executable, base, failures):
|
||||
from dx_productivity.trial_task_center import batch_supervisor
|
||||
from dx_productivity.trial_task_center.batch_resources import pending_resources
|
||||
from dx_productivity.trial_task_center.bom_plan import write_bom_template
|
||||
from dx_productivity.trial_task_center.isolated_cad import IsolatedCadSession
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.task_queue import execute_task_queue
|
||||
from private_desktop_observer import PrivateDesktopObserver
|
||||
|
||||
source, library, executable = [path.resolve(strict=True) for path in (source, library, executable)]
|
||||
assert executable.name == 'DxPartReplace.exe'
|
||||
output = base.resolve() / ('run-' + uuid4().hex)
|
||||
assert output.is_relative_to(REPO / 'build' / 'issue-3-frozen-startup-recovery')
|
||||
output.mkdir(parents=True, exist_ok=False)
|
||||
source_hashes, library_hash = hashes(source), hashlib.sha256(library.read_bytes()).hexdigest()
|
||||
folder = output / 'projects' / source.name
|
||||
shutil.copytree(source, folder)
|
||||
project = folder / (source.name + '.prj')
|
||||
text, count = re.subn(r'(?m)^KEY CentralLibrary .*$', lambda _: f'KEY CentralLibrary "{library.as_posix()}"',
|
||||
project.read_text(encoding='utf-8-sig'))
|
||||
assert count == 1
|
||||
project.write_text(text, encoding='utf-8')
|
||||
report = {'mode': 'source_supervised_parent_frozen_worker', 'gui_parent_covered': False,
|
||||
'frozen_executable': str(executable), 'exe_sha256': hashlib.sha256(executable.read_bytes()).hexdigest(),
|
||||
'project': str(project), 'requested_startup_failures': failures, 'launches': []}
|
||||
original_command = batch_supervisor._command
|
||||
def frozen_command(request):
|
||||
command = (str(executable), '--batch-worker', str(request))
|
||||
report['launches'].append({'command': command, 'request': read_json(request)})
|
||||
save(output / 'report.json', report)
|
||||
return command
|
||||
batch_supervisor._command = frozen_command
|
||||
store = TaskStore(output / 'tasks.db')
|
||||
service = TaskCenterService(store, configure_logging(output / 'data'), supervised=True)
|
||||
def independent(label):
|
||||
evidence = output / 'independent-read' / 'quick-run-history' / ('sheet-discovery-' + label)
|
||||
evidence.mkdir(parents=True)
|
||||
destination, spec = output / (label + '.json'), evidence / 'spec.json'
|
||||
save(spec, {'project': str(project), 'destination': str(destination)})
|
||||
with IsolatedCadSession(evidence) as cad:
|
||||
process = cad.start_worker((sys.executable, str(REPO / 'tools/probe_marker_write_failure_live.py'),
|
||||
'--inspect-file', str(spec)), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
try:
|
||||
code = process.wait(timeout=240)
|
||||
(evidence / 'worker-output.txt').write_bytes(process.stdout.read() + b'\n' + process.stderr.read())
|
||||
assert code == 0, (evidence / 'worker-output.txt').read_text(errors='replace')
|
||||
finally:
|
||||
process.close()
|
||||
return read_json(destination)
|
||||
started = time.monotonic()
|
||||
try:
|
||||
with PrivateDesktopObserver(output):
|
||||
print('Independent BEFORE (source read-only inspector)', flush=True)
|
||||
before = independent('before')
|
||||
tasks = service.create_project_tasks(output / 'projects', selected_paths=(project,))
|
||||
assert len(tasks) == 1
|
||||
bom = write_bom_template(output / 'input.xlsx', ({'project_path': str(project),
|
||||
'old_part_number': '4100-CA0000-T7', 'new_part_number': '4100-CA0000-T0',
|
||||
'refdes': 'RV05', 'enabled': True, 'sync_nc': True},))
|
||||
print('BOM import in frozen supervisor', flush=True)
|
||||
imported = service.inspect_bom_batch(tasks[0].task_id, bom,
|
||||
on_progress=lambda message: print(message, flush=True))
|
||||
assert imported.passed_rows == 1 and imported.preview is not None
|
||||
service.save_bom_batch(imported.preview, {item.task_id: item.rows for item in imported.preview.projects})
|
||||
cancel = Event()
|
||||
print('Frozen queue with owned startup CAD failure', flush=True)
|
||||
with StartupFault(output / 'data', output, failures, cancel) as injector:
|
||||
result = execute_task_queue(service, (tasks[0].task_id,), stop_event=cancel)
|
||||
report['queue_result'] = asdict(result)
|
||||
report['faults'], report['fault_errors'] = injector.faults, injector.errors
|
||||
assert not injector.errors and len(injector.faults) == failures
|
||||
execution = [item for item in report['launches'] if item['request']['operation'] == 'execute_task_queue']
|
||||
assert len(execution) == 1
|
||||
request = Path(execution[0]['command'][-1])
|
||||
session = read_json(request.parent / 'session.json')
|
||||
report['execution_session'] = session
|
||||
assert session['state'] == 'released' and session['generation'] == 2
|
||||
assert len(session['retries']) == 1 and next(iter(session['retries'].values()))['count'] == 1
|
||||
for fault in injector.faults:
|
||||
path = Path(fault['marker'])
|
||||
assert read_json(path)['state'] == 'prepared'
|
||||
assert hashlib.sha256(path.read_bytes()).hexdigest() == fault['marker_sha256_before']
|
||||
assert not pending_resources(output / 'data')
|
||||
assert not batch_supervisor.pending_supervisors(output / 'data')
|
||||
first_release, last_release = session['generation_releases'][0], session['generation_releases'][-1]
|
||||
assert first_release['cleanup']['desktop_confirmation']['released']
|
||||
second_create = last_release['cleanup']['desktop_handle_history'][0]['time_ns']
|
||||
assert first_release['released_at'] * 1e9 <= second_create
|
||||
report['old_desktop_released_before_second_create'] = True
|
||||
status_files = {str(path.relative_to(output)): path.read_text(encoding='utf-8-sig')
|
||||
for path in (output / 'data').rglob('status.tsv')}
|
||||
report['status_files'] = status_files
|
||||
saved_count = sum(sum(line.split('\t')[1:2] == ['saved_ok'] for line in text.splitlines())
|
||||
for text in status_files.values())
|
||||
assert saved_count == (1 if failures == 1 else 0)
|
||||
assert all(entry.state == ('completed' if failures == 1 else 'blocked') for entry in result.entries)
|
||||
print('Independent AFTER (source read-only inspector)', flush=True)
|
||||
after = independent('after')
|
||||
report['graph_comparison'] = compare_saved(before, after) if failures == 1 else {'identical': before == after}
|
||||
if failures == 2:
|
||||
assert before == after
|
||||
observation = read_json(output / 'desktop-observation.json')
|
||||
assert not observation['observer_errors']
|
||||
assert observation['input_desktop_before'] == observation['input_desktop_after']
|
||||
assert not observation['owned_cad_became_foreground'] and not observation['owned_cad_on_input_desktop']
|
||||
assert all(observation['cad_exit_confirmed'].values()) and all(observation['desktop_absent'].values())
|
||||
report['verified'] = True
|
||||
except BaseException as error:
|
||||
report.update(verified=False, error=repr(error), traceback=traceback.format_exc())
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
batch_supervisor._command = original_command
|
||||
store.close()
|
||||
from dx_productivity.trial_task_center.legacy_paths import cleanup_worker_aliases
|
||||
report['alias_cleanup_errors'] = list(map(str, cleanup_worker_aliases()))
|
||||
report['source_unchanged'] = hashes(source) == source_hashes
|
||||
report['library_unchanged'] = hashlib.sha256(library.read_bytes()).hexdigest() == library_hash
|
||||
report['elapsed_seconds'] = time.monotonic() - started
|
||||
save(output / 'report.json', report)
|
||||
print(json.dumps({key: report.get(key) for key in ('verified', 'error', 'elapsed_seconds')}, ensure_ascii=False), flush=True)
|
||||
return 0 if (report.get('verified') and report['source_unchanged'] and report['library_unchanged']
|
||||
and not report['alias_cleanup_errors']) else 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--source', type=Path, required=True)
|
||||
parser.add_argument('--library', type=Path, required=True)
|
||||
parser.add_argument('--executable', type=Path, default=REPO / 'dist/DxPartReplace/DxPartReplace.exe')
|
||||
parser.add_argument('--output', type=Path, default=REPO / 'build/issue-3-frozen-startup-recovery')
|
||||
parser.add_argument('--failures', type=int, choices=(1, 2), default=1)
|
||||
args = parser.parse_args()
|
||||
raise SystemExit(run(args.source, args.library, args.executable, args.output, args.failures))
|
||||
@@ -0,0 +1,310 @@
|
||||
"""Source-only real CAD fail-closed probe using Windows deny-delete file handles.
|
||||
|
||||
Only this test process patches _prepare_attempts. Production sources and VBS are
|
||||
unchanged. All project operations target a newly copied disposable project.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from dataclasses import asdict
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from threading import Event
|
||||
import time
|
||||
import traceback
|
||||
from uuid import uuid4
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(REPO / 'src'))
|
||||
sys.path.insert(0, str(REPO / 'tools'))
|
||||
|
||||
from probe_batch_runtime_live import hashes, inspect_native, save
|
||||
|
||||
|
||||
def read_all_sheets(spec_path):
|
||||
"""Discover all sheets in a private CAD, then use the independent full reader."""
|
||||
import pythoncom
|
||||
import win32com.client
|
||||
from dx_productivity.trial_task_center.cad_identity import project_file_path, verify_application
|
||||
spec = json.loads(spec_path.read_text(encoding='utf-8'))
|
||||
pythoncom.CoInitialize()
|
||||
app = None
|
||||
try:
|
||||
app = win32com.client.GetActiveObject('Viewdraw.Application')
|
||||
verify_application(app, require_view=False)
|
||||
assert not project_file_path(app)
|
||||
assert app.OpenProject(spec['project'])
|
||||
documents = app.SchematicSheetDocuments()
|
||||
spec['sheets'] = sorted((str(schematic), str(sheet))
|
||||
for schematic in documents.GetAvailableSchematics()
|
||||
for sheet in documents.GetAvailableSheets(str(schematic)))
|
||||
assert spec['sheets']
|
||||
documents = None
|
||||
app.CloseProject()
|
||||
assert not project_file_path(app)
|
||||
save(spec_path, spec)
|
||||
finally:
|
||||
if app is not None and project_file_path(app):
|
||||
app.CloseProject()
|
||||
app = None
|
||||
pythoncom.CoUninitialize()
|
||||
inspect_native(spec_path)
|
||||
|
||||
|
||||
def verify_publish_failure(output):
|
||||
reports = list((output / 'data').rglob('*.native.json'))
|
||||
matches = []
|
||||
for path in reports:
|
||||
value = json.loads(path.read_text(encoding='utf-8'))
|
||||
trace = value.get('traceback', '')
|
||||
if ('mark_snapshot_mutation' in trace and 'mark_mutation_possible' in trace
|
||||
and 'os.replace' in trace and 'PermissionError' in trace):
|
||||
matches.append({'path': str(path.relative_to(output)), 'error': value['error'],
|
||||
'traceback': trace, 'stage': value.get('stage')})
|
||||
assert len(matches) == 1, 'expected one actual native marker publication failure'
|
||||
return matches[0]
|
||||
|
||||
|
||||
def verify_existing(output):
|
||||
"""Read back one already attempted run; never executes BOM or queue business."""
|
||||
import win32api
|
||||
from dx_productivity.trial_task_center.batch_resources import pending_resources
|
||||
from dx_productivity.trial_task_center.isolated_cad import IsolatedCadSession
|
||||
from private_desktop_observer import PrivateDesktopObserver
|
||||
output = output.resolve(strict=True)
|
||||
assert output.is_relative_to(REPO / 'build/issue-3-marker-failure')
|
||||
original = json.loads((output / 'report.json').read_text(encoding='utf-8'))
|
||||
assert original['prepare_calls'] == 1 and len(original['locked_markers']) == 1
|
||||
assert original['queue_result']['entries'][0]['state'] == 'blocked'
|
||||
assert all('\tsaved_ok\t' not in text for text in original['status_files'].values())
|
||||
failure = verify_publish_failure(output)
|
||||
locked = original['locked_markers'][0]
|
||||
marker = Path(locked['path'])
|
||||
assert hashlib.sha256(marker.read_bytes()).hexdigest() == locked['sha256_before']
|
||||
assert json.loads(marker.read_bytes())['state'] == 'prepared'
|
||||
resource = json.loads((marker.parents[3] / 'session.json').read_text(encoding='utf-8'))
|
||||
assert resource['state'] == 'released' and resource['retries'] == {}
|
||||
assert not pending_resources(output / 'data')
|
||||
verification = output / 'readback-only'
|
||||
verification.mkdir(exist_ok=False)
|
||||
spec, destination = verification / 'spec.json', verification / 'after.json'
|
||||
save(spec, {'project': original['project'], 'destination': str(destination)})
|
||||
report = {'mode': 'read_only_existing_marker_failure', 'business_replayed': False,
|
||||
'source_coverage': True, 'frozen_coverage': False, 'marker_publish_failure': failure}
|
||||
started = time.monotonic()
|
||||
try:
|
||||
with PrivateDesktopObserver(verification) as observer:
|
||||
with IsolatedCadSession(verification / 'private-cad') as cad:
|
||||
observer.records[cad.desktop.name] = dict(cad.record)
|
||||
observer.handles[cad.cad.pid] = win32api.OpenProcess(0x100000, False, cad.cad.pid)
|
||||
process = cad.start_worker((sys.executable, str(Path(__file__).resolve()),
|
||||
'--inspect-file', str(spec)), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
try:
|
||||
code = process.wait(timeout=240)
|
||||
(verification / 'worker-output.txt').write_bytes(process.stdout.read() + b'\n' + process.stderr.read())
|
||||
assert code == 0, (verification / 'worker-output.txt').read_text(errors='replace')
|
||||
finally:
|
||||
process.close()
|
||||
before = json.loads((output / 'before.json').read_text(encoding='utf-8'))
|
||||
after = json.loads(destination.read_text(encoding='utf-8'))
|
||||
assert before == after, 'complete schematic attributes/pins changed'
|
||||
observation = json.loads((verification / 'desktop-observation.json').read_text(encoding='utf-8'))
|
||||
assert not observation['observer_errors']
|
||||
assert all(observation['cad_exit_confirmed'].values()) and all(observation['desktop_absent'].values())
|
||||
assert not observation['owned_cad_became_foreground'] and not observation['owned_cad_on_input_desktop']
|
||||
assert observation['input_desktop_before'] == observation['input_desktop_after']
|
||||
assert hashes(Path(original['source'])) == original['source_files_before']
|
||||
assert hashlib.sha256(Path(original['library']).read_bytes()).hexdigest() == original['library_sha256']
|
||||
assert hashlib.sha256(marker.read_bytes()).hexdigest() == locked['sha256_before']
|
||||
report.update(verified=True, graph_identical=True, components=len(before),
|
||||
pins=sum(len(row['pins']) for row in before.values()),
|
||||
sheets=sorted({(row['schematic'], row['sheet']) for row in before.values()}),
|
||||
target=[row for row in after.values() if row['refdes'] == 'RV05'],
|
||||
marker_unchanged=True, resources_released=True, source_unchanged=True, library_unchanged=True)
|
||||
finally:
|
||||
from dx_productivity.trial_task_center.legacy_paths import cleanup_worker_aliases
|
||||
report['alias_cleanup_errors'] = list(map(str, cleanup_worker_aliases()))
|
||||
report['elapsed_seconds'] = time.monotonic() - started
|
||||
save(verification / 'verification.json', report)
|
||||
print(json.dumps(report, ensure_ascii=False), flush=True)
|
||||
return 0 if report.get('verified') and not report['alias_cleanup_errors'] else 1
|
||||
|
||||
|
||||
def run(source, library, base):
|
||||
from dx_productivity.trial_task_center.batch_resources import pending_resources
|
||||
from dx_productivity.trial_task_center.batch_runtime import BatchCadRuntime
|
||||
from dx_productivity.trial_task_center.bom_plan import write_bom_template
|
||||
from dx_productivity.trial_task_center.isolated_cad import IsolatedCadSession
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.path_access import internal_path
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.task_queue import execute_task_queue
|
||||
from private_desktop_observer import PrivateDesktopObserver
|
||||
import win32con
|
||||
import win32file
|
||||
|
||||
source, library = source.resolve(strict=True), library.resolve(strict=True)
|
||||
output = base.resolve() / ('run-' + uuid4().hex)
|
||||
assert output.is_relative_to(REPO / 'build' / 'issue-3-marker-failure')
|
||||
output.mkdir(parents=True, exist_ok=False)
|
||||
source_before, library_before = hashes(source), hashlib.sha256(library.read_bytes()).hexdigest()
|
||||
folder = output / 'projects' / source.name
|
||||
shutil.copytree(source, folder)
|
||||
project = folder / (source.name + '.prj')
|
||||
text, count = re.subn(r'(?m)^KEY CentralLibrary .*$',
|
||||
lambda _: f'KEY CentralLibrary "{library.as_posix()}"', project.read_text(encoding='utf-8-sig'))
|
||||
assert count == 1
|
||||
project.write_text(text, encoding='utf-8')
|
||||
report = {'mode': 'source_real_cad_marker_publish_failure', 'frozen_coverage': False,
|
||||
'source': str(source), 'project': str(project), 'library': str(library),
|
||||
'old': '4100-CA0000-T7', 'new': '4100-CA0000-T0', 'refdes': 'RV05',
|
||||
'copy_part_number_to_value': False, 'locked_markers': [], 'prepare_calls': 0,
|
||||
'steps': [], 'source_files_before': source_before, 'library_sha256': library_before}
|
||||
save(output / 'source-under-test.json', {'git_head': subprocess.check_output(
|
||||
['git', 'rev-parse', 'HEAD'], cwd=REPO, text=True).strip(),
|
||||
'sha256': {str(path.relative_to(REPO)): hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
for folder_name, pattern in (('src', '*.py'), ('integration', '*.vbs'))
|
||||
for path in (REPO / folder_name).rglob(pattern)}})
|
||||
store = TaskStore(output / 'tasks.db')
|
||||
service = TaskCenterService(store, configure_logging(output / 'data'))
|
||||
handles = []
|
||||
original_prepare = BatchCadRuntime._prepare_attempts
|
||||
runtime = None
|
||||
def guarded_prepare(owner, root, command, attempt_id, generation):
|
||||
result = original_prepare(owner, root, command, attempt_id, generation)
|
||||
report['prepare_calls'] += 1
|
||||
_, directory, expected = result
|
||||
for path, record in expected:
|
||||
assert record['state'] == 'prepared'
|
||||
raw = internal_path(path).read_bytes()
|
||||
handle = win32file.CreateFile(str(internal_path(path)), win32con.GENERIC_READ,
|
||||
win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE, None,
|
||||
win32con.OPEN_EXISTING, win32con.FILE_ATTRIBUTE_NORMAL, None)
|
||||
handles.append(handle)
|
||||
assert internal_path(path).read_bytes() == raw
|
||||
report['locked_markers'].append({'path': str(path), 'handle': int(handle),
|
||||
'share_read': True, 'share_write': True, 'share_delete': False,
|
||||
'sha256_before': hashlib.sha256(raw).hexdigest(), 'prepared': record})
|
||||
save(output / 'report.json', report)
|
||||
return result
|
||||
def independent(label):
|
||||
evidence = output / 'independent-read' / 'quick-run-history' / ('sheet-discovery-' + label)
|
||||
evidence.mkdir(parents=True)
|
||||
destination, spec = output / (label + '.json'), evidence / 'spec.json'
|
||||
save(spec, {'project': str(project), 'destination': str(destination)})
|
||||
with IsolatedCadSession(evidence) as cad:
|
||||
process = cad.start_worker((sys.executable, str(Path(__file__).resolve()),
|
||||
'--inspect-file', str(spec)), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
try:
|
||||
code = process.wait(timeout=240)
|
||||
(evidence / 'worker-output.txt').write_bytes(process.stdout.read() + b'\n' + process.stderr.read())
|
||||
assert code == 0, (evidence / 'worker-output.txt').read_text(errors='replace')
|
||||
finally:
|
||||
process.close()
|
||||
return json.loads(destination.read_text(encoding='utf-8'))
|
||||
started = time.monotonic()
|
||||
try:
|
||||
with PrivateDesktopObserver(output):
|
||||
print('Independent full-project BEFORE read', flush=True)
|
||||
before = independent('before')
|
||||
assert [row['part_number'] for row in before.values() if row['refdes'] == 'RV05'] == [report['old']]
|
||||
tasks = service.create_project_tasks(output / 'projects', selected_paths=(project,))
|
||||
assert len(tasks) == 1
|
||||
bom = write_bom_template(output / 'input.xlsx', ({'project_path': str(project),
|
||||
'old_part_number': report['old'], 'new_part_number': report['new'],
|
||||
'refdes': 'RV05', 'enabled': True, 'sync_nc': True},))
|
||||
imported = service.inspect_bom_batch(tasks[0].task_id, bom,
|
||||
on_progress=lambda message: print(message, flush=True))
|
||||
assert imported.passed_rows == 1 and imported.preview is not None
|
||||
service.save_bom_batch(imported.preview, {item.task_id: item.rows for item in imported.preview.projects})
|
||||
print('Production queue with real deny-delete prepared marker handle', flush=True)
|
||||
BatchCadRuntime._prepare_attempts = guarded_prepare
|
||||
runtime = BatchCadRuntime(output / 'data', 'marker_write_failure')
|
||||
try:
|
||||
with runtime:
|
||||
result = execute_task_queue(service, (tasks[0].task_id,), stop_event=Event(), runtime=runtime)
|
||||
runtime.result = result
|
||||
report['queue_result'] = asdict(result)
|
||||
finally:
|
||||
BatchCadRuntime._prepare_attempts = original_prepare
|
||||
for handle in handles:
|
||||
handle.Close()
|
||||
handles.clear()
|
||||
assert report['prepare_calls'] == 1 and len(report['locked_markers']) == 1
|
||||
assert all(entry.state != 'completed' for entry in result.entries)
|
||||
for locked in report['locked_markers']:
|
||||
raw = internal_path(locked['path']).read_bytes()
|
||||
locked['sha256_after'] = hashlib.sha256(raw).hexdigest()
|
||||
assert locked['sha256_after'] == locked['sha256_before']
|
||||
assert json.loads(raw)['state'] == 'prepared'
|
||||
statuses = {str(path.relative_to(output)): path.read_text(encoding='utf-8-sig')
|
||||
for path in (output / 'data').rglob('status.tsv')}
|
||||
report['status_files'] = statuses
|
||||
assert statuses and not any('\tsaved_ok\t' in text for text in statuses.values())
|
||||
report['marker_publish_failure'] = verify_publish_failure(output)
|
||||
assert not pending_resources(output / 'data')
|
||||
report['resource_record'] = json.loads(runtime.path.read_text(encoding='utf-8'))
|
||||
assert report['resource_record']['state'] == 'released'
|
||||
assert report['resource_record']['retries'] == {}
|
||||
report['business_execution_attempts'] = report['prepare_calls']
|
||||
print('Independent full-project AFTER read', flush=True)
|
||||
after = independent('after')
|
||||
assert before == after, 'full component attributes or pin connectivity changed'
|
||||
report['graph_comparison'] = {'identical': True, 'components': len(before),
|
||||
'sheets': sorted({(row['schematic'], row['sheet']) for row in before.values()}),
|
||||
'pins': sum(len(row['pins']) for row in before.values()),
|
||||
'all_component_attributes_positions_and_pin_connections_compared': True}
|
||||
observation = json.loads((output / 'desktop-observation.json').read_text(encoding='utf-8'))
|
||||
assert not observation['observer_errors']
|
||||
assert observation['input_desktop_before'] == observation['input_desktop_after']
|
||||
assert not observation['owned_cad_became_foreground'] and not observation['owned_cad_on_input_desktop']
|
||||
assert all(observation['cad_exit_confirmed'].values()) and all(observation['desktop_absent'].values())
|
||||
report['verified'] = True
|
||||
except BaseException as failure:
|
||||
report.update(verified=False, error=repr(failure), traceback=traceback.format_exc())
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
BatchCadRuntime._prepare_attempts = original_prepare
|
||||
for handle in handles:
|
||||
handle.Close()
|
||||
if runtime is not None:
|
||||
try:
|
||||
runtime.close(reason='probe_finally')
|
||||
except Exception as failure:
|
||||
report['final_cleanup_error'] = repr(failure)
|
||||
store.close()
|
||||
from dx_productivity.trial_task_center.legacy_paths import cleanup_worker_aliases
|
||||
report['alias_cleanup_errors'] = list(map(str, cleanup_worker_aliases()))
|
||||
report['source_unchanged'] = hashes(source) == source_before
|
||||
report['library_unchanged'] = hashlib.sha256(library.read_bytes()).hexdigest() == library_before
|
||||
report['elapsed_seconds'] = time.monotonic() - started
|
||||
save(output / 'report.json', report)
|
||||
print(json.dumps({'output': str(output), **{key: report.get(key) for key in
|
||||
('verified', 'error', 'source_unchanged', 'library_unchanged', 'graph_comparison', 'elapsed_seconds')}},
|
||||
ensure_ascii=False), flush=True)
|
||||
return 0 if (report.get('verified') and report['source_unchanged'] and report['library_unchanged']
|
||||
and not report['alias_cleanup_errors'] and not report.get('final_cleanup_error')) else 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--inspect-file', type=Path)
|
||||
parser.add_argument('--verify-existing', type=Path, help='Read back only; no queue or business execution')
|
||||
parser.add_argument('--source', type=Path)
|
||||
parser.add_argument('--library', type=Path)
|
||||
parser.add_argument('--output', type=Path, default=REPO / 'build/issue-3-marker-failure')
|
||||
args = parser.parse_args()
|
||||
if args.inspect_file:
|
||||
read_all_sheets(args.inspect_file)
|
||||
elif args.verify_existing:
|
||||
raise SystemExit(verify_existing(args.verify_existing))
|
||||
else:
|
||||
assert args.source and args.library
|
||||
raise SystemExit(run(args.source, args.library, args.output))
|
||||
@@ -0,0 +1,129 @@
|
||||
"""Request ordinary stop during save on disposable projects; never replay a run."""
|
||||
from dataclasses import asdict
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
from threading import Event, Thread
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
sys.path[:0] = [str(REPO / 'src'), str(REPO / 'tools')]
|
||||
from probe_batch_runtime_live import hashes, compare_saved, save, OLD, NEW, NAMES
|
||||
from dx_productivity.trial_task_center.isolated_cad import IsolatedCadSession
|
||||
from dx_productivity.trial_task_center.logging_setup import configure_logging
|
||||
from dx_productivity.trial_task_center.service import TaskCenterService
|
||||
from dx_productivity.trial_task_center.store import TaskStore
|
||||
from dx_productivity.trial_task_center.bom_plan import write_bom_template
|
||||
from dx_productivity.trial_task_center.stop_control import StopControl
|
||||
from dx_productivity.trial_task_center.task_queue import execute_task_queue
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--source-root', required=True, type=Path)
|
||||
parser.add_argument('--library', required=True, type=Path)
|
||||
parser.add_argument('--run-native', action='store_true', required=True)
|
||||
args = parser.parse_args()
|
||||
output = REPO / 'build' / ('safe-stop-live-' + uuid4().hex[:10])
|
||||
output.mkdir()
|
||||
source_root, library = args.source_root.resolve(strict=True), args.library.resolve(strict=True)
|
||||
source_hashes = {name: hashes(source_root / name) for name in NAMES}
|
||||
library_hash = hashlib.sha256(library.read_bytes()).hexdigest()
|
||||
projects = []
|
||||
for name in NAMES:
|
||||
folder = output / 'projects' / name
|
||||
shutil.copytree(source_root / name, folder)
|
||||
project = folder / (name + '.prj')
|
||||
text, count = re.subn(r'(?m)^KEY CentralLibrary .*$', lambda _: f'KEY CentralLibrary "{library.as_posix()}"',
|
||||
project.read_text(encoding='utf-8-sig'))
|
||||
assert count == 1
|
||||
project.write_text(text, encoding='utf-8')
|
||||
projects.append(project)
|
||||
report = {'scope': 'source supervised service, production VBS and real CAD; no GUI or frozen EXE',
|
||||
'output': str(output), 'projects': list(map(str, projects))}
|
||||
print(json.dumps(report, ensure_ascii=False), flush=True)
|
||||
store = TaskStore(output / 'tasks.db')
|
||||
service = TaskCenterService(store, configure_logging(output / 'data'), supervised=True)
|
||||
control, finished = StopControl(), Event()
|
||||
watcher = None
|
||||
def inspect(project, sheets, label):
|
||||
evidence = output / (label + '-' + project.stem)
|
||||
evidence.mkdir()
|
||||
spec, result = evidence / 'spec.json', evidence / 'graph.json'
|
||||
save(spec, {'project': str(project), 'sheets': sorted(sheets), 'destination': str(result)})
|
||||
with IsolatedCadSession(evidence) as cad:
|
||||
child = cad.start_worker((sys.executable, str(REPO / 'tools/probe_batch_runtime_live.py'),
|
||||
'--repo', str(REPO), '--inspect-file', str(spec), '--run-native'), stdout=-1, stderr=-1)
|
||||
try:
|
||||
code = child.wait(timeout=180)
|
||||
(evidence / 'worker-output.txt').write_bytes(child.stdout.read() + b'\n' + child.stderr.read())
|
||||
assert code == 0, (evidence / 'worker-output.txt').read_text(errors='replace')
|
||||
finally:
|
||||
child.close()
|
||||
return json.loads(result.read_text(encoding='utf-8'))
|
||||
try:
|
||||
tasks = service.create_project_tasks(output / 'projects', selected_paths=tuple(projects))
|
||||
bom = write_bom_template(output / 'input.xlsx', ({'project_path': str(p), 'old_part_number': OLD,
|
||||
'new_part_number': NEW, 'refdes': 'RV05', 'enabled': True, 'sync_nc': True} for p in projects))
|
||||
imported = service.inspect_bom_batch(tasks[0].task_id, bom)
|
||||
assert imported.passed_rows == 2 and imported.preview is not None
|
||||
service.save_bom_batch(imported.preview, {p.task_id: p.rows for p in imported.preview.projects})
|
||||
before, sheet_sets = {}, {}
|
||||
for project in projects:
|
||||
rows = next(p.rows for p in imported.preview.projects if p.project_path.resolve() == project.resolve())
|
||||
sheet_sets[project.stem] = {(row.target.schematic_name, row.target.sheet_number) for row in rows}
|
||||
before[project.stem] = inspect(project, sheet_sets[project.stem], 'before')
|
||||
def stop_during_save():
|
||||
while not finished.wait(.01):
|
||||
for path in (output / 'data/tasks').glob('*/quick-run/quick-part-replace.log'):
|
||||
try:
|
||||
text = path.read_text(encoding='utf-8-sig')
|
||||
except OSError:
|
||||
continue
|
||||
if 'save_requested job=' in text:
|
||||
report['stop_requested'] = {'time_ns': time.time_ns(), 'log': str(path),
|
||||
'trigger': 'production save_requested log; ordinary stop only'}
|
||||
control.set()
|
||||
return
|
||||
watcher = Thread(target=stop_during_save, daemon=True)
|
||||
watcher.start()
|
||||
result = execute_task_queue(service, tuple(task.task_id for task in tasks), stop_event=control)
|
||||
finished.set()
|
||||
watcher.join(3)
|
||||
report['queue_result'] = asdict(result)
|
||||
assert report.get('stop_requested') and not control.force_event.is_set()
|
||||
assert result.paused and [row.state for row in result.entries] == ['completed', 'pending'], result
|
||||
report['comparisons'] = []
|
||||
for index, project in enumerate(projects):
|
||||
after = inspect(project, sheet_sets[project.stem], 'after')
|
||||
if index == 0:
|
||||
report['comparisons'].append(compare_saved(before[project.stem], after))
|
||||
else:
|
||||
assert after == before[project.stem], 'second project was changed after stop'
|
||||
report['comparisons'].append({'unchanged': True, 'components': len(after)})
|
||||
assert not service.pending_batch_resources()
|
||||
assert not list((output / 'data').rglob('uncertain.flag'))
|
||||
report['verified'] = True
|
||||
except BaseException as error:
|
||||
import traceback
|
||||
report.update(error=repr(error), traceback=traceback.format_exc())
|
||||
raise
|
||||
finally:
|
||||
finished.set()
|
||||
if watcher is not None:
|
||||
watcher.join(3)
|
||||
store.close()
|
||||
report['source_unchanged'] = all(hashes(source_root / name) == source_hashes[name] for name in NAMES)
|
||||
report['library_lmc_unchanged'] = hashlib.sha256(library.read_bytes()).hexdigest() == library_hash
|
||||
save(output / 'report.json', report)
|
||||
print(json.dumps(report, ensure_ascii=False), flush=True)
|
||||
return 0 if report.get('verified') and report['source_unchanged'] and report['library_lmc_unchanged'] else 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
Reference in new issue
Block a user