2 Commits
12 changed files with 765 additions and 135 deletions

No files matched your search

+12 -2
View File
@@ -18,12 +18,22 @@
导出.cmd --input "F:\工程目录" --output "F:\导出结果" --yes
导出.cmd --mentor-home "D:\Mentor\7.9.5EE\SDD_HOME" --input "F:\工程目录" --yes
导出.cmd --retry "F:\导出结果\批次_..." --yes
导出.cmd --input "F:\工程目录" --output "F:\导出结果" --jobs 3 --yes
导出.cmd --input "F:\工程目录" --jobs 1 --yes
不指定参数进入问答。--yes 适合使用固定参数重复运行;发现多套 Mentor 时需要 --mentor-home 指定。--timeout 可以调整单工具等待上限,默认 60 秒。
默认同时运行 3 个独立导出任务,可用 --jobs 1 切换串行,--jobs 2 到 8 调整并发上限。各工具使用同一已保存输入的独立副本;工程按顺序处理,库件读取在 ASCII 平铺属性就绪后与其他导出重叠,同库缓存继续共享。快照名称探测也参与同一并发上限,不额外占用任务槽。
工具退出通过进程通知立即唤醒,不额外睡眠;收尾与其他导出重叠执行。后台进程自然退出的宽限期最多 2 秒,已退出则立即结束等待。若仍有自有后台进程需要终止,会保留文件并记录提示,不标为已验证成功,也不阻塞其他任务。运行记录分别保存工具执行与收尾耗时,工程记录保存实际并发数与总耗时。
按 Ctrl+C 请求停止,等待工具收尾;不要直接关闭窗口。下次使用 --retry 指定批次,可恢复未完成项。成功产物只有在输入、工具和输出摘要仍匹配时才复用;源工程变化会完整重新导出该工程。
退出码:0 全部请求项成功;1 含扫描问题/部分成功/失败;2 环境或输入配置错误;130 已取消。
默认尽力导出并持续完成批次。单项异常、缺工具/缺库、超时、格式检查提示或进程收尾提示,都记录下来并继续后续工具和工程;报告文件被占用也不阻塞导出。只有用户主动按 Ctrl+C 才停止批次。
生成了非空文件便保留在正常分类目录,并显示「已导出」或「已导出(有提示)」。没有生成文件的工程显示「已跳过(无导出文件)」,单项显示「未导出」或「已跳过」。格式检查、非零退出和源文件变化保留为提示;带提示的产物不会作为已验证成功结果复用。收尾不确定时保留临时副本及自有进程句柄,继续使用独立副本运行其余任务。
退出码:0 批次已跑完(允许存在跳过项和提示);2 启动环境或输入配置错误,未能开始;130 用户主动取消。判断具体取得了哪些文件,应查看 HTML/JSON 中的逐项结果。
## 输出内容
@@ -38,7 +48,7 @@
工具、模式、快照、输入/输出摘要分别保存。同名字段不互相覆盖;分层属性、平铺有效值、网表身份表和当前中央库是不同来源。先看「导出汇总.html」,再进入工程的逐项说明。
“成功”表示请求的导出项生成并通过产物检查,不代表换料正确、ERC 通过或全部制造几何已核验。缺工具、COM 不匹配、缺库或不支持的内容均明确列出,不使用空文件冒充成功。
「已导出」表示文件已取得,不代表换料正确、ERC 通过或全部制造几何已核验。有检查提示时会注明具体原因;缺工具、COM 不匹配、缺库或不支持的内容均列出并跳过。
## 兼容与边界
+12
View File
@@ -0,0 +1,12 @@
# MentorExport 0.1.1
2026-11-09。根据工单 #1 后用户确认的新规则,默认以取得文件和跑完批次为目标。工单只有程序附件,没有导出批次日志,因此本版调整运行策略,不声称已复现现场未知故障。
- 格式检查不通过、工具非零退出或超时但生成了文件:保留到正常分类目录,显示「已导出(有提示)」;原始退出码、格式检查及输入摘要继续记录。
- 只取得部分导出内容,也不把整个工程判为失败;无产物时显示「已跳过(无导出文件)」。
- 收尾不确定、缺库、缺工具、预检帮助不匹配及单工程异常均不停止后续导出;收尾不确定仍保留临时副本与句柄证据。
- HTML 或记录文件写入异常改为提示;单个输出复制异常不会阻止其他文件保留。
- 正常结束批次统一退出 0,用户取消为 130,无法启动的配置错误为 2。退出 0 表示批次跑完,不代表所有内容完整。
- 带提示的产物不参与「已验证成功」重试复用。源工程和中央库仍只读,工具运行在隔离副本上。
本次不修改网表解析和比对功能,也不自动改动工程或重试换料。
+15
View File
@@ -0,0 +1,15 @@
# MentorExport 0.1.2
2026-11-09。加速原理图批量导出,保留 0.1.1 的单项问题不阻塞批次策略。
- 默认 3 个导出任务并发,支持 `--jobs 1` 串行和 `--jobs 2` 到 `--jobs 8`。问答模式直接采用默认 3,无需额外配置。
- 每工具独立数据库副本、输出、日志、私有桌面和进程 Job;仅协调线程更新工程记录,输出分类和逐项顺序稳定。
- ASCII 平铺属性完成后即可开始库件读取,与其他格式导出重叠;同库缓存继续在顺序处理的工程间共享。
- 快照探测与正文导出重叠,同样计入并发上限。
- 工具退出改为进程通知唤醒,消除固定轮询睡眠;收尾等待与其他导出重叠。保留后台进程最多 2 秒的自然退出宽限期,避免过早终止导致下次重试需要重导。
- 若后台进程仍需强制结束,文件照常保留、显示提示,不作为已验证成功结果复用;库件记录也不进入已验证缓存。进程退出确认和工作副本保留机制继续生效。
- 移除一次重复的库件记录写入,新增工具执行/进程收尾耗时及工程并发信息。
下载完整 ZIP 解压,双击 `导出.cmd`。本机需已有 Mentor 与可用许可,Python 和依赖已预置。实际兼容环境仍为 EE7.9.5;其他安装的原生库读取沿用已验证指纹准入。
实测同一对真实工程:v0.1.1 串行 156.015 秒,新版默认并发 74.594 秒,耗时减少 52.2%。22 项均取得且通过检查,EDF/KYN 连接和器件属性、关联库记录与串行结果相同;重试校验后复用耗时 13.609 秒。64 项自动测试通过。完整范围见 [并发验收](../validation-2026-11-09-parallel.md),实际速度取决于工程和机器环境。
@@ -2,6 +2,8 @@
日期:2026-11-06。状态:已获用户确认并实施;最终验收见 ../../validation-2026-11-06.md。
2026-11-09 更新:用户明确要求尽力导出、单项问题不阻塞批次。下文原始设计中的严格全成功判定和收尾不确定时停止批次,已被 0.1.1 的持续导出策略取代,现行规则见 ../../releases/v0.1.1.md。
## 已确认要求
- 独立、可复制的工具文件夹,以命令行启动,中文问答式交互。
+45
View File
@@ -0,0 +1,45 @@
# 0.1.2 并发导出验收
2026-11-09。本机 Windows x64、Mentor EE7.9.5。默认每工程 3 个导出任务并发,工程顺序处理,同库缓存继续共享。`--jobs 1` 保留串行入口,最大并发 8。
## 自动测试及审查
`python -X utf8 -m pytest -q`:**64 passed in 34.73s**。
新增回归先失败后通过,覆盖独立副本上的真实调度重叠、并发上限、稳定输出顺序、快照与正文导出重叠、库件在 ASCII 就绪后提前运行、并发取消后收尾、CLI 参数传递、无效并发数、进程退出通知唤醒,以及强制收尾产物不能作为已验证结果复用。
独立审查确认协调线程独占工程记录写入,worker 使用独立状态与副本,同库分区缓存仍为单写入者。审查发现的强制结束后误入成功缓存已修正,普通导出和原生库读取都保留产物但标为提示;末轮针对进程等待与库缓存复查无待处理发现。
## 同输入实测
从之前已验收的两个真实工程复制固定输入,分别使用不可修改的 v0.1.1 发行包和最终 v0.1.2 发行包。每次是独立新进程和新批次,未用 `--retry` 代替首次导出。
| 项目 | v0.1.1 串行 | v0.1.2 默认 3 并发 |
| --- | ---: | ---: |
| 两工程完整导出 | 156.015 秒 | 74.594 秒 |
| 取得的导出项 | 22 | 22 |
| 带提示项 | 0 | 0 |
| 批次退出码 | 0 | 0 |
本机样例耗时降低 **52.2%**,约为原速度的 **2.09 倍**。不同工程、磁盘负载及许可环境下不保证同一比例。
两工程逐一解析 EDIF 网表与 KYN:器件、字符串属性、物理引脚、网络及连接集合与串行结果完全一致。关联库件原生记录逐项完全一致。14 个工程输入文件摘要未变,工程内源数据库和源库检查均为 source_unchanged=true。
最终包复制到中文及空格目录,从 `C:\Windows\Temp` 通过真实 CMD 无参数问答启动,PATH 仅保留 System32,PYTHONHOME/PYTHONPATH 为无效目录。自动发现环境并使用预置 Python 成功。
随后使用 `--retry --jobs 1 --yes` 验证串行入口及重复使用:13.609 秒,退出 0,22 项成功产物校验后复用。664 个发行文件与 manifest、ZIP 内容均核对一致。
## 等待策略
固定 50 ms 轮询睡眠改为进程句柄等待,工具退出即唤醒;仍以最多 50 ms 检查用户取消和超时。正常后台进程保留最多 2 秒自然退出宽限期,等待与其他任务重叠。
曾以 0.25 秒宽限期测试,出现 14 项强制收尾提示,串行重试需重新导出并耗时 68.188 秒。最终保留 2 秒正常宽限期,解决重复运行退化;不靠过早终止后台进程制造加速。
## 证据
- 串行基线及初轮对照:`build/parallel-benchmark-20261109-141823/benchmark.json`。
- 最终实际问答、串行重试及逐项比较:`build/parallel-release-acceptance-20261109-143025/acceptance.json`。
- 原始逐工具记录新增 started_at、tool_seconds、cleanup_seconds;工程记录新增 execution.jobs 与 execution.seconds。
- 最终 ZIP SHA256:`28616ced54ea893058d6dcde00025a9a72122814ae4f0021ee790ff9ed7bf22c`。
本次未增加跨工程并发或原生库分区并发;主要加速来自独立导出任务并行、库件提前开始、快照探测重叠、进程退出通知及减少重复记录写入。
+28
View File
@@ -0,0 +1,28 @@
# 0.1.1 持续导出验收
2026-11-09。按用户明确要求调整:取得文件即保留,单项问题不阻塞后续工具和工程,完整跑完批次退出 0;仅主动取消为 130,无法启动为 2。
## 自动回归
`python -X utf8 -m pytest -q`:**55 passed in 69.90s**。
覆盖非零退出仍有文件、未识别格式、文件复制/摘要异常、伴随文件保留、日志被占用、预检返回异常及抛异常、中央库不可读、原文件导出中变化、进程收尾不确定后继续、临时副本删除失败、普通工程异常后继续、用户主动取消以及带提示的 ASCII 仍可提供库件引用。带提示产物不会作为已验证成功结果复用。
独立审查发现的遗漏均已修正并加入先失败后通过的回归;最后两项针对 ASCII 主文件选择与逐文件元数据错误的复查无待处理发现。
## 最终便携包实际批次
将最终 ZIP 对应工具复制到中文和空格目录,从 `C:\Windows\Temp` 使用真实 CMD 入口。PATH 仅保留 System32,PYTHONHOME/PYTHONPATH 指向无效路径;仍使用预置 Python 并自动发现本机 Mentor EE7.9.5。
- 真实工程 A:只在验收副本中清空中央库路径,10 项生成,库件读取跳过,工程标记「已导出(有提示)」。
- 真实工程 B:A 之后继续执行,全部 11 项生成。
- 第三个扫描条目缺数据库:记录为扫描问题,不阻止另外两个工程。
- 首轮实际导出 159.828 秒,退出码 0;随后重试 11.219 秒,退出码 0。
- 15 个验收输入文件摘要前后完全一致;两个工程的 source_unchanged 均为 true。
- 664 个发行文件及 ZIP 内文件与完整性清单一致。
机器可读证据:`build/best-effort-acceptance-20261109-110803/acceptance.json`。
最终发布 ZIP SHA256:`01ab5b5eec7f32f43d6670d9f4bfd893c505e7c73336587a25fbe1afd911e69b`。
工单 #1 原附件只有程序与配置,没有实际导出批次日志,本次未复现其未知现场根因;本版验证的是用户指定的新判定和继续执行策略。也未将导出文件标记为电气正确性校验通过。
+1 -1
View File
@@ -1,2 +1,2 @@
"""Portable Mentor saved-schematic exports."""
__version__ = "0.1.0"
__version__ = "0.1.2"
+64 -33
View File
@@ -95,7 +95,9 @@ def previous_results(record):
def run_batch(projects, problems, installation, output_root, scan_root, *, previous=None,
stop_event=None, timeout=60, exporter=None, progress=print):
stop_event=None, timeout=60, exporter=None, progress=print, jobs=3):
if not 1 <= jobs <= 8:
raise ValueError("并发数必须为 1 到 8")
if exporter is None:
from .exports import export_project
exporter = export_project
@@ -113,61 +115,83 @@ def run_batch(projects, problems, installation, output_root, scan_root, *, previ
"scan_root": str(Path(scan_root).resolve()), "output_root": str(output_root),
"created": datetime.now().astimezone().isoformat(), "installation": installation,
"owner": process_identity(os.getpid()),
"problems": problems, "projects": []}
write_json(batch / "环境信息.json", installation)
"problems": problems, "projects": [], "warnings": [], "execution": {"jobs": jobs}}
def warn(message):
if message not in record["warnings"]:
record["warnings"].append(message)
progress("提示:" + message)
def checkpoint():
try:
write_report(batch, record)
except Exception as error:
warn("汇总报告暂未写入,继续导出:" + str(error))
try:
write_json(batch / "batch.json", record)
except Exception as error:
warn("批次记录暂未写入,继续导出:" + str(error))
def save_result(row):
try:
write_json(Path(row["folder"]) / "export.json", row["result"])
except Exception as error:
warn(row["project"]["name"] + ":工程记录暂未写入,继续下一工程:" + str(error))
try:
write_json(batch / "环境信息.json", installation)
except Exception as error:
warn("环境记录暂未写入,继续导出:" + str(error))
prior = previous_results(previous or {})
for project in projects:
record["projects"].append({"project": project, "folder": str(batch / project_folder_name(project, scan_root)), "status": "pending"})
write_json(batch / "batch.json", record)
write_report(batch, record)
cleanup_uncertain = False
checkpoint()
for index, row in enumerate(record["projects"], 1):
if stop_event.is_set():
row["status"] = "cancelled"
row["result"] = {"status": "cancelled", "reason": "用户停止,尚未执行", "items": []}
write_json(Path(row["folder"]) / "export.json", row["result"])
save_result(row)
continue
project = row["project"]
row["status"] = "running"
write_json(batch / "batch.json", record)
checkpoint()
progress(f"\n[{index}/{len(projects)}] {project['name']}")
try:
result = exporter(project, installation, Path(row["folder"]), shared,
previous=prior.get(project["path"]), stop_event=stop_event,
timeout=timeout, progress=progress)
timeout=timeout, progress=progress, jobs=jobs)
row["result"] = result
row["status"] = result["status"]
if result.get("cleanup_uncertain"):
cleanup_uncertain = True
stop_event.set()
warn(project["name"] + ":进程收尾有提示,保留临时副本并继续下一工程")
except KeyboardInterrupt:
stop_event.set()
row["status"] = "cancelled"
row["result"] = load_json(Path(row["folder"]) / "export.json", {"items": []})
row["result"].update(status="cancelled", reason="用户停止")
except Exception as exc:
row["status"] = "failed"
row["status"] = "skipped"
row["result"] = load_json(Path(row["folder"]) / "export.json", {"items": []})
row["result"].update(status="failed", reason=str(exc))
if any(item.get("outputs") for item in row["result"].get("items", [])):
row["status"] = "exported"
row["result"].update(status=row["status"], reason=str(exc))
folder = Path(row["folder"]) / "90_运行日志"
folder.mkdir(parents=True, exist_ok=True)
(folder / "exception.txt").write_text(traceback.format_exc(), encoding="utf-8")
try:
folder.mkdir(parents=True, exist_ok=True)
(folder / "exception.txt").write_text(traceback.format_exc(), encoding="utf-8")
except OSError as log_error:
warn(project["name"] + ":异常日志暂未写入:" + str(log_error))
warn(project["name"] + ":" + str(exc) + ";已跳过并继续")
if getattr(exc, "cleanup_uncertain", False):
cleanup_uncertain = True
stop_event.set()
row["result"]["cleanup_uncertain"] = True
Path(row["folder"]).mkdir(parents=True, exist_ok=True)
write_json(Path(row["folder"]) / "export.json", row["result"])
write_json(batch / "batch.json", record)
write_report(batch, record)
save_result(row)
checkpoint()
progress("结果:" + LABELS.get(row["status"], row["status"]))
cancelled = stop_event.is_set() or any(p["status"] == "cancelled" for p in record["projects"])
record["state"] = "attention_required" if cleanup_uncertain else ("cancelled" if cancelled else "complete")
record["state"] = "cancelled" if cancelled else "complete"
record["finished"] = datetime.now().astimezone().isoformat()
code = 1 if cleanup_uncertain else (130 if cancelled else (1 if problems or not projects or any(p["status"] != "success" for p in record["projects"]) else 0))
code = 130 if cancelled else 0
record["exit_code"] = code
write_json(batch / "batch.json", record)
write_report(batch, record)
checkpoint()
return batch, code
@@ -219,6 +243,7 @@ def main(argv=None):
parser.add_argument("--mentor-home", help="指定 Mentor SDD_HOME")
parser.add_argument("--retry", help="重试已有批次,指定批次目录或 batch.json")
parser.add_argument("--timeout", type=float, default=60, help="单工具超时秒数,默认 60")
parser.add_argument("--jobs", "-j", type=int, default=3, help="同时运行的导出任务数,默认 3;1 为串行,最多 8")
parser.add_argument("--yes", "-y", action="store_true", help="采用已给定参数运行,不显示交互菜单")
parser.add_argument("--check", action="store_true", help="只检查预置环境与 Mentor")
parser.add_argument("--json", action="store_true", help="环境检查以 JSON 输出")
@@ -227,23 +252,29 @@ def main(argv=None):
try:
if args.timeout <= 0:
raise ValueError("超时必须大于 0")
if not 1 <= args.jobs <= 8:
raise ValueError("并发数必须为 1 到 8")
errors = runtime_check()
if errors:
raise ValueError("预置运行环境不完整,请重新解压发行包:\n" + "\n".join(errors))
settings = next((v for p in _settings_paths() if (v := load_json(p)) is not None), {})
installation = _choose_installation(args, settings)
diagnostic_root = Path(os.environ.get("LOCALAPPDATA", str(Path.home()))) / "MentorExport/diagnostics" / uuid.uuid4().hex
installation["preflight"] = preflight_installation(installation, diagnostic_root / "mentor-help")
try:
installation["preflight"] = preflight_installation(installation, diagnostic_root / "mentor-help")
except Exception as error:
installation["preflight"] = {"status": "failed", "reason": str(error)}
if installation["preflight"]["status"] != "success":
raise ValueError(installation["preflight"]["reason"] + "\n" + str(diagnostic_root))
print("提示:" + installation["preflight"]["reason"] + ";继续尝试各导出工具。\n" + str(diagnostic_root))
if args.check:
import json
print(json.dumps({"version": __version__, "python": sys.executable, "runtime_ok": True,
"installation": installation}, ensure_ascii=False, indent=2) if args.json else
f"运行环境正常。Mentor:{installation['version']} {installation['root']}")
return 0
return 0 if installation["preflight"]["status"] == "success" else 2
print(f"\nMentorExport {__version__} · 原理图详细导出")
print(f"Mentor:{installation['version']} {installation['root']}")
print(f"并发任务:{args.jobs}(--jobs 1 可切换串行)")
print("读取已保存的原理图。请先保存目标工程;工具仅在副本上运行。")
if not args.retry and settings.get("output"):
candidates = sorted(Path(settings["output"]).glob("批次_*/batch.json"), reverse=True)[:30]
@@ -252,7 +283,7 @@ def main(argv=None):
print("发现未完成批次:" + str(unfinished[0].parent))
if not args.yes and _ask("输入 1 重试该批次;回车开始新批次") == "1":
return main(["--retry", str(unfinished[0]), "--mentor-home", installation["root"],
"--timeout", str(args.timeout)])
"--timeout", str(args.timeout), "--jobs", str(args.jobs)])
previous = None
if args.retry:
previous_path = Path(args.retry)
@@ -304,14 +335,14 @@ def main(argv=None):
signal.signal(signal.SIGINT, cancel)
try:
batch, code = run_batch(projects, problems, installation, output_root, scan_root,
previous=previous, stop_event=stop, timeout=args.timeout)
previous=previous, stop_event=stop, timeout=args.timeout, jobs=args.jobs)
finally:
signal.signal(signal.SIGINT, old_handler)
record = load_json(batch / "batch.json")
record = load_json(batch / "batch.json", {"projects": []})
counts = {}
for row in record["projects"]:
counts[row["status"]] = counts.get(row["status"], 0) + 1
print("\n本批结束:" + " ".join(f"{LABELS.get(k, k)} {v}" for k, v in counts.items()))
print("\n本批已结束(单项问题不影响后续工程):" + " ".join(f"{LABELS.get(k, k)} {v}" for k, v in counts.items()))
print("汇总报告:" + str(batch / "导出汇总.html"))
print("下次重试:导出.cmd --retry \"" + str(batch) + "\"")
if not args.yes:
@@ -322,7 +353,7 @@ def main(argv=None):
if action in ("1", "2"):
os.startfile(str(batch if action == "1" else batch / "导出汇总.html"))
elif action == "3":
return main(["--retry", str(batch), "--mentor-home", installation["root"], "--timeout", str(args.timeout)])
return main(["--retry", str(batch), "--mentor-home", installation["root"], "--timeout", str(args.timeout), "--jobs", str(args.jobs)])
return code
except KeyboardInterrupt:
print("\n已取消。")
+167 -73
View File
@@ -1,6 +1,7 @@
"""Saved-project exports through short lived, owned, isolated tool processes."""
from copy import deepcopy
from contextlib import contextmanager
from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED
import ctypes
from datetime import datetime, timezone
import hashlib
@@ -9,6 +10,7 @@ import os
from pathlib import Path
import re
import shutil
import subprocess
import tempfile
import time
import uuid
@@ -38,16 +40,30 @@ _UNCERTAIN_DESKTOPS = []
_LIBRARY_CACHE = {}
def _checkpoint(path, record):
try:
write_json(path, record)
except Exception as error:
message = str(path) + ": " + str(error)
if message not in record.setdefault("warnings", []):
record["warnings"].append(message)
@contextmanager
def _workspace(result):
def _workspace(result, receipt=None):
path = Path(tempfile.mkdtemp(prefix="mexp-"))
try:
yield path
finally:
if result.get("cleanup_uncertain"):
if result.get("cleanup_uncertain") or result.get("workspace_preserved"):
result.setdefault("workspace_preserved", str(path))
else:
shutil.rmtree(path)
try:
shutil.rmtree(path)
except OSError as error:
result.update(workspace_preserved=str(path), workspace_cleanup_error=str(error))
if receipt is not None:
_checkpoint(receipt, result)
def _digest(value):
@@ -80,7 +96,7 @@ def _run(command, cwd, environment, log_dir, *, stop_event, timeout):
log_dir.mkdir(parents=True, exist_ok=True)
desktop = PrivateDesktop("mexp-" + uuid.uuid4().hex, enabled=True)
result = {"command": list(map(str, command)), "cwd": str(cwd), "exit_code": None,
"timed_out": False, "cancelled": False}
"timed_out": False, "cancelled": False, "started_at": datetime.now(timezone.utc).isoformat()}
started = time.monotonic()
try:
with desktop:
@@ -99,19 +115,25 @@ def _run(command, cwd, environment, log_dir, *, stop_event, timeout):
except Exception as error:
result["dialog_inspection_error"] = str(error)
break
time.sleep(0.05)
try:
process.wait(timeout=min(0.05, max(0, timeout - (time.monotonic() - started))))
except subprocess.TimeoutExpired:
pass
result["exit_code"] = process.poll()
result["tool_seconds"] = round(time.monotonic() - started, 3)
except Exception as error:
result["error"] = str(error)
finally:
result["seconds"] = round(time.monotonic() - started, 3)
if "tool_seconds" in result:
result["cleanup_seconds"] = round(result["seconds"] - result["tool_seconds"], 3)
result["cleanup"] = desktop.cleanup_record
cleanup = result["cleanup"] or {}
if cleanup.get("errors") or any(p.get("job_active_processes", 0) for p in cleanup.get("processes", [])):
result["cleanup_uncertain"] = True
# Retain owned handles for further cleanup instead of abandoning live resources.
_UNCERTAIN_DESKTOPS.append(desktop)
write_json(log_dir / "run.json", result)
_checkpoint(log_dir / "run.json", result)
return result
@@ -286,85 +308,83 @@ def _status(items):
return "cancelled"
if states and all(s == "success" for s in states):
return "success"
return "partial" if "success" in states else "failed"
return "exported" if any(i.get("outputs") for i in items) else "skipped"
def export_project(project, installation, project_dir, shared_dir, *, previous=None, stop_event=None, timeout=45, progress=print):
def export_project(project, installation, project_dir, shared_dir, *, previous=None, stop_event=None, timeout=45, progress=print, jobs=3):
if not 1 <= jobs <= 8:
raise ValueError("并发数必须为 1 到 8")
project_started = time.monotonic()
project_dir = Path(project_dir).resolve()
project_dir.mkdir(parents=True, exist_ok=True)
shared_dir = Path(shared_dir).resolve()
result = {"status": "running", "project": deepcopy(project), "captured_at": datetime.now(timezone.utc).isoformat(),
"source_fingerprint": None, "source_hashes": {}, "items": [], "source_unchanged": None,
"execution": {"jobs": jobs},
"coverage_limits": ["Exports preserve each tool/mode separately; no ERC or manufacturing certification.",
"CMS requests read mode but may report ReadOnly=False; only isolated copies are opened.",
"Cell coordinate units and rotation remain native integers; multi-slot/swappable mappings unverified."]}
state = project_dir / "export.json"
write_json(state, result)
if _UNCERTAIN_DESKTOPS:
result.update(status="failed", cleanup_uncertain=True,
reason="A previous owned process has unconfirmed cleanup; restart only after resolving its recorded Job/processes")
write_json(state, result)
return result
with _workspace(result) as temporary:
_checkpoint(state, result)
with _workspace(result, state) as temporary:
temp = _short(temporary)
captured = temp / "saved"
captured.mkdir()
try:
result["source_hashes"] = _capture(project, captured)
result["library_source_hashes"] = _hashes(_library_sources(project))
try:
result["library_source_hashes"] = _hashes(_library_sources(project))
except Exception as error:
result["library_source_hashes"] = {}
result["library_source_error"] = str(error)
result["coverage_limits"].append("Library fingerprint unavailable: " + str(error))
fingerprint = _digest({"source": result["source_hashes"], "snapshot": project["snapshot"],
"root_block": project["root_block"], "configuration": project.get("configuration"),
"library": result["library_source_hashes"]})
result["source_fingerprint"] = fingerprint
except Exception as error:
result.update(status="failed", reason=str(error), source_unchanged=False)
write_json(state, result)
result.update(status="skipped", reason=str(error), source_unchanged=False)
_checkpoint(state, result)
return result
environment = _environment(installation)
old_items = {i["key"]: i for i in (previous or {}).get("items", [])}
discovery = {"status": "failed", "names": []}
result["snapshot_discovery"] = discovery
try:
snapshot_work = temp / "snapshot-discovery"
shutil.copytree(captured, snapshot_work)
discovery.update(_snapshot_metadata(project, installation, snapshot_work,
project_dir / "90_运行日志" / "snapshots", stop_event=stop_event,
timeout=min(timeout, 20)))
if discovery.get("cleanup_uncertain"):
result.update(cleanup_uncertain=True, workspace_preserved=str(temp))
except Exception as error:
discovery.update(status="failed", reason=str(error))
if discovery["status"] != "success":
result["coverage_limits"].append("Other snapshot names were not obtained: " + discovery.get("reason", ""))
write_json(state, result)
for key, tool, folder in ITEMS:
def discover():
discovery = {"status": "failed", "names": []}
try:
snapshot_work = temp / "snapshot-discovery"
shutil.copytree(captured, snapshot_work)
discovery.update(_snapshot_metadata(project, installation, snapshot_work,
project_dir / "90_运行日志" / "snapshots", stop_event=stop_event,
timeout=min(timeout, 20)))
except Exception as error:
discovery.update(status="failed", reason=str(error))
return discovery
def execute(spec, task_state):
key, tool, folder = spec
output = None
item = {"key": key, "status": "failed", "reason": "", "outputs": [], "provenance": {}}
result["items"].append(item)
try:
progress(project["name"] + ": " + key)
identity = _tool_identity(key, tool, installation)
item["provenance"] = {"tool": identity, "snapshot": project["snapshot"], "root_block": project["root_block"],
"installation": installation["root"], "version": installation.get("version"),
"input_fingerprint": fingerprint, "reused": False}
if result.get("cleanup_uncertain"):
item.update(status="cancelled", reason="Not launched: previous owned process cleanup is uncertain")
continue
if stop_event is not None and stop_event.is_set():
item.update(status="cancelled", reason="Cancelled before launch")
continue
return item, task_state
if _reusable(old_items.get(key), fingerprint, identity):
item.update(deepcopy(old_items[key]))
item["provenance"]["reused"] = True
item["reason"] = "Verified previous successful output reused"
continue
return item, task_state
if key == "library":
_export_library(item, project, installation, result, project_dir / folder, shared_dir,
_export_library(item, project, installation, task_state, project_dir / folder, shared_dir,
temp, environment, stop_event, timeout)
continue
return item, task_state
if not identity.get("sha256") or (tool == "cms" and not identity.get("matches")):
item.update(status="unavailable", reason="Tool missing or COM registration does not match selected installation")
continue
return item, task_state
work = temp / key
shutil.copytree(captured, work)
output = work / "output"
@@ -375,8 +395,8 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
item["provenance"].update(run)
item["provenance"]["log_dir"] = str(log_dir)
if run.get("cleanup_uncertain"):
result.update(cleanup_uncertain=True, workspace_preserved=str(temp))
item["reason"] = "Owned process cleanup is uncertain; workspace preserved and later tools stopped"
task_state.update(cleanup_uncertain=True, workspace_preserved=str(temp))
item["reason"] = "Owned process cleanup is uncertain; workspace preserved; remaining exports continue"
elif run.get("cancelled"):
item.update(status="cancelled", reason="Cancelled while tool was running")
elif run.get("timed_out"):
@@ -384,7 +404,8 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
elif run.get("error") or run.get("exit_code") != 0 or run.get("cleanup", {}).get("errors"):
item["reason"] = run.get("error") or "Tool exited unsuccessfully or process cleanup failed"
else:
files = _validate(key, output)
_validate(key, output)
files = sorted(p for p in output.rglob("*") if p.is_file() and p.stat().st_size)
destination = project_dir / folder
published = []
for file in files:
@@ -393,26 +414,85 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
shutil.copyfile(file, target)
published.append(target)
item.update(status="success", outputs=[str(p) for p in published])
item["validation"] = {"status": "passed"}
item["provenance"]["output_files"] = [_file_info(p) for p in published]
if any(process.get("forced_termination") for process in run.get("cleanup", {}).get("processes", [])):
item.update(status="exported", reason="Files retained; owned background processes exceeded shutdown grace and were terminated; background completion unverified")
except Exception as error:
item.update(status="failed", reason=str(error))
finally:
if item["status"] in ("failed", "cancelled") and output is not None:
saved = []
publication_errors = []
try:
evidence = project_dir / "90_运行日志" / key / "failed-output"
saved = []
destination = project_dir / folder
for source in output.rglob("*"):
if source.is_file():
target = evidence / source.relative_to(output)
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(source, target)
saved.append(_file_info(target))
if saved:
item["provenance"]["failed_output_files"] = saved
try:
target = destination / source.relative_to(output)
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(source, target)
if target.stat().st_size:
saved.append(_file_info(target))
except OSError as error:
publication_errors.append(str(error))
task_state.setdefault("workspace_preserved", str(temp))
try:
if source.is_file() and source.stat().st_size:
saved.append(_file_info(source))
except OSError as metadata_error:
saved.append({"path": str(source), "size": None, "sha256": None,
"metadata_error": str(metadata_error)})
except OSError as error:
item["provenance"]["failure_evidence_error"] = str(error)
result["status"] = "running"
write_json(state, result)
publication_errors.append(str(error))
if saved:
item["outputs"] = [file["path"] for file in saved]
item["provenance"]["output_files"] = saved
if publication_errors:
item["provenance"]["publication_errors"] = publication_errors
if item["status"] == "failed" and item.get("outputs"):
item["status"] = "exported"
if item["status"] == "exported":
item["validation"] = {"status": "warning", "reason": item["reason"]}
return item, task_state
result["items"] = [{"key": key, "status": "pending", "outputs": [], "reason": "", "provenance": {}}
for key, tool, folder in ITEMS]
specs = {spec[0]: spec for spec in ITEMS}
positions = {item["key"]: index for index, item in enumerate(result["items"])}
ready = ["ascii_flat", "snapshots"] + [key for key, tool, folder in ITEMS if key not in ("ascii_flat", "library")]
# Only the coordinator publishes receipts; workers own distinct clones, items and logs.
with ThreadPoolExecutor(max_workers=jobs, thread_name_prefix="mentor-export") as pool:
active = {}
while ready or active:
while ready and len(active) < jobs:
key = ready.pop(0)
if key == "snapshots":
future = pool.submit(discover)
else:
task_state = {"items": list(result["items"]),
"library_source_hashes": result["library_source_hashes"]}
future = pool.submit(execute, specs[key], task_state)
active[future] = key
completed, _ = wait(active, return_when=FIRST_COMPLETED)
for future in completed:
key = active.pop(future)
if key == "snapshots":
metadata = future.result()
result["snapshot_discovery"] = metadata
task_state = metadata
if metadata["status"] != "success":
result["coverage_limits"].append("Other snapshot names were not obtained: " + metadata.get("reason", ""))
else:
item, task_state = future.result()
result["items"][positions[key]] = item
if key == "ascii_flat":
ready.insert(0, "library")
if task_state.get("cleanup_uncertain"):
result["cleanup_uncertain"] = True
if task_state.get("workspace_preserved"):
result["workspace_preserved"] = str(temp)
_checkpoint(state, result)
try:
after = _hashes(_inputs(project))
result["source_hashes_after"] = after
@@ -425,9 +505,11 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
if not result["source_unchanged"]:
for item in result["items"]:
if item["status"] == "success":
item.update(status="failed", reason="Source changed during export; save and run a full new attempt")
result["status"] = "failed" if result.get("cleanup_uncertain") else _status(result["items"])
write_json(state, result)
item.update(status="exported", reason="Source changed during export; captured output retained; re-export to verify current source",
validation={"status": "warning", "reason": "Source changed during export"})
result["execution"]["seconds"] = round(time.monotonic() - project_started, 3)
result["status"] = _status(result["items"])
_checkpoint(state, result)
return result
@@ -464,8 +546,6 @@ def list_snapshots(project, installation, work_dir):
executable = installation.get("tools", {}).get("constraints_xml")
if not executable:
return []
if _UNCERTAIN_DESKTOPS:
raise RuntimeError("A prior owned process has unconfirmed cleanup; snapshot discovery stopped")
workspace_state = {}
with _workspace(workspace_state) as temporary:
work = _short(temporary)
@@ -491,11 +571,15 @@ def _export_library(item, project, installation, result, destination, shared_dir
item.update(status="unavailable", reason="Central library missing or contains no local PDB/CEL partitions")
return
references = {"parts": set(), "cell": set()}
flat = next((i for i in result["items"] if i["key"] == "ascii_flat" and i["status"] == "success"), None)
flat = next((i for i in result["items"] if i["key"] == "ascii_flat" and i["status"] in ("success", "exported") and i.get("outputs")), None)
if not flat:
item.update(status="unavailable", reason="Effective ASCII properties required to identify actually used parts/cells")
return
for line in read_text(flat["outputs"][0]).splitlines():
report_path = next((Path(path) for path in flat["outputs"] if Path(path).name == "connections.txt"), None)
if report_path is None:
item.update(status="unavailable", reason="ASCII properties report was not generated")
return
for line in read_text(report_path).splitlines():
match = re.match(r"\s*Prop\s+(Part Number|Cell Name)\s+(.+?)\s*$", line)
if match:
references["parts" if match[1] == "Part Number" else "cell"].add(match[2])
@@ -562,17 +646,21 @@ def _export_library(item, project, installation, result, destination, shared_dir
log = destination.parent / "90_运行日志" / "library" / f"{index:03d}"
command = [str(HELPERS / "native_read.exe"), mode, str(dll_dir), str(copied)]
run = _run(command, work, environment, log, stop_event=stop_event, timeout=timeout)
if run.get("cleanup_uncertain"):
result.update(cleanup_uncertain=True, workspace_preserved=str(temp))
run["partition"] = relative
run["log_dir"] = str(log)
run["stdout_retained"] = True
write_json(log / "run.json", run)
_checkpoint(log / "run.json", run)
data["runs"].append(run)
persist()
if run.get("cleanup_uncertain"):
result.update(cleanup_uncertain=True, workspace_preserved=str(temp))
item.update(status="failed", reason="Library reader cleanup uncertain; workspace preserved and batch must stop")
item.update(status="failed", reason="Library reader cleanup uncertain; workspace preserved; remaining exports continue")
item["provenance"]["uncertain_run"] = run
return
data["errors"].append({"partition": relative, "reason": item["reason"]})
persist()
continue
if run.get("cancelled"):
item.update(status="cancelled", reason="Library reading cancelled")
return
@@ -580,6 +668,9 @@ def _export_library(item, project, installation, result, destination, shared_dir
data["errors"].append({"partition": relative, "reason": run.get("error") or "Native read failed/timeout/cleanup error"})
persist()
continue
trusted_run = not any(process.get("forced_termination") for process in run.get("cleanup", {}).get("processes", []))
if not trusted_run:
data["errors"].append({"partition": relative, "reason": "Owned background process terminated during shutdown; records retained without verified reuse"})
try:
native = json.loads((log / "stdout.txt").read_text(encoding="utf-8"))
_validate_native(native, mode)
@@ -595,15 +686,16 @@ def _export_library(item, project, installation, result, destination, shared_dir
if known:
old_names = {row["name"] for row in selected}
selected.extend(row for row in known["selected"] if row["name"] not in old_names)
partitions[relative] = {"names": {row["name"] for row in native["records"]},
"selected": selected, "run": deepcopy(run)}
if trusted_run:
partitions[relative] = {"names": {row["name"] for row in native["records"]},
"selected": selected, "run": deepcopy(run)}
# Remove unrelated full-library stdout only after selected records are durable.
persist()
(log / "stdout.txt").unlink()
run["stdout_retained"] = False
write_json(log / "run.json", run)
partitions[relative]["run"] = deepcopy(run)
persist()
_checkpoint(log / "run.json", run)
if trusted_run:
partitions[relative]["run"] = deepcopy(run)
except (OSError, ValueError) as error:
data["errors"].append({"partition": relative, "reason": str(error)})
persist()
@@ -629,7 +721,9 @@ def _export_library(item, project, installation, result, destination, shared_dir
if data["errors"] or any(missing.values()) or any(ambiguous.values()):
item.update(status="failed", reason="Partial library coverage: missing/ambiguous names or failed partitions; see index")
else:
item.update(status="success", reason="Used part/cell records read from fingerprinted shared library")
item.update(status="success" if flat["status"] == "success" else "exported",
reason="Used part/cell records read from fingerprinted shared library" +
("; references came from ASCII export with warnings" if flat["status"] != "success" else ""))
def _validate_native(native, mode):
+14 -5
View File
@@ -5,8 +5,8 @@ from pathlib import Path
import os
from urllib.parse import quote
LABELS = {"success": "成功", "partial": "部分成功", "failed": "失败",
"unavailable": "不可用", "cancelled": "已取消", "running": "执行中", "pending": "待执行"}
LABELS = {"success": "已导出", "exported": "已导出(有提示)", "partial": "已导出(有提示)", "failed": "未导出",
"skipped": "已跳过(无导出文件)", "unavailable": "已跳过", "cancelled": "已取消", "running": "执行中", "pending": "待执行"}
STYLE = """body{font:15px/1.65 'Segoe UI','Microsoft YaHei',sans-serif;background:#f4f6f9;color:#243246;margin:32px auto;max-width:1150px;padding:0 24px}
table{border-collapse:collapse;width:100%;background:white;margin:16px 0}th,td{padding:10px 14px;border:1px solid #dde3ec;text-align:left;vertical-align:top}th{background:#eaf0f8}
a{color:#1756a9;overflow-wrap:anywhere}.success{color:#137146}.failed{color:#b42318}.partial,.unavailable{color:#925600}code{background:#e9edf4;padding:2px 5px}small{color:#5c6a7d}.card{background:white;border:1px solid #dde3ec;border-radius:10px;padding:18px;margin:16px 0}"""
@@ -34,7 +34,6 @@ def write_report(batch, record):
for row in record.get("projects", []):
project = row["project"]
folder = Path(row["folder"])
folder.mkdir(parents=True, exist_ok=True)
result = row.get("result", {})
items = []
for item in result.get("items", []):
@@ -53,20 +52,30 @@ def write_report(batch, record):
f'来源:{escape(str(project["path"]))}<br>'
f'根块:{escape(str(project.get("root_block", "")))} 快照:{escape(str(project.get("snapshot", "")))}<br>'
f'数据库快照名称(仅记录):{escape(snapshot_names)}<br>'
+ (f'暂存文件位置:{link(result["workspace_preserved"], folder)}<br>' if result.get("workspace_preserved") else '') +
f'{escape(str(result.get("reason", "")))}</div>'
'<p>各工具和模式的输出分别保留;同名字段可能具有不同来源。导出成功不等于工程电气正确性校验通过。</p>'
'<table><tr><th>导出项</th><th>来源</th><th>状态</th><th>说明</th><th>产物</th></tr>' + "".join(items) + '</table>'
+ ('<h2>覆盖限制</h2><ul>' + limits + '</ul>' if limits else '')
+ link(folder / "export.json", folder, "工程原始记录 JSON")
+ " " + link(folder / "90_运行日志", folder, "运行日志"))
(folder / "工程说明.html").write_text(page(project["name"], details), encoding="utf-8")
try:
folder.mkdir(parents=True, exist_ok=True)
(folder / "工程说明.html").write_text(page(project["name"], details), encoding="utf-8")
except OSError as error:
message = project["name"] + ":工程说明暂未写入:" + str(error)
if message not in record.setdefault("warnings", []):
record["warnings"].append(message)
rows.append(f'<tr><td>{link(folder / "工程说明.html", batch, project["name"])}</td>'
f'<td class="{escape(row["status"])}">{LABELS.get(row["status"], escape(row["status"]))}</td>'
f'<td>{escape(str(result.get("reason", "")))}</td><td>{escape(str(project["path"]))}</td></tr>')
issues = "".join(f'<li>{escape(str(p.get("path", "")))}:{escape(str(p.get("reason", "")))}</li>' for p in record.get("problems", []))
content = (f'<div class="card">{escape(summary or "没有可执行工程")}<br>批次状态:{escape(record.get("state", ""))}</div>'
warnings = "".join('<li>' + escape(str(message)) + '</li>' for message in record.get("warnings", []))
batch_state = {"complete": "已完成", "running": "执行中", "cancelled": "已取消"}.get(record.get("state"), record.get("state", ""))
content = (f'<div class="card">{escape(summary or "没有可执行工程")}<br>批次状态:{escape(batch_state)}<br>单项问题记录提示,批次继续执行。</div>'
'<p>默认读取已保存的原理图;不同工具的字段和覆盖范围分别记录。点击工程名查看逐项结果。</p>'
'<table><tr><th>工程</th><th>结果</th><th>说明</th><th>源文件</th></tr>' + "".join(rows) + '</table>'
+ ('<h2>扫描问题</h2><ul>' + issues + '</ul>' if issues else '')
+ ('<h2>运行提示</h2><ul>' + warnings + '</ul>' if warnings else '')
+ link(batch / "batch.json", batch, "批次记录 JSON"))
(batch / "导出汇总.html").write_text(page("Mentor 原理图批量导出", content), encoding="utf-8")
+82 -2
View File
@@ -12,8 +12,8 @@ def test_batch_does_not_stop_after_one_project_failure(tmp_path):
projects = [{"name": name, "path": str(tmp_path / (name + ".prj"))} for name in ("坏工程", "好工程")]
batch, code = run_batch(projects, [], {}, tmp_path / "out", tmp_path, exporter=exporter)
state = json.loads((batch / "batch.json").read_text(encoding="utf-8"))
assert code == 1 and state["state"] == "complete"
assert [p["status"] for p in state["projects"]] == ["failed", "success"]
assert code == 0 and state["state"] == "complete"
assert [p["status"] for p in state["projects"]] == ["skipped", "success"]
assert (batch / "导出汇总.html").is_file()
@@ -69,3 +69,83 @@ def test_retry_cannot_claim_running_owner_has_exited():
import pytest
with pytest.raises(ValueError, match="运行"):
assert_retry_safe({"state": "running", "owner": process_identity(os.getpid())})
def test_uncertain_cleanup_does_not_cancel_following_project(tmp_path):
from mentorexport.cli import run_batch
visited = []
def exporter(project, installation, folder, shared, **options):
visited.append(project["name"])
return {"status": "skipped", "cleanup_uncertain": True, "items": []}
projects = [{"name": name, "path": str(tmp_path / (name + ".prj"))} for name in ("工程A", "工程B")]
stop = Event()
batch, code = run_batch(projects, [], {}, tmp_path / "out", tmp_path, exporter=exporter, stop_event=stop)
state = json.loads((batch / "batch.json").read_text(encoding="utf-8"))
assert visited == ["工程A", "工程B"]
assert code == 0 and state["state"] == "complete" and not stop.is_set()
assert state["warnings"]
def test_report_error_does_not_block_batch(tmp_path, monkeypatch):
from mentorexport import cli
visited = []
def broken_report(*args):
raise OSError("HTML file is locked")
def exporter(project, installation, folder, shared, **options):
visited.append(project["name"])
return {"status": "success", "items": []}
monkeypatch.setattr(cli, "write_report", broken_report)
projects = [{"name": name, "path": str(tmp_path / (name + ".prj"))} for name in ("工程A", "工程B")]
batch, code = cli.run_batch(projects, [], {}, tmp_path / "out", tmp_path, exporter=exporter)
record = json.loads((batch / "batch.json").read_text(encoding="utf-8"))
assert visited == ["工程A", "工程B"] and code == 0
assert any("HTML file is locked" in message for message in record["warnings"])
def test_scan_problems_and_no_projects_do_not_return_failed_batch(tmp_path):
from mentorexport.cli import run_batch
batch, code = run_batch([], [{"path": "missing.prj", "reason": "missing database"}], {}, tmp_path / "out", tmp_path)
state = json.loads((batch / "batch.json").read_text(encoding="utf-8"))
assert code == 0 and state["state"] == "complete" and state["problems"]
def test_export_continues_despite_help_preflight_failure(tmp_path, monkeypatch):
from mentorexport import cli
monkeypatch.setattr(cli, "runtime_check", lambda: [])
monkeypatch.setattr(cli, "_choose_installation", lambda *args: {"root": str(tmp_path), "version": "test"})
monkeypatch.setattr(cli, "preflight_installation", lambda *args: {"status": "failed", "reason": "help did not match"})
monkeypatch.setattr(cli, "_settings_paths", lambda: [])
monkeypatch.setattr(cli, "_save_settings", lambda value: None)
result = cli.main(["--input", str(tmp_path), "--output", str(tmp_path / "out"), "--yes"])
assert result == 0
assert list((tmp_path / "out").glob("批次_*/batch.json"))
def test_preflight_exception_is_a_warning_not_a_batch_blocker(tmp_path, monkeypatch):
from mentorexport import cli
monkeypatch.setattr(cli, "runtime_check", lambda: [])
monkeypatch.setattr(cli, "_choose_installation", lambda *args: {"root": str(tmp_path), "version": "test"})
def locked(*args): raise PermissionError("diagnostic directory locked")
monkeypatch.setattr(cli, "preflight_installation", locked)
monkeypatch.setattr(cli, "_settings_paths", lambda: [])
monkeypatch.setattr(cli, "_save_settings", lambda value: None)
assert cli.main(["--input", str(tmp_path), "--output", str(tmp_path / "out"), "--yes"]) == 0
assert list((tmp_path / "out").glob("批次_*/batch.json"))
def test_batch_passes_worker_limit_and_records_execution_mode(tmp_path):
from mentorexport.cli import run_batch
limits = []
def exporter(project, installation, folder, shared, **options):
limits.append(options["jobs"])
return {"status": "success", "items": []}
batch, code = run_batch([{"name": "工程", "path": str(tmp_path / "a.prj")}], [], {},
tmp_path / "out", tmp_path, exporter=exporter, jobs=2)
record = json.loads((batch / "batch.json").read_text(encoding="utf-8"))
assert code == 0 and limits == [2] and record["execution"]["jobs"] == 2
def test_invalid_worker_limit_does_not_start_export(tmp_path):
from mentorexport.cli import main
for jobs in ("0", "9"):
assert main(["--input", str(tmp_path), "--jobs", jobs, "--yes"]) == 2
+323 -19
View File
@@ -25,6 +25,163 @@ def backend():
return exports
def test_parallel_exports_overlap_on_distinct_clones_and_keep_receipt_order(sample, monkeypatch):
from threading import Barrier, Lock
exports = backend()
project, install, root = sample
barrier, lock = Barrier(2), Lock()
calls, workspaces = [], []
run_ascii = ascii_runner(calls)
active = maximum = 0
def runner(command, cwd, environment, log_dir, **kwargs):
nonlocal active, maximum
with lock:
active += 1
maximum = max(maximum, active)
workspaces.append(Path(cwd))
try:
barrier.wait(timeout=5)
return run_ascii(command, cwd, environment, log_dir, **kwargs)
finally:
with lock:
active -= 1
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "parallel", root / "shared", jobs=2, progress=lambda _: None)
assert maximum == 2 and len(set(workspaces)) == 2
assert [item["key"] for item in result["items"]] == [spec[0] for spec in exports.ITEMS]
assert [i["key"] for i in result["items"] if i["status"] == "success"] == ["ascii_flat", "ascii_hier"]
assert result["source_unchanged"] and result["execution"]["jobs"] == 2
def test_library_starts_after_flat_without_waiting_for_other_exports(sample, monkeypatch):
from threading import Event
exports = backend()
project, install, root = sample
library_started = Event()
runner = ascii_runner([])
def run(command, *args, **kwargs):
if "-h" in command:
assert library_started.wait(5), "library waited for unrelated exports"
return runner(command, *args, **kwargs)
def library(item, project, installation, result, *args):
flat = next(i for i in result["items"] if i["key"] == "ascii_flat")
assert flat["status"] == "success" and Path(flat["outputs"][0]).is_file()
library_started.set()
item.update(status="unavailable", reason="fixture without library")
monkeypatch.setattr(exports, "_run", run)
monkeypatch.setattr(exports, "_export_library", library)
result = exports.export_project(project, install, root / "parallel", root / "shared", jobs=2, progress=lambda _: None)
assert library_started.is_set() and result["items"][1]["status"] == "success"
def test_snapshot_listing_overlaps_with_exports(sample, monkeypatch):
from threading import Barrier
exports = backend()
project, install, root = sample
install["tools"]["constraints_xml"] = install["tools"]["ascii"]
barrier = Barrier(2)
runner = ascii_runner([])
def run(command, cwd, environment, log_dir, **kwargs):
if "-list" in command or "-f" in command:
barrier.wait(timeout=5)
if "-list" in command:
Path(log_dir).mkdir(parents=True)
(Path(log_dir) / "stdout.txt").write_text(" -> Saved\n")
return {"exit_code": 0}
return runner(command, cwd, environment, log_dir, **kwargs)
monkeypatch.setattr(exports, "_run", run)
result = exports.export_project(project, install, root / "parallel", root / "shared", jobs=2, progress=lambda _: None)
assert result["snapshot_discovery"]["status"] == "success"
assert result["items"][0]["status"] == "success"
def test_forced_descendant_shutdown_keeps_files_but_never_reuses_as_verified(sample, monkeypatch):
exports = backend()
project, install, root = sample
calls = []
ascii_run = ascii_runner(calls)
def runner(*args, **kwargs):
run = ascii_run(*args, **kwargs)
run["cleanup"] = {"errors": [], "processes": [{"forced_termination": True,
"forced_root_termination": False, "forced_descendant_pids": [123],
"termination_reason": "descendant_grace_expired", "job_active_processes": 0}]}
return run
monkeypatch.setattr(exports, "_run", runner)
first = exports.export_project(project, install, root / "first", root / "shared", jobs=2, progress=lambda _: None)
assert first["items"][0]["status"] == "exported" and first["items"][0]["outputs"]
assert first["items"][0]["validation"]["status"] == "warning"
exports.export_project(project, install, root / "retry", root / "shared", jobs=2, previous=first, progress=lambda _: None)
assert len(calls) == 4
def test_forced_library_shutdown_retains_records_without_trusted_cache(sample, monkeypatch):
exports = backend()
project, install, root = sample
library = root / "library/PartsDBLibs"
library.mkdir(parents=True)
(library / "a.pdb").write_bytes(b"parts")
project["library"] = str(library.parent)
monkeypatch.setattr(exports, "VALIDATED_DLLS", {})
monkeypatch.setattr(exports, "_LIBRARY_CACHE", {})
calls = []
def runner(command, cwd, environment, log_dir, **kwargs):
if "-f" in command or "-h" in command:
output = Path(next(arg.split("=", 1)[1] for arg in command if arg.startswith("-file=")))
output.write_text("iCDB Ascii Report\n" + ("Block Top\n" if "-h" in command else "Config default\n") + "\tProp Part Number USED\n")
return dict(exit_code=0, cleanup={"errors": []})
calls.append(command)
Path(log_dir).mkdir(parents=True, exist_ok=True)
(Path(log_dir) / "stdout.txt").write_text(json.dumps({"mode": "parts", "count": 1, "records": [{"name": "USED"}]}), encoding="utf-8")
return dict(exit_code=0, cleanup={"errors": [], "processes": [{"forced_termination": True}]})
monkeypatch.setattr(exports, "_run", runner)
first = exports.export_project(project, install, root / "first", root / "shared", jobs=2, progress=lambda _: None)
item = next(i for i in first["items"] if i["key"] == "library")
assert item["status"] == "exported" and item["validation"]["status"] == "warning"
data = next(json.loads(Path(path).read_text(encoding="utf-8")) for path in item["outputs"] if Path(path).name.startswith("used-"))
assert data["records"]["parts"][0]["name"] == "USED" and data["complete"] is False
exports.export_project(project, install, root / "retry", root / "shared", jobs=2, previous=first, progress=lambda _: None)
assert len(calls) == 2
def test_parallel_cancel_drains_active_tasks_without_launching_queued_vendor_work(sample, monkeypatch):
from threading import Barrier
exports = backend()
project, install, root = sample
stop, barrier = Event(), Barrier(2)
calls = []
def runner(command, cwd, *args, **kwargs):
calls.append(command)
barrier.wait(timeout=5)
stop.set()
return dict(exit_code=None, cancelled=True, cleanup={"errors": []})
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "cancel", root / "shared", jobs=2,
stop_event=stop, progress=lambda _: None)
assert len(calls) == 2 and result["status"] == "cancelled"
assert all(item["status"] == "cancelled" for item in result["items"])
def test_process_exit_notification_advances_without_polling_delay(tmp_path, monkeypatch):
exports = backend()
class Process:
pid = 123
finished = False
def poll(self): return 0 if self.finished else None
def wait(self, timeout=None):
self.finished = True
return 0
class Desktop:
cleanup_record = {"errors": [], "processes": []}
def __init__(self, *args, **kwargs): pass
def __enter__(self): return self
def __exit__(self, *args): pass
def start_process(self, *args, **kwargs): return Process()
def inspect_dialogs(self, pid): return []
monkeypatch.setattr(exports, "PrivateDesktop", Desktop)
result = exports._run(["fixture"], tmp_path, {}, tmp_path / "logs", stop_event=None, timeout=0.15)
assert result["exit_code"] == 0 and not result["timed_out"]
def ascii_runner(calls):
def run(command, cwd, environment, log_dir, *, stop_event, timeout):
calls.append(command)
@@ -110,7 +267,8 @@ def test_modified_output_and_changed_tool_are_not_reused(sample, monkeypatch):
assert len(calls) == 5
def test_used_library_only_is_shared_and_source_fingerprinted(sample, monkeypatch):
@pytest.mark.parametrize("ascii_warning", [False, True])
def test_used_library_only_is_shared_and_source_fingerprinted(sample, monkeypatch, ascii_warning):
exports = backend()
project, install, root = sample
library = root / "library"
@@ -132,6 +290,7 @@ def test_used_library_only_is_shared_and_source_fingerprinted(sample, monkeypatc
out = Path(next(p.split("=", 1)[1] for p in command if p.startswith("-file=")))
out.write_text("iCDB Ascii Report\n" + ("Block Top\n" if "-h" in command else "Config default\n") +
"".join(" Symbol id\n\tProp Part Number " + part + "\n\tProp Cell Name FOOTPRINT\n" for part in active_parts))
(out.parent / "companion.txt").write_text("additional exported data")
else:
native_calls.append(command)
mode = command[1]
@@ -139,11 +298,11 @@ def test_used_library_only_is_shared_and_source_fingerprinted(sample, monkeypatc
data = {"mode": mode, "count": len(rows), "records": rows}
Path(log_dir).mkdir(parents=True, exist_ok=True)
(Path(log_dir) / "stdout.txt").write_text(json.dumps(data), encoding="utf-8")
return dict(exit_code=0, cleanup={"errors": []})
return dict(exit_code=9 if ascii_warning and ("-f" in command or "-h" in command) else 0, cleanup={"errors": []})
monkeypatch.setattr(exports, "_run", runner)
first = exports.export_project(project, install, root / "one", root / "shared", progress=lambda _: None)
item = next(i for i in first["items"] if i["key"] == "library")
assert item["status"] == "success", item
assert item["status"] == ("exported" if ascii_warning else "success"), item
assert len(native_calls) == 2
assert all("UNUSED" not in p.read_text(encoding="utf-8") for p in (root / "shared").rglob("*.json"))
exports.export_project(project, install, root / "two", root / "new-shared", previous=first, progress=lambda _: None)
@@ -154,7 +313,7 @@ def test_used_library_only_is_shared_and_source_fingerprinted(sample, monkeypatc
assert len(native_calls) == 4
active_parts.append("NEW")
fourth = exports.export_project(project, install, root / "four", root / "shared", progress=lambda _: None)
assert next(i for i in fourth["items"] if i["key"] == "library")["status"] == "success"
assert next(i for i in fourth["items"] if i["key"] == "library")["status"] == ("exported" if ascii_warning else "success")
assert len(native_calls) == 5, "only partition containing previously unread NEW should rerun"
@@ -204,7 +363,7 @@ def test_batch_records_snapshot_names_from_captured_input(sample, monkeypatch):
assert result["project"]["snapshot"] == "Saved"
def test_uncertain_snapshot_cleanup_stops_batch_and_preserves_copy(sample, monkeypatch):
def test_uncertain_snapshot_cleanup_continues_exports_and_preserves_copy(sample, monkeypatch):
exports = backend()
project, install, root = sample
install["tools"]["constraints_xml"] = install["tools"]["ascii"]
@@ -214,14 +373,15 @@ def test_uncertain_snapshot_cleanup_stops_batch_and_preserves_copy(sample, monke
return dict(exit_code=0, cleanup_uncertain=True, cleanup={"errors": ["exit unconfirmed"]})
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
assert len(calls) == 1 and "-list" in calls[0]
assert len(calls) == 3 and any("-list" in call for call in calls)
assert result["cleanup_uncertain"] is True
assert result["status"] == "failed"
assert all(i["status"] == "cancelled" for i in result["items"])
assert result["status"] == "skipped"
assert not any(i["status"] == "cancelled" for i in result["items"])
preserved = Path(result["snapshot_discovery"]["workspace_preserved"])
assert (preserved / "database/icdb.dat").is_file()
exports.shutil.rmtree(preserved)
exports.shutil.rmtree(Path(result["workspace_preserved"]))
if Path(result["workspace_preserved"]) != preserved and Path(result["workspace_preserved"]).exists():
exports.shutil.rmtree(Path(result["workspace_preserved"]))
@pytest.mark.parametrize("key,name,text", [
@@ -256,7 +416,7 @@ def test_empty_oat_is_not_a_successful_pair(tmp_path):
exports._validate("wir_oat", tmp_path)
def test_failed_format_output_is_retained_as_failure_evidence(sample, monkeypatch):
def test_unrecognized_format_is_exported_with_warning_in_normal_category(sample, monkeypatch):
exports = backend()
project, install, root = sample
def malformed(command, cwd, environment, log_dir, **kwargs):
@@ -266,9 +426,14 @@ def test_failed_format_output_is_retained_as_failure_evidence(sample, monkeypatc
monkeypatch.setattr(exports, "_run", malformed)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
item = result["items"][0]
assert item["status"] == "failed"
assert item["provenance"]["failed_output_files"]
assert Path(item["provenance"]["failed_output_files"][0]["path"]).read_text() == "partial or malformed vendor output"
assert item["status"] == "exported"
assert item["validation"]["status"] == "warning"
path = Path(item["outputs"][0])
assert "02_器件与引脚属性" in str(path) and "90_运行日志" not in str(path)
assert path.read_text() == "partial or malformed vendor output"
assert result["status"] == "exported"
again = exports.export_project(project, install, root / "retry", root / "shared", previous=result, progress=lambda _: None)
assert not again["items"][0]["provenance"]["reused"]
@pytest.mark.parametrize("field", [{"hr": -2147467259, "value": None}, {"hr": 0, "value": None}])
@@ -323,7 +488,7 @@ def test_native_partition_diagnostics_and_accumulated_data_survive_failure(sampl
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
item = next(i for i in result["items"] if i["key"] == "library")
assert item["status"] == ("cancelled" if outcome == "cancel" else "failed")
assert item["status"] == ("cancelled" if outcome == "cancel" else "exported")
logs = root / "attempt/90_运行日志/library"
assert len(list(logs.rglob("stderr.txt"))) == 2
assert len(list(logs.rglob("run.json"))) == 2
@@ -338,7 +503,7 @@ def test_native_partition_diagnostics_and_accumulated_data_survive_failure(sampl
assert str(data_files[0]) in item["outputs"]
def test_uncertain_cleanup_preserves_workspace_and_stops_remaining_tools(sample, monkeypatch):
def test_uncertain_cleanup_preserves_workspace_and_continues_remaining_tools(sample, monkeypatch):
exports = backend()
project, install, root = sample
calls = []
@@ -347,13 +512,13 @@ def test_uncertain_cleanup_preserves_workspace_and_stops_remaining_tools(sample,
return dict(exit_code=0, cleanup_uncertain=True, cleanup={"errors": ["owned process exit unconfirmed"]})
monkeypatch.setattr(exports, "_run", uncertain)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
assert len(calls) == 1
assert len(calls) == 2
assert result["cleanup_uncertain"] is True
preserved = Path(result["workspace_preserved"])
assert preserved.is_dir()
assert calls[0].is_dir()
assert result["status"] == "failed"
assert all(i["status"] == "cancelled" for i in result["items"][1:])
assert result["status"] == "skipped"
assert not any(i["status"] == "cancelled" for i in result["items"])
# No real process is running in this synthetic boundary failure.
exports.shutil.rmtree(preserved)
@@ -369,11 +534,150 @@ def test_capture_rejects_a_source_that_changes_while_copying(sample, monkeypatch
return value
monkeypatch.setattr(exports.shutil, "copyfile", changing)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
assert result["status"] == "failed"
assert result["status"] == "skipped"
assert "changed" in result["reason"].lower()
assert not result["items"]
def test_nonzero_tool_with_output_is_retained_and_later_tool_runs(sample, monkeypatch):
exports = backend()
project, install, root = sample
calls = []
real_runner = ascii_runner(calls)
def runner(*args, **kwargs):
run = real_runner(*args, **kwargs)
run["exit_code"] = 7
return run
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
assert len(calls) == 2
assert all(item["status"] == "exported" and Path(item["outputs"][0]).is_file() for item in result["items"][:2])
assert result["items"][0]["provenance"]["exit_code"] == 7
@pytest.mark.parametrize("lock", ["copy", "metadata"])
def test_one_locked_output_does_not_hide_other_generated_files(sample, monkeypatch, lock):
exports = backend()
project, install, root = sample
def runner(command, cwd, environment, log_dir, **kwargs):
output = Path(next(p.split("=", 1)[1] for p in command if p.startswith("-file="))).parent
(output / "a.txt").write_text("exported A")
(output / "b.txt").write_text("exported B")
return dict(exit_code=8, cleanup={"errors": []})
real_copy = exports.shutil.copyfile
def copy(source, target, *args, **kwargs):
if Path(source).name == "a.txt":
raise PermissionError("one output is locked")
return real_copy(source, target, *args, **kwargs)
monkeypatch.setattr(exports, "_run", runner)
if lock == "copy":
monkeypatch.setattr(exports.shutil, "copyfile", copy)
else:
info = exports._file_info
def locked_info(path):
if Path(path).name == "a.txt":
raise PermissionError("one output hash is unavailable")
return info(path)
monkeypatch.setattr(exports, "_file_info", locked_info)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
assert result["items"][0]["status"] == "exported"
assert {Path(path).read_text() for path in result["items"][0]["outputs"]} == {"exported A", "exported B"}
assert result["workspace_preserved"]
exports.shutil.rmtree(Path(result["workspace_preserved"]))
def test_source_changed_after_capture_keeps_outputs_as_warning(sample, monkeypatch):
exports = backend()
project, install, root = sample
real_runner = ascii_runner([])
def runner(*args, **kwargs):
run = real_runner(*args, **kwargs)
Path(project["database"], "icdb.dat").write_bytes(b"new source saved during export")
return run
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
assert result["source_unchanged"] is False and result["status"] == "exported"
assert all(item["status"] == "exported" and item["outputs"] for item in result["items"][:2])
def test_locked_receipt_does_not_stop_following_tools(sample, monkeypatch):
exports = backend()
project, install, root = sample
calls = []
monkeypatch.setattr(exports, "_run", ascii_runner(calls))
write = exports.write_json
def locked(path, value):
if Path(path).name == "export.json":
raise PermissionError("receipt is locked")
return write(path, value)
monkeypatch.setattr(exports, "write_json", locked)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
assert len(calls) == 2 and result["items"][1]["outputs"]
assert result["warnings"]
def test_unreadable_library_fingerprint_does_not_block_schematic_tools(sample, monkeypatch):
exports = backend()
project, install, root = sample
calls = []
hashes = exports._hashes
monkeypatch.setattr(exports, "_library_sources", lambda _: {"parts": root / "parts.pdb"})
def unreadable(files):
if "parts" in files:
raise PermissionError("central library unreadable")
return hashes(files)
monkeypatch.setattr(exports, "_hashes", unreadable)
monkeypatch.setattr(exports, "_run", ascii_runner(calls))
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
assert len(calls) == 2 and result["items"][0]["outputs"]
def test_successful_export_also_retains_companion_files(sample, monkeypatch):
exports = backend()
project, install, root = sample
run = ascii_runner([])
def runner(command, cwd, environment, log_dir, **kwargs):
result = run(command, cwd, environment, log_dir, **kwargs)
(Path(cwd) / "output/companion.txt").write_text("additional exported data")
return result
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
assert {Path(path).name for path in result["items"][0]["outputs"]} == {"connections.txt", "companion.txt"}
def test_workspace_cleanup_error_is_recorded_and_receipt_updated(sample, monkeypatch):
exports = backend()
project, install, root = sample
monkeypatch.setattr(exports, "_run", ascii_runner([]))
remove = exports.shutil.rmtree
def locked(path):
raise PermissionError("temporary workspace locked")
monkeypatch.setattr(exports.shutil, "rmtree", locked)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
receipt = json.loads((root / "attempt/export.json").read_text(encoding="utf-8"))
assert receipt["workspace_preserved"] == result["workspace_preserved"]
assert "locked" in result["workspace_cleanup_error"]
remove(Path(result["workspace_preserved"]))
def test_run_receipt_write_error_keeps_cleanup_uncertainty(tmp_path, monkeypatch):
exports = backend()
class Desktop:
cleanup_record = {"errors": ["unconfirmed owned process"], "processes": []}
def __init__(self, *args, **kwargs): pass
def __enter__(self): return self
def __exit__(self, *args): pass
def start_process(self, *args, **kwargs): return self
def poll(self): return 0
pid = 123
monkeypatch.setattr(exports, "PrivateDesktop", Desktop)
monkeypatch.setattr(exports, "_UNCERTAIN_DESKTOPS", [])
def locked(*args): raise PermissionError("run receipt locked")
monkeypatch.setattr(exports, "write_json", locked)
result = exports._run(["test"], tmp_path, {}, tmp_path / "logs", stop_event=None, timeout=1)
assert result["cleanup_uncertain"] and result["warnings"]
@pytest.mark.skipif(__import__("os").name != "nt", reason="Windows private desktop")
@pytest.mark.parametrize("cancel", [False, True])
def test_owned_process_timeout_and_active_cancel_leave_no_job_processes(tmp_path, cancel):