Run isolated export tasks concurrently and overlap cleanup

This commit is contained in:
阿白叔 committed 2026-11-09 14:35:48 +08:00
1 parent 5b974c9b45
commit 8029d146cb
8 files changed
+341 -41

No files matched your search

+6
View File
@@ -18,9 +18,15 @@
导出.cmd --input "F:\工程目录" --output "F:\导出结果" --yes 导出.cmd --input "F:\工程目录" --output "F:\导出结果" --yes
导出.cmd --mentor-home "D:\Mentor\7.9.5EE\SDD_HOME" --input "F:\工程目录" --yes 导出.cmd --mentor-home "D:\Mentor\7.9.5EE\SDD_HOME" --input "F:\工程目录" --yes
导出.cmd --retry "F:\导出结果\批次_..." --yes 导出.cmd --retry "F:\导出结果\批次_..." --yes
导出.cmd --input "F:\工程目录" --output "F:\导出结果" --jobs 3 --yes
导出.cmd --input "F:\工程目录" --jobs 1 --yes
不指定参数进入问答。--yes 适合使用固定参数重复运行;发现多套 Mentor 时需要 --mentor-home 指定。--timeout 可以调整单工具等待上限,默认 60 秒。 不指定参数进入问答。--yes 适合使用固定参数重复运行;发现多套 Mentor 时需要 --mentor-home 指定。--timeout 可以调整单工具等待上限,默认 60 秒。
默认同时运行 3 个独立导出任务,可用 --jobs 1 切换串行,--jobs 2 到 8 调整并发上限。各工具使用同一已保存输入的独立副本;工程按顺序处理,库件读取在 ASCII 平铺属性就绪后与其他导出重叠,同库缓存继续共享。快照名称探测也参与同一并发上限,不额外占用任务槽。
工具退出通过进程通知立即唤醒,不额外睡眠;收尾与其他导出重叠执行。后台进程自然退出的宽限期最多 2 秒,已退出则立即结束等待。若仍有自有后台进程需要终止,会保留文件并记录提示,不标为已验证成功,也不阻塞其他任务。运行记录分别保存工具执行与收尾耗时,工程记录保存实际并发数与总耗时。
按 Ctrl+C 请求停止,等待工具收尾;不要直接关闭窗口。下次使用 --retry 指定批次,可恢复未完成项。成功产物只有在输入、工具和输出摘要仍匹配时才复用;源工程变化会完整重新导出该工程。 按 Ctrl+C 请求停止,等待工具收尾;不要直接关闭窗口。下次使用 --retry 指定批次,可恢复未完成项。成功产物只有在输入、工具和输出摘要仍匹配时才复用;源工程变化会完整重新导出该工程。
默认尽力导出并持续完成批次。单项异常、缺工具/缺库、超时、格式检查提示或进程收尾提示,都记录下来并继续后续工具和工程;报告文件被占用也不阻塞导出。只有用户主动按 Ctrl+C 才停止批次。 默认尽力导出并持续完成批次。单项异常、缺工具/缺库、超时、格式检查提示或进程收尾提示,都记录下来并继续后续工具和工程;报告文件被占用也不阻塞导出。只有用户主动按 Ctrl+C 才停止批次。
+15
View File
@@ -0,0 +1,15 @@
# MentorExport 0.1.2
2026-11-09。加速原理图批量导出,保留 0.1.1 的单项问题不阻塞批次策略。
- 默认 3 个导出任务并发,支持 `--jobs 1` 串行和 `--jobs 2` 到 `--jobs 8`。问答模式直接采用默认 3,无需额外配置。
- 每工具独立数据库副本、输出、日志、私有桌面和进程 Job;仅协调线程更新工程记录,输出分类和逐项顺序稳定。
- ASCII 平铺属性完成后即可开始库件读取,与其他格式导出重叠;同库缓存继续在顺序处理的工程间共享。
- 快照探测与正文导出重叠,同样计入并发上限。
- 工具退出改为进程通知唤醒,消除固定轮询睡眠;收尾等待与其他导出重叠。保留后台进程最多 2 秒的自然退出宽限期,避免过早终止导致下次重试需要重导。
- 若后台进程仍需强制结束,文件照常保留、显示提示,不作为已验证成功结果复用;库件记录也不进入已验证缓存。进程退出确认和工作副本保留机制继续生效。
- 移除一次重复的库件记录写入,新增工具执行/进程收尾耗时及工程并发信息。
下载完整 ZIP 解压,双击 `导出.cmd`。本机需已有 Mentor 与可用许可,Python 和依赖已预置。实际兼容环境仍为 EE7.9.5;其他安装的原生库读取沿用已验证指纹准入。
实测同一对真实工程:v0.1.1 串行 156.015 秒,新版默认并发 74.594 秒,耗时减少 52.2%。22 项均取得且通过检查,EDF/KYN 连接和器件属性、关联库记录与串行结果相同;重试校验后复用耗时 13.609 秒。64 项自动测试通过。完整范围见 [并发验收](../validation-2026-11-09-parallel.md),实际速度取决于工程和机器环境。
+45
View File
@@ -0,0 +1,45 @@
# 0.1.2 并发导出验收
2026-11-09。本机 Windows x64、Mentor EE7.9.5。默认每工程 3 个导出任务并发,工程顺序处理,同库缓存继续共享。`--jobs 1` 保留串行入口,最大并发 8。
## 自动测试及审查
`python -X utf8 -m pytest -q`:**64 passed in 34.73s**。
新增回归先失败后通过,覆盖独立副本上的真实调度重叠、并发上限、稳定输出顺序、快照与正文导出重叠、库件在 ASCII 就绪后提前运行、并发取消后收尾、CLI 参数传递、无效并发数、进程退出通知唤醒,以及强制收尾产物不能作为已验证结果复用。
独立审查确认协调线程独占工程记录写入,worker 使用独立状态与副本,同库分区缓存仍为单写入者。审查发现的强制结束后误入成功缓存已修正,普通导出和原生库读取都保留产物但标为提示;末轮针对进程等待与库缓存复查无待处理发现。
## 同输入实测
从之前已验收的两个真实工程复制固定输入,分别使用不可修改的 v0.1.1 发行包和最终 v0.1.2 发行包。每次是独立新进程和新批次,未用 `--retry` 代替首次导出。
| 项目 | v0.1.1 串行 | v0.1.2 默认 3 并发 |
| --- | ---: | ---: |
| 两工程完整导出 | 156.015 秒 | 74.594 秒 |
| 取得的导出项 | 22 | 22 |
| 带提示项 | 0 | 0 |
| 批次退出码 | 0 | 0 |
本机样例耗时降低 **52.2%**,约为原速度的 **2.09 倍**。不同工程、磁盘负载及许可环境下不保证同一比例。
两工程逐一解析 EDIF 网表与 KYN:器件、字符串属性、物理引脚、网络及连接集合与串行结果完全一致。关联库件原生记录逐项完全一致。14 个工程输入文件摘要未变,工程内源数据库和源库检查均为 source_unchanged=true。
最终包复制到中文及空格目录,从 `C:\Windows\Temp` 通过真实 CMD 无参数问答启动,PATH 仅保留 System32,PYTHONHOME/PYTHONPATH 为无效目录。自动发现环境并使用预置 Python 成功。
随后使用 `--retry --jobs 1 --yes` 验证串行入口及重复使用:13.609 秒,退出 0,22 项成功产物校验后复用。664 个发行文件与 manifest、ZIP 内容均核对一致。
## 等待策略
固定 50 ms 轮询睡眠改为进程句柄等待,工具退出即唤醒;仍以最多 50 ms 检查用户取消和超时。正常后台进程保留最多 2 秒自然退出宽限期,等待与其他任务重叠。
曾以 0.25 秒宽限期测试,出现 14 项强制收尾提示,串行重试需重新导出并耗时 68.188 秒。最终保留 2 秒正常宽限期,解决重复运行退化;不靠过早终止后台进程制造加速。
## 证据
- 串行基线及初轮对照:`build/parallel-benchmark-20261109-141823/benchmark.json`。
- 最终实际问答、串行重试及逐项比较:`build/parallel-release-acceptance-20261109-143025/acceptance.json`。
- 原始逐工具记录新增 started_at、tool_seconds、cleanup_seconds;工程记录新增 execution.jobs 与 execution.seconds。
- 最终 ZIP SHA256:`28616ced54ea893058d6dcde00025a9a72122814ae4f0021ee790ff9ed7bf22c`。
本次未增加跨工程并发或原生库分区并发;主要加速来自独立导出任务并行、库件提前开始、快照探测重叠、进程退出通知及减少重复记录写入。
+1 -1
View File
@@ -1,2 +1,2 @@
"""Portable Mentor saved-schematic exports.""" """Portable Mentor saved-schematic exports."""
__version__ = "0.1.1" __version__ = "0.1.2"
+12 -6
View File
@@ -95,7 +95,9 @@ def previous_results(record):
def run_batch(projects, problems, installation, output_root, scan_root, *, previous=None, def run_batch(projects, problems, installation, output_root, scan_root, *, previous=None,
stop_event=None, timeout=60, exporter=None, progress=print): stop_event=None, timeout=60, exporter=None, progress=print, jobs=3):
if not 1 <= jobs <= 8:
raise ValueError("并发数必须为 1 到 8")
if exporter is None: if exporter is None:
from .exports import export_project from .exports import export_project
exporter = export_project exporter = export_project
@@ -113,7 +115,7 @@ def run_batch(projects, problems, installation, output_root, scan_root, *, previ
"scan_root": str(Path(scan_root).resolve()), "output_root": str(output_root), "scan_root": str(Path(scan_root).resolve()), "output_root": str(output_root),
"created": datetime.now().astimezone().isoformat(), "installation": installation, "created": datetime.now().astimezone().isoformat(), "installation": installation,
"owner": process_identity(os.getpid()), "owner": process_identity(os.getpid()),
"problems": problems, "projects": [], "warnings": []} "problems": problems, "projects": [], "warnings": [], "execution": {"jobs": jobs}}
def warn(message): def warn(message):
if message not in record["warnings"]: if message not in record["warnings"]:
record["warnings"].append(message) record["warnings"].append(message)
@@ -156,7 +158,7 @@ def run_batch(projects, problems, installation, output_root, scan_root, *, previ
try: try:
result = exporter(project, installation, Path(row["folder"]), shared, result = exporter(project, installation, Path(row["folder"]), shared,
previous=prior.get(project["path"]), stop_event=stop_event, previous=prior.get(project["path"]), stop_event=stop_event,
timeout=timeout, progress=progress) timeout=timeout, progress=progress, jobs=jobs)
row["result"] = result row["result"] = result
row["status"] = result["status"] row["status"] = result["status"]
if result.get("cleanup_uncertain"): if result.get("cleanup_uncertain"):
@@ -241,6 +243,7 @@ def main(argv=None):
parser.add_argument("--mentor-home", help="指定 Mentor SDD_HOME") parser.add_argument("--mentor-home", help="指定 Mentor SDD_HOME")
parser.add_argument("--retry", help="重试已有批次,指定批次目录或 batch.json") parser.add_argument("--retry", help="重试已有批次,指定批次目录或 batch.json")
parser.add_argument("--timeout", type=float, default=60, help="单工具超时秒数,默认 60") parser.add_argument("--timeout", type=float, default=60, help="单工具超时秒数,默认 60")
parser.add_argument("--jobs", "-j", type=int, default=3, help="同时运行的导出任务数,默认 3;1 为串行,最多 8")
parser.add_argument("--yes", "-y", action="store_true", help="采用已给定参数运行,不显示交互菜单") parser.add_argument("--yes", "-y", action="store_true", help="采用已给定参数运行,不显示交互菜单")
parser.add_argument("--check", action="store_true", help="只检查预置环境与 Mentor") parser.add_argument("--check", action="store_true", help="只检查预置环境与 Mentor")
parser.add_argument("--json", action="store_true", help="环境检查以 JSON 输出") parser.add_argument("--json", action="store_true", help="环境检查以 JSON 输出")
@@ -249,6 +252,8 @@ def main(argv=None):
try: try:
if args.timeout <= 0: if args.timeout <= 0:
raise ValueError("超时必须大于 0") raise ValueError("超时必须大于 0")
if not 1 <= args.jobs <= 8:
raise ValueError("并发数必须为 1 到 8")
errors = runtime_check() errors = runtime_check()
if errors: if errors:
raise ValueError("预置运行环境不完整,请重新解压发行包:\n" + "\n".join(errors)) raise ValueError("预置运行环境不完整,请重新解压发行包:\n" + "\n".join(errors))
@@ -269,6 +274,7 @@ def main(argv=None):
return 0 if installation["preflight"]["status"] == "success" else 2 return 0 if installation["preflight"]["status"] == "success" else 2
print(f"\nMentorExport {__version__} · 原理图详细导出") print(f"\nMentorExport {__version__} · 原理图详细导出")
print(f"Mentor:{installation['version']} {installation['root']}") print(f"Mentor:{installation['version']} {installation['root']}")
print(f"并发任务:{args.jobs}(--jobs 1 可切换串行)")
print("读取已保存的原理图。请先保存目标工程;工具仅在副本上运行。") print("读取已保存的原理图。请先保存目标工程;工具仅在副本上运行。")
if not args.retry and settings.get("output"): if not args.retry and settings.get("output"):
candidates = sorted(Path(settings["output"]).glob("批次_*/batch.json"), reverse=True)[:30] candidates = sorted(Path(settings["output"]).glob("批次_*/batch.json"), reverse=True)[:30]
@@ -277,7 +283,7 @@ def main(argv=None):
print("发现未完成批次:" + str(unfinished[0].parent)) print("发现未完成批次:" + str(unfinished[0].parent))
if not args.yes and _ask("输入 1 重试该批次;回车开始新批次") == "1": if not args.yes and _ask("输入 1 重试该批次;回车开始新批次") == "1":
return main(["--retry", str(unfinished[0]), "--mentor-home", installation["root"], return main(["--retry", str(unfinished[0]), "--mentor-home", installation["root"],
"--timeout", str(args.timeout)]) "--timeout", str(args.timeout), "--jobs", str(args.jobs)])
previous = None previous = None
if args.retry: if args.retry:
previous_path = Path(args.retry) previous_path = Path(args.retry)
@@ -329,7 +335,7 @@ def main(argv=None):
signal.signal(signal.SIGINT, cancel) signal.signal(signal.SIGINT, cancel)
try: try:
batch, code = run_batch(projects, problems, installation, output_root, scan_root, batch, code = run_batch(projects, problems, installation, output_root, scan_root,
previous=previous, stop_event=stop, timeout=args.timeout) previous=previous, stop_event=stop, timeout=args.timeout, jobs=args.jobs)
finally: finally:
signal.signal(signal.SIGINT, old_handler) signal.signal(signal.SIGINT, old_handler)
record = load_json(batch / "batch.json", {"projects": []}) record = load_json(batch / "batch.json", {"projects": []})
@@ -347,7 +353,7 @@ def main(argv=None):
if action in ("1", "2"): if action in ("1", "2"):
os.startfile(str(batch if action == "1" else batch / "导出汇总.html")) os.startfile(str(batch if action == "1" else batch / "导出汇总.html"))
elif action == "3": elif action == "3":
return main(["--retry", str(batch), "--mentor-home", installation["root"], "--timeout", str(args.timeout)]) return main(["--retry", str(batch), "--mentor-home", installation["root"], "--timeout", str(args.timeout), "--jobs", str(args.jobs)])
return code return code
except KeyboardInterrupt: except KeyboardInterrupt:
print("\n已取消。") print("\n已取消。")
+86 -33
View File
@@ -1,6 +1,7 @@
"""Saved-project exports through short lived, owned, isolated tool processes.""" """Saved-project exports through short lived, owned, isolated tool processes."""
from copy import deepcopy from copy import deepcopy
from contextlib import contextmanager from contextlib import contextmanager
from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED
import ctypes import ctypes
from datetime import datetime, timezone from datetime import datetime, timezone
import hashlib import hashlib
@@ -9,6 +10,7 @@ import os
from pathlib import Path from pathlib import Path
import re import re
import shutil import shutil
import subprocess
import tempfile import tempfile
import time import time
import uuid import uuid
@@ -94,7 +96,7 @@ def _run(command, cwd, environment, log_dir, *, stop_event, timeout):
log_dir.mkdir(parents=True, exist_ok=True) log_dir.mkdir(parents=True, exist_ok=True)
desktop = PrivateDesktop("mexp-" + uuid.uuid4().hex, enabled=True) desktop = PrivateDesktop("mexp-" + uuid.uuid4().hex, enabled=True)
result = {"command": list(map(str, command)), "cwd": str(cwd), "exit_code": None, result = {"command": list(map(str, command)), "cwd": str(cwd), "exit_code": None,
"timed_out": False, "cancelled": False} "timed_out": False, "cancelled": False, "started_at": datetime.now(timezone.utc).isoformat()}
started = time.monotonic() started = time.monotonic()
try: try:
with desktop: with desktop:
@@ -113,12 +115,18 @@ def _run(command, cwd, environment, log_dir, *, stop_event, timeout):
except Exception as error: except Exception as error:
result["dialog_inspection_error"] = str(error) result["dialog_inspection_error"] = str(error)
break break
time.sleep(0.05) try:
process.wait(timeout=min(0.05, max(0, timeout - (time.monotonic() - started))))
except subprocess.TimeoutExpired:
pass
result["exit_code"] = process.poll() result["exit_code"] = process.poll()
result["tool_seconds"] = round(time.monotonic() - started, 3)
except Exception as error: except Exception as error:
result["error"] = str(error) result["error"] = str(error)
finally: finally:
result["seconds"] = round(time.monotonic() - started, 3) result["seconds"] = round(time.monotonic() - started, 3)
if "tool_seconds" in result:
result["cleanup_seconds"] = round(result["seconds"] - result["tool_seconds"], 3)
result["cleanup"] = desktop.cleanup_record result["cleanup"] = desktop.cleanup_record
cleanup = result["cleanup"] or {} cleanup = result["cleanup"] or {}
if cleanup.get("errors") or any(p.get("job_active_processes", 0) for p in cleanup.get("processes", [])): if cleanup.get("errors") or any(p.get("job_active_processes", 0) for p in cleanup.get("processes", [])):
@@ -303,12 +311,16 @@ def _status(items):
return "exported" if any(i.get("outputs") for i in items) else "skipped" return "exported" if any(i.get("outputs") for i in items) else "skipped"
def export_project(project, installation, project_dir, shared_dir, *, previous=None, stop_event=None, timeout=45, progress=print): def export_project(project, installation, project_dir, shared_dir, *, previous=None, stop_event=None, timeout=45, progress=print, jobs=3):
if not 1 <= jobs <= 8:
raise ValueError("并发数必须为 1 到 8")
project_started = time.monotonic()
project_dir = Path(project_dir).resolve() project_dir = Path(project_dir).resolve()
project_dir.mkdir(parents=True, exist_ok=True) project_dir.mkdir(parents=True, exist_ok=True)
shared_dir = Path(shared_dir).resolve() shared_dir = Path(shared_dir).resolve()
result = {"status": "running", "project": deepcopy(project), "captured_at": datetime.now(timezone.utc).isoformat(), result = {"status": "running", "project": deepcopy(project), "captured_at": datetime.now(timezone.utc).isoformat(),
"source_fingerprint": None, "source_hashes": {}, "items": [], "source_unchanged": None, "source_fingerprint": None, "source_hashes": {}, "items": [], "source_unchanged": None,
"execution": {"jobs": jobs},
"coverage_limits": ["Exports preserve each tool/mode separately; no ERC or manufacturing certification.", "coverage_limits": ["Exports preserve each tool/mode separately; no ERC or manufacturing certification.",
"CMS requests read mode but may report ReadOnly=False; only isolated copies are opened.", "CMS requests read mode but may report ReadOnly=False; only isolated copies are opened.",
"Cell coordinate units and rotation remain native integers; multi-slot/swappable mappings unverified."]} "Cell coordinate units and rotation remain native integers; multi-slot/swappable mappings unverified."]}
@@ -336,25 +348,22 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
return result return result
environment = _environment(installation) environment = _environment(installation)
old_items = {i["key"]: i for i in (previous or {}).get("items", [])} old_items = {i["key"]: i for i in (previous or {}).get("items", [])}
discovery = {"status": "failed", "names": []} def discover():
result["snapshot_discovery"] = discovery discovery = {"status": "failed", "names": []}
try: try:
snapshot_work = temp / "snapshot-discovery" snapshot_work = temp / "snapshot-discovery"
shutil.copytree(captured, snapshot_work) shutil.copytree(captured, snapshot_work)
discovery.update(_snapshot_metadata(project, installation, snapshot_work, discovery.update(_snapshot_metadata(project, installation, snapshot_work,
project_dir / "90_运行日志" / "snapshots", stop_event=stop_event, project_dir / "90_运行日志" / "snapshots", stop_event=stop_event,
timeout=min(timeout, 20))) timeout=min(timeout, 20)))
if discovery.get("cleanup_uncertain"): except Exception as error:
result.update(cleanup_uncertain=True, workspace_preserved=str(temp)) discovery.update(status="failed", reason=str(error))
except Exception as error: return discovery
discovery.update(status="failed", reason=str(error))
if discovery["status"] != "success": def execute(spec, task_state):
result["coverage_limits"].append("Other snapshot names were not obtained: " + discovery.get("reason", "")) key, tool, folder = spec
_checkpoint(state, result)
for key, tool, folder in ITEMS:
output = None output = None
item = {"key": key, "status": "failed", "reason": "", "outputs": [], "provenance": {}} item = {"key": key, "status": "failed", "reason": "", "outputs": [], "provenance": {}}
result["items"].append(item)
try: try:
progress(project["name"] + ": " + key) progress(project["name"] + ": " + key)
identity = _tool_identity(key, tool, installation) identity = _tool_identity(key, tool, installation)
@@ -363,19 +372,19 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
"input_fingerprint": fingerprint, "reused": False} "input_fingerprint": fingerprint, "reused": False}
if stop_event is not None and stop_event.is_set(): if stop_event is not None and stop_event.is_set():
item.update(status="cancelled", reason="Cancelled before launch") item.update(status="cancelled", reason="Cancelled before launch")
continue return item, task_state
if _reusable(old_items.get(key), fingerprint, identity): if _reusable(old_items.get(key), fingerprint, identity):
item.update(deepcopy(old_items[key])) item.update(deepcopy(old_items[key]))
item["provenance"]["reused"] = True item["provenance"]["reused"] = True
item["reason"] = "Verified previous successful output reused" item["reason"] = "Verified previous successful output reused"
continue return item, task_state
if key == "library": if key == "library":
_export_library(item, project, installation, result, project_dir / folder, shared_dir, _export_library(item, project, installation, task_state, project_dir / folder, shared_dir,
temp, environment, stop_event, timeout) temp, environment, stop_event, timeout)
continue return item, task_state
if not identity.get("sha256") or (tool == "cms" and not identity.get("matches")): if not identity.get("sha256") or (tool == "cms" and not identity.get("matches")):
item.update(status="unavailable", reason="Tool missing or COM registration does not match selected installation") item.update(status="unavailable", reason="Tool missing or COM registration does not match selected installation")
continue return item, task_state
work = temp / key work = temp / key
shutil.copytree(captured, work) shutil.copytree(captured, work)
output = work / "output" output = work / "output"
@@ -386,7 +395,7 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
item["provenance"].update(run) item["provenance"].update(run)
item["provenance"]["log_dir"] = str(log_dir) item["provenance"]["log_dir"] = str(log_dir)
if run.get("cleanup_uncertain"): if run.get("cleanup_uncertain"):
result.update(cleanup_uncertain=True, workspace_preserved=str(temp)) task_state.update(cleanup_uncertain=True, workspace_preserved=str(temp))
item["reason"] = "Owned process cleanup is uncertain; workspace preserved; remaining exports continue" item["reason"] = "Owned process cleanup is uncertain; workspace preserved; remaining exports continue"
elif run.get("cancelled"): elif run.get("cancelled"):
item.update(status="cancelled", reason="Cancelled while tool was running") item.update(status="cancelled", reason="Cancelled while tool was running")
@@ -407,6 +416,8 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
item.update(status="success", outputs=[str(p) for p in published]) item.update(status="success", outputs=[str(p) for p in published])
item["validation"] = {"status": "passed"} item["validation"] = {"status": "passed"}
item["provenance"]["output_files"] = [_file_info(p) for p in published] item["provenance"]["output_files"] = [_file_info(p) for p in published]
if any(process.get("forced_termination") for process in run.get("cleanup", {}).get("processes", [])):
item.update(status="exported", reason="Files retained; owned background processes exceeded shutdown grace and were terminated; background completion unverified")
except Exception as error: except Exception as error:
item.update(status="failed", reason=str(error)) item.update(status="failed", reason=str(error))
finally: finally:
@@ -425,7 +436,7 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
saved.append(_file_info(target)) saved.append(_file_info(target))
except OSError as error: except OSError as error:
publication_errors.append(str(error)) publication_errors.append(str(error))
result.setdefault("workspace_preserved", str(temp)) task_state.setdefault("workspace_preserved", str(temp))
try: try:
if source.is_file() and source.stat().st_size: if source.is_file() and source.stat().st_size:
saved.append(_file_info(source)) saved.append(_file_info(source))
@@ -443,8 +454,45 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
item["status"] = "exported" item["status"] = "exported"
if item["status"] == "exported": if item["status"] == "exported":
item["validation"] = {"status": "warning", "reason": item["reason"]} item["validation"] = {"status": "warning", "reason": item["reason"]}
result["status"] = "running" return item, task_state
_checkpoint(state, result)
result["items"] = [{"key": key, "status": "pending", "outputs": [], "reason": "", "provenance": {}}
for key, tool, folder in ITEMS]
specs = {spec[0]: spec for spec in ITEMS}
positions = {item["key"]: index for index, item in enumerate(result["items"])}
ready = ["ascii_flat", "snapshots"] + [key for key, tool, folder in ITEMS if key not in ("ascii_flat", "library")]
# Only the coordinator publishes receipts; workers own distinct clones, items and logs.
with ThreadPoolExecutor(max_workers=jobs, thread_name_prefix="mentor-export") as pool:
active = {}
while ready or active:
while ready and len(active) < jobs:
key = ready.pop(0)
if key == "snapshots":
future = pool.submit(discover)
else:
task_state = {"items": list(result["items"]),
"library_source_hashes": result["library_source_hashes"]}
future = pool.submit(execute, specs[key], task_state)
active[future] = key
completed, _ = wait(active, return_when=FIRST_COMPLETED)
for future in completed:
key = active.pop(future)
if key == "snapshots":
metadata = future.result()
result["snapshot_discovery"] = metadata
task_state = metadata
if metadata["status"] != "success":
result["coverage_limits"].append("Other snapshot names were not obtained: " + metadata.get("reason", ""))
else:
item, task_state = future.result()
result["items"][positions[key]] = item
if key == "ascii_flat":
ready.insert(0, "library")
if task_state.get("cleanup_uncertain"):
result["cleanup_uncertain"] = True
if task_state.get("workspace_preserved"):
result["workspace_preserved"] = str(temp)
_checkpoint(state, result)
try: try:
after = _hashes(_inputs(project)) after = _hashes(_inputs(project))
result["source_hashes_after"] = after result["source_hashes_after"] = after
@@ -459,6 +507,7 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
if item["status"] == "success": if item["status"] == "success":
item.update(status="exported", reason="Source changed during export; captured output retained; re-export to verify current source", item.update(status="exported", reason="Source changed during export; captured output retained; re-export to verify current source",
validation={"status": "warning", "reason": "Source changed during export"}) validation={"status": "warning", "reason": "Source changed during export"})
result["execution"]["seconds"] = round(time.monotonic() - project_started, 3)
result["status"] = _status(result["items"]) result["status"] = _status(result["items"])
_checkpoint(state, result) _checkpoint(state, result)
return result return result
@@ -619,6 +668,9 @@ def _export_library(item, project, installation, result, destination, shared_dir
data["errors"].append({"partition": relative, "reason": run.get("error") or "Native read failed/timeout/cleanup error"}) data["errors"].append({"partition": relative, "reason": run.get("error") or "Native read failed/timeout/cleanup error"})
persist() persist()
continue continue
trusted_run = not any(process.get("forced_termination") for process in run.get("cleanup", {}).get("processes", []))
if not trusted_run:
data["errors"].append({"partition": relative, "reason": "Owned background process terminated during shutdown; records retained without verified reuse"})
try: try:
native = json.loads((log / "stdout.txt").read_text(encoding="utf-8")) native = json.loads((log / "stdout.txt").read_text(encoding="utf-8"))
_validate_native(native, mode) _validate_native(native, mode)
@@ -634,15 +686,16 @@ def _export_library(item, project, installation, result, destination, shared_dir
if known: if known:
old_names = {row["name"] for row in selected} old_names = {row["name"] for row in selected}
selected.extend(row for row in known["selected"] if row["name"] not in old_names) selected.extend(row for row in known["selected"] if row["name"] not in old_names)
partitions[relative] = {"names": {row["name"] for row in native["records"]}, if trusted_run:
"selected": selected, "run": deepcopy(run)} partitions[relative] = {"names": {row["name"] for row in native["records"]},
"selected": selected, "run": deepcopy(run)}
# Remove unrelated full-library stdout only after selected records are durable. # Remove unrelated full-library stdout only after selected records are durable.
persist() persist()
(log / "stdout.txt").unlink() (log / "stdout.txt").unlink()
run["stdout_retained"] = False run["stdout_retained"] = False
_checkpoint(log / "run.json", run) _checkpoint(log / "run.json", run)
partitions[relative]["run"] = deepcopy(run) if trusted_run:
persist() partitions[relative]["run"] = deepcopy(run)
except (OSError, ValueError) as error: except (OSError, ValueError) as error:
data["errors"].append({"partition": relative, "reason": str(error)}) data["errors"].append({"partition": relative, "reason": str(error)})
persist() persist()
+18
View File
@@ -131,3 +131,21 @@ def test_preflight_exception_is_a_warning_not_a_batch_blocker(tmp_path, monkeypa
monkeypatch.setattr(cli, "_save_settings", lambda value: None) monkeypatch.setattr(cli, "_save_settings", lambda value: None)
assert cli.main(["--input", str(tmp_path), "--output", str(tmp_path / "out"), "--yes"]) == 0 assert cli.main(["--input", str(tmp_path), "--output", str(tmp_path / "out"), "--yes"]) == 0
assert list((tmp_path / "out").glob("批次_*/batch.json")) assert list((tmp_path / "out").glob("批次_*/batch.json"))
def test_batch_passes_worker_limit_and_records_execution_mode(tmp_path):
from mentorexport.cli import run_batch
limits = []
def exporter(project, installation, folder, shared, **options):
limits.append(options["jobs"])
return {"status": "success", "items": []}
batch, code = run_batch([{"name": "工程", "path": str(tmp_path / "a.prj")}], [], {},
tmp_path / "out", tmp_path, exporter=exporter, jobs=2)
record = json.loads((batch / "batch.json").read_text(encoding="utf-8"))
assert code == 0 and limits == [2] and record["execution"]["jobs"] == 2
def test_invalid_worker_limit_does_not_start_export(tmp_path):
from mentorexport.cli import main
for jobs in ("0", "9"):
assert main(["--input", str(tmp_path), "--jobs", jobs, "--yes"]) == 2
+158 -1
View File
@@ -25,6 +25,163 @@ def backend():
return exports return exports
def test_parallel_exports_overlap_on_distinct_clones_and_keep_receipt_order(sample, monkeypatch):
from threading import Barrier, Lock
exports = backend()
project, install, root = sample
barrier, lock = Barrier(2), Lock()
calls, workspaces = [], []
run_ascii = ascii_runner(calls)
active = maximum = 0
def runner(command, cwd, environment, log_dir, **kwargs):
nonlocal active, maximum
with lock:
active += 1
maximum = max(maximum, active)
workspaces.append(Path(cwd))
try:
barrier.wait(timeout=5)
return run_ascii(command, cwd, environment, log_dir, **kwargs)
finally:
with lock:
active -= 1
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "parallel", root / "shared", jobs=2, progress=lambda _: None)
assert maximum == 2 and len(set(workspaces)) == 2
assert [item["key"] for item in result["items"]] == [spec[0] for spec in exports.ITEMS]
assert [i["key"] for i in result["items"] if i["status"] == "success"] == ["ascii_flat", "ascii_hier"]
assert result["source_unchanged"] and result["execution"]["jobs"] == 2
def test_library_starts_after_flat_without_waiting_for_other_exports(sample, monkeypatch):
from threading import Event
exports = backend()
project, install, root = sample
library_started = Event()
runner = ascii_runner([])
def run(command, *args, **kwargs):
if "-h" in command:
assert library_started.wait(5), "library waited for unrelated exports"
return runner(command, *args, **kwargs)
def library(item, project, installation, result, *args):
flat = next(i for i in result["items"] if i["key"] == "ascii_flat")
assert flat["status"] == "success" and Path(flat["outputs"][0]).is_file()
library_started.set()
item.update(status="unavailable", reason="fixture without library")
monkeypatch.setattr(exports, "_run", run)
monkeypatch.setattr(exports, "_export_library", library)
result = exports.export_project(project, install, root / "parallel", root / "shared", jobs=2, progress=lambda _: None)
assert library_started.is_set() and result["items"][1]["status"] == "success"
def test_snapshot_listing_overlaps_with_exports(sample, monkeypatch):
from threading import Barrier
exports = backend()
project, install, root = sample
install["tools"]["constraints_xml"] = install["tools"]["ascii"]
barrier = Barrier(2)
runner = ascii_runner([])
def run(command, cwd, environment, log_dir, **kwargs):
if "-list" in command or "-f" in command:
barrier.wait(timeout=5)
if "-list" in command:
Path(log_dir).mkdir(parents=True)
(Path(log_dir) / "stdout.txt").write_text(" -> Saved\n")
return {"exit_code": 0}
return runner(command, cwd, environment, log_dir, **kwargs)
monkeypatch.setattr(exports, "_run", run)
result = exports.export_project(project, install, root / "parallel", root / "shared", jobs=2, progress=lambda _: None)
assert result["snapshot_discovery"]["status"] == "success"
assert result["items"][0]["status"] == "success"
def test_forced_descendant_shutdown_keeps_files_but_never_reuses_as_verified(sample, monkeypatch):
exports = backend()
project, install, root = sample
calls = []
ascii_run = ascii_runner(calls)
def runner(*args, **kwargs):
run = ascii_run(*args, **kwargs)
run["cleanup"] = {"errors": [], "processes": [{"forced_termination": True,
"forced_root_termination": False, "forced_descendant_pids": [123],
"termination_reason": "descendant_grace_expired", "job_active_processes": 0}]}
return run
monkeypatch.setattr(exports, "_run", runner)
first = exports.export_project(project, install, root / "first", root / "shared", jobs=2, progress=lambda _: None)
assert first["items"][0]["status"] == "exported" and first["items"][0]["outputs"]
assert first["items"][0]["validation"]["status"] == "warning"
exports.export_project(project, install, root / "retry", root / "shared", jobs=2, previous=first, progress=lambda _: None)
assert len(calls) == 4
def test_forced_library_shutdown_retains_records_without_trusted_cache(sample, monkeypatch):
exports = backend()
project, install, root = sample
library = root / "library/PartsDBLibs"
library.mkdir(parents=True)
(library / "a.pdb").write_bytes(b"parts")
project["library"] = str(library.parent)
monkeypatch.setattr(exports, "VALIDATED_DLLS", {})
monkeypatch.setattr(exports, "_LIBRARY_CACHE", {})
calls = []
def runner(command, cwd, environment, log_dir, **kwargs):
if "-f" in command or "-h" in command:
output = Path(next(arg.split("=", 1)[1] for arg in command if arg.startswith("-file=")))
output.write_text("iCDB Ascii Report\n" + ("Block Top\n" if "-h" in command else "Config default\n") + "\tProp Part Number USED\n")
return dict(exit_code=0, cleanup={"errors": []})
calls.append(command)
Path(log_dir).mkdir(parents=True, exist_ok=True)
(Path(log_dir) / "stdout.txt").write_text(json.dumps({"mode": "parts", "count": 1, "records": [{"name": "USED"}]}), encoding="utf-8")
return dict(exit_code=0, cleanup={"errors": [], "processes": [{"forced_termination": True}]})
monkeypatch.setattr(exports, "_run", runner)
first = exports.export_project(project, install, root / "first", root / "shared", jobs=2, progress=lambda _: None)
item = next(i for i in first["items"] if i["key"] == "library")
assert item["status"] == "exported" and item["validation"]["status"] == "warning"
data = next(json.loads(Path(path).read_text(encoding="utf-8")) for path in item["outputs"] if Path(path).name.startswith("used-"))
assert data["records"]["parts"][0]["name"] == "USED" and data["complete"] is False
exports.export_project(project, install, root / "retry", root / "shared", jobs=2, previous=first, progress=lambda _: None)
assert len(calls) == 2
def test_parallel_cancel_drains_active_tasks_without_launching_queued_vendor_work(sample, monkeypatch):
from threading import Barrier
exports = backend()
project, install, root = sample
stop, barrier = Event(), Barrier(2)
calls = []
def runner(command, cwd, *args, **kwargs):
calls.append(command)
barrier.wait(timeout=5)
stop.set()
return dict(exit_code=None, cancelled=True, cleanup={"errors": []})
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "cancel", root / "shared", jobs=2,
stop_event=stop, progress=lambda _: None)
assert len(calls) == 2 and result["status"] == "cancelled"
assert all(item["status"] == "cancelled" for item in result["items"])
def test_process_exit_notification_advances_without_polling_delay(tmp_path, monkeypatch):
exports = backend()
class Process:
pid = 123
finished = False
def poll(self): return 0 if self.finished else None
def wait(self, timeout=None):
self.finished = True
return 0
class Desktop:
cleanup_record = {"errors": [], "processes": []}
def __init__(self, *args, **kwargs): pass
def __enter__(self): return self
def __exit__(self, *args): pass
def start_process(self, *args, **kwargs): return Process()
def inspect_dialogs(self, pid): return []
monkeypatch.setattr(exports, "PrivateDesktop", Desktop)
result = exports._run(["fixture"], tmp_path, {}, tmp_path / "logs", stop_event=None, timeout=0.15)
assert result["exit_code"] == 0 and not result["timed_out"]
def ascii_runner(calls): def ascii_runner(calls):
def run(command, cwd, environment, log_dir, *, stop_event, timeout): def run(command, cwd, environment, log_dir, *, stop_event, timeout):
calls.append(command) calls.append(command)
@@ -216,7 +373,7 @@ def test_uncertain_snapshot_cleanup_continues_exports_and_preserves_copy(sample,
return dict(exit_code=0, cleanup_uncertain=True, cleanup={"errors": ["exit unconfirmed"]}) return dict(exit_code=0, cleanup_uncertain=True, cleanup={"errors": ["exit unconfirmed"]})
monkeypatch.setattr(exports, "_run", runner) monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None) result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
assert len(calls) == 3 and "-list" in calls[0] assert len(calls) == 3 and any("-list" in call for call in calls)
assert result["cleanup_uncertain"] is True assert result["cleanup_uncertain"] is True
assert result["status"] == "skipped" assert result["status"] == "skipped"
assert not any(i["status"] == "cancelled" for i in result["items"]) assert not any(i["status"] == "cancelled" for i in result["items"])