Run isolated export tasks concurrently and overlap cleanup

This commit is contained in:
阿白叔 committed 2026-11-09 14:35:48 +08:00
1 parent 5b974c9b45
commit 8029d146cb
8 files changed
+341 -41

No files matched your search

+6
View File
@@ -18,9 +18,15 @@
导出.cmd --input "F:\工程目录" --output "F:\导出结果" --yes
导出.cmd --mentor-home "D:\Mentor\7.9.5EE\SDD_HOME" --input "F:\工程目录" --yes
导出.cmd --retry "F:\导出结果\批次_..." --yes
导出.cmd --input "F:\工程目录" --output "F:\导出结果" --jobs 3 --yes
导出.cmd --input "F:\工程目录" --jobs 1 --yes
不指定参数进入问答。--yes 适合使用固定参数重复运行;发现多套 Mentor 时需要 --mentor-home 指定。--timeout 可以调整单工具等待上限,默认 60 秒。
默认同时运行 3 个独立导出任务,可用 --jobs 1 切换串行,--jobs 2 到 8 调整并发上限。各工具使用同一已保存输入的独立副本;工程按顺序处理,库件读取在 ASCII 平铺属性就绪后与其他导出重叠,同库缓存继续共享。快照名称探测也参与同一并发上限,不额外占用任务槽。
工具退出通过进程通知立即唤醒,不额外睡眠;收尾与其他导出重叠执行。后台进程自然退出的宽限期最多 2 秒,已退出则立即结束等待。若仍有自有后台进程需要终止,会保留文件并记录提示,不标为已验证成功,也不阻塞其他任务。运行记录分别保存工具执行与收尾耗时,工程记录保存实际并发数与总耗时。
按 Ctrl+C 请求停止,等待工具收尾;不要直接关闭窗口。下次使用 --retry 指定批次,可恢复未完成项。成功产物只有在输入、工具和输出摘要仍匹配时才复用;源工程变化会完整重新导出该工程。
默认尽力导出并持续完成批次。单项异常、缺工具/缺库、超时、格式检查提示或进程收尾提示,都记录下来并继续后续工具和工程;报告文件被占用也不阻塞导出。只有用户主动按 Ctrl+C 才停止批次。
+15
View File
@@ -0,0 +1,15 @@
# MentorExport 0.1.2
2026-11-09。加速原理图批量导出,保留 0.1.1 的单项问题不阻塞批次策略。
- 默认 3 个导出任务并发,支持 `--jobs 1` 串行和 `--jobs 2` 到 `--jobs 8`。问答模式直接采用默认 3,无需额外配置。
- 每工具独立数据库副本、输出、日志、私有桌面和进程 Job;仅协调线程更新工程记录,输出分类和逐项顺序稳定。
- ASCII 平铺属性完成后即可开始库件读取,与其他格式导出重叠;同库缓存继续在顺序处理的工程间共享。
- 快照探测与正文导出重叠,同样计入并发上限。
- 工具退出改为进程通知唤醒,消除固定轮询睡眠;收尾等待与其他导出重叠。保留后台进程最多 2 秒的自然退出宽限期,避免过早终止导致下次重试需要重导。
- 若后台进程仍需强制结束,文件照常保留、显示提示,不作为已验证成功结果复用;库件记录也不进入已验证缓存。进程退出确认和工作副本保留机制继续生效。
- 移除一次重复的库件记录写入,新增工具执行/进程收尾耗时及工程并发信息。
下载完整 ZIP 解压,双击 `导出.cmd`。本机需已有 Mentor 与可用许可,Python 和依赖已预置。实际兼容环境仍为 EE7.9.5;其他安装的原生库读取沿用已验证指纹准入。
实测同一对真实工程:v0.1.1 串行 156.015 秒,新版默认并发 74.594 秒,耗时减少 52.2%。22 项均取得且通过检查,EDF/KYN 连接和器件属性、关联库记录与串行结果相同;重试校验后复用耗时 13.609 秒。64 项自动测试通过。完整范围见 [并发验收](../validation-2026-11-09-parallel.md),实际速度取决于工程和机器环境。
+45
View File
@@ -0,0 +1,45 @@
# 0.1.2 并发导出验收
2026-11-09。本机 Windows x64、Mentor EE7.9.5。默认每工程 3 个导出任务并发,工程顺序处理,同库缓存继续共享。`--jobs 1` 保留串行入口,最大并发 8。
## 自动测试及审查
`python -X utf8 -m pytest -q`:**64 passed in 34.73s**。
新增回归先失败后通过,覆盖独立副本上的真实调度重叠、并发上限、稳定输出顺序、快照与正文导出重叠、库件在 ASCII 就绪后提前运行、并发取消后收尾、CLI 参数传递、无效并发数、进程退出通知唤醒,以及强制收尾产物不能作为已验证结果复用。
独立审查确认协调线程独占工程记录写入,worker 使用独立状态与副本,同库分区缓存仍为单写入者。审查发现的强制结束后误入成功缓存已修正,普通导出和原生库读取都保留产物但标为提示;末轮针对进程等待与库缓存复查无待处理发现。
## 同输入实测
从之前已验收的两个真实工程复制固定输入,分别使用不可修改的 v0.1.1 发行包和最终 v0.1.2 发行包。每次是独立新进程和新批次,未用 `--retry` 代替首次导出。
| 项目 | v0.1.1 串行 | v0.1.2 默认 3 并发 |
| --- | ---: | ---: |
| 两工程完整导出 | 156.015 秒 | 74.594 秒 |
| 取得的导出项 | 22 | 22 |
| 带提示项 | 0 | 0 |
| 批次退出码 | 0 | 0 |
本机样例耗时降低 **52.2%**,约为原速度的 **2.09 倍**。不同工程、磁盘负载及许可环境下不保证同一比例。
两工程逐一解析 EDIF 网表与 KYN:器件、字符串属性、物理引脚、网络及连接集合与串行结果完全一致。关联库件原生记录逐项完全一致。14 个工程输入文件摘要未变,工程内源数据库和源库检查均为 source_unchanged=true。
最终包复制到中文及空格目录,从 `C:\Windows\Temp` 通过真实 CMD 无参数问答启动,PATH 仅保留 System32,PYTHONHOME/PYTHONPATH 为无效目录。自动发现环境并使用预置 Python 成功。
随后使用 `--retry --jobs 1 --yes` 验证串行入口及重复使用:13.609 秒,退出 0,22 项成功产物校验后复用。664 个发行文件与 manifest、ZIP 内容均核对一致。
## 等待策略
固定 50 ms 轮询睡眠改为进程句柄等待,工具退出即唤醒;仍以最多 50 ms 检查用户取消和超时。正常后台进程保留最多 2 秒自然退出宽限期,等待与其他任务重叠。
曾以 0.25 秒宽限期测试,出现 14 项强制收尾提示,串行重试需重新导出并耗时 68.188 秒。最终保留 2 秒正常宽限期,解决重复运行退化;不靠过早终止后台进程制造加速。
## 证据
- 串行基线及初轮对照:`build/parallel-benchmark-20261109-141823/benchmark.json`。
- 最终实际问答、串行重试及逐项比较:`build/parallel-release-acceptance-20261109-143025/acceptance.json`。
- 原始逐工具记录新增 started_at、tool_seconds、cleanup_seconds;工程记录新增 execution.jobs 与 execution.seconds。
- 最终 ZIP SHA256:`28616ced54ea893058d6dcde00025a9a72122814ae4f0021ee790ff9ed7bf22c`。
本次未增加跨工程并发或原生库分区并发;主要加速来自独立导出任务并行、库件提前开始、快照探测重叠、进程退出通知及减少重复记录写入。
+1 -1
View File
@@ -1,2 +1,2 @@
"""Portable Mentor saved-schematic exports."""
__version__ = "0.1.1"
__version__ = "0.1.2"
+12 -6
View File
@@ -95,7 +95,9 @@ def previous_results(record):
def run_batch(projects, problems, installation, output_root, scan_root, *, previous=None,
stop_event=None, timeout=60, exporter=None, progress=print):
stop_event=None, timeout=60, exporter=None, progress=print, jobs=3):
if not 1 <= jobs <= 8:
raise ValueError("并发数必须为 1 到 8")
if exporter is None:
from .exports import export_project
exporter = export_project
@@ -113,7 +115,7 @@ def run_batch(projects, problems, installation, output_root, scan_root, *, previ
"scan_root": str(Path(scan_root).resolve()), "output_root": str(output_root),
"created": datetime.now().astimezone().isoformat(), "installation": installation,
"owner": process_identity(os.getpid()),
"problems": problems, "projects": [], "warnings": []}
"problems": problems, "projects": [], "warnings": [], "execution": {"jobs": jobs}}
def warn(message):
if message not in record["warnings"]:
record["warnings"].append(message)
@@ -156,7 +158,7 @@ def run_batch(projects, problems, installation, output_root, scan_root, *, previ
try:
result = exporter(project, installation, Path(row["folder"]), shared,
previous=prior.get(project["path"]), stop_event=stop_event,
timeout=timeout, progress=progress)
timeout=timeout, progress=progress, jobs=jobs)
row["result"] = result
row["status"] = result["status"]
if result.get("cleanup_uncertain"):
@@ -241,6 +243,7 @@ def main(argv=None):
parser.add_argument("--mentor-home", help="指定 Mentor SDD_HOME")
parser.add_argument("--retry", help="重试已有批次,指定批次目录或 batch.json")
parser.add_argument("--timeout", type=float, default=60, help="单工具超时秒数,默认 60")
parser.add_argument("--jobs", "-j", type=int, default=3, help="同时运行的导出任务数,默认 3;1 为串行,最多 8")
parser.add_argument("--yes", "-y", action="store_true", help="采用已给定参数运行,不显示交互菜单")
parser.add_argument("--check", action="store_true", help="只检查预置环境与 Mentor")
parser.add_argument("--json", action="store_true", help="环境检查以 JSON 输出")
@@ -249,6 +252,8 @@ def main(argv=None):
try:
if args.timeout <= 0:
raise ValueError("超时必须大于 0")
if not 1 <= args.jobs <= 8:
raise ValueError("并发数必须为 1 到 8")
errors = runtime_check()
if errors:
raise ValueError("预置运行环境不完整,请重新解压发行包:\n" + "\n".join(errors))
@@ -269,6 +274,7 @@ def main(argv=None):
return 0 if installation["preflight"]["status"] == "success" else 2
print(f"\nMentorExport {__version__} · 原理图详细导出")
print(f"Mentor:{installation['version']} {installation['root']}")
print(f"并发任务:{args.jobs}(--jobs 1 可切换串行)")
print("读取已保存的原理图。请先保存目标工程;工具仅在副本上运行。")
if not args.retry and settings.get("output"):
candidates = sorted(Path(settings["output"]).glob("批次_*/batch.json"), reverse=True)[:30]
@@ -277,7 +283,7 @@ def main(argv=None):
print("发现未完成批次:" + str(unfinished[0].parent))
if not args.yes and _ask("输入 1 重试该批次;回车开始新批次") == "1":
return main(["--retry", str(unfinished[0]), "--mentor-home", installation["root"],
"--timeout", str(args.timeout)])
"--timeout", str(args.timeout), "--jobs", str(args.jobs)])
previous = None
if args.retry:
previous_path = Path(args.retry)
@@ -329,7 +335,7 @@ def main(argv=None):
signal.signal(signal.SIGINT, cancel)
try:
batch, code = run_batch(projects, problems, installation, output_root, scan_root,
previous=previous, stop_event=stop, timeout=args.timeout)
previous=previous, stop_event=stop, timeout=args.timeout, jobs=args.jobs)
finally:
signal.signal(signal.SIGINT, old_handler)
record = load_json(batch / "batch.json", {"projects": []})
@@ -347,7 +353,7 @@ def main(argv=None):
if action in ("1", "2"):
os.startfile(str(batch if action == "1" else batch / "导出汇总.html"))
elif action == "3":
return main(["--retry", str(batch), "--mentor-home", installation["root"], "--timeout", str(args.timeout)])
return main(["--retry", str(batch), "--mentor-home", installation["root"], "--timeout", str(args.timeout), "--jobs", str(args.jobs)])
return code
except KeyboardInterrupt:
print("\n已取消。")
+86 -33
View File
@@ -1,6 +1,7 @@
"""Saved-project exports through short lived, owned, isolated tool processes."""
from copy import deepcopy
from contextlib import contextmanager
from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED
import ctypes
from datetime import datetime, timezone
import hashlib
@@ -9,6 +10,7 @@ import os
from pathlib import Path
import re
import shutil
import subprocess
import tempfile
import time
import uuid
@@ -94,7 +96,7 @@ def _run(command, cwd, environment, log_dir, *, stop_event, timeout):
log_dir.mkdir(parents=True, exist_ok=True)
desktop = PrivateDesktop("mexp-" + uuid.uuid4().hex, enabled=True)
result = {"command": list(map(str, command)), "cwd": str(cwd), "exit_code": None,
"timed_out": False, "cancelled": False}
"timed_out": False, "cancelled": False, "started_at": datetime.now(timezone.utc).isoformat()}
started = time.monotonic()
try:
with desktop:
@@ -113,12 +115,18 @@ def _run(command, cwd, environment, log_dir, *, stop_event, timeout):
except Exception as error:
result["dialog_inspection_error"] = str(error)
break
time.sleep(0.05)
try:
process.wait(timeout=min(0.05, max(0, timeout - (time.monotonic() - started))))
except subprocess.TimeoutExpired:
pass
result["exit_code"] = process.poll()
result["tool_seconds"] = round(time.monotonic() - started, 3)
except Exception as error:
result["error"] = str(error)
finally:
result["seconds"] = round(time.monotonic() - started, 3)
if "tool_seconds" in result:
result["cleanup_seconds"] = round(result["seconds"] - result["tool_seconds"], 3)
result["cleanup"] = desktop.cleanup_record
cleanup = result["cleanup"] or {}
if cleanup.get("errors") or any(p.get("job_active_processes", 0) for p in cleanup.get("processes", [])):
@@ -303,12 +311,16 @@ def _status(items):
return "exported" if any(i.get("outputs") for i in items) else "skipped"
def export_project(project, installation, project_dir, shared_dir, *, previous=None, stop_event=None, timeout=45, progress=print):
def export_project(project, installation, project_dir, shared_dir, *, previous=None, stop_event=None, timeout=45, progress=print, jobs=3):
if not 1 <= jobs <= 8:
raise ValueError("并发数必须为 1 到 8")
project_started = time.monotonic()
project_dir = Path(project_dir).resolve()
project_dir.mkdir(parents=True, exist_ok=True)
shared_dir = Path(shared_dir).resolve()
result = {"status": "running", "project": deepcopy(project), "captured_at": datetime.now(timezone.utc).isoformat(),
"source_fingerprint": None, "source_hashes": {}, "items": [], "source_unchanged": None,
"execution": {"jobs": jobs},
"coverage_limits": ["Exports preserve each tool/mode separately; no ERC or manufacturing certification.",
"CMS requests read mode but may report ReadOnly=False; only isolated copies are opened.",
"Cell coordinate units and rotation remain native integers; multi-slot/swappable mappings unverified."]}
@@ -336,25 +348,22 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
return result
environment = _environment(installation)
old_items = {i["key"]: i for i in (previous or {}).get("items", [])}
discovery = {"status": "failed", "names": []}
result["snapshot_discovery"] = discovery
try:
snapshot_work = temp / "snapshot-discovery"
shutil.copytree(captured, snapshot_work)
discovery.update(_snapshot_metadata(project, installation, snapshot_work,
project_dir / "90_运行日志" / "snapshots", stop_event=stop_event,
timeout=min(timeout, 20)))
if discovery.get("cleanup_uncertain"):
result.update(cleanup_uncertain=True, workspace_preserved=str(temp))
except Exception as error:
discovery.update(status="failed", reason=str(error))
if discovery["status"] != "success":
result["coverage_limits"].append("Other snapshot names were not obtained: " + discovery.get("reason", ""))
_checkpoint(state, result)
for key, tool, folder in ITEMS:
def discover():
discovery = {"status": "failed", "names": []}
try:
snapshot_work = temp / "snapshot-discovery"
shutil.copytree(captured, snapshot_work)
discovery.update(_snapshot_metadata(project, installation, snapshot_work,
project_dir / "90_运行日志" / "snapshots", stop_event=stop_event,
timeout=min(timeout, 20)))
except Exception as error:
discovery.update(status="failed", reason=str(error))
return discovery
def execute(spec, task_state):
key, tool, folder = spec
output = None
item = {"key": key, "status": "failed", "reason": "", "outputs": [], "provenance": {}}
result["items"].append(item)
try:
progress(project["name"] + ": " + key)
identity = _tool_identity(key, tool, installation)
@@ -363,19 +372,19 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
"input_fingerprint": fingerprint, "reused": False}
if stop_event is not None and stop_event.is_set():
item.update(status="cancelled", reason="Cancelled before launch")
continue
return item, task_state
if _reusable(old_items.get(key), fingerprint, identity):
item.update(deepcopy(old_items[key]))
item["provenance"]["reused"] = True
item["reason"] = "Verified previous successful output reused"
continue
return item, task_state
if key == "library":
_export_library(item, project, installation, result, project_dir / folder, shared_dir,
_export_library(item, project, installation, task_state, project_dir / folder, shared_dir,
temp, environment, stop_event, timeout)
continue
return item, task_state
if not identity.get("sha256") or (tool == "cms" and not identity.get("matches")):
item.update(status="unavailable", reason="Tool missing or COM registration does not match selected installation")
continue
return item, task_state
work = temp / key
shutil.copytree(captured, work)
output = work / "output"
@@ -386,7 +395,7 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
item["provenance"].update(run)
item["provenance"]["log_dir"] = str(log_dir)
if run.get("cleanup_uncertain"):
result.update(cleanup_uncertain=True, workspace_preserved=str(temp))
task_state.update(cleanup_uncertain=True, workspace_preserved=str(temp))
item["reason"] = "Owned process cleanup is uncertain; workspace preserved; remaining exports continue"
elif run.get("cancelled"):
item.update(status="cancelled", reason="Cancelled while tool was running")
@@ -407,6 +416,8 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
item.update(status="success", outputs=[str(p) for p in published])
item["validation"] = {"status": "passed"}
item["provenance"]["output_files"] = [_file_info(p) for p in published]
if any(process.get("forced_termination") for process in run.get("cleanup", {}).get("processes", [])):
item.update(status="exported", reason="Files retained; owned background processes exceeded shutdown grace and were terminated; background completion unverified")
except Exception as error:
item.update(status="failed", reason=str(error))
finally:
@@ -425,7 +436,7 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
saved.append(_file_info(target))
except OSError as error:
publication_errors.append(str(error))
result.setdefault("workspace_preserved", str(temp))
task_state.setdefault("workspace_preserved", str(temp))
try:
if source.is_file() and source.stat().st_size:
saved.append(_file_info(source))
@@ -443,8 +454,45 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
item["status"] = "exported"
if item["status"] == "exported":
item["validation"] = {"status": "warning", "reason": item["reason"]}
result["status"] = "running"
_checkpoint(state, result)
return item, task_state
result["items"] = [{"key": key, "status": "pending", "outputs": [], "reason": "", "provenance": {}}
for key, tool, folder in ITEMS]
specs = {spec[0]: spec for spec in ITEMS}
positions = {item["key"]: index for index, item in enumerate(result["items"])}
ready = ["ascii_flat", "snapshots"] + [key for key, tool, folder in ITEMS if key not in ("ascii_flat", "library")]
# Only the coordinator publishes receipts; workers own distinct clones, items and logs.
with ThreadPoolExecutor(max_workers=jobs, thread_name_prefix="mentor-export") as pool:
active = {}
while ready or active:
while ready and len(active) < jobs:
key = ready.pop(0)
if key == "snapshots":
future = pool.submit(discover)
else:
task_state = {"items": list(result["items"]),
"library_source_hashes": result["library_source_hashes"]}
future = pool.submit(execute, specs[key], task_state)
active[future] = key
completed, _ = wait(active, return_when=FIRST_COMPLETED)
for future in completed:
key = active.pop(future)
if key == "snapshots":
metadata = future.result()
result["snapshot_discovery"] = metadata
task_state = metadata
if metadata["status"] != "success":
result["coverage_limits"].append("Other snapshot names were not obtained: " + metadata.get("reason", ""))
else:
item, task_state = future.result()
result["items"][positions[key]] = item
if key == "ascii_flat":
ready.insert(0, "library")
if task_state.get("cleanup_uncertain"):
result["cleanup_uncertain"] = True
if task_state.get("workspace_preserved"):
result["workspace_preserved"] = str(temp)
_checkpoint(state, result)
try:
after = _hashes(_inputs(project))
result["source_hashes_after"] = after
@@ -459,6 +507,7 @@ def export_project(project, installation, project_dir, shared_dir, *, previous=N
if item["status"] == "success":
item.update(status="exported", reason="Source changed during export; captured output retained; re-export to verify current source",
validation={"status": "warning", "reason": "Source changed during export"})
result["execution"]["seconds"] = round(time.monotonic() - project_started, 3)
result["status"] = _status(result["items"])
_checkpoint(state, result)
return result
@@ -619,6 +668,9 @@ def _export_library(item, project, installation, result, destination, shared_dir
data["errors"].append({"partition": relative, "reason": run.get("error") or "Native read failed/timeout/cleanup error"})
persist()
continue
trusted_run = not any(process.get("forced_termination") for process in run.get("cleanup", {}).get("processes", []))
if not trusted_run:
data["errors"].append({"partition": relative, "reason": "Owned background process terminated during shutdown; records retained without verified reuse"})
try:
native = json.loads((log / "stdout.txt").read_text(encoding="utf-8"))
_validate_native(native, mode)
@@ -634,15 +686,16 @@ def _export_library(item, project, installation, result, destination, shared_dir
if known:
old_names = {row["name"] for row in selected}
selected.extend(row for row in known["selected"] if row["name"] not in old_names)
partitions[relative] = {"names": {row["name"] for row in native["records"]},
"selected": selected, "run": deepcopy(run)}
if trusted_run:
partitions[relative] = {"names": {row["name"] for row in native["records"]},
"selected": selected, "run": deepcopy(run)}
# Remove unrelated full-library stdout only after selected records are durable.
persist()
(log / "stdout.txt").unlink()
run["stdout_retained"] = False
_checkpoint(log / "run.json", run)
partitions[relative]["run"] = deepcopy(run)
persist()
if trusted_run:
partitions[relative]["run"] = deepcopy(run)
except (OSError, ValueError) as error:
data["errors"].append({"partition": relative, "reason": str(error)})
persist()
+18
View File
@@ -131,3 +131,21 @@ def test_preflight_exception_is_a_warning_not_a_batch_blocker(tmp_path, monkeypa
monkeypatch.setattr(cli, "_save_settings", lambda value: None)
assert cli.main(["--input", str(tmp_path), "--output", str(tmp_path / "out"), "--yes"]) == 0
assert list((tmp_path / "out").glob("批次_*/batch.json"))
def test_batch_passes_worker_limit_and_records_execution_mode(tmp_path):
from mentorexport.cli import run_batch
limits = []
def exporter(project, installation, folder, shared, **options):
limits.append(options["jobs"])
return {"status": "success", "items": []}
batch, code = run_batch([{"name": "工程", "path": str(tmp_path / "a.prj")}], [], {},
tmp_path / "out", tmp_path, exporter=exporter, jobs=2)
record = json.loads((batch / "batch.json").read_text(encoding="utf-8"))
assert code == 0 and limits == [2] and record["execution"]["jobs"] == 2
def test_invalid_worker_limit_does_not_start_export(tmp_path):
from mentorexport.cli import main
for jobs in ("0", "9"):
assert main(["--input", str(tmp_path), "--jobs", jobs, "--yes"]) == 2
+158 -1
View File
@@ -25,6 +25,163 @@ def backend():
return exports
def test_parallel_exports_overlap_on_distinct_clones_and_keep_receipt_order(sample, monkeypatch):
from threading import Barrier, Lock
exports = backend()
project, install, root = sample
barrier, lock = Barrier(2), Lock()
calls, workspaces = [], []
run_ascii = ascii_runner(calls)
active = maximum = 0
def runner(command, cwd, environment, log_dir, **kwargs):
nonlocal active, maximum
with lock:
active += 1
maximum = max(maximum, active)
workspaces.append(Path(cwd))
try:
barrier.wait(timeout=5)
return run_ascii(command, cwd, environment, log_dir, **kwargs)
finally:
with lock:
active -= 1
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "parallel", root / "shared", jobs=2, progress=lambda _: None)
assert maximum == 2 and len(set(workspaces)) == 2
assert [item["key"] for item in result["items"]] == [spec[0] for spec in exports.ITEMS]
assert [i["key"] for i in result["items"] if i["status"] == "success"] == ["ascii_flat", "ascii_hier"]
assert result["source_unchanged"] and result["execution"]["jobs"] == 2
def test_library_starts_after_flat_without_waiting_for_other_exports(sample, monkeypatch):
from threading import Event
exports = backend()
project, install, root = sample
library_started = Event()
runner = ascii_runner([])
def run(command, *args, **kwargs):
if "-h" in command:
assert library_started.wait(5), "library waited for unrelated exports"
return runner(command, *args, **kwargs)
def library(item, project, installation, result, *args):
flat = next(i for i in result["items"] if i["key"] == "ascii_flat")
assert flat["status"] == "success" and Path(flat["outputs"][0]).is_file()
library_started.set()
item.update(status="unavailable", reason="fixture without library")
monkeypatch.setattr(exports, "_run", run)
monkeypatch.setattr(exports, "_export_library", library)
result = exports.export_project(project, install, root / "parallel", root / "shared", jobs=2, progress=lambda _: None)
assert library_started.is_set() and result["items"][1]["status"] == "success"
def test_snapshot_listing_overlaps_with_exports(sample, monkeypatch):
from threading import Barrier
exports = backend()
project, install, root = sample
install["tools"]["constraints_xml"] = install["tools"]["ascii"]
barrier = Barrier(2)
runner = ascii_runner([])
def run(command, cwd, environment, log_dir, **kwargs):
if "-list" in command or "-f" in command:
barrier.wait(timeout=5)
if "-list" in command:
Path(log_dir).mkdir(parents=True)
(Path(log_dir) / "stdout.txt").write_text(" -> Saved\n")
return {"exit_code": 0}
return runner(command, cwd, environment, log_dir, **kwargs)
monkeypatch.setattr(exports, "_run", run)
result = exports.export_project(project, install, root / "parallel", root / "shared", jobs=2, progress=lambda _: None)
assert result["snapshot_discovery"]["status"] == "success"
assert result["items"][0]["status"] == "success"
def test_forced_descendant_shutdown_keeps_files_but_never_reuses_as_verified(sample, monkeypatch):
exports = backend()
project, install, root = sample
calls = []
ascii_run = ascii_runner(calls)
def runner(*args, **kwargs):
run = ascii_run(*args, **kwargs)
run["cleanup"] = {"errors": [], "processes": [{"forced_termination": True,
"forced_root_termination": False, "forced_descendant_pids": [123],
"termination_reason": "descendant_grace_expired", "job_active_processes": 0}]}
return run
monkeypatch.setattr(exports, "_run", runner)
first = exports.export_project(project, install, root / "first", root / "shared", jobs=2, progress=lambda _: None)
assert first["items"][0]["status"] == "exported" and first["items"][0]["outputs"]
assert first["items"][0]["validation"]["status"] == "warning"
exports.export_project(project, install, root / "retry", root / "shared", jobs=2, previous=first, progress=lambda _: None)
assert len(calls) == 4
def test_forced_library_shutdown_retains_records_without_trusted_cache(sample, monkeypatch):
exports = backend()
project, install, root = sample
library = root / "library/PartsDBLibs"
library.mkdir(parents=True)
(library / "a.pdb").write_bytes(b"parts")
project["library"] = str(library.parent)
monkeypatch.setattr(exports, "VALIDATED_DLLS", {})
monkeypatch.setattr(exports, "_LIBRARY_CACHE", {})
calls = []
def runner(command, cwd, environment, log_dir, **kwargs):
if "-f" in command or "-h" in command:
output = Path(next(arg.split("=", 1)[1] for arg in command if arg.startswith("-file=")))
output.write_text("iCDB Ascii Report\n" + ("Block Top\n" if "-h" in command else "Config default\n") + "\tProp Part Number USED\n")
return dict(exit_code=0, cleanup={"errors": []})
calls.append(command)
Path(log_dir).mkdir(parents=True, exist_ok=True)
(Path(log_dir) / "stdout.txt").write_text(json.dumps({"mode": "parts", "count": 1, "records": [{"name": "USED"}]}), encoding="utf-8")
return dict(exit_code=0, cleanup={"errors": [], "processes": [{"forced_termination": True}]})
monkeypatch.setattr(exports, "_run", runner)
first = exports.export_project(project, install, root / "first", root / "shared", jobs=2, progress=lambda _: None)
item = next(i for i in first["items"] if i["key"] == "library")
assert item["status"] == "exported" and item["validation"]["status"] == "warning"
data = next(json.loads(Path(path).read_text(encoding="utf-8")) for path in item["outputs"] if Path(path).name.startswith("used-"))
assert data["records"]["parts"][0]["name"] == "USED" and data["complete"] is False
exports.export_project(project, install, root / "retry", root / "shared", jobs=2, previous=first, progress=lambda _: None)
assert len(calls) == 2
def test_parallel_cancel_drains_active_tasks_without_launching_queued_vendor_work(sample, monkeypatch):
from threading import Barrier
exports = backend()
project, install, root = sample
stop, barrier = Event(), Barrier(2)
calls = []
def runner(command, cwd, *args, **kwargs):
calls.append(command)
barrier.wait(timeout=5)
stop.set()
return dict(exit_code=None, cancelled=True, cleanup={"errors": []})
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "cancel", root / "shared", jobs=2,
stop_event=stop, progress=lambda _: None)
assert len(calls) == 2 and result["status"] == "cancelled"
assert all(item["status"] == "cancelled" for item in result["items"])
def test_process_exit_notification_advances_without_polling_delay(tmp_path, monkeypatch):
exports = backend()
class Process:
pid = 123
finished = False
def poll(self): return 0 if self.finished else None
def wait(self, timeout=None):
self.finished = True
return 0
class Desktop:
cleanup_record = {"errors": [], "processes": []}
def __init__(self, *args, **kwargs): pass
def __enter__(self): return self
def __exit__(self, *args): pass
def start_process(self, *args, **kwargs): return Process()
def inspect_dialogs(self, pid): return []
monkeypatch.setattr(exports, "PrivateDesktop", Desktop)
result = exports._run(["fixture"], tmp_path, {}, tmp_path / "logs", stop_event=None, timeout=0.15)
assert result["exit_code"] == 0 and not result["timed_out"]
def ascii_runner(calls):
def run(command, cwd, environment, log_dir, *, stop_event, timeout):
calls.append(command)
@@ -216,7 +373,7 @@ def test_uncertain_snapshot_cleanup_continues_exports_and_preserves_copy(sample,
return dict(exit_code=0, cleanup_uncertain=True, cleanup={"errors": ["exit unconfirmed"]})
monkeypatch.setattr(exports, "_run", runner)
result = exports.export_project(project, install, root / "attempt", root / "shared", progress=lambda _: None)
assert len(calls) == 3 and "-list" in calls[0]
assert len(calls) == 3 and any("-list" in call for call in calls)
assert result["cleanup_uncertain"] is True
assert result["status"] == "skipped"
assert not any(i["status"] == "cancelled" for i in result["items"])