diff --git a/README.md b/README.md index 2e625d1..8bcf39a 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ **最新版本:V1.1.2** · [下载便携包](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/download/v1.1.2/DxPartReplace-1.1.2.zip) · [版本说明与附件](https://git.simgreek.com/simgreek/dxdesigner-part-replacement/releases/tag/v1.1.2) · [完整使用说明](docs/user/v1.1.0-project-tasks.md) -当前开发分支新增了[目标封装选择](docs/user/target-cell-selection.md):在料号规则中读取并选择合法封装,未选择时阻止执行;BOM 保持原 11 列。此功能尚未打包到上面的 V1.1.2 下载包。 +当前开发分支新增了[目标封装选择](docs/user/target-cell-selection.md):目标封装可留空,此时工具不主动覆写封装,可后续与中心库同步;需要指定时,在料号规则中读取并选择合法封装。BOM 保持原 11 列。此功能尚未打包到上面的 V1.1.2 下载包。 ## 窗口操作教程 diff --git a/docs/reviews/release-queue-filesystem-fix-review-20261019.md b/docs/reviews/release-queue-filesystem-fix-review-20261019.md new file mode 100644 index 0000000..a70c1ed --- /dev/null +++ b/docs/reviews/release-queue-filesystem-fix-review-20261019.md @@ -0,0 +1,34 @@ +# Queue filesystem optimization review — 2026-10-19 + +## Verdict + +**Approved for the final reviewed diff. No remaining Critical or Important findings.** The initial rotation exception-semantics issue identified during review has been corrected. The optimization retains live filesystem validation, per-step evidence, fsync, atomic command publication, and file identity checks. + +Reviewed `legacy_paths.py`, `retention.py`, `logging_setup.py`, `quick_run.py`, the new filesystem-work tests, relevant existing long-path/logging/cache/retention coverage, and supplied XML/metrics artifacts. No tests, CAD, application operations, source changes, or Git mutations were performed. This report is the only output written. + +## Final code review + +- `_prepare_project_access` uses a local `seen` set keyed by the exact raw project-path field. Repeated commands resolve each input once within that preparation call. Different raw paths are not silently merged, and the set is discarded afterward. Strict short-path/alias identity checks and before/after-worker access validation remain. This is not a lifetime cache of filesystem trust. +- `rotate_text_log` performs one fresh `stat` for normal files. If that query raises `OSError` or `ValueError`, the exceptional branch invokes `Path.exists()` to retain pathlib's own platform-specific distinction between an absent/invalid path and a genuine filesystem error. False returns false; true re-raises the original exception; permission and other non-ignored errors from `exists` propagate. No blanket `OSError` suppression was added. +- This final exceptional branch fixes the intermediate implementation that caught only `FileNotFoundError`. On the installed Python 3.12, the former `exists()` also treated ENOTDIR, EBADF, ELOOP, specific Windows path errors, and nonencodable paths as absent. Keeping the platform predicate in the cold branch avoids duplicating those rules in the application. +- `_task_directory` and `_append_log` perform a live `is_dir()` check on every call and still execute `mkdir(parents=True, exist_ok=True)` when needed. Existing directories avoid the failed-mkdir path; removed directories are recreated. File collisions, missing ancestors, and permission failures still pass through the existing filesystem operations and error paths. A race removing the directory after the check remains an error, as was already possible after the previous mkdir call. +- Task ID validation is unchanged. Existing symlink/junction handling is unchanged. Rotation's link checks, thresholds, backup ordering, and rename operations remain in place. +- The event pre/post stamps and handle identity comparison, sequence invalidation, flush/fsync, current-command atomic writer, post-worker command comparison, `batch.request` job binding, backup-state durability, executor shutdown, and every step's event remain unchanged by this diff. + +## Test evidence inspected + +| Artifact | Tests | Failures | Errors | Skipped | +| --- | ---: | ---: | ---: | ---: | +| `build/release-queue-filesystem-red-20261019.xml` | 3 | 3 | 0 | 0 | +| `build/release-queue-filesystem-green-20261019.xml` | 19 | 0 | 0 | 0 | +| `build/release-queue-filesystem-errors-red-20261019.xml` | 8 | 3 | 0 | 0 | +| `build/release-queue-filesystem-final-20261019.xml` | 24 | 0 | 0 | 0 | +| `build/release-queue-performance-optimized-20261019.xml` | 1 | 0 | 0 | 0 | + +The new tests check distinct-input resolution counts across two preparation calls, one-stat rotation for an existing small log, avoidance of redundant mkdir, recreation after removal, original absence/error semantics, and continued permission-error propagation. Existing suites cover path redirection, event sequence cache invalidation, rotation, and canonical immediate event visibility. The artifacts were read, not rerun by this reviewer. + +## Performance and release limits + +The unchanged independent 1,000-step test's supplied optimized metrics record **22.5226234 seconds**, 91 saved steps, and 2,210,321 peak bytes with tracemalloc enabled. The source still requires `elapsed < 30`, `peak < 128 MiB`, complete progress, exact saved/no-match counts, and 1,000 unique calls. No gate relaxation or workload reduction was found. + +The final exception correction affects the cold error branch; the measured normal-file optimization remains the single-stat path. This static review does not substitute for building the final frozen artifact with the reviewed optimization. The coordinator's completed v2 live CAD acceptance applies to the preceding artifact; any final release claim must identify the rebuilt package and its applicable verification evidence. diff --git a/docs/reviews/release-queue-performance-review-20261019.md b/docs/reviews/release-queue-performance-review-20261019.md new file mode 100644 index 0000000..4e319ca --- /dev/null +++ b/docs/reviews/release-queue-performance-review-20261019.md @@ -0,0 +1,80 @@ +# Read-only queue performance review — 2026-10-19 + +## Conclusion + +There are concrete redundant filesystem operations that can be removed while retaining durable writes and identity checks. The strongest simple candidate is deduplicating project access preparation before resolving paths. Logging also repeats metadata/directory work. **The current profile does not prove these changes alone will bring the independent 40.90-second result below 30 seconds.** A measured result is required; the threshold should remain unchanged. + +No source edits, tests, CAD, application operations, or Git mutations were performed. This report is the only output written. + +## Evidence and interpretation + +`build/release-queue-profile-20261019-v2/stress-metrics.json` records 1,000 steps, 91 saved steps, 36.731 seconds, and approximately 2.34 MB peak memory. The separate independent 40.90-second failure was supplied by the coordinator. The profiled test retains tracemalloc, so Python/path allocation reductions can matter as well as disk calls. + +Selected cumulative profile costs: + +| Path | Calls | Cumulative seconds | Interpretation | +| --- | ---: | ---: | --- | +| `execute_step` | 1,000 | 20.756 | Broad parent; includes child work below | +| `_write_current_command` | 1,000 | 8.341 | Predominantly required durable command publication | +| `append_task_event` | approximately 1,043 | 8.301 | Required event evidence plus repeated metadata work | +| `_io.open` | approximately 7,131 | 7.129 | Global cost; includes required files and test worker | +| `nt.stat` | approximately 14,663 | 4.375 | Global cost; some redundant calls, many identity checks | +| `nt.fsync` | approximately 2,063 | 3.939 | Required durability; retain | +| `_prepare_project_access` | 1 | 2.654 | Repeats project resolution across command rows | +| `nt.replace` | approximately 1,020 | 2.005 | Atomic publication; retain | + +These costs overlap and must not be added. In particular, `BatchBackupCoordinator.__exit__/shutdown/join` appears as about 33 seconds, overlapping the queue/worker-thread work in this profile. It is not evidence of an additional exclusive 33-second cleanup operation. The test stubs the actual backup copy; removing `shutdown(wait=True)` would weaken cleanup rather than establish a performance fix. + +## Ranked optimization candidates + +### 1. Deduplicate project paths before expensive access preparation + +**Location:** `legacy_paths.py:107`, `_prepare_project_access`. + +The helper reads all command rows and executes `Path(value).resolve(strict=False)` before testing whether that project's access pair was already produced. This test has 1,000 commands but only 20 project paths. For ordinary short paths, no access row is produced at all, so the existing `pair not in rows` guard never helps. + +Use an invocation-local set of raw project-path field values and process each distinct value once before resolution, file queries, short-path lookup, or alias creation. Keep the access pair output unique and retain strict alias-to-project identity validation. Do not cache across invocations or remove `validate_project_access` before/after workers. + +**Expected effect:** expensive project preparation falls from 1,000 to 20 evaluations in this workload. The profile assigns 2.654 seconds to the entire helper, which bounds the available saving; it cannot recover the whole regression alone. + +**Verification:** repeated command rows should produce the identical access manifest, with resolution/short-path calls scaling with distinct project strings. Existing long-path/redirected-entry tests must still pass. Per-worker identity checks must still reject an alias changed after preparation. + +### 2. Replace existing-log `exists()` plus `stat()` with one fresh metadata read + +**Location:** `retention.py:29`, `rotate_text_log`. + +For each normal existing log, `path.exists()` performs a stat, immediately followed by `path.stat().st_size`. One fresh stat can answer both existence and size. Preserve the missing-file behavior and error handling deliberately, and retain all link checks and backup rename checks when rotation is needed. + +This removes one filesystem metadata call per rotation check without changing log bytes, event count, flush/fsync, sequence calculation, rotation threshold, or atomic operations. It also benefits callers beyond the event log. + +**Verification:** missing log, existing below-threshold log, rotation, linked logs/backups, and filesystem error behavior. Instrument metadata calls for the below-threshold existing-file case; it should use one query rather than two. Do not replace live checks with a permanently cached file size. + +### 3. Avoid failed mkdir calls for already-existing log directories + +**Locations:** `logging_setup.py:225` (`_task_directory`) and `quick_run.py:1082` (`_append_log`). + +Both call `mkdir(parents=True, exist_ok=True)` on every append. On an existing Windows directory, pathlib attempts mkdir, catches the existing-directory error, and checks whether the path is a directory. A fresh `is_dir()` fast path can avoid that failed mkdir call; create the directory with the existing code only when the check is false. Keep task-ID validation on every call. + +This retains a live directory check and recovery after directory deletion rather than trusting an unvalidated lifetime cache. It is a small optimization with uncertain timing benefit, not a guaranteed multi-second saving. + +**Verification:** existing directory, removed directory recreated on next append, non-directory collision, invalid task IDs, permission failures, and concurrent first append. Event contents, sequences, immediate readability, and fsync count must remain unchanged. + +### 4. Cache pure command/session path construction, not filesystem trust + +`execute_step` repeatedly constructs the same `batch.request`, `current-command.tsv`, `current-state.tsv`, and related parent paths. A session can retain immutable Path/string values established at startup while still performing every live file stamp, per-step command write, request write, and post-worker identity comparison. + +The high pathlib allocation counts make this a plausible secondary saving under tracemalloc, but the profile does not isolate its exclusive cost. Prefer the first three candidates before adding session fields. Never cache a stamp as proof that a file remains unchanged. + +## Work that should remain + +- The pre-append event path stamp, post-fsync handle stat, and final path stamp serve different purposes. The final path-vs-handle comparison explicitly accounts for Windows ctime semantics and binds the sequence cache to the file actually written. Do not remove it as an apparent duplicate. External append/replacement and rotation must continue invalidating the cache (`test_event_append_cache.py`). +- `current-command.tsv` contains a different confirmed job on each step. Its fsync plus atomic replace and the post-worker stamp comparison are not redundant with the full queue or status file. Preserve deletion/change detection. +- `batch.request` binds the worker's batch mode to the expected job ID; VBS reads and compares it. The write is not shown to be useless merely because the command line also contains a job ID. Skipping it would be a protocol change requiring separate evidence, not the safe first optimization. +- Keep every `quick_run_executed` event and its timing/state/identity fields, event fsync, uncertain markers, backup state publication, atomic queue commit, and executor cleanup boundary. +- Do not disable tracemalloc, reduce the simulated queue, skip saved steps, falsify statuses, move required work outside the measured operation, or loosen the 30-second gate. + +## Suggested validation sequence + +Implement one candidate at a time only after the active real acceptance run is finished. First verify semantic guards and call-count reduction, then rerun the unchanged independent 1,000-item test in the same environment. Once correctness checks pass, measure the isolated performance gate and retain the raw metrics. Use a new profile only if the gate still fails or the result leaves the bottleneck uncertain; use per-thread wall spans/counters for backup overlap rather than subtracting the 33-second shutdown attribution. + +There is a defensible low-risk optimization direction, but no evidence-backed complete solution to the 10.90-second deficit yet. diff --git a/docs/testing/2026-10-19-release-preparation.md b/docs/testing/2026-10-19-release-preparation.md index 7d5ca65..8a538df 100644 --- a/docs/testing/2026-10-19-release-preparation.md +++ b/docs/testing/2026-10-19-release-preparation.md @@ -43,4 +43,8 @@ 验收工具同步补齐:BOM 检查所有 CLI 尝试的资源清理,不能被后续成功记录掩盖;执行阶段读取本次新队列检查点,遇 failed/blocked/skipped 立即失败,不能只等待可能保留 RUNNING 的任务状态。相关反例先失败、修复后 10 passed;未修改产品任务状态语义。 -独立千步模拟为 40.90267 秒,未通过小于 30 秒的原门槛,尚未放宽阈值。冻结多工程替换、保存重开和后续独立核验仍待完成,此处不宣称正式发布就绪。 +修正夹具后的 v2 冻结验收通过:两工程(路径 370 / 143 字符)均完成替换及保存重开核验,执行总耗时 278.312 秒。独立新 CAD 回读证实 RV05 改为新料号、Part Label 与 Value 为 0R,RV11 料号与 Value 未变。层次与展开网络均比对 57 个器件、103 个引脚、22 个网络,除目标属性外其余 56 个器件、引脚与网络连接不变。日志窗口和新一轮检查通过;1221 次执行观察中主桌面未切换、文件均可读,自有 CAD 未出现在主桌面,进程及私有桌面已释放。原工程与中央库跟踪的 1033 个文件哈希均未变。完整证据在 build/packaged-optional-acceptance-20261019-v2。 + +独立千步模拟初测 40.90267 秒,未通过小于 30 秒的原门槛。审查确认并移除重复工作:同次队列准备对重复工程路径先去重;日志轮转正常分支合并 exists/stat,异常分支保留 pathlib 平台语义;已有日志目录先检查,避免重复失败的 mkdir。没有改 fsync、原子提交、逐步事件、实时身份戳或门槛。优化后独立模拟 22.52262 秒、峰值 2210321 字节,通过原门槛;这不是实际 CAD 替换速度。长路径、事件缓存、轮转与日志异常语义共 24 项通过。 + +上述 v2 冻结实机证据对应 e2f76c3 的产品代码;后续仅文件操作优化仍须构建最终候选并确认。尚不发布或覆盖现有版本附件。 diff --git a/docs/user/media/target-cell-rule-editor.png b/docs/user/media/target-cell-rule-editor.png index b58c4ef..ab1339f 100644 Binary files a/docs/user/media/target-cell-rule-editor.png and b/docs/user/media/target-cell-rule-editor.png differ diff --git a/src/dx_productivity/trial_task_center/legacy_paths.py b/src/dx_productivity/trial_task_center/legacy_paths.py index 4a94ee4..40c59f2 100644 --- a/src/dx_productivity/trial_task_center/legacy_paths.py +++ b/src/dx_productivity/trial_task_center/legacy_paths.py @@ -110,11 +110,13 @@ def _prepare_project_access(root: Path): if not command.is_file(): return rows = [] + seen = set() with command.open(encoding='utf-8-sig', newline='') as stream: for row in csv.DictReader(stream, delimiter='\t'): value = row.get('project_path') - if not value: + if not value or value in seen: continue + seen.add(value) project = Path(value).resolve(strict=False) if (_fits(project.parent) and len(str(project)) < 260) or not project.is_file(): continue diff --git a/src/dx_productivity/trial_task_center/logging_setup.py b/src/dx_productivity/trial_task_center/logging_setup.py index dc3c732..d2f95a9 100644 --- a/src/dx_productivity/trial_task_center/logging_setup.py +++ b/src/dx_productivity/trial_task_center/logging_setup.py @@ -226,7 +226,8 @@ def _task_directory(context: LoggingContext, task_id: str) -> Path: if not task_id or Path(task_id).name != task_id or task_id in {".", ".."}: raise TaskCenterError("task_id", "task event requires a safe task identifier") path = context.root / "tasks" / task_id - path.mkdir(parents=True, exist_ok=True) + if not path.is_dir(): + path.mkdir(parents=True, exist_ok=True) return path diff --git a/src/dx_productivity/trial_task_center/quick_run.py b/src/dx_productivity/trial_task_center/quick_run.py index d602933..d54ffb3 100644 --- a/src/dx_productivity/trial_task_center/quick_run.py +++ b/src/dx_productivity/trial_task_center/quick_run.py @@ -1080,7 +1080,8 @@ def _read_status(status_path: Path) -> tuple[tuple[str, str], ...]: def _append_log(log_path: Path, entry: str, *, check_rotation: bool = True) -> None: - log_path.parent.mkdir(parents=True, exist_ok=True) + if not log_path.parent.is_dir(): + log_path.parent.mkdir(parents=True, exist_ok=True) if check_rotation: rotate_text_log(log_path) with log_path.open("a", encoding="utf-8", newline="\n") as handle: diff --git a/src/dx_productivity/trial_task_center/retention.py b/src/dx_productivity/trial_task_center/retention.py index f9e1844..0b6ac3b 100644 --- a/src/dx_productivity/trial_task_center/retention.py +++ b/src/dx_productivity/trial_task_center/retention.py @@ -35,7 +35,15 @@ def rotate_text_log( path = Path(path) if max_bytes < 1 or backup_count < 1: raise ValueError("log retention limits must be positive") - if not path.exists() or path.stat().st_size <= max_bytes: + try: + size = path.stat().st_size + except (OSError, ValueError): + # Preserve pathlib's platform-specific absence/error handling. Only + # exceptional paths need the second query; normal logs use one stat. + if not path.exists(): + return False + raise + if size <= max_bytes: return False if _is_link(path): raise OSError(f"refusing to rotate linked log: {path}") diff --git a/tests/v1_1/test_queue_filesystem_work.py b/tests/v1_1/test_queue_filesystem_work.py new file mode 100644 index 0000000..8d95b8f --- /dev/null +++ b/tests/v1_1/test_queue_filesystem_work.py @@ -0,0 +1,74 @@ +"""Bound repeated filesystem work without caching mutable file identities.""" +from pathlib import Path +from types import SimpleNamespace +import errno +import pytest + +from dx_productivity.trial_task_center import legacy_paths, logging_setup, quick_run, retention + + +def test_project_access_resolves_each_distinct_input_once_per_preparation(tmp_path,monkeypatch): + project=tmp_path/'board.prj';project.write_text('fixture') + (tmp_path/'command.tsv').write_text('project_path\n'+(str(project)+'\n')*1000,encoding='utf8') + original=Path.resolve;calls=[] + def resolve(path,*args,**kwargs): + if path==project:calls.append(path) + return original(path,*args,**kwargs) + monkeypatch.setattr(Path,'resolve',resolve) + legacy_paths._prepare_project_access(tmp_path) + assert len(calls)==1 + assert not (tmp_path/'project-access.tsv').exists() + legacy_paths._prepare_project_access(tmp_path) + assert len(calls)==2 # No lifetime cache of filesystem trust. + + +def test_below_threshold_log_rotation_needs_only_one_live_stat(tmp_path,monkeypatch): + log=tmp_path/'events.jsonl';log.write_text('record') + original=Path.stat;calls=[] + def stat(path,*args,**kwargs): + if path==log:calls.append(path) + return original(path,*args,**kwargs) + monkeypatch.setattr(Path,'stat',stat) + assert not retention.rotate_text_log(log) + assert len(calls)==1 + + +def test_existing_log_directories_avoid_mkdir_and_recreate_after_removal(tmp_path,monkeypatch): + context=SimpleNamespace(root=tmp_path) + directory=tmp_path/'tasks/task';directory.mkdir(parents=True) + original=Path.mkdir;calls=[] + def mkdir(path,*args,**kwargs): + if path==directory:calls.append(path) + return original(path,*args,**kwargs) + monkeypatch.setattr(Path,'mkdir',mkdir) + assert logging_setup._task_directory(context,'task')==directory + log=directory/'worker.log' + quick_run._append_log(log,'first') + assert calls==[] + log.unlink();directory.rmdir() + quick_run._append_log(log,'second') + assert calls==[directory] + assert log.read_text()=='second\n' + + +@pytest.mark.parametrize('error',[FileNotFoundError(errno.ENOENT,'missing'), + NotADirectoryError(errno.ENOTDIR,'not a directory'),OSError(errno.EBADF,'bad handle'), + ValueError('embedded null character')]) +def test_log_rotation_keeps_path_exists_absence_semantics(tmp_path,monkeypatch,error): + log=tmp_path/'missing.jsonl' + original=Path.stat + def stat(path,*args,**kwargs): + if path==log:raise error + return original(path,*args,**kwargs) + monkeypatch.setattr(Path,'stat',stat) + assert retention.rotate_text_log(log) is False + + +def test_log_rotation_still_reports_permission_errors(tmp_path,monkeypatch): + log=tmp_path/'denied.jsonl' + original=Path.stat + def stat(path,*args,**kwargs): + if path==log:raise PermissionError(errno.EACCES,'denied') + return original(path,*args,**kwargs) + monkeypatch.setattr(Path,'stat',stat) + with pytest.raises(PermissionError):retention.rotate_text_log(log)