diff --git a/README.md b/README.md index 4ebc830..b9a4842 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,10 @@ 本目录是独立的 CBB 中心库上传后校验工具,不属于 Part Replacement,也不负责 CBB 工程复制或中心库上传。 +针对 `The last server session ... lost the connection to the database`,修复分支增加 +`ICDB_REPAIR_REQUIRED` 专项识别、共享符号库路径诊断及停止批次处理。现场原生修复步骤及实现边界见 +[iCDB 旧服务器会话失联分析](docs/diagnostics/2026-10-27-icdb-lost-server-session.md)。该分支不会自动修复受损数据库。 + ## 功能边界 | 工具 | 处理对象 | 核心动作 | 结果 | @@ -135,7 +139,7 @@ python -m cbb_refresh_verify - **不注入鼠标或键盘输入,不切换 Windows 桌面,不改写中心库源文件。** 生产路径在本次运行创建的私有桌面内启动中心库目录绑定的 ReusableCircuit PCB,随后用 COM 完成刷新、保存和关闭;只读状态桥在结束后确认 LMC 的 READY。Library Manager 菜单派发仅作为显式诊断路径保留。`tests/test_safety.py` 会在离线测试中阻止真实输入注入 API 重新进入生产代码。 - 所有自动化都跑在本进程创建的**私有 Windows 桌面**上:用户看不到窗口,光标不会被移动。 -- 弹窗只认领本次运行自己启动的进程,并在该进程的私有桌面内投递 `BM_CLICK`。数据库异常恢复提示默认选择“最后一次用户保存的数据库”;其他弹窗优先确认/继续/恢复,只有 Project Integration 进度窗(仅有 Cancel)保持不动。所有弹窗文本和实际控件 ID 都写入 `dialogs.jsonl`。 +- 弹窗只认领本次运行的私有桌面/进程,依据已知规则投递 `BM_CLICK`。数据库恢复、会话失联和设计锁接管提示会阻断并保留现场;不会自动选择恢复或接管。未知弹窗也会阻断,Project Integration 进度窗保持观察。动作/错误弹窗和实际控件 ID 写入 `dialogs.jsonl`。 - 超时或失败时停止自动化客户端,保留可能仍在写库的原生会话并停止批次。监督器核对进程是否退出,记录 `process_exit_confirmed` 或 `process_retained`;无法确认释放时输出 `recovery-required.json`,不以强杀代替数据库安全关闭。 ## 环境要求 @@ -163,7 +167,7 @@ Windows 版本判断以 Build 为准:`22000` 及以上归为 Win11,`10240– ## 批量策略 -每个 CBB 独立执行并写入自己的 evidence 和结果行。缺少 PCB/PRJ、内容检查失败、刷新失败或超时只影响当前项,记录 `BLOCKED/FAIL/TIMEOUT` 后继续下一项;不会因为单项失败终止整批。只有点击“当前项完成后停止”(或执行器返回 `QUEUE_STOP`)才把后续项目标为 `SKIPPED`。 +每个 CBB 独立执行并写入自己的 evidence 和结果行。可确认安全的单项失败允许继续;遇到 `ICDB_REPAIR_REQUIRED`、超时、原生会话释放未确认等情况,会停止整批并把后续项目标为 `SKIPPED`。点击“当前项完成后停止”或执行器返回 `QUEUE_STOP` 也会停止队列。 GUI 中状态为 `绿色 / Ready` 的条目不能勾选、也不会进入批量队列。 @@ -196,7 +200,7 @@ Mentor 桥耗时需在目标机器另行测量。 - `save.log`:`LICENSE_OK / SAVE_OK / IS_SAVED / POST_SAVE_SYNCHRONIZED / CLOSE_OK / QUIT_OK`; - `result.json`:状态、中心库标志读取结果、阻断原因; - `image-status-map.json`:按中心库路径隔离的本机动态 `image_id` 校准快照(当前格式 `schema_version=2`);旧版本或无版本缓存会自动忽略并重新校准,日志会记录当前样本、缓存样本、Ready 主 ID、未映射 ID 和逐项决策原因; -- `backup/`:仅本地盘操作前的稳定文件副本;网络盘/UNC 跳过前置递归备份(`backup_skipped`);`database/LogFiles/tmp` 等易变目录和日志文件会排除,备份超时或失败不会阻断 PCB 校验。每个 CBB 结束后默认自动删除该目录,只保留 `events.jsonl`、`dialogs.jsonl`、`save.log` 和 `result.json` 等轻量证据。 +- `backup/`:仅本地盘操作前的稳定文件副本;网络盘/UNC 跳过前置递归备份(`backup_skipped`);`database/LogFiles/tmp` 等易变目录和日志文件会排除,备份超时或失败不会阻断 PCB 校验。仅在成功且确认安全后清理该目录,失败时保留已有副本及日志;它不是完整数据库回滚备份。 超时采用无进展计时:界面和批量请求默认 600 秒无进展超时;它不是整个 CBB 的固定总时限,而是连续 600 秒没有可观察进展时才终止。PCB/COM 阶段切换、Project Integration 状态变化、保存标记变化和弹窗处理都会刷新计时器;重复的状态轮询不算进展。需要更快失败时仍可在界面将单项预算调低,但 30 秒只适合已确认 PCB 启动稳定的环境。 diff --git a/docs/diagnostics/2026-10-27-icdb-lost-server-session.md b/docs/diagnostics/2026-10-27-icdb-lost-server-session.md new file mode 100644 index 0000000..3f7b29d --- /dev/null +++ b/docs/diagnostics/2026-10-27-icdb-lost-server-session.md @@ -0,0 +1,103 @@ +# iCDB 旧服务器会话失联:分析、恢复步骤与分支改动 + +分支:`codex/icdb-lost-session-diagnostics`,基于 `a11eb3c`(`v0.2.2-beta.1`)。 + +## 错误的含义和影响范围 + +现场消息为: + +```text +The last server session on different server machine lost the connection +to the database. Please use iCDB Project Backup 'Repair project' +functionality to fix this issue. +Machine: [200091-CBB] +Session: [2026-10-24 08.22.29-10308] +``` + +原生 iCDB 检测到先前服务器会话的数据库状态需要检查/修复。这不是 +“当前网络连接已经恢复就可以继续”的提示,也不能仅凭它认定端口被 Windows 保留。 +`Machine` 和 `Session` 是故障记录中的原服务器、会话标识,不能当作当前进程 PID。 + +用户贴出的路径包含 `5:\5ymbolLibsReusableBlockssymb`,存在文字识别/转录歧义。 +本机隔离样本确实存在 `SymbolLibs\__ReusableBlocksSymb\icdb.dat`:如果现场原文 +指向这个结构,问题影响的是中心库共享符号数据库,而非单个 CBB 的 +`ReusableBlocksLP\\Work\database`。必须以现场原始日志为准,不能自动把 +`5:` 改成 `S:`,或根据 CBB 名称推导修复目标。 + +异常关闭、原服务器/网络存储断开、写库期间复制工程、跨机器访问不一致的 +路径,都可能涉及此类状态。现有消息不能区分这些原因,也不能证明 +“校验失败一定导致损坏”;数据库异常本身同样会导致校验失败。共享符号库 +出问题时,不同失败 CBB 可能只是访问了同一个有问题的数据库。 + +## 现场恢复步骤 + +1. 停止批量校验,核对 `200091-CBB` 和其他访问同一中心库的客户端。 + 正常关闭占用该目标的 Library Manager、PCB、原理图/符号编辑会话,并确认 + 原数据库服务器的实际状态;只看本机没有 PCB 或 TCP 探测失败不足以确认独占。 +2. 在会话正常关闭后保留完整、一致的工程/中心库副本。共享库问题需包括 + `.lmc`、相关 `SymbolLibs` 数据库、原有 `ProjectBackup`/数据库备份和日志。 + 工具日常的 `backup/` 排除了 `database` 等目录,网络操作还可能跳过备份, + 因而不能把它当作这次修复所需的完整回滚副本。 +3. 使用现场相同 Mentor 版本的 **iCDB Project Backup**,先核对打开的工程或 + 备份存储确实关联报错数据库,再执行 **Project → Repair**(错误提示称为 + `Repair project`)。如果目标是共享符号库,而工具无法识别对应库/存储, + 应由库管理员按该版本支持的库恢复流程处理,不能改选无关的 Work PRJ。 +4. 保留原生工具的修复输出及 `Project Repair` 备份,确认修复结果。 + 不通过删除 `sAddress.adr`、锁文件、`icdb.dat`,也不通过强杀数据库服务器 + 来代替 Repair。如果原生修复失败,再依据原生日志选择一致备份恢复。 +5. 正常重新打开原先失败的目标,先复测一个 CBB,检查本轮原生日志、 + Project Integration、Save/Close 和进程释放,再以 `IsVerifiedReuseBlock` + 返回 `READY/raw=1` 确认校验成功。之后再扩大批次。 + +本机已确认原生入口: + +```text +%SDD_HOME%\iCDB\win32\bin\iCDBProjectBackup.exe +``` + +只读检查其命令行帮助显示 `-p `、`-ver`、`-h`;菜单资源确认存在 +`Project → Repair`。没有查到公开的直接 Repair 命令行开关,因此本分支没有 +编造 `-repair` 参数,也没有自动投递写库修复命令。曾在本机既有隔离样本上 +打开原生管理界面进行入口检查,未执行 Repair;本机没有用户本次故障库, +未在 `200091-CBB` 或现场网络库上执行恢复。 + +## 代码修复 + +- 在内容检查输出、实时弹窗和本轮 `PCBLogFile.txt`、`ForwardAnnotation.txt`、 + `MsgBoxResponse.log`、`PartPkg.log` 中识别完整的服务器会话失联消息。 +- 分类为 `ICDB_REPAIR_REQUIRED`,保留原始错误和 `Machine/Path/Session`, + 区分共享符号库、Work 工程数据库及不能确定的目标;写入 + `diagnostics/icdb-recovery.json` 并在失败清单显示恢复说明。 +- 该错误优先于通用打开失败、JobPrefs 错误及清理失败;即使内容检查同时 + 输出成功文字和退出码 0,也不能变成 PASS。标记 `safe_to_continue=False`, + 后续条目标记 SKIPPED,沿用现有会话、证据、网络映射保留机制。 +- 将当前日志与启动前检查点比较,支持追加、截断、等长/增长重写、UTF-16 + 及 UTF-8;恢复识别每次最多读取两个 128 KiB 片段。不能确认日志新旧时 + 保留原始失败,不据旧记录推荐本轮修复。重写后保留的同一历史会话不作为 + 新证据;可验证的追加则允许识别同一会话再次报错。 +- 通用 `Could not run server` 不再直接推断端口保留并提示重启 WinNAT。 + 启动前独立端口绑定探测保留原逻辑。 + +## 验证和边界 + +原分支基线:`python -m pytest -q`,383 项通过。 +新增回归包括原错误、共享库身份、内容检查停止队列、多行集成日志、 +日志重写及历史错误隔离、编码、证据写入失败和监督器失败摘要。 + +最终验证: + +- 全量 Python 测试 **417 项通过**,含 34 项新增回归;仍有基线已有的 + 一条 pywinauto STA COM 模式提示。 +- 两组 PowerShell 测试、`python -m compileall -q src tests`、 + `git diff --check` 通过。 +- 只读解析仓库留存的真实 TUNER_R836_A 现场日志,正确提取到机器 `EDA03`、 + `S:\ReusableBlocksLP\TUNER_R836_A\Work\database` 及会话 + `2025-01-21 14.44.46-2323356`。该记录仅用于解析验证,不代表本次现场故障。 +- 独立代码复查发现的集成日志遗漏、通用端口建议覆盖具体原因、历史日志 + 重写误判、不可读/超出窗口的基线问题,均补充回归并通过复核。 + +本分支修复自动化的错误识别、误导性建议和失败后的继续执行风险, +**没有修复现场受损数据库,也没有验证原生 Repair 的现场效果**。 +恢复识别仅覆盖已知日志末尾的有限窗口;错误出现在窗口外、相同历史会话 +在重写时被原样重报、超出基线窗口的大日志发生重写、以及现场不同日志位置/格式, +仍需结合完整原生日志确认。这些不能确认新旧的情况不会自动提升为本轮修复诊断。 diff --git a/src/cbb_refresh_verify/dialog_rules.py b/src/cbb_refresh_verify/dialog_rules.py index eb9feae..204bfdb 100644 --- a/src/cbb_refresh_verify/dialog_rules.py +++ b/src/cbb_refresh_verify/dialog_rules.py @@ -193,6 +193,10 @@ def decide(snapshot: WindowSnapshot, rules: Sequence[DialogRule]) -> RuleDecisio and not snapshot.controls): return RuleDecision(rule_id='ime-floating-language-bar', category='OBSERVE_PROGRESS') combined = normalize_text(' '.join((snapshot.title, *snapshot.texts))) + from .icdb_recovery import ERROR_CODE, parse_session_loss + if parse_session_loss(combined): + return RuleDecision(error_code=ERROR_CODE, + detail=' | '.join(snapshot.texts) or snapshot.title) # These prompts require knowledge of database ownership or a recovery # choice. A generic Yes/restore rule cannot establish either fact. if any(marker in combined for marker in ( diff --git a/src/cbb_refresh_verify/dialog_supervisor.py b/src/cbb_refresh_verify/dialog_supervisor.py index 31ad5a3..1915f0d 100644 --- a/src/cbb_refresh_verify/dialog_supervisor.py +++ b/src/cbb_refresh_verify/dialog_supervisor.py @@ -351,6 +351,11 @@ def classify_dialog(snapshot: DialogSnapshot) -> DialogDecision: joined = " ".join(snapshot.texts).casefold() combined = f"{title} {joined}" + from .icdb_recovery import ERROR_CODE, parse_session_loss + if parse_session_loss(combined): + return DialogDecision(error_code=ERROR_CODE, + detail=' | '.join(snapshot.texts) or snapshot.title) + if ("problem encountered while opening one of the database files" in combined and "loading of the job has failed" in combined): return DialogDecision(action="decline_database_log", diff --git a/src/cbb_refresh_verify/icdb_recovery.py b/src/cbb_refresh_verify/icdb_recovery.py new file mode 100644 index 0000000..a9387bb --- /dev/null +++ b/src/cbb_refresh_verify/icdb_recovery.py @@ -0,0 +1,144 @@ +"""Recognize native iCDB recovery requests without changing database state.""" +import hashlib +import ntpath +from pathlib import Path +import re + + +ERROR_CODE = 'ICDB_REPAIR_REQUIRED' +LOG_LIMIT = 128 * 1024 +RECOVERY_ACTION = ( + '停止批次并确认原服务器及其他客户端已正常关闭;保留完整一致副本后,' + "使用同版本 iCDB Project Backup 的 Repair project 检查并修复正确目标。" + '共享符号库问题需核对中心库,不能只修复单个 CBB 的 Work 工程。' + '不要删除 sAddress.adr/锁文件;修复后重新打开、校验并确认权威 READY。' +) +_SESSION_LOSS = re.compile( + r'\bthe\s+last\s+server\s+session\s+on\s+different\s+server\s+machine\s+' + r'lost\s+the\s+connection\s+to\s+the\s+database\b', re.IGNORECASE) + + +def parse_session_loss(text): + """Keep the native target verbatim; never infer a local repair target.""" + text = str(text or '') + match = _SESSION_LOSS.search(text) + if match is None: + return None + # A log may contain several sessions. Use the identity following this + # error, never an unrelated Machine/Path header earlier in the log. + message = text[match.start():] + next_error = _SESSION_LOSS.search(message, 1) + if next_error: + message = message[:next_error.start()] + record = {'error_code': ERROR_CODE, 'native_message': message, + 'machine': '', 'database_path': '', 'session': '', 'scope': 'unknown'} + for native_key, key in (('machine', 'machine'), ('path', 'database_path'), + ('session', 'session')): + field = re.search( + rf'\b{native_key}\s*:\s*(?:\[([^\]]*)\]|([^\r\n]*?)(?=\s+\b(?:machine|path|session)\s*:|[\r\n]|$))', + message, re.IGNORECASE) + if field: + record[key] = (field.group(1) if field.group(1) is not None else field.group(2)).strip() + parts = ntpath.normpath(record['database_path']).casefold().split('\\') + if 'symbollibs' in parts: + record['scope'] = 'central_library' + elif 'work' in parts and parts[-1] == 'database': + record['scope'] = 'project' + return record + + +def recovery_detail(record): + target = ('中心库共享符号数据库' if record['scope'] == 'central_library' else + '工程数据库' if record['scope'] == 'project' else '报错中指定的数据库') + return ( + f"{ERROR_CODE}: {target}的上次服务器会话失联。\n" + f"{record['native_message']}\n" + f'{RECOVERY_ACTION}' + ) + + +def snapshot_log(path): + """Fingerprint a bounded tail before launch, including a missing log.""" + try: + with Path(path).open('rb') as stream: + stream.seek(0, 2) + size = stream.tell() + stream.seek(max(0, size-LOG_LIMIT)) + tail = stream.read(LOG_LIMIT) + losses = [_session_identity(record) for record in _session_losses(decode_log(tail))] + return {'size': size, 'tail_sha256': hashlib.sha256(tail).hexdigest(), + 'truncated': size > LOG_LIMIT, + 'session_losses': losses} + except FileNotFoundError: + return {'size': 0, 'tail_sha256': hashlib.sha256(b'').hexdigest()} + except OSError as error: + return {'error': str(error)} + + +def _read_log_delta(path, baseline): + """Read new bytes, detecting truncate/rewrite as well as append. + + Unknown baselines cannot prove freshness. At most two bounded tails are + read, even when a native log is large or located on a network share. + """ + if not baseline or 'error' in baseline: + return b'', False + old_size = int(baseline['size']) + try: + with Path(path).open('rb') as stream: + stream.seek(0, 2) + size = stream.tell() + offset = 0 + appended = False + if size >= old_size: + stream.seek(max(0, old_size-LOG_LIMIT)) + old_tail = stream.read(min(old_size, LOG_LIMIT)) + if hashlib.sha256(old_tail).hexdigest() == baseline['tail_sha256']: + offset = old_size + appended = True + stream.seek(max(offset, size-LOG_LIMIT)) + return stream.read(LOG_LIMIT), appended + except OSError: + return b'', False + + +def _session_losses(text): + matches = list(_SESSION_LOSS.finditer(text)) + for index, match in enumerate(matches): + end = matches[index+1].start() if index+1 < len(matches) else len(text) + yield parse_session_loss(text[match.start():end]) + + +def _session_identity(record): + return [record[key] for key in ('machine', 'database_path', 'session')] + + +def read_log_failure(path, baseline): + """Return new log text and a recovery request with evidence of freshness. + + A rewritten header does not prove that a retained old session failed + again. In that ambiguous case keep the generic failure; a verified append + can report even the same old session as a current native error. + """ + data, appended = _read_log_delta(path, baseline) + text = decode_log(data) + if not appended and baseline and baseline.get('truncated'): + # A rewrite may expose an old error outside our bounded baseline. + # Without positive append evidence its age cannot be established. + return text, None + historical = baseline.get('session_losses', []) if baseline else [] + recovery = next((record for record in _session_losses(text) + if appended or _session_identity(record) not in historical), None) + return text, recovery + + +def decode_log(data): + if data.startswith((b'\xff\xfe', b'\xfe\xff')): + return data.decode('utf-16', errors='replace') + # Appended UTF-16 bytes and bounded tails no longer contain a BOM. + sample = data[:200] + if sample[1::2].count(0) > len(sample)//8: + return data.decode('utf-16-le', errors='replace') + if sample[::2].count(0) > len(sample)//8: + return data.decode('utf-16-be', errors='replace') + return data.decode('utf-8', errors='replace') diff --git a/src/cbb_refresh_verify/mentor_adapter.py b/src/cbb_refresh_verify/mentor_adapter.py index d482d7d..1cb3c8a 100644 --- a/src/cbb_refresh_verify/mentor_adapter.py +++ b/src/cbb_refresh_verify/mentor_adapter.py @@ -647,10 +647,8 @@ def execute_job(job, emit): evidence=str(root/'diagnostics/preflight.json')) cert_project = block_root/'Cert'/Path(job['project']).name pcb_log = Path(job['pcb']).parent/'LogFiles'/'PCBLogFile.txt' - try: - pcb_log_offset = pcb_log.stat().st_size - except OSError: - pcb_log_offset = 0 + from .icdb_recovery import snapshot_log + pcb_log_offset = snapshot_log(pcb_log) integration_log_dir = Path(job['pcb']).parent / 'LogFiles' integration_offsets = {} for name in ('ForwardAnnotation.txt', 'MsgBoxResponse.log', 'PartPkg.log'): @@ -664,6 +662,26 @@ def execute_job(job, emit): } except OSError: integration_offsets[path] = {'size': 0, 'mtime_ns': None, 'digest': ''} + integration_offsets[path]['icdb_baseline'] = snapshot_log(path) + try: + (root/'diagnostics/startup-log-baseline.json').write_text(json.dumps({ + 'source': str(pcb_log), 'baseline': pcb_log_offset, + 'integration_logs': {str(path): value['icdb_baseline'] + for path, value in integration_offsets.items()}, + }, ensure_ascii=False), encoding='utf-8') + except OSError as error: + emit(stage='startup_log_baseline_unavailable', detail=str(error)) + + def native_error(): + # Inspect every source before returning a generic server/packager + # failure: either can be a consequence of a lost library session. + startup = pcb_log_reports_missing_server(job['pcb'], offset=pcb_log_offset) + integration = pcb_log_reports_integration_error(job['pcb'], offsets=integration_offsets) + if 'ICDB_REPAIR_REQUIRED' in startup: + return startup + if 'ICDB_REPAIR_REQUIRED' in integration: + return integration + return startup or integration emit(stage='pcb_baseline', pids=sorted(pcb_before)) lm_session = None lm_pid = None @@ -735,7 +753,7 @@ def execute_job(job, emit): process_alive=lambda: any(candidate_pid in {lm_pid, worker_pid, worker_parent_pid} for candidate_pid, _, _ in pcb_processes()), emit=emit, - error_reader=lambda: pcb_log_reports_missing_server(job['pcb'], offset=pcb_log_offset), + error_reader=native_error, ) # Do not acquire MGCPCBAutomationLicensing from this 64-bit Python # worker. EE7.9 registers that automation class for the 32-bit host; @@ -746,7 +764,7 @@ def execute_job(job, emit): pi, emit=emit, timeout_seconds=job['timeout'], - error_reader=lambda: pcb_log_reports_integration_error(job['pcb'], offsets=integration_offsets), + error_reader=native_error, ) # The x86 helper acquires its own COM references. Keeping these Python # references alive prevents the native PCB server from exiting cleanly. @@ -1040,17 +1058,27 @@ def icdb_ports_bindable(ports=(10000, 10050, 10100)): def pcb_log_reports_missing_server(pcb_path, *, offset=0): - """Detect the same failure from ExpeditionPCB's own log, as a second signal.""" + """Detect startup failure from this run, preserving native recovery causes.""" + from .icdb_recovery import (decode_log, parse_session_loss, read_log_failure, + recovery_detail) log = Path(pcb_path).parent/'LogFiles'/'PCBLogFile.txt' try: - with log.open('rb') as stream: - stream.seek(max(0, offset)) - text = stream.read().decode('utf-8', errors='replace') + if isinstance(offset, dict): + text, recovery = read_log_failure(log, offset) + else: + # Retain the byte-offset interface for existing diagnostic callers. + with log.open('rb') as stream: + stream.seek(max(0, offset)) + text = decode_log(stream.read()) + recovery = parse_session_loss(text) except Exception: return '' + if recovery: + return recovery_detail(recovery) for marker in ("could not bring it alive", 'Could not run server'): if marker.lower() in text.lower(): - return RESERVED_PORT_MESSAGE + return (f'DESIGN_SERVER_FAILED: {marker}。请检查本轮原生日志中的数据库/服务器原因;' + '此信息本身不能证明端口被系统保留。') return '' @@ -1065,6 +1093,7 @@ _INTEGRATION_ERROR_LINE = re.compile( def pcb_log_reports_integration_error(pcb_path, *, offsets=None): """Read explicit Project Integration errors produced by this run.""" + from .icdb_recovery import parse_session_loss, read_log_failure, recovery_detail log_dir = Path(pcb_path).parent / 'LogFiles' offsets = offsets or {} details = [] @@ -1073,6 +1102,19 @@ def pcb_log_reports_integration_error(pcb_path, *, offsets=None): try: size = path.stat().st_size baseline = offsets.get(path, offsets.get(str(path), 0)) + has_icdb_baseline = isinstance(baseline, dict) and any( + key in baseline for key in ('icdb_baseline', 'tail_sha256', 'error')) + icdb_baseline = baseline.get('icdb_baseline', baseline) if isinstance(baseline, dict) else None + if icdb_baseline and 'tail_sha256' in icdb_baseline: + text, recovery = read_log_failure(path, icdb_baseline) + if recovery: + return recovery_detail(recovery) + if 'tail_sha256' in baseline: + # New bounded checkpoints can also be supplied directly. + details.extend(line.strip() for line in text.splitlines() + if _INTEGRATION_ERROR_LINE.search(line.strip()) + and not parse_session_loss(line)) + continue if isinstance(baseline, dict): baseline_size = int(baseline.get('size', 0) or 0) baseline_mtime = baseline.get('mtime_ns') @@ -1103,8 +1145,17 @@ def pcb_log_reports_integration_error(pcb_path, *, offsets=None): text = stream.read().decode('utf-8', errors='replace') except OSError: continue + # Legacy byte offsets/digests remain accepted by offline callers. + if not has_icdb_baseline: + recovery = parse_session_loss(text) + if recovery: + return recovery_detail(recovery) for line in text.splitlines(): cleaned = line.strip() + if has_icdb_baseline and parse_session_loss(cleaned): + # read_log_failure already rejected this retained old session. + # Do not let the generic ERROR filter promote it again. + continue if _INTEGRATION_ERROR_LINE.search(cleaned) and cleaned not in details: details.append(cleaned) return ' | '.join(details[:8]) diff --git a/src/cbb_refresh_verify/verify_diagnostics.py b/src/cbb_refresh_verify/verify_diagnostics.py index caeb03d..c5a671b 100644 --- a/src/cbb_refresh_verify/verify_diagnostics.py +++ b/src/cbb_refresh_verify/verify_diagnostics.py @@ -113,6 +113,8 @@ def collect_native_failure_logs(pcb_path, evidence_root, *, max_bytes=NATIVE_LOG target = destination/(label+'-'+source.name) target.write_bytes(data) item.update(evidence=str(target), copied_bytes=len(data)) + if label == 'pcb-logfiles' and source.name != 'ReusableBlock.txt': + _collect_icdb_recovery(source, evidence_root) except FileNotFoundError: item['exists'] = False except OSError as error: @@ -120,6 +122,23 @@ def collect_native_failure_logs(pcb_path, evidence_root, *, max_bytes=NATIVE_LOG return _write_record(destination.parent/'native-logs.json', record) +def _collect_icdb_recovery(source, evidence_root): + """Only promote a native error whose log changed after this launch.""" + from .icdb_recovery import read_log_failure + root = Path(evidence_root)/'diagnostics' + try: + baseline = json.loads((root/'startup-log-baseline.json').read_text('utf-8')) + checkpoint = (baseline['baseline'] if baseline['source'] == str(source) else + baseline.get('integration_logs', {}).get(str(source))) + if checkpoint is None: + return + _, recovery = read_log_failure(source, checkpoint) + if recovery: + _write_record(root/'icdb-recovery.json', {**recovery, 'source': str(source)}) + except (OSError, ValueError, KeyError, TypeError): + pass # Unknown freshness must not replace the original failure. + + def failure_logs_main(request_path): job = json.loads(Path(request_path).read_text(encoding='utf-8')) diff --git a/src/cbb_refresh_verify/verify_runner.py b/src/cbb_refresh_verify/verify_runner.py index cdf6c15..f102c5c 100644 --- a/src/cbb_refresh_verify/verify_runner.py +++ b/src/cbb_refresh_verify/verify_runner.py @@ -127,6 +127,9 @@ def classify_output(exit_code: int | None, output: str, *, timed_out: bool = Fal """Never invent success: PASS needs the success text and no error marker.""" if timed_out: return 'TIMEOUT' + from .icdb_recovery import parse_session_loss + if parse_session_loss(output): + return 'FAIL' if exit_code == 0 and SUCCESS_TEXT.search(output) and not ERROR_TEXT.search(output): return 'PASS' if ERROR_TEXT.search(output): @@ -277,6 +280,9 @@ def read_library_flag_with_retry(reader, library, block_name, *, attempts=3, def classify_failure_category(detail): text = str(detail or '') lower = text.casefold() + from .icdb_recovery import ERROR_CODE, parse_session_loss + if ERROR_CODE.casefold() in lower or parse_session_loss(text): + return ERROR_CODE if any(code in lower for code in ( 'pcb_release_unconfirmed', 'icdb_release_unconfirmed', 'icdb_session_unconfirmed', 'pcb_session_active', 'database_recovery_required')): @@ -554,6 +560,31 @@ def _run_verify_serialized(request, *, progress=None, flag_reader=None, process_ pass finally: timing.end('failure_diagnostics') + if result.status not in {'PASS', 'SKIPPED', 'QUEUE_STOP'}: + from .icdb_recovery import ERROR_CODE, RECOVERY_ACTION, parse_session_loss, recovery_detail + recovery = parse_session_loss(result.stderr) + if recovery is None and result.evidence_path: + try: + saved = json.loads(result.evidence_path.with_name('diagnostics').joinpath( + 'icdb-recovery.json').read_text('utf-8')) + recovery = parse_session_loss(saved.get('native_message', '')) + except (OSError, ValueError, AttributeError): + pass + if recovery or ERROR_CODE.casefold() in result.stderr.casefold(): + detail = result.stderr + if recovery and RECOVERY_ACTION not in detail: + detail = (detail + '\n' + recovery_detail(recovery)).strip() + result = replace(result, safe_to_continue=False, stderr=detail, + error_category=ERROR_CODE) + if recovery and result.evidence_path: + try: + record_path = result.evidence_path.parent/'diagnostics/icdb-recovery.json' + record_path.parent.mkdir(parents=True, exist_ok=True) + if not record_path.is_file(): + record_path.write_text(json.dumps(recovery, ensure_ascii=False, indent=2), + encoding='utf-8') + except OSError: + pass durations, unfinished = timing.snapshot() result = replace(result, timings_seconds=durations, incomplete_timing_phases=unfinished, error_category=(result.error_category or classify_failure_category(result.stderr) diff --git a/tests/test_icdb_recovery.py b/tests/test_icdb_recovery.py new file mode 100644 index 0000000..6790dfa --- /dev/null +++ b/tests/test_icdb_recovery.py @@ -0,0 +1,310 @@ +import json +from pathlib import Path + +import pytest + +from cbb_refresh_verify import mentor_adapter as adapter +from cbb_refresh_verify import verify_runner as runner + + +ERROR = ( + "The last server session on different server machine lost the connection\n" + "to the database. Please use iCDB Project Backup 'Repair project' functionality " + "to fix this issue. (Machine:[200091-CBB] Path:" + "[S:\\SymbolLibs\\__ReusableBlocksSymb] session:[2026-10-24 08.22.29-10308])" +) + + +def test_native_session_loss_has_priority_over_generic_load_and_cleanup_errors(): + detail = 'DATABASE_LOAD_FAILED: Could not load Job Preferences DB\n' + ERROR + detail += '\nPCB_RELEASE_UNCONFIRMED' + assert runner.classify_failure_category(detail) == 'ICDB_REPAIR_REQUIRED' + + +def test_live_repair_dialog_blocks_without_clicking(): + from test_dialog_rules import prompt, ASSETS + from cbb_refresh_verify.dialog_rules import decide, load_rules + from cbb_refresh_verify.dialog_supervisor import classify_dialog, DialogSnapshot, DialogControl + decision = decide(prompt(ERROR), load_rules(ASSETS/'dialog_rules.v1.json')) + legacy = classify_dialog(DialogSnapshot(1, 2, 'Expedition PCB', ( + DialogControl(3, 1, 'Static', ERROR, True),))) + for item in (decision, legacy): + assert item.error_code == 'ICDB_REPAIR_REQUIRED' + assert not item.action + assert '200091-CBB' in item.detail + + +def test_startup_probe_reports_session_loss_instead_of_port_reset(tmp_path): + log = tmp_path/'LogFiles/PCBLogFile.txt' + log.parent.mkdir() + log.write_text(ERROR + '\nCould not run server', encoding='utf-8') + detail = adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb') + assert 'ICDB_REPAIR_REQUIRED' in detail + assert '__ReusableBlocksSymb' in detail + assert 'Repair project' in detail + assert 'net stop winnat' not in detail + + +def test_generic_server_log_does_not_claim_port_reservation(tmp_path): + log = tmp_path/'LogFiles/PCBLogFile.txt' + log.parent.mkdir() + log.write_text('Could not run server', encoding='utf-8') + detail = adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb') + assert 'DESIGN_SERVER_FAILED' in detail + assert 'WinError 10013' not in detail + assert 'net stop winnat' not in detail + + +@pytest.mark.parametrize('mode', ['append', 'shorter', 'same_size', 'longer']) +def test_current_run_probe_handles_appended_and_rewritten_logs(tmp_path, mode): + from cbb_refresh_verify.icdb_recovery import snapshot_log + log = tmp_path/'LogFiles/PCBLogFile.txt' + log.parent.mkdir() + old = 'old harmless text\n' if mode == 'longer' else 'x'*(len(ERROR.encode())+40) + if mode == 'same_size': + old = 'x'*len(ERROR.encode()) + log.write_text(old, encoding='utf-8') + baseline = snapshot_log(log) + log.write_text(old+ERROR if mode == 'append' else ERROR, encoding='utf-8') + assert 'ICDB_REPAIR_REQUIRED' in adapter.pcb_log_reports_missing_server( + tmp_path/'CBB.pcb', offset=baseline) + + +def test_old_loss_record_does_not_fail_a_new_run(tmp_path): + from cbb_refresh_verify.icdb_recovery import snapshot_log + log = tmp_path/'LogFiles/PCBLogFile.txt' + log.parent.mkdir() + log.write_text(ERROR, encoding='utf-8') + baseline = snapshot_log(log) + assert adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb', offset=baseline) == '' + with log.open('a', encoding='utf-8') as stream: + stream.write('\nNormal startup completed\n') + assert adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb', offset=baseline) == '' + + +def test_rewritten_header_does_not_promote_retained_old_session_loss(tmp_path): + from cbb_refresh_verify.icdb_recovery import snapshot_log + log = tmp_path/'LogFiles/PCBLogFile.txt' + log.parent.mkdir() + log.write_text('old header\n'+ERROR, encoding='utf-8') + baseline = snapshot_log(log) + log.write_text('new header\n'+ERROR+'\nNormal startup\n', encoding='utf-8') + assert adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb', offset=baseline) == '' + + +def test_repeated_old_session_loss_is_current_when_verifiably_appended(tmp_path): + from cbb_refresh_verify.icdb_recovery import snapshot_log + log = tmp_path/'LogFiles/PCBLogFile.txt' + log.parent.mkdir() + log.write_text(ERROR+'\n', encoding='utf-8') + baseline = snapshot_log(log) + with log.open('a', encoding='utf-8') as stream: + stream.write(ERROR) + assert 'ICDB_REPAIR_REQUIRED' in adapter.pcb_log_reports_missing_server( + tmp_path/'CBB.pcb', offset=baseline) + + +@pytest.mark.parametrize('name', ['MsgBoxResponse.log', 'ForwardAnnotation.txt', 'PartPkg.log']) +def test_integration_log_retains_multiline_loss_identity(tmp_path, name): + from cbb_refresh_verify.icdb_recovery import snapshot_log + log = tmp_path/'LogFiles'/name + log.parent.mkdir() + baseline = snapshot_log(log) + log.write_text(ERROR.replace(' (Machine:', '\nMachine:').replace(' Path:', '\nPath:') + .replace(' session:', '\nSession:'), encoding='utf-8') + detail = adapter.pcb_log_reports_integration_error(tmp_path/'CBB.pcb', offsets={log: baseline}) + assert 'ICDB_REPAIR_REQUIRED' in detail + assert '200091-CBB' in detail + assert '__ReusableBlocksSymb' in detail + assert '2026-10-24 08.22.29-10308' in detail + + +@pytest.mark.parametrize('production_checkpoint', [False, True]) +def test_integration_generic_filter_does_not_repromote_historical_repair(tmp_path, production_checkpoint): + import hashlib + from cbb_refresh_verify.icdb_recovery import snapshot_log + log = tmp_path/'LogFiles/MsgBoxResponse.log' + log.parent.mkdir() + old = 'old header\nERROR: '+ERROR.replace('\n', ' ') + log.write_text(old, encoding='utf-8') + baseline = snapshot_log(log) + if production_checkpoint: + raw = log.read_bytes() + baseline = {'size': len(raw), 'digest': hashlib.sha256(raw).hexdigest(), + 'icdb_baseline': baseline} + log.write_text(old.replace('old header', 'new header')+'\nnormal startup', encoding='utf-8') + detail = adapter.pcb_log_reports_integration_error(tmp_path/'CBB.pcb', offsets={log: baseline}) + assert 'ICDB_REPAIR_REQUIRED' not in detail + assert runner.classify_failure_category(detail) != 'ICDB_REPAIR_REQUIRED' + + +def test_unreadable_integration_baseline_never_falls_back_to_historical_log(tmp_path): + log = tmp_path/'LogFiles/MsgBoxResponse.log' + log.parent.mkdir() + log.write_text('ERROR: '+ERROR.replace('\n', ' '), encoding='utf-8') + baseline = {'size': 0, 'mtime_ns': None, 'digest': '', + 'icdb_baseline': {'error': 'access denied'}} + detail = adapter.pcb_log_reports_integration_error(tmp_path/'CBB.pcb', offsets={log: baseline}) + assert runner.classify_failure_category(detail) != 'ICDB_REPAIR_REQUIRED' + + +def test_rewrite_of_large_log_does_not_promote_error_outside_baseline_window(tmp_path): + from cbb_refresh_verify.icdb_recovery import LOG_LIMIT, snapshot_log + log = tmp_path/'LogFiles/PCBLogFile.txt' + log.parent.mkdir() + log.write_text(ERROR+'\n'+'normal\n'*LOG_LIMIT, encoding='utf-8') + baseline = snapshot_log(log) + log.write_text('changed header\n'+ERROR, encoding='utf-8') + assert adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb', offset=baseline) == '' + + +def test_integration_session_loss_has_priority_over_generic_startup_error(tmp_path, monkeypatch): + from test_verify_worker_stability import exercise_worker + from cbb_refresh_verify.pcb_bridge import PcbBridgeError + job, events, execute = exercise_worker(tmp_path, monkeypatch) + def attach(job, provider, **kwargs): + logs = Path(job['pcb']).parent/'LogFiles' + logs.mkdir() + (logs/'PCBLogFile.txt').write_text('Could not run server', encoding='utf-8') + (logs/'MsgBoxResponse.log').write_text(ERROR, encoding='utf-8') + detail = kwargs['error_reader']() + assert 'ICDB_REPAIR_REQUIRED' in detail + assert 'net stop winnat' not in detail + raise PcbBridgeError('PCB_STARTUP_FAILED', detail) + monkeypatch.setattr('cbb_refresh_verify.pcb_bridge.wait_for_target_document', attach) + with pytest.raises(PcbBridgeError, match='ICDB_REPAIR_REQUIRED'): + execute() + assert not any(event['stage'] == 'save_start' for event in events) + + +def test_optional_baseline_evidence_write_failure_does_not_block_verify(tmp_path, monkeypatch): + from test_verify_worker_stability import exercise_worker + job, events, execute = exercise_worker(tmp_path, monkeypatch) + original = Path.write_text + def deny_baseline(path, *args, **kwargs): + if path.name == 'startup-log-baseline.json': + raise PermissionError('diagnostic evidence unavailable') + return original(path, *args, **kwargs) + monkeypatch.setattr(Path, 'write_text', deny_baseline) + assert execute()['status'] == 'PASS' + + +def test_recovery_record_preserves_identity_and_distinguishes_shared_library(): + from cbb_refresh_verify.icdb_recovery import parse_session_loss + record = parse_session_loss(ERROR) + assert record['machine'] == '200091-CBB' + assert record['database_path'] == r'S:\SymbolLibs\__ReusableBlocksSymb' + assert record['session'] == '2026-10-24 08.22.29-10308' + assert record['scope'] == 'central_library' + assert parse_session_loss(ERROR.replace(r'S:\SymbolLibs\__ReusableBlocksSymb', + r'S:\ReusableBlocksLP\CBB\Work\database'))['scope'] == 'project' + assert parse_session_loss('Could not run server') is None + + +@pytest.mark.parametrize('encoding', ['utf-8', 'utf-16', 'utf-16-le']) +def test_new_utf16_or_utf8_log_errors_are_detected_after_append(tmp_path, encoding): + from cbb_refresh_verify.icdb_recovery import snapshot_log + log = tmp_path/'LogFiles/PCBLogFile.txt' + log.parent.mkdir() + log.write_bytes('old text\n'.encode(encoding)) + baseline = snapshot_log(log) + with log.open('ab') as stream: + stream.write(ERROR.encode('utf-16-le' if encoding == 'utf-16' else encoding)) + assert 'ICDB_REPAIR_REQUIRED' in adapter.pcb_log_reports_missing_server( + tmp_path/'CBB.pcb', offset=baseline) + + +@pytest.mark.parametrize('baseline_state', ['stale', 'absent', 'unreadable']) +def test_historical_log_evidence_never_creates_a_repair_recommendation(tmp_path, baseline_state): + from cbb_refresh_verify.icdb_recovery import snapshot_log + from cbb_refresh_verify.verify_diagnostics import collect_native_failure_logs + pcb = tmp_path/'Work/PCB/CBB.pcb' + log = pcb.parent/'LogFiles/PCBLogFile.txt' + log.parent.mkdir(parents=True) + log.write_text(ERROR, encoding='utf-8') + evidence = tmp_path/'evidence' + (evidence/'diagnostics').mkdir(parents=True) + if baseline_state != 'absent': + baseline = snapshot_log(log) if baseline_state == 'stale' else {'error': 'access denied'} + (evidence/'diagnostics/startup-log-baseline.json').write_text(json.dumps({ + 'source': str(log), 'baseline': baseline}), encoding='utf-8') + collect_native_failure_logs(pcb, evidence) + assert not (evidence/'diagnostics/icdb-recovery.json').exists() + + +def test_content_success_text_cannot_override_native_session_loss(): + assert runner.classify_output(0, 'content checking completed successfully\n'+ERROR) == 'FAIL' + + +@pytest.mark.parametrize('name', ['PCBLogFile.txt', 'MsgBoxResponse.log', 'ForwardAnnotation.txt', 'PartPkg.log']) +def test_native_log_capture_generates_only_fresh_repair_evidence(tmp_path, name): + from cbb_refresh_verify.icdb_recovery import snapshot_log + from cbb_refresh_verify.verify_diagnostics import collect_native_failure_logs + pcb = tmp_path/'Work/PCB/CBB.pcb' + log = pcb.parent/'LogFiles'/name + log.parent.mkdir(parents=True) + evidence = tmp_path/'evidence' + (evidence/'diagnostics').mkdir(parents=True) + log.write_text('old text', encoding='utf-8') + (evidence/'diagnostics/startup-log-baseline.json').write_text(json.dumps({ + 'source': str(pcb.parent/'LogFiles/PCBLogFile.txt'), + 'baseline': snapshot_log(pcb.parent/'LogFiles/PCBLogFile.txt'), + 'integration_logs': {str(log): snapshot_log(log)}}), encoding='utf-8') + log.write_text(ERROR, encoding='utf-8') + collect_native_failure_logs(pcb, evidence) + report = json.loads((evidence/'diagnostics/icdb-recovery.json').read_text('utf-8')) + assert report['scope'] == 'central_library' + assert report['source'] == str(log) + assert log.read_text('utf-8') == ERROR + + +def test_contents_session_loss_stops_batch_before_any_pcb_launch(tmp_path, monkeypatch): + from test_verify_runner import fixture, request + fixture(tmp_path) + monkeypatch.setattr(runner, 'resolve_importer', lambda *_: Path('ReuseBlockImporter.exe')) + calls = [] + def content_check(*args): + calls.append(args) + return 1, ERROR, '', False + results = runner.run_batch([request(tmp_path, libspec='EE', dms_login_config='login')]*2, + verifier=lambda req: runner.run_verify(req, process_runner=content_check, + port_check=lambda: pytest.fail('PCB must not start'))) + assert len(calls) == 1 + assert results[0].error_category == 'ICDB_REPAIR_REQUIRED' + assert results[0].safe_to_continue is False + assert 'Repair project' in results[0].stderr + assert results[1].status == 'SKIPPED' + + +def test_supervisor_enriches_generic_failure_from_fresh_native_log(tmp_path, monkeypatch): + from test_verify_runner import fixture, request + from cbb_refresh_verify.icdb_recovery import snapshot_log + from cbb_refresh_verify.verify_diagnostics import failure_logs_main + work = fixture(tmp_path) + log = work/'PCB/LogFiles/PCBLogFile.txt' + log.parent.mkdir() + class Worker: + desktop = None + returncode = 1 + def __init__(self, command, *_a, **_kw): + root = Path(command[-1]).parent + (root/'diagnostics').mkdir() + (root/'diagnostics/startup-log-baseline.json').write_text(json.dumps({ + 'source': str(log), 'baseline': snapshot_log(log)}), encoding='utf-8') + log.write_text(ERROR, encoding='utf-8') + (root/'outcome.json').write_text(json.dumps({ + 'status': 'FAIL', 'stderr': 'MENTOR_DATABASE_LOAD_FAILED: opening failed', + 'safe_to_continue': False}), encoding='utf-8') + def poll(self): return 1 + def close(self): pass + monkeypatch.setattr('cbb_refresh_verify.desktop_worker.DesktopWorker', Worker) + monkeypatch.setattr(runner, '_cleanup_session_processes', lambda *a, **k: {'safe_to_continue': True}) + monkeypatch.setattr('cbb_refresh_verify.verify_diagnostics.capture_failure_logs_with_timeout', failure_logs_main) + result = runner.run_verify(request(tmp_path), port_check=lambda: (True, '')) + assert result.error_category == 'ICDB_REPAIR_REQUIRED' + assert result.safe_to_continue is False + assert 'opening failed' in result.stderr + assert '200091-CBB' in result.stderr + assert '__ReusableBlocksSymb' in result.stderr + assert '2026-10-24 08.22.29-10308' in result.stderr + assert json.loads(result.evidence_path.read_text('utf-8'))['error_category'] == 'ICDB_REPAIR_REQUIRED'