fix: diagnose iCDB lost sessions and stop unsafe verification

This commit is contained in:
阿白叔 committed 2026-10-27 16:38:51 +08:00
1 parent a11eb3c766
commit 8a517d6f38
9 files changed
+685 -14

No files matched your search

+7 -3
View File
@@ -4,6 +4,10 @@
本目录是独立的 CBB 中心库上传后校验工具,不属于 Part Replacement,也不负责 CBB 工程复制或中心库上传。
针对 `The last server session ... lost the connection to the database`,修复分支增加
`ICDB_REPAIR_REQUIRED` 专项识别、共享符号库路径诊断及停止批次处理。现场原生修复步骤及实现边界见
[iCDB 旧服务器会话失联分析](docs/diagnostics/2026-10-27-icdb-lost-server-session.md)。该分支不会自动修复受损数据库。
## 功能边界
| 工具 | 处理对象 | 核心动作 | 结果 |
@@ -135,7 +139,7 @@ python -m cbb_refresh_verify
- **不注入鼠标或键盘输入,不切换 Windows 桌面,不改写中心库源文件。** 生产路径在本次运行创建的私有桌面内启动中心库目录绑定的 ReusableCircuit PCB,随后用 COM 完成刷新、保存和关闭;只读状态桥在结束后确认 LMC 的 READY。Library Manager 菜单派发仅作为显式诊断路径保留。`tests/test_safety.py` 会在离线测试中阻止真实输入注入 API 重新进入生产代码。
- 所有自动化都跑在本进程创建的**私有 Windows 桌面**上:用户看不到窗口,光标不会被移动。
- 弹窗只认领本次运行自己启动的进程,并在该进程的私有桌面内投递 `BM_CLICK`。数据库异常恢复提示默认选择“最后一次用户保存的数据库”;其他弹窗优先确认/继续/恢复,只有 Project Integration 进度窗(仅有 Cancel)保持不动。所有弹窗文本和实际控件 ID 都写入 `dialogs.jsonl`。
- 弹窗只认领本次运行的私有桌面/进程,依据已知规则投递 `BM_CLICK`。数据库恢复、会话失联和设计锁接管提示会阻断并保留现场;不会自动选择恢复或接管。未知弹窗也会阻断,Project Integration 进度窗保持观察。动作/错误弹窗和实际控件 ID 写入 `dialogs.jsonl`。
- 超时或失败时停止自动化客户端,保留可能仍在写库的原生会话并停止批次。监督器核对进程是否退出,记录 `process_exit_confirmed` 或 `process_retained`;无法确认释放时输出 `recovery-required.json`,不以强杀代替数据库安全关闭。
## 环境要求
@@ -163,7 +167,7 @@ Windows 版本判断以 Build 为准:`22000` 及以上归为 Win11,`10240–
## 批量策略
每个 CBB 独立执行并写入自己的 evidence 和结果行。缺少 PCB/PRJ、内容检查失败、刷新失败或超时只影响当前项,记录 `BLOCKED/FAIL/TIMEOUT` 后继续下一项;不会因为单项失败终止整批。只有点击“当前项完成后停止”(或执行器返回 `QUEUE_STOP`)才把后续项目标为 `SKIPPED`。
每个 CBB 独立执行并写入自己的 evidence 和结果行。可确认安全的单项失败允许继续;遇到 `ICDB_REPAIR_REQUIRED`、超时、原生会话释放未确认等情况,会停止整批并把后续项目标为 `SKIPPED`。点击“当前项完成后停止”或执行器返回 `QUEUE_STOP` 也会停止队列。
GUI 中状态为 `绿色 / Ready` 的条目不能勾选、也不会进入批量队列。
@@ -196,7 +200,7 @@ Mentor 桥耗时需在目标机器另行测量。
- `save.log`:`LICENSE_OK / SAVE_OK / IS_SAVED / POST_SAVE_SYNCHRONIZED / CLOSE_OK / QUIT_OK`;
- `result.json`:状态、中心库标志读取结果、阻断原因;
- `image-status-map.json`:按中心库路径隔离的本机动态 `image_id` 校准快照(当前格式 `schema_version=2`);旧版本或无版本缓存会自动忽略并重新校准,日志会记录当前样本、缓存样本、Ready 主 ID、未映射 ID 和逐项决策原因;
- `backup/`:仅本地盘操作前的稳定文件副本;网络盘/UNC 跳过前置递归备份(`backup_skipped`);`database/LogFiles/tmp` 等易变目录和日志文件会排除,备份超时或失败不会阻断 PCB 校验。每个 CBB 结束后默认自动删除该目录,只保留 `events.jsonl`、`dialogs.jsonl`、`save.log` 和 `result.json` 等轻量证据。
- `backup/`:仅本地盘操作前的稳定文件副本;网络盘/UNC 跳过前置递归备份(`backup_skipped`);`database/LogFiles/tmp` 等易变目录和日志文件会排除,备份超时或失败不会阻断 PCB 校验。仅在成功且确认安全后清理该目录,失败时保留已有副本及日志;它不是完整数据库回滚备份。
超时采用无进展计时:界面和批量请求默认 600 秒无进展超时;它不是整个 CBB 的固定总时限,而是连续 600 秒没有可观察进展时才终止。PCB/COM 阶段切换、Project Integration 状态变化、保存标记变化和弹窗处理都会刷新计时器;重复的状态轮询不算进展。需要更快失败时仍可在界面将单项预算调低,但 30 秒只适合已确认 PCB 启动稳定的环境。
@@ -0,0 +1,103 @@
# iCDB 旧服务器会话失联:分析、恢复步骤与分支改动
分支:`codex/icdb-lost-session-diagnostics`,基于 `a11eb3c`(`v0.2.2-beta.1`)。
## 错误的含义和影响范围
现场消息为:
```text
The last server session on different server machine lost the connection
to the database. Please use iCDB Project Backup 'Repair project'
functionality to fix this issue.
Machine: [200091-CBB]
Session: [2026-10-24 08.22.29-10308]
```
原生 iCDB 检测到先前服务器会话的数据库状态需要检查/修复。这不是
“当前网络连接已经恢复就可以继续”的提示,也不能仅凭它认定端口被 Windows 保留。
`Machine` 和 `Session` 是故障记录中的原服务器、会话标识,不能当作当前进程 PID。
用户贴出的路径包含 `5:\5ymbolLibsReusableBlockssymb`,存在文字识别/转录歧义。
本机隔离样本确实存在 `SymbolLibs\__ReusableBlocksSymb\icdb.dat`:如果现场原文
指向这个结构,问题影响的是中心库共享符号数据库,而非单个 CBB 的
`ReusableBlocksLP\<CBB>\Work\database`。必须以现场原始日志为准,不能自动把
`5:` 改成 `S:`,或根据 CBB 名称推导修复目标。
异常关闭、原服务器/网络存储断开、写库期间复制工程、跨机器访问不一致的
路径,都可能涉及此类状态。现有消息不能区分这些原因,也不能证明
“校验失败一定导致损坏”;数据库异常本身同样会导致校验失败。共享符号库
出问题时,不同失败 CBB 可能只是访问了同一个有问题的数据库。
## 现场恢复步骤
1. 停止批量校验,核对 `200091-CBB` 和其他访问同一中心库的客户端。
正常关闭占用该目标的 Library Manager、PCB、原理图/符号编辑会话,并确认
原数据库服务器的实际状态;只看本机没有 PCB 或 TCP 探测失败不足以确认独占。
2. 在会话正常关闭后保留完整、一致的工程/中心库副本。共享库问题需包括
`.lmc`、相关 `SymbolLibs` 数据库、原有 `ProjectBackup`/数据库备份和日志。
工具日常的 `backup/` 排除了 `database` 等目录,网络操作还可能跳过备份,
因而不能把它当作这次修复所需的完整回滚副本。
3. 使用现场相同 Mentor 版本的 **iCDB Project Backup**,先核对打开的工程或
备份存储确实关联报错数据库,再执行 **Project → Repair**(错误提示称为
`Repair project`)。如果目标是共享符号库,而工具无法识别对应库/存储,
应由库管理员按该版本支持的库恢复流程处理,不能改选无关的 Work PRJ。
4. 保留原生工具的修复输出及 `Project Repair` 备份,确认修复结果。
不通过删除 `sAddress.adr`、锁文件、`icdb.dat`,也不通过强杀数据库服务器
来代替 Repair。如果原生修复失败,再依据原生日志选择一致备份恢复。
5. 正常重新打开原先失败的目标,先复测一个 CBB,检查本轮原生日志、
Project Integration、Save/Close 和进程释放,再以 `IsVerifiedReuseBlock`
返回 `READY/raw=1` 确认校验成功。之后再扩大批次。
本机已确认原生入口:
```text
%SDD_HOME%\iCDB\win32\bin\iCDBProjectBackup.exe
```
只读检查其命令行帮助显示 `-p <project file>`、`-ver`、`-h`;菜单资源确认存在
`Project → Repair`。没有查到公开的直接 Repair 命令行开关,因此本分支没有
编造 `-repair` 参数,也没有自动投递写库修复命令。曾在本机既有隔离样本上
打开原生管理界面进行入口检查,未执行 Repair;本机没有用户本次故障库,
未在 `200091-CBB` 或现场网络库上执行恢复。
## 代码修复
- 在内容检查输出、实时弹窗和本轮 `PCBLogFile.txt`、`ForwardAnnotation.txt`、
`MsgBoxResponse.log`、`PartPkg.log` 中识别完整的服务器会话失联消息。
- 分类为 `ICDB_REPAIR_REQUIRED`,保留原始错误和 `Machine/Path/Session`,
区分共享符号库、Work 工程数据库及不能确定的目标;写入
`diagnostics/icdb-recovery.json` 并在失败清单显示恢复说明。
- 该错误优先于通用打开失败、JobPrefs 错误及清理失败;即使内容检查同时
输出成功文字和退出码 0,也不能变成 PASS。标记 `safe_to_continue=False`,
后续条目标记 SKIPPED,沿用现有会话、证据、网络映射保留机制。
- 将当前日志与启动前检查点比较,支持追加、截断、等长/增长重写、UTF-16
及 UTF-8;恢复识别每次最多读取两个 128 KiB 片段。不能确认日志新旧时
保留原始失败,不据旧记录推荐本轮修复。重写后保留的同一历史会话不作为
新证据;可验证的追加则允许识别同一会话再次报错。
- 通用 `Could not run server` 不再直接推断端口保留并提示重启 WinNAT。
启动前独立端口绑定探测保留原逻辑。
## 验证和边界
原分支基线:`python -m pytest -q`,383 项通过。
新增回归包括原错误、共享库身份、内容检查停止队列、多行集成日志、
日志重写及历史错误隔离、编码、证据写入失败和监督器失败摘要。
最终验证:
- 全量 Python 测试 **417 项通过**,含 34 项新增回归;仍有基线已有的
一条 pywinauto STA COM 模式提示。
- 两组 PowerShell 测试、`python -m compileall -q src tests`、
`git diff --check` 通过。
- 只读解析仓库留存的真实 TUNER_R836_A 现场日志,正确提取到机器 `EDA03`、
`S:\ReusableBlocksLP\TUNER_R836_A\Work\database` 及会话
`2025-01-21 14.44.46-2323356`。该记录仅用于解析验证,不代表本次现场故障。
- 独立代码复查发现的集成日志遗漏、通用端口建议覆盖具体原因、历史日志
重写误判、不可读/超出窗口的基线问题,均补充回归并通过复核。
本分支修复自动化的错误识别、误导性建议和失败后的继续执行风险,
**没有修复现场受损数据库,也没有验证原生 Repair 的现场效果**。
恢复识别仅覆盖已知日志末尾的有限窗口;错误出现在窗口外、相同历史会话
在重写时被原样重报、超出基线窗口的大日志发生重写、以及现场不同日志位置/格式,
仍需结合完整原生日志确认。这些不能确认新旧的情况不会自动提升为本轮修复诊断。
+4
View File
@@ -193,6 +193,10 @@ def decide(snapshot: WindowSnapshot, rules: Sequence[DialogRule]) -> RuleDecisio
and not snapshot.controls):
return RuleDecision(rule_id='ime-floating-language-bar', category='OBSERVE_PROGRESS')
combined = normalize_text(' '.join((snapshot.title, *snapshot.texts)))
from .icdb_recovery import ERROR_CODE, parse_session_loss
if parse_session_loss(combined):
return RuleDecision(error_code=ERROR_CODE,
detail=' | '.join(snapshot.texts) or snapshot.title)
# These prompts require knowledge of database ownership or a recovery
# choice. A generic Yes/restore rule cannot establish either fact.
if any(marker in combined for marker in (
@@ -351,6 +351,11 @@ def classify_dialog(snapshot: DialogSnapshot) -> DialogDecision:
joined = " ".join(snapshot.texts).casefold()
combined = f"{title} {joined}"
from .icdb_recovery import ERROR_CODE, parse_session_loss
if parse_session_loss(combined):
return DialogDecision(error_code=ERROR_CODE,
detail=' | '.join(snapshot.texts) or snapshot.title)
if ("problem encountered while opening one of the database files" in combined
and "loading of the job has failed" in combined):
return DialogDecision(action="decline_database_log",
+144
View File
@@ -0,0 +1,144 @@
"""Recognize native iCDB recovery requests without changing database state."""
import hashlib
import ntpath
from pathlib import Path
import re
ERROR_CODE = 'ICDB_REPAIR_REQUIRED'
LOG_LIMIT = 128 * 1024
RECOVERY_ACTION = (
'停止批次并确认原服务器及其他客户端已正常关闭;保留完整一致副本后,'
"使用同版本 iCDB Project Backup 的 Repair project 检查并修复正确目标。"
'共享符号库问题需核对中心库,不能只修复单个 CBB 的 Work 工程。'
'不要删除 sAddress.adr/锁文件;修复后重新打开、校验并确认权威 READY。'
)
_SESSION_LOSS = re.compile(
r'\bthe\s+last\s+server\s+session\s+on\s+different\s+server\s+machine\s+'
r'lost\s+the\s+connection\s+to\s+the\s+database\b', re.IGNORECASE)
def parse_session_loss(text):
"""Keep the native target verbatim; never infer a local repair target."""
text = str(text or '')
match = _SESSION_LOSS.search(text)
if match is None:
return None
# A log may contain several sessions. Use the identity following this
# error, never an unrelated Machine/Path header earlier in the log.
message = text[match.start():]
next_error = _SESSION_LOSS.search(message, 1)
if next_error:
message = message[:next_error.start()]
record = {'error_code': ERROR_CODE, 'native_message': message,
'machine': '', 'database_path': '', 'session': '', 'scope': 'unknown'}
for native_key, key in (('machine', 'machine'), ('path', 'database_path'),
('session', 'session')):
field = re.search(
rf'\b{native_key}\s*:\s*(?:\[([^\]]*)\]|([^\r\n]*?)(?=\s+\b(?:machine|path|session)\s*:|[\r\n]|$))',
message, re.IGNORECASE)
if field:
record[key] = (field.group(1) if field.group(1) is not None else field.group(2)).strip()
parts = ntpath.normpath(record['database_path']).casefold().split('\\')
if 'symbollibs' in parts:
record['scope'] = 'central_library'
elif 'work' in parts and parts[-1] == 'database':
record['scope'] = 'project'
return record
def recovery_detail(record):
target = ('中心库共享符号数据库' if record['scope'] == 'central_library' else
'工程数据库' if record['scope'] == 'project' else '报错中指定的数据库')
return (
f"{ERROR_CODE}: {target}的上次服务器会话失联。\n"
f"{record['native_message']}\n"
f'{RECOVERY_ACTION}'
)
def snapshot_log(path):
"""Fingerprint a bounded tail before launch, including a missing log."""
try:
with Path(path).open('rb') as stream:
stream.seek(0, 2)
size = stream.tell()
stream.seek(max(0, size-LOG_LIMIT))
tail = stream.read(LOG_LIMIT)
losses = [_session_identity(record) for record in _session_losses(decode_log(tail))]
return {'size': size, 'tail_sha256': hashlib.sha256(tail).hexdigest(),
'truncated': size > LOG_LIMIT,
'session_losses': losses}
except FileNotFoundError:
return {'size': 0, 'tail_sha256': hashlib.sha256(b'').hexdigest()}
except OSError as error:
return {'error': str(error)}
def _read_log_delta(path, baseline):
"""Read new bytes, detecting truncate/rewrite as well as append.
Unknown baselines cannot prove freshness. At most two bounded tails are
read, even when a native log is large or located on a network share.
"""
if not baseline or 'error' in baseline:
return b'', False
old_size = int(baseline['size'])
try:
with Path(path).open('rb') as stream:
stream.seek(0, 2)
size = stream.tell()
offset = 0
appended = False
if size >= old_size:
stream.seek(max(0, old_size-LOG_LIMIT))
old_tail = stream.read(min(old_size, LOG_LIMIT))
if hashlib.sha256(old_tail).hexdigest() == baseline['tail_sha256']:
offset = old_size
appended = True
stream.seek(max(offset, size-LOG_LIMIT))
return stream.read(LOG_LIMIT), appended
except OSError:
return b'', False
def _session_losses(text):
matches = list(_SESSION_LOSS.finditer(text))
for index, match in enumerate(matches):
end = matches[index+1].start() if index+1 < len(matches) else len(text)
yield parse_session_loss(text[match.start():end])
def _session_identity(record):
return [record[key] for key in ('machine', 'database_path', 'session')]
def read_log_failure(path, baseline):
"""Return new log text and a recovery request with evidence of freshness.
A rewritten header does not prove that a retained old session failed
again. In that ambiguous case keep the generic failure; a verified append
can report even the same old session as a current native error.
"""
data, appended = _read_log_delta(path, baseline)
text = decode_log(data)
if not appended and baseline and baseline.get('truncated'):
# A rewrite may expose an old error outside our bounded baseline.
# Without positive append evidence its age cannot be established.
return text, None
historical = baseline.get('session_losses', []) if baseline else []
recovery = next((record for record in _session_losses(text)
if appended or _session_identity(record) not in historical), None)
return text, recovery
def decode_log(data):
if data.startswith((b'\xff\xfe', b'\xfe\xff')):
return data.decode('utf-16', errors='replace')
# Appended UTF-16 bytes and bounded tails no longer contain a BOM.
sample = data[:200]
if sample[1::2].count(0) > len(sample)//8:
return data.decode('utf-16-le', errors='replace')
if sample[::2].count(0) > len(sample)//8:
return data.decode('utf-16-be', errors='replace')
return data.decode('utf-8', errors='replace')
+62 -11
View File
@@ -647,10 +647,8 @@ def execute_job(job, emit):
evidence=str(root/'diagnostics/preflight.json'))
cert_project = block_root/'Cert'/Path(job['project']).name
pcb_log = Path(job['pcb']).parent/'LogFiles'/'PCBLogFile.txt'
try:
pcb_log_offset = pcb_log.stat().st_size
except OSError:
pcb_log_offset = 0
from .icdb_recovery import snapshot_log
pcb_log_offset = snapshot_log(pcb_log)
integration_log_dir = Path(job['pcb']).parent / 'LogFiles'
integration_offsets = {}
for name in ('ForwardAnnotation.txt', 'MsgBoxResponse.log', 'PartPkg.log'):
@@ -664,6 +662,26 @@ def execute_job(job, emit):
}
except OSError:
integration_offsets[path] = {'size': 0, 'mtime_ns': None, 'digest': ''}
integration_offsets[path]['icdb_baseline'] = snapshot_log(path)
try:
(root/'diagnostics/startup-log-baseline.json').write_text(json.dumps({
'source': str(pcb_log), 'baseline': pcb_log_offset,
'integration_logs': {str(path): value['icdb_baseline']
for path, value in integration_offsets.items()},
}, ensure_ascii=False), encoding='utf-8')
except OSError as error:
emit(stage='startup_log_baseline_unavailable', detail=str(error))
def native_error():
# Inspect every source before returning a generic server/packager
# failure: either can be a consequence of a lost library session.
startup = pcb_log_reports_missing_server(job['pcb'], offset=pcb_log_offset)
integration = pcb_log_reports_integration_error(job['pcb'], offsets=integration_offsets)
if 'ICDB_REPAIR_REQUIRED' in startup:
return startup
if 'ICDB_REPAIR_REQUIRED' in integration:
return integration
return startup or integration
emit(stage='pcb_baseline', pids=sorted(pcb_before))
lm_session = None
lm_pid = None
@@ -735,7 +753,7 @@ def execute_job(job, emit):
process_alive=lambda: any(candidate_pid in {lm_pid, worker_pid, worker_parent_pid}
for candidate_pid, _, _ in pcb_processes()),
emit=emit,
error_reader=lambda: pcb_log_reports_missing_server(job['pcb'], offset=pcb_log_offset),
error_reader=native_error,
)
# Do not acquire MGCPCBAutomationLicensing from this 64-bit Python
# worker. EE7.9 registers that automation class for the 32-bit host;
@@ -746,7 +764,7 @@ def execute_job(job, emit):
pi,
emit=emit,
timeout_seconds=job['timeout'],
error_reader=lambda: pcb_log_reports_integration_error(job['pcb'], offsets=integration_offsets),
error_reader=native_error,
)
# The x86 helper acquires its own COM references. Keeping these Python
# references alive prevents the native PCB server from exiting cleanly.
@@ -1040,17 +1058,27 @@ def icdb_ports_bindable(ports=(10000, 10050, 10100)):
def pcb_log_reports_missing_server(pcb_path, *, offset=0):
"""Detect the same failure from ExpeditionPCB's own log, as a second signal."""
"""Detect startup failure from this run, preserving native recovery causes."""
from .icdb_recovery import (decode_log, parse_session_loss, read_log_failure,
recovery_detail)
log = Path(pcb_path).parent/'LogFiles'/'PCBLogFile.txt'
try:
with log.open('rb') as stream:
stream.seek(max(0, offset))
text = stream.read().decode('utf-8', errors='replace')
if isinstance(offset, dict):
text, recovery = read_log_failure(log, offset)
else:
# Retain the byte-offset interface for existing diagnostic callers.
with log.open('rb') as stream:
stream.seek(max(0, offset))
text = decode_log(stream.read())
recovery = parse_session_loss(text)
except Exception:
return ''
if recovery:
return recovery_detail(recovery)
for marker in ("could not bring it alive", 'Could not run server'):
if marker.lower() in text.lower():
return RESERVED_PORT_MESSAGE
return (f'DESIGN_SERVER_FAILED: {marker}。请检查本轮原生日志中的数据库/服务器原因;'
'此信息本身不能证明端口被系统保留。')
return ''
@@ -1065,6 +1093,7 @@ _INTEGRATION_ERROR_LINE = re.compile(
def pcb_log_reports_integration_error(pcb_path, *, offsets=None):
"""Read explicit Project Integration errors produced by this run."""
from .icdb_recovery import parse_session_loss, read_log_failure, recovery_detail
log_dir = Path(pcb_path).parent / 'LogFiles'
offsets = offsets or {}
details = []
@@ -1073,6 +1102,19 @@ def pcb_log_reports_integration_error(pcb_path, *, offsets=None):
try:
size = path.stat().st_size
baseline = offsets.get(path, offsets.get(str(path), 0))
has_icdb_baseline = isinstance(baseline, dict) and any(
key in baseline for key in ('icdb_baseline', 'tail_sha256', 'error'))
icdb_baseline = baseline.get('icdb_baseline', baseline) if isinstance(baseline, dict) else None
if icdb_baseline and 'tail_sha256' in icdb_baseline:
text, recovery = read_log_failure(path, icdb_baseline)
if recovery:
return recovery_detail(recovery)
if 'tail_sha256' in baseline:
# New bounded checkpoints can also be supplied directly.
details.extend(line.strip() for line in text.splitlines()
if _INTEGRATION_ERROR_LINE.search(line.strip())
and not parse_session_loss(line))
continue
if isinstance(baseline, dict):
baseline_size = int(baseline.get('size', 0) or 0)
baseline_mtime = baseline.get('mtime_ns')
@@ -1103,8 +1145,17 @@ def pcb_log_reports_integration_error(pcb_path, *, offsets=None):
text = stream.read().decode('utf-8', errors='replace')
except OSError:
continue
# Legacy byte offsets/digests remain accepted by offline callers.
if not has_icdb_baseline:
recovery = parse_session_loss(text)
if recovery:
return recovery_detail(recovery)
for line in text.splitlines():
cleaned = line.strip()
if has_icdb_baseline and parse_session_loss(cleaned):
# read_log_failure already rejected this retained old session.
# Do not let the generic ERROR filter promote it again.
continue
if _INTEGRATION_ERROR_LINE.search(cleaned) and cleaned not in details:
details.append(cleaned)
return ' | '.join(details[:8])
@@ -113,6 +113,8 @@ def collect_native_failure_logs(pcb_path, evidence_root, *, max_bytes=NATIVE_LOG
target = destination/(label+'-'+source.name)
target.write_bytes(data)
item.update(evidence=str(target), copied_bytes=len(data))
if label == 'pcb-logfiles' and source.name != 'ReusableBlock.txt':
_collect_icdb_recovery(source, evidence_root)
except FileNotFoundError:
item['exists'] = False
except OSError as error:
@@ -120,6 +122,23 @@ def collect_native_failure_logs(pcb_path, evidence_root, *, max_bytes=NATIVE_LOG
return _write_record(destination.parent/'native-logs.json', record)
def _collect_icdb_recovery(source, evidence_root):
"""Only promote a native error whose log changed after this launch."""
from .icdb_recovery import read_log_failure
root = Path(evidence_root)/'diagnostics'
try:
baseline = json.loads((root/'startup-log-baseline.json').read_text('utf-8'))
checkpoint = (baseline['baseline'] if baseline['source'] == str(source) else
baseline.get('integration_logs', {}).get(str(source)))
if checkpoint is None:
return
_, recovery = read_log_failure(source, checkpoint)
if recovery:
_write_record(root/'icdb-recovery.json', {**recovery, 'source': str(source)})
except (OSError, ValueError, KeyError, TypeError):
pass # Unknown freshness must not replace the original failure.
def failure_logs_main(request_path):
job = json.loads(Path(request_path).read_text(encoding='utf-8'))
+31
View File
@@ -127,6 +127,9 @@ def classify_output(exit_code: int | None, output: str, *, timed_out: bool = Fal
"""Never invent success: PASS needs the success text and no error marker."""
if timed_out:
return 'TIMEOUT'
from .icdb_recovery import parse_session_loss
if parse_session_loss(output):
return 'FAIL'
if exit_code == 0 and SUCCESS_TEXT.search(output) and not ERROR_TEXT.search(output):
return 'PASS'
if ERROR_TEXT.search(output):
@@ -277,6 +280,9 @@ def read_library_flag_with_retry(reader, library, block_name, *, attempts=3,
def classify_failure_category(detail):
text = str(detail or '')
lower = text.casefold()
from .icdb_recovery import ERROR_CODE, parse_session_loss
if ERROR_CODE.casefold() in lower or parse_session_loss(text):
return ERROR_CODE
if any(code in lower for code in (
'pcb_release_unconfirmed', 'icdb_release_unconfirmed',
'icdb_session_unconfirmed', 'pcb_session_active', 'database_recovery_required')):
@@ -554,6 +560,31 @@ def _run_verify_serialized(request, *, progress=None, flag_reader=None, process_
pass
finally:
timing.end('failure_diagnostics')
if result.status not in {'PASS', 'SKIPPED', 'QUEUE_STOP'}:
from .icdb_recovery import ERROR_CODE, RECOVERY_ACTION, parse_session_loss, recovery_detail
recovery = parse_session_loss(result.stderr)
if recovery is None and result.evidence_path:
try:
saved = json.loads(result.evidence_path.with_name('diagnostics').joinpath(
'icdb-recovery.json').read_text('utf-8'))
recovery = parse_session_loss(saved.get('native_message', ''))
except (OSError, ValueError, AttributeError):
pass
if recovery or ERROR_CODE.casefold() in result.stderr.casefold():
detail = result.stderr
if recovery and RECOVERY_ACTION not in detail:
detail = (detail + '\n' + recovery_detail(recovery)).strip()
result = replace(result, safe_to_continue=False, stderr=detail,
error_category=ERROR_CODE)
if recovery and result.evidence_path:
try:
record_path = result.evidence_path.parent/'diagnostics/icdb-recovery.json'
record_path.parent.mkdir(parents=True, exist_ok=True)
if not record_path.is_file():
record_path.write_text(json.dumps(recovery, ensure_ascii=False, indent=2),
encoding='utf-8')
except OSError:
pass
durations, unfinished = timing.snapshot()
result = replace(result, timings_seconds=durations, incomplete_timing_phases=unfinished,
error_category=(result.error_category or classify_failure_category(result.stderr)
+310
View File
@@ -0,0 +1,310 @@
import json
from pathlib import Path
import pytest
from cbb_refresh_verify import mentor_adapter as adapter
from cbb_refresh_verify import verify_runner as runner
ERROR = (
"The last server session on different server machine lost the connection\n"
"to the database. Please use iCDB Project Backup 'Repair project' functionality "
"to fix this issue. (Machine:[200091-CBB] Path:"
"[S:\\SymbolLibs\\__ReusableBlocksSymb] session:[2026-10-24 08.22.29-10308])"
)
def test_native_session_loss_has_priority_over_generic_load_and_cleanup_errors():
detail = 'DATABASE_LOAD_FAILED: Could not load Job Preferences DB\n' + ERROR
detail += '\nPCB_RELEASE_UNCONFIRMED'
assert runner.classify_failure_category(detail) == 'ICDB_REPAIR_REQUIRED'
def test_live_repair_dialog_blocks_without_clicking():
from test_dialog_rules import prompt, ASSETS
from cbb_refresh_verify.dialog_rules import decide, load_rules
from cbb_refresh_verify.dialog_supervisor import classify_dialog, DialogSnapshot, DialogControl
decision = decide(prompt(ERROR), load_rules(ASSETS/'dialog_rules.v1.json'))
legacy = classify_dialog(DialogSnapshot(1, 2, 'Expedition PCB', (
DialogControl(3, 1, 'Static', ERROR, True),)))
for item in (decision, legacy):
assert item.error_code == 'ICDB_REPAIR_REQUIRED'
assert not item.action
assert '200091-CBB' in item.detail
def test_startup_probe_reports_session_loss_instead_of_port_reset(tmp_path):
log = tmp_path/'LogFiles/PCBLogFile.txt'
log.parent.mkdir()
log.write_text(ERROR + '\nCould not run server', encoding='utf-8')
detail = adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb')
assert 'ICDB_REPAIR_REQUIRED' in detail
assert '__ReusableBlocksSymb' in detail
assert 'Repair project' in detail
assert 'net stop winnat' not in detail
def test_generic_server_log_does_not_claim_port_reservation(tmp_path):
log = tmp_path/'LogFiles/PCBLogFile.txt'
log.parent.mkdir()
log.write_text('Could not run server', encoding='utf-8')
detail = adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb')
assert 'DESIGN_SERVER_FAILED' in detail
assert 'WinError 10013' not in detail
assert 'net stop winnat' not in detail
@pytest.mark.parametrize('mode', ['append', 'shorter', 'same_size', 'longer'])
def test_current_run_probe_handles_appended_and_rewritten_logs(tmp_path, mode):
from cbb_refresh_verify.icdb_recovery import snapshot_log
log = tmp_path/'LogFiles/PCBLogFile.txt'
log.parent.mkdir()
old = 'old harmless text\n' if mode == 'longer' else 'x'*(len(ERROR.encode())+40)
if mode == 'same_size':
old = 'x'*len(ERROR.encode())
log.write_text(old, encoding='utf-8')
baseline = snapshot_log(log)
log.write_text(old+ERROR if mode == 'append' else ERROR, encoding='utf-8')
assert 'ICDB_REPAIR_REQUIRED' in adapter.pcb_log_reports_missing_server(
tmp_path/'CBB.pcb', offset=baseline)
def test_old_loss_record_does_not_fail_a_new_run(tmp_path):
from cbb_refresh_verify.icdb_recovery import snapshot_log
log = tmp_path/'LogFiles/PCBLogFile.txt'
log.parent.mkdir()
log.write_text(ERROR, encoding='utf-8')
baseline = snapshot_log(log)
assert adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb', offset=baseline) == ''
with log.open('a', encoding='utf-8') as stream:
stream.write('\nNormal startup completed\n')
assert adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb', offset=baseline) == ''
def test_rewritten_header_does_not_promote_retained_old_session_loss(tmp_path):
from cbb_refresh_verify.icdb_recovery import snapshot_log
log = tmp_path/'LogFiles/PCBLogFile.txt'
log.parent.mkdir()
log.write_text('old header\n'+ERROR, encoding='utf-8')
baseline = snapshot_log(log)
log.write_text('new header\n'+ERROR+'\nNormal startup\n', encoding='utf-8')
assert adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb', offset=baseline) == ''
def test_repeated_old_session_loss_is_current_when_verifiably_appended(tmp_path):
from cbb_refresh_verify.icdb_recovery import snapshot_log
log = tmp_path/'LogFiles/PCBLogFile.txt'
log.parent.mkdir()
log.write_text(ERROR+'\n', encoding='utf-8')
baseline = snapshot_log(log)
with log.open('a', encoding='utf-8') as stream:
stream.write(ERROR)
assert 'ICDB_REPAIR_REQUIRED' in adapter.pcb_log_reports_missing_server(
tmp_path/'CBB.pcb', offset=baseline)
@pytest.mark.parametrize('name', ['MsgBoxResponse.log', 'ForwardAnnotation.txt', 'PartPkg.log'])
def test_integration_log_retains_multiline_loss_identity(tmp_path, name):
from cbb_refresh_verify.icdb_recovery import snapshot_log
log = tmp_path/'LogFiles'/name
log.parent.mkdir()
baseline = snapshot_log(log)
log.write_text(ERROR.replace(' (Machine:', '\nMachine:').replace(' Path:', '\nPath:')
.replace(' session:', '\nSession:'), encoding='utf-8')
detail = adapter.pcb_log_reports_integration_error(tmp_path/'CBB.pcb', offsets={log: baseline})
assert 'ICDB_REPAIR_REQUIRED' in detail
assert '200091-CBB' in detail
assert '__ReusableBlocksSymb' in detail
assert '2026-10-24 08.22.29-10308' in detail
@pytest.mark.parametrize('production_checkpoint', [False, True])
def test_integration_generic_filter_does_not_repromote_historical_repair(tmp_path, production_checkpoint):
import hashlib
from cbb_refresh_verify.icdb_recovery import snapshot_log
log = tmp_path/'LogFiles/MsgBoxResponse.log'
log.parent.mkdir()
old = 'old header\nERROR: '+ERROR.replace('\n', ' ')
log.write_text(old, encoding='utf-8')
baseline = snapshot_log(log)
if production_checkpoint:
raw = log.read_bytes()
baseline = {'size': len(raw), 'digest': hashlib.sha256(raw).hexdigest(),
'icdb_baseline': baseline}
log.write_text(old.replace('old header', 'new header')+'\nnormal startup', encoding='utf-8')
detail = adapter.pcb_log_reports_integration_error(tmp_path/'CBB.pcb', offsets={log: baseline})
assert 'ICDB_REPAIR_REQUIRED' not in detail
assert runner.classify_failure_category(detail) != 'ICDB_REPAIR_REQUIRED'
def test_unreadable_integration_baseline_never_falls_back_to_historical_log(tmp_path):
log = tmp_path/'LogFiles/MsgBoxResponse.log'
log.parent.mkdir()
log.write_text('ERROR: '+ERROR.replace('\n', ' '), encoding='utf-8')
baseline = {'size': 0, 'mtime_ns': None, 'digest': '',
'icdb_baseline': {'error': 'access denied'}}
detail = adapter.pcb_log_reports_integration_error(tmp_path/'CBB.pcb', offsets={log: baseline})
assert runner.classify_failure_category(detail) != 'ICDB_REPAIR_REQUIRED'
def test_rewrite_of_large_log_does_not_promote_error_outside_baseline_window(tmp_path):
from cbb_refresh_verify.icdb_recovery import LOG_LIMIT, snapshot_log
log = tmp_path/'LogFiles/PCBLogFile.txt'
log.parent.mkdir()
log.write_text(ERROR+'\n'+'normal\n'*LOG_LIMIT, encoding='utf-8')
baseline = snapshot_log(log)
log.write_text('changed header\n'+ERROR, encoding='utf-8')
assert adapter.pcb_log_reports_missing_server(tmp_path/'CBB.pcb', offset=baseline) == ''
def test_integration_session_loss_has_priority_over_generic_startup_error(tmp_path, monkeypatch):
from test_verify_worker_stability import exercise_worker
from cbb_refresh_verify.pcb_bridge import PcbBridgeError
job, events, execute = exercise_worker(tmp_path, monkeypatch)
def attach(job, provider, **kwargs):
logs = Path(job['pcb']).parent/'LogFiles'
logs.mkdir()
(logs/'PCBLogFile.txt').write_text('Could not run server', encoding='utf-8')
(logs/'MsgBoxResponse.log').write_text(ERROR, encoding='utf-8')
detail = kwargs['error_reader']()
assert 'ICDB_REPAIR_REQUIRED' in detail
assert 'net stop winnat' not in detail
raise PcbBridgeError('PCB_STARTUP_FAILED', detail)
monkeypatch.setattr('cbb_refresh_verify.pcb_bridge.wait_for_target_document', attach)
with pytest.raises(PcbBridgeError, match='ICDB_REPAIR_REQUIRED'):
execute()
assert not any(event['stage'] == 'save_start' for event in events)
def test_optional_baseline_evidence_write_failure_does_not_block_verify(tmp_path, monkeypatch):
from test_verify_worker_stability import exercise_worker
job, events, execute = exercise_worker(tmp_path, monkeypatch)
original = Path.write_text
def deny_baseline(path, *args, **kwargs):
if path.name == 'startup-log-baseline.json':
raise PermissionError('diagnostic evidence unavailable')
return original(path, *args, **kwargs)
monkeypatch.setattr(Path, 'write_text', deny_baseline)
assert execute()['status'] == 'PASS'
def test_recovery_record_preserves_identity_and_distinguishes_shared_library():
from cbb_refresh_verify.icdb_recovery import parse_session_loss
record = parse_session_loss(ERROR)
assert record['machine'] == '200091-CBB'
assert record['database_path'] == r'S:\SymbolLibs\__ReusableBlocksSymb'
assert record['session'] == '2026-10-24 08.22.29-10308'
assert record['scope'] == 'central_library'
assert parse_session_loss(ERROR.replace(r'S:\SymbolLibs\__ReusableBlocksSymb',
r'S:\ReusableBlocksLP\CBB\Work\database'))['scope'] == 'project'
assert parse_session_loss('Could not run server') is None
@pytest.mark.parametrize('encoding', ['utf-8', 'utf-16', 'utf-16-le'])
def test_new_utf16_or_utf8_log_errors_are_detected_after_append(tmp_path, encoding):
from cbb_refresh_verify.icdb_recovery import snapshot_log
log = tmp_path/'LogFiles/PCBLogFile.txt'
log.parent.mkdir()
log.write_bytes('old text\n'.encode(encoding))
baseline = snapshot_log(log)
with log.open('ab') as stream:
stream.write(ERROR.encode('utf-16-le' if encoding == 'utf-16' else encoding))
assert 'ICDB_REPAIR_REQUIRED' in adapter.pcb_log_reports_missing_server(
tmp_path/'CBB.pcb', offset=baseline)
@pytest.mark.parametrize('baseline_state', ['stale', 'absent', 'unreadable'])
def test_historical_log_evidence_never_creates_a_repair_recommendation(tmp_path, baseline_state):
from cbb_refresh_verify.icdb_recovery import snapshot_log
from cbb_refresh_verify.verify_diagnostics import collect_native_failure_logs
pcb = tmp_path/'Work/PCB/CBB.pcb'
log = pcb.parent/'LogFiles/PCBLogFile.txt'
log.parent.mkdir(parents=True)
log.write_text(ERROR, encoding='utf-8')
evidence = tmp_path/'evidence'
(evidence/'diagnostics').mkdir(parents=True)
if baseline_state != 'absent':
baseline = snapshot_log(log) if baseline_state == 'stale' else {'error': 'access denied'}
(evidence/'diagnostics/startup-log-baseline.json').write_text(json.dumps({
'source': str(log), 'baseline': baseline}), encoding='utf-8')
collect_native_failure_logs(pcb, evidence)
assert not (evidence/'diagnostics/icdb-recovery.json').exists()
def test_content_success_text_cannot_override_native_session_loss():
assert runner.classify_output(0, 'content checking completed successfully\n'+ERROR) == 'FAIL'
@pytest.mark.parametrize('name', ['PCBLogFile.txt', 'MsgBoxResponse.log', 'ForwardAnnotation.txt', 'PartPkg.log'])
def test_native_log_capture_generates_only_fresh_repair_evidence(tmp_path, name):
from cbb_refresh_verify.icdb_recovery import snapshot_log
from cbb_refresh_verify.verify_diagnostics import collect_native_failure_logs
pcb = tmp_path/'Work/PCB/CBB.pcb'
log = pcb.parent/'LogFiles'/name
log.parent.mkdir(parents=True)
evidence = tmp_path/'evidence'
(evidence/'diagnostics').mkdir(parents=True)
log.write_text('old text', encoding='utf-8')
(evidence/'diagnostics/startup-log-baseline.json').write_text(json.dumps({
'source': str(pcb.parent/'LogFiles/PCBLogFile.txt'),
'baseline': snapshot_log(pcb.parent/'LogFiles/PCBLogFile.txt'),
'integration_logs': {str(log): snapshot_log(log)}}), encoding='utf-8')
log.write_text(ERROR, encoding='utf-8')
collect_native_failure_logs(pcb, evidence)
report = json.loads((evidence/'diagnostics/icdb-recovery.json').read_text('utf-8'))
assert report['scope'] == 'central_library'
assert report['source'] == str(log)
assert log.read_text('utf-8') == ERROR
def test_contents_session_loss_stops_batch_before_any_pcb_launch(tmp_path, monkeypatch):
from test_verify_runner import fixture, request
fixture(tmp_path)
monkeypatch.setattr(runner, 'resolve_importer', lambda *_: Path('ReuseBlockImporter.exe'))
calls = []
def content_check(*args):
calls.append(args)
return 1, ERROR, '', False
results = runner.run_batch([request(tmp_path, libspec='EE', dms_login_config='login')]*2,
verifier=lambda req: runner.run_verify(req, process_runner=content_check,
port_check=lambda: pytest.fail('PCB must not start')))
assert len(calls) == 1
assert results[0].error_category == 'ICDB_REPAIR_REQUIRED'
assert results[0].safe_to_continue is False
assert 'Repair project' in results[0].stderr
assert results[1].status == 'SKIPPED'
def test_supervisor_enriches_generic_failure_from_fresh_native_log(tmp_path, monkeypatch):
from test_verify_runner import fixture, request
from cbb_refresh_verify.icdb_recovery import snapshot_log
from cbb_refresh_verify.verify_diagnostics import failure_logs_main
work = fixture(tmp_path)
log = work/'PCB/LogFiles/PCBLogFile.txt'
log.parent.mkdir()
class Worker:
desktop = None
returncode = 1
def __init__(self, command, *_a, **_kw):
root = Path(command[-1]).parent
(root/'diagnostics').mkdir()
(root/'diagnostics/startup-log-baseline.json').write_text(json.dumps({
'source': str(log), 'baseline': snapshot_log(log)}), encoding='utf-8')
log.write_text(ERROR, encoding='utf-8')
(root/'outcome.json').write_text(json.dumps({
'status': 'FAIL', 'stderr': 'MENTOR_DATABASE_LOAD_FAILED: opening failed',
'safe_to_continue': False}), encoding='utf-8')
def poll(self): return 1
def close(self): pass
monkeypatch.setattr('cbb_refresh_verify.desktop_worker.DesktopWorker', Worker)
monkeypatch.setattr(runner, '_cleanup_session_processes', lambda *a, **k: {'safe_to_continue': True})
monkeypatch.setattr('cbb_refresh_verify.verify_diagnostics.capture_failure_logs_with_timeout', failure_logs_main)
result = runner.run_verify(request(tmp_path), port_check=lambda: (True, ''))
assert result.error_category == 'ICDB_REPAIR_REQUIRED'
assert result.safe_to_continue is False
assert 'opening failed' in result.stderr
assert '200091-CBB' in result.stderr
assert '__ReusableBlocksSymb' in result.stderr
assert '2026-10-24 08.22.29-10308' in result.stderr
assert json.loads(result.evidence_path.read_text('utf-8'))['error_category'] == 'ICDB_REPAIR_REQUIRED'