fix: classify ICDB repair outcomes and gate released recovery
This commit is contained in:
1 parent
44b808b8e2
commit
aa760c2839
12 files changed
+751
-27
No files matched your search
@@ -78,3 +78,64 @@ error and historical-backup recovery, not the field lost-session auto-repair
|
||||
trigger. The test does not change production handling of old-backup recovery.
|
||||
|
||||
READY filtering, hash-scope changes and CBB CRUD are outside this focused fix.
|
||||
|
||||
## Approved Task 1 follow-up: lifecycle and repair outcomes
|
||||
|
||||
The empty-PCB close path now receives the launcher process handle and waits for
|
||||
its normal exit after PCB exits. It checks database addresses again during that
|
||||
wait, so a late address record prevents a false `CLOSED` result. The startup
|
||||
record includes PCB, launcher and address release state. A connected COM server
|
||||
with no active document emits `pcb_document_wait`; it remains in startup and
|
||||
continues reading the current native failure rather than implying identity.
|
||||
|
||||
Native Repair reads bounded Output text and records only the text appended after
|
||||
this run's Repair invocation as operation evidence. Database logs are restricted
|
||||
to the current native PID and start time. Output, native log copies, the complete
|
||||
backup, native result and attempt record remain available under `native-repair`.
|
||||
Detailed native evidence is preserved separately from its directory path.
|
||||
|
||||
The result distinguishes `MISSING_HEALTHY_BACKUP`, `NATIVE_REPAIR_FAILED`,
|
||||
`DATABASE_BUSY`, `UNKNOWN_DIALOG`, and `TIMEOUT_UNRELEASED`. A failed tool receives
|
||||
cooperative close only after a terminal native error, an enabled exact-project
|
||||
main window, no other windows/prompts and no target database address. The worker
|
||||
then waits for normal process exit. Unknown/custom prompts and incomplete native
|
||||
operations receive no close action. A failure classification can coexist with
|
||||
unreleased state; that state always stops the queue. A busy-project message in
|
||||
this fresh native process's initial loading Output prevents Repair invocation
|
||||
entirely; it is not hidden by the later invocation baseline.
|
||||
|
||||
Before reopening or continuing, the parent independently checks relevant PCB,
|
||||
Library Manager, Repair, importer and close-helper processes, all known project/
|
||||
PCB/shared-library addresses, and exclusive read access to every existing known
|
||||
database. Process-observation errors and exclusive-access failures block reuse.
|
||||
Only a conclusively failed, released, independently revalidated project-local
|
||||
target may allow the next CBB. Shared or unknown scope, busy owners, timeouts,
|
||||
unknown prompts and historical restores stop for review.
|
||||
|
||||
Native `REPAIRED` is retained as `native_status`; summaries count a repair only
|
||||
after the same request reopens, completes Verify and returns authoritative
|
||||
`PASS/GREEN` with confirmed release. A failed reopen becomes
|
||||
`REPAIR_VALIDATION_FAILED`; an unconfirmed native release prevents reopening.
|
||||
Historical restoration remains `RESTORED_REVIEW_REQUIRED`. The original error,
|
||||
stage and evidence references, a `pre-repair-result.json` snapshot, native
|
||||
evidence and validation evidence remain attached even if the retry raises.
|
||||
|
||||
`tests/fixtures/issue2-lost-session-sanitized.json` contains selected fields from
|
||||
the actual issue attachment, with paths, machine and process identities
|
||||
sanitized. Replay asserts `PCB_STARTING`, the selected PRJ/database, normal PCB
|
||||
and launcher release before native Repair, and exactly one automatic repair and
|
||||
one reopen even if the opening error repeats. Separate replays cover the actual
|
||||
2026-10-28 missing-backup Output, delayed exits/addresses, unknown custom prompts,
|
||||
successful native repair and historical restoration. These are nonnative tests.
|
||||
|
||||
The attachment contains logs only. No matching `1GPHY_8211` faulty project was
|
||||
found in the sibling `Skyworth_CBB_Lib` collection. This host runs Windows 11;
|
||||
Windows 10 behavior and recovery of the real field database remain unverified.
|
||||
The coordinator's prepared `diagnostics/issue2-frozen-20261102` fixture damages a
|
||||
copied database header; it does not reproduce field lost-server-session loss.
|
||||
Candidate packaging and frozen/native tests remain coordinator work after review.
|
||||
|
||||
Task 1 checks: full suite once, 532 passed with the existing pywinauto COM warning;
|
||||
after the final initial-busy gate regression, the related final suite passed all
|
||||
234 tests. Compileall and `git diff --check` passed. The coordinator will run
|
||||
candidate/frozen validation against the final implementation.
|
||||
@@ -25,7 +25,7 @@ def _command_args(command):
|
||||
|
||||
|
||||
def close_failed_startup(job, pcb_pid, *, baseline_pids=(), emit=lambda **event: None,
|
||||
timeout_seconds=20):
|
||||
timeout_seconds=20, launcher_process=None):
|
||||
"""Normally close our empty PCB after a diagnosed database startup failure.
|
||||
|
||||
No COM attach: there is no document, and the global ROT can name another
|
||||
@@ -38,6 +38,10 @@ def close_failed_startup(job, pcb_pid, *, baseline_pids=(), emit=lambda **event:
|
||||
from .icdb_ownership import unreleased_addresses
|
||||
root = Path(job['root'])
|
||||
result = {'status': 'RETAINED', 'pcb_pid': pcb_pid}
|
||||
result['release'] = {'pcb_exited': False, 'launcher_exited': launcher_process is None,
|
||||
'safe_to_continue': False}
|
||||
if launcher_process is not None:
|
||||
result['launcher_pid'] = launcher_process.pid
|
||||
try:
|
||||
diagnosed = False
|
||||
diagnosis_path = root/'diagnostics/icdb-recovery.json'
|
||||
@@ -68,8 +72,20 @@ def close_failed_startup(job, pcb_pid, *, baseline_pids=(), emit=lambda **event:
|
||||
while True:
|
||||
records = [r for r in _process_details('ExpeditionPCB.exe') if r[0] == pcb_pid]
|
||||
if not records:
|
||||
result['status'] = 'CLOSED'
|
||||
break
|
||||
addresses = unreleased_addresses(job['lmc_file'], job['project'], job['pcb'])
|
||||
launcher_exited = launcher_process is None or launcher_process.poll() is not None
|
||||
result['release'] = {'pcb_exited': True, 'launcher_exited': launcher_exited,
|
||||
'addresses': addresses,
|
||||
'safe_to_continue': launcher_exited and not addresses}
|
||||
if result['release']['safe_to_continue']:
|
||||
result['status'] = 'CLOSED'
|
||||
result.pop('detail', None)
|
||||
break
|
||||
result['detail'] = 'STARTUP_RELEASE_UNCONFIRMED: 等待启动器退出和数据库地址释放'
|
||||
if time.monotonic() >= deadline:
|
||||
break
|
||||
time.sleep(.2)
|
||||
continue
|
||||
_, executable, command, _ = records[0]
|
||||
args = _command_args(command)
|
||||
if (Path(executable).name.casefold() != 'expeditionpcb.exe'
|
||||
|
||||
@@ -5,7 +5,12 @@ import json
|
||||
|
||||
def maintenance_counts(results):
|
||||
attempts = [attempt for result in results for attempt in result.recovery_attempts]
|
||||
repaired = sum(attempt.get('status') == 'REPAIRED' for attempt in attempts)
|
||||
repaired = sum(attempt.get('status') == 'REPAIRED'
|
||||
and attempt.get('native_status') == 'REPAIRED'
|
||||
and attempt.get('validation_passed') is True
|
||||
and result.status == 'PASS' and result.library_flag == 'GREEN'
|
||||
and result.safe_to_continue
|
||||
for result in results for attempt in result.recovery_attempts)
|
||||
review = sum(attempt.get('status') == 'RESTORED_REVIEW_REQUIRED' for attempt in attempts)
|
||||
return {'total': len(results), 'passed': sum(r.status == 'PASS' for r in results),
|
||||
'skipped': sum(r.status == 'SKIPPED' for r in results),
|
||||
|
||||
@@ -866,7 +866,8 @@ def execute_job(job, emit):
|
||||
elif not completed and worker_pid is not None:
|
||||
try:
|
||||
from .failed_session import close_failed_startup
|
||||
failed_close = close_failed_startup(job, worker_pid, baseline_pids=pcb_before, emit=emit)
|
||||
failed_close = close_failed_startup(job, worker_pid, baseline_pids=pcb_before, emit=emit,
|
||||
launcher_process=launcher_process)
|
||||
if failed_close.get('status') == 'CLOSED' and lm_session is not None:
|
||||
lm_session.close()
|
||||
except Exception as error:
|
||||
|
||||
@@ -23,6 +23,8 @@ def select_repair_target(request, diagnosis):
|
||||
raise ValueError('REPAIR_TARGET_UNCONFIRMED: 缺少原生失联会话信息')
|
||||
if diagnosis.get('scope') == 'central_library':
|
||||
raise ValueError('SHARED_DATABASE_REPAIR_UNSUPPORTED: 当前原生 Repair 入口要求 PRJ,不能改修无关的 Work 工程')
|
||||
if diagnosis.get('scope') != 'project':
|
||||
raise ValueError('REPAIR_TARGET_UNCONFIRMED: 数据库所有权范围未知')
|
||||
project, pcb = preflight(request)
|
||||
try:
|
||||
reported = Path(diagnosis.get('database_path', '')).resolve(strict=True)
|
||||
@@ -115,7 +117,8 @@ def attempt_recovery(request, diagnosis, evidence_root, *, progress=None):
|
||||
marker = root/'attempt.json'
|
||||
if marker.exists():
|
||||
return {'status': 'BLOCKED', 'detail': 'REPAIR_ALREADY_ATTEMPTED', 'evidence': str(root)}
|
||||
record = {'status': 'BLOCKED', 'evidence': str(root)}
|
||||
record = {'status': 'BLOCKED', 'evidence_path': str(root),
|
||||
'release': {'safe_to_continue': False}}
|
||||
try:
|
||||
process_record = Path(evidence_root)/'content-check-process.json'
|
||||
if process_record.is_file():
|
||||
@@ -137,12 +140,51 @@ def attempt_recovery(request, diagnosis, evidence_root, *, progress=None):
|
||||
callback = (lambda event: progress({'block': request.block_name, **event})) if progress else None
|
||||
record = run_native_repair(target, root, mentor_bin=request.mentor_bin,
|
||||
timeout_seconds=max(60, request.timeout_seconds), progress=callback)
|
||||
record['release'] = repair_release_state(request, target, record)
|
||||
record['evidence_path'] = str(root)
|
||||
except Exception as error:
|
||||
record['detail'] = str(error)
|
||||
marker.write_text(json.dumps(record, ensure_ascii=False, indent=2), encoding='utf-8')
|
||||
return record
|
||||
|
||||
|
||||
def repair_release_state(request, target, native_result):
|
||||
"""Require normal exit, owner/address absence and exclusive database access."""
|
||||
from .mentor_adapter import pcb_processes, _process_details
|
||||
from .icdb_ownership import database_directories, unreleased_addresses
|
||||
from .verify_runner import preflight
|
||||
result = {**native_result.get('release', {}), 'safe_to_continue': False,
|
||||
'database_exclusive': False}
|
||||
try:
|
||||
project, pcb = preflight(request)
|
||||
owners = [int(r[0]) for r in pcb_processes()]
|
||||
for name in ('LibraryManager.exe', 'iCDBProjectBackup.exe', 'ReuseBlockImporter.exe'):
|
||||
owners.extend(int(r[0]) for r in _process_details(name))
|
||||
for pid, _exe, command, _parent in _process_details('cscript.exe'):
|
||||
if (not command or str(project).casefold() in command.casefold()
|
||||
or str(pcb).casefold() in command.casefold()):
|
||||
owners.append(int(pid))
|
||||
result.update(remaining_pids=sorted(set(owners)),
|
||||
addresses=unreleased_addresses(request.lmc_file, project, pcb))
|
||||
if (native_result.get('release', {}).get('safe_to_continue') is not True
|
||||
or owners or result['addresses']):
|
||||
return result
|
||||
import win32con, win32file
|
||||
files = [target.database/'icdb.dat']
|
||||
files.extend(folder/'icdb.dat' for folder in database_directories(request.lmc_file, project, pcb)
|
||||
if (folder/'icdb.dat').is_file())
|
||||
result['exclusive_databases'] = []
|
||||
for database_file in dict.fromkeys(files):
|
||||
handle = win32file.CreateFile(str(database_file), win32con.GENERIC_READ,
|
||||
0, None, win32con.OPEN_EXISTING, win32con.FILE_ATTRIBUTE_NORMAL, None)
|
||||
handle.Close()
|
||||
result['exclusive_databases'].append(str(database_file))
|
||||
result.update(database_exclusive=True, safe_to_continue=True)
|
||||
except Exception as error:
|
||||
result['error'] = str(error)
|
||||
return result
|
||||
|
||||
|
||||
def run_native_repair(target, root, *, mentor_bin='', timeout_seconds=300, progress=None):
|
||||
"""Supervise the GUI on its own desktop; a repair timeout never kills it."""
|
||||
from .desktop_worker import DesktopWorker
|
||||
@@ -176,11 +218,12 @@ def run_native_repair(target, root, *, mentor_bin='', timeout_seconds=300, progr
|
||||
outcome = root/'repair-result.json'
|
||||
if worker.poll() is None or not outcome.is_file():
|
||||
worker.request_stop()
|
||||
return {'status': 'RETAINED', 'worker_pid': worker.pid, 'desktop': desktop,
|
||||
'detail': '原生修复尚未确认结束,保留操作并停止批次', 'evidence': str(root)}
|
||||
return {'status': 'TIMEOUT_UNRELEASED', 'worker_pid': worker.pid, 'desktop': desktop,
|
||||
'release': {'safe_to_continue': False},
|
||||
'detail': '原生修复尚未确认结束,保留操作并停止批次', 'evidence_path': str(root)}
|
||||
result = json.loads(outcome.read_text('utf-8'))
|
||||
if worker.returncode != 0 and result.get('status') == 'REPAIRED':
|
||||
result['status'] = 'UNCONFIRMED'
|
||||
return {**result, 'evidence': str(root)}
|
||||
return {**result, 'evidence_path': str(root)}
|
||||
finally:
|
||||
worker.close()
|
||||
@@ -5,6 +5,7 @@ import json
|
||||
import ntpath
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
import zipfile
|
||||
@@ -30,6 +31,23 @@ def repair_completed(diagnostic_text, database_text, *, native_backup):
|
||||
and 'iCDBProjectBackupRepair' in database_text and 'IC: Closing project' in database_text)
|
||||
|
||||
|
||||
def classify_repair_evidence(diagnostic, database, restore, output):
|
||||
"""Classify this invocation only; an error alone does not prove native idle."""
|
||||
combined = '\n'.join((diagnostic, database, restore, output))
|
||||
terminal_error = bool(re.search(r'Repair project error\s*\[[^\r\n]+\]', output, re.I))
|
||||
closed_error = ('Diagnostic: FINISH' in diagnostic and 'IC: Closing project' in database)
|
||||
if re.search(r'project is in use|database.*(?:in use|busy)|other clients are connected', combined, re.I):
|
||||
outcome = 'DATABASE_BUSY'
|
||||
elif re.search(r'there is no backup of (?:missing )?iCDB|no (?:healthy|valid) backup', combined, re.I):
|
||||
outcome = 'MISSING_HEALTHY_BACKUP'
|
||||
elif re.search(r'\bERROR\s*:|not repaired|repair.*failed|Repair project error', combined, re.I):
|
||||
outcome = 'NATIVE_REPAIR_FAILED'
|
||||
else:
|
||||
outcome = 'UNCONFIRMED'
|
||||
return {'outcome': outcome, 'failure_complete': bool(
|
||||
outcome != 'UNCONFIRMED' and (terminal_error or closed_error))}
|
||||
|
||||
|
||||
def _current_desktop_name():
|
||||
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
|
||||
user32 = ctypes.WinDLL('user32', use_last_error=True)
|
||||
@@ -97,12 +115,36 @@ def _windows(pid):
|
||||
return windows
|
||||
|
||||
|
||||
def _native_evidence(project, database, pid, started_ns, baseline):
|
||||
def _native_output(windows):
|
||||
"""Read Output without UI activation; a hung control has a bounded deadline."""
|
||||
user32 = ctypes.WinDLL('user32', use_last_error=True)
|
||||
user32.SendMessageTimeoutW.argtypes = [wintypes.HWND, wintypes.UINT, wintypes.WPARAM,
|
||||
wintypes.LPARAM, wintypes.UINT, wintypes.UINT, ctypes.POINTER(ctypes.c_size_t)]
|
||||
user32.SendMessageTimeoutW.restype = wintypes.LPARAM
|
||||
texts = []
|
||||
for window in windows:
|
||||
for child in window['children']:
|
||||
if not child['class'].casefold().startswith('richedit'):
|
||||
continue
|
||||
buffer = ctypes.create_unicode_buffer(131072)
|
||||
copied = ctypes.c_size_t()
|
||||
if not user32.SendMessageTimeoutW(child['hwnd'], 0x000D, len(buffer),
|
||||
ctypes.addressof(buffer), 0x0002, 200, ctypes.byref(copied)):
|
||||
return None
|
||||
if copied.value >= len(buffer)-1:
|
||||
return None # A truncated prefix cannot establish the current operation.
|
||||
texts.append(buffer.value)
|
||||
return '\n'.join(texts) if texts else None
|
||||
|
||||
|
||||
def _native_evidence(project, database, pid, started_ns, baseline, *, output_text=''):
|
||||
"""Use only this native process's new diagnostic/close and backup evidence."""
|
||||
log_paths = []
|
||||
def read_logs(folder):
|
||||
text = []
|
||||
for path in (database/'cdbsvr'/folder).glob(f'*-{pid}/*.txt'):
|
||||
if path.stat().st_mtime_ns >= started_ns:
|
||||
log_paths.append(str(path))
|
||||
text.append(path.read_bytes()[-256*1024:].decode('utf-8', errors='replace'))
|
||||
return '\n'.join(text)
|
||||
from .icdb_recovery import read_log_failure
|
||||
@@ -122,9 +164,13 @@ def _native_evidence(project, database, pid, started_ns, baseline):
|
||||
diagnostic, db_log = read_logs('dbcheck'), read_logs('log')
|
||||
restore_log = read_logs('dbrestore')
|
||||
restored = re.search(r'iCDB database restored from backup:\s*([^\r\n]+)', restore_log)
|
||||
valid = (not restored and str(database).casefold() in db_log.casefold()
|
||||
classification = classify_repair_evidence(diagnostic, db_log, restore_log, output_text)
|
||||
valid = (classification['outcome'] == 'UNCONFIRMED' and not restored
|
||||
and str(database).casefold() in db_log.casefold()
|
||||
and repair_completed(diagnostic, db_log, native_backup=native_backup))
|
||||
return {'confirmed': bool(valid), 'native_backup': native_backup,
|
||||
return {**classification, 'confirmed': bool(valid), 'native_backup': native_backup,
|
||||
'output_tail': output_text[-8192:], 'log_paths': log_paths,
|
||||
'backup_log_tail': text[-8192:],
|
||||
'diagnostic_tail': diagnostic[-8192:], 'database_tail': db_log[-8192:],
|
||||
'restored_from_backup': restored.group(1).strip() if restored else '',
|
||||
'restore_complete': bool(restored and '[0] DONE' in restore_log
|
||||
@@ -140,7 +186,8 @@ def repair_worker_main(job_path):
|
||||
job = json.loads(Path(job_path).read_text('utf-8'))
|
||||
root = Path(job['root'])
|
||||
project, database = Path(job['project']), Path(job['database'])
|
||||
result = {'status': 'BLOCKED', 'project': str(project), 'database': str(database)}
|
||||
result = {'status': 'BLOCKED', 'project': str(project), 'database': str(database),
|
||||
'release': {'safe_to_continue': False}}
|
||||
process = None
|
||||
def emit(**event):
|
||||
with (root/'repair-events.jsonl').open('a', encoding='utf-8') as stream:
|
||||
@@ -175,19 +222,38 @@ def repair_worker_main(job_path):
|
||||
deadline = time.monotonic()+max(1, float(job['timeout']))
|
||||
invoked = confirmed = closing = False
|
||||
evidence = None
|
||||
output_baseline = None
|
||||
output = ''
|
||||
while process.poll() is None and time.monotonic()<deadline:
|
||||
windows = _windows(process.pid)
|
||||
main = next((w for w in windows if w['class'].startswith('Afx:')
|
||||
and w['title'].casefold() == ('iCDB Project Backup - '+str(project)).casefold()), None)
|
||||
mains = [w for w in windows if w['class'].startswith('Afx:')
|
||||
and w['title'].casefold() == ('iCDB Project Backup - '+str(project)).casefold()]
|
||||
main = mains[0] if len(mains) == 1 else None
|
||||
dialogs = [w for w in windows if w['class'] == '#32770']
|
||||
if not invoked and main and not dialogs and main['enabled']:
|
||||
unknown = [w for w in windows if w is not main and w not in dialogs]
|
||||
current_output = _native_output(windows)
|
||||
busy_before_start = (not invoked and current_output is not None
|
||||
and classify_repair_evidence('', '', '', current_output)['outcome'] == 'DATABASE_BUSY')
|
||||
if busy_before_start:
|
||||
output = current_output
|
||||
result.update(status='DATABASE_BUSY', detail='原生工具加载工程时报告占用;未请求 Repair')
|
||||
if not invoked and not busy_before_start and main and not dialogs and not unknown and main['enabled']:
|
||||
if Path(job['cancel_signal']).exists():
|
||||
raise ValueError('REPAIR_CANCELLED_BEFORE_START')
|
||||
output_baseline = current_output
|
||||
win32gui.PostMessage(main['hwnd'], win32con.WM_COMMAND, command_id, 0)
|
||||
invoked = True
|
||||
emit(stage='native_repair_requested', command=command_id)
|
||||
if invoked and not confirmed and main:
|
||||
for dialog in dialogs:
|
||||
if dialogs or unknown:
|
||||
known = [d for d in dialogs if invoked and not confirmed and main
|
||||
and confirmation_allowed(main['title'], project,
|
||||
'\n'.join(c['text'] for c in d['children']))]
|
||||
if unknown or len(dialogs) != 1 or len(known) != 1:
|
||||
result.update(status='UNKNOWN_DIALOG', unknown_dialogs=[*unknown, *dialogs],
|
||||
detail='未知原生弹窗;未发送操作,保留原生工具')
|
||||
# Capture evidence below before stopping supervision.
|
||||
else:
|
||||
dialog = known[0]
|
||||
text = '\n'.join(c['text'] for c in dialog['children'])
|
||||
if confirmation_allowed(main['title'], project, text):
|
||||
if Path(job['cancel_signal']).exists():
|
||||
@@ -198,16 +264,40 @@ def repair_worker_main(job_path):
|
||||
win32gui.PostMessage(button['hwnd'], win32con.BM_CLICK, 0, 0)
|
||||
confirmed = True
|
||||
emit(stage='native_repair_confirmed', text=text)
|
||||
if (invoked and output_baseline is not None and current_output is not None
|
||||
and current_output.startswith(output_baseline)):
|
||||
output = current_output[len(output_baseline):]
|
||||
evidence = _native_evidence(project, database, process.pid, started_ns, baseline,
|
||||
output_text=output)
|
||||
(root/'native-output.json').write_text(json.dumps({
|
||||
'native_pid': process.pid, 'baseline': output_baseline, 'current': current_output,
|
||||
'invocation_output': output}, ensure_ascii=False, indent=2), encoding='utf-8')
|
||||
if result['status'] == 'UNKNOWN_DIALOG' or busy_before_start:
|
||||
break
|
||||
if confirmed and not closing and main and not dialogs and main['enabled']:
|
||||
evidence = _native_evidence(project, database, process.pid, started_ns, baseline)
|
||||
if (evidence['confirmed'] or (evidence.get('restore_complete')
|
||||
if (not (database/'cdbsvr/sAddress.adr').exists() and (
|
||||
evidence['failure_complete'] or evidence['confirmed'] or (evidence.get('restore_complete')
|
||||
and evidence['native_backup']
|
||||
and not (database/'cdbsvr/sAddress.adr').exists())):
|
||||
and evidence['outcome'] == 'UNCONFIRMED'))):
|
||||
win32gui.PostMessage(main['hwnd'], win32con.WM_CLOSE, 0, 0)
|
||||
closing = True
|
||||
emit(stage='native_repair_finished', **evidence)
|
||||
time.sleep(.2)
|
||||
if (closing and process.poll() == 0 and evidence and evidence.get('restore_complete')
|
||||
if evidence:
|
||||
result['evidence'] = evidence
|
||||
for log_path in evidence['log_paths']:
|
||||
source = Path(log_path)
|
||||
destination = root/'native-logs'/source.relative_to(database/'cdbsvr')
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(source, destination)
|
||||
addresses = [str(database/'cdbsvr/sAddress.adr')] if (database/'cdbsvr/sAddress.adr').exists() else []
|
||||
result['release'] = {'native_exited': process.poll() is not None, 'addresses': addresses,
|
||||
'safe_to_continue': process.poll() is not None and not addresses}
|
||||
if result['status'] == 'UNKNOWN_DIALOG':
|
||||
pass
|
||||
elif evidence and evidence['outcome'] != 'UNCONFIRMED':
|
||||
result.update(status=evidence['outcome'], detail='原生 Repair 失败;详见本次日志和 Output')
|
||||
elif (closing and process.poll() == 0 and evidence and evidence.get('restore_complete')
|
||||
and not (database/'cdbsvr/sAddress.adr').exists()):
|
||||
result.update(status='RESTORED_REVIEW_REQUIRED', evidence=evidence,
|
||||
detail='原生工具从已有备份恢复数据库;需核对恢复版本和工程内容,未自动重试校验')
|
||||
@@ -215,7 +305,7 @@ def repair_worker_main(job_path):
|
||||
and not (database/'cdbsvr/sAddress.adr').exists()):
|
||||
result.update(status='REPAIRED', evidence=evidence)
|
||||
else:
|
||||
result.update(status='RETAINED' if process.poll() is None else 'UNCONFIRMED',
|
||||
result.update(status='TIMEOUT_UNRELEASED' if process.poll() is None else 'UNCONFIRMED',
|
||||
detail='原生 Repair 未取得完整成功/释放证据,未重试校验')
|
||||
except Exception as error:
|
||||
result.update(detail=str(error), status='RETAINED' if process and process.poll() is None else 'BLOCKED')
|
||||
|
||||
@@ -103,6 +103,11 @@ def wait_for_target_document(job, app_provider, *, process_alive, emit=lambda **
|
||||
try:
|
||||
app = app_provider()
|
||||
document = app.ActiveDocument
|
||||
if not document:
|
||||
observation = ('PCB_DOCUMENT_NOT_READY', 'COM 已连接,尚无目标文档')
|
||||
if observation != last_observation:
|
||||
emit(stage='pcb_document_wait', error_code=observation[0], detail=observation[1])
|
||||
last_observation = observation
|
||||
if document:
|
||||
try:
|
||||
integration = require_document_identity(document, job)
|
||||
|
||||
@@ -525,14 +525,54 @@ def run_verify(request, *, progress=None, flag_reader=None, process_runner=None,
|
||||
try:
|
||||
diagnosis = json.loads((result.evidence_path.parent/'diagnostics/icdb-recovery.json').read_text('utf-8'))
|
||||
original = result
|
||||
snapshot = original.evidence_path.with_name('pre-repair-result.json')
|
||||
snapshot.write_text(json.dumps(asdict(original), ensure_ascii=False, default=str, indent=2),
|
||||
encoding='utf-8')
|
||||
recovery = attempt_recovery(request, diagnosis, result.evidence_path.parent, progress=progress)
|
||||
recovery = {**recovery, 'original_error': original.stderr,
|
||||
'original_evidence': str(original.evidence_path)}
|
||||
if recovery.get('status') == 'REPAIRED':
|
||||
'original_evidence': str(original.evidence_path),
|
||||
'original_result_snapshot': str(snapshot),
|
||||
'original_stage': original.error_stage,
|
||||
'native_status': recovery.get('status'), 'validation_passed': False}
|
||||
if (recovery.get('status') == 'REPAIRED'
|
||||
and recovery.get('release', {}).get('safe_to_continue') is True):
|
||||
result = replace(original, recovery_attempts=(recovery,))
|
||||
recovery['status'] = 'REPAIR_VALIDATION_FAILED'
|
||||
result = _run_verify_serialized(
|
||||
replace(request, auto_repair=False), progress=progress, flag_reader=flag_reader,
|
||||
process_runner=process_runner, port_check=port_check,
|
||||
dialog_supervisor_factory=dialog_supervisor_factory)
|
||||
recovery['validation_evidence'] = str(result.evidence_path or '')
|
||||
recovery['validation_passed'] = (result.status == 'PASS'
|
||||
and result.library_flag == 'GREEN' and result.safe_to_continue)
|
||||
if not recovery['validation_passed']:
|
||||
recovery['status'] = 'REPAIR_VALIDATION_FAILED'
|
||||
if result.status == 'PASS':
|
||||
result = replace(result, status='FAIL',
|
||||
stderr=(result.stderr+'\nREPAIR_VALIDATION_UNCONFIRMED: 未取得校验 GREEN').strip())
|
||||
else:
|
||||
recovery['status'] = 'REPAIRED'
|
||||
elif recovery.get('status') == 'REPAIRED':
|
||||
recovery['status'] = 'REPAIR_RELEASE_UNCONFIRMED'
|
||||
result = replace(result, safe_to_continue=False)
|
||||
else:
|
||||
# A failed local database must not block unrelated CBBs once
|
||||
# every owner is released. Re-check target ownership here so
|
||||
# an incomplete repair record cannot authorize the queue.
|
||||
safe = False
|
||||
if (recovery.get('status') in {'MISSING_HEALTHY_BACKUP', 'NATIVE_REPAIR_FAILED'}
|
||||
and recovery.get('evidence', {}).get('failure_complete') is True
|
||||
and recovery.get('release', {}).get('safe_to_continue') is True):
|
||||
from .native_repair import select_repair_target
|
||||
try:
|
||||
target = select_repair_target(request, diagnosis)
|
||||
safe = (Path(recovery.get('project', '')).resolve() == target.project
|
||||
and Path(recovery.get('database', '')).resolve() == target.database)
|
||||
except (ValueError, OSError):
|
||||
pass
|
||||
result = replace(result, safe_to_continue=safe, stderr=(result.stderr + '\n'
|
||||
+ str(recovery.get('status', 'REPAIR_UNCONFIRMED')) + ': '
|
||||
+ str(recovery.get('detail', '原生 Repair 未完成恢复'))).strip())
|
||||
result = replace(result, recovery_attempts=(recovery,))
|
||||
result.evidence_path.write_text(json.dumps({'block_name': request.block_name, **asdict(result)},
|
||||
ensure_ascii=False, default=str, indent=2), encoding='utf-8')
|
||||
@@ -540,6 +580,12 @@ def run_verify(request, *, progress=None, flag_reader=None, process_runner=None,
|
||||
result = replace(result, status='FAIL' if result.status == 'PASS' else result.status,
|
||||
safe_to_continue=False,
|
||||
stderr=(result.stderr+f'\nREPAIR_UNCONFIRMED: {error}').strip())
|
||||
if result.evidence_path:
|
||||
try:
|
||||
result.evidence_path.write_text(json.dumps({'block_name': request.block_name, **asdict(result)},
|
||||
ensure_ascii=False, default=str, indent=2), encoding='utf-8')
|
||||
except OSError:
|
||||
pass # Keep the native/repair errors even if evidence storage fails.
|
||||
return result
|
||||
finally:
|
||||
if acquired:
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"source": "Issue 2 data.zip, verify-86f90ffba911; selected fields only; paths and process identities sanitized",
|
||||
"limitations": "Logs only. No field database, executable fault reproduction or recovery claim.",
|
||||
"block": "1GPHY_8211",
|
||||
"diagnosis": {
|
||||
"error_code": "ICDB_REPAIR_REQUIRED",
|
||||
"scope": "project",
|
||||
"machine": "FIELD-PC",
|
||||
"session": "2026-10-20 14.03.54-18212",
|
||||
"database_relative": "Work/database"
|
||||
},
|
||||
"native_error": "The last server session on different server machine lost the connection to the database. Please use iCDB Project Backup 'Repair project' functionality to fix this issue. (Machine:[FIELD-PC] Path:[<library>/ReusableBlocksLP/1GPHY_8211/Work/database] session:[2026-10-20 14.03.54-18212])",
|
||||
"observed_events": [
|
||||
{"stage": "pcb_launch", "strategy": "reusable_circuit_direct"},
|
||||
{"stage": "pcb_launcher_started", "launcher_pid": 11},
|
||||
{"stage": "pcb_worker_started", "pcb_pid": 12, "binding": "commandline"},
|
||||
{"stage": "native_failure_logs"},
|
||||
{"stage": "process_retained", "safe_to_continue": false},
|
||||
{"stage": "failed", "error": "VERIFY_CANCELLED: stop and normally close target document"}
|
||||
],
|
||||
"dialog": {
|
||||
"pid": 12,
|
||||
"phase": "PCB_STARTING",
|
||||
"action": "acknowledge_fatal",
|
||||
"error_code": "MENTOR_DATABASE_LOAD_FAILED",
|
||||
"text": "Problem encountered while opening one of the database files. See PCBLogFile.txt for details. Loading of the job has failed. Do you wish to view the log file now?"
|
||||
},
|
||||
"original_result": {"status": "FAIL", "error_category": "ICDB_REPAIR_REQUIRED", "error_stage": "PCB_STARTING"},
|
||||
"original_repair": {"status": "BLOCKED", "detail": "REPAIR_DATABASE_IN_USE"}
|
||||
}
|
||||
@@ -156,3 +156,33 @@ def test_failed_startup_rejects_unrelated_failure_dialog(tmp_path, monkeypatch,
|
||||
'pid': pid, 'phase': phase, 'error_code': 'MENTOR_DATABASE_LOAD_FAILED'})+'\n')
|
||||
assert failed_session.close_failed_startup(job, 12, timeout_seconds=0)['status'] == 'RETAINED'
|
||||
assert not messages
|
||||
|
||||
|
||||
@pytest.mark.parametrize('exits', [True, False])
|
||||
def test_startup_close_waits_for_launcher_after_empty_pcb_exit(tmp_path, monkeypatch, exits):
|
||||
from types import SimpleNamespace
|
||||
job, records, messages = startup_fixture(tmp_path, monkeypatch)
|
||||
ticks = [0.0]
|
||||
launcher = SimpleNamespace(pid=11, poll=lambda: 0 if exits and ticks[0] >= .4 else None)
|
||||
monkeypatch.setattr(failed_session, 'time', SimpleNamespace(
|
||||
monotonic=lambda: ticks[0], sleep=lambda seconds: ticks.__setitem__(0, ticks[0]+seconds)))
|
||||
result = failed_session.close_failed_startup(job, 12, launcher_process=launcher, timeout_seconds=1)
|
||||
assert ticks[0] >= .4
|
||||
assert result['status'] == ('CLOSED' if exits else 'RETAINED')
|
||||
assert result['release']['launcher_exited'] is exits
|
||||
assert result['release']['pcb_exited'] is True
|
||||
assert len(messages) == 1
|
||||
|
||||
|
||||
def test_startup_exit_does_not_hide_late_database_address(tmp_path, monkeypatch):
|
||||
import win32gui
|
||||
job, records, messages = startup_fixture(tmp_path, monkeypatch)
|
||||
def close(*args):
|
||||
records.clear()
|
||||
address = tmp_path/'database/cdbsvr/sAddress.adr'
|
||||
address.parent.mkdir(parents=True)
|
||||
address.write_text('still owned')
|
||||
monkeypatch.setattr(win32gui, 'PostMessage', close)
|
||||
result = failed_session.close_failed_startup(job, 12, timeout_seconds=0)
|
||||
assert result['status'] == 'RETAINED'
|
||||
assert result['release']['addresses']
|
||||
+387
-2
@@ -22,7 +22,7 @@ def test_repair_target_matches_actual_database_not_guessed_cbb_name(tmp_path):
|
||||
assert target.database == target.project.parent/'database'
|
||||
|
||||
|
||||
@pytest.mark.parametrize('change', ['foreign', 'shared', 'missing_session'])
|
||||
@pytest.mark.parametrize('change', ['foreign', 'shared', 'missing_session', 'unknown_scope'])
|
||||
def test_repair_target_rejects_unresolved_or_shared_database(tmp_path, change):
|
||||
from cbb_refresh_verify.native_repair import select_repair_target
|
||||
request, record, target = target_fixture(tmp_path)
|
||||
@@ -30,6 +30,7 @@ def test_repair_target_rejects_unresolved_or_shared_database(tmp_path, change):
|
||||
if change == 'shared':
|
||||
record.update(scope='central_library', database_path=str(tmp_path/'SymbolLibs/__ReusableBlocksSymb'))
|
||||
if change == 'missing_session': record['session'] = ''
|
||||
if change == 'unknown_scope': record['scope'] = 'unknown'
|
||||
with pytest.raises(ValueError):
|
||||
select_repair_target(request, record)
|
||||
|
||||
@@ -134,12 +135,41 @@ def test_automatic_repair_retries_only_once_and_preserves_original_failure(tmp_p
|
||||
calls.append(req)
|
||||
return original
|
||||
monkeypatch.setattr(runner, '_run_verify_serialized', verify)
|
||||
monkeypatch.setattr(native_repair, 'attempt_recovery', lambda *a, **kw: {'status':'REPAIRED'})
|
||||
monkeypatch.setattr(native_repair, 'attempt_recovery', lambda *a, **kw:
|
||||
{'status':'REPAIRED', 'release': {'safe_to_continue': True}})
|
||||
outcome = runner.run_verify(request)
|
||||
assert len(calls) == 2
|
||||
assert outcome.status == 'FAIL'
|
||||
assert outcome.stderr == 'native lost session'
|
||||
assert outcome.recovery_attempts[0]['original_error'] == 'native lost session'
|
||||
assert outcome.recovery_attempts[0]['status'] == 'REPAIR_VALIDATION_FAILED'
|
||||
assert outcome.recovery_attempts[0]['native_status'] == 'REPAIRED'
|
||||
|
||||
|
||||
@pytest.mark.parametrize('text,expected,terminal', [
|
||||
('There is no backup of iCDB.\nRepair project error [Cannot repair the project - there is no backup of missing iCDB].',
|
||||
'MISSING_HEALTHY_BACKUP', True),
|
||||
('Repair project error [Database validation failed].', 'NATIVE_REPAIR_FAILED', True),
|
||||
('Project is in use. Full functionality is unavailable.', 'DATABASE_BUSY', False),
|
||||
('Repairing the project. Diagnostic: START', 'UNCONFIRMED', False),
|
||||
])
|
||||
def test_current_native_output_classifies_failure_and_terminal_evidence(text, expected, terminal):
|
||||
from cbb_refresh_verify.native_repair_ui import classify_repair_evidence
|
||||
result = classify_repair_evidence('', '', '', text)
|
||||
assert result['outcome'] == expected
|
||||
assert result['failure_complete'] is terminal
|
||||
|
||||
|
||||
def test_historical_no_backup_output_replay_is_conclusive_failure():
|
||||
from cbb_refresh_verify.native_repair_ui import classify_repair_evidence
|
||||
# Sanitized actual native Output from the 2026-10-28 no-history fixture.
|
||||
text = ('Repairing the project [<project>/CBB.prj].\n'
|
||||
'Repair backup [repair.zip] created successfully.\n'
|
||||
'The iCDB database is corrupted.\nThere is no backup of iCDB.\n'
|
||||
'Repair project error [Cannot repair the project - there is no backup of missing iCDB].\n'
|
||||
'The project is probably in inconsistent state. Please restore the project from one of previous backups.')
|
||||
result = classify_repair_evidence('', '', '', text)
|
||||
assert result == {'outcome': 'MISSING_HEALTHY_BACKUP', 'failure_complete': True}
|
||||
|
||||
|
||||
@pytest.mark.parametrize('repair_status', ['BLOCKED', 'RESTORED_REVIEW_REQUIRED'])
|
||||
@@ -264,3 +294,358 @@ def test_backup_detects_a_file_added_during_the_copy(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(native_repair.shutil, 'copy2', changed)
|
||||
with pytest.raises(OSError, match='REPAIR_BACKUP_CHANGED'):
|
||||
native_repair.backup_project(target, tmp_path/'backup')
|
||||
|
||||
|
||||
def repair_worker_fixture(tmp_path, monkeypatch, scenario):
|
||||
"""Replay native window/log transitions without launching a native client."""
|
||||
from types import SimpleNamespace
|
||||
import win32con, win32gui
|
||||
from cbb_refresh_verify import native_repair_ui as ui, native_repair, mentor_adapter
|
||||
request, diagnosis, target = target_fixture(tmp_path)
|
||||
root = tmp_path/'worker'
|
||||
root.mkdir()
|
||||
job = root/'request.json'
|
||||
job.write_text(json.dumps({'project': str(target.project), 'database': str(target.database),
|
||||
'root': str(root), 'cancel_signal': str(root/'cancel.signal'), 'timeout': 1,
|
||||
'supervisor_pid': 123, 'desktop': 'CbbRepair_test'}))
|
||||
ticks = [0.0]
|
||||
state = {'requested': False, 'confirmed': False, 'closed': False}
|
||||
actions = []
|
||||
monkeypatch.setattr(ui, 'time', SimpleNamespace(monotonic=lambda: ticks[0], time_ns=lambda: 0,
|
||||
sleep=lambda seconds: ticks.__setitem__(0, ticks[0]+seconds)))
|
||||
monkeypatch.setattr(ui, '_current_desktop_name', lambda: 'CbbRepair_test')
|
||||
monkeypatch.setattr(ui, '_repair_command', lambda *_: 32924)
|
||||
monkeypatch.setattr(ui, 'discover_sdd_home', lambda *_: tmp_path, raising=False)
|
||||
monkeypatch.setattr('cbb_refresh_verify.mentor_paths.discover_sdd_home', lambda *_: tmp_path)
|
||||
monkeypatch.setattr('cbb_refresh_verify.desktop_worker.start_supervisor_watchdog', lambda *a, **kw: None)
|
||||
monkeypatch.setattr(native_repair, 'backup_project', lambda *a: {})
|
||||
monkeypatch.setattr(mentor_adapter, 'pcb_processes', lambda: [])
|
||||
monkeypatch.setattr(mentor_adapter, '_process_details', lambda *_: [])
|
||||
monkeypatch.setattr(ui.subprocess, 'Popen', lambda *a, **kw: SimpleNamespace(
|
||||
pid=777, poll=lambda: 0 if state['closed'] and ticks[0] >= .6 else None))
|
||||
terminal = 'Repair project error [Cannot repair the project - there is no backup of missing iCDB].'
|
||||
def output(_):
|
||||
if scenario == 'busy_before': return 'Project is in use. Full functionality is unavailable.'
|
||||
if scenario == 'stale': return terminal
|
||||
if not state['confirmed']: return ''
|
||||
if scenario == 'busy': return 'Repair project error [Project is in use].'
|
||||
if scenario == 'active': return 'Repairing the project. Please wait.'
|
||||
if scenario in {'success', 'restore'}: return 'Repair project finished.'
|
||||
return terminal
|
||||
monkeypatch.setattr(ui, '_native_output', output, raising=False)
|
||||
def windows(_):
|
||||
main = {'hwnd': 100, 'class': 'Afx:main', 'title': 'iCDB Project Backup - '+str(target.project),
|
||||
'enabled': True, 'children': []}
|
||||
if state['confirmed'] and scenario in {'unknown', 'custom_prompt'}:
|
||||
return [main, {'hwnd': 300, 'class': 'CustomDialog' if scenario == 'custom_prompt' else '#32770', 'title': 'Unexpected',
|
||||
'enabled': True, 'children': [{'text': 'Overwrite design?', 'class': 'Button',
|
||||
'id': 1, 'hwnd': 301, 'enabled': True}]}]
|
||||
if state['requested'] and not state['confirmed']:
|
||||
return [main, {'hwnd': 200, 'class': '#32770', 'title': '', 'enabled': True,
|
||||
'children': [{'hwnd': 201, 'class': 'Button', 'id': 1, 'enabled': True,
|
||||
'text': 'The project content will be validated and repaired if necessary.\n'
|
||||
'A project backup will be created before any action take place.\n'
|
||||
'You will be able to revert all changes using that backup.'}]}]
|
||||
return [main]
|
||||
monkeypatch.setattr(ui, '_windows', windows)
|
||||
def post(hwnd, message, *args):
|
||||
actions.append((hwnd, message))
|
||||
if message == win32con.WM_COMMAND: state['requested'] = True
|
||||
if message == win32con.BM_CLICK:
|
||||
state['confirmed'] = True
|
||||
if scenario in {'success', 'restore'}:
|
||||
import zipfile
|
||||
log = target.project.parent/'Log Files/ProjectBackupLog.txt'
|
||||
log.parent.mkdir(parents=True)
|
||||
log.write_text('Successfully created [Project Repair] backup [repair.zip]')
|
||||
archive = target.project.parent/'ProjectBackup/backups/repair.zip'
|
||||
archive.parent.mkdir(parents=True)
|
||||
with zipfile.ZipFile(archive, 'w') as zipped: zipped.writestr('database/icdb.dat', b'backup')
|
||||
logs = {'dbcheck': 'Diagnostic: START\nDiagnostic: FINISH',
|
||||
'log': f'{target.database}\niCDBProjectBackupRepair\nIC: Closing project'}
|
||||
if scenario == 'restore':
|
||||
logs['dbrestore'] = 'iCDB database restored from backup: old.zip\n[0] DONE'
|
||||
for folder, text in logs.items():
|
||||
path = target.database/'cdbsvr'/folder/'new-777/log.txt'
|
||||
path.parent.mkdir(parents=True)
|
||||
path.write_text(text)
|
||||
if scenario == 'address':
|
||||
address = target.database/'cdbsvr/sAddress.adr'
|
||||
address.parent.mkdir(parents=True)
|
||||
address.write_text('owner')
|
||||
if message == win32con.WM_CLOSE: state['closed'] = True
|
||||
monkeypatch.setattr(win32gui, 'PostMessage', post)
|
||||
return ui, root, job, actions
|
||||
|
||||
|
||||
@pytest.mark.parametrize('scenario,expected,closed', [
|
||||
('failed', 'MISSING_HEALTHY_BACKUP', True), ('busy', 'DATABASE_BUSY', True),
|
||||
('unknown', 'UNKNOWN_DIALOG', False), ('custom_prompt', 'UNKNOWN_DIALOG', False),
|
||||
('active', 'TIMEOUT_UNRELEASED', False),
|
||||
('stale', 'TIMEOUT_UNRELEASED', False), ('address', 'MISSING_HEALTHY_BACKUP', False),
|
||||
])
|
||||
def test_native_worker_closes_only_conclusively_failed_idle_tool(tmp_path, monkeypatch, scenario, expected, closed):
|
||||
import win32con
|
||||
ui, root, job, actions = repair_worker_fixture(tmp_path, monkeypatch, scenario)
|
||||
assert ui.repair_worker_main(job) == 1
|
||||
result = json.loads((root/'repair-result.json').read_text('utf-8'))
|
||||
assert result['status'] == expected
|
||||
assert result['release']['safe_to_continue'] is closed
|
||||
assert ((100, win32con.WM_CLOSE) in actions) is closed
|
||||
assert actions[:2] == [(100, win32con.WM_COMMAND), (201, win32con.BM_CLICK)]
|
||||
assert bool(result['evidence']['output_tail']) is (scenario != 'stale')
|
||||
if scenario == 'unknown': assert result['unknown_dialogs'][0]['title'] == 'Unexpected'
|
||||
|
||||
|
||||
@pytest.mark.parametrize('scenario,expected', [('success', 'REPAIRED'), ('restore', 'RESTORED_REVIEW_REQUIRED')])
|
||||
def test_native_worker_preserves_success_and_historical_restore_distinction(tmp_path, monkeypatch, scenario, expected):
|
||||
ui, root, job, actions = repair_worker_fixture(tmp_path, monkeypatch, scenario)
|
||||
assert ui.repair_worker_main(job) == (0 if scenario == 'success' else 1)
|
||||
result = json.loads((root/'repair-result.json').read_text('utf-8'))
|
||||
assert result['status'] == expected
|
||||
assert result['release']['safe_to_continue'] is True
|
||||
assert result['evidence']['native_backup'] is True
|
||||
assert result['evidence']['confirmed'] is (scenario == 'success')
|
||||
assert (root/'native-logs/dbcheck/new-777/log.txt').exists()
|
||||
|
||||
|
||||
def test_native_busy_project_on_load_never_invokes_repair(tmp_path, monkeypatch):
|
||||
ui, root, job, actions = repair_worker_fixture(tmp_path, monkeypatch, 'busy_before')
|
||||
assert ui.repair_worker_main(job) == 1
|
||||
result = json.loads((root/'repair-result.json').read_text('utf-8'))
|
||||
assert result['status'] == 'DATABASE_BUSY'
|
||||
assert actions == []
|
||||
assert not result['release']['safe_to_continue']
|
||||
|
||||
|
||||
@pytest.mark.parametrize('status,released,scope,expected', [
|
||||
('MISSING_HEALTHY_BACKUP', True, 'project', True),
|
||||
('NATIVE_REPAIR_FAILED', True, 'project', True),
|
||||
('MISSING_HEALTHY_BACKUP', False, 'project', False),
|
||||
('MISSING_HEALTHY_BACKUP', True, 'central_library', False),
|
||||
('MISSING_HEALTHY_BACKUP', True, 'unknown', False),
|
||||
('UNKNOWN_DIALOG', True, 'project', False), ('TIMEOUT_UNRELEASED', True, 'project', False),
|
||||
])
|
||||
def test_failed_repair_continues_only_released_project_local_items(
|
||||
tmp_path, monkeypatch, status, released, scope, expected):
|
||||
from cbb_refresh_verify import verify_runner as runner, native_repair
|
||||
request, diagnosis, target = target_fixture(tmp_path)
|
||||
diagnosis['scope'] = scope
|
||||
evidence = tmp_path/'first/result.json'
|
||||
evidence.parent.mkdir()
|
||||
(evidence.parent/'diagnostics').mkdir()
|
||||
(evidence.parent/'diagnostics/icdb-recovery.json').write_text(json.dumps(diagnosis))
|
||||
original = runner.VerifyResult('FAIL', stderr='original session lost', safe_to_continue=False,
|
||||
error_category='ICDB_REPAIR_REQUIRED', error_stage='PCB_STARTING', evidence_path=evidence)
|
||||
monkeypatch.setattr(runner, '_run_verify_serialized', lambda *a, **kw: original)
|
||||
monkeypatch.setattr(native_repair, 'attempt_recovery', lambda *a, **kw: {
|
||||
'status': status, 'release': {'safe_to_continue': released},
|
||||
'evidence': {'failure_complete': True}, 'project': str(target.project),
|
||||
'database': str(target.database)})
|
||||
result = runner.run_verify(request)
|
||||
assert result.safe_to_continue is expected
|
||||
assert result.status == 'FAIL'
|
||||
assert result.error_stage == 'PCB_STARTING'
|
||||
assert result.stderr.startswith('original session lost')
|
||||
assert status in result.stderr
|
||||
assert result.recovery_attempts[0]['original_stage'] == 'PCB_STARTING'
|
||||
|
||||
|
||||
@pytest.mark.parametrize('validation_status,flag,expected', [
|
||||
('PASS', 'GREEN', True), ('PASS', 'UNKNOWN', False), ('FAIL', 'GREEN', False),
|
||||
])
|
||||
def test_maintenance_counts_require_native_success_then_verify_green(
|
||||
tmp_path, monkeypatch, validation_status, flag, expected):
|
||||
from cbb_refresh_verify import verify_runner as runner, native_repair
|
||||
from cbb_refresh_verify.icdb_maintenance import maintenance_counts
|
||||
request, diagnosis, target = target_fixture(tmp_path)
|
||||
evidence = tmp_path/'first/result.json'
|
||||
evidence.parent.mkdir()
|
||||
(evidence.parent/'diagnostics').mkdir()
|
||||
(evidence.parent/'diagnostics/icdb-recovery.json').write_text(json.dumps(diagnosis))
|
||||
original = runner.VerifyResult('FAIL', stderr='original lost session', safe_to_continue=False,
|
||||
error_category='ICDB_REPAIR_REQUIRED', evidence_path=evidence)
|
||||
evidence.write_text(json.dumps({'stderr': original.stderr}))
|
||||
validated = runner.VerifyResult(validation_status, library_flag=flag,
|
||||
evidence_path=tmp_path/'second.json', safe_to_continue=True)
|
||||
calls = []
|
||||
def verify(req, **kwargs):
|
||||
calls.append(req)
|
||||
return original if len(calls) == 1 else validated
|
||||
monkeypatch.setattr(runner, '_run_verify_serialized', verify)
|
||||
monkeypatch.setattr(native_repair, 'attempt_recovery', lambda *a, **kw:
|
||||
{'status': 'REPAIRED', 'release': {'safe_to_continue': True}})
|
||||
result = runner.run_verify(request)
|
||||
assert len(calls) == 2 and calls[1].auto_repair is False
|
||||
assert result.recovery_attempts[0]['validation_passed'] is expected
|
||||
assert maintenance_counts([result])['repaired'] == int(expected)
|
||||
assert original.evidence_path.exists(), 'Original result evidence must remain available'
|
||||
assert json.loads(original.evidence_path.read_text('utf-8'))['stderr'] == 'original lost session'
|
||||
|
||||
|
||||
def test_native_success_without_release_does_not_reopen(tmp_path, monkeypatch):
|
||||
from cbb_refresh_verify import verify_runner as runner, native_repair
|
||||
request, diagnosis, target = target_fixture(tmp_path)
|
||||
evidence = tmp_path/'first/result.json'
|
||||
evidence.parent.mkdir()
|
||||
(evidence.parent/'diagnostics').mkdir()
|
||||
(evidence.parent/'diagnostics/icdb-recovery.json').write_text(json.dumps(diagnosis))
|
||||
calls = []
|
||||
def verify(*args, **kwargs):
|
||||
calls.append('open')
|
||||
return runner.VerifyResult('FAIL', error_category='ICDB_REPAIR_REQUIRED',
|
||||
safe_to_continue=False, evidence_path=evidence)
|
||||
monkeypatch.setattr(runner, '_run_verify_serialized', verify)
|
||||
monkeypatch.setattr(native_repair, 'attempt_recovery', lambda *a, **kw: {'status': 'REPAIRED'})
|
||||
result = runner.run_verify(request)
|
||||
assert calls == ['open']
|
||||
assert not result.safe_to_continue
|
||||
assert result.recovery_attempts[0]['status'] == 'REPAIR_RELEASE_UNCONFIRMED'
|
||||
|
||||
|
||||
@pytest.mark.parametrize('obstacle', ['none', 'launcher', 'repair_tool', 'importer', 'address', 'handle', 'shared_handle', 'probe'])
|
||||
def test_repair_release_requires_process_address_and_database_handle_release(tmp_path, monkeypatch, obstacle):
|
||||
import win32con, win32file
|
||||
from cbb_refresh_verify import native_repair, mentor_adapter
|
||||
request, diagnosis, target = target_fixture(tmp_path)
|
||||
monkeypatch.setattr(mentor_adapter, 'pcb_processes', lambda:
|
||||
[(123, 'ExpeditionPCB.exe', 'launcher')] if obstacle == 'launcher' else [])
|
||||
def processes(name):
|
||||
if obstacle == 'probe': raise OSError('process observation unavailable')
|
||||
if ((obstacle == 'repair_tool' and name == 'iCDBProjectBackup.exe')
|
||||
or (obstacle == 'importer' and name == 'ReuseBlockImporter.exe')):
|
||||
return [(456, name, 'unconfirmed owner', 0)]
|
||||
return []
|
||||
monkeypatch.setattr(mentor_adapter, '_process_details', processes)
|
||||
if obstacle == 'address':
|
||||
address = target.database/'cdbsvr/sAddress.adr'
|
||||
address.parent.mkdir(parents=True)
|
||||
address.touch()
|
||||
handle = None
|
||||
try:
|
||||
if obstacle in {'handle', 'shared_handle'}:
|
||||
database_file = target.database/'icdb.dat'
|
||||
if obstacle == 'shared_handle':
|
||||
database_file = request.lmc_file.parent/'SymbolLibs/__ReusableBlocksSymb/icdb.dat'
|
||||
database_file.parent.mkdir(parents=True)
|
||||
database_file.touch()
|
||||
handle = win32file.CreateFile(str(database_file), win32con.GENERIC_READ,
|
||||
0, None, win32con.OPEN_EXISTING, 0, None)
|
||||
result = native_repair.repair_release_state(request, target,
|
||||
{'release': {'safe_to_continue': True}})
|
||||
assert result['safe_to_continue'] is (obstacle == 'none')
|
||||
finally:
|
||||
if handle: handle.Close()
|
||||
|
||||
|
||||
def test_attempt_persists_distinct_native_failure_and_release_result(tmp_path, monkeypatch):
|
||||
from cbb_refresh_verify import native_repair, mentor_adapter
|
||||
request, diagnosis, target = target_fixture(tmp_path)
|
||||
evidence = tmp_path/'evidence'
|
||||
evidence.mkdir()
|
||||
monkeypatch.setattr(mentor_adapter, 'pcb_processes', lambda: [])
|
||||
monkeypatch.setattr(mentor_adapter, '_process_details', lambda *_: [])
|
||||
monkeypatch.setattr(native_repair, 'run_native_repair', lambda *a, **kw: {
|
||||
'status': 'MISSING_HEALTHY_BACKUP', 'project': str(target.project),
|
||||
'database': str(target.database), 'release': {'safe_to_continue': True},
|
||||
'evidence': {'failure_complete': True, 'output_tail': 'no backup'}})
|
||||
result = native_repair.attempt_recovery(request, diagnosis, evidence)
|
||||
assert result['status'] == 'MISSING_HEALTHY_BACKUP'
|
||||
assert result['release']['database_exclusive'] is True
|
||||
assert result['release']['safe_to_continue'] is True
|
||||
assert isinstance(result['evidence'], dict)
|
||||
assert result['evidence_path'] == str(evidence/'native-repair')
|
||||
assert json.loads((evidence/'native-repair/attempt.json').read_text('utf-8')) == result
|
||||
|
||||
|
||||
def test_retry_exception_keeps_original_error_and_native_repair_evidence(tmp_path, monkeypatch):
|
||||
from cbb_refresh_verify import verify_runner as runner, native_repair
|
||||
request, diagnosis, target = target_fixture(tmp_path)
|
||||
evidence = tmp_path/'first/result.json'
|
||||
evidence.parent.mkdir()
|
||||
(evidence.parent/'diagnostics').mkdir()
|
||||
(evidence.parent/'diagnostics/icdb-recovery.json').write_text(json.dumps(diagnosis))
|
||||
calls = []
|
||||
def verify(*a, **kw):
|
||||
calls.append('open')
|
||||
if len(calls) > 1: raise OSError('retry infrastructure failure')
|
||||
return runner.VerifyResult('FAIL', stderr='original native error', safe_to_continue=False,
|
||||
error_category='ICDB_REPAIR_REQUIRED', evidence_path=evidence)
|
||||
monkeypatch.setattr(runner, '_run_verify_serialized', verify)
|
||||
monkeypatch.setattr(native_repair, 'attempt_recovery', lambda *a, **kw: {
|
||||
'status': 'REPAIRED', 'release': {'safe_to_continue': True},
|
||||
'evidence': {'confirmed': True, 'diagnostic_tail': 'original repair evidence'}})
|
||||
result = runner.run_verify(request)
|
||||
assert not result.safe_to_continue
|
||||
assert 'original native error' in result.stderr
|
||||
assert 'retry infrastructure failure' in result.stderr
|
||||
assert result.recovery_attempts[0]['evidence']['diagnostic_tail'] == 'original repair evidence'
|
||||
assert result.recovery_attempts[0]['status'] == 'REPAIR_VALIDATION_FAILED'
|
||||
assert json.loads(evidence.read_text('utf-8'))['recovery_attempts']
|
||||
|
||||
|
||||
def test_sanitized_field_open_failure_replay_closes_launcher_then_repairs_once(tmp_path, monkeypatch):
|
||||
from types import SimpleNamespace
|
||||
from dataclasses import replace
|
||||
from test_failed_startup import startup_fixture
|
||||
from cbb_refresh_verify import verify_runner as runner, native_repair, failed_session, mentor_adapter
|
||||
from cbb_refresh_verify.pcb_bridge import wait_for_target_document, PcbBridgeError
|
||||
replay = json.loads((Path(__file__).parent/'fixtures/issue2-lost-session-sanitized.json').read_text('utf-8'))
|
||||
request, diagnosis, target = target_fixture(tmp_path)
|
||||
# Retain a different PRJ name: the field CBB name never authorizes guessing it.
|
||||
block_root = target.project.parent.parent
|
||||
block_root.rename(block_root.with_name(replay['block']))
|
||||
request = replace(request, block_name=replay['block'])
|
||||
project, pcb = runner.preflight(request)
|
||||
diagnosis['database_path'] = str(project.parent/'database')
|
||||
target = native_repair.select_repair_target(request, diagnosis)
|
||||
job, records, messages = startup_fixture(tmp_path, monkeypatch)
|
||||
job.update(project=str(project), pcb=str(pcb), lmc_file=str(request.lmc_file))
|
||||
records[0] = (12, records[0][1], f'ExpeditionPCB.exe "{pcb}" -ReusableCircuit', 11)
|
||||
diagnosis.update(session=replay['diagnosis']['session'], machine='FIELD-PC')
|
||||
(tmp_path/'diagnostics/icdb-recovery.json').write_text(json.dumps(diagnosis))
|
||||
(tmp_path/'dialogs.jsonl').write_text(json.dumps(replay['dialog'])+'\n')
|
||||
ticks = [0.0]
|
||||
monkeypatch.setattr(failed_session, 'time', SimpleNamespace(monotonic=lambda: ticks[0],
|
||||
sleep=lambda seconds: ticks.__setitem__(0, ticks[0]+seconds)))
|
||||
launcher = SimpleNamespace(pid=11, poll=lambda: 0 if ticks[0] >= .4 else None)
|
||||
monkeypatch.setattr(mentor_adapter, 'pcb_processes', lambda: [])
|
||||
trace = []
|
||||
evidence = tmp_path/'result.json'
|
||||
original = runner.VerifyResult('FAIL', stderr=replay['native_error'], safe_to_continue=False,
|
||||
error_category=replay['original_result']['error_category'],
|
||||
error_stage=replay['original_result']['error_stage'], evidence_path=evidence)
|
||||
assert runner.classify_failure_category(original.stderr) == original.error_category
|
||||
assert runner.phase_for_event(replay['observed_events'][2]).value == original.error_stage
|
||||
opens = []
|
||||
def verify(req, **kwargs):
|
||||
opens.append(req)
|
||||
trace.append('open')
|
||||
if len(opens) == 1:
|
||||
errors = iter(('', replay['native_error']))
|
||||
with pytest.raises(PcbBridgeError, match='Repair project'):
|
||||
wait_for_target_document(job, lambda: SimpleNamespace(ActiveDocument=None),
|
||||
process_alive=lambda: True, error_reader=lambda: next(errors),
|
||||
emit=lambda **e: trace.append(e['stage']), sleep=lambda _: None)
|
||||
closed = failed_session.close_failed_startup(job, 12, launcher_process=launcher,
|
||||
emit=lambda **e: trace.append(e['stage']), timeout_seconds=1)
|
||||
assert closed['status'] == 'CLOSED'
|
||||
trace.append('launcher_exited')
|
||||
return original # Repeated field error after one repair must not loop.
|
||||
def repair(actual, root, **kwargs):
|
||||
assert actual == target
|
||||
assert trace[-1] == 'launcher_exited'
|
||||
assert not records and launcher.poll() == 0
|
||||
trace.append('native_repair')
|
||||
return {'status': 'REPAIRED', 'project': str(actual.project), 'database': str(actual.database),
|
||||
'release': {'safe_to_continue': True}, 'evidence': {'confirmed': True}}
|
||||
monkeypatch.setattr(runner, '_run_verify_serialized', verify)
|
||||
monkeypatch.setattr(native_repair, 'run_native_repair', repair)
|
||||
result = runner.run_verify(request)
|
||||
assert trace == ['open', 'pcb_document_wait', 'startup_close_requested', 'startup_close_result',
|
||||
'launcher_exited', 'native_repair', 'open']
|
||||
assert len(opens) == 2 and opens[1].auto_repair is False
|
||||
assert result.error_stage == 'PCB_STARTING'
|
||||
assert result.recovery_attempts[0]['status'] == 'REPAIR_VALIDATION_FAILED'
|
||||
assert result.recovery_attempts[0]['original_error'] == replay['native_error']
|
||||
assert json.loads((tmp_path/'pre-repair-result.json').read_text('utf-8'))['stderr'] == replay['native_error']
|
||||
@@ -89,3 +89,15 @@ def test_success_requires_save_and_pcb_exit():
|
||||
assert execution_outcome(save_complete=True, pcb_exited=True)['status'] == 'PASS'
|
||||
assert execution_outcome(save_complete=False, pcb_exited=True)['status'] == 'FAIL'
|
||||
assert execution_outcome(save_complete=True, pcb_exited=False)['status'] == 'FAIL'
|
||||
|
||||
|
||||
def test_connected_com_without_document_remains_startup_and_checks_native_failure(tmp_path):
|
||||
errors = iter(('', 'ICDB_REPAIR_REQUIRED: original lost session'))
|
||||
events = []
|
||||
with pytest.raises(PcbBridgeError) as raised:
|
||||
wait_for_target_document(job(tmp_path), lambda: Application(None), process_alive=lambda: True,
|
||||
error_reader=lambda: next(errors), emit=lambda **e: events.append(e), sleep=lambda _: None)
|
||||
assert raised.value.error_code == 'PCB_STARTUP_FAILED'
|
||||
assert 'original lost session' in str(raised.value)
|
||||
assert [e['stage'] for e in events] == ['pcb_document_wait']
|
||||
assert events[0]['error_code'] == 'PCB_DOCUMENT_NOT_READY'
|
||||
Reference in new issue
Block a user