fix: preserve engineering identity in scoped runtime exclusions

This commit is contained in:
阿白叔 committed 2026-11-02 19:50:26 +08:00
1 parent 29c154d2ca
commit 3415e010ff
4 files changed
+288 -12

No files matched your search

@@ -1,6 +1,6 @@
# Task 3 report: scoped primary manifests and complete verified backups
# Scoped recovery manifests and complete verified backups
Status: implementation and verification complete. Started on b90a9bb; paused at
Status: implemented; review finding P1 addressed with focused validation. Started on b90a9bb; paused at
the RED checkpoint for the coordinator's native startup-window repair fix and
resumed on 877e3b4. No native UI/process/live-library actions were performed by
this implementer. Original untracked review.diff was preserved and not staged.
@@ -19,6 +19,10 @@ incoming Work preparation, immediately before target equality, closing the old
late-source-change window. Source idle/address ownership is rechecked there as
well: excluding address bytes from semantic hashes does not bypass idle checks.
Initial implementation: 29c154d. The review correction is described below.
Durable documentation lives here; an ignored review copy remains at
.superpowers/sdd/2026-11-02-issue-2-management/task-3-report.md.
Scoped production files: backup_recovery.py and new scoped_manifest.py. Scoped
tests: test_backup_recovery.py and new test_scoped_manifest.py. No Verify public
types, Task 2 helpers/READY logic, native repair classifiers/process whitelist,
@@ -33,12 +37,33 @@ categories. Rules are case-insensitive; manifests preserve relative path case.
Explicit semantic exclusions, still copied and metadata-validated:
- Directory components: log, logs, temp, tmp, __pycache__, backup, backups,
output, outputs, report, reports, gerber, gerbers, odb++, and historical
Work.before-restore-* directory components.
- Runtime/history child directories **at a project workspace root**, its PCB
workspace root, a selected Work/Cert root, or the library root: log, logs,
temp, tmp, __pycache__, backup, backups, output, outputs, report, reports,
gerber, gerbers, odb++, Log Files, LogFiles, and Work.before-restore-*.
- ProjectBackup/backups only at those workspace roots. Other ProjectBackup
content remains conservative primary data.
- Known native runtime below database/cdbsvr and PCB/database/cdbsvr: log,
dbcheck and dbrestore subtrees. In shared library databases, those same
subtrees below SymbolLibs/<partition>/cdbsvr or <shared-root>/cdbsvr. Other
unknown files below cdbsvr remain primary; the directory is not dropped.
- File suffixes: .log, .tmp, .temp, .lck, .lock, .bak, .backup, .pdf.
- File names: sAddress.adr and editor files starting with ~$.
Directory exclusions are contextual rather than arbitrary ancestor matching.
ReusableBlocksLP/<selected-block>/Work and Cert are structural identity
components, never runtime labels. CBB names Logs, Temp, Backup, Output, Report,
Log Files, ProjectBackup and Work.before-restore-engineering remain valid and
their primary files are hashed in both semantic manifests and complete-copy
validation. Case variants have the same protection.
Shared dependency partitions and nested engineering folders keep their names:
SymbolLibs/temp/icdb.dat, SymbolLibs/Temp_a/device.sym,
SymbolLibs/Backup/device.sym, hdl/Logs/design.v and hdl/Backup/design.v are
primary. Runtime directory labels apply at the known roots above, not inside
arbitrary engineering/dependency hierarchies. File suffix/name exclusions are
applied to files, not to engineering directories with matching suffixes.
The uploader-compatible .<stem>.cbb-upload.lock falls under .lock, so it is
runtime and never primary. It is **not** silently removed from complete copies.
@@ -137,6 +162,10 @@ added. Empty directory metadata is intentionally not part of semantic equality.
## RED/GREEN and verification evidence
The original full-suite evidence below belongs to initial commit 29c154d. The
P1 review correction has fresh focused validation, and has **not** rerun the
full suite; the coordinator requested focused tests only for this fix.
1. Initial RED, before production edits:
`python -m pytest tests/test_scoped_manifest.py tests/test_backup_recovery.py -q`
-> **29 failed, 26 passed in 7.89s**. Failures demonstrated missing versioned
@@ -167,6 +196,41 @@ source content changes with preserved metadata, complete copy membership and
nonprimary size consistency, idle/address rechecks, retained copies on failure,
and unchanged native reopen/retained-Work recovery behavior.
## Review correction: contextual runtime directory classification
Review found that the initial unconditional directory ancestor rules could
classify valid selected CBB names (Logs/Temp/Backup) or real SymbolLibs/temp
engineering partitions as runtime. That removed primary paths from both the
baseline manifest and later conflict comparison, and from copy byte checks.
The classifier now resolves the structural role first and only applies the
runtime directory rules at documented workspace/native DB locations. Semantic
directory traversal and complete-copy byte classification share that rule.
Complete tree copy/metadata validation remains unchanged. This also protects
nested engineering folders with names Logs/Backup, and avoids broad dropping
of unknown cdbsvr files.
Independent regressions cover Work and Cert under 11 colliding CBB names/case
variants, real SymbolLibs/temp and Temp_a partitions, arbitrary nested
engineering folders, source and destination corruption with unchanged size/
mtime, independent locked database destination corruption, and end-to-end
recovery conflicts that previously overwrote concurrent target/shared edits.
Known native Log Files, ProjectBackup/backups and DB server log/dbcheck/
dbrestore data remain semantically ignored and completely copied.
Review-fix RED:
`python -m pytest tests/test_scoped_manifest.py tests/test_backup_recovery.py -q --tb=short`
-> **47 failed, 74 passed in 18.41s**, before production changes. The failures
included silently accepted primary corruption and installs over concurrent
target/shared edits. Additional conservative-boundary regressions prevented
reintroducing the same bug through Log Files/ProjectBackup/retained-Work names
or broad cdbsvr directory exclusion.
Final review-fix focused GREEN:
`python -m pytest tests/test_scoped_manifest.py tests/test_backup_recovery.py tests/test_backup_recovery_dialog.py -q --tb=short`
-> **141 passed in 18.28s**. No native UI/process actions, live-library mutation,
or full-suite run was performed during this fix round.
## Limitations / follow-up boundary
- No real field lost-session reproduction, native GUI success reproduction, or
+53 -7
View File
@@ -18,12 +18,35 @@ MANIFEST_VERSION = 1
_NONPRIMARY_DIRECTORIES = frozenset({
'log', 'logs', 'temp', 'tmp', '__pycache__', 'backup', 'backups',
'output', 'outputs', 'report', 'reports', 'gerber', 'gerbers', 'odb++',
'log files', 'logfiles',
})
_NONPRIMARY_SUFFIXES = frozenset({
'.log', '.tmp', '.temp', '.lck', '.lock', '.bak', '.backup', '.pdf',
})
def _root_runtime(directories):
"""Known runtime/history children of a library or project workspace root."""
return bool(directories and (
directories[0] in _NONPRIMARY_DIRECTORIES
or directories[0].startswith('work.before-restore-')
or directories[:2] == ('projectbackup', 'backups')))
def _native_runtime(directories, database_prefix):
runtime_prefix = database_prefix + ('cdbsvr',)
return (directories[:len(runtime_prefix)] == runtime_prefix
and len(directories) > len(runtime_prefix)
and directories[len(runtime_prefix)] in ('log', 'dbcheck', 'dbrestore'))
def _project_runtime(directories):
return (_root_runtime(directories)
or (directories[:1] == ('pcb',) and _root_runtime(directories[1:]))
or _native_runtime(directories, ('database',))
or _native_runtime(directories, ('pcb', 'database')))
@dataclass(frozen=True)
class ManifestScope:
"""Project root by default; library selects one CBB and shared dependencies."""
@@ -46,21 +69,44 @@ class ManifestScope:
def includes_directory(self, relative):
parts = tuple(part.casefold() for part in Path(relative).parts)
if any(part in _NONPRIMARY_DIRECTORIES or part.startswith('work.before-restore-')
for part in parts):
if self._runtime_path(parts, directory=True):
return False
if self.kind == 'project' or parts[0] != 'reusableblockslp':
if self.kind == 'project' or not parts or parts[0] != 'reusableblockslp':
return True
return ((len(parts) < 2 or parts[1] == self.block_name.casefold())
and (len(parts) < 3 or parts[2] in ('work', 'cert')))
def _runtime_path(self, parts, *, directory=False):
"""Recognize runtime locations only in their structural context.
Generic names such as ``Logs`` are runtime when they are a project-root
or selected Work/Cert child. The same name deeper in an engineering
partition (or in shared SymbolLibs data) remains primary.
"""
directories = parts if directory else parts[:-1]
if not directory and (Path(parts[-1]).suffix in _NONPRIMARY_SUFFIXES
or parts[-1] == 'saddress.adr' or parts[-1].startswith('~$')):
return True
if self.kind == 'project':
return _project_runtime(directories)
if directories[:1] == ('reusableblockslp',):
# Identity and Work/Cert role components never act as runtime names.
if (len(directories) >= 3 and directories[1] == self.block_name.casefold()
and directories[2] in ('work', 'cert')):
return _project_runtime(directories[3:])
return False
if _root_runtime(directories):
return True
# Shared dependency names/partitions are engineering data. Only native
# server runtime subtrees within known DB positions are excluded.
if directories[:1] == ('symbollibs',) and len(directories) >= 2:
return _native_runtime(directories, directories[:2])
return bool(directories and _native_runtime(directories, directories[:1]))
def classify(self, relative):
"""Return a semantic scope label or None; paths are relative to the root."""
parts = tuple(part.casefold() for part in Path(relative).parts)
if (any(part in _NONPRIMARY_DIRECTORIES or part.startswith('work.before-restore-')
for part in parts[:-1])
or Path(parts[-1]).suffix in _NONPRIMARY_SUFFIXES
or parts[-1] == 'saddress.adr' or parts[-1].startswith('~$')):
if self._runtime_path(parts):
return None
if self.kind == 'project':
return 'project'
+46
View File
@@ -280,3 +280,49 @@ def test_source_address_record_is_rechecked_after_validation(recovery_files):
assert 'ownership' in result.stderr
assert address.read_bytes() == b'live owner'
assert (original.parent/'database/icdb.dat').read_bytes() == b'damaged'
@pytest.mark.parametrize('name', ['Logs', 'tEmP', 'bAcKuP', 'Output'])
@pytest.mark.parametrize('role', ['Work', 'Cert'])
def test_cbb_names_matching_runtime_labels_do_not_hide_target_conflicts(tmp_path, monkeypatch, name, role):
from cbb_refresh_verify.backup_recovery import recover_from_backup
monkeypatch.setattr(Path, 'home', lambda: tmp_path/'profile')
library = tmp_path/'target/target.lmc'
original = project(library.parent/f'ReusableBlocksLP/{name}/Work', name, b'damaged')
library.write_bytes(b'library')
source = project(tmp_path/'backup/Work', name, b'healthy')
request = VerifyRequest(library, name, tmp_path/'evidence')
conflict = original.parent.parent/role/'database/icdb.dat'
conflict.parent.mkdir(parents=True, exist_ok=True)
if role == 'Cert':
conflict.write_bytes(b'cert before')
calls = []
def verify(*args, **kwargs):
calls.append(args)
conflict.write_bytes(b'concurrent target edit')
return VerifyResult('PASS', library_flag='GREEN')
result = recover_from_backup(request, source, verifier=verify, idle_check=lambda *_: None)
assert result.status == 'BLOCKED'
assert len(calls) == 1
assert conflict.read_bytes() == b'concurrent target edit'
assert not list(original.parent.parent.glob('Work.before-restore-*'))
@pytest.mark.parametrize('relative', ['SymbolLibs/temp/icdb.dat', 'SymbolLibs/Temp_a/device.sym',
'SharedDependency/hdl/Logs/design.v'])
def test_shared_partition_engineering_conflicts_block_install(recovery_files, relative):
from cbb_refresh_verify.backup_recovery import recover_from_backup
request, original, source = recovery_files
conflict = request.lmc_file.parent/relative
conflict.parent.mkdir(parents=True, exist_ok=True)
conflict.write_bytes(b'shared before')
calls = []
def verify(*args, **kwargs):
calls.append(args)
conflict.write_bytes(b'concurrent shared edit')
return VerifyResult('PASS', library_flag='GREEN')
result = recover_from_backup(request, source, verifier=verify, idle_check=lambda *_: None)
assert result.status == 'BLOCKED'
assert len(calls) == 1
assert (original.parent/'database/icdb.dat').read_bytes() == b'damaged'
assert conflict.read_bytes() == b'concurrent shared edit'
+120
View File
@@ -213,3 +213,123 @@ def test_only_primary_paths_are_read_for_sha256_and_manifest_is_relocatable(tmp_
assert recovery._manifest(source, scope=scope) == manifest
assert recovery._manifest(destination, scope=scope) == manifest
assert {p.relative_to(destination).as_posix() for p in destination.rglob('*') if p.is_file()} == primary | excluded
@pytest.mark.parametrize('block', ['Logs', 'lOgS', 'Temp', 'tEmP', 'Backup', 'bAcKuP', 'Output', 'rEpOrT',
'Log Files', 'ProjectBackup', 'Work.before-restore-engineering'])
@pytest.mark.parametrize('role', ['Work', 'Cert'])
def test_structural_cbb_identity_is_primary_in_semantic_and_copy_manifests(tmp_path, block, role):
from cbb_refresh_verify.scoped_manifest import ManifestScope
scope = ManifestScope.library(block, 'target.lmc')
source, destination = tmp_path/'source', tmp_path/'destination'
relative = f'ReusableBlocksLP/{block}/{role}/database/icdb.dat'
primary = put(source, relative)
runtime = f'ReusableBlocksLP/{block}/{role}/logs/session.txt'
put(source, runtime)
before = recovery._copy_verified(source, destination, scope=scope)
assert set(before['files']) == {relative}
assert (destination/runtime).read_bytes() == b'engineering'
assert recovery._manifest(source, scope=scope) == before
primary.write_bytes(b'concurrent primary edit')
assert recovery._manifest(source, scope=scope) != before
@pytest.mark.parametrize('relative', [
'SymbolLibs/temp/icdb.dat', 'SymbolLibs/Temp_a/device.sym',
'SymbolLibs/Logs/device.sym', 'SymbolLibs/BACKUP/device.sym',
'SymbolLibs/temp/cdbsvr/persistent.bin',
'ReusableBlocksLP/A/Work/hdl/Logs/design.v',
'ReusableBlocksLP/A/Cert/hdl/bAcKuP/design.v',
'SymbolLibs/temp/hdl/Backup/design.v', 'SharedDependency/hdl/Logs/design.v',
])
def test_library_engineering_partition_names_are_not_runtime_categories(tmp_path, relative):
from cbb_refresh_verify.scoped_manifest import ManifestScope
scope = ManifestScope.library('A', 'target.lmc')
source, destination = tmp_path/'source', tmp_path/'destination'
primary = put(source, relative)
before = recovery._copy_verified(source, destination, scope=scope)
assert relative in before['files']
assert recovery._manifest(source, scope=scope) == before
primary.write_bytes(b'concurrent shared/design edit')
assert recovery._manifest(source, scope=scope) != before
@pytest.mark.parametrize('relative', ['hdl/Logs/design.v', 'hdl/Backup/design.v', 'symbols/Temp/device.sym',
'hdl/Log Files/design.v', 'hdl/ProjectBackup/backups/design.v',
'hdl/Work.before-restore-engineering/design.v', 'database/cdbsvr/persistent.bin'])
def test_nested_project_engineering_names_remain_primary(tmp_path, relative):
primary = put(tmp_path, relative)
before = recovery._manifest(tmp_path)
assert relative in before['files']
primary.unlink()
assert recovery._manifest(tmp_path) != before
@pytest.mark.parametrize('relative', [
'ReusableBlocksLP/Logs/Work/Logs.prj',
'ReusableBlocksLP/Backup/Cert/Backup.prj',
'SymbolLibs/temp/device.sym', 'SharedDependency/hdl/Logs/design.v',
])
@pytest.mark.parametrize('side', ['source', 'destination'])
def test_colliding_engineering_names_cannot_bypass_primary_copy_byte_checks(tmp_path, monkeypatch, relative, side):
from cbb_refresh_verify.scoped_manifest import ManifestScope
block = relative.split('/')[1] if relative.startswith('ReusableBlocksLP/') else 'A'
scope = ManifestScope.library(block, 'target.lmc')
source, destination = tmp_path/'source', tmp_path/'destination'
put(source, relative)
real_copy = shutil.copy2
def copy_then_corrupt(src, dst, *args, **kwargs):
result = real_copy(src, dst, *args, **kwargs)
corrupt = Path(src) if side == 'source' else Path(dst)
metadata = corrupt.stat()
corrupt.write_bytes(b'X' * metadata.st_size)
os.utime(corrupt, ns=(metadata.st_atime_ns, metadata.st_mtime_ns))
return result
monkeypatch.setattr(shutil, 'copy2', copy_then_corrupt)
with pytest.raises(OSError):
recovery._copy_verified(source, destination, scope=scope)
assert (destination/relative).exists()
@pytest.mark.parametrize('prefix', ['', 'ReusableBlocksLP/Logs/Work/', 'ReusableBlocksLP/Temp/Cert/'])
@pytest.mark.parametrize('relative', [
'Log Files/ProjectBackupLog.txt', 'ProjectBackup/backups/old.zip',
'database/cdbsvr/log/session/text.txt', 'database/cdbsvr/dbcheck/session/text.txt',
'PCB/database/cdbsvr/dbrestore/session/text.txt',
])
def test_known_native_runtime_and_history_locations_are_ignored_contextually(tmp_path, prefix, relative):
from cbb_refresh_verify.scoped_manifest import ManifestScope
scope = ManifestScope.library(prefix.split('/')[1], 'target.lmc') if prefix else None
runtime = put(tmp_path, prefix + relative)
before = recovery._manifest(tmp_path, scope=scope)
assert not before['files']
runtime.write_bytes(b'updated runtime')
assert recovery._manifest(tmp_path, scope=scope) == before
@pytest.mark.parametrize('relative', ['SymbolLibs/temp/icdb.dat',
'ReusableBlocksLP/Logs/Work/database/icdb.dat'])
def test_locked_critical_bytes_in_colliding_names_are_checked_independently(tmp_path, relative):
from cbb_refresh_verify.scoped_manifest import ManifestScope
source, destination = tmp_path/'source', tmp_path/'destination'
put(source, relative)
def corrupt_after_copy(event):
if event['copied_files'] == event['total_files']:
copied = destination/relative
metadata = copied.stat()
copied.write_bytes(b'X' * metadata.st_size)
os.utime(copied, ns=(metadata.st_atime_ns, metadata.st_mtime_ns))
with pytest.raises(OSError):
recovery._copy_verified(source, destination,
scope=ManifestScope.library('Logs', 'target.lmc'), progress=corrupt_after_copy)
assert (source/relative).read_bytes() == b'engineering'
assert (destination/relative).read_bytes() == b'XXXXXXXXXXX'
@pytest.mark.parametrize('relative', ['SymbolLibs/temp/cdbsvr/log/session/text.txt',
'SymbolLibs/Temp_a/cdbsvr/dbcheck/session/text.txt',
'CellDB/cdbsvr/dbrestore/session/text.txt'])
def test_shared_native_runtime_has_explicit_database_context(tmp_path, relative):
from cbb_refresh_verify.scoped_manifest import ManifestScope
put(tmp_path, relative)
assert not recovery._manifest(tmp_path, scope=ManifestScope.library('A', 'target.lmc'))['files']