fix: preserve engineering identity in scoped runtime exclusions
This commit is contained in:
1 parent
29c154d2ca
commit
3415e010ff
4 files changed
+288
-12
No files matched your search
+69
-5
@@ -1,6 +1,6 @@
|
||||
# Task 3 report: scoped primary manifests and complete verified backups
|
||||
# Scoped recovery manifests and complete verified backups
|
||||
|
||||
Status: implementation and verification complete. Started on b90a9bb; paused at
|
||||
Status: implemented; review finding P1 addressed with focused validation. Started on b90a9bb; paused at
|
||||
the RED checkpoint for the coordinator's native startup-window repair fix and
|
||||
resumed on 877e3b4. No native UI/process/live-library actions were performed by
|
||||
this implementer. Original untracked review.diff was preserved and not staged.
|
||||
@@ -19,6 +19,10 @@ incoming Work preparation, immediately before target equality, closing the old
|
||||
late-source-change window. Source idle/address ownership is rechecked there as
|
||||
well: excluding address bytes from semantic hashes does not bypass idle checks.
|
||||
|
||||
Initial implementation: 29c154d. The review correction is described below.
|
||||
Durable documentation lives here; an ignored review copy remains at
|
||||
.superpowers/sdd/2026-11-02-issue-2-management/task-3-report.md.
|
||||
|
||||
Scoped production files: backup_recovery.py and new scoped_manifest.py. Scoped
|
||||
tests: test_backup_recovery.py and new test_scoped_manifest.py. No Verify public
|
||||
types, Task 2 helpers/READY logic, native repair classifiers/process whitelist,
|
||||
@@ -33,12 +37,33 @@ categories. Rules are case-insensitive; manifests preserve relative path case.
|
||||
|
||||
Explicit semantic exclusions, still copied and metadata-validated:
|
||||
|
||||
- Directory components: log, logs, temp, tmp, __pycache__, backup, backups,
|
||||
output, outputs, report, reports, gerber, gerbers, odb++, and historical
|
||||
Work.before-restore-* directory components.
|
||||
- Runtime/history child directories **at a project workspace root**, its PCB
|
||||
workspace root, a selected Work/Cert root, or the library root: log, logs,
|
||||
temp, tmp, __pycache__, backup, backups, output, outputs, report, reports,
|
||||
gerber, gerbers, odb++, Log Files, LogFiles, and Work.before-restore-*.
|
||||
- ProjectBackup/backups only at those workspace roots. Other ProjectBackup
|
||||
content remains conservative primary data.
|
||||
- Known native runtime below database/cdbsvr and PCB/database/cdbsvr: log,
|
||||
dbcheck and dbrestore subtrees. In shared library databases, those same
|
||||
subtrees below SymbolLibs/<partition>/cdbsvr or <shared-root>/cdbsvr. Other
|
||||
unknown files below cdbsvr remain primary; the directory is not dropped.
|
||||
- File suffixes: .log, .tmp, .temp, .lck, .lock, .bak, .backup, .pdf.
|
||||
- File names: sAddress.adr and editor files starting with ~$.
|
||||
|
||||
Directory exclusions are contextual rather than arbitrary ancestor matching.
|
||||
ReusableBlocksLP/<selected-block>/Work and Cert are structural identity
|
||||
components, never runtime labels. CBB names Logs, Temp, Backup, Output, Report,
|
||||
Log Files, ProjectBackup and Work.before-restore-engineering remain valid and
|
||||
their primary files are hashed in both semantic manifests and complete-copy
|
||||
validation. Case variants have the same protection.
|
||||
|
||||
Shared dependency partitions and nested engineering folders keep their names:
|
||||
SymbolLibs/temp/icdb.dat, SymbolLibs/Temp_a/device.sym,
|
||||
SymbolLibs/Backup/device.sym, hdl/Logs/design.v and hdl/Backup/design.v are
|
||||
primary. Runtime directory labels apply at the known roots above, not inside
|
||||
arbitrary engineering/dependency hierarchies. File suffix/name exclusions are
|
||||
applied to files, not to engineering directories with matching suffixes.
|
||||
|
||||
The uploader-compatible .<stem>.cbb-upload.lock falls under .lock, so it is
|
||||
runtime and never primary. It is **not** silently removed from complete copies.
|
||||
|
||||
@@ -137,6 +162,10 @@ added. Empty directory metadata is intentionally not part of semantic equality.
|
||||
|
||||
## RED/GREEN and verification evidence
|
||||
|
||||
The original full-suite evidence below belongs to initial commit 29c154d. The
|
||||
P1 review correction has fresh focused validation, and has **not** rerun the
|
||||
full suite; the coordinator requested focused tests only for this fix.
|
||||
|
||||
1. Initial RED, before production edits:
|
||||
`python -m pytest tests/test_scoped_manifest.py tests/test_backup_recovery.py -q`
|
||||
-> **29 failed, 26 passed in 7.89s**. Failures demonstrated missing versioned
|
||||
@@ -167,6 +196,41 @@ source content changes with preserved metadata, complete copy membership and
|
||||
nonprimary size consistency, idle/address rechecks, retained copies on failure,
|
||||
and unchanged native reopen/retained-Work recovery behavior.
|
||||
|
||||
## Review correction: contextual runtime directory classification
|
||||
|
||||
Review found that the initial unconditional directory ancestor rules could
|
||||
classify valid selected CBB names (Logs/Temp/Backup) or real SymbolLibs/temp
|
||||
engineering partitions as runtime. That removed primary paths from both the
|
||||
baseline manifest and later conflict comparison, and from copy byte checks.
|
||||
|
||||
The classifier now resolves the structural role first and only applies the
|
||||
runtime directory rules at documented workspace/native DB locations. Semantic
|
||||
directory traversal and complete-copy byte classification share that rule.
|
||||
Complete tree copy/metadata validation remains unchanged. This also protects
|
||||
nested engineering folders with names Logs/Backup, and avoids broad dropping
|
||||
of unknown cdbsvr files.
|
||||
|
||||
Independent regressions cover Work and Cert under 11 colliding CBB names/case
|
||||
variants, real SymbolLibs/temp and Temp_a partitions, arbitrary nested
|
||||
engineering folders, source and destination corruption with unchanged size/
|
||||
mtime, independent locked database destination corruption, and end-to-end
|
||||
recovery conflicts that previously overwrote concurrent target/shared edits.
|
||||
Known native Log Files, ProjectBackup/backups and DB server log/dbcheck/
|
||||
dbrestore data remain semantically ignored and completely copied.
|
||||
|
||||
Review-fix RED:
|
||||
`python -m pytest tests/test_scoped_manifest.py tests/test_backup_recovery.py -q --tb=short`
|
||||
-> **47 failed, 74 passed in 18.41s**, before production changes. The failures
|
||||
included silently accepted primary corruption and installs over concurrent
|
||||
target/shared edits. Additional conservative-boundary regressions prevented
|
||||
reintroducing the same bug through Log Files/ProjectBackup/retained-Work names
|
||||
or broad cdbsvr directory exclusion.
|
||||
|
||||
Final review-fix focused GREEN:
|
||||
`python -m pytest tests/test_scoped_manifest.py tests/test_backup_recovery.py tests/test_backup_recovery_dialog.py -q --tb=short`
|
||||
-> **141 passed in 18.28s**. No native UI/process actions, live-library mutation,
|
||||
or full-suite run was performed during this fix round.
|
||||
|
||||
## Limitations / follow-up boundary
|
||||
|
||||
- No real field lost-session reproduction, native GUI success reproduction, or
|
||||
@@ -18,12 +18,35 @@ MANIFEST_VERSION = 1
|
||||
_NONPRIMARY_DIRECTORIES = frozenset({
|
||||
'log', 'logs', 'temp', 'tmp', '__pycache__', 'backup', 'backups',
|
||||
'output', 'outputs', 'report', 'reports', 'gerber', 'gerbers', 'odb++',
|
||||
'log files', 'logfiles',
|
||||
})
|
||||
_NONPRIMARY_SUFFIXES = frozenset({
|
||||
'.log', '.tmp', '.temp', '.lck', '.lock', '.bak', '.backup', '.pdf',
|
||||
})
|
||||
|
||||
|
||||
def _root_runtime(directories):
|
||||
"""Known runtime/history children of a library or project workspace root."""
|
||||
return bool(directories and (
|
||||
directories[0] in _NONPRIMARY_DIRECTORIES
|
||||
or directories[0].startswith('work.before-restore-')
|
||||
or directories[:2] == ('projectbackup', 'backups')))
|
||||
|
||||
|
||||
def _native_runtime(directories, database_prefix):
|
||||
runtime_prefix = database_prefix + ('cdbsvr',)
|
||||
return (directories[:len(runtime_prefix)] == runtime_prefix
|
||||
and len(directories) > len(runtime_prefix)
|
||||
and directories[len(runtime_prefix)] in ('log', 'dbcheck', 'dbrestore'))
|
||||
|
||||
|
||||
def _project_runtime(directories):
|
||||
return (_root_runtime(directories)
|
||||
or (directories[:1] == ('pcb',) and _root_runtime(directories[1:]))
|
||||
or _native_runtime(directories, ('database',))
|
||||
or _native_runtime(directories, ('pcb', 'database')))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ManifestScope:
|
||||
"""Project root by default; library selects one CBB and shared dependencies."""
|
||||
@@ -46,21 +69,44 @@ class ManifestScope:
|
||||
|
||||
def includes_directory(self, relative):
|
||||
parts = tuple(part.casefold() for part in Path(relative).parts)
|
||||
if any(part in _NONPRIMARY_DIRECTORIES or part.startswith('work.before-restore-')
|
||||
for part in parts):
|
||||
if self._runtime_path(parts, directory=True):
|
||||
return False
|
||||
if self.kind == 'project' or parts[0] != 'reusableblockslp':
|
||||
if self.kind == 'project' or not parts or parts[0] != 'reusableblockslp':
|
||||
return True
|
||||
return ((len(parts) < 2 or parts[1] == self.block_name.casefold())
|
||||
and (len(parts) < 3 or parts[2] in ('work', 'cert')))
|
||||
|
||||
def _runtime_path(self, parts, *, directory=False):
|
||||
"""Recognize runtime locations only in their structural context.
|
||||
|
||||
Generic names such as ``Logs`` are runtime when they are a project-root
|
||||
or selected Work/Cert child. The same name deeper in an engineering
|
||||
partition (or in shared SymbolLibs data) remains primary.
|
||||
"""
|
||||
directories = parts if directory else parts[:-1]
|
||||
if not directory and (Path(parts[-1]).suffix in _NONPRIMARY_SUFFIXES
|
||||
or parts[-1] == 'saddress.adr' or parts[-1].startswith('~$')):
|
||||
return True
|
||||
if self.kind == 'project':
|
||||
return _project_runtime(directories)
|
||||
if directories[:1] == ('reusableblockslp',):
|
||||
# Identity and Work/Cert role components never act as runtime names.
|
||||
if (len(directories) >= 3 and directories[1] == self.block_name.casefold()
|
||||
and directories[2] in ('work', 'cert')):
|
||||
return _project_runtime(directories[3:])
|
||||
return False
|
||||
if _root_runtime(directories):
|
||||
return True
|
||||
# Shared dependency names/partitions are engineering data. Only native
|
||||
# server runtime subtrees within known DB positions are excluded.
|
||||
if directories[:1] == ('symbollibs',) and len(directories) >= 2:
|
||||
return _native_runtime(directories, directories[:2])
|
||||
return bool(directories and _native_runtime(directories, directories[:1]))
|
||||
|
||||
def classify(self, relative):
|
||||
"""Return a semantic scope label or None; paths are relative to the root."""
|
||||
parts = tuple(part.casefold() for part in Path(relative).parts)
|
||||
if (any(part in _NONPRIMARY_DIRECTORIES or part.startswith('work.before-restore-')
|
||||
for part in parts[:-1])
|
||||
or Path(parts[-1]).suffix in _NONPRIMARY_SUFFIXES
|
||||
or parts[-1] == 'saddress.adr' or parts[-1].startswith('~$')):
|
||||
if self._runtime_path(parts):
|
||||
return None
|
||||
if self.kind == 'project':
|
||||
return 'project'
|
||||
|
||||
@@ -280,3 +280,49 @@ def test_source_address_record_is_rechecked_after_validation(recovery_files):
|
||||
assert 'ownership' in result.stderr
|
||||
assert address.read_bytes() == b'live owner'
|
||||
assert (original.parent/'database/icdb.dat').read_bytes() == b'damaged'
|
||||
|
||||
|
||||
@pytest.mark.parametrize('name', ['Logs', 'tEmP', 'bAcKuP', 'Output'])
|
||||
@pytest.mark.parametrize('role', ['Work', 'Cert'])
|
||||
def test_cbb_names_matching_runtime_labels_do_not_hide_target_conflicts(tmp_path, monkeypatch, name, role):
|
||||
from cbb_refresh_verify.backup_recovery import recover_from_backup
|
||||
monkeypatch.setattr(Path, 'home', lambda: tmp_path/'profile')
|
||||
library = tmp_path/'target/target.lmc'
|
||||
original = project(library.parent/f'ReusableBlocksLP/{name}/Work', name, b'damaged')
|
||||
library.write_bytes(b'library')
|
||||
source = project(tmp_path/'backup/Work', name, b'healthy')
|
||||
request = VerifyRequest(library, name, tmp_path/'evidence')
|
||||
conflict = original.parent.parent/role/'database/icdb.dat'
|
||||
conflict.parent.mkdir(parents=True, exist_ok=True)
|
||||
if role == 'Cert':
|
||||
conflict.write_bytes(b'cert before')
|
||||
calls = []
|
||||
def verify(*args, **kwargs):
|
||||
calls.append(args)
|
||||
conflict.write_bytes(b'concurrent target edit')
|
||||
return VerifyResult('PASS', library_flag='GREEN')
|
||||
result = recover_from_backup(request, source, verifier=verify, idle_check=lambda *_: None)
|
||||
assert result.status == 'BLOCKED'
|
||||
assert len(calls) == 1
|
||||
assert conflict.read_bytes() == b'concurrent target edit'
|
||||
assert not list(original.parent.parent.glob('Work.before-restore-*'))
|
||||
|
||||
|
||||
@pytest.mark.parametrize('relative', ['SymbolLibs/temp/icdb.dat', 'SymbolLibs/Temp_a/device.sym',
|
||||
'SharedDependency/hdl/Logs/design.v'])
|
||||
def test_shared_partition_engineering_conflicts_block_install(recovery_files, relative):
|
||||
from cbb_refresh_verify.backup_recovery import recover_from_backup
|
||||
request, original, source = recovery_files
|
||||
conflict = request.lmc_file.parent/relative
|
||||
conflict.parent.mkdir(parents=True, exist_ok=True)
|
||||
conflict.write_bytes(b'shared before')
|
||||
calls = []
|
||||
def verify(*args, **kwargs):
|
||||
calls.append(args)
|
||||
conflict.write_bytes(b'concurrent shared edit')
|
||||
return VerifyResult('PASS', library_flag='GREEN')
|
||||
result = recover_from_backup(request, source, verifier=verify, idle_check=lambda *_: None)
|
||||
assert result.status == 'BLOCKED'
|
||||
assert len(calls) == 1
|
||||
assert (original.parent/'database/icdb.dat').read_bytes() == b'damaged'
|
||||
assert conflict.read_bytes() == b'concurrent shared edit'
|
||||
@@ -213,3 +213,123 @@ def test_only_primary_paths_are_read_for_sha256_and_manifest_is_relocatable(tmp_
|
||||
assert recovery._manifest(source, scope=scope) == manifest
|
||||
assert recovery._manifest(destination, scope=scope) == manifest
|
||||
assert {p.relative_to(destination).as_posix() for p in destination.rglob('*') if p.is_file()} == primary | excluded
|
||||
|
||||
|
||||
@pytest.mark.parametrize('block', ['Logs', 'lOgS', 'Temp', 'tEmP', 'Backup', 'bAcKuP', 'Output', 'rEpOrT',
|
||||
'Log Files', 'ProjectBackup', 'Work.before-restore-engineering'])
|
||||
@pytest.mark.parametrize('role', ['Work', 'Cert'])
|
||||
def test_structural_cbb_identity_is_primary_in_semantic_and_copy_manifests(tmp_path, block, role):
|
||||
from cbb_refresh_verify.scoped_manifest import ManifestScope
|
||||
scope = ManifestScope.library(block, 'target.lmc')
|
||||
source, destination = tmp_path/'source', tmp_path/'destination'
|
||||
relative = f'ReusableBlocksLP/{block}/{role}/database/icdb.dat'
|
||||
primary = put(source, relative)
|
||||
runtime = f'ReusableBlocksLP/{block}/{role}/logs/session.txt'
|
||||
put(source, runtime)
|
||||
before = recovery._copy_verified(source, destination, scope=scope)
|
||||
assert set(before['files']) == {relative}
|
||||
assert (destination/runtime).read_bytes() == b'engineering'
|
||||
assert recovery._manifest(source, scope=scope) == before
|
||||
primary.write_bytes(b'concurrent primary edit')
|
||||
assert recovery._manifest(source, scope=scope) != before
|
||||
|
||||
|
||||
@pytest.mark.parametrize('relative', [
|
||||
'SymbolLibs/temp/icdb.dat', 'SymbolLibs/Temp_a/device.sym',
|
||||
'SymbolLibs/Logs/device.sym', 'SymbolLibs/BACKUP/device.sym',
|
||||
'SymbolLibs/temp/cdbsvr/persistent.bin',
|
||||
'ReusableBlocksLP/A/Work/hdl/Logs/design.v',
|
||||
'ReusableBlocksLP/A/Cert/hdl/bAcKuP/design.v',
|
||||
'SymbolLibs/temp/hdl/Backup/design.v', 'SharedDependency/hdl/Logs/design.v',
|
||||
])
|
||||
def test_library_engineering_partition_names_are_not_runtime_categories(tmp_path, relative):
|
||||
from cbb_refresh_verify.scoped_manifest import ManifestScope
|
||||
scope = ManifestScope.library('A', 'target.lmc')
|
||||
source, destination = tmp_path/'source', tmp_path/'destination'
|
||||
primary = put(source, relative)
|
||||
before = recovery._copy_verified(source, destination, scope=scope)
|
||||
assert relative in before['files']
|
||||
assert recovery._manifest(source, scope=scope) == before
|
||||
primary.write_bytes(b'concurrent shared/design edit')
|
||||
assert recovery._manifest(source, scope=scope) != before
|
||||
|
||||
|
||||
@pytest.mark.parametrize('relative', ['hdl/Logs/design.v', 'hdl/Backup/design.v', 'symbols/Temp/device.sym',
|
||||
'hdl/Log Files/design.v', 'hdl/ProjectBackup/backups/design.v',
|
||||
'hdl/Work.before-restore-engineering/design.v', 'database/cdbsvr/persistent.bin'])
|
||||
def test_nested_project_engineering_names_remain_primary(tmp_path, relative):
|
||||
primary = put(tmp_path, relative)
|
||||
before = recovery._manifest(tmp_path)
|
||||
assert relative in before['files']
|
||||
primary.unlink()
|
||||
assert recovery._manifest(tmp_path) != before
|
||||
|
||||
|
||||
@pytest.mark.parametrize('relative', [
|
||||
'ReusableBlocksLP/Logs/Work/Logs.prj',
|
||||
'ReusableBlocksLP/Backup/Cert/Backup.prj',
|
||||
'SymbolLibs/temp/device.sym', 'SharedDependency/hdl/Logs/design.v',
|
||||
])
|
||||
@pytest.mark.parametrize('side', ['source', 'destination'])
|
||||
def test_colliding_engineering_names_cannot_bypass_primary_copy_byte_checks(tmp_path, monkeypatch, relative, side):
|
||||
from cbb_refresh_verify.scoped_manifest import ManifestScope
|
||||
block = relative.split('/')[1] if relative.startswith('ReusableBlocksLP/') else 'A'
|
||||
scope = ManifestScope.library(block, 'target.lmc')
|
||||
source, destination = tmp_path/'source', tmp_path/'destination'
|
||||
put(source, relative)
|
||||
real_copy = shutil.copy2
|
||||
def copy_then_corrupt(src, dst, *args, **kwargs):
|
||||
result = real_copy(src, dst, *args, **kwargs)
|
||||
corrupt = Path(src) if side == 'source' else Path(dst)
|
||||
metadata = corrupt.stat()
|
||||
corrupt.write_bytes(b'X' * metadata.st_size)
|
||||
os.utime(corrupt, ns=(metadata.st_atime_ns, metadata.st_mtime_ns))
|
||||
return result
|
||||
monkeypatch.setattr(shutil, 'copy2', copy_then_corrupt)
|
||||
with pytest.raises(OSError):
|
||||
recovery._copy_verified(source, destination, scope=scope)
|
||||
assert (destination/relative).exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize('prefix', ['', 'ReusableBlocksLP/Logs/Work/', 'ReusableBlocksLP/Temp/Cert/'])
|
||||
@pytest.mark.parametrize('relative', [
|
||||
'Log Files/ProjectBackupLog.txt', 'ProjectBackup/backups/old.zip',
|
||||
'database/cdbsvr/log/session/text.txt', 'database/cdbsvr/dbcheck/session/text.txt',
|
||||
'PCB/database/cdbsvr/dbrestore/session/text.txt',
|
||||
])
|
||||
def test_known_native_runtime_and_history_locations_are_ignored_contextually(tmp_path, prefix, relative):
|
||||
from cbb_refresh_verify.scoped_manifest import ManifestScope
|
||||
scope = ManifestScope.library(prefix.split('/')[1], 'target.lmc') if prefix else None
|
||||
runtime = put(tmp_path, prefix + relative)
|
||||
before = recovery._manifest(tmp_path, scope=scope)
|
||||
assert not before['files']
|
||||
runtime.write_bytes(b'updated runtime')
|
||||
assert recovery._manifest(tmp_path, scope=scope) == before
|
||||
|
||||
|
||||
@pytest.mark.parametrize('relative', ['SymbolLibs/temp/icdb.dat',
|
||||
'ReusableBlocksLP/Logs/Work/database/icdb.dat'])
|
||||
def test_locked_critical_bytes_in_colliding_names_are_checked_independently(tmp_path, relative):
|
||||
from cbb_refresh_verify.scoped_manifest import ManifestScope
|
||||
source, destination = tmp_path/'source', tmp_path/'destination'
|
||||
put(source, relative)
|
||||
def corrupt_after_copy(event):
|
||||
if event['copied_files'] == event['total_files']:
|
||||
copied = destination/relative
|
||||
metadata = copied.stat()
|
||||
copied.write_bytes(b'X' * metadata.st_size)
|
||||
os.utime(copied, ns=(metadata.st_atime_ns, metadata.st_mtime_ns))
|
||||
with pytest.raises(OSError):
|
||||
recovery._copy_verified(source, destination,
|
||||
scope=ManifestScope.library('Logs', 'target.lmc'), progress=corrupt_after_copy)
|
||||
assert (source/relative).read_bytes() == b'engineering'
|
||||
assert (destination/relative).read_bytes() == b'XXXXXXXXXXX'
|
||||
|
||||
|
||||
@pytest.mark.parametrize('relative', ['SymbolLibs/temp/cdbsvr/log/session/text.txt',
|
||||
'SymbolLibs/Temp_a/cdbsvr/dbcheck/session/text.txt',
|
||||
'CellDB/cdbsvr/dbrestore/session/text.txt'])
|
||||
def test_shared_native_runtime_has_explicit_database_context(tmp_path, relative):
|
||||
from cbb_refresh_verify.scoped_manifest import ManifestScope
|
||||
put(tmp_path, relative)
|
||||
assert not recovery._manifest(tmp_path, scope=ManifestScope.library('A', 'target.lmc'))['files']
|
||||
Reference in new issue
Block a user